app.getSelfPath not working on put handlers

This is a fix for CerboGX environments (Venus OS from Victron) : app.getSelfPath() always returns 'undefined' due to the definition of the handlers, outside of the main plugin function ('app' is not defined at this time).

Sorry, I don't quite understand:

how is this related to Cerbo/Venus? If there's a bug it should affect all platforms, there's nothing Venus OS specific here
even with the require calls in the module init all the modules are initialised with the app that the plugin gets on construction and they reference the app in closure when calling app.getSelfPath. None of this code would have ever worked if the problem is what you describe in the PR description

Can you elaborate?

The "related to CerboGX" is only suspicion because I didn't test elsewhere (I was on my boat with only an iPad to develop).
All I know is that the signalk-autopilot addon doesn't work on my CerboGX (it is up-to-date, but this is the version 1.46.3 of Signal-K server which is installed in Venus OS). It is correctly displaying autopilot state and heading information, but refuse to change the heading (-/+ 1 or 10) with the error "Autopilot not in auto or wind mode".
After some tests, I've found that app.getSelfPath() always returns "undefined" when called from put handlers, but works well from the main plugin function. Then, moving these lines inside the main plugin function solved this problem. I can't explain it (my nodejs skills aren't high enough) but I can suppose it is related to the context where the handler functions are defined or imported.
And as the bug has never been reported by someone else, I suppose it is related to the Venus OS environment (maybe the nodejs version and/or the signal-k version).
Anyway, this discovery saved my day and now I can pilot my boat from the bow with my phone ;-)


Support for alternative runtimes

Docker recently started supporting alternative runtimes with the --runtime flag.
This allows users to use alternative runtimes to runc such as Kata (which I'd like to use on some pods due to additional benefits of kernel isolation on external facing applications).
Additionally, cri-dockerd support was just merged into K3s, allowing users to continue to use Docker post upgrade to Kubernetes 1.24 and above.
Unfortunately, those who choose to keep using Docker with Kubernetes will be constrained to the lack of alternative runtime support meaning native Kubernetes support in form of RuntimeClasses is made redundant as the CRI layer is a bottleneck.
I think adding support for passing through the requested runtime should not be too difficult so I was wondering if it's something that could be implemented?

/label enhancement

need --runtime flag support

Hi,
I'd like to start this discussion again, for my own purpose I solved this problem like this: https://github.com/KWasm/cri-dockerd/pull/2
There are two problems, in this example I use the containerd-shim-spin:

The RunPodSandbox call contains the runtimeClassName e.g. spin but Docker needs io.containerd.spin.v1. That value is then passed to containerd which is looking for a binary called containerd-shim-spin-v1.
The second problem is that the RunPodSandbox call contains the runtimeClassName but starts a pause container which should be handled by runc, but the CreateContainer call does not contain the runtimeClassName but it is needed here to create the container with the right runtime.

As a solution to the first problem I introduced the --runtime-handler flag which is a map[string]string and is used to define the mapping at startup e.g. --runtime-handler spin=io.containerd.soin.v1
For the second problem, I had no better idea than to store the runtime handler as a label of the pause container which represents the pod. May someone has a better idea.
What do you think?

@evol262 This is a bit more complicated for two reasons:

daemon.json runtime config only concerns itself with the concrete runtime invoked by the containerd runc shim. It more-or-less is the path to runc (or crun).
The only way to set a non-runc-shim runtime is to use --runtime in the individual container creation -- we special-case containerd shim names.

In short the config for setting a default runtime is currently a bit busted as the code is so brittle that it's blocked on a major refactor of config-handling. So for now, --runtime is the way to do this, unless you want to change the path of runc/use a drop-in replacement like crun.

Thanks, @evol262!
I just realized that I was confused by the term "alternative runtimes", it means other OCI compliant runtimes than runc, like crun, youki, or gvisor. Right? And they can be configured in daemon.json and passed by docker run --runtime <name_configured_in_daemon.json>.
What I am working on is using alternative containerd shims like described in https://docs.docker.com/desktop/wasm/. And alternative containerd shims are also passed via docker run --runtime <io,containerd.ShimNane.Version>.
Thanks @neersighted, would it be an option to pass the runtimeClassName as it is and we change the daemon.json so that it is possible to add containerd shims as runtime like:
 "runtimes": {
    "spin": {
      "shim": "io.containerd.spin.v1"
    },
 ...

That would address @evol262 concern that docker should be the leading system to manage the runtimes (and shims).

I think it would make sense to pass values through without any opinion in cri-dockerd. I just explained why daemon.json is not an option right now -- that will take some time, upstream, to fix.
I also want to push back on your config snippet -- shim is not the right level of abstraction, as to containerd, the shim is the runtime. Instead we've leaked the io.containerd.runc.v2 implementation details (runc, crun, other compatible implementations) into Moby, which was a mistake caused by the migration from the v1 to v2 runtime APIs.
In short, no opinions in cri-dockerd makes sense, and the work on making alternate runtimes easier to use continues upstream, though most of the people involved in making it happen are working on other issues right now. I can bring it up in today's maintainer's call to see where people are at on returning to it.

Understood, I can see that there is no way to solve this problem in cri-containerd.
For my purpose, a hacky solution is good enough, so I have no need to push further on this topic.
Thanks for the discussion 😊

1. Docker does not "need" that particular formatting. containerd-compatible runtimes do, but they are two separate things, which is part of the reason hat this was not initially addressed. The user stories need some scoping. When you did this, did you also configure it in daemon.json, or pass a container-compatible runtime blindly?

Blindling passing a containerd-compatible runtime name is the correct way to use it. Any shimv2 runtime on containerd's PATH can be used with Docker without any daemon.json configuration. See https://github.com/moby/moby/pull/43800 for more details. From the Engine API client's perspective, there is no meaningful distinction between shimv2 runtimes and configured-in-daemon.json runtimes.

2. in general, the "normal" runtime is almost certainly the "right" one for a container as do-nothing as pause

I disagree. Mixing runtimes for containers in a pod could break Kata and other runtimes which aren't namespace-based. Kata, for instance, runs all containers in a pod inside a single VM and likely wouldn't take too kindly to being asked to start a container in a sandbox it didn't also create.

In short the config for setting a default runtime is currently a bit busted as the code is so brittle that it's blocked on a major refactor of config-handling. So for now, --runtime is the way to do this, unless you want to change the path of runc/use a drop-in replacement like crun.

No, shimv2 runtimes can be set as the default. https://github.com/moby/moby/pull/43978

would it be an option to pass the ~runtimeClassName~ RuntimeClass handler as it is ~and we change the daemon.json~ so that it is possible to add containerd shims as runtime

FTFY. Kubernetes RuntimeClass objects are already a mapping from names to runtimes (runtimeClassName -> handler) and the Docker daemon has a mapping from runtimes to configs (explicit daemon.json config for legacy, implicit identity mappings for now for shimv2) so there is no need to put another layer of mapping names to runtimes into cri-dockerd.

Looks like I missed the follow-up PR, thanks for clarifying @corhere!

Blindly passing a containerd-compatible runtime name is the correct way to use it for moby. cri-dockerd can take it as a given that the administrator has hopefully set up the right node annotations so the scheduler will place it on somewhere that it's configured, but the feedback loop around failing to schedule/start when blindly passing is terrible, and a richer way to do it is much better.

From a practical POV, the kind of validation in your PR doesn't really belong in cri-dockerd. Installation/management of alternative runtimes (either the actual binary or the configuration of /etc/docker/daemon.json) is an administrator concern, and passing it off to Docker and letting it try to spawn a container is a more coherent workflow than trying to fail slightly sooner unless the right set of options is specified in a second location (no matter what cri-dockerd thinks, if docker itself doesn't know about the alternative runtime, it's still going to fail there).

So should cri-dockerd validate the name of the runtime it passes to the daemon, or not?

It's not mixing runtimes for containers.

What were you suggesting, then? I read it as "don't bother starting the pause container with the pod's configured runtime; using the daemon's default runtime to start pause containers is always fine, even when the workload containers are to be started with a different runtime." Using a different runtime to start the sandbox container than the pod's application containers would lead to broken behaviour with some runtimes which use isolation techniques other than Linux namespaces.

The only purpose of the pod-infra-container-image is to create a namespace which the containers can graft onto, which is completely irrelevant with kata anyway, since the isolation is via KVM

That's the only purpose of the pause container with namespace-based runtimes. The image may be irrelevant to Kata, but the act of starting a pause container is necessary for other purposes.

support for Kata with pods is finicky enough anyway, because the container's config needs to be annotated to tell Kata how to handle it regardless -- something which cri-dockerd can't do anyway, since docker/moby handles that passthrough to containerd. It's not in the scope here.

This feature request is titled "Support for alternative runtimes." Kata is an alternative runtime. While it may be the case that extra work above and beyond what is being considered here may be needed before cri-dockerd can be compatible with Kata or some other runtime, I don't see why that would justify making design decisions which would would be actively hostile to allowing cri-dockerd to be compatible with a subset of runtimes in the future.
As an aside, I have been working on adding first-class support for Kata to moby. While cri-dockerd can't ask the daemon to add OCI annotations today, it'd be fairly trivial to add that functionality into the daemon. Kata is already ready for us, too.

The /containers/{}/start request will return an HTTP 500 response if the runtime is unavailable.
< HTTP/1.1 500 Internal Server Error
< Api-Version: 1.43
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Fri, 17 Feb 2023 03:41:04 GMT
< Content-Length: 218
<
{"message":"failed to create task for container: failed to start shim: failed to resolve runtime path: runtime \"io.containerd.foo.bar\" binary not installed \"containerd-shim-foo-bar\": file does not exist: unknown"}

Hypothetical discussions about what could or could not (or should or should not) be done about firecracker, kata, and other container drivers with different isolation models is a waste of time and energy for everyone involved until whatever time the support is actually in the daemon, exposed in the API, and widely-available enough that there's an actual user request for fulfilling it.

docker run --runtime io.containerd.kata.v2 works today with v23.0.1. This thread is the user request. The only thing hypothetical is support from cri-dockerd. Are there any other blockers aside from the annotation?

FTFY. Kubernetes RuntimeClass objects are already a mapping from names to runtimes (runtimeClassName -> handler) and the Docker daemon has a mapping from runtimes to configs (explicit daemon.json config for legacy, implicit identity mappings for now for shimv2) so there is no need to put another layer of mapping names to runtimes into cri-dockerd.

I agree that cri-dockerd should not have a mapping and I just hacked something together for a demo. But in my example I created a Pod runtimeClassName: wasmedge which is mapped to a handler by the following runtime class:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
handler: io.containerd.wasmedge.v1

That leads to the error that the handler is not valid
The RuntimeClass "wasmedge" is invalid: handler: Invalid value: "io.containerd.wasmedge.v1": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')
@corhere
When I can't have io.containerd.wasmedge.v1 as handler value which valid value can I pass to docker as runtime that is explicit or implicit mapped to 'io.containerd.wasmedge.v1'?  (to run this example on Kubernetes)

The daemon runtimes config is surfaced to the client at GET /info.
# curl -s --unix-socket /var/run/docker.sock http://./v1.42/info | jq .Runtimes
{
  "crun": {
    "runtimeType": "io.containerd.runc.v2",
    "options": {
      "BinaryName": "/usr/local/bin/crun"
    }
  },
  "runc": {
    "path": "runc"
  }
}

any update?
I found pod.RuntimeClass not set to non runc, it always be runc
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: nvidia
handler: docker
---

....
    spec:
      runtimeClassName: nvidia

$ docker inspect be4d | grep -i runtime
            "Runtime": "runc",


No way to determine if underlying connection has been killed

In the event that client.quit(); is called I would expected client.status to update to something like disconnecting or closing and then to close once complete. That does not happen.
    console.log(this._redisClient?.status);
    await this._redisClient?.quit();
    console.log(this._redisClient?.status);
    await this._redisClient?.quit();
    console.log(this._redisClient?.status);

    OUTPUT

ready
ready
[ioredis] Unhandled error event: Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:164:15)
    at writeGeneric (node:internal/stream_base_commons:155:3)

The same applies when calling await this._redisClient?.quit(); too.
Incidentally it would be good to get some tsdocs on the statuses so it's obvious what they mean, for example what's the difference between end and close
https://github.com/luin/ioredis/blob/main/lib/Redis.ts#L37-L44

After some investigation it seems the issue is that there is no intermediate state before end. Like reconnecting or connecting there should be an ending status which indicates that the connection is in the process of being closed, up until that point it responds as ready even though it's being killed.

I did this to cleanly close the client with Fastify.
const registerRedis = (server: FastifyInstance) => {
  const redisClient = new Redis(process.env.REDIS_URL || '', { disconnectTimeout: 100 })
  server.decorate('redis', redisClient)

  server.addHook('onClose', async (instance) => {
    await instance.redis.quit()
    await new Promise((resolve) => {
      instance.redis.disconnect()
      setTimeout(() => { resolve(null) }, 100)
    })
  })
}


Is OpenSSL_version_num() deprecated?

The function OpenSSL_version_num() is not marked as deprecated in the header file or in the source code or in libcrypto.num. But in the documentation, it is described in the "BACKWARD COMPATIBILITY" section which says this at the beginning of it:

For compatibility, some older macros and functions are retained or
synthesised.
They are all considered deprecated.

Its also listed as deprecated in the synopsis section of the man page.
The situation is similar with the OPENSSL_VERSION_NUMBER macro, i.e. its not marked as deprecated in the header file, but it is in the docs. This one is additionally marked as deprecated in other.syms.
It seems very unclear whether they really are deprecated or not.

Yes, I'm not really sure why we wanted to deprecate this function. It's actually quite useful to have a single value to represent the whole version number - and it exists in old versions too. In my response to #17516 I wanted to say that the easiest way to compare the header version against the library version is to compare OPENSSL_VERSION_NUMBER against the output of OpenSSL_version_num() but felt unable to because of the ambiguous deprecation status.

I would treat this issue as a documentation bug and simply undeprecate. But perhaps we need to discuss this within OTC?

But perhaps we need to discuss this within OTC?

Yes. I think so

Question for OTC: Should OPENSSL_VERSION_NUMBER and OpenSSL_version_num() be undeprecated in the documentation? Alternatively should they be marked as deprecated in the header files?

Because the bit arrangement in that number has changed over time, resulting in, uhm, pretty creative was to handle it, I thought that the new macros were much easier to interpret.
Deprecating the old was meant to be in the interest of not trying to have two ways to check.

I agree that this should be undeprecated in the documentation - the documentation is wrong.
Warning that this has challenges when we get to a much later major version would have been more appropriate IMHO.

These should be undeprecated.

OTC Decision: the OPENSSL_VERSION_NUMBER macro and OpenSSL_version_num() function should be undeprecated on the 3.0 branch in the documentation


[BUG] Unresponsive picam2.stop_encoder() when using with threads

Please only report one bug per issue!
A live stream program unresponsive at picam2.stop_encoder() (also, picam2.stop_recording())
Describe the bug
I wrote a python program to manage pi camera that it started recording only when there is any client request, and when there is no client left, the pi camera will stop working to reduce CPU usage and temperature.
Unfortunately, after I upgraded from picam2 v.0.3.12 to 0.3.14, the program stop working at picam2.stop_recording(). I tried separated the command into two part and discovered that the problem is with the line picam2.stop_encoder()
To Reproduce
This is the part of program that manage start/stop recording pi camera.
def generate():
    try:
        if output is None:
            output = StreamingOutput()

        with output.condition:
            active_sessions += 1
            if active_sessions == 1:
                picam2.start_recording(MJPEGEncoder(), FileOutput(output))

        while True:
            try:
                with output.condition:
                    output.condition.wait(1)
                    if output.frame is not None:
                        frame = output.frame
                    else:
                        continue
                yield (b'--FRAME\r\n'
                       b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
            except KeyboardInterrupt:
                break

    except Exception as e:
        syslog.syslog(syslog.LOG_WARNING, f'Removed streaming client: {str(e)}')

    finally:
        with output.condition:
            output.in_use = False
            active_sessions -= 1

            if active_sessions == 0:
                try:
                    output = None
                    picam2.stop()
                    picam2.stop_encoder()
                except Exception as e:
                    print(e)

Expected behaviour
picam2.stop_encoder() stop encoding properly and run the next command
Console Output, Screenshots
N/A
Hardware :
Raspberry pi Zero with Pi Camera V.2
Additional context
My Raspberry Pi runs with Raspberry Pi OS Bookworm

I wonder if trying to stop the encoders while holding the output.condition is causing a deadlock. Because they might still be trying to flush out the last frames, and won't be able to get that lock. It certainly feels like something to avoid. Maybe try moving picam2.stop() and picam2.stop_encoder() outside with with output.condition.

I wonder if trying to stop the encoders while holding the output.condition is causing a deadlock. Because they might still be trying to flush out the last frames, and won't be able to get that lock. It certainly feels like something to avoid. Maybe try moving picam2.stop() and picam2.stop_encoder() outside with with output.condition.

I tried revising the code by your suggestion and the result seems better. Now for one client session, the camera stream can start and stop camera properly. But for two or more clients access the streaming concurrently, After all sessions were closed, a new client cannot access anymore. Try debugging and see that the program stop at picam2.stop_encoder().
Below is my code revision, also added the StreamingOutput class. I am not good at threading (and threading.condition) usage. I wondered if it is my improper programming, or the library. However, with picamera2 0.3.12, the code runs with multiple client sessions properly.
class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = threading.Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

def generate():
    global picam2, output, active_sessions

    try:
        if output is None:
            output = StreamingOutput()

        with output.condition:
            active_sessions += 1
            if active_sessions == 1:
                picam2.start_recording(MJPEGEncoder(), FileOutput(output))
        print(active_sessions)
        while True:
            try:
                with output.condition:
                    output.condition.wait()
                    if output.frame is not None:
                        frame = output.frame
                    else:
                        continue
                yield (b'--FRAME\r\n'
                       b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n')
            except KeyboardInterrupt:
                break

    except Exception as e:
        syslog.syslog(syslog.LOG_WARNING, f'Removed streaming client: {str(e)}')

    finally:
        active_sessions -= 1
        if active_sessions == 0:
            output = None
            picam2.stop_encoder()
            picam2.stop()

I think this issue relates to MJPEGEncoder's parent Class. v4l2_encoder.py. that the latest commit address the issue of preventing dropping final frames.

As you say, the thing that's changed is that when you stop an encoder, it flushes any pending output (this was reported as a bug previously because it had received and encoded frames which were then lost).
It's certainly true that when when you take a lock inside an API using some kind of callback (as you are in StreamingOutput.write), then you probably shouldn't hold the same lock when you call into the API. It's quite common for libraries to behave like this, and that's what was happening previously.
I can only debug really code the I'm sent if it's completely self-contained (I can just copy/paste and it works), preferably avoids using other packages that I don't have or don't understand, and are absolutely as short as you can make them. On this occasion, I've tried to turn your example into something I can run:
import io
import time
from threading import Condition, Event, Lock, Thread

from picamera2 import Picamera2
from picamera2.encoders import MJPEGEncoder
from picamera2.outputs import FileOutput

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

output = StreamingOutput()
picam2 = Picamera2()
lock = Lock()
active_sessions = 0

def generate(abort):
    global output, picam2, active_sessions

    with lock:
        active_sessions += 1
        print("active_sessions", active_sessions)
        if active_sessions == 1:
            print("Starting")
            picam2.start_recording(MJPEGEncoder(), FileOutput(output))

    while True:
        with output.condition:
            output.condition.wait(1)
            if output.frame:
                yield()

        if abort.is_set():
            break

    with lock:
        active_sessions -= 1
        print("active_sessions", active_sessions)
        if active_sessions == 0:
            print("Stopping")
            picam2.stop()
            picam2.stop_encoder()

def thread_func(abort):
    for _ in generate(abort):
        pass

aborts = [Event() for _ in range(5)]
threads = [Thread(target=thread_func, args=(abort, )) for abort in aborts]

for thread in threads:
    time.sleep(1)
    thread.start()

for i, abort in enumerate(aborts):
    time.sleep(1)
    abort.set()
    threads[i].join()

If your application is still seeing deadlocks, maybe put a print statement just before each attempt to grab each lock, another when you acquire the lock, and a final print every time you release it.

Thank you for your advice. Your code help me a lot. It solved problem of RuntimeError and BrokenPipError that I am researching for a solution. I will learn more on Python thread programming :)
However, I tried running your code

As you say, the thing that's changed is that when you stop an encoder, it flushes any pending output (this was reported as a bug previously because it had received and encoded frames which were then lost).
It's certainly true that when when you take a lock inside an API using some kind of callback (as you are in StreamingOutput.write), then you probably shouldn't hold the same lock when you call into the API. It's quite common for libraries to behave like this, and that's what was happening previously.
I can only debug really code the I'm sent if it's completely self-contained (I can just copy/paste and it works), preferably avoids using other packages that I don't have or don't understand, and are absolutely as short as you can make them. On this occasion, I've tried to turn your example into something I can run:
import io
import time
from threading import Condition, Event, Lock, Thread

from picamera2 import Picamera2
from picamera2.encoders import MJPEGEncoder
from picamera2.outputs import FileOutput

class StreamingOutput(io.BufferedIOBase):
    def __init__(self):
        self.frame = None
        self.condition = Condition()

    def write(self, buf):
        with self.condition:
            self.frame = buf
            self.condition.notify_all()

output = StreamingOutput()
picam2 = Picamera2()
lock = Lock()
active_sessions = 0

def generate(abort):
    global output, picam2, active_sessions

    with lock:
        active_sessions += 1
        print("active_sessions", active_sessions)
        if active_sessions == 1:
            print("Starting")
            picam2.start_recording(MJPEGEncoder(), FileOutput(output))

    while True:
        with output.condition:
            output.condition.wait(1)
            if output.frame:
                yield()

        if abort.is_set():
            break

    with lock:
        active_sessions -= 1
        print("active_sessions", active_sessions)
        if active_sessions == 0:
            print("Stopping")
            picam2.stop()
            picam2.stop_encoder()

def thread_func(abort):
    for _ in generate(abort):
        pass

aborts = [Event() for _ in range(5)]
threads = [Thread(target=thread_func, args=(abort, )) for abort in aborts]

for thread in threads:
    time.sleep(1)
    thread.start()

for i, abort in enumerate(aborts):
    time.sleep(1)
    abort.set()
    threads[i].join()

If your application is still seeing deadlocks, maybe put a print statement just before each attempt to grab each lock, another when you acquire the lock, and a final print every time you release it.

Thank you for your advice. Your code help me a lot. It solved the problems of RuntimeError and BrokenPipError that I am researching for solution. I will learn more on Python thread programming :)
I tried running your code without modification. On Picamera2 v 0.3.12, the program run and exit completely without error. However, when I run on Picamera2 v 0.3.14, the program printed and stop at Stopping and wait for something forever.
$ python3 test.py
[0:21:53.035673678] [2559]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+99-4a23664b
[0:21:53.263820789] [2563]  WARN RPiSdn sdn.cpp:39 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:21:53.281247645] [2563]  WARN RPI vc4.cpp:390 Mismatch between Unicam and CamHelper for embedded data usage!
[0:21:53.287279595] [2563]  INFO RPI vc4.cpp:444 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[0:21:53.289452577] [2563]  INFO RPI pipeline_base.cpp:1142 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[0:21:53.307860424] [2559]  INFO Camera camera_manager.cpp:284 libcamera v0.1.0+99-4a23664b
[0:21:53.512487730] [2566]  WARN RPiSdn sdn.cpp:39 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:21:53.529040593] [2566]  WARN RPI vc4.cpp:390 Mismatch between Unicam and CamHelper for embedded data usage!
[0:21:53.536173534] [2566]  INFO RPI vc4.cpp:444 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
[0:21:53.538350516] [2566]  INFO RPI pipeline_base.cpp:1142 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
active_sessions 1
Starting
[0:21:54.625777525] [2570]  INFO Camera camera.cpp:1181 configuring streams: (0) 1280x720-XBGR8888 (1) 1920x1080-SBGGR10_CSI2P
[0:21:54.630337488] [2566]  INFO RPI vc4.cpp:608 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
active_sessions 2
active_sessions 3
active_sessions 4
active_sessions 5
active_sessions 4
active_sessions 3
active_sessions 2
active_sessions 1
active_sessions 0
Stopping

I modified the code by your suggestion:
    with lock:
        active_sessions -= 1
        print("active_sessions", active_sessions)
        if active_sessions == 0:
            print("Stopping")
            picam2.stop()
            print("After stop")
            picam2.stop_encoder()
            print("After stop_encoder")

def thread_func(abort):
    for _ in generate(abort):
        pass

aborts = [Event() for _ in range(5)]
threads = [Thread(target=thread_func, args=(abort, )) for abort in aborts]

for thread in threads:
    time.sleep(1)
    thread.start()

for i, abort in enumerate(aborts):
    time.sleep(1)
    abort.set()
    threads[i].join()

print("After thread.join")

with picamera2 v0.3.14 (installed by apt) the result are:
active_sessions 2
active_sessions 3
active_sessions 4
active_sessions 5
active_sessions 4
active_sessions 3
active_sessions 2
active_sessions 1
active_sessions 0
Stopping
After stop

and definitely, with picamera v0.3.12 (installed by pip):
active_sessions 2
active_sessions 3
active_sessions 4
active_sessions 5
active_sessions 4
active_sessions 3
active_sessions 2
active_sessions 1
active_sessions 0
Stopping
After stop
After stop_encoder
After thread.join
$

Certainly, I will help test this issue!

Today, I just upgraded the raspi-firmware to version (1:1.20231024+ds-1+rpt1, 1:1.20231024+ds-1+rpt2). After the upgrade, I reinstalled picamera2 v0.3.14 and conducted a test. Fortunately, the problem has been resolved. Your analysis appears to be accurate based on this evidence.
I will continue to run tests to see if any incidents occur.

Hmm, I not entirely convinced. I installed the latest image and firmware yesterday and found that the problem comes and goes a bit. My inclination is to include the workaround as there's not much to it, and it will have no effect when things work properly.  See here: https://github.com/raspberrypi/picamera2/pull/864.

I saw similar behavior in my photobooth code. It was a bit random, more likely on a rpi 3b+ to happen, less likely on the pi 4.
unfortunately I could not reduce it to a code to confirm, so I hope this workaround proposed will solve my case also 👍


bug: when user asks for route by name '+foo-route' do not create '++f…

…oo-route'

Just a few nits.

LGTM.


Embedding added for fb and twitter

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fixes
Test URLs:

Original: https://www.sunstar.com

Before: https://main--sunstar--hlxsites.hlx.live/_drafts/piyush/twitter-and-facebook-embed

After: https://fb-plugin--sunstar--hlxsites.hlx.live/_drafts/piyush/twitter-and-facebook-embed

Original: https://www.sunstar.com/healthy-thinking/how-to-develop-a-healthy-habit

Before: https://main--sunstar--hlxsites.hlx.live/healthy-thinking/how-to-develop-a-healthy-habit

After: https://fb-plugin--sunstar--hlxsites.hlx.live/healthy-thinking/how-to-develop-a-healthy-habit

[ ]  If you are adding a new block or a variation to an existing block, please fill below:
Block library path: https://--sunstar--hlxsites.hlx.page/tools/sidekick/library.html?plugin=blocks&path=/sidekick/blocks/&index=0

@ehrro Facebook plugin is broken for me on the following page. Can you please specify what all permissions we need to give to user centrics.
https://www.sunstar.com/healthy-thinking/category/holistic-health-care/
Also for https://www.sunstar.com/healthy-thinking/how-to-develop-a-healthy-habit/
Instagram plugin is broken


[bug] (valgrind findings) Uninitialized memory use

🐛 BUG REPORT

Ran a quick valgrind on the xcash-dpops testnet branch. It popped up a few uninitialized memory findings I thought I'd share here.
Mostly just issues such as an attempt to delete a poll with this uninitialized epoll_ctl structure here,  here, here, here:
==856572== Syscall param epoll_ctl(event) points to uninitialised byte(s)
==856572==    at 0x4AD2B2E: epoll_ctl (syscall-template.S:78)
==856572==    by 0x26733F: new_socket_thread (server_functions.c:211)
==856572==    by 0x26A7A1: socket_receive_data_thread (server_functions.c:923)
==856572==    by 0x497B608: start_thread (pthread_create.c:477)
==856572==    by 0x4AD2162: clone (clone.S:95)
And this socket close call on an unitialized structure from here, here:
==856572== Syscall param close(fd) contains uninitialised byte(s)
==856572==    at 0x49863FB: close (close.c:27)
==856572==    by 0x25E9F4: get_delegates_online_status (block_verifiers_update_functions.c:1300)
==856572==    by 0x224C2E: start_new_round (block_verifiers_functions.c:115)
==856572==    by 0x24791E: current_block_height_timer_thread (block_verifiers_thread_server_functions.c:186)
==856572==    by 0x497B608: start_thread (pthread_create.c:477)
==856572==    by 0x4AD2162: clone (clone.S:95)
And a couple missed stack buffer initializations here (missed the 'data' initialization) and here (missed the 'data3' initialization):
==856572== Conditional jump or move depends on uninitialised value(s)
==856572==    at 0x25E555: get_delegates_online_status (block_verifiers_update_functions.c:1238)
==856572==    by 0x224C2E: start_new_round (block_verifiers_functions.c:115)
==856572==    by 0x24791E: current_block_height_timer_thread (block_verifiers_thread_server_functions.c:186)
==856572==    by 0x497B608: start_thread (pthread_create.c:477)
==856572==    by 0x4AD2162: clone (clone.S:95)
And here and here:
==900800== Conditional jump or move depends on uninitialised value(s)
==900800==    at 0x483EFA9: __strlen_sse2 (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==900800==    by 0x1E6739: parse_json_data (string_functions.c:57)
==900800==    by 0x22418E: server_receive_data_socket_node_to_node (block_verifiers_server_functions.c:415)
==900800==    by 0x269E2E: socket_thread (server_functions.c:856)
==900800==    by 0x26A923: socket_receive_data_thread (server_functions.c:932)
==900800==    by 0x497B608: start_thread (pthread_create.c:477)
==900800==    by 0x4AD2162: clone (clone.S:95)
And here:
==901150== Conditional jump or move depends on uninitialised value(s)
==901150==    at 0x483EFA9: __strlen_sse2 (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==901150==    by 0x1EA987: get_delegates_total_voters_count (shared_delegate_website_thread_server_functions.c:241)
==901150==    by 0x26B5A6: block_verifiers_add_reserve_proof_check_if_data_is_valid (delegate_server_functions.c:146)
==901150==    by 0x26C008: server_receive_data_socket_node_to_block_verifiers_add_reserve_proof (delegate_server_functions.c:353)
==901150==    by 0x269A32: socket_thread (server_functions.c:753)
==901150==    by 0x26A93C: socket_receive_data_thread (server_functions.c:932)
==901150==    by 0x497B608: start_thread (pthread_create.c:477)
==901150==    by 0x4AD2162: clone (clone.S:95)
Also valgrind doesn't seem to like the individual structure member approach taken here:
  // initialize the reserve_proof struct
  memset(reserve_proof.block_verifier_public_address,0,sizeof(reserve_proof.block_verifier_public_address));
  memset(reserve_proof.public_address_created_reserve_proof,0,sizeof(reserve_proof.public_address_created_reserve_proof));
  memset(reserve_proof.public_address_voted_for,0,sizeof(reserve_proof.public_address_voted_for));
  memset(reserve_proof.reserve_proof_amount,0,sizeof(reserve_proof.reserve_proof_amount));
  memset(reserve_proof.reserve_proof,0,sizeof(reserve_proof.reserve_proof));

I would probably just clear it in one fell swoop:
  memset(&reserve_proof,0,sizeof(reserve_proof));

Expected Behavior
Memory should be initialized before referenced.
Current Behavior
Some missed initializations could potentially result in wrong file handles being closed. Also that one stack buffer could wind up with random data appended to the string. Maybe...
Possible Solution
valgrind seemed happy with a few simple memsets. i.e.:
memset(events,0,sizeof(events));
memset(block_verifiers_send_data_socket,0,sizeof(block_verifiers_send_data_socket));
memset(data,0,sizeof(data));
etc...
I also recommend wrapping the EPOLL_CTL_DEL and the close() calls in a check for a valid socket as such:
   for (count = 0; count < TOTAL_BLOCK_VERIFIERS; count++)
   {
-    epoll_ctl(epoll_fd_copy, EPOLL_CTL_DEL, block_verifiers_send_data_socket[count].socket, &events[count]);
-    close(block_verifiers_send_data_socket[count].socket);
+    if (block_verifiers_send_data_socket[count].socket > 0)
+    {
+      epoll_ctl(epoll_fd_copy, EPOLL_CTL_DEL, block_verifiers_send_data_socket[count].socket, &events[count]);
+      close(block_verifiers_send_data_socket[count].socket);
+    }
   }

Steps to Reproduce (for bugs)
run xcash-dpops under valgrind. I used this in my testnet machine /usr/lib/systemd/system/xcash-dpops.service:
ExecStart=/usr/bin/valgrind --max-stackframe=2100800 --num-callers=20 --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=/tmp/valgrind-dpops-out-%%p-%%n.val /home/xcash_user/xcash-official/xcash-dpops/build/xcash-dpops --block-verifiers-secret-key ***secret***  --shared-delegates-website --fee 12.345678 --minimum-amount 10000 --start-time 122 0 8 16 56

Context

Your Environment

Version used: latest testnet build
Server type and version: Crappy discount Contabo dedicated server :)
Operating System and version: Ubuntu 20.04.4 LTS

Thanks a lot for this @CygnusMiner
I will get to looking at it, and applying fixes or asking questions
do you mind running one on the remote-data branch as well and posting any new findings
this way we can leave this issue open while we switch on the testnet to rd and close it when done
Thanks!

Sure. I'm attaching a full valgrind file (prior to any memory checking since this is an uninitialized memory defect). It does the initialization checking prior to gathering memory leaks (which really clutter up the file) so I can catch it here. I'll attach it compressed.
The initialization warnings start at line 206:
valgrind-remote-data.val.zip
This is a valgrind run on a debug build of a clone of xcash-dpops with a git checkout remote-data using the following flags:
/usr/bin/valgrind --max-stackframe=2100800 --num-callers=20 --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose

I looked over and fixed all but the epoll struct initializes since it does not say that in the documentation is necessary.
I will look at the attached file next

Thanks Zach! The latest clone of the remote-data branch looks good. The only outstanding uninitialized errors are the Syscall param epoll_ctl(event) points to uninitialised byte(s) and Googling that seems to show a consensus that it's a valgrind false positive. I would say this issue has been addressed.

oh does the txt file not show any more?

Not that I can tell. It seemed to show the same findings from the dpops-test branch that I created the first defect against. I believe you've addressed all of the outstanding uninitialized findings thus far.
I can't guarantee that all have been hit, though. It seems valgrind is slowing my dpops down enough that it cant make it past a simple reserve bytes check...
Checking reserve bytes
The database is not synced with the majority of block verifiers, syncing the database from a random block verifier that is in the majority
Syncing the reserve bytes database
Getting the database data from dpops-test-4.xcash.foundation
Could not receive data from dpops-test-4.xcash.foundation
Connecting to another block verifier
Getting the database data from franckiki2.ddns.net
Could not receive data from franckiki2.ddns.net
Connecting to another block verifier
Getting the database data from <IP_ADDRESS>
Could not receive data from <IP_ADDRESS>
Connecting to another block verifier
Getting the database data from testnode-nl.xcash.rocks
Could not receive data from testnode-nl.xcash.rocks
Connecting to another block verifier
Getting the database data from umxnet.ursamajor.eu
Could not receive data from umxnet.ursamajor.eu
Connecting to another block verifier
Getting the database data from <IP_ADDRESS>
Could not receive data from <IP_ADDRESS>
Connecting to another block verifier
etc...

I'm going to have to spend time trying to reduce the overhead. Maybe I'll turn memleak checking off completely...


http middleware

Codecov Report
Base: 86.16% // Head: 86.16% // No change to project coverage :thumbsup:

Coverage data is based on head (6a9fe59) compared to base (250a92f).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #78   +/-   ##
=======================================
  Coverage   86.16%   86.16%
=======================================
  Files          42       42
  Lines        4871     4871
=======================================
  Hits         4197     4197
  Misses        590      590
  Partials       84       84

Flag
Coverage Δ

go_tests
86.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Cron expression error

I have tried to set up cron via drone CLI  weekly with custom time.
But I faced with a problem. When I used same pattern for weekly cron which I used for daily cron. I faced with an error
For daily : 12 14 * * * * is ok
For Weekly 12 14 * * 7  I see an error.
{"message":"Invalid Cronjob Expression"}

Expected
Should be same pattern to each cron or fill additional info in docs

This is expected because the cron library we use requires you to specify seconds:

The cron expression represents a set of times, using 6 space-separated fields, including seconds.

You can learn more and see some examples in the docs:
https://docs.drone.io/cron/#expressions

@bradrydzewski I have tried to run  bash drone cron add gelleson/drone-repo "weekly"  "0 14 14 * * 7" --branch main client error 400: {"message":"Invalid Cronjob Expression"} inspired by docs. But I still see an error

What was wrong?

I found :) Sorry  :)


Reduce instance size to xlarge.

The motivation for the change to use x2large was because of running out of disk space on a few macos builds. We should be ok to move to xlarge. I will keep an eye out for OOMS/disk space issues etc.

@kgpai There are CI failures. Please, take a look.

Abandoning Diff.


Ordinal date formatting for elm-date-extra Format

Needed Ordinal Date formatting for current project, thought other people might like the ability to get Ordinal days. Very simple implementation of the ordinal.
Used %E and %-D as representation for the equivalent space and non padded numbers (%e, %-d).

I added a few tests and i notice that the padded Ordinal Day always adds a space.
The intention of padding was that single digit fields get padding so they align with double digit fields.

I added a test, and adjusted it so it only pads single digits.
I have pushed the changes to the repo with test, but have not published yet as I just though of a complication.
I've been building this with support of languages.
You're addition is English specific at the moment.
I will need to move it to the Config system.
Something like the following for each I18n file for each of the existing languages.
dayOfMonthWithSuffix: Int -> String
dayOfMonthWithSuffix day =
  case day of
    1 -> "1st"
    21 -> "21st"
    2 -> "2nd"
    22 -> "22nd"
    3 -> "3rd"
    23 -> "23rd"
    31 -> "31st"
    _ -> (toString day) ++ "th"

Also I think Ordinal may not be a good name.
I had a quick hunt on the ruby document page I have been using as a guide for what the character codes in format do. http://apidock.com/ruby/DateTime/strftime  it has a different meaning for ordinal.
Also the characters you selected collide with the ruby API  have bee using as a guide for character format codes.
I think picking something that does not collide with the ruby ones is a better idea.
from %'E' to %'es'
from %'-D' to '%'-ds'
With the 's' standing for suffix, can't think of a better solution at the moment.

I have moved the core into language config and wired it up.
Published 8.0.0 with it.
Finish, French and Polish output only integers at the moment as I don't know what they might need to if they even have this idiom in there language.


Add a prototype tab; support ingest and search with guardrails

Description
This PR adds a Prototype tab that can be used for testing a workflow end-to-end. Note the placement of executing this in UX is TBD, so I've placed in its own sandboxed tab for now for ease of use. It dynamically parses out the necessary fields to generate both 1/ a valid document schema to ingest to the configured index, and 2/ a valid query to run some variation of neural search against the configured index, all based on the workflow's use case. For example, a SEMANTIC_SEARCH use case will have the index mapping configured with a specific set of fields (input field, vector field, model ID), defining the document schema, and a particular way to execute a search with the neural query clause utilizing those fields. This will all be different than other neural search variations, that will have different schemas and a different construction of valid queries to execute.
In particular (current scope is only semantic search), we have UI guardrails to make everything readonly besides:

the value of the plaintext field used when ingesting
the value of the plaintext field used when querying
The rest of the construction is all generated based on the use case (for now only semantic search), and the values of the configured fields (input field, vector field, model ID, index name).

Note this is just a first start, we can easily expose more or less depending on how much flexibility is desired (e.g., freeform queries, opening up editing of k or other neural clause params, etc.)
Implementation details:

added Ingestor component to format, execute, and display response from running an indexing command against an index
added QueryExecutor component to format, execute, and display a formatted list of results from running a query against an index
added base Prototype component to handle global prototype state and tab state
added data_extractor_utils to organize all utility fns used for parsing the workflow data to get different fields based on the workflow use case. this will be expanded upon and refactored as more use cases are added
onboarded search index and index document APIs
minor handling of empty state in prototype tab when the workflow has not been provisioned and/or no resources are available

Demo video:

creating a semantic search workflow and defining all of the configured fields
provisioning to set up the index and ingest pipeline
using Ingest tab for ingesting some sample documents, only requiring plaintext input
using Query tab for searching against those documents, only requiring plaintext input
error handling on empty search
basic validation of semantic search working as expected

screen-capture (31).webm
Issues Resolved
Makes progress on #68
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Looks awesome, quick question, if provisioning takes a long time, should we have some refresh capability or it will auto query until provisioning is done?

Looks awesome, quick question, if provisioning takes a long time, should we have some refresh capability or it will auto query until provisioning is done?

Yep, currently it's very basic, only does one refresh after 5s, no auto refresh. My idea to improve will be logic in the state rendering, and if it's in a transitive state (provisioning/deprovisioning), perform auto-refresh on some 1 or 2s interval or something. Not that fine-grained yet though

Looks awesome, quick question, if provisioning takes a long time, should we have some refresh capability or it will auto query until provisioning is done?

Yep, currently it's very basic, only does one refresh after 5s, no auto refresh. My idea to improve will be logic in the state rendering, and if it's in a transitive state (provisioning/deprovisioning), perform auto-refresh on some 1 or 2s interval or something. Not that fine-grained yet though

Sounds good, even a refresh button later on if some jobs take a while like reindex or deploying a large pre trained model, we have refresh buttons all over core OSD too

Looks awesome, quick question, if provisioning takes a long time, should we have some refresh capability or it will auto query until provisioning is done?

Yep, currently it's very basic, only does one refresh after 5s, no auto refresh. My idea to improve will be logic in the state rendering, and if it's in a transitive state (provisioning/deprovisioning), perform auto-refresh on some 1 or 2s interval or something. Not that fine-grained yet though

Sounds good, even a refresh button later on if some jobs take a while like reindex or deploying a large pre trained model, we have refresh buttons all over core OSD too

Agreed. I was thinking similar. Same with on the base workflow list page, can be helpful. I've already seen on my local single-node cluster how the pretrained models can take up a good 10-15s to finish getting deployed locally.

@ohltyler are we not adding tests for now?

@ohltyler are we not adding tests for now?

No. Not until there's even an idea of a final proposed design, it's not worth adding as the code is too fluid


Bug 2040880: Degrade the OCP cluster if driver is found

Previously we did not degrade or mark cluster as unpgradeable when we can't connect to vcenter.
This PR changes that behaviour and always degrades the cluster if we previously installed the CSI driver and one or more of CSI driver pre-requisties is unsatisfied. This is necessary because CSI driver will be crash looping or being non-functional anyway when this happens.
xref https://bugzilla.redhat.com/show_bug.cgi?id=2040880

/lgtm
/retest


Predict and update API for inference

It seems like we're starting to converge on the idea that on top of the transition and observation methods defined for SSMs, it would be helpful to provide some consistent structure for inference algorithms through the introduction of predict and update methods (named form the two steps forming the Kalman filter).
These two APIs would be fairly orthogonal in design, but with the inference methods calling on the SSM methods.
I feel that we need to be careful to implement these in a flexible way, and will provided some examples of what the interface could look like below.
Initial discussions of the issue are linked directly below.

Use in bootstrap PF and SIR
For the bootstrap filter, we would simply have predict call transition.
For general SIR, predict would use the supplied proposal function and then the update method would call transition_logdensity for reweighting.
Use in Kalman Filter
Suppose that the transition method is written with sufficient generality that it works on both a particle and a GaussianDistribution (this requires transition to return a distribution which is then sampled from).
function transition(dyn, state, ...)
    return dyn.A * state + Gaussian(zeros(dyn.d), dyn.Q))

Then, this transition method can be used in both the Kalman filter and a regular particle filter by dispatching on where state is a float or Gaussian.
Use in Hidden Markov Models
A similar approach can be taken as for the Kalman filter if we define a Multinomial(p) with methods
# Assuming matrix is stochastic
*(A::Matrix, d::Multinomial) = Multinomial(A * d)

It's not clear how you would make this work with an integer state though (i.e. how you'd run a particle filter with this transition function).

Had a more in-depth think about how this would apply to the Kalman filter and it seems like this four methods alone are insufficient.
Naturally we would have
function transition(dyn::LinearGaussian, state::Gaussian)
    return dyn.A * state + Gaussian(zeros(dyn.d), dyn.Q)
end

function observation(dyn::LinearGaussian, state::Gaussian)
    return dyn.H * state + Gaussian(zeros(dyn.d), dyn.R)
end

The prediction step (estimate of p(x_k | y_{1:k-1}) is obvious,
function predict(estimate::Gaussian, dyn::LinearGaussian, alg::KalmanFilter)
    return transition(dyn, state)

Recall the update step for the Kalman filter. This is less obvious.
Calling yi - observation(dyn, estimate) where estimate is the result of predict gives us the innovation distribution.
The trouble is, we then need to computer the Kalman gain, and then update the posterior covariance. Both of these operations require the H matrix of the model, which is not available.
And f you were going to make this available, you might as well make all A, b, Q, R, etc. available and we're back at the old interface.
I feel like this sort of problem is only going to become more common with more complex filters or smoothing algorithms.

I don't think we can make this completely general, but we could maybe impose more structure on the LatentDynamics and ObservationProcess processes ?
struct AdditiveNoiseLinearObservationProcess <: ObservationProcess
  Q::Matrix
  H::Matrix
end

noise_covariance(obs::AdditiveNoiseLinearObservationProcess) = obs.Q

Suppose that the transition method is written with sufficient generality that it works on both a particle and a GaussianDistribution (this requires transition to return a distribution which is then sampled from).

I think this implies an inherent difference in the meaning behind each dispatch of transition. The Gaussian dispatch propagates the entire filtered distribution, not just an individual state. I would instead dispatch predict and update on the sample, and leave transition and observation unused for the Kalman filter.
function predict(states::Gaussian, proc::LinearGaussianLatentDynamics, ::KalmanFilter)
    @unpack A, Q = proc

    predicted_states = let μ = states.μ, Σ = states.Σ
        Gaussian(A*μ, A*Σ*A' + Q)
    end

    return predicted_states
end

function update(predicted_states::Gaussian, proc::LinearGaussianObservationProcess, observation, ::KalmanFilter)
    @unpack H, R = proc

    states, residual, S = GaussianDistributions.correct(
        predicted_states,
        Gaussian(observation, R), H
    )

    log_marginal = logpdf(
        Gaussian(zero(residual), Symmetric(S)),
        residual
    )

    return states, log_marginal
end

I personally don't see anything wrong with this design pattern, but maybe there's something I'm missing.

I agree that both of the above approaches are sensible.
To add some context that is from a previous PR, the motivation for this issue was noting that the draft implementation of the Kalman filter didn't actually use the transition or observation methods from the SSM, rather requesting the model matrices and vectors [1].
The above three comments seem to suggest that it's much cleaner to not force the use of transition and observation, even though they are often useful.
Regardless of this, the predict/update structure for inference algorithms could be useful.
[1] Note, these were supplied by calc_A, calc_b etc. rather than accessing fields to, 1. allow these matrices/vectors to depend on control variables (required for Rao-Blackwellisation), 2. allow for alternative parameterisations, e.g. OU dynamics are parameterised by a single mean-reversion parameter which A can be generated from.

A small caveats is update would most likely introduce something like an AbstractFilterAlgorithm type to allow for dispatch.

Looking at @charlesknipp example, makes me think that the predict/update API probably sits somewhere in an AbstractFilter program that subscribes to the SSMProblems API.
Here for example:

function predict(
        ::AbstractRNG,
        particles::Gaussian,
        model::LinearGaussianModel,
        ::KalmanFilter,
        step::Integer,
        extra
    )
    # this can be replaced with linear_system(proc, step, extra)
    @unpack A, Q = model.latent_dynamics

    predicted_particles = let μ = particles.μ, Σ = particles.Σ
        Gaussian(A*μ, A*Σ*A' + Q)
    end

    return predicted_particles
end

We could leverage the SSM structure:
function predict(rng, model, ...)
    state = transition(model.latent_dynamics, state, step, extra)

    A, Q = linear_system(model.latent_dynamics, step, extra)
    predicted_state = let μ = state.μ, Σ = state.Σ
        # In practice, for a more general Kalman filter
        # we'd probably expect some sort of `update_covariance` / `update_mean` handlers
        Gaussian(μ, A*Σ*A' + Q)
    end
    return predicted_state
end

Looking at @charlesknipp example, makes me think that the predict/update API probably sits somewhere in an AbstractFilter program that subscribes to the SSMProblems API.

Definitely agree with this. You can then have step be the same as calling predict then update.
Allows for nice decomposability. I could run (predict, update)x4 as 4 pieces of data come in and then predict ahead 3 steps by just calling predict.
I think the two things we need to check before committing to this are:

Whether this predict/update distinction is still valid for more complex models
How would we cache values for the backwards smoothing pass to avoid recomputing these

This was moved to GeneralisedFilters.jl and implemented in TuringLang/GeneralisedFilters.jl#7


Update ingress-controllers.md

Adding FortiADC

Hello @erezyfntn, thank you for your contribution! Please can you sign the CLA so that we can review the PR?

@erezyfntn Hello there! We're still looking for you to sign the CLA so that we can review this PR. Thanks so much

@erezyfntn 👋🏻 Please feel free to /reopen when you're ready to sign the CLA.
/close


Update Logger implementation

See Typescript implementation mobilejazz/harmony-typescript#5.

Done and merged to master.


Can not load colllection and another nft when one in collection failed to display

Address get this problem:

5Hdmob81pfjKUfb3wChbTFB5GYmZ2yaVJJfphNoAcjn7rJAb

Version test: https://file.cdmteck.com/s/4oimk9zDJ8R5Aai/download?path=%2F&files=./upgrade-ui-build-1.1.3-2023-07-24T05-30-56UTC.zip
Validation passed


Close file descriptors for redo process

Fixes https://github.com/neondatabase/neon/issues/1814
Tested manually using lsof .zenith/pageserver.pid

I believe this solution is suboptimal. If the goal is to fix just #1814, it would suffice to ensure that somebody sets the FD_CLOEXEC flag for the pidfile's fd. Since we use https://github.com/knsd/daemonize, this is exactly the entity which should do that for us. Unfortunately, that's not the case for the version 0.4.1. Compare this to the main branch: the code is there, but the fix hasn't made it to the crates.io yet (I wonder why).
One could argue that there may be more fds like that. Fortunately, std already uses cloexec where possible (see the docs for exec). This leads me to think that we should do two three things instead:

Somehow use the latest version of daemonize, or better yet, get rid of it. That's not how one should daemonize services in this day and age anyway.
Audit the whole project for other places where we don't set cloexec either due to the broken deps or sloppiness with libs like nix.
Insert an assert! to the pre_exec to blow up the fork if we detect any stray fds.

I believe this solution is suboptimal. If the goal is to fix just #1814, it would suffice to ensure that somebody sets the FD_CLOEXEC flag for the pidfile's fd. Since we use https://github.com/knsd/daemonize, this is exactly the entity which should do that for us. Unfortunately, that's not the case for the version 0.4.1. Compare this to the main branch: the code is there, but the fix hasn't made it to the crates.io yet (~I wonder why,~ the project seems to be dead: last commit on 31 Jul 2021).
One could argue that there may be more fds like that. Fortunately, std already uses cloexec where possible (see the docs for exec). This leads me to think that we should do two three things instead:
* Somehow use the latest version of daemonize, or better yet, get rid of it. [That's not how one should daemonize services in this day and age anyway](https://0pointer.de/public/systemd-man/daemon.html#New-Style%20Daemons).

* Audit the whole project for other places where we don't set `cloexec` either due to the broken deps or sloppiness with libs like nix.

* Insert an `assert!` to the `pre_exec` to blow up the fork if we detect any stray fds.

Yes https://github.com/neondatabase/neon/issues/1814#issuecomment-1141365290
Short term: Would you rather deal with unreleased commits from daemonize, or this new close_fds crate? We might need this crate anyway for the pre_exec assertion that you mention, unless there's a simpler way to correctly iterate open fds in a multi-thread program. Or alternatively we can do this check after exec, in the postgres code.
Long term: Moving away from daemonize also fixes https://github.com/neondatabase/neon/issues/1840, among other things. It's a bigger project though, we shouldn't block a/reliability fixes on this transition.


Saving array, object to 1 key (1 record)

Thanks for your great package!
It would be nice if your package gave an option of saving array, object (after serialize()) to 1 key, like wordpress option.

Objects aren't supported. Everything is a native value as we use json_encode. (int/float/string/array). Feel free to extend a setting store and using serialize, except it won't be compatible with any other programming language.

I definitely wouldn't want to implement anything using serialize(), though JSON might be an option. However, I don't see a strong argument for why you'd want to do that over using one table row for each setting value - could you elaborate on why you'd like this feature?


bpf: Fix Go format

Invalid format breaks builds in the dev VM.
Signed-off-by: Jarno Rajahalme<EMAIL_ADDRESS>

This change is

Coverage decreased (-0.01%) to 42.585% when pulling 9f373da0c239eb5f68bf728b539fb43f866bf22d on pr/fix-go-format into 8d81abdb8c1eb6212a0faf713ebe70177b0cde68 on master.

I've just noticed it, sorry about that, thanks for the fix.


Add required files

Added 3 files (each one in a separate commit):

index.html file that includes a header with the text "Hello Microverse!". Used the h1 tag for that.
style.css file that includes one class with styles for the header (make it in favorite color).
test.md file with the text "This file should be ignored by git".

Completely finish requiring task and review done


{cdn}skip cdn migration commit ut

skip cdn migration commit ut to fix https://github.com/Azure/azure-cli/issues/30577

Thank you for your contribution! We will review the pull request and get back to you soon.

/azp run


Auth Cookie Expiration

My auth cookie gets expired after a hour.How do i extend the span?
I have tried adding cookie.expiration and other ways in startup.cs but nothing worked.
My app is in .net core version.
Please help..

Serenity uses .NET Core authentication so you should check DOCs for that.


how play youtube link in GSYplayer

hi
how play youtube link ex: https://www.youtube.com/watch?v=ZIbNKBx88_E
please write code here
thanks

this is a webUrl，not video stream url


Run the entire analysis together

Runs the entire analysis by:

Runs /analysis and keeps polling until safety_analysis is done.
Then, runs api.results and keeps polling until highlight_video is done.
Then downloads the results, unzips them, and opens the file location

This is what the GUI looks like when I test on the SCOPE mac.  I can see what I can change in the .ui file to fix this.

I made a PR that fixes the button size here: https://github.com/santosfamilyfoundation/SantosGUI/pull/46

Going to merge. Just going to dev, not master, and I feel that it's fine to merge.


[TOR2e] added dark theme support 03.08.00 (new feature)

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

New Sheet Details

The name of this game is: <   >  (i.e. Dungeons & Dragons 5th Edition, The Dresden Files RPG)

The publisher of this game is: <   > (i.e. Wizards of the Coast, Evil Hat)

The name of this game system/family is: <   > (i.e. Dungeons & Dragons, FATE)

[ ] I have followed the Character Sheets Standards when building this sheet.

[ ] This sheet has been made on behalf of, or by, the game's publisher.
[ ] This game is not a traditionally published game, but a copy of the game rules can be purchased/downloaded/found at: <   >
[ ] This sheet is for an unofficial fan game, modification to an existing game, or a homebrew system.

Changes / Description
03.08.00 (Build 29)

added dark theme support for all sheets and roll-templates and message templates

Character Sheet Info Roll20 Internal Use only.


Migrate REP button and tooltip

Desktop:
https://www.figma.com/file/bZh7tOiLm6Ls9Hl4RL6qMC5X/Augur-Redesign?node-id=12885%3A588

[ ] Change the button type (blue button is too big for the new nav bar) HIGH
[ ] Switch out the tooltip icon (new icon being used everywhere) LOW

Mobile:
https://www.figma.com/file/bZh7tOiLm6Ls9Hl4RL6qMC5X/Augur-Redesign?node-id=12888%3A934

[ ] Change the button type (blue button is too big for the new nav bar) HIGH
[ ] Add the tooltip HIGH

So I pushed up this PR, but I would recommend adding the new styling for the tooltips in a separate PR. The tooltips look ok right now and if it's low priority, it could be punted.

tooltip in build looks better anyway imo

Closing as not priority.


Added --year argument

Will add "/timeline?year=" year to user_profile_url before navigating
Using the URL "https://www.facebook.com/[USER]/timeline?year=[YEAR]" to limit deletions to that time period.

Thanks for the contribution! I've integrated your changes into https://github.com/weskerfoot/DeleteFB/pull/23 since they were conflicting


Maps are included in NPM package v4.5.0, but source files are missing

Map files starts:
{"version":3,"sources":["../tmp/<file_name>.ts"],"names":[],"mappings":...
But those files are missing

As of<EMAIL_ADDRESS>following Angular Package Format v5.0 format for the package. The spec doesn't put source files in the package, so will continue not to add them.


nomination-pools: clear pool metadata after a pool is deleted.

In fn dissolve_pool, the corresponding pool metadata has to be removed as well.
Currently, we forget to do this.
Also requires a small migration: iterate over all metadatas, if one does not have a corresponding bonded_pool, delete it. The migration could also be executed manually with providing the right system::kill_storage.

Would like to attempt this

What do you mean by migration?


added error message for 500 to logger

Description
Added error messages for 500 error to event service. When 500 error happens, the logger will have a brief message about the reason of the error.
Fixes #841
Review Time Estimate
Please give your idea of how soon this pull request needs to be reviewed by selecting one of the options below. This can be based on the criticality of the issue at hand and/or other relevant factors.

[ ] Immediately
[x] Within a week
[ ] When possible

Type of changes
Please select a relevant option:

[ ] Bug fix (non-breaking change which fixes an issue).
[x] New feature (non-breaking change which adds functionality).
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).
[ ] Other (any another change that does not fall in one of the above categories.)

Checklist:
Please select all applicable options:

[x] I have signed the Rokwire Contributor License Agreement (CLA). (Any contributor who is not an employee of the University of Illinois whose official duties include contributing to the Rokwire software, or who is not paid by the Rokwire project, needs to sign the CLA before their contribution can be accepted.)
[x] I have updated the CHANGELOG.
[x] I have read the Contributor Guidelines.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[ ] My change requires updating the documentation.
[ ] I have made necessary changes to the documentation.
[ ] I have added tests related to my changes.
[ ] My changes generate no new warnings.
[ ] New and existing unit tests pass locally with my changes.
[ ] Any dependent changes have been merged and published in downstream modules.

@ywkim312 @sandeep-ps I can not leave comments on the code line. I am not sure why.

import configs.messages as msgs might change to import controllers.messages as msgs

import configs.messages as msgs might change to import controllers.messages as msgs

changed.

main.py's line 44 makes an error based on situation. Maybe you'd better make it as separate variable in conf instead of hard code it.

main.py's line 44 makes an error based on situation. Maybe you'd better make it as separate variable in conf instead of hard code it.

@ywkim312 are you leaving the comments at the different pr?

main.py's line 44 makes an error based on situation. Maybe you'd better make it as separate variable in conf instead of hard code it.

oops sorry. thank you for catching it. =)

@bingzhang, I'm going to merge this as you had approved it earlier. Please let me know if there are any concerns. Thanks.


Fix shared option handling

closes: #7730
closes: #7546
Changes:

make option sharing an opt-in for controller
shared options is a single object held by the controller
controllers are always reset first, even when animations are disabled
getSharedOptions takes only one parameter (resolved options)
updateSharedOptions takes the new options as parameter
animations update the target with reference to shared options, after all animations are done.
'default' used instead of falsy modes

https://codepen.io/kurkle/pen/poyeEmJ

@benmccann interested in this at all?

IE11 needs a polyfill:
SCRIPT5009: 'Promise' is undefined


Stylesheet tweaks

Fixes #213
Fixes #210

Awesome. Thank you.

Will need to rebase this PR in order to make it mergable.


Remove LeakCanary from library

Codecov Report

Merging #228 (176f265) into main (1653efc) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 176f265 differs from pull request most recent head 623a68c. Consider uploading reports for the commit 623a68c to get more accurate results

@@           Coverage Diff           @@
##             main     #228   +/-   ##
=======================================
  Coverage   74.25%   74.25%
=======================================
  Files          66       66
  Lines        2975     2975
  Branches      477      477
=======================================
  Hits         2209     2209
  Misses        500      500
  Partials      266      266

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


mainWindow's windowLevel should not be changed

Changing the main window's windowLevel will cause some expected issues, such as using the library with Intercom: https://github.com/intercom/intercom-ios because the Intercom window is using windowLevel = 0.1
This library should use a value less than 0 to be the windowLevel for it's private passcodeLockWindow, and never change the windowLevel of the main window.
When presenting the passcodeLockWindow, it's window level can be a bigger number (such UIWindowLevelStatusBar - 1), but once it's dismissed, it should be set back to a number less than UIWindowLevelNormal.

Sounds good indeed. Thanks for all the info shared. Are you @bull-xu-stride willing to open a PR for this one?


feat✨: add resource parallelization

Description

Parallelized GCP resource computation using the ThreadPoolExecutor().
Improves tool runtime from ~37s to ~12s.

@mshudrak @ZetaTwo

Tested today the change. I see quite strange behavior when I run the scanner. It basically hangs forever and can't start the enumeration
python3 -m gcp_scanner -g - -o res -l INFO -wc 1
<skip>
Inspecting project <project_name_here>
^CException ignored in atexit callback: <function _exit_function at 0x7f20baaa39a0>
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/util.py", line 360, in _exit_function
    _run_finalizers()
  File "/usr/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/lib/python3.10/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 199, in _finalize_join
    thread.join()
  File "/usr/lib/python3.10/threading.py", line 1096, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
KeyboardInterrupt:

You can reproduce it by removing res output folder. It seems for me like some error happens in the child process and the parent waits for it to exit forever.

Looking into it.

Important note: it is likely related to previous commit with process based parallelization not this one

I was not able to reproduce the hanging behavior, but got similar result if the path to the output doesn't exists. I get the following output (the program terminates after the inspecting <<project_name>>):
python3 scanner.py -l INFO -wc 8 -o res -k ../output/sakeys
2023-08-25 18:33:47 - INFO - Retrieving credentials from ../output/sakeys/gcp-scanner-test-project-01-ba40a00eb1fe.json
2023-08-25 18:33:47 - INFO - >> current service account<EMAIL_ADDRESS>2023-08-25 18:33:47 - INFO - Retrieving projects list
Inspecting project gcp-scanner-393515 for Impersonation
Inspecting project gcp-scanner-test-project-02 for Impersonation
Inspecting project gcp-scanner-test-project-01 for Impersonation
Inspecting project gcp-scanner-393515
Inspecting project gcp-scanner-test-project-02
Inspecting project gcp-scanner-test-project-01

The possible solution would be to exit the program if the folder doesn't exist. This way, we eliminate the assumption that the path to output is valid.

I think this issue persist for any other errors that happens in the child process. You could try to reproduce by injecting an error in one of the child processes. I did my test in a fresh venv and have this issue constantly.

I think we need to dig deeper in that problem in order to understand why it is happening given it is critical issue that block the scan.

There could be differencies in how we execute it. I can't reproduce it with GCP SA key either but when I acquire creds from gcloud it is happenning (-g -) parameter.

Pylint is complaining about the recent change. PTAL.

@mshudrak done with the changes :)


Remove shebang

This allows the module to be used in the browser.

this breaks it as a command, can you make a bin.js file that has the #!

Done, I've moved the shebang and the if(!module.parent) to bin.js, fixed the permissions, and updated package.json with the new path. Please let me know if there's anything else, thanks!

@dominictarr can we merge this please?

merged into 1.4.3

Thanks!


Reporting - Daily new applications

Describe the task
We need a new dashboard (page or section) within the HR module to show the analytical data. This will be visible to only super-admins and HR admins.
Expected behavior
A clear and concise design for the HR analytics dashboard. The basic use case would be to show a report that displays daily new applications.
Some suggestions that the design team may consider:

Graph (line or bar) that shows the count of new applications
By default it can be past 7 days but there can be an option to increase/decrease the report duration.
There may be more options like showing daily new applications based on college or job profile. Maybe in the future.

Design team to share the link for design files.

PRD for the project: https://docs.google.com/document/d/170KT7y7_venU_uF1lpKnTJbXjj4qOu3Qk22q-pfsMtk/edit?usp=sharing

Prototype link: https://www.figma.com/proto/ZjX8ogSM5yrZGhvO5vEWJn/Employee-Portal?page-id=0%3A1&node-id=353%3A97&viewport=356%2C-1133%2C0.17737901210784912&scaling=min-zoom&starting-point-node-id=353%3A97

Design file link: https://www.figma.com/file/ZjX8ogSM5yrZGhvO5vEWJn/Employee-Portal?node-id=0%3A1

This looks great 🎉


Update azure-sdk-authenticate-hosted-applications.md

This steps may be still interesting, but should not be recommended to customers

@KarlErickson - Would you be able to review and approve these proposed changes?
#label:"aq-pr-triaged"

@jess-johnson-msft ping

@DavidCBerry13  - I added you as a reviewer for this PR, as you have been contributing and expanding the Python SDK Auth content.

@DavidCBerry13 ping

Closing this issue because we have just reworked the whole SDK auth section and this file has been replaced by new content.  Working with Laurent though to make sure the intent of this change is captured in the new content.


⚠ Bofrost.com has degraded performance

In f3cc923, Bofrost.com (https://www.bofrost.com) experienced degraded performance:

HTTP code: 200
Response time: 716 ms

Resolved: Bofrost.com performance has improved in 29ff881.


Implement the K8s sub-system for Galley.

This PR implements completes the core K8s sub-system implementation for Galley.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: ozevren
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

galley/OWNERS [ozevren]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: ayj, ozevren
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

galley/OWNERS [ayj,ozevren]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ozevren: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/e2e-dashboard.sh
644ab75f1f9a6c013ffa4e42b2408f6ec17f95b0
link
/test e2e-dashboard

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


CI broken

The tests run in CI are currently somewhat broken by what it seems two issues.
I will try to nail them down, but am currently a bit low on time. So if anyone wants to search the problem, feel free to examine the current travis-results.

Part of this is fixed in 01e28bbcf642a5b4b9d29ea0dea63e35305cd833

OK, the test run complete, this is fixed. The problem is a regression in pytest.


SReLU activation

Added the recently proposed SReLU activation.
http://arxiv.org/abs/1512.07030
see issue #1662

The added tests are failing.

They were failing due to inconsistency between the TensorFlow and the Theano backends. TF's ReLU did not support a non const slope for the negative part - reported as #1681. It is now fixed and all the tests are passing.

@fchollet thanks for the feedback. Now testing the type of alpha. Tests seems to be failing randomly on MNIST download. I pushed 2 commits with no code difference, one failed on python2.7+theano and the other python3.4+tensorflow. Its probably temporary though.
==================================== ERRORS ====================================
______________ ERROR collecting tests/keras/test_regularizers.py _______________
tests/keras/test_regularizers.py:22: in <module>
    (X_train, y_train), (X_test, y_test) = mnist.load_data()
keras/datasets/mnist.py:17: in load_data
    data = cPickle.load(f)
../../../miniconda/envs/test-environment/lib/python2.7/gzip.py:464: in readline
    c = self.read(readsize)
../../../miniconda/envs/test-environment/lib/python2.7/gzip.py:268: in read
    self._read(readsize)
../../../miniconda/envs/test-environment/lib/python2.7/gzip.py:315: in _read
    self._read_eof()
../../../miniconda/envs/test-environment/lib/python2.7/gzip.py:354: in _read_eof
    hex(self.crc)))
E   IOError: CRC check failed 0x5159ec5c != 0x7a59d592L
______________ ERROR collecting tests/keras/test_regularizers.py _______________
tests/keras/test_regularizers.py:22: in <module>
    (X_train, y_train), (X_test, y_test) = mnist.load_data()
keras/datasets/mnist.py:9: in load_data
    path = get_file(path, origin="https://s3.amazonaws.com/img-datasets/mnist.pkl.gz")
keras/datasets/data_utils.py:42: in get_file
    ParanoidURLopener().retrieve(origin, fpath, dl_progress)
../../../miniconda/envs/test-environment/lib/python2.7/urllib.py:289: in retrieve
    "of %i bytes" % (read, size), result)
E   ContentTooShortError: retrieval incomplete: got only 1383448 out of 15296311 bytes
------------------------------- Captured stdout --------------------------------
Downloading data from https://s3.amazonaws.com/img-datasets/mnist.pkl.gz
 1384448/15296311 [=>............................] - ETA: 822s

These errors appear to be connection issues on the Travis server. It doesn't matter...

This looks good to me, please squash your commits and I will merge.

@fchollet done


Fix DLL unloading on Linux

Fixes https://github.com/Unvanquished/Unvanquished/issues/1792

Some of these apparently come from crunch:
daemon/libs/crunch/crnlib/crn_math.h
inline uint count_trailing_zero_bits(uint v) {
  uint c = 32;  // c will be the number of zero bits on the right

  static const unsigned int B[] = {0x55555555, 0x33333333, 0x0F0F0F0F, 0x00FF00FF, 0x0000FFFF};
  static const unsigned int S[] = {1, 2, 4, 8, 16};  // Our Magic Binary Numbers

Out of curiosity, how did you find it? Is this useful for the dll bin or for the engine?
If you've tested it, LGTM.

I recently fixed the same problem (dlclose doesn't unload the DLL) for Mac and recognized the symptom of duplicate emoticon warnings. After confirming that was the problem, I googled for "dlclose no effect" or something like that and learned about STB_GNU_UNIQUE. And ran the commands suggested to list STB_GNU_UNIQUE symbols (I didn't see anything from Crunch, presumably because it was optimized out in a release build).
The flag is only needed for objects that go into DLLs. I just applied it to everything because that's easiest and has no ill effects.


Improve logging

I think this PR is ready @yunhailuo.

The PR is good itself. But I'll wait until we get replies from GDC and figure out the "right" way to download from GDC. On hold for now.


Firewalla with NextDNS

I set this up:
https://github.com/Control-D-Inc/ctrld/wiki/NextDNS-Mode
But the service seems to hang after an hour or so.  This has happened multiple times since setting it up tonight.  It works great when it's working, but then it stops working and I see a "connection refused" to the Firewalla IP message with running a dig command on the Firewalla.
I'm seeing this in the syslog:
Dec 25 23:35:31 localhost bash[2008190]: dnsmasq: illegal repeated keyword at line 6 of /home/pi/.firewalla/config/dnsmasq_local/ctrld
Dec 25 23:35:31 localhost bash[2008190]: file /home/pi/.firewalla/run/dns_proxy.strict_new.bf.data can't be opened

Thanks for reporting.
Sounds like something is triggered on Firewalla causing a conflict with the cache-size that is added by ctrld.

Maybe this helps:  https://www.reddit.com/r/pihole/comments/45v71b/pihole_dnsmasq_cachesize_directive/

Maybe this helps:
https://www.davekb.com/browse_computer_tips:dnsmasq_illegal_repeated_keyword_at_line_n_of_etc_dnsmasq.d_caching-dns.conf:txt
https://www.reddit.com/r/pihole/comments/45v71b/pihole_dnsmasq_cachesize_directive/

Yes, we know that there could be a conflict. But the thing is that Firewalla only trigger "something" after ctrld running for a while, which causes the conflict happens.
We have a plan to tackle this in future releases.

I have a test Firewalla device on my desk, never encountered this issue after many weeks of operations.  @cirrusflyer Do you know what exactly triggers this to happen? Can you provide steps to reproduce?
In the mean time, you can also run ctrld in this mode (article is for pfsense, but its the same): https://github.com/Control-D-Inc/ctrld/wiki/pfSense-and-OPNsense-Operations-Guide#dont-touch-my-configs

I've removed the cache-size line from the file, and I'm seeing how long things are stable.  Maybe just coincidence as well.  Going on a couple hours now.

Also, see here:  https://www.reddit.com/r/firewalla/comments/udh52a/firerouter_dns_restarting_often/
Seems the restart every couple minutes is normal.

We'll get this addressed in the next version.

This is done and available in v1.3.4

Great.  Remind me how to update Firewalla with NextDNS?

Re-run the installer command from https://github.com/Control-D-Inc/ctrld/wiki/NextDNS-Mode
It will upgrade the instance.


Productionizing chime a bit for upcoming user test.

In prep for our first user test, making Chime easier to deploy to production. Changes include:
Switching to rbenv as a way to get a consistent, modern Ruby.
Removing the github-pages dependency and pulling in Jekyll only.
Adding some logging to the python app to make debugging production issues easier.
Adding server deploy instructions.

I will review this today

🍻


'az ml computetarget create computeinstance' doesn't work properly for VNet

Description:
When securing AML workspace and associated resources by using VNet and private link/endpoints, after setting up an AML workspace with a private endpoint, I want to create compute instance which should also be put into the VNet. Here's the command I'm using:
az ml computetarget create computeinstance --name ci-gary1-test --vm-size Standard_DS3_v2 --vnet-name ${azurerm_subnet.compute_subnet.virtual_network_name} --subnet-name ${azurerm_subnet.compute_subnet.name} --vnet-resourcegroup-name ${azurerm_subnet.compute_subnet.resource_group_name} --resource-group ${azurerm_machine_learning_workspace.aml_ws.resource_group_name} --workspace-name ${azurerm_machine_learning_workspace.aml_ws.name}
By using this command, when I log into the AML workspace though a jump host, I can see the created 'ci-gary1-test' with gray links of 'jupyterlab, jupyter...' which is totally not usable. Please see the screen shot below. Note the compute instance 'wg655361' was created through the portal.

Would you please advise how to solve this issue?
Thanks in advance!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 70b332b2-b671-a5e8-0db0-38aa1378dbaf
Version Independent ID: 13cd812a-48e4-dd85-3cc8-a0b357a5a453
Content: Create and manage a compute instance - Azure Machine Learning
Content Source: articles/machine-learning/how-to-create-manage-compute-instance.md
Service: machine-learning
Sub-service: core
GitHub Login: @sdgilley
Microsoft Alias: sgilley

@gary918 Thanks for the question. We are investigating the issue and will update you shortly.

@gary918 confirming you see the issue with CLI but it is working fine through portal?

@gary918 Thanks, Can you please confirm is it issue with the CLI.

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.

@gary918 - can you try adding a compute instance from the studio interface and let us know if that one works?  We're trying to figure out if this problem occurs only when using the CLI.
Thanks,
Sheri

@sdgilley - Yes. I confirm that I can add the compute instance from the studio interface. The one named 'wg655361' was created in that way. If you run the CLI in the cloud shell, the created instance will be like 'ci-gary1-test'.
If you do want to reproduce the case, you can run the terraform scripts in https://github.com/csiebler/azure-machine-learning-terraform.

Any update about this issue, @sdgilley ?

Hi, @Blackmist , thanks a lot for your reply. Understood that this is out of your working scope. I'll contact Azure Support later.


Separate S3Control methods from S3

The S3-module contained three methods that were technically part of S3Control:

delete_public_access_block()
get_public_access_block()
put_public_access_block

Because they were part of the S3-module, it needed the S3-mock:
@mock_s3
def test():
    client = boto3.client("s3control")
    ...

This PR extracts the logic into a dedicated module, making it more discoverable and obvious that we support these methods.
This means users now have to use the dedicated mock_s3control:
@mock_s3control
def test():
    client = boto3.client("s3control")
    ...

This is a breaking change, so it will only be merged as part of the next major release.
Note that none of the existing logic or tests have been changed as part of this PR - only moved around.

@mikegrima I believe you implemented this feature originally - any objections to this change from your side?

Hi @bblommers! As long as all of the previous functionality will work properly then I'm OK with the change.  Is the breaking change that instead of mock_s3, its mock_s3control?

Correct @mikegrima - it's just a matter of using the new decorator from 3.x. None of the logic has changed.

LGTM

Thanks @mikegrima!

Codecov Report

Merging #4745 (77d2ad5) into master (9c8744f) will increase coverage by 0.00%.
The diff coverage is 97.20%.

@@           Coverage Diff           @@
##           master    #4745   +/-   ##
=======================================
  Coverage   95.01%   95.02%
=======================================
  Files         575      580    +5
  Lines       61324    61343   +19
=======================================
+ Hits        58268    58289   +21
+ Misses       3056     3054    -2

Flag
Coverage Δ

servertests
36.29% <25.87%> (+0.01%)
:arrow_up:

unittests
94.95% <97.20%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

moto/backend_index.py
100.00% <ø> (ø)

moto/s3/config.py
100.00% <ø> (+3.66%)
:arrow_up:

moto/s3/models.py
95.22% <ø> (-0.08%)
:arrow_down:

moto/s3/urls.py
100.00% <ø> (ø)

moto/s3control/config.py
93.44% <93.44%> (ø)

moto/__init__.py
98.42% <100.00%> (+0.01%)
:arrow_up:

moto/config/models.py
99.34% <100.00%> (+<0.01%)
:arrow_up:

moto/s3/responses.py
95.52% <100.00%> (-0.01%)
:arrow_down:

moto/s3control/__init__.py
100.00% <100.00%> (ø)

moto/s3control/models.py
100.00% <100.00%> (ø)

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c8744f...77d2ad5. Read the comment docs.


The first "ALARM Disarmed" default notification on iOS cannot be deleted with swipe. Requires clicking it and opening home assistant app before it will go away

Alarmo Version
1.9.4
HA Version
2022.7
Bug description
I can provide a video of this behavior if necessary.
This has been witnessed multiple times as I am new to HA and have been setting up HA and Alarmo over the last few weeks.
Essentially, upon disarming through the alarm panel, I have configured Alarmo to send a push notification when the alarm is disarmed.
The message arrives properly, however, it does not allow swiping to remove it.
 I get an arming notification as well as a second disarm notification. ---> The arming notification can be swiped to delete and the second disarm notification can be swiped to delete. The only way I have been able to get the first arming notification to go away is to click it which opens the HA iOS app. This is reproducible if you would like to see a video or screen recording of the issue.
Thank you for any help you can provide with this!
Steps to reproduce
No response
Relevant log output
No response

I just ran an additional test, I took the system to "armed away" and got a notification, it is swipable to the left and gives options such as "clear" and "options". I then Disarmed the alarm, the notification that came with the disarm would not allow swiping to the left to clear it.
Maybe this is how you intended this to work? (First disarm can't be "acknowledge" or maybe it is a bug.
Thank you for looking into this and sorry for dropping another message. I just wanted to give you that info as I was testing a little bit more.

This issue has possibly went away for me by deleting the message automation and recreating it within the Alarmo configuration panel. Craziest thing I've ever seen. Maybe this can be closed and if it starts happening again, I will let you know.

Actually, I'm seeing this with other notifications from home assistant so I think this may be an HA issue possibly.

The notifications sent by Alarmo rely on the notify services of HA.
I cannot influence the behaviour from the side of Alarmo, the configuration is limited to the parameters which HA allows.
Perhaps you could play with the parameters, following the examples on this page.
You could send a 'raw' notification from the Developer Tools -> Services page in HA.
If the problem persists, I think its best to open a bug report in the HA repository (and check if there is already one open for this).
At this point I don't know how to follow up on this report.

Not a problem and I agree with you. Thank your for your response. I will close the issue so it’s not clogging up things


Switch to the keycloak-openshift base image.

Some of the permissions tweaks are already included in this
upstream image, simplifies our Dockerfile and also makes
it easier for my bazel experiments.
Need to test in staging to confirm this image runs ok in the openshift environment. Works fine in local dev under docker-compose.

Tested and confirmed on openshift, keycloak deployment #34 is running fine.


Revert "Upgrade to Curator 2.13.0"

Reverts vespa-engine/vespa#11600
This upgrade failed last time (some system tests failed), so have a revert ready for merging. We are now using a newer ZooKeeper version (3.5.6), so let's try again. Please review only

Some tests failed this time as well


MKButton doesn't work on Android

it is working for me on Android API 22 and 23,  with react native 0.21, mdk 0.3.0  (I use the xxFab buttons).

Sorry about that, i had something wrong with my settings. MKButtons works well


wrong include path generated for module node-addon-api

my addon project depends on node-addon-api and I add the include as follow:
'include_dirs': [
        "<!@(node -p \"require('node-addon-api').include\")"
 ]

but the include path to node-addon-api is wrong in VC project which generated by "node-gyp configure":
..\..node_modulesnode-addon-api
it should be:
..\..\node_modules\node-addon-api
I ran node command (node -p "require('node-addon-api').include") and it returned the correct path.

Node Version: v14.7.0
Platform: Windows 10
Compiler: Visual Studio Community 2017
Module:

Odd, I guess the \ is being treated as escaping. What happens if you remove the @?

@rvagg Thanks.
The issue is really fixed after remove @

@nodejs/addon-api hey, the docs for node-addon-api have the @ but we don't use it for the NAN docs. Perhaps this should change? https://github.com/nodejs/node-addon-api/blob/master/doc/setup.md, I don't think it's necessary if it's just one item being returned, not an array, and in this case it seems to be to blame for doing some extra escaping on Windows, which I don't get, but maybe @nodejs/gyp understands.