Controlling which color spaces appear in menus

There has been wide agreement that it would be helpful for config authors to be able to limit which color spaces appear in application menus.  Indeed, several features of OCIO v2 are aimed at improving this situation and a number of PRs have been merged.  At this point, it seems that it would be useful to summarize this topic and pull together the results of various conversations to make sure that we are all on the same page as to how this will work in v2 and decide on any remaining tweaks.
NOTE:  To clarify the scope of this issue, it is only concerning color space menus.  Applications also need to create menus for displays and views, but the display/view section of the config already filters a list of views based on what display is selected.  Furthermore, the new viewing rules allows an application to further filter the views based on the color space of the image being viewed.  So display/view filtering is out of scope for the present issue.
OVERVIEW
To recap and frame the color space menu problem, there are several new and existing mechanisms that are relevant:

The roles may be used to specify that there should be no user choice (i.e., a fixed color space should be used).  An example of this that is used in Maya is the "rendering" role.  If this role is present, the application rendering space menu in Preferences is locked to this color space.

The new inactive_colorspaces property of a config or the corresponding environment variable may be used to remove color spaces from application menus while still allowing them to be used, if necessary.  For example, an old scene file may use deprecated color space names.  Leaving them in the config allows an application to read and process the old files but adding those color spaces to the inactive list means they won't show up in menus by default.  Note that an application may temporarily add a given space to the menu in this scenario when displaying what was assigned to a given asset in the scene.

The proposed "aliases" feature (Issue #1129) would provide another way to prevent color spaces from appearing in menus while still allowing them to be used elsewhere (e.g., to be used in file rule paths).

An application may include all roles in color space menus in special circumstances.  This statement might seem contrary to several discussions in TSC meetings concluding that it is generally undesirable for applications to add roles to color space menus.  (If a config author wants them to be in the menu, a color space with the desired name could be added.)  However, there are some situations where roles could be appropriate.  For example, if an application allowed editing of file rules, it would be appropriate to allow a role to be selected as a rule's color space.

The family attribute of a color space may be used to group the menu contents into hierarchical menus.  (This does not actually control whether a space appears but could be used to similar effect, e.g., by putting spaces under a "DO NOT USE" heading, which is sometimes used as a workaround in v1.)

Other color space attributes such as encoding, isdata, and equalitygroup could also be used to filter color spaces.

The new display_colorspaces section of the config could provide another means for an applications to sort types of color spaces.

And last, but not least, the primary purpose of the new categories attribute of a color space is to filter what shows up in each menu.

APP HELPERS
In order for these capabilities to be implemented in a uniform manner, there needs to be some example code provided.  The ColorSpaceMenuHelper class in the App Helpers code is intended to serve this purpose and already supports a number of the above mechanisms (nos. 1, 2, 4, 5, and 8).  The App Helpers were introduced in PR #952 and PR #1220 (in progress) moves it into the core library.
CATEGORIES
The item that needs the most elaboration is the last one: categories.  The categories feature was added in PR #627 and has been discussed in Issue #623 and at many meetings.  However we still need to get alignment on what set of official strings will be documented and how the App Helpers code should filter based on them.
During the discussions, there have been two main use-cases that have been discussed:
Menu-based (or Task-based) filtering -- In this scenario, the categories contain a description of the task that a given application menu is used for.  For example, Maya has a menu for assigning a color space to a file texture and another menu for selecting the rendering (working) space.  Clearly, a different set of color spaces are appropriate for those two tasks.  From an app developer's point of view, these category string(s) would be predetermined and fixed for a given menu (and they should be documented by the app developer for the benefit of config authors).  A baseline set of strings should be documented by OCIO to promote consistency among developers but, as with roles, it should be possible to introduce more without requiring any library changes.
Artist-based filtering -- In this scenario, the categories contain a description of the type of person using the application based on what type of tasks they focus on.  For example, in a large studio there are departments for modeling, layout, look-dev, lighting, compositing, and many others.  The degree to which a given sort of artist is knowledgeable about color spaces and is expected to be able to select which ones are appropriate varies a lot with their role.  Hence it makes sense to filter color spaces based on this.  From an app developer's point of view, the category string(s) being looked for are not predetermined and the expectation is that they would need to be set at runtime (e.g., by an environment variable).  However, it would still be useful for OCIO to document a baseline set of strings to promote consistency (and again with the understanding that it does not need to be an exhaustive list).
Another way of looking at this is that the app developer hard-codes one set of keys (the menu categories) and the app end-user provides another set of keys at runtime (the artist categories) and the intersection of these is what filters what is provided by the config author.  In other words, another way of framing the problem is to think of it in terms of a set of hard-coded app-based categories plus a set of runtime end-user-based categories.
That brings us to the question of how the App Helpers example code should filter color space lists using categories.  Currently, the code takes one list of strings and will select all color spaces that contain any of those strings (in other words, a union or logical OR).  If no strings are provided, or the config does not use categories, the color spaces are not filtered by them.
The limitation of the existing algorithm is that it does not allow expressing a logical AND relationship, which seems like it would be necessary to capture the menu-type AND artist-type intersection that is desired.
This issue proposes extending the App Helpers code to take two lists of categories rather than one -- an app-supplied list and an end-user-supplied list.  The following algorithm would then be used:
if (no categories are provided to App Helpers)  --> don't filter based on categories

app_set = FindColorSpaces(app_categories)
end_user_set = FindColorSpaces(end_user_categories)

if ( !isempty(app_set) && !isempty(end_user_set) )  --> use the intersection of the two sets
else if ( !isempty(app_set) )  --> filter using the app categories
else if ( !isempty(end_user_set) )  --> filter using the end-user categories
else  --> don't filter based on categories

MENU-BASED CATEGORIES STRINGS
Here is a proposed baseline set of Menu-based categories:

file-io -- For menus setting the color space of imported or exported files.
working-space -- For menus setting a working color space.
look-process-space -- For menus setting the process space for applying a CDL or LUT.

OTHER FILTERING
Note that the above menu-based categories are intended to be used in connection with the "encoding" property of the color space (Issue #1031, merged in PR #1038).  So for example, rather than define "scene_linear_working_space", "log_working_space", etc., there is only a "working-space" category.  If an application needs scene-linear working spaces, they would provide App Helpers with a category of "working-space" and an encoding of "scene-linear".  Hence another refinement proposed here is to allow an encoding to be presented to the ColorSpaceMenuHelper class (which is not currently implemented).
Similarly, it seems that it could be useful to further filter the list to show either all color spaces, only display color spaces (Issue #926, merged in PR #939), or only non-display color spaces.  (The Config class API already supports this, but it could be useful to add that to ColorSpaceMenuHelper too.)
ARTIST-BASED CATEGORY STRINGS
The Artist-based categories are a bit harder to enumerate.  There are a lot of different department names in a big VFX or animation studio and it might be difficult to agree on a representative list.  Furthermore, there may be little point to distinguish between certain types of artists if they have similar needs in terms of color management (e.g., modeling vs. layout).  So perhaps one way of approaching it is to delineate between 3d (modeling, animation, lighting, etc.) and 2d (editorial, compositing, etc.) departments/jobs and based on the amount of access needed?  Here is a proposed baseline set:

basic-3d -- For artists in 3d departments needing only basic color management.
advanced-3d -- For artists in 3d departments needing advanced color management access.
basic-2d -- For artists in 2d departments needing only basic color management.
advanced-2d -- For artists in 2d departments needing advanced color management access.

SUMMARY
In summary, here is the proposal:
Add to ColorSpaceMenuHelper in App Helpers:

The ability to specify a second list of categories, to be used in intersection with the first (existing) set (thereby allowing both an app-based set and an end-user-based set).
The ability to specify an encoding to further filter the categories results.
The ability to further filter the results to all color spaces, only display color spaces, or only non-display color spaces.

Document the set of Menu-based and Artist-based categories listed above as the recommended baseline set.

Dig it!
Here are some thoughts and ideas, increasingly out-of-scope:

I think the proposed algorithm for handling app-demanded and user-provided categories is great.

I also really like the idea of using the color space "encoding" property to get menu-based categories "for free." I suppose "encoding" values are expected (but not required?) to be one of: [data, scene-linear, display-linear, log, sdr-video, hdr-video, sdr-cinema, hdr-cinema] ...?

Any consideration given to sort order? How are merged / filtered color spaces presented? Are hierarchical menus preserved when presenting filtered color spaces? Should special consideration be given to colorspaces that are assigned to certain roles? (e.g., floating the color space assigned to the "color_timing" role to the top of a "look-process-space" menu?)

I would like the option to enable a standard "strict-sorting" method that forces families, followed by color spaces, to be listed in alphabetical order. (And I think it should be an option specified in the config, as opposed to an environment variable set at runtime).

Even though we have other ways of explicitly enabling / disabling color space visibility, I still think users would find it useful to be able to hide (from unfiltered menus) color spaces belonging to certain families. I was thinking an intuitive mechanism for this might be to filter any color spaces whose family string is prefixed with a period, or includes a separator delimiter immediately followed by a period. For instances, color spaces under the  ".DO_NOT_USE" and "foo/.bar" families would be hidden, but not necessarily color spaces directly under "foo" or "foo/mew" or what have you. Thoughts?

How are ViewTransform categories used?

As mentioned in the TSC meeting, I'd prefer something explicit over 'magic' hiding based on naming of the family string pattern matching some set of values.
Kevin

This proposal has now been implemented in PR #1220.  Thanks for the comments above, here are the responses based on the latest work:
Zach,

Yes you are correct about the encodings.
The sort order basically preserves the ordering of the config, please refer to the comments in OpenColorAppHelpers.h for more details.
Hierarchical menus are preserved by the filtering, but note that the application ultimately controls how they order and present the hierarchical menus since creating them is more complicated than simply iterating over a list.
The ViewTransform categories are not currently used but the idea was that they could ultimately be used to provide some additional filtering on Views, similar to what has been done for color spaces.

Mark, per your request, we have not added the proposed category strings listed above into the header files.  Currently they are only used in the unit tests as plausible real-world examples.

Closing this as fixed via PR #1220.

Closing this as fixed via PR #1220.


Documentation is broken

Just as a heads up, the link to the documentation on crates.io is broken.

I'll be pushing 0.3 version of crate to crates.io soon, where the docs all link to docs.rs now. Thanks for the reminder that I need to do that soon.

Crate.io apparently has a bug, the documentation link did not update when 0.3 was published. Crates.io at least shows the latest readme which has the correct links, just the main crates.io link is broken for now.

This looks resolved now


Publish camps index page

This issue has two-steps:

Migrate camps data from Google forms (DevOps)
Lilach started to collect camps data with Google forms, so she has a Doc format with all the current data. Technically it has to be store in Spark's database, mind to adapt the table column names.

Contact Lilach Givaty to get the data.
The data will be used in Camps Index page (in Midburn.org)

Serving camps_index page (Front-end)
Need to serve the static web-page from this repo: https://github.com/Midburn/camps_index
in oldsite.midburn.org - at first, later will be published in midburn.org

i got them

Done.
https://midburn-camps.firebaseapp.com/


agent: configure nvidia GPU access for the container

With nvidia-container-cli and nvidia driver installed into the rootfs of kata vm, if environment variable NVIDIA_VISIBLE_DEVICES is set in the OCI spec, the agent will configure GPU access for the container by leveraging nvidia-container-cli from project libnvidia-container.
Fixes: #735
Signed-off-by: Aohan Yang<EMAIL_ADDRESS>

Nice. It's quite a hardware specific change to the agent, but, I'm not sure we have any other alternative mechanism (like plugins) right now...
Let's see if we can track down some NVIDIA folks to pull in - maybe @guptaNswati  and @RenaudWasTaken ??

Hello!
A note on devices generally, we are working on CDI (https://github.com/RenaudWasTaken/cdi) with a few other runtime maintainers so that runtimes don't have to write vendor specific code. It seems that this might be useful here :)
Outside of that, on the actual PR, it seems like you might just want to run the nvidia-container-cli, or more likely the nvidia-container-toolkit, as a prestart hook since:

All the work mounting libraries, devices and executable is already handled by that binary
We do a bit more work around "preflight checks" ensuring that the container can actually run on that GPU
You'll get the latest features container features
It will give you a more consistent behavior with Docker / Podman / LXC

Outside of that (since CDI is not finished), on the actual PR, it seems like you might just want to run the nvidia-container-cli, or more likely the nvidia-container-toolkit, as a prestart hook since:

I have tried to add nvidia-container-runtime-hook as a prestart hook just like what nvidia-container-runtime did, but the container failed. As I am not really know how nvidia-container-runtime-hook works，I can‘t work out the error. At last, I took the approach in the PR.

Thanks @RenaudWasTaken. Actually, I have tried to add nvidia-container-runtime-hook as a prestart hook just like what nvidia-container-runtime did at first, but the container failed.

Happy to help this work! Do you have any error logs ?

@RenaudWasTaken
Here are the logs collected by running kata-collect-data.sh. Kata runtime and agent debug enabled. Hope it  helps.
out.txt

@jodh-intel
@RenaudWasTaken should be your point of contact now :)

Thanks @flx42 and @RenaudWasTaken.
@RyanAoh - if you haven't already found it, here's how you can run Kata "stand-alone" (without a  container manager):

https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#running-standalone

@RenaudWasTaken  I got the following error message. I think it could be helpful :
kata-gpu-test-1 containerd[5229]: time="2020-02-27T08:56:43.968868354-05:00" level=debug msg="reading guest console" ID=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 sandbox=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 source=virtcontainers subsystem=kata_agent vmconsole="time=\"2020-02-27T13:56:43.961858575Z\" level=debug msg=\"create container err: rpc error: code = Internal desc = Could not run process: container_linux.go:346: starting container process caused \\\"process_linux.go:449: container [init](url) caused \\\\\\\"process_linux.go:432: running prestart hook 0 caused \\\\\\\\\\\\\\\"error running hook: exit status 1, stdout: , stderr: could not open OCI spec: open /config.json: no such file or directory\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\"\\\\\\\"\\\"\" debug_console=false name=kata-agent pid=156 sandbox=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 source=agent"

@RenaudWasTaken  I got the following error message. I think it could be helpful :
kata-gpu-test-1 containerd[5229]: time="2020-02-27T08:56:43.968868354-05:00" level=debug msg="reading guest console" ID=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 sandbox=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 source=virtcontainers subsystem=kata_agent vmconsole="time=\"2020-02-27T13:56:43.961858575Z\" level=debug msg=\"create container err: rpc error: code = Internal desc = Could not run process: container_linux.go:346: starting container process caused \\\"process_linux.go:449: container [init](url) caused \\\\\\\"process_linux.go:432: running prestart hook 0 caused \\\\\\\\\\\\\\\"error running hook: exit status 1, stdout: , stderr: could not open OCI spec: open /config.json: no such file or directory\\\\\\\\\\\\\\\\n\\\\\\\\\\\\\\\"\\\\\\\"\\\"\" debug_console=false name=kata-agent pid=156 sandbox=282803f53f697766a9fbb31b46b45ac67f275366d29f3555b37d21df539a8fb8 source=agent"

Yep! That's a helpful error, looks like the hook state doesn't contain the path to the container bundle :)
We need to look into how the runtime invokes hooks :)

Yep! That's a helpful error, looks like the hook state doesn't contain the path to the container bundle :)

Thanks @RenaudWasTaken. I solved it refering to your reply. I missed the step to write the OCI spec to a file so that hooks can read it.(em...Since I am not very know how the runtime invokes hooks ).
Not, It can work well with  adding nvidia-container-runtime-hook as a prestart hook. Maybe  an update for this PR later.

Huh，with the agent working well with adding nvidia-container-runtime-hook. I got the the best way to configure nvidia GPU access for the container with no code changes.

config guest_hook_path = "/usr/share/oci/hooks" in the configuration file
locate "nvidia-container-toolkit" to "${ROOTFS_DIR}/usr/share/oci/hooks/prestart" (not nvidia-container-runtime-hook, since a symbolic is invalid)
Then the agent will add it. And the nvidia-container-toolkit will decide whether to configure nvidia GPU access by calling nvidia-container-cli  accord the env "NVIDIA_VISIBLE_DEVICES"

So, I think this PR and the issue #735 coud be closed. But maybe we need a detailed document to guide us how to use nvidia gpu device with kata containers.

Huh，with the agent working well with adding nvidia-container-runtime-hook. I got the the best way to configure nvidia GPU access for the container with no code changes.

config guest_hook_path = "/usr/share/oci/hooks" in the configuration file
locate "nvidia-container-toolkit" to "${ROOTFS_DIR}/usr/share/oci/hooks/prestart" (not nvidia-container-runtime-hook, since a symbolic is invalid)
Then the agent will add it. And the nvidia-container-toolkit will decide whether to configure nvidia GPU access by calling nvidia-container-cli  accord the env "NVIDIA_VISIBLE_DEVICES"

So, I think this PR and the issue #735 coud be closed. But maybe we need a detailed document to guide us how to use nvidia gpu device with kata containers.
Thanks everyone :)

Hi @RyanAoh, maybe you could help as I'm trying to get this working, and you're comment is very promising!
I cannot get the 2nd step to work - I added nvidia-container-toolkit to the root fs and rebuilt the kata image, set the hooks, but then when running the container the hook complains about missing nvidia-container-cli. So, I then copy all of nvidia-container-cli list --binaries and nvidia-container-cli list --libraries listed in your initial #704 comment, and then get the problem with libnvidia-container.so.1: cannot open shared object file: No such file or directory. Where would I run ldconfig without customising the agent like you did in this PR?

Where would I run ldconfig without customising the agent like you did in this PR?

emm, try to run ldconfig /path/to/your_copy_dest in the container before running nvidia-smi ?


Feature/issue 12 view competition entries

@miltontony review please

@miltontony forgot to ask for review, please review


Optimize SQL based on parameter nullability

This is part of #17543 - Queries really slow due to null checks
Problem was that at the time we performed null semantics and related optimizations we didn't know the value of parameters, so we always had to assume they can be nullable, which resulted in additional IS NULL checks being created.
However, later in the pipeline we know the parameter values and we can optimize out those checks for parameters whose values are not null.

depends on https://github.com/aspnet/EntityFrameworkCore/pull/18035


Update README.md to move OCR Project to Completed.

Moving OCR Project to Completed

@tulasiram58827 please resolve the conflicts.

@tulasiram58827 please resolve the conflicts.

Done.


Lower the default MacOS deployment target in the XCode project

The MacOS deployment target in the XCode project is set by default to 12.3. I may be wrong but from what I understand, a Mac on version 12.3 can run applications that target a lower version but it doesn't work the other way around. A Mac on version 11 can't run applications with a target version of 12.
I think we should target in our MacOS build the lowest version possible that supports the APIs we are using. I have tried running with a target of 10.9 (which is the lowest version proposed on XCode by default) and it seems to work alright.
This would make the configuration of the XCode IDE easier for new developers. The MacOS build would also support a wider range of devices.

Yes that makes sense. I just went with the defaults feom xcode. Lets just use the lowest shown by xcode. We probably can go even lower.
I think we only depend (implicitly) on UIKit/AppKit, Metal and QuartzCore.


[stemcell] network access denied; how to enable?

I want to create stemcells that pull down non-apt assets during some additional stages. Currently the rake stemcell:build task is blocking public internet access some how. If I add http_proxy=http://localhost:3142 then apt-get commands work (I assume from a local cache outside of the chroot).
How do I allow other http traffic to continue out to the internet?

Closing - stemcell execution environment has changed quite a bit since this was created and I doubt it's still an issue. Hopefully you're not needing to create custom stemcells anymore. If so, would like to hear why and if they can be done in a release instead of further stemcell sprawl; or feel free to create an issue in the new stemcell repo, cloudfoundry/bosh-linux-stemcell-builder.


Report the correct repo URL even if GitHub lies

Fixes https://github.com/fregante/github-url-detection/issues/53
Fixes https://github.com/sindresorhus/refined-github/issues/3743

I'll test it in the morning. It's crazy that they are inconsistent with everything lately!

GitHub entered the move fast break things phase

I am trying to find the place you said "when github switches to react this extension (RGH) is a goner"

Found it! https://github.com/sindresorhus/refined-github/pull/1159

That hasn't happened yet, but they've been switching to utility classes, which is almost as bad.


Avoid SSLError: Cannot create a client socket with a PROTOCOL_TLS_SERVER context

When we build mailman3 in Fedora with Python 3.10.0rc2,
we see the following problem:
Traceback (most recent call last):
  File "/builddir/build/BUILD/mailman-3.3.4/src/mailman/testing/layers.py", line 297, in setUp
    cls.smtpd.start()
  File "/builddir/build/BUILD/mailman-3.3.4/src/mailman/testing/mta.py", line 177, in start
    super().start()
  File "/usr/lib/python3.10/site-packages/aiosmtpd/controller.py", line 288, in start
    self._trigger_server()
  File "/usr/lib/python3.10/site-packages/aiosmtpd/controller.py", line 481, in _trigger_server
    InetMixin._trigger_server(self)
  File "/usr/lib/python3.10/site-packages/aiosmtpd/controller.py", line 428, in _trigger_server
    s = stk.enter_context(self.ssl_context.wrap_socket(s))
  File "/usr/lib64/python3.10/ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.10/ssl.py", line 1061, in _create
    self._sslobj = self._context._wrap_socket(
ssl.SSLError: Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)

This makes the problem go away.
Disclaimer: I have no idea what I'm doing here.

What do these changes do?
They check if the context we are about to wrap to is not server context.
Are there changes in behavior for the user?
I hope not.
Related issue number
Related to https://github.com/aio-libs/aiosmtpd/issues/277
Checklist

[x] I think the code is well written
[x] Unit tests for the changes exist (the original tests suite)
[ ] tox testenvs have been executed in the following environments:

[ ] Linux (Ubuntu 18.04, Ubuntu 20.04, Arch): {py36,py37,py38,py39}-{nocov,cov,diffcov}, qa, docs
[ ] Windows (7, 10): {py36,py37,py38,py39}-{nocov,cov,diffcov}
[ ] WSL 1.0 (Ubuntu 18.04): {py36,py37,py38,py39}-{nocov,cov,diffcov}, pypy3-{nocov,cov}, qa, docs
[ ] FreeBSD (12.2, 12.1, 11.4): {py36,pypy3}-{nocov,cov,diffcov}, qa
[ ] Cygwin: py36-{nocov,cov,diffcov}, qa, docs

[ ] Documentation reflects the changes
[ ] Add a news fragment into the NEWS.rst file

Add under the "aiosmtpd-next" section, creating one if necessary

You may create subsections to group the changes, if you like

Use full sentences with correct case and punctuation
Refer to relevant Issue if applicable

I am sorry but I won't be able to finish this checklist. Maybe my change is good, maybe not, but it helped me in mailman3 in Fedora.

I'm wondering if this is an issue in mailman3 - if they incorrectly created an SSL context - or used the wrong SSL context.

That is certainly possible.
In fact, before this patch:
 = 3 failed, 531 passed, 2 skipped, 126 warnings, 27 errors in 186.11s (0:03:06) =

After this patch:
=========== 3 failed, 558 passed, 2 skipped, 126 warnings in 55.14s ============

@hroncok I'm not positive, but based on this comment I don't think this is necessary.
Would you be able to take a look this week? We're aiming to get v1.5 released ~ mid/late December with some of these assorted fixes, especially the ones that enable support for the newer versions of Python.

Would you be able to take a look this week?

Sorry, I will not be able to look into this.

No worries, thanks for letting me know!

@hroncok can you verify that 1.4.3 solves this problem?
If so we can close this PR without merging.

Looks like we still patch this in Fedora by using this PR. I can try removing the patch and updating the package early January.

Looks like we still patch this in Fedora by using this PR. I can try removing the patch and updating the package early January.

Okay.
I'm close to releasing 1.4.4 though, that one will fix some DeprecationWarnings.
So in January, if possible tell us if 1.4.4 is good enough, and if it is, we can close this without merging.

I can no longer reproduce the problem with 1.4.3 and recent Python 3.10.

I can no longer reproduce the problem with 1.4.3 and recent Python 3.10.

Awesome!
Thanks for verifying 👍🏼


Remove Download option from Voicemail Task

Remove the Download option from Voice mail tasks audio player (click 3 dots menu).
Allowing agents to download a recording with potentially sensitive info may be a concern for some customers.

One thing to consider, even if we remove the option, the user can still download the recording if they know how to use the browser developer tools.


pythonPackages.pygments: 2.3.1 -> 2.4.2

Motivation for this change
Update to latest Pygments.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @FRidh

Thanks. Pushed to staging.


Geom as flatgeobuf from database

Enhanced list action (the one responding to GET at /items...) so that it can be called with either of the following parameters:

geom_as=wkb: returning items with wkb-encoded geometry instead of json (the default)
only_geom_as=fgb (FlatGeobuffer): downloading the geometry (and nothing else) of the items filtered in the current queryset.

As most implementions of FlatGeobuffer, the one used here (PostGIS') does not support encoding to FlatGeobuffer heterogenous containers, such as a mix of geometric and non-geometric fields, unless one goes through the extra step of providing a specification and compiling.

@3nids So we have this data:

which I try to transform with:
query = """
    WITH rows AS (SELECT ST_GeomFromText(geom, %s) FROM signalo_roads_road WHERE id IN %s)
    SELECT encode(ST_AsFlatGeobuf(rows), 'base64') FROM rows
"""
cur.execute(
    query,
    (
        GEOMETRY_SRID,
        pks,
    ),
)

resulting in this error:
ERROR: test_geom_fgb (signalo.roads.tests.TestRoads)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.InternalError_: parse error - invalid geometry
HINT:  "01" <-- parse error at position 2 within geometry

Not sure where the mistake is.

Superseded by #89.


Allow search for EC2 AMIs

Would be nice if some official AMIs were easily available, so that usage instructions for simple use-cases would not involve of "look up ami names on community site" step :)
Related:

https://github.com/test-kitchen/kitchen-ec2/blob/master/data/amis.json
https://github.com/test-kitchen/kitchen-ec2/issues/117

Or perhaps simplest would be to allow a user to drop a amis.json or amis.yml file into the .chef/ dir where the knife.rb exists.

@patcon
According to this reference https://github.com/test-kitchen/kitchen-ec2/issues/117#issuecomment-103604021 , adding the .json file won't help as aws changes AMI's Id.
Probably we can add new option something like --image-name in knife-ec2  which will take image name as a value. Example- "centos-6.4" and internally knife-ec2 will find corresponding "ami-id" for specified image.

Yes, this would be an AMI search like what was implemented in test-kitchen here: https://github.com/test-kitchen/kitchen-ec2/pull/17

The MVP of this was released in knife-ec2 0.15. There's more work to do still in the next round.

@btm , can this issue be closed because the basic requirement is fulfilled in knife-ec2 0.15.0?


function object not valid to initialize analytic expression

I have encountered the following situation：
function object not valid to initialize analytic expression
why ？
I see that the number of sql function parameters can not be greater than 3.
But Why?

Is there a function for analytic purposes with more than three parameters? Is it possible to use any function here?

Can you give an example of a valid SQL using there a function with more than three arguments?

So can we close this issue?


Don't use list_on_cluster to gather issues

The cartridge_issues.list_on_cluster function gathers issues from entire cluster (obviously). The same does prometheus. It results in N^2 network requests. Prometheus queries every instances metrics and for every request, list_on_cluster does N more.
I think the metrics module should collect local issues only.

I suggest to gather only local issues in this metric and add new metric e.g. cluster_issues and API to enable cluster issues only. It could be called on instances with special role (e.g. router)

After some discussion we decided not to change this metric behavior and add function to disable that metric.

Will you leave it enabled by default?

After some discussion we decided not to change this metric behaviour and add function to disable that metric.

I just share some details that we recently faced when try to deploy huge cluster.
There where several hundreds instances on one physical hosts. So one moment each Tarantool instance had established connections to each instance in our cluster. The problem is that amount of ports is limited. Finally, we lost an ability to connect to our host and there are some problems when Tarantools try to connect to each other (e.g. for replication).
Is it our official position to consume all available ports? @vasiliy-t @ZolotarevIgor-mail-ru I suggest to reconsider your opinion about this issue. Actually it's a bug, not "refactoring".


Add S4TF crasher test

Regression test for #59467.

Implemented in https://github.com/apple/swift/pull/60158


most.create in chain causes later publisher call

Recently timing of publisher call in most.create bothers me a bit. Following discussions in #72, I created an experiment to illustrate the problem I have:
var most = require('most');

let push = () => {};

let s = most.just().chain(() => most.create(add => push = add)); // print only 2
// let s = most.create(add => push = add); // print 1, 2

var tapped = s.tap(x => console.log(x));

tapped.drain();
push(1);

setTimeout(() => push(2), 0);

If I understand correctly the versions with and without chain should be equal in semantics. The current situation is a bit problematic for me because if I use chain, I will miss some events without the ugly setTimeout.

@yarray I appreciate that you provided a simplified example.  However, calling push right after calling drain isn't valid. Most.js never delivers an event on a stream in the same call stack as drain/observe/forEach is called, just as promises never invoke callbacks within the same call stack as resolve or then is called.  So the function passed to chain will never be called in the same call stack as drain.
In general, writing code outside of a stream that implicitly depends on the timing of events inside a stream is asking for trouble, just as any implicit async timing dependency is asking for trouble.  It might work by accident in some cases, but timing dependencies really should be expressed explicitly using stream combinators.
Could you explain a bit more about what your real scenario is so we can try to help?  Thanks!

@briancavalier I seem to understand now...  So code like
setTimeout(() => push(2), 0);

is in fact not reliable either?
I currently investigate to combine virtual dom with most.js. The hard thing is how to create event stream from dynamically added nodes. Virtual dom libraries like snabbdom often provide ways to specify event handlers. However it cannot be transferred easily to stream with fromEvent. Previously I want to do something like that:
const clickStream = vnodeStream.map(vnode => {
    return most.create(add => {
        vnode.hook.create = (_, newNode) => {
            newNode.elm.onclick = add;
        };
    });
}).join();

clickStream.observe(e => {
    // do something
});

vnodeStream.reduce(patch);

The code is quite weird and with your explanation now I can tell that the weirdness comes from timing.
I've seen your TodoMVC example and it seems a possible way is to take advantage of event bubbling and listen all events on a static ancestor element. Do you think this approach is the best or perhaps the only correct way to deal with this situation?
Thanks for your reply and this awesome library.

@yarray as a user of both most and snabbdom i would have to say yes that using event-delegation is ideal in the situation you present above if i understand what you are trying to achieve.. that essentially could the best way to do that regardless if you attach event listeners natively (addEventListener) or with something like fromEvent in most just my $0.02

@davidchase Thanks for your suggestion. The approach worked until I came across some events that do not bubble, like "focus" and "blur". There are bubbling alternatives for them but the browser support is not that good. So currently there seems no perfect solution yet...

@yarray you are correct focus and blur need to be captured the opposite of bubbling and i believe the support for focusin and focusout is everywhere but mozilla as they observe some type of bug in the implementation and therefore can be poly-filled with a CustomEvent quite easily if needed

you could also use most.create as @briancavalier suggested to manually listen to non-bubbling events
something like:
var captureEventStream = function(event, source){
  return most.create(function(add, end, error) {
     source.addEventListener(event, add, true);
      return function() {
          source.removeEventListener(event, add, true);
      }
  });
};

@davidchase Thanks for your information especially the CustomEvent part. Also captureEventStream seems good but I am still not sure how to apply this method to virtual dom. Generally we can only use hook to access DOM element so it seems that we still cannot use the captured stream in proper scopes. Do you have some solutions for that?
By the way I feel the approach way too complicated. Though I also agree that push is dangerous for FRP, as discussed in #72,  it seems to be the most elegant way to handle this situation, at least for snabbdom.

vnode.hook.create = (_, newNode) => {
newNode.elm.onclick = add;
};

Im trying to understand your example above, but in snabbdom isnt it true you have to create the hooks objects within the hyperscript (h) in the above example i think hook would be undefined therefore it cant be done like that, no?
So we are on the same page when are you trying to add listeners (iow create the fromEvent stream) on the nodes? before you call patch(container) ?

@davidchase Yes it will not work as you observed. The example is given just to illustrate the hardness when trying to combine most's publisher and virtual dom libraries like snabbdom. As for the timing, I have not found a perfect place to attach the publisher to the hook of virtual dom. Could you please offer a more complete example if you have some successful experiences with captureEventStream ?

I guess im not quite following the issue... instead of using hooks why not listen to a root node and filter until you get your newly adding elements after you've patched the dom?
Also it maybe easier if you want to chat some more on gitter about your situation to see if i can provide some help.

@davidchase Listening to a root node is possible (and perhaps with polyfill for focuson) is possible. However it seems a little unnecessarily complex for me (it will quickly become tedious to check what happens on the root node when the DOM hierarchy becomes more complex). Also this way the event stream will be in a callback, which may bring some inconvenience. I'd say that I prefer the approach of Elm to bind streams directly to virtual dom.
Many thanks for your suggestions about gitter. Unfortunately I am quite busy these days... Hope I can find some time and investigate more on this topic later.

@yarray I think the remaining question here is about having an imperative adapter for virtual dom.  You can use most-subject.
Closing, but if you have more questions, please ask here or in gitter.  Thanks.


There is no data/OpticalFlow.cpp file in this repository

As the title, the file didn't be put on the repository.
I saw the comment on README.md. The modification of the file is about removing logs spamming to console.
I try to remove them by myself. However, I found there are some error logs during training phase.
"All the pixels are invalid in estimation Laplacian noise!!!
Something severely wrong happened!!!"
Are these error logs OK for this project?
Thank you for taking the time to read this issue.

Hello, thanks for the notice, we added .cpp file and updated README.
This error is OK, it occurs when some data regions are filled with black pixels


Allow creation of apps without tagging them

Checklist

[ ] tests added
[x] commits conform to the Commit message format
[ ] documentation updated

Squash commit message
Allow creation of apps without tagging them. The documentation does not need to be updated since it already implies this is possible.

Create an issue: https://github.com/qlik-trial/gopherciser/issues :P

Created an issue regarding the missing feature.  https://github.com/qlik-trial/gopherciser/issues/608


Update apiDefinition.swagger.json

Please check the following conditions for your PR.

[ ] apiDefinition.swagger.json is validated using paconn validate command.
[ ] apiProperties.json has a valid brand color. Invalid brand colors are #007ee5 and #ffffff.

[[certifycommunityconnector]]

test

[[certifycommunityconnector]] 1

[[certify-connector]]


vignettes(cellosaurus): describe annotation steps

need to explain how to annotate to other fields:
example from the gdsc pipeline
field_map <- list(
    "id" = "id",
    "ac" = "accession",
    "sy" = "synonyms",
    "misspelling" = "misspellings",
    "di" = "diseases",
    "ca" = "category",
    "sx" = "sexOfCell",
    "ag" = "ageAtSampling",
    "derived-from-site" = "samplingSite"
)

fields <- names(field_map)

sampleMetadata[, paste0("cellosaurus.", fields) := {
    mapped <- AnnotationGx::mapCell2Accession(
        as.character(GDSC.COSMIC_ID),
        from = "dr",
        to = fields,
        BPPARAM = BPPARAM
    )
    return(mapped[, 1:length(fields)])
    }

also how to map from something other than "name"


vimPlugins.lspecho-nvim: init at 2024-10-06

lspecho-nvim is a small Neovim plugin to show the LSP progress in the message bar, released under the MIT license.
Project homepage

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 2 packages built:

vimPlugins.lspecho-nvim
vimPluginsUpdater


Updated Pinata API to newest functions and IPFSSpec removed sys.append from top of notebook

This may throw an error because credentials are need for the Pinata API

Failed on check


Random closing of connection

Report from user:
"I started implementing TLS1.1 and TLS 1.2 and I am getting some unexpected behavior from the socket.  The socket will just disconnect randomly without an error.  I have tried it on your demo program and gotten the same results when I use a server port of 20000."

There is a socket poll that tries to determine if the the socket is still open, and if it detects that it is closed - if closes itself.  Apparently there is some sort of false positive tricking it into closing when it shouldn't.
I've updated the polling mechanism to make sure that it checks several times before it decides that it is really closed.


[Feature] Support reading from petrel in UnconditionalImageDataset

Support selecting petrel as backend in UnconditionalImageDataset.

@plyfager This pull request has been approved, please consider merging it.


[Bug]: Zitadel account auto creation not working using Okta as an OIDC provider

Preflight Checklist

[X] I could not find a solution in the documentation, the existing issues or discussions
[X] I have joined the ZITADEL chat

Environment
Self-hosted
Version
v2.42.1
Database
CockroachDB
Database Version
v23.2
Describe the problem caused by this bug
Currently we are trying to implement Okta within Zitadel. We already configured Okta as an authentication option in zitadel at the organization level.
We configured the OIDC provider to automatically create an user with the options in the first screenshot.
Also in "Login Behavior and Security" the option "Username Password allowed" was unchecked, and checked the "Disable login with email address", so that causes that, when loading the zitadel authorized link, it takes us to the Okta login directly (which is nice).
We authenticate in Okta without issues; however then, instead of landing into our application (the URL configured as callback), we are taken into a zitadel screen that just says "External User not found" (2nd screenshot).
It seems like auto creation feature is not working.
If I enable "Account creation allowed" I'm able to create an account, but that's not what we want. Also, it is weird that, having to create a zitadel account to link it to an external account, you are asked to verify the email, and being asked to enable MFA when you already have that on okta side, and verify the email address.
To reproduce

Configure Okta as an OIDC provider using these instructions: https://zitadel.com/docs/guides/integrate/identity-providers/okta
Enabling the following checkboxes for the OIDC provider:

Automatic Creation
Automatic Update

Enable the OIDC provider
In "Login Behavior and Security":

Uncheck "Username Password allowed"
Check "Disable login with email address"

Navigate to URL https://[your-domain]/oauth/v2/authorize?client_id=243656644137910285%40some_pry&redirect_uri=http%3A%2F%2Flocalhost%3A3005%2Fcallback&response_type=code&scope=openid+profile+email+urn%3Azitadel%3Aiam%3Aorg%3Aproject%3Aid%3A243656593084776461%3Aaud&state=65f27fb8cd234dc992242a4f328cbb15&code_challenge=__16J2R6S_c6maMfpRxeUtPmZil7X1Gi2M8GlrlIch8&code_challenge_method=S256&response_mode=query using your instance parameters
Authenticate with Okta
Zitadel user creation page loads instead of creating user automatically

Screenshots

Expected behavior
To get the zitadel user account automatically created, and right after the okta login, get redirected to the configured redirect_uri
Operating System
Device: Mac
OS: Sonoma 14.1.1
Browser: Firefox
Version: 120.0.1
Relevant Configuration
No response
Additional Context
No response

Regarding third bullet already found out what fields I do need to populate, using zitadel logger prints

Hei @oscarcalderonc
You are absolutely right, I wasn't actually fully aware that those infos are not sent.
Will definitely improve the docs.
If you do have an example action that works, and fills the information correctly I would also be glad to get an example. Otherwise will figure out how to do it. But can't promise an eta yet. :)

Sure! I ended up using zitadel logger to print the information coming from Okta and based on that I made this simple script:
let logger = require("zitadel/log")

function mapOktaOauth(ctx, api) {
  logger.log('Populating extra information for new Okta user');

  if (ctx.v1.externalUser.externalIdpId != "243980621926367243") {
    return
  }
  api.setFirstName(ctx.v1.providerInfo.name);
  api.setLastName(ctx.v1.providerInfo.name);
  api.setEmailVerified(ctx.v1.providerInfo.email)
  api.setEmail(ctx.v1.providerInfo.email)
  api.setPreferredUsername(ctx.v1.providerInfo.preferred_username)
}

I have added your action to the actions repository: https://github.com/zitadel/actions/pull/25
And extended the docs: https://github.com/zitadel/zitadel/pull/7060
Thanks for your help.


Polyglot syntax highlighting not working?

Syntax highlighting works great for Clojure, I read in the book that it should work for other languages too. (but probably ::clerk/opts should be  ::clerk/render-opts)
Unfortunately, when I tried to use this feature, I could only get it to work for Forth, which is very strange. Maybe I'm doing something wrong?
Any help would be appreciated
Clerk version: io.github.nextjournal/clerk {:mvn/version "0.17.1102"}
Clojure version org.clojure/clojure {:mvn/version "1.11.1"}
I'm using simple md notebook with code blocks
def hello():
  print("world");

And I see in the network tab that relevant parsers are loading.

Interestingly syntactic tree for python, c++, js, etc looks very simple compared to clojure, so i guess smth is wrong here.

I think there's 2 issues here:

as observed by @romanatexn options should be passed as ::clerk/render-opts, I thought we normalise from the older version but we no longer do that.
even with that in place, our codemirror highlighting has been messed up with some dependency update. I used the markdown fences notebook to bisect where language options are passed down correctly, last good is dc51320f and first bad is 93e21941 (93e21941642509a598cee8c71ec58f0634667672).

That said, we're importing code mirror language-data dynamically and that's already the latest version available. So I'll keep checking other options.

In fact https://github.com/nextjournal/clerk/commit/93e21941642509a598cee8c71ec58f0634667672#diff-cc47688828e6345582744b8a90a5b22ee617035b69c25d7861f47499ab386793 bumped language-data. The previous version just cannot be restored because is no longer compatible with the codemirror stack we have now...


Merge aws-resolver-rules-operator & dns-aws-operator

Check if we want to use a command line argument or the annotation aws.giantswarm.io/resolver-rules-owner-account (which is currently used in cluster-aws and aws-resolver-rules-operator), and remove one of them

Done in https://github.com/giantswarm/aws-resolver-rules-operator/releases/tag/v0.7.0


Moveable Party Frames

Alt+Shift+Drag when out of combat. Also a small fix for threeDPortraits.
threeDPortraits is still a bit off but this is better than it was.

Thanks


ERROR Failed to read certificate: EISDIR: illegal operation on a directory, read

code-server version: 1.604-vsc1.32.0
OS version: Debian GNU/Linux 9 (stretch)

Description
I try to secure my setup but I got the following problem :
ERROR Failed to read certificate: EISDIR: illegal operation on a directory, read
Steps to Reproduce

wget https://github.com/codercom/code-server/releases/download/1.604-vsc1.32.0/code-server1.604-vsc1.32.0-linux-x64.tar.gz
tar xvfz code-server1.604-vsc1.32.0-linux-x64.tar.gz
cd code-server1.604-vsc1.32.0-linux-x64
mv code-server /bin
mkdir ~/certs && cd ~/certs
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ~/certs/MyKey.key -out ~/certs/MyCertificate.crt
code-server --cert=~/certs/MyCertificate.crt --cert-key=~/certs/MyKey.key

I have the same issue.
And I have solved this by using Relative Path.
// $pwd = /home/ubuntu
// MyCertificate.crt and MKey.key are in the /home/ubuntu/certs
./code-server ~/your_project_path --cert=certs/MyCertificate.crt --cert-key=certs/MyKey.key

If you use spaces it should work.
code-server --cert ~/certs/MyCertificate.crt --cert-key ~/certs/MyKey.key

Without those your shell won't expand ~ so it'll be looking for a directory literally called ~.


Missing implementation for arm_fir_decimate_f64

Currently, there is an implementation for arm_fir_f64, but there is no equivalent arm_fir_decimate_f64 as there are with the f32 variants of the same functions.
I could provide a reference implementation PR based on arm_fir_decimate_f32 if desired (removing any f32 intrinsic optimizations)

@torfinnberset Thanks. Don't hesitate to submit PR.
Note that currently I won't be able to quickly test / merge your PR because I am working on other things. But I'll do as soon as I can find some bandwidth.


GtRemoteRunner: keep status in an object that doesn't require mutex access

Requesting status from headless remote runners is slow due to multiple calls and allowing for possible mutex waits.  Move the status to a separate object that is kept up to date by the runner and doesn't require mutex access.


app dev: add middleware so frontend doesn't query database directly

Issue was opened while in development, closing for now while project is in maintenance mode.


docs: update api docs

Update Location Serializer to include location detail api url.
Deletes unnessassry DestinationMinimumSerializer.
Update Destination API view to only inculde destinations data.
Add markdown for api docs.

Hello @awaisdar001! Thanks for opening this PR.


[BUG] random error when downloading media

Describe the bug
Sometimes, some connections cant download one file on "messages.upsert"
error: Error: error:1C800064:Provider routines::bad decrypt
      at Decipheriv.final (node:internal/crypto/cipher:199:29)
      at Transform.final [as _final] (/app/node_modules/@whiskeysockets/baileys/lib/Utils/messages-media.js:501:31)
      at callFinal (node:internal/streams/writable:698:12)
      at prefinish (node:internal/streams/writable:710:7)
      at finishMaybe (node:internal/streams/writable:720:5)
      at Writable.end (node:internal/streams/writable:634:5)
      at IncomingMessage.onend (node:internal/streams/readable:748:10)
      at Object.onceWrapper (node:events:631:28)
      at IncomingMessage.emit (node:events:529:35)
      at endReadableNT (node:internal/streams/readable:1400:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    library: 'Provider routines',
    reason: 'bad decrypt',
    code: 'ERR_OSSL_BAD_DECRYPT'
  },

To Reproduce
There is no steps, it occurs randomly
Expected behavior
A clear and concise description of what you expected to happen.
Environment (please complete the following information):

Is this on a server? YES, azure
What do your connectOptions look like?

makeWASocket({
           logger: loggerBaileys,
           printQRInTerminal: false,
           auth: state as AuthenticationState,
           version,
           msgRetryCounterCache,
           browser: Browsers.macOS("My app"),
           shouldIgnoreJid: jid =>
             isJidBroadcast(jid) || isJidStatusBroadcast(jid),
           getMessage: async key => {
             const msg = await GetWbotMessage(key.id || "");
             return msg?.message ?? undefined;
           }
         })

I'm not using store

Do you have multiple clients on the same IP? YES
Are you using a proxy? NO

Additional context
i do something like:
try {
media = await downloadMediaMessage();
} catch {
media = await downloadContentFromMessage()
}
` ``

@PurpShell any ideia why i'm having this error? maybe i've seen other people report it a long time ago, but no fixes yet

error: 2024-08-06T14:21:38.763Z Error: error:1C800064:Provider routines::bad decrypt
still occuring...


UI improvements

Working with @parhamin2010 and @mirekys

I tried to check the PR and meanwhile a critical point is understood. there is no invitation-accept process designed in the new UI.
we can generate the invitation token but there is no textbox or input to paste the token and finalize the connection.

there is no textbox or input to paste the token and finalize the connection

Correct. Well, the page still exists but there no longer is a navigation menu item pointing to it.
This is because the only valid way to accept an invite is through https://sciencemesh.cesnet.cz/iop/meshdir?token=asdf&providerDomain=cern.ch
This redirects to https://cloud.pondersource.org/index.php/apps/sciencemesh/accept?token=asdf&providerDomain=cern.ch
And there, we concluded that the "user@host" input box is useless, it's too complex for the user, it should be a hidden form field, automatically populated from the URL query string.

This was a design decision from our design brainstorm session with @mirekys

Like this: https://files.gitter.im/634ea5646da03739849e00df/HxUX/Screenshot-2023-02-03-at-15.11.31.jpg

So the current UI is not similar to the image that you attached.

Last commit


Working 7413

Compiling and working. Thank you for your awesome work on this tool!

@cleydyr Thank you for fixing this, merged! There are still frontend bugs that I'm working on, but struggling to fix them as JQuery is gone, frontend structure is more React friendly, so I'm replacing JQuery methods to bare Javascript, but some of them do not work somehow...


Two minor CMake changes

Use the default CMake install directories from GNUInstallDirs
Try to use the system pugixml first if it exists, compile in the bundled pugixml otherwise

I've added the pugixml folder to the qmake include path so now it builds again.


Many fields in the current IFF form are marked required in the editor interface but on the public application form they are not required.

See the different between https://apply.opentech.fund/internet-freedom-fund/ and the edit page for that application. I can apply with very little information.

I believe it works, if one edit the correct application form. Look for the one used by the current round to see the changes.
Best to edit all the instances of the forms so it becomes correct the next round as well.
The single checkboxes lack a "required" box but there is a pr under testing that fixes that.

Totally not an issue, closing. Now wondering if when an application is copied the required is also copied. Will test and create a ticket if issue arises.


Add child-parent relations for STAC records

The collections are showing up on the metadata details page. STAC records are all at the item level on the metadata page, it would be nice to populate the collection-level information for STAC records when translating it to GeoCore format.
Example on metadata page: an example here https://app-dev.geo.ca/result?id=085024ac-5a48-427a-a2ea-d62af73f2142&lang=en.
Related GeoCore:https://geocore-dev.metadata.geo.ca/085024ac-5a48-427a-a2ea-d62af73f2142.geojson
@bo-lu help wanted from you. I know the GeoCore parentIdentifier field can be used to identify child-parent, does it require a specific format?

I think this is complete.

Yes, this is completed. Codes are committed to dev and are merged into the main brunch. Code related to this issue is in this commit: https://github.com/Canadian-Geospatial-Platform/stac_to_geocore/commit/082ce78747f15e7b69915a72e889e563b509bc79


Missing values on the UI but present in the code

Describe the Bug
I have created a workflow using VSCode where I am calling another workflow hosted in the same logic app and passing the required parameters. When the code is published to Azure, I see the call workflow action but the parameters go missing on the UI but it's seen in the underlying code.
Steps to Reproduce the Bug or Issue

Create two logic app standard workflows in VSCode
call one workflow - "exceptions handler" and allow the HTTP trigger to accept input parameters.
from the 2nd workflow, call the exception handler workflow and pass some values as parameters.
publish the code to Azure DevOps --> deploy to --> Azure Logic App Standard
open the 2nd workflow, go to the call-workflow action and see if the parameters are still shown.

Workflow JSON
No response
Expected behavior
Whatever I see on VSCode UI, I should be able to see the exact same thing in Azure workflow UI
Screenshots or Videos
No response
Platform And Browser

OS: [Windows]
Browser: [Edge]

Additional context
No response
AB#17409527

@Eric-B-Wu: The workflow was created on the Visual Studio code designer (local) and everything looks fine there. When the code is published to Azure and I go to the portal to see it, I notice the parameters and the values in the Call workflow actions have disappeared.

@mhireholi - I was able to successfully deploy the parent-child workflow from vscode to Azure and view the same parameters in Azure portal. There might be some issues in the vscode extension in not showing all parameters for child workflow correctly today and that should be fixed in coming days.
But I do not see the parameters not appearing in portal issue yet.
Can you try this with latest extension and show us some screenshots both in vscode UI and portal?

Closing this issue since it is not reproducible


ZeroDivisionError

When I launch the program, I've got this error :
"  File "D:*\deepmusic\composer.py", line 255, in _main_train
next_batch_test = batches_test[self.glob_step % len(batches_test)]  # Generate test batches in a cycling way (test set smaller than train set)
ZeroDivisionError: integer division or modulo by zero  "
I've tried with others midi and with the included one but i still get the error even if i got 4 train and 1 test.
I don't see where it come from...
Thanks in advance.

pls someone answer ..i have the same problem as well

i fixed it ..with help ...somewehere above that error it propaply writes down:
Restoring dataset from C:\data/samples\ragtimemusic-relative.pkl...
just  delete ragtimemusic-relative.pkl in the path it says and run again =D

haha its me again...for some reason this worked and for other reasons it stopped working...idk whats going on ...actually u have to change the batchsize .... try running            main.py --batch_size 1   ... and see if it works ..and when it those ..choose a batch size that is smaller than the number of midis u have in the program ...like if u have 1000 choose 50 ..or so not 1...from what i understood one is not good..i think. and the normal patchsize of the program is 64.. idk why it worked before ..but its working for me again ...and if it finds no model ...make an empty file model.ckpt   ..in ur save/model folder.

Creating the model.ckpt file leads to some music being generated. But what does this mean? Is it learning from the files that have been used for training?


bug #4276: Fix snap_create for qcow2

Previous snapshots should be looked up on the remote host.

issue was already resolved in commit c1a9c0b1233a6e3aae66424ff3bf969db649a3ef


Failed to load shared library

I looked at this similar issue, but the cause of the error is different for me.
OS: Amazon Linux
Version: Latest version of llama_cpp_python
Commands / code / output:
$ python
Python 3.9.16 (main, Feb 23 2023, 00:00:00)
[GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import llama_cpp
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/llama_cpp.py", line 67, in _load_shared_library
    return ctypes.CDLL(str(_lib_path), winmode=0)
  File "/usr/lib64/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/libllama.so: undefined symbol: _ZTVN10__cxxabiv120__si_class_type_infoE

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "/home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/llama_cpp.py", line 80, in <module>
    _lib = _load_shared_library(_lib_base_name)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/llama_cpp.py", line 69, in _load_shared_library
    raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library '/home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/libllama.so': /home/ec2-user/.local/lib/python3.9/site-packages/llama_cpp/libllama.so: undefined symbol: _ZTVN10__cxxabiv120__si_class_type_infoE

I figured out the issue rather quickly. I had an env variable called CXX set while installing the wheel, and I had to unset CXX
Maybe there is a way to fix this for others? I feel like the installation shouldn't break just because of having that env variable set 😊

Glad you fixed your compile environment. Can you close this issue?


Update apicall.js

Routing the call to emi rates and get the data to be sent to caller

I have merged your pull request.


Mismatch between Validator and Vizarr viewer

Hi ,
I just checked out the latest https://deploy-preview-53--ngff-challenge.netlify.app/
and realized the NGFF-validator seems perfectly fine for my Lightsheet image but the proposed Vizarr viewer throws an error: Error: server replied with "Group is missing multiscales specification." when loading the resource
When I add /0 at the end, it works fine (except the fucked up B&C settings for the channels 😄 )
Is there something I can add/change to make it work properly?

Ah - Thanks for the feedback. Hmm - I realise I don't have the information stored in the CSV to know whether to add a /0 to the URL for vizarr. I actually look that up for each zarr as part of the Thumbnail generation, so I could use that to fix the link. Alternatively I could add into the csv the path-to-image, which would improve performance of thumbnail loading.
I'll look into it...

Adding the path-to-image seems tricky -- in this case it works because there's only one image, but then you lose the RO-Crate, no?

I guess what I mean is adding a relative path/to/thumbnail/zarr.json and not changing the original url/to/image.zarr.
This would also fix the thumbnail for images where the bioformats2raw series is not /0. I'm currently making the assumption of /0  for thumbnails (so I don't have to load the XML just to render a thumbnail) but it breaks like https://deploy-preview-36--ome-ngff-validator.netlify.app/?source=https://radosgw.public.os.wwu.de/n4bi-goe/Platynereis-H2B-TL.ome.zarr

I guess what I mean is adding a relative path/to/thumbnail/zarr.json

Gotcha.


docs: missing text elements break  section

The engine has a feature to automatically make <details> sections into expandable boxes e.g.

however when there's no text paragraphs or ### md headers inside <details></details>, the result is

even if there's other contents inside the details section, for example a fenced code block.

it's expected behavior. ### defined the name for the expandable section

Yes I understand. I just think it should just not crate the expandable section when it doesn't find any text to use for the tittle, instead of causing an error. But OK!


Update section "Should I use hashHistory?"

Added some much needed information about no-no's about using hashHistories. Although much are left behind, such as hashes are good for single-page applications, this is not something that we should comment on(in my eyes).

Moving this to react-router-tutorial section.

That's not really a great summary of the browser support story, and it doesn't really touch on the SSR thing.
Thanks for drawing our attention to this section, though; we will make a note to update it.

Great, thanks for the feedback!


Configure support tools for backend

(Mostly) Python:

pre-commit
ruff
poetry
pytest
pytest-cov
codecov?
mkdocs

Part of #3

We need to somehow merge poetry expectations for project with how FastAPI backends work.
Something alike mv app backend/pec_framework_web_app so no longer backend is stored in app directory. See poetry documentation.
poetry add<EMAIL_ADDRESS><EMAIL_ADDRESS>
etc.
git rm requirements.txt

Merge this stuff ^^^^ with general python project template.
To make this stuff build
feature/pec-framework-web-app-backend

Ctrl + Shift + F -> "pec_framework_data_consumer"
poetry shell

poetry install

poe run-unit-tests

poe run-type-checks

poe install-hooks

run-code-quality-checks

mkdocs build

poetry build

FROM python:3.11.6-slim

WORKDIR /code

ENV PIP_DISABLE_PIP_VERSION_CHECK 1

Please add:

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

#COPY ./backend /code/backend
#COPY ./pyproject.toml /code/

Don't leave comments in code.
COPY . /code/

RUN rm docker*
RUN rm Docker*

Use .dockerignore instead.
RUN apt-get -y update
RUN apt-get -y install git

RUN pip install --user poetry

To reduce image size and align with how things are supposed to be used, instead use requirements.txt file generated with poetry export --ouput "requirements.txt"
ENV PATH="/root/.local/bin:${PATH}"

RUN poetry config virtualenvs.create false
RUN poetry install --no-interaction --no-ansi

WORKDIR /code/backend

#COPY ./requirements.txt /code/requirements.txt

#RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

Comments again.
Make sure that changes done in source code are reflected in container content.


Smart Water Management

Logic Legends Project Submission

Closed as requested changes were not applied; happy to review again with changes made in future if you wish to re-open PR.


test: add multiple environments

Fixes https://github.com/anonrig/fast-querystring/issues/41

Codecov Report

Merging #45 (9b18762) into main (3063aad) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 9b18762 differs from pull request most recent head 9ca43eb. Consider uploading reports for the commit 9ca43eb to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff            @@
##              main       #45   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines          311       311
  Branches        74        74
=========================================
  Hits           311       311

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


API_AUDIENCE example is incorrect.

API_AUDIENCE variable should be https://funcapi.<tenantname>.onmicrosoft.com instead, otherwise it will occur error code invalid_claims.

Could you please tell which Python version supports in the documentation because this did not support Python version 3.8?

@Sangtong check this out, it worked for me in Python 3.8.
It is loosely inspired from this sample repo.

@inspiralpatterns Thank you for your reply.


integrate digital meter(BE)

Integrate digital meter as a sensor.
My Niko home control 2 is connected to my digital meter's p1 port.
Reading from digital meter will be auto discovered as new sensor entity called "energyhome".
Sorry for my bad coding, since I don't have any experience with Python.. And also new to home automation.

@YangLin404 thx! But this is also implemented in the rewrite. See https://github.com/joleys/niko-home-control-II/pull/49


Redirect HTTP to HTTPS

https://dekeo.atlassian.net/browse/MN-1247

Closed in favor of redirecting by load balancer: https://repost.aws/knowledge-center/elb-redirect-http-to-https-using-alb


Add details around account and password reqs

@JohanFreelancer9 Copy editing is needed for this PR. Thanks.

@SerdarSoysal PR has been copyedited and is ready for final review, could you please check and merge? Thanks!


Does Clarity UI Support Targets in the Progress bars..

Select one ...  (check one with "x")
[ ] bug
[ ] feature request
[ ] enhancement

Expected behavior

Actual behavior

Reproduction of behavior

Include a link to the reproduction scenario you created by forking one of the Clarity Plunker Templates:

Environment details

Angular version: 4.x.x

Clarity version:

OS and version:

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

@sunpasup
What do you mean by "targets"?

I think the suggestion was to have steps at specified percentage points
Something like

@juanmendes
We call this a "timeline". We have open issues for a vertical and horizontal version (#353 and #1633 ).
Please subscribe to those issues for updates. Thanks!


GLObjects handle semantics rewrite

Rewrites of most of the GLObjects as Raw* Handle types;
Gives const-correct behavior to GLObjects through GLMutability with GLConst and GLMutable;
Raw GLObjects have correct reference semantics and behave similarly to fancy-pointers;
Provides extrinsic ownership for GLObjects through container types like GLUnique;
Implements helpers for render targets: ViewAttachment, UniqueAttachment, RenderTarget,  SwapChain.

Probably best to rename Object aliases to UniqueObject before merging. So Texture2D -> UniqueTexture2D, etc.


Remove once_cell dependency (?)

I think it's no longer used; the lazy feature is now in Rust mainline, right?

(or if I do use the dependency, we can switch to native rust instead?)


Adds ability to customize and access engine-specific types

This allows end-users to use new features added compatibly to an engine without updating waPC first. This also allows generic use of engine-specific features without having to copy/paste tests in the underlying library.
This begins with wasmer, as that what was needed by #40

note: wazero already has this feature, but seems fine to change it to the options pattern as in this project, that's the preferred approach per discussion in #40, so basically if this is acceptable, I'll make everything look the same except using the correct engine-specific nouns.


[tools] remove --flutter-repo flag in flutter test

This option has a misleading name, is not used in any of our builds, not useful to our users, and its benefits can be easily gained using a shell alias.
Fixes #1939
/cc @Hixie @devoncarew

lgtm

Maybe we should make travis/test.sh robust enough to be run as a replacement by developers?

I just use travis/test.sh personally. Sometimes I use a variant of test.sh that uses --engine-debug but is otherwise unchanged.

Just ran travis/test.sh myself. Worked great. Could certainly use an environment variable for specifying the engine to run with.


WIP-Choose random ports if no configuration is given

@SylvainCorlay @JohanMabille Do you mind reviewing this? I'm not sure about the design.

I have a segfault locally, I cannot reproduce it on the CI

@JohanMabille segfault fixed (it was in xeus-cling), I think this PR is ready :)

Almost young padawan, almost ;)


update the remnants of table watcher with 3s default polling frequency

Related to https://github.com/IBM/kui/issues/1918


Hash link to start editing of a JSON style

It would be great have a chance to create "Edit" links going to Maputnik instance (in our case on https://editor.openmaptiles.org/ - opening for editing directly a given JSON style (in future GitHub #3, but now at least loading the style from a given HTTP link).
Such a link can be also put into the GitHub repos of our styles.

Yes I want that too #34. :+1:


Toggle visibility of tabs hotkey behaves differently than right click when used with Show live tabs only

Checklist

[X] I'm reporting a problem with Chatterino
[X] I've verified that I'm running the most recent nightly build or stable release
[X] I've looked for my problem on the wiki
[X] I've searched the issues and pull requests for similar looking reports

Describe your issue
When using Show live tabs only, using right click to toggle visibility of tabs will toggle back to only showing live channels, but when using the Ctrl+U hotkey it toggles back to showing all tabs regardless of live or not.
Screenshots
right click example:
https://github.com/Chatterino/chatterino2/assets/62087307/5e0b3ca2-f73a-4019-8cf9-1a9e91194088
hotkey example:
https://github.com/Chatterino/chatterino2/assets/62087307/6e1212f9-c3c6-458a-a7c7-a255ecdaff11
OS and Chatterino Version
Chatterino Nightly 2.5.0 (commit 3aead093) built on 2024-04-21 with Qt 6.5.0, MSVC 193833135 Running on Windows 11 Version 23H2, kernel: 10.0.22631

I'd say this is a bug with the CTRL+U hotkey, as it will always show all tabs: https://github.com/Chatterino/chatterino2/blob/3aead09339ffff0fbfd9f8a7a5ea9eb551ad8fb8/src/widgets/Window.cpp#L648-L653
The menu action is identical to pressing CTRL+Shift+L (toggle live tabs only).

I believe this is a bug...

I believe this is a bug...

Yes Neri's comment right above yours indicates this and also points to where the problem is.


Change database model for job data

Right now job data is stored such that one entry corresponds to one job and one participant. The actual data is stored as a JSON in a text field. This is problematic for two reasons:

If a job is repeated often, this results in very large JSON text blob that contains the combined results from all repetitions. This results in the database crashing, out-of-sort memory errors, and other symptoms that we've discussed elsewhere. (#152 , #150, #92 )
If an experiment is removed or changed, or if a participant is removed, the data is removed as well. This is undesirable because the data should remain available even after such changes.

To fix this, the database model should be refactored such that multiple runs of the same job are stored as separate entries, and that these entries remain accessible when linked experiments / participants are removed.

@smathot @nclaidiere finally found some time and started working on this. I think the new proposed way of storing and retrieving data will be an improvement overall. There is only one caveat:

and that these entries remain accessible when linked experiments / participants are removed.

I need to relationally link the data to at least one of these database records, and I have chosen studies. This implies though that if a study is deleted the stored data will go with it. For participants, I have chosen to embed their data in the data that is stored for the job, so participants can be safely deleted but their data will remain.
@nclaidiere I remember you sent me this a while ago but I cannot find it anymore, but can you send me an export or dump of your current database? I want to convert the old data tabel structure to the new one in a migration, but I want to see if this works well with the production data, to be really sure nothing will break.

@dschreij Thanks for picking this up. Can you hold this thought for a few days though? @nclaidiere @kristian-lange and I are having a meeting next Wednesday to discuss the next steps for OpenMonkeyMind, and this one of the things that's on the agenda. I'll get back to you!

Hello, I'm one of the IT in the Laboratory of @nclaidiere.
I did some investigation, and relative to some experiment's design, sometimes there is an UPDATE's query in the table  "job_states". I understand this table has all the job and sometimes there are one, two or prehaps three json data in the same row. From my point view this the problem we are looking for. I will try to correct that with the rewrite plugin of mysql but it's not in the Docker image use by omm-server.

Hi @Loic-Bonnier, you are right in your assessment. I have already started with decoupling the job data with the job_states as you can see in the commits that are attached to this issue. Feel free to inspect them and improve them.

@smathot I'm looking at the database schema and stumbled over certain tables that have only one column: dtypes, job_statuses_, access_permission, user_types, participation_statuses_. They all have the same columns: an id and a name. I fail to grasp the purpose of them - why not just incorporate them into the table where they have relation to?
E.g. Instead of having two tables for the job status

one could have just one table

and this would reduce the complexity of our schema, makes it easier to handle and develop. Or do I miss some purpose of them?

Allow me to chime in. This was a deliberate design choice (and a discussion
that pops up once in a while again). But I am from the camp that believes
that enums are evil 👀. See this article that explains nicely why
http://komlenic.com/244/8-reasons-why-mysqls-enum-data-type-is-evil/
Especially point 4 has often bitten me, as has point 2 when you need to
change something to the member list.
Met vriendelijke groet / with kind regards
Daniël Schreij
On Fri, Jul 12, 2024, 14:54 Kristian Lange @.***> wrote:

@smathot https://github.com/smathot I'm looking at the database schema
and stumbled over certain tables that have only one column: dtypes,
job_statuses_, access_permission, user_types,
participation_statuses_. They all have the same columns: an id and a
name. I fail to grasp the purpose of them - why not just incorporate
them into the table where they have relation to?
E.g. Instead of having two tables for the job status
image.png (view on web)
https://github.com/user-attachments/assets/a6f38f99-eb52-472f-9456-980e9f2ed802
one could have just one table
image.png (view on web)
https://github.com/user-attachments/assets/aa94d2a8-381c-41a2-b0d2-fe79fd125eb0
and this would reduce the complexity of our schema, makes it easier to
handle and develop. Or do I miss some purpose of them?
—
Reply to this email directly, view it on GitHub
https://github.com/open-cogsci/omm-server/issues/155#issuecomment-2225518134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFIHFE455DYN4HVFYQAYMLZL7GZLAVCNFSM6AAAAABF3EBPOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGUYTQMJTGQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

@dschreij Hi  Daniël, nice you chimed in :). I wasn't aware of this apparent war, people on one side supporting enum types and people on the other side supporting look-up tables. I belong to a third type of people, admittedly a minory, that argue one could just put your status/type data in a simple varchar or tinyint and keep it simple without an extra table or enum.
Advantages

simple schema: no enum type, no extra look-up table
simple queries: no table joins
easy to read (everyone understands "pending" - not everyone "1")

Disadvantages

varchar uses (a bit) more disk space
people need to be disciplined and actually only using the appropriate values
renaming of such status/type data means to go through every row in the table and update the value

For a small project (like OMM) I'm happy to live with the disadvantages and enjoy the advantages.
E.g.
I think this
await ptcp.jobs()
        .where('study_id', study.id)
        .whereInPivot('status', ['pending', 'started'])
        .withPivot(['status'])
        .orderBy('pivot_status', 'desc')
        .orderBy('position', 'asc')
        .firstOrFail()

is easier to read than this
await ptcp.jobs()
        .where('study_id', study.id)
        .whereInPivot('status_id', [1, 2])
        .withPivot(['status_id'])
        .with('variables.dtype')
        .orderBy('pivot_status_id', 'desc')
        .orderBy('position', 'asc')
        .firstOrFail()

.

The main disadvantage of using a reference table that I can see is that this may require an additional join and thus reduce performance. But this may not actually happen in practice, because we are generally not interested in the text description of these fields, and thus we are likely not actually joining the reference tables in our queries. (Or am I wrong about that? If we are regularly joining the reference tables, then that's a performance issue.)
@kristian-lange You mention using a simple tinyint instead. This is actually no different from using a reference table, except that in that case the tinyint also serves as an index to the reference table. The varchar approach I don't like because it makes the table bigger and potentially less performant.
In other words: @kristian-lange, restructure as you see fit while considering that:

There is no need to change the database structure unless there is a clear reason for it. The main reason would be to reduce the number of operations and joins to increase performance.
We don't want to use varchar columns for enum-like data.
There may not be a fundamental difference between your tinyint approach and @dschreij 's reference tables.

Is that helpful?

I'm still trying to get familiar with the database schema and where it could potentially be improved to solve some problems we actually face. For me this discussion about the look-up table / reference table is more like an exercise before tackling the real problems. I agree with @smathot that there is no need to change the schema regarding this.

You mention using a simple tinyint instead. This is actually no different from using a reference table, except that in that case the tinyint also serves as an index to the reference table.

I actually meant only an tinyint and no lookup table. The meaning of the different int values would have been coded in the application. It's similar to what we are doing currently. We seldom actually look up the meaning in the look-up table and we do the database queries with the id, e.g. .whereInPivot('status_id', [1, 2]).

Hi @dschreij, could you please help me to understand why you have chosen a certain way. You wrote:

I need to relationally link the data to at least one of these database records, and I have chosen studies.

And from your commit (#4accc39) I can see that you planed a table study_data that both links the tables studies and participants.

For me the natural choice to attach the table that contains the job's result data would be to link it to the job_states table where the job's result data are currently stored (in field data of type json). Why did you choose to link it to studies and participants.

For participants, I have chosen to embed their data in the data that is stored for the job, so participants can be safely deleted but their data will remain.

Can you please elaborate on this? Are their data for participants that have to be stored additionally? Are those data different from the job's result data? What do you mean with 'embed' and 'data that is stored for the job'? Do you mean the jobs table (But this table does not have a data field)?
And I have two partially related questions:

Why does the table job_states not have an id? Was it a deliberate choice? What were the reasons?

Why is the participant_id sometimes stored as a varchar (e.g. table session) and sometimes as an int (e.g. job_states or participations)?

Hi Kristian,
Why did you choose to link it to studies and participants.
The main reason to no longer attach job results to the job_states table is that there is a (new) requirement that the data should be retained if the job table is wiped. Currently, if one wants to upload a new job table file, the old jobs are deleted first, along with their data, before the new job table is composed from the uploaded file. To mitigate deleting all result data, I had to decouple it from jobs and tie it to the study instead.
Can you please elaborate on this? Are their data for participants that have to be stored additionally? Are those data different from the job's result data?
Yes, if you view a participant record you will see one can specify extra data in yaml that pertain to the participant, like gender, age, etc. These need to be stored with each trial/job result, so they are exported for each job data row when the data is saved as excel.
Why does the table job_states not have an id? Was it a deliberate choice? What were the reasons?
This is a pivot table, right? Those never really need ids as they have a composite primary key consisting of the participant ID and job ID if I remember correctly. Any combination of these are always unique. You will rarely use ids from a pivot table to retrieve data, as they are only used to be able to join 2 many-many related tables.
Why is the participant_id sometimes stored as a varchar (e.g. table session) and sometimes as an int (e.g. job_states or participations)
I can't clearly recall, but I think in the session table, the link to the participant is made using its identifier string, and not its ID. This is because the omm client needed to be able to manipulate the session data and using the identifier is easier that using the id for that case. I should have chosen a better name for this participant_id in the session table, so this is my bad
On Sun, Jul 14, 2024, 23:31 Kristian Lange<EMAIL_ADDRESS>wrote:
Hi @dschreijhttps://github.com/dschreij, could you please help me to understand why you have chosen a certain way. You wrote:
I need to relationally link the data to at least one of these database records, and I have chosen studies.
And from your commit (#4accc39) I can see that you planed a table study_data that both links the tables studies and participants.
image.png (view on web)https://github.com/user-attachments/assets/87abd619-5506-4d7f-aa73-1bdb977fc152
For me the natural choice to attach the table that contains the job's result data would be to link it to the job_states table where the job's result data are currently stored (in field data of type json). Why did you choose to link it to studies and participants.
For participants, I have chosen to embed their data in the data that is stored for the job, so participants can be safely deleted but their data will remain.
Can you please elaborate on this? Are their data for participants that have to be stored additionally? Are those data different from the job's result data? What do you mean with 'embed' and 'data that is stored for the job'? Do you mean the jobs table (But this table does not have a data field)?
And I have two partially related questions:

Why does the table job_states not have an id? Was it a deliberate choice? What were the reasons?

Why is the participant_id sometimes stored as a varchar (e.g. table session) and sometimes as an int (e.g. job_states or participations)?

—
Reply to this email directly, view it on GitHubhttps://github.com/open-cogsci/omm-server/issues/155#issuecomment-2227488731, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFIHFB2WYPQXMQUBR6TRWTZMLU3XAVCNFSM6AAAAABF3EBPOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGQ4DQNZTGE.
You are receiving this because you were mentioned.

Just to add some formatting to Daniel's answer ;).
Hi Kristian,

Why did you choose to link it to studies and participants.

The main reason to no longer attach job results to the job_states table is that there is a (new) requirement that the data should be retained if the job table is wiped. Currently, if one wants to upload a new job table file, the old jobs are deleted first, along with their data, before the new job table is composed from the uploaded file. To mitigate deleting all result data, I had to decouple it from jobs and tie it to the study instead.

Can you please elaborate on this? Are their data for participants that have to be stored additionally? Are those data different from the job's result data?

Yes, if you view a participant record you will see one can specify extra data in yaml that pertain to the participant, like gender, age, etc. These need to be stored with each trial/job result, so they are exported for each job data row when the data is saved as excel.

Why does the table job_states not have an id? Was it a deliberate choice? What were the reasons? This is a pivot table, right?

Those never really need ids as they have a composite primary key consisting of the participant ID and job ID if I remember correctly. Any combination of these are always unique. You will rarely use ids from a pivot table to retrieve data, as they are only used to be able to join 2 many-many related tables.

Why is the participant_id sometimes stored as a varchar (e.g. table session) and sometimes as an int (e.g. job_states or participations)

I can't clearly recall, but I think in the session table, the link to the participant is made using its identifier string, and not its ID. This is because the omm client needed to be able to manipulate the session data and using the identifier is easier that using the id for that case. I should have chosen a better name for this participant_id in the session table, so this is my bad

@kristian-lange Can this issue be closed now? Or are there other database changes that still need to be implemented?

@smathot We still have to merge the branch database_seeder into master. I made a PR for this: https://github.com/open-cogsci/omm-server/pull/186


Made a robot 'prepForStart' method for uponStart opmode usage

There were a variety of different opmodes doing different things, so I made a single chunk of code (not a command: We aren't scheduling a command for start, we're directly running code), and fixed all the 16750 opmodes to call that new function in the uponStart method. @Magkhuu and @braelynandthefrogs  please take a look.

I approve! Makes the code a lot more concise. :)


[BUG]: Drizzle Studio does not show correct value for timestamp

What version of drizzle-orm are you using?
Latest
What version of drizzle-kit are you using?
No response
Describe the Bug
Drizzle studio

pgAdmin

Expected behavior
Should display actual value
Environment & setup
No response

Should be fixed by https://github.com/drizzle-team/drizzle-orm/pull/1659.
It's already in drizzle-orm@beta. You can check this tag, and if it was fixed there, it should be in the latest version soon, and I'll close this issue

@AndriiSherman I was under the impression we weren't going to use this PR, I was getting ready to close the one for json/jsonb.