parallel test fail on RISC-V platform

I am trying to build the netcdf 4.9.0 on Arch Linux RISC-V. And I met two issues that I can't resolve when I run testing. The following tests FAILED (Collpased because the log are too long):

176 - nc_test4_run_par_test (Failed)
176/255 Testing: nc_test4_run_par_test
176/255 Test: nc_test4_run_par_test
Command: "/usr/bin/bash" "-c" "export srcdir=/build/netcdf-openmpi/src/netcdf-c-4.9.0/nc_test4;export TOPSRCDIR=/build/netcdf-openmpi/src/netcdf-c-4.9.0;/build/netcdf-openmpi/src/build/nc_test4/run_par_test.sh run_par_test.sh "
Directory: /build/netcdf-openmpi/src/build/nc_test4
"nc_test4_run_par_test" start time: Oct 10 19:15 CST
Output:
----------------------------------------------------------

Testing MPI parallel I/O with various other mode flags...

*** Testing illegal mode combinations
*** Testing create + MPIO + fletcher32
[avimitin:15301] *** Process received signal ***
[avimitin:15301] Signal: Segmentation fault (11)
[avimitin:15301] Signal code: Address not mapped (1)
[avimitin:15301] Failing at address: (nil)
[avimitin:15301] [ 0] linux-vdso.so.1(__vdso_rt_sigreturn+0x0)[0x3f80bfb800]
[avimitin:15301] [ 1] /usr/lib/libopen-pal.so.40(+0x55508)[0x3f8010b508]
[avimitin:15301] [ 2] /usr/lib/libmpi.so.40(PMPI_Bcast+0x2a8)[0x3f80974f74]
[avimitin:15301] [ 3] /usr/lib/libhdf5.so.200(+0x2cd25a)[0x3f806cd25a]
[avimitin:15301] [ 4] /usr/lib/libhdf5.so.200(H5FD_open+0xde)[0x3f8050a9c0]
[avimitin:15301] [ 5] /usr/lib/libhdf5.so.200(H5F_open+0x498)[0x3f804f92a8]
[avimitin:15301] [ 6] /usr/lib/libhdf5.so.200(H5VL__native_file_create+0x1c)[0x3f806a5dd0]
[avimitin:15301] [ 7] /usr/lib/libhdf5.so.200(+0x28a786)[0x3f8068a786]
[avimitin:15301] [ 8] /usr/lib/libhdf5.so.200(H5VL_file_create+0x82)[0x3f8068f052]
[avimitin:15301] [ 9] /usr/lib/libhdf5.so.200(H5Fcreate+0xe8)[0x3f804ee908]
[avimitin:15301] [10] /build/netcdf-openmpi/src/build/liblib/libnetcdf.so.19(nc4_H5Fcreate+0x44)[0x3f80a90562]
[avimitin:15301] [11] /build/netcdf-openmpi/src/build/liblib/libnetcdf.so.19(NC4_create+0x346)[0x3f80a908ec]
[avimitin:15301] [12] /build/netcdf-openmpi/src/build/liblib/libnetcdf.so.19(NC_create+0x170)[0x3f80a54af0]
[avimitin:15301] [13] /build/netcdf-openmpi/src/build/liblib/libnetcdf.so.19(nc_create_par+0x46)[0x3f80a63626]
[avimitin:15301] [14] ./tst_mode(main+0x64)[0x2abb703b94]
[avimitin:15301] [15] /usr/lib/libc.so.6(+0x26b04)[0x3f807f4b04]
[avimitin:15301] [16] /usr/lib/libc.so.6(__libc_start_main+0x74)[0x3f807f4bac]
[avimitin:15301] [17] ./tst_mode(+0xefc)[0x2abb703efc]
[avimitin:15301] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node avimitin exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
<end of output>
Test time =   3.30 sec
----------------------------------------------------------
Test Failed.
"nc_test4_run_par_test" end time: Oct 10 19:15 CST
"nc_test4_run_par_test" time elapsed: 00:00:03
----------------------------------------------------------

203 - h5_test_run_par_tests (Failed)
203/255 Testing: h5_test_run_par_tests
203/255 Test: h5_test_run_par_tests
Command: "/usr/bin/bash" "-c" "export srcdir=/build/netcdf-openmpi/src/netcdf-c-4.9.0/h5_test;export TOPSRCDIR=/build/netcdf-openmpi/src/netcdf-c-4.9.0;/build/netcdf-openmpi/src/build/h5_test/run_par_tests.sh run_par_tests.sh "
Directory: /build/netcdf-openmpi/src/build/h5_test
"h5_test_run_par_tests" start time: Oct 10 19:17 CST
Output:
----------------------------------------------------------

Testing parallel I/O with HDF5...
[avimitin:15344] *** Process received signal ***
[avimitin:15344] Signal: Segmentation fault (11)
[avimitin:15344] Signal code: Address not mapped (1)
[avimitin:15344] Failing at address: (nil)
[avimitin:15344] [ 0] linux-vdso.so.1(__vdso_rt_sigreturn+0x0)[0x3fbea1b800]
[avimitin:15344] [ 1] /usr/lib/libopen-pal.so.40(+0x55508)[0x3fbe265508]
[avimitin:15344] [ 2] /usr/lib/libmpi.so.40(PMPI_Bcast+0x2a8)[0x3fbe55cf74]
[avimitin:15344] [ 3] /usr/lib/libhdf5.so.200(+0x2cd25a)[0x3fbe8cd25a]
[avimitin:15344] [ 4] /usr/lib/libhdf5.so.200(H5FD_open+0xde)[0x3fbe70a9c0]
[avimitin:15344] [ 5] /usr/lib/libhdf5.so.200(H5F_open+0x498)[0x3fbe6f92a8]
[avimitin:15344] [ 6] /usr/lib/libhdf5.so.200(H5VL__native_file_create+0x1c)[0x3fbe8a5dd0]
[avimitin:15344] [ 7] /usr/lib/libhdf5.so.200(+0x28a786)[0x3fbe88a786]
[avimitin:15344] [ 8] /usr/lib/libhdf5.so.200(H5VL_file_create+0x82)[0x3fbe88f052]
[avimitin:15344] [ 9] /usr/lib/libhdf5.so.200(H5Fcreate+0xe8)[0x3fbe6ee908]
[avimitin:15344] [10] ./tst_h_par(main+0x15c)[0x2ac9da535c]
[avimitin:15344] [11] /usr/lib/libc.so.6(+0x26b04)[0x3fbe3dcb04]
[avimitin:15344] [12] /usr/lib/libc.so.6(__libc_start_main+0x74)[0x3fbe3dcbac]
[avimitin:15344] [13] ./tst_h_par(+0x1cec)[0x2ac9da5cec]
[avimitin:15344] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 0 on node avimitin exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
<end of output>
Test time =   2.90 sec
----------------------------------------------------------
Test Failed.
"h5_test_run_par_tests" end time: Oct 10 19:17 CST
"h5_test_run_par_tests" time elapsed: 00:00:02
----------------------------------------------------------

Platform information

OS: Arch Linux 5.19.2

Arch: riscv64

Machine: SiFive Unmatched

Build command:
export CC=mpicc
cmake -B build -S ${_pkg}-c-${pkgver} \
-DCMAKE_INSTALL_PREFIX=/usr \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_BUILD_TYPE=Release \
-DENABLE_BYTERANGE=OFF \
-DENABLE_CDF5=ON \
-DENABLE_PNETCDF=OFF \
-DENABLE_DAP_LONG_TESTS=ON \
-DENABLE_DAP_REMOTE_TESTS=ON \
-DENABLE_EXAMPLE_TESTS=ON \
-DENABLE_EXTRA_TESTS=ON \
-DENABLE_FILTER_TESTING=ON \
-DENABLE_LARGE_FILE_TESTS=ON \
-DENABLE_UNIT_TESTS=ON
make -C build

Test command:
export OMPI_MCA_rmaps_base_oversubscribe=yes
export OMPI_MCA_opal_warn_on_missing_libcuda=0

make -C build test

This is tricky for us to debug, unfortunately, given that we don't have access to RISC-V hardware, and the same configuration appears to work locally (for what that is worth).  I suspect the issue is related to the installed HDF5 library, however.  Given that h5_test_run_par_tests is a pure HDF5 test, no netCDF interface involved, I have to conclude that the issue is rooted in HDF5.  This would explain why the corresponding netCDF test failed as well.  Are you able to install libhdf5 manually, with parallel enabled, in this environment? If so, are you able to run make check and see if the llibhdf5 tests pass?

I am using the openmpi version of the hdf5, and it seems like there are some bug in the openmpi: https://github.com/archlinux/svntogit-community/blob/packages/hdf5-openmpi/trunk/PKGBUILD#L88, and Arch Linux just bypass the test failure. So this issue is not related to the netcdf.
Thanks for your guidance!


(DO NOT MERGE) Prototype of API key support in GAX

The intention is to make it easy for APIs which do support API keys to expose that, without providing misleading properties to other client builders.

https://github.com/googleapis/google-cloud-dotnet/pull/11217 shows how this would be used.

A different alternative to consider:

Most of the API key support in the base class is as it was before
Remove GetEffectiveSettings, but add a protected AdditionalCallSettings property in the base class which normally returns null, but would return any extra call settings, e.g. for an API key or anything else we want
Either generate an EffectiveSettings property, or generate an internal static GetEffectiveSettings(XyzSettings settings, CallSettings additionalSettings) method in each concrete settings class.

Generating actual implementation feels nasty, but so does the GetEffectiveSettings method in this PR. Urgh. I'm still thinking about this, basically.

I've continued thinking about this, and while the whole "clone the settings" part is a bit annoying, I think it's okay for now. Maybe in the next GAX major version we should make the builder generic in the settings type as well, which would allow us to generate less per-API code, I suspect. Will note that, but for now I'm happy to go with this, and change the generator to unconditionally generate the call to GetEffectiveSettings. (We should look into it for hand-written client builders...)
Thoughts?

I've been thinking about this as well and I haven't come up with anything that's clean. A tweak to your alternative solution might be to:

Like yours: Add the CallSettings AdditionalCallSettings property on the base client builder and have the base client builder deal with everything API Key related (and other similar ones in the future).
On the XYZClient change the generated Create method to accept both XYZSettings and CallSettings additionalCallSettings.
Have the XYZClient.Create method calculate the effective XYZSettings by cloning and merging the passed XYZSettings with the additionalCallSettings if necessary.

It's still generating implementation but in a place where we are already generating related implementation, for instance XYZClientImpl's constructor calculates effective settings in case the passed settings are null, so, it's just a little bit more of that.
Otherwise, this is fine as well. I do +1 strongly making the client builder base generic on the settings type on the next major version.

I've been wanting to avoid adding a new Create overload - otherwise I completely agree that would be a good call. No huge urgency for this, so I'll keep pondering options.

It's been a couple of months now, and I haven't figured out anything better than this. Shall we just go with this? I can't remember offhand how easy it is to write tests for this sort of thing, but I can definitely look into it.

Yep, I'm fine with this approach, I don't think we can do much better.

This is now ready for review and merging. There are tests for GetEffectiveSettings - a lot of the rest is hard to test due to ADC.


Multichannel training for longitudinal studies

This is PR to allow longitudinal studies/datasets to be trained when the Multichannel option is selected.
The file structure of these datasets includes folders and filenames with "ses-*" strings.
The derivatives are loaded in a straightforward way when they are present for all images.
However, since the multichannel training requires a single GT for multiple input images, there are a few options for associating a derivative to an image (the code follows this order):

The derivative of the image (normal approach)
The derivative of the same CONTRAST from a different session
The derivative from a different contrast (but also selected in the config file) from the same session
The derivative from a different contrast (but also selected in the config file) from a different session

The reasoning behind this is that with the previous structure of the code the images that didn't have a GT would be ignored.
However in the multi-channel training case we can still use them.
There is a display that is logged on luguru to inform the user/devs of the association of an image to a different GT.
The inspiration for accommodating these cases comes from the MS-Challenge dataset that only has a single GT for the second time point.
https://github.com/ivadomed/ms-challenge-2021
This is still in draft mode
The training/testing work
Segmenting is still WIP
Related to https://github.com/ivadomed/ivadomed/issues/788
Comments welcome.

thanks @jcohenadad
I'll have to create a new dataset with fake namings I guess since we don't have any publicly available longitudinal data to add for automatic download through: https://github.com/ivadomed/ivadomed/blob/master/ivadomed/scripts/download_data.py#L156
@mariehbourget you mentioned you had a few microscopy files with ses- that can help generalize the script, but I can't locate the message you sent. Can you please remind me?

@mariehbourget you mentioned you had a few microscopy files with ses- that can help generalize the script, but I can't locate the message you sent. Can you please remind me?

Yes, they are in our data testing dataset here: https://github.com/ivadomed/data-testing/tree/master/microscopy_png/sub-rat3, only 1 subject and 2 sessions, though.

You are right that the dataframe does not include the files with missing derivatives (for MS-challenge: files from the first timepoint). I had not thought about that before. So there are like 2 main issues to resolve:

How to modify the filtering of the dataframe to include those files

How to fill the multichannel dict. From what I understand, this is not yet addressed in this PR, and I'm not sure how this works with training.

I just made some modifications for multichannel dict:
The association to a new derivative that I described above is inherited all the way to fill_multichannel_dict
Here is an example of a variable within fill_multichannel_dict for an image that does not have a derivative, but inherits all the derivatives from a different session.
(['/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-01/anat/sub-013_ses-01_FLAIR.nii.gz', '/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-02/anat/sub-013_ses-02_FLAIR.nii.gz'],
['/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/derivatives/labels/sub-013/ses-02/anat/sub-013_ses-02_FLAIR_lesion-manual-rater1.nii.gz'],
None,

[{'path': '/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-01/anat/sub-013_ses-01_FLAIR.nii.gz', 'filename': 'sub-013_ses-01_FLAIR.nii.gz', 'datatype': 'anat', 'extension': '.nii.gz', 'participant_id': 'sub-013', 'session': '01', 'subject': '013', 'suffix': 'FLAIR', 'contrast': 'FLAIR',
'input_filenames': '/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-01/anat/sub-013_ses-01_FLAIR.nii.gz', 'gt_filenames': ['/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/derivatives/labels/sub-013/ses-02/anat/sub-013_ses-02_FLAIR_lesion-manual-rater1.nii.gz'], 'slice_index': 255, 'coord': None},

{'path': '/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-02/anat/sub-013_ses-02_FLAIR.nii.gz', 'filename': 'sub-013_ses-02_FLAIR.nii.gz', 'datatype': 'anat', 'extension': '.nii.gz', 'participant_id': 'sub-013', 'session': '02', 'subject': '013', 'suffix': 'FLAIR', 'contrast': 'FLAIR',
'input_filenames': '/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/sub-013/ses-02/anat/sub-013_ses-02_FLAIR.nii.gz', 'gt_filenames': ['/home/nas/Consulting/ivado_project/Datasets/ms_challenge/LongitudinalMultipleSclerosisLesionSegmentationChallengeMiccai21/training/_curated/derivatives/labels/sub-013/ses-02/anat/sub-013_ses-02_FLAIR_lesion-manual-rater1.nii.gz'], 'slice_index': 255, 'coord': None}])

This is still WIP.

Personal note:
the number of the model inputs is now dependent on the number of sessions within the dataset.
In other words, the dataset dictates these values and not the config file:
https://github.com/ivadomed/ivadomed/blob/master/ivadomed/main.py#L169
Things get a bit more complicated when a dataset has a combination of longitudinal and normal subjects that have one or multiple contrasts (even a single contrast can be located into multiple sessions, affecting the # of inputs)
Discuss strategy on how to approach this.
Add a maximum number of inputs?

A quick update on this PR status.
A minor issue was blocking the unit tests from passing, which has been fixed (dict key issue) and should be passing existing unit tests without major issues.
HOWEVER, the BIGGER issue is

We do not have a proper unit/integration test for multi-sessions multi-contrasts currently.

Some radical attempts to adopt a small 3 subjects, 4~5 sessions, T1/T2/FLAIR/PD subset of multi-sessions-contrasts scavenged MS2015 challenge dataset (Public domain at https://smart-stats-tools.org/lesion-challenge-2015) is currently ongoing at https://github.com/MotionCorrect/data_multi-sessions-contrasts and will be migrated to IvadoMed once they are fully complete.

We are running into issues getting the MS subjects' data loaded for some unknown reasons (despite derivatives being quite well loaded. This is going to take some quite time as from what I have seen as it is even more complicated than the MS2021 dataset.
However, to truly properly implement multi-sessions support and user configurable session specification, a lot more remains in that direction. You can view some of the more radical extreme changes being introduced at yd/sessions_to_multichannel branch along with earlier attempt to adopt the dataset (since we don't have a way to auto download the 1.5GB test set yet). A lot of streamlining of that is also absolutely needed in addition to try it out on MS2021 _curated dataset once the access issue has been sorted out. I am still struggle to properly form the multi-sessional, multi-contrasts BIDSDict and I will need substantial more time to investigate that issue.
Depending on the needs of the competition, I am uncertain if what Konstantinos has completed is good enough as it is to merge or we need to go to full throttle with multi-contrast/sessions support as started on the other branch.
Lastly, just want to be clear, pretty much ALL changes in this branch are introduced by Konstantinos. My more crazier experimental stuff are not anywhere here...

Pull Request Test Coverage Report for Build 934512753

23 of 69 (33.33%)  changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage remained the same at 69.575%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

ivadomed/loader/loader.py
12
16
75.0%

ivadomed/loader/utils.py
10
29
34.48%

ivadomed/main.py
1
24
4.17%

Files with Coverage Reduction
New Missed Lines
%

ivadomed/main.py
1
13.18%

Totals

Change from base Build 916354062:
0%

Covered Lines:
4059

Relevant Lines:
5834

💛  - Coveralls

I am still struggle to properly form the multi-sessional, multi-contrasts BIDSDict and I will need substantial more time to investigate that issue.
Depending on the needs of the competition, I am uncertain if what Konstantinos has completed is good enough as it is to merge or we need to go to full throttle with multi-contrast/sessions support as started on the other branch.

If I recall correctly from the last meetings, the feature implemented in this branch by @mpompolas was working for the purpose of the challenge, in this https://github.com/ivadomed/ivadomed/pull/814#issuecomment-852551171, the filename_pairs seems correct.
Our concern with merging was that the feature was specific to the challenge and not easily generalizable to other situation. If that is the case, would it be possible to do the training for the challenge on this branch without merging?

I managed to get it working for the challenge with a hardcoded entry for the number of inputs on:
https://github.com/ivadomed/ivadomed/blob/master/ivadomed/main.py#L169
model_params["in_channel"] = 2
@dyt811 is working on generalizing on other datasets and also introducing new variables within the config.json file for multichannel filtering selections

@mariehbourget was SPOT on for the issue. A new updated dataset with PROPER capitalization is now up there. @cakester is wrapping up the loader unit tests and I can start testing out the potential new experimental flag related to session specification from user and single source of ground truth implementation to help generalize better.

More clear explanation will be given at the upcoming presentation. This is a technical brief on the direction the upcoming PR will head towards.

Unit tests for loading multi-sessional-multi contrasts data are passing BUT many edge cases are not even remotely being covered because, assumptions below are key potential contentions and will be extensively discussed in the upcoming meetings. It has been proposed/added in the radical branch yd/sessions_to_multichannels.

A session based filter key has been proposed/added. This enable restricting and targeting specific session x modality to ensure a consistent input for the model input channel for Main.py (not yet implemented).

All modality data must be present for that subject to be included.

A ground truth loader configure key has been proposed/added in the same branch to allow cross-session, cross-modality ground truth specification (i.e. one ground truth to rule them all PER subject). Key assumption going forward will be all data must be aligned to each other BEFORE IvadoMed. This way, it drastically reduce the complexity needed to do ground truth match across modality, across sessions.

Once those edge cases are covered, the only remaining major issues is to ensure model_params["in_channel"] is properly updated. At that point, we can tentatively say that we support multi-session multi-contrasts data with huge caveat on how the data must be pre-aligned across sessions AND modalities before trying ivadomed and config.json must include details about the new target_ground_truth and target_sessions key for the model to know: 1) what ground truth to consider, 2) which sessions data to ensure are present.

We have tentatively achieved what we set up in terms of subject/session/contrast inclusion/exclusion approach set out and built a full suite of unit tests leverage pytest-cases around it and they are passing in the other branch.

It was quite radical/experimental and may not be exactly what people have in mind when it comes to supporting multi-contrasts/sessions. However, it is a point to start and require much further refinement and improvement in the upcoming days before it can be ready to integrate with the rest of the main ivadomed code due to amount of changes it introduce (i.e. a new mandatory config keywords).
The presentation summarizing the approach is here: https://docs.google.com/presentation/d/1yJJTsciiRCPSw9bRUUPpEmqle6jIO2qcfiuRALYAL4s/edit?usp=sharing

Thank you @dyt811 and @cakester for the report and update, it helps a lot to understand the different scenarios. I have a few questions/comments:

I don’t have any experience with multichannel training so I’m not sure about the behavior on slides 5, 11 and 12 i.e. everytime we keep a GT from a modality or session that is not specified in the config file. I think we should confirm/discuss with the ivadomed team.

In the presentation, there is always only 1 ground-truth available for each multichannel input. What happens when several GT are available, is the “best” match in terms of modality/sessions is kept?

I’m not sure I understand the purpose of target_ground_truth and the difference with target_suffix. When there are several target_suffix, it means that we train a multi-class model, so we need 1 GT per class (each target_suffix represents a class). So unless I’m mistaken, multi-channel and multi-class are two separate things and if we filter with a single “target_ground_truth”, we won’t be able to do multi-class training.

If I understand correctly, the new features are implemented in another branch (yd/sessions_to_multichannel), which also includes a major refactoring of the loader. If that is the case, maybe we should open a separate draft PR to be able to visualize the changes and discuss further your approach.

So to summarize the current direction this PR will take. Per last week's discussion here are the clarifications on the questions.

I don’t have any experience with multichannel training so I’m not sure about the behavior on slides 5, 11 and 12 i.e. everytime we keep a GT from a modality or session that is not specified in the config file. I think we should confirm/discuss with the ivadomed team.

The team concurred on some of the basic key assumptions:

We can safely assume all Session and Modalities are properly aligned.
We will not use a new key but instead we will ensure that at least ONE of EACH of the "target_suffix" key is present at the subject level (across Sessions and Modalities).
We require user to specify the new specific target_sessions key to use. We require user to specify the modalities required. The mandatory requirements are used to help ensure the right model input channels. If missing one modality or session, we discard subject.

In the presentation, there is always only 1 ground-truth available for each multichannel input. What happens when several GT are available (ex: GT for sessions 1 and 2 are available), is the “best” match in terms of modality/sessions with the input is kept?

So far, we are defaulting to the very first GT if more than one is specified.

I’m not sure I understand the purpose of target_ground_truth and the difference with target_suffix. When there are several target_suffix, it means that we train a multi-class model, so we need 1 GT per class (each target_suffix represents a class). So unless I’m mistaken, multi-channel and multi-class are two separate things and if we filter with a single “target_ground_truth”, we won’t be able to do multi-class training.

The target ground truth key has been removed in favour of repurposing the target_suffix instead.

If I understand correctly, the new features are implemented in another branch (yd/sessions_to_multichannel), which also includes a major refactoring of the loader. If that is the case, maybe we should open a separate draft PR to be able to visualize the changes and discuss further your approach.

Yes. As soon as an end to end training is completed successfully, we will breakdown the new branch into bite size chunk PRs to the main branch to ease the adaption. So far, we have not been able to even achieve end to end training EVEN with the feature branch so as soon as we hit that mile stone, we will start migrating it towards the main branch.

This PR is now being closed as we are reforging it in a new PR post the major loader.py class refactoring.
Rest assured, ALL changes introduced here as of 2021-12-12T115559EST has been manually checked and migrated over to the new PR branch as of commit df1e319dd4700a50a50a8320439ce030320b5730 made on yd/multi_session_support_reforged branch which will subsume existing work from kn/sessions_to_multichannel as well as the more radical exploratory branch formed earlier yd/sessions_to_multichannel as part of Project 5 In progress items.

Do not delete this branch until the new PR is fully merged and ready.


update readme : display of "swarmctl node ls" is not correct

Signed-off-by: Xiaoxu Chen<EMAIL_ADDRESS>

Thanks!
LGTM
/cc @sfsmithcha

LGTM


Add a non-regression test on our evaluation script

Currently, when we develop our evaluation scripts, the good practice is to add a new test with pytest in the /test folder. However, this test is performed on specific test knowledge base such as tiny.json that are small on purpose. We should also add a non regression test on larger dataset such as fsquad.json in order to verify that the behavior is still the same at some specific parameters.

Almost done. Last check to do with DILA dataset /clients/dila/knowledge_base/squad.json

Score are Ok @guillaumecherel


Bug/added themed context

Table of Contents
Link to GitHub issues it solves
closes #12
Description
Added themed context to fix resolution issue
Contributing has more information and tips for a great pull request.

@zeobot set lc to pass


Add results panel (dynamically updating)

The results of the round and match should be fed into a 'result' panel or similar to assess the warrior's performance

very basic impl in #49


Ensure correct seeding behaviour. Using the self.np_random rather tha…

This is a PR to solve issue #461
Ensure correct seeding behaviour. Using the self.np_random rather than the np.random. This makes sure:

Deterministic env resetting.
Different seeds for envs running in parallel, e.g. Async Vector Env of gymnasium.
Make the seed in the reset function useful, following the gymnasium interface.

Thanks for your reply.


Wiki article request: The curse of CORS and how to overcome it

If you're working on the Twitch Zipline and getting a error message instead of data, you are likely a victim of a Cross-Origin Resource Sharing (CORS) restriction.
I'd be glad to point to CORS documentation and explain how I solved it, but others would need to add to it to make the article complete.

I've made an entry here: https://github.com/FreeCodeCamp/FreeCodeCamp/wiki/Zipline:-Use-the-Twitchtv-JSON-API. Please feel free to correct/expand.

I think the article mentioned above has been moved to:
https://github.com/FreeCodeCamp/FreeCodeCamp/wiki/Zipline-Use-the-Twitchtv-JSON-API

@pdsoper thanks for posting this issue!
@leonfeng thanks for creating that Wiki article, @krisgesling thanks for noticing that article has been moved to Zipline Use the Twitchtv JSON API.
Closing this issue as the article is created.


(Question) How to show the marker in my map after picking a place in autocomplete

How can I achieve this after selecting a place (text input) in autocomplete component I want to display it to my map and show also the marker on it
here is my code

and this is my function

I tried several times it took 3 days but still no progress hope someone can help me thanks!

Hello! Did you try to use these examples as base? Also, are you being able to show markers on your map? Because after that, all you need to do is to change the values of them and everything works fine :)

Hello! Did you try to use these examples as base? Also, are you being able to show markers on your map? Because after that, all you need to do is to change the values of them and everything works fine :)

Yes It show on my map :D

heres my updated code
const markers = ref([{
  position: {
    lat: 14.5069174,
    lng:<PHONE_NUMBER>,
  }
}])

<GMapMarker
  :key="index"
  v-for="(m, index) in markers"
  :position="m.position"
  :clickable="true"
  :draggable="true"
/>

<!-- Search -->
<GMapAutocomplete
   placeholder="Search Location"
   @place_changed="setPlace"
   :options="{
   componentRestrictions: { country: `ph` },
   fields: [`address_components`, `geometry`],
   types: [`establishment`],
   }"
>
</GMapAutocomplete>

My Real Concern was If  I pick a place in autocomplete I want to show the marker on my map

Still cant find how to show it I hope you understand what I mean hahaha thanks!

I got it now :D
heres what I added
<script setup>
import { ref, onMounted } from 'vue';

const center = ref({
       lat: 14.5069174,
       lng:<PHONE_NUMBER>,
});

const markers = ref([{
  position: {
    lat: 14.5069174,
    lng:<PHONE_NUMBER>,
  },
}])

const place = ref({});

function setPlace(place) {

  markerSelected.value = {
    lat: place.geometry.location.lat(),
    lng: place.geometry.location.lng(),
  }

  navigator.geolocation.getCurrentPosition(position => {
    center.value = {
      lat: place.geometry.location.lat(),
        lng: place.geometry.location.lng(),
    };
  });

    markers.value = [
    {
      position: {
        lat: place.geometry.location.lat(),
        lng: place.geometry.location.lng(),
        }
    },
      ];
}

onMounted(() => {
  setPlace(place);
})

</script>

and in my autocomplete and markers base still the same

I'm glad you handle it!
Let me know if I can help you about this again!

Sure!  I will, haha thanks!


HDDS-10708. Create configuration option for key/truststore factory default algorithm

What changes were proposed in this pull request?
Key- and TrustManager factory default algorithms might not be cryptographically compliant. An option is created here so they can be specified in config files and can be whitelisted once whitelisting is implemented.
HDDS-10708
This builds on HDDS-10701 and its pr: https://github.com/apache/ozone/pull/6548 so it should remain draft until that is merged.
How was this patch tested?
green ci:
https://github.com/Galsza/ozone/actions/runs/8781366734

@Galsza, I believe we can use the same approach here as discussed in this comment, and we can just close this PR.


add rebalance heart beat

Codecov Report

Merging #33 (6011b43) into main (46f95d7) will decrease coverage by 0.40%.
The diff coverage is 42.10%.

@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
- Coverage   59.67%   59.26%   -0.41%
==========================================
  Files          11       11
  Lines         801      820      +19
==========================================
+ Hits          478      486       +8
- Misses        261      270       +9
- Partials       62       64       +2

Impacted Files
Coverage Δ

pkg/kafsar/kafka_impl.go
46.93% <0.00%> (-0.35%)
:arrow_down:

pkg/kafsar/group_coordinator_impl.go
67.72% <47.05%> (-2.50%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46f95d7...6011b43. Read the comment docs.


Minor readme fix for autoconf install instructions

I copied/pasted the install instructions and it failed. Seems like there's just a typo in the readme.

It should be cpputest_build... it is now fixed though. Thanks!


Added option to add custom font to theme

I was trying to change the font but couldn't find the option, not even explorin the .ini file.
Could you paste some instructions? Thanks in advance. Great app!

LyricsFont = "Arial"

I believe the example above should work if I'm not mistaken. Add it to the configuration, as it's not there by default.

Thanks for the answer, I tested and it works as the program starts, however as long as you change the size of the font it automatically resets the font style.


Support auto-start conversations

https://help.mypurecloud.com/articles/configure-messenger/#:~:text=Automatically Start Conversations

https://developer.genesys.cloud/commdigital/digital/webmessaging/websocketapi#customer-join

This looks like a cool project 👊, I wonder If I could implement this testing on my projects?, and if there is any way I could collaborate to gain experience will be great.


WIP: Enable Cuda

Something that was tricky in the staged-recipes PR, particularly since it was either-or between cos7 and cuda.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Ah, the error in arch_list[-1] += '+PTX' made me think of cuda arches, and indeed the just needed a correct TORCH_CUDA_ARCH_LIST. 🥳
Since https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/1890 hasn't been merged yet, we'll need to pin the pytorch-version to the same that we're building with though...
Also I want to merge #1 first.


Application Thread Exception

What steps will reproduce the problem?
1.Open SuperPutty <IP_ADDRESS>, right click in sessions area
2.Click New Session
3. Fill out information for session, click save

What is the expected output? What do you see instead?
Expected to allow me to save session.  Image of what appears:
http://s17.postimg.org/guysvwydr/8_21_2014_9_55_38_PM.png

What version of the product are you using? On what operating system?
<IP_ADDRESS>

Please provide any additional information below.

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 22 Aug 2014 at 3:03
Attachments:

[8-21-2014 9-55-38 PM.png](https://storage.googleapis.com/google-code-attachments/superputty/issue-453/comment-0/8-21-2014 9-55-38 PM.png)

Closing this issue but all hope is not lost! We still want to help.
See: https://github.com/jimradford/superputty/wiki/Legacy-Imported-Issues on how to help us to help you!


Error after starting file browser

After installing the plugin and reopening Neovim, I opened my file browser, Neotree, and got the error in the attached screenshot.
I do not know if it's my file browser or what, but I wanted to make the issue known.

Chris fixed this on stream!


[Bing Channel] Cannot publish the bot to Bing

System Information (Required)

SDK Language: C#
SDK Version: <IP_ADDRESS>
Development Environment:

Issue Description
can not publish the bot to bing, the submit for review button is always disabled.
Code Example
NA, the bot handle is FindIt
Steps to Reproduce the Issue
When I submitted the bot for review to bing, the submit for review button is disabled, despite i filled in all the information.
Expected Behavior
I should be able to submit the bot bing.
Actual Results
When I submitted the bot for review to bing, the submit for review button is disabled, despite i filled in all the information.
Thanks.

That's a bug in the configuration page. I will take a look at it and get a fix asap.

@anarchitect Thanks for reporting. We're working on a fix for this now.

@anarchitect this is fixed now, please verify.
Thanks a lot for reporting the issue.


BeagleSoftware.BeagleEditor version 2024.4

Checklist for Pull Requests

[x] Have you signed the Contributor License Agreement?
[x] Is there a linked Issue?

Manifests

[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?

[x] Does your manifest conform to the 1.6 schema?

Note: <path> is the directory's name containing the manifest you're submitting.

Resolves: #174329

Microsoft Reviewers: Open in CodeFlow

Why no one accepting the package?
Hello! Is anyone home?

Why no one accepting the package? Hello! Is anyone home?

Moderators is on break today because today is sunday (or saturday in american time)
They will continue accepting the packages mostly starting monday.

Btw,

Just add: - Resolves: #174329 to the description on this PR so that it closes the issue when this PR is merged.
Originally posted by @vikingnope in https://github.com/microsoft/winget-pkgs/issues/174327#issuecomment-2361244912

Oh, no! How to enable auto-merge, again?

(or saturday in american time)
Where? in Seattle?

Hey! It is 7:00! Start working and accept my package!

Please! Please merge this repository!

Which timezone are you from?

Iran

And now it is Monday (At least in my time zone)

Why no one is accepting my package? All of packages accepted except mine

Why no one is accepting my package? All of packages accepted except mine

Usually, PR that has a https://github.com/microsoft/winget-pkgs/labels/New-Package label needs to be tested more thoroughly, like manually install and playing with the application a little bit, that is done by @stephengillie.
Other PR that was accepted by community moderators are just either a package update or deletion of obsolete packages.
It's 3AM right now in their timezone, so just be patient 😉

@KarbitsCode Microsoft is not only in America. It has many branches in many U.S. cities and even at other countries

@KarbitsCode Microsoft is not only in America. It has many branches in many U.S. cities and even at other countries

I know.
Actually, i meant @stephengillie timezone, the only moderator that verifies this kind of cases.

@ItzLevvie, this is a duplicate to mine https://github.com/microsoft/winget-pkgs/pull/174344, so since you merged mine we should close this

And the workflow you opened a PR for it, doesn't work.

And the workflow you opened a PR for it, doesn't work.

For the workflow to work, the PR has to be merged, so from now onwards it should work

Why no one accepting the package? Hello! Is anyone home?

Moderators is on break today because today is sunday (or saturday in american time) They will continue accepting the packages usually starting monday.

While we have moderators scattered around the planet, the bulk of moderation is indeed currently performed during business hours in the Pacific Time Zone.


[8.x] Add a method to sort keys in a collection using a callback

I added a method to the Collection classes to sort the keys of the items using a callback. For example, this helps you to sort item keys case insensitive.
With this method, the following code:
$items = $collection->all();

uksort($items, 'strnatcasecmp');

$items = collect($items);

can be replaced with this code:
$collection->sortKeysUsing('strnatcasecmp');

I can already do this via sortBy?

@taylorotwell Not entirely. The new method provides much more flexibility. Take this example to sort items where keys represent version number strings:

Your example does not sort this correctly. However,
collect([...])->sortKeysUsing('version_compare');

would do the trick.

@taylorotwell @driesvints With the argument in my last reply, would this feature be worth reconsidering?

@redelschaap best to just attempt a new PR.


feat(TestScheduler): Support use of emoji in marble syntax

TestScheduler.parseMarbles should correcty read an emoji as single value such as in:
-------✅---🔥---|
Description: Currently TestScheduler doesn't correctly support emojis in marble syntax because the string is read using marbles[i] and it doesn't consider that emojis are encoded as surrogate pairs.
Using toArray from lodash the string is correctly splitted into an equivalent array, where each emoji is considered as single character in parseMarbles.
All unit tests pass but the build is failing with Rollup and I'm not familiar with it, how could I resolve? I've tried importing toArray differently but nothing works.

Build error

><EMAIL_ADDRESS>clean_dist_global /Users/jiayi/Desktop/Repo/rxjs
> shx rm -rf ./dist/global
node_modules/lodash/lodash.js (17097:8) The 'this' keyword is equivalent to 'undefined' at the top level of an ES module, and has been rewritten. See https://github.com/rollup/rollup/wiki/Troubleshooting#this-is-undefined for more information
(node:3276) UnhandledPromiseRejectionWarning: Error: 'toArray' is not exported by node_modules/lodash/lodash.js (imported by dist/esm5_for_rollup/internal/testing/TestScheduler.js). For help fixing this error see https://github.com/rollup/rollup/wiki/Troubleshooting#name-is-not-exported-by-module
at Module.trace (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:7739:29)
at ModuleScope.findDeclaration (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:7361:22)
at Scope.findDeclaration (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5361:39)
at Scope.findDeclaration (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5361:39)
at Node.bind (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5846:28)
at /Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5155:50
at Node.eachChild (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5172:5)
at Node.bind (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5155:7)
at /Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5155:50
at Node.eachChild (/Users/jiayi/Desktop/Repo/rxjs/node_modules/rollup/dist/rollup.js:5169:19)
(node:3276) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3276) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

<EMAIL_ADDRESS>build_closure_core /Users/jiayi/Desktop/Repo/rxjs
node ./tools/make-closure-core.js

fs.js:660
return binding.open(pathModule.toNamespacedPath(path),
^
Error: ENOENT: no such file or directory, open 'dist/global/rxjs.umd.js'
at Object.fs.openSync (fs.js:660:18)
at Object.fs.readFileSync (fs.js:565:33)
at Object. (/Users/jiayi/Desktop/Repo/rxjs/tools/make-closure-core.js:4:17)
at Module._compile (internal/modules/cjs/loader.js:654:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:665:10)
at Module.load (internal/modules/cjs/loader.js:566:32)
at tryModuleLoad (internal/modules/cjs/loader.js:506:12)
at Function.Module._load (internal/modules/cjs/loader.js:498:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:695:10)
at startup (internal/bootstrap/node.js:201:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:516:3)

@kwonoj I removed the dependency on lodash by copying its minimal implementation into utils/stringToArray.ts. I added a comment about the source of the implementation to credit Lodash, but I'm not sure if it's completely okay both legally and as Code of Conduct. FYI Lodash has MIT License.
Otherwise I honestly don't know how to implement it differently and yet keeping it minimal like only Lodash can do. Unicode handling is a pain in JS.

Coverage increased (+0.01%) to 96.96% when pulling 903a9f8806e1f519596795c7fdbcfda29beba99a on jiayihu:emoji-marbles into 01eefff2132f29dfe51476782c3e7f2db48f88f2 on ReactiveX:master.

I'll need to think about this more, given that it does require a decent amount of code to add to the repository, and I'm unsure the benefits are worth it. We'll need to discuss this one at the next team meeting.

Neat! Though I'm personally neutral-to-slightly against this as I don't imagine it being used widely enough to justify the additional complexity and feature lock-in. e.g. if we end up moving to a parser generator some day (as some of my prototypes have) and used flex as the lexer we'd have to support these non-ascii characters ourselves instead of deferring to the lodash helper. Totally doable, but it's a lot more complexity for little gain.
Of course, my opinions can always be changed! I welcome constructive debate and any potential real world benefits to this addition 👍 👍

No problem @benlesh , I had my doubts myself after seeing how much difficult is working with emojis. It was fun nevertheless 👍Thank you all for your time!


Enhancement: Add proposer to blocks

Summary
The only way now to retrieve the proposer of a block is from Algod and in msgpack and encoded. It would be helpful to have it available from the Indexer.
Scope/Requirements
The addition of proposer to the data model, a migration script to add it to existing indexers, and adding it to the API.
Urgency/Relative Priority
It is helpful, but not urgent

Looking at this now -
We can add the proposer to the block in the indexer. This will require a DB migration to add the column. We can do this.
Backfilling the data requires a more serious migration, requires getting all the historical blocks again. We are not sure we need to do this.

This is addressed via https://github.com/algorand/indexer/pull/1608


As a server admin, node to node connections can be authenticated with HTTP Basic Auth

https://github.com/Zhipeng-Chang/CMPUT404-Winter-2019-GroupProject/pull/112


List Tools: Add List Tools

暂时不要合并它。
同样出于某种原因，虽然菜单确实适用于大多数块，但当您更改编辑目标时，某些块的值会随机重置。我想这是涡轮扭曲中的一个错误？

类似的拓展我做过，但是你的更棒！

暂时不要合并它。
同样出于某种原因，虽然菜单确实适用于大多数块，但当您更改编辑目标时，某些块的值会随机重置。我想这是涡轮扭曲中的一个错误？

类似的拓展我做过，但是你的更棒！

what

The 3 blocks that don't save their menu values. I believe it might be to do with the fact there's 2 variable arguments.

暂时不要合并它。
同样出于某种原因，虽然菜单确实适用于大多数块，但当您更改编辑目标时，某些块的值会随机重置。我想这是涡轮扭曲中的一个错误？

类似的拓展我做过，但是你的更棒！

what

google translated:

Don't merge it yet. !image
Also for some reason, while the menu does work for most blocks, when you change the edit target some blocks' values reset randomly. I guess it's a bug in Turbo Twist?

I did similar extensions, but yours is even better!

(probably quoting your entire message)

Hope this gets added soon

Thumbnail:

lists plus?

olo

As much as I despise looking at this palette with open-menu inputs (and it goes against my philosophy for "plus" extensions), the menus don't work without acceptReporters enabled for whatever reason and I want this to be as merge-ready as possible.

HOW DOES IT SORT SO FAST?? It sorted over 9999 random numbers from 1 to 9999 almost INSTANTLY!

That's a funny looking error

I do not have access to an SVG editor that can address the validation errors, should i just leave out the thumbnail?

Congrats on merging Motion+!

Congrats on merging Motion+!

Lily was not the creator of More Motion.

Congrats on merging Motion+!

Lily was not the creator of More Motion.

That notification had me excited for a second, you owe me an Arizona iced tea /j

Congrats on merging Motion+!

Lily was not the creator of More Motion.

That notification had me excited for a second, you owe me an Arizona iced tea /j

You owe me my extension back /j

This branch has conflicts that must be resolved

Still waiting for this and Events+ to be merged

hope some of those comments help

hope some of those comments help

The only one I'm unsure about is the randomise one, but aside from that I can get all of that done in the next 10 minutes lol

I have resolved most of what you brought up, the one thing I've left is randomise since I simply don't know how to write that in JS, and it works fine as is. As it's randomise and not, say, a sort, it's not exactly a breaking change if I do decide to change it later (if it really becomes an issue)

It's been almost 3 months and this is still not merged

How do you delete an issue when it's locked?

thats rude
no
it can take months for a pr to get merged
who put you incharge?

From: horizontal-shading @.>
Sent: Tuesday, October 3, 2023 1:16 PM
To: TurboWarp/extensions @.>
Cc: Surv @.>; Comment @.>
Subject: Re: [TurboWarp/extensions] List Tools: Add List Tools (PR #623)
Someone please close this PR as it's never coming.
—
Reply to this email directly, view it on GitHubhttps://github.com/TurboWarp/extensions/pull/623#issuecomment-1745401370, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAGGRICMD4OO25UZFKWUXG3X5RB6DAVCNFSM6AAAAAAZPUAMEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBVGQYDCMZXGA.
You are receiving this because you commented.Message ID: @.***>

thats rude 2. no 3. it can take months for a pr to get merged 4. who put you incharge?
…
________________________________ From: horizontal-shading @.> Sent: Tuesday, October 3, 2023 1:16 PM To: TurboWarp/extensions @.> Cc: Surv @.>; Comment @.> Subject: Re: [TurboWarp/extensions] List Tools: Add List Tools (PR #623) Someone please close this PR as it's never coming. — Reply to this email directly, view it on GitHub<#623 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAGGRICMD4OO25UZFKWUXG3X5RB6DAVCNFSM6AAAAAAZPUAMEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBVGQYDCMZXGA. You are receiving this because you commented.Message ID: @.***>

Nobody...

I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.

I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.

Exactly but it's been months lol

no excuses

From: horizontal-shading @.>
Sent: Monday, October 9, 2023 5:16 PM
To: TurboWarp/extensions @.>
Cc: Surv @.>; Comment @.>
Subject: Re: [TurboWarp/extensions] List Tools: Add List Tools (PR #623)
I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.
Exactly but it's been months lol
—
Reply to this email directly, view it on GitHubhttps://github.com/TurboWarp/extensions/pull/623#issuecomment-1753891011, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAGGRIGN6D4BNLSO7ZPFYK3X6RSR7AVCNFSM6AAAAAAZPUAMEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJTHA4TCMBRGE.

________________________________ From: LilyMakesThings @.> Sent: Tuesday, October 10, 2023 4:03 PM To: TurboWarp/extensions @.> Cc: Surv @.>; Comment @.> Subject: Re: [TurboWarp/extensions] List Tools: Add List Tools (PR #623) no excuses plenty of reasons, actually, just have some patience — Reply to this email directly, view it on GitHub<#623 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BAGGRIFLAARXHANRM5ALXMLX6WS2LAVCNFSM6AAAAAAZPUAMEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJWGE2DCNZUHA. You are receiving this because you commented.Message ID: @.***>

I wasn't trying to be rude.

why not accept reporters in list selection 😢

I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.

Exactly but it's been months lol

It's not like people are getting paid to do this...

I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.

Exactly but it's been months lol

It's not like people are getting paid to do this...

Umm, that was an old reply...

I think some people here need to be more patient when it comes to open source updates. I've seen so many comments on this repo about how updates are "never coming". It's like this for any open source project.

Exactly but it's been months lol

It's not like people are getting paid to do this...

Umm, that was an old reply...

...

Has this been merged yet?

no

do people still celebrate half-birthdays, because we've got a really fun one coming up

do people still celebrate half-birthdays, because we've got a really fun one coming up

YES! It's gonna be merged!

do people still celebrate half-birthdays, because we've got a really fun one coming up

YES! It's gonna be merged!

Quite the opposite

do people still celebrate half-birthdays, because we've got a really fun one coming up

YES! It's gonna be merged!

Quite the opposite

slowly frowns

Can't wait for the 6 month anniversary, i guess

This is my new account, hopefully List Tools gets added soon. And I apologize for the rude remarks.

i believe this was discussed before but is there a reason the menus are name-based instead of id-based?

i believe this was discussed before but is there a reason the menus are name-based instead of id-based?

To allow people to put stuff like join blocks or variables in the menus to determine where list goes (parity with the "var and list" extension, essentially)
Suffers from a similar problem to Asset Manager in that regard

thanks

To everyone, I apologize for all the inconvenience my old account has caused.

metal

Started processing extension additions?

YESSSSSSSS FINALLY


Metadata

Is it possible to add more metadata fields?

rating
views
category
original URL
available formats

Is it possible to rename certain metadata fields for easier metadata management? E.g.

author→ARTIST

For reference, here is the comparison between ytcog-dl and youtube-dl metadata:
youtube-dl
root@Joys-MacBook-Pro ~# youtube-dl -c --add-meta --xattrs P6C3szts5C4
[youtube] P6C3szts5C4: Downloading webpage
[download] Kuchizuke Diamond - Yamada-kun and the Seven Witches OP [Piano]-P6C3szts5C4.webm has already been downloaded and merged
[ffmpeg] Adding metadata to 'Kuchizuke Diamond - Yamada-kun and the Seven Witches OP [Piano]-P6C3szts5C4.webm'
[metadata] Writing metadata to file's xattrs
root@Joys-MacBook-Pro ~# ffprobe /private/var/root/Kuchizuke\ Diamond\ -\ Yamada-kun\ and\ the\ Seven\ Witches\ OP\ \[Piano\]-P6C3szts5C4.webm
Input #0, matroska,webm, from '/private/var/root/Kuchizuke Diamond - Yamada-kun and the Seven Witches OP [Piano]-P6C3szts5C4.webm':
  Metadata:
    title           : Kuchizuke Diamond - Yamada-kun and the Seven Witches OP [Piano]
    PURL            : https://www.youtube.com/watch?v=P6C3szts5C4
    ARTIST          : Animenz Piano Sheets
    DATE            : 20150619
    KEYWORDS        : くちづけDiamond,Kuchizuke Diamond,山田くんと七人の魔女,Yamada-kun and the Seven Witches,Weaver,ピアノ,Piano,OP,Opening,オープニング
    AUTHOR          : Animenz Piano Sheets
    DESCRIPTION     : Hey everyone, sorry for the lack of updates recently!
                    : I was having some piano exams, and therefore I was busier than usual.
                    :
                    : Anyway, with almost 3 weeks delay, I have finally uploaded my piano cover of Kuchizuke Diamond, the OP song of yamada-kun and the seven witches. (It's also one of of my favorite Opening songs of the spring anime season 2015)
                    :
                    : Also, I have decided to arrange this song in my "old" style:
                    : Almost 100% faithful to the original song and without any "original material" - basically, not too many experiments.
                    : This transcription is also a good exercise for the left hand, because you have to train your accuracy with the left hand in order to play the fast left hand jumps. I even think this is one of my few transcriptions, where the left hand is more difficult to
                    : The most difficult part is actually the "left hand melody" beginning at 0:36, since the right hand is playing an entirely different rhythm than the left hand.
                    : Initially, I included an "original ending" for the ending segment of the sheet, but after a few edits, I decided to end it similar to the original song - the music suddenly stops.
                    :
                    : Okay, that would be everything I want to say about this piece.
                    : My next (and final) upload for this anime season is going to be the ED song of Oregairu S2 -  "everyday world" (ballad version).
                    : It's going to be one of my easier pieces. (fo' real this time! I promise ;_;)
                    :
                    : See you next week!

                    : I was having some piano exams, and therefore I was busier than usual.
                    :
                    : Anyway, with almost 3 weeks delay, I have finally uploaded my piano cover of Kuchizuke Diamond, the OP song of yamada-kun and the seven witches. (It's also one of of my favorite Opening songs of the spring anime season 2015)
                    :
                    : Also, I have decided to arrange this song in my "old" style:
                    : Almost 100% faithful to the original song and without any "original material" - basically, not too many experiments.
                    : This transcription is also a good exercise for the left hand, because you have to train your accuracy with the left hand in order to play the fast left hand jumps. I even think this is one of my few transcriptions, where the left hand is more difficult to
                    : The most difficult part is actually the "left hand melody" beginning at 0:36, since the right hand is playing an entirely different rhythm than the left hand.
                    : Initially, I included an "original ending" for the ending segment of the sheet, but after a few edits, I decided to end it similar to the original song - the music suddenly stops.
                    :
                    : Okay, that would be everything I want to say about this piece.
                    : My next (and final) upload for this anime season is going to be the ED song of Oregairu S2 -  "everyday world" (ballad version).
                    : It's going to be one of my easier pieces. (fo' real this time! I promise ;_;)
                    :
                    : See you next week!

                    : I was having some piano exams, and therefore I was busier than usual.
                    :
                    : Anyway, with almost 3 weeks delay, I have finally uploaded my piano cover of Kuchizuke Diamond, the OP song of yamada-kun and the seven witches. (It's also one of of my favorite Opening songs of the spring anime season 2015)
                    :
                    : Also, I have decided to arrange this song in my "old" style:
                    : Almost 100% faithful to the original song and without any "original material" - basically, not too many experiments.
                    : This transcription is also a good exercise for the left hand, because you have to train your accuracy with the left hand in order to play the fast left hand jumps. I even think this is one of my few transcriptions, where the left hand is more difficult to
                    : The most difficult part is actually the "left hand melody" beginning at 0:36, since the right hand is playing an entirely different rhythm than the left hand.
                    : Initially, I included an "original ending" for the ending segment of the sheet, but after a few edits, I decided to end it similar to the original song - the music suddenly stops.
                    :
                    : Okay, that would be everything I want to say about this piece.
                    : My next (and final) upload for this anime season is going to be the ED song of Oregairu S2 -  "everyday world" (ballad version).
                    : It's going to be one of my easier pieces. (fo' real this time! I promise ;_;)
                    :
                    : See you next week!
                     You can extract video related data using the ${template} method using anything that is on the video properties list
I haven't tested all of these and ffmpeg plus the codec standards might reject some of the things you try.


Make OUI check optional

It would be great if the OUI check was optional, particularly to understand the effects of randomized MAC addresses where they fall outside of the recognised ranges.

its updated now, you can reinstall with
pip install --upgrade howmanypeoplearearound


Introduce constants for blkn

Must come after #33

This Pull Request meets the following criteria:

[x] Tests have been added/adjusted for my new feature
[x] New Components are registered in index.js of my project

Referencing this in: #35


[KYUUBI #4219]invert the pattern match branch order

Why are the changes needed?

It does work after  several tests, that inverting the pattern match branch order in CatalogShim_v3_0#getColumnsByCatalog, which ensures BI tools can list columns on Delta Lake tables in all schemas.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[ ] Run test locally before make a pull request

can you check the failed tests?


Feature request: enable funding of accounts from an external wallet

Whether creating new accounts or depositing into existing ones, there should be an option to fund accounts via an external deposit mechanism -- read: not directly from the connected LND node. This is useful for a few reasons:

It eliminates a source of double-fee deposits; if I want to deposit more BTC into my pool account but my LND node doesn't have enough free balance, I have to first deposit BTC into my LND wallet (and thus, pay a fee for that transaction) and then run DepositAccount() to transfer that BTC from LND to pool (thus, paying a second transaction fee).
This would enable rapid, "on-the-go" deposits; as above, instead of needing to wait for two transactions to confirm, I could simply deposit BTC straight into my Pool account and be ready to go within the allotted confirmation target.
It eliminates a source of headaches with regard to using LND; currently, LND will tie up its entire wallet balance as unusable for every single withdrawal until said withdrawal transaction confirms. Allowing for external deposits will remove this factor, allowing for simultaneous use of LND as well as Pool.

Pool already has a mechanism for creating a deposit address and for awaiting the confirmation of a transaction before fully opening the account. Given this, it should be possible for poold to be funded via an external wallet.
My thought process is to have the account creation function create an address which can be deposited to, and then watch for deposits to that address, rather than expecting a specific Outpoint from LND.
I can spend a little time implementing some of this via a PR, as well.

This is likely a bit more involved than you anticipate. We originally examined this type of deposit, but then went with the current model as it's: simpler and also ends up saving on chain fees in a batch given each participant has only a single output. Using a single deposit address would also mean address re-use for each user of pool that used the feature. Generating new addresses each time is possible, but would then require some interaction from the server so it can properly increment the "account nonce" on its end each time.

LND will tie up its entire wallet balance as unusable for every single withdrawal until said withdrawal transaction confirms

It's possible to spend these outputs in an unconfirmed manner using the existing lnd APIs. It's also relatively straight forward to allow poold to spend lnd's unconfirmed change to service a deposit or new account creation as well.

One other complication here is that it would then be possible to have partial account expiry (so some of your account UTXOs expire while others are still valid), which complicates the account management state machine. This would be alleviated somewhat by having auto-renew implemented (so renew an account each time it's part of a batch), which is planned as well.

It's also relatively straight forward to allow poold to spend lnd's unconfirmed change to service a deposit or new account creation as well.

Can you point me to where I'd find documentation on that? It would be hella useful for my use case.

The code doesn't allow it at the moment, but the relevant section would be:

For new accounts: https://github.com/lightninglabs/pool/blob/91ce63795066f212f25d056812c0725221f49a81/account/manager.go#L503
For deposits: https://github.com/lightninglabs/pool/blob/91ce63795066f212f25d056812c0725221f49a81/account/manager.go#L1600

For new accounts, we'll want to perform manual coin selection as done with deposits instead of using SendOutputs, which does not allow specific inputs to be used.


setting nodejs timeout values to match ALB config by adding server.ke…

…epAliveTimeout
Proposed changes

What changed

Why did it change

Issue tracking

ATB-XXX

Checklists

Environment variables or secrets

[ ] No environment variables or secrets were added or changed

[ ] Application configuration is up-to-date
[ ] Documented in the README
[ ] Added to deployment steps
[ ] Added to local startup config

[ ] Let the tech lead know if the Dockerfile has changed

Other considerations

[ ] Demo to a BA, TA, and the team.
[ ] Recorded demo shared on #govuk-accounts-tech
[ ] Update README with any new instructions or tasks

Hi @jessica-li-01 - could fill in the PR title to describe what has happened and fill in the form providing what it is asking for.

Hi @jessica-li-01 - could fill in the PR title :  to describe what has happened and fill in the form providing what it is asking for.

Hi @jessica-li-01 - could fill in the PR title to describe what has happened and fill in the form providing what it is asking for.


Make zod faster

I found repo with validators benchmarks. Zod in it not so good as I expected.
I wrote simple test for my model and custom validation function, which fully corresponds to the Zod scheme.
When running, I saw a 15x (1500%) execution speed difference.

My performance test code:
const z = require('zod')

const DATA = {
    id: 1,
    yo: 1,
    email<EMAIL_ADDRESS>    username: 'admin',
    score: 0,
    roles: ['owner'],
    userProfiles: [
        { profileId: 1, profileName: 'Yo!1' },
        { profileId: 2, profileName: 'Yo!2' },
    ],
}
const TRY_COUNT = 100000

function validateZod(data, tryCount) {
    const userSchema = z.object({
        id: z.number(),
        email: z.string(),
        username: z.string(),
        score: z.number(),
        roles: z.array(
            z.enum(['admin', 'manager', 'owner', 'customer'])
        ),
        userProfiles: z.array(
            z.object({ profileId: z.number(), profileName: z.string() })
        ),
    })

    for(let i = 0; i < tryCount; i += 1) {
        userSchema.parse(data)
    }
}

function validateCustom(data, tryCount) {
    function parse(data) {
        if (!data || typeof data !== 'object') {
            throw new Error('data is not object')
        }

        const out = {}

        if (typeof data.id !== 'number') {
            throw new Error('id is not number')
        }
        out.id = data.id

        if (typeof data.email !== 'string') {
            throw new Error('email is not string')
        }
        out.email = data.email

        if (typeof data.username !== 'string') {
            throw new Error('username is not string')
        }
        out.username = data.username

        if (typeof data.score !== 'number') {
            throw new Error('score is not number')
        }
        out.score = data.score

        if (!Array.isArray(data.roles)) {
            throw new Error('roles is not array')
        }
        out.roles = data.roles.map((data, i) => {
            if (!data || typeof data !== 'string') {
                throw new Error(`roles[${i}] is not string`)
            }

            if (!['admin', 'manager', 'owner', 'customer'].includes(data)) {
                throw new Error(`roles[${i}] is not string`)
            }

            return data
        })

        if (!Array.isArray(data.userProfiles)) {
            throw new Error('userProfiles is not array')
        }
        out.userProfiles = data.userProfiles.map((data, i) => {
            if (!data || typeof data !== 'object') {
                throw new Error(`userProfiles[${i}] is not object`)
            }

            const out = {}

            if (typeof data.profileId !== 'number') {
                throw new Error(`userProfiles[${i}].profileId is not number`)
            }
            out.profileId = data.profileId

            if (typeof data.profileName !== 'string') {
                throw new Error(`userProfiles[${i}].profileName is not string`)
            }
            out.profileName = data.profileName

            return out
        })

        return out
    }

    for(let i = 0; i < tryCount; i += 1) {
        parse(data)
    }
}

// Zod test

let startTs = Date.now()
let ticksStart = process.hrtime.bigint()

validateZod(DATA, TRY_COUNT)

let ticksEnd = process.hrtime.bigint()
let endTs = Date.now()

console.log(`ZOD: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

// Custom validation test

startTs = Date.now()
ticksStart = process.hrtime.bigint()

validateCustom(DATA, TRY_COUNT)

ticksEnd = process.hrtime.bigint()
endTs = Date.now()

console.log(`CUSTOM: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

I know the overhead is unavoidable, but maybe you can optimize the code by precompiling the Zod-schema and make 15x gap smaller.
My suggestion is to compile the Zod-schema with a new Function() (at least for simple cases) and use it when calling validation.
function compileSchema(zodSchemaDefinition) {
    let code = ''

    if (zodSchemaDefinition.nullable) {
        code += `if (value === null) return null;\n`
    }

    if (zodSchemaDefinition.optional) {
        code += `if (value === undefined) return undefined;\n`
    }

    if (zodSchemaDefinition.type !== 'string') {
        // MVP compiler: only for strings at the moment
        return null
    }

    code += `if (typeof value !== 'string') throw new Error('value is not a string');\n`

    if (zodSchemaDefinition.min) {
        code += `if (value.length < ${zodSchemaDefinition.min}) throw new Error('Minimum value length is ${zodSchemaDefinition.min}');\n`
    }

    code += 'return value;'

    return new Function('value', code)
}

// Schema must get from Zod schema metadata
const validator = compileSchema({
    nullable: true,
    type: 'string',
    min: 2,
})

console.log('Generated validator function:')
console.log(validator.toString())
console.log()
console.log('Value "qwe" validation: ', JSON.stringify(validator('qwe')))
console.log()
console.log('Next validation of number 1 must throw error')
console.log(validator(1))

Did I write something wrong or just have no idea how to implement this?

I'm sure you did nothing wrong, and if you submit a pull request to implement that change I'm sure many will be interested.

I'm working on a solution that uses a similar process. The difficulty does not lie in algorithmic logic but rather in the cleanliness of the solution.
In one afternoon I have already created a "build" of x function string, number and Object.
This is the result:
const zodSchema = zod.object({
    firstname: zod.string().trim().toLowerCase().max(15).min(2),
    lastname: zod.string().trim().toUpperCase().max(15).min(2),
    age: zod.coerce.number().min(16),
    email: zod.string().email(),
    addresse: zod.object({
        postCode: zod.coerce.string().regex(/[0-9]+/),
        city: zod.string().max(50),
        number: zod.number()
    }).strict()
}).strict()

const zodAcceleratorSchema = ZodAccelerator.make(zodSchema);

let startTs = Date.now()
let ticksStart = process.hrtime.bigint()

for(let i = 0; i < 10000; i += 1) {
    zodAcceleratorSchema({
        firstname: "  Mike ",
        lastname: " gnogno  ",
        age: 21,
        email<EMAIL_ADDRESS>        addresse: {
            postCode: 22778,
            city: "Paris",
            number: 67
        }
    })
}

let ticksEnd = process.hrtime.bigint()
let endTs = Date.now()

console.log(`ZodAccelerator: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

startTs = Date.now()
ticksStart = process.hrtime.bigint()

for(let i = 0; i < 10000; i += 1) {
    zodSchema.parse({
        firstname: "  Mike ",
        lastname: " gnogno  ",
        age: 21,
        email<EMAIL_ADDRESS>        addresse: {
            postCode: 22778,
            city: "Paris",
            number: 67
        }
    })
}

ticksEnd = process.hrtime.bigint()
endTs = Date.now()

console.log(`ZOD: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

Hi @colinhacks, I think I found a good recipe for zod !
The diagram below could be accelerated and not just a little.
const zodSchema = zod.object({
    firstname: zod.string().trim(),
    lastname: zod.string().nullable(),
    age: zod.coerce.number(),
    email: zod.string(),
    gender: zod.enum(["boy", "girl"]),
    connected: zod.boolean(),
    createdAt: zod.coerce.date(),
    addresse: zod.object({
        postCode: zod.coerce.string().transform((val) => val + "turbodab"),
        city: zod.string(),
        number: zod.number()
    }),
    test: zod.tuple([zod.string().trim(), zod.number()]).rest(zod.string().default("lolo")),
    tutu: zod.union([ // responsible for the slowdown
        zod.literal("123"),
        zod.literal("456"),
        zod.object({
            test: zod.string()
        }),
    ]).optional().catch("123")
}).array()

parsed data :
const data = Array.from({length: 10}).fill({
    firstname: "  Mike ",
    lastname: null,
    age: 21,
    email<EMAIL_ADDRESS>    gender: "girl",
    connected: true,
    createdAt: "2024-09-13",
    addresse: {
        postCode: 22778,
        city: "Paris",
        number: 67
    },
    test: ["temp  ", 1, "cheese", "ok", undefined],
    tutu: "litote",
})

test :
let startTs = Date.now()
let ticksStart = process.hrtime.bigint()

for(let i = 0; i < 100000; i += 1) {
    zodSchema.parse(data)

    // zodSchema.parse("testtests")
}

let ticksEnd = process.hrtime.bigint()
let endTs = Date.now()

console.log(`Zod: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

startTs = Date.now()
ticksStart = process.hrtime.bigint()

for(let i = 0; i < 100000; i += 1) {
    zodAccelerator.parse(data)

    // zodAcceleratorSchema("testtests")
}

ticksEnd = process.hrtime.bigint()
endTs = Date.now()

console.log(`ZodAccelerator: Time sec ${(endTs - startTs) / 1000}, ticks: ${Math.round(Number(ticksEnd - ticksStart) / 1000)}`)

https://github.com/duplojs/duplojs-zod-accelerator

I’m all for making Zod faster, but I’d like to still be able to run it on the edge. A lot of the reason why some of those other validators are faster is because they use eval or new Function.
@mathcovax I couldn’t find any documentation explaining the approach you used for accelerating, but it seems it’s using eval?

I’m all for making Zod faster, but I’d like to still be able to run it on the edge. A lot of the reason why some of those other validators are faster is because they use eval or new Function.
@mathcovax I couldn’t find any documentation explaining the approach you used for accelerating, but it seems it’s using eval?

hello 🙂, actually I was a bit stingy in terms of explanation ^^'.
eval or new Function there is no difference, the strategy remains the same.
For faster execution, a custom function is built in a string which will then interpret pars the functions eval or new Function.
It’s not complex has realized, the challenges are architectural. It’s not easy to make code that manipulates string maintainable.

Thanks for confirming. Unfortunately a lot of browser Content Security Policies will not allow dynamic code execution, so if this were to become the default approach used by Zod, it would make it impossible to use it in those environments. See Next.js docs for example. Fine if it's opt-in or opt-out. Another approach I've seen used (by Ajv I believe?) is making it possible to compile the schemas with a CLI utility so that there is no dynamic code execution at runtime.

I reassure you, use zodAccelerator does not prevent use zod normal. zodAccelerator just create a function will start from a schema.
the strategy you are talking about remains the same, it generates a function in a string but instead of using "eval" it creates a file containing the function then it asks for import.
It makes me think that I have to set up the possibliter of just generated a chain skipping the step of the "eval".
Thanks for the info 😉

any workaround for zodunion slowness from https://github.com/colinhacks/zod/issues/2613#issuecomment-1964700733 ?

une solution de contournement pour zodunion et autres lenteurs du #2613 (commentaire) ?
À l'heure actuelle, je reçois 500 Ko de données via WebSockets et il faut 608 ms pour les valider, ce qui est trop, je pense.

hi, this issue has been solved. Zod is almost x100 faster on the operation fat union with zodAccelerator. watch the bench mark in the read me

Yes, but your solution does not work in strict csp environments


add insecure oci support

Please provide a description of this PR:
fixed https://github.com/istio/istio/issues/36571

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@zirain: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
616ef3f40f349e980e1134e84c09c9cb0c0d5a09
link
true
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bianpengyuan as we talk on Slack, ptal

@daixiang0 @bianpengyuan @costinm
Let me summarize:

for long-term, we should support imagePullSecret on WasmPlugin, IIRC, @bianpengyuan is working on
for test/no-production env, users can simply turn INSECURE_WASM_PLUGIN  on.

updated, @bianpengyuan ptal

replace INSECURE_WASM_PLUGIN with WASM_INSECURE_REGISTRIES
support plain text repo

@zirain: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-security-multicluster_istio
618e5fadc0960a55b90d501b9b598a3319ecf864
link
true
/test integ-security-multicluster_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test integ-security-multicluster_istio

LGTM, do we have the ability to run unit tests against this?

not sure about it, will take a look in the near future.
@bianpengyuan need one more approval, PTAL

Still need doc for this.

Still need doc for this.

any idea about the right place for a doc? maybe istio.io?

I think the document can be with imagepull secret

Failed to enable this feature:
istioctl install --set meshConfig.defaultConfig.proxyMetadata.WASM_INSECURE_REGISTRIES=my-registery

Any interesting log at proxy? Turning on debug log would help debugging.

Same log as below:
could not fetch Wasm OCI image: could not parse url in image reference: Get "https://registry.fi.intel.com/v2/": x509: certificate signed by unknown authority

Same log as below:
could not fetch Wasm OCI image: could not parse url in image reference: Get "https://registry.fi.intel.com/v2/": x509: certificate signed by unknown authority

let me take a look at this, seems something error with the plaintext support on
// fallback to http based request, inspired by [helm](https://github.com/helm/helm/blob/12f1bc0acdeb675a8c50a78462ed3917fb7b2e37/pkg/registry/client.go#L594)
	_, err = remote.Get(ref)
	if err != nil && strings.Contains(err.Error(), "server gave HTTP response") {
		wasmLog.Infof("fetch with plain text from %s", url)
		return name.ParseReference(url, name.Insecure)
	}

I'll take a look at this

@zirain I think https://github.com/istio/istio/blob/50c0cc307ee273e34e35e324fc130371bd536bdf/pkg/wasm/imagefetcher.go#L151 should return nil here instead of err.

@zirain I think
https://github.com/istio/istio/blob/50c0cc307ee273e34e35e324fc130371bd536bdf/pkg/wasm/imagefetcher.go#L151
should return nil here instead of err.

seems I forgot to respect fetchOpts when doing remote.Get, I need to rebuild my test env to verify.

In response to a cherrypick label: new pull request created: #37532


[LIVY-124] Allow jar/pyfile/file to be uploaded when creating a batch session

What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/LIVY-124
Allow upload jar/python/other files only with Livy http connection for batch session.

Create batch session with delayed flag
Upload files with set-file, add-file, add-jar and/or and-pyfile
Start session with start (i.e. submit to spark)

How was this patch tested?

New unittest for batch session
New unittest for batch servlet
Tested manually against my spark cluster

@pmsgd would you please give an example of how to use this new feature?

@jerryshao We have separate platform and customer apps. They are connected only through http proxy with authentication and authorization. There is no direct hdfs access, so we need upload apps to spark cluster with Livy.
Submit example:

POST /batches with delayed = true
Upload files, one or more of POST /batches/{batchId}/set-file,   /batches/{batchId}/add-jar, /batches/{batchId}/add-pyfile and /batches/{batchId}/add-file - this set file, jars, pyFiles and files variables in batch request
GET /batches/{batchId}/start - finish upload and really submit request to Spark
the rest is like standard batch submission

Codecov Report

Merging #91 into master will decrease coverage by 0.41%.
The diff coverage is 53.01%.

@@             Coverage Diff              @@
##             master      #91      +/-   ##
============================================
- Coverage     71.49%   71.08%   -0.42%
- Complexity      793      800       +7
============================================
  Files            97       97
  Lines          5402     5474      +72
  Branches        801      821      +20
============================================
+ Hits           3862     3891      +29
- Misses         1019     1049      +30
- Partials        521      534      +13

Impacted Files
Coverage Δ
Complexity Δ

.../apache/livy/server/batch/CreateBatchRequest.scala
67.64% <100%> (+2.02%)
20 <1> (+2)
:arrow_up:

...apache/livy/server/batch/BatchSessionServlet.scala
51.85% <28.12%> (-33.87%)
3 <0> (ø)

...la/org/apache/livy/server/batch/BatchSession.scala
78.57% <66.66%> (-7.95%)
18 <7> (+5)

...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala
52.94% <0%> (-2.95%)
0% <0%> (ø)

...main/java/org/apache/livy/rsc/ContextLauncher.java
81.86% <0%> (-2.46%)
18% <0%> (ø)

...ain/java/org/apache/livy/rsc/driver/RSCDriver.java
79.41% <0%> (-0.85%)
42% <0%> (-1%)

...in/java/org/apache/livy/rsc/rpc/RpcDispatcher.java
67% <0%> (+3%)
20% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e3f45a0...5c343de. Read the comment docs.

Thanks for the info. The API usage seems not so straightforward from my point. Let me think a bit on this thing.

My goal was small addition with minimal changes to current API. So far only batch request is enhanced with delayed flag plus 4 new methods for file uploads. First call (batch create) creates hdfs temp area for uploads and ensures that all files will be deleted after job is finished. Uploads should be definitely separate calls for each file. API could be only single upload method with type parameter but it will completely different from interactive session. I think similar API for batches and interactive sessions is better. Last call is "start" - do you know better way how to let Livy know that all files are uploaded and job could be submitted ?

I was thinking if we can:

associate file uploading with session creation post request, so that we can handle this in one request, not sure if Http protocol support this.
Don't change the semantics of session creation request, upload files to Livy Server before session creation, and use these files for session creation.

I understand that so far your approach might be the easiest way to achieve this feature, but it also makes the request a little strange, I'm just thinking if there's any better solution for this.

yes - it is possible (see https://tools.ietf.org/html/rfc7578#section-4.3) but not very common. For better reliability it is better to split upload into separate calls (lack of reconnect support). I don't know how scalatra/livy handle file uploads but very common practice is to load complete file(s) into memory before processing. We can run into memory problems with larger apps.
if file is uploaded before session it can hang on hdfs indefinitely without session. Some timeout is possible but such maintenance service looks like basic rest hdfs API. Even more, interactive session upload should be then also refactored and handled this way (at least as optional upload possibility).

Can you explain more about this "if file is uploaded before session it can hang on hdfs indefinitely without session"? I'm not sure why it will be hang on hdfs indefinitely?

Imagine this scenario:

client uploads some file
client crashes
file is uploaded to hdfs but session is never started and file never deleted

After some time hdfs will be filled with these "orphan" files. I think periodic maintenace task for deleting old unused files is worse than one rest call.

After some time hdfs will be filled with these "orphan" files. I think periodic maintenace task for deleting old unused files is worse than one rest call.

This should not be a big problem. We don't have to upload files to hdfs, we can cache the files locally in LivyServer, when LivyServer starts Spark Application, SparkSubmit will upload the dependencies automatically. Also we can add retention mechanism to clean orphaned files.

Can you please check how other Spark service handle such file upload issue? like Toree, Spark JobServer, or maybe others.

Keeping old unused files is bad idea everywhere. New retention mechanism is possible but why not use something what is currently in use and working ? And if we go with uploads independent to batches next step is similar filesystem like in JobServer.

@zjffdu @ajbozarth can you please comment on this feature, especially on the API design side. Thanks!

How about we design new APIs for such such usage scenario?

request a new session/batch id from Livy server.
Start preparing new resources(jars, py files, archieves).
submit JSON data to start this session/batch

This is a similar process to submit application on YARN: 1) request a new application id. 2) preparing all the resources to launch application. 3) start application.
What do you think @pmsgd ?

You almost exactly described this pull request. The only difference is between 1. and 3. where the most batch information are send in 1. and not 3.. 3. is only command to start spark job.

Though they're similar, the semantics are different. In your proposal you're doing POST/POST/GET, but here is GET/POST/POST.
I was thinking that:

We should add the new API endpoints to support this feature, not just one flag in the request.
We should support both interactive and batch session.
We should manage the dependencies locally in LivyServer, not in HDFS.

Are you really talking about complete API rework for batches and interactive sessions ? My goal was to keep current API as is, only with minimal backward compatible changes. I definitely have not enough time and scala skills for such big task.

Yes, I'm thinking of adding a new API set for such usage, since this is quite common in the environment where user cannot submit dependencies to HDFS before launching the session. We can add a version flag to support this new API set. There shouldn't have lots of changes in the background.

FYI I'm reading through both the discussion and the code on this trying to solidify my position before giving a longer review/opinion

Ok so I've been thinking about it for a while and though I have quite a few issues with the details of this implementation (which I'll submit as a detailed review if decide to move forward with this strategy), I like this strategy better. I'll put my more detailed thoughts into the new JIRA: https://issues.apache.org/jira/browse/LIVY-471

Any progress in this functionality. For me, supporting uploads of jars via the REST API is very desirable.

Hello Martin, no further progress from me - we decided to use another solution.

What is the "another solution"?
Can you point me to a PR?

I am not authorized to post this information but it is commercial solution, not open source.

it's a useful feature, with this user can submit local jars to remote spark cluster.


Add clear text button to some inputs

Added a Clear text button for removing text from some inputs/textarea
Resolves: https://trello.com/c/J17Is4tI/1130-publisher-clearing-a-text-area-2

Thanks Panos. Can you ask Rachel to review this?

Thanks Panos. Can you ask Rachel to review this?

Will do, no problem


No failing else in init command file tree processing

https://github.com/wuespace/telestion-client/blob/d3a6f8fc4ce0930aa7250478811409318ce32066/packages/telestion-client-cli/src/lib/init/process-template-tree.js#L56
We should probably include an else branch here to fail with a useful error message so that future breaking changes can easily be detected.

Disregard because of implementation in 7f0aac19b24438a990bfc34577673f7107594d7a


Capabilities support for Azure Cosmos DB and minor fixes

This checklist is used to make sure that common guidelines for a pull request are followed.
General Guidelines

[X] The PR has modified HISTORY.rst describing any customer-facing, functional changes. Note that this does not include changes only to help content. (see Modifying change log).

Command Guidelines

[X] Each command and parameter has a meaningful description.
[X] Each new command has a test.

(see Authoring Command Modules)

View a preview at https://prompt.ws/r/Azure/azure-cli/5451
This is an experimental preview for @microsoft users.

@tjprescott I have added tests for all of the control-plane commands, including the changes for the commands added in this pull request. The tests for data-plane commands will be worked on by @moderakh

Address the CI failures but otherwise, LGTM! Thanks @dmakwana!


 I'm using your Player now for over 15 years already, cause my old PCs back in the days couldnt handle the "modern" players out there well together with gaming :P
Okay, to the issue :
When you input a Audio CD with 3 Tracks for example and switch to a CD with more Tracks, it dont load at all or only list till the count of the previous CD (3 Tracks before, next CD is 5 but still only shows 3 from 5)
When i restart the programm it's correct again.
Keep it up ! I will still use your Player xD

Hi!
I wasn't able to reproduce this issue (I tried ejecting a CD with 9 tracks, then loading a CD with 13 tracks).
When you say switching CDs, are you switching between multiple drives? If that's the case, then it'll probably have to wait until I get another external drive to test with.
One bug I did notice was that you can delete CD tracks from the list with the delete key, which you shouldn't be able to. They do show up again on a restart (or ejecting/loading the disc), but I'll tidy that up for the next version.
Thanks,
James

Hmm, im using Win7 64bit. Sometimes even when no CD is loaded at first it dont load it at all. I tested it with a program of my own and it works there. Didnt looked into the source what your using, but im using the window msg "WM_DEVICECHANGE" with testing the wparm value for "DBT_DEVICEARRIVAL" and "DBT_DEVICEREMOVECOMPLETE" , which works pretty fine. It happends with the my physical drive. I got 1 physical DVD/CD drive and 3 virtual drives from deamon tools. Dont know if this helps.
PS : It even seems like it dont even recognize my cd switching.

It's taken awhile, but a finally got around to getting a couple of extra external CD drives to test with.
I couldn't reproduce the exact issue of seeing the wrong track count, but it did reveal some related issues and performance problems, so these have hopefully now been resolved in v4.14.


fix: disable refetch on focus

Disable refetching data on focus

:tada: This PR is included in version 1.29.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Can Play Video that MediaDetector Can't Read

Hey all,
I'm having some issues getting the WPF-MediaKit libray MediaDetector class to read in some video files -- specifically, some mp4 files. I can play them (graphically speaking) just fine using the library, but MediaDetector doesn't seem to be able to load the files, which is a feature I need so that I can save video thumbnails. LoadMedia() thinks that there are no streams.
Seeing as how both the playback functionality and MediaDetector use DirectShow, I'm a bit confused as to why this is happening. I've gotten MediaDetector to work before, but I reinstalled my DirectShow video codecs and suddenly it doesn't work.
The filters I am attempting to use are here: https://github.com/Nevcairiel/LAVFilters . Before installing the filters, I cannot play the specified mp4 files, but after I install them, MediaKit is able to play them...so the filters themselves seem to be fine.
Perhaps the put_Filter() function of DirectShow.DES.IMediaDet needs to be used to load filters first?
Thanks for any assistance.

hi,
ran into the same issue when trying to get the AvgFrameRate from a File (mjpeg).
Did you solve this issue somehow?

@simonbuehler : No, I wasn't able to figure out how to solve the issue unfortunately.
I actually ending up not using WPF-MediaKit at all in my project. I needed to take dynamic screenshots of a video on a background thread without rendering a MediaElement to the screen, and I actually figured out that you can do so via native C# WPF code (this is possible because MediaElement can actually natively play MP4; don't know about mjpeg).
I can't post much of the code due to it being for my client, but if you need to do something similar, I'll point you to http://stackoverflow.com/questions/423373/how-do-i-write-a-method-to-open-start-playing-then-return-the-duration-of-an-a and then tell you that you can get an image of a MediaPlayer element that you've created and used to open media with...
RenderTargetBitmap bmp = new RenderTargetBitmap(640, 480, 96, 96, PixelFormats.Pbgra32);
DrawingVisual dv = new DrawingVisual();
using (DrawingContext dc = dv.RenderOpen())
{
    dc.DrawVideo(player, new Rect(0, 0, 640, 480));
}
bmp.Render(dv);

Good luck! Sorry I wasn't able to help.

For grabbing an image see the Wiki FAQ and offscreen see Wiki Headless Examples.
I do not know how ti fix the MediaDetector and since nobody has offered a PR to fix that I am closing the issue.


the resolution and classes of imagenet dataset

I found that in DiffBIR, the resolution of imagenet data is set to 512x512, but as far as I know, the resolution of imagenet is 256x256. And I have another question, which classes of image net do you choose? image1k or 11k? Thank you very much

We use center crop to generate GT with a resolution of 512.
ImageNet-1k, about 1.3M training images.

thanks for reply! I have reproduced the results. I close this issue