Difference between a task being enabled and a task executing

Using:
root@kapacitor1:/tmp# kapacitor version
Kapacitor 0.2.3 (git: master 7ea4a548108e82a214d6c7a94b66a59ffeda1bd9)

What does it mean for a task to be enabled, but not executing? For example:
root@kapacitor1:/tmp# kapacitor list tasks
Name                          Type      Enabled   Executing Databases and Retention Policies
disk_alert                    stream    true      false     ["telegraf"."default"]

Where Enabled is set to true, but Executing is set to false. When this happens, redefining and/or re-enabling doesn't seem to help. Deleting the task and re-adding does work, though. I've also been re-adding and modifying the tickscript fairly often, so I'm not sure if that could have an effect. There also doesn't seem to be anything logged when this occurs.
So my questions are:

Is this expected?
Is this a symptom of an error in the tickscript or tickscript's execution?
Is there a way to 're-execute' a task?

If a TICKscript is not valid etc you will get an error when you try to enable the task and will be able to fix it at that point. But it is possible to right a TICKscript that is valid syntactically but could have an error based on the data the task is given(i.e. you could reference a field name that doesn't exist). If you get an that is data dependent the task will stop executing.
When a task is enabled but it is not executing it means it had an error and has therefore stopped executing. Using kapacitor show disk_alert you can get more information on the state of the task and why it is not executing.
Running kapacitor reload disk_alert should re-execute the task but if its getting an error when the task first starts running then it will quickly change to Executing: false state again.

Thanks for the explanation @nathanielc! It ended up being a type issue in one of my lambda expressions. I also had debug logging on, so I missed the error message.
Thanks again!


Fix HeaderCreator leaking file descriptors

Fixes #127 with little cleanup

Thanks!
Tests are failing, not even compiling. Please fix,

@hseeberger Codacy suggests to do something with scala.util.Try instead of using try-catch-finally. I don't shee how Try can help here. Do you see a more idiomatic way to implement the try-catch part?

Should compile now, missed the usage in test.

Codacy is wrong! Using Try for implementation only (not as return value) needlessly allocates and should be avoided. Good old try-catch is just fine.


MacOS electron app is not code signed

What steps did you take and what happened:
Downloaded the Octant 0.17 electron build and opened it.  MacOS displayed a dialog saying "Octant" cannot be opened because the developer cannot be verified.

This dialog can be worked around by alt-clicking on the app and then selecting open.
What did you expect to happen:
The app should open on launch like any other Mac application.
Anything else you would like to add:
Details on code signing
Environment:

Octant version (use octant version): 0.17.0
Kubernetes version (use kubectl version): n/a
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): 11.2.1

We are working through the process of getting code signing in place for macOS. This involves some hurdle traversal within VMware.


[PHPhotoLibrary authorizationStatus] status is not correct in iOS14+

it is not correct using [PHPhotoLibrary authorizationStatus] to get the authorization status in iOS14+. such as when you limit your app can access only some photos, then [PHPhotoLibrary authorizationStatus] will return PHAuthorizationStatusAuthorized which is not correct.
to solve the problem, you should use [PHPhotoLibrary authorizationStatusForAccessLevel:PHAccessLevelReadWrite] in iOS14+ to get the correct result.

Hi @zhouxing5311
I'm not sure I can get the context of the issue. Please elaborate the issue in more detail by indicating whether it is related to the Flutter framework or a specific plugin you are using.

@zhouxing5311
I try modifying authorizationStatus to authorizationStatusForAccessLevel:PHAccessLevelReadWrite but the result is still the same. I'm testing on iPhone 14 Pro Max, iOS 16.0.

Demo
https://user-images.githubusercontent.com/104349824/196847304-9161e386-497e-4049-9790-78a95635bc7f.mov

Modified snippet code
- (void)checkPhotoAuthorizationForAccessLevel API_AVAILABLE(ios(14)) {
  PHAuthorizationStatus status = [PHPhotoLibrary authorizationStatusForAccessLevel:PHAccessLevelReadWrite];

Which iOS version did you try?

@zhouxing5311 I try modifying authorizationStatus to authorizationStatusForAccessLevel:PHAccessLevelReadWrite but the result is still the same. I'm testing on iPhone 14 Pro Max, iOS 16.0.
Demo
Screen.Recording.2022-10-20.at.10.02.09.mov
Modified snippet code
Which iOS version did you try?

maybe i did't make my point, image selecting behavior has no problem, the question is PHPhotoLibrary authorizationStatus return the result 'PHAuthorizationStatusAuthorized', but the result should be 'PHAuthorizationStatusLimited' , here is my code result.

the reason I bring this up is that i want to show diffrent image select page when your permission is limitted by modifying the source code, so i need to get the correct authorizationStatus. maybe you will add the feature like this to source code?

or when i finish this, i can submit a pull request. i think this is a better experience for user who selected the limited photos. the present behavior may confusing the user that the original image selecting is invalid.

@zhouxing5311
Please check this issue if it's similar to your case: #110176
Also, I re-test this thing again, looks like it has been implemented similar to what you're describing (although the Select more photos dialog shows only once and the design is different), please see in demo:

Demo (Select more photos)
https://user-images.githubusercontent.com/104349824/196854649-e08a285c-77b1-443d-adf9-1f14bc45d588.mov

@zhouxing5311 Please check this issue if it's similar to your case: #110176
Also, I re-test this thing again, looks like it has been implemented similar to what you're describing (although the Select more photos dialog shows only once and the design is different), please see in demo:
Demo (Select more photos)

hi, i don't think this is a similar case, the handling for limit photos is not suitable i think. after selecting the special photos, next time the image picker should show the photos only which is selected by user at the begin time. or the previous selecting is useless, users can not understand what's meaning of the begining select.

@zhouxing5311 Labeling the issue as a proposal for further insight from the team.
Plugin: image_picker_ios: ^0.8.6+1

flutter doctor -v (stable and master)
[✓] Flutter (Channel stable, 3.3.5, on macOS 12.6 21G115 darwin-x64, locale en-VN)
    • Flutter version 3.3.5 on channel stable at /Users/huynq/Documents/GitHub/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision d9111f6402 (9 hours ago), 2022-10-19 12:27:13 -0700
    • Engine revision 3ad69d7be3
    • Dart version 2.18.2
    • DevTools version 2.15.0

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode14.0.1.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 70.0.5
    • Dart plugin version 222.4167.21

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.72.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.50.0

[✓] Connected device (3 available)
    • iPhone 14 Pro Max (mobile) • 332079AF-E895-4685-910E-7B1E18B0C6B8 • ios            • com.apple.CoreSimulator.SimRuntime.iOS-16-0 (simulator)
    • macOS (desktop)            • macos                                • darwin-x64     • macOS 12.6 21G115 darwin-x64
    • Chrome (web)               • chrome                               • web-javascript • Google Chrome 106.0.5249.119

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[!] Flutter (Channel master, 3.5.0-10.0.pre.67, on macOS 12.6 21G115 darwin-x64, locale en-VN)
    • Flutter version 3.5.0-10.0.pre.67 on channel master at /Users/huynq/Documents/GitHub/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision e62d519cb5 (18 minutes ago), 2022-10-24 21:56:13 -0400
    • Engine revision 75bfcd73ca
    • Dart version 2.19.0 (build 2.19.0-330.0.dev)
    • DevTools version 2.18.0
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/huynq/Library/Android/sdk
    • Platform android-33, build-tools 31.0.0
    • ANDROID_HOME = /Users/huynq/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
    • Xcode at /Applications/Xcode14.0.1.app/Contents/Developer
    • Build 14A400
    • CocoaPods version 1.11.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2021.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] IntelliJ IDEA Community Edition (version 2022.2.2)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 70.0.5
    • Dart plugin version 222.4167.21

[✓] IntelliJ IDEA Community Edition (version 2022.1.1)
    • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart

[✓] VS Code (version 1.72.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.50.0

[✓] Connected device (3 available)
    • SM T225 (mobile) • R9JT3004VRJ • android-arm64  • Android 12 (API 31)
    • macOS (desktop)  • macos       • darwin-x64     • macOS 12.6 21G115 darwin-x64
    • Chrome (web)     • chrome      • web-javascript • Google Chrome 106.0.5249.119

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Most of the discussion above has been focused on the code, but without any context on what the behavior is, which makes it hard to understand what exactly is being reported. What is the developer and/or end-user impact of the code changes you are suggesting?

hi, the behavior we wants to is this. image_picker will open this page when users used the limiting photo permission.
https://user-images.githubusercontent.com/14874355/198476452-6821888e-c910-4ea0-b177-740947c65b3a.mov
i have finish this at my forked repo, you can this change. this is only a suggestion, the behavior now is ok too.
https://github.com/zhouxing5311/plugins/commit/67f4eea7371aefdc01b0dbedfced260c7d091674

Hi!
We are facing this issue too. If the user only gives access to some of their photos, while picking photos all of them can be picked. It is not the expected behaviour. Only the selected photos should be shown while picking images.
So for example in the video captured by @huycozy :

at the beginning we can see the selected photos - only the first row, the first three images are selected
after that while picking photos all six photos can be picked. The expected behaviour would be that while picking photos only the 3 should be shown to select from.

Hi!
We are facing this issue too. If the user only gives access to some of their photos, while picking photos all of them can be picked. It is not the expected behaviour. Only the selected photos should be shown while picking images. So for example in the video captured by @huycozy :

at the beginning we can see the selected photos - only the first row, the first three images are selected
after that while picking photos all six photos can be picked. The expected behaviour would be that while picking photos only the first 3 should be shown to select from.

i have changed some code in my forked repo to change the behavior. you can see the details in this change https://github.com/zhouxing5311/plugins/commit/67f4eea7371aefdc01b0dbedfced260c7d091674
if you want to test the new behavior, you can add the following override code in your pubspec.yaml.
dependency_overrides:
  image_picker_ios:
    git:
      url: https://gitee.com/zhouxing5318/image_picker_ios

cc @vashworth


Fix riscv32 atomics and fix tests on 32-bit platforms

This fixes tests on 32-bit platforms. Previously, the Encode would encode a usize and decode a u64. This works fine on 64-bit platforms, but on 32-bit platforms this results in a mismatch.
Force usize to be encoded as a u64 even on 32-bit platforms.
Additionally, the 32-bit riscv target does not implement AtomicU64 or AtomicI64, so remove those impelementations on that platform.

Thanks for the suggestions. I've annotated all Atomic implementations.
I assume that Bool is the same as 8, and usize is the same as ptr.
I'm not sure what the best way is to deprecate the feature, so I removed it wholesale.

Can you pull in the new cross-platform build pipeline, and see if bincode now compiles for your target arch?

It does build for my platform still, so it looks good to me!
[10:10:39 am] E:/Code/bincode> cargo build --lib --target riscv32imac-unknown-xous-elf
   Compiling virtue v0.0.7
   Compiling bincode_derive v2.0.0-rc.1 (E:\Code\bincode\derive)
   Compiling bincode v2.0.0-rc.1 (E:\Code\bincode)
    Finished dev [unoptimized + debuginfo] target(s) in 2.39s
[10:10:43 am] E:/Code/bincode>

If you run cargo fmt and cargo clippy I think this is good to merge

Alright, I just ran cargo fmt to fix that up.

Thanks!


Use react-frame-component for preview pane

- Summary
Uses react-frame-component for preview pane instead of custom iframe code. Fixes #307.

- Test plan
Existing tests pass; manually tested that previewing and scrolling work correctly in both Firefox and Chrome.

- Description for the changelog
Use react-frame-component for preview pane

@Benaiah update still coming for this?


Add a trialType descriptor to Trial entity

Many breeding programs have different phases to them: 1st year / 2nd year / 3rd year, Preliminary Yield Trial / Advanced Yield Trial / Area Trial, etc. Adding a trialType descriptor to a Trial entity would be a good way to capture these top-level categories.
In annuals, entries from one phase are selected and advanced to the next phase in the following growing season. While check lines often remain consistent, the entries tested in a given trial phase/type will typically be completely different each year. The entries in the 2024 Advanced Yield Trial (AYT24) are different from those in the 2023 Advanced Yield Trial (AYT23).
The Study entities of a given type could be grouped into a single Trial entity (all Advanced Yield Trial studies become a single AYT Trial) or into separate Trial entities (there is an AYT22 Trial, an AYT23, an AYT24, etc.).
In either case, a user would need to parse names, descriptions, or dates in order to either separate the data (e.g. pull only the most recent year of data from a Trial containing multiple years of AYT Studies) or combine it (e.g. merge multiple AYT Trial entities in order to look at trends over time).
If you had a trialType descriptor you could create a unique Trial for each year's unique phase, keeping them distinct for easier access to only a specific cycle, while having a quick way to group them into top-level categories.
This would be similar to the existing studyType descriptor in the Study entity. Having both studyType and trialType would also give users the ability to contain multiple studyTypes in a single trialType.
For instance, many programs evaluate a given phase of breeding lines in separate yield trials and disease trials. You could now have a Trial entity with a trialType of "1st-year evaluation" that contained some Studies with studyType "Small-plot yield trial" and some with studyType "Headrow stem rust nursery"
It would also simplify the process of labeling finite (non-cyclical) experiments. For example, say a breeder runs a GWAS experiment across ten unique site-years, forming ten Studies grouped into a single Trial. They wish to label this "GWAS" somehow, so they can pull it along with data from other GWAS experiments. Instead of doing this ten times, adding it to each studyType field, they could add it once to the overall Trial entity.

@tyrwh note that you have already trialStage : The current stage of this trial. Example include Advance, Elite, EPT, EST, Prelim, PST.
Much of what you describe can be considered implementation (breeding program) specific.  You could still have some documented guidelines / examples. Perhaps some documented example use cases is required.

@daveneti Is trialStage a descriptor already you mean? I don't see that on the 2.1 release specifications. If that exists already, then that works perfectly

AHHHH! Yes you are right, I was looking at our internal extensions to BrAPI. In that case, can I propose to call the descriptor 'trialStage'. I think that trialType is too generic and that what you are proposing above is more a stage?

PS, you can have a trialType, which is like a studyType. This would be things like 'Yield Trial', 'Nursery' etc. Whereas 1st year / 2nd year / 3rd year, Preliminary Yield Trial / Advanced Yield Trial etc would be stages in a 'Yield Trial'.

Haha ok great, I was worried I just didn't actually read the docs before posting
Adding the specific trialStage would valuable I think. It's a close to universal concept, if not universal, in annual crop breeding. I think it would be hard to find a diagrammatic representation of annual crop breeding scheme where you could not say "the rows, columns, or boxes represent stage in the sense that we're using the term here"
If it's applicable to a significant chunk of breeding program activities, that seems like grounds to think about inclusion, even if it may be broadly not applicable to some types of trial.
Applied breeding programs may not have a publication tied to many of their trials (if any), but it's important metadata to enough research programs that it warrants inclusion as part of the universal Trials spec. Same with GPS - if you're in annuals and don't own an RTK planter, you probably won't have observation-level GPS coords. But it's useful in enough cases that it makes sense to be a part of the universal Observations spec.
I think adding the more generic trialType would still be useful even if trialStage is added, for a few reasons-
Having both trialType and studyType allows you to capture two meanings of "type"
If you perform 3 Studies, grouped in one Trial, to evaluate drought resistance for an eventual GWAS, and you have a single Type descriptor, is it more important to capture the type of experiment - "GWAS" or "diversity panel screening"- or the type of trait examined- "drought screening"?
There are cases where you may want to query on either, and it's hard to have both with studyType alone. You could concatenate them and call it "Drought GWAS", but then you're trying to parse substrings
Having both trialType and studyType allows you to capture two granularity levels of "type"
Most cereal programs I have seen run separate yield trials and disease evaluations on at least one phase of breeding lines. Often the disease evaluations themselves are subdivided, with different diseases evaluated in separate trials so they can each be inoculated singly.
So say you have three disease Studies on distinct diseases from a single evaluation phase, forming a single Trial to capture all your disease ratings from that germplasm set.
On a Study level, the studyType could just be "disease evaluation", but if could be very useful to differentiate using studyType of "Northern leaf blight evaluation", "Gray leaf spot evaluation", and "Smut evaluation" and have the trialType be "disease evaluation".
You would have the ability to easily query either to find data on any particular disease or on any diseases, whereas with studyType alone you'd have to choose one and find a workaround for the other.


Make registration screen more convincing and cheerful

Problem
As a new user use the mobile app and decide to modify a product, we ask him to register.
It is a big step for most user to take and current screen to do so lacks explanation and encouragement.
Proposed solution
Change current screen to:

welcome and encourage (We are so happy you want to contribute)
explain why we need them to signup (for quality)

It can also be a screen displayed before asking registration.

@teolemon  I think you have mockups !

https://www.figma.com/design/nFMjewFAOa8c4ahtob7CAB/Mobile-App-Design-(Quentin)?node-id=6633-20361&node-type=canvas&t=40yFSMFjg863zlb0-0


Check failed

F tensorflow/core/framework/tensor_shape.cc:211] Check failed: num_elements_ <= kMaxElements<PHONE_NUMBER>864 vs.<PHONE_NUMBER>776)
I can't figure out why the above error happens. What's wrong ?

There's a hard bound on the number of elements in a Tensor (about a trillion), do you really need to use a 9 trillion element tensor?

I found I have created a big SparseTensor


test: E2E on replica cluster with basic monitoring queries

Closes #355
Signed-off-by: Jitendra Wadle<EMAIL_ADDRESS>

/test

/test


Use asyncio.sleep during backoff

https://github.com/okta/okta-sdk-python/blob/master/okta/request_executor.py#L337
In the linked line of code time.sleep is used instead of asyncio.sleep. This can cause some very long (and unnecessary) waiting time.

This has been resolved in 2.9.0


Merge all inline operations into main operations (InlineMerge should be Merge).

Will be a part of build > v1.9.0-beta1


p2p: test SecretConnection handshake

Write a test for Secret connection, which covers the whole handshake (exchanging ephemeral keys and so on) #happypath. We probably also want to cover additional scenarios like when a peer is not responding or responding with invalid data (do we want to try fuzzing here?) at each step of the handshake.
Acceptance Criteria

[x] happy path test
[ ] test where the remote peer is not responding
[ ] test where the remote peer provides invalid data

    #[test]
    fn test_evil_peer_does_not_share_eph_key() {
        let (_pipe1, pipe2) = pipe::bipipe_buffered();

        let thread1 = thread::spawn(|| {
            let mut csprng = OsRng {};
            let privkey1: ed25519::Keypair = ed25519::Keypair::generate(&mut csprng);
            let conn1 = SecretConnection::new(pipe2, &privkey1, Version::V0_34);
            assert_eq!(conn1.is_ok(), true);
        });

        thread1.join().expect("thread1 has panicked");
    }

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

    #[test]
    fn test_evil_peer_does_not_share_eph_key() {
        let (_pipe1, pipe2) = pipe::bipipe_buffered();

        let thread1 = thread::spawn(|| {
            let mut csprng = OsRng {};
            let privkey1: ed25519::Keypair = ed25519::Keypair::generate(&mut csprng);
            let conn1 = SecretConnection::new(pipe2, &privkey1, Version::V0_34);
            assert_eq!(conn1.is_ok(), true);
        });

        thread1.join().expect("thread1 has panicked");
    }

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

This could be done in a follow-up after the tests are finalised, as first improvement based on it.

test where peer is not responding fails at the moment because we're using blocking reads (read_exact). we probably need to use mio or threads (which will require static lifetime) to read with a timeout.

This could be done in a follow-up after the tests are finalised, as first improvement based on it.


[REVIEW] Multi-column groupby support

closes #121
This PR aims to address three different cases requiring "mutli-column" groupby statistics:

"Joint" Encoding: As requested in #121, the user may want to perform a "joint" encoding of multiple categorical columns. this is relevant when the elements in distinct columns correspond to the same "kind" of data.  In this case, we want to concatenate the targeted columns to perform the groupby operation (in GroupbyStatistics).

"Combo" Encoding: The user may also want to perform a groupby on two different kinds of columns.  This would be required to convert 2+ categorical columns into a single encoded column, or to join groupby-aggregation statistics on multiple reference columns.

Multi-column Groupby Join (self explanatory - uses "combo" version of Groupby aggregation)

TODO:

[x] Support both multi-column groupby scenarios in CategoryStatistics (concat_groups arg controls multi-column behavior)
[x] Support encode_type="joint" for multi-column Categorify
[x] Support encode_type="combo" for multi-column Categorify
[x] Support multi-column GroupBy Op (now called JoinGroupby)

RFC:

[ ] This PR renames CategoryStatistics to GroupbyStatistics -- Okay ?
[ ] This PR renames GroupBy to JoinGroupby -- Okay ?
[ ] This PR dissallows replacement for Categorify for encode_type="combo" -- Okay?

CI should start passing when the latest cudf is picked up :)

@rjzamora Thanks so much for this PR.  I did a small test with a toy example. The resulted new_gdf  from JoinGroupby op does not seem correct. This is my test:
gdf = cudf.DataFrame({'A': [1, 2, 2, 6, 6, 8, 3], 'B': [2, 4, 4, 7, 7, 2, 5], 'C': [1.0 , 4.0, 4.0, 8.0, 8.0, 5.0, 1.0]})

proc = nvt.Workflow(
    cat_names=['A', 'B'],
    cont_names=['C'],
    label_name=[])

#drop_duplicates with groupby
proc.add_feature(JoinGroupby(columns=['A', 'B'],
                             cont_names=['C'],
                             stats=['count']))

train_dataset = nvt.Dataset(gdf, engine='parquet')

proc.apply(train_dataset, apply_offline=True, record_stats=True, output_path='./dataset/test/', shuffle=False, out_files_per_proc=1)

This is generated new_gdf:
new_gdf= cudf.read_parquet('./dataset/test/*.parquet')
new_gdf.head()
	C	A	B	A_count	B_count
0	1.0	1	2	1	2
1	4.0	2	4	2	2
2	4.0	2	4	2	2
3	8.0	6	7	2	2
4	8.0	6	7	2	2

The expected output should be like below (we group the gdf based on A and B columns, apply count on the C column):
	A	B	C	C_count
0	1	2	1.0	1
1	2	4	4.0	2
2	2	4	4.0	2
3	6	7	8.0	2
4	6	7	8.0	2
5	8	2	5.0	1
6	3	5	1.0	1

also from nvt.ops.GroupbyStatistics() we should be getting cat_stats.C.parquet, but I get cat_stats.A.parquet and cat_stats.B.parquet stat files.
Please let me know if I am missing or confusing anything here.

We only perform a “count” aggregation on the groups (cat colums), since the “count” of “C” is just the count of the group (the values in “C” make no difference). If you asked for a “sum“ or “std” aggregation, you would see a “C”-specific agg column, but the name will still specify a corresponding cat column.
Also, we only write parquet files for each Cat group, not each continuous column.  So, you should not see a cat_stats.C.parquet - the C-specific stats must be in the other files.  However, since you only asked for “count”, there are no C-specific stats

@rjzamora thanks.  I agree on this logic: We only perform a “count” aggregation on the groups (cat colums). But still I am a bit confused on the generated results. Let's talk offline.

@rjzamora thanks. I agree on this logic: We only perform a “count” aggregation on the groups (cat colums). But still I am a bit confused on the generated results. Let's talk offline.

Sounds good.  In case it helps... I jut realized this discussion is under the "Multi-column" PR, so you probably intended for the results to correspond to a multi-column groupby.  If so, you need to do: JoinGroupby(columns=[['A', 'B']], ...) (note that the outer list allows multiple column groups, and the inner list is the actual multi-column group).  If you make that change, you should get a "cat_stats.A_B.parquet" file that looks like:

   A  B  A_B_count
0  1  2          1
1  2  4          2
2  3  5          1
3  6  7          2
4  8  2          1

This probably gives you waht you expect after the merge.


Handle zocalo returning multiple crystals

In some samples there could be more than one crystal. In this case we're prototyping Zocalo returning a list of centres, with the strength of each one. The json returned from zocalo will look something like:
[
  {
    "centre_of_mass": [
      4.636612650133922,
      6.038044420033806,
      2.612136870930547
    ],
    "max_count": 156032.0,
    "max_voxel": [
      3,
      7,
      1
    ],
    "n_voxels": 65,
    "total_count": 6671044.0
  },
  {
    "centre_of_mass": [
      3.5,
      11.5,
      11.039976728723405
    ],
    "max_count": 103952.0,
    "max_voxel": [
      3,
      11,
      11
    ],
    "n_voxels": 2,
    "total_count": 192512.0
  }
]

Acceptance Criteria

Artemis moves to the correct centre in this case (sensibly choosing which crystal to go to)
System tests and fake_zocalo updated to reflect this

Testing on this will be happening 26/02/23

By way of example, for https://ispyb.diamond.ac.uk/dc/visit/cm31105-5/dcg/8424506 Zocalo will return a json string looking something like:
[
  {
    "bounding_box": [
      [
        0,
        1,
        1
      ],
      [
        5,
        9,
        10
      ]
    ],
    "centre_of_mass": [
      2.902023886839894,
      5.55764956832089,
      5.045271936984979
    ],
    "max_count": 156032.0,
    "max_voxel": [
      1,
      7,
      3
    ],
    "n_voxels": 161,
    "total_count": 12083976.0
  },
  {
    "bounding_box": [
      [
        8,
        10,
        2
      ],
      [
        12,
        12,
        4
      ]
    ],
    "centre_of_mass": [
      10.430489554536866,
      11.006540224656192,
      3.1069566434545264
    ],
    "max_count": 103952.0,
    "max_voxel": [
      11,
      11,
      3
    ],
    "n_voxels": 9,
    "total_count": 571540.0
  }
]

The results will be sorted by total_count in descending order.
Note: I have fixed the axis ordering in the values returned by Zocalo, such that they will now be x,y,z rather than z,y,x

Testing went well and Artemis handles this fine. We should explicitly do the ordering of results though, rather than assuming Zocalo  will do so

As discussed, we should order the results again Artemis side. Whilst not 100% necessary we agreed that the deciding on order was Artemis' responsibility. This also makes it easier for us to do different ordering in the future too


Docs: Create PXE docs

make it clear PXE is run client side  (update diagram in foundatoinal concetpts)
what it actually is, what it stores, how it stores it
what are oracles
put into advanced concepts and core concepts

closed via #4021


does nncf support post training quantization and jupyter notebook?

Hello,
Thanks for this great project. But

does nncf support post training quantization? I did not find related information/example yet.
it seems nncf does not  jupyter notebook?  There will be error if import modules in notebook.

Thanks.

Greetings, @liwei46!

The NNCF is based upon PyTorch, which itself is a training framework. Now, you could possibly use it just to import models and convert them to ONNX (if you have no GPU, for instance), but this does not seem to be the major use case for PyTorch.
Anyway, it is possible to do post-training quantization with NNCF, in a sense. Just specify the "range" and the "batchnorm_adaptation" initializer sections in the NNCF config as described in the docs: one, two. Note that you should supply a dataset for the initialization to happen (you can use a helper function for that), since initializing activation quantizers entails running the model inference one or multiple times. Once you do that, during the create_compressed_model call both initializations will happen while only using forward passes of the model (no backward), and the model that is output from that create_compressed_model will then contain all quantizers configured as best as currently possible with NNCF without running training. You can try exporting the model to ONNX at that point right away, or running evaluation scripts directly in PyTorch to see if the resulting no-training compressed model quality is suitable for your purposes.
Still, for best results, I would recommend to actually train the NNCF-wrapped model obtained after the create_compressed_model call so that the quantizer parameters and the model weights can adapt together to the low-precision execution scenario. This is what NNCF specializes in.
If you uncompromisingly require post-training quantization for OpenVINO-powered inference, then you can use a tool that is specialized exactly in that - the Post Training Optimization Toolkit, distributed as part of the OpenVINO packages. It has a larger scale of algorithms that specifically improve post-training quantization performance and quality. See POT documentation and general OpenVINO docs for instructions on how to use it.

Could you please give more details on the errors during Jupyter import and post logs/versions of NNCF and Jupyter you used? We will be happy to look into that and enable Jupyter notebook imports if for any reason it is broken right now.

Thanks for the quick and helpful reply.
Below is the error message when I type:
from nncf.dynamic_graph.version_agnostic_op_names import VersionAgnosticNames in jupyter notebook.
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-76d95093b247> in <module>
----> 1 from nncf.dynamic_graph.version_agnostic_op_names import VersionAgnosticNames

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/__init__.py in <module>
     15
     16 # Required for correct COMPRESSION_ALGORITHMS registry functioning
---> 17 from .binarization import algo as binarization_algo
     18 from .quantization import algo as quantization_algo
     19 from .sparsity.const import algo as const_sparsity_algo

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/binarization/algo.py in <module>
     18 from torch import nn
     19
---> 20 from nncf.algo_selector import COMPRESSION_ALGORITHMS
     21 from nncf.binarization.layers import BINARIZATION_MODULES, BinarizationMode, WeightBinarizer, ActivationBinarizer, \
     22     ActivationBinarizationScaleThreshold, BaseBinarizer

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/algo_selector.py in <module>
     12 """
     13
---> 14 from .compression_method_api import CompressionAlgorithmBuilder, CompressionAlgorithmController
     15 from .registry import Registry
     16

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/compression_method_api.py in <module>
     27 from nncf.config import NNCFConfig
     28 from nncf.dynamic_graph.graph_builder import create_mock_tensor
---> 29 from nncf.initialization import DataLoaderBNAdaptationRunner
     30 from nncf.nncf_logger import logger as nncf_logger
     31 from nncf.nncf_network import NNCFNetwork

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/initialization.py in <module>
      9
     10 from nncf.nncf_logger import logger as nncf_logger
---> 11 from nncf.quantization.init_range import MinMaxInitializer, ThreeSigmaInitializer, MeanMinMaxInitializer
     12 from nncf.quantization.init_range import PercentileInitializer
     13 from nncf.structures import QuantizationPrecisionInitArgs, QuantizationRangeInitArgs, BNAdaptationInitArgs

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/quantization/__init__.py in <module>
     12 """
     13 # Required for correct QUANTIZATION_MODULES registry functioning
---> 14 from . import layers

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/quantization/layers.py in <module>
     23 from nncf.functions import clamp
     24 from nncf.nncf_logger import logger as nncf_logger
---> 25 from .quantize_functions import symmetric_quantize, asymmetric_quantize, ExportQuantizeToFakeQuantize, \
     26     get_scale_zp_from_input_low_input_high, ExportQuantizeToONNXQuantDequant, TuneRange
     27 from ..layer_utils import COMPRESSION_MODULES

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/quantization/quantize_functions.py in <module>
     14 import warnings
     15
---> 16 from .extensions import QuantizedFunctionsCPU, QuantizedFunctionsCUDA
     17 from ..dynamic_graph.patch_pytorch import register_operator
     18 from ..functions import STRound, clamp

~/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/quantization/extensions.py in <module>
     39 QuantizedFunctionsCPU = load(
     40     'quantized_functions_cpu', CPU_EXT_SRC_LIST, extra_include_paths=EXT_INCLUDE_DIRS,
---> 41     verbose=False
     42 )
     43

~/anaconda3/envs/nncf/lib/python3.7/site-packages/torch-1.5.0-py3.7-linux-x86_64.egg/torch/utils/cpp_extension.py in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module)
    896         verbose,
    897         with_cuda,
--> 898         is_python_module)
    899
    900

~/anaconda3/envs/nncf/lib/python3.7/site-packages/torch-1.5.0-py3.7-linux-x86_64.egg/torch/utils/cpp_extension.py in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module)
   1063         build_arguments=[extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths],
   1064         build_directory=build_directory,
-> 1065         with_cuda=with_cuda
   1066     )
   1067     if version > 0:

~/anaconda3/envs/nncf/lib/python3.7/site-packages/torch-1.5.0-py3.7-linux-x86_64.egg/torch/utils/_cpp_extension_versioner.py in bump_version_if_changed(self, name, source_files, build_arguments, build_directory, with_cuda)
     42                                 with_cuda):
     43         hash_value = 0
---> 44         hash_value = hash_source_files(hash_value, source_files)
     45         hash_value = hash_build_arguments(hash_value, build_arguments)
     46         hash_value = update_hash(hash_value, build_directory)

~/anaconda3/envs/nncf/lib/python3.7/site-packages/torch-1.5.0-py3.7-linux-x86_64.egg/torch/utils/_cpp_extension_versioner.py in hash_source_files(hash_value, source_files)
     14 def hash_source_files(hash_value, source_files):
     15     for filename in source_files:
---> 16         with open(filename) as file:
     17             hash_value = update_hash(hash_value, file.read())
     18     return hash_value

FileNotFoundError: [Errno 2] No such file or directory: '/home/wei_li/anaconda3/envs/nncf/lib/python3.7/site-packages/nncf-1.3.2-py3.7.egg/nncf/extensions/src/quantization/cpu/functions_cpu.cpp'

@liwei46 try uninstalling NNCF and reinstalling the latest version from develop (or the v1.4.1 release version at least). There have been some packaging problems with the v1.3.2 of the exact kind that you are describing, that should be fixed starting from v1.4.

thanks.


[refactor & pipelineX][pick fix] Pick fix of predicate pushdown to pipelineX

pick https://github.com/apache/doris/pull/25901 to pipeline
refactor is_acting_on_a_slot, ref: https://github.com/apache/doris/pull/25950

run buildall

run buildall


Update scripts to use new bugbug HTTP answer

Following https://github.com/mozilla/bugbug/pull/670, the classification field
was renamed into class, update the needed scripts.
Wait until https://tools.taskcluster.net/groups/RCUA7UVDSVmZVoiQM948QQ/tasks/B6SiGklhSBOR9hbq_7fxQQ/details has finished before merging this PR.

Coverage remained the same at 25.597% when pulling add3b4d0388f8804156329d7e85965e3af40b48e on Lothiraldan:update-new-bugbug-http-response into 30eac8c6c0f61785ca26c1f1333230cb6ad51d51 on mozilla:master.

The new service has been deployed, merging


chore: add a test case to check naming function with capitalized values

Signed-off-by: ankitm123<EMAIL_ADDRESS>

/approve


Jupyter Notebook does not execute !python or %run in the selected conda environment.

Environment data

VS Code version: 1.53.0
Extension version (available under the Extensions sidebar): v2021.1.502429796
OS and version: macOS 10.15.7
Python version (& distribution if applicable, e.g. Anaconda): 3.7.9
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Relevant/affected Python packages and their versions: XXX
Relevant/affected Python-related VS Code extensions and their versions: XXX
Value of the python.languageServer setting: Jedi

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]
Expected behaviour
Python script run using !python or %run within a cell in a jupyter notebook uses the same environment as the jupyter notebook
Actual behaviour
Python script uses base conda environment rather than the selected conda environment
Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Open Jupyter notebook in VS Code. Select a kernel with a Conda environment with a package that isn't in the base env.
Run a .py script using ! python .py or %run .py that imports a package in the selected conda environment.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

To address the issues I deleted VS Code by opening my Mac Applications folder, right clicking on VS Code, selecting Move to Trash. I then downloaded VS Code 1.52.1 from here, unzipped the download, dragged VS Code back to my Applications folder. Opened VS Code and the Jupyter Notebook displayed above, re-ran the cells and NO ERROR! Conclusion: there is an issue with 1.53.0!
So, reverting from 1.53.0 back to 1.52.1 got me back to work but it doesn't fix the 1.53.0 issue.


chore(sdk/ts): bump tsx dependency

Thanks to https://github.com/privatenumber/tsx/issues/582 being resolved, we can bump tsx to its latest version.

Typescript integ tests are failing like:

Hmm strange, I'm going to rerun the CI, just to see if its flacky or not

I convert this PR back to draft since it seems it breaks the current code, I need to investigate why.

I found a fix, by updating the package.json with "type": "module", it solves the issue.
It seems tsx also dropped commonJS support like in #7636
However, it seems v4.15.6 fixes the issue


fix: delete broken layer when duplication detected

When a layer is duplicated but not recorded in layer_db, it should be treated as "broken" and be deleted. Then the newly pulled layer data can be written, instead of raising an error and exit.
cc @arronwy

Do we need to wait after the code freeze phase?