Documents not summarizing

We have received another incident of a document accumulating a large number (40k) of OPs without summarization. The ICM incident contains a Kusto query for the document.
We are seeing a lot of:
fluid:telemetry:DeltaManager:WrapError	Nack: Submit a summary before inserting additional operations		Nack: Submit a summary before inserting additional operations
fluid:telemetry:Container:ContainerWarning	Service busy/throttled: Nack: Submit a summary before inserting additional operations		Service busy/throttled: Nack: Submit a summary before inserting additional operations
fluid:telemetry:DeltaManager:WrapError	Nack: Submit a summary before inserting additional operations		Nack: Submit a summary before inserting additional operations
fluid:telemetry:Container:ContainerWarning	Service busy/throttled: Nack: Submit a summary before inserting additional operations		Service busy/throttled: Nack: Submit a summary before inserting additional operations
fluid:telemetry:Container:ContainerWarning	Service busy/throttled: Nack: Submit a summary before inserting additional operations		Service busy/throttled: Nack: Submit a summary before inserting additional operations
fluid:telemetry:DeltaManager:WrapError	Nack: Submit a summary before inserting additional operations		Nack: Submit a summary before inserting additional operations
fluid:telemetry:DeltaManager:WrapError	Nack: Submit a summary before inserting additional operations		Nack: Submit a summary before inserting additional operations

And the summarizer logs show the following pattern:
fluid:telemetry:SummaryManager:CreatingSummarizer
fluid:telemetry:Summarizer:StoppingSummarizer		parentNotConnected
(...)
fluid:telemetry:SummaryStatus:Behind
fluid:telemetry:SummaryManager:CreatingSummarizer
fluid:telemetry:Summarizer:StoppingSummarizer		parentNotConnected
(...)
fluid:telemetry:SummaryManager:CreatingSummarizer
fluid:telemetry:Summarizer:StoppingSummarizer		parentNotConnected
fluid:telemetry:SummaryStatus:Behind
fluid:telemetry:Summarizer:NotStarted		DifferentComputedSummarizer
fluid:telemetry:SummaryManager:RunningSummarizer_end
(...)
fluid:telemetry:SummaryManager:CreatingSummarizer
fluid:telemetry:SummaryStatus:Behind
fluid:telemetry:Summarizer:StoppingSummarizer		parentShouldNotSummarize
fluid:telemetry:SummaryStatus:Behind
fluid:telemetry:SummaryManager:CreatingSummarizer
fluid:telemetry:Summarizer:StoppingSummarizer		parentNotConnected

Incident 257910811 may be related, however that particular instance recovered by itself.
cc @vladsud @pleath

First, some related issue I've been looking yesterday, with deeper investigation into 2 recent docs that hit similar problems: https://github.com/microsoft/FluidFramework/issues/7589
When it comes to docId == "M%2F1pui%2FeRmFe%2FD8nLpqwO1YhTAYYENnGQUOL4g8Kx%2B4%3D", here are some observations:
First events started on 2021-08-31, for loaderId == "14e785a3-5e6e-45fd-838c-7f126c00d931", runtimeVersion = 0.46.0
What I see is that whenever interactive client disconnects due to 10K ops nack, summarizer client does not close and keeps loading just fine, but then eventually when it fully catches up ("connects"), it closes itself with fluid:telemetry:Summarizer:NotStarted	DifferentComputedSummarizer
I.e. fluid:telemetry:Summarizer:StoppingSummarizer / parentNotConnected event that fires before tells us that this summarizer should go away, but it proceeds forward despite of it to evaluate if last summary needs to be performed (which gets us out of this state), but it never happens as by the time summarizing client is fully up to date, it bails out due to DifferentComputedSummarizer check.
This query looks at overall picture - both successes & failures:

union Office_Fluid_FluidRuntime_*
| where Data_docId == "M%2F1pui%2FeRmFe%2FD8nLpqwO1YhTAYYENnGQUOL4g8Kx%2B4%3D"
| where Data_clientType == "noninteractive/summarizer"
| where Data_reason != "" or Data_error != "" or Data_eventName contains "Summarize_end" or Data_eventName contains "Summarize_cancel"
| where Data_eventName notcontains "DeltaManager"
| where Data_eventName notcontains "Container"
| project Event_Sequence, Event_Time, Data_eventName, Data_reason, Data_error, Data_clientType, Data_containerId

What I see from here is

For most part, when summaries are not going, parentNotConnected is followed by DifferentComputedSummarizer, and the later is the core reason we should look into why summaries are not being created.

Note that DifferentComputedSummarizer keyword does not exist in main any more. Looking through PRs, I've removed this code in https://github.com/microsoft/FluidFramework/pull/7230/files, which points to # 4 point in description of this PR on why this check was wrong and caused last Summary to not run properly - exactly the case we observer here!

There are cases where parentNotConnected is not followed by DifferentComputedSummarizer, but these cases are rarer, and it feels like it should be lower priority case to look into. For example, there is a block of such cases at the end (after 2021-09-07T17:29:35.75Z)

That said, if you look at nacks, you will see that there are a lot of nacks on 09/14. And looking at details (loaderId == "8adea59a-a1ec-4d1d-8855-8ff7ba60148c") you see that summarizer client was not attempted to be created for very long time after last summarizer quite. This looks to be https://github.com/microsoft/FluidFramework/pull/7321 problem. I checked - it is not in 0.46!

So to sum up, we need latest fixes. I have not checked, but I believe all of them are on 0.47 (need to double check), and all these sessions are for runtime = 0.46, so we are missing all of the proactive work I've done in this space.

This seems to be related to the old 10K ops issue, and possible fixed?


Swift 2.0 and bitcode enable

Hello,
I move my project to swift 2.0 on XCode 7.0 and I have to disable bitcode to ensure that my app will compile.
I get this error with bitcode enable:
ld: '/Users/xxxxxxx/Documents/WORK/myproject/kingpin-frameworks/iOS/kingpin.framework/kingpin' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target. for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
But with bitcode disable I got an other warning when I try to push my app on itunes connect :
the resulting api analysis file is too large
Thanks for your help

@notiles, thanks for reporting this. I'll take a look at it asap within 2 days.

Perfect, Thanks a lot.

@notiles could you please try kingpin-frameworks from enable-bitcode branch? There I have enabled ENABLE_BITCODE and rebuilt frameworks with it.
Unfortunately I do not have time to test how submission to iTunesConnect works so I am asking you to do that. Let me know if that works for you.

Ok I will try it today! Thanks a lot.

@stanislaw I build the app with bitcode enable but now it's an other framework which is not bitcode enabled in my app...
But KingPin look working well with bitcode enable !!
Thanks a lot.

Good to know! Thanks for pushing this update live :)


Update model.py

The previous code was using the 24h model for forecasts > 24h. However, the step would only go up by 6h. Also for a 120h forecast (e.g.), the previous code would have used the 6h model which is not advised due to error growth.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Example not compiling

I have cloned the repo, ran npm i, and then npm start, but it is failing to compile. I am wondering if I need to do some configuration on my side?
Here is the error log:
[1/14] Building src/ReactDOMRe.mlast
FAILED: src/ReactDOMRe.mlast
/usr/local/lib/node_modules/bs-platform/lib/bsc.exe -pp "/usr/local/lib/node_modules/bs-platform/lib/refmt3.exe --print binary"    -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/ben/develop/reroute/examples/node_modules/bs-platform/lib/ocaml' -bs-no-version-header -no-alias-deps -color always -c -o src/ReactDOMRe.mlast -bs-syntax-only -bs-binary-ast -impl /Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReactDOMRe.re
File "/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReactDOMRe.re", line 26, characters 6-7:
Error: 2960: <UNKNOWN SYNTAX ERROR>
File "/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReactDOMRe.re", line 1:
Error: Error while running external preprocessor
Command line: /usr/local/lib/node_modules/bs-platform/lib/refmt3.exe --print binary '/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReactDOMRe.re' > /var/folders/1c/6kl9yv6x59d58kc181lwnbdr0000gn/T/ocamlpp439446

[2/14] Building src/ReasonReact.mlast
FAILED: src/ReasonReact.mlast
/usr/local/lib/node_modules/bs-platform/lib/bsc.exe -pp "/usr/local/lib/node_modules/bs-platform/lib/refmt3.exe --print binary"    -w -30-40+6+7+27+32..39+44+45+101 -nostdlib -I '/Users/ben/develop/reroute/examples/node_modules/bs-platform/lib/ocaml' -bs-no-version-header -no-alias-deps -color always -c -o src/ReasonReact.mlast -bs-syntax-only -bs-binary-ast -impl /Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReasonReact.re
File "/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReasonReact.re", line 82, characters 2-3:
Error: 373: <UNKNOWN SYNTAX ERROR>
File "/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReasonReact.re", line 1:
Error: Error while running external preprocessor
Command line: /usr/local/lib/node_modules/bs-platform/lib/refmt3.exe --print binary '/Users/ben/develop/reroute/examples/node_modules/reason-react/src/ReasonReact.re' > /var/folders/1c/6kl9yv6x59d58kc181lwnbdr0000gn/T/ocamlpp9703a1

[7/14] Building src/ReactEventRe.mliast
ninja: error: rebuilding 'build.ninja': subcommand failed
Failure: /usr/local/lib/node_modules/bs-platform/lib/ninja.exe
 Location: /Users/ben/develop/reroute/examples/node_modules/reason-react/lib/bs

So sorry! It appears bs-platform had gone stale on my machine. I got it cleaned and rebuilt, and it's working.


HDDS-3312. Remove server-side dependencies from hdds/ozone-common.

What changes were proposed in this pull request?
hadoop-ozone/common and hadoop-hdds/common projects are common between client and server. Therefore we should remove any server side utilities / dependencies from them as they would be added to the client classpath which makes harder the Ozone client / ozonefs adoption.

hadoop-hdds should depend on a minimal set of hadoop dependencies (can be managed a separated technical project / pom.xml)
code shared between server side projects (and not with the client) should be moved to the framework
OM related code should be moved to the ozone-manager instead of ozone/common
For technical reason some test utilities are imported to the Ozone code base (from Hadoop)

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3312
How was this patch tested?
Full integration test. This is a project organization patch without real change of the logic. If the clients can work without class-not-found exception. It should be fine.

It will have a few conflicts with #701 so if you need to update this anyway, I'd appreciate if you could review my PR. :)

@adoroszlai: #701 is already conflicting. Do a rebase and I will merge it.

Thanks for merging #701 first.  In return I have merged master into this branch and resolved conflicts: https://github.com/adoroszlai/hadoop-ozone/commit/0a76d0cf0b02016962b4b32bb58f29fe4090f20f
Checkstyle still fails due to the new module hadoop-hdds-hadoop-dependency-client, fix is in progress in #768.

Checkstyle problem is fixed on master.
Hadoop-related acceptance tests are failing:
hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs
==============================================================================
Test hadoop dfs                                                       | FAIL |
1 != 0

due to
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/ozone/shaded/org/apache/http/client/utils/URIBuilder
	at org.apache.hadoop.fs.ozone.BasicOzoneFileSystem.initialize(BasicOzoneFileSystem.java:141)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
	at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
	at org.apache.hadoop.fs.shell.CommandWithDestination.getRemoteDestination(CommandWithDestination.java:195)
	at org.apache.hadoop.fs.shell.CopyCommands$Put.processOptions(CopyCommands.java:259)
	at org.apache.hadoop.fs.shell.Command.run(Command.java:175)
	at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.ozone.shaded.org.apache.http.client.utils.URIBuilder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

https://github.com/apache/hadoop-ozone/pull/744/checks?check_run_id=558258420
https://github.com/adoroszlai/hadoop-ozone/runs/559314792

Merged to the master. Thanks the review (and the help with the rebase) @adoroszlai


Show image from camera live at Internet, Is possible ?

Looking the webcam-capture-live-streaming
I'm thinking if is possible let the images live at Browser captured of Webcam.default() ?
For example:
I created an Jetty embedded in my application java Desktop, I would like to knows  if using the Webcam.default() for capture images live, I transmit
online, to see this image accessing the address defined in my Jetty embedded, as this:
http://ip:8084/Live Streaming

and I could see the images from any place live .
My Jetty embedded is it:
Server server= new Server(8084);
WebAppContext context = new WebAppContext(server, null,"/");
context.addServlet(new ServletHolder(new LiveStreamingServlet()), "/LiveStreaming");  // LiveStreaming is fake!
server.setHandler(context);
server.start();
server.join();

I'm trying to understand it, for implements after.
Thks !

@mcicolella, if you are still interested, here you can find very basic example demonstrating how images feed can be transported to web page over WebSockets.
https://github.com/sarxos/webcam-capture/tree/master/webcam-capture-examples/webcam-capture-websockets


feat: social media sharing

Fix prettier warnings.
[warn] src/app/[locale]/app/(public)/share-presentation/[id]/credentialPanel.tsx
[warn] src/app/[locale]/app/(public)/share-presentation/[id]/formatedView.tsx
[warn] src/app/[locale]/app/(public)/share-presentation/[id]/page.tsx
[warn] src/components/ShareCredentialModal/index.tsx
[warn] Code style issues found in 4 files. Run Prettier to fix.

Resolve prettier issues:
./src/app/[locale]/app/(public)/share-presentation/[id]/page.tsx
1:10  Error: 'ResolvingMetadata' is defined but never used.  unused-imports/no-unused-imports


Strike range and formatting

Added an input to the study to limit strike range. Cleaned up some formatting.

Merging changes that add strike range.


Update MD file on new branch 2017-11-13_22-23-08

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef1.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Test support for esp devices

Let's check the efforts needed to add a sample or at least support for the esp framework from espressif.
There is a request here for support on top of the esp-ieee802154 crate.
Since this zigbee crate is planned as no_std it should be possible to archive.

I'd be interested to test this for my project :) https://github.com/maufl/sensor2


Support higher versions of react-bootstrap

Thank you so much for your package!
I am using your package in a project that includes react-bootstrap 2.4.x and it works great. But I have to include --legacy-peer-deps to install because it uses a higher version

@jaywilz any chance you and tag a new release for NPM?
Thank so much!

@skylerkatz I was going to until realising react-bootstrap 2.0.0 introduces breaking changes.
I'll need to spend some time creating a new version, then will release on NPM

Thanks for your response.
Let me know if there is anything you would like some help with and I can take a stab at it as well.

Let me know if there is anything you would like some help with and I can take a stab at it as well.

There's a draft PR: https://github.com/jaywilz/react-bootstrap-country-select/pull/16
Feel free to contribute, if you fancy it


Links to commits open new tab

Closes: #1210
Added newTab prop to the Link component which allows toggling of the target='_blank' attribute of the  element

Given these are links to external sites, do we need to worry about adding a rel="noopener" for a bit of added security?

Given these are links to external sites, do we need to worry about adding a rel="noopener" for a bit of added security?

I'm pretty sure browsers now have that functionality built in (https://stackoverflow.com/questions/50709625/link-with-target-blank-and-rel-noopener-noreferrer-still-vulnerable/50709760) - but I'll throw it in there cause it can't hurt right?


Scrolling messes up embedded terminal layout

When scrolling in the embedded Terminal (Containers view) the content gets messed up (e.g. line breaks appear).

Yes, i could reproduce it. Its also the same issue on https://linuxcontainers.org/lxd/try-it
Maybe i have to update on xterm.js (fork of term.js)


Fix helper menu

Make helper menu image more accurate.
Estimated time taken: 45 s

Fixed by b9a9858.


[Bug] Firebase not work with GoogleSignIn SDK

[REQUIRED] Please fill in the following fields:

Unity editor version: 2020.3.31f1
Firebase Unity SDK version: 8.8.0
Source you installed the SDK: .unitypackage (.unitypackage or Unity Package Manager)
Problematic Firebase Component: Database (Auth, Database, etc.)
Other Firebase Components in use: Auth (Auth, Database, etc.)
Additional SDKs you are using: GoogleSignIn-Unity (Facebook, AdMob, etc.)
Platform you are using the Unity editor on: Windows (Mac, Windows, or Linux)
Platform you are targeting: Android (iOS, Android, and/or desktop)
Scripting Runtime: IL2CPP (Mono, and/or IL2CPP)

[REQUIRED] Please describe the issue here:
(Please list the full steps to reproduce the issue. Include device logs, Unity logs, and stack traces if available.)
Steps to reproduce:
Have you been able to reproduce this issue with just the Firebase Unity quickstarts (this GitHub project)?
no
What's the issue repro rate? (eg 100%, 1/5 etc)
100%
What happened? How can we make the problem occur?
I have implemented GoogleSignIn-Unity for authentication via google as a provider on Firebase.
I removed the Unity.Compat.dll and Unity.Task.dll files inside the Parse / Plugins root because the GoogleSignIn SDK for Unity was giving me the incompatibility error with the Tasks. The two dlls were already disabled but this was obviously not enough, I had to remove them. And here we have two strange problems:

Since I removed the two dlls in the root (obviously leaving those inside the dontnet45 folder enabled) Firebase uses the mscorlib tasks instead of the Firebase ones (and in fact all the Tasks magically become awaitable), and I have absolutely no idea of why it does so but above all how to make it re-point to the Firebase Tasks. The only way seems to put the two Net 3.5 dlls back in the root (disabled) but of course I can't do that as GoogleSignIn breaks ...

In the built Android (.apk) version any call I make to FirebaseDatabase be it a GetValueAsync (), a SetValueAsync () or any other operation block at that point! (This problem for example does not occur with FirebaseAuth). Even doing a development build i saw that it doesn't even enter the ContinueWithOnMainThread. In the Editor, however, everything works.

It is useless to say that in this way it is not possible to work, remove two dlls because otherwise nothing works, but removing them generates other problems. So what's the solution?
If you have a downloadable sample project that reproduces the bug you're reporting, you will
likely receive a faster response on your issue.
Example project with Google SignIn + FirebaseAuth + FirebaseDatabase:
https://we.tl/t-yiRfKZ12pT
The project has only the dotnet45 dlls inside the Parse / Plugins folder, but by opening the TestFirebaseTask script you can see that the methods of both GoogleSignIn and FirebaseApp are pointing to mscorlib tasks instead of Unity.Task and this causes all the problems mentioned above
See this attachment and you can see in visual studio popup Task referenced to mscorlib instead Unity.Task
https://ibb.co/kmRYtTD
Relevant Code:
// TODO(you): code here to reproduce the problem

Hi @gabry90,
Thanks for the information you provided. In order to identify what's causing this behavior, could you provide the complete steps needed to replicate the issue? I've downloaded the sample project you provided, but I couldn't seem to find a scene for the build.
Additionally, could you provide the error logs resulted from the incompatibility with the tasks as you've mentioned?

There is no scene for the build because it is not needed. Just open the TestFirebaseTaskClass script to see that the GoogleSingIn and Firebase tasks are pointing to those of mscorlib and not to those of Unity.Task.
Re-adding the two .Net 3.5 dlls, disabled, in the root of Parse\Plugins the console does not generate errors, and this is strange, but inside GoogleSignIn Visual Studio highlights the conflict error. In fact, intellisense cannot identify the correct return type if you try to ignore the error.
Here is the link to the error screen: https://ibb.co/Tc9q2Yx
As for the second problem I can no longer find it, I honestly do not know why the calls to Firebase Database were pending without entering the callbacks. It would be advisable, whatever the circumstances, to always call the callbacks avoiding silent errors and leave the execution of the code pending without the possibility of resuming it.

Hi @gabry90,
I looked at the error message you provided which apparently is in the GoogleSignIn.cs file, and it doesn't seem to occur on my end, regardless if the .dll files were disabled or not.
Also, does the TestFirebaseTaskClass.cs file look like this in your implementation?

Please provide more context on how the tasks are "pointing to those of mscorlib".
Additionally, version 8.9.0 of the SDK has been released recently. Could you try it out and see if it makes a difference?

tried to update firebase SDK to version 8.9.0 but the problem is the same. I attach 2 screenshots:

https://ibb.co/pPyPj9D: Unity.Task.dll and Unity.Compat.dll (.Net 3.5) present but disabled.

https://ibb.co/pPyPj9D: Unity.Task.dll and Unity.Compat.dll (.Net 3.5) removed. Error gone but the ContinueWithOnMainThread method extension points to mscorlib instead of Unity.Task. Without this extension the Firebase Tasks will correctly point to Unity.Task.

That is very odd that you are seeing different behavior when it is present but disabled, vs removed.  It should be the same, where Unity is just not aware of that file.  Is it possible that it is still enabled, I know the UI for enabling/disabling is a bit complicated, with the one checkbox changing what all the other checkboxes do.
The two dlls under the dotnet45 folder are there to pass along the definitions to the underlying System.Threading.Tasks.Task, which was not present in older versions of Unity, but is now present.
When you added the Firebase SDKs, did you add the ones from the dotnet4 folder?  As those should be the ones with the correct settings.

That is very odd that you are seeing different behavior when it is present but disabled, vs removed. It should be the same, where Unity is just not aware of that file. Is it possible that it is still enabled, I know the UI for enabling/disabling is a bit complicated, with the one checkbox changing what all the other checkboxes do.
The two dlls under the dotnet45 folder are there to pass along the definitions to the underlying System.Threading.Tasks.Task, which was not present in older versions of Unity, but is now present.
When you added the Firebase SDKs, did you add the ones from the dotnet4 folder? As those should be the ones with the correct settings.

I know how dll imports work, I have also worked with asmdef in much more complicated contexts, finding myself also managing any conflicts. I have already tried different solutions and combinations and the error looks like that. As long as the Net 3.5 dlls exist (disabled) everything works, when I go to remove them, however, it seems that the Net 4.5 dlls alone are not enough. I also came to think that it is a Visaul Studio intellisense problem or something related more to the development FE than to Unity ... at the point I don't know what to think anymore ...

Hi @gabry90,
I'm sorry to say that we cannot reproduce this here and the issue seems to be in the behavior of the editor, which doesn't seem to be correctly disabling or enabling the libraries.  If there's something that you can provide that we can easily reproduce then we welcome it and we can investigate further.  Otherwise I recommend reaching out to Unity support to determine why the editor doesn't properly honor the disable / enable operations.


[Duplicate of #4672]: How machine learning is revolutionizing the agricultural industry

Article checklist
Attention
Our Peer Reviewer and Content Moderator teams do NOT provide any revisions services. All revisions and edits should be completed on your own forked repo (as to not take up room in the queue) in order for our team to review them all in a timely manner.
ANY ARTICLE SUBMITTED WITH GLARING ERRORS WILL BE IMMEDIATELY CLOSED.

As a rule of thumb - please be sure to only submit articles (pull requests) that are fully polished and ready to be published. Be sure to go through our resources documents for extra and 3rd party (vetted) resources to help improve overall technical writing.

Pre-submission checks

Please ensure that you have only one open topic suggestion (issue) + in-review article (linked pull request) at a time. This will ensure that we complete the review process in a timely manner from inception to publishing

If you have not already, please go over our Resources Page for more tips and tricks on how to improve your overall technical writing so reviews are swifter, increase the chance of a payout and provide more value to our readers.

For first-time contributors and for more details on our submission guidelines, see our Contributing Guidelines.

Formatting and structure checks

[x] Have you used the correct folder and file structure? - see Contribution Guidelines

[x] Is your article properly formatted in Markdown? - see Github Markdown guide

[x] Have you used our preferred heading structure? - H3 (###) and up

[x] Does your article follow any of the suggested structure formats? - see example formats

[x] Is your article over 750 words? - Tool: Wordcounter.net

[x] Is the article you are submitting an in-depth and unique article? Does it go beyond what is in the official docs and what is covered in other blog sites. See these articles as examples.

[x] Does your article provide enough value and detail about your topic? - Articles should be clear, accurate, and fully explained.

[x] Can your article be understood by beginners? - Assume the audience is smart but has no prior exposure to the common terminology in your article.

[x] Have you included a hero image that is 600x400 pixels, under 300KB in size, and have the copyright to use it? - Tools: Pixlr Image Editor for resizing and Unsplash for Creative Commons images

Grammar & plagiarism checks

[x] Have you spellchecked and grammar checked your article? - Tools: Write&Improve - ProWritingAid - Grammarly
[x] Please place your article through a 3rd party plagiarism checker. We suggest using Quetext, this tool is free and has a daily limit. We typically accept articles with 10% or less.
[x] Have you checked your article for readability? - Tool: Hemmingway
[x] Have you added sources for quotes and images that aren't yours?

Technical checks

[x] Are your code snippets properly formatted for syntax highlighting - see Syntax guide
[x] Have you checked your code runs correctly and you've highlighted all necessary dependencies for installation?
[x] Are the software programs and packages you're highlighting in your article up to date, using current versions and not deprecated?

Topic suggestion this closes

Remove the backticks and add the issue number below to link and close your Topic Suggestion (issue) when your article has been published (PR has been merged). See this video for more details.

This closes #4374

Finally, delete the article checklist notes in blockquotes and submit your PR. We look forward to reviewing your article.

@srishilesh. Sorry I took too long to revise the document. I lost my laptop and I had to use my friend's to upload. I did make a mistake and committed using her profile, do I need to recommit again?

@srishilesh. Sorry I took too long to revise the document. I lost my laptop and I had to use my friend's to upload. I did make a mistake and committed using her profile, do I need to recommit again?

@Ann-tech348
Not a problem.
Will start my review ASAP

Duplicate of #4672

Alright, thank you

Formatting looks good. Proceeding with tech review.

I'll complete my tech review by tomorrow

Alright, thank you

Thank you so much

@Ann-tech348
Good afternoon  and thank you for submitting your topic to the EngEd program.
Made some edits to the sections I was able to review. Some sentences were a bit difficult to understand. & after some careful consideration it struck us that this topic may be a bit over saturated throughout other blog sites.
We typically refrain from publishing content that is covered widely on the net or other blogs. We're more interested in original, practitioner-focused content that takes a deeper dive into programming-centric concepts.
But in order to approve the topic it has to serve value to the larger developer community at large. Please feel free to suggest an alternate topic to explore. 🚀
cc @srishilesh

Well noted. I am sad to be honest. I wish the issue of topic being over saturated came earlier before commencing the writing. I will keep suggesting a more  original, practitioner-focused content


Added contributer Deven Phillips

Fixed.

I'm sorry, but I need to validate all requests before I can merge them. Would you mind giving me a link to a commit that was successfully merged into one of the official Vert.x repositories? Thank you so much!
By the way, I just changed the community page. The contributors are now in a JSON array and the HTML code is automatically generated in the template. This way we can make sure all entries look the same. I can try to update your pull request, but if you like you can also do it yourself.
Thanks,
Michel

Here's one link: https://github.com/vert-x3/vertx-examples/pull/3

Some Maven archetypes I created for the project.
https://github.com/InfoSec812/vertx-js-archetype
https://github.com/JUGGL/vertx-java-archetype

Finally, conference workshop I did on Vert.x
https://github.com/JUGGL/codepalousa-vertx-chat

Thanks a lot and sorry again for being to picky :-)

Deven,
What do think about your entry on the community page http://vert-x3.github.io/community/ ? I think it looks a bit off due to the long list of contributions, but that's just my personal opinion. Just let me know what you think.
Cheers,
Michel


Add cryostat to enable JFR recordings for Keycloak.

It proved possible to add Cryostat to do the JFR recordings. Login is disabled like with all other tools here to enable fast turnaround times for developers working on this.
For an OpenShift setup where the routes would be publicly accessible this would then be changed.
I did a test for Keycloak and downloaded the JFR report. The first flamegraph showed truncated stack dumps (default would be 64?), therefore I configured the stackdepth to 512.
The option to create JFR recordings is enabled by default, but no JFR recording is created until this is triggered.
The profiling created is regular profiling, not async profiling. The profiling will therefore suffer from the safepoint bias problem. See async profiler for details:

This project is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on the HotSpot JVM.

For now, not using async profiling should be good enough until proven otherwise.
Here a screenshot of the UI.

Closes #98

@hmlnarik - this PR shows a solution how JFR can be part of this setup. While Cryostat provides a point-and-click Web interace, it also has a REST API to interact with it: https://github.com/cryostatio/cryostat/blob/main/HTTP_API.md
While it supports regular profiling and not async profiling, I will assume until proven otherwise that it serves the need for JFR recordings.

@kami619 - I'd be happy if you could give it a try and test the following:

spin up Minikube with these changes (the upgrade.sh script should work here)
create a JFR recordings and download it
open such a recording for example in Eclipse Mission Control https://adoptium.net/jmc

Thanks!

Cryostat can initiate JFR recording via JMX and downloads the created file?

@hmlnarik - yes.

Hi @kami619 - I fixed the problem with a missing placeholder for the hostname. I also added an increased timeout when retrieving the cadvisor metrics. Please give it another try. Thanks!

this works as expected, thanks for this great utility addition 🥇 @ahus1


How to support OCB ?

OCB needs a different image.
Idea: replace supported_repos and build_images configs by an ordered list of regexes on repos and target branch to select the image:
[
    {
        "repo": "^oca/ocb$",
        "branch": "^15.0$",
        "builds": [
            {
                "image": "ghcr.io/oca/oca-ci/py3.8-ocb15.0:latest"
            }
        ]
    },
    {
        "repo": "^oca/.*",
        "branch": "^15.0$",
        "builds": [
            {
                "label": "odoo",
                "image": "ghcr.io/oca/oca-ci/py3.8-odoo15.0:latest"
            },
            {
                "label": "ocb",
                "image": "ghcr.io/oca/oca-ci/py3.8-ocb15.0:latest"
            }
        ]
    },
    {
        "repo": "^oca/ocb$",
        "branch": "^14.0$",
        "builds": [
            {
                "image": "ghcr.io/oca/oca-ci/py3.6-ocb14.0:latest"
            }
        ]
    },
    {
        "repo": "^oca/.*",
        "branch": "^14.0$",
        "builds": [
            {
                "image": "ghcr.io/oca/oca-ci/py3.6-odoo14.0:latest"
            }
        ]
    }
]

The first match wins. And we can imagine starting more than one build for each.

The first part above is done. Next step is to add, env, secret_env and template_vars in BuildSettings next to image, and in k8s make_deployment_vars() merge these with build_env, build_secret_env and build_template_vars so they are passed to the kubefiles rendering and the kustomization.yaml.jinja template.
Then we can have an environment variable that default to oca_install_addons that can be overridden to pip install -e . for the OCA/OCB repo.


⚠ Bofrost.at has degraded performance

In 8823ba4, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1077 ms

Resolved: Bofrost.at performance has improved in 8cc86c5.


popupWidth cannot set less than 267.

popupWidth cannot set less than 267. Nothing happens when set popupWidth < 267, but wider when set popupWidth > 267.

@Nexirus fix


launchy: Update to version 2.10.79

Looks like the releases on github are more up to date, so let's switch over to them.
One issue I found while using it: (this is independent of the version update)
Looks like the persistence of launchy.ini is not working properly. I suspect the app deletes and recreates the file on every settings change leading to a decoupled file in the persistent folder and a regular, non-linked file in the app folder. Any suggestions about that? Should I file it in a separate issue?

@Ash258 any idea how to solve the persistence problem? Have you seen this behavior with other apps?

https://github.com/lukesampson/scoop-extras/search?q=uninstaller+Copy-Item&unscoped_q=uninstaller+Copy-Item

Thanks for the hint, works like a charm!
From my perspective everything is fine now and we are good to merge.


Black window where player should be

Sometime in the past 24 hours a presumed YouTube update broke YouTubeCenter completely.  Type of player selected or other options don't seem to matter.
Console shows:
TypeError: window.matchMedia is not a function[Learn More]  base.js:7279:111
	<anonymous> https://www.youtube.com/yts/jsbin/player-vflRhCLRy/en_GB/base.js:7279:111
	<anonymous> https://www.youtube.com/yts/jsbin/player-vflRhCLRy/en_GB/base.js:1:20
TypeError: content is null[Learn More]  watch:23059:11
	resize YouTubeCenter.js:23059:11
	ytcenter.player._resize YouTubeCenter.js:23280:16
	ytcenter.player._updateResize YouTubeCenter.js:22918:9
	mutationCallback YouTubeCenter.js:2485:11
TypeError: window.matchMedia is not a function[Learn More]  base.js:7279:111
	<anonymous> https://www.youtube.com/yts/jsbin/player-vflRhCLRy/en_GB/base.js:7279:111
	<anonymous> https://www.youtube.com/yts/jsbin/player-vflRhCLRy/en_GB/base.js:1:20
TypeError: uw.yt.player is undefined[Learn More]  watch:10870:11
	load YouTubeCenter.js:10870:11
	onload

I realise that maintenance on this was stopped so I don't expect we'll get an official fix for this, but perhaps a workaround can be found.

Someone made a PR which is now compiled. Go to https://github.com/YePpHa/YouTubeCenter/tree/master/dist and install the new version.

The one for Chrome is version 548 still :-)

@ShamblerDK I seem to have forgotten to use the certificate.

Ah, didn't realise the Issue page filtered out closed ones. This is a duplicate of 2481  which ix now fixed in build 549.

The Maxthon version appears to be build 548, too.

@Green45 as far as I can see https://github.com/YePpHa/YouTubeCenter/raw/master/dist/YouTubeCenter.mxaddon is build 549.

Claims to be Build 548 when I check the "about" info on YouTube. Additionally, it still has the "black screen only"-bug, which I assume was meant to be fixed in 549 for all browsers?

@Green45 perhaps there's a caching issue in Maxthon. Have you tried to uninstall the extension. Close the browser. Open the browser and install the new version?

Turn it off and turn it back on again, in other words.
And you know what? It worked. It ALWAYS works.

@Green45 glad to be of some help :)


Out of context glFinish

I was running apitrace to track down a different issue when I noticed that glFinish() is called with no gl context set.
It's most likely this one: https://github.com/swaywm/wlroots/blob/e77c046cf98f53a9307deeb8940d67938c078986/backend/drm/drm.c#L962

That one probably lost the gl context because of af2f69e6c1578f848d3e3222804444ff9cff83df

We should properly make the mgpu surface current before glFinish, and unset it after.

Removed in https://github.com/swaywm/wlroots/commit/60001a75a276f931b02d78a95cedfd0098b88617


[RC-1.21] Unable to search by Author or Subject in My Resources page

@sblack-usu, @pkdash
I was able to narrow down this error to a problem with a context variable not reaching the frontend:
https://github.com/hydroshare/hydroshare/blob/develop/hs_core/templates/pages/my-resources.html#L269
However, this only happens in our current deployment on beta. I tried to reproduce this error locally with no success. We might need to investigate directly in the VM in order to find out what's going on.
Testing locally:

Testing beta deployment:

Since this data doesn't make it to the template in beta, the DataTables' scripts can't use the information to filter correctly. The same happens with the Subjects column.

@pkdash, I was able to reproduce this issue locally today. Could you pull the most recent changes and try again? We might be able to debug and find the problem now.

@Maurier I am using the RC-1.21 now locally. Not able to reproduce this issue locally.

This is still a problem on the new 1.21 beta RC just deployed.  If I create a resource with a unique keyword "kw123" on www in MyResources if I search on the MyResources search bar for kw123, I find the resource, but on beta I do not.
Also, if I add another author and on www search on that auther in the search bar, I find the resource, but do not on beta.

@dtarb and @Maurier I have now created a PR #3282 as a hotfix for this issue.

the pr has been merged into the RC and deployed to beta., @Maurier @dtarb @pkdash, We are planning to release 1.21 on Tuesday

Filtering on authors and keywords seem to work now on beta.

Seems the page loading did not suffer much due to not prefetching the metadata.

@Maurier I think this can be closed.


Use a log-scale axis in contour plots if a parameter type is LogUniformDistribution.

This PR changes the axes of a contour plot if the types of parameters are LogUniformDistribution.
An example:

x: UniformDistribution
y: LogUniformDistribution
z: LogUniformDistribution

Plots generated by the current master branch:

Plots generated by this branch:

Code:
import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_loguniform('y', 1e-7, 1e-2)
    z = trial.suggest_loguniform('z', 1e-5, 1)
    return x**2 + 10**y + 10**z

study = optuna.create_study(direction='maximize', sampler=optuna.samplers.RandomSampler())
study.optimize(objective, n_trials=100, gc_after_trial=False)

optuna.visualization.plot_contour(study)

The change itself looks nice, but the dots are a bit too big for the figures attached to the description.
But this might be up to the size of the display and should be accessed in another PR if needed.


retryFor isn't flexible enough to be convenient

Let's consider the following snippet:
    def dispatch(): ZIO[Throwable, String] = {
      val id = "xxx"
      val timeout: FiniteDuration = 2.seconds

      val onTimeout = new RuntimeException() // TODO: ZIO interface isn't nice
      val responses = new ConcurrentHashMap[String, String]

      IO.sync {
        id
      }
        .flatMap {
          id =>
            Option(responses.get(id)) match {
              case None =>
                IO.fail(onTimeout)
              case Some(value) =>
                IO.point(Some(value))
            }
        }
        .retryFor[Option[String]](None)(id => id)(timeout)
        .flatMap {
          case Some(value) =>
            logger.debug(s"Have response: $value")
            IO.point(value)

          case None =>
            IO.terminate(new TimeoutException(s"No response for $id in $timeout"))
        }
    }

So, here I'm doing a simple thing - I'm attempting to read from an external mutable state until I get a value or timeout occurs. Though I had to do that dirty trick with onTimeout because .retryFor does not allow me to modify "left" branch.
I guess we should implement another kind of retryFor which would allow us to consider last failure value returned from our action. Though it may be kinda tricky to implement because atm retryFor just races timeout actin with the primary action, so there is no way to get the latest value.

We are reworking the retry combinators on #141 and when it's done you can complain about anything except about lack of flexibility :laughing:

Would it be possible to consider last action error while handling timeout and change left branch type? Would be perfect if so.

@pshirshov Good spot! Looking at your motivating use case, I was able to generalize the retry method on IO under the new composable retry policy framework.
Now you get something like this (simplified):
trait IO[E, A] {
  def retryOrElse[S, E2](policy: Retry[E, S], orElse: (E, S) => IO[E, A]): IO[E2, A] = ???
}

You also a more powerful variant (retryOrElse0), but I don't think you need that power here.
These methods allow you to look at the last error, and decide to change the error type.
Is this sufficient?

Yeah, that should be enough. Thank you!

So, now it looks like
  def poll(id: RpcPacketId, interval: FiniteDuration, timeout: FiniteDuration): IO[Nothing, Option[RawResponse]] = {
    IO.sleep(interval)
      .flatMap {
        _ =>
          checkResponse(id) match {
            case None =>
              IO.fail(None)
            case Some(value) =>
              IO.point(Some(value))
          }
      }
      .retryOrElse(Retry.duration(timeout), {
        (_: Any, _: Any) =>
          IO.point(None)
      })
  }

Which is lot better.

This is the best I could do:
(IO.sleep(interval) *>
  IO.sync(checkResponse(id)).flatMap { case None => IO.fail(()); case x => IO.now(x) })
  .retryOrElse(Schedule.duration(timeout), (_, _) => IO.point(None))

The main problem is that it's just awkward to convert from Option[A] => IO[Unit, A]. Probably we could add, IO.fromOption to do that. We already have fromEither.
Then it could be:
(IO.sleep(interval) *> IO.fromOption(IO.sync(checkResponse(id))))
  .retryOrElse(Schedule.duration(timeout), (_, _) => IO.point(None))

which is pretty elegant.


[DUOS-1740][risk=no] Doc updates for running https and cors-enabled

Addresses
https://broadworkbench.atlassian.net/browse/DUOS-1740

Have you read Terra's Contributing Guide lately? If not, do that first.

Label PR with a Jira ticket number and include a link to the ticket
Label PR with a security risk modifier [no, low, medium, high]
PR describes scope of changes
Get a minimum of one thumbs worth of review, preferably two if enough team members are available
Get PO sign-off for all non-trivial UI or workflow changes
Verify all tests go green
Test this change deployed correctly and works on dev environment after deployment

Closing in favor of https://github.com/DataBiosphere/duos-ui/pull/1735


Fix #1078: Citation not styled properly

Turns out our demo docs werent being rendered by sphinx as citations so this may have slipped though the sphinx 2 fixes.
I updated the demo docs and included the need style fixes.

The citation display looks incorrect in this PR

This should match the surrounding styles, and should have matched the html4 output

That should be how it looks before this PR. with this or I get:

It looks like you didn't build the assets on this PR, so the PR build output looks broken. Can you update this PR with the built assets?

Sure, but we should figure out what cased the assets to no longer be built.

So, these assets always had to be manually built when opening/changing a PR, which is actually one of the reasons we don't want to check these assets in to the repository. It makes doing PR review tricky, and contributors need to know to build assets on their PRs after each change.
By removing the assets from the repository, and only building them when needed -- on RTD/CI and at release time -- we can keep these files out of PR and review entirely.

It would seem that in order to get this PR in, we need to merge latest master and then rebuild assets. I'll be getting a dev env up and hopefully I can push that easily after that :crossed_fingers:

Verified that I got the intended output that @Blendify mentioned in https://github.com/readthedocs/sphinx_rtd_theme/pull/1080#issuecomment-801495939
If tests build, I'm going to :ship:

(sorry needed another merge for conflicts on theme.css to go away)
@agjohnson should we switch on "Squash and Merge" as a default?

@benjaoming indeed :+1: Squash and merge should be our default on repositories now

Going to track this in #1307
Btw we now have Squash and Merge as the default :+1:


Getting exception while scep enrollment from device

Hi,
I am trying to connect to scep server to get certificate but getting following exception
org.jscep.client.ClientException: org.jscep.transport.response.InvalidContentTypeException: Expected [application/x-x509-ca-cert, application/x-x509-ca-ra-cert], but was text/html;charset=ISO-8859-1
Can you tell me the reason for this
Thanks & Regards,
Ashiq Sayyad

Your SCEP server appears to be serving HTML content.  I would consider visiting the endpoint using a web browser first.  This isn't a jscep issue, so I'm closing it.

I believe your SCEP server is serving up HTML.  This isn't a jscep bug.
On 25 Oct 2016 7:15 am, "AshiqKony"<EMAIL_ADDRESS>wrote:
Hi,
I am trying to connect to scep server to get certificate but getting
following exception
org.jscep.client.ClientException:
org.jscep.transport.response.InvalidContentTypeException:
Expected [application/x-x509-ca-cert, application/x-x509-ca-ra-cert], but
was text/html;charset=ISO-8859-1
Can you tell me the reason for this
Thanks & Regards,
Ashiq Sayyad
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jscep/jscep/issues/68, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnaO_3ph87-yM458FsCdob7m33UpiChks5q3Z5mgaJpZM4KfmYj
.


Version 0.0.15 ClusterRoles and ClusterRoleBindings for argocd-application-controller and argocd-server not showing as connected with the ArgoCD

Describe the bug
We install ArgoCD Operator and create ArgoCD Server in a namespace on OpenShift 4.7 by defining necessary manifest files in GIT and let a central ArgoCD-server in same OpenShift cluster sync these manifests. Like this: Central ArgoCD v0.0.14 --> Namespace ArgoCD v0.0.15.
What I observed in GUI in the central ArgoCD Server after upgradring the namespace ArgoCD Operator from 0.0.14 to 0.0.15 was that ClusterRoles and ClusterRoleBindings regarding argocd-application-controller and argocd-server showed as disconnected from the namesace ArgoCD server/object. See the screenshot below for the resources marked as yellow and ready for pruning.
If I copy the ownerReferences from one of the other resources, i.e rolebinding, and add this to one of the clusterroles or clusterrolebindings, they get connected to the central ArgoCD. Now showing as synced, but not connected to the namespace ArgoCD.
ownerReferences:

apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: ArgoCD
name: argocd
uid: d412eb32-e506-4976-bccf-ce578f0d9812

To Reproduce
Steps to reproduce the behavior:
See description above
Expected behavior
The clusterroles and clusterrolebindings for argocd-application-controller and argocd-server to be connected to the namespace ArgoCD in the GUI of the central ArgoCD Server.
Screenshots

Added yaml manifestes for namespace argocd and rolebinding.
namespace_argocd.yaml.txt
namespace_crb_argocd-server.yaml.txt


Unable to implement classic COM type using winrt::implements

In the version of C++/WinRT that is part of the Windows SDK, the following code in which I am using C++/WinRT to provide me an implementation of IUnknown compiles correctly:
class AppVisibilitySink : public winrt::implements<AppVisibilitySink, IAppVisibilityEvents> {
private:
	std::function<void(bool)> m_startOpenedCallback;

public:
	inline AppVisibilitySink(std::function<void(bool)> startOpenedCallback) :
		m_startOpenedCallback(std::move(startOpenedCallback))
	{ }

	IFACEMETHODIMP LauncherVisibilityChange(BOOL currentVisibleState) override
	{
		try
		{
			m_startOpenedCallback(currentVisibleState);
			return S_OK;
		}
		catch (...)
		{
			return winrt::to_hresult();
		}
	}

	IFACEMETHODIMP AppVisibilityOnMonitorChanged(HMONITOR, MONITOR_APP_VISIBILITY, MONITOR_APP_VISIBILITY) override
	{
		return S_OK;
	}
};

I am now trying to write some pages in XAML to host in XAML Islands, so I was automatically upgraded to the NuGet version of C++/WinRT by Visual Studio. That code (among other) doesn't compiles anymore, because of the override specification on the functions. The compiler is saying that I did not override any base class method. After being clueless and removing the override to see if I can get more info, I get a (not really) better insight into the error:

The trick is that you need to include Unknwn.h before any C++/WinRT headers to enable COM interop support. For example:
#include <Unknwn.h>
#include "winrt/Windows.Foundation.h"

Since C++/WinRT has no dependency on the Windows SDK, it has no way of knowing that you would like to implement COM interfaces that rely on the definition of IUnknown as defined by the Windows SDK.

Thanks, issue resolved. Header inclusion order in another file that included this one is what had broke it.


NeoTree & Telescope open in wrong CWD

Did you check docs and existing issues?

[X] I have read all the LazyVim docs
[X] I have searched the existing issues of LazyVim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
NVIM v0.10.0-dev-2982+gea1c9f60e
Operating system/version
MacoS
Describe the bug
Hello, if i open a spesific directory it falls back always to the root kinda. So i opened in my terminal nvim joomla. The first time the file browser shows the right directory, then i press leader e or E to close and reopen, then it shows me everything above too  and in telescope i got not the find files picker but instead the git files. I don't know if I'm doing something wrong or is this the normal behavoir??

Steps To Reproduce
Open a spesific directory with nvim folder,
press leader e twice
press leader ff
Expected Behavior
Open neo-tree only in that folder as cwd
set that folder as cwd for telescope
Repro
-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  "folke/LazyVim",
  -- add any other plugins here
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

You should use <leader>E and <leader>fF which are the corresponding keymaps for opening in cwd instead of root_dir.

I just tried it but it still behaves the same way

Yes in options.lua file.

Ok i tried it out, but it still behaves the same...

I don't understand what you're saying. It behaves correctly for me. Can you explain what exactly you're doing?

I have (had) this problem and added @dpetka2001 's fix which has worked for me. Thanks

Like in the video. If i open on the first time neovim, it shows me the right folder if i close and reopen it shows me all the folders...
https://github.com/LazyVim/LazyVim/assets/92093874/b2af30f7-ebc8-49ba-b193-fbb1b486dccf

Just make a new keymap for neo-tree that executes Neotree toggle.

Ok this worked if i do Neotree toggle. And what about telescope, how do I get it to only search the folder and subfolders I'm in?

The current working directory cwd is the one that you invoke Neovim. Not the file/directory argument that you pass to Neovim. So, if you want to search from a specific directory downwards, you should invoke Neovim in the specific directory that you want.

Do the following test if you'd like. nvim --clean joomla/ and then inside Neovim (which will open as vanilla Neovim without plugins and configuration) do :pwd. You'll see that it the directory from where you invoked Neovim. This is default Neovim behavior.


How should the fix for the dependencies be handled?

I am installing the packages one by one in al_ds and found that the following packages will throw errors when run after installation.
# oauth2client==4.1.3 #bug
# dql==0.5.26 #bug
# snowflake-connector-python==3.0.2 #bug
# firebolt-sdk  #bug

I have not tested the following disabled dependency packages.
# pymapd==0.19.0
# ibm-db>=2.0.9

@justinclift
For these dependency packages, they should be tested one by one. If there are issues, they can either be fixed or disabled, but there needs to be progress in resolving them. I suggest that for those data sources that are no longer actively updated, if problems occur, they can be directly removed from support.

Yeah, with the exception of the oauth2client package, I reckon disabling those other ones for now is probably the right move.
The oauth2client one will probably turn out to be important though, as OAuth is commonly used in web based authentication.  So there could be a bunch of people that need it.
That being said, I agree with you. Lets just remove all of these for now and see how the resulting preview image goes.
It seems like it should go ok, and we'll have a working preview image that people can use.  In theory (!). :smile:

I just found that the Python backend cannot run properly after installing these packages locally. Since I am not very familiar with these dependencies, this is only provided as a reference for everyone.

No worries.  Letting people know is a good idea. :smile:


exporter show no smartmeter_ metrics only the python default ones

I tried running this exporter to collect data from an EMH mMe 4.0.
I can confirm that the smart meter is sending out sml frames which I can see using the example sml_server from the libsml project:
> ./sml_server /dev/ttyUSB0
1-0:96.50.1*1#EMH#
1-0:96.1.0*255#XX XX XX XX XX #
1-0:1.8.0*255#616.4#Wh
1-0:16.7.0*255#-7#W
1-0:96.50.1*1#EMH#
1-0:96.1.0*255#XX XX XX XX XX #
1-0:1.8.0*255#616.4#Wh
1-0:16.7.0*255#-33#W
1-0:96.50.1*1#EMH#
1-0:96.1.0*255#XX XX XX XX XX #
1-0:1.8.0*255#616.4#Wh
1-0:16.7.0*255#-38#W
...

Unfortunately the exporter is not showing any smart meter metrics but a single exception when its starting up:
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<SmlProtocol._reconnect() done, defined at /usr/local/lib/python3.9/site-packages/sml/asyncio.py:107> exception=AttributeError("'NoneType' object has no attribute 'call_soon'")>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sml/asyncio.py", line 109, in _reconnect
    await self._disconnect()
  File "/usr/local/lib/python3.9/site-packages/sml/asyncio.py", line 140, in _disconnect
    self._transport.abort()
  File "/usr/local/lib/python3.9/site-packages/serial_asyncio/__init__.py", line 192, in abort
    self._abort(None)
  File "/usr/local/lib/python3.9/site-packages/serial_asyncio/__init__.py", line 396, in _abort
    self._loop.call_soon(self._call_connection_lost, exc)
AttributeError: 'NoneType' object has no attribute 'call_soon'

I was building the python package myself because I had to remove the vendor and device_id thing because my smart meter is not sending those apparently.
I'm not a python export so I'm pretty clueless about what is going wrong here. Help would be appreciated.
UPDATE: Looking at the output from sml_server it seems like my smart meter is sending out vendor and device_id. But with the unmodified 0.1.4 version I get zero output, no exception but also no metrics. Looking at the code I figured it was still waiting for the vendor or device_id but maybe I was wrong there.
UPDATE 2: Looking at it again the obis number for vendor on device_id seem to be different for my smart meter. So I think I do have the problem with the upstream version that the exporter waits for those indefinitely.

I have only ever tested this against one model of smart meters, also an EMH, so I think it's fair to say that I may have introduced assumptions, that don't work for your smart meter.
Does yours actually spew out a binary format that you're decoding using sml_server? Can you try an example of pysml, the SML library used in this project?
https://github.com/mtdcr/pysml/tree/master/examples

With #10 there is an example where you need to add the proper OBIS IDs to identify the device. Only when vendor/model have been detected will SML frames be accepted.
The PR also adds more log messages that should help debug this scenario, if you are still interested.


updated deps & readme, cleaned up scripts

Updated dependencies
Cleaned up NPM scripts, removed redundant & deprecated ones
Fixed e2e:live
Updated README with info on DLLs, external stylesheets, and updated scripts
Travis script also tests dev/dll build
Added delay in Travis script to reduce likelihood of failure

Resolves #1353

Protractor ^5.0.0 dropped support Node 5. If we want to update Protractor further we will need to drop support for Node 5. In my opinion we should support only Node lastest and Node LTS.

Why did we missed to add the language-service?

@katallaxie you haven't submitted a pull request for i18n yet!
https://angular.io/docs/ts/latest/cookbook/i18n.html


DTSPB-3177: after replacements

Is line 52 required to update to afterSuccess as well?

I have changed that one too now


Improve loading times

HuskTowns data loading takes forever on servers with a large playerbase and lots of player and town data.

This was the big goal:tm: of the rewrite. I need to run some more serious benchmarks, but yeah, the new approach specifically targeted at claims should make their loading near instantaneous. It might take a few seconds on massive servers, but they shouldn't take 15 minutes to load a few hundred claims anymore.


Different result from available results

I am reproducing the MASH pipeline, and I found the result of mashr is different from the available results (produced by Hao),
the code in this step is

library(mashr)
dat = readRDS('/mnt/vast/hpc/csg/rf2872/Work/MASH_csg.q/output/Ast_Exc_Inh_Mic_OPC_Oli.rds')
set.seed(1)
random.subset = sample(1:nrow(dat$random.b), min(6000, nrow(dat$random.b)))
random.subset = mash_set_data(dat$random.b[random.subset,], dat$random.s[random.subset,], alpha=1, zero_Bhat_Shat_reset = 1E3)
vhatprior = mash_estimate_corr_em(random.subset, readRDS('/mnt/vast/hpc/csg/rf2872/Work/MASH_csg.q/MASH_6_celltypes2/Ast_Exc_Inh_Mic_OPC_Oli.EZ.prior.rds'), max_iter = 6)
vhat = vhatprior$V
saveRDS(vhat, '/mnt/vast/hpc/csg/rf2872/Work/MASH_csg.q/MASH_6_celltypes/Ast_Exc_Inh_Mic_OPC_Oli.EZ.V_mle.rds')

Here is the result of my job,

and below is the result of Hao's job

The random subset may cause a tiny difference from others. But there are some of the values reversed. Is that normal?

@rfeng2023 The result should be identical if you set seed ... did you use the same container? It could make a diffeerence if major R version changes also changed behavior of seed such that set.seed(1) is not the same between different versions R or related libraries?
To solve the problem: since you are worried of the small differences particularly the sign differences, my suggestion is to rerun but using a much larger random.b sample and see how it works ? for example, take 4 random SNPs per gene so you get many such SNPs.

To solve the problem: since you are worried of the small differences particularly the sign differences, my suggestion is to rerun but using a much larger random.b sample and see how it works ? for example, take 4 random SNPs per gene so you get many such SNPs.

We can hold on to this. Because picking these SNPs should be built int othe updated pipeline with fine-mapping CS in mind. We can revisit at that point. But it is good for you (and important) that you keep documenting these problems. @rfeng2023


Clarity in complex client versions

Improve clarity in the client version with more than 20 catalogs. In RDMO, many different institutions (= clients) can be managed together in one RDMO installation. With many different question catalogs per client, a clear assignment of client and question catalogs would be useful.
[This issue is from 7th RDMO community meeting on March 2, 2022]

We can only address this when implementing a new front-end: https://github.com/rdmorganiser/rdmo/issues/518

In the new interface, catalogs (and everything else) can now be filtered by site and uri prefix.

Can be closed by 2.0.0?


Support for warnings

Does the library support the _warning field that can come back? This is documented here:
https://www.geocod.io/docs/#warnings
As an example:
curl --request GET \
  --url 'https://api.geocod.io/v1.7/geocode?api_key=<SNIPPED>&fields=acs-social&q=Road%20X%2C%20Hot%20Springs%2C%20AR&limit=1'

returns
{
	"input": {
		"address_components": {
			"street": "Rd",
			"formatted_street": "Rd",
			"city": "Hot Springs",
			"state": "AR",
			"country": "US"
		},
		"formatted_address": "Rd, Hot Springs, AR"
	},
	"results": [
		{
			"address_components": {
				"city": "Hot Springs",
				"county": "Garland County",
				"state": "AR",
				"zip": "71901",
				"country": "US"
			},
			"formatted_address": "Hot Springs, AR 71901",
			"location": {
				"lat": 34.526453,
				"lng": -92.974072
			},
			"accuracy": 0.45,
			"accuracy_type": "place",
			"source": "TIGER\/Line® dataset from the US Census Bureau",
			"fields": {
				"census": {
					"2021": {
						"census_year": 2021,
						"state_fips": "05",
						"county_fips": "05051",
						"tract_code": "010502",
						"block_code": "1001",
						"block_group": "1",
						"full_fips": "050510105021001",
						"place": null,
						"metro_micro_statistical_area": {
							"name": "Hot Springs, AR",
							"area_code": "26300",
							"type": "metropolitan"
						},
						"combined_statistical_area": {
							"name": "Hot Springs-Malvern, AR",
							"area_code": "284"
						},
						"metropolitan_division": null,
						"source": "US Census Bureau"
					}
				}
			},
			"_warnings": [
				"acs-social was skipped since result is not street-level"
			]
		}
	]
}

I don't see any way to read the _warnings field.
In addition, the documentation states the _warnings field can also show up on the same level as results.
{
  "input": {
    ...
  },
  "results": [
    ...
  ],
  "_warnings": [
    "Ignoring parameter \"postalcode\" as it was not expected. Did you mean \"postal_code\"? See full list of valid parameters here: https://www.geocod.io/docs/"
  ]
}

I assume there just needs to be new fields added to GeocodingResponse and GeocodingResult.

By the way, the fix is simple. Let me know if you want me to submit a pull request.

Hi @ctmay4! You're more than welcome to submit a pull request. If not I'll try to get to this soon

Pull request is ready.


Fix request.cf's botManagement.score description

The description was backwards. Score 1 is bot, 99 is human.

I have read the CLA Document and I hereby sign the CLA.

recheck

cc @mrbbot @kentonv 🙏


How do I pass the None value explicitly?

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Typer documentation, with the integrated search.
[X] I already searched in Google "How to X in Typer" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to Typer but to Click.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
from typing import Optional

import typer

def main(
    limit: Optional[int] = typer.Option(11),
):
    print(limit)

if __name__ == "__main__":
    typer.run(main)

Description
How do I pass the None value for an Optional argument that has a non-None default value?
Operating System
macOS
Operating System Details
No response
Typer Version
0.7.0
Python Version
Python 3.9.13
Additional Context
No response

Semantically None is used to communicate the absence of a specified value in typer. So you can't as far as I can tell set it explicitly or enable functionality to allow it.
My proposal is that you either use a special value (for example -1 or 0) to indicate the limit, or to create a special --disable-limit flag, like so:
from typing import Optional

import typer

def main1(
    limit: Optional[int] = typer.Option(11, help="A limit value of 0 disables the limit.")
):
    if limit > 0:
        print(limit)
    else:
        print("unlimited")

def main2(
    limit: Optional[int] = 11,
    disable_limit: bool = typer.Option(False, "--disable-limit")
):
    if disable_limit:
        limit = None
    print(limit)

if __name__ == "__main__":
    typer.run(main1)

Update here: with custom parsers, you should be able to implement this as of typer 0.8.0!
from typing import Optional
import typer

def int_or_none(raw: str) -> Optional[int]:
    return None if raw == 'None' else int(raw)

def main(
    limit: Optional[int] = typer.Option(11, parser=int_or_none),
):
    print(limit)

if __name__ == "__main__":
    typer.run(main)

Usage:
$ /usr/bin/python3 main.py
11
$ /usr/bin/python3 main.py --limit None
None
$ /usr/bin/python3 main.py --limit 3
3

The name of the parser is even displayed in the help string:
$ /usr/bin/python3 main.py --help
Usage: main.py [OPTIONS]

Options:
  --limit INT_OR_NONE  [default: 11]
  --help               Show this message and exit.

You could modify the metavar or help field of your Option for additional clarify.


We need a logo for this project

We need a logo for this project. Ideally it should be something that fits nicely into circle and square since our design language has room for a circle logo and the favicon will be a square. But it's important that it does not look anything like the OpenAI / ChatGPT logo.
Aesthetically it should not require color or fine detail in order to recognize it. It should look great in greyscale and it should be recognizable even as small as the favicon. Hopefully DALL-E can help us.  Maybe you have to reply before I can put you on it.

Let’s see if it works now.

GRMNStudio_HostedGPT_Brand_PPT_01.pdf
@krschacht
My friend has come up with a couple of suggestions.
Feel free to go ahead and choose the one you like best. After that, I'll add the logo to the relevant places in our layout and upload a couple of variants to Google Drive, Dropbox or whatever other place we'd like to use to store this type of asset for now.

@florianfelsing
Tell your friend thank you! These are two very good options!! Your friend is talented.  I’m going to look at them a few more times in different context and consider different trade offs. Let me circle back tomorrow or over the weekend, but I’ll pick soon in order to keep us moving forward on this.

@florianfelsing okay I’ve tried them out in different contexts and at different sizes. I actually think this one is the best:

Visually, it’s more symmetrical so it looks nicer when placed inside a square or circle. It has a nice distinctive shape which remains recognizable even when you get down to favicon sizes. And it has a really nice optical illusion effect. You can stare at it and see it as a spiral galaxy (arms swirling around a square) or as a 3d box coming off the page. :) It’s like that classic: is it a face or a vase.
I think what would be good to add to the repo are:

A master vector file (AI or whatever file format it was made in)
Is it easy to get an SVG of this? Then we can use that throughout the app in all different contexts.
 These will be good for places where we have to upload a logo like to a Twitter account.

Let me know if you think of any other considerations.

Hi @florianfelsing do you think you'll have time this week to help finalize this? I'm thinking about doing a small push this week to get more awareness of HostedGPT. I have a hunch that the Claude 3 support might be reason enough for some people to want to try it since there's a good amount of buzz about Claude 3 and they don't have a nice consumer-friendly UI.


Set up repo for reporting client

The python client should be located in its own repository, which needs to be set up:

Create new repo frequenz-client-reporting-python (according to https://github.com/frequenz-floss/frequenz-sdk-python/wiki/APIs-stack-and-repositories-structure)
Configure the repo using repo config as described here: https://frequenz-floss.github.io/frequenz-repo-config-python/

This PR for the microgrid API client can be used as a reference: https://github.com/frequenz-floss/frequenz-client-microgrid-python/pull/1.

Why is this issue in this repo?


Support cluster mode to deploy ?

Support cluster mode to deploy ?
I've deployed and test, It's very cool，But I have multiple GPU nodes. Do we support cluster mode to deploy?
If not,  We plan to support it?

we are planning to add support for distributing training task to remote GPU machine in next Major version.

we are planning to add support for distributing training task to remote GPU machine in next Major version.
ok，thanks, When do we plan to release the next major version？

June, probably.

ok，thanks


[5.2] Account for __isset changes in PHP 7

PHP 7 has fixed a bug with __isset which affects both the native isset and empty methods.  This causes specific issues with checking isset or empty on relations in Eloquent.  In PHP 7 checking if a property exists on an unloaded relation, for example isset($this->relation->id) is always returning false because unlike PHP 5.6, PHP 7 is now checking the offset of each attribute before chaining to the next one.  In PHP 5.6 it would eager load the relation without checking the offset.  This change brings back the intended behavior of the core Eloquent model __isset method for PHP 7.
For reference, please check the following link, specifically Nikita Popov's comment (core PHP dev) - https://bugs.php.net/bug.php?id=69659

Thank you so much. Could you please send this to 5.1 though, since we'd like to patch bugs on our LTS version. ;)


fix for bits type

In manual tests came out that bits type was not properly cast. OnWrite function got uint64 instead of []string. This fix solves that.

TBH, i think the uint64 probably is the right return for Value()

Also, I noticed FreeCONF was missing support for BitList so I added it and thought following name changes would be better:
Decimal -> Positions
StringVals -> Labels
I should have submitted a PR for you to review first but I ran short on time, sorry.


Constant modeling for getitem

🚀 Feature
It would be useful if we would model constants and use that to optimize away getitem.
Motivation
The ThunderFX path sometimes ends up giving us some rather silly graphs:
  class DynamoModule(torch.nn.Module):
    def forward(self):
        scale_t = torch.tensor([2])
        getitem = scale_t[0]
        return (getitem, scale_t)

Our trace ends up correspondingly silly:
def computation():
  scale_t = torch.tensor([2], device=torch.device("cpu"), dtype=None)  # scale_t: "cpu i64[1]"
    # scale_t = ltorch.tensor([2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # scale_t: "cpu i64[1]"
      # scale_t = prims.tensor_from_sequence([2], dtype=None, device=devices.Device("cpu"))  # scale_t: "cpu i64[1]"
  getitem = Tensor.__getitem__(scale_t, 0)  # getitem: "cpu i64[]"
    # getitem = ltorch.getitem(scale_t, 0)  # getitem: "cpu i64[]"
      # t7 = prims.slice_prim(scale_t, [0], [1], [1])  # t7: "cpu i64[1]"
      # getitem = prims.squeeze(t7, (0,))  # getitem: "cpu i64[]"
  return (getitem, scale_t)

g28.py - self contained test case.
Pitch
I'd love to see us produce this trace:
def computation():
  scale_t = torch.tensor([2], device=torch.device("cpu"), dtype=None)  # scale_t: "cpu i64[1]"
    # scale_t = ltorch.tensor([2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # scale_t: "cpu i64[1]"
      # scale_t = prims.tensor_from_sequence([2], dtype=None, device=devices.Device("cpu"))  # scale_t: "cpu i64[1]"
  return (2, scale_t)

i.e. we elide the getitem call and just use a 2 directly. foo = 2 and return (foo, scale_t) is also reasonable for now.
Alternatives
Another idea is to do constant folding and DCE at the FX graph level, and if the graph is empty then we just don't even invoke thunder.jit at all. Then Thunder is only tasked with handling "real" computations.
If we reject this entirely, we could say that the user is responsible for e.g. torch.compiler.allow_in_graphing appropriately such that we never get isolated graphs like the one above.
Additional context
These are happening in a variety of subgraphs from the NeVA case, and I suspect we'll see them often as we look into supporting other networks verbatim.
cc @tfogal

I for one would love to see a constant folding pass.

I have an example pass as transform for this specific case.
I assume down the line FusionExecutor (or transform_for_execution) may change the exact computation based on this, so this is a pre_prologue transform which is called before these execution passes is applied.
NOTE - Generated trace is in the snippet.
import torch
import thunder
from thunder.core.trace import from_trace
from thunder.core import utils
from collections.abc import Sequence
from thunder.core.pytree import tree_flatten
from thunder.core.proxies import variableify, Variable
import numbers

class GetItemFromScalarTensorConstantFold(thunder.Transform):
    def transform_traces_pre_prologue(self, prologue_trc, computation_trc, epilogue_trc, **kwargs):
        # Create a new trace
        const_folded_trace = from_trace(computation_trc)
        const_folded_trace.bound_symbols = computation_trc.bound_symbols

        producers = utils.producers(computation_trc)
        swap_map: dict[Variable, numbers.Number] = {}
        for bsym in const_folded_trace.bound_symbols:
            # If symbol `getitem` with zero-dim tensor output.
            if bsym.sym.id == thunder.torch.getitem.id and bsym.output.shape == ():
                getitem_bsym = bsym  # Nicer name for bsym
                getitem_producer = producers[getitem_bsym.args[0]]
                # Check if producer if `torch.tensor`.
                if getitem_producer.sym.id in (thunder.torch.tensor.id,):
                    # TODO: Support folding scalar elem from multi-dim tensor.
                    tensor_value = getitem_producer.args[0]
                    flat_values, _ = tree_flatten(tensor_value)
                    if isinstance(tensor_value, Sequence) and len(flat_values) == 1:
                        tensor_value = tensor_value[0]
                    elif isinstance(tensor_value, numbers.Number):
                        pass
                    else:
                        # tensor_values are multi-elem or
                        # it could be an ndarray, etc (though currently thunder only supports Number or Sequence).
                        continue

                    swap_map[variableify(getitem_bsym.output)] = tensor_value

        updated_bsyms = []
        for bsym in const_folded_trace.bound_symbols:
            updated_bsyms.append(bsym.from_bsym_swap_proxies(swap_map))

        const_folded_trace.bound_symbols = updated_bsyms
        const_folded_trace.set_provenance("GetItem from Scalar Tensor Const Folding pass")
        return prologue_trc, const_folded_trace, epilogue_trc

def forward():
    scale_t = torch.tensor([2])
    getitem = scale_t[0]
    return (getitem, scale_t)

jf = thunder.jit(forward, transforms=[GetItemFromScalarTensorConstantFold()])
jf()

print(thunder.last_traces(jf)[-1])

# def computation():
#   scale_t = torch.tensor([2], device=torch.device("cpu"), dtype=None)  # scale_t: "cpu i64[1]"
#     # scale_t = ltorch.tensor([2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # scale_t: "cpu i64[1]"
#       # scale_t = prims.tensor_from_sequence([2], dtype=None, device=devices.Device("cpu"))  # scale_t: "cpu i64[1]"
#   return (2, scale_t)

def forward(x):
    scale_t = torch.tensor([2])
    getitem = scale_t[0]  # This will be const folded

    # This won't be folded for now because `torch.tensor` has more than 1 elem.
    # Though, will get it working in future.
    getitem_2 = torch.tensor([2, 2])[0]
    return x + getitem + getitem_2

jf = thunder.jit(forward, transforms=[GetItemFromScalarTensorConstantFold()])
jf(torch.randn(3, 3, requires_grad=True))

print(thunder.last_traces(jf)[-1])

# def computation(x):
#   # x: "cpu f32[3, 3]"
#   t3 = torch.tensor([2, 2], device=torch.device("cpu"), dtype=None)  # t3: "cpu i64[2]"
#     # t3 = ltorch.tensor([2, 2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # t3: "cpu i64[2]"
#       # t3 = prims.tensor_from_sequence([2, 2], dtype=None, device=devices.Device("cpu"))  # t3: "cpu i64[2]"
#   t7 = torch.add(x, 2.0)  # t7: "cpu f32[3, 3]"
#     # t7 = ltorch.add(x, 2.0, alpha=None)  # t7: "cpu f32[3, 3]"
#       # t7 = prims.add(x, 2.0)  # t7: "cpu f32[3, 3]"
#   t11 = torch_slice_prim_impl(t3, [0], [1], [1])  # t11: "cpu i64[1]"
#   del t3
#   getitem_2 = torch.squeeze(t11, (0,))  # getitem_2: "cpu i64[]"
#     # getitem_2 = ltorch.squeeze(t11, (0,))  # getitem_2: "cpu i64[]"
#       # getitem_2 = prims.squeeze(t11, (0,))  # getitem_2: "cpu i64[]"
#   del t11
#   t14 = Tensor.to(getitem_2, torch.float32, copy=True)  # t14: "cpu f32[]"
#     # t14 = ltorch.to(getitem_2, torch.float32, None, device=None, dtype=None, copy=True, memory_format=None)  # t14: "cpu f32[]"
#       # t14 = prims.convert_element_type(getitem_2, dtypes.float32)  # t14: "cpu f32[]"
#   del getitem_2
#   t9 = torch.add(t7, t14)  # t9: "cpu f32[3, 3]"
#     # t9 = ltorch.add(t7, t14, alpha=None)  # t9: "cpu f32[3, 3]"
#       # t9 = prims.add(t7, t14)  # t9: "cpu f32[3, 3]"
#   del t7, t14
#   return {'output': t9, 'flat_args': [x], 'flat_output': (t9,)}, ((), ())

Just wanted to double check if this is what you had in mind. Also, should it be present by default in thunder.jit with an opt-out flag? Or should we provide this as a transform and user can opt-in by passing it via transforms argument to thunder.jit.

As per offline chat with @t-vi, here is a more generic approach. Is this roughly what you had in mind?
Thanks @t-vi for the idea and suggestions!!
import torch
import thunder
from thunder.core.trace import from_trace, TraceCtx, tracectx
from thunder.core import utils
from collections.abc import Sequence
from thunder.core.pytree import tree_flatten
from thunder.core.proxies import variableify, Variable, TensorProxy, NumberProxy
from thunder.core.symbol import BoundSymbol
from thunder.core.dtypes import to_dtype
from thunder.core.devices import to_device
import numbers
from thunder.core.proxies import ProxyTag
from thunder.torch import _torch_to_thunder_function_map

_thunder_to_torch_function_map = {v: k for k,v in _torch_to_thunder_function_map.items()}

ProxyTag.register_tag("CONSTANT_VALUE")

def is_constant(proxy):
    if isinstance(proxy, TensorProxy) and ProxyTag.CONSTANT_VALUE in proxy.tags:
        return True
    elif isinstance(proxy, NumberProxy) and proxy.is_static_constrained():
        return True
    return False

def compute_with_materialized_tensor(bsym, const_values):

    def materialize_args(a):
        if isinstance(a, TensorProxy):
            return const_values[variableify(a)]
        elif isinstance(a, NumberProxy):
            return a.value
        return a

    new_args = tuple(map(materialize_args, bsym.args))
    new_kwargs = {k: materialize_args(v) for k, v in bsym.kwargs.items()}
    torch_fn = _thunder_to_torch_function_map.get(bsym.sym, None)
    if torch_fn is None:
        return
    return torch_fn(*new_args, **new_kwargs)

class ConstantFold(thunder.Transform):
    def __init__(self, const_fold_scalar_tensor_from_getitem=True):
        self.const_fold_scalar_tensor_from_getitem = const_fold_scalar_tensor_from_getitem

    def transform_traces_pre_prologue(self, prologue_trc, computation_trc, epilogue_trc, **kwargs):
        # Create a new trace
        const_folded_trace = from_trace(computation_trc)
        const_folded_trace.bound_symbols = computation_trc.bound_symbols

        # Factory functions whose value we know.
        TENSOR_FACTORY = (thunder.torch.tensor.id, thunder.torch.ones.id, thunder.torch.zeros.id,)
        # TODO: Clear them at the last usage in trace.
        const_tensors: dict[Variable, BoundSymbol] = {}

        # Tag output from factory functions as constant value.
        for bsym in const_folded_trace.bound_symbols:
            if bsym.sym.id in TENSOR_FACTORY:
                bsym.output.tags.add(ProxyTag.CONSTANT_VALUE)
                torch_fn = _thunder_to_torch_function_map[bsym.sym]
                t = torch_fn(*bsym.args, **bsym.kwargs)
                const_tensors[variableify(bsym.output)] = t

        new_bsyms = []
        swap_map_getitem = {}
        for bsym in const_folded_trace.bound_symbols:
            # If bsym has constant inputs, try to compute the output.
            if all(map(is_constant, bsym.flat_proxy_args)) and bsym.sym.id not in TENSOR_FACTORY:
                if bsym.flat_args == []:  # eg, unpack_trivial
                    continue
                new_concrete_output = compute_with_materialized_tensor(bsym, const_tensors)
                if new_concrete_output is not None:  # Might happen for `python_return` as it won't have mapping in `_thunder_to_torch_map`.

                    # Create a new symbol with same output proxy but which will now represent the computed constant value.
                    # eg. t = known_tensor + 1 where known_tensor = torch.tensor(2) --> t = torch.tensor(3)
                    new_bsym = BoundSymbol(thunder.prims.tensor_from_sequence,
                                           (new_concrete_output.tolist(),), {'dtype': to_dtype(new_concrete_output.dtype), 'device': to_device(new_concrete_output.device)},
                                           output=bsym.output)
                    new_bsyms.append(new_bsym)

                    # Update const_tensors (so that usage of the output of this symbol will also be used for further computation.)
                    const_tensors[variableify(bsym.output)] = new_concrete_output
                    bsym.output.tags.add(ProxyTag.CONSTANT_VALUE)
                    if self.const_fold_scalar_tensor_from_getitem and bsym.sym.id == thunder.torch.getitem.id and bsym.output.shape == ():
                        swap_map_getitem[variableify(bsym.output)] = new_concrete_output.tolist()
                    continue

            # BoundSymbol with non-constant inputs, keep it as-is
            new_bsyms.append(bsym)

        del const_tensors

        const_folded_trace.bound_symbols = new_bsyms

        # Replace 0-dim tensors from getitem with Python scalars.
        if self.const_fold_scalar_tensor_from_getitem:
            new_bsyms = []
            for bsym in const_folded_trace.bound_symbols:
                new_bsyms.append(bsym.from_bsym_swap_proxies(swap_map_getitem))

        const_folded_trace.bound_symbols = new_bsyms
        const_folded_trace.set_provenance("Const Folding pass")
        return prologue_trc, const_folded_trace, epilogue_trc

def forward():
    scale_t = torch.tensor([2])
    getitem = scale_t[0]
    return (getitem, scale_t)

jf = thunder.jit(forward, transforms=[ConstantFold()])
jf()

print(thunder.last_traces(jf)[-1])

# def computation():
#   scale_t = torch.tensor([2], device=torch.device("cpu"), dtype=None)  # scale_t: "cpu i64[1]"
#     # scale_t = ltorch.tensor([2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # scale_t: "cpu i64[1]"
#       # scale_t = prims.tensor_from_sequence([2], dtype=None, device=devices.Device("cpu"))  # scale_t: "cpu i64[1]"
#   return (2, scale_t)

def forward(x):
    scale_t = torch.tensor([2])
    getitem = scale_t[0]
    getitem_2 = torch.tensor([2, 2])[0]
    return x + getitem + getitem_2

jf = thunder.jit(forward, transforms=[ConstantFold()])
jf(torch.randn(3, 3, requires_grad=True))

# print(thunder.last_traces(jf)[-1])

# def computation(x):
#   # x: "cpu f32[3, 3]"
#   t7 = torch.add(x, 2.0)  # t7: "cpu f32[3, 3]"
#     # t7 = ltorch.add(x, 2.0, alpha=None)  # t7: "cpu f32[3, 3]"
#       # t7 = prims.add(x, 2.0)  # t7: "cpu f32[3, 3]"
#   t9 = torch.add(t7, 2.0)  # t9: "cpu f32[3, 3]"
#     # t9 = ltorch.add(t7, 2.0, alpha=None)  # t9: "cpu f32[3, 3]"
#       # t9 = prims.add(t7, 2.0)  # t9: "cpu f32[3, 3]"
#   del t7
#   return {'output': t9, 'flat_args': [x], 'flat_output': (t9,)}, ((), ())

def forward(x):
    scale_t = torch.tensor([2], dtype=torch.float16)
    ones_t = torch.ones(1, dtype=torch.float32)
    s1 = scale_t * 2
    s2 = scale_t / 1
    s3 = s1 * s2
    ones_mul_10 = ones_t * 10
    return x[0, 0] + s3 + ones_mul_10

jf = thunder.jit(forward, transforms=[ConstantFold()])
forward(torch.randn(3, 3, requires_grad=True))
jf(torch.randn(3, 3, requires_grad=False))

# print(thunder.last_traces(jf)[-1])

# def computation(x):
#   # x: "cpu f32[3, 3]"
#   s3 = torch.tensor([8.0], device=torch.device("cpu"), dtype=torch.float16)  # s3: "cpu f16[1]"
#     # s3 = ltorch.tensor([8.0], device=torch.device("cpu"), dtype=torch.float16, requires_grad=False, pin_memory=False)  # s3: "cpu f16[1]"
#       # s3 = prims.tensor_from_sequence([8.0], dtype=dtypes.float16, device=devices.Device("cpu"))  # s3: "cpu f16[1]"
#   ones_mul_10 = torch.tensor([10.0], device=torch.device("cpu"), dtype=torch.float32)  # ones_mul_10: "cpu f32[1]"
#     # ones_mul_10 = ltorch.tensor([10.0], device=torch.device("cpu"), dtype=torch.float32, requires_grad=False, pin_memory=False)  # ones_mul_10: "cpu f32[1]"
#       # ones_mul_10 = prims.tensor_from_sequence([10.0], dtype=dtypes.float32, device=devices.Device("cpu"))  # ones_mul_10: "cpu f32[1]"
#   t14 = Tensor.__getitem__(x, (0, 0))  # t14: "cpu f32[]"
#     # t14 = ltorch.getitem(x, (0, 0))  # t14: "cpu f32[]"
#       # (_, _) = prims.shape(x)
#       # t27 = prims.slice_prim(x, [0, 0], [1, 1], [1, 1])  # t27: "cpu f32[1, 1]"
#       # t14 = prims.squeeze(t27, (0, 1))  # t14: "cpu f32[]"
#   t16 = torch.add(t14, s3)  # t16: "cpu f32[1]"
#     # t16 = ltorch.add(t14, s3, alpha=None)  # t16: "cpu f32[1]"
#       # t29 = prims.convert_element_type(s3, dtypes.float32)  # t29: "cpu f32[1]"
#       # t16 = prims.add(t14, t29)  # t16: "cpu f32[1]"
#   del t14, s3
#   t17 = torch.add(t16, ones_mul_10)  # t17: "cpu f32[1]"
#     # t17 = ltorch.add(t16, ones_mul_10, alpha=None)  # t17: "cpu f32[1]"
#       # t17 = prims.add(t16, ones_mul_10)  # t17: "cpu f32[1]"
#   del t16, ones_mul_10
#   return t17
# print(thunder.last_traces(jf)[-1])

@kshitij12345 Why make this a pass?
What if the implementation of torch.item (or prims.item?) looked like:
def item(a: TensorLike, /) -> Number:
	if is_constexpr(a):
		return a.constexpr_value

    return prims.item(a)

And what if calling torch.ones and other tensor factories appropriately labeled tensors as constexpr when given constexpr inputs?
The reason why I'd prefer to keep the logic with the operators, if possible, is that a pass (as in the above) has to implement operator-specific logic away from the operators. If a pass changes the behavior of specific operators and doesn't require cross-operator analysis then it might best be implemented at the operator level.
For example, would we prefer a pass that translated prims.mul(a, 1) to a nullop, or would we want the mul operation itself to contain the logic that makes it a nullop? I think putting the logic with the operators implements the transform through abstract interpretation, which is nice.
Maybe @tfogal wants to argue for a transform over operator-logic changes, however.

@mruberry thanks for the suggestion. However, I am leaning a bit more towards pass.
Example snippet -
def forward():
    scale_t = torch.tensor([2])
    getitem = ((scale_t * 10 ) - 2)[0]
    return (getitem, scale_t)

For the above snippet, with operation based approach as you have mentioned, mul, neg and torch.getitem (and any other deterministic) would have to know about the constant propagation, I think it could lead to coverage problem. And I imagine, in case of bugs, it would be easy to reason and debug through the pass than to look at all operators and figure why 1 or 2 operator are doing something funny. So, I feel this is more suitable for the pass which will take care of this logic (using PyTorch eager for computation).  Wdyt?

If a pass changes the behavior of specific operators and doesn't require cross-operator analysis then it might best be implemented at the operator level.

I don't think this pass would change the behaviour of operator (except for the getitem, I guess) as the pass should be following same semantics just at compile time.
Curious to hear other opinions as well.

@tfogal regarding the original request to inline the output of getitem, one thing I wanted to highlight was scale_t[0] is a scalar tensor and not a Python scalar. scale_t[0].item() will return a Python scalar, so do we expect a Python scalar?
Original Graph -
  class DynamoModule(torch.nn.Module):
    def forward(self):
        scale_t = torch.tensor([2])
        getitem = scale_t[0]
        return (getitem, scale_t)

Expected Computation Graph (from PR description)
def computation():
  scale_t = torch.tensor([2], device=torch.device("cpu"), dtype=None)  # scale_t: "cpu i64[1]"
    # scale_t = ltorch.tensor([2], device=torch.device("cpu"), dtype=None, requires_grad=False, pin_memory=False)  # scale_t: "cpu i64[1]"
      # scale_t = prims.tensor_from_sequence([2], dtype=None, device=devices.Device("cpu"))  # scale_t: "cpu i64[1]"
  return (2, scale_t)

PyTorch Eager behavior -
>>> import torch
>>> torch.randn(3)[0]
tensor(-0.3535)
>>> torch.randn(3)[0].item()
0.9367182850837708

@mruberry thanks for the suggestion. However, I am leaning a bit more towards pass.
Example snippet -
def forward():
    scale_t = torch.tensor([2])
    getitem = ((scale_t * 10 ) - 2)[0]
    return (getitem, scale_t)

For the above snippet, with operation based approach as you have mentioned, mul, neg and torch.getitem (and any other deterministic operation) would have to know about the constant propagation, I think it could lead to coverage problem. And I imagine, in case of bugs, it would be easy to reason and debug through the pass than to look at all operators and figure why 1 or 2 operator are doing something funny. So, I feel this is more suitable for the pass which will take care of this logic (using PyTorch eager for computation). Wdyt?

I think it's easier for always on per-operator logic to be with the operator description. Otherwise if people update an operator, or add a new operator that can reason about constexpr inputs, then both the operator and the pass need to be updated. Essentially the logic for an operation is split into two places. If another pass did the same then a single operator could have its logic in three places, and so on.
The maybe_convert_to_dtype operation:
https://github.com/Lightning-AI/lightning-thunder/blob/b4ca020b6e194ac9cd6ce47a28c04df963110a83/thunder/clang/__init__.py#L150
Follows this logic. In the case of a no-op it performs a no-op that is removed by DCE. Putting that logic into a pass would just mean looking at multiple places to understand what the operation actually did, and people would have to know which other passes applied to the operation (or didn't), too. I think it would be confusing.
Transforms/passes are best used for optional per-operator transformations (like grad) or transformations that reason about multiple operations (like distributed).

If a pass changes the behavior of specific operators and doesn't require cross-operator analysis then it might best be implemented at the operator level.

I don't think this pass would change the behaviour of operator (except for the getitem, I guess) as the pass should be following same semantics just at compile time.

My point is that this would be an "always on" and "per operator" transform, which I think is the category to implement with the operator logic.
Implementing things at the operator level is also typically nicer for debuggability and maintainability, as the manipulation is often more direct and reproducible, not requiring modifying an entire pass to experiment.

do we expect a Python scalar?

You are correct, of course; I should have looked at what torch does and not what we do :-). That the graph happened to name the tensor getitem despite never calling item() certainly didn't help me.
Anyway, in general I would say the rule is match torch, but in this case it seems we've been returning a scalar instead of a tensor for a while and it seems to have been fine. Maybe torch is just setup to accept a raw Python value and a scalar tensor interchangeably? If we can get away with the raw Python value, that seems cheaper and so is reasonable to me. I'm curious as to other's thoughts, though, about whether strict adherence to what torch would do is desired here.

do we expect a Python scalar?

You are correct, of course; I should have looked at what torch does and not what we do :-). That the graph happened to name the tensor getitem despite never calling item() certainly didn't help me.
Anyway, in general I would say the rule is match torch, but in this case it seems we've been returning a scalar instead of a tensor for a while and it seems to have been fine. Maybe torch is just setup to accept a raw Python value and a scalar tensor interchangeably? If we can get away with the raw Python value, that seems cheaper and so is reasonable to me. I'm curious as to other's thoughts, though, about whether strict adherence to what torch would do is desired here.

If PyTorch returns a tensor here, we should probably return a tensor. While PyTorch may allow tensors or numbers (almost always interchangeably), not all Python code does.

Closing this issue as we have added ConstantFolding as a transform in #1273. Have filed a separate issue on how it could be added as a default transform to thunder.jit in #1299