[RFC/style] Add a style guide or best practices document for RISC-V assembly.

I've spend the last few days reviewing a large (well, dozens of lines, but I think that's a lot) of assembly code, and making significant comments on style and best practice. Seeing as how the volume of assembly will only increase, I think it's prudent to make explicit decisions about what we want our assembly files to look like.
I've sketched out some ideas in my fork for what kinds of things I've spotted in code review and which I think we should explicitly require. I'm hoping to get feedback on:

The list of explicit callouts in the draft (too many, too few).
The content of the explicit callouts (a few things in it are my personal preference, and really just a starting point for discussion).
Whether the callouts for inline assembly belong in the C/C++ style guide, instead. Google has no rules on inline assembly, so we don't get anything for free from using their style guide.

Also, I'm not sure if this change is of appropriate scope for an RFC; I figure it is, but I'll move this discussion to a different venue if that would be more appropriate.
Meta: https://docs.opentitan.org/doc/project/rfc_process/ says I should be tagging this with the Type:RFC label, which doesn't seem to exist yet... oops?

this is great! thanks for getting this started!
For the inline assembly portion, maybe we should link to it also from the
C-style guide?
For the examples, I'm partial to the style used in the verilog style guide (
https://github.com/lowRISC/style-guides/blob/master/VerilogCodingStyle.md#begin--end)
where each good / bad examples is in its own code block.  This feels more
clean to me, although I could go either way on this one.
On Thu, Nov 7, 2019 at 12:15 PM Miguel Young<EMAIL_ADDRESS>wrote:

I've spend the last few days reviewing a large (well, dozens of lines, but
I think that's a lot) of assembly code, and making significant comments on
style and best practice. Seeing as how the volume of assembly will only
increase, I think it's prudent to make explicit decisions about what we
want our assembly files to look like.
I've sketched out some ideas in my fork
https://github.com/mcy/opentitan-1/blob/16a109e899bc540090f9207fa98607fb6b316b77/doc/rm/asm_coding_style.md
for what kinds of things I've spotted in code review and which I think we
should explicitly require. I'm hoping to get feedback on:

The list of explicit callouts in the draft (too many, too few).
The content of the explicit callouts (a few things in it are my
personal preference, and really just a starting point for discussion).
Whether the callouts for inline assembly belong in the C/C++ style
guide, instead. Google has no rules on inline assembly, so we don't get
anything for free from using their style guide.

Also, I'm not sure if this change is of appropriate scope for an RFC; I
figure it is, but I'll move this discussion to a different venue if that
would be more appropriate.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/lowRISC/opentitan/issues/925?email_source=notifications&email_token=AAH2RSRMO2G5M57SHIGJDRTQSRZOTA5CNFSM4JKNADYKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HXWYOIQ,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH2RSUZTG2ZY62POK4WQ7DQSRZOTANCNFSM4JKNADYA
.

I also think this is a great idea - thanks for kicking it off. CCing @lenary and @luismarques.
First on the meta point about the RFC process: In terms of the lifetime described in the RFC process doc - you're currently at the "Optional: seek feedback on whether the RFC is desired stage" and I guess overlapping that with getting early feedback. So the path forwrads is to collect feedback on this thread, work it into your proposal and then we can mark it as an RFC ready for review when feedback has stabilised. Note the Technical Committee described in that doc is in the process of being instantiated, but I hope that's resolved pretty soon and we can use this as a test case for the process.
Feedback/ideas:

Add a rule on the use of CSR names in assembly. My recommendation would be that CSR names should be used for CSRs defined in the standard (e.g. mtvec), but they should be #defined constants for non-standard extensions in order to minimise toolchain requirements.
Add a rule banning the use of the 'weakly typed' aliases, like add $rd, $rs1, $simm12 -> addi $rd, $rs1, $simm12 - they're only going to confuse people. These aliases exist both for the basic integer operations and csr[r]w/csr[r]s/csr[r]c
The advice to use pseudoinstructions like call and la is obviously correct. You might want to add an example and express an opinion on the aliases for sgt, beqz, bgt etc etc. Note that assemblers accept sgt/sgtu/bgt/ble/bgtu/bleu but will not print them when disassembling. In contrast, beqz/bnez/blez/bgez/bltz/bgtz will be printed when disassembling. Should that influence the decision on whether they are recommended or not?
I personally don't like lb $rd, ($rs1) and sb $rs2, ($rs1) etc and think the 0($rs1) forms should always be used.
It's implicit in your examples, but you clearly prefer // rather than # style comments. This should be made explicit.
In general our aim is to promote these style guides to lowRISC-wide style guides (as has happened with the SystemVerilog guide), this might mean we want to tweak some wording and maybe think slightly beyond Ibex with that in mind
Do we want guidance on indicating register usage? I suppose it depends how gnarly we expect any hand-written assembly to be...
This might be reasonable place to put a short notice about when assembly should/shouldn't be used. If you can use C instead you probably should. If you can use a compiler intrinsic (especially a target-indepent one like __builtin_clz) then you should. etc.

Other suggestions:

Always use .p2alignor .balign rather than .align. Assemblers for many architectures treat the .align operand as the alignment in bytes. For RISC-V it's power of two. Be explicit to avoid confusion.
Maybe more controversial: Always use .byte/.2byte/.4byte/.8byte. Justification: you don't want to have to second-guess whether .word and .long are 8 bytes for 64-bit.

Maybe more controversial: Always use .byte/.2byte/.4byte/.8byte. Justification: you don't want to have to second-guess whether .word and .long are 8 bytes for 64-bit.

I would absolutely endorse that requirement.  If I never had to read another .dw again I'd be happy.

@asb Thanks for the clarification and feedback! I think all of your comments are great ideas and gone ahead and incorporated the straightforward ones. Let me know your thoughts on the wording. Could I ask you to elaborate on:

What do you mean by "weakly typed" aliases? Why do you think they're confusing? An immediate always looks like an immediate, IMO.
Aliases for compare/branch instructions. I honestly don't care what the disasm looks like, since I don't expect that to be eminently readable, so I don't think it makes sense to have that affect our style guide. What do you think?

@asb Thanks for the clarification and feedback! I think all of your comments are great ideas and gone ahead and incorporated the straightforward ones. Let me know your thoughts on the wording. Could I ask you to elaborate on:
* What do you mean by "weakly typed" aliases? Why do you think they're confusing? An immediate always looks like an immediate, IMO.

Because the add instruction is a reg-reg operation and the addi instruction is reg-imm. Maybe it's personal preference, but I just find it really weird that add with reg-imm operations is accepted. I'll admit my suggestion is basically completely motivated by a gut reaction to it, but a more logical reason might be that it's not necessarily obvious if add a0, a0, 1 is a typo for add a0, a0, a1.

* Aliases for compare/branch instructions. I honestly don't care what the disasm looks like, since I don't expect that to be eminently readable, so I don't think it makes sense to have that affect our style guide. What do you think?

I don't really see a reason not to encourage them as much as any other pseudoinstruction. I agree with you.

Because the add instruction is a reg-reg operation and the addi instruction is reg-imm. Maybe it's personal preference, but I just find it really weird that add with reg-imm operations is accepted. I'll admit my suggestion is basically completely motivated by a gut reaction to it, but a more logical reason might be that it's not necessarily obvious if add a0, a0, 1 is a typo for add a0, a0, a1.

Is there such a thing as a linter for assembly languages? I somehow doubt it, but I feel the need to ask now...

I've sometimes wanted to implement a "strict mode" for the LLVM RISC-V MC layer which would disallow those particular aliases (we implemented them to get complete gas compatibility). I'm not aware of an asm linter.

So, a few other suggestions:

(Hand-written) Assembly files must have the .S extension (so other tooling knows what they are). .S will have the preprocessor invoked on them when assembled, whereas .s will not.

Compressed instruction mnemonics should be avoided

At least in LLVM, if you compile an assembly file with -march=rv32i*c, then all compressible instructions will be compressed, even if the uncompressed mnemonic was written.
Both LLVM and gas should recognise unimp as the all zeroes instruction (which is defined to always be illegal). This is actually a pseudo-instruction for two c.unimp instructions, as there is no 32-bit all zeroes instruction.
One possible exception for this rule is where it needs to be explicitly shown that code will fit into a certain number of bytes. I don't know if this issue will arise.

We should agree some set of prefixes for the non-standard CSR #defines, so they don't conflict with future definitions of standard CSR names.

The use of . (as a label for the current point) should be avoided. It is not obviously a label, and it's pretty easy to miss during review.

The exception here is the use of . when calculating function sizes, which seems reasonable.

Please can you justify the leading whitespace in your inline asm example with two instructions? I would find something like the following more reasonable:
asm volatile(
  "la sp, _stack_start;"
  "tail _crt0;"
  ::: "memory");

where the string literals do not contain leading whitespace. You get enough indenting due to the C/C++ requiring indenting of arguments, surely?

There is also a question as to whether we want a style guide for linker scripts. This should perhaps be discussed separately.

@lenary

My style guide already mandates the .S expression, so we're good there!
My style guide already suggests prefixing CSR defines with CSR_, but people aren't a fan I'm ok with bikeshedding a different prefix.
My justification for the extra space in inline asm is for labels:

asm volatile(
  "loop:"
  "  j loop;");

If we expect to rarely if ever have labels in inline asm, we can have the rule be: "two space indent required if and only if the asm snippet has labels in it".
All your other suggestions are great; I'll add them to the proposed style guide.
Re: .ld scripts: I think that's a separate question that I'd like a domain expert to comment on. I have not written enough ld to have opinions.

@mcy

For ibex-specific CSRs, might be worth explicitly calling out that they're Ibex-specific rather than opentitan-specific.
Ah, I see the label issue. I don't have a good grasp on how common labels in inline assembly are. Either version of the rule is fine by me.

Another thing that came up today on an ibex pull request is the use of li vs la. I would prefer us to stick to a stricter usage of these two pseudo-instructions:

la should only be used for loading symbol addresses (i.e. the second operand must be a symbol, I think gas is now more relaxed about what it accepts in the second operand).
li should be used for all other constant loads, including loading values introduced by #define or similar.
This pair of rules may need more nuance around symbol+offset - it would be good to hear more opinions about it.

@lenary
Hmm, good points. For CSRs, I propose the obvious CSR_<design>_<name> format; in the Ibex case, CSRs would start with CSR_IBEX_, while CSRs specific to the opentitan top would be CSR_OT_. How do you suggest documenting the choice of value for <design>? Do you have a better idea for how to format this?
I agree that we should be strict about la. I don't have a lot of experience with corner-cases of RISC-V assemblers, though... is la comparable to x86 lea, where you might write something morally equivalent to leaq 4(symbol), %rax?

@mcy I think CSR_IBEX_ and CSR_OT_ should be enough for the moment, and if we need more prefixes we can amend the guide later.
Looking further at la and friends. la will use the Global offset table in PIC-mode (and just emit pc-relative operands in non-PIC mode). I think this means it will never allow a symbol plus an offset.
lla seems to always emit pc-relative operands, and does allow symbol + offset. So I think RISC-V's lla <reg>, <symbol>+<offset> is the closest equivalent to x86's lea <offset>(<symbol>), <reg>, with the added restriction that RISC-V's lla does need a symbol where x86's lea does not.
LLVM's definitions of these pseudo-instructions seems to be quite a bit stricter than gas'.
For the moment, let's stick to la for symbols without offsets, and li for expressions.

* Both LLVM and gas should recognise `unimp` as the all zeroes instruction (which is defined to always be illegal). This is actually a pseudo-instruction for two `c.unimp` instructions, as there is no 32-bit all zeroes instruction.

Actually unimp expands to a fairly arbitrary 32-bit instruction rather than 2*c.unimp. c.unimp is at least defined so it is always an illegal instructions (the all zeros 16-bit instruction).

@asb Would you suggest instead requiring a pair of c.unimp instructions anywhere we want all-zeroes? Maybe we could create a OT_UNIMP macro to the same effect?

@asb Would you suggest instead requiring a pair of c.unimp instructions anywhere we want all-zeroes? Maybe we could create a OT_UNIMP macro to the same effect?

Although I don't much like the definition of unimp, I think it's ok to use it. Plus it's going to be converted to c.unimp whenever targeting a machine with the C extension anyway. I think __builtin_trap.
Going through #997, I'm not fully sure about the rules you're introducing for execution "falling off". I suppose the justification for inserting something more than the compiler would for C is that the chance for error is significantly higher. But is there are a reason you're typically preferring the infinite loop rather than a trapping instruction?

Another point of consideration: guidance on using __attribute__(naked) to avoid unnecessary function epilogue/prologue generation for functions that just contain inline asm.

Regarding https://github.com/lowRISC/opentitan/pull/997: That's great question. I think I made some fairly arbitrary choices there, since I expected it to be either temporary or source of a good discussion.
I think we should settle on a specific C idiom for an execution sink, be it an infinite wfi loop or an instruction that is guaranteed to trap. My de facto proposal is basically
asm volatile("unimp");
for (;;) { asm volatile(""); }

The infinite loop with empty volatile assembly is simply so that

The expression is treated as "never returning", which is useful for noreturn functions and for optimizations in general (telling the compiler that a basic block is a dead end is usually a good idea).
The infinite loop has "side effects" (it totally has side effects, I wrote volatile), to prevent UB.

Regarding naked functions... I don't know how I feel about them. They're kind of dangerous, and I don't think prologues and epilogues should be too bad for inline asm? There's no way a single assembly block will trigger spills, so there will effectively be no prologue and the epilogue will be a single ret. I think I trust the register allocator to not be completely stupid here. Is there anything else you're worried about sneaking into the prologue/epilogue?

@asb Is there anything I (or you?) should be doing to move this forward? I worry about blocking further changes in .S files over style considerations.

@mcy Alex will be back next week. How about you go ahead and do a google doc with the wording of the style guide based on the discussion we had so far?

@imphil I actually already have a Markdown version that I haven't PR'ed yet. I can copy it to a google doc and we can do the comments there, but I think it'd be a bit easier for me to just open the PR and use that to bikeshed it.
I don't care terribly much, so let me know which one you prefer.

Sure, just open a PR then.

As discussed in/after the recent software meeting, I think this is uncontentious and doesn't need to go through the RFC process. In general future style guides really should go through that process, but given it's not quite spun up yet we're best off landing this (assuming there are no major concerns on #1067) for expediency.


⚠ Bofrost.fr has degraded performance

In 8551739, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 883 ms

Resolved: Bofrost.fr performance has improved in e587c29 after 10 minutes.


Title attributions

Change to title attributions on all content in PMMI sites.

 Please turn on (display) Author (Contact) title at the article (content page - ex. Article, video, podcast, etc) level. NOT displayed at the card level.


update v7 useForm errors references

Updated any const { errors } = useForm(); to const { formState: { errors } } = useForm(); in the code examples.
Did not perform the update to any of the files in the v6 directory.

thank you very much 🙏


Remove racing condition in making pythainlp data directory

Rationale
To my understanding, PyThaiNLP will always create a new data directory if it does not yet exist. In the original code, the non-existence of the data directory is check before the creation of the directory is attempted. Although this might work fine in single-processing environment, but it will fail arbitrarily if this code is deployed under multi-processing environment. That is, it is possible that two processes reach line 51 (of the original code) around the same time (passing the if-check), but one of the processes will fail before the other will create the same directory before it. This is called racing condition.
Solution
This patch will solve such issue by simply use exist_ok=True flag to ignore errors when the directory already exists.
Please review.

Thanks for a thougthful contribution!
The test failed because deepcut doesnot compatible with TensorFlow 2.0 (not related to your code).
To fix this, in setup.py, change "tensorflow" to "tensorflow>=1.14.0,<2" (there are two spots) and push the code again (this has been fixed in the latest dev branch).

Coverage decreased (-0.004%) to 90.232% when pulling 5c9b0052e97b79ca13056b30f8c99fa53918bf0a on abhabongse:fix-makedir-racing-condition into 3e320e9f8c0fa1a6cbc756df75b123b9c72045e0 on PyThaiNLP:dev.


Add Memo

Our memos are represented as a String.  This suggestion is to create a simple Memo(String) class to enforce invariants (e.g. memo length, legal characters).  This helps app and UI development, as invalid memos can be caught sooner.  This also prevents transposition of values, given that some of our APIs have multiple strings being passed in.

Closing as done.


kubectl autoscale: Add --mem-percent option

What would you like to be added:
CPU usage is one of major keys for HPA, and we can specify the target average CPU utilization as --cpu-percent option.
Memory usage also is one of them, and it is useful to support specifying the target utilization as --mem-percent option.
Why is this needed:
We can create the HPA instance of the target memory utilization from YAML file now.
However it is useful to create the instance with kubectl command line because we don't need to maintain the YAML file on user side.

/close
autoscale is an imperative command, and at present the direction of kubectl is to provide imperative commands as learning tools for users, which necessitates a reduction in the number of required flags to facilitate that learning by not over whelming the user. For more advanced use cases (such as described in this issue) we would advise that users should be using the declarative tools to accomplish these tasks.

@mpuckett159
Thank you for your comment.
Could you help me understand current direction of kubectl?

at present the direction of kubectl

Do you have any links about the above direction like KEPs, issues or slack discussion?

the declarative tools

What kind of tools is impled here? kustomize or something?

Declarative would be the apply command, which can be used in conjunction with kustomize yes.
Here are some sources for you regarding this direction and the differences between the tools:

Kubernetes Object Management - this has a few more links within it for further details
Presentation by SIG lead Katrina Verey from KubeCon NA 2022 - In this video (starting from the 8:08 timestamp) Katrina explains more about the differences between imperative vs declarative object management, the how and why, and she also explains some of the reasoning behind why production systems should not be used for production management. She also details why we highly recommend the use of apply and declarative object management instead of imperative object management. This presentation has been given a few times I believe, this is the most recent I'm aware of.

@mpuckett159 Thank you so much for your advice!
That helps me much :-)


Authz request object example in

The example of Authz request in https://docs.italia.it/italia/spid/spid-cie-oidc-docs/it/versione-corrente/esempi.html#en-6-authorization-request contains a payload non entirely compliant to the specifications.
at this moment the decode payload is the one below
"{\r\n \"iss\": \"s6BhdRkqt3\",\r\n \"aud\": \"https://server.example.com\",\r\n \"response_type\": \"code id_token\",\r\n \"client_id\": \"s6BhdRkqt3\",\r\n \"redirect_uri\": \"https://client.example.org/cb\",\r\n \"scope\": \"openid\",\r\n \"state\": \"af0ifjsldkj\",\r\n \"nonce\": \"n-0S6_WzA2Mj\",\r\n \"max_age\": 86400,\r\n \"claims\": \r\n  {\r\n   \"userinfo\": \r\n    {\r\n     \"given_name\": {\"essential\": true},\r\n     "

Resolved here https://github.com/italia/spid-cie-oidc-docs/pull/183/commits/bc3c804b31a02c14b343ec6df1d59c9fc9f6bbec


Add a new investor to the whitelist automatically when kyc is disabled.

If KYC is disabled, you can set the KYC status to the default value. If you set KYC status kyc_verified, investors will be added to the whitelist. See #119.


34 js read directory [Ready for Review]

Fixes #34
What was changed?

Implemented a JavaScript function to:
Read all files in a specified directory.
Filter files to process only those with a .tbe extension.
Parse and extract metadata from TBE files.
Return the metadata as a JSON object.

Why was it changed?
This functionality fulfills the requirement in Issue #34, enabling users to retrieve metadata without processing the entire file content, which improves efficiency for metadata-focused operations.
How was it changed?

Created a javascript_tbe directory with a structured sub-directory utils for modular utility functions.
Added code for file reading, filtering, and metadata parsing within their respective files.
Tested the implementation by running the entry point script read_TBE.js using Node.js, which successfully processed sample TBE files and returned expected metadata results.

@DeviPriya-Kolla update, and conflict resolved


gnome.yelp-xsl: 40.2 -> 41.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for gnome.yelp-xsl is: "Yelp's universal stylesheets for Mallard and DocBook"
meta.homepage for gnome.yelp-xsl is: "https://wiki.gnome.org/Apps/Yelp"
meta.changelog for gnome.yelp-xsl is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 41.0 with grep in /nix/store/7wxiqsjy9qh9ks5ndplls0byf5rvsm97-yelp-xsl-41.0

found 41.0 in filename of file in /nix/store/7wxiqsjy9qh9ks5ndplls0byf5rvsm97-yelp-xsl-41.0

directory tree listing: https://gist.github.com/597abf9d8ba1098964d624d73b53d664

du listing: https://gist.github.com/5a6c7b41629869500551a5b91c367386

Rebuild report (if merged into master) (click to expand)

482 total rebuild path(s)

198 package rebuild(s)

198 x86_64-linux rebuild(s)
131 i686-linux rebuild(s)
25 x86_64-darwin rebuild(s)
128 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
Sylk
adapta-gtk-theme
ajour
altair
anki-bin
anytype
appimage-run
appimage-run-tests
apple-music-electron
betterdiscord-installer
bitwig-studio1
bitwig-studio2
bluefish
chrome-gnome-shell
chrysalis
cinnamon.cinnamon-common
cinnamon.cinnamon-gsettings-overrides
cinnamon.cinnamon-screensaver
cinnamon.muffin
cinnamon.xviewer
crypto-org-wallet
denemo
devdocs-desktop
electron-mail
electronplayer
empathy
evince
flexoptix-app
font-manager
glabels
glom
gnome-user-docs
gnome.aisleriot
gnome.anjuta
gnome.empathy
gnome.evince
gnome.gnome-applets
gnome.gnome-books
gnome.gnome-control-center
gnome.gnome-documents
gnome.gnome-session
gnome.gnome-shell
gnome.gnome-terminal
gnome.gnome-tweak-tool
gnome.gnome-tweaks
gnome.gnome-user-docs
gnome.gnome_control_center
gnome.gnome_session
gnome.gnome_shell
gnome.gnome_terminal

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/7wxiqsjy9qh9ks5ndplls0byf5rvsm97-yelp-xsl-41.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A gnome.yelp-xsl https://github.com/r-ryantm/nixpkgs/archive/a2b96bbb738122945a63ffddea18829db62a96aa.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/7wxiqsjy9qh9ks5ndplls0byf5rvsm97-yelp-xsl-41.0
ls -la /nix/store/7wxiqsjy9qh9ks5ndplls0byf5rvsm97-yelp-xsl-41.0/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings
cc @hedning @jtojnar @dasj19 @maxeaubrey for testing.

Needs to wait for GNOME 41 packaging starts.


Is that possible to know the line number of element?

Hi, I am using your project as XML parser.  And I also want do some mark up on my UI.
It will be easier if I know the line number. But I didn't find any possible method.
Is that possible to know the line line number of specific XMLIndexer I am dealing with?

Are you asking if it is possible to know the line number if you were speaking of text?
For example:
<xml>
  <contents>
    <content>First</content>
    <content>Second</content>
  </contents>
</xml>

And then the element with the text of Second would have a line number of 4?
That information isn't provided by the library today; however, it looks like NSXMLParser has a lineNumber property so it is possible that this functionality could be added. It'd be difficult to know which elements would need to expose this, though.
For example, attributes can be on their own line. Even textual content can be on its own line, but today it is only exposed as a string property.

Thanks, understand your concern. I didn't play with NSXMLParser, so not sure how it behaves. The lineNumber and columnNumber are just NSInteger. If that will be added on your XMLElement, it's more logical to expose it as a range.
For example: the line range of content is (2, 5)
<xml>
  <contents>
    <content>First</content>
    <content>Second</content>
  </contents>
</xml>

Actually, I am trying to mark up some inline attributes and leaf attributes on my UI with the original XML, so 'columnNumber' is also a good thing.
For example: mark the minimumLineSpacing="10" and <size key="itemSize" width="50" height="50"/>
<collectionViewFlowLayout key="collectionViewLayout" minimumLineSpacing="10" minimumInteritemSpacing="10" id="Rhn-xg-EmY">
                                    <size key="itemSize" width="50" height="50"/>
                                    <size key="headerReferenceSize" width="0.0" height="0.0"/>
                                    <size key="footerReferenceSize" width="0.0" height="0.0"/>
                                    <inset key="sectionInset" minX="0.0" minY="0.0" maxX="0.0" maxY="0.0"/>
                                </collectionViewFlowLayout>

Yeah, great point having it be modeled as a range - that's spot on. I'm certainly open the possibility of adding it at some point. I'll tag this as a feature request.


Support Baichuan models

Thanks for sharing Repo.
Can you please provide model file to support Baichuan models.

Hi, support for models beyond Llama will be available later.


feat: migrate KYCWall class component to functional

Details

Migration of KYCWall class component to functional one
Fixed Issues

$ https://github.com/Expensify/App/issues/16170
PROPOSAL:
Tests

[ ] log into the app
[ ] press the FAB and request money from e.g. workspace that you are a part of (to make it easier and not have to jump between accounts)
[ ] select the said workspace and press on pay with Expensify
[ ] confirm that the pop up shows in which you can select an option and be navigated to it
[ ] Verify that no errors appear in the JS console

Offline tests

QA Steps

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android / native
[x] Android / Chrome
[x] iOS / native
[x] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome

Mobile Web - Safari

Desktop

iOS

Android

LGTM
The only thing im wondering about is the PaymentMethods.kycWallRef
It seems really hacky together with the continueAction method and its the only place where it is being set. However it would probably require refactoring other components too so I think there is no point in changing it here

@abdulrahuman5196 hey, apologies for missing that part, added the steps now

Reviewer Checklist

[x] I have verified the author checklist is complete (all boxes are checked off).
[x] I verified the correct issue is linked in the ### Fixed Issues section above
[x] I verified testing steps are clear and they cover the changes made in this PR

[x] I verified the steps for local testing are in the Tests section
[x] I verified the steps for Staging and/or Production testing are in the QA steps section
[x] I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)

[x] I checked that screenshots or videos are included for tests on all platforms
[ ] I included screenshots or videos for tests on all platforms
[x] I verified tests pass on all platforms & I tested again on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[ ] If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
[x] I verified proper code patterns were followed (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method
[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I verified that this PR follows the guidelines as stated in the Review Guidelines
[x] I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[ ] I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

https://github.com/Expensify/App/assets/46707890/3ab667e1-214a-42fd-b490-db9cb7436fc6

Mobile Web - Chrome

https://github.com/Expensify/App/assets/46707890/53f28190-aa1d-4c02-8f02-280ded4426af

Mobile Web - Safari

https://github.com/Expensify/App/assets/46707890/a49a00d7-6889-438c-81a7-73ec870272a7

Desktop

https://github.com/Expensify/App/assets/46707890/00d12dd1-6d95-4d55-8fe0-59cf6c0bb2c1

iOS

https://github.com/Expensify/App/assets/46707890/04491a08-430c-41dc-9fb3-06f2560b9ef3

Android

https://github.com/Expensify/App/assets/46707890/233281b7-7804-40e3-b952-d90806a6cd19

@bondydaa Kindly run the workflows before review/merging.

@abdulrahuman5196 @bondydaa
hey, just wanted to ask, if there is any blocker with this PR getting merged?

looks like conflicts

@bondydaa
well just my luck ;) for the 5 days since C+ approved the changes there were no conflicts, either way the conflicts are fixed now, please have a look when you get a chance, thanks!

@abdulrahuman5196 @bondydaa
hey again, could we get this over the line, please? I've been fixing conflicts on this for the past three days as a new one comes up each day, the pr has been approved by C+ over a week ago. Thanks!

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging by https://github.com/bondydaa in version: 1.3.88-1 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

this started causing a crash so we reverted it here https://github.com/Expensify/App/pull/30110

The bigger reason for the revert is we have a conference starting sunday which we've been testing specifically the flows of https://github.com/Expensify/App/pull/29573 to ensure they work for that event.
I should have held this PR initially but wasn't thinking when I merged it.

🚀 Deployed to staging by https://github.com/bondydaa in version: 1.3.88-1 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

🚀 Deployed to production by https://github.com/chiragsalian in version: 1.3.88-11 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅


textureLod conversion seems off

float4 depth_ao = tex2Dlod(a_thing, float4(tex, 0.0f, 0.0f)); gets converted to vec4 depth_ao = textureLod(Ssao, vec4(xsv_TEXCOORD0, 0.0f, 0.0f));
The translation doesn't validate:
ERROR: 0:23: 'textureLod' : no matching overloaded function found
ERROR: 0:23: '=' :  cannot convert from 'const float' to 'temp 4-component vector of float'
ERROR: 0:23: '' :  syntax error
ERROR: 3 compilation errors.  No code generated.

Thanks for the hint, I haven't tested the compiler much with Shader Model 3 shaders.
So this is erroneously ignored in the converter so far.

Fixed with 2aa19f1
Currently, the conversion for these intrinsics always creates a temporary varibale, but maybe I can enhance the output when the temporary variable is avoidable.


Give users the option of disabling auto-frame-track

In this PR we are including an option to disable auto frame track in
user settings (by default true) as it was discussed in
go/stadia-orbit-auto-frame-track-alignment. The option is related
about Orbit adding automatically the default Frame Track at the start
of the session.
The logic is a bit more complex as it seems.

The option is true by default.
Setting the flag to false won't erase the already added default frame
track. It will apply for the next session.
Setting the flag to true will add the frame track during the next
LoadAllSymbols call. This will be modified in a future commit, I
think we should add it as soon as possible (by actually triggering
AddDefaultFrameTrack method) but only once (so users are able to
erase it manually: TODO tracked in http://b/239675491), but I'm open
to suggestions if you think that we should only add it in the next
session to be consistent.

http://screenshot/BYU8NuWCCr3h93u
Bug: http://b/239148938
Test: Started/ended sessions flipping the option and took several
captures. Modified E2E tests (https://github.com/google/orbit/pull/3998)
and run them manually. In progress: Creating a signed build and triggering
E2E tests.

I didn't look at the code in detail, but one comment regarding the screenshot: It could be confusing to users to have this option that refers to the whole session amidst the other options that refer to the next capture. Especially since the text next to the "OK" Button states that the options will come into effect with the next capture, and I don't think the tooltip helps much.
At the very least I'd make the tooltip more explicit that this is a session-wide setting. The way I read the tooltip is that if I enable this setting at the beginning of a session, a frametrack will be added. But I think it's only added if the setting has already been active at the beginning of a session, i.e. before symbol loading, is that correct? That feels confusing.
Did you consider adding the option in a separate place in the options Window to make this clearer? You could e.g. add a toolbox or tab widget with 2 pages, one for the "Next Capture" options and one for the "Next Session" options, and add the text explaining when these settings come into effect within those pages instead of next to the "OK" button.

At the very least I'd make the tooltip more explicit that this is a session-wide setting. The way I read the tooltip is that if I enable this setting at the beginning of a session, a frametrack will be added. But I think it's only added if the setting has already been active at the beginning of a session, i.e. before symbol loading, is that correct? That feels confusing.

A few things. The message and the tooltip was agreed with Ronald before the creation of this PR. The motivation was making it clear that if the user erase the automatic FrameTrack manually, it won't appear in the following captures, so basically that it will be added only once during a session (which is currently a TODO tracked in http://b/239675491).

The way I read the tooltip is that if I enable this setting at the beginning of a session, a frametrack will be added. But I think it's only added if the setting has already been active at the beginning of a session, i.e. before symbol loading, is that correct? That feels confusing.

No, is what you got from the tooltip. The already added FrameTrack won't be erased (because the complexity related to erasing a FrameTrack) but if the user is choosing the option to be true again, I will add it as soon as possible, because I think it makes sense (TODO).

Did you consider adding the option in a separate place in the options Window to make this clearer? You could e.g. add a toolbox or tab widget with 2 pages, one for the "Next Capture" options and one for the "Next Session" options, and add the text explaining when these settings come into effect within those pages instead of next to the "OK" button.

No, I found it very complex. We are just speaking about a border case-I don't expect the user to select/deselect this option very often. I will simply clarify using a better message making it as clear as possible the current behavior. My plan, discussed with Ronald, is to insert it ASAP but not remove the already added FrameTrack. . And also only add it once (at the start of the session OR when the user enable it). I could add "the Frame Track won't be removed for this session", what do you think?

but if the user is choosing the option to be true again, I will add it as soon as possible

Ok, that makes sense. Then I find the option less confusing than I initially thought. Thanks for clarifying, then I'm fine with those changes. Maybe as you suggested add something like "Disabling this option will not remove an existing frame track for the current session" or so.

Addressed Ronald's comment and added "Disabling this option will not remove any existing frame track for the current session" here.
Henning gave me ok in merging it without his approval, so I will merge this in the next hour.


yarn error: Incorrect peer dependency<EMAIL_ADDRESS>

I think peerDependency should be removed or updated to match any React Native.

isn't > 30 enough? or which native version is currently used?

Closing this until feedback is provided


Usecase2 - Infrastructure provisioning with ImageStreamer

Description
Usecase2 - Infrastructure provisioning with ImageStreamer
Issues Resolved
NA
Check List

[ ] New functionality includes testing.

[x] All tests pass ($ rake test).

[ ] New functionality has been documented in the README if applicable.

[ ] New functionality has been thoroughly documented in the examples (please include helpful comments).

[ ] Changes are documented in the CHANGELOG.

Below provided is the usecase:
UseCase - 2
Infrastructure provisioning with OS on Synergy with Image Streamer
•	Create server profile template with:
o	Boot settings
o	Deployment network
o	Management network - 6
o	OS deployment plan - 1
•	Create server profile with OS deployment plan options filled in and assign to hardware
•	Power on server (image streamer deploys OS)


Preview Images Are Obscured by the Menu

After the ComfyUI update, the preview images of the Image Feed feature are obscured by the menu.
Old：

New：

Bugs occur for versions 0.2.3 and above

fixed thanks to @DrJKL
https://github.com/pythongosssss/ComfyUI-Custom-Scripts/pull/368


Permit Simple and GLSL450 memory model in WEBGPU_0

Now that WebGPU ingests WGSL instead of SPIR-V,
there is no need to be so strict about the memory model.
Allow any memory model that is already allowed by Vulkan 1.0,
either directly or via an existing.

Rebasing to see if the smoketest bot works.


Rename 'pagelib_theme_div_do_not_apply_baseline' to 'notheme'

https://phabricator.wikimedia.org/T229286
classifyElements is by far the slowest part of preparing mobile-html. It's not sustainable to keep adding exclusions to the list of specific filenames and query selectors.
I'm proposing we change pagelib_theme_div_do_not_apply_baseline pagelib_theme_div_do_not_apply_baseline to notheme and request editors add it to the appropriate templates ahead of the launch of mobile-html. This will allow the mobile-html endpoint to stop calling classifyElements. There is some precedent for this with nomobile (now obviated by TemplateStyles) but will need to confirm that this is a viable option.

Does it still need to be confirmed as a viable solution? Waiting on that to merge this PR and https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/544854

@thesocialdev It still needs to be confirmed that this is a viable solution and there's not a better option. Adding the hold tag for now. https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/544854 can be merged independently.


Pump Chunk Marking

For each resampled (chunked) pump csv file, did you only mark 1 chunk as True?
E.g. if there is a pump at 2019-03-1 17.00 and I chunked my csv data into 5 second chunks (and only taking into consideration the pump day and 1 day before and after), I only marked the chunk from 17.00.00 to 17.00.05 as True.
This leaves me with an extremely imbalanced dataset so that a RandomForrestClassifier ends up predicting every chunk as False.
What am I missing here?
------- Offtopic -----------
Also thank you guys for your effort to collect all the data. I enjoyed reading your paper too and got lots of useful information out of it. It's a welcome distraction to fiddle around with your data during all the restrictions :)

Hello, @fleerdayo :)
I've been trying to reverse engineer the paper model for the last two weeks:)
I've been able to achieve 77.907 % recall but ridiculously low 0.185 % precision:(
I use imblearn.ensemble.BalancedRandomForestClassifier to undersample the data.
I tried to cut off 30 minutes after each pump chunk because the paper says that "...Once a pump is detected we pause our classifier for 30 minutes to avoid multiple alerts for the same event..."
However it does not help.
So I believe that the data should be filtered before training. The paper says that the authors picked only 104 samples out of 175.
Maybe this is the main reason of so many false positives?
Let me know if you're still interested. We could collaborate:) I see that the authors are not responding here:) Maybe they are too busy...or too rich:) Just kidding:)
Btw I'm ready to share my code and collaborate with whoever is interested including the authors:)

Hi @RaibekTussupbekov
An up-to-date and bigger dataset might help with that.
I'm up for collaboration, please send me an email.

Hey @RaibekTussupbekov , did you mange to make this work ? I also encounter many issues with the dataset.
Thanks !

@RazcoDev Not yet


Getting error : Error registering device for DPS : Resolved after performing few additional steps in IoT central Application which seem to be currently missing in the document

Although the document specifies : "Complete the Create an Azure IoT Central application quickstart to create an IoT Central application using the Custom app > Custom application template." ,
it is not evident if we need to go to the next step of creating a 'simulated device' or not.
I initially only created an application and did not add 'simulated device'. After this I was getting error specified in point#4 under 'Build the code'. Even after several retries I was still getting the same error.
To fix this, I added a device in my IOTC application, with device id mxchip-001 and then clicked Approved. This fixed the issue.
Request you to please look into this and update the documentation if needed.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 9319a8d0-18e8-f56f-2b08-3b1657243c3d
Version Independent ID: bcac81e9-1437-5ef4-9ecd-aab7f7c1c260
Content: Tutorial - Connect an IoT Plug and Play (preview) device to Azure IoT Central
Content Source: articles/iot-central/core/tutorial-connect-pnp-device.md
Service: iot-central
GitHub Login: @dominicbetts
Microsoft Alias: dobett

@isagrwl Thanks for the feedback! We are currently investigating and will update you shortly.

Hi @isagrwl - thanks for getting in touch. You shouldn't need to add a device template to complete this tutorial. I just stepped through it and although the first time I ran the device code I saw the DPS error, the second time I ran it (as per the note in the article), it worked as expected. Can you confirm that in your IoT Central application Auto approve is enabled on the Administration > Device connection page?
For reference, here's what I saw when I ran the client code (some parameters redacted):
c:\temp\central-app\devkit_device\cmake>.\Release\devkit_device.exe mxchip-001 -myscopeid- -myprimarykey-
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_prov_transport_process_json_reply Line:658 Provisioning Failure: OperationId: 4.22d9be6e95a09645.5f6b2530-1db4-4135-b3c7-51a52cb631d8 - Date: 2020-02-20T14:27:58.0656187Z - Msg: Custom allocation failed with status code: 404
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_transport_mqtt_common.c Func:_prov_transport_common_mqtt_dowork Line:1014 Unable to process registration reply.
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_on_transport_registration_data Line:770 Failure retrieving data from the provisioning service
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\central-app\devkit_device\main.c Func:_provisioningRegisterCallback Line:82 DPS Provisioning callback called with error state 7
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\central-app\devkit_device\main.c Func:_registerDevice Line:165 Error registering device for DPS

c:\temp\central-app\devkit_device\cmake>.\Release\devkit_device.exe mxchip-001 -myscopeid- -myprimarykey-
Info: Provisioning callback indicates success.  iothubUri=iotc-........7.azure-devices.net, deviceId=mxchip-001
Info: DPS successfully registered.  Continuing on to creation of IoTHub device client handle.
Info: Successfully created DigitalTwin device with connectionString=<****>, deviceHandle=<00FA0B08>
Info: DigitalTwin Interface : Changing interface state on interface deviceinfo from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: DEVICEINFO_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=<urn:azureiot:DeviceManagement:DeviceInformation:1>, interfaceInstanceName=<deviceinfo>, handle=<00FA5800>
Info: DigitalTwin Interface : Changing interface state on interface sensors from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SENSORS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=<urn:mxchip:built_in_sensors:1>, interfaceInstanceName=<sensors>, handle=<00FA5A80>
Info: DigitalTwin Interface : Changing interface state on interface leds from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: LEDS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=<urn:mxchip:built_in_leds:1>, interfaceInstanceName=<leds>, handle=<00FA5350>
Info: DigitalTwin Interface : Changing interface state on interface screen from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SCREEN_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=<urn:mxchip:screen:1>, interfaceInstanceName=<screen>, handle=<00FA5710>
Info: DigitalTwin Interface : Changing interface state on interface settings from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SETTINGS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=<urn:mxchip:settings:1>, interfaceInstanceName=<settings>, handle=<00FA5670>
Info: DigitalTwin Interface : Changing interface state on interface deviceinfo from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE
Info: DigitalTwin Interface : Changing interface state on interface sensors from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE
Info: DigitalTwin Interface : Changing interface state on interface leds from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE

Thanks Dominic for trying it out. It’s good to know that the setup is working for you. Yes, I do have Auto approve selected by default. I tried to follow the document as is but still facing the issue after multiple tries.
Few other observations:
The error I receive is different from yours:
E:\CSS\SDKs\central_app\devkit_device\cmake>.\Release\devkit_device.exe mxchip-001 --scope id-- --primary key--
Error: Time:Thu Feb 20 11:49:28 2020 File:E:\CSS\SDKs\vcpkg\vcpkg_pnp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_prov_transport_process_json_reply Line:670 Unsuccessful json encountered: {"errorCode":401000,"trackingId":"42a2c6b6-75b3-4ba8-9a34-09c345d0f2bb","message":"Unauthorized","timestampUtc":"2020-02-20T19:49:28.2529747Z"}
Error: Time:Thu Feb 20 11:49:28 2020 File:E:\CSS\SDKs\vcpkg\vcpkg_pnp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_transport_mqtt_common.c Func:_prov_transport_common_mqtt_dowork Line:1014 Unable to process registration reply.
Error: Time:Thu Feb 20 11:49:28 2020 File:E:\CSS\SDKs\vcpkg\vcpkg_pnp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_on_transport_registration_data Line:770 Failure retrieving data from the provisioning service
Error: Time:Thu Feb 20 11:49:28 2020 File:E:\CSS\SDKs\central_app\devkit_device\main.c Func:_provisioningRegisterCallback Line:82 DPS Provisioning callback called with error state 7
Error: Time:Thu Feb 20 11:49:28 2020 File:E:\CSS\SDKs\central_app\devkit_device\main.c Func:_registerDevice Line:165 Error registering device for DPS
Few other observations:
If I use the ‘primary key’, I get the above error but Device templates and Devices section in IOTC is still blank.
I replaced ‘primary key’ with ‘device key’, I do get the same error as above but Devices and Device templates now show the device and template. But I see that the device is not Simulated and am not getting any data.
I got the ‘Primary key’ after clicking the link ‘View Keys’ on the Device connection page.
I selected ‘CMake Project on Windows’ as the project type instead of ‘MXChip IoT DevKit Project’ as suggested in the documentation.
The documentation seems pretty straightforward but somehow I am missing something.
Thanks,
Ishu
From: Dominic Betts<EMAIL_ADDRESS>Sent: Thursday, February 20, 2020 8:12 AM
To: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Cc: Ishu Agarwal<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/azure-docs] Getting error : Error registering device for DPS : Resolved after performing few additional steps in IoT central Application which seem to be currently missing in the document (#48511)
Hi @isagrwlhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fisagrwl&data=02|01|Ishu.Agarwal%40microsoft.com|523e7473cfb14a34fa3208d7b61f9f31|72f988bf86f141af91ab2d7cd011db47|1|0|637178119238177102&sdata=aXEtFnhBTBYfRRonegrAaZmTg%2BUcxstKjZy8nLpTKz8%3D&reserved=0 - thanks for getting in touch. You shouldn't need to add a device template to complete this tutorial. I just stepped through it and although the first time I ran the device code I saw the DPS error, the second time I ran it (as per the note in the article), it worked as expected. Can you confirm that in your IoT Central application Auto approve is enabled on the Administration > Device connection page?
For reference, here's what I saw when I ran the client code (some parameters redacted):
c:\temp\central-app\devkit_device\cmake>.\Release\devkit_device.exe mxchip-001 -myscopeid- -myprimarykey-
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_prov_transport_process_json_reply Line:658 Provisioning Failure: OperationId: 4.22d9be6e95a09645.5f6b2530-1db4-4135-b3c7-51a52cb631d8 - Date: 2020-02-20T14:27:58.0656187Z - Msg: Custom allocation failed with status code: 404
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_transport_mqtt_common.c Func:_prov_transport_common_mqtt_dowork Line:1014 Unable to process registration reply.
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\vcpkg\buildtrees\azure-iot-sdk-c\src\8ff96160cc-bd25b8f0be\provisioning_client\src\prov_device_ll_client.c Func:_on_transport_registration_data Line:770 Failure retrieving data from the provisioning service
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\central-app\devkit_device\main.c Func:_provisioningRegisterCallback Line:82 DPS Provisioning callback called with error state 7
Error: Time:Thu Feb 20 14:28:01 2020 File:C:\temp\central-app\devkit_device\main.c Func:_registerDevice Line:165 Error registering device for DPS
c:\temp\central-app\devkit_device\cmake>.\Release\devkit_device.exe mxchip-001 -myscopeid- -myprimarykey-
Info: Provisioning callback indicates success.  iothubUri=iotc-........7.azure-devices.net, deviceId=mxchip-001
Info: DPS successfully registered.  Continuing on to creation of IoTHub device client handle.
Info: Successfully created DigitalTwin device with connectionString=<****>, deviceHandle=<00FA0B08>
Info: DigitalTwin Interface : Changing interface state on interface deviceinfo from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: DEVICEINFO_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=urn:azureiot:1:DeviceManagement:DeviceInformation, interfaceInstanceName=, handle=<00FA5800>
Info: DigitalTwin Interface : Changing interface state on interface sensors from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SENSORS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=urn:mxchip:1:built_in_sensors, interfaceInstanceName=, handle=<00FA5A80>
Info: DigitalTwin Interface : Changing interface state on interface leds from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: LEDS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=urn:mxchip:1:built_in_leds, interfaceInstanceName=, handle=<00FA5350>
Info: DigitalTwin Interface : Changing interface state on interface screen from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SCREEN_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=urn:mxchip:1:screen, interfaceInstanceName=, handle=<00FA5710>
Info: DigitalTwin Interface : Changing interface state on interface settings from DT_INTERFACE_STATE_INVALID to DT_INTERFACE_STATE_CREATED
Info: SETTINGS_INTERFACE: Created DIGITALTWIN_INTERFACE_CLIENT_HANDLE successfully for interfaceId=urn:mxchip:1:settings, interfaceInstanceName=, handle=<00FA5670>
Info: DigitalTwin Interface : Changing interface state on interface deviceinfo from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE
Info: DigitalTwin Interface : Changing interface state on interface sensors from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE
Info: DigitalTwin Interface : Changing interface state on interface leds from DT_INTERFACE_STATE_CREATED to DT_INTERFACE_STATE_BOUND_TO_CLIENT_HANDLE
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F48511%3Femail_source%3Dnotifications%26email_token%3DAOL3I6HK6CBS3EXMCE44ECDRD2TVFA5CNFSM4KYDQWX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMO5ZVY%23issuecomment-589159639&data=02|01|Ishu.Agarwal%40microsoft.com|523e7473cfb14a34fa3208d7b61f9f31|72f988bf86f141af91ab2d7cd011db47|1|0|637178119238187101&sdata=J%2FegUDumcuUw7lakiZpbwkDfx%2FM3rYTm7pnuYOhmYpc%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOL3I6CTWZB3K3A3O2WXCUDRD2TVFANCNFSM4KYDQWXQ&data=02|01|Ishu.Agarwal%40microsoft.com|523e7473cfb14a34fa3208d7b61f9f31|72f988bf86f141af91ab2d7cd011db47|1|0|637178119238187101&sdata=Ligv3rBooII6Vt6%2B6aRwDlX44zlT0GRXtNWjBBCI2Mc%3D&reserved=0.

Hi @isagrwl - I'm investigating further as to why you're seeing different results - I'll be back with an update shortly. Thanks for your patience.

Thanks again for your patience.
At present, IoT Central applications created using the Custom application template do not support IoT Plug and Play devices - IoT Plug and Play is currently in public preview. For the time being, if you want to connect a PnP device to your IoT Central application you should contact technical support and request that PnP support is enabled in your application (follow the Technical support link from the help menu in your application). I hadn't realized that support for PnP had been enabled in the application I was using to test.
If you want to try out connecting a non-PnP client to your IoT Central app, we have a new tutorial: https://docs.microsoft.com/azure/iot-central/core/tutorial-connect-device
I hope that explains everything! I'll close this issue now, but feel free to reopen it if you have any follow-up questions.
#please-close

Thanks @donimicbetts, that does explain why it doesn't work.
But shouldn't that be mentioned as part of the document?
The document asks to create a Custom application for Connecting a PnP device but in reality Custom applications are not supporting PnP devices. So, nobody will be able to get this working.
I have spent hours to get this working and there will be other Customers trying to use PnP and will face the same issue.
The only limitation the document states is that of the Region (Will be good to have a link here to : https://docs.microsoft.com/en-us/azure/iot-pnp/overview-iot-plug-and-play#regional-availability ).
Also, how will one ensure that the IOTC application gets created in one of the supported Regions as it just provides United States while creating the application. May be if I am able to create the application in the supported region, it may work for me.

Hi @isagrwl - thanks for the comments. We added a note the article last week explaining that it's currently necessary to contact technical support to enable PnP functionality in an IoT Central application. I'll update the article (and leave a comment here) as soon as this restriction is removed and you can use the PnP functionality directly. Thanks for your patience.

Hi @isagrwl - just to follow up on a couple of items. PnP should be enabled on IoT Hub in all regions now - and there's no longer a need to contact support to have PnP enabled on any new IoT Central applications you create. Thank you.


How to partition update an native statefulset in batch

We're using a quite conservative k8s strategy, and now on 1.24, without alpha features. (No MaxUnavailable field.) Most of our workloads are native StatefulSet and we cannot change it.
Now we want to make its rollout quicker with 25% pods MaxUnavailable. I write a rollout like this:
apiVersion: rollouts.kruise.io/v1alpha1
kind: Rollout
metadata:
  name: foo
  annotations:
    rollouts.kruise.io/rolling-style: partition
spec:
  objectRef:
    workloadRef:
      apiVersion: apps/v1
      kind: StatefulSet
      name: foo
  strategy:
    canary:
      steps:
      - replicas: 1
      - replicas: 25%
      - replicas: 50%
      - replicas: 75%
      - replicas: 100%

The sts: podManagementPolicy is Parallel, updateStrategy is type: RollingUpdate
Now rollout works, but during its steps, it still rollout its pods one by one like native, not in batch. I want inner step, just remove all pods in this step. Is this supported? Or do I config something wrong?
Thanks.

I'm afraid it is not supported, kruise rollout still relies on the referenced workload to manage the pod, and it make no sense to support the maxUnavailable of native statefulset below 1.24.


Feature Request: Match on any arbitrary header and value

Would love to be able to match on any header and header value
e.g.
(header x-custom-header /value/)

For what it's worth, there's a GmailMessage.getHeader function that is useful for this.

As the rules are read in a single time, and then emails are parsed, we would need to collect a set of all headers mentioned in the rules. Then when emails are parsed, the dictionary of headers could be populated using GmailMessage.getHeader

I've started to work on a PR for this and #35 which are very related.

OK, I've created PR #67. Try it out!


📚 [Documentation] - Creating a CODE_OF_CONDUCT.md file [GSSoC'23]

Description ✍
I can create a CODE_OF_CONDUCT.md file to enhance the documentation of the project.
Are you contributing for a program ?
GSSoC2023
Code of Conduct

[X] I have read and agree to the Contributing Guidelines of this project.

Already present man... check here - https://github.com/amupedia2021/Project-Amupedia/blob/master/.github/CODE_OF_CONDUCT.md

Firest check the code base and all.

ok @AmanJain18 . I am so sorry. I wasted your time. Sorry for that.

No worries, check all the issue first, if not found ur issue open, then open new issue and go through the full codebase and then come up with new and nice issue.
we would welcome u. ✌🏻🚀💪🏻


New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions

I looked through the “8.1 Expressions” and searched the Visit User Manual v3.2.0 high and low
to find and answer to a very trivial question.
How can I using Expressions create a new scalar mesh variable which
is an existing mesh variable, say, density dens normalized by the
maximum of dens over the whole mesh?
Just to try to make my question clearer.
Say in FORTRAN terms is I have 3D array dens(:,:,:) I want
dens_normalized(:,:,:) = dens(:,:,:)/maxval(dens)

Here is some brief python code that should get you headed in the right direction:
OpenDatabase("/usr/gapps/visit/data/curv3d.silo")
AddPlot("Pseudocolor", "d")
DrawPlots()
SetQueryFloatFormat("%g")
res = Query("Max", use_actual_data=1)
maxval = GetQueryOutputValue()
DefineScalarExpression("maxval", "cell_constant(curvmesh3d, " + str(maxval) + ")")

What does this do? It opens a database, adds a plot of one of its variables, draws it, and then queries the max value. It then creates a new expression that paints the result of that query across the entire mesh. You can use this expression in other expressions (so you can normalize by the maximum, for example) and you can plot it. You'll want to tweak the name, and change it to nodal_constant instead of cell_constant if you want this for a nodal variable.
Let me know if you have further questions.

Wonderful!
Now, what do I do with this code?
I mean where should I put it and how do I run it?
Can it create a new scalar variable and add it to the list
of existing scalars?
I always work interactively, open databases using button
@.***
then I add desired “Pseudocolor” variable using Add button
@.***
And then I add Operators (slicing, selecting mostly) if I need.
When I needed the normalized variable I, for the first time, I went
to “Expressions” and tried to program-in the expression for the new
variable.
So having the database already opened and plot drawn,
where should I add this code so that it would run all the time I change the
time cycle or just redraw the plot.
Or can it create a new variable dens_normalized and add it to the list of
existing variables which I then can Pseudocolor select?
I see that in the “Expression” there is a tab called “Python expression editor”:
@.***
I am almost there!
Thanks,
Alex
From: Justin Privitera @.>
Sent: Monday, August 19, 2024 6:00 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
Here is some brief python code that should get you headed in the right direction:
OpenDatabase("/usr/gapps/visit/data/curv3d.silo")
AddPlot("Pseudocolor", "d")
DrawPlots()
SetQueryFloatFormat("%g")
res = Query("Max", use_actual_data=1)
maxval = GetQueryOutputValue()
DefineScalarExpression("maxval", "cell_constant(curvmesh3d, " + str(maxval) + ")")
What does this do? It opens a database, adds a plot of one of its variables, draws it, and then queries the max value. It then creates a new expression that paints the result of that query across the entire mesh. You can use this expression in other expressions (so you can normalize by the maximum, for example) and you can plot it. You'll want to tweak the name, and change it to nodal_constant instead of cell_constant if you want this for a nodal variable.
Let me know if you have further questions.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2297531057__;Iw!!CGUSO5OYRnA7CQ!aZsXWp0nnd2BZaRfRCweTCuOXlHBszKbjJi0T5GIKnB_pL4IcE57Ua5hpU8nnn0kyLI0yUprsUMKa9AxwSfb4_sPiGN5$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQTMMRRQOSXYWCKRMWTZSJTGLAVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGUZTCMBVG4__;!!CGUSO5OYRnA7CQ!aZsXWp0nnd2BZaRfRCweTCuOXlHBszKbjJi0T5GIKnB_pL4IcE57Ua5hpU8nnn0kyLI0yUprsUMKa9AxwSfb436tlMHl$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

You have a few options. You could put the code I wrote into a python script myscript.py and run with visit -s myscript.py. If you'd like to work entirely within the GUI, you could open up Controls/Command (Ctrl+Shift+C) and then paste the last few lines of my code there and hit execute. Just these lines should do the trick.
SetQueryFloatFormat("%g")
res = Query("Max", use_actual_data=1)
maxval = GetQueryOutputValue()
DefineScalarExpression("maxval", "cell_constant(curvmesh3d, " + str(maxval) + ")")

After you've hit execute, you should see a new variable called maxval appear in the appropriate menus for scalar mesh variables. You'll be able to use it in expressions. Just remember to change the mesh name in the expression to your mesh name, use nodal_constant instead of cell_constant if you want a nodal variable, and make sure you run the max query on a plot of the variable you are interested in getting the maximum of.

What we've done with this code is create an expression that has a hardcoded value. If you change the timestep, that value won't change. If you want it to change, you'll have to re-run this code. You may also have to redraw plots that use this. There is a way to create macros in the Command window; I would start there to see about making something that is easy to run repeatedly.

Thanks, Justin, for all the help!
I see.
One way I see that I relatively simple is find a way to run the script so
it would add the new, max_desn, variable to ALL time frames?
So that it would appear and behave as true variable such as dens?
I would use the script once upon opening the file and then use max_dens in Expression.
I would have to rerun it for every opened database etc.
Or is it better to figure out the way to run the script every time I change the time or press Draw button?
I am not familiar nor intended to go that deep into Visit when I started it.
I thought it would be something trivial.
Thanks,
Alex
From: Justin Privitera @.>
Sent: Monday, August 19, 2024 6:35 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
What we've done with this code is create an expression that has a hardcoded value. If you change the timestep, that value won't change. If you want it to change, you'll have to re-run this code. You may also have to redraw plots that use this. There is a way to create macros in the Command window; I would start there to see about making something that is easy to run repeatedly.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2297613955__;Iw!!CGUSO5OYRnA7CQ!eAe2fyD5NWKpltUbraYVlCFdM39ldl0ByCuwmbS3GjaiITVQGtYha6SXVvTr55os1PtXCLyyzETudiwrEr5qnqSQxF04$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQRXCC6B53TUMGND7VDZSJXHNAVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGYYTGOJVGU__;!!CGUSO5OYRnA7CQ!eAe2fyD5NWKpltUbraYVlCFdM39ldl0ByCuwmbS3GjaiITVQGtYha6SXVvTr55os1PtXCLyyzETudiwrEr5qngQBd-SX$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

There is not a simple way (that I know of) to create a max variable that changes as you change the timestep. Perhaps other @visit-dav/visit-developers might have ideas or be willing to weigh in.
We could write a script that creates a separate maximum value variable for each time step. Then you would have access to them at any time step. We could even load those values into a mesh array variable, such that the nth entry in the array corresponds to timestep n. This isn't what you asked for though.
Our issue is that the query system isn't talking to the expression language. We run the query to determine the max, which can change every timestep. But we take that maximum and make a variable for it with the expression system.
If you want it to behave like the other variables and change over time, there are some complicated options we could explore, in lieu of adding your request as a feature in an upcoming release. One such solution would be to export your data in a format like Blueprint. Then I could write you a script that would read the Blueprint files, add a variable to each timestep that contains the maximum for some other variable, and write the files back out. Then you could read that result into VisIt.

We could write a script that creates a separate maximum value variable for each time step. Then you would have access to them at any time step.
If I can have access to it in your Expression dialog window by whatever name it is,
it would work. For example if max_dens or max_elv would correspond to max over the mesh to
dens or elv AND I could use it to form an Expression such as
elv_normalized = elv/max_elv
and each time step max_elv will be just that max(elv) over the mesh.
It would work
Our issue is that the query system isn't talking to the expression language.
Yes that is an issue!
If I could use a reduction function in the Expression.
Say, if you introduce REDUCTION function that would use query machinery or any other machiner.
Call them reduce(dens,max) or reduce(dens,min) or reduce(dens,sum)
and offer them along with other math function in Expression.
It would work!
Or just introduce max_mesh(), sum_mesh() etc functions.
As I said, I thought I was asking to do something trivial. Which takes
just a single line in pretty much all modern languages,
e.g. in Matlab:
elv_normalized = elv/max(elv,’all’)
This is not worth my and your time to pursue any complicated options!
BUT I hope VISIT developers could introduce something along the lines we discussed here.
It will make the data analysis part so much more convenient and easier.
Thanks for all your time and help!
From: Justin Privitera @.>
Sent: Monday, August 19, 2024 7:13 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
There is not a simple way (that I know of) to create a max variable that changes as you change the timestep. Perhaps other @visit-dav/visit-developershttps://urldefense.com/v3/__https:/github.com/orgs/visit-dav/teams/visit-developers__;!!CGUSO5OYRnA7CQ!fJZCAtCyV0fu2VEV0IRBsXJJA7mhuRRgoLPHOjA80OQLKJOQA1Qct-gBH84Es9Jj22cbWV_IOjD8G8FC23XpjdI9AmNS$ might have ideas or be willing to weigh in.
We could write a script that creates a separate maximum value variable for each time step. Then you would have access to them at any time step. We could even load those values into a mesh array variable, such that the nth entry in the array corresponds to timestep n. This isn't what you asked for though.
Our issue is that the query system isn't talking to the expression language. We run the query to determine the max, which can change every timestep. But we take that maximum and make a variable for it with the expression system.
If you want it to behave like the other variables and change over time, there are some complicated options we could explore, in lieu of adding your request as a feature in an upcoming release. One such solution would be to export your data in a format like Blueprint. Then I could write you a script that would read the Blueprint files, add a variable to each timestep that contains the maximum for some other variable, and write the files back out. Then you could read that result into VisIt.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2297673315__;Iw!!CGUSO5OYRnA7CQ!fJZCAtCyV0fu2VEV0IRBsXJJA7mhuRRgoLPHOjA80OQLKJOQA1Qct-gBH84Es9Jj22cbWV_IOjD8G8FC23XpjZq-7zUI$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQUVDAZEQKNTMSCWE6DZSJ3XDAVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJXGY3TGMZRGU__;!!CGUSO5OYRnA7CQ!fJZCAtCyV0fu2VEV0IRBsXJJA7mhuRRgoLPHOjA80OQLKJOQA1Qct-gBH84Es9Jj22cbWV_IOjD8G8FC23XpjagV_D4A$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Thanks, Justin, for the time and the effort!
Let me describe what I do currently, so we are clear if what you are proposing is
any better and worth spending time on.
The way I work with the Visit is to create a Pseudocolor plots of one or a few variables
and add slicing/selecting operators and then click time slider to go forward and backward
in time and observe and sample the evolution of the plotted quantities.
For example, in my case I simultaneously plot density (dens) and laser deposition (elv)
on one plot such as here:
@.***
The problem is that laser deposition changes in amplitude dramatically from time step to time step.
but I need to see where it is deposited and the amplitude of the modulations, regardless of the
magnitude. Hence, I wante to plot the laser deposition from say 0.9 to 1 w/r to its MAX value
at each time step to be able to see the deposition as I slide the time sider.
Currently, since I don’t have a max value of deposition available as a variable,
I, at each time step, take the max value which is shown on the color bar of the left of the plot
(see 1.354e+04 in the plot above) and use it to manually set min and max for this current time step.
When I move the slider I repeat.
Although, the process is tedious and cumbersome, it is clear, simple, straightforward.
So, any solution (such as 1. below) that will require me to special treat each time step is not
going to make my life easier and more comfortable, thus should not be considered as
a solution. At any rate it is worse “solution” than the process above.
If 2. below allows me to use this array of MAX values of elv in an Expression
construct such that it will automatically do what I have to do now manually,
I would consider it as a solution and will happily use it.
But if it still requires me to special treat it each time step, it is not worth it.
A real “fix” would be to extend the Expressions to access Query or
add REDUCE(elv,MAX) or REDUC(dens,MIN) functions to Expressions.
Some such things.
From: Justin Privitera @.>
Sent: Tuesday, August 20, 2024 12:10 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
We could write a script that creates a separate maximum value variable for each time step. Then you would have access to them at any time step.
I think your options would look like this:

If you have N timesteps, you get N variables, each called max_for_timestep_X where X is in the range 0 to N - 1. If you want to use any of those, you'll have to specify the one you want for each timestep. This isn't ideal.
We could load all of those variables into an array variable, such that you would get at the max for a given timestep by specifying the timestep as an array index: max_for_timestep[X] where X is in the range 0 to N - 1.

Neither of these options are great but they can get you mostly to where you are trying to go, as long as you are willing to put up with the idiosyncrasies. Since you opened a ticket, we will discuss as a team your feature request and determine where it should fit into our release schedule.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2299228126__;Iw!!CGUSO5OYRnA7CQ!YCj10AAW_evlWYIp0WAHH1tbBOJmqu4lEafsDTATprr_RIBauwJOdIldXYY1tv2Pil3bgRamvJWgDcg6szZekaet_nSI$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQUWKTJUO3EL6BN5DC3ZSNS6VAVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJZGIZDQMJSGY__;!!CGUSO5OYRnA7CQ!YCj10AAW_evlWYIp0WAHH1tbBOJmqu4lEafsDTATprr_RIBauwJOdIldXYY1tv2Pil3bgRamvJWgDcg6szZekbm5YA5D$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

We'll plan to add new expressions to our expression language for max, min, standard deviation, basically the same menu of expressions as for data binning. These expressions will need new names to differentiate them from what is there. They will broadcast the result across the entire mesh, so special_max(d) will give you a scalar mesh variable that is constant across the mesh that records the maximum value of d.
We should also make it so that max(), min(), and other associated functions can take a single argument without crashing the compute engine.

Sounds like a good plan!
What would be the meaning of max() and min() with a single argument?
Now if you, max(velx, vely), returns a scalar mesh variable which in a zone/node
is the max of velx and vely in this zone/node.
Similarly max(velx,1) is the max between velx and 1.
But what will max(velx) return?
From: Justin Privitera @.>
Sent: Tuesday, August 20, 2024 6:52 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
We'll plan to add new expressions to our expression language for max, min, standard deviation, basically the same menu of expressions as for data binning. These expressions will need new names to differentiate them from what is there. They will broadcast the result across the entire mesh, so special_max(d) will give you a scalar mesh variable that is constant across the mesh that records the maximum value of d.
We should also make it so that max(), min(), and other associated functions can take a single argument without crashing the compute engine.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2299892730__;Iw!!CGUSO5OYRnA7CQ!ZT71YHGKNxSyIDBzzB0XkIcaIuH_8YHi0Vyk1HCFIFDKXzl2v4FjusWypKTRWtjJJFiUpEAHiC-03ik_Yv5NsArh2vbA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQX3FAB26TNNFGEX4NTZSPCA7AVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJZHA4TENZTGA__;!!CGUSO5OYRnA7CQ!ZT71YHGKNxSyIDBzzB0XkIcaIuH_8YHi0Vyk1HCFIFDKXzl2v4FjusWypKTRWtjJJFiUpEAHiC-03ik_Yv5NsODiIg4P$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

What would be the meaning of max() and min() with a single argument?

max(d) for some scalar mesh variable d would evaluate to d. max() ought to take any number of variables and evaluate to the maximum for each node or zone across all variables that were provided. If a single variable is provided, then the maximum is simply that variable. This is a better solution than crashing the compute engine. It also keeps the function consistent.

Ok. It makes sense.
From: Justin Privitera @.>
Sent: Tuesday, August 20, 2024 7:07 PM
To: visit-dav/visit @.>
Cc: Shvydky, Alexander @.>; Author @.>
Subject: [EXT] Re: [visit-dav/visit] New scalar mesh variable = existing scalar mesh variable divided by the maximum of of it over the whole mesh in Expressions (Issue #19747)
What would be the meaning of max() and min() with a single argument?
max(d) for some scalar mesh variable d would evaluate to d. max() ought to take any number of variables and evaluate to the maximum for each node or zone across all variables that were provided. If a single variable is provided, then the maximum is simply that variable. This is a better solution than crashing the compute engine. It also keeps the function consistent.
—
Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/visit-dav/visit/issues/19747*issuecomment-2299906656__;Iw!!CGUSO5OYRnA7CQ!ZX-zY-3o2Y3Nltt8TLYztwNMt9AnwXX05vEZFGxmziiuRyPJNCnxSFm6oa2uFzF5-izEdzsrF4C2AcJECnblWPjmi1UE$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BKUOYQQ3GAUFIEQWE3YGTTLZSPD2NAVCNFSM6AAAAABMYUMT22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJZHEYDMNRVGY__;!!CGUSO5OYRnA7CQ!ZX-zY-3o2Y3Nltt8TLYztwNMt9AnwXX05vEZFGxmziiuRyPJNCnxSFm6oa2uFzF5-izEdzsrF4C2AcJECnblWDJMLz_1$.
You are receiving this because you authored the thread.Message ID<EMAIL_ADDRESS>

Expressions to be included:

Average
Minimum
Maximum
Standard Deviation
Variance
Sum
Count
RMS
PDF

@JustinPrivitera add a ticket for the fix to normal max, min, so that they don't error on single inputs.


Concat then encode

FFMPeg allows the input files to be concatenated and passed in to the mp3 codec. At a glance I can't see how this could be done. Could I please get some pointers?

Had the same question. Did you ever figure this out?


Update arangodb-java-driver to 7.7.0

About this PR
📦 Updates com.arangodb:arangodb-java-driver from 7.5.1 to 7.7.0
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.arangodb", artifactId = "arangodb-java-driver" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.arangodb", artifactId = "arangodb-java-driver" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #293.


added infrastructure resources to java kustomization.yaml

This is to avoid multiple Kustomization manifests in Flux. At the moment, Flux add-on in EKS Blueprints CDK does not support multiple Kustomizations.
I though believe that it is a better approach if whenever possible a single path is provided to Flux and the kustomization.yaml file in there takes care of referencing all files and paths needed.

@freschri We are reverting this PR merge. We cant do this PR because this will impact terraform solution because in terraform, java mon is addon to Infra mon on EKS. So we cant do this. You have to wait for multi flux path support or add new Kustomization


Added specs

Added rspec and sqlite3 gems for testing.
Wrote specs for .pluck_to_hash and .pluck_h alias to test
basic functionality and errors.
Advice: https://travis-ci.org/ makes it easy to set up automatic
builds for free with a simple addition of a .travis.yml file
if this PR is accepted it might be worth it to set that up.

That's pretty cool @yez
Thank you!

Glad to contribute, @girishso!


KAFKA-9058: Lift queriable and materialized restrictions on FK Join

Update foreign key joins not to require materialization or queriability to function.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Seems this needs a rebase, as the other PR updating the tests to use TDD was merged.

Thanks for the review, @mjsax !

Thanks for the review, @bbejeck !
Since the only comment is about line length, I think I'd prefer to leave the PR alone at this point because the lines are marginal, and we're already one hour into the three hour build. (If that's ok with you).

Thanks @mjsax and @bbejeck !