Optimal encoding of Code 128 with option "allowxc"

This should encode Code 128 optimally - at least with the option "allowxc". Without that option the encoder will switch to the ASCII range before using character set C. This prevents problems with decoders that have "alternative ways" of dealing with the combination of extended ASCII and character set C. I believe that there are other ways to achieve optimal encoding, but I also think that this is quite an elegant solution. I can't really take credit for it because as with most things it's been done before in some way or another - although perhaps not in the context of Code 128. It is most likely more demanding in terms of memory usage and processing time than the current encoder. It's a brute force encoder if you will. I hope it is as easy to read and understand as the current encoder. I fully understand if it is "too much". It's mainly a way for me of procrastinating on a different project.

Nice work! But as I said on PR #276, I don't think disallowing allowxc is necessary.
On my measurements (with allowxc) it's a fair bit slower than PR #276, e.g. using this script:
% Prefixed with `loadctx`, `unloadctx`, `raiseerror`, `processoptions`, `parseinput`, `renlinear`
% and current master `code128`, with this PR #278 as `lode128` and PR #276 as `mode128`

-2 vmreclaim

/havel true def /havem true def
/totc 0 def /totl 0 def /totm 0 def
/data [
    (123456ABCD)
    (^031^031_^127^159^031^159^159^159^15912345``^255^000^127^255^224^224^159`)
    (123456ABCD123456ABCD123456ABCD123456ABCD123456ABCD123456ABCD)
] def

1 1 3 {
    ==

havel {
    2 vmreclaim

    data {
        /datum exch def
        /difl 0 def
        /startt usertime def
        1 1 100 {
            pop
            datum (dontdraw parse allowxc) /lode128 /uk.co.terryburton.bwipp findresource exec
            pop
        } for
        /endt usertime def
        /difl difl endt startt sub add def
        (    lode128 difl ) print difl ==
        /totl totl difl add def
    } forall
} if

havem {
    2 vmreclaim

    data {
        /datum exch def
        /difm 0 def
        /startt usertime def
        1 1 100 {
            pop
            datum (dontdraw parse) /mode128 /uk.co.terryburton.bwipp findresource exec
            pop
        } for
        /endt usertime def
        /difm difm endt startt sub add def
        (    mode128 difm ) print difm ==
        /totm totm difm add def
    } forall
} if

    2 vmreclaim

    data {
        /datum exch def
        /difc 0 def
        /startt usertime def
        1 1 100 {
            pop
            datum (dontdraw parse) /code128 /uk.co.terryburton.bwipp findresource exec
            pop
        } for
        /endt usertime def
        /difc difc endt startt sub add def
        (    code128 difc ) print difc ==
        /totc totc difc add def
    } forall

} for

havel { (mode128 totl ) print totl == } if
havem { (mode128 totm ) print totm == } if
(code128 totc ) print totc ==

I get the following results (gs 10.02.1, Intel Core i9-9900K)
1
    lode128 difl 80
    lode128 difl 150
    lode128 difl 320
    mode128 difm 30
    mode128 difm 50
    mode128 difm 90
    code128 difc 20
    code128 difc 30
    code128 difc 50
2
    lode128 difl 70
    lode128 difl 140
    lode128 difl 310
    mode128 difm 30
    mode128 difm 50
    mode128 difm 100
    code128 difc 20
    code128 difc 30
    code128 difc 50
3
    lode128 difl 60
    lode128 difl 150
    lode128 difl 320
    mode128 difm 30
    mode128 difm 40
    mode128 difm 100
    code128 difc 20
    code128 difc 30
    code128 difc 50
mode128 totl 1600
mode128 totm 520
code128 totc 300

I've done some inner loop optimizations on the same branch that cut the time in half.

Time consumption is now on par with mode128 if my calculations are correct. However the final optimization takes it too far in my opinion. It bakes in priority and unrolls the inner loop.

I added options to avoid character set C and to avoid extended ASCII latch codes (shift codes are still allowed). Can you think of more options that might be interesting?

avoidc
avoidxlatch

This branch now addresses issue #23 and it dips into #58. The way the encoder works lends itself to these sort of issues.

This encoder only uses the charmaps array to map function codes and GS1-128 linkage (which I still don't know what is). I'm inclined to remove it and define dictionaries seta, setb, and setc directly. Is there an argument for keeping charmaps and charvals around?

I went ahead and did that. This improved performance. Time consumption should now be below mode128 although not as low as the original. Probably because the dictionaries are smaller and lookup cheaper.

Fab stuff!
I don't think the option "avoidxlatch" is particularly useful. Also I'd avoid the word "avoid" - it has a suggestion of "trying one's best" - I'd use "suppress" instead. So "suppressc" and "suppressxc"...
For "avoidxc"/"suppressxc", which I concede could be useful for compatibility, "xc" is pretty cryptic, though it's hard to come up with something that isn't very long - perhaps "unlatchextbeforec"??
Note the PR is missing the latest commit to master ([fd7966c]).
Nit-picks: there's trailing white space on a number of lines; "character set A/B/C" -> "code set A/B/C".
I'll be closing PR #276 in preference to this, though probably not till tomorrow (need to do some more testing).

I've been absent from this discussion, but dropping in to say I'm happy with how things are going in terms of review. - and thanks!
Lack of time rather than lack of interest, on my part. Please don't let that demotivate...

My editor was not configured for removing trailing whitespace on postscript documents. Let me take this opportunity to complain about github treating postscript files as binary. I noticed the issue you raised with linguist.
Do my comments in the postscript file make sense - should I cut down on them or add more?

Your comments are great and fine by me! (There's a few I'd cut down a little but only a very few.)

Do it! :-)

Ha! I'll leave it to Terry to decide... (Also there're so minor it's not worth mentioning really.)

Can I add a few more changes?

You can add as many as you like! Nothing's finalized, any testing I do is automated so is easily re-run.

Okay at this point I'm really just noodling.

Do you think this principle can be applied in other places?

One that comes to mind immediately is MaxiCode, where the 144 module size limits input to be short enough (138 characters) that it might be feasible...
(Note you still haven't rebased this PR to the latest master.)

I clicked the sync button to merge now - is that the wrong way to go
about it?

Venlig hilsen
Aksel Bue Worm Jensen
Lyngbakken 1
3540 Lynge
+45 31 22 42 45
@.***
On 28.10.2024 15:00, gitlost wrote:

One that comes to mind immediately is MaxiCode, where the 144 module
size limits input to be short enough (138 characters) that it might be
feasible...
(Note you still haven't rebased this PR to the latest master.)
--
Reply to this email directly, view it on GitHub [1], or unsubscribe
[2].
You are receiving this because you authored the thread.Message ID:
@.***>

Links:
[1]
https://github.com/bwipp/postscriptbarcode/pull/278#issuecomment-2441673980
[2]
https://github.com/notifications/unsubscribe-auth/AB6HDF56BBDSUFG4JLOXDALZ5YYQHAVCNFSM6AAAAABQSUB4G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBRGY3TGOJYGA

That's cool, thanks!

Wow MaxiCode looks funky! :-) I don't think I've ever seen one of those
in real life. I don't know if I've ever seen a Code 128 either to be
honest, but Code 128 should be pretty common.

Venlig hilsen
Aksel Bue Worm Jensen
Lyngbakken 1
3540 Lynge
+45 31 22 42 45
@.***
On 28.10.2024 15:00, gitlost wrote:

One that comes to mind immediately is MaxiCode, where the 144 module
size limits input to be short enough (138 characters) that it might be
feasible...
(Note you still haven't rebased this PR to the latest master.)
--
Reply to this email directly, view it on GitHub [1], or unsubscribe
[2].
You are receiving this because you authored the thread.Message ID:
@.***>

Links:
[1]
https://github.com/bwipp/postscriptbarcode/pull/278#issuecomment-2441673980
[2]
https://github.com/notifications/unsubscribe-auth/AB6HDF56BBDSUFG4JLOXDALZ5YYQHAVCNFSM6AAAAABQSUB4G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBRGY3TGOJYGA

I have some ideas for auspost as well but that's mostly concerning the
Reed-Solomon calculations.

Venlig hilsen
Aksel Bue Worm Jensen
Lyngbakken 1
3540 Lynge
+45 31 22 42 45
@.***
On 28.10.2024 15:00, gitlost wrote:

One that comes to mind immediately is MaxiCode, where the 144 module
size limits input to be short enough (138 characters) that it might be
feasible...
(Note you still haven't rebased this PR to the latest master.)
--
Reply to this email directly, view it on GitHub [1], or unsubscribe
[2].
You are receiving this because you authored the thread.Message ID:
@.***>

Links:
[1]
https://github.com/bwipp/postscriptbarcode/pull/278#issuecomment-2441673980
[2]
https://github.com/notifications/unsubscribe-auth/AB6HDF56BBDSUFG4JLOXDALZ5YYQHAVCNFSM6AAAAABQSUB4G2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBRGY3TGOJYGA

Oh man I didn't know you could comment here via e-mail. My e-mail address has been harvested long ago, but I don't want to volunteer more information than I have to ;-)

Yes, that was unfortunate. I'm sure though that Micro$oft can be trusted to use your data responsibly (gaffaw).

There may be some things I would like to do differently to keep Code 128 consistent with MaxiCode if I ever get that to work.

I would like to discuss maxicode with the aim of potentially making a pull-request. How should I go about it - start a discussion or simply make the pull-request and move the discussion there?

Wow those improvements seem to make it even slightly faster than the current code!
That's good to know about variables being ok compared to stack, which as you say gets very very old very quickly.
Re MaxiCode, from my point of view just opening a PR would be the way to go...

Just noticed the edit re sub-optimal - will check it out.

I'm preparing a pull request with a test case that exposes it - at least I think it does, it's up for discussion.

Just need to rebase to latest master again.
Latest figures from me:
len data
 10 (123456ABCD)
 20 (123456abcd123456ABCD)
 35 (123456ABCD^031^031abcd^192^192123456^224)
 57 (^031^031_^127^159^031^159^159^159^15912345``^255^000^127^255^224^224^159`)
 60 (123456abcd123456ABCD123456abcd123456ABCD123456abcd123456ABCD)
100<PHONE_NUMBER>123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890)

1
    lode128 difl  10  20  30  30  50  80
    code128 difc  20  20  30  30  50 140
2
    lode128 difl  10  20  20  30  50  80
    code128 difc  10  30  30  30  40 140
3
    lode128 difl  10  20  30  20  50  80
    code128 difc  20  20  30  30  40 140

    lode128 tots  30  60  80  80 150 240 totl  640
    code128 tots  50  70  90  90 130 420 totc  850

I shall endeavour to take a look this week.
It would be helpful if there was a simple outline description of the algorithm posted either here or as comments in the code.

I agree, but I've looked at it for so long now that all comments have started seeming like noise. The key points are that the algorithm is based around a table of best lengths and best prior states. They are separate tables now, but that's beside the point. The table has a row for each character of the message with an entry for each state. Each entry contains the best possible encoded length of the message up to that point ending in the corresponding state. If the message cannot be encoded in a given state then length will be "longer" so that it doesn't get picked as a best prior state. The table also contains the best possible prior state for making a state shift to the corresponding state. That's an optimization. Previously I calculated the best prior state on the fly, so I did a lot of the same calculations over and over. There's still a tiny bit of that. It's a trade-off between memory consumption and redundancy. The calculation takes into consideration the total encoding length of the prior state and the length of the latch from prior state. Does that make sense?. I've looked at it for so long now that I start to go blind to my own difficulties implementing it. I have some ideas for using less memory and perhaps better naming; I've been using "target" and "origin" a lot. For Maxicode I'm using "code set" and "prior set". I can't use "code set" for Code 128 because the code sets are named A, B, and C, but the states are named A0, B0, etc. I might stick with "state" for Code 128 but use "prior state" instead of "origin". It seems more appropriate. Naming matter - especially for understanding.

Let me take this opportunity to complain about github treating postscript files as binary. I noticed the issue you raised with linguist.

I have made some changes that should result in GitHub showing diffs for the PS sources.
@lyngklip So might I ask that you rebase once more since this should facilitate GitHub-based review... (Sorry for the inconvenience.)

The table has a row for each character of the message with an entry for each state. Each entry contains the best possible encoded length of the message up to that point ending in the corresponding state. If the message cannot be encoded in a given state then length will be "longer" so that it doesn't get picked as a best prior state.

Much like the length optimisation presented in the specification for Aztec Code...

The table also contains the best possible prior state for making a state shift to the corresponding state. That's an optimization. Previously I calculated the best prior state on the fly, so I did a lot of the same calculations over and over. There's still a tiny bit of that. It's a trade-off between memory consumption and redundancy.

Attention to the runtime is good.
My main concern with any algorithm is that it makes progress and doesn't degrade into something that has a high time complexity based on length, which might be gamed by a malicious choice of input to exhaust resources.
I haven't looked at how this applies yet.

The calculation takes into consideration the total encoding length of the prior state and the length of the latch from prior state. Does that make sense?

Yes.
One thing that you might also consider taking a look at with regard to harmonisation is Codeblock F, which is essentially stacked Code 128. No pressure.

Much like the length optimisation presented in the specification for Aztec Code...
I will have a look.

My main concern with any algorithm is that it makes progress and doesn't degrade into something that has a high time complexity based on length, which might be gamed by a malicious choice of input to exhaust resources.
I haven't looked at how this applies yet.
I think you will find that the encoder has complexity O(n).

One thing that you might also consider taking a look at with regard to harmonisation is Codeblock F, which is essentially stacked Code 128. No pressure.
I will have a look at that too.

@lyngklip Your latest push does not contain any of your own commits. Don't worry - hold off a second...
I'm now going to merge your code behind a feature toggle and we can realign this PR once that's done.

oh! I missed your comment - and my own push, not sure what I did there, err heh

@lyngklip Okay...
Within your optimal-code128 please do the following please:
git fetch --all
git branch optimal-code128-saved
git reset --hard fe2c2185f3353a78231c5802bf66e477b643d156

Then copy the following files into their respective location, check that you are satisfied with them, and commit.

Ensure that you preserve any modifications that you want to keep (that are not checked in anywhere) before issuing git reset --hard.

(Check the edits if you are following along by email. GitHub stripped out the links.)

Holy smoke, that's confusing

I don't have the Aztec spec, but I took a look at Aztec in BWIPP. The latch tables and other tables look a lot like what I'm suggesting for Code 128. I've looked in vain for history tables corresponding to Code 128 so my first guess is that the algorithms are not identical. Perhaps Aztec encoding doesn't require much history.
I will look more at Aztec. Bar codes are ridiculously, wonderfully complex for what seem like simple things.

Holy smoke, that's confusing - are we there? GitHub must close the pull-request when I force-push. I love Git - Linus is a genius - but my Git skills are limited.

It looks like you've aligned your branch with the current HEAD of BWIPP, but you haven't staged your own changes on top of this. I.e. there is nothing extra in your branch to review. Because of this, GitHub decided to close the PR.
If you stage the files that I linked to (based on your earlier work), commit them, then make and commit your recent changes, then push the result then what will appear hear will be consistent and in a form that can be reviewed online. (And I will be able to re-open the PR.)

Ignore that. GitHub wasn't showing your recent commits until I reopened the PR.
No action on you for now.

Ignore that. GitHub wasn't showing your recent commits until I reopened the PR.

That confused me too. Then again being confused is my MO.

@lyngklip You missed a file, so the tests aren't currently passing.
Please do something like this from the project root:
wget https://raw.githubusercontent.com/bwipp/postscriptbarcode/aa30a21fcc39cbc71997f1891ce58bbe6189dd3a/tests/ps_tests/code128.ps
mv code128.ps tests/ps_tests/code128.ps
make && make test
git add tests/ps_tests/code128.ps
git commit -m 'tests: Fix up for new Code 128 encoder'
git push
```

I should name the states and use those names instead of numbers. Like the Aztec encoder.
I suspect one could make a cookie-cutter encoder. Someone could turn it into a C++ template...

I should name the states and use those names instead of numbers. Like the Aztec encoder. Although for inner-loop code that hurts performance...

For Code 128 I'm not too bothered about performance because the messages are short. The choice is yours...

I suspect one could make a cookie-cutter encoder. Someone could turn it into a C++ template...

That's a laudable goal. I'm in two minds about keeping a "specification-based" optimiser as a fallback for each symbology. Decoders can be frustratingly fragile!

I would like to remove STP from the dictionaries. The encoder only uses the dictionaries to look up FNC1-3 and GS1-128 linkage codes. I understand you might want to keep the char maps for consistency. Consistency is good. The maps impact performance though.

I'm not too concerned about micro-optimising. Removing things that are barely used (providing we have comments against uses of magic numbers is fine though.

Will be offline for a couple of hours now.

@lyngklip My what name / handle would you like your participation to be credited:

https://github.com/bwipp/postscriptbarcode/blob/master/THANKS.md
https://slides.com/terryburton/barcode-writer-in-pure-postscript#/19

@lyngklip By what name / handle would you like your participation to be credited:

I made the hall of fame?  That is pretty cool. Bue Jensen please :-)

@lyngklip By what name / handle would you like your participation to be credited:

I made the hall of fame? That is pretty cool. Bue Jensen please :-)

Occasional and sustained interest by others has made the project what it is, since my investment has necessarily waxed and waned.

By the way, there's a ISO/IEC 24778:2024 now.

By the way, there's an ISO/IEC 24778:2024 now.

Each of the barcode symbology specifications is being worked on and released with changes to align them with "decimal grading", which relates to scoring verification parameters with greater fidelity.
The current tranche of work doesn't present much concern for encoders.

@gitlost I'll probably invert the feature flag and cut a new BWIPP release at the point that you tell me that your port to Zint is adequately tested. (FYI only. I'm not actively pushing for it.)

I ended up deciding to keep the existing Divide-and-Conquer algorithm, adapting it to use many of the techniques here - the only reason being that Zint has a manual Code Set selection which I couldn't get to work properly with the backtracking algorithm despite numerous attempts at hacking it.
I did have a fully functioning direct port though, which passed all my tests (except those involving manual Code Set Selection), and was around 20% faster than the Divide-and-Conquer algorithm.


Implement a Config Screen for the UI

[ ] Change Tilesets by choosing from a list of pre-defined tilesets
[ ] Add a "Reset Highscores" button that will reset all highscores, as soon as those are implemented

Spiltting up this issue in #9 and #7


feat(storage): implement "public access prevention"

This is a new metadata field in the storage Bucket. This PR also fixes
some typos and blunders in the existing tests for
BucketIamConfiguration.
Fixes #5491

This change is

/cc: @tritone @shaffeeullah FYI

Rebased to resolve conflicts.

Rebased to resolve conflicts.

Okay, this has been dragging for too long and still has several weeks before it is ready. Closing for now, will reopen when needed.


Support for Bulk Formats / Databases

Overview
Hello there! 👋 I'm currently running experiments with a substantial number of recordings exceeding 100k. Managing these experiments becomes a bit unwieldy when dealing with over 100k individual files, particularly considering that many of them may lack labels and contain only headers.
I primarily work with Python for data processing, and to streamline my workflow and reduce waiting times, I'm currently converting these extensive directories into single parquet files.
In this context, it would be immensely beneficial if BirdNET-Analyser could natively support bulk formats such as parquet. Alternatively, an even more powerful solution would be the inclusion of support for databases like SQLite. This enhancement would not only optimize storage by eliminating redundant headers but also facilitate efficient tracking of numerous experiments.
Requested Features

Bulk Format Support: Integration of bulk formats, such as parquet, to enable more efficient handling of large datasets.
Database Support: Introduction of support for databases, specifically SQLite, to enhance the organization and tracking of experiments.

Benefits

Storage Optimization: Reducing the number of files by utilizing bulk formats or databases would significantly optimize storage space, especially when many files have minimal content (e.g., only headers).
Enhanced Speed: Accessing and processing a smaller number of larger files generally results in improved speed.
Efficient Experiment Management: Database support, particularly with SQLite, would simplify the process of managing and tracking multiple experiments, enhancing overall workflow efficiency.

I believe these enhancements would greatly benefit users dealing with extensive datasets and contribute to a more streamlined and efficient user experience. Looking forward to the possibility of incorporating these features into future releases! 🚀
Thank you a lot for this great project! 👍

EDIT:
To give a perspective of space:
I have an experiment, where:

The directory containing all CSV label files occupies: 2.8 GB
When merging all CSV files into a single Parquet file, the space requirement is reduced to: 115 MB

You're right, it gets kind of complicated for extremely large datasets. I've always been a fan of file-based results formats because they're easy to access (i.e. you can always open a text file, but might need extra code to read an embedded format). However, I acknowledge that there is an upside in supporting other exchange formats. I'm not sure if the analyzer is the right repo to implement these things, because it evolved more into an application rather than a package or framework. The same goes for the Python bindings (#121) which I think could be a great feature, but maybe something that birdnetlib should cover. What do you think?

Thank you for your response :)
I'm not against the results in single file formats. I also like them. Databases might potentially exceed the framework, but the effort for bulk formats should be manageable. I quite appreciate the command-line-based approach of this repository. I'm thinking of an additional command-line parameter, such as analyze.py --rtype r --merge csv/parquet.
birdnetlib for binding: ✔
The issue is that many functions there have not been implemented yet, for example, the ability to output embeddings, training, or exporting segments.
In my opinion, the effort for producing combined CSV data or Parquet files using libraries like Pandas or Dask should be limited.


Fix the radio button focus outline isn't circle

Details

The radio button focus outline is currently in square and we want to make it circle.
Fixed Issues

$ https://github.com/Expensify/App/issues/49501
PROPOSAL: https://github.com/Expensify/App/issues/49501#issuecomment-2370236219
Tests

Same as QA Steps

[x] Verify that no errors appear in the JS console

Offline tests

Same as QA Steps
QA Steps

Press FAB > Start chat
Press "Add to group" on any user
Keep pressing Tab until it's focused on the selected circle button
Verify the blue outline is a circle
Now, go to workspace workflows page
Press Connect bank account
Choose Connect online with Plaid
Complete the Plaid flow
On the "Choose an account" page, keep pressing Tab until it's focused on one of the radio buttons
Verify the blue outline is a circle

Android/iOS: no blue outline, just make sure the looks still the same

[x] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] Android: Native
[x] Android: mWeb Chrome
[x] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:

[x] I verified that all the inputs inside a form are aligned with each other.
[x] I added Design label and/or tagged @Expensify/design so the design team can review the changes.

[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

MacOS: Chrome / Safari

MacOS: Desktop

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.


Fetch instrumentation experimental package

Description
This PR will add the experimental package responsible for instrumenting the fetch API.
Checklist

[ ] Tests added
[ ] Changelog updated
[ ] Documentation updated

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


CRI: lib: rename package to rkt

This PR adds one commit on top of #3149 reflecting my comment https://github.com/coreos/rkt/pull/3149#issuecomment-245000341.
@yifan-gu feel free to cherry-pick this on top of your PR #3149 and close this if you agree with the changes.

LGTM

Closing this in favor of using the dedicated https://github.com/coreos/rkt/tree/cri branch for development around CRI.


Order with type ORDER_TYPE_STOP_LOSS causes API Error code 1106

Hi.
I'm trying to create a stop loss order, using this code (with other values than in the example):
order = self.client.create_order(            symbol="ETHBTC",            side=SIDE_SELL,            type=ORDER_TYPE_STOP_LOSS,            quantity="1.000",            price="1.000")
Running the code causes this error from the Binance API: binance.exceptions.BinanceAPIException: APIError(code=-1106): Parameter 'price' sent when not required.
How can I place a stop loss order?

If you want a stop loss, try order_limit_sell instead. Relevant documentation here, and you'd use the parameter stopPrice

If you want a stop loss, try order_limit_sell instead. Relevant documentation here, and you'd use the parameter stopPrice

Thanks for the quick reply!
I tried this code: order = self.client.order_limit_sell(             symbol=trade_pair_name,             quantity=quantity,             stopPrice=price)
The result was this:  binance.exceptions.BinanceAPIException: APIError(code=-1106): Parameter 'stopPrice' sent when not required.
What am I missing?

By the way, I think I got this to work:
order = self.client.create_order(symbol=trade_pair_name, side=SIDE_SELL, type=ORDER_TYPE_STOP_LOSS, quantity=quantity, stopPrice=price).
If I'm not mistaking, the stopPrice parameter is not documentet in the API documentation.

I don't think STOP_LOSS is supported anymore, only STOP_LOSS_LIMIT

I keep having *** binance.exceptions.BinanceAPIException: APIError(code=-1116): Invalid orderType. for :
 client.futures_create_order(symbol=_symbol, side=client.SIDE_SELL,
type=client.ORDER_TYPE_STOP_LOSS_LIMIT, quantity=1, stopPrice=80.0)

I don't think STOP_LOSS is supported anymore, only STOP_LOSS_LIMIT

It is actually the opposite.

I'm testing on hedge mode. I aim to create a new order (market open) and set stop loss at 1% below the avgPrice.
What I am currently doing to achieve this:

open a long position on hedge mode

order = client.futures_create_order(
    symbol=trade_symbol,
    side=Client.SIDE_BUY,
    type=Client.ORDER_TYPE_MARKET,
    quantity=quantity,
    positionSide="LONG"
)

wait for order to get filled: order["status"] == "FILLED"
get the order["avgPrice"] and calculate the price 1% below avgPrice call stopPrice
create a stop loss order

order = client.futures_create_order(
    symbol=trade_symbol,
    side=Client.SIDE_SELL,
    type=Client.ORDER_TYPE_STOP_LOSS,
    quantity=quantity,
    positionSide="LONG",
    stopPrice=stopPrice,
)

I cant get the stop loss order to work, getting this error APIError(code=-1116): Invalid orderType.. I have tried both ORDER_TYPE_STOP_LOSS_LIMIT and ORDER_TYPE_STOP_LOSS. What am I missing? Or is there a easier way to achieve what I want to?


Update internship providers

Trello card
update internship providers list
Context
Monthly internship providers list update
Changes proposed in this pull request
Guidance to review

Looks good to me - to be merged once the more general GitHub issues are resolved


gvim display can become messed up after scrollup

Describe the bug
The screen may not always update properly in the windows gui after a cursor movement (via "{") causes the display to scroll downwards while both cursorline and scroll offset (2) are set.
To Reproduce
Detailed steps to reproduce the behavior:

Run gvim on a file large enough to require scrolling. Enable cursorline and set scrolloff=2.
Scroll down the file a little.
Move the cursor upwards by pressing '{'.
After a few presses, the display may become corrupted.

Expected behavior
The display corruption does not happen.
Screenshots
This screenshot shows display before scroll:

This screenshot shows display after scroll (shift-{):

Environment (please complete the following information):

gVim 8.2 with patches 1-1955
OS: Windows 10 and Windows 8.1 both x64
Terminal: GUI

Additional context
This does not occur on every press of "{" which causes the display to scroll, but it does consistently for the text in the above screenshots, so I suspect something about the combination of the text and syntax highlighting perhaps. Edit: No, the problem occurs even with syntax highlighting off.
The problem does not happen if cursorline is disabled or scroll offset is set to 0.
I don't know when the problem was introduced. I tried going back to 8.2.1880 and the behaviour was still there but I just hadn't noticed.

Update: I have tried to identify the problem but I'm not familiar enough with the code base to track it down.
I have found that the screen corruption only seems to occur if:
cursorline is set,
Lines are wrapped because the view is not wide enough,
The wrapped line is out of the view,
Pressing '{' causes the view to scroll so that the wrapped line is now in view.
I don't think the corruption is triggered when pressing '}'.
Before:

After pressing '{' you can see the screen is now messed up with the cursorline artefact being left behind after the scroll:

Then after pressing '}' you can see that the screen remains messed up:

A screen refresh corrects the corruption, until it happens again of course.
Cheers

Have you tried doing the same in a terminal/console?  I wonder if this is a problem specifically in the GUI, the Windows GUI implementation or the generic drawing code.

I can reproduce the bug consistently with this, with both Vim and gVim:

Open cmd.exe
vim --clean -c"set co=40 lines=20 cursorline scrolloff=2" +120 $VIMRUNTIME\doc\options.txt
zt
CTRL-U

Result:

Vim 8.2.3197 x64, from https://github.com/vim/vim-win32-installer/releases

OK, now I can reproduce it.  The scroll amount is probably not computed correctly.

Fixed with patch 8.2.3204

Thank you for the fix. However, in the commit message (c9e7e344ed3), you give me credit even though it really was @basilisk0315 who found and reported the bug.

Thank you for the fix. However, in the commit message (c9e7e344ed3), you
give me credit even though it really was @basilisk0315 who found and
reported the bug.

Ah, I just noticed some strange user name "basilisk0315", didn't realize
that this is John Marriott.  I don't credit user names, only real names.
Anyway, you provided the reproduction steps, that matters a lot.
--
hundred-and-one symptoms of being an internet addict:
219. Your spouse has his or her lawyer deliver the divorce papers...
via e-mail.
/// Bram Moolenaar --<EMAIL_ADDRESS>-- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Thanks Garbriel for the succinct repro instructions.
Thanks Bram for the fix. It has been bugging me for a while now and I am glad that it it has been sorted.
Cheers All
John


Allow config settings with environment variables

#147
Config is the facade and gets first from env then from toml.
All references to toml change to config.
What are your thoughts?

Closes https://github.com/rust-embedded/cross/pull/460.

I changed the Environment struct so that it can be constructed with an optional Map. I did not like the cfg(test) inside the "not test" code.
Makes this also sense for the Configuration? It uses an Environment (that must be created for tests to be independent of env variables during test). The env for config could also be made optional.

@reitermarkus Hi could you have a look? I would like to bring this to a an end

bors try

bors r+


use singular for some post operation ids

This change updates the REST API schema so that the POST operation for comments, comment reactions, and webhooks all use the singular form. E.g., postWebhook instead of postWebhooks. This is a little more natural and aligns with other operations that use singular when the subject is a single object, e.g. getWebhook.

More changes incoming, closing so we can lump them all together.


[stable/envoy] Add general configuration options to make restart-less reload real

Is this a new chart
No
What this PR does / why we need it:
Add the stable/envoy chart more configuration options for wider use-cases.
Special notes for your reviewer:
@josdotso I've added several configuration options that seemed general and useful enough to be included in this chart, while I was making restart-less config reload available to Envoy deployed from this chart.
The number of changes are relatively large but I think it's reviewable if you see each commit 🙏
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[ ] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/retest

/assign @josdotso

/assign @scottrigby

@mumoshu Also for newly introduced values, we need:

Variables are documented in the README.md

@scottrigby Thanks again for reviewing!
The template format fixed in 9146322 and the documentation has been updated in 502eb53.
Would you mind taking a look again?

LOC changed in 9146322 is large but I can say it doesn't break anything according to my manual test with helm-diff that produced no functional diff before/after the change.
I used a fairly large values.yaml with a lot of feature coverage for testing so I'd say it's safe: https://github.com/mumoshu/envoy-xds-configmap-loader/blob/master/example/values.yaml

/lgtm


Shutdown no working

the shutdown command works after adding sudo command before shutdown -h now in the node_helper.js file

Thanks for tour app


Fix Neptun link

Previous link leads to a 404 site. The corrected one includes a graphic and a short description of Neptun 426.

Good job!


MPU Calibration

Still two things are not implemented.

void complementary_filter(int16_t *acce_raw_value, int16_t *gyro_raw_value, float *complementary_angle, float *mpu_offset) -> this doesn't internally use mpu_offset on gyro values

inside read_mpu6050(), call to complementary_filter function doesn't use the offset values defined above.

Can you please mark the review as resolved if you have did it @RISHI27-dot ?

Other than that LGTM, @gautam-dev-maker can you do one final review?

LGTM!


Kata Containers 1.9.2

rootless: Fix rangeUID parsing (Backport to 1.9)
rootless: Fix cgroup creation logic for rootless(Backport to 1.9)

639f033 rootless: Fix rangeUID parsing
fd95e7e rootless: Fix cgroup creation logic for rootless

/test


Fix turbomind session_len and update docs

Motivation
Fix turbomind session_len and update docs
Modification
Please briefly describe what modification is made in this PR.
BC-breaking (Optional)
Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
Use cases (Optional)
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
Checklist
Before PR:

[ ] Pre-commit or other linting tools are used to fix the potential lint issues.
[ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
[ ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
[ ] The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

[ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
[ ] CLA has been signed and all committers have signed the CLA in this PR.

@lvhan028 @irexyc
Hi, could you take a look at this PR？

TurboMind初始化的参数感觉不够，长文本的话至少还需要 rope_scaling_factor。感觉增加一个engine config的参数比较好。类似这种
internlm_20b = dict(
        type=TurboMindModel,
        abbr='internlm-20b-turbomind',
        path="internlm/internlm-20b",
        engine_config=TurbomindEngineConfig(session_len=2048, max_batch_size=8, rope_scaling_factor=1.0),
        max_out_len=100,
        batch_size=8,
        concurrency=8,
        run_cfg=dict(num_gpus=1, num_procs=1),
    )

另外目前generation 的时候目前是写死top_p,top_k了。是不是把EngineGenerationConfig作为TurboMindModel初始化的参数比较好。

hf模型直接评测的时候，也设置了top_p, top_k了么？

hf模型直接评测的时候，也设置了top_p, top_k了么？

我不确定测评的时候是不是要关闭sampling，不过看opencompass的config，有一些模型初始化的时候有generation_kwargs
hf 模型一般会有一个generation_config.json。有的模型用了，比如https://huggingface.co/lmsys/vicuna-7b-v1.5/blob/main/generation_config.json
有的写到了modeling里面，比如https://huggingface.co/internlm/internlm2-chat-20b/blob/main/modeling_internlm2.py#L1158-L1159

@irexyc @lvhan028 hi, pls. take some time to review this PR. THX


Feature Request - Add option to no use debrid service

in my country there is no need to use a debrid service, why i have to use one with this application?

:/ sorry i'm going to wait for an actual solution, thank you for the answer, this project have future!

Initial level changes for streaming torrents are in.
It takes some time to start streaming. If it's lagging, pause for some time for the download to get ahead. (maybe it's my download speed being slow, idk)
On settings there is torrenting section

For nya you'll have to give a mirror that works for you.

Change your download server here https://premiumize.me/speedtest i recommend not using automatic, cloudflare, or direct add those tend to not play well with players. Personally digital ocean Toronto is my best server

Very

Initial level changes for streaming torrents are in. It takes some time to start streaming. If it's lagging, pause for some time for the download to get ahead. (maybe it's my download speed being slow, idk)
On settings there is torrenting section
For nya you'll have to give a mirror that works for you.

Change your download server here https://premiumize.me/speedtest i recommend not using automatic, cloudflare, or direct add those tend to not play well with players. Personally digital ocean Toronto is my best server

Added option to use VLC media player instead of windows media player.
VLC is better than WMP at streaming MKV files.
So if you're using torrents, use Vlc media player, otherwise use the default. (Vlc package is still in pre-release)

Playing with debrid service works perfectly now.
If you don't have a debrid service, I'm downloading the torrent while streaming, so it might some time to start playing.
I've added, download information to know if something is happening.

Video will start playing after first section is downloaded. After this i didn't notice any buffering (unless you seek).
Download will go to. \AppData\Local\Totoro\Torrents
Contents of this folder will get cleared at startup. so, if you want to keep the downloaded files, you might want to move them somewhere else before starting.


Zend\Console\Console and Zend\Console\Request not found

In ZF\ContentNegotiation\Factory\RequestFactory we have a dependency on Zend\Console\Console and Zend\Console\Request but those are not defined in the composer.json.  Now when i call composer install --no-dev --o i get the error ( ! ) Fatal error: Uncaught Error: Class 'Zend\Console\Console' not found in /srv/www/kp-webservice/vendor/zfcampus/zf-content-negotiation/src/Factory/RequestFactory.php on line 22

zend-console is only required for calling Zend\Console\Console::isConsole() which does nothing more than return PHP_SAPI === 'cli'.
In my opinion zend-console should be suggested dependency only required for using console routes. This means the check if it's a command line environment should be done without zend-console.

+1
This is definitley a bug!
In my opinion there should be a travis build with --no-dev --o and one of the following steps:

add zend console to require of composer.json
or
remove dependency from RequestFactory.php
(e.g. https://github.com/marc-mabe/zf-content-negotiation )

Are there any opinions from the maintainers? ping @weierophinney
Best regards
alex

+1
The same was the case with me.
I agree with the @alexz707.
In my case it occurred while trying to deploying the apigility in heroku.
Heroku does not allow deploy in dev mode. But the error occurs in production mode.

From composer.json:
    "suggest": {
        "zendframework/zend-console": "^2.3, if you intend to use the RequestFactory"
    },

https://github.com/zfcampus/zf-content-negotiation/blob/master/composer.json#L47
And:
$ composer install --no-dev -o
Loading composer repositories with package information
Installing dependencies from lock file
Nothing to install or update
Generating optimized autoload files

All seems to be fine.
If you'd like to use RequestFactory you must define zend-console dependency explicitly in your composer.

This repository has been closed and moved to laminas-api-tools/api-tools-content-negotiation; a new issue has been opened at https://github.com/laminas-api-tools/api-tools-content-negotiation/issues/6.


MKV not converting

Hey love the work that you do 👍
So my problem is that I have one particular file that isn't converting. All my other mkv files are converting fine, its just for some reason this one isn't. Looking through the log file in debug mode the error that comes up is:

2017-05-08 19:44:25 - mkvtomp4 - ERROR - Unabled to create external subtitle file for stream 2.
Traceback (most recent call last):
File "C:\Users\mdizo\Documents\Scripts\sickbeard_mp4_automator-master\mkvtomp4.py", line 506, in generateOptions
for timecode in conv:
File "C:\Users\mdizo\Documents\Scripts\sickbeard_mp4_automator-master\converter_init_.py", line 243, in convert
timeout=timeout, preopts=preopts, postopts=postopts):
File "C:\Users\mdizo\Documents\Scripts\sickbeard_mp4_automator-master\converter\ffmpeg.py", line 548, in convert
total_output, pid=p.pid)

If I run this through manual.py it converts like it normally does. Any ideas?

for more info this is my autoprocess.ini

[MP4]
ffmpeg = C:\Users\mdizo\Documents\Scripts\ffmpeg\bin\ffmpeg.exe
ffprobe = C:\Users\mdizo\Documents\Scripts\ffmpeg\bin\ffprobe.exe
threads = auto
output_directory =
copy_to =
move_to = C:\Users\mdizo\Downloads\Torrents\CONVERTED
output_extension = mp4
output_format = mp4
delete_original = True
relocate_moov = True
video-codec = h265
video-bitrate =
video-crf = 22
video-max-width =
h264-max-level =
use-qsv-decoder-with-encoder = True
ios-audio = False
ios-first-track-only = False
ios-audio-filter =
max-audio-channels =
audio-codec = ac3
audio-language =
audio-default-language =
audio-channel-bitrate =
audio-filter =
subtitle-codec = mov_text
subtitle-language = eng
subtitle-default-language =
subtitle-encoding =
fullpathguess = True
convert-mp4 = False
tagfile = False
tag-language = eng
download-artwork = Poster
download-subs = False
embed-subs = False
sub-providers = addic7ed,podnapisi,thesubdb,opensubtitles
permissions = 0777
post-process = False
pix-fmt =
aac_adtstoasc = False
postopts =
preopts =

[uTorrent]
convert = True
couchpotato-label = couchpotato
sickbeard-label = sickbeard
sonarr-label = sonarr
bypass-label = bypass
sickrage-label = sickrage
webui = true
action_before = stop
action_after =
host = http://localhost:36793/
username = sonarr
password = ********
output_directory =
radarr-label = radarr

What client?

utorrent
The run program is:
C:\Python27\python.exe C:\Users\mdizo\Documents\Scripts\sickbeard_mp4_automator-master\uTorrentPostProcess.py %L %T %D %K %F %I %N

can you try using C:\Python27*pythonw.exe*?

It looks like its running into an issue trying to create the external subtitle file
Not sure why that is unfortunately
You could try disabling that feature and see if the issue resolves
I'll take a look and see if I can sort out why though or at least transition this to a noncritical error


improve Project SameAs logic

use strings.EqualFold instead of strings.ToLower with equal
operation when comparing string ignore case
BenchmarkSameAsToLower-8     	29738065	        39.1 ns/op	       0 B/op	       0 allocs/op
BenchmarkSameAsEqualFold-8   	42597748	        25.5 ns/op	       0 B/op	       0 allocs/op

thanks for the feedback, I updated the test case data to exercise the case folding.


Fixups

A bunch of fixups:

Fixes CI ARM builds on master branch
Override alpine openrc service cmd writing over /etc/rancher/k3s/* instead of overwriting the default initrd service
Enable openntpd on alpine
Re-enable tests on alpine (hopefully they should pass now)

green tests: https://github.com/c3os-io/c3os/actions/runs/2309279631, adding just couple of unrelated fixups


Syntax highlighting for javascript files not working

Lapce Version
The Lapce version you are using, which can be found in "About Lapce" at the top right settings icon.

nightly

System information
the operating system used, including its version, e.g. Windows 10, Ubuntu 18.04
Ubuntu 22.04 Kernel 6.2.0-10027-tuxedo
Describe the bug
Syntax highlighting for JS files does not work. Other filetypes (go, py, php) working fine. This is also on self compiled lapce.

The logfile shows:
ERROR lapce_core::language: Encountered QueryError { row: 25, column: 1, offset: 551, message: "function", kind: NodeType } while trying to construct HighlightConfiguration for JavaScript
Deleting the config folders in .config and .local/share and restarting lapce shows the same error in log.


feat(apps/prod/tekton/configs): add build flow for pingcap/advanced-statefulset

currently on push on master events.
Signed-off-by: wuhuizuo<EMAIL_ADDRESS>

/approve


Remove/block use of accept in properties section of broadcast service

@cmebarrow please review when you can .

retest this please

retest this please

LGTM, re kicked off build.  Please feel free to merge if it goes green

@mjolie This is hanging on a test run both on travis and on my local test machine.
Running org.kaazing.gateway.service.broadcast.BroadcastServiceTest
testBroadcast(org.kaazing.gateway.service.broadcast.BroadcastServiceTest) starting
TestClient 1:  received message >|<
TestClient 3:  received message >|<
TestClient 2:  received message >|<
TestClient 1:  received message >|<
TestClient 3:  received message >|<
TestClient 2:  received message >|<
BroadcastServiceTest.testBroadcast success
testSlowConsumer(org.kaazing.gateway.service.broadcast.BroadcastServiceTest) starting
service started
FastTestBackendProducer receive buffer size: 408300
FastTestBackendProducer send buffer size: 146988
FastTestBackendProducer targetBytes: 1110576
FastTestBackendProducer batch size: 13882
SlowTestClient 1: socket is Socket[addr=localhost/<IP_ADDRESS>,port=9880,localport=61323]
SlowTestClient receive buffer size: 408300
SlowTestClient send buffer size: 146988
Time: 11  Producer, sent 13884 bytes
Time: 72  Producer, sent 27768 bytes
Time: 139  Producer, sent 41652 bytes
Time: 200  Producer, sent 55536 bytes
Time: 262  Producer, sent 69420 bytes
Time: 328  Producer, sent 83304 bytes

Up to 1500+ iterations before I killed it

@dpwspoon I am not able to reproduce this locally, did you test with my branch ?

Yes, I used your branch to reproduce locally

Actually I ran
git checkout -b mjolie-rmAccepPropBCS develop
git pull https://github.com/mjolie/gateway.git rmAccepPropBCS

Which is a merge of your branch and develop

cleared myself as the assignee, please add me back once this is ready

@cmebarrow Hi Chris assigning to you to ask for help , Can you please take a look at my branch , there is one test that gets trapped in infinite loop only on MAC/Unix, but I am not able to figure out why.  When I run on windows this socket does get closed and the test passes ?
You can look at the travis build log: but really it just shows the loop, for:
testSlowConsumer(org.kaazing.gateway.service.broadcast.BroadcastServiceTest) starting
:
:
:
:.
SlowTestClient 1: iteration #69 has 0 bytes available
Time: 70158  Producer, sent 48781206 bytes
Time: 70221  Producer, sent 48827313 bytes
Time: 70285  Producer, sent 48873420 bytes
Time: 70349  Producer, sent 48919527 bytes
Time: 70413  Producer, sent 48965634 bytes
Time: 70477  Producer, sent 49011741 bytes
Time: 70540  Producer, sent 49057848 bytes
Time: 70605  Producer, sent 49103955 bytes
Time: 70668  Producer, sent 49150062 bytes
Time: 70733  Producer, sent 49196169 bytes
Time: 70797  Producer, sent 49242276 bytes
Time: 70867  Producer, sent 49288383 bytes
Time: 70941  Producer, sent 49334490 bytes
Time: 71015  Producer, sent 49380597 bytes
Time: 71079  Producer, sent 49426704 bytes

@cmebarrow  updated from develop pls see my last comment above

@mjolie I have looked at the test concluded the issue is that the run loop in SlowTestClient is not detecting socket close on some platforms because it is not attempting to read the socket. My pull request https://github.com/mjolie/gateway/pull/1 should fix the issue hopefully. Just merge it - it will add the changes to your branch, then we can see if the Travis build on your PR then passes.


Missing 'end' in /bart/matlab/bart.m @ branch v0.4.04

Hey,
just downloaded the last update (v0.4.04) and there is a missing 'end' in /bart/matlab/bart.m at line 33.
Not existing in master branch, but I did use the link of the gh-page to download and than got to this version.
Best regards
Max

This should be fixed in the new version. Thank you.


Executing input groups concurrently by default

Same logic for concurrent=True for tasks, it is even difficult to set up a case for concurrent=False for input groups, so it makes sense to set concurrent=True as default.

Wait, I think you've justified concurrent =False  by default by the user case that behavior on later input will depend on previous input. Is that a valid user case?

The example I gave in the doc is
sum = 0
input: for_each={'i': range(4)}
sum += i
print(f'sum is {sum} at index {_index}')

and the use of concurrent=False here is to pass sum to the next step, instead of having sum=0 for each input group. I am not sure if this is a good user case but I can imagine that users might expect concurrent=False when they, for example, process some data line by line and expect the output in the same order.
This is indeed a weaker case compared to task: concurrent=False, and I just try to set a reasonable default that is safe, less surprise, and clean (e.g. do not have to write concurrent=True everywhere).

and I just try to set a reasonable default that is safe, less surprise, and clean

In terms of "safe", maybe concurrent = False; in terms of cleaness then maybe concurrent = True because that'd perhaps be the most user case. I have a feeling you cannot have both :)

OK, then. Let us keep concurrent=False as default.


Target offline

The backup to OneDrive ran without any problems for a long time, but now Hyper Backup always reports "Target: offline", which is definitely not true.  What could be the problem?

Getting the same issue now after probably over a year of things working. Did you ever get a solution @hdau54 , or did it resolve on its own?


Use BOOST_TEST for map equality in word_count

Fix exercism/exercism.io#2616
BOOST_REQUIRE_EQUAL_COLLECTIONS isn't valid in the current version of boost (1.59.0). Using BOOST_TEST in the word_count test suite fixes a compilation error as described in exercism/exercism.io#2616

Unfortunately, this breaks the code for boost versions prior to 1.59.
They were supposed to retain backwards compatibility.   Boost 1.59 is the default in brew which is how I got it.
(I'm on OS X)
On Tue, Dec 29, 2015 at 5:38 PM Richard Thomson<EMAIL_ADDRESS>wrote:

Unfortunately, this breaks the code for boost versions prior to 1.59.
They were supposed to retain backwards compatibility. This is basically a
bug in boost.test 1.59
—
Reply to this email directly or view it on GitHub
https://github.com/exercism/xcpp/pull/58#issuecomment-167893459.

I'm preparing a change that will correct this problem and be independent of the version of Boost you have.

@mhamrah Can you pull the boost branch of the xcpp repository and verify that my changes work for you now?

Unfortunately this didn't work.

git checkout upstream/boost
cd word-count
mkdir build
mv example.h word_count.h
mv example.cpp word_count.cpp
cd build
cmake -G "Unix Makefiles" ..
make

Output:
❯ make
Scanning dependencies of target word-count
[ 33%] Building CXX object CMakeFiles/word-count.dir/word_count_test.cpp.o
In file included from /Users/mhamrah/dev/gh/xcpp/word-count/word_count_test.cpp:5:
/Users/mhamrah/dev/gh/xcpp/word-count/require_equal_containers.h:71:8: error: no template
      named 'print_log_value'; did you mean 'tt_detail::print_log_value'?
struct print_log_value<std::pair<int, int>>
       ^~~~~~~~~~~~~~~
       tt_detail::print_log_value
// rest omitted

Boost 1.59

@mhamrah Thanks for trying that.  Unfortunately I do not have access to  Mac for testing.  Can you try making the following changes to require_equal_containers.h and see if this fixes it?
diff --git a/word-count/require_equal_containers.h b/word-count/require_equal_containers.h
index 48925e1..5d1641c 100644
--- a/word-count/require_equal_containers.h
+++ b/word-count/require_equal_containers.h
@@ -65,6 +65,8 @@ inline wrap_stringstream &operator<<(wrap_stringstream &str, std::pair<K, V> con

 namespace test_tools
 {
+namespace tt_detail
+{

 // teach Boost.Test how to print std::pair with BOOST_REQUIRE_EQUAL
 template<>
@@ -76,6 +78,7 @@ struct print_log_value<std::pair<int, int>>
     }
 };

+} // namespace tt_detail
 } // namespace test_tools
 } // namespace boost

I just noticed that the error you're getting is on line 71, but line 71 is in the < 1.59 branch of the code.  Can you look at version.hpp in your Boost distribution and tell me the value of BOOST_VERSION?  In my Boost 1.59 distrbution, boost_1_59_0/boost/version.hpp, BOOST_VERSION is defined to be 105900.

The defined BOOST_VERSION is 105900.
I wanted to make sure I'm reading that diff correctly: it just looks like you're enclosing test_tools in tt_detail, right?

Yes, but that is in the section for Boost < 1.59, which shouldn't come into play at all.  On MSVC with Boost 1.59, things are compiling just fine for me.  I'm going to try on a linux machine and see what I get.

It didn't appear that BOOST_VERSION was defined anywhere. When I include <boost/version.hpp> in require_equal_containers.h, I get a different error:
In file included from /usr/local/include/boost/test/tools/old/impl.hpp:24:
/usr/local/include/boost/test/tools/detail/print_helper.hpp:50:14: error: call to function
      'operator<<' that is neither visible in the template definition nor found by
      argument-dependent lookup
        ostr << t;

This is also the same error which occurs if I apply your patch above without include the boost/version.hpp file.
Mike

OK, I was able to reproduce the problem on linux.  (The Travis CI builds use boost 1.55 and I couldn't find a linux PPA containing boost 1.59.)
At the top of require_equal_containers.h, after the line with #include <vector>, add the following:
#include <boost/version.hpp>

And tell me if that fixes the problem for you.
I appreciate you assistance in testing this!

@LegalizeAdulthood  happy to help. Not sure if you saw my earlier comment regarding boost_version? I think we posted at similar times. Unfortunately I couldn't get the type-specific print_Log_output to work. https://github.com/exercism/xcpp/pull/58#issuecomment-168571019.
some more debug output:
In file included from /usr/local/include/boost/test/tools/old/impl.hpp:24:
/usr/local/include/boost/test/tools/detail/print_helper.hpp:50:14: error: call to
      function 'operator<<' that is neither visible in the template definition nor found
      by argument-dependent lookup
        ostr << t;
             ^
/usr/local/include/boost/test/tools/detail/print_helper.hpp:172:5: note: in instantiation
      of member function
      'boost::test_tools::tt_detail::print_log_value<std::__1::pair<const
      std::__1::basic_string<char>, int> >::operator()' requested here
    print_log_value<T>()( ostr, ph.m_t );
    ^

@mhamrah Yes, I saw that you have Boost 1.59 as reported by BOOST_VERSION.  However, the error diagnostic you posted indicated that BOOST_VERSION wasn't defined when processing require_equal_container.h.  (This is why you got to the #else portion of the #if.)
Did you add the #include <boost/version.hpp> at the top of require_equal_containers.h?

Is that the entire error output?

It's not. Full output;
~/dev/gh/xcpp/word-count/build remotes/upstream/boost*
❯ make
Scanning dependencies of target word-count
[ 33%] Building CXX object CMakeFiles/word-count.dir/word_count_test.cpp.o
In file included from
/Users/mhamrah/dev/gh/xcpp/word-count/word_count_test.cpp:3:
In file included from /usr/local/include/boost/test/unit_test.hpp:18:
In file included from /usr/local/include/boost/test/test_tools.hpp:38:
In file included from /usr/local/include/boost/test/tools/old/impl.hpp:24:
/usr/local/include/boost/test/tools/detail/print_helper.hpp:50:14: error:
call to
      function 'operator<<' that is neither visible in the template
definition nor found
      by argument-dependent lookup
        ostr << t;
             ^
/usr/local/include/boost/test/tools/detail/print_helper.hpp:172:5: note: in
instantiation
      of member function
      'boost::test_tools::tt_detail::print_log_value<std::__1::pair<const
      std::__1::basic_string<char>, int> >::operator()' requested here
    print_log_value<T>()( ostr, ph.m_t );
    ^
/usr/local/include/boost/test/utils/wrap_stringstream.hpp:66:19: note: in
instantiation
      of function template specialization
      'boost::test_tools::tt_detail::operator<<<std::__1::pair<const
      std::__1::basic_string<char>, int> >' requested here
    targ.stream() << t;
                  ^
/usr/local/include/boost/test/tools/old/impl.hpp:212:19: note: in
instantiation of
      function template specialization 'boost::operator<<<char,
      boost::test_tools::tt_detail::print_helper_t<std::__1::pair<const
      std::__1::basic_string<char>, int> > >' requested here
                  <<
::boost::test_tools::tt_detail::print_helper(*left_begin)
                  ^
/Users/mhamrah/dev/gh/xcpp/word-count/word_count_test.cpp:15:5: note: in
instantiation of
      function template specialization

'boost::test_tools::tt_detail::equal_coll_impl::operator()<std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<std::__1::basic_string<char>,
      int>,
std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char>,
      int>, void *> *, long> >,

std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<std::__1::basic_string<char>,
      int>,
std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char>,
      int>, void *> *, long> > >' requested here
    REQUIRE_EQUAL_CONTAINERS(expected, actual);
    ^
/Users/mhamrah/dev/gh/xcpp/word-count/require_equal_containers.h:86:5:
note: expanded
      from macro 'REQUIRE_EQUAL_CONTAINERS'
    BOOST_REQUIRE_EQUAL_COLLECTIONS(left_.begin(), left_.end(),
right_.begin(), r...
    ^
/usr/local/include/boost/test/tools/old/interface.hpp:245:5: note: expanded
from macro
      'BOOST_REQUIRE_EQUAL_COLLECTIONS'
    BOOST_TEST_TOOL_IMPL( 1,
::boost::test_tools::tt_detail::equal_coll_impl(),         \
    ^
/usr/local/include/boost/test/tools/old/interface.hpp:66:9: note: expanded
from macro
      'BOOST_TEST_TOOL_IMPL'
        BOOST_JOIN( BOOST_TEST_TOOL_PASS_PRED, frwd_type )( P, ARGS ),
     \
        ^
note: (skipping 3 expansions in backtrace; use -fmacro-backtrace-limit=0 to
see all)
<scratch space>:88:1: note: expanded from here
BOOST_TEST_TOOL_PASS_PRED1
^
/usr/local/include/boost/test/tools/old/interface.hpp:48:49: note: expanded
from macro
      'BOOST_TEST_TOOL_PASS_PRED1'
#define BOOST_TEST_TOOL_PASS_PRED1( P, ARGS ) P BOOST_PP_SEQ_TO_TUPLE(ARGS)
                                                ^
/usr/local/include/boost/preprocessor/seq/to_tuple.hpp:21:40: note:
expanded from macro
      'BOOST_PP_SEQ_TO_TUPLE'
#    define BOOST_PP_SEQ_TO_TUPLE(seq) (BOOST_PP_SEQ_ENUM(seq))
                                       ^
/Users/mhamrah/dev/gh/xcpp/word-count/require_equal_containers.h:32:22:
note:
      'operator<<' should be declared prior to the call site
inline std::ostream &operator<<(std::ostream &str, std::pair<K, V> const&
item)
                     ^
1 error generated.
make[2]: *** [CMakeFiles/word-count.dir/word_count_test.cpp.o] Error 1
make[1]: *** [CMakeFiles/word-count.dir/all] Error 2
make: *** [all] Error 2

On Tue, Jan 5, 2016 at 12:26 AM Richard Thomson<EMAIL_ADDRESS>wrote:

Is that the entire error output?
—
Reply to this email directly or view it on GitHub
https://github.com/exercism/xcpp/pull/58#issuecomment-168902063.

Can you please create a gist with your copy of word_count_test.cpp and require_equal_containers.h

https://gist.github.com/mhamrah/79ed07a00209754a0ab1
The word_count.{h, cpp} is from the example.
On Tue, Jan 5, 2016 at 10:13 PM Richard Thomson<EMAIL_ADDRESS>wrote:

Can you please create a gist with your copy of word_count_test.cpp and
require_equal_containers.h
—
Reply to this email directly or view it on GitHub
https://github.com/exercism/xcpp/pull/58#issuecomment-169207301.

The require_equal_containers.h header looks good.  I also need to see your copy of the test source file, word_count_test.cpp.

Updated gist.

OK, the first compile error is on line 3 of word_count_test.cpp which is the inclusion of the boost.test unit test header.  I don't understand why that would cause an error.  Are you compiling with clang or with gcc?  Can you do make VERBOSE=1 so I can see the full compile command line and then for whatever compiler cmake is using (probably /usr/bin/c++), run that compiler with --version.

Basically, what's in your compile output doesn't make any sense to me based on those files, so I must be missing something.
I was able to reproduce your original error when I compiled on linux and then the inclusion of the version header from boost fixed it on linux.  MacOS (for this code anyway) shouldn't be significantly different and I'm just really confused why it isn't working for you, particularly since the compiler output you provided indicates an error as soon as we include the boost test headers.

Thanks for putting so much effort into this, you really don't have to do it on my behalf, but I'm happy to help. If you want to punt on this, simply close the pr.
But here's the verbose compiler output:
https://gist.github.com/mhamrah/b32a9f97eb6edadb5ea8
and the compiler info:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.0.0
Thread model: posix
So maybe something is up with Apple's llvm?

Well, we need to figure out the problem here otherwise any Mac user is going to get stuck like this.
Is there any chance we could do some sort of desktop sharing so I could try things out directly on your machine while you watch?

OK, let's try "binary search" to find the offending code in the test file.

In other words, your test cpp file just contains the #include lines and nothing else.

@sunzenshen Can you help out with testing the boost branch of the xcpp repository with the latest boost from homebrew?

@LegalizeAdulthood I'll make some time later this week (tomorrow?) to catch up on this thread and clone the repository to test. Send a search party if you don't hear from me as late as Sunday. :snowman:

@sunzenshen Thanks.  I'm going to see if I can track down a Mac for testing this as well.  Ultimately I need to be able to test stuff on Macs on my own.

@LegalizeAdulthood Cloned the xcpp repo, checked out the boost branch, and encountered the following compilation with boost 1.59:
https://gist.github.com/sunzenshen/915085e43b283935f5eb

@sunzenshen Thanks.  That is the same error that @mhamrah was getting, so I think this means I need to get a Mac myself with Boost >= 1.58 and figure out what is going on.

@LegalizeAdulthood Sorry I've been MIA. I'm happy to do a desktop share so you can poke around. I'm EST and evenings are best. Shoot me an email (on my profile) and we can work out the details. I'll also try and see if I can make progress.

I've merged my solution for Windows/linux into master, it may need some refinement for Mac, but I haven't had a chance to test yet.  Therefore I'm going to close this pull request.


Redesign profile card , and assest optimizations for better loading

hi
i can redesign the card to look even better.

can you pls assign me this task and this design is matching with the website even more :)
i can add assest optimizations for better loading

hi
i can redesign the card to look even better.
can you pls assign me this task and this design is matching with the website even more :)
i can add assest optimizations for better loading

after pushing the changes, create pr


Instead of only looking at patches-base line in the spec file, allow for filtering out patches by regex

Problem we are trying to solve:
Because our CI configuration is part of the branch we use to generate patches with the help of rdopkg update-patches action, our current workflow for modifying it is as follows:

Propose a change that affects the CI config in gerrit
Wait for CI, see if results are satisfactory and merge the change

Instead of being done, we now have to also do the following set of tasks that make us sad:

rebase the branch that the patch just merged on so as to move the CI config patch to the "bottom" so that it will not be caught and added to dist-git on the next run of rdopkg update-patches
Update the spec file in dist-git by just changing the patches_base comment line (sometimes this is not possible at all for reasons I won't get into here. In this case we are very sad).
Push the dist-git change
Force push the branch we just rebased to gerrit. Since CI config does not get worked on a lot in parallel this does not really break people, but it is very brittle and, honestly, annoying.

What we propose to do:
Add a --filter-out (or similar) switch to the update-patches action which would allow for an arbitrary regex to be passed, this regex would filter out any patches that would otherwise end up in the dist-git repo, and have commit messages that match the regex (or at least title lines). So we could do something like:
$rdopkg update-patches --filter-out "^CI-CONFIG.*"  which would generate patches from a) commits above patches_base so that we are backwards compatible b) that have commit messages that **do not start with "CI-CONFIG"
If the maintainers of rdopkg are on board with this idea - I'd be happy to hack up a patch to do this.

Copying comment from proposed patch:
:+1: for # filter_out "magic" comment
:-1: for adding --filter-out new CLI option
Too expand, this should be more reliable, developers would be forgetting to use CLI option.

"Add patches_ignore magic comment" was merged.


change SA1600 rulset to Error

Reference Issue: https://github.com/hardkoded/puppeteer-sharp/issues/1384

@MajeQafouri did you miss any commit here?

@MajeQafouri these issues are about enabling that rule and fixing the errors that come from enabling it.

@MajeQafouri these issues are about enabling that rule and fixing the errors that come from enabling it.

Ok, I will check it again

@kblok
During the build process, StyleCop.Analyzers package does not allow to build the solution, because of the lack of documentation. do you have any suggestions?

If SA1600 wasn't set as an error before it's because we have missing documentation :)
I see that much of the missing documentation is from internal elements. I would try disabling that using this.
After that, we will need to add all the missing documentation. Constructors tend to have the same style, the same for exceptions. pptr.dev is a good source of content.


PublishTemplatesCommand failed with Laravel 5.0

I run the command:
$ php artisan generate:publish-templates

then, I get some errors with publishing package config.
In PublishTemplatesCommand.php line 32:
$this->call('config:publish', ['package' => 'way/generators']);

I think it should be:
$this->call('publish:config', ['package' => 'way/generators']);

In PublishTemplatesCommand.php line 63:
$configPath = app_path('config/packages/way/generators/config.php');

I think it should be:
$configPath = base_path('config/packages/way/generators/config.php');

This is Laravel 4 Generators, not 5.


Keep NavigationServer active while SceneTree is paused

Can't speak for physics that shares a similar faith but deactivating the entire NavigationServer on a SceneTree pause is wrong.
The NavigationAgents already react to SceneTree pauses on their own.
Related to https://github.com/godotengine/godot/issues/73655

This really depends how NavigationServer works. On the physics side, you can't just really pause individual bodies because it's not truly pausing them (this is why you have to select how you want them paused). If on navigation server, pausing the agent effectively pauses them and nothing continues on background, them this PR makes sense and it makes no sense pausing the whole server.

The problem with the current deactive is if the entire NavigationServer gets deactivated it also does no longer synchronizes the navigation maps or statistics (or speaking of the future would not process async pathqueries). This means even those navigation related nodes that users set to always process would no longer work for them or receive updates.
While the NavigationServer should not be deactivated by pauses I guess it would not hurt to add a true pause mode that suspends the avoidance processing or can be used to suspend individual agents.
The NavigationAgent nodes are already following their parent pause / processing mode but there is the case when users work with server-created agents only. They can be paused but it is not straight-forward to do so as it requires removing them from their navigation map or removing their avoidance callback.
So I think we can keep this pr open for now and I will see if this can be added without to much of a rework required.

Added a pause property to NavAgent and a pause function to the NavigationServer API.
NavigationAgents already follow their parent node process mode in case of pauses but this gives users that only use the NavigationServer API a way to control pauses themself.

Seems to me that the original PR could've been merged as is this new functionality could've been made as a dedicated PR. Now this needs to be reviewed again and we can't even merge the original fix :eyes:

@YuriSizov I reverted to the original pr that just removes the server pause and will make a dedicated pr for the agent pause mode.

Thanks!


Add basic highlighting for SAS language. (fix #515)

Part of this is based on the SAS package for Sublime (by Jiangtang) and the implementation in GeSHi, conforming to the discussion in #515. The rest comes from the documentation. I think the highlighting is quite basic, there is a lot of room for improvements, but it's a start.
See the same code in GeSHi, Sublime and Prism. (images not inlined because of their dimensions)

From a quick look, LGTM. Merging.


RuntimeError: dictionary changed size during iteration

Scons 4.0.1 on Gentoo Linux.
I didn't get and answer on Discord, so posting here
  File "/usr/lib/python3.8/site-packages/SCons/Environment.py", line 1407, in Clone
    clone._dict = semi_deepcopy_dict(self._dict, ['BUILDERS'])
  File "/usr/lib/python3.8/site-packages/SCons/Util.py", line 539, in semi_deepcopy_dict
    for key, val in x.items():
RuntimeError: dictionary changed size during iteration:

My company uses Scons for a huge project, so it's sometimes hard to say what's going on or how Scons was invoked.
But quite likely the code above was executed in a thread and I don't know if it's eligible.
I believe the easiest fix is to get a list of keys and not items iterator.
something like this probably
def semi_deepcopy_dict(x, exclude = [] ):
    copy = {}
    for key in list(x.keys()):
        # The regular Python copy.deepcopy() also deepcopies the key,
        # as follows:
        #
        #    copy[semi_deepcopy(key)] = semi_deepcopy(val)
        #
        # Doesn't seem like we need to, but we'll comment it just in case.
        if key not in exclude:
            copy[key] = semi_deepcopy(x[key])
    return copy

Your approach is fine and should work to avoid, but the problem is a little surprising: the code in question was evolved from Python's copy module (long ago, it has evolved in other ways since) and it has the same basic construct:
    for key, value in x.items():
        y[deepcopy(key, memo)] = deepcopy(value, memo)
    return y

and this doesn't seem to be causing people to hit this.
In the scons case the surprise is that cloning an environment shouldn't be happening in a threaded context.

Your approach is fine and should work to avoid, but the problem is a little surprising: the code in question was evolved from Python's copy module (long ago, it has evolved in other ways since) and it has the same basic construct:
    for key, value in x.items():
        y[deepcopy(key, memo)] = deepcopy(value, memo)
    return y

and this doesn't seem to be causing people to hit this.
In the scons case the surprise is that cloning an environment shouldn't be happening in a threaded context.

You'll need to provide a reproducer for us to be able to help/fix this.
Please come back to discord or the users mailing list.
Also we'd need more of the stack trace to help.
As I'm sure you read when you opened this issue we prefer to discuss issues in users mailing list, IRC, or discord before anyone files an issue.

You'll need to provide a reproducer for us to be able to help/fix this.
Please come back to discord or the users mailing list.
Also we'd need more of the stack trace to help.
As I'm sure you read when you opened this issue we prefer to discuss issues in users mailing list, IRC, or discord before anyone files an issue.

Hi guys.
Nobody answers on Discord and honestly I hate using mailing lists.
So maybe someone else with the same problem will discuss this with you.
With regards to stack, there are just entries from our product, nothing interesting. But one of the entries mentions starting a thread. I don't want to expose any stuff which is internal for the company, sorry.
We plan to have a private Scons fork for now, since we ca't wait till it's resolved.
Sorry for bothering you with such non-standard issues.

Hi guys.
Nobody answers on Discord and honestly I hate using mailing lists.
So maybe someone else with the same problem will discuss this with you.
With regards to stack, there are just entries from our product, nothing interesting. But one of the entries mentions starting a thread. I don't want to expose any stuff which is internal for the company, sorry.
We plan to have a private Scons fork for now, since we ca't wait till it's resolved.
Sorry for bothering you with such non-standard issues.

One more thing to mention. The issue was noticed while porting our build system from Python2+some old Scons to Python3+Scons4.
So apparently the backward compatibility was broken. Basically it's fine to break it with major releases. But unfortunately for us it may require some significant redesign. Even though I haven't reviewed the problem deeply from our side yet.

One more thing to mention. The issue was noticed while porting our build system from Python2+some old Scons to Python3+Scons4.
So apparently the backward compatibility was broken. Basically it's fine to break it with major releases. But unfortunately for us it may require some significant redesign. Even though I haven't reviewed the problem deeply from our side yet.

That this exposes with Py3 is no surprise: that's when the keys, values, items dictionary methods started returning views by default, as opposed to needing ask for one (e.g. Py2's iteritems).  It's not my call but I don't see any problem with your change.

That this exposes with Py3 is no surprise: that's when the keys, values, items dictionary methods started returning views by default, as opposed to needing ask for one (e.g. Py2's iteritems).  It's not my call but I don't see any problem with your change.

@Ri0n - you're clearly many timezones out of sync with many of our community. Expecting instant responses on discord is unrealistic.
Also we need a reproducer so that we can add a test to cover this issue. So it IS worth the time.

@Ri0n - you're clearly many timezones out of sync with many of our community. Expecting instant responses on discord is unrealistic.
Also we need a reproducer so that we can add a test to cover this issue. So it IS worth the time.

sure. I'll make a reproducer. Unfortunately it won't be soon.

sure. I'll make a reproducer. Unfortunately it won't be soon.

@Ri0n We've had other reports of this now.  It seems in the specific known case, a modification to an env (well, in this case a Clone of one) was being initiated in an Action function.  We don't think that's a great idea - SCons pretty much expects the various created environments to be stable once the build phase kicks off - and at the very least, at that point, you're in multi-threaded mode and thread safety becomes an issue.  Is there any chance you have enough information left over to know if that applies to your case as well?

In my case we have a huge project and there is a lot of code around Scons splitted up into modules responsible for various project parts or just some common helpers.
On start we load all the modules. Some of them are heavy on initialization, so we load them asynchronously in threads. On initialization the modules add various stuff to the env. Some of them clone the env first and then add its stuff to the clone, others just add to the existing env.
Eventually we join all the threads. And after that start building the project.

Honestly SCons was never designed or coded to support initializing the build via multithreaded SConscript logic.
And there never has been (thus far) the intent to support such usage.
If you're seeing the dictionary in question being changed while you're cloning it, that should tell you you have unpredictable unguarded writes happening, and could yield very unpredictable repeatability.
There are many very large builds which aren't using multithreading to initialize the logic.
Generally SConscript/SConstruct processing time is a small fraction of any build..
How many source files are you dealing with?
Are you running non-trivial python logic in your SConscript logic?

Hard to say how many files are compiled with SCons. But there are 16k just c/c++ files in the project.
Regarding the Clone() on the initialization it seems theres is just one place in our code and it does't care much about consistency of the cloned env. It roughly does the following (kind of shallow clone)
  android = env.Clone()
  dict = android.Dictionary()
  keys = dict.keys()
  for key in keys:
    del android[key]
  env['android'] = android

It's where it often fails on the first line. Regarding the non-trivial logic it's everywhere there. The framework is designed to be totally crossplatform with regards to boh host and target.

That's kind of an interesting usage, what's the idea? You get a duplicate environment with all the construction variables removed but the methods intact? Or is the pseudo-code description leaving something out?

Nothing left. the code is almost as is. Then the cloned env is populated with some variables afterwards.
I'm not really sure why it's needed. The guys who implemented this left the company long ago. From the code the strong necessity is not obvious.
The only related comment in the code I see
# Clone the environment and clear it out.  I want the class not the data

Well I start thinking I just have to try to refactor this without any cloning and check if it works :-)

Might be worth a look, yes.  If you find this is really what you need to do there's a cheap shortcut (this isn't considered an "official" way, so you didn't hear it from me :-) ):
android = env.Clone()
android._dict = {}

But you're actually paying quite a cost - the Clone does quite a lot of fiddling, including rerunning tool initialization, to make sure the vars get set up right and then those are just tossed... there may be a better way to handle.

Can you disabled the threaded initialization and see how long it takes?
16k files is not that many...
There are much larger builds around.
Also an OverrideEnvironment() may do what you want, but please try disabling the threaded sconscript processing first?


worker role path has to be removed from the config map

Description
I have to create my worker roles with specific naming schema (including a path). But the nodes won't join if the worker arn in the configMap has the path included. See https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html#access-denied-managed-node-groups
Versions

Module version [Required]: 18.24.1

Terraform version: 0.14.11

Provider version(s):
Using hashicorp/tls v3.4.0 from the shared cache directory
Using hashicorp/null v3.1.1 from the shared cache directory
Using hashicorp/kubernetes v2.11.0 from the shared cache directory
Using hashicorp/template v2.2.0 from the shared cache directory
Using hashicorp/aws v4.19.0 from the shared cache directory

Reproduction Code [Required]
create a worker role with a path arn:aws:iam::xxxxxx:role/development/ccs-office/office-ir-eks-worker
the node-group creation will fail (though everything will come up and will be running). removing "development/ccs-office/" onlye in the configMap is already enough to let the nodes join.
Expected behavior
joining nodes
Actual behavior
node group created but creatinf is failing as nodes don't join in time

another way to solve it (instead of the user fiddling around with this problem .... like I did) would be to just remove the path from the role when the configmap resource is created. as far as I can say everything else works with the path.

you should fix this with regex on ARN. just remove the path.
that way - a user can create a role with a path and not suffer from this issue.
join("/", values(regex("(?P<prefix>arn:aws:iam::[0-9]+:role)/.+/(?P<role>[^/]*)", aws_iam_role.this.arn)))


Key error for some stations

There is a good chance this is a user error, but I am running into the following error, specifically when pulling GHCND and PRCP data. If I follow the example and generate a response, there appears to be data, but using to_dataframe() throws the following error for some stations:
File "/PycharmProjects/Regenerate/.venv/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3553, in run_code     exec(code_obj, self.user_global_ns, self.user_ns)   File "<ipython-input-15-7a2ac4d974af>", line 1, in <module>     response.to_dataframe()   File "/PycharmProjects/Regenerate/.venv/lib/python3.12/site-packages/pyncei/bot.py", line 1068, in to_dataframe     df = pd.DataFrame(self.values())          ^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/PycharmProjects/Regenerate/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 832, in __init__     data = list(data)            ^^^^^^^^^^   File "/PycharmProjects/Regenerate/.venv/lib/python3.12/site-packages/pyncei/bot.py", line 1010, in values     yield {k: val[k] for k in self.key_order if k in keys}               ~~~^^^ KeyError: 'station'
Is there anything I can do to avoid this or check for this issue before running to_dataframe() to avoid erroring out?

Can you please provide an example of code that produces this error?

Thank you for the quick response!
year = 2023
response = NCEIResponse()
while year >= 2000:
response.extend(
ncei.get_data(
datasetid="GHCND",
stationid='GHCND:USC00140637',
datatypeid=["PRCP"],
startdate=date(year, 1, 1),
enddate=date(year, 12, 31)
)
)
    year -= 1

df_precip_temp = response.to_dataframe()
This is an example of one of the stations that throws this error for me. There are lots, such as GHCND:USC00347390 that work as expected.

This gives a similar error, but with a different: KeyError: 'elevation
x = ["FIPS:56"]
stations = ncei.get_stations(
datasetid="GHCND",
datatypeid=["PRCP"],
locationid=x,
startdate=mindate,
enddate=maxdate,
)
df_stations = stations.to_dataframe()

Both these errors result from bugs in how this library handles missing data. In the first example, it looks like there is a gap in the data for that station between 1951 and 2003; the missing years are causing the errors. In the second, certain stations are missing the elevation parameter. I've patched the issue on GitHub but expect it will be a bit before I do a new release. In the meantime, you can install the GitHub code as follows:
git clone https://github.com/adamancer/pyncei
cd pyncei
pip install .

And here is a version of your code that should catch the missing years. It does require you to use the development code.
response = NCEIResponse()
for year in range(2000, 2024):
    resp = ncei.get_data(
        datasetid="GHCND",
        stationid='GHCND:USC00140637',
        datatypeid=["PRCP"],
        startdate=date(year, 1, 1),
        enddate=date(year, 12, 31)
    )

    if resp:
        response.extend(resp)
    else:
        print(f"No data found for {year}")

response.to_dataframe()

Let me know if that solves the problem for you.

Thanks for digging into this! It works now for USC00140637 and generally seems to be getting a higher success rate, but there are still some stations that are failing. For example:
USC00031459
USC00145870
USC00340017
I have this set up to pull stations within a radius of a point, so these are just a random selection of ones that failed.

Can you please provide the code that is producing the error? When I plug those stations into the code above, it seems to run fine.

Here is my uncommented, data science-esque code in all of its inefficient glory... Maybe I did something wrong and am still using the original pyncei code?
lat = 35.00
lon = -97.05
distance = 125 #km

df_stations = pd.read_csv('stations.csv')
gdf_stations = gpd.GeoDataFrame(df_stations,
                                geometry=gpd.points_from_xy(df_stations['longitude'], df_stations['latitude']),
                                crs='EPSG:4326')

gdf_stations_proj = gdf_stations.to_crs('EPSG:3395')
site = gpd.GeoSeries([Point(lon, lat)], crs='EPSG:4326').to_crs('EPSG:3395')

gdf_stations_proj['distance'] = gdf_stations_proj.distance(site[0])
gdf_ref = gdf_stations_proj[
gdf_stations_proj['distance'] <= distance * 1000]  # Filter for distances within set distance
df_precip = pd.DataFrame()

for id in gdf_ref["id"].unique():
    year = 2023
    ncei = NCEIBot("PRxtmfIOSNHxDEmIjHaYrYfjvUFXnSMN", cache_name="ncei")
    response = NCEIResponse()
    for year in range(2000, 2024):
        resp = ncei.get_data(
            datasetid="GHCND",
            stationid=id,
            datatypeid=["PRCP"],
            startdate=date(year, 1, 1),
            enddate=date(year, 12, 31)
            )
        if resp:
            response.extend(resp)
        else:
            print(f"No data found for {year}")

    df_precip_temp = response.to_dataframe()
    df_precip = pd.concat([df_precip, df_precip_temp])

I am also attaching a copy of stations.csv which is a bulk pull using ncei.get_stations
stations.csv

Hmm I can't reproduce the error without falling back to the release version on PyPI. I'm a little mystified by the error popping up for these stations but not for the station we discussed earlier. Is the traceback the same?
Can you run pip freeze in your command line and locate pyncei in the output? If you've installed it from PyPI, it should show up as pyncei==1.0, otherwise there should be a path to a file on your computer.
And a friendly word of warning--you don't want to share an API token publicly. I tried to obscure it above but it's still in the comment history. Be careful pasting code in a public forum.

Yep, you're right, I didnt install the updated version correctly the first time (still not sure how that station ran, I checked it like 3 times). Anyways, appreciate the help!

Hello, I'm having a similar issue. I checked the version of the package and the version is 1.0. I also checked if NOAA has a request response, it seems that the server is providing data, but the package can't convert it into a data frame.
I would appreciate it if you could help me with this.

I fixed the issue. When the response is 1, it is actually missing. As a result, they make mistakes when stitching data from different years. I modified the code to fix the occurrence of this exception.


Integrated Snap support

[x] I have read the contribution documentation for this project.
[x] I agree to follow the code of conduct that this project follows, as appropriate.
[x] The changes are appropriately documented (if applicable).
[x] The changes have sufficient test coverage (if applicable).
[x] The testsuite passes successfully on my local machine (if applicable).

Summarize your changes:
Adds builtin support for making Snap distributables, and publishing them to the Snap store. Both are accomplished via electron-installer-snap and the result of work done during the September 2017 Ubuntu Rally and January 2018 Snapcraft Summit.
Closes #25.

I was thinking about it, and it would make more sense to link to https://www.npmjs.com/search?q=electron-forge-maker and https://www.npmjs.com/search?q=electron-forge-publisher in the README, rather than enumerate them all and make the Forge README turn into something like awesome-electron-forge (despite the badge that's on the top of the README :smile:).
@develar as discussed in person, you'll need to rename electron-forge-maker-snap to electron-forge-maker-builderSnap in order for people to use it.


fix config locations

i still do not understand what is the point of making $HOME more clutter to maintain, even when we try to make it Clean / Neat
issue is regarding config-location

$HOME/.erdtreerc
make location good , i mean standards of unix as `.config/et/.etrc'

some work around or TIPS which i have been using to tackle it
in Window (coz i use it for now ) :(
in $profile put this
$env:ERDTREE_CONFIG_PATH="path/to/conf/"
in Unix
in .bashrc put
export ERDTREE_CONFIG_PATH=path

Yup I made an issue for this here https://github.com/solidiquis/erdtree/issues/63
I will make this change in the next minor release before monday.

Let me know if you have any further issues :]


Spelling: Change cian to cyan

The documentation uses cian, proper spelling is cyan.

Fun fact: cyan in Spanish is cian. I did merge the pull request for the sake of consistency though.