Standardize units managment in indices

Pull Request Checklist:

[x] This PR addresses an already opened issue (for bug fixes / features)

This PR fixes #639

[x] Tests for the changes have been added (for bug fixes / features)
[x] Documentation has been added / updated (for bug fixes / features)
[x] HISTORY.rst has been updated (with summary of main changes)
[x] bumpversion (major / minor / patch) has been called on this branch
[ ] Tags have been pushed (git push --tags)

What kind of change does this PR introduce?
This explicitly set the 'units' attribute for several indices where it was assumed that the computation would remove them. When using xr.set_options(keep_attrs=True), simple xarray operation will keep the original attributes (duh) and that was problematic for some of the indices. The most common example is all those "threshold-count" indices : the "compare - resample - sum" chain of operation will retain attributes when the option is activated. However, units change from whatever the input was in to "days".  In the declare_units decorator, if units are set on the output, it converts the output to the required units. The error was coming from there : it received "degK" or "mm/day" instead of nothing.  When units are not set on the output, it simply sets the required ones.   This PR fixes those cases, and some others.

Does this PR introduce a breaking change?
No.

Other information:
I did not find a way to properly test this without adding a test for all indices, which I was to lazy to do. Thus, if xarray changes its behaviour, the issue could happen again with the remaining indices that do not set the units explicitly.

While fixing this, I noted a few things:
1) Do we really want declare_units to set the units AND test if they are already set?
I think it sould do either, but not both. I.e. : Either declare units and override whatever is there or not. Or act a as metadata check : the unit-setting action has to be done in the indice.
2) Threshold-count methods could be more flexible.
Now that we have xr.infer_freq, we could open all threshold-count indicators to accept other sampling (regular) frequencies than daily. infer_freq could give the base unit and when the sum in done we put that as units (H => "hours", D => "days", etc). Offsets with multiples (ex: "3H") could be managed by multiplying the output by that factor.
Not sure if there are many use cases though.

I'm currently running tox.
@Ouranosinc/xclim-core I raised two issues in this PR, please chime in if you have time and opinions.

So we have two categories of indices:

unit changing (UC)
non unit changing (NUC)

and two contexts:

keep_attrs=True (KA)
keep_attrs=False (NKA)

UC requires explicit unit specification.

NKA: Done by declare_units
KA: Failure -> requires change to indice function

NUC
-NKA: Done by declare units

KA : OK. Declare units can check dimensionality, but this is typically redundant since we're checking input units already.

I suspect all the degree day indices will need the same approach, no ?
Your question 1 is still open. Probably worth a discussion.

@tlogan2000 @Zeitsperre surtout
After discussion with David, decision has been made to make the following conceptual change to indices and the declare_units decorator:
Indices must take care of assigning the proper units to their output. The declare_units decorator only checks the input data and stores information about the required input units/dimension. It doesn't modify the output.
The problem was that declare_units was acting on 2 levels : checking for user errors and for developer errors, the latter being undesirable. It's role in assigning units when they were not present on the indice output was prone to error in some cases, especially with all the  keep_attrs business, as raised above. To keep the unit flexibility* in indices, implementing a "smart" unit management code outside of the individual indice seemed to complex a task, so we decided to leave it to the indice developper to think, test and solve all unit cases.
* By unit flexibility, I mean the possibility, for example, to input tas in °F and retrieve °F, in a simple temperature indice. Or even input "°F" and retrieve "delta_F days" in the output of a degree-days indice.
What that means is numerous little changes to ensure good units on indices ouputs and the removal of the out_units argument of declare_units. Output units will have to be explicitly explained in the docstring.

I'm not sure I completely agree with this .. but might need a more concrete example.
Suppose I set a xr.set_options(keep_attrs=True) globally then proceed to calculate xci.wetdays on precip. What is in the output as a units attr?   If it remains mm/s then I think this is a bit of a step back for the code.

The goal is to be independent of the keep_attrs setting. Your example currently fails if keep_attrs is True. We want to force the unit attribution in the indice so that doesn't happen. The wetdays example is simple because that means only a simple : out.attrs['units'] = "days", as done in the PR. For more complex cases, like degree-days, a bit more thought must be put into the code, in order to keep the flexibility in input units.

The goal is to be independent of the keep_attrs setting. Your example currently fails if keep_attrs is True: declare_units raises an Error as it receives "mm/s" units.
We want to force the unit attribution in the indice so that doesn't happen. The wetdays example is simple because that means only a simple : out.attrs['units'] = "days", as done in the PR. For more complex cases, like degree-days, a bit more thought must be put into the code, in order to keep the flexibility in input units.

Ok I think I have my head around it now.

Ready for review!
This PR will break everything that expected a specific unit in the output of indices, that might be a lot! The indicators are not impacted, and that API should be considered more stable anyway.
So now indicators take care of attributing the final units themselves. In a lot of cases, this mean more flexibility than before, often more than the title suggest. I added a few functions to help:
xclim.core.units.to_agg_units
Infers the sampling frequency of the input and uses that to get the appropriate units of the aggregation. Example : if a "count" operation was applied along a daily dataset, like say "tx_days_above", the output unit is "day". The same operation on hourly data returns "h". It also covers the "6H" case by multiplying the data by 6 and setting "h" as units.
Similarly for "delta_prod", for operations like (a - thresh_a).integrate('time'), such as degree days. Daily temperature in "K", gives "K d". Or 6-hourly temperature in "°C" gives "h delta_degC".  Note that I modified pint2cfunits so that delta_degree units are kept as such. It should be cf-compliant and reduce unit issues.
This function allows a lot of indices that were written for daily data to be more flexible and work with any sampling frequency that is convertible to a unit (ex:  QS-DEC doesn't work, "season" is not a temporal unit.).
For now, it defaults to assuming daily input when the frequency inference fails (xr.infer_freq(da) is None). As such, it doesn't change the previous behaviour, but that seems inelegant at best for me...  Should we raise an error instead? That would mean that badly constructed timeseries would fail. Main case : data converted to a different calendar using convert_calendar but without padding missing dates with NaN. I.e : a "365_day" converted to "366_day" will miss the 29th of Feb and its frequency will not be inferrable.
xclim.core.units.rate2amount
Useful to go from precipitation rate to amount. Multiplies by the sampling period (daily => * 1 d). If the sampling period cannot be found (xr.infer_freq -> None), computes it point by point (da.time.diff('time')). Units are reduced on the output (` "mm / d" * 1 h => "mm").
Docstrings
Docstrings have been modified so that the output dimension or units are listed in the "Returns" section, after the type annotation. I didn't write these in a parsable way, but rather a the most specific indicator I can give given the new flexibility.
Ex:
Returns
----------
xarray.DataArray, [dimensionless]
   <Long name ....>
# or
xarray.DataArray, [same as tasmin]
# or
xarray.DataArray, [temperature]
"""

Passes all tests on all flavors, except py37-bottleneck (which is broken), py37-windows and macOS (duh).
@Zeitsperre  in the last commit, I changed the config of pydocstyle to skip the whole /tests/ directory. Also changed the "black" tox env so that it installs xclim normally, pylint was raising import-error issues everywhere otherwise. Oh and excluded the tests directory in black.

@Zeitsperre in the last commit, I changed the config of pydocstyle to skip the whole /tests/ directory. Also changed the "black" tox env so that it installs xclim normally, pylint was raising import-error issues everywhere otherwise. Oh and excluded the tests directory in black.

Thanks for running tox against a bunch of environments. It's been very helpful to have that in the absence of Travis CI. For pylint, given that tests are installed as a pseudo-module, it requires that Xclim be installed. I can see that it's in the doctests build already, so rather than install xclim in the black build, just remove pylint from black. Surface-level code style checks shouldn't require xclim to be installed.
Excluding tests from black isn't what I'd do, but I don't see an issue with doing that. LGTM.

Excluding tests from black isn't what I'd do,

Woups you are right, I didn't think this completely through...  Thanks!

@huard Review when you have time, also I think I'll prefer to merge this first before finishing #634 as they both act on the same code. I'll start testing against finch after merging this also.
As a summary of difference between the two PRs:

This PR tries to maximize unit flexibility. It removes output unit declaration and makes sure units are handled and set explicitly (or almost) in the indice function. The output unit information is moved to the "Returns" section of the docstring, but in a human-readable manner only.
#634 propagates input units info to the indicator. It removes all input units info from the indice docstring but includes them in new generated indicator docstrings.

@huard Added doc examples where needed.
@tlogan2000  I realized I didn't finish the standardizing work in _anuclim.py, it is done now. I also saw two indices that were computing total precipitation using the src_timestep arg and multiple conditionals, all to convert rates to amounts. I replaced the blocks with my new rate2amount function, that should hopefully behave the same. Except that it infers the source timestep OR computes it from the time axis.
And for all: I made another "breaking" change : Instead of relying to what is hardcoded in the functions, declare_units now reads the output's units, parses them with pint and rewrites them to string with pint2cfunits. This ensures more consistency. We can discuss that in a few minutes.


Apiml start script variables

changes for latest round of externalized variables

@janan07 Hello Andrew, hope you had great Easter! Would you find time to look at this PR please?

Hi @janan07, are there additional changes or is your review complete?

Reviewed and edited by tech info

Much appreciated!


fix: added conditional rendering for overlay element

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 62.277%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
227

Relevant Lines:
346

💛  - Coveralls


fix(obstacle_stop_planner): fix debug topic name

Related Issue(required)
The plotJuggler config can not be loaded correctly.

Description(required)
Fix topic names used there.

Review Procedure(required)

Related PR(optional)

Pre-Review Checklist for the PR Author
PR Author should check the checkboxes below when creating the PR.

[x] Read commit-guidelines
[x] Assign PR to reviewer

If you are adding new package following items are required:

[ ] Documentation with description of the package is available
[ ] A sample launch file and parameter file are available if the package contains executable nodes

Checklist for the PR Reviewer
Reviewers should check the checkboxes below before approval.

[ ] Commits are properly organized and messages are according to the guideline
[ ] PR title describes the changes

Post-Review Checklist for the PR Author
PR Author should check the checkboxes below before merging.

[ ] All open points are addressed and tracked via issues or tickets

CI Checks

Build and test for PR / build-and-test-pr: Required to pass before the merge.
Build and test for PR / clang-tidy-pr: NOT required to pass before the merge. It is up to the reviewer(s). Found false positives? See the [guidelines][clang-tidy-guidelines].
Check spelling: NOT required to pass before the merge. It is up to the reviewer(s). See here if you want to add some words to the spell check dictionary.

Could you also rename /control/trajectory_follower/velocity_controller/debug_values/data.N to /planning/scenario_planning/lane_driving/motion_planning/adaptive_cruise_controller/debug/debug_values/data.N

Could you also rename ...

Roger. And I've added its module name on the topic namespace.

@TakaHoribe
Could you fix clang-format?

pre-commit.ci run

SORRY


Remove focus & hover states from bg-variant mixin

About this code:
https://github.com/twbs/bootstrap/blob/8896827bd072a120705a0576a5bcde69c8ce2cda/scss/mixins/_background-variant.scss#L9-L14
I think we should remove it. IMO focus and hover background states don't belong in these utilities and we are also not defining an active state here. We have the buttons with backgrounds, if this doesn't cover the needs I think the developer will need to write custom code, because it's quite specific.
I guess we can just remove the mixin, just use the background-color property in v5 and use the deprecate mixin to deprecate this in v4.
Thoughts?

Big fan of this. Nix the mixin and the hover/focus states. We'll inevitably get the question of how to set hover styles via utilities, but I don't think that's a path we should go down. Maybe provide some guidance in our docs though for how to customize that kind of stuff?

Maybe provide some guidance in our docs though for how to customize that kind of stuff?

If we ditch the mixin, we can use the utility API https://github.com/twbs/bootstrap/issues/28367 (or whatever it will be called) and  clarify this in its documentation section.

We'll inevitably get the question of how to set hover styles via utilities, but I don't think that's a path we should go down

I agree, I experimented with hover utilities before, but you'll end up with elements tons of utility classes. It's just better to create a custom component if an element needs hover states to get a better overview of all the properties.


sbteclipse doesn't support scala compiler version setting

Based on scala version sbteclipse should append following properties into
e.g. for scala version 2.10 projects
.settings/org.scala-ide.sdt.core.prefs
scala.compiler.additionalParams=...-Xsource\:2.10
scala.compiler.installation=2.10

Here's the same bug for gradle: https://discuss.gradle.org/t/eclipse-plugin-does-not-set-scala-version/17852

I develop scala with maven in eclipse.
Eclipse's setting scala compiler version(also used in build path) should be the same with scala-library version, or cause "Error: Could not find or load main class",  I always change build path by hand to match scala-library version, it's fussy.
sbteclipse supports compiler version setting will be better.

/cc #239 which seems to contain a partial fix for this, but will only work for exact version 2.10, which is unfortunate.  Ref commit 1bbf64c0cf8a9942afb56bcd228179d1019f0b9e .

Any news on this one? I hit it too.


Probability test framework didn't catch bug with vectorization

Description
This pull request went in: https://github.com/stan-dev/math/pull/1331
And later was reverted: https://github.com/stan-dev/math/pull/1530
@mcol had a look why the testing framework did not catch the problem: https://github.com/stan-dev/math/pull/1752#issuecomment-596207740
It looks like the code here tests in psuedo-code:
N * logp(a) = logp(rep(a, N))

We should also test the more general:
logp(a) + logp(b) + ... = logp({a, b, ...})

This issue is a two parter:

Confirm that such a test would actually have caught https://github.com/stan-dev/math/pull/1331
If it would have, then confirm this issue and make a test. If it doesn't, delete this issue.

Current Version:
v3.2.0

This was a duplicate but it seems to reference a different issue: https://github.com/stan-dev/math/issues/1978


[Bug] Metadata for many USI mods should not require Community Resource Kit.

Background

Operating System:  (K)Ubuntu Linux
CKAN Version: 1.30.4
KSP Version:  1.12.3

Have you made any manual changes to your GameData folder (i.e., not via CKAN)?
No
Problem
Describe the bug
Many USI mods require Community Resource pack when they shouldn't.
Many are inheriting this from USI Core, which does require it.
Desired behavior
In all but a few necessary cases, USI Core should be considered recommended instead of Required, as should Community Resource Pack.

The netkans for USI mods are all self-managed in https://github.com/UmbraSpaceIndustries/CKAN. Please open the issue there.


Change agreement/terms text

In gitlab by @A-Lawrence on Aug 20, 2016, 18:28
The text for the agreement/terms is currently worded incorrectly.
Current wording is:

I confirm that I have read the Visiting & Transferring Controller Policy
Your application will be rejected if it is later found you haven't read this document.
I confirm that I have consolidated my rating by controlling for 50 hours in my home division
A rating that has not been consolidated cannot be considered for a visit or transfer.
When consolidating my current rating, the hours were spent on a suitable position for my rating
For example, if you are a C1 your 50 hours must have been spent on CTR or FSS type positions.
I last transferred region, division or VACC in excess of 90 days prior to the start of my application
Applicants may only transfer regions, divisions or VACCs once every 90 days.
I will complete my local induction plan (if required), or attain full competency within 90 days
Any application not completing this induction will be transferred back to their previous region/division.
I confirm that I am not presently staff in my home division OR that if I am staff, I will relinquish my role if successful
You may only hold a staff position in your home division.

Will await review by @bjoplin

In gitlab by @bjoplin on Aug 20, 2016, 18:35

I confirm that I have read the Visiting & Transferring Controller Policy
Your application will be rejected if you do not meet the requirements of this policy.
I confirm that I have consolidated my rating by controlling for 50 hours in my home division
A rating that has not been consolidated cannot be considered for an application to visit or transfer.
When consolidating my current rating, the hours were spent on a suitable position for my rating
For example, if you are a C1 your 50 hours must have been spent on CTR or FSS type positions.
I last transferred region, division or VACC at least 90 days prior to the start of my application
Applicants may only transfer regions, divisions or VACCs once every 90 days.
I will complete my local induction plan (if required), or make every attempt to attain full competency within 90 days
Any application not completing this induction will be transferred back to their previous region/division.
I confirm that I do not hold a staff position in my home division OR that if I do hold a staff position, I understand that this must be relinquish if my application to transfer is successful.
This only applies to applications for transfer - visiting applicants may retain any staff position within their home Region/Division.

In gitlab by @A-Lawrence on Aug 20, 2016, 19:03
Removed ~160393 label

In gitlab by @A-Lawrence on Aug 20, 2016, 20:16
Removed ~98471 label


Difference between code and paper for sequence construction in AR-RNTR

Hello! I found that the implementations of category and index of parent for sequence construction in AR-RNTR are different between your released code and the paper. Are they equivalent?

Hello! I found that the implementations of category and index of parent for sequence construction in AR-RNTR are different between your released code and the paper. Are they equivalent?

"Seq2bznodelist" seems to have not been called

Hello! I found that the implementations of category and index of parent for sequence construction in AR-RNTR are different between your released code and the paper. Are they equivalent?

"Seq2bznodelist" seems to have not been called

@EchoQiHeng
It is called in simple_test_pts function. Besides, all of the topo types are set to start, continue, fork and merge instead of  Ancestor, Lineal, Offshoot and Clone. And the integer for the index of parent are implemented as fork from and merge into in the code.


[1.18.2] Server crash while generating new terrain.

Describe the bug
Server crash while generating new terrain.
To Reproduce
Load a ridiculous number of mods, then run around in server to generate new chunks.
Log
[crash-2024-03-30_20.24.59-server.txt]
(https://github.com/DynamicTreesTeam/DynamicTrees/files/14813698/crash-2024-03-30_20.24.59-server.txt)

you have 40+ mods that are not running...


See error during Docker build

When building using docker seeing the below issue. using Docker version 17.06.0-ce, build 02c1d87
on Ubuntu 16.04

See also "/tmp/eos/build/CMakeFiles/CMakeOutput.log".
The command '/bin/sh -c cd /tmp && git clone https://github.com/EOSIO/eos.git --recursive   && cd eos && mkdir build && cd build   && cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_STANDARD=14 -DCMAKE_INSTALL_PREFIX=/opt/eos ../   && make -j$(nproc) && make install   && rm -rf /tmp/eos*' returned a non-zero code: 1

Digging a little deeper, this is the CMake error

-- Configuring Eos on Linux
CMake Error at CMakeLists.txt:162 (add_subdirectory):
The source directory
/tmp/eos/contracts
does not contain a CMakeLists.txt file.

I just added an empty CMakeLists.txt file, which solved this problem for me.


add awesome hosts

:+1: for the awesome hosts file list

Please read the contributing guidelines and the awesome manifest.
Especially this part:

Wait at least a couple of weeks after creating a list before submitting to give it a chance to mature.

Ping me here in a few weeks and I'll reopen ;)

Each items in your list could use a description.


Xcode 8.1

Created Swift bug: https://bugs.swift.org/browse/SR-3081

Great! Waiting for release )


bug(preimage): Investigate oracle reader flakes

Overview
The oracle reader tests flaked in this workflow: https://github.com/ethereum-optimism/kona/actions/runs/8541657676/job/23401491592
Investigate the failure and fix it. Pretty sure this is just a race condition in the tokio test, though we should expect that the pipe properly blocks.

Closed w/ #96

Just experienced this flake again locally. Re-opening so we investigate again.

Haven't seen these in a good while after the latest updates. Closing for now.


Adds support for it.only expressions.

Fixes #29.

Thanks! This is really useful :D
alas, I have some bad news though, this code has moved into Jest here - I am planning on migrating this repo to the new code today hopefully - so this PR  will need to Jest I'm afraid

https://github.com/facebook/jest/pull/2289

nice one - thanks @bookman25

@bookman25 I'm going to add you as an repo collaborator, no extra pressure or anything - but I don't want you to feel blocked if you need to do something

and thanks @mute


Inventory state tracking

In addition to using global vars to track items in Twinsen's inventory, each inventory slot also has a "state". This is used to determine which variant of an item Twinsen holds. The affected items are:

robes vs wizard robes
blowgun vs blowtron
protopack vs jetpack
etc

These changes implement support for tracking that state. Individual commit messages have details.
Preview here: https://pr-649.lba2remake.net

btw @fraserjgordon you can merge the code yourself as long you have at least 1 approval.


[DCOS_OSS-4245]  (WIP) Left align owner column

Remove uid from propsToRight in getClassName()
Closes DCOS_OSS-4245

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Hey @sharynneazhar it works! Thank you for your contribution 🙇 Let us get this through the CI... 😅

:tada: This PR is included in version 2.36.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Portal: Deleting a department should "delete" its membership

Instead of preventing deleting a department if it has membership, deleting should simply flag all memberships as "ended" (#302), and then the department should also be deleted (or similarly soft-deleted/hidden).

Test and confirm please

Deleting Dock (which is in fact supposed to go away) failed with a 500
error.
On Fri, Oct 18, 2019 at 10:00 PM Aric Stewart<EMAIL_ADDRESS>wrote:

Test and confirm please
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/CON-In-A-Box/CIAB-Signin/issues/303?email_source=notifications&email_token=AAISDXE2QJYLTDB6IOSZZLDQPJZ3TA5CNFSM4JA6SIAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBW47HQ#issuecomment-544067486,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAISDXC4E6BGFMADT7GJTLLQPJZ3TANCNFSM4JA6SIAA
.

Correction made, please try again.

This now appears to work! Yay!
On Sat, Oct 19, 2019 at 8:18 PM Aric Stewart<EMAIL_ADDRESS>wrote:

Correction made, please try again.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/CON-In-A-Box/CIAB-Signin/issues/303?email_source=notifications&email_token=AAISDXBMZFC2ZPTLYPR2ET3QPOWXJA5CNFSM4JA6SIAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBYAAIY#issuecomment-544210979,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAISDXCNZTUDTPFR2YJHA43QPOWXJANCNFSM4JA6SIAA
.


Allow twilio whatsapp verify channel

Not ready yet hence creating a draft PR as i'm going to sleep, will add more commits in subsequent days/weeks.
But wanted some of your opinions 1st on your preferences on how you would prefer me to structure things.
My goal is this:
https://www.twilio.com/docs/verify/whatsapp
Then if it work well we can try add Vonage/Nexmo 1 which allow you to just press Yes or No in whatsapp to verify the phone number. Without the need for OTP code
https://developer.vonage.com/en/verify/verify-v2/guides/using-whatsapp-interactive

hmm nvm i shall close this 1st


audit: unused imports

Bug on the scarb test. We are obliged to add the debug::PrintTrait in order to initiate the testing process.


Default chunking has changed

With xarray > v2023.09.0, the chunking behaviour of xarray's open_dataset has changed. Now:

chunks="auto" returns variables with chunk sizes as multiples of the chunks in the nc file,
chunks={} returns variables with the exact nc chunk sizes.

The chunking tutorial in the documentation does not reflect this, but should.

See https://github.com/intake/intake-esm/issues/632


Search Old Codes

We literally have a slack channel called #good-codes where we end up sharing/saving code snippets. I would like to add 2 commands into this for saving and searching old codebases.
The architecture would be:

Write to my/org github gist whenever I save a snippet
generate some docstrings for it automatically
Embed the doc string
and have a search feature.

Would work with 2 additional shortcuts Cmd+option+S and command+space
Replaces bulky software like pieces.app

Are you proposing a native integration with slack for searching code snippets?

Please repost with a clearer title and description. It seems like you are just proposing the ability to search over a codebase.


coordinates of structure 4 and 5

Is anyone doing the 4 and 5 structure? If so, what are their coordinates?

ok. 4 and 5 seem tough to understand its geometry.


Migrates e2e tests from ginkgo,gomega to testing,testify

To call ginkgo,gomega indirect would be putting things lightly. A combination of an indirect testing library and long-running tests are a recipe for frustration. I've begun a process to remove ginkgo,gomega with a couple tests
that I had been looking at.
See #127
Some benefits

Before, we both had tests using normal testing,testify and also ginkgo,gomega. Now we only have way to write tests: testing,testify.
testing,testify are more popular and easier to learn than ginkgo,gomega. testing is the standard library described in all golang books.  testify is a simple assertion library 10x more popular than gomega in terms of star count. Neither require learning BDD concepts.
Before, our running of e2e tests was significantly different than our other non-gingko tests in the same project. It literally required compilation of a platform-specific binary named 'e2e', adding a lot of complexity to CI and local setup.
Before, individual tests cannot be executed via normal means, such as Goland. Instead, you have to run the suite file. Now, you can execute and more importantly debug individual tests instead of via a suite wrapper.
Before, writing tests not only required understanding of the subject, but also the ginkgo runtime. Otherwise, statements like var _ = Describe("getenvoy extension init", func() { seem pure magic.
Before, the nature of test execution required a combination of understanding the go platform and also ginkgo
Before, updating Ginkgo/gomega would skew the dependency tree, due to their dependencies on two protobuf libraries. Now, the test runtime does not interfere with main code.
Before, the same test took more lines of code to achieve the same table test. Less code is less to maintain.
Some other notes about migration off are captured in this issue https://github.com/openservicemesh/osm/issues/1704

Notable differences
Gomega had a different way of logging. For example instead of using a logging library, it would use statements like this:
By("changing to the output directory") which translate into output STEP: changing to the output directory. This is mainly to satisfy BDD idioms and is meant to be interpreted amongst the many lines of normal log output. For cases where the points inside the test should be logged, we should use normal logging instead. However, I don't recommend making log statements unless they are meaningful and clarify progress.
Also, those not used to it will notice use of normal go defer commands instead of the before/after test hooks used in tools like gomega. This can take getting used to, but it is also the same as normal go code, so shouldn't be time lost. There are some cases where multiple tear-down steps are needed and those steps are more visible now. That's not necessarily bad because these typically are also the most complex tests and have a lot going on. Seeing the various setup requirements shows problem areas that help knowing when debugging.

Here's an example of the tests executing in Goland

+1 Personally, I'm all in for this migration, and I've been frustrated too.
I would like to have @yskopets's opinition.

@codefromthecrypt

A combination of an indirect testing library and long-running tests are a recipe for frustration.

Can you explain what it means ? What has changed after this PR ?

While the rationale is fairly straightforward, the following issue captures a lot of it nicely

Personally, I don't agree with this rationale.
I don't see from the linked issue how much support it received.
It's labeled with needs discussion for a reason.
If I understand the linked issue correctly, it has only 1 argument - "Let's move to Go std library because Ginkgo/Gomega is not a Go std library."
Sure. Developers choose Ginkgo/Gomega when they want to do BDD-style testing (which is uniform across all programming languages). If Go std library cannot offer it, developers have to switch to other libraries that can.

I've begun a process to remove gomega with a couple tests that I had been looking at.

I'm looking at the replacement code of this PR and I don't understand what component has become better now:

is it more readable ?
does it give better assertion messages when the test fails ?
does it leave a better trail in the test log ?

I don't mind finishing the e2e suite

It's a team/community work.
Please be mindful, though, about people who will have to maintain these tests.
If you're not sticking around to maintain the refactored code, please consider the effect of such changes on other developers.

The highest value from this so far is that the nested test is directly executable now, as opposed to unrunnable in Goland, for example. I would like to hear from others besides you @yskopets because I have noticed a lot of projects copy/paste things even when they don't agree with it. There are likely more people who have only tolerated this gomega thing due to difficulty getting rid of it.

@mathetake I'd like to merge what's here. Later, I will do a PR to finish the rest, but landing incremental progress makes debugging #129 far easier, which is what started all of this really.

a different test failed while looking at go:embed and debuggability problems are the same as the one I've already converted. I'll go ahead and finish this PR vs doing things piece-wise. I don't usually like huge PRs, but I like whack-a-mole less

found this bug while visiting the push tests I'll rebase once it is merged https://github.com/tetratelabs/getenvoy/pull/147

will be another day or two to finish. the "run" tests are rather gnarly so saved them for last :p

the "getenvoy extension run" unit tests are now done. I still need to do the e2e for that, and that's probably where I'll stop as the "getenvoy extension" tests were the more difficult ones I ran into, leading to this issue. I'll update soon and probably separate out the "unit tests" from the e2e.
I quote "unit tests" as they aren't really like normal unit tests, as they do invoke OS commands etc. The whole unit test run could be much faster if this was faked out, resulting in less temp dirs and process calls. I won't have time to address that, though.

I pulled the unit test changes into a separate PR as they were instrumental in figuring out what was going on in the e2e tests. https://github.com/tetratelabs/getenvoy/pull/151
This test is purely addressing the e2e, but other benefits such as less dependencies won't be realized until other tests stop using gingko/gomega. Sadly, just this part took so much time I don't have time to port all remaining tests. OTOH, it should be far easier with this and #151 in.

run phases fail on timeout locally and in CI for rust. I'll look into it, but I raised an issue as generally speaking everything rust related takes a huge amount of time. We should probably bump the matrix so that the language e2e runs is managed externally and defaults to tinygo https://github.com/tetratelabs/getenvoy/issues/152

kicked e2e tests due to #145 (current script we are using routinely times out installing docker on osx)

green build now. @mathetake PTAL

I believe the value props @codefromthecrypt mentioned are now achieved, and e2e tests now become much simpler than before. Notably, 1) not using Ginkgo or any related stuff would definitely lower the onboarding cost for folks both from Go and other languages including me, and also 2) the better integration with IDEs from zero. LGTM to me now.

thanks for the review and pointers @mathetake, as well the support behind the scenese. this is definitely better now

re-kicked the osx job as it is failing for the usual reason:
Waiting for docker service to be in the running state. 	If Docker is not ready within 10 minutes, it's better to fail the current CI Job and let the comitter to re-start the job manually

@musaprg any comments?

Personally, +1 to this PR, though I'm not sure it would be the best way in a community. I prefer the standard way in Go and I think it makes current tests more simple and readable.

also, ginkgo/gomega is a little bit complicated for contributors who new to them, for me at first as well.

note that now we can run individual test in a click of button


Reworked current delete command for vendor delete

The command format is updated as per mentioned in #77
delete v/1

Deleting multiple items was not implemented due to the complexity of testing
The command is extended in such a way adding delete for events should be trivial

no idea why my codecov no passing despite adding test cases that accounts for it

You might want to open an issue for this PR just to follow the workflow requirements for the tP. Just admin work—the PR can remain as is.


Blog: Why you need a low code platform

Description

Related Issue(s)
close https://github.com/FlowFuse/website/issues/2059
close https://github.com/FlowFuse/website/issues/2060

Checklist

[ ] I have read the contribution guidelines
[ ] I have considered the performance impact of these changes
[ ] Suitable unit/system level tests have been added and they pass
[ ] Documentation has been updated

@Hasmin-AC,  I added some customer success stories.  Can you review and merge if you are happy?  I am off the rest of the day and would like this to go out today.

I think its actually better to summarize the stories rather than direct quoting them under a title like How enterprises benefit from low code.

@Yndira-E Can you change the tile please to match the new title?

@Hasmin-AC

I think its actually better to summarize the stories rather than direct quoting them under a title like How enterprises benefit from low code.

Done

@gdziuba

@Yndira-E Can you change the tile please to match the new title?

Done


First test scenarios for Flux gradients

Work done during JuliaCon Hackathon; preparing test for neural networks

Thanks so much for your contribution! It may take a few days but I'll get back to you, give you some feedback on the code and take it from here if that's okay. I'm sorry that the task was so daunting, but very grateful that you agreed to give it a try :) And if you need any more resources on AD, let me know

Codecov Report
Attention: Patch coverage is 83.33333% with 1 line in your changes missing coverage. Please review.

Project coverage is 40.28%. Comparing base (0db3be8) to head (1e29dd6).

Files
Patch %
Lines

DifferentiationInterface/test/runtests.jl
83.33%
1 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (0db3be8) and HEAD (1e29dd6). Click for more details.
HEAD has 42 uploads less than BASE

Flag
BASE (0db3be8)
HEAD (1e29dd6)

49
7

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #352       +/-   ##
===========================================
- Coverage   96.91%   40.28%   -56.63%
===========================================
  Files         101       71       -30
  Lines        4824     2626     -2198
===========================================
- Hits         4675     1058     -3617
- Misses        149     1568     +1419

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@CarloLucibello I got the first batch of tests working with Zygote. The syntax is a bit counter-intuitive because DI doesn't yet support multiple arguments, so I have to put the data x inside the loss closure, but it gets the job done for now. The Enzyme tests fail on two cases though, the ones involving Flux.Scale and Flux.SkipConnection. Any idea what could be different about those two? @wsmoses could this be one of the cases where we want to Duplicate the function f itself for the time being?

Actually this is probably a case of exactly the opposite in which doing so would be harmful. Marking the closure as differentiable would mean that Enzyme would also end up computing the derivative of the input [and then throwing it away].

Per the error message, it looks like the closure is causing issues where it can't be compiled due to mixed activity [e.g. one element of the closure being differentiable the other not]. As the error message states, you can enable runtime activity to resolve at a slight [but not asymptotic performance loss]. But also using multiple arguments here in Enzyme directly would also presumably resolve this

Per the error message, it looks like the closure is causing issues where it can't be compiled due to mixed activity [e.g. one element of the closure being differentiable the other not]. As the error message states, you can enable runtime activity to resolve at a slight [but not asymptotic performance loss].

I tried enabling runtime activity but the stack traces just got angrier 😱

But also using multiple arguments here in Enzyme directly would also presumably resolve this

Agreed, but setting this up in the single-argument case also helped me resolve some other issues to pave the way for non-array testing, so it's a great first step. For the multi-argument case, what I need most of all is inspiration and ideas to improve my initial API proposal at https://github.com/gdalle/DifferentiationInterface.jl/issues/311#issuecomment-2228066950. The idea is to be slightly different than Enzyme, and closer to the mathematical concepts that users may be familiar with.

Merging this without the Enzyme tests because there's already some useful stuff in there.


chore: Bump dependency versions

Signed-off-by: Vyom-Yadav<EMAIL_ADDRESS>
Purpose of PR?:
Bumps Dependency versions and go version in:

pkg/KubeArmorHostPolicy
pkg/KubeArmorPolicy

Bumps go version in CI tasks.

Thanks for looking into this.
But I don't think we need to update deps for the said pkgs. They are being deprecated in favour of pkg/KubeArmorController.

Thanks for looking into this.
But I don't think we need to update deps for the said pkgs. They are being deprecated in favour of pkg/KubeArmorController.

These dependencies are referenced in github.com/kubearmor/KubeArmor/tests which is imported in github.com/kubearmor/discovery-engine/tests. So either the above-mentioned go modules should be made free from these dependencies or we need to update them.
These dependencies prohibit updating dependencies in other packages.

The primary problem is the usage of github.com/kubearmor/KubeArmor/tests in github.com/kubearmor/discovery-engine/tests.

Can we migrate deps in those (or create an issue for the same) to leverage pkg/KubeArmorController instead? It has all the methods that other 2 packages export.

we've already an open issue with PRs for it #1113 @Vyom-Yadav PTAL and we can close any one.

we've already an open issue with PRs for it #1113 @Vyom-Yadav PTAL and we can close any one.

Since that one removes the dependency from KubeArmorHostPolicy and KubeArmorPolicy and updates the dependencies too, that one solves more problems.
Please bump dependencies in that PR to the latest compatible, that PR hasn't been updated in a while. You can close this one.

@rksharma95 @Vyom-Yadav is is possible to have this or PRs in #1113 merged before the upcoming v0.10 release this week?
FYI, @Aryan-sharma11 is also updating repo wide dependencies in #1218 before adding renovate bot. However in favor of the above, I think we can change that PR to not update the controllers. Thoughts?

@DelusionalOptimist yes definitely we can get PRs in #1113 merged asap. I will make it a priority for today.


Fix Rails 5.2 incompatibility

Following my comment on PR #3, I propose this change to fix the error raised on app loading with Rails 5.2 :
NoMethodError: undefined method `sqlite3=' for ActiveRecord::Base:Class
.../rails_wordpress/lib/rails_wordpress/railtie.rb:9

My opinion is that the gem should not change the belongs_to_required_by_default config of its parent application. In my company, we use rails_wordpress as a Rails engine in a larger app, with another database and other models living aside. They should not be concerned by this setting. So I suggest to replace this global config by individual optional: true on every belongs_to association declaration (as suggested by @brijeshror himself in his initial PR #3). I added a test on creating a Post without parent.
I also update the Rails locked dependency (used in tests) to latest 5.2 version.

It would be good if this could get sorted out as we are trying to use this under Rails 5.2 as well.


Use get-function for calling Sass functions

What
src/tools/_font_url.scss and src/tools/_image_url.scss should use get-function instead of passing a string directly; e.g. @return call(get-function($govuk-font-url-function), $filename);
Why
get-function($string) is deprecated and adds loads of warnings to the log
DEPRECATION WARNING: Passing a string to call() is deprecated and will be illegal
in Sass 4.0. Use call(get-function("image-url")) instead.

I have a PR for this, but I can no longer push it.

Thanks for raising Alex!

I have a PR for this, but I can no longer push it.

Are you able to open a pull request from a fork?

@36degrees I am, but I was to lazy to cherry pick the commit. Will do that.

Thanks Alex – appreciate it!
To avoid cherry picking I think you should be able to do this by forking the repo in the GitHub interface and then adding it as a new remote (renaming the existing remote to upstream):
git remote rename origin upstream
git remote add origin git@github.com:alex-ju/govuk-frontend.git
You should then be able to push your branch – I think…!


PR build doesn't show any results in ADO pipeline under scan tabs

I am using folloiwng tasks in ADO pipeline. It works fine and publish the SAIRF test results under Scans tab  with branch build but same thing doesn't work for PR build.

task: Gitleaks@2
displayName: "Scan secrets > Gitleaks"
inputs:
scanlocation: '$(workdir)'
configtype: 'predefined'
predefinedconfigfile: 'GitleaksUdmCombo.toml'
reportformat: 'sarif'
taskfail: false # true Fail the task if secrets are found
verbose: true

Branch build pipeline output

PR build pipeline output

Please suggest.

So, after checking logs it shows:
Only commits belonging to this Pull Request are scanned.
When we run gitleaks on branch level it scans all the commits and code but for PR branch/build it scans only PR commit.

Can you tell me what you expect and what you get?
Also check the readme and check the scanmode:
all' will scan all commits.
'prevalidation' will scan only the commits that are part of a Pull Request.
'changes' will scan only the changes between this build and the previous build.
'smart' will detect the best scanmode.
'nogit' will run GitLeaks in no-git mode (flat file scan).
'custom' will allow you to provide custom -log-opts.

@JoostVoskuil  Thank you !!! yeah scanall : 'all' worked and published all the results.
Thank you again !


Celestia Core Devs Call 14

Celestia Core Devs Call 14

Meeting Information
When: Wednesday, July 24, 4:00 PM UTC, 60 minutes max
Where: Youtube Livestream
Agenda

Roll Call

Consensus:

Core/app Team
Strangelove
IBC team
Binary Builders
Informal

DA

celestia-node
Eiger

Working Groups Updates

Updates from DA Working Group
Updates from Interface Working Group
Updates from ZK Working Group

Updates on CIPs (WIP):

CIP-23

Reminder to revisit CIP-21 and CIP-23 and potentially move them to next status.

CIP-22 moved to review.

Updates on Lemongrass hardfork:

We plan to cut celestia-app v2.0.0 by the end of this week.
celestia-app v2.0.0 should be supported by celestia-node v0.16.0 which is ETA August 1.
In early August, we plan to roll out celestia-node v0.16.0 and celestia-app v2.0.0 to Arabica.
Conduit hasn't been able to test celestia-app v2.0.0 release candidates on an ephemeral testnet yet.
If you run a roll-up, indexer, or service on top of Celestia: please deploy to Arabica and Mocha so that we can collect signal on if your roll-up, indexer, or service works as expected after the Lemongrass hard-fork is activated on those testnets.


Logo for vagga

I've started a contest for a logo on 99designs: http://99d.me/c/hgpa
This is a place to discuss options there (just started, so nothing to see at the moment), but you can also take a look at the text or maybe add more logo ideas here.

Here is the link to current contest entries: https://99designs.com/contests/814798/entries
I will create a poll later on, so you can choose and provide feedback easier later (it only first day of a contest yet). But feel free to comment on this issue.

Here is the preliminary poll: https://99designs.com/contests/poll/3bf78334c0
Please provide some feedback! Rating by the stars helps, commenting on the design will help more.


CI

Continuous integration should be setup to use a Concourse pipeline. This should follow the style of other cloudfoundry projects, such as s3cli or bosh-cli, using a dedicated ci directory.

#8 is a blocker.


Fix #2870

This prints a warning message if the memory set by virtualbox is below 640mb.

Hm, I think perhaps it might be better to document that we don't recommend running VMs with less than, say, 512MB of memory.  I think it's reasonable to draw a boundary somewhere, for instance in some situations we require proper network access for Machine to function properly.

Let's do some doc then...


Add fix for live video content sidebar being inverted

Added selector for an issue where all the sidebar content in FB live videos were inverted.
Attached is a screenshot of the issue:

I've also appended this fix with the selector for BBC.com's logo not showing up as outlined in issue #52 .

Please, keep the alphabetical order by URL.

@alexanderby done

@alexanderby I also found an error with Twitter Summary card titles and content that I've added as well.
Screenshots:

Sorry that I haven't made a style guide, but for the next fix try to preserve whitespaces, like at other samples.

Will do, my bad.
Also, it looks like that Twitter fix wound up inverting all images/videos on Twitter.
I've fixed this in my latest commit.


Lisence type

Hi! Thank you for this job, Could you add license type to understand how we can use it, please.

Hi! We added a license file to the main directory.

Hi! We added a license file to the main directory.

Thank you


[FLINK-7847][avro] Fix typo in jackson shading pattern

What is the purpose of the change
This PR fixes a funny typo in the jackson shading pattern of flink-avro.

👍 😂

@aljoscha do you want to include this in your avro PR?

Jip, I included your commit. (In the correctly relocated POM)


Vertical dimension sort by time between threads

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ ] Bug
[x] Requirement
[ ] Feature or performance improvement

Requirement or improvement
The picture below is from a tracing in Sharding-Sphere:

There are 4 threads appear in this tracing: the main thread, Sharding-Sphere-0, Sharding-Sphere-1, Sharding-Sphere-2.
The tracing order in main thread is correct: parsing -> getConnection -> executeSQL -> close.
But it's disordered between threads: Sharding-Sphere-0, Sharding-Sphere-1, Sharding-Sphere-2 should happen after getConnection while before close.
I believe skywalking didn't sort tracing beween threads, and this kind of sort is the requirement.

Yes, this feature is still missing. I don't know, maybe 6.0.0-RC to fix this? @peng-yongsheng thoughts?

Closing. We have shardingsphere plugin from the official team, today.


Add border-radius to the .form-check element

We've decided that the focus-ring around interactive elements should never add a border-radius to an element and instead the element needs to provide rounded-corners by itself and the focus-ring will follow this default form.
To align with the Figma Design, we need to add border-radius to the .form-check element, so the focus-ring will show up with rounded-corners.
Dependant on #2855.

If a border radius is needed, use the content slot of the mixin to declare the border radius. Will be added in #2855


Update README_zh.md

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

The PR will be merged when CI checks succeed. Thanks for your contribution!


Avoid importing everything using *

Like for example in
01_trajectory_data_preperation/01_transformation_straight_traj.py

from numpy import *
from helper import *

and in other scripts as well.
Import only what you need. For example import numpy as np or from helper import transformation_coord

Done


Add basic support for underline and strikethrough decorations

Ah. I should have tested this before merging. Something is not quite right...

Fixed! It just needed to be transform rather than self.transform.


nvgpu softmax cuda version

Issue:
https://github.com/NervanaSystems/ngraph/issues/2001
Change the cuda version algorithm to the same as our cpu version:
if refer to CUDNN version, it's from:
CUDNN_SOFTMAX_FAST
This implementation applies the straightforward softmax operation.

to
CUDNN_SOFTMAX_ACCURATE
This implementation scales each point of the softmax input domain by its maximum value to avoid potential floating point overflows in the softmax evaluation.

NOTE1: cudnn version only support reduce_axis = 0;
NOTE2: performance issue because https://github.com/NervanaSystems/ngraph/issues/2013
Result:
before:
Comparing results for Softmax_11
[INFO] 2018-11-06T07:15:24z test/util/all_close_f.cpp 75        1.02089e-35 is not close to nan
[INFO] 2018-11-06T07:15:24z test/util/all_close_f.cpp 75        5.4829e-33 is not close to nan
[INFO] 2018-11-06T07:15:24z test/util/all_close_f.cpp 75        4.61425e-24 is not close to nan
[INFO] 2018-11-06T07:15:24z test/util/all_close_f.cpp 75        5.80109e-28 is not close to nan
[INFO] 2018-11-06T07:15:24z test/util/all_close_f.cpp 75        0 is not close to nan

after:
Comparing results for Softmax_11
  14 [INFO] 2018-11-07T01:44:08z test/util/all_close_f.cpp 75    1.02089e-35 is not close to 1.02034e-35
  15 [INFO] 2018-11-07T01:44:08z test/util/all_close_f.cpp 75    4.61425e-24 is not close to 4.61631e-24
  16 [INFO] 2018-11-07T01:44:08z test/util/all_close_f.cpp 75    8.77843e-30 is not close to 8.78233e-30
  17 [INFO] 2018-11-07T01:44:08z test/util/all_close_f.cpp 75    5.02628e-30 is not close to 5.02361e-30
  18 [INFO] 2018-11-07T01:44:08z test/util/all_close_f.cpp 75    5.10065e-33 is not close to 5.10542e-33

close to CPU result now.

@csullivan @AyanmoI
sure. what's failed on cuda reduce?

@csullivan I don't have a measure yet for accuracy. but should be good.


net/http: read: connection reset by peer under high load

What version of Go are you using (go version)?
go1.8.3
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="linux"
What did you do?

Edit /etc/sysctl.conf and reload with sysctl -p.
Edit /etc/security/limits.conf.
Execute the following server and client (on different hosts) to simulate multiple HTTP requests:
Server https://play.golang.org/p/B3UCQ_4mWm
Client https://play.golang.org/p/_XQRqp8K5b

What did you expect to see?
No errors at all.
What did you see instead?
Errors like the following:
Get http://<addr>:8080: read tcp [::1]:65244->[::1]:8080: read: connection reset by peer
Get http://<addr>:8080: write tcp [::1]:62575->[::1]:8080: write: broken pipe
Get http://<addr>:8080: dial tcp [::1]:8080: getsockopt: connection reset by peer
Get http://<addr>:8080: readLoopPeekFailLocked: read tcp [::1]:51466->[::1]:8080: read: connection reset by peer

I've been seeing similar issues with a similar set up where the server is on Go 1.6 while the client, which is making many concurrent requests, is on Go 1.9.2. Are we aware of what causes this? Seems like some kind of unintentional in-build rate limiting?

Go has no built-in rate limiting.
If you get broken network connections, they might be real (a peer really did disconnect), or they might be from hitting kernel limits on one of the sides.

Can anybody reproduce this and investigate?

Adding a comment to remind myself but:
I’ve seen similar issues with a should have time to investigate in the next month or so.

In my opinion this issue occurs when the response is streamed into a dynamic allocated buffer.
No error is thrown when conn.Read() is send to a buffer with a fixed length.
Error is thrown sporadic when response is read via:
io.Copy(&buf, conn)
ioutil.ReadAll(conn)
conn.Read() streamed until EOF
Notice: even when the error is thrown, the response is buffered correct and complete.

I am seeing this error frequently (read: connection reset by peer) when testing an HTTP 1.1 client that is generating heavy load on a server.  I have not checked if the problem goes away with HTTP/2. (It could, since requests would be interleaved on a single connection).
I am running the client and server both on the same "localhost", and both are running a managed pool of "worker" goroutines in an attempt to limit the demand for resources.
The client and server each have 350+ goroutines (as reported by runtime.NumGoroutine())and are running as root on a Mac laptop with the max number of open files and sockets maxed out at 8192 (ulimit -n 8192).
I can run the same code on Ubuntu 18.04 LTS with more cores, memory, sockets, etc, and I eventually run into the same error message, although at a higher traffic load.
If there is a kernel limit, it would be nice to know what it is, so we could write our code so that it intentionally stays below that limit, rather than tweaking parameters and hoping it won't be a problem.
I see this topic was active a year ago, ... has there been any progress on this in the past year @bradfitz ?

Can I retry the HTTP request in second?

I also see this issue under macOS where server and client talk over localhost. I don't know the exact cause of the problem but running netstat during high load displays very large number of connections in TIME_WAIT state. Either I exhaust file descriptors or local ports.
Or both. I keep seeing connection reset by peer until the number of TIME_WAIT connections is about 7k. If I let the test run even further, I eventually get connect: can't assign requested address error which suggests that I exhausted local ports. The number of TIME_WAIT connections is at 15k.
For my project that's not needed and I could solve the problem by limiting number of concurrent connections. Solving the problem in my code by limiting number of goroutines seemed arbitrary and still occasionally failed. Limiting MaxConnsPerHost in http transport did the trick and problem went away completely.

Also hitting this issue. Im running 1000 concurrent connections at it eventually hits this also on OSX with file descriptors bumped up from default (256) to 4096.  See a lot of stuff about this, possibly only an issue on OSX

I'm also running into this issue with the latest release of Go. No matter what I try, I keep getting these "connection reset by peer". Netstat show's hundreds of connections in TIME_WAIT state, even after shutting down my client.
transport = &http.Transport{ 		MaxIdleConns:        1, 		MaxIdleConnsPerHost: 1,                 IdleConnTimeout: 5 * time.Second }
The problem still happens with a Max Idle Connections of 1, and using runtime.GOMAXPROCS(1).

I manage to solve my TIME_WAIT problem using this article: http://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/

@gabrielcalderon it doesn't fix the problem completely. Idle connections do not limit the number of connections Go could open. If you exhausted idle connections for any reason, Go will start opening new ones and cause TIME_WAIT problem again. What fixes the problem is MaxConnsPerHost knob. It limits the number of connections Go could open and blocks subsequent requests if it's exhausted. You want to put it equal or higher than idle connections.

Just to update. my issue was related to OSX and its defaults.. having tweakes the following settings I can now push 1000 concurrent transactions
sudo ulimit -n 6049
sudo sysctl -w kern.ipc.somaxconn=1024

The only way I was able to fix it was to limit the number of concurrent transactions to be half the number of MaxIdleConns.

@REPTILEHAUS  Thanks for that.. This worked well for me.. Don't think "ulimit" was an issue, as it was set to "unlimited" already.
Can someone give an insight into, what could be the side-effects, if I were to increase kern.ipc.somaxconn to an abnormally high number ?

Just to update. my issue was related to OSX and its defaults.. having tweakes the following settings I can now push 1000 concurrent transactions
sudo ulimit -n 6049
sudo sysctl -w kern.ipc.somaxconn=1024

Works for me on Mac OS.

This is an absolute golden thread of lost knowledge.

have same issue

Just to update. my issue was related to OSX and its defaults.. having tweakes the following settings I can now push 1000 concurrent transactions sudo ulimit -n 6049 sudo sysctl -w kern.ipc.somaxconn=1024

For me, after doing this change, the error changed to connect: resource temporarily unavailable

Thanks.
I personally think that this is not primarily because golang itself, almost nothing to do with it. We can only limit the connection in golang codebase. But to solve this, I believe it just a matter of kernel tuning in a desired OS, linux/mac/windows.
Open file descriptors and the maximum number of connections that can be queued according to @REPTILEHAUS' solution is still working like a charm in OSX Macbook Air M1, 16GB RAM.  I was about to benchmark my golang app in high concurrency number, now there is no such connection reset in 1000 concurrency.
sudo ulimit -n 6049
sudo sysctl -w kern.ipc.somaxconn=1024

So, every machine have their own unique variable to tune its performance, it is just a matter how we adjust them accordingly.

Thanks.
I personally think that this is not primarily because golang itself, almost nothing to do with it. We can only limit the connection in golang codebase. But to solve this, I believe it just a matter of kernel tuning in a desired OS, linux/mac/windows.
Open file descriptors and the maximum number of connections that can be queued according to @REPTILEHAUS' solution is still working like a charm in OSX Macbook Air M1, 16GB RAM. I was about to benchmark my golang app in high concurrency number, now there is no such connection reset in 1000 concurrency.
sudo ulimit -n 6049
sudo sysctl -w kern.ipc.somaxconn=1024

So, every machine have their own unique variable to tune its performance, it is just a matter how we adjust them accordingly.

Awesome, thx bro.


Uppercase first letter in nativeName

The language, that i changed, need show uppercase, how like other
Why:
Closes ISSUE
What's being changed (if available, include any code snippets, screenshots, or gifs):
/lib/languages.js - languages constant, ru block and nativeName parameter.
Check off the following:

[x] I have reviewed my changes in staging (look for the "Automatically generated comment" and click the links in the "Preview" column to view your latest changes).
[x] For content changes, I have completed the self-review checklist.

@dmitryburov great catch! Thanks for spotting this and opening a PR ⚡
However, I noticed that this PR is trying to merge into our repo-sync as opposed to the main branch. Could you submit this PR against the main branch?

@cmwilson21 I changed the target on this branch. And approved it. I don't know what your exact process is for landing things, but it looks good to me to go in now.

أُرسلت من الـ iPhoneفي 09/12/2022 الساعة 5:03 م، كتب/كتبت Peter Bengtsson @.***>:
@cmwilson21 I changed the target on this branch. And approved it. I don't know what your exact process is for landing things, but it looks good to me to go in now.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

@peterbe Thank you! I'll remember that option for next time. 😃 ✨

@dmitryburov Thanks for submitting this PR! I'll get it merged down once the checks have passed ✅


Declaration dts files cannot represent enums nested in interfaces

TypeScript Version:  3.5.0
Search Terms: enum in interface declaration
Code
For JavaScript code being edited with Monaco, DTS libraries are used to inform Monaco of the symbols which can be autocompleted.
In our case, our library permits code like the following:
Lib.setEnum(Lib.MyEnum.ONE);
Lib.chain().setEnum(Lib.MyEnum.ONE);

The expected way to represent this would be something along the lines of:
// lib.d.ts
interface Lib {
    chain(): Lib;
    setEnum(val: Lib.MyEnum);
}
const Lib: Lib;
module Lib {
    enum MyEnum { ONE }
}

Expected behavior:
TypeScript is aware of the symbols Lib.chain, Lib.setEnum, Lib.MyEnum.
Actual behavior:
As written, with module Lib at the end of lib.d.ts, only Lib.chain and Lib.setEnum autocomplete.  Lib.setEnum takes an any though instead of Lib.MyEnum.
If you move module Lib to the top of lib.d.ts, only Lib.MyEnum autocompletes.

This one is tricky. enums define both a type and a value, so the containing namespace Lib defines a value too. That means it can't merge with any other values, like const Lib. But in order for references like Lib.setEnum to resolve, the value Lib needs to contain setEnum. So you need to declare the functions on the namespace instead of the interface.
This leaves you with a problem of referencing the type that Lib defines in chain. Confusingly, it's not Lib since the namespace Lib is a value, not a type. It's typeof Lib.
declare namespace Lib {
    export function chain(): typeof Lib
    export function setEnum(val: Lib.MyEnum): any;
    enum MyEnum { ONE }
}

Lib.setEnum(Lib.MyEnum.ONE);
Lib.chain().setEnum('hi'); // expect error here


[WIP] Rakefileなどをリポジトリルートに移動する

Codecov Report

Merging #130 into master will increase coverage by 0.18%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #130      +/-   ##
==========================================
+ Coverage   86.07%   86.26%   +0.18%
==========================================
  Files         188      189       +1
  Lines       21826    22186     +360
==========================================
+ Hits        18786    19138     +352
- Misses       3040     3048       +8

Impacted Files
Coverage Δ

diceBot/JamesBond.rb

diceBot/NightmareHunterDeep.rb

diceBot/Cthulhu7th.rb

diceBot/EarthDawn.rb

diceBot/CodeLayerd.rb

diceBot/SwordWorld2_5.rb

diceBot/Paranoia.rb

diceBot/StratoShout.rb

bcdiceCore.rb

diceBot/Avandner.rb

... and 367 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6dc09be...5a5d7e4. Read the comment docs.


Build a query parameter to make unav version configurable

Adds a query parameter to unav where one can pass the version of unav to debug

Resolves: MWPW-147126
Test URLs:

Before: https://main--milo--adobecom.hlx.live/drafts/blaishram/unav-test
After: https://gnav--milo--adobecom.hlx.live/drafts/blaishram/unav-test?unav=1.0&martech=off

verified the change in Gnav branch. now Unav JS , Css and other libraries are rendering as per the unav version parameter passed in url as query parameter. PFB

@bandana147 @salonijain3

I agree with @narcis-radu's suggestion. Additionally, I'd argue unav is not very descriptive. I'd expect to be able to turn a feature on or off with such wording. I'd adapt the name to reflect that the value is impacting the version.

@overmyheadandbody Does unavVersion sounds good?


added dragIgnoreFrom and resizeIgnoreFrom options to GridItem.vue

I simply used "webpack --progress --hide-modules --config webpack.build.js && cross-env NODE_ENV=production webpack --progress --hide-modules --config  webpack.build.min.js" for the build. Was that okay? The files in dist changed pretty much. That's why I'm a little concerned.

sorry. I used the wrong branch.


Relational Database/Learn Bash By Building a Boilerplate

Affected page

https://www.freecodecamp.org/learn/relational-database/learn-bash-by-building-a-boilerplate/build-a-boilerplate
Your code

Expected behavior

At the some point, tutorial told me that add a file named images. Even i cheated from tips, the error was there. It says there is already a file named images. But i check it with list command and it's not there. Then i close the tutorial to replay the course. I can't open it now. I guess it's an extension named "CodeRoad" but i am not able to see tutorial somehow. I am not sure the problem is me or a bug etc.
Screenshots

System

Device: [Laptop]
OS: [Windows 10]
Browser: [chrome]
Version: [100.0.4896.127]

Additional context

Note: Cookies are allowed all the time.


Error In ffmpeg on docker container

Recently whern I try to do a docker-compose up -d it gets stuck on the middle of the process, by running the container log it displays:

whisper has been successfully installed.
Traceback (most recent call last):
File "/subgen/./subgen.py", line 43, in
import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg'

And gets stuck here.

Look further up in your log and see if the ffmpeg module is installing.  Either it's failing to install it, or it's looking in the wrong place for it.

here is the full log output from a clean run from zero:
Installing numpy...
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.26.3)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
numpy has been successfully installed.
Installing stable-ts...
Requirement already satisfied: stable-ts in /usr/local/lib/python3.10/dist-packages (2.15.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from stable-ts) (1.26.3)
Requirement already satisfied: openai-whisper==20231117 in /usr/local/lib/python3.10/dist-packages (from stable-ts) (20231117)
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from stable-ts) (2.1.2)
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from stable-ts) (4.66.1)
Requirement already satisfied: torchaudio in /usr/local/lib/python3.10/dist-packages (from stable-ts) (2.1.2)
Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from openai-whisper==20231117->stable-ts) (0.58.1)
Requirement already satisfied: triton<3,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from openai-whisper==20231117->stable-ts) (2.1.0)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.10/dist-packages (from openai-whisper==20231117->stable-ts) (10.2.0)
Requirement already satisfied: tiktoken in /usr/local/lib/python3.10/dist-packages (from openai-whisper==20231117->stable-ts) (0.5.2)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (2023.12.2)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (12.1.105)
Requirement already satisfied: nvidia-cublas-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-curand-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cusparse-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (2.18.1)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (12.1.105)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (12.1.105)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (1.12)
Requirement already satisfied: nvidia-cusolver-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (3.1.3)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (4.9.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (3.2.1)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (3.13.1)
Requirement already satisfied: nvidia-cufft-cu12==<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from torch->stable-ts) (<IP_ADDRESS>)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==<IP_ADDRESS>->torch->stable-ts) (12.3.101)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->stable-ts) (2.1.4)
Requirement already satisfied: llvmlite<0.42,>=0.41.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->openai-whisper==20231117->stable-ts) (0.41.1)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->stable-ts) (1.3.0)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->openai-whisper==20231117->stable-ts) (2023.12.25)
Requirement already satisfied: requests>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from tiktoken->openai-whisper==20231117->stable-ts) (2.31.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken->openai-whisper==20231117->stable-ts) (2.1.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken->openai-whisper==20231117->stable-ts) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken->openai-whisper==20231117->stable-ts) (2023.11.17)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.26.0->tiktoken->openai-whisper==20231117->stable-ts) (3.3.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
stable-ts has been successfully installed.
Installing fastapi...
Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (0.109.0)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from fastapi) (2.5.3)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from fastapi) (4.9.0)
Requirement already satisfied: starlette<0.36.0,>=0.35.0 in /usr/local/lib/python3.10/dist-packages (from fastapi) (0.35.1)
Requirement already satisfied: pydantic-core==2.14.6 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi) (2.14.6)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi) (0.6.0)
Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.36.0,>=0.35.0->fastapi) (4.2.0)
Requirement already satisfied: exceptiongroup>=1.0.2 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.36.0,>=0.35.0->fastapi) (1.2.0)
Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.36.0,>=0.35.0->fastapi) (1.3.0)
Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.36.0,>=0.35.0->fastapi) (3.6)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fastapi has been successfully installed.
Installing requests...
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests) (2023.11.17)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests) (2.1.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
requests has been successfully installed.
Installing faster-whisper...
Requirement already satisfied: faster-whisper in /usr/local/lib/python3.10/dist-packages (0.10.0)
Requirement already satisfied: ctranslate2<4,>=3.22 in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (3.24.0)
Requirement already satisfied: tokenizers<0.16,>=0.13 in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (0.15.1)
Requirement already satisfied: huggingface-hub>=0.13 in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (0.20.3)
Requirement already satisfied: av==10.* in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (10.0.0)
Requirement already satisfied: onnxruntime<2,>=1.14 in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (1.16.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from ctranslate2<4,>=3.22->faster-whisper) (1.26.3)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from ctranslate2<4,>=3.22->faster-whisper) (59.6.0)
Requirement already satisfied: pyyaml<7,>=5.3 in /usr/local/lib/python3.10/dist-packages (from ctranslate2<4,>=3.22->faster-whisper) (6.0.1)
Requirement already satisfied: typing-extensions>=<IP_ADDRESS> in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (4.9.0)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (4.66.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (2023.12.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (3.13.1)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (2.31.0)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13->faster-whisper) (23.2)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from onnxruntime<2,>=1.14->faster-whisper) (1.12)
Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from onnxruntime<2,>=1.14->faster-whisper) (4.25.2)
Requirement already satisfied: flatbuffers in /usr/local/lib/python3.10/dist-packages (from onnxruntime<2,>=1.14->faster-whisper) (23.5.26)
Requirement already satisfied: coloredlogs in /usr/local/lib/python3.10/dist-packages (from onnxruntime<2,>=1.14->faster-whisper) (15.0.1)
Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs->onnxruntime<2,>=1.14->faster-whisper) (10.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.13->faster-whisper) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.13->faster-whisper) (2023.11.17)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.13->faster-whisper) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.13->faster-whisper) (3.6)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->onnxruntime<2,>=1.14->faster-whisper) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
faster-whisper has been successfully installed.
Installing uvicorn...
Requirement already satisfied: uvicorn in /usr/local/lib/python3.10/dist-packages (0.27.0)
Requirement already satisfied: typing-extensions>=4.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn) (4.9.0)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn) (0.14.0)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn) (8.1.7)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
uvicorn has been successfully installed.
Installing python-multipart...
Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (0.0.6)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
python-multipart has been successfully installed.
Installing whisper...
Requirement already satisfied: whisper in /usr/local/lib/python3.10/dist-packages (1.1.10)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from whisper) (1.16.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
whisper has been successfully installed.
Traceback (most recent call last):
File "/subgen/./subgen.py", line 43, in
import ffmpeg
ModuleNotFoundError: No module named 'ffmpeg'

I misremembered.  ffmpeg is installed in the docker image @ https://github.com/McCloudS/subgen/blob/f02cb3d8709869245c6398c7aeed8b2db26f4809/Dockerfile#L5C1-L5C68.  How old is your build?  Did you try a whole new pull?  Did you use the volume mount "/usr/local/lib/python3.10/dist-packages"?  You could remove it if so.
The ffmpeg python module is installed from faster-whisper inside the av module.  Which shows installed correctly in your log : "Requirement already satisfied: av==10.* in /usr/local/lib/python3.10/dist-packages (from faster-whisper) (10.0.0)"

tried with a new pull and after clearing docker build cache same results, using the default cpu version of the image

Hello, I noticed the same error just now. All fresh, never tried this before. I added ffmpeg-python to subgen.py and was able to pass the error.
I then ran into a GPU related error, but that is probably unrelated...  CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

Apologies if you were working on more changes, but I saw the latest commits and gave it a spin and found a different error:
[+] Running 2/2
 ✔ Network subgen_default  Created                                                                                0.0s
 ✔ Container subgen        Created                                                                                18.1s
Attaching to subgen
subgen  |
subgen  | ==========
subgen  | == CUDA ==
subgen  | ==========
subgen  |
subgen  | CUDA Version 11.8.0
subgen  |
subgen  | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
subgen  |
subgen  | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
subgen  | By pulling and using the container, you accept the terms and conditions of this license:
subgen  | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
subgen  |
subgen  | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
subgen  |
subgen  |   File "/subgen/./subgen.py", line 21
subgen  |     import ffmpeg-python
subgen  |                  ^
subgen  | SyntaxError: invalid syntax
subgen exited with code 1

Explicitly adding python-ffmpeg to the Dockerfile list of pip packages to install makes everything run fine. cuda seems to be working too.
You said whisper should install it, as it is a dependency, but is there anything wrong with specifying it directly?
Thanks!

Thanks, I'll give it a shot.  I hadn't pulled it and checked the change yet.  Does subgen.py need import ffmpeg or ffmpeg-python?

Confirm, I do not get that issues anymore

Thanks for the feedback!  Persistent volume is working, so it won't re-download the models if you don't have one mounted.

Working again, thanks!


Improve Arm (And other Arch) Context Swap Performance

Is your enhancement proposal related to a problem? Please describe.
Some benchmarks shows Zephyr behind in context swap performance compared to ThreadX.
Describe the solution you'd like
Avoid any branch-link (function call) operations in PendSV handling for Arm, likely other archs could have the same idea implemented.
Every bl op is a potential pipeline flush, certainly some lost context, we almost immediately call out to a C function handler for PendSV handling (used for Arm context swap). There's several other bl ops involved depending on which options are involved.
ThreadX avoids almost all bl ops except a hook for swap in/swap out that is opt in. Otherwise has ~80 asm instructions for PendSV handling. Clearly has some performance implications somewhere here, maybe partially due to the branch out of inline asm. Perhaps other things, needs investigating.
https://github.com/eclipse-threadx/threadx/blob/master/ports_arch/ARMv7-M/threadx/gnu/src/tx_thread_schedule.S#L131
https://github.com/zephyrproject-rtos/zephyr/blob/main/arch/arm/core/cortex_m/swap_helper.S#L56
Describe alternatives you've considered
Not doing anything
Additional context
Benchmark report showing difference in context swap performance, on a cortex-m4
https://www.dropbox.com/scl/fi/opimwfbvkd9coeprc7d5h/Beningo_RtosPerformance_2024_Report.pdf?rlkey=s3n007s6hgubnj37ovto88bs2&e=3&dl=0
In large part the difference is due to our MPU usage for hw stack protection by default, but this isn't the only thing playing a part.

Reminds me of https://github.com/zephyrproject-rtos/zephyr/pull/65071#pullrequestreview-2020028109 ...

I still argue that eliminating PendSV on the context switch path entirely would be even better.  Other architectures don't work that way, it's specific to cortex-m.

Maybe some evidence to support this idea here https://gvpress.com/journals/IJSH/vol9_no2/10.pdf

I still argue that eliminating PendSV on the context switch path entirely would be even better. Other architectures don't work that way, it's specific to cortex-m.

I'm new to this subject, but isn't the benefit of PendSV is that it allows having minimal context switching code for all Cortex-M CPUs? Also, the fact that it is asynchronous allows avoiding context switching in the middle of ISR & thus arguably improves ISR handling.
https://developer.arm.com/documentation/107706/0100/System-exceptions/Pended-SVC---PendSV

@JarmouniA it does reduce code size by unifying the "context switch on interrupt exit" and "synchronous/cooperative context switch" cases (by effectively making the latter a trap to an interrupt).  But:

As a cost, it pays for a full interrupt entry/exit on the cooperative case, which is how we ended up here.
It's actually something you can do on almost all architectures if you think about it, almost everything has the ability to flag a low priority interrupt to be handled after the current ISR returns.  The fact that no one else does this should be informative[1].
And my personal peeve: it makes context-switch non-atomic.  The kernel likes to be able to set some state based on the old/new threads it's selected and then call swap()/switch() to effect that.  But that breaks on cortex-m because the actual context switch won't happen until the PendSV exception, which can be preempted by an interrupt!  There's some really subtle/horrible code gated under CONFIG_SWAP_NONATOMIC that we have to carry to track this, and I'd love to see it die.

[1] OK, it does need to be mentiond that cortex-m specifically has very light weight interrupts, something other architectures are much weaker at.  But even this falls down when you start adding more stuff: nested interrupts, MPU/MMU/FPU state handling, stack switching, etc...  pollute that pretty badly when you start turning features on.

It’s also not potentially slower given the paper I linked. Seems like a compounding set of reasons to try and make a smaller swap with inline asm and avoid many quirks. Who’s gonna try it?


Adding grafana dashboards based on prometheus metrics

Adding grafana dashboards based on promtheus metrics for standard and k8s config

modifcations done