CRITICAL BUG: Multi GPU training possibly not implemented correctly, resulting models potentially under-trained

Hello,
I hope you are doing well. While using your codebase for SlotFormer and building on it for a while now, I have just discovered a critical bug in the way that the nerv toolkit implements multi gpu training with Distributed Data Parallel. I have opened an issue for it in that repo here.
Summary: DDP is not implemented correctly. There are two main problems, one major and one minor:

The models on each GPU do not communicate with each other, and are trained differently and diverge, which is wrong behavior. In essence, the result when training using 8 GPUs for 80 epochs, is that every GPU trains the starting model differently, and performs gradient calculation and model updates separately, based on the input batch for the current GPU. The result is 8 separate models, each trained on the equivalent of roughly 10 epochs.
When training for an epoch, the indices across the GPUs are not mutually exclusive, so technically we get slightly different behavior from single GPU training. When using torch distributed sampler properly, each GPU gets a unique subset of the data that does not intersect with any of the other gpus. Although this does not affect performance significantly as the models still see the entirety of the data on average, it be might good to keep that in mind.

This bug affects all experiments using multiple GPUs with the nerv library. So far, I have come across SlotFormer, SlotDiffusion, and EventClip.
Multi GPU training issue
The main issue arises in the forward pass logic here.
When a model is wrapped in DDP module, the DDP module handles the forward and backward pass across the GPUs such that the gradients are synced during the background pass. However, for this to function correctly, the forward pass must be called directly using the DDP forward specifically.
In the code, the forward pass is done by bypassing this forward and accessing the inner variable module in order to access the wrapped model. This completely skips the gradient synchronization and thus leads to each GPU calculating the gradient for only its own batch and updating accordingly. This can be easily tested by comparing the parameters across the different ranks, and they diverge to a large extent as the training goes. In essence, every GPU trains a separate model independently. This does not affect the correctness, because each model is correctly trained, and in the end of the training the model on rank 0/gpu 0 is saved.
However: what this means is that the computation of all GPUs other than the first GPU are completely wasted, and that in every epoch, the model on GPU 0 is trained on 1/N of the data, where N is the number of GPUs. As a result, increasing the number of GPUs means the model is trained using less data each epoch compared to using a single GPU.
After 5 epochs, I checked the difference in L2 norms of each parameter between the main GPU and other ranks, and here are the largest differences, which I measured using my toolkit here just for convenience:
----------------------------------------------------------------------
|        Parameter        | Ranks compared |  L2 norm of difference  |
----------------------------------------------------------------------
|module.decoder.3.0.weight|    0 <-> 3     |        2.4202e+01       |
|module.decoder.3.0.weight|    0 <-> 2     |        2.3539e+01       |
|module.decoder.3.0.weight|    0 <-> 1     |        2.2490e+01       |
|module.decoder.0.0.weight|    0 <-> 2     |        1.5555e+01       |
|module.decoder.0.0.weight|    0 <-> 1     |        1.5359e+01       |
|module.decoder.0.0.weight|    0 <-> 3     |        1.5071e+01       |
|module.decoder.2.0.weight|    0 <-> 2     |        1.4756e+01       |
|module.decoder.2.0.weight|    0 <-> 3     |        1.4714e+01       |
|module.decoder.2.0.weight|    0 <-> 1     |        1.4555e+01       |
|module.encoder.2.0.weight|    0 <-> 1     |        1.4034e+01       |
----------------------------------------------------------------------

As you can see, the L2 norm becomes massive after a while, meaning the models are completely different on each GPU, which is not desired behavior.
The solution to this problem is to reword the loss calculation logic in DDP, such the the wrapped model is a model that returns the desired loss to be optimized directly in the forward pass, so that calling
loss = ddp_wrapped_model(batch)

can be used as is. This ensures the syncing is happening and that the training signal is from all the available GPUs.
If this issue is fixed, this means that we can train the models in this project much faster with multiple GPUs than currently, which should be a very strong improvement.
Moreover, this also means that results reported for X epochs on N gpus should be roughly equivalent to training using 1 GPU for X/N epochs, assuming the last checkpoint is taken in both cases. This might imply either the models that were trained with multiple GPUs are either 1) undertrained, and can perform better or 2) trained enough, but could be trained much faster with proper multigpu support.
Multi GPU data sampling issue
This is a relatively minor issue but I still wanted to bring attention to it. In the distributed sampler, here is the logic of generating the indices on each GPU:
indices = torch.randperm(len(self.dataset))

which are then divided into partitions, one for each GPU here:
indices = indices[self.rank:self.total_size:self.num_replicas]

However, the indices generated on each GPU are completely independent and not the same, which leads to the subsets of indices on each GPU overlapping.
The best way to solve this, which is how it is implemented in the official torch repository as follows:
g = torch.Generator()
g.manual_seed(self.seed + self.epoch)
indices = torch.randperm(len(self.dataset), generator=g).tolist()

In this arrangement, the generation of the indices on every GPU is fixed by the seed and the current epoch, which must be explicitly set using sampler.set_epoch at the beginning of each epoch before iterating over the data, and each GPU gets a non-overlapping subset of the data.
I checked the possible overlapping of the indices using this method, and found such overlap in all epochs I tested in your training pipeline, unlike the torch sampler.
In the end, this is not a critical bug like the first one, but it does mean that the sampling is different in cases of single GPU (every GPU sees the entire dataset) vs multi GPU (every GPU sees 1/N randomly sampled of the data with replacement, so all GPUs together don't see the entire dataset (probabilistically very unlikely)). It might not matter much if at all for final performance.
Since I have been using this code for some experiments for SaVI/SlotFormer, please let me know if you need any help in addressing the core of the issue, would be happy to connect and explain in more detail/assist in any way I can.

DDP gradient sync solved with the commit d3fe9c5
DDP non-overlapping data sampler solved with commit ccc8a8d
Refer to https://github.com/pairlab/SlotFormer/issues/9 for performance benchmarking


Random Access

Maris said:

i need to make my files seekable e.g. my record has an ID and i want to jump exactly to that record (instead of iterating across the whole file). It works fine i only need the byte offset of the written file after each record, which is then appended to a dict. In default avro a function aclled sync() does that (in Datafile.py class DataFileWriter(). It essentially forces to write the data and then returns tell() of the file. Is there any chance that something like this will be build in?

There's StoredKeyValueFile maybe we should do this.


chore: unnecessary use of fmt.Sprint or fmt.Sprintf

Checklist

[ ] make test-all (UNIX) passes. CI will also test this
[ ] unit and/or integration tests are included (if applicable)
[ ] documentation is changed or added (if applicable)

Description of change

This looks good, if you resolve the conflicts, I'll merge it.
Thank you for taking the time to do this. Need to add some linting sometime to the CI process. :thinking:


Service to aaactl communication: require a subscription

Need to finalize and test the functionality for a service (ixctl, prefixctl etc.) to contact aaactl to check if user has a subscription for a service, and if not make sure to set it up.
I think this can be mostly opaque to the user, with the exception of requiring them to set up billing if billing is not set up yet.

For requiring billing information, that should probably be a separate check depending on whether the user's usage actually requires billing?
If we want to do something like allowing a certain usage for free, it makes sense to only ask the customer for billing once they actually need to go past the free threshold
@grizz

For requiring billing information, that should probably be a separate check depending on whether the user's usage actually requires billing?
If we want to do something like allowing a certain usage for free, it makes sense to only ask the customer for billing once they actually need to go past the free threshold
@grizz


Build Luigi App with React

Tutorial: https://developers.sap.com/tutorials/luigi-app-react.html
Step 12: Add "Product Detail" view to Luigi app
For Step 12.1, it looks like there's a missing import for Products.jsx because of the subsequent step:
<Route path="/products" component={Products} />
<Route path='/productDetail/:id' component={ProductDetail} />

Hello,
Please refer to step 10.2, Products component needs to be created at this step.
Thanks.

Hello,
Please refer to step 10.2, Products component needs to be created at this step.
Thanks.

@stanleychh kindly review the steps. There was a usage but no import -- see screenshot below:

@stanleychh kindly review the steps. There was a usage but no import -- see screenshot below:

@jcailan , thanks for pointing this out. We will put this step in.

@jcailan , thanks for pointing this out. We will put this step in.

Hello,
Thank you for bringing up this issue. The tutorials have been updated in https://github.com/SAPDocuments/Tutorials/pull/6425

Hello,
Thank you for bringing up this issue. The tutorials have been updated in https://github.com/SAPDocuments/Tutorials/pull/6425

Thank you for your feedback. The issue seems to be resolved, so I am closing the issue.
If you still have questions, feel free to reopen the issue.

Thank you for your feedback. The issue seems to be resolved, so I am closing the issue.
If you still have questions, feel free to reopen the issue.

I checked step 12 again, and it looks like no changes were made:

I checked step 12 again, and it looks like no changes were made:

@MichaelCzcz i'm not able to reopen this issue.

@MichaelCzcz i'm not able to reopen this issue.


add support to jsdom

While trying to implement tests for my styled-components I was constantly facing the following error

TypeError: Cannot read property 'cssRules' of undefined

I then managed to track its root as being the issue https://github.com/tmpvar/jsdom/issues/992.
Other browsers are also affected by this, not only jsdom, as can be seen at https://github.com/threepointone/glamor/issues/58.
Glamor has already fixed this with the commit https://github.com/threepointone/glamor/commit/98d4f44ba4335abd0e63bf946edff04ec51d3a41, and this PR brings the main change to styled-component.

This PR should fix https://github.com/styled-components/jest-styled-components/issues/2 and it could also be related to https://github.com/styled-components/styled-components/issues/124#issuecomment-273914589

Hey, thank you for the PR. If you could add a changelog entry I am happy to merge this. 🎉

there we go! :)


[output] Backoff support for all PagerDuty related outputs

to: @ryandeivert @jacknagz
cc: @airbnb/streamalert-maintainers
size: small
Background
There were some issues with the PagerDuty multiple APIs that was making StreamAlert to lose alerts becase the API access was being rate limited. With these changes the requests sent to the API will retry when a problem is detected, that way we can effectively mitigate the encountered rate limits, due to high volume alerts.
Changes

Added changes to the base output class to add retry methods for both GET and POST requests. This way the support for backoff/retry can be progressively added in each single output without breaking existing code.
Added specific support in PagerDuty outputs to use the backoff/retry code.
Modified existing tests to support new approach.

Testing
$ rm .coverage && ./tests/scripts/unit_tests.sh
...
TOTAL                                                  2617     75    97%
----------------------------------------------------------------------
Ran 470 tests in 13.873s

OK

$ ./tests/scripts/pylint.sh

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)

$ python manage.py lambda test --processor rule all
...
StreamAlertCLI [INFO]: (59/59) Successful Tests
StreamAlertCLI [INFO]: (93/93) Alert Tests Passed
StreamAlertCLI [INFO]: Completed

Coverage increased (+0.002%) to 95.333% when pulling af60dda91a2e51719205e96ea6041ec788d380fb on javier-streamalert-pagerduty-backoff into 0253d1c216c3ae8e251b24b8f15b3899c3e44236 on master.

Coverage increased (+0.1%) to 95.44% when pulling 4e1298ebd94aba3e8dcc38e900239f00629dcef6 on javier-streamalert-pagerduty-backoff into 0253d1c216c3ae8e251b24b8f15b3899c3e44236 on master.

Coverage increased (+0.1%) to 95.44% when pulling 308f8705d8cc040decac230d6fe52fad10532a18 on javier-streamalert-pagerduty-backoff into 0253d1c216c3ae8e251b24b8f15b3899c3e44236 on master.

Implements this https://github.com/airbnb/streamalert/issues/363


run sequentially the unit test of codemirror-promql

It may help to stabilize the job test_ui.
Signed-off-by: Augustin Husson<EMAIL_ADDRESS>

:+1:


Execute pg_resetwal in docker Unraid

Hi,
Following a freeze of my Unraid server version 6.12.9, I'm unable to start the Immich Postgres DB docker anymore. I receive the following logs messages when I try :
LOG:  starting PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
LOG:  listening on IPv4 address "<IP_ADDRESS>", port 5432
LOG:  listening on IPv6 address "::", port 5432
LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
LOG:  database system was shut down at 2024-04-02 17:57:27 CEST
LOG:  unexpected pageaddr 1/5AD0E000 in WAL segment<PHONE_NUMBER>00000100000060, LSN 1/60D0E000, offset 13688832
LOG:  invalid checkpoint record
PANIC:  could not locate a valid checkpoint record
LOG:  startup process (PID 29) was terminated by signal 6: Aborted
LOG:  aborting startup due to startup process failure
INFO  service::utils::clean] Find directory "pg_vectors/indexes/17536".
INFO  service::utils::clean] Find directory "pg_vectors/indexes/17541".
INFO  service::utils::clean] Find directory "pg_vectors/indexes/17541/segments/6c6cf080-e06e-482e-b87b-70e0330f4656".
LOG:  database system is shut down
After some reasearch, and  in order to clean de invalid checkpoint status, I apparently need to execute the command :
"su postgres pg_resetwal -f /var/lib/postgresql/data"
But when I try I receive the followinbg error:
/usr/lib/postgresql/16/bin/pg_resetwal: /usr/lib/postgresql/16/bin/pg_resetwal: cannot execute binary file
From there, I can't find any solution to the issue. Would someone have an idea on how to solve that problem?

I was going to say this isn't a pgvecto.rs issue, but the cannot execute binary file is strange. I think the binary in the container might be for the wrong architecture.

According to manual, the arguments after <user> are passed to the shell. So it's an intended behaivor. You can use su to login or use sudo to run programs.


typo: unknow -> unknown

Ⅰ. Describe what this PR did
Ⅱ. Does this pull request fix one issue?

Ⅲ. Describe how you did it
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #959 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #959   +/-   ##
=======================================
  Coverage   13.06%   13.06%
=======================================
  Files         123      123
  Lines        8235     8235
=======================================
  Hits         1076     1076
  Misses       7063     7063
  Partials       96       96

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 979666b...cce7d11. Read the comment docs.

LGTM, thanks a lot for your work. @faycheng


Need to validate that new articles have at least one subject

I was very sure we had included this a long tie ago, but it seems to be gone now, so articles are able to be created without subjects.
I believe we had a test for this as well, which is obviously gone now too, since nothing is failing.
Or am i just imagining that we had that validation...

This has been fixed and is in production.


chore: no-op dummy commit to test RHIDP-4146...

chore: no-op dummy commit to test RHIDP-4146 from a personal branch
Signed-off-by: Nick Boldt<EMAIL_ADDRESS>

/retest


Dev Text: Learn More links

Figure out what is needed for the Learn More

[ ] for reporting form Invalid outcome
[ ] participation token section
[x] market creation
[ ] forking banner

searching the code base for learn more will give you the places some just link to docs.augur.net.
Is that correct?

addressed here https://github.com/AugurProject/augur/issues/4846 and https://github.com/AugurProject/augur/issues/4848


Update readme sample links to point to main branch

Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[ ] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[ ] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
API changes are not detected in this pull request.

@weidongxu-microsoft, Can you please help me review this PR whenever you have time? I created this PR to update the links in easm package README

@weidongxu-microsoft Thanks for pointing that out, I have removed unused classes in this PR. Can you help me merge this PR whenever you have time?


ci: include speech/api in the build

Note that I changed the code to compile with CMake + vcpkg. That makes it easier to keep the code up to date.
Instead of building the protos using some custom Makefile, I just used the proto libraries from google-cloud-cpp.

PTAL


Added ability to format picklist selection using tab key

Fix for issue #22. Styling for hovering on picklist improved so cursor now displays as a pointer.

Thanks @niick2112


Remove mostRecentContent from HomeSplash.

Summary
Removes the mostRecent and mostRecentContent declarations from HomeSplash.  They weren't used.

Test plan
Nothing should change.

you are awesome.


Lazy.eval - Window is Not Defined Error

Although the data is coming through and the page is rendering fine, I am seeing a window is not defined error. Has anyone seen this / know the source of the issue?
⨯ Internal error: ReferenceError: window is not defined
    at Lazy.eval [as fn] (./node_modules/@glideapps/glide-data-grid/dist/esm/common/browser-detect.js:25:73)
    at get value [as value] (./node_modules/@glideapps/glide-data-grid/dist/esm/common/browser-detect.js:14:45)
    at realizeKeybinds (./node_modules/@glideapps/glide-data-grid/dist/esm/data-editor/data-editor-keybindings.js:67:87)
    at eval (./node_modules/@glideapps/glide-data-grid/dist/esm/data-editor/data-editor-keybindings.js:129:16)

I am running this app on NextJS 14 in a client component.
What is a good place to start looking for the source of the issue?

I also have this issue

This error occurs because Next.js attempts to perform server-side rendering (SSR) by default, even for components marked with "use client". During SSR, the window object is not available since it's a browser-specific API.
To resolve this issue with @glideapps/glide-data-grid, the temporary solution is :

Use dynamic import for the DataEditor component
Disable SSR for this specific component

like
const DataEditor = dynamic(
  () => import("@glideapps/glide-data-grid").then((mod) => mod.DataEditor),
  { ssr: false },
);

But the proper solution should not use window during ssr time,
for example
// this code will run on server and client
const Component = () => {
  // on server, it will throw the error because we don't have window
  const windowWidth = window.innerWidth;
  return
Width: {windowWidth}
;
}

const Component = () => {
// useEffect will only run in client
const [windowWidth, setWindowWidth] = useState(0);
useEffect(() => {
setWindowWidth(window.innerWidth);
}, []);
}


Added rate limiting table and displaying for endpoints

Caveats: This pulls the data from default.js which shouldn't make a different right now but might later, sadly config does not allow being instantiated with different working directories.
Pro: Automatic rate limit table building, YAY!
@connor4312 If you have a minute it would be totally cool if you could add a little database or cache icon to the endpoints (like the lock icon for secured endpoints), I didn't really understand how your glyph icon font thingy works 😛

@SimonSchick, thanks for your PR! By analyzing the blame information on this pull request, we identified @connor4312 and @ProbablePrime to be potential reviewers

lgtm

Re to the conifg: what I would do is either:

submit a PR to node-config to add subdirectory / instanced loading or
create a tiny standalone module that requires the config module and outputs the config to stdout before exiting. Then you can subprocess that module and set the working directory appropriately. This is a slightly ugly solution but isn't too hacky and should be stable enough.

I'll look into creating a PR for config since it seems like a pretty straight-forward process.

Can we get this merged for now? We want to roll out rate limiting so it would be pretty good to have this in.

Sure go ahead and merge it, i'll get you an icon. I also need to add an icon for deprecation. feature/FANCY ICONS incoming soon :)


added windows 11

after reading this article, i confirmed windows 11 is supported

Learn Build status updates of commit ed22195:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

windows/security/threat-protection/security-policy-settings/devices-allow-undock-without-having-to-log-on.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the learn.microsoft.com contributor guidesPost your question in the Learn support channel


Improve environment variable handling in integ tests

Fixes gradle/gradle-private#3821

@bot-gradle test this

I've triggered the following builds for you:

PullRequestFeedback build

Is this not a breaking change? Do we need to deprecate this behavior first? Or did this already fail with a bad error message

The change should only affect our integration test harness. Nothing in production uses GradleExecuter, even Testkit.

@bot-gradle test and merge

I've triggered a build for you. Click here to see all failures if there's any.

Pre-tested commit build failed. Click here to see all build failures.

@bot-gradle test ACC

@bot-gradle test and merge

I've triggered a build for you. Click here to see all failures if there's any.

@bot-gradle test and merge

I've triggered a build for you. Click here to see all failures if there's any.


Rework CephBackend and rely to the lib-common

Change glance-operator's CephBackend. Do not define our own versions of GetCephOsdCaps, GetCephRbdUser, GetCephGlancePool, and validateMons and their
GlanceAPICephBackend and CephPoolSpec structs.
Instead import them from lib-common [1].
[1] https://github.com/openstack-k8s-operators/lib-common/pull/51
Signed-off-by: John Fulton<EMAIL_ADDRESS>

@fmount should we include a change to the CRDs in both the glance and cinder operators with this move? I'd rather the following names were consistent.
CephClusterFSID to ClusterFSID
CephClientKey to ClientKey
CephClusterMonHosts to ClusterMons
CephPoolSpec to PoolSpec
CephPoolName to PoolName

@fmount should we include a change to the CRDs in both the glance and cinder operators with this move? I'd rather the following names were consistent.
CephClusterFSID to ClusterFSID
CephClientKey to ClientKey
CephClusterMonHosts to ClusterMons
CephPoolSpec to PoolSpec
CephPoolName to PoolName

Yes, best to be consistent and follow a single naming convention. I'm wondering if your idea is to change those occurrences in the glanceapi config as per my previous comment.

@fultonj It might just be my environment, but I can't seem to run the operator with the patch due to dependency issues.  Does OPERATOR_TEMPLATES=./templates make run work for you?

@abays I tested it with OPERATOR_TEMPLATES=$PWD/templates ./bin/manager (after building) and used it to create an image on a Ceph server. I have to rebase for the new go.mod files anyway. I'll make sure it works with OPERATOR_TEMPLATES=./templates make run when I re-submit.


Revert "NHM District Level Data: All datasets"

Reverts datacommonsorg/data#482

Reason to revert this PR: Updated the NHM data with district-wise data for all datasets, cleaned the directory structure and created a new and updated PR


Always ensure generated modules have cmdlets in them

Since this SDK is built on a beta API, changes are made to the service on a daily. We need to ensure that when we generate modules, we check to see if their OpenAPI docs have paths in them. This results in modules with no cmdlets Teams.Channel.
AB#4724

If the OpenAPIDocument has no paths we should return a 400 error from the service to indicate that that the query parameters were not good.  This will allow the PowerShell generation to not create a module for this.


Install dependency via requirements.txt

Remove requirements in setup.py, use requirements.txt instead.
This change will cause failure when run CodeCov.
Fixed CodeCov failure by add pip instal -r requirements.txt before running CodeCov

Codecov Report

Merging #271 (bced94b) into main (eb0ba44) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head bced94b differs from pull request most recent head 203ac69. Consider uploading reports for the commit 203ac69 to get more accurate results

@@           Coverage Diff           @@
##             main     #271   +/-   ##
=======================================
  Coverage   71.99%   71.99%
=======================================
  Files         178      178
  Lines        6106     6106
  Branches      636      636
=======================================
  Hits         4396     4396
  Misses       1488     1488
  Partials      222      222

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb0ba44...203ac69. Read the comment docs.

/approve
/lgtm


Weight should be optional on the Centroid analysis

So you can calculate also simple centroids.
cc/ @javisantana

being done by arce

Yes, it's waiting for review here: https://github.com/CartoDB/cartodb/pull/8407


Caching build artifacts in order to split large CI job into multiple smaller jobs?

Hello,
i have a question regarding the best practice to split a large CI job that is composed of multiple executions of gradle-build-action, or of steps that effectively depend on the artifacts generated by Gradle.
Would gradle-build-action cache the jar produced by the jar task for instance ?
If not, could i cache it myself by tweaking the gradle-build-action configuration, or by using actions/cache ? And if I were to do so by using actions/cache, would subsequent executions of gradle-build-action in a different workflow/job consider the restored cache as enough not to trigger the dependent tasks again ?

The gradle-build-action will save and restore content in the Gradle User Home directory. When you run a jar task the output is saved in the Project build directory, and so won't be saved by this action.
However, the Gradle Local Build Cache is stored in the Gradle User Home, and so will be saved and restored by this action. All .class files and other expensive outputs are saved in the build cache (assuming it's enabled). Note that the actual jar file outputs are not saved in the build cache, since these are normally inexpensive to recreate from the underlying .class files.
So while you could theoretically save the build directory using actions/cache, I think you're better off enabling the Gradle Build Cache and leveraging the built-in support of this action.


Disallowed importing concrete models without an application.

Removed fragile algorithm to find which application a model belongs to.
Fixed #21680, #21719. Refs #21794.

@timgraham I checked that there's nothing else to do in the code.
If I remember correctly, you had seen a paragraph of docs related to this that didn't make sense. I couldn't find it again. Do you remember where it is? If that helps, the "Features removed in 1.9" section of the release notes is correct.

Perhaps we should write a test for this...

... of course, anyone who dares change this code will feel free change the test, so I don't see much point in writing an artificial test.

Just the doc you already updated -- I already replaced the weird text with the current description. Check the 1.8 docs for the old version (which could be fixed) https://docs.djangoproject.com/en/1.8/ref/models/options/#app-label

LGTM

I don't understand why the tests don't pass, it looks like Jenkins didn't pick my fix. I'll rebase on top of the next commit to master and force push to see if I can get the tests to pass.


fix(core): serialize bigint in metadata to string

Issue #, if available: closes #616
Description of changes:
This PR introduces a new method called serialize to the prototype of the Segment and Subsegment objects. This method wraps the JSON.stringify() call that was previously made by the toString() and format() methods of these two classes.
When calling the JSON.stringify() function, the method also passes a replacer function. This replacer for now includes an if statement that casts BigInt objects to string. This avoids runtimes errors described in the linked issue.
By extracting the method, customers can easily override it and extend the replacer to suit their needs.
In addition to the method, the PR includes also changes to the type definition and the addition of a test.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6a9f999) 83.41% compared to head (63cb239) 83.49%.
 Consider uploading reports for the commit 5eed194 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #619      +/-   ##
==========================================
+ Coverage   83.41%   83.49%   +0.07%
==========================================
  Files          37       37
  Lines        1797     1805       +8
==========================================
+ Hits         1499     1507       +8
  Misses        298      298

see 3 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @carolabadeer thanks for the review and the comments, I have committed both your suggestions!


Correct some word errors and add a picture in readme.

Description

Correct some word errors.
Add a new picture to show the architecture.

If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
All SDK Contribution checklist:

[x] The pull request does not introduce [breaking changes]
[ ] CHANGELOG is updated for new features, bug fixes or other significant changes.
[ ] I have read the contribution guidelines.

General Guidelines and Best Practices

[x] Title of the pull request is clear and informative.
[ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

[ ] Pull request includes test coverage for the included changes.

API change check
API changes are not detected in this pull request.


Add colours dynamically.

The enhancement was mentioned in #8.

The method names are :
patternView.setPathColor(pathColor);

patternView.setCircleColor(pathColor);

patternView.setDotColor(pathColor);

Fix at version 1.0.6.


enterPrecommit is almost always double verifying the block

Feature Request
Summary
enterPrecommit almost always runs ValidateBlock, however in most cases this is duplicate logic as we have already prevoted the block. We only enter a lock after this line! This is the line where we run validate block: https://github.com/cometbft/cometbft/blob/main/internal/consensus/state.go#L1667
I suggest that we change this line here, to be one of two things:

Only validate the block after checking some cache for whether we have already validated this block.
Check  "did we prevote this block", and only ran ValidateBlock if we did not.

Problem Definition
We see we have duplicate VerifyBlock logic on synced full nodes:

This operation is expensive, especially in consensus critical contexts!
Proposal
I think that making a cache for checking if we've already validated a block is better, as it seems easier to also avoid extra redundant validations for finalizeCommit. I haven't read that logic, but guessing by how all 3 of these validate block calls are pretty similar in the amount of time spent, I'm guessing its a redundant ValidateBlock as well.

We had a discussion at some point regarding the multiple validations of the same block. I found this comment:
https://github.com/cometbft/knowledge-base/pull/14#issuecomment-1708062228

A similar discussion also here: https://github.com/cometbft/cometbft/pull/1391

We need to separate the validation of a block by the state-machine/blockchain, and the validation of a block by the application. The first validation, whether a block can be appended to the blockchain, could indeed be cached.

Just checked, we are indeed also doing an extra validate Block call in FinalizeCommit, that would be mitigated with a simple cache :)

Ah we are actually double validating the block in FinalizeCommit -- though this one is super easy to fix using ApplyVerifiedBlock:

Hi @ValarDragon @cason chiming in here
First of all, the caching mechanism that @ValarDragon is proposing shouldn't be hard to do, and would probably do the trick. So this is still and option.
However, going through @cason's old comments on the algorithm's valid(v) (which were made in the context of non-deterministic ProcessProposal, i.e., RFC 105) I think we can go further if we agree that the following is true:

valid(v) can be thought of consisting of two parts: CometBFT-valid and App-valid. The former being ValidateBlock, the latter being ProcessProposal.
The modifications made to the Tendermint algorithm in RFC 105 are correct (in particular, the calls to valid(v) that were removed

If we agree this is true, then we can conclude that CometBFT-valid can just be called where and only where App-valid is called; without affecting the algorithm's correctness.
If I didn't terribly miss anything in my reasoning, then the solution I would implement here is aligning ValidateBlock with ProcessProposal; IOW calling ValidateBlock only when we call ProcessProposal.
@cason what do you think?

@ValarDragon I just posted #2960 for review. We need to wait for @cason to come back from vacation (he's back next Mon). I would like to have his opinion on correctness.
In the meantime, do you think you folks could measure if this PR improves the performance and how much? Thanks!

Ah your right, that would argue that we can safely remove those two calls! That sounds good to me, will try to get it tested on mainnet soon! (@czarcas7ic is working on some automation for this, so that way we can also be getting benchmarks that adapt for the fixes we've already gotten in as well!)
From my understanding though, this would save those calls and be a notable time reduction!

The change in #2928 is nonetheless really relevant.

@cason what do you think?

I added my comments/concerns to https://github.com/cometbft/cometbft/pull/2960.

Following @cason's observations and comments, and an internal discussion, this is how we agreed to move forward:

We keep the part where we align the checks in enterPrecommit
But we revert the hunks that are removing the defensive panics
And, in order to keep the performance improvement that motivated this issue initially, we'll add a cache in memory as @ValarDragon was suggesting


select2_from_ajax does not show the field value when update ( with entity => false )

I'm using this field type
$this->crud->addField([             'label' => 'Booking',             'type' => 'select2_from_ajax',             'name' => 'booking',             'entity' => false,             'attribute' => 'booking',             'model' => "App\Models\PayByLink",             'data_source' => url('/admin/bookings/ajax-bookings-avalon-options'),         ]);
when I create a new entry it  works, the info is stored in database, but when I edit the entry the field shows empty.
I'm using entity False because this field is not a relationship, just a text field.  I use the same solution in many places in that project and I have the same problem, if i set entity to false the field shows empty
My workaround for now is to show the field as a regular text field in the update view  in order to show the stored value, but I would like to use the ajax search in the update view too.
PHP VERSION:
PHP 8.1.12-1ubuntu4.3 (cli) (built: Aug 17 2023 17:37:48) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.12, Copyright (c) Zend Technologies
with Zend OPcache v8.1.12-1ubuntu4.3, Copyright (c), by Zend Technologies
LARAVEL VERSION:
<IP_ADDRESS>
BACKPACK PACKAGE VERSIONS:
backpack/backupmanager: v5.0.0
backpack/basset: 1.2.1
backpack/crud: 6.3.0
backpack/devtools: 2.0.3
backpack/editable-columns: 3.0.2
backpack/generators: v4.0.2
backpack/logmanager: v5.0.1
backpack/permissionmanager: 7.1.1
backpack/pro: 2.0.18
backpack/settings: 3.1.0
backpack/theme-coreuiv2: 1.2.2
backpack/theme-coreuiv4: 1.1.1
backpack/theme-tabler: 1.1.1

Hi @miquelangeld ,
Huh... turns out this is a known issue (see https://github.com/Laravel-Backpack/CRUD/issues/3195 ), don't know why we haven't fixed this yet 🤷♂ Sorry for that.
TLDR:

the select2_from_ajax field requires a model
when you want a select2 from an API endpoint... you might not have a Model for that column (no relationship, nothing)

Possible solutions:

Solution A. We make select2_from_ajax work without a Model. Could be a breaking change though.
Solution B. We create a new field, select2_from_api that has zero relationship logic. Doesn't work with relationships at all.

What do you think @pxpm ? Any better solution? Which one do you think we should do?

Hi @tabacitu  In this project in all cases there is a linked model which I need to feed up with external data (sometimes an external api or sometimes an endpoint within the same project but from external databases, in fact the data is inserted correctly for the model PayByLink in the create view. the problem comes when I want to update and the field doesn't shows the stored data.
If I may, the idea of a select2_from_api field type may be a solution less disruptive for you, the need to query external data to fill a model field it's very present in all my projects.

Hmm @miquelangeld then maybe it's how you've defined that Model... sounds like it doesn't get picked up as a "normal" Eloquent model. What did you use for your "fake" models, did you use https://laravel-news.com/laravel-sushi ? I've used it a bunch of times and it's GREAT for creating Fake models that actually get their data from an array/file/api/etc.

Hi  @tabacitu I don't know if you have had a chance to take a look at my last update. Thanks in advance

Hi, any chance to get some news about this? Thanks!

@miquelangeld this is a technical limitation of the select2_from_ajax. Sorry, you are not doing nothing wrong.😞
We plan to create select2_from_api to avoid messing (with the already messy) select2 from ajax code that should just fit in your use case.
Can't promisse a date, but it's on our January roadmap if everything goes accordingly.

Cheers

Thanks @pxpm Great!

I close the issue since select2_json_from_api is released!!


Implementation of Switch component

React implementation of the Fabric Switch component.
I would appreciate some help in regards to accessibility. My assumption would be that it should be tabable, as well as toggleable with the spacebar.

We need to define a focus state for this component (with Adi's help), but I think we get the a11y "for free" via the input/label elements? I'm able to navigate to the component via keyboard-nav and (de)activate it.

We need to define a focus state for this component (with Adi's help), but I think we get at least some of the a11y "for free" via the input/label elements? I'm able to navigate to the component via keyboard-nav and (de)activate it.

I agree with you that it needs a focus state. I'm able to tab to the component (no visual focus though), but not toggle the component's value with my keyboard. Perhaps because it's an onClick and not onChange?

#13 solved the keyboard toggling for me. Does it still work as expected for you?

bcd4aac solved the keyboard toggling for me. Does it still work as expected for you?

Looks like accessible name (label text) is missing. The switch role requires an accessible name.
Unfortunately, an ARIA switch does not get accessible name from a native label element via the "for" attribute either (tested in Chrome with JAWS just for fun), so that's also a no-go.
Maybe a solution could be to use a button element instead of input/label? The button could wrap the component's children (which would act as the text content) alongside a span element that visually indicates the toggle state?

Thank you, @tor-martin-storsletten. I've now added support for the aria-labels and changed the input to a button for semantic accessibility.
I also went ahead and applied Tailwind's focus styles, @pearofducks, which we can change anytime. E.g. after consulting Adi.

Sounds good!


Release

Codecov Report
Base: 28.53% // Head: 28.53% // No change to project coverage :thumbsup:

Coverage data is based on head (0b8e3be) compared to base (eaf2b60).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #428   +/-   ##
=======================================
  Coverage   28.53%   28.53%
=======================================
  Files          54       54
  Lines        4921     4921
=======================================
  Hits         1404     1404
  Misses       3385     3385
  Partials      132      132

Flag
Coverage Δ

unittests
28.53% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Is there a Find and replace shortcut in AsciidocFX?

I use AsciidocFX for writing technical documentation and sometimes need to replace certain text. I am unable to find the replace command or shortcut. Tried the mac Cmd + H and and Cmd+Opt+H and few others, but with no luck.
Update: Checked the documentation and found http://asciidocfx.com/#truesome-shortcuts and https://github.com/ajaxorg/ace/wiki/Default-Keyboard-Shortcuts from there. The Command-Option-F does not work on 1.5.4 version on Mac.
Any suggestions please?

Hello @sengopal, could you please try to push Command-F twice ?

Does it work for you @sengopal  ?

@rahmanusta Thanks so much. It worked.  Could we please add it to the shortcut list. It would really help for searchers like me :)

You are welcome @sengopal, could you please edit Readme.adoc file for that?
Thanks

Of course :) Raised https://github.com/asciidocfx/AsciidocFX/pull/273 to add this to the Readme.adoc.

Thank you dear :)


Unable to render HTML (thus PDF) correctly when no images are referenced in md

Hey @BaileyJM02
I discovered this bug when I was trying to convert multiple markdown files into PDFs in a project.
I have created a repository for demonstration purpose: https://github.com/bernardkkt/md2pdf
There are 4 markdown files inside the docs folder: 01.md, 02.md, 03.md and 04.md. All except 03.md have the <img> tag. For 01.md and 02.md, the <img> tag points to a local image file, whereas for 04.md the <img> tag points to an internet link. The output files in HTML and PDF for these 3 files are fine.
However, when a markdown file contains no <img> tags at all, which is the case in 03.md, it will only be rendered with one word undefined. I have found the root cause and attempted to fix it, and if you are fine with the changes I will update this pull request after restoring the action.yml file.
You may refer to this attachment, which contains all the HTML and PDF files after running your GitHub Actions in my repository.
I am relying on a workaround to circumvent the issue, so there's no rush to include this fix if it needs more time for you to go through.

Sorry for the late reply, been super busy over the last couple of months - please mark this as not a draft when you see this and I'll merge.

@BaileyJM02 Me too, wasn't around for some time. It is now ready for review :)


Ollama test stability

We disabled tests in #203 for some of the Ollama integration work.
This is tracking the fact they are disabled and what we need to do to re-enable them.

Fixed with #222


use_shlex required with argparse api

I'm not sure why yet, but the extras type arguments fail to be parsed when handled through the legacy parser

maybe

Doesn't seem to be the case


fix: Rename legacy line and area charts

SUMMARY
Prior to the enablement of the generic chart x-axis, the relevant ECharts charts included the "Time-series" prefix whereas several of the deprecated NVD3 charts didn't:

After the enablement the "Time-series" prefix was correctly dropped from the ECharts charts (to reflect their more general form),

however this resulted in a somewhat of a convoluted UX, i.e., the legacy NVD3 charts only work when the x-axis is a time-series (temporal) and thus I thought the names should be updated to reflect this.
This PR renames the following NVD3 chart types:

Area Chart (legacy) → Time-series Area Chart (legacy)
Line Chart (legacy) → Time-series Line Chart (legacy)

It's worth noting that the the legacy NVD3 temporal bar chart was already named appropriately given it needed to distinguish itself from the legacy NVD3 non-temporal bar chart.

Finally I haven't changed any of the translations, but the keys have been updated so translations will still exist even though the don't include the Time-series prefix. I looked into updating these via Google Translate, but I'm not entirely sure how accurate they are.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
BEFORE

AFTER

TESTING INSTRUCTIONS
CI.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

/testenv down

@michael-s-molina would you mind taking a look at this?

@michael-s-molina would you mind taking a look at this? Note it's listed as a chore, but I'm not certain whether in fact it would be considered a fix for 4.0 given it ensures parity with how other legacy NVD3 charts are named within the generic chart x-axis realm.

I think you can consider this as a fix.

Ruh Roh.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 60.54%. Comparing base (2e5f3ed) to head (1aa4bb7).
Report is 17 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #28113      +/-   ##
==========================================
+ Coverage   60.49%   60.54%   +0.04%
==========================================
  Files        1931     1933       +2
  Lines       76241    76489     +248
  Branches     8566     8566
==========================================
+ Hits        46122    46309     +187
- Misses      28015    28076      +61
  Partials     2104     2104

Flag
Coverage Δ

hive
49.21% <ø> (+0.04%)
:arrow_up:

presto
53.83% <ø> (+0.03%)
:arrow_up:

python
63.57% <ø> (+0.08%)
:arrow_up:

unit
57.72% <ø> (+0.09%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@rusackas as a code owner would you mind reviewing this change?


Move PictureSequenceInfoField further up in PictureInfo

The PictureSequenceInfoField should be visible without scrolling.

This is superseded by #641, as decided by Werner (moving the field up would move the linked texts field down, which are present and relevant more frequently).


[IOS] Swipe to back on IOS

Hi,
At first, your lib is great!
Second, how can I swipe to go back to previous screen like Ios native ?

@hoanghai9650 Hi, thanks for advise, going to add it (iOS swipe back action, or else smth to interact/pass into lib)

Research result:
iOS nativ nav seems to use somthing close to android shared-transition (especially for top bar)
Animation itself is easy to replicate with
Navigation(
                ...
                contentTransformProvider = { navigationSlideInOut(it) },
                ...
            )

It is also possible to add global wraper (box + touchHandled) with touch action to perform global back action (contact me for more details if needed)
Will wait to shared-element-transition to be fixed and release, next will add fully functional example (or maybe direct implementation)

PS: feature/predictive-back - here is some very basic example on how it migh looks

UPD: work in progress, https://github.com/ComposeGears/Tiamat/tree/feature/content-transition-impl here is impl. Waiting for google to fix seekable transition bug


WIP: adding OCI output mode as buildctl does

Describe the changes in this pull request using active verbs such as Add, Remove, Replace ...
This PR has one little issue it is WIP. I am resolving it by discussion with the maintainers
Closes #604

@h4l0gen are you still working on this issue?

Hi @ashnamehrotra, sorry I am deeply involved in some other project's engagements. Please feel free to assign it to someone who wants to work on it. Again sorry that I am not able to find time ;)

@h4l0gen no problem, thank you for getting back to me. I will close this PR for now.


ir_dist alignment stuck

Describe the bug
While running ir_dist with alignment metric, the python process is stuck after the progress bar for "VJ" has been completed. The distance calculation for "VDJ" cannot get started and the second progress bar does not appear for a long time. After I hit the "Interrupt" button twice, I then see a lot of subprocesses get started after the first interruption, making me think the function was stuck somewhere.
KeyboardInterrupt                         Traceback (most recent call last)
File ~/mambaforge/envs/compbio/lib/python3.10/site-packages/tqdm/contrib/concurrent.py:51, in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     49 with PoolExecutor(max_workers=max_workers, initializer=tqdm_class.set_lock,
     50                   initargs=(lk,)) as ex:
---> 51     return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/process.py:766, in ProcessPoolExecutor.map(self, fn, timeout, chunksize, *iterables)
    764     raise ValueError("chunksize must be >= 1.")
--> 766 results = super().map(partial(_process_chunk, fn),
    767                       _get_chunks(*iterables, chunksize=chunksize),
    768                       timeout=timeout)
    769 return _chain_from_iterable_of_lists(results)

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/_base.py:610, in Executor.map(self, fn, timeout, chunksize, *iterables)
    608     end_time = timeout + time.monotonic()
--> 610 fs = [self.submit(fn, *args) for args in zip(*iterables)]
    612 # Yield must be hidden in closure so that the futures are submitted
    613 # before the first iterator value is required.

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/_base.py:610, in <listcomp>(.0)
    608     end_time = timeout + time.monotonic()
--> 610 fs = [self.submit(fn, *args) for args in zip(*iterables)]
    612 # Yield must be hidden in closure so that the futures are submitted
    613 # before the first iterator value is required.

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/process.py:734, in ProcessPoolExecutor.submit(self, fn, *args, **kwargs)
    733 # Wake up queue management thread
--> 734 self._executor_manager_thread_wakeup.wakeup()
    736 if self._safe_to_dynamically_spawn_children:

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/process.py:79, in _ThreadWakeup.wakeup(self)
     78 if not self._closed:
---> 79     self._writer.send_bytes(b"")

File ~/mambaforge/envs/compbio/lib/python3.10/multiprocessing/connection.py:200, in _ConnectionBase.send_bytes(self, buf, offset, size)
    199     raise ValueError("buffer length < offset + size")
--> 200 self._send_bytes(m[offset:offset + size])

File ~/mambaforge/envs/compbio/lib/python3.10/multiprocessing/connection.py:411, in Connection._send_bytes(self, buf)
    406 else:
    407     # Issue #20540: concatenate before sending, to avoid delays due
    408     # to Nagle's algorithm on a TCP socket.
    409     # Also note we want to avoid sending a 0-length buffer separately,
    410     # to avoid "broken pipe" errors if the other end closed the pipe.
--> 411     self._send(header + buf)

File ~/mambaforge/envs/compbio/lib/python3.10/multiprocessing/connection.py:368, in Connection._send(self, buf, write)
    367 while True:
--> 368     n = write(self._handle, buf)
    369     remaining -= n

KeyboardInterrupt:

During handling of the above exception, another exception occurred:

KeyboardInterrupt                         Traceback (most recent call last)
/users/rng/proj/tlc/src/sc/tcr_test.py in line 2
      98 #%%
----> 99 ir.pp.ir_dist(mdata, vdjdb, metric='alignment', sequence='aa',
      100     cutoff=30, n_jobs=128)

File ~/mambaforge/envs/compbio/lib/python3.10/site-packages/scirpy/ir_dist/__init__.py:220, in _ir_dist(adata, reference, metric, cutoff, sequence, key_added, inplace, n_jobs, airr_mod, airr_key, chain_idx_key, airr_mod_ref, airr_key_ref, chain_idx_key_ref)
    218 for chain_type in ["VJ", "VDJ"]:
    219     logging.info(f"Computing sequence x sequence distance matrix for {chain_type} sequences.")  # type: ignore
--> 220     result[chain_type]["distances"] = dist_calc.calc_dist_mat(
    221         result[chain_type]["seqs"], result[chain_type].get("seqs2", None)
    222     ).tocsr()
    224 # return or store results
    225 if inplace:

File ~/mambaforge/envs/compbio/lib/python3.10/site-packages/scirpy/ir_dist/metrics.py:200, in ParallelDistanceCalculator.calc_dist_mat(self, seqs, seqs2)
    197 # precompute blocks as list to have total number of blocks for progressbar
    198 blocks = list(self._block_iter(seqs, seqs2, self.block_size))
--> 200 block_results = process_map(
    201     self._compute_block,
    202     *zip(*blocks),
    203     max_workers=self.n_jobs if self.n_jobs is not None else cpu_count(),
    204     chunksize=50,
    205     tqdm_class=tqdm,
    206     total=len(blocks),
    207 )
    209 try:
    210     dists, rows, cols = zip(*itertools.chain(*block_results))

File ~/mambaforge/envs/compbio/lib/python3.10/site-packages/tqdm/contrib/concurrent.py:105, in process_map(fn, *iterables, **tqdm_kwargs)
    103     tqdm_kwargs = tqdm_kwargs.copy()
    104     tqdm_kwargs["lock_name"] = "mp_lock"
--> 105 return _executor_map(ProcessPoolExecutor, fn, *iterables, **tqdm_kwargs)

File ~/mambaforge/envs/compbio/lib/python3.10/site-packages/tqdm/contrib/concurrent.py:49, in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     46 lock_name = kwargs.pop("lock_name", "")
     47 with ensure_lock(tqdm_class, lock_name=lock_name) as lk:
     48     # share lock in case workers are already using `tqdm`
---> 49     with PoolExecutor(max_workers=max_workers, initializer=tqdm_class.set_lock,
     50                       initargs=(lk,)) as ex:
     51         return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/_base.py:649, in Executor.__exit__(self, exc_type, exc_val, exc_tb)
    648 def __exit__(self, exc_type, exc_val, exc_tb):
--> 649     self.shutdown(wait=True)
    650     return False

File ~/mambaforge/envs/compbio/lib/python3.10/concurrent/futures/process.py:780, in ProcessPoolExecutor.shutdown(self, wait, cancel_futures)
    777         self._executor_manager_thread_wakeup.wakeup()
    779 if self._executor_manager_thread is not None and wait:
--> 780     self._executor_manager_thread.join()
    781 # To reduce the risk of opening too many files, remove references to
    782 # objects that use file descriptors.
    783 self._executor_manager_thread = None

File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1096, in Thread.join(self, timeout)
   1093     raise RuntimeError("cannot join current thread")
   1095 if timeout is None:
-> 1096     self._wait_for_tstate_lock()
   1097 else:
   1098     # the behavior of a negative timeout isn't documented, but
   1099     # historically .join(timeout=x) for x<0 has acted as if timeout=0
   1100     self._wait_for_tstate_lock(timeout=max(timeout, 0))

File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1116, in Thread._wait_for_tstate_lock(self, block, timeout)
   1113     return
   1115 try:
-> 1116     if lock.acquire(block, timeout):
   1117         lock.release()
   1118         self._stop()

KeyboardInterrupt:

To Reproduce
ir.pp.ir_dist(mdata, vdjdb, metric='alignment', sequence='aa', cutoff=30, n_jobs=128)

I had also run the same command with cutoff=15 and it sometimes can run to completion. I have repeatedly had the same hanging behavior when cutoff=15.
I had tried also running with n_jobs=1 and it was still stuck after VJ dist mat calculation.
System
256 cores, 1TB RAM

OS: Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-146-generic x86_64)
Python version 3.10.11
Versions of libraries involved

-----
anndata     0.9.2
scanpy      1.9.3
-----
Levenshtein                 0.21.1
PIL                         9.5.0
adjustText                  NA
airr                        1.4.1
argcomplete                 NA
asttokens                   NA
awkward                     2.3.1
awkward_cpp                 NA
backcall                    0.2.0
certifi                     2023.11.17
cffi                        1.15.1
charset_normalizer          3.1.0
cloudpickle                 2.2.1
colorama                    0.4.6
comm                        0.1.3
cupy                        12.2.0
cupy_backends               NA
cupyx                       NA
cycler                      0.10.0
cython_runtime              NA
cytoolz                     0.12.0
dask                        2023.6.0
dateutil                    2.8.2
debugpy                     1.6.7
decorator                   5.1.1
defusedxml                  0.7.1
exceptiongroup              1.1.1
executing                   1.2.0
fastrlock                   0.8.2
google                      NA
h5py                        3.9.0
idna                        3.4
igraph                      0.10.8
iniconfig                   NA
ipykernel                   6.23.1
ipywidgets                  8.0.6
jedi                        0.18.2
jinja2                      3.1.2
joblib                      1.2.0
kiwisolver                  1.4.4
leidenalg                   0.9.1
llvmlite                    0.40.0
louvain                     0.8.0
markupsafe                  2.1.3
matplotlib                  3.7.1
matplotlib_inline           0.1.6
mpl_toolkits                NA
mudata                      0.2.3
muon                        0.1.5
natsort                     8.3.1
networkx                    3.1
numba                       0.57.0
numexpr                     2.8.4
numpy                       1.24.3
packaging                   23.1
pandas                      1.5.3
parso                       0.8.3
patsy                       0.5.3
pexpect                     4.8.0
pickleshare                 0.7.5
pkg_resources               NA
platformdirs                3.5.3
pluggy                      1.0.0
pooch                       v1.7.0
prompt_toolkit              3.0.38
psutil                      5.9.5
ptyprocess                  0.7.0
pure_eval                   0.2.2
py                          NA
pycparser                   2.21
pydev_ipython               NA
pydevconsole                NA
pydevd                      2.9.5
pydevd_file_utils           NA
pydevd_plugins              NA
pydevd_tracing              NA
pygments                    2.15.1
pynndescent                 0.5.10
pyparsing                   3.0.9
pytest                      7.3.2
pytz                        2023.3
rapidfuzz                   2.15.1
requests                    2.31.0
rpy2                        3.5.11
scanwrap                    NA
scipy                       1.10.1
scirpy                      0.13.0
seaborn                     0.12.2
session_info                1.0.0
setuptools                  67.7.2
six                         1.16.0
sklearn                     1.2.2
socks                       1.7.1
stack_data                  0.6.2
statsmodels                 0.14.0
texttable                   1.6.7
threadpoolctl               3.1.0
tlz                         0.12.0
toolz                       0.12.0
tornado                     6.3.2
tqdm                        4.65.0
tracerlib                   NA
traitlets                   5.9.0
typing_extensions           NA
tzlocal                     NA
umap                        0.5.3
urllib3                     2.0.3
wcwidth                     0.2.6
yaml                        6.0
yamlordereddictloader       NA
zipp                        NA
zmq                         25.1.0
zoneinfo                    NA
-----
IPython             8.14.0
jupyter_client      8.2.0
jupyter_core        5.3.0
-----
Python 3.10.11 | packaged by conda-forge | (main, May 10 2023, 18:58:44) [GCC 11.3.0]
Linux-5.4.0-146-generic-x86_64-with-glibc2.31
-----
Session information updated at 2023-12-07 12:33

Additional context

Hi Rachel,
thanks for reporting this. I'm not entirely sure what could be causing this issue.
How many cells does your mdata have? And do you have the same problem when you use a small example dataset (e.g. scirpy.datasets.wu2020_3k()?

My mdata has 71706 cells in the airr modality and iedb database has 34509 TCRs.
I don't have the same problem if I use a small dataset.
I think something happened at the end of VJ dist mat calculation, where a thread fails to shut down and then it gets stuck in lock indefinitely, suggested by this traceback:
File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1096, in Thread.join(self, timeout)
   1093     raise RuntimeError("cannot join current thread")
   1095 if timeout is None:
-> 1096     self._wait_for_tstate_lock()
   1097 else:
   1098     # the behavior of a negative timeout isn't documented, but
   1099     # historically .join(timeout=x) for x<0 has acted as if timeout=0
   1100     self._wait_for_tstate_lock(timeout=max(timeout, 0))

File ~/mambaforge/envs/compbio/lib/python3.10/threading.py:1116, in Thread._wait_for_tstate_lock(self, block, timeout)
   1113     return
   1115 try:
-> 1116     if lock.acquire(block, timeout):
   1117         lock.release()
   1118         self._stop()

KeyboardInterrupt:

Could it be a generic threading problem? https://github.com/open-telemetry/opentelemetry-python/issues/2284

You could try to increase the block size, which will result in less threads that could be stuck :D The only downside is that the progressbar doesn't update as continuously:
dist_calc = ir.ir_dist.metrics.AlignmentDistanceCalculator(cutoff, block_size=5000, n_jobs=32, ...)
sc.pp.ir_dist(adata, metric=dist_calc)

Hi @racng,
I believe I found a solution to this in #473.
joblib.Parallel is allegedly more robust than the multiprocessing module from the standard library.
An added benefit is that you can also use the dask backend via joblib.parallel_config to distribute the workload across multiple machines. The version from that PR also has an improved version of the alignment distance that is faster.
Would be great if you can give it a try! You can install it with
pip install git+git@github.com:icbi-lab/scirpy.git@ir-dist-parallelism


Trouble using Make to build examples.

I'm seeing build errors with the different TFLM build attempts I've been making. I have not tried a Bazel build, if there are build instructions on how to do a Bazel build for this repo on Windows I would try that, currently I am trying to build with a Make project. I am successfully building Tensorflow Lite (from the main Tensorflow repo) with CMake, but with that build I don't have clarity on what is just needed for TFLM.
Build attempt #1:
Following the first Micro Speech example "Deploy to ARC EM SDP" found here:
https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/examples/micro_speech
Command line commands from an empty folder:
git clone https://github.com/tensorflow/tflite-micro tflite-micro
cd tflite-micro
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=arc_emsdp ARC_TAGS=reduce_codesize OPTIMIZED_KERNEL_DIR=arc_mli generate_micro_speech_mock_make_project
Result: Several packages are downloaded, but the end result is "make: *** No rule to make target 'generate_micro_speech_mock_make_project'.  Stop."
Build attempt #2:
Following the  Micro Speech example "Deploy to ESP32" found here:
https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/examples/micro_speech
ESP IDF should be properly installed.
Command line commands from an empty folder:
git clone https://github.com/tensorflow/tflite-micro tflite-micro
cd tflite-micro
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=esp generate_micro_speech_esp_project
Result:
There are a couple "FIND: Parameter format not correct" messages and "File not found - *.cc", "File not found - *.h". It appears that pigweed is downloaded, along with some other packages. Ultimately, the build fails with "make: *** No rule to make target 'generate_micro_speech_esp_project'.  Stop.".
Build attempt #3:
Following the Micro Speech example "Deploy to Sparkfun Edge" found here:
https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/examples/micro_speech
Command line commands from an empty folder:
git clone https://github.com/tensorflow/tflite-micro tflite-micro
cd tflite-micro
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=sparkfun_edge OPTIMIZED_KERNEL_DIR=cmsis_nn micro_speech_bin
Results: Essentially the same results as build attempt #2 above.
Build attempt #4:
Following the Micro Speech example "Deploy to STM32F746" found here:
https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/examples/micro_speech
Command line commands from an empty folder:
git clone https://github.com/tensorflow/tflite-micro tflite-micro
cd tflite-micro
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=disco_f746ng OPTIMIZED_KERNEL_DIR=cmsis_nn generate_micro_speech_mbed_project
Same results as above.
Build attempt #5:
Following the Hello World example "Deploy to SparkFun Edge" found here:
https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/examples/hello_world
Command line commands from an empty folder:
git clone https://github.com/tensorflow/tflite-micro tflite-micro
cd tflite-micro
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=sparkfun_edge hello_world_bin
Same results as above.

Windows is currently fully community supported and the builds are not part of any CI or something that the TFLM engineers are set up to test. Often errors are the result of some incompatibility with the shell commands (which appears to be the case here).
Bazel is linux only and TFLM does not have any support for cmake at this time.
I would recommend attempting to build the reference kernels and pasting the full error log from the following command:
make -f tensorflow/lite/micro/tools/make/Makefile hello_world

Also, if the final goal is to use TFLM from within an IDE, I would suggest trying out the project generation script. Something like (copied from https://github.com/tensorflow/tflite-micro/blob/16a9b1e8bfd69a73fcadff210aecf63085d9cbd4/tensorflow/lite/micro/tools/ci_build/test_project_generation.sh):
git clone tensorflow/tflite-micro tflite-micro
cd tflite-micro
python3 tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py /tmp/tflm-out -e hello_world
cp tensorflow/lite/micro/tools/project_generation/Makefile /tmp/tflm-out
cd /tmp/tflm-out
make -j8 examples

If you can get these steps to work on Windows, you can also use the tree  from /tmp/tflm-out in your IDE.

Thank you! I am attaching the errors for the two recommendations in "make hello_world errors.txt" (the first step suggested) and "python create_tflm_tree errors.txt". The python tree build might work if the first one worked?
I didn't know about the python tree build script, that could be helpful. Hopefully I can fix this hello_world make build and perhaps get some traction.
make hello_world errors.txt
python create_tflm_tree errors.txt

The error that you are seeing is:
FIND: Parameter format not correct
FIND: Parameter format not correct
File not found - *.cc
File not found - *.h

possibly coming from https://github.com/tensorflow/tflite-micro/blob/16a9b1e8bfd69a73fcadff210aecf63085d9cbd4/tensorflow/lite/micro/tools/make/Makefile#L404-L408
If you can figure out how to get this working on Windows, that might get you unblocked.

Thank you... so this makefile will not be tested to support Windows? If I work through it perhaps it could be a PR?
Just knowing that I'm not doing something else wrong and that I need to debug this Makefile is very helpful.

Correct, the makefile is not tested on Windows.
But any fixes that you have can certainly be reviewed and merged so please feel free to make a pull request.

Hello @advaitjain,
I was able to get back to this task and work through the issues keeping everything from working on Windows. I have added four python scripts to the make/ folder, and modified for makefile or bash scripts to fix the issues, I hope incorporating a few python scripts to work around the problem of running "Find" on Windows is OK.
If these changes could be pulled in, I would then be able to run the project generator, then from there automate adding the TFLM source code to a Visual Studio C/C++ project. How do you recommend I submit these changes for review? A normal pull request?
Note: I don't see any ..\google Make_internal.inc files, so I wasn't able to fully test that.

Note: In addition, the list_third_party_sources target in the Makefile doesn't appear to correctly print the third party source files. The  list_third_party_headers target seems to work properly. When you run the create_tflm_tree.py script, the third party source code doesn't get copied over as a result of this issue.

Just following up on this - Do you have a pull request with your changes? I'd be happy to take a look.

Hi Pete!
I believe I attempted a pull request a few weeks ago but wasn't able to, I'll check again. I have a tool I've written that pulls everything down from GitHub, adds python scripts, modifies makefiles, runs the make file, makes modifications to third party libraries, adds everything to a Visual Studio project, and everything builds and runs perfectly with no errors or warnings. I've slept several times since a few weeks ago when I did that, I need to make sure a pull request will cover everything.

@petewarden FYI I have signed a CLA, but it appears my "Create Pull Request" button is still disabled. Do I just need to wait a little while to see that enabled?

Hi @petewarden and @advaitjain ,
Below is an update on my progress, in case you are interested. Everything is pulled into Visual Studio and Virtuoso and building properly. I'm only appearing to see the "yes" get detected with the speech demo data:
https://youtu.be/Hb0-PUMRyV8
I'm very close to having everything up and running, real system audio should be streaming correctly, I'm just not sure if there's an issue with the LatestAudioTimestamp() that I'm using. If the mock audio worked, then I think I should be able to get the live audio stream working, but at the moment the mock audio isn't quite working either...
Are you still interested in a pull request from me to pull in the changes to let TFLM compile in Visual Studio without errors/warnings?
Thanks!

@petewarden I'm also wondering if the micro speech demo isn't finished yet? I see a note about needing to fix a big with the start_ms being less than zero. Also, it appears that inference is only performed every 100ms when using the mock data, there are 5 new slices (20ms each) added each pass through loop(), but then inference only happens once per pass through loop(), so the LSTM might be expecting to be evaluated on a 20ms window stride, but actually evaluating every 100ms?

Are you still interested in a pull request from me to pull in the changes to let TFLM compile in Visual Studio without errors/warnings?

Assuming that the change fit well with the existing build system, we would be open to a PR that adds support for Visual Studio.
Also w.r.t. https://github.com/tensorflow/tflite-micro/issues/407#issuecomment-901469738, we can figure out what a reasonable path forward would be.

hi,
I'm getting similar issues here.
I have WSL in my windows OS and already clone the repo from tflite-micro. I have everything in the Ubuntu distribution and installed dependencies asked (numpy and Pillow). However, whenever in tfllite-micro calling the Makefile I get this:
sudo make -f tensorflow/lite/micro/tools/make/Makefile test_hello_world_test
tensorflow/lite/micro/tools/make/downloads/flatbuffers already exists, skipping the download.
tensorflow/lite/micro/tools/make/downloads/kissfft already exists, skipping the download.
tensorflow/lite/micro/tools/make/downloads/pigweed already exists, skipping the download.
make: *** No rule to make target 'test_hello_world_test'.  Stop.
hello_world folder exists in examples but inside there are evaluate files (.py evaluate_test.cc etc)
I'm getting a bit crazy about this and a hand would be appreciated.
All the best,
Javier


Uncaught UnboundLocalError in main.py line 319

Aider version: <module 'aider.version' from '/home/dmtarmey/aider_venv/lib/python3.12/site-packages/aider/version.py'>
Python version: 3.12.6
Platform: Linux-6.11.0-5-cachyos-x86_64-with-glibc2.40
Python implementation: CPython
Virtual environment: Yes
OS: Linux 6.11.0-5-cachyos (64bit)
Git version: git version 2.46.1
An uncaught exception occurred:
Traceback (most recent call last):
  File "aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "main.py", line 564, in main
    if not sanity_check_repo(repo, io):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "main.py", line 319, in sanity_check_repo
    if bad_ver:
       ^^^^^^^
UnboundLocalError: cannot access local variable 'bad_ver' where it is not associated with a value

Thank you for filing this issue.
The fix is available in the main branch. You can get it by installing the latest version from GitHub:
aider --install-main-branch

# or.

python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git

If you have a chance to try it, please report if it works better for you.


Support Variable Interest Rates

Hi,
Been experimenting with an interest rate that isn't static - so far, looking at the docs, it seems that the library is mostly built around a rate fixed for the lifetime of the loan. I see in some other countries it seems a 30-year fix is possible, but where I live there is no chance of fixing for more than 5 years, and many products are trackers or variable. A common product is also an initial fixed rate, which then reverts to a tracker/SVR after the fix ends in 1/2/3/5 years, so I have so far managed to simulate this situation by creating two separate amortisation schedules - an initial one, for the fixed rate, and then one at a higher (but still assumed invariable, for now) rate for the rest of the product life, without having to break into the builder classes. However, it would be a nice feature if we could pass in an iterable of different interest rates for each month, or perhaps take some type of generator object, to forecast changes in the base rates and to simulate variable mortgages.
Thanks for the nice work BTW. I'm connecting this straight into KNIME, (eventually) mangling the data tables in there and calling out to the JAR - sooo much better than any more awful spreadsheets :D

Hello,
Sorry for the long response.
You are right - I didn't think about variable interest rates while developing this lib.
I like the idea of some objects with custom implementation for providing interest rates for each month.
I'll try to show you the experimental implementation in the near future and we can discuss it.
Thank you for your request!

Sounds good! More than happy to give a try and provide feedback - although I'm not a Java developer just FYI, so that will limit how helpful I can be. In C++ I think using generics with something overloading operator() is a nice way of doing it, so I like what you're describing. Could then maybe provide a couple of somple classes that implement this interface and provide for typical patterns using just a few args in the ctor - stepped rates, linearly-increasing, feeding items from an array, etc. etc.
Cheers :D

I'm going to be looking at all this (simulating real rates, that is) this weekend, just if you happen to have anything prepared! Should I check out (and checkout) this branch: feature/custom-interest-rate ?

Yes, you can find the changes in this branch. Make sure you have the updated version because my last push was several minutes ago.
I've added a parameter in the loan class which is an entry point for providing custom monthly interest rates. It is a functional interface with input and output.
This function will be called from the calculator implementation for every monthly payment
The input is additional data that is needed for a calculation. For now, I've added only the payment number to this class.
The must be an interest rate for the requested month.
Usage example you can find in LoanAmortizationCalculatorTest#shouldUseCustomInterestRate
   @Test
    void shouldUseCustomInterestRate() {
        final List<BigDecimal> rates = Arrays.asList(
                BigDecimal.valueOf(0.0036),
                BigDecimal.valueOf(0.0031),
                BigDecimal.valueOf(0.0031),
                BigDecimal.valueOf(0.0053),
                BigDecimal.valueOf(0.0022));

        final Loan loan = Loan.builder()
                .amount(BigDecimal.valueOf(500000.32))
                .term(5)
                .monthlyInterestRateProvider(input -> rates.get(input.getPaymentNumber()))
                .build();

        LoanAmortization amortization = calculator.calculate(loan);
        assertNotNull(amortization);

        for (MonthlyPayment payment : amortization.getMonthlyPayments()) {
            assertEquals(rates.get(payment.getMonthNumber()), payment.getInterestRate());
        }
    }

It is a concept and I didn't check the correctness of calculations. So if you have real examples of such loan schedules I would be happy to use them in test data.

Okay, I have been fighting Java today, so will probably pull tomorrow morning. I have done some basic comparisions myself to test, but I don't have any accurate schedules to compare against so far, just looking at general numbers (introductory payments, follow-on payments, ..). Harder to find these as a consumer - do you know anywhere I can get/make some? If not, I think I have some spreadsheets that should generate full tables. I may as well also make a repo for my KNIME workflow and binding code, so you could use that for testing if you wanted at all.


Reduced Travis CI runtime

This addresses the very long runtimes of Travis CI builds (#1070). The number of builds is reduced and each build is kept within some minutes.
Currently the total time has been decreased from 2 1/2 hours to 5 minutes.
CI Builds

[x] Clang
[x] GCC
[x] C++11 on / off
[x] CMake
[x] Autotools
[x] WCL
[x] OS X (clang only)
[x] GTest
[x] Coverage

Cron Builds
In addition to the faster CI builds, long running steps are moved into Cron Builds.

[x] Full check (make full_check; autotools only)

OS X builds take badly long to spawn. Sometimes > 10 minutes.

I have configured a cron build for my fork (daily); let's see if it works.

Changes Unknown when pulling 13f73ebcd4f0e68e9defaac5a910e00f455a0070 on offa:issue1070/ci into ** on cpputest:master**.

Changes Unknown when pulling 772db1cb918c058b9f9790940da7e59ea5db52b8 on offa:issue1070/ci into ** on cpputest:master**.

Looks like the check_special_situations target on autotools isn't supposed to run on linux build!?

Could you also run the OSX at least on Chron. We do find problems with that build.

Could you also run the OSX at least on Chron. We do find problems with that build.

Hmm, have to figure out how to add this as cron-only.

Changes Unknown when pulling 4a1dda4d361b974b7ac048b426b030c53bd5c89a on offa:issue1070/ci into ** on cpputest:master**.

Waiting for MaxOSX Cron job ?

I have to find out, how to add a OS X build to Cron's only. If it's added as a usual job, it'll be included the normal builds as well (at least it needs to spawn) – this will increase the feedback time to hours (= bad).
Unfortunately there's no "add-this-build-if-cron-only" setting yet.

An idea to handle this limitation (just an idea yet …):

Build the ci jobs as usual
If successful merge it into a "nightly" branch
Build cron's from the "nightly" branch (full builds, checks, environments)
Extension possible: If successful here, push nightly tag and pack a release to that version

Hmm.. ok, though I think they'll fix the spawn time in the future. So, we'd better keep it out for now and not complicate it too much, right? I'll merge this. Thanks!

Yes, that's the best solution at the moment I guess.
@basvodde If not done yet, you have to add a cron job at travis (doc).


🐛  [ERROR:flutter/shell/common/shell.cc(93)] Dart Error: Dart_LookupLibrary: library 'packagename/utils/fcm.dart' not found.

Bug report
Describe the bug
Those error happen on start up application and when received the notification from background in release mode. Foreground notification work well.

Additional context
FirebaseMessaging is called in fcm class

Flutter doctor
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[√] Flutter (Channel stable, 3.3.0, on Microsoft Windows [Version 10.0.19044.2006], locale en-US)
Checking Android licenses is taking an unexpectedly long time...[√] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[√] Chrome - develop for the web
[!] Visual Studio - develop for Windows (Visual Studio Enterprise 2022 17.2.6)
    X Visual Studio is missing necessary components. Please re-run the Visual Studio installer for the "Desktop development with C++" workload, and include these components:
        MSVC v142 - VS 2019 C++ x64/x86 build tools
         - If there are multiple build tool versions available, install the latest
        C++ CMake tools for Windows
        Windows 10 SDK
[√] Android Studio (version 2021.2)
[√] Connected device (4 available)
[√] HTTP Host Availability

! Doctor found issues in 1 category.

Flutter dependencies
Click To Expand
environment:
  sdk: ">=2.17.6 <3.3.0"

dependencies:
  flutter:
    sdk: flutter
  flutter_localizations:
    sdk: flutter
  flutter_svg: ^0.22.0
  fl_chart: ^0.55.0
  intl: ^0.17.0
  rxdart: ^0.27.3
  shared_preferences: ^2.0.15
  fluttertoast: ^8.0.9
  retrofit: ^3.0.1+1
  logger: ^1.1.0
  json_annotation: ^4.5.0
  package_info_plus: ^1.4.2
  device_info_plus: ^3.2.3
  firebase_core: ^1.17.0
  firebase_crashlytics: ^2.8.0
  firebase_analytics: ^9.1.8
  firebase_messaging: ^12.0.0
  awesome_notifications: ^0.6.21
  android_intent: ^2.0.2
  device_apps: ^2.2.0
  syncfusion_flutter_gauges: ^20.1.61
  connectivity_plus: ^2.3.0
  need_resume: ^1.0.6
  cached_network_image: ^3.1.0
  shamsi_date: ^0.15.0
  liquid_progress_indicator: ^0.4.0
  persian_number_utility: ^1.1.1
  flutter_custom_tabs: ^1.0.4
  substring_highlight: ^1.0.33
  app_settings: ^4.1.6
  open_store: ^0.2.4
  marquee: ^2.2.0
  flutter_html: ^2.0.0
  url_launcher: ^6.1.5
  image_picker: ^0.8.5+3
  path_provider: ^2.0.10
  file_picker: ^4.5.1
  filesize: ^2.0.1
  open_filex: ^3.4.0
  share_plus: ^4.0.4
  google_sign_in: ^5.3.1
  devicelocale: ^0.5.2
  tapsell_plus: ^2.1.7
  google_mobile_ads: ^2.0.1
  chewie: ^1.3.3
  video_player: ^2.4.2
  carousel_slider: ^4.1.1
  flutter_web_frame: ^0.0.3
  universal_io: ^2.0.4
  flutter_linkify: ^5.0.2
  uuid: ^3.0.6
  progress_indicators: ^1.0.0
  url_strategy: ^0.2.0
  flutter_math_fork: ^0.4.2+2
  sms_autofill: ^2.2.0
  visibility_detector: ^0.3.3
  steps_indicator: ^1.3.0
  flutter_staggered_grid_view: ^0.6.1
  audioplayers: ^0.20.1
  wakelock: ^0.6.1+2
  permission_handler: ^9.2.0
  idb_sqflite: ^1.1.0
  sqflite_common_ffi: ^2.1.1
  sqflite: ^2.0.2+1
  universal_html: ^2.0.8
  metrix_plugin: ^1.4.2
  just_audio: ^0.9.27
  flick_video_player: ^0.5.0
  dotted_border: ^2.0.0

dependency_overrides:
  provider: ^6.0.3

dev_dependencies:
  flutter_lints: ^1.0.4
  json_serializable: ^6.2.0
  retrofit_generator: ^4.0.1
  build_runner: ^2.1.11
  device_preview: ^1.1.0
  flutter_test:
    sdk: flutter

Same me!

@Zeinab-Kouhkan  @SarahGhorbani
The error is similar to https://github.com/firebase/flutterfire/issues/9446, so solution is as mentioned https://github.com/firebase/flutterfire/issues/9446#issuecomment-1240554285.
Use the solution mentioned which should resolve the issue. Closing for now as this is already fixed.


[md-dialog] Presets not working

Since v0.7.0, we can't get events from md-dialog presets (alert, confirm).
The @click event was not updated.
I submitted a PR to fix this : #466

Ok already done in fe1d3ce977fb49d77db8b6301895c9dc9ef02088.


Update TUnit to 0.2.210

This PR contains the following updates:

Package
Type
Update
Change

TUnit
nuget
patch
0.2.181 -> 0.2.210

TUnit.Assertions
nuget
patch
0.2.181 -> 0.2.210

TUnit.Core
nuget
patch
0.2.181 -> 0.2.210

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about these updates again.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

Edited/Blocked Notification
Renovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR.
You can manually request rebase by checking the rebase/retry box above.
⚠ Warning: custom changes will be lost.


source local kubebuilder assests before running tests

increase duration b/n fast reconcile events in tests

Signed-off-by: Leela Venkaiah G<EMAIL_ADDRESS>

in #191 I had kubebuilder assets in the default path where envtest looks for
that shadowed local binaries which should be used in the unit tests
this PR sources local binaries before running tests

I was running #188 in a relatively less resource machine and that slightly effected the reconcile time for ratelimiter
Based on feedback from @bindrad, I found reconciler is exhausting 500 fast reconciles and the so increased the duration b/n fast reconcile triggers

# make test
/root/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1alpha1/zz_generated.deepcopy.go
go vet ./...
/root/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /root/toolbox/orc/ocs-osd-deployer/testbin
test -f /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh || curl -sSLo /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.6.3/hack/setup-envtest.sh
source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; fetch_envtest_tools /root/toolbox/orc/ocs-osd-deployer/testbin
Using cached envtest tools from /root/toolbox/orc/ocs-osd-deployer/testbin
>>>> New Change <<<< source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=converged go test ./... -coverprofile converged-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 96.197s coverage: 76.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    17.077s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]
>>>> New Change <<<< source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=provider go test ./... -coverprofile provider-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 95.710s coverage: 74.9% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    15.333s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]
>>>> New Change <<<< source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=consumer go test ./... -coverprofile consumer-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 84.468s coverage: 72.6% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    15.587s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]

restested
# make test
/root/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1alpha1/zz_generated.deepcopy.go
go vet ./...
/root/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
mkdir -p /root/toolbox/orc/ocs-osd-deployer/testbin
test -f /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh || curl -sSLo /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.6.3/hack/setup-envtest.sh
source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; fetch_envtest_tools /root/toolbox/orc/ocs-osd-deployer/testbin
Using cached envtest tools from /root/toolbox/orc/ocs-osd-deployer/testbin
source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=converged go test ./... -coverprofile converged-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 95.986s coverage: 76.8% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    15.964s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]
source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=provider go test ./... -coverprofile provider-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 95.046s coverage: 74.9% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    16.105s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]
source /root/toolbox/orc/ocs-osd-deployer/testbin/setup-envtest.sh; setup_envtest_env /root/toolbox/orc/ocs-osd-deployer/testbin; \
DEPLOYMENT_TYPE=consumer go test ./... -coverprofile consumer-cover.out
setting up env vars
?       github.com/red-hat-storage/ocs-osd-deployer     [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/api/v1alpha1        [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/controllers 85.520s coverage: 73.5% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/readinessProbe      [no test files]
ok      github.com/red-hat-storage/ocs-osd-deployer/readinessProbe/readiness    13.420s coverage: 73.3% of statements
?       github.com/red-hat-storage/ocs-osd-deployer/templates   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/testutils   [no test files]
?       github.com/red-hat-storage/ocs-osd-deployer/utils       [no test files]


Vineyard is no longer a hard dependency for the client package.

Related issue number
Fixes #779

The issue has been fixed on main and a release will but published tomorrow.


Create a document for Firefox Accounts

cc @bcolsson
I think this was done? The document might need a refresh though
https://github.com/mozilla-l10n/localizer-documentation/blob/main/src/webprojects/fxa.md


[stable/jaeger-operator] fixed link to elasticseach from incubator to stable

@dzmitry-lahoda
Please bump chart to the latest version. Thank you.

@dzmitry-lahoda please bump the chart version
/assign


[enchancment] Updated requirements

Coverage remained the same at 100.0% when pulling 608d54e2cad0f62e23a958ca12ed85705a5dfa6f on oltarzewskik:requirements into 37d545e2fda4bb65a93610b9f857986821f93709 on openwisp:master.

I fixed the build in a way that I think is cleaner. The distro is more up to date and the solution should be more robust.
See my changes here: https://github.com/openwisp/django-loci/pull/27


[ui] Add stories to document the components

Storybook seems to be a nice tool to document the ui components

@aulneau @kyranjamie thoughts on this tool?

I'd love for us to use Storybook. This'll help tremendously building out and testing components. Thanks @friedger

I'm very open to it, too! There is a pr here that you can work from @friedger https://github.com/blockstack/ui/pull/48

Closing this in favor of https://github.com/blockstack/ui/issues/50


Accessing t helper in an initializer

Hi, how can I access the t helper in an initializer?
I'm trying container.lookup('utils.t') but it does not work. Any ideas?
Thanks.

@tehmaestro as suggested in the README this library has been deprecated in favor of ember-i18n

Ok, but I cannot update to Ember 1.12 as of right now ..

Found a way. For anyone searching this, use container.lookup('utils:t') to get the t helper, but make sure you run your initializer after the t initializer: after: ['bla', 'bla', 't'],