Get last non-null value instead of latest XBRL filing.

Overview
Closes #3309.
What problem does this address?
A bunch of utilities sent diff-only reports to update their 2021 data, which led to some important XBRL tables being mostly-null.
What did you change?
After much debate and investigation, we decided to change the XBRL deduplication to take the latest non-null value (across multiple filings) instead of taking the value from the latest single filing.
Testing
How did you make sure this worked? How can a reviewer verify this?
@cmgosnell I'd love if you could take a look and see if this affects rate base table in the expected way.
# To-do list
- [ ] If updating analyses or data processing functions: make sure to update or write data validation tests (e.g.,  `test_minmax_rows()`)
- [ ] Update the [release notes](../docs/release_notes.rst): reference the PR and related issues.
- [ ] Ensure docs build, unit & integration tests, and test coverage pass locally with `make pytest-coverage` (otherwise the merge queue may reject your PR)
- [ ] Review the PR yourself and call out any questions or issues you have
- [ ] For minor ETL changes or data additions, once `make pytest-coverage` passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and [run relevant data validation tests](https://catalystcoop-pudl.readthedocs.io/en/latest/dev/testing.html#data-validation) using `pytest` and `--live-dbs`.
- [ ] For significant ETL, data coverage or analysis changes, once `make pytest-coverage` passes, ensure the full ETL runs locally and [run data validation tests](https://catalystcoop-pudl.readthedocs.io/en/latest/dev/testing.html#data-validation) using `make pytest-validate` (a ~10 hour run). If you can't run this locally, run the `build-deploy-pudl` GitHub Action (or ask someone with permissions to). Then, check the logs on the `#pudl-deployments` Slack channel or `gs://builds.catalyst.coop`.

before your change
Correcting 95.02% of all non-null reported values (439/462) out of a total of 143284 original records.
post change

cc @e-belfer

Oy, the integration tests failed with a bunch of error thresholds being broken:

```
ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type balancing_authority -y 2022 --limit-by-state --no-dissolve -o -balancing_authority_geometry_limited.parquet-expected_cols0] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type balancing_authority -y 2021 -y 2022 --no-dissolve -o -balancing_authority_geometry.parquet-expected_cols1] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/console_scripts_test.py::test_pudl_service_territories[inprocess-pudl_service_territories --entity-type utility -y 2022 --dissolve -o -utility_geometry_dissolved.parquet-expected_cols2] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/datasette_metadata_test.py::test_database_metadata - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_annual_eia_outputs[gen_fuel_by_generator_energy_source_owner_eia923] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_null_rows[mcoe_generators-0.9] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_outputs_by_table_suffix[eia861] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_outputs_by_table_suffix[ferc714] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_ferc714_outputs[out_ferc714__summarized_demand] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_ferc714_outputs[out_ferc714__respondents_with_fips] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_service_territory_outputs[out_eia861__yearly_balancing_authority_service_territory] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_service_territory_outputs[out_eia861__yearly_utility_service_territory] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
ERROR test/integration/output_test.py::test_state_demand_outputs[out_ferc714__hourly_estimated_state_demand] - AssertionError: Found errors while running tests on the calculations:
                                  error_frequency  tolerance_error_frequency  is_error_error_frequency  relative_error_magnitude  tolerance_relative_error_magnitude  is_error_relative_error_magnitude  null_calculated_value_frequency  tolerance_null_calculated_value_frequency is_error_null_calculated_value_frequency  null_reported_value_frequency  tolerance_null_reported_value_frequency  is_error_null_reported_value_frequency
group     table_name group_value
ungrouped ungrouped  ungrouped           0.059571                      0.058                      True                       NaN                               0.045                              False                         0.014635                                        0.7                                    False                       0.539281                                      1.0                                   False
```

What seems a little strange is that the same dang dataframe is getting reported in every failing test - that seems surprising to me. I don't know quite what to make of it but I'll see if I can get this breaking locally for now. @zaneselvans this seems like something you might have some insight into?

I think the reason the same dataframe is coming up repeatedly is that it's not a test failure, it's a defensive check which is erroring out in the ETL phase, and so you're getting identical "errors" raised in every test which depends on the fixture that's generating the errors.

Ahhh that makes sense. OK! I'll go look at where those checks are (presumably) hard coded and see if I can make heads or tails of it.

I'd probably try to materialize the exploded tables locally, which I think would exercise this check, and then you could fiddle with the parameters that set the acceptable error levels.  However, this could reveal real issues. @cmgosnell will be the most familiar with what the error levels are supposed to look like and why the new data might be setting off these alarm bells.

OK, so core_ferc1__yearly_depreciation_changes_sched219 is newly failing in the fast ETL.
I compared these calculation reconciliation metrics across this branch & the main branch:

metric
new value (apply-diffs)
old value (main)
threshold

error frequency
0.059571
0.05792
0.058

relative error magnitude
NaN
NaN
0.045

null calc value freq
0.014635
0.015244
0.7

null reported value freq
0.539281
0.560459
1.0

So the error frequency is newly a little higher, and the null calcs/reported values are both less frequent.

If that's the only thing tripping it up, and it's only happening in the fast and not the full ETL, then upping the allowed error frequency to 0.06 seems fine.  Those thresholds are arbitrary, and really intended to detect changes so they can be evaluated for significance / tracked down if they're wacked.

I ran the fast ETL through the Dagster UI locally and thankfully core_ferc1__yearly_depreciation_changes_sched219 was the only asset that failed. I bumped the error frequency to 0.06 and re-ran it, and it passed. So I think we should just make that tiny adjustment to get this merged in.
@cmgosnell do you think we should check the other error metrics for the other FERC 1 assets, and ratchet them down if we have significantly lower error rates with the new diff-based extractions? Do we expect this to have broad impacts across the board? I can't remember if we had an easy way of generating all the error tolerance outputs.
I also noticed 2 failing asset checks in the fast ETL, which was a little surprising:

_core_eia860__fgd_equipment_fgd_equipment_null_check
`_core_eia860__cooling_equipment_cooling_equipment_null_cols'

I'm glad y'all figured out how to tweak the error tolerances! ( i don't super love how it ended up, structurally/ergonomically )
It makes sense to me that more non-null values in data would result in at least slightly worse error metircs. It is very standard for the fast ETL to require higher error tolerances and that is not a big change. Believe it or not the XBRL data is less calculable with its own calculations as the old data. So it makes sense that adding more XBRL values -> more errors.
I was hoping to spend a lil time after this gets merged in to see if null frequency tolerance can be turned down a bit on any of the tables.
We do have a way to check all of the metrics all at once. that's this asset: _core_ferc1__calculation_metric_checks. That runs all of the calc checks on all of the tables that have non-null params in reconcile_table_calculations (i.e. are we checking their calcs). It outputs a table of all of the error metrics with standard tolerances in the table (it was a lil too complicated/deemed not worth it to use table-specific tolerances in this asset).


EFS enhancements

Codecov Report

Merging #4940 (2bf3496) into master (382ff2b) will increase coverage by 0.02%.
The diff coverage is 98.52%.

@@            Coverage Diff             @@
##           master    #4940      +/-   ##
==========================================
+ Coverage   95.53%   95.56%   +0.02%
==========================================
  Files         619      619
  Lines       65332    65143     -189
==========================================
- Hits        62417    62255     -162
+ Misses       2915     2888      -27

Flag
Coverage Δ

servertests
36.16% <36.02%> (-0.03%)
:arrow_down:

unittests
95.50% <98.52%> (+0.03%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

moto/efs/urls.py
100.00% <ø> (ø)

moto/efs/models.py
91.09% <97.43%> (+2.52%)
:arrow_up:

moto/efs/exceptions.py
100.00% <100.00%> (ø)

moto/efs/responses.py
100.00% <100.00%> (ø)

moto/eks/exceptions.py
95.45% <0.00%> (-4.55%)
:arrow_down:

moto/acm/responses.py
79.20% <0.00%> (-4.03%)
:arrow_down:

moto/route53/responses.py
96.69% <0.00%> (-0.89%)
:arrow_down:

moto/resourcegroupstaggingapi/models.py
83.66% <0.00%> (-0.58%)
:arrow_down:

moto/sns/models.py
96.83% <0.00%> (-0.40%)
:arrow_down:

moto/elbv2/responses.py
93.93% <0.00%> (-0.34%)
:arrow_down:

... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 382ff2b...2bf3496. Read the comment docs.


Unneeded import for ZipWriter docs

In ZipWriter there is an unneeded import
https://github.com/zip-rs/zip/blob/21a20584bc9e05dfa4f3c5b0bc420a1389fae2c3/src/write.rs#L77C9-L77C37

Fixed by https://github.com/zip-rs/zip2/commit/66e0b825290d7da2ea399c15ac2b5bc06339e6ae. If you still see an issue, please reopen it at https://github.com/zip-rs/zip2, since this repo is no longer maintained.


support app dir cloning from github

corrected & rebased with master to resolve conflicts.

Any reason not to merge it?


Poor Terminal Performance

VSCode Version:  1.35.0
OS Version: Win 10 19H1

Tried with --disable-gpu,  didn't make any difference.
Tried with terminal.integrated.rendererType = dom , didn't make any difference.
Steps to Reproduce:

Open powershell in terminal
dir -Recurse

Does this issue occur when all extensions are disabled?: Yes
Benchmark Profile-Fast.zip
Profile-Slow.zip

So the fast one is using the canvas renderer and the slow one is using the DOM renderer. Could you set terminal.integrated.rendererType to canvas and do another profile on the slow machine?

Something else to try is setting terminal.integrated.windowsEnableConpty to false and relaunching the terminal, if that fixes the problem then this is a duplicate of https://github.com/microsoft/vscode/issues/74620

setting terminal.integrated.windowsEnableConpty to false fixed the issue

How long is it supposed to take to print 10,000 lines of ints?
This (F#) code takes upwards of 2 seconds to complete in F# Interactive in VSCode and 0.08 seconds in VS2019:
``for i in 1..10000 do printfn "%A" i`
VSCode:

VS2019:

Tried --disable-gpu, turning off windowsEnableConpty, changing renderType, uninstalling VSCode and removing %appdata% and .vscode, a different computer...

@reinux that looks like https://github.com/microsoft/vscode/issues/74620


"None of the above" answer misplaced

In many questions, the answers "All of the above" and "None of the above" are misplaces. Sometimes above the other questions, sometimes below. They should always be the last ones.

yeah, maybe... I can consider a pull request that change all related questions ;)

https://github.com/certificationy/php-pack/pull/12


save and load history conversation

It is possible to save and load the conversation (with Rasa)?
Let's say I want to save the conversation with user A.
A logs in and chats with the bot.
A close the browser.
Next day A logs in again and see the conversation of the previous day.
So in the front end part there are functions like save/export conversation and load/import conversation?

The rasa webchat saves conversations in browser storage by default, you can choose to save it either through local storage or session storage ( it will be kept longer in localStorage) please look at the readme for more details on how to choose the storage option.


React Komposer 2.x

We've been pretty busy with some internal apps these days. Sorry for not updating.
When we are building it and looking at some of these issues, I think we need to re-think about the API.
Here's what we are looking to have in 2.x (Which will be out pretty soon).
import { makeComposer, merge } from 'react-komposer';
// Additional dataLoader generators are located in a different NPM package.
// These goes into the same repo. But multiple NPM modules. That's to have their own NPM modules for each of these.
import getPromiseLoader from 'react-komposer-promise';
import getRxLoader from 'react-komposer-rx';

// We need to this once in an app.
// With this, we could create our own base compose utility with our own options.
// Additionally, this'll allow us to use react-komposer under React Native since
// we don't have any loading and error components.
const compose = makeComposer({
  loading: () => (
Loading..
),
  error: (error) => alert(error.message),
  pure: true,
  propsToWatch: [], // Now dataLoader is re-run only when one of the props mentioned is changed. In this case, it only runs once.
  context: { // This is how we pass addition stuff to the dataLoader.
    reduxStore: {}
  }
});

// Now there's two callbacks onData and onError.
// That's because, we don't usually use onError.
// We may handle it via our own component rather using something global.
// We use the onError specially in prototyping the app.
const dataLoader = (props, onData, onError) => {
  // Fetch from the data.
  // We can access the context via this.
  //   For an example, we can get the reduxStore with `this.reduxStore`
  return aa.subscribe();
};

const now = Rx.Observable.interval(1000)
  .map(() => new Date().toString());

// Now composeAll is renamed into merge
const DataAwareComponent = merge(
  compose(dataLoader, { propsToWatch: ['appId'] }), // We could alter options like this.
  compose(getPromiseLoader(Promise.resolve(100), 'promiseData')),
  compose(getRxLoader(now, 'date')),
)(<Component/>);

We should also look at documenting how to use recompose if needed.

Expecting

Nice @arunoda! Looking forward to it

I'm HIGHLY opposed to using this to access context. Why not pass it in as an argument?
For that matter, I think the args that are passed in should be an object rather than ordered parameters:
const dataLoader = ({props, onData, onError, context}) => {
  // Fetch from the data.
  return aa.subscribe();
};

This API is much cleaner and you can simply ignore the arguments that you don't use.

@clayne11 yep. I agree and I found it confusing specially with the arrow functions.
So, now I am using it as you are describing.
It's now on v2 branch and we are using it on production.

@arunoda when I install v2 with meteor npm install --save git://github.com/kadirahq/react-komposer#v2 it throws an error
Unable to resolve some modules:

  "./compose" in /C/meteor/project/app/node_modules/react-komposer/dist/index.js (web.browser)

How soon will v2 merge to master?

Hm, I think the old pattern/usage was simpler.

by the way, react-komposer share some functionality with https://github.com/acdlite/recompose (like composeAll / merge is mostly the same like recompose' compose). Did you think about using it under the hood?

Guys, v2 is almost ready and I've added docs as well.

@macrozone we only use a single feature of recompose which is merge. It's a few lines of code and I hope the API is bit different too.
We use react-komposer for data loading, but after that it's possible to use recompose or use recomposed UI components with react-komposer.

Closing since I've released it. But feel free to open new issues for questions.


chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
5fa5912 -> dd53916

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
minor
v1.36.0 -> v1.37.0

go (source)
toolchain
minor
1.22.6 -> 1.23.0

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.37.0
Compare Source
Features

allow deep folders in stack config path (#851) (fd252ef)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

/run pipeline

/run pipeline

/run pipeline

/run pipeline

The failure should be addressed in #213 and the renovate automation should clear this request up within a week.

ℹ Artifact update notice
File name: tests/go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

9 additional dependencies were updated

Details:

Package
Change

github.com/IBM-Cloud/bluemix-go
v0.0.0-20240423071914-9e96525baef4 -> v0.0.0-20240719075425-078fcb3a55be

github.com/IBM-Cloud/power-go-client
v1.6.0 -> v1.7.1

github.com/IBM/go-sdk-core/v5
v5.17.3 -> v5.17.4

github.com/IBM/platform-services-go-sdk
v0.63.1 -> v0.66.0

github.com/hashicorp/go-retryablehttp
v0.7.5 -> v0.7.7

golang.org/x/crypto
v0.24.0 -> v0.26.0

golang.org/x/sync
v0.7.0 -> v0.8.0

golang.org/x/sys
v0.21.0 -> v0.23.0

golang.org/x/text
v0.16.0 -> v0.17.0

/run pipeline

ℹ Artifact update notice
File name: tests/go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

9 additional dependencies were updated

Details:

Package
Change

github.com/IBM-Cloud/bluemix-go
v0.0.0-20240423071914-9e96525baef4 -> v0.0.0-20240719075425-078fcb3a55be

github.com/IBM-Cloud/power-go-client
v1.6.0 -> v1.7.1

github.com/IBM/go-sdk-core/v5
v5.17.3 -> v5.17.4

github.com/IBM/platform-services-go-sdk
v0.63.1 -> v0.67.0

github.com/hashicorp/go-retryablehttp
v0.7.5 -> v0.7.7

golang.org/x/crypto
v0.24.0 -> v0.26.0

golang.org/x/sync
v0.7.0 -> v0.8.0

golang.org/x/sys
v0.21.0 -> v0.23.0

golang.org/x/text
v0.16.0 -> v0.17.0

/run pipeline

/run pipeline

:tada: This PR is included in version 1.10.8 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Adds more benchmarks, bugfixes

Adds autoscaling, prediction serving, and matmul benchmarks.
Among other things, this PR:

Enables close functionality that GCs user library threads.
Fixes Python's random library not being sufficiently random by using SystemRandom.
Implements backoff and load management for schedulers.
Implements cache eviction.

LGTM for the changes related to my code.


Zerotier Plugin

Hi,
I've created a new plugin for opnsense that enables Zerotier. Please can it be reviewed and considered for inclusion.
Thank you.
-=david=-

@dharrigan yay, thanks, this is exciting!

so far so, good, thanks! I'm applying two commits to make the service stop / start according to its configuration: if no networks are enabled, the service is stopped / parked.


Issue with RDLC with apps compiled in pilelines

I am not sure if this is an issue with Al-GO directly, but I am a bit desperate with this issue already.
I have one customer with custom document report layouts. One update introduced an error saying that one field is not present in the dataset. But I do only get the error on the customers sandbox, not locally in my docker container.
After further investigation I can now say that this definitely related to the compiling in AL-Go.
If I deploy the app generated from AL-GO manually, as PTE I still get the error. So its not related to the deployment.
When I compile the exact same app locally with cs code and upload the app as PTE, I do not get the error.
Any idea what could be causing this or how to track down this issue?
It starts to become quite annoying.
This issue persists since over one month now trough several versions.

If you compile the app in both the pipeline and with VS Code and then use BcContainerHelper Extract-AppFileToFolder with both apps - what differences do you see?

Thank you very much!
The issue was a dependency. I did not create a new release for the dependency, therefore AL-Go pulled on older version of the dependency.


The difference between latest and previous version's result.

Hello,
I ran epubchecks of un epub file in different versions and older version seems to show a right result.
At older version's result is following.
MAC-HOSOYA-M:epubcheck-3.0.1 hosoya$ java -jar epubcheck-3.0.1.jar /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub
Epubcheck Version 3.0.1
`Validating against EPUB version 3.0` `WARNING: /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/toc.ncx: meta@dtb:uid content '{%%uid%%}' should conform to unique-identifier in content.opf: 'urn:8443976911000000000Q'` `ERROR: /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/Text/p00132.xhtml(11,61): value of attribute "id" is invalid; must be an XML name without colons`
Check finished with warnings or errors
The latest version's (4.0.2) one is,
MAC-HOSOYA-M:epubcheck-4.0.2 hosoya$ java -jar epubcheck.jar /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub
Validating using EPUB version 3.0.1 rules.
INFO(RSC-022): /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/Images/cover.jpg(-1,-1): Cannot check image details (requires Java version 7 or higher).
INFO(RSC-022): /Users/hosoya/Desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/Images/p00001.jpg(-1,-1): Cannot check image details (requires Java version 7 or higher).
...
No errors or warnings detected.
epubcheck completed
The parts in question of opf file, ncx, xhtml files are,

opf
<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://www.idpf.org/2007/opf" prefix="rendition: http://www.idpf.org/vocab/rendition/# ebpaj: http://www.ebpaj.jp/" unique-identifier="books" version="3.0" xml:lang="ja">
...
    <dc:identifier id="books">urn:8443976911000000000Q</dc:identifier>
...

ncx
<ncx xmlns="http://www.daisy.org/z3986/2005/ncx/" version="2005-1">
  <head>
    <meta name="dtb:uid" content="{%%uid%%}" />
...

p00132.xhtml
...
  <body>
    <div class="main" id="0ead6ac30fff432bf463a908065267f8">
...

I'm wondering the latest version has a problem in checking the error or warning above.
I would appreciate your reply in advance.

The id error in the 3.0.1 checker was a mistake. HTML5 does not require that IDs be valid XML IDs.
The dtb:uid mismatch is currently only reported if you use the --usage flag. See https://github.com/IDPF/epubcheck/issues/669

Mr. Garrish,
Thank you for your reply.
I tried the --usage flag, but it wasn't be reported, though I don't know the reason.
The following are all that's connected with opf or ncx.

USAGE(OPF-058): /Users/hosoya/desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/toc.xhtml(-1,-1): Spine item has no TOC entry reference.
USAGE(HTM-010): /Users/hosoya/desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/toc.ncx(2,68): Namespace uri 'http://www.daisy.org/z3986/2005/ncx/' was found.
USAGE(OPF-059): /Users/hosoya/desktop/fixed/20170206_forautomation/20170509/8443976911000000000Q.epub/OEBPS/toc.ncx(-1,-1): Spine item has no NCX entry reference.

But now I understand the previous issue, so I would wait for either of the two suggestions there (https://github.com/IDPF/epubcheck/issues/669#issuecomment-270646511) would be taken in the next version.
I appreciate your comment.

Closing this issue as author @hosoya-m is waiting for fix in #669

This has been fixed with PullRequest #798 and will be included in the upcoming release v4.1. Please see #669 and #798 for discussion on the topic.


fix: styles on .user-prompt-buttons

As in this snapshot, if you create empty prompts without editing the conent, nothing is wrong see A, but as long as you have long contents, well, the button groups starts displaying incorrectly, see B.
The reason is,
.user-prompt-header {
        max-width: calc(100% - 100px);
}

it only leaves 100px for the 3 buttons with full prompt contents.  I don't think to reduce the max width of the content is a good idea, because we also need to display the content in mobile devices.
so the only way i can do is to slim the buttons,  originally, each button has a padding of 10px, which takes up 20 px space, so i reduced it to padding: 7px;, not only that, i see the gap between two buttons from 5px to 2px.
by the way, I recommend using  css properties of column-gap or row-gap in both flex and grid layouts , which is sightly better than
&:not(:last-child) {
            margin-right: 5px;
          }

also I commented out height: 100%;, if you need, you may change it back.

LGTM, this style bug has been existing for a long time, thx.


[UX] Better dedup packages we nix build, and improve UX printouts

Summary
We have two separate steps for "installing packages".

We use nix build to add to the nix store.
We later ensure the nix profile is updated to have items pointing to the nix store packages.

For the first step, we need to check if it is in the nix store. For this, we check nix path ls <path>. We get <path> from nix path-info <installable>. Previously, we were doing this de-duplication by checking for missing package from the nix profile, rather than directly querying the nix store.
For the second step, we can remove the [1/3] <package> being removed print outs. This was already removed for packages being added before. Reason: this step should be quick! We've already added the packages to the nix store. It also confuses users who are likely not aware of a nix profile concept, nor should we make them aware of it.
Alternatives:
Also tried nix build --profile <path> <installable> but this doesn't seem to return the installed package in nix profile list. Not sure why, but it doesmean its not useful for de-duplicating packages that remain to be installed.
Upside:
Ths store-path deduping is great. Works much faster. Re-doing devbox add <package> for a package already in /nix/store is much faster since we skip the nix build step entirely. As it should be.
Downside:
A user adding or removing packages directly from editing the devbox.json will not see any nice per-package-steps installation output. (Note, this was changed when adding nix-proflile-from-flake, but mentioning for completeness of analyzing the UX)
How was it tested?
devbox add a package already in store no longer shows the per-package installation steps
devbox shell subsequently does not again show we are adding the package.
devbox add a package not in store does show the per-package-steps installation output.
Need to do more rigorous testing:

[ ] add a local flake
[ ] add a remote flake

@gcurtis to clarify, prior to this PR the de-duplication was happening based on the nix profile even for the nix build. Hence this PR.
I added this to the Summary to clarify:

Previously, we were doing this de-duplication by checking for missing package from the nix profile, rather than directly querying the nix store.

Regarding:

You might want to try out nix build --log-format internal-json to get a structured log of which packages Nix is building (or downloading). There's also a --dry-run flag if it helps to know what needs to be build ahead of time.

IIUC I think this would be useful to show the incremental progress being made while installing. @mikeland73 yesterday made a change to enable this output to be shown. See #1826

For completeness, I'll share my findings.
nix build --log-format internal-json doesn't seem to have useful output:
❯ nix build --log-format internal-json  github:nixos/nixpkgs/5233fd2ba76a3accb5aaa999c00509a11fd0793c#cowsay --extra-experimental-features 'nix-command flakes' --
json
@nix {"action":"start","id":406196532019200,"level":4,"parent":0,"text":"evaluating derivation 'github:nixos/nixpkgs/5233fd2ba76a3accb5aaa999c00509a11fd0793c#cowsay'","type":0}
@nix {"action":"stop","id":406196532019200}
@nix {"action":"start","id":406213711888384,"level":6,"parent":0,"text":"querying info about missing paths","type":0}
@nix {"action":"stop","id":406213711888384}
@nix {"action":"start","id":406213711888385,"level":0,"parent":0,"text":"","type":102}
@nix {"action":"start","id":406213711888386,"level":0,"parent":0,"text":"","type":104}
@nix {"action":"start","id":406213711888387,"level":0,"parent":0,"text":"","type":103}
@nix {"action":"result","fields":[0,1,0,0],"id":406213711888386,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":406213711888387,"type":105}
@nix {"action":"result","fields":[101,0],"id":406213711888385,"type":106}
@nix {"action":"result","fields":[100,0],"id":406213711888385,"type":106}
@nix {"action":"start","id":406213711888388,"level":6,"parent":0,"text":"querying info about missing paths","type":0}
@nix {"action":"stop","id":406213711888388}
@nix {"action":"result","fields":[0,0,0,0],"id":406213711888386,"type":105}
@nix {"action":"result","fields":[0,0,0,0],"id":406213711888387,"type":105}
@nix {"action":"result","fields":[101,0],"id":406213711888385,"type":106}
@nix {"action":"result","fields":[100,0],"id":406213711888385,"type":106}
@nix {"action":"stop","id":406213711888387}
@nix {"action":"stop","id":406213711888386}
@nix {"action":"stop","id":406213711888385}

nix build --json --dry-run is more promising. We could pick out the store-paths from the outputs below:
❯ nix build github:nixos/nixpkgs/5233fd2ba76a3accb5aaa999c00509a11fd0793c#cowsay --extra-experimental-features 'nix-command flakes' --json --dry-run
[{"drvPath":"/nix/store/wp73jzcp45x22a54jmb3x5hz9fhaf6y4-cowsay-3.04.drv","outputs":{"man":"/nix/store/njfycpgk5ky9q53h6wmh2z131gh64wfn-cowsay-3.04-man","out":"/nix/store/awaw8fix4bcjxsyg86srbdr3pxkw8wqg-cowsay-3.04"}}]

but the format can change based on the <installable> kind:
❯ nix build --dry-run --json /nix/store/awaw8fix4bcjxsyg86srbdr3pxkw8wqg-cowsay-3.04 --extra-experimental-features nix-command
["/nix/store/awaw8fix4bcjxsyg86srbdr3pxkw8wqg-cowsay-3.04"]

In contrast, nix path-info output is more predictable for the <installable> variants I've seen, so feels better to continue with that.

LGTM!
Can you confirm that doing devbox install on a fresh project with a few packages shows output.

confirmed!

Should we rename packagesToInstallInProfile to packagesToInstallInStore ?

yes, done!


TreeView.SelectedItem is set to null before being set to the newly selected item.

When selecting different items in a TreeView the value for the SelectedItem property
is briefly set to null before being set to the newly selected item.
Consider the following ViewModel:
public class MainWindowViewModel : ViewModelBase
{
    public ObservableCollection<string> Items {get; set;} = new ObservableCollection<string>();

    [Reactive]
    public string SelectedItem {get; set;}
    public MainWindowViewModel()
    {
        for (int i = 0; i < 10; i++)
        {
            Items.Add($"Item {i:##}");
        }

        this.WhenAnyValue(x => x.SelectedItem)
            .Subscribe(x => Console.WriteLine($"Changed: {x ?? "null"}"));
    }
}

As you select different items in the TreeView the console will output:
Changed: Item 6
Changed: null
Changed: Item 4
Changed: null
Changed: Item 6
Changed: null
Changed: Item 4
Changed: null
Changed: Item 6
Changed: null
Changed: Item 3
Changed: null
Changed: Item 7

This behavior differs from WPF and UWP.
Sample repository

It is even worst than that, SelectedItem get sets to null and back to the original value when selecting the currently selected item. Breaks all my logic.

Yeah, if you could test/review https://github.com/AvaloniaUI/Avalonia/pull/3583 we can try and get that merged.

@grokys Unfortunately I am unable to run Avalonia from the branch refactor/use-selectionmodel. I keep getting a BadImageException because of HarfBuzzSharp. I did a fresh clone and checked out that branch. Compiled fine but execution fails.
When Avalonia is configured with UseSkia(), the exception occurs in Avalonia.Skia.FontManagerImpl (line 16).
When Avalonia is configured with UseDirect2D1() the exception occurs in Avalonia.Direct2D1.Media.GlyphTypefaceImpl (line 22).

I've just updated that branch to latest master, hopefully this should fix it.

Still not working. Looking more into the output folder, the wrong native libHarfBuzzSharp.dll gets copied, i.e. the x86 version (766 KB) instead of the x64 version (889 KB).

@Gillibald ☝ have you seen this before?

I think that's a bug with the tooling. With AnyCPU it somehow fails to use the right native assets. Will try to figure out what is happening.

But I guess this is expected. To be safe just build against a specific platform. This would only work if you ship both native bins and select the right one at runtime.

@Gillibald that seems to work for now if I specify explicitly the target.
@grokys I played a bit with the TreeView sample and it didn't work as expected, especially when deleting the selection. At that point I don't know if the issue is with the SelectionModel or with the code in TreeViewPage.PageViewModel but basically after multi-selecting all nodes from 1 to 9 and clicking on "Remove" only odd-indexed nodes were properly removed.
(sorry uploading images seems to be blocked at my work office)

Bug described above is still there. And If I select from 1 to 9 while pressing SHIFT the catalog sample crashes (NullReferenceException).

iginal value w

version: 0.10.18, This issue doesn't seem to be resolved.
Problem recurrence: https://github.com/chenjing1294/avalonia_test


Create uni-mannheim/students.txt

This change adds the domain<EMAIL_ADDRESS>to the repository.  The University of Mannheim uses this domain for the email addresses of its students.  The official website of the University of Mannheim is: https://www.uni-mannheim.de/.

@fionapa First stage of verification passed, official domain is uni-mannheim.de as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@fionapa Thank you for your patience while waiting for a review and processing your request. Pull request merged.


ERROR is occurding when trying to write something in create new mod "mod name" and "mod ID"

What did you enter? I can't seem to reproduce it.

Oh lmao I forgot that I opened that issue
Its fixed for me a long time ago


flaky test: compliance scanner

User Story

1) Failure:
--
  | 30_jobs_docker_spec.rb#test_0005_works [/var/lib/buildkite-agent/builds/single-use-privileged-i-0634b28ea655af127-1/chef-oss/.golang/chef-automate-master-verify/src/github.com/chef/automate/components/compliance-service/api/tests/30_jobs_docker_spec.rb:669]:
  | expected | 0s
  | actual | 1m 30s
  | @@ -1 +1 @@
  | -1
  | +2

https://buildkite.com/chef-oss/chef-automate-master-verify/builds/1796#db69b150-b637-418c-837e-4e70f13322c5
we've seen this multiple times.
i believe it's related to https://github.com/chef/automate/pull/377/files and https://github.com/chef/automate/pull/399 ....sorry :/

Let me have a look at this bad boy!

Looking into this issue, I noticed that for this nodes list query:
MANAGER_GRPC nodes, :list, Nodes::Query.new(filters: [
      Common::Filter.new(key: "state", values: ["TERMINATED"]),
      Common::Filter.new(key: "environment", values: ["trou"]),
      Common::Filter.new(key: "group", values: ["mak", "do"])
    ])

the node manager is doing the following SQL query to get the list of nodes:
SELECT
  n.connection_error,
  n.id,
  n.last_contact,
  n.last_job,
  n.manager,
  n.name,
  n.platform,
  n.platform_version,
  n.report_id,
  n.source_state,
  n.status,
  n.target_config,
  n.last_scan,
  n.last_run,
  n.projects_data,
  COALESCE(('[' || string_agg('{"key":"' || t.key || '"' || ',"value": "' || t.value || '"}', ',') || ']'), '[]') :: JSON AS tags,
  COALESCE(array_to_json(array_remove(array_agg(DISTINCT m.manager_id), NULL)), '[]') AS manager_ids,
  COALESCE(array_to_json(array_remove(array_agg(p.project_id), NULL)), '[]') AS projects,
  COUNT(*) OVER () AS total_count
FROM nodes n
  LEFT JOIN nodes_tags nt ON n.id = nt.node_id
  LEFT JOIN tags t ON t.id = nt.tag_id
  LEFT JOIN node_managers_nodes m on n.id = m.node_id
  LEFT JOIN nodes_projects np on np.node_id = n.id
  LEFT JOIN projects p on np.project_id = p.id
WHERE (n.source_state IN ('TERMINATED') AND t.key LIKE 'group' AND t.value LIKE 'mak%' OR t.value LIKE 'do%' OR t.key LIKE 'environment' AND t.value LIKE 'trou%')
GROUP BY n.id
ORDER BY LOWER(n.name) asc
LIMIT 100
OFFSET 0;

Because of the lack of parentheses and the ORed t.key LIKE 'environment' in the WHERE clause where tags are matched, the n.source_state IN ('TERMINATED') condition is being ignored.
If the right parentheses are used along with AND for the t.key LIKE 'environment' condition:
WHERE n.source_state IN ('TERMINATED') AND (t.key LIKE 'group' AND (t.value LIKE 'mak%' OR t.value LIKE 'do%') AND t.key LIKE 'environment' AND t.value LIKE 'trou%')

0 nodes are returned. This is because of the multiple tag based conditions that are applied to each row after the LEFT join.

I managed to come up with this query that uses an EXIST + SELECT for each tag used for filtering:
SELECT
  n.connection_error,
  n.id,
  n.last_contact,
  n.last_job,
  n.manager,
  n.name,
  n.platform,
  n.platform_version,
  n.report_id,
  n.source_state,
  n.status,
  n.target_config,
  n.last_scan,
  n.last_run,
  n.projects_data,
  COALESCE(('[' || string_agg('{"key":"' || t.key || '"' || ',"value": "' || t.value || '"}', ',') || ']'), '[]') :: JSON AS tags,
  COALESCE(array_to_json(array_remove(array_agg(DISTINCT m.manager_id), NULL)), '[]') AS manager_ids,
  COALESCE(array_to_json(array_remove(array_agg(p.project_id), NULL)), '[]') AS projects,
  COUNT(*) OVER () AS total_count
FROM nodes n
  LEFT JOIN nodes_tags nt ON n.id = nt.node_id
  LEFT JOIN tags t ON t.id = nt.tag_id
  LEFT JOIN node_managers_nodes m on n.id = m.node_id
  LEFT JOIN nodes_projects np on np.node_id = n.id
  LEFT JOIN projects p on np.project_id = p.id
WHERE
  n.source_state IN ('TERMINATED') AND
  EXISTS (SELECT * FROM nodes_tags nt LEFT JOIN tags t ON nt.tag_id = t.id WHERE nt.node_id = n.id AND t.key LIKE 'environment' AND t.value LIKE 'trou%') AND
  EXISTS (SELECT * FROM nodes_tags nt LEFT JOIN tags t ON nt.tag_id = t.id WHERE nt.node_id = n.id AND t.key LIKE 'group' AND (t.value LIKE 'mak%' OR t.value LIKE 'do%'))
GROUP BY n.id
ORDER BY LOWER(n.name) asc
LIMIT 100
OFFSET 0;

 connection_error |                  id                  |    last_contact     |               last_job               | manager  |          name           | platform | platform_version | report_id | source_state |   status    |                                     target_config                                      |
               last_scan                                                            | last_run | projects_data |                                                     tags                                                      |               manager_ids                | projects | total_count
------------------+--------------------------------------+---------------------+--------------------------------------+----------+-------------------------+----------+------------------+-----------+--------------+-------------+----------------------------------------------------------------------------------------+---------------------------------------------
------------------------------------------------------------------------------------+----------+---------------+---------------------------------------------------------------------------------------------------------------+------------------------------------------+----------+-------------
                  | 509f845c-f833-4635-9e21-5dd30e1141c4 | 2019-06-17 10:31:07 | 48cdefa5-08ce-4d27-943d-819e38a074e1 | automate | My Existing Docker Node | debian   | 9.7              |           | TERMINATED   | unreachable | {"secrets":["098ecb0d-f440-4f27-acd2-386269c863e1"],"backend":"docker","host":"cc_pg"} | {"ID":"f6046a69-b7a0-4b88-9b89-8e0bc9f7cbc6"
,"Status":"SKIPPED","PenultimateStatus":"SKIPPED","EndTime":"2019-06-17T10:31:06Z"} | {}       | null          | [{"key":"group","value": "doers"},{"key":"group","value": "makers"},{"key":"environment","value": "trouble"}] | ["e69dc612-7e67-43f2-9b19-256afd385820"] | []       |           1
(1 row)

which correctly matches only the TERMINATED node that has tags environment:trouble, group:doers and group:makers

ooooh nice find @alexpop !!

I paired with Rick today and looked for an alternative solution to the EXISTS with inner SELECTS, which was returning in 3 minutes for tag only filtering for 30k nodes.
We came up with the following SELECT with HAVING which we saw up to 100 times faster than the inner SELECTS version:
SELECT
  n.connection_error,
  n.id,
  n.last_contact,
  n.last_job,
  n.manager,
  n.name,
  n.platform,
  n.platform_version,
  n.report_id,
  n.source_state,
  n.status,
  n.target_config,
  n.last_scan,
  n.last_run,
  n.projects_data,
  COALESCE(('[' || string_agg('{"key":"' || t.key || '"' || ',"value": "' || t.value || '"}', ',') || ']'), '[]') :: JSON AS tags,
  COALESCE(array_to_json(array_remove(array_agg(DISTINCT m.manager_id), NULL)), '[]') AS manager_ids,
  COALESCE(array_to_json(array_remove(array_agg(p.project_id), NULL)), '[]') AS projects,
  COUNT(*) OVER () AS total_count
FROM nodes n
  LEFT JOIN nodes_tags nt ON n.id = nt.node_id
  LEFT JOIN tags t ON t.id = nt.tag_id
  LEFT JOIN node_managers_nodes m on n.id = m.node_id
  LEFT JOIN nodes_projects np on np.node_id = n.id
  LEFT JOIN projects p on np.project_id = p.id
WHERE (n.source_state IN ('TERMINATED'))
GROUP BY n.id
HAVING (array_agg(t.key || ' ' || t.value) @> ARRAY ['group doers'] OR array_agg(t.key || ' ' || t.value) @> ARRAY ['group makers']) AND  array_agg(t.key || ' ' || t.value) @> ARRAY ['environment trouble']
ORDER BY LOWER(n.name) asc
LIMIT 100
OFFSET 0;

Nice! We could also be missing some indexes for all these different types of queries.

The array_agg solution works but can't do wildcard tags filtering.
I switched to string_agg instead and wildcard(%) filtering is possible with LIKE. Check it out:
SELECT
  n.connection_error,
  n.id,
  n.last_contact,
  n.last_job,
  n.manager,
  n.name,
  n.platform,
  n.platform_version,
  n.report_id,
  n.source_state,
  n.status,
  n.target_config,
  n.last_scan,
  n.last_run,
  n.projects_data,
  COALESCE(('[' || string_agg('{"key":"' || t.key || '"' || ',"value": "' || t.value || '"}', ',') || ']'), '[]') :: JSON AS tags,
  COALESCE(array_to_json(array_remove(array_agg(DISTINCT m.manager_id), NULL)), '[]') AS manager_ids,
  COALESCE(array_to_json(array_remove(array_agg(p.project_id), NULL)), '[]') AS projects,
  COUNT(*) OVER () AS total_count
FROM nodes n
  LEFT JOIN nodes_tags nt ON n.id = nt.node_id
  LEFT JOIN tags t ON t.id = nt.tag_id
  LEFT JOIN node_managers_nodes m on n.id = m.node_id
  LEFT JOIN nodes_projects np on np.node_id = n.id
  LEFT JOIN projects p on np.project_id = p.id
WHERE (n.source_state IN ('TERMINATED'))
GROUP BY n.id
HAVING (string_agg(',' || t.key || ':' || t.value, '') LIKE '%,environment:trou%') AND (string_agg(',' || t.key || ':' || t.value, '') LIKE '%,group:mak%' OR string_agg(',' || t.key || ':' || t.value, '') LIKE '%,group:do%')
ORDER BY LOWER(n.name) asc
LIMIT 100
OFFSET 0;

For NOT to work when nodes have no tags, I had to concat the value of string_agg before LIKING it:
For example:
HAVING NOT concat('',string_agg(',' || t.key || ':' || t.value, '')) LIKE '%,department:%'

Looks like this was passing through and generating an invalid postgres query with my refactor:
    # Get the nodes with a department tag without passing the values array:
    nodes_list = MANAGER_GRPC manager, :search_nodes, Manager::NodeQuery.new(
      node_manager_id: "e69dc612-7e67-43f2-9b19-256afd385820",
      query: Manager::Query.new(
        filter_map: [
          Common::Filter.new(key: "department"),
        ]
      )
    )

I updated the code to search for any tag value in this case.

I added a number of extra tests for tags filtering.
Did another performance test with 10k nodes with 11k tags between them.
The SELECT with WHERE filtering using EXISTS + SELECT averaged at 5 seconds response time without the n.source_state IN ('TERMINATED') condition that filters at node level.
Same filter condition but with HAVING + string_agg as the PR is using comes back in around 0.2 seconds on Postgres in Docker for Mac.

Node manager integration tests are failing on https://github.com/chef/automate/pull/662 because we issue a GetNodes here:
https://github.com/chef/automate/blob/1e73466860fa95a087d060aefd6e0c0d2bd535d2/components/nodemanager-service/pgdb/nodes_integration_test.go#L220
with filters:
Key:    "tacos",    Values: []string{"yes"}
Key:    "nachos",   Values: []string{"yes"}

These used to be OR'ed before and returned two nodes. They are now AND'ed and we don't have nodes that share those two tags.
Before I update the test I want to clarify that AND is what we want here.

thinking through this..
the tag keys are different, which means this is the same thing as doing something like an environment filter and a platform filter - which we AND together.
So, for consistency, AND for tags of different keys, OR for tags of same keys - so ya, AND is what we want here.   might be worth noting that logic in the docs: https://github.com/chef/automate/blob/master/components/automate-chef-io/content/docs/nodes.md

Thank you for confirming Victoria.
I updated the tests added two tags filtering examples to nodes.md for the OR and AND cases.
PR is green now.


Adding MixMaterial to MaterialsLibrary

TODO: finish getAnimatables and Clone

Why the PR, that is approved, has not been validated ? I can not test it today to send comments to Julien.
What is in TODO does not prevent the operation of the material and was planned I guess with another PR.

Just the code has been approved, I finalize this evening then it should be merged if

you confirm that it works as you want
or it’ll be merged and I’ll pull request next changes

Ok, but I can not confirm that it works before it is merged.

Oh ok, this because the 'WhatsNew' was not edited I guess for this new material.

I just noticed that getAnimatables () and clone () are already present. Is this imcomplete? because when I compare with the fieldMateriel, everything seems finished.
By cons I'm not sure, but in function hasTexture(), there is only the _mixTexture1 and not the _mixTexture2. Same thing in the function dispose(), it only has the _mixTexture1. Is this normal?
I hope this PR will soon be merged.

I tried to convert the ts file to js for testing. it works, but I have two problem:

MixMap1 and MixMap 2 mix together creating a bad paint effect.
And texture 4 and 8 does not give textures, but gives transparency between different layers.

I had idea to use 8 different textures. For example if on mixMap 1 I make earth, sand and rocks, then with mixMap2 I repaint snow on the peaks of mountains, I do not find myself with snow, but with a mix between the snow and rock which gives something ugly.
Basically, the textures from 1 to 8 can this mix, but not the two mixTextures. I do not know if I explain well. I hope.
Is there the possibility that mixMap1 and mixMap2 do not mix and that texture 4 and 8 create texture and not transparency.
I wonder if this should not be 6 textures that should instead of 8, because the alpha channel does not give a fourth texture, but an erase of RGB.
Otherwise I have no error, it works, apart from the mixture of 2 mixMap and alpha channel
Maybe for alpha I have to use a different color than black which corresponds to an alpha of 1? that's what white would make an alpha of 0 ? I saw in the shader that the apha is 1.0 - mixColor.a.
Thank you Julien

That's right ! Then I should add an option for that (mixMap 2 overrides mixmap 1)

No, I did not test like that, I would do it by testing with your patches.

@deltakosh do you want me to push the dist files as well ?

No I will do it:)


update to python 3.12

based off of https://github.com/SwissDataScienceCenter/renku-data-services/pull/158 review that one first
EDIT (by @olevski): I added the deploy comment to make sure everything is ok. Just to be safe and make sure everything deploys.
/deploy extra-values=global.gitlab.url=https://gitlab.dev.renku.ch,global.gitlab.registry.host=registry.dev.renku.ch/

You can access the deployment of this PR at https://renku-ci-ds-167.dev.renku.ch

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

168 of 174 (96.55%)  changed or added relevant lines in 50 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.09%) to 89.305%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

components/renku_data_services/authn/keycloak.py
3
4
75.0%

components/renku_data_services/project/models.py
5
6
83.33%

components/renku_data_services/users/kc_api.py
5
6
83.33%

components/renku_data_services/app_config/config.py
10
13
76.92%

Totals

Change from base Build<PHONE_NUMBER>:
0.09%

Covered Lines:
5720

Relevant Lines:
6405

💛  - Coveralls

Tearing down the temporary RenkuLab deplyoment for this PR.


MSVS vs MSVC vs MSC

#71 introduces MSVS keyword for ide, MSVC is used internally to mark Microsoft C/C++ compiler. So what keywords should we use ?

MSC is used by the compiler itself - e.g. MSC_VER, etc. We can nust stick to it.

Also, MSVS is not common at all, maybe we should change it, too.

How do you name the argument then to generate ide project ? --msc ?

Can we go with -ide=vs2013 ?

Yep, this can work, though it's a bit long. Maybe we also should add some aliases like bf vs2013 and\or bf vs.

Indeed, I'll work on it in a branch and merge it afterwards.


service : dev : Enhance commit message linting

See https://github.com/intel/dffml/pull/1076#issuecomment-864556863 for full details
Related Issue #1040

[x] 1.  All commits starting with test shouldn't contain any test_  in the rest of the parts

Implement a new mutation which is only triggered when the commit starts with test

[x] 2. Substitutions are required for certain cases , where a word should translate to a path

Add a substitution step which would replace such cases with their respective actual paths.

[ ] 3.  Checking for function/class names in files/folders

Add a mutation which checks for a valid , file/directory path by recursively removing segments from the end of it until it finds a valid file/directory path , and then it searches for the removed segments in those files.
for ex: In util: net: cached_download(): Changed logging , the mutation should remove the cached_download() part and check for util/net as a path with extensions mutation, and by this it will resolve to dffml/util/net.py, and then we've to search in that file for the removed part , i.e. cached_download() in this file, similarly if it resolved to a directory then we'd need to search all the files in that directory.
This might sound complicated but it can be implemented easily as a body_mutation , we'd need this category for other things as well curretly we have only prefix_mutations and suffix_mutations only , this body mutation should only be applied if all the removed parts are found in their respective file/directory paths.

[ ] 4. Capitalization should be done for certain parts of the path

Implement a body mutation to handle capitalisation of last and second last parts of the path.

[ ] 5. Special keywords should be ignored

A simple body mutation to filter these words can be implemented.

[ ] 6. Investigate why this is failing and fix it, related to 1. above
[ ] 7. Add support for "Merge branch x into y" type of commits.
[ ] 8. Add support for Reverted " < some valid commit > " type of commits.
[ ] 9. Add rule that all that checks whether the first letter of every section of the commit is capitalised
[ ] 10. Fix the bug introduced while implementing 1. above

[ ] Investigate why this is failing and fix it, related to 1. above

There is some problem with commits related to tests, most probably related to point 1 above:
Some failing examples:
tests: Renamed `accuracy` to `score`

model: vowpalWabbit: tests: Renamed `accuracy` to `score`

model: scikit: tests: Renamed `accuracy` to `score`

model: pytorch: tests: Renamed `accuracy` to `score`

We might want to go with https://commitizen-tools.github.io/commitizen/

Looks great !! I guess it might be the best way to do it as we're sort of re-inventing the wheel here, I will look into it and come up with a plan to port our lint commit CMD to use this. Thanks for sharing 🙂 .

https://github.com/wagoid/commitlint-github-action

We might want to go with https://commitizen-tools.github.io/commitizen/, let's discuss this today

We can also look into this one: https://github.com/wagoid/commitlint-github-action


Direct: Adds version 3.29.0

Pull Request Template
Description

Adds Direct Version 3.29.0, which upgrades System.Configuration.ConfigurationManager dependency from 4.7.0 to 6.0.0

This dependency version is tied to the .NET 6.0 release, which still supports .NET Standard 2.0, so it is not a breaking change.
This dependency was upgraded due to a security/compliance alert.

Performance: Replaces DateTime.UtcNow.ToString("r") with Rfc1123DateTimeCache.UtcNow()
Fixes ObjectDisposedException during Bounded Staleness/Strong barrier requests

Micro Benchmark Test Results:

Method
Mean
Error
StdDev
Median
Gen 0
Allocated

DateTimeToString
58.878 ns
1.9800 ns
5.8070 ns
57.634 ns
0.0185
80 B

Rfc1123DateTimeCacheUtcNow
2.298 ns
0.0778 ns
0.2280 ns
2.228 ns
-
-

Type of change

[x] New feature (non-breaking change which adds functionality)

Closing issues
Part 1 of 2 for Issue #3274

Adds Direct Version 3.29.0, which upgrades System.Configuration.ConfigurationManager dependency from 4.7.0 to 6.0.0
This is a breaking change when the CosmosDB SDK is used in Azure Function .Net Core 3.1.


More natural language instead of a slash

Just a simple message edit.

Codecov Report

Merging #3318 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #3318   +/-   ##
======================================
  Coverage     7.5%    7.5%
======================================
  Files         255     255
  Lines       11206   11206
  Branches      895     895
======================================
  Hits          841     841
  Misses      10303   10303
  Partials       62      62

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 56c4018...9b5ce79. Read the comment docs.


unable connect to my azure account

I got this error message when i try to connect to my default directory
{
"name": null,
"message": "{"source":{"accountId":"00000000-0000-0000-6120-b6bc3cb8a860.9188040d-6c67-4c5b-b112-36a304b66dad:666446FFA39D8AF9E3F8ADB1ACD75DE573935F2F62E51D8AFD6362BDDCA142C3","tenantId":"c4c26657-ca32-43ab-ba31-b12786456227"},"resourceId":"https://management.core.windows.net/\",\"code\":\"AADSTS50020\",\"message\":\"User account '{EUII Hidden}' from identity provider 'live.com' does not exist in tenant 'Default Directory' and cannot access the application 'e0001ebd-e920-49b8-a27e-92e573ec0af6'(Storage Explorer) in that tenant. The account needs to be added as an external user in the tenant first. Sign out and sign in again with a different Azure Active Directory user account. Trace ID: 8489cc2d-acd5-44cd-b06a-21819fa60a00 Correlation ID: 3dff0360-19ee-490c-ae4e-89a4fadf1bea Timestamp: 2024-08-23 12:34:10Z"}"
}

@khould7207

Please provide the information requested in our bug template, including version, platform, and step you followed that resulted in your error.
Is Storage Explorer and all required components up-to-date?
Can you follow the error's suggestion of signing out and signing back in?

Closing due to inactivity. If you are still experiencing this issue, please respond with additional information, and we can revisit.


Use short identifier from SPDX License List

https://spdx.org/licenses/Apache-2.0.html
From https://hex.pm/docs/publish

It is recommended to use SPDX License identifier.

Thanks! Much appreciated!
Sorry for the long wait...


Add model class creation

Bug Report or Feature Request (mark with an x)
- [ ] bug report -> please search issues before submitting
- [x] feature request

Versions.
@angular/cli: 1.2.0
node: 6.6.0
os: win32 x64
@angular/animations: 4.2.5
@angular/common: 4.2.5
@angular/compiler: 4.2.5
@angular/core: 4.2.5
@angular/forms: 4.2.5
@angular/http: 4.2.5
@angular/material: 2.0.0-beta.7
@angular/platform-browser: 4.2.5
@angular/platform-browser-dynamic: 4.2.5
@angular/router: 4.2.5
@angular/cli: 1.2.0
@angular/compiler-cli: 4.2.5
@angular/language-service: 4.2.5

Repro steps.

The log given by the failure.

Desired functionality.
Posibility to create a class model look like xxx.model.ts and xxxx.model.spec.ts via ng generate command

Mention any other details that might be useful.
May be we could reused ng generate class with a new flag -model or create a new generate method look like ng generate model

ng g class blah.model

Thanks, i have discovered this way  few second after issue creation


Passthrough TagCollection for things like barcode assignments

yes

Just talked about this with Brett.  So, the challenge is what to do when the barcodes don't match within a zmw.  We decided that we will emit a UINT.MAXVALUE for this case, let me know if there are any complaints.

Just committed f0b58ad with ability to pass through barcodes and tests to verify, closing isssue.

I don't know if I'm on board with this ... I'd much rather have it go through bam2bam for rebarcoding ... the case you describe above requiring UINT.MAXVALUE is a telltale sign that this passthrough is not necessarily well-formed, there could be pathological cases where a huge number of ZMWs are labeled thusly.

@bnbowman can comment on the feasibility, I was not aware that bam2bam could add barcodes to a ccs.bam file.  How does it adjudicate when the same ZMW has different barcodes called by the adapters?

There's also the problem of the LocalContextFlags, which are not necessarily passed through here, in particular BARCODE_BEFORE and BARCODE_AFTER.

We would just use the CCS to recall the barcodes, not the constituent subreads. The CCS is an entirely different beast, a completely novel sequence generated from the subreads. While it might be true sometimes, or even often, the properties of the subreads do not necessarily perpetuate into the CCS. Some concepts, like queryStart and queryEnd, for example, are not provided. I would advocate that most (if not any) kind of "downstream" sequencing analysis done on the subreads (such as barcode calling) must be redone for the CCS separately.

I would also really, really like to dissuade everyone from on-the-fly specification-making as in the case of the UINT.MAXVALUE -- it leads to the implementation leading ahead of the specification, essentially creating undocumented behavior. If this is really deemed the best way to address this problem, please update https://github.com/PacificBiosciences/PacBioFileFormats/blob/3.0/BAM.rst as well

The barcode sequences are not in the input.bam that CCS takes, but in a separate index file based on my conversation with Brett.  I am strongly against having CCS take a dependency on another file which is not guaranteed to be present or in sync with the input file.  People always have the option to ignore the bc tags, the output is identical except for the addition of these fields.
But we should probably define the problem better before discussing more.  Right now, there are no "downstream" sequencing analysis tools that use CCS barcodes or any consumer save Brett, so any discussion is ungrounded and academic.   Brett asked for the feature so I dropped it in, but I know you created the original issue.  What were you thinking was the point of these tags and who would use them?

The BAM fileformats have only a passing mention of ccs.bam files, the whole thing is undocumented (e.g. look at the QUAL field description here: https://github.com/PacificBiosciences/PacBioFileFormats/blob/3.0/BAM.rst)... we need to fix that.

A user intending on using barcodes with CCS should probably instead provide the barcode sequences as a separate, downstream analysis step from CCS generation (and bam2bam should do the trick, @bnbowman can speak to whether this makes sense). When I opened the issue originally, it didn't immediately occur to me what to do about the mixed barcode case, or that this even made sense. Now that you've gone ahead and implemented it, it's clear there are some issues wrt to providing a first-class barcoding experience with CCS--yanking the data from the subreads is not necessarily a well-formed operation, and things like BARCODE_BEFORE and _AFTER do not translate. I'm not advocating that we automatically rebarcode for the user (requiring the gross dependency, as you say, but barcoding in general has this problem and it is not unique to CCS). It might perhaps be sufficient to just leave the user a message that we detected and are discarding barcoding data, and that the CCS sequences will need to be reanalyzed using bam2bam.

@evolvedmicrobe, wrt to QUAL values, we absolutely, most definitely should.

I have multiple reasons for wanting to pass-through barcode information:
1). Simplicity - this way I only need one model for calling / scoring
barcodes
2) Chimera filtering - planned feature comparing barcodes across a ZMW to
catch 2P sequences, which I can't do on CCS
3). Clashing namespace - BC and BQ tags would then have two meanings,
discordant with other barcoding results / data
4).  Breaks the PPA model - barcoding is now supposed to be an upstream
prepossessing step a la adapter finding
On Tue, Nov 24, 2015, 12:57 PM Nigel Delaney<EMAIL_ADDRESS>wrote:

The barcode sequences are not in the input.bam that CCS takes, but in a
separate index file based on my conversation with Brett. I am strongly
against having CCS take a dependency on another file which is not
guaranteed to be present or in sync with the input file. People always have
the option to ignore the bc tags, the output is identical except for the
addition of these fields.
But we should probably define the problem better before discussing more.
Right now, there are no "downstream" sequencing analysis tools that use CCS
barcodes or any consumer save Brett, so any discussion is ungrounded and
academic. Brett asked for the feature so I dropped it in, but I know you
created the original issue. What were you thinking was the point of these
tags and who would use them?
—
Reply to this email directly or view it on GitHub
https://github.com/PacificBiosciences/pbccs/issues/41#issuecomment-159402847
.

Fair enough, though a test as to whether the preexisting model suffices for CCS could obviate this concern.
I don't think it's fair that the existence of an inapplicable analysis should count for or against.. you have the problem either way..?
I don't understand this point. We would just re-barcode the CCS sequences and populate the correct LocalContextFlags. To what double meaning are you referring?
This is perhaps the best point so far, but bam2bam was already put into place for dealing with scenarios such as this. "Supposed to be" shouldn't interfere with doing things the correct way.

I agree with Brett's concern about the BQ/BC tags, it's a general problem... CCS and raw reads have a lot of similarities in terms of metrics, but they aren't exactly the same so it's confusing to give them the same label at times.  But that's not what's preventing everyone from using our awesome instrument right now.
It sounds like we have no actual external use case for these yet, but they could be valuable for R&D.  There are other BAM tags this applies to, e.g. the "mt", "ma", "ms" tags, and I have been wanting to put in a TAG that gives "RQ" as the error probability, rather than 1 minus that probability to maintain precision.
So how about this, for this bc tag and the others which at present only are consumed by R&D, we only generate them if pbccs is compiled with a special flag, that way we have them for R&D but they don't go into the release build.  We could then revisit that decision whenever we have a defined use case that could actually affect or benefit a customer.  Thoughts?

I can get behind this plan of action 100%

I don't understand this point. We would just re-barcode the CCS
sequences and populate the correct LocalContextFlags.

I don't see the CX tag as relevant to this discussion - it is a
subread-specific tag, not a ZMW-specific tag, so it implies nothing about
BC/BQ other than their existence.  So passing through BC and BQ but not CX
is perfectly reasonable.
Or in a practical example - the CX tag on the first subread can refer to a
condition where it has a single barcode on one side, and that one barcode
may be a reverse-sided barcode.  But the BC tag for that same subread will
have both the Forward and Reverse barcodes, and in the F--R orientation.

To what double meaning are you referring?

BC and BQ tags being calculated by two different algorithms for Raw/CCS
data, and therefore potentially having different problems / ranges /
behaviors.

This is perhaps the best point so far, but bam2bam was already put into
place for dealing with scenarios such as this. "Supposed to be" shouldn't
interfere with doing things the correct way.

Bam2bam does not understand CCS files, or even single BAM files (absent the
LQ/Scraps partner file).  This would require a completely new code-path.
On Tue, Nov 24, 2015 at 1:14 PM, Lance Hepler<EMAIL_ADDRESS>wrote:

Fair enough, though a test as to whether the preexisting model suffices
for CCS could obviate this concern.
I don't think it's fair that the existence of an inapplicable analysis
should count for or against.. you have the problem either way..?
I don't understand this point. We would just re-barcode the CCS
sequences and populate the correct LocalContextFlags. To what double
meaning are you referring?
This is perhaps the best point so far, but bam2bam was already put into
place for dealing with scenarios such as this. "Supposed to be" shouldn't
interfere with doing things the correct way.

—
Reply to this email directly or view it on GitHub
https://github.com/PacificBiosciences/pbccs/issues/41#issuecomment-159407006
.

So how about this, for this bc tag and the others which at present only
are consumed by R&D, we only generate them if pbccs is compiled with a
special flag,

That should suffice for now.
On Tue, Nov 24, 2015 at 1:42 PM, Nigel Delaney<EMAIL_ADDRESS>wrote:

I agree with Brett's concern about the BQ/BC tags, it's a general
problem... CCS and raw reads have a lot of similarities in terms of
metrics, but they aren't exactly the same so it's confusing to give them
the same label at times. But that's not what's preventing everyone from
using our awesome instrument right now.
It sounds like we have no actual external use case for these yet, but they
could be valuable for R&D. There are other BAM tags this applies to, e.g.
the "mt", "ma", "ms" tags, and I have been wanting to put in a TAG that
gives "RQ" as the error probability, rather than 1 minus that probability
to maintain precision.
So how about this, for this bc tag and the others which at present only
are consumed by R&D, we only generate them if pbccs is compiled with a
special flag, that way we have them for R&D but they don't go into the
release build. We could then revisit that decision whenever we have a
defined use case that could actually affect or benefit a customer. Thoughts?
—
Reply to this email directly or view it on GitHub
https://github.com/PacificBiosciences/pbccs/issues/41#issuecomment-159413648
.

https://github.com/PacificBiosciences/pbccs/pull/90/files


MOB-25: Follower & Following Pages

What's up?
We need to follower and following page.
To Do

[x]  Implementing followers & following users page.

Deadline
17.12.2022 @12.00
Additional Information
No response
Reviewers
@MustafaEmreErengul

No action required, closed.


Increase contrast ration of editor

The contrast ration of the editor's background and most code is <4.5, which is too little.

The WebAIM guidelines recommend an AA (minimum) contrast ratio of 4.5:1 for all text.

https://developers.google.com/web/fundamentals/accessibility/accessible-styles#color_and_contrast

closing since the site has been rewritten. please file at https://github.com/sveltejs/svelte.dev/issues/new if the issue persists


fix(ffe-accordion): Background transition for blue accordion

This commit adds a transition on background for blue accordion. This is
done to improve the visual quality of the accordion when transitioning
between hover and normal state.
Fixes #398

Coverage remained the same at 80.521% when pulling 85110f70e85b6010d7bf4a73f0e39e1ae3a5104d on fix-accordion-background-transition into 4e3ee7cf336e08c13e5fe047adfe051e7b3bea2e on develop.


Exit > 0 when tests fail

It does if you run smoke_report at the end? :) https://github.com/asm89/smoke.sh/blob/34ba26aa28279dceaacd16e2b38f51cda6398853/smoke.sh#L50


RxJava 2 support

Any thoughts about how to introduce it in a better manner without dropping RxJava 1 support?
Also object methods have to be reworked since they are emitting null if there is no requested object.

Totally forgot about Maybe - it's perfect for that case.

Yeah, there is some work required to do RxJava v2 support.
TL;TR plan:

Wait for final release.
Add basic support for Flowable because it supports backpressure which is
required for IO operations.
Add support for Single/Completable.
Add/Think about support for Maybe.

RxJava 2 can work with RxJava 1 in same classpath, so  I don't see problems
with that except naming of the methods (I think we'll just add "2" to it).
On 2 Oct 2016 10:30 am, "Ilya Zorin"<EMAIL_ADDRESS>wrote:

Totally forgot about Maybe - it's perfect for that case.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pushtorefresh/storio/issues/685#issuecomment-250974064,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3FCuyaL5AWs-sb4gOL3psnCF8K3Pks5qv8ABgaJpZM4KHoyr
.

Not sure if I understand you correctly, so you want to have both RxJava and RxJava 2 in the same library? It sounds not really good having both of these dependencies when you need only one of them in your project.
Regarding supporting Maybe - it's a perfect fit for object methods since null is prohibited in RxJava 2 and was kinda hacky in a first place.

StorIO has just provided rxjava dependency - you can use rx only when you already have this dependency in your project

Thanks for clarification @nikitin-da!
Anyway, I will be happy to help with this task.

Great, let's wait for release then :)
On 3 Oct 2016 5:00 am, "Ilya Zorin"<EMAIL_ADDRESS>wrote:

Thanks for clarification @nikitin-da https://github.com/nikitin-da!
Anyway, I will be happy to help with this task.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pushtorefresh/storio/issues/685#issuecomment-251058128,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3LHi2oPNnNQosC_7eDxso5pEMCCMks5qwMQlgaJpZM4KHoyr
.

https://github.com/ReactiveX/RxJava/releases/tag/v2.0.0

Yup, still waiting for more stable build though, but we can start migration.

Good reason to lay the foundation for StorIO v2, huh?

Hah, probably, but we're using semantic versioning so StorIO will become v2 as soon as we break API, adding RxJava v2 is planned as non-breaking change at the moment, so I guess StorIO will still be v1.
BUT: I'd like to keep both RxJava v1 and v2 only for the period of migration (in our internal and yours, users's) apps from RxJava v1 to v2 and when we'll see that RxJava v1 is not actively used anymore -> we'll drop support for RxJava v1 and that'll in fact require us to upgrade major version of StorIO!
@Adambl4 @nikitin-da @geralt-encore sounds good to you?

I agree, both versions will be required for a certain period of time

Sounds good 👍

Though I admit that with https://github.com/akarnokd/RxJava2Interop from @akarnokd we can leave only RxJava v2 support and force users to use interop library to convert types between v1 and v2, but without extension methods support in plain Java it'll be non-fluent and add a lot of boilerplate to the code.

What about using interop library internally? Completely migrate to v2 and provide methods for v1 interop via the library. Then just wipe it after some time.

RxJava1 has 5k methods, RxJava2 another 9k.
Forcing users to have v2 in classpath if they don't need it is a crime against humanity.

I don't like the idea to keep RxJava v2 methods only.
On the one hand it will speed up migration, but on the other hand it will lead to boilerplate on java6 and step towards to multidex 😱 on this period.
👍 for asRxObservable and asRxObservable2
Sure it will be more difficult for us, but much more democratic for users

@geralt-encore

What about using interop library internally? Completely migrate to v2 and provide methods for v1 interop via the library. Then just wipe it after some time.

:+1:
PR welcome!
@Adambl4 nobody will force you download RxJava v2, we'll add it same way we added RxJava v1 support, it's optional dependency!

Any updates on RxJava2 support?

Please use adapter library from @akarnokd for now.
On Wed, Dec 14, 2016, 16:09 outofdate<EMAIL_ADDRESS>wrote:

Any updates on RxJava2 support?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pushtorefresh/storio/issues/685#issuecomment-267029569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3KftfTT5dtjyUF6Srpk04XLWnztOks5rH-qAgaJpZM4KHoyr
.

Bump!, any progress on RxJava v2 support?

Not yet, starting integrating it into work projects, then I'll look into support for StorIO, PRs welcome!

I guess I can start working on what we agreed in https://github.com/pushtorefresh/storio/issues/767.
There are still a couple of open questions:

Should we expose database only as Flowable or with a new one Observable as well?
What are we going to do with nulls in object method?

Everything else looks pretty clear to me.
Also, I want to add Maybe to both object and objects method, but it should be addressed after the migration in a separate PR, I guess.

@artem-zinnatullin so I have been looking into RxJava 2 migration lately and I was thinking since 2.0 is going to introduce API breaking changes because of interceptors we can introduce more of them to make migration easier?
I was thinking about:

Migrating completely to RxJava 2
Changing Observable to Flowable everywhere and method names as well to reflect it
Removing object method since it doesn't fit into RxJava 2 non-null model

I understand that it is not the best way for migration I honestly don't see any way to migrate while maintaining backwards compatibility since we haven't done anything about it for quite some time. And I think that it is better to make migration happen like that than don't make it at all.
btw I need some help to implement OnSubscribeExecuteAsBlocking for RxJava2. The current implementation is based on some internal RxJava APIs which I am not aware of. OnSubscribeExecuteAsBlockingSingle and OnSubscribeExecuteAsBlockingCompletable were pretty straightforward to implement, but I have no idea how to correctly rewrite OnSubscribeExecuteAsBlocking.

Your points are fare, lemme start migration so you and @nikitin-da could
catch up and continue the work?
On Sun, May 21, 2017, 15:49 Ilya Zorin<EMAIL_ADDRESS>wrote:

@artem-zinnatullin https://github.com/artem-zinnatullin so I have been
looking into RxJava 2 migration lately and I was thinking since 2.0 is
going to introduce API breaking changes because of interceptors we can
introduce more of them to make migration easier?
I was thinking about:

Migrating completely to RxJava 2
Changing Observable to Flowable everywhere and method names as well
to reflect it
Removing object method since it doesn't fit into RxJava 2 non-null
model

I understand that it is not the best way for migration I honestly don't
see any way to migrate while maintaining backwards compatibility since we
haven't done anything about it for quite some time. And I think that it is
better to make migration happen like that than don't make it at all.
btw I need some help to implement OnSubscribeExecuteAsBlocking for
RxJava2. The current implementation is based on some internal RxJava APIs
which I am not aware of. OnSubscribeExecuteAsBlockingSingle and
OnSubscribeExecuteAsBlockingCompletable were pretty straightforward to
implement, but I have no idea how to correctly rewrite
OnSubscribeExecuteAsBlocking.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pushtorefresh/storio/issues/685#issuecomment-302934766,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA7B3LcxMkM7AW3fN_V2mcIbkxw0XV4xks5r8DLpgaJpZM4KHoyr
.

Yes! It would be really nice if you made sure that the foundation works correctly since you are the guy who originally wrote it and you know rx way better than me =)
I can take care of all the more high-level stuff, tests and etc.

So I've started migration yesterday, it'll definitely take a lot of evenings after work to complete. So what I'll do is — convert all StorIO main/src to RxJava 2 and use Interop library in tests so tests won't be modified that much. Then I'll create a PR so you could review it and merge, and then we'll be able to convert tests to RxJava 2 little by little.
Does that sound good?

Sound nice! It will allow to do it iteratively and together 👍

Sounds good! Let me know if can help you. I would be happy to help as much as possible.

Hello, guys! Are there any news about rxjava 2 integration in library?

All work is public and can be found in initial-2.x branch, I've migrated almost all StorIO codebase to RxJava 2.x
I need to make couple tests however, regarding need in Flowable and then a week of evenings or so to finish the work. If someone wants to continue my work, I'm more than happy to make a PR to 2.x branch so somebody could work on top of it

I am ready to help. I was stuck on some low-level parts when I was trying to migrate it myself. I assume that you already have dealt with them so I am ready to help with whatever needed.

For nullabillity in RxJava2, we need a way to emit a null field for when a record is not found while keeping a hot stream, so Maybe does not suffice.
Has this been considered? Maybe some type of converter/adapter? Or does .onErrorReturn() suffice to catch the null pointer exception and return our own Optional implemention (does this style of error handling have side effects?)

I was thinking about the same thing recently. Converter/adapter won't work since they apply after the emission. Emitting nulls and catching them with onErrorReturn() seems like an anti-pattern to be honest. The only solution that I see is to bundle some Optional implementation with StorIO and use it for that case.

Or does .onErrorReturn() suffice to catch the null pointer exception and return our own Optional implemention (does this style of error handling have side effects?)

This definitely does not work, because it terminates the stream.

The only solution that I see is to bundle some Optional implementation with StorIO and use it for that case.

This seems like the best solution. Just make sure that it's not called Optional and something more specific to this lib to not confuse it with our own Optional implantations.

Will you add a migration guide?

@Rainer-Lang Right now it is only available via https://github.com/akarnokd/RxJava2Interop
When we will finish build in rxjava2 support of cause we should add some readme about migration

@nikitin-da Thanks for your feedback.

@artem-zinnatullin can I continue migration based on your 2.x branch or is it still early and some crucial things are missing?

@geralt-encore I was stuck on this issue https://github.com/ReactiveX/RxJava/issues/5234 but it seems some workaround was released https://github.com/ReactiveX/RxJava/pull/5590

ah, you are working on it! Didn't know about that. So let me know if you need any help then.

@nikitin-da @geralt-encore Thanks for the update.
BTW, I don't want to use the Interop-lib for something so essential like the database.

@geralt-encore Is there any ETA on RxJava2 support? We are heavily dependent on StorIO for some basic functionality and we would like to see if we need to investigate for other solutions :) !!

@pavlospt sorry for delay.
Migration is in progress in branch https://github.com/pushtorefresh/storio/commits/az/initial-2.x
There are couple of unresolved issues for now: optional wrapper and some problems with class loader for use cases without rx dependency

I hope we will resolve it in 2 weeks

@nikitin-da No need to say sorry :) Thank you for your work on the project and for the insights :) !!


Remove php 5.3 from travis

I've just pulled this out from #23 as it seems to be an easy merge & I thought it might simply things a little

Removing 5.3 did end up in #23 so, so this PR can be closed off.