Changed include to support wildcards

I felt like the changes to ParseDirectives should be bundled with this commit, feel free to tell me otherwise.

ParseDirectives for the last param should always be a table. Even if it just has a single item. I think having string/table isn't nice behaviour.

ParseDirectives is has been made into a single block.

@EpicG Would this break anything you're working on? I remember us chatting about remaking the ParseDirectives to refactor it properly for chaining directive checking.

Was testing this a bit and came up with a few things:
You can't have more than two of the same statement, preprocessor.lua:125
err returned from ParseDirectives is a table, can't be concatenated in editor.lua:877
directive_include doesn't work properly when starting with *
Maybe change include directive example to show wildcards

Those are not the same statement, one is singular one is plural
Fixed err
fixed starting with *
Added example to show wildcards

I meant when you have two of the same directive, they are going to be overwritten in the directives table.
directives[ directive ] = args
If directive is equal to "include" then what ever was there before is just going to be overwritten.

Ohhhh I see now. gotcha. Will fix that asap.


[TOPI] Simplify GPU NMS IR and optimize a bit

I spent some time studying GPU NMS implementation in detail, and found a number of possible improvements. I simplified the IR quite a bit, the main NMS routine now consists only of two kernel calls (initialization and the triangle loop). I also removed unnecessary computation and memory movement. I believe what I have now is a much simpler, and faster NMS.
In particular, on PyTorch Mask RCNN workload (4900 boxes), this PR cuts more than 1000 micro seconds. This is actually significant because PyTorch NMS takes only 200 micro seconds. NMS from GluonCV SSD is also faster now by 2x (24 ms -> 12 ms, see blow).
Our NMS is still super slow for PyTorch models due to the serial triangle loop (fused_vision_non_max_suppression_kernel2 below):  PyTorch NMS doesn't have topk or score_threshold, so we need to take all boxes into account. Optimizing and parallelizing the triangle loop is an important future work that I plan on doing.
nvprof output from running PyTorch MaskRCNN
Before:
            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   99.95%  2.43544s         1  2.43544s  2.43544s  2.43544s  fused_vision_non_max_suppression_kernel2
                    0.02%  443.24us         1  443.24us  443.24us  443.24us  fused_vision_non_max_suppression_kernel6
                    0.02%  420.55us         1  420.55us  420.55us  420.55us  fused_vision_non_max_suppression_kernel1
                    0.01%  253.40us         1  253.40us  253.40us  253.40us  fused_vision_non_max_suppression_kernel5
                    0.00%  23.298us         1  23.298us  23.298us  23.298us  void cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700, bool=1, float, int, int>(int const *, cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700, bool=1, float, int, int>*, cub::DeviceRa
dixSortPolicy<float, int, int>::Policy700 const *, cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700, bool=1, float, int, int>**, bool=1, int, int)

After:
            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:  100.00%  2.17494s         1  2.17494s  2.17494s  2.17494s  fused_vision_non_max_suppression_kernel2
                    0.00%  22.401us         1  22.401us  22.401us  22.401us  void cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700
, bool=1, float, int, int>(int const *, cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700, bool=1, float, int, int>*, cub::DeviceRa
dixSortPolicy<float, int, int>::Policy700 const *, cub::DeviceRadixSortSingleTileKernel<cub::DeviceRadixSortPolicy<float, int, int>::Policy700, bool=1, float, int, int>**, b
ool=1, int, int)
                    0.00%  14.976us        10  1.4970us     512ns  6.7840us  [CUDA memcpy HtoD]
                    0.00%  5.4720us         1  5.4720us  5.4720us  5.4720us  fused_vision_non_max_suppression_kernel1

GluonCV SSD
Before:
            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   31.27%  24.388ms         1  24.388ms  24.388ms  24.388ms  fused_vision_non_max_suppression_kernel2
                   18.44%  14.378ms       206  69.795us     544ns  970.55us  [CUDA memcpy HtoD]
                    4.59%  3.5769ms         1  3.5769ms  3.5769ms  3.5769ms  fused_vision_non_max_suppression_kernel4
                    3.82%  2.9766ms         1  2.9766ms  2.9766ms  2.9766ms  fused_vision_get_valid_counts_kernel1
                    3.64%  2.8394ms         1  2.8394ms  2.8394ms  2.8394ms  fused_nn_conv2d_add_4_kernel0

After:
            Type  Time(%)      Time     Calls       Avg       Min       Max  Name
 GPU activities:   24.25%  14.487ms       206  70.325us     544ns  987.28us  [CUDA memcpy HtoD]
                   20.90%  12.489ms         1  12.489ms  12.489ms  12.489ms  fused_vision_non_max_suppression_kernel2
                    4.77%  2.8497ms         1  2.8497ms  2.8497ms  2.8497ms  fused_nn_conv2d_add_4_kernel0
                    4.29%  2.5619ms         1  2.5619ms  2.5619ms  2.5619ms  fused_nn_conv2d_add_11_kernel0
                    3.09%  1.8472ms         1  1.8472ms  1.8472ms  1.8472ms  fused_nn_conv2d_add_nn_relu_3_kernel0

There is some issue when running one of NMS tests in test_any.py, which is not run on CI due to flaky sort. Convert to draft until I resolve this problem.

Thanks @masahi @mbrookhart


Your data-lift is my data-lift: Reimplement data-lift as a data attribute parser.

This PR reimplements data-lift as a data attribute parser. This stemmed from a Lift hack day a few months ago, and I'm just now getting around to wrapping this up...
Few points of concern:

I want to go through at some point and clean up the copy-pasting of helper methods that I did wrt MetaData parsing.
I'm not entirely sure why I'm having to manually check for the parallel attribute in my parser. My understanding was that if my parser generates a lift: node, that should be processed subsequently by the regular SnippetNode parser, which does checks for parallel processing.
Can someone make sure my implementation of handling parallel is even functional while we're at it? lol

All specs are passing, though. So that's something.
closes #1582

I fixed the last comment I had. Going to use this as a guinea pig for Travis for a bit.

Okay, now that M3 is out I think we'll merge this fine feller. Going to have Travis give it a second look just to see what happens >_>

Less go!


Update details about platformsh/upsun plans in 2024 plans

The Issue
Add a bit of nuance about platform.sh/upsun goals for 2024.

Rebased after fixing tests.


Unable to execute a Direct Flow without specifying a globalSender

We're currently upgrading the signature-api-client-java from version 2.2. to version 4.6.
In our setup, we use a DirectClient with no global sender specified in the ClientConfiguration.
Instead we specify the sender’s organization number on each and every signature job.
In version 4.6, we encounter a problem with this method call, which throws an exception regardless if the job.getSender() is present or not.
no.digipost.signature.client.asice.CreateASiCE#createASiCE(JOB job):Line 40
        Sender sender = job.getSender()
                .orElse(globalSender
                .orElseThrow(SENDER_NOT_SPECIFIED));

The current workaround is to specify a dummy value for globalSender.

That is an awkwardly embarrassing bug, indeed. And it must have been there since the ability to specify sender per job was introduced 5 years ago.
Simple fix though: replace the code snippet with:
Sender sender = job.getSender()
                .orElseget(() -> globalSender
                .orElseThrow(SENDER_NOT_SPECIFIED));

to avoid invoking .orElseThrow on globalSender unless job.getSender() returns Optional.empty().
If you want to submit a pull-request, feel free! I will, though, be able to fix it myself at least within a few hours from now. And release v4.6.1 with the fix.
Thanks for reporting!

There is a fix in v4.6.1, available in Maven Central Repository. There is also a pending release of v5.0.1 which should soon also be available.


Step 3 Grant the user missing, missmatch  vs.

The REST example does mention a parameter called "".
The corresponding description does mention a parameter called "".
Would be good to align both so it is much easier to follow.
thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 564fcf4e-69b9-a8b0-a362-c4effeca4a8b
Version Independent ID: 7ae1feea-e9ca-bc0a-c984-81b19f4339b8
Content: Grant access to create Azure Enterprise subscriptions
Content Source: articles/cost-management-billing/manage/grant-access-to-create-subscription.md
Service: cost-management-billing
Sub-service: billing
GitHub Login: @bandersmsft
Microsoft Alias: banders

@cpinotossi Thanks a lot for your feedback, we will review and update the document accordingly.

Thanks a lot @cpinotossi for bringing this to notice. I have raised a PR to update the document , once reviewed and approved it will merge with the live document.
At this point it appears that there's no further edits or updates needed to the documentation, other than what is already listed above. We will go ahead and close this issue for now, however if you have any further feedback please tag us in your reply and will be happy to reopen the thread and continue the conversation. Thank you


Question: with Gulp how do you only SFTP files which have changed after minification?

Using gulp and gulp-sftp, I can't seem to only upload the file that has changed after CSS minification.
The semi-working snippet below begins by compiling the CSS and then continues to watch for changes in the src dir. It follows on by watching for changes in the dist dir (where minified CSS is stored) in order to upload that file to a web server.
However, this does not work as gulp is uploading everything rather than only the file that has changed and been minified.
var gulp = require('gulp'),
    changed = require('gulp-changed'),
    minifycss = require('gulp-minify-css'),
    sftp = require('gulp-sftp')
;

var src = './src/',
    dist = './dist/';

var srcStyles = src + '**/*.css',
    distStyles = dist + '**/*.css';

var host = 'ftp.xxxx.xx.xx',
    auth = 'keyMain',
    remotePath = 'public_html';

gulp.task('compilecss', function(){
    gulp.src(srcStyles)
        .pipe(changed(dist))
        .pipe(minifycss({keepBreaks: true}))
        .pipe(gulp.dest(dist))
    ;
});

gulp.task('uploadcss', function(){
    gulp.src(distStyles)
        .pipe(changed(dist))
        .pipe(sftp({
            host: host,
            auth: auth,
            remotePath: remotePath
        }))
    ;
});

gulp.task('main', function(){
    gulp.start('compilecss');
});

gulp.task('watch', function(){
    gulp.watch(srcStyles, ['compilecss']);
    gulp.watch(distStyles, ['uploadcss']);
});

gulp.task('default', ['main', 'watch']);

@contra Using rimraf doesn't work if you are trying to delete multiple files, which is contrary to trying to put all the cleans in one task. Either that or I can't figure out how it works since it can't take an array of file paths and can't be called twice.


Review hard code on web port

In the docker-compose need to change the WEB_PORT to 8080 or update the port mapping to use 8888
WEB_PORT: 8080
ports:

8080:8080

Suggest by @svfreitas

Fixed in the latest version in the main branch at https://github.com/thangchung/go-coffeeshop/commit/22d33190a0cf81630878e47bafae2de7c0abb783. Feel free to re-open it if there are still any things else to fix.


RFC: deposit: multiuser submission

Problem:
Currently, deposit module is dependent on having a current user, and the data model is tied to a user id.
This prevents both:

Guest submissions in the deposit module (needed by Inspire).
Multiple users access to a single deposition (needed by Zenodo).

Proposal:

Remove user id property from deposition model.
Add model for controlling access to deposition (e.g. many-to-many relationship between a depositions and (users or access tokens)).
Allow access to a deposition based on an access token (to support guest submissions). This allow long running guest submissions, and sharing a guest submission. Likely the access token should expire and deposition cleaned be up after a while.

Co-authored-by: @lnielsen-cern, @kneczaj @pedrogaudencio @giokokos

Closing this for now as no one are working on it. Eventually we will need on Zenodo and reopen then.

In cases like this, we should also change the milestone for better book keeping.  E.g. we can use a latter milestone like "2.5" (if resources are planned), or use "someday" (if unknown).  Otherwise this issue would not be easily distinguishable from any regularly closed RFC for 2.2.
P.S. In the CERN Open Data project we use "someday" as a milestone, not as a label.  This seems more appropriate in this case for better bookkeeping.  Let's see IRL if we change it globally too.


Privacy Settings dialog box does NOT show up on Iphone

When I click on the privacy settings nothing happens. I've tried it on both safari and chrome on an Iphone.
TO go to Privacy Settings - Go to the top menu and click privacy settings

Fixed!

Thanks for letting us know of the issue

Thanks for fixing it.


Chat history is not saved for the user

STR:

Start a conversation from a user
Log in with the same credentials on another device
Check chat history
Send smth to the operator and check the chat history on another device

Actual result: in portal.infobip are created 2 different chats after these steps / chat history is not the same on the UI for the same user

Hi @Marieye,
we will check it and let  you know. Thank you for report.

We found a problem. There was expired key. So we continued the operation of the key, and now issue doesn't reproduce.

Thank you for info, closing the issue.


RangeError: Maximum call stack size exceeded

Bug report

What is the current behavior?
After running npm run prod, webpack throws,
➜  aem-test git:(main) ✗ npm run prod

><EMAIL_ADDRESS>prod
> NODE_ENV=production webpack --config ./webpack.prod.js && clientlib --verbose

/Users/kolivar/aem-test/node_modules/webpack/lib/util/runtime.js:60
exports.forEachRuntime = (runtime, fn, deterministicOrder = false) => {
                         ^

RangeError: Maximum call stack size exceeded
    at exports.forEachRuntime (/Users/kolivar/aem-test/node_modules/webpack/lib/util/runtime.js:60:26)
    at ExportInfo.setUsed (/Users/kolivar/aem-test/node_modules/webpack/lib/ExportsInfo.js:1017:4)
    at ExportsInfo.setUsedWithoutInfo (/Users/kolivar/aem-test/node_modules/webpack/lib/ExportsInfo.js:392:31)
    at processReferencedModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:80:25)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:270:9)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:206:10)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:206:10)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:206:10)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:206:10)
    at processModule (/Users/kolivar/aem-test/node_modules/webpack/lib/FlagDependencyUsagePlugin.js:206:10)

If the current behavior is a bug, please provide the steps to reproduce.

The application should successfully be built as with using the command npm run dev.
If it can also help investigating this issue, I first thought it was maybe related to MiniCssExtractPlugin. Here some more details on that issue https://github.com/webpack-contrib/mini-css-extract-plugin/issues/733.
However, the problem may also be in webpack.

Other relevant information:
webpack version:  5.65.0
Node.js version: 16.11.1
Operating System: Mac OS Big Sur 11.5.2 (20G95)
Additional tools:
npm version: 8.0.0
yarn version: 1.22.5
webpack-cli version: 4.9.1
webpack-dev-server version: 4.7.1
webpack-merge version: 5.8.0
mini-css-extract-plugin version: 2.4.5

I look at source code of glob-import-loader and there are some problems too:

import has absolute paths, i.e. import "/home/evilebottnawi/IdeaProjects/aem-test/src/main/webpack/site/main.ts";, and it is wrong, because import should be URL
Also import should have relative paths (relative to the file where it is used), not absolute, because it will be bad for cache

Why you need this loader? I would recommend to be as close as possible with ESM spec and do not use this loader.

Anyway I found a problem, your code is
// Other code
import "./**/*.ts";

After glob you have:
import "/home/evilebottnawi/IdeaProjects/aem-test/src/main/webpack/site/main.ts";

i.e. you load main.ts inside main.ts -  circular dependencies, webpack tries to optimize your code (css modules generated by mini-css-extract-plugin in referenced module and webpack tried to understand can we drop some code for this, https://webpack.js.org/configuration/optimization/#optimizationusedexports) and here we have a loop (and a problem), because. Adding more modules will fix the problem (i.e. just create foo.ts and you will see it). But yes, we should generate the better error message here.
So solution is exclude main.ts, I tried using /**/!(*main).ts, but loader emits an error. So I recommend to fix circular dependency, because I'm sure this is not the behavior you expect (note - circular dependencies have some limitations in webpack). And using glob can be very dangerous because you can accidental include/import something in your code and it will be bad for your app size/perf, especial using **/*.

I look at source code of glob-import-loader and there are some problems too:

import has absolute paths, i.e. import "/home/evilebottnawi/IdeaProjects/aem-test/src/main/webpack/site/main.ts";, and it is wrong, because import should be URL
Also import should have relative paths (relative to the file where it is used), not absolute, because it will be bad for cache

Why you need this loader? I would recommend to be as close as possible with ESM spec and do not use this loader.

I am using a generated project and this is what I get out of the box,
https://github.com/adobe/aem-project-archetype/tree/develop/src/main/archetype/ui.frontend.general
https://github.com/adobe/aem-project-archetype/blob/develop/src/main/archetype/ui.frontend.general/src/main/webpack/site/main.ts
The thing is that it works fine if I don't upgrade the npm packages. Eg. webpack...
As you can notice, the package.json contains outdated packages.

There is the problem with circular dependencies, I am afraid it works before, but works invalid, using import "./**/*.ts"; should not include itself, it is wrong

@alexander-akait Thanks for your reply, I will then remove this glob-import-loader.

You can try to rewrite glob to exclude file, but I am failed with it, maybe due old version of glob

Importing itself shouldn't lead to an error... That might be a bug.

or at least a better message? As @alexander-akait mentioned

You can try to rewrite glob to exclude file, but I am failed with it, maybe due old version of glob

@soluml?

@kevinolivar thanks, yea I'll take a look at it tomorrow. Patching glob-import-loader to prevent a file from including itself should be simple to do, but you still have to be mindful that generic globs as you've implemented them can still cause you to include more than intended. It's not a standard feature so be sure to check out the docs for the loader. I'll tag you once an update is available.
@alexander-akait some glob features (special characters) aren't available due to incompatibilities with underlying webpack tooling. The loader docs calls these out.
Also, even though the loader should exclude itself from the import, it's worth looking into Sokra's note that self importing shouldn't cause an error. I hadn't run into this problem with Webpack 4 when I forked/built the loader.

@sokra It is due mini-css-extract-plugin, when we have one CssModule and import itself in one file, we have a loop

@kevinolivar - glob-import-loader has been updated which excludes the resourcePath from being glob imported.

@soluml thanks mate, appreciate. I think at the end, as a good practice and as you and @alexander-akait mentioned, it is good not relying on glob but rather ESM spec. Just to avoid importing undesirable files.
Is there anything else open to address? Or this issue can be closed?
From my understanding, maybe for @alexander-akait a better way of handling circular dependencies with clearer error message if that issue was caused by the mini-css-extract-plugin (and not webpack)?
I really appreciate your help @alexander-akait and @soluml. @sokra too for your input as well.

Let's keep open, webpack still should not throw an error on circular dependencies

@kevinolivar All the loader does is make it easier to import large numbers of files in a single easy to read statement. This can be helpful in component libraries, or in AEM's case, importing all component JS/TS into a single clientlib. In these cases though, it's best to be more specific.. something like: import './components/**/.ts' where we'd want to import all component TypeScript into a site clientlib for example.
You can also do something similar to this with Webpack's require.context, but then, you don't get the advantages of ESM so this tries to give you the best of both worlds.
The loader is also handy in that it leverages the enhanced-loader package under the covers, meaning you can also use globbing with things like aliases: import 'MY_COMPONENTS_FOLDER/**.ts';. This is also the reason why some globbing characters can't be used.
Hopefully that makes sense. If you have 20+ odd components that you need to import into a single entry for clientlib purposes, this may really help you out. Any new components will automatically be added to the clientlib which I've found very handy when working on big teams. If you're just importing a few files here and there, best to be explicit and skip the globbing. Any non-globbed imports will be completely ignored by the loader so there shouldn't be any harm in using it on an as needed basis.

@soluml I myself never used glob but always manually included files I needed. Eg. exporting my components through a single index.ts.
I guess case by case. Eg. team size, project size...
Your explanation makes absolutely sense, thanks again for the clarifications.

Importing itself shouldn't lead to an error... That might be a bug.

Any news regarding this?

@vankop glob-import-loader fixed the problem, the main problem here https://github.com/webpack/webpack/issues/15060#issuecomment-1002093799 (but maybe we fixed it?), need to check it, circular import itself, try
main.ts
import "./main.ts";

@alexander-akait
circular import itself works fine in js and css (mini-css) I have checked

Thank you guys!


storage: vectored get does not handle aux key range

Steps to reproduce
with the new aux file v2 pull request https://github.com/neondatabase/neon/pull/7401, we now have a full key range that needs to be ignored.
an alternative solution is to specify "no_ancestor_branch" when calling the interface so that all key-values within the scan operation does not fall into the ancestor branch, so that we avoid per-key decision on whether to look into the ancestor.
Expected result
Actual result
Environment
Logs, links

We believe https://github.com/neondatabase/neon/pull/7404 was the fix for this - @skyzh please reopen if not.


Fingerprint 23 RAV4 LE

Description:
Added fingerprint for Engine ECU for 2023 RAV4 LE
Verification:
Tested on personal vehicle. Initially wasn't recognized by openpilot and with this fix openpilot works successfully.

Be sure to include the dongleID of the device this firmware version was gathered from

Be sure to include the dongleID of the device this firmware version was gathered from

dongleID is cb4c4fc7644d7b30

@sebmueller01 are you on Discord? Are you aware this car uses a different steering API and so it doesn't work on master yet? We have a channel you might be interested in joining here: https://discord.com/channels/469524606043160576/1047684141963022376

Yup, currently running the toyota-lta branch forked that includes the updated ECU on my C3. I'm in the discord channel and I was told to target master for my PR. Definitely willing to help out in any way I can to get longitudinal working :)


Created for level: 90609201

reject
the delay is not correct making the level unsynced


Ecog plot

Taking over https://github.com/mne-tools/mne-python/pull/3976
Closes https://github.com/mne-tools/mne-python/issues/3971

Codecov Report

Merging #3977 into master will decrease coverage by -12.78%.

@@             Coverage Diff             @@
##           master    #3977       +/-   ##
===========================================
- Coverage   85.95%   73.17%   -12.78%
===========================================
  Files         349      349
  Lines       62496    62509       +13
  Branches     9569     9570        +1
===========================================
- Hits        53720    45743     -7977
- Misses       6088    14465     +8377
+ Partials     2688     2301      -387

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 296154c...d28f14b. Read the comment docs.

This looks like the right fix at one level. However, I thought there was another level -- in his use case it should be able to create a layout on the fly because the electrode positions are given in the montage, no?

it should be able to create a layout on the fly because the electrode positions are given in the montage, no?

I think it's doing that. The locations come from the montage, right? The layout is generated below if len(name_idx) < len(chs).

Ahh you're right, I didn't realize it was generating a custom one in there. LGTM then!
@MartinPerez can you see if this fixes the error?

I realize also raw.plot_psd_topo() fails directly due to some layout issue

AttributeError                            Traceback (most recent call last)
 in ()
----> 1 raw.plot_psd_topo()
/media/mfpgt/bigdata/PHD_related/open_source_contributions/mne-python/mne/io/base.pyc in plot_psd_topo(self, tmin, tmax, fmin, fmax, proj, n_fft, n_overlap, layout, color, fig_facecolor, axis_facecolor, dB, show, block, n_jobs, verbose)
1649                                  axis_facecolor=axis_facecolor, dB=dB,
1650                                  show=show, block=block, n_jobs=n_jobs,
-> 1651                                  verbose=verbose)
1652
1653     def estimate_rank(self, tstart=0.0, tstop=30.0, tol=1e-4,
/media/mfpgt/bigdata/PHD_related/open_source_contributions/mne-python/mne/viz/raw.pyc in plot_raw_psd_topo(raw, tmin, tmax, fmin, fmax, proj, n_fft, n_overlap, layout, color, fig_facecolor, axis_facecolor, dB, show, block, n_jobs, verbose)
959                      axis_facecolor=axis_facecolor,
960                      fig_facecolor=fig_facecolor, x_label='Frequency (Hz)',
--> 961                      unified=True, y_label=y_label)
962
963     try:
/media/mfpgt/bigdata/PHD_related/open_source_contributions/mne-python/mne/viz/topo.pyc in _plot_topo(info, times, show_func, click_func, layout, vmin, vmax, ylim, colorbar, border, axis_facecolor, fig_facecolor, cmap, layout_scale, title, x_label, y_label, font_color, unified, img)
177                                     unified=unified, img=img)
178
--> 179     for ax, ch_idx in my_topo_plot:
180         if layout.kind == 'Vectorview-all' and ylim is not None:
181             this_type = {'mag': 0, 'grad': 1}[channel_type(info, ch_idx)]
/media/mfpgt/bigdata/PHD_related/open_source_contributions/mne-python/mne/viz/topo.pyc in _iter_topography(info, layout, on_pick, fig, fig_facecolor, axis_facecolor, axis_spinecolor, layout_scale, unified, img)
99         fig.canvas.mpl_connect('button_press_event', callback)
100
--> 101     pos = layout.pos.copy()
102     if layout_scale:
103         pos[:, :2] *= layout_scale
AttributeError: 'NoneType' object has no attribute 'pos'

I confirm @jaeilepp fix works. @Eric89GXL I realize my main issue was similar to #3877, I attempted the steps you suggested without success. I took the MNI coord in mm to m, then followed the rest of the steps, but did not find where to use the convert=True argument. When trying plot_trans with three example sensors I get the points drawn, but in a plot_psd(average=False) I get an empty 2d projection. I verified from snapshot_brain_montage i get xy coordinates with exorbitant values around 500, when original sensor coordinates in m are around 0.03 so I dont really understand where transformations are taking place.
I append example images:

Here is my example code:
import mne
import numpy as np
from mayavi import mlab

lpa = [-0.1, 0, 0]
rpa = [0.1, 0, 0]
nasion = [0, 0.1, 0]
locations = [[0., 0., 0.], [0.03, -0.03, 0.03], [0.04, -0.01, 0.02]]
names = ['lala', 'lolo', 'lili']
mon = mne.channels.DigMontage(lpa=lpa, rpa=rpa, nasion=nasion, dig_ch_pos=dict(zip(names, locations)))
info = mne.create_info(names, 512., 'ecog', mon)
raw = mne.io.RawArray(np.random.rand(3, 10000), info)
fig = mne.viz.plot_trans(info, trans=None)
mlab.view(200, 70)
xy, im = mne.viz.snapshot_brain_montage(fig, mon)

As an extra clarification, I selected those arbitrary example coordinates expecting them to be close to the center of the head in the 2d projection plot, as they correspond to MNI coordinates.

The layout generation failed because _cart_to_sph returned nans when converting (0., 0., 0.). I added a line which converts nans back to zeros.

@jaeilepp I confirm the nan change makes the electrodes visible in the 2d projection and clearly reveals my lack of understanding to implement the correct transformations (one point should be centered in the plot). Here image from plot_sensors(). Still plot_psd_topo() has the same error as before.

Ok, yes I see your point. I believe the points are simply centralized since usually the points are somewhat evenly spread around the scalp and AFAIK the origin is not necessarily at the center.

AFAIK the origin is not necessarily at the center.

We might want to ensure our layouts are all in head coordinates if possible. Then our layouts have a true center, and that center will correspond to the center in these plots.

I think I figured out a couple of things, first that being able to pass the head_pos argument to _check_outlines method in topomap.py might effectively give complete control of the plot. So I managed to make raw.plot_sensors() give me what I want by doing this, although it breaks with raw.plot_psd().
Second that find_layout() in layout.py gives automatically a None layout for "ecog" completely ignoring the info file. So I think in line 453 an (elif has_ecog) could be added to return a sensible layout.
I tried successfully to eliminate the raw.plot_psd_topo crash:
    has_ecog = FIFF.FIFFV_ECOG_CH in channel_types
    elif has_ecog:
        locations = np.array([c['loc'][:3] for c in chs])
        pos = _pol_to_cart(_cart_to_sph(locations)[:, 1:][:, ::-1])
        return generate_2d_layout(pos)

hope this helps...

Now the custom made layout has origin at the center of the head. When a layout is found with find_layout, it still scales the coordinates to run from zero upwards. I'm thinking that maybe it should be left as is, since it's working fine AFAIK.

What do you mean scales the coordinate to run from 0 upwards?  If 0 is the center, then there would be negative coordinates no? Could you please plot an example?  The 69 electrodes colored by their number would look like this:

I think occupying from bottom to top automatically could be very misleading if that is what you mean. For example in this electrode set I have no electrodes from the lower half of the brain. I append the coordinates in case it helps to test.
coords.json.zip

I could also give a hand implementing the same type of topo visualization you have with the head, but with a mni glass brain template, if its a desirable feature. I tried to read from the code the type of interpolations you were doing to achieve the nice gradients and contours but I admit it was difficult for me to follow the plotting code.

I think we can always assume (or convert MEG to) head coordinates. The case
of MNI coordinates should be converted to head coordinates ahead of time,
and we have multiple issues open about how we need to change our code to
facilitate that so it's a separate issue.

What do you mean scales the coordinate to run from 0 upwards?

Each coordinate in range [0, 1].

@jaeilepp I tested the branch with my electrodes, it looks like this at the moment. The center is effectively at the center of the head. But playing with the nasion, lpa and rpa did not have any influence in the plot (I changed them hoping to squeeze the electrodes to fix the plot). Reasonable values would be 0.08 for lpa and rpa according to a MNI template. In the case of nasion it would be 0.08 but electrodes on the back would fall out of the template since its -0.11 for the back, so I guess 0.11 would be better as nasion to keep everything inside.

Also trying to select a portion of the plot would throw another error

IndexError                                Traceback (most recent call last)
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/backends/backend_qt5.pyc in mouseReleaseEvent(self, event)
298         if button is not None:
299             FigureCanvasBase.button_release_event(self, x, y, button,
--> 300                                                   guiEvent=event)
301         if DEBUG:
302             print('button released')
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/backend_bases.pyc in button_release_event(self, x, y, button, guiEvent)
1930         s = 'button_release_event'
1931         event = MouseEvent(s, self, x, y, button, self._key, guiEvent=guiEvent)
-> 1932         self.callbacks.process(s, event)
1933         self._button = None
1934
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/cbook.pyc in process(self, s, *args, **kwargs)
547             for cid, proxy in list(six.iteritems(self.callbacks[s])):
548                 try:
--> 549                     proxy(*args, **kwargs)
550                 except ReferenceError:
551                     self._remove_proxy(proxy)
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/cbook.pyc in call(self, *args, **kwargs)
414             mtd = self.func
415         # invoke the callable and return the result
--> 416         return mtd(*args, **kwargs)
417
418     def eq(self, other):
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/widgets.pyc in release(self, event)
1291             event = self._clean_event(event)
1292             self.eventrelease = event
-> 1293             self._release(event)
1294             self.eventpress = None
1295             self.eventrelease = None
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/widgets.pyc in _release(self, event)
1535         if self.minspan is not None and span < self.minspan:
1536             return
-> 1537         self.onselect(vmin, vmax)
1538         self.pressv = None
1539         return False
/media/mfpgt/bigdata/PHD_related/open_source_contributions/mne-python/mne/viz/evoked.pyc in _line_plot_onselect(xmin, xmax, ch_types, info, data, times, text, psd)
102     maxidx = np.abs(times - xmax).argmin()
103     fig, axarr = plt.subplots(1, len(ch_types), squeeze=False,
--> 104                               figsize=(3 * len(ch_types), 3))
105
106     for idx, ch_type in enumerate(ch_types):
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/pyplot.pyc in subplots(nrows, ncols, sharex, sharey, squeeze, subplot_kw, gridspec_kw, **fig_kw)
1210
1211     # Create first subplot separately, so we can share it if requested
-> 1212     ax0 = fig.add_subplot(gs[0, 0], **subplot_kw)
1213     axarr[0] = ax0
1214
/home/mfpgt/miniconda2/envs/blackboard/lib/python2.7/site-packages/matplotlib/gridspec.pyc in getitem(self, key)
166                     k2 += ncols
167                 if k2 >= ncols or k2 < 0 :
--> 168                     raise IndexError("index out of range")
169                 col1, col2 = k2, k2+1
170
IndexError: index out of range
the tested code is this:
    lpa = [-0.08, 0, 0]
    rpa = [0.08, 0, 0]
    nasion = [0, 0.11, 0]
    locations = (np.array(data['coords']) / 1000.).tolist()
    names = data['info']['ch_names']
    mon = mne.channels.DigMontage(lpa=lpa, rpa=rpa, nasion=nasion, dig_ch_pos=dict(zip(names, locations)))
    info = mne.create_info(names, 512., 'ecog', mon)
    raw = mne.io.RawArray(data['lfp'][0], info)

BTW @MartinPerez thanks for working through these bugs with us. Tightening up these coordinate frame and plotting issues is a high priority in the next month or so

Its my pleasure @Eric89GXL, thank you for responding so quickly to these issues :)

I think the problem @MartinPerez is experiencing stems from fitting the 3d locations to a sphere. So in addition we would need to have a way of disabling that. I'm not sure how the API would look like.

So I'm thinking maybe just dump the z-coordinate when the coordinate frame is provided. So just have one param for lims and use only xy-coordinates. This would only work for data that does not have a layout available. Another option would be to expose layout for plot_psd and other functions that allow spatial_colors.

So I'm thinking maybe just dump the z-coordinate when the coordinate frame is provided.

When/why are we providing a coordinate frame? Again, I think these "plots on heads" should always plot sensors into head coords (for EEG they should be there already, for MEG they should be transformed using dev_head_t if possible).
The ECoG use case where things are often in MRI coords is in my mind a separate issue that should be solved by providing a way to get them to head coords one way or another, even if by simple identity transform (and we have issues open for that already).

Another option would be to expose layout for plot_psd and other functions that allow spatial_colors.

That would be fine, and just expose e.g. a projection parameter in the layout-making function or something?

fitting the 3d locations to a sphere

Actually it seems we don't fit it to a sphere, but use spherical (theta, pol) as (r, theta) for polar->cartesian. I wonder why we do it like that.

It's standard eeg practice
Alex
On 24 Feb 2017, 08:08 +0100, jaeilepp<EMAIL_ADDRESS>wrote:

fitting the 3d locations to a sphere

Actually it seems we don't fit it to a sphere, but use spherical (theta, pol) as (r, theta) for polar->cartesian (https://github.com/mne-tools/mne-python/blob/master/mne/channels/layout.py#L730). I wonder why we do it like that.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (https://github.com/mne-tools/mne-python/pull/3977#issuecomment-282220753), or mute the thread (https://github.com/notifications/unsubscribe-auth/AAJ1HF4hQD_FonJRUnQ57vVmpfHvzoJTks5rfoH4gaJpZM4L7xV2).

Well, with that practice it is impossible to plot points that are not on a sphere accurately. For instance, locations like [.5, .5, 0] and [50, 50, 0] get transformed to the same point.

what do you suggest?

It would probably be best to take into account the radial distance in _pol_to_cart.

@MartinPerez it looks like @jaeilepp's latest changes should fix your latest bug, can you try it?

@Eric89GXL, @jaeilepp now the electrodes are inside the skull! Although lpa, rpa and nasion are still ignored and the error on selection remains. It looks like this now:

You mean that you want to see fiducials plotted also if they are present in
the dig?

I understood from the example you gave in another post that the fiducials would control the position of the electrodes relative to the boundaries of the head. Right now the electrodes remain in the same plotted position independently of any fiducial value provided. Am I wrong about this?

So for example I expected decreasing the value of lpa and rpa would increase the spread of the electrodes.

Although of course I can modify the spread by changing directly the coordinates, so I would not worry about this if the fiducials are not meant to be used in this way, it would be good to clarify this in the example for ecog once all errors are fixed. Here with modified coordinates:

@MartinPerez my feeling is that you're trying to fit a square in a round
hole.
EcOG is not visualized like EEG and I doubt you can make a paper figure with
EcOG displayed as an EEG topo. So I am fine if it works ok but let's not
allocate
too much energy for this. In case you missed it it's how EcOG would be
visualized
one the brain http://martinos.org/mne/dev/auto_tutorials/plot_ecog.html

@agramfort I see your point, is not my intention. I do find the topo visualizations extremely useful to explore the data. Also I find the type of analysis I am doing on comparing high gamma power time series, not very different from some MEG analysis I saw on examples.
In particular being able to visualize in topo with contours the clusters from a spatiotemporal cluster f test and some butterfly plots in other cases could be very instructive and nice to present to others :). It would be a pity to not be able to take advantage of this functions only due to determining electrode positions on the plot.
Said this I agree that is not a priority and would understand that if adding an Ecog case for topo is too high mantainance for a small user base, then might not be worth it. Anyway I have already managed to take advantage of everything else by just considering the electrodes as 'mag' and disabling any undesired assumptions on data corrections in some functions to replace for mine, mostly related to baseline correction :).
There are other cases in epoch plots were just plotting the time series would also fail since the 'ecog' case is not considered. Anyway I might have the wrong idea about the type of analysis you want to support for ecog too, if you quickly clarify this for me I will focus on testing those first to give you a hand with what is an actual priority and PR a nice example/tutorial after :).

well for me no function should crash if ch_type == ecog
Worse case it should disable some options if it's not relevant for ecog.
For topos I would say either we disable topos for ECoG as it's not standard
or we accept it looks 'funny"
Hacking it to mag is therefore not ok…

well if disabling options and topo is just a matter of adding some few ifs around the code in the right places, I think it would be more sensible for this PR to help test that. If its more or less the same amount of work (just passing the corrected electrode positions that @jaeilepp already fixed and let topo do the rest) I would vote to go for the funny and warn about it in the tutorial. WDYT?

ok for the funny but I am not sure we should show ecog topos in tutorials
which are meant to show the good practice.
you can just not use spatial_colors=True and things will go easy with
ecog. right?

I don't see a big problem with drawing a head with the sensors for ECoG, we
just say the sensors are inside the head instead of outside in that case.
If we properly define the coordinate frames it's reasonable anyway.

sure but the spacing between ecog electrodes is too small to make a proper/clean topo. It gives you a sense of where they are true that's why I am saying it will just look funny...

Now the interactive selection should work for drawing topomaps. This is not thoroughly tested, but maybe I could add a test with random ecog data.

I just checked, it does not fail anymore, but the topomap shows electrode positions in the old way. Which btw is a similar behavior to raw.plot_sensors()

@MartinPerez I hope you don't interpret such images. What you're looking at is huge interpolation artifacts due to inadequate sampling of the sensors. Again looking at topographies like this for ECoG does not make sense.

Again looking at topographies like this for ECoG does not make sense.

Agreed. Maybe I should revert allowing ecog for topomaps?

I would say yes.

@agramfort and @jaeilepp I agree like this of course can not be even looked at. I thought you would show interpolation within the new electrode positions (as they appear now above the time series), otherwise I agree it would be just better to disable topo completely. Plot_sensors would also need to be disabled or the electrode positions fixed to be coherent with the one shown in the times series.

Plot_sensors would also need to be disabled or the electrode positions fixed to be coherent with the one shown in the times series.

@jaeilepp you up for fixing that in this PR, too, or should that be separate?
Would it be possible/good to unify all of these layout searches? Or is that already done and you just need to set to_sphere=False somewhere?

@jaeilepp you up for fixing that in this PR, too, or should that be separate?

I'll do it now.

is that already done and you just need to set to_sphere=False somewhere?

Yeah something like that. Is it okay to do it always or should we add a param to plot_sensors?

Here's the difference on neuromag:
To sphere

Not to sphere

I think we should add a param and have it default to True, it's backward compatible that way, too.

Agreed and done. Ready for review.

@MartinPerez are you happy with the fixes in this one?

@Eric89GXL and @jaeilepp I tested all plot functions in raw, epochs and evoked to see if appropriate error messages are prompted. So raw is all fine now, I wonder if you would like to also raise the same type of error of the selection on plot_psd_topo() to make clear that this function should not be used. Right now it just shows an empty black screen.
Then in epochs all is fine and the plot functions not supposed to be used throw a no channel type found exception, which makes sense.
In evoked plot_topo does not work correctly simply failing in a weird line instead of declaring that no channel type found or that ecog is not allowed. While plot_topomap throws the right exception that no channel type is found. Also if plot_joint is attempted it throws the right exception of no plottable channel type found but do not close the initialized plot, so when another plot like sensors is called then it also shows the incomplete plot from plot joint.
These would be all the remaining unexpected errors in plot functions with ecog. Maybe it makes sense to just finish taking care of these in this PR? Everything else seems to work fine. I will move my analysis to ecog and report any other failure not related to plots if they arrive.

Thanks @MartinPerez for checking.
@jaeilepp did your changes cover these (from @MartinPerez):

"raise the same type of error of the selection on plot_psd_topo() to make clear that this function should not be used."
(Although maybe it should just be disabled...?)

"evoked plot_topo does not work correctly simply failing in a weird line instead of declaring that no channel type found or that ecog is not allowed"

"if plot_joint is attempted it throws the right exception of no plottable channel type found but do not close the initialized plot"

Still not quite addressed -- vectorize?

Other than my last little comment, +1 for merge

why do you need to warning unless you think the generated version will not
look good?

It is an API change of sort. Before, if the layout could not be found, it would
return None

well it would generate it from sensor locations and otherwise yes return None.

and the predefined layouts are not using the exact channel locations.

we only have predefined layouts for MEG right?

we only have predefined layouts for MEG right?

Yeah, I think so. I'll remove the on_failure option

thx @jaeilepp


Faster (and smaller) WASM runtime builds

This tackles underlying issue from #12671 (partially) from a different angle by simply not compiling any dependencies in substrate-wasm-builder when compiling WASM version of the crate in the first place.
It is a bit awkward that WasmBuilder is copy-pasted and not implementing any shared trait with regular one, but I decided it might be acceptable for now.
What this means is that runtime is build faster and smaller.
target/debug/wbuild/node-template-runtime/target size:

before ~1.5G
after ~0.6G

I didn't bother measuring compile times though.

Replaced with #12790


Feature Limitations for Business Central SaaS Web Client

Hello
I cannot find any information about the BC web client's features limitations.
This page shows information only for Nav 2018 / Nav 2017.
Our customers want to switch from BC OnPrem Windows Client (BC v14) to BC OnPrem Web Client (BC v14) and need to know the "incompatibility" of the web client.
Can you help us ?
thank you

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d2b3a845-b388-9c69-478f-87cc70b87819
Version Independent ID: af209cd5-e554-fe3f-30f0-05d771b21a17
Content: Feature Limitations of the Web Client - Dynamics NAV
Content Source: dynamics-nav/Feature-Limitations-of-the-Microsoft-Dynamics-NAV-Web-Client.md
Product: dynamics-nav-2018
GitHub Login: @jswymer

Hello @jnpatrouillault ,
There is no equivalent article for Dynamics 365 Business Central. While the Windows client is still available in version 14 on-premises, resources went into adding productivity features to the Web client. For more information, see the release plan for version 14 (https://docs.microsoft.com/en-us/business-applications-release-notes/April19/dynamics365-business-central/planned-features) and version 15 (https://docs.microsoft.com/en-us/dynamics365-release-plan/2019wave2/dynamics365-business-central/modern-clients).
Optionally, you can show them a demo of the latest version of Business Central online, which is very close to closing the last gaps.

The lack of an equivalent article for BC causes questions such as
https://www.yammer.com/dynamicsnavdev/#/Threads/show?threadId=1619878328786944
I could only say 'the NAV 2018 limitations probably still apply, mostly, maybe'... when really we should have a corresponding article about the modern BC Web Client, detailing which features that RTC-familiar developers will find are (still) missing.

Hello Daniel,
We disagree on the subject, which is OK. As mentioned in several Yammer-threads, the modern clients are based on the web client, which has undergone many, many improvements in the past 5 years with far more capabilities than it had before. We're not going to publish a list of differences between the deprecated Windows client and the modern clients because it doesn't make sense. There are things user can do with the modern clients that they were not able to do with the Windows client; and there are things that don't make sense in a browser that made more sense in the installed client.

You have a good point!
What makes more sense: all the things that are still accepted by the compiler, but will later at runtime fail or simply do nothing... should not be allowed to compile. I'll start opening separate issues on the AL repo for those.


SR-IOV and Discrete Device Assignment and WSL2 and Legacy hardware

Your Windows build number:  Microsoft Windows [Version 10.0.18362.239]

What you're doing and what's happening:  I have an oldish box (3930k, ga-x79-up4, 32GB quad channel). I recently had (I think) a fleeting run with tensorflow gpu accelerated in a docker container! I also had hyper-v server running in hyper-v too. And then the latest update to windows dropped and the  world changed for the worse, again. Nothing worked, a complete reinstall of docker (well I think it was) did no good and now linux containers for docker no longer work. There seems to be legacy crud in the filesystem and registry, and I do not have the hours to try and do your jobs (the collective MS 'your'). All I want to do is get stuff done without continually getting the rug pulled out from under my feet! To cap it off, during my toil, I ran up a powershell script (SurveyDDA.ps1) that was meant to tell me if my hardware could handle the brave new world of SR-IOV, Discrete Device Assignment and hyper-v. Now I don't know and the script told me that it has disabled something incomprehensible! For security! A SURVEY SCRIPT! Argh! I have uninstalled the hyper-v role, and several other windows features that I have no clue as to what they do... I have installed virtual-box 6, and docker-toolbox, which runs on a virtual-box vm. SO now I can have dockers running again without (most) incomprehensible fails. This not satisfying though. I had GPU pass-through, I had TensorFlow in containers, I had a process of booting into a hyper-v mode and booting into a virtual-box mode, and whilst not being convenient, it was a working solution. Now I don't - so here is the question: With the advent of WSL2, what will the ramifications be for SR-IOV, Discrete Device Assignment, legacy hardware and catastrophic (in the bad sense) forced updates. In other words; should I even bother even when forced too by your egregious updates...

What's wrong / what should be happening instead: The crux of the question is that you (MS) keep on breaking in unfixable ways that are incomprehensible without devoting too much of my precious time to working out what went wrong.

Is there a way I can safely and securely use WSL2? For anything? Is it worth me devoting any more time to it? to MS? You keep on ripping the rug out from under my feet, and no, I WILL NOT BE DRIVEN INTO THE CLOUD...

Ran course.

Ran course... ? great comment!


Update single-file.md

Summary
Note that single file deployment is not available on Windows 7
Fixes https://github.com/dotnet/sdk/issues/16499

ping @adegeo for review


Update to 0.18.10

This is a good version to update to, because the garbage collector is a lot stricter now. Tests pass too. An easy win.

i've noticed we are using as-loader instead of the official as/loader. I think it would be best if we support the official load than as-loader. I feel this will have better future support and optimizations then a fork, or a spoon for that matter.


kind restic and other tools benchmark report

Hello,
I'm currently doing benchmarks for deduplication backup tools, including restic.
I decided to write a script that would:

Install the backup programs
Prepare the source server
Prepare local targets / remote targets
Run backup and restore benchmarks
Use public available data (linux kernel sources as git repo) and checkout various git tags to simulate user changes in the dataset

The idea of the script would be to have reproductible results, the only changing factor being the machine specs & network link between sources and targets.
So far, I've run two sets of benchmarks, each done locally and remotely.
You can find the results at https://github.com/deajan/backup-bench
I'd love you to review the recipe I used for restic, and perhaps guide me on what parameters to use to get maximum performance.
Any remarks / ideas / PRs are welcome.
I've also made a comparaison table of some features of the backup solutions I'm benchmarking.
I still miss some informations for some of the backup programs.
Would you mind having a look at the comparaison table and fill the question marks related to the features of restic ?
Also, if restic has an interesting feature I didn't list, I'll be happy to extend the comparaison.
PS: I'm trying to be as unbiased as possible when it comes to those benchmarks, please forgive me if I didn't treat your program with the parameters it deserves.
Also, I've created the same issue in every git repo of the backup tools I'm testing, so every author / team / community member can judge / improve the instructions for better benchmarking.

Just some notes.

I'd love you to review the recipe I used for restic, and perhaps guide me on what parameters to use to get maximum performance.
Any remarks / ideas / PRs are welcome.

What do you understand as maximum performance? You can disable compression or enable maximum compression for smaller backupsizes, see https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html?highlight=max#compression

Also, if restic has an interesting feature I didn't list, I'll be happy to extend the comparaison.

Restic has its rest-server https://github.com/restic/rest-server and this has an append-only mode. With this you can make trojan safe backups, since backups can only be added but not removed. Very cool feature.
Some of your table
Are metadatas encrypted too ? Yes, see https://restic.readthedocs.io/en/latest/100_references.html#threat-model
Can encrypted / compressed data be guessed (CRIME/BREACH style attacks)? Restic is quite immune to watermark attacks https://restic.readthedocs.io/en/latest/100_references.html#backups-and-deduplication
Can a compromised client delete backups? Depends on the backend and how it is configured. With rest-server even automated backups are secure (see above)
Can a compromised client restore encrypted data? Depends if you store the password on the client or not
Can a repo be synchronized to another repo ? Yes, simplest solution is using rsync or using integrated mechanism, between different repositories https://restic.readthedocs.io/en/latest/045_working_with_repos.html#copying-snapshots-between-repositories

Thanks for your answers. I've updated my comparaison table.

What do you understand as maximum performance? You can disable compression or enable maximum compression for smaller backupsizes, see https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html?highlight=max#compression

I meant best overall settings. Indeed maxing out compression would reduce filesize. But I try to not compare apples to oranges, ie have identical compression across all backup programs.

Can a compromised client restore encrypted data? Depends if you store the password on the client or not

I am supposed to have the password on the client in order to add data to the encrypted repository, ain't it ?
Other backup software have mechanisms like a "backup only" password which does not allow to decrypt data.
How is this solved using restic ?
So far restic does best on remote restore operations, due to rest_server which has amazing speeds.
I was thinking there might be some settings to improve backup speed, like thread settings or something alike.

I meant best overall settings. Indeed maxing out compression would reduce filesize. But I try to not compare apples to oranges, ie have identical compression across all backup programs.

Best overall settings is very subjective I would say. I prefer best compression over speed, because the backup is meant for longterm storage.

I am supposed to have the password on the client in order to add data to the encrypted repository, ain't it ?
Other backup software have mechanisms like a "backup only" password which does not allow to decrypt data.
How is this solved using restic ?

Currently yes. However I think when your client is compromised you have a lot more problems than restoring data from the backup. However there is this feature request https://github.com/restic/restic/issues/187 that would support asymmetric encrytion. This is intended to enable deduplication between servers without the possibility to get access to the data of the other servers. But this would of course also fit your use case: Encryption key only to prevent a compromised client to restore old backups.
I was thinking there might be some settings to improve backup speed, like thread settings or something alike.
You could disable compression to increase backup speed when backup speed limit is due to CPU usage. However I would not recommend this. About one day ago there has been added a feature to tune settings to increase backup speed, especially for fast disks like SSD, see https://github.com/restic/restic/pull/2750

Regarding the comparison table:

"Redundant index copies": restic can rebuild the index from the pack files if necessary
"Continue restore on bad blocks in repository": Partially. in most cases yes, restore currently will fail if data is missing. Workaround for that are excluding the affected files during restore. Or using #2876 to remove the file from the snapshot.
"Data checksumming": I don't understand the answers. HMAC and reed-solomon serve totally different purposes. restic uses SHA256 hashes to verify that the data is as expected (plus an AEAD cipher to prevent tampering with the encrypted chunks).
"Supports backup XATTRs": yes
"Supports backup ACLs": only if these are mapped to XATTRs
"Supports sparse files": that PR is not yet included in a released restic version
"Does the backup software support pre/post execution hooks?": No, we consider scheduling to not be restic's responsibility
"Can a repo be synchronized to another repo ?": the copy command can transfer snapshots between repositories

Comments to the comparison:

For measurement durations in the single-digit second range it would be a good idea to use timestamps with a higher resolution. Otherwise the quantization can lead to a rather high measurement error.
The average file size in the qemu image backup looks odd: "17344875" kb. That rather looks like a file size in bytes, not kilobytes. The file count also seems to be wrong. For restic 0.14.0 I'd expect about 7000.
"Getting restic SFTP to work with a different SSH port made me roam restic forums and try various setups. Didn't succeed in getting RESTIC_REPOSITORY variable to work with that configuration.". Just add a matching entry to you ssh_config, and then use the corresponding hostname/alias. Or use the URL syntax: sftp://host:port/$path. If $path is absolute, then you'll have two consecutive slashes: sftp://host:port//path.

About one day ago there has been added a feature to tune settings to increase backup speed, especially for fast disks like SSD, see #2750

That might be interesting for SSD to SSD backups. Or when using 10Gbit Ethernet. In almost all other cases, I wouldn't expect that increasing the number of file readers makes any significant difference.
The available tuning options are listed at https://restic.readthedocs.io/en/latest/047_tuning_backup_parameters.html .

Further comments:

For the initial git benchmark it is a bit misleading to show the gzip/xz compressed size of the kernel tree including the .git directory, but afterwards excluding .git for the actual backups. The latter part also seems to be missing from the description. Actually, it would be rather interesting to see how small the repo could be.
To make the results more readable, file size in MB would be preferable.
How high is the latency between both systems? And what is the maximum bandwidth?

That might be interesting for SSD to SSD backups. Or when using 10Gbit Ethernet. In almost all other cases, I wouldn't expect that increasing the number of file readers makes any significant difference.

Depends. Some benchmark:
backend rest-server on encrypted spinning disk on Raspberry PI 4. Backend is limited in this configuration to about 300 - 400 MBit/s throughput.
Backupsystem: S-ATA SSD, Ryzen 5 Pro 5650GE (6 Cores, 12 Threads)
Env variables
set RESTIC_PACK_SIZE=32
set RESTIC_COMPRESSION=max
TL;DR
With default read concurreny it takes 201 seconds, with read-concurreny of 6 it takes 104 seconds. So speed / performance has almost doubled.
Feel free to move this benchmark to another thread.
Details:
With default read-concurrency of 2 adding a folder with 512 MP3s (2,52 GB) and 44 txt files (total 2,48 KB in size):
repository 0878d514 opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 3:21
snapshot 251eecde saved

CPU usage is about 40 to 50 %.
With read concurrency of 4
repository 802b8e7c opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 2:01
snapshot d05e3f9e saved

CPU usage is about 80 - 85 %
With read concurreny of 5:
repository 7091443a opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 1:50
snapshot 77411c83 saved

With read concurrency of 6
repository 26c58c17 opened (repository version 2) successfully, password is correct
created new cache in C:\Users\XXX\AppData\Local\restic
no parent snapshot found, will read all files

Files:         556 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 2.526 GiB (2.504 GiB stored)

processed 556 files, 2.526 GiB in 1:44
snapshot 81474633 saved

Hmm, you're right. The archiver pipeline can stall after reading a file while waiting for all blobs to be queued for uploading:
https://github.com/restic/restic/blob/7112a132c3c568497309ac034f1da62bdf4738a1/internal/archiver/file_saver.go#L199
Fixing that would require fixing #3869 and then tweaking the blob saving to complete the FutureBlob right after checking whether the blob has to be uploaded or already exists. Or alternatively rework the FileSaver part which waits for the FutureBlobs to be completely asynchronous.
The first variant is probably the easier way. Although I haven't thought much about the second alternative.

@JsBergbau Could you give PR #3955 a try? That should improve the archiver performance far enough that my initial assessment on the usefulness of --read-concurrency is correct again.

Closing as we've hopefully answered the questions.


Live action version?

Is there something like Anime4k but for live action videos like movies and TV shows? It works great on anime and other animated media but doesn't work so well for live action of course, so I'm wondering what other projects like Anime4k I should look for.

There's a few of them, like FSRCNNX and ravu, but they are based on older machine learning architectures/training methods that do not take in account video compression and other degradations. I don't know how effective it would be to train a restore shader for live action but it might be a future project.

I'm actually interested in perhaps doing this myself, any pointers I could look at for how you did it with anime? I assume if we get enough training examples, we could substitute the anime training data with live action data and see how it fares. That's just my assumption however, could be wrong.

You might also want to have a look at AviSynth AiUpscale.
I've just added it to my config and from my limited testing so far I get almost identical results as with FSRCNNX + adaptive sharpen (curve_height 0.5) with identical GPU usage afaict.
Regarding training data you should start off with very high quality source material and then generate training data from it in a variety of bitrates (and maybe even different codecs) at whatever scaling factors you want to support. That way it should be able to deal with compression artifacts.

@christoph-heinrich take a look at Nvidia Image Scaling, I've been using it in mpv (https://gist.github.com/agyild/7e8951915b2bf24526a9343d951db214) and it's a lot sharper than AviSynth AiUpscale. For being a non-machine-learning shader, it's actually very impressive.
I'll collect training data as you mentioned, that's a good idea to segment into various codecs. Is there a way to have a drop in replacement with Anime4k, ie is there a folder of training data that Anime4k was trained on where I can simply replace the previous training data with my live action one? Or is that not possible?

@satvikpendem Thanks for the tip. NVScale introduces some haloing, but AviSynth AiUpscale (medium sharp) makes compression artifacts more prominent, so it depends on the video which one ends up looking better. AiUpscale usually gets closer to what a better quality source would look like imo.

segment into various codecs

Presumably the same model should be able to handle different codecs as long as it gets trained with a variety of them, but as usual with neural networks, it's probably best to find what works through trial & error.

is there a folder of training data that Anime4k was trained on where I can simply replace the previous training data

I've never tried training Anime4k myself or looked at the source code, but since it has to take the training data from somewhere, you can probably just look at the code and replace the path.
Due to the increased complexity over lineart I'd assume you'll have to use a bigger network then Anime4k to get decent results. I also don't know how well the multi stage approach from Anime4k lends itself to non-animated content.


Activation fails on ABAP Cloud instance

When pulling the CLOUD branch into our ABAP cloud instance, the following errors and warnings occur:

The pull process is stuck now on object 56.
Kind Regards,
Steven

HI Steven,
the issue might be related to a language version problem of the authorization field artefact. Please open a ticket on component BC-SEC-AUT-PFC with the given details.
Thanks and best regards,
Marina

Names of any custom objects in SAP should be started with Z.
So, impossible to export from github to /DMO

Hi alexsei, if you are wokring on an ABAP environment instance on SAP BTP or S/4HANA Cloud, public edition you can directly download the repository with /DMO/ objects. If you are working on SAP S/4 HANA, on premise S/4 HANA Cloud, private edition, you have to set up the namespace first and then you can download the /DMO/ objects of this repository. See README:

Hi Marina,
We works in S/4HANA Cloud, public edition
Use the abapGit plug-in to import the ABAP Flight Reference Scenario by executing the following steps:

1. In your ABAP cloud project, create the ABAP package /DMO/FLIGHT (using the superpackage /DMO/SAP) as the target package for the demo content to be downloaded (leave the suggested values unchanged when following the steps in the package creation wizard).

"Namespace /DMO/ does not exist"
Besides that, there is the followng in README:

Hi aleksei,
thanks for the update. The restriction in the documentation is indeed wrong. The namespace /DMO/ should be available in S/4HANA Cloud, public edition systems.
Please open a ticket on component BC-SRV-NWD-XBR with your system details, so we can look into this.
Thanks and best regards, Marina

Hi Marina,
Could you please explain, in brief, how to open a ticket on this issue.
Thank you.
Best regards,
Aleksei

Hi alexsei,
you can create incidents on the support launchpad: https://launchpad.support.sap.com/
Best regards,
Marina

Hi Marina,
Field 'System' is locked on the incident page.

Best regards,
Aleksei

Hi alexsei, unfortunately I can't help you with that. Please refer to the support options for creating an incident here or get information in this video: https://www.youtube.com/watch?v=GtTTfB80Wb4&t=97s
Best regards,
Marina


Unable to bundle extension with ESBUILD

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.66.2
OS Version: Windows 10 / Windows 11 and OSX

Steps to Reproduce:

Follow these instructions https://code.visualstudio.com/api/working-with-extensions/bundling-extension#run-esbuild
The output shows the extension still isn't bundled

D:\Projects\VSCODE_Extensions\kodi-localize>vsce package
Executing prepublish script 'npm run vscode:prepublish'...

><EMAIL_ADDRESS>vscode:prepublish
> npm run esbuild-base -- --minify

><EMAIL_ADDRESS>esbuild-base
> esbuild ./src/extension.ts --bundle --outfile=out/main.js --external:vscode --format=cjs --platform=node "--minify"

  out\main.js  309.8kb

Done in 28ms
 WARNING  A 'repository' field is missing from the 'package.json' manifest file.
Do you want to continue? [y/N] y
This extension consists of 563 files, out of which 246 are JavaScript files. For performance reasons, you should bundle your extension: https://aka.ms/vscode-bundle-extension . You should also exclude unnecessary files by adding them to your .vscodeignore: https://aka.ms/vscode-vscodeignore
 DONE  Packaged: D:\Projects\VSCODE_Extensions\kodi-localize\kodi-localize-0.1.3.vsix (563 files, 1.25MB)

Referring to this issue #136831 the user noted:-

Solution turned out to be adding out/ and node_modules/ to .vscodeignore. Docs are just a little confusing in terms of what steps are for webpack and which steps are for esbuild.

After adding node_modules/  to my .vscodeignore my extension now bundles.

DONE  Packaged: D:\Projects\VSCODE_Extensions\kodi-localize\kodi-localize-0.1.3.vsix (11 files, 86.44KB)
However the issue is my extension no longer activates or runs, before bundling the extension works fine.

similar problem here with a selfmade zowe extension... (1.66.0, win 10). Also esbuild, if "node_modules/" is removed from vscodeignore file it is working like expected (und quite big)...

Any help please?

Hm, I'm having trouble reproducing what you're running into following the instructions in the docs. The .vscodeignore file that we recommend in the publishing step https://code.visualstudio.com/api/working-with-extensions/bundling-extension#publishing already include node_modules. When I run with that, I see that node_modules gets excluded as I would expect.
This section is linked to in the esbuild portion of the docs:

Finally, you will want to update your .vscodeignore file so that compiled files are included in the published extension. Check out the Publishing section for more details.

Are there additional repro steps I should use for this?

@connor4312 ended up here with excluding the needed modules with ! in the vscodeignore file
There are a lot of the them needed for the extension (can be seen when starting the extension with the developer tools)..., the needed modules are not added (i guess the esbuild plugin would be needed ).

@kraabrsg I don't think you're running into the same issue as the author. If you use esbuild, node_modules are not needed.

Thanks @connor4312 I was wondering the same

so if esbuild is used to bundle out to a single file guess vscodeignore file is not needed?
should the esbuild prepublish be followed with "vscode:package": "vsce package --no-dependencies"? I don't see the corresponding followup packaging command in either the docs or the linked example https://github.com/microsoft/vscode-test-adapter-converter/blob/main/package.json

Could this be related to esbuild's inability to process dynamic imports https://github.com/evanw/esbuild/issues/700 ? When I add  --log-level=debug option to esbuild I see a number of This call to "require" will not be bundled because the argument is not a string literal. Could this be why esbuild is not building the extension correctly? Is there a non-trivial vscode extension project out there that uses esbuild?

so if esbuild is used to bundle out to a single file guess vscodeignore file is not needed?

You still want the ignore file to omit your node_modules, sources, and so on.

Could this be why esbuild is not building the extension correctly?

Yes, as the message says, it won't bundle that statement.

Is there a non-trivial vscode extension project out there that uses esbuild?

Many of our new extensions use esbuild, e.g. the built-in notebook renderers extension

@kraabrsg I don't think you're running into the same issue as the author. If you use esbuild, node_modules are not needed.

@connor4312 well it is exactly the same case.
1.) adding all node_modules to the vscodeignore file
2.) when bundling the extension with ESBUILD it is working, but when loading the extension and investigating with the developer tools you see errors:
module 'xxx' not found
3.)excluding those modules one by one (vscodeignore) untill no error is displayed, then everything works. But that was quite a daunting task, there must be better ways to do this...

Is there a way to explicitly include a glob pattern e.g. node_modules with.vscodeignore ?
When vsce package --no-dependencies is specified, it doesn't include node_modules even without any .vscodeignore file. Some clarification on this would be much appreciated.

Is there a way to explicitly include a glob pattern e.g. node_modules with.vscodeignore ?

 This works the same as a .gitignore.

but when loading the extension and investigating with the developer tools you see errors: module 'xxx' not found

I don't have enough info what you posted to provide advice. Usually you'll get warnings or errors from your bundler if it can't resolve and import, such as what Mohan posted. Unless you import things in esoteric ways that the bundler doesn't know to look for. Otherwise, it's expected that you'll get a single static file (or set of files, depending on your configuration) that can be consumed without node_modules.

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


JSON returned is different in debug vs release mode

while the implementation worked perfectly in debug mode, i notice my code crashes in release versions and realized the returning JSON structure is quite different in these modes. these screenshots may explain somehow.
example loadInventory data in debug:

example loadInventory data in release:

i'm using react-native v 0.57
any idea why this happens?
as you may see the json keys are replaced by 'a','b','c',....

for anybody else that has a similar issue the reason appears to be using proguard in the release build.
adding this line to my proguard-rules.pro helped solve the problem:
-keep class com.contoriel.cafebazaar.** { *; }


[CU-26pmd15] Start TypeScript migration

Background
The website is built off of only JS and doesn't use any PropTypes and therefore has no type checking at all. I'd like to migrate this project to TypeScript to only get all the benefits of TS, but also as a way to demonstrate my TS knowledge and skills.
This PR starts the migration by:

configuring TypeScript in the project
updating the packages to include type definitions
updating linting rules and scripts
migrating gatsby-* config files to TS
migrating the <CurrentYear> component to TS to make sure everything is working.

Bonus:
This PR updates the /resume URL redirect from the old 2021 CV version to the current 2022 CV version
Changes

Adds tsconfig
Migrates gatsby-browser to TS
Migrates gatsby-config to TS
Migrates gatsby-node to TS
Updates the ESLint config for TS rules
Migrates the <CurrentYear> component to TS
Updates the /resume URL to point to the current 2022 CV

How to test

Browse the website as usual and make sure there are no unexpected errors

Task linked: CU-26pmd15 Add TypeScript


Compability warning during startup [BUG]

Module Version: v0.10.01
Describe the bug
There is a warning message after startup from this module.
To Reproduce
Just restart Foundry.
Screenshots
setup.js:145
[trigger-happy] The module "trigger-happy" contains compendium pack data which uses the deprecated "entity" field which must be migrated to "type"
Browser:
Google Chrome
Foundry Version:
Version 10 - Build 288
Game System:
DnD5e 2.0.3

Trigger happy is abbandoned in favor of MATT

I'll update the v10 release to fix the compatibility warning. But as mentioned the module is not being developed any further.


Error with custom image/title

When I try to add my image and description, am seeing this error. Does the input image need to be in any specific [format/resolution?]
Text(0.5, 1.0, 'Cosine similarity between text and image features')Error in callback <function install_repl_displayhook..post_execute at 0x7fd3eb1d1730> (for post_execute):
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/matplotlib/pyplot.py in post_execute()
107             def post_execute():
108                 if matplotlib.is_interactive():
--> 109                     draw_all()
110
111             # IPython >= 2
13 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/colors.py in call(self, value, clip)
1015             result.fill(0)   # Or should it be all masked?  Or 0.5?
1016         elif vmin > vmax:
-> 1017             raise ValueError("minvalue must be less than or equal to maxvalue")
1018         else:
1019             if clip:
ValueError: minvalue must be less than or equal to maxvalue
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/IPython/core/formatters.py in call(self, obj)
332                 pass
333             else:
--> 334                 return printer(obj)
335             # Finally look for special method names
336             method = get_real_method(obj, self.print_method)
14 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/colors.py in call(self, value, clip)
1015             result.fill(0)   # Or should it be all masked?  Or 0.5?
1016         elif vmin > vmax:
-> 1017             raise ValueError("minvalue must be less than or equal to maxvalue")
1018         else:
1019             if clip:
ValueError: minvalue must be less than or equal to maxvalue

Hi, I believe this happened while calling plt.imshow on the similarity matrix. (let me know if it wasn't)
I had only vmax=0.3 specified, and this error might have happened when the minimum value of your matrix was above 0.3.
In the latest commit (ebd0e35aac063855694a1e6d2f6b2181acb7af0b) I added vmin=0.1 to the imshow() call, which should prevent this error. Let me know if this fixes the problem - thanks!

Hi, I believe this happened while calling plt.imshow on the similarity matrix. (let me know if it wasn't)
I had only vmax=0.3 specified, and this error might have happened when the minimum value of your matrix was above 0.3.
In the latest commit (ebd0e35aac063855694a1e6d2f6b2181acb7af0b) I added vmin=0.1 to the imshow() call, which should prevent this error. Let me know if this fixes the problem - thanks!


Snapping to Common Distances

Feature Description
As and author I want to snap to common distances based on other elements on the page for a consistent look
Please see this video on slack. .mov is not allowed in Github.

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Snapping to Common Distances
Note: multi-selection is treated as a single element.

QA Instructions

You need 3 equal size elements
Position 2 next to each other
Take the third and try to position it with teh same distance between the first 2 elements.

Closing this. See PR #488 and #285

@samitron7 to add screenshots

@samitron7 I know this ticket is closed but you mentioned you added a screencast link here but I don't see it.

I have linked the slack message containing the .mov and added the screenshot to the issue.

Tracking https://github.com/daybrush/moveable/issues/200


BCDB Models migration

Currently when you change model schema you will get an error and you wont be able to load your old data because capnp schema changed without updating the data so you won't be able to de-serialize it using the old schema, the Idea here is to create a migration tool to handle schema changes and update old data.
Auto migration
auto migrations can be applied in certain cases without losing data like:
1- adding a new field to schema (with default value)
2- deleting an existing field
Custom migration
in more complex cases we can't automatically migrate data, in this case the user should be asked to provide a migration script:
1- changing a field type
2- adding new fields without default value
Migration scripts
migration scripts should be generated in both cases and the user is asked to write down his migration scripts in case of custom migration
Note: if the schema doesn't exist there will be no need for migration and the server will load the most recent one, otherwise it will start form the current version and start applying the migrations to reach the most recent version
questions:
1- should the migration tool be a separate tool that the user should run explicitly? (This will make sure that the user is aware of the data changes that will happen due to the migrations)
2- where should the migration scripts should be stored?
3- should these migration be applied only when the version is changed, or whenever the schema changed
suggested scenario:
1- the user changes schema of any bcdb model
2- when the user tries to access data from bcdb an error should be raised ("the schema has been changed you should migrate
3- then the user should run the migration command
4- if auto migration can be applied it will do it otherwise it will ask the user to implement the custom migration script

➤ Bola E.Nasr commented:
Auto Migrations
already implemented ( with the same url )
Custom migration
all we need change url to versions like ( @jumpscale.version.1` and<EMAIL_ADDRESS>and send it as parameter to new function and migrate ( and you can send in args the fields you need it to migrate )
should these migration be applied only when the version is changed

➤ Bola E.Nasr commented:
Custom migration
if we have 2 models have the same name we will overwrite otherwise we will append
Example:
base
@url = jumpscale.test.1
name* = "" (S)
test1 = "" (S)
test2 = "" (S)
data :
id = 1
name = "test"
test1=""
test2""
id = 2
name= "tst"
test1=""
test2=""second
@url = jumpscale.test.2
name* = "" (S)
test1 = "" (S)
test2 = "" (S)
test3 = "" (S)
data :
id = 1
name = "test"
test1="dmdm"
test2 = "test"
test3 = ""
id = 2
name= "tst_2"
test1=""
test2=""
test3 = ""After migration will be
@url = jumpscale.test.1
name* = "" (S)
test1 = "" (S)
test2 = "" (S)
test3 = "" (S)
data :
id = 1
name = "test"
test1="dmdm"
test2 = "test"
test3 = ""
id = 2
name= "tst"
test1=""
test2=""
test3 = ""
id = 3
name= "tst_2"
test1=""
test2=""
test3 = ""

➤ Bola E.Nasr commented:
still working on
j.data.bcdb.migrate("jumpscale.tfchain.client.1","jumpscale.tfchain.client", test2="test") but we have problem in schema id  can't get it anymore after migration
hopefully finish it today


Calling a method on a class for the first time on a secondary thread fails.

This seems to be an issue with caching not occurring properly, despite a ThreadGuard being held.
To workaround, call any method on a class from the main thread and all subsequent calls will work.

I started looking at this and I've discovered that this is inconsistent behavior between Android and vanilla java.  Curiously, I can write a test that only fails on Android but the identical test fails with vanilla Java.
Updates to follow (and if folks have seen this outside of Android I'm keen to know).

I believe this relates to the problem:
https://groups.google.com/g/android-ndk/c/eYL9tYkvefs
In fact, there are a couple threads posted which seem to be this specific Android issue.
I have a couple ideas I can try. I think if I cache the loader during the OnLoad call, it hopefully will work.  I need to make sure this wouldn't induce breaking changes elsewhere (and, I presume it would work, but the devil is in the details).

Possibly also this:
https://stackoverflow.com/questions/13263340/findclass-from-any-thread-in-android-jni

So, I've invested a non-zero amount of time trying to resolve this, and I think this cannot be resolved without breaking or confusing behaviour for end users .
The issue is that any JNI call is effectively bound to the loader of the class that initiated the call.  For Android, this means the application class loader, but when new threads are spun up, they do not have the same classloader, and fail to find the user defined class.
The only workaround is to call loadClass ahead of time, but JNI Bind doesn't know what class names are going to be used, so it actually can't.  I think I could go to complicated lengths to work around this (e.g. having some type of callback mechanism), but I think it would be pretty unexpected for an end user to have this happen (and could cause side effects).
I'll try add a syntax like so to workaround, but at the end of the day I think there's nothing I can do unfortunately.
JvmRef jvm_ref {vm};
jvm_ref.Warm<kClassUsedOnOtherThread>()

So, I continued to plug away at this, and it turns out the mistake was partly a cacheing mistake.  TL;DR: I managed to get a fallback class loader to work for classes that fail lookup.
The unfortunate part is that it requires some extra setup from the caller, however, it's fairly minor.  Basically the caller just has to provide a single instance of the host activity so that it can cache the application cache loader (you won't have to spell out each class type which would be easy to forget).

FYI I uploaded the Android test that I used to test this fix here.
Sadly, I haven't managed to get these tests running under Bazel, but it at least shows the correct flow.


Performance issue with bulkPut method

Hello, folks.
I have a performance issue with bulkPut method. I put about 30000 data into db and spent almost 10s to finish whole bulkPut operation. The figure shown below was recorded by devtool.

My data structure is shown below:

And here is the code I used to put data:
        try {
          const nextData = helper.clone(data);
          store = store.concat(data);
        if (data.length === 0) {
          context.db[TREE_SCHEMA].bulkPut(store);
        }
          observer.next(nextData);
        } catch (e) {
          observer.error({ code: 2017, detail: e });
        }

I found the idbstore.put method and req.onerror = errorHandler; spent a lot of time.

Please help me solving this problem. Did I miss something in document or error usage with Dexie? Or it's the limit with indexedDB....

The time spent on bulkPutting 30,000 objects depends on several things:

The number of indexes your objects have
How much nested objects you have in your objects (structured cloning algorithm will spend a lot more time cloning objects if they have nested structures)
How large data each object contains
Hardware (CPU / mem / disk)

It's a little of a surprise though that it spends time setting the value of req.onerror though! Are you using Angular and zoneJS ( it could intercept event setters which would make it slower).

Thanks for quick response!
I added the data structure figure in my post. Sorry for missing information.
It's surprised me too. Originally I thought indexedDB was free with thread and I don't need to care about the UI hangs issue by using indexedDB...  I used React and rxjs libraries. The compiling tool is webpack 4.x.

Seems objects don't have deep structures, which is good. How is the hardware (Mem + CPU mainly)? Do you have a lot of free RAM to play with when you run it? Did only the call to bulkPut() take 10 seconds for 30,000 items or was it the entire code that took that time?

..and which properties are you indexing?

You can use bulkAdd instead of bulkPut if you can make sure that incoming data does not exist in you DB.

export function getExistingValues(
table: DBCoreTable,
req: AddRequest | PutRequest | DeleteRequest,
effectiveKeys: Key[],
) {
return req.type === 'add'
? Promise.resolve(new Array(req.values.length))
: table.getMany({ trans: req.trans, keys: effectiveKeys });
}
@dfahlander  can Dexie just get primeKey in this function, why it need to get data, it costs time when bulkPut a large amout of data I think.


Fixed toObject to work for strings in IE8 and Rhino. Added test spec for forEach.

Fixes #90.

Looks good. o/

Looks good to me.

I've actually incorporated this fix into #96 which contains a better fix.


[Bug] Subscription with schema fails if there is an AUTO-CONSUME consumer attached to a topic

Search before asking

[X] I searched in the issues and found nothing similar.

Version
2.10.1, master
Minimal reproduce step

create an auto-consumer : bin/pulsar-client consume topic-a -s sub0 -n 0 -t Shared -st auto_consume
create a consumer on same topic, same sub this time with a schema

    @Test
    private void test() throws Exception {
        try (PulsarClient client = PulsarClient.builder()
            .serviceUrl("pulsar://localhost:6650")
            .build()) {
            try (Consumer<CustomObject> ignored = client.newConsumer(Schema.AVRO(CustomObject.class))
                .topic("topic-a")
                .subscriptionType(SubscriptionType.Shared)
                .subscriptionName("sub0")
                .subscribe()) {
            }
        }
    }

==> failure with IncompatibleSchemaException("Topic does not have schema to check")
stop the first consumer and start the test again ==> success

What did you expect to see?
Not failing
What did you see instead?
Failing
Anything else?
This happens because in PersistentTopic::addSchemaIfIdleOrCheckCompatible, we call checkSchemaCompatibleForConsumer if there are active consumers but no topic schema.
And checkSchemaCompatibleForConsumer fails if there's no schema.
Also addSchemaIfIdleOrCheckCompatible will not try to add the new consumer schema.
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Fixed by #17353

No it's not fixed by https://github.com/apache/pulsar/pull/17353.
Can we reopen ?

reopened

Fixed by #17449


Start Next.js implementation based on POC

Start of the Next.js implementation based on POC work which was done. This includes:

Introduction of the sitecore-jss-nextjs package - which is currently just a complete pass-through to sitecore-jss-react
nextjs sample template added with a minimum (working) implementation on Next.js v10

Not included here (which are in POC) are Experience Editor support, and GraphQL-based (e.g. XM Delivery) layout data consumption.
Note we've got mixed TypeScript / JS at the moment. This will be resolved (going all TS) in a subsequent task (along with proper linting setup).

@ambrauer I'm not sure, do we need to take almost all stuff from sitecore-jss-react and put it into sitecore-jss-nextjs?
Probably, you took same package.json, test configs (setup) from sitecore-jss-react. Maybe we can setup a minimal test configuration later?

@sc-illiakovalenko I think much will remain a pass-through to sitecore-jss-react, we'll eventually be introducing nextjs-specific components (e.g. Link, Image) that will then replace/extend those. And correct, I took the same test setup just to have a starting point. For this initial PR, the main goal of the sitecore-jss-nextjs package was to not have any references to sitecore-jss-react in the nextjs sample app.


Add types to redux map functions

For the mapStateToProps and mapDispatchToProps functions in day 2 parts 7-9, I think they might be a little easier to understand if the return types are specified. I also made some other minor updates to those functions.

I'm purposefully keeping types away just to make the demo code to be simpler. So I'm closing this one for now.


[Feature] cnf-testsuite generate_config should prompt to overwrite specified output-file YML config if the file exists already

Is your feature request related to a problem? Please describe.
When generating a cnf-testsuite.yml config file using generate_config option, it will not overwrite the existing file if it exists in the specified output-file= argument. This can lead to a new generate config file that is not valid for testing.
Describe the solution you'd like
cnf-testsuite should prompt if you use the same config-file to overwrite or keep, exit immediately if you do not want to overwrite so you know to use a different output file name for your config.

@agentpoyo can this issue be closed?


Added observed and expected frequencies to chi-square statistics

This is needed to log observed and expected frequencies.

Coverage remained the same at 92.208% when pulling fc586b06256820d05dbd3e94e757d5f443795e5d on chi_square_more_statistics into 4ab75fa463c7596361f53ccbd72438366748639f on master.


Tracebacks are very long

You can make it better by passing --tb=short to pytest, but nbsmoke should probably at least cut out all the nbconvert part of the traceback...

Closed by #13


Adding support for multipart content-type header.

Adding support to detect multipart content type. Based on https://en.wikipedia.org/wiki/MIME#Multipart_messages

Adding isValid property to contentType.
Adding isMultipart property to contentType.

Thanks, but I wish you'd opened an issue to discuss this design first. This doesn't seem like a good fit for this module to me based on the code, but I'm willing to discuss.

What is the use case for this?
How does this help jsdom?
Why can't you accomplish this using a library built on top of the parser, e.g. is-validish-multipart that takes a parsed content type?
Do you have specification references for these distinctions? Wikipedia does not count.
Why do you consider e.g. application/json an invalid MIME type? (per isValid())
Why do you consider multipart/mixed not multipart? (per isMultipart())
Why do you not validate boundary strings?


Loaded the content page only once

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- Ginger/Ginger/RunSetPageLib/RunsetOperationsPage.xaml.cs  1

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- Ginger/Ginger/RunSetPageLib/RunsetOperationsPage.xaml.cs  1

See the complete overview on Codacy


Use requestAnimationFrame for digest debounce

It may make sense to use requestAnimationFrame rather than setTimeout to debounce the digest. See discussion with @bahmutov in https://github.com/curran/reactive-model/issues/9#issuecomment-208004321.

This is to be exposed in reactive-model https://github.com/curran/reactive-model/issues/11