Many reads with "no alignment" in nanopolish phase-reads

Hi Jared,
I alluded to this a little while ago and finally got around to checking the output files.
I've run nanopolish independently on small chunks of the fastq because of speed issues - may not be necessary since your last update, but here's the concatenated log files.
[post-run summary] total reads: 17390, unparseable: 0, qc fail: 1, could not calibrate: 8, no alignment: 2508, bad fast5: 0
[post-run summary] total reads: 16622, unparseable: 0, qc fail: 1, could not calibrate: 1, no alignment: 1871, bad fast5: 0
[post-run summary] total reads: 17322, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 2036, bad fast5: 0
[post-run summary] total reads: 16483, unparseable: 0, qc fail: 2, could not calibrate: 1, no alignment: 1664, bad fast5: 0
[post-run summary] total reads: 16950, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 2238, bad fast5: 0
[post-run summary] total reads: 16606, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 4756, bad fast5: 0
[post-run summary] total reads: 17507, unparseable: 0, qc fail: 2, could not calibrate: 4, no alignment: 4713, bad fast5: 0
[post-run summary] total reads: 16906, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 3792, bad fast5: 0
[post-run summary] total reads: 18697, unparseable: 0, qc fail: 1, could not calibrate: 10, no alignment: 9149, bad fast5: 0
[post-run summary] total reads: 37632, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 36130, bad fast5: 0
[post-run summary] total reads: 17091, unparseable: 0, qc fail: 2, could not calibrate: 2, no alignment: 5448, bad fast5: 0
[post-run summary] total reads: 46413, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 45828, bad fast5: 0
[post-run summary] total reads: 37288, unparseable: 0, qc fail: 0, could not calibrate: 2, no alignment: 31405, bad fast5: 0
[post-run summary] total reads: 45334, unparseable: 0, qc fail: 0, could not calibrate: 2, no alignment: 44527, bad fast5: 0
[post-run summary] total reads: 34664, unparseable: 0, qc fail: 1, could not calibrate: 2, no alignment: 33380, bad fast5: 0
[post-run summary] total reads: 38680, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 38122, bad fast5: 0
[post-run summary] total reads: 45740, unparseable: 0, qc fail: 0, could not calibrate: 7, no alignment: 45356, bad fast5: 0
[post-run summary] total reads: 32654, unparseable: 0, qc fail: 0, could not calibrate: 9, no alignment: 26612, bad fast5: 0
[post-run summary] total reads: 43679, unparseable: 0, qc fail: 0, could not calibrate: 5, no alignment: 43181, bad fast5: 0
[post-run summary] total reads: 30299, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 22882, bad fast5: 0
[post-run summary] total reads: 17182, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 2919, bad fast5: 0
[post-run summary] total reads: 18034, unparseable: 0, qc fail: 1, could not calibrate: 1, no alignment: 9295, bad fast5: 0
[post-run summary] total reads: 17653, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 3554, bad fast5: 0
[post-run summary] total reads: 17248, unparseable: 0, qc fail: 3, could not calibrate: 4, no alignment: 2784, bad fast5: 0
[post-run summary] total reads: 17967, unparseable: 0, qc fail: 7, could not calibrate: 1, no alignment: 3927, bad fast5: 0
[post-run summary] total reads: 17127, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 3332, bad fast5: 0
[post-run summary] total reads: 18104, unparseable: 0, qc fail: 1, could not calibrate: 6, no alignment: 3903, bad fast5: 0
[post-run summary] total reads: 17721, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 7663, bad fast5: 0
[post-run summary] total reads: 18736, unparseable: 0, qc fail: 0, could not calibrate: 10, no alignment: 10177, bad fast5: 0
[post-run summary] total reads: 20527, unparseable: 0, qc fail: 0, could not calibrate: 10, no alignment: 14915, bad fast5: 0
[post-run summary] total reads: 37394, unparseable: 0, qc fail: 0, could not calibrate: 4, no alignment: 36416, bad fast5: 0
[post-run summary] total reads: 35629, unparseable: 0, qc fail: 0, could not calibrate: 4, no alignment: 29800, bad fast5: 0
[post-run summary] total reads: 19455, unparseable: 0, qc fail: 3, could not calibrate: 0, no alignment: 12641, bad fast5: 0
[post-run summary] total reads: 47682, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 47176, bad fast5: 0
[post-run summary] total reads: 47855, unparseable: 0, qc fail: 0, could not calibrate: 0, no alignment: 47299, bad fast5: 0
[post-run summary] total reads: 24629, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 14590, bad fast5: 0
[post-run summary] total reads: 16432, unparseable: 0, qc fail: 0, could not calibrate: 0, no alignment: 2339, bad fast5: 0
[post-run summary] total reads: 16943, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 3764, bad fast5: 0
[post-run summary] total reads: 16163, unparseable: 0, qc fail: 2, could not calibrate: 0, no alignment: 2477, bad fast5: 0

As you can see, there is a very large number of reads with no alignment - sometimes nearly all of the reads in a single fastq. Any idea what the cause of this could be?
Thanks,
Scott

Did you do any trimming/preprocessing of the reads, or are they straight
from Albacore?
Jared
On Mon, Mar 5, 2018 at 11:00 AM, Scott Gigante<EMAIL_ADDRESS>wrote:

Hi Jared,
I alluded to this a little while ago and finally got around to checking
the output files.
I've run nanopolish independently on small chunks of the fastq because of
speed issues - may not be necessary since your last update, but here's the
concatenated log files.
[post-run summary] total reads: 17390, unparseable: 0, qc fail: 1, could not calibrate: 8, no alignment: 2508, bad fast5: 0
[post-run summary] total reads: 16622, unparseable: 0, qc fail: 1, could not calibrate: 1, no alignment: 1871, bad fast5: 0
[post-run summary] total reads: 17322, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 2036, bad fast5: 0
[post-run summary] total reads: 16483, unparseable: 0, qc fail: 2, could not calibrate: 1, no alignment: 1664, bad fast5: 0
[post-run summary] total reads: 16950, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 2238, bad fast5: 0
[post-run summary] total reads: 16606, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 4756, bad fast5: 0
[post-run summary] total reads: 17507, unparseable: 0, qc fail: 2, could not calibrate: 4, no alignment: 4713, bad fast5: 0
[post-run summary] total reads: 16906, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 3792, bad fast5: 0
[post-run summary] total reads: 18697, unparseable: 0, qc fail: 1, could not calibrate: 10, no alignment: 9149, bad fast5: 0
[post-run summary] total reads: 37632, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 36130, bad fast5: 0
[post-run summary] total reads: 17091, unparseable: 0, qc fail: 2, could not calibrate: 2, no alignment: 5448, bad fast5: 0
[post-run summary] total reads: 46413, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 45828, bad fast5: 0
[post-run summary] total reads: 37288, unparseable: 0, qc fail: 0, could not calibrate: 2, no alignment: 31405, bad fast5: 0
[post-run summary] total reads: 45334, unparseable: 0, qc fail: 0, could not calibrate: 2, no alignment: 44527, bad fast5: 0
[post-run summary] total reads: 34664, unparseable: 0, qc fail: 1, could not calibrate: 2, no alignment: 33380, bad fast5: 0
[post-run summary] total reads: 38680, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 38122, bad fast5: 0
[post-run summary] total reads: 45740, unparseable: 0, qc fail: 0, could not calibrate: 7, no alignment: 45356, bad fast5: 0
[post-run summary] total reads: 32654, unparseable: 0, qc fail: 0, could not calibrate: 9, no alignment: 26612, bad fast5: 0
[post-run summary] total reads: 43679, unparseable: 0, qc fail: 0, could not calibrate: 5, no alignment: 43181, bad fast5: 0
[post-run summary] total reads: 30299, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 22882, bad fast5: 0
[post-run summary] total reads: 17182, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 2919, bad fast5: 0
[post-run summary] total reads: 18034, unparseable: 0, qc fail: 1, could not calibrate: 1, no alignment: 9295, bad fast5: 0
[post-run summary] total reads: 17653, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 3554, bad fast5: 0
[post-run summary] total reads: 17248, unparseable: 0, qc fail: 3, could not calibrate: 4, no alignment: 2784, bad fast5: 0
[post-run summary] total reads: 17967, unparseable: 0, qc fail: 7, could not calibrate: 1, no alignment: 3927, bad fast5: 0
[post-run summary] total reads: 17127, unparseable: 0, qc fail: 2, could not calibrate: 3, no alignment: 3332, bad fast5: 0
[post-run summary] total reads: 18104, unparseable: 0, qc fail: 1, could not calibrate: 6, no alignment: 3903, bad fast5: 0
[post-run summary] total reads: 17721, unparseable: 0, qc fail: 0, could not calibrate: 6, no alignment: 7663, bad fast5: 0
[post-run summary] total reads: 18736, unparseable: 0, qc fail: 0, could not calibrate: 10, no alignment: 10177, bad fast5: 0
[post-run summary] total reads: 20527, unparseable: 0, qc fail: 0, could not calibrate: 10, no alignment: 14915, bad fast5: 0
[post-run summary] total reads: 37394, unparseable: 0, qc fail: 0, could not calibrate: 4, no alignment: 36416, bad fast5: 0
[post-run summary] total reads: 35629, unparseable: 0, qc fail: 0, could not calibrate: 4, no alignment: 29800, bad fast5: 0
[post-run summary] total reads: 19455, unparseable: 0, qc fail: 3, could not calibrate: 0, no alignment: 12641, bad fast5: 0
[post-run summary] total reads: 47682, unparseable: 0, qc fail: 0, could not calibrate: 1, no alignment: 47176, bad fast5: 0
[post-run summary] total reads: 47855, unparseable: 0, qc fail: 0, could not calibrate: 0, no alignment: 47299, bad fast5: 0
[post-run summary] total reads: 24629, unparseable: 0, qc fail: 0, could not calibrate: 3, no alignment: 14590, bad fast5: 0
[post-run summary] total reads: 16432, unparseable: 0, qc fail: 0, could not calibrate: 0, no alignment: 2339, bad fast5: 0
[post-run summary] total reads: 16943, unparseable: 0, qc fail: 1, could not calibrate: 3, no alignment: 3764, bad fast5: 0
[post-run summary] total reads: 16163, unparseable: 0, qc fail: 2, could not calibrate: 0, no alignment: 2477, bad fast5: 0
As you can see, there is a very large number of reads with no alignment -
sometimes nearly all of the reads in a single fastq. Any idea what the
cause of this could be?
Thanks,
Scott
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/jts/nanopolish/issues/352, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXxnzRT8BDPYCTfTN53UHf-R7zDhVJWks5tbWETgaJpZM4SciIq
.

They are straight from albacore. You think these could all just be bad reads? Do you have a recommendation of which package to use for preprocessing?

I prefer not to preprocess unless there are a lot of adapters that need to be cleaned up.
Can you check whether the alignments of the reads look ok? Do you see a lot of low quality alignments (low map qual, only part of the read aligns)?

Here's some quick QC on the fastq that had 36k no alignment out of 37k reads.
MAPQ looks fine.

Read lengths are a little short.

The BAM shows lots of supplementaries:
37691 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 secondary
27691 + 0 supplementary
0 + 0 duplicates
37394 + 0 mapped (99.21%:-nan%)

Distribution across chromosomes looks fine, it's not all mapping to one region that happens to look like an adapter.

Any ideas?

Thanks for the QC. Nothing obvious comes to mind, could you send me some example reads?

Sent you a Google Drive invite with the fast5s and fastq.

Great, thanks. I'll try to look soon.
Jared
On Mon, Mar 5, 2018 at 3:03 PM, Scott Gigante<EMAIL_ADDRESS>wrote:

Sent you a Google Drive invite with the fast5s and fastq.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jts/nanopolish/issues/352#issuecomment-370546042, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAXxn4QwzNXAlm1EIOYVg_CqaJT-JxyWks5tbZohgaJpZM4SciIq
.

Thanks so much, I appreciate it.

Just checking in after our email discussion - is this still an issue or shall I close?

Thanks Jared. I think it’s safe to say this was a problem somewhere in the pipeline before nanopolish – your code was just the first with strict enough QC to raise a complaint about the data. Thanks again!

Ok no problem - let me know if you need any more help.


Additional code form was broken trying to get cleaned_data field not …

AdditionCode form was broken, this fixes it.
Needs a test... or really, we should test that POSTing to all forms doesn't cause an exception in the same was as we did for GETs.
Before:

Cheers @gabelton I really should borrow the code from your test to make sure this has coverage :)

Codecov Report

Merging #454 (0ced390) into master (6177dff) will increase coverage by 0.09%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #454      +/-   ##
==========================================
+ Coverage   91.74%   91.83%   +0.09%
==========================================
  Files         285      285
  Lines       17630    17646      +16
  Branches     1474     1475       +1
==========================================
+ Hits        16174    16205      +31
+ Misses       1171     1155      -16
- Partials      285      286       +1

Impacted Files
Coverage Δ

additional_codes/forms.py
91.46% <ø> (+17.07%)
:arrow_up:

additional_codes/tests/test_views.py
100.00% <100.00%> (ø)

common/forms.py
76.72% <0.00%> (+0.86%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6177dff...0ced390. Read the comment docs.


Sidebar does not toggle correctly with Semantic UI RTL build

Hi,
Using a sidebar with an RTL build, when it opens, it uses double the space on the right side and when it closes it leaves a visible area on the left side of screen and messes up the elements on the page.
I could not use the fiddle as instructed in guidelines because it is a ltr version and could not find a cdn for rtl version to setup another fiddle so I uploaded the files to my server for testing and reproducing the problem.
http://behnood.darjade.ir/sidebar.html
it is a ui inverted vertical menu sidebar that is toggle by a ui labeled icon button.
I removed all other html and css and js to make sure nothing is interfering with this.

I have the exact same issue with version 2.1.8

Probably not relevant by now, but you can just give the sidebar element direction:ltr whilst the rest of the body is rtl. It doesn't completely solve the issue, the sidebar doesn't react well wen set to right, but if its on the left it will work great.

Any updates? I have this issue on 2.2.6. The sidebar is not working properly when using RTL mode.

Hi, the original poster’s test case is unavailable, can someone please make a screenshot of the issue at least?

Closing the issue until there’s a test case.

This is the code I'm using:

    ראשי

   ...

When I type , it looks like this: (Which is fine) https://i.gyazo.com/6e9c2919303d90119808a12f0a43341e.mp4
But when I type , it starts to do something like this:
https://i.gyazo.com/30b01f2ca48c100614a841bbbf863f07.mp4
I think it has to do with this rule:
.ui.visible.right.sidebar ~ .fixed, .ui.visible.right.sidebar ~ .pusher {
    -webkit-transform: translate3d(260px, 0, 0);
    transform: translate3d(260px, 0, 0);
}

I tried to do
transform: translate3d(260px, 0, 0) translateX(-100%);

But it refuses to slide to the other side..

When I changed these lines on sidebar.js, it fixed the issue only on fullscreen:
from this:
distance.left = -width;
distance.right = width;

to this:
distance.left = width;
distance.right = -width;

When I try to see how it works on mobile (using chrome dev tools), it looks like chrome is stretching the document from the left to the right, and that's why we don't see the menu. Because the menu is in the right and we need to horizontally scroll to the right to see him. And when we do that, we're triggering a touchmove event which closes the menu while we try to scroll to the right.
I haven't found a good solution of how to fix this. I've tested it on FireFox and it looks like it works pretty well there.

tap


CSS update

This PR updates the CSS across the codebase. The majority of the changes consist of:
Form Structure
Better Code Readability
CSS best practices
Implementation
This will be the first part of CSS changes. Next tasks would be to reduce bloat and utilize more immutable classes from Basscss.

LGTM 👍 . @sumitarora ?

@danielschifano In the popup.html I am seeing extra underline.

@sumitarora fixed and updated.

🚢


use const properly, esp in function signatures

I wasn't diligent enough with my use of float* vs const float*. The former is used for outputs, the latter for inputs, to all functions. We'd like to refactor the function declarations to follow this paradigm, to be more defensive.

This one?
int check_tensor(float *a, float b, int n, const char label) { this one generates a couple of warnings
test_gpt2.cu(139): warning #2464-D: conversion from a string literal to "char *" is deprecated
test_gpt2.cu(141): warning #2464-D: conversion from a string literal to "char *" is deprecated
There's also these but wasn't sure if there's some incoming changes to use these variables?
test_gpt2.cu(56): warning #177-D: variable "maxT" was declared but never referenced
test_gpt2.cu(57): warning #177-D: variable "L" was declared but never referenced
test_gpt2.cu(95): warning C4101: 'losses': unreferenced local variable


Unable to add new key:val to nested object

myobject = new Map({a: {1: 1, 2: 2, 3: 3}})
patches = new List([new Map({op: 'add', path: '/a/4', value: 4})])
patch(myobject, patches).toJS()
>>Object {a: 4}

// expected: {a: {1: 1, 2: 2, 3: 3, 4: 4}}

Seems like a bug, there should definitely be a way to add a value to a sub-object without having to use replace.  That's the point of having a path after all right?
Thanks!  I'll work on a PR to fix it when I get some time after this weekend.

This looks like a bug when the key of the Map is of type number.

@pirate have you tried myobject = Immutable.fromJS({a: {1: 1, 2: 2, 3: 3}})  It seems you are creating a shallow Map with a nested native object.

That's right @pedroteixeira Just tested it here and it works
> myobject = Immutable.fromJS({a: {1: 1, 2: 2, 3: 3}})
=> Map { a: Map { 1: 1, 2: 2, 3: 3 } }
> patches = new Immutable.List([new Immutable.Map({op: 'add', path: '/a/4', value: 4})])
=> List [ Map { op: "add", path: "/a/4", value: 4 } ]
> patch(myobject, patches).toJS()
=> { a:
   { '1': 1,
     '2': 2,
     '3': 3,
     '4': 4 } }

@pirate using fromJS gives the expected output, since the value of a will be a Immutable Map as well (instead of a JS Object)

Thanks! You're totally right, I don't know why I didn't realize the nested object wasn't an immutable Map. Works like a charm!


docker-compose for dev

test with ghcr.io/devoxx4kids-npo/littil-backend:latest after https://github.com/Devoxx4Kids-NPO/littil-backend/pull/313 has been merged and built


Final project: Morgan Kindel and Julia Burke

Looks good. I strongly encourage you to use pre-existing tools to achieve all of these details. Your project will be creating a pipeline to run all of those tools. Snakemake or Prefect would be two tools that will help you create such a workflow.
See my comment here about working together.
See my comment here about balancing the project goals of ticking off details on the rubric etc.

@jnburke4

Can you guys just leave a comment here about how the merging/merge conflicts went. I'd like to get a sense of how much of time you lost to that.
Also your code is nicely modularized in that it is split into notebooks. If you can make functions out off the some of the code there some easy points there. And some docstrings and assertions (or even tests).

Can you guys just leave a comment here about how the merging/merge conflicts went. I'd like to get a sense of how much of time you lost to that.
Also your code is nicely modularized in that it is split into notebooks. If you can make functions out off the some of the code there some easy points there. And some docstrings and assertions (or even tests).

The merging was surprisingly extremely difficult, I couldn't say how much time we have spent sorting out where to put things but it was many hours. Ultimately, we decided since most of our work was here in this repository, we would post here. We merged some but this was working out better.
We split functions into tests as well, thanks for the suggestions!

Ok good to know thanks. Yea we didn't work through that. There are various strategies to make it easier. It is especially difficult with Jupyter notebooks though.

#superceded by #32

Excellent job. I appreciate the toils with trying to resolve git merge conflicts. I didn't know, otherwise I could have provided some strategies for improving the situation. With that said you still created a nicely organized, modular project, with lots of comments.
Areas for improvement for your future in coding:
References to pre-existing solutions could be better. You use opencv to solve your problem: mentioning this and the reasons for this decision would be useful for others considering reusing your code. Even if "my lab uses it" that's fine but demonstration of awareness of the context to your decision is important
Look at some of the other students' projects regarding testing: adding tests in the tests directory will make any projects far more scalable and maintainable. Especially in a collaborative setting, tests provide sanity to the development process, ensuring that changes introduced do not break previously working functionality.
Python modules should typically define functions and not run them or much other code (besides imports and some general variable declaration).
If you do wish to run the file as an executable too you can wrap that bit of code like this:
__name__ == '__main__':
    filecount()
    create_dict()
    create_template()
    match_images()

You have stored a few too many images in the repository. Ideally you would store this somewhere else. It quickly become unmanageable to work with git if you have such a large size on disk. Solutions include:

Make the data much smaller (often impractical)
storing the images separately on a webserver
storing the data in a git submodule
using git-annex or datalad

Regarding the "package":
In your setup.py file you are missing a comma after the url argument and you should include your dependencies in the install_requires argument (opencv-python, PIL etc)
Overall your attention to python style is good. A notable exception is lines of code that are too long. Moving forward keep an eye on the pep8 style guide to continue to grow in this respect
Well done.


InstrumentationBase vs BasePlugin And BasePlugin.unpatch lifecyle

Question 1
While writing plugin what is the difference between extending  InstrumentationBase and BasePlugin any difference.
Question 2
When exactly BasePlugin.unpatch is called and for what purpose, do we have any lifecycle ?

[x] This only affects the JavaScript OpenTelemetry library
[ ] This may affect other libraries, but I would like to get opinions here first

You can find more details about the difference here. You should write plugin using the InstrumentationBase now and to answer your second question, patch is called when we enable the plugin and unpatch when we disable it (so we stop generating new spans).

Thanks for letting me know, I have started writing a plugin (https://github.com/mnadeem/opentelemetry-plugin-mssql)  using the old mechanism, I will convert it into new mechanizm

I have converted plugin to instrumentation
However I see that, inspite of init method being called the patch method is not being called,

The plugin approach works like a charm, here is the test case.
Any suggestion would be greatly affected.

Hi @vmarchaud
I have written a blog on how to write plugin to auto instrument a library
https://reachmnadeem.wordpress.com/2021/02/22/opentelemetry-automatic-instrumentation-of-a-nodejs-library/
P.S : Appreciate if you could help me with above query.

@mnadeem You need to instanciate the MssqlInstrumentation and .enable() it before requiring mssql so it get patched. You can refer to other instrumentation tests

Thanks a bundle @vmarchaud, you are just awesome

BasePlugin is deprecated and will be removed as soon as all plugins will be converted to new instrumentation class. With regards to naming the patch function is not present in instrumentation. The instrumentation also allows you to patch more packages and individual files - this was not possible with plugin so this is huge and the biggest difference between those 2 classes. The instrumentation is enabled by default unless you call it with config option to disable it. The most important is to load all instrumentation before a real usage - not event importing / requiring it so it can be patched correctly. The instrumentation needs to be loaded also, you can do it by either creating a new instance and setting things manually or you can use registerInstrumentations where you pass the new instance of you instrumentation there. In our main repo all plugins have been converted already, so you might have a look for details how all of this looks like.

It's now long removed. Closing.


Added disk usage cli status

Added Disk usage percentage and disk space allocated to store artifacts.
Added check for available disk space
Fixed counting of nodes in status

@MitaliBo I am merging this for you.


README refers to a deprecated method for deployment

Suggested deployment method still works, but may not work for long. Would be better to suggest a  BiocManager  equivalent
> options(repos = BiocInstaller::biocinstallRepos())
Warning message:
'BiocInstaller' and 'biocLite()' are deprecated, use the 'BiocManager' CRAN
  package instead.

Might need to use: BiocManager::repositories()

This is issue is will likely be close with introduction of renv


Possible bug

Hi, I really like the package.  A good alternative to Vienna (which isn't on pypi).
But when verifing your example sequence output against Vienna's RNAfold, it looks like seqfold misses two pair bonds?  the lowercased g-c and a-u
         GGGAGGUCgUUACAUCUGGGUAAcaCCGGUACUGAUCCGGuGACCUCCC
RNAfold  (((((((((((((......)))))(((((.......))))))))))))) (-25.10)
seqfold  ((((((((.((((......))))..((((.......)))).)))))))) (-25.5)

Hey there and thanks for pointing this out!
I poked around and I think that it's from a diff in the underlying energy functions and penalties applied to multibranching/bifurcations:
$ seqfold GGGAGGTCGTTACATCTGGGTAACACCGGTACTGATCCGGTGACCTCCC --celcius 32 --dot-bracket --sub-structures
GGGAGGTCGTTACATCTGGGTAACACCGGTACTGATCCGGTGACCTCCC
((((((((.((((......))))..((((.......)))).))))))))
   i    j    ddg  description
   0   48   -1.9  STACK:GG/CC
   1   47   -1.9  STACK:GG/CC
   2   46   -1.4  STACK:GA/CT
   3   45   -1.4  STACK:AG/TC
   4   44   -1.9  STACK:GG/CC
   5   43   -1.6  STACK:GT/CA
   6   42   -1.4  STACK:TC/AG
   7   41   -0.5  BIFURCATION:4n/3h
   9   22   -1.1  STACK:TT/AA
  10   21   -0.7  STACK:TA/AT

At the BIFURCATION line it's saying that in its minimum free energy structure, it's seeing 4 free nucleotides as being the best choice. Working backwards, that came from this paper: https://pubmed.ncbi.nlm.nih.gov/15139820/
DNA_MULTIBRANCH: MultiBranch = (2.6, 0.2, 0.2, 2.0)
"""a, b, c, d in a linear multi-branch energy change function.

Inferred from:
Supplemental Material: Annu.Rev.Biophs.Biomol.Struct.33:415-40
doi: 10.1146/annurev.biophys.32.110601.141800
The Thermodynamics of DNA Structural Motifs
SantaLucia and Hicks, 2004
"""

In other words the energy functions and algo I adapted from those papers applies a greater penalty to bulge inbetween the branches than RNAFold:

I'm not saying that seqfold is right or that seqfold is wrong, it's just got a slightly diff approach, esp with regard to the energy penalties. I purposefully did not look at the RNAseq energies (or source code) to avoid running afoul of its license. This is all adapted from the public stuff. There's a chance seqfold is right and RNASeq is wrong... but in this particular case they're more similar that usual, I'd say
I hope this helps for context. I'm thinking it might help folks if I write up a pinned issue summarizing the list of differences that I'm aware, of which this is one

Also I appreciate this:

Hi, I really like the package. A good alternative to Vienna (which isn't on pypi).

I hope the package remains useful


[Royce] Update UG Format

Coverage remained the same at 79.091% when pulling 4396e3898d884349be3bacf27b8c5c283014e5af on royceljh:UG1.4 into 40113b9e8fc32bc7d9f350678b7c0286fad9c3ed on CS2103AUG2017-T11-B1:master.

Coverage remained the same at 79.091% when pulling 52140df4f0d871280429ee3b41ad88c1c468d125 on royceljh:UG1.4 into 40113b9e8fc32bc7d9f350678b7c0286fad9c3ed on CS2103AUG2017-T11-B1:master.

Coverage increased (+2.6%) to 81.719% when pulling 5e5e5f2e37472e3e96e8fb3b3714c8cf7649b3f1 on royceljh:UG1.4 into 40113b9e8fc32bc7d9f350678b7c0286fad9c3ed on CS2103AUG2017-T11-B1:master.

Coverage increased (+2.6%) to 81.719% when pulling 2e6a79868af87dca14c672e6af85d40e00395dc2 on royceljh:UG1.4 into 40113b9e8fc32bc7d9f350678b7c0286fad9c3ed on CS2103AUG2017-T11-B1:master.


Fix multi version removal of x-kubernetes-preserve-unknown-field

What this PR does / why we need it:
With adding v1beta1 the filter-crd broke causing invalid legacy CRDs. This adds the new field paths to remove in a multi version validation.
Release note:

NONE

/kind bug

/assign @munnerz

/test pull-cert-manager-e2e-v1-11
/test pull-cert-manager-e2e-v1-12
/test pull-cert-manager-e2e-v1-13
/test pull-cert-manager-e2e-v1-14
/test pull-cert-manager-e2e-v1-15
/test pull-cert-manager-e2e-v1-16
/test pull-cert-manager-e2e-v1-17

@wallrj e2e takes care of that, it was due to an outage in the alerting system of the Kubernetes testgrid that we were unaware of this happening.


Extend WellLog to offer an optional acquisition context

Finished criteria:
This issue is finished when the update is presented to the OSDU forum.
What is the scope of this issue?
Show how to import existing version of Welllog and model the extension in AKM and generate a CSV-file for import in OSDU.
Why
To implement a way to update OSDU Schema giving a better overview over the change and also better visualization of the details.
How

[ ] Run a test prototype of the whole procedure from import of existing to export of CSV.
[ ] Agree on best practice for content and layout of the modelviews suggested. Se ex.: https://github.com/octo-org/octo-repo/issues/45
[ ] Implement changes and enhancement necessary or wanted  in AKMM
[ ] Demo

Who
@dagkarlsen , @SnorreFossland
When
March 2024
Milestone  ....

Hei Snorre,
Here is my suggestion for a demo use case:
Goal: We need to extend WellLog to offer an optional acquisition context.
Prerequisites:

  There is an independent modeling exercise to define a WellLogAcquisition object. Its main purpose is to hold all details about a well log acquisition, which produces raw WellLog instances in potentially multiple runs, which contain multiple passes. (current proposal WellLogAcquisition.1.0.0 (1)).

  WellLogAcquisition.1.0.0 (1) is treated as ‘black box’, it is just there defined as a master-data group-type entity.

  WellLog.1.5.0 you already have, the next minor version, which should be extended – this is the basis or equivalent for the AKM import ().

In AKM

  You ‘somehow’ create a representation for the master-data WellLogAcquisition (without the detail inside)

  You create an abstract fragment AbstractLogAcquisitionRunPass with version 1.0.0. (The happy outcome is attached with the same name). But you create the prop-links for (the descriptions are in the attached proposal spreadsheet)

a.       WellLogAcquisitionID (string) – this is the relationship to WellLogAcquisition (aka foreign key) – I made this a mandatory property (JSON schema: required)
b.       RunIdentifier (string)
c.       PassIdentifier (string)

  The meta model is used to declare the WellLogAcquisitionID property as ‘relationship’ to master-data WellLogAcquisition.

   You may not need to ‘show’ all the details, which we are only adding new elements.

a.       You add a new property to WellLog ‘AcquisitionDetail’, type nested object by reference, referring to an abstract group-type, the name AbstractLogAcquisitionRunPass and the version 1.0.0 (we need to uniquely identify the semantic version number we include/use as composition).
Now you can explain the change by creating views (let AKM shine). Eventually, the approval could be done in AKM and you could fill the “State” column Y with Accepted (for the demo).
Now the CSV ‘copy by value’ starts:

  Create a new abstract proposal for AbstractLogAcquisitionRunPass.1.0.0.xlsx – I attached a AbstractTemplate.1.0.0.xlsx for that purpose. It would have to be renamed as the name has to match the main type name.

  Open the WellLog.1.5.0 and add the one new row (or if you are brave and can recreate the entire content, the entire main 1.5.0 sheet.

Hilsen,
Thomas

Modelview with overvew of the types infolved

WellLogAcquisition and AbstractLogAcquisitionRunPass

Snorre,
Jeg skal se mer på OSDU product and organization modelling i morgen.
Frank


Horozontal chart squeezed

I am trying to use Chartist togheter with Knockout and i cannot figure out what is happening. The data from knockout is correct, and all the labels and the series are there. The problem comes when it is rendering:

Everything is 1px high, labels and everything. I cannot reproduce this problem in a Fiddle either.
The only thing I can imagine can be the problem is that the div is not visible as first, but loads a few seconds after, to give the user a better experience (have a spinner while loading).
Do you know about this?

We need code to help you out. Please re-open if still a problem.


[CC-27289] Update SDK to latest CC API spec

The immediate motivation for the upgrade is to leverage the new cloud-neutral private endpoint connections API in the Terraform provider. Steps to generate patch in this PR:
$ make fetch-latest-spec
$ make
$ git tag v1.8.0

Thanks Ryan and Chris !!
I saw some questionable things (unrelated to this PR) in the generated SDK, like AddIPAllowlist2. Will open a Jira to track those separately.


get_debug_queries should return the bind key along with the Queries

The get_debug_queries does not provide the bind the query was executed on.
When I have multiple database connections it can be difficult to debug and test the queries.

Closing, matching PR was also created.


[2.9.x] macros 2.6.2 (was 2.5.9)

About this PR
📦 Updates com.softwaremill.macwire:macros from 2.5.9 to 2.6.2
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.softwaremill.macwire", artifactId = "macros" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.softwaremill.macwire", artifactId = "macros" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #771.


Update application-proxy-configure-custom-domain.md

We must make the article clear for non-Azure DNS customers.

@githubarpyka : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@githubarpyka : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@Court72 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@kenwith
Thanks for the review! I've incorporated your edit. IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#assign:@MicrosoftDocs/public-repo-pr-review-team

The following users don't have valid GitHub IDs, or are not collaborators on this repo: microsoftdocs/public-repo-pr-review-team

#sign-off

@githubarpyka - Hi Arpad - it is easier to make these changes in private repo so it can be merged with less review. Thanks.


Update Lodestar flags for v1.0.0

Updated flags to be compatible with v1.0.0 updates. For updates on all flag changes, please see: https://github.com/chainsafe/lodestar/releases/tag/v1.0.0

Thanks for fixing this! Merging.


[sticky] Don't Work

Since updating the foundation version from 6.5.3 to 6.6.0, this is the only problem I had.
The Sticky PLUGIN stopped working.
On the documentation page, it doesn't work either. Which can also give a hint.
https://get.foundation/sites/docs/sticky.html

Foundation version(s) used: Foundation v6.6.0
Browser(s) name and version(s): Chrome v78.0
Device, Operating System and version: MacOS High Sierra

Hi @actycrea,
thanks for reporting this. There is already an issue about this and this is fixed in the develop branch.
See https://github.com/foundation/foundation-sites/issues/11928 and https://github.com/foundation/foundation-sites/pull/11925
Next time please also use our issue template and check if there is an issue about this.


feat(planning_evaluator): add a trigger to choice whether to output metrics to log folder

Description
Add a bool parameter output_metrics, if it's true, the planning_evaluator node writes the statics of the metrics measured during its lifetime to <ros2_logging_directory>/autoware_metrics/<node_name>-<time_stamp>.json when shut down.
This feature will be used to collect and analyze test results in real vehicle tests and evaluator tests.
Related links
Parent Issue:

Link

How was this PR tested?
Psim with output_metrics=true:
$ cat /home/temkeikem/.ros/log/autoware_metrics/autoware_planning_evaluator-2024-11-26-19-00-31.json
{
    "acceleration/count": 285,
    "acceleration/description": "Trajectory_acceleration[m/s²]",
    "acceleration/max": 1.0012520551681519,
    "acceleration/mean": 0.234304107391435,
    "acceleration/min": -0.9976022839546204,
    # ignore many outputs.
    "yaw_deviation/count": 285,
    "yaw_deviation/description": "Yaw_deviation[rad]",
    "yaw_deviation/max": 0.02092383717239943,
    "yaw_deviation/mean": -9.644752318082534e-06,
    "yaw_deviation/min": -0.02153717392870902
}

Notes for reviewers
None.
Interface changes
None.

Effects on system behavior
None.

ping @kyoichi-sugahara @maxime-clem  :pray:


Do not render DevTools during SSR

This PR makes the DevTools client-side-only to avoid hydration issues (see https://github.com/tannerlinsley/react-query/issues/1774).
It renders ReactQueryDevtools as null during SSR, and then renders normally after mount.
I'm experiencing a failed test on my machine, even on the master branch without any change of mine:
Expected: "[\"query-2\"]"
    Received: "[\"query-1\"]"

      324 |     fireEvent.change(sortSelect, { target: { value: 'Last Updated' } })
      325 |     queries = await screen.findAllByText(/\["query-[1-3]"\]/)
    > 326 |     expect(queries[0]?.textContent).toEqual(query2Hash)
          |                                     ^
      327 |     expect(queries[1]?.textContent).toEqual(query3Hash)
      328 |     expect(queries[2]?.textContent).toEqual(query1Hash)
      329 |

      at src/devtools/tests/devtools.test.tsx:326:37

With my changes, the failed test is different:
Expected: "[\"query-2\"]"
    Received: "[\"query-3\"]"

      324 |     fireEvent.change(sortSelect, { target: { value: 'Last Updated' } })
      325 |     queries = await screen.findAllByText(/\["query-[1-3]"\]/)
    > 326 |     expect(queries[0]?.textContent).toEqual(query2Hash)
          |                                     ^
      327 |     expect(queries[1]?.textContent).toEqual(query3Hash)
      328 |     expect(queries[2]?.textContent).toEqual(query1Hash)
      329 |

      at src/devtools/tests/devtools.test.tsx:326:37

Maybe @prateek3255 would know what's going on?

I'm experiencing a failed test on my machine, even on the master branch without any change of mine:

hmm, it works fine on CI 🤷

:tada: This PR is included in version 3.21.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Math Magicians: setup

Use Create React App (CRA) to set up the project.
Make sure the application is running without any errors.

Thanks for the review @Ghiftee !


ArtifactStore implementation for CosmosDB

This PR provides a CosmosDBArtifactStore implementation to enable using CosmosDB for storing subjects, whisks and activation in Azure CosmosDB
This is currently work in progress and opening this early so as to get feedback on design progress
Build Results
This PR requires test which need account keys to access CosmosDB. Untill it gets merged to master following build runs are being done in my fork with keys configured

Branch artifact-store-cosmos (backing branch of this PR) - Runs the CouchDBArtifactStoreTests based on test suite introduced with #3517

Branch artifact-store-cosmos-ci - Runs the complete OpenWhisk suite with ArtifactStoreProvider set to CosmosDBArtifactStoreProvider. Some test here are being ignored. See below for more details

For test coverage see here
Description
Currently OpenWhisk supports CouchDB for storing various entities like actions, rules, subjects, activations etc. This PR provides a CosmosDB based implementation for ArtifactStore SPI
Usage of CosmosDB Java SDK
CosmosDB supports various modes to connect to it. For our usage we have two options

REST API
Async Java SDK
Sync Java SDK

Compared to CouchDB performing queries against CosmosDB requires client side computation which involves sending queries to each partition and then collect and merge the result set. The Async Java SDK takes care of all these interactions and provides a simplified reactive api based on RxJava
Given the complexity involved in performing various operations against CosmosDB this PR uses the Java SDK to simplify and speed up the implementation
Design Considerations
Data Model
Below is the json representation of whisk.system/utils/echo action saved in CouchDB and CosmosDB

CouchDB
CosmosDB

{
  "_id": "whisk.system/utils/echo",
  "_rev": "1-975e42c7be63291bf0ff2500e24caaae",
  "name": "echo",
  "publish": false,
  "annotations": [
    {
      "key": "exec",
      "value": "nodejs:6"
    },
    {
      "key": "parameters",
      "value": [
        {
          "description": "Any JSON entity",
          "name": "payload",
          "required": false
        }
      ]
    }
  ],
  "version": "0.0.1",
  "updated":<PHONE_NUMBER>521,
  "entityType": "action",
  "exec": {
    "kind": "nodejs:6",
    "code": "code",
    "binary": false
  },
  "parameters": [],
  "limits": {
    "timeout": 60000,
    "memory": 256,
    "logs": 10
  },
  "namespace": "whisk.system/utils"
}

{
  "id": "whisk.system|utils|echo",
  "name": "echo",
  "publish": false,
  "annotations": [
    {
      "value": "nodejs:6",
      "key": "exec"
    },
    {
      "value": [
        {
          "name": "payload",
          "description": "Any JSON entity",
          "required": false
        }
      ],
      "key": "parameters"
    }
  ],
  "version": "0.0.1",
  "updated":<PHONE_NUMBER>877,
  "entityType": "action",
  "exec": {
    "code": "code",
    "kind": "nodejs:6",
    "binary": false
  },
  "parameters": [],
  "limits": {
    "memory": 256,
    "logs": 10,
    "timeout": 60000
  },
  "namespace": "whisk.system/utils",
  "_c": {
    "rootns": "whisk.system"
  },
  "_self": "dbs/MZFtAA==/colls/MZFtAO2-Ig4=/docs/MZFtAO2-Ig4cAAAAAAAAAA==/",
  "_rid": "MZFtAO2-Ig4cAAAAAAAAAA==",
  "_etag": "\"09004181-0000-0000-0000-5adc75090000\"",
  "_attachments": "attachments/",
  "_ts":<PHONE_NUMBER>
}

Below are some keys aspects on how data is stored in CosmosDB
id
The id field in CosmosDB is referred as id (in CouchDB its _id). Per Resource Id docs

The following characters are restricted and cannot be used in the Id property: '/', '', '?', '#'

As / cannot be used CosmosDBArtifactStore would replace / with |. So whisk.system/utils/echo is stored as whisk.system|utils|echo
Per OpenWhisk Naming Restrictions

The subsequent characters can be alphanumeric, spaces, or any of the following: _, @, ., -

So no other char needs this treatment
Revision Handling
CosmosDB supports optimistic concurrency control (OCC) through HTTP entity tags, or ETags. So for all practical purpose _rev is mapped to _etag field of CosmosDB and all semantics around concurrent updates are supported via that.
Computed Fields
CosmosDBArtifactStore uses a subdocument under _c to store some computed fields which are referred in queries. Such fields are computed by CouchDB view logic and here these are set at time of inserting/updating the docs via ArtifactStore API.
So any logic which directly manipulates the objects need to also support these fields
Limits
Following are some of the limits enforced by CosmosDB. Currently OpenWhisk does not enforce any restriction on some of these so there is a change in behaviour

Max document size is 2 MB
The id must not exceed 255 characters

Partitioning Strategy
One of the key aspect of using CosmosDB is to determine the partitioning key. This can be decided on per collection basis
For now this PR uses /id itself also as the partitioning key. This has following pros and cons
Cons

All the queries would need to be sent to all partitions. This would lead to higher RU consumption and possibly higher latency in query responses

Note that currently query in CouchDB also hits all partitions and thus performance decreases when number of partitions are large. There are some discussions to support querying for specific partitions. That would make it similar to CosmosDB.
Pros
From production usage perspective OpenWhisk does not perform any query in critical path of activation. There are calls to DB are based on specific id lookups. This enables that read load would be distributed across actual partitions and thus would lead to fair utilization of resource units.
If we use namespace as partition key then it can lead to situation where if number of activity related to that namespace increase considerably then it may become hot and thus exhause the resource quota. Requests to the same partition key can't exceed the provisioned throughput allocated to a partition and will be rate-limited.
Given that queries are mostly performed during authoring workflow and increase in latency in query responses may be ok
Attachment Storage
As part of this PR the attachments are being stored in CosmosDB itself using its attachment support. This is similar to how attachments are supported in CouchDB. This enables storing upto 10GB of attachments.
Going forward once AttachmentStore SPI PR gets merged this would be change to use the SPI with default implementation using CosmosDB attachment support with an option to use say S3/Azure BlobStore based SPI implementation
Query Execution
CosmosDBArtifactStore maps the CouchDB views to CosmosDB Queries. Following are details around how the views are mapped
Generic Whisks View

CosmosDB
CouchDB

SELECT
    {
      "name":r['name'],
      "publish": r['publish'],
      "annotations": r['annotations'],
      "version": r['version'],
      "id": r['id'],
      "updated": r['updated'],
      "exec": {
        "binary": r['exec']['binary']
      },
      "limits": r['limits'],
      "namespace": r['namespace']
    }
FROM   root r
WHERE  r.entitytype = @entityType
       AND ( r.namespace = @namespace
              OR r._c.rootns = @namespace )
ORDER  BY r.updated DESC

function view(doc) {
    var PATHSEP = "/";
    var isAction = function (doc) { return (doc.exec !== undefined) };
    if (isAction(doc)) try {
        var ns = doc.namespace.split(PATHSEP);
        var root = ns[0];
        var value = {
            namespace: doc.namespace,
            name: doc.name,
            version: doc.version,
            publish: doc.publish,
            annotations: doc.annotations,
            limits: doc.limits,
            exec: { binary: doc.exec.binary || false},
            updated: doc.updated
        };
        emit([doc.namespace, doc.updated], value);
        if (root !== doc.namespace) {
            emit([root, doc.updated], value);
        }
    } catch (e) {}
}

Above example shows how a whisks.v2.1.0/actions view is mapped to query. Few things to note here

Property names in select clause use Quoted Property Accessor as some of the property names like end is a reserved word.
So CosmosDBUtil would use the mode by default for all fields selected
Projecting JSON - Select clause attempts to return a minimum superset of field names as specified in view javascript.
This enables computing some  fields like exec on server side via DocumentHandler.transformViewResult

Similar sql is generated for following views

whisks.v2.1.0-actions
whisks.v2.1.0-packages
whisks.v2.1.0-rules
whisks.v2.1.0-triggers

For above view the query is performed by matching namespace in full r.namespace or on root namespace r._c.rootns and r.entityType set to required type
Public Packages Query
For public package there is a check for binding

CosmosDB
CouchDB

SELECT
    {
      "name": r['name'],
      "publish": r['publish'],
      "annotations": r['annotations'],
      "version": r['version'],
      "id": r['id'],
      "updated": r['updated'],
      "namespace": r['namespace']
    } AS view
FROM     root r
WHERE    r.publish = true
AND      (
                  NOT IS_OBJECT(r.binding)
         OR       r.binding = {})
AND      r.entitytype = @entityType
AND      (
                  r.namespace = @namespace
         OR       r._c.rootns = @namespace)
AND      r.updated >= @since
ORDER BY r.updated DESC

function view(doc) {
  var isPublicPackage = function(doc) {
    return Object.keys(doc.binding).length == 0 and doc.publish;
  }
  if (isPublicPackage(doc)) try {
    var value = {
      namespace: doc.namespace,
      name: doc.name,
      version: doc.version,
      publish: doc.publish,
      annotations: doc.annotations,
      updated: doc.updated,
      binding: false
    };
    emit([doc.namespace, doc.updated], value);
  } catch (e) {}
}

Activations Query
For whisks-filters.v2.1.0/activations view the query is made on r._c.nspath which is a computed field based view logic

CosmosDB
CouchDB

SELECT
{
  "name": r['name'],
  "activationId": r['activationId'],
  "publish": r['publish'],
  "annotations": r['annotations'],
  "version": r['version'],
  response: {
    "statusCode": r['response']['statusCode']
  },
  "id": r['id'],
  "cause": r['cause'],
  "end": r['end'],
  "start": r['start'],
  "namespace": r['namespace']
} AS VIEW
FROM   root r
WHERE  r._c.nspath = @nsvalue
ORDER  BY r.start DESC

function (doc) {
  var PATHSEP = "/";
  var isActivation = function (doc) { return (doc.activationId !== undefined) };
  var summarize = function (doc) {
    var endtime = doc.end !== 0 ? doc.end : undefined;
    return {
        namespace: doc.namespace,
        name: doc.name,
        version: doc.version,
        publish: doc.publish,
        annotations: doc.annotations,
        activationId: doc.activationId,
        start: doc.start,
        end: endtime,
        duration: endtime !== undefined ? endtime - doc.start : undefined,
        cause: doc.cause,
        statusCode: (endtime !== undefined AND doc.response !== undefined AND doc.response.statusCode !== undefined) ? doc.response.statusCode : undefined
      }
  };
var pathFilter = function(doc) {
for (i = 0; i < doc.annotations.length; i++) {
var a = doc.annotations[i];
if (a.key == "path") try {
var p = a.value.split(PATHSEP);
if (p.length == 3) {
return p1 + PATHSEP + doc.name;
} else return doc.name;
} catch (e) {
return doc.name;
}
}
return doc.name;
}
if (isActivation(doc)) try {
var value = summarize(doc)
emit([doc.namespace+PATHSEP+pathFilter(doc), doc.start], value);
} catch (e) {}
}

Subject Query
subjects/identities
At time of authentication subjects/identities view is used. For that below query is invoked which makes use of INNER JOIN to query on namespaces sub document

CosmosDB
CouchDB

SELECT r AS VIEW
FROM   root r
JOIN   n IN r.namespaces
WHERE
    (NOT(is_defined(r.blocked))  OR r.blocked = falses)
AND (
     (r.uuid = @uuid AND r.KEY = @key)
       OR (n.uuid = @uuid AND n.KEY = @key)
    )

function (doc) {
  if(doc.uuid AND doc.key AND!doc.blocked) {
    var v = {namespace: doc.subject, uuid: doc.uuid, key: doc.key};
    emit([doc.subject], v);
    emit([doc.uuid, doc.key], v);
  }
  if(doc.namespaces AND !doc.blocked) {
    doc.namespaces.forEach(function(namespace) {
      var v = {_id: namespace.name + '/limits', namespace: namespace.name, uuid: namespace.uuid, key: namespace.key};
      emit([namespace.name], v);
      emit([namespace.uuid, namespace.key], v);
    });
  }
}

namespaceThrottlings/blockedNamespaces
This view is used by NamespaceBlacklist logic to peridically query for blacklisted namespace

CosmosDB
CouchDB

SELECT r AS VIEW
FROM   root r
WHERE  r.blocked = true
        OR r.concurrentinvocations = 0
        OR r.invocationsperminute = 0

function (doc) {
  if (doc._id.indexOf("/limits") >= 0) {
    if (doc.concurrentInvocations === 0 || doc.invocationsPerMinute === 0) {
      var namespace = doc._id.replace("/limits", "");
      emit(namespace, 1);
    }
  } else if (doc.subject && doc.namespaces && doc.blocked) {
    doc.namespaces.forEach(function(namespace) {
      emit(namespace.name, 1);
    });
  }
}

Shutdown Handling
All CosmosDBArtifactStore instances (3 in total for whisks, activations and subjects) share the same AsyncDocumentClient instance. So to handle the shutdown call properly this PR uses a ReferenceCounted approach where all instance share same reference and once the reference count reaches zero the client instance is closed
Client side skipping
ArtifactStore.query API supports skip to support paginated query over large dataset. For CouchDB the query is handled by one of the shards which in turn queries all the other shards and then merges the result on server side (i.e. scatter and gather)
For CosmosDB the query results are collected from various partitions and merged by the sdk. Thus  skip is implemented by dropping the query results. Due to this going to later pages in a query result would be slow.
Batching not supported
ArtifactStoreProvider.makeStore exposes a batching flag to enable batch insert mode for activations. CosmosDB does not have a bulk insert api so this flag is currently ignored.
Azure docs suggest using stored procedures to achive bulk insert support. However that requires all document meant to be inserted to be part of same partition. With our usage of id field as partition key this would not help much
Testing Approach
Some of the tests from this PR need to access the CosmosDB service and thus require the account credentials. These credentials would be provided by environment variables. Based on discussion on dev following appraoch is taken.

Tests would always run on travis runs for apache/incubator-openwhisk master
For other runs

If the travis has required env variables configured tests would run
If required environment variables are not found then they would be skipped
On my fork the credentials are configured and thus tests would be executed

Pending Work

[x] Attachment Support
[x] Query around subject and namespaces
[x] Indexes for various collections
[x] CI Integration

Third Party Dependencies
This PR pulls in following major dependencies

com.microsoft.azure:azure-cosmosdb:1.0.0

io.reactivex:rxnetty:0.4.20
org.json:json:20140107

io.reactivex:rxjava-reactive-streams:1.2.1
io.reactivex:rxscala_2.11:0.26.5

Dependency on org.json
org.json dependency library license is marked as Category-X and hence such a library cannot be used as a dependency in Apache Project. For this Azure/azure-cosmosdb-java#29 has been opened and is being addressed

Trying out
To try out the CosmosDB integration locally (for now only tests) follow steps below

Create a free server from Try CosmosDB and select SQL API
From within portal look for Azure Cosmos DB account and go to Keys section
Copy the URI and PRIMARY KEY
Specify following environment variables

COSMOSDB_KEY - Use the PRIMARY KEY value here
COSMOSDB_ENDPOINT - Use the URI value here
COSMOSDB_NAME - Set it to openwhisk

With this it would be possible to run CosmosDBArtifactStoreTests locally.
Once #3689 is done it would be possible to run complete OpenWhisk setup using CosmosDB via ansible setup scripts
Related issue and scope
My changes affect the following components

[x] Data stores (e.g., CouchDB)
[X] Tests
[x] Deployment
[X] Documentation

Types of changes

[x] Enhancement or new feature (adds new functionality).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[x] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Test suite run result
In a separate branch travis is configured to run whole OW test suite as being run currently for master with default ArtifactStoreProvider switched to CosmosDBArtifactStoreProvider. In such a run some test got failed. Below is the analysis for them

WskRestEntitlementTests - These test make use of wskadmin to provision a custom user. Currently wskadmin does not support CosmosDB. Hence for now test being ignored
ActionLimitsTests - Some of the test here got failed due to Azure/azure-cosmosdb-java#33. PR raised for that. For now being ignored
SequenceMigrationTests - These are passing now

Makes sense to update wskadmin to also handle momdodb/cosmosdb ?

Makes sense to update wskadmin to also handle momdodb/cosmosdb ?

Yes this aspect was discussed sometime back here. Need to figure out right way to abstract that in python and refactor wskadmin. Would do that in a separate PR

Indexing Policy
CosmosDB provides rich indexing options which support

Indexing mode - consistent or lazy
Hash Index - For equality check
Range Index - For range check and order by
Control Precision - Allows to make trade-offs between index storage overhead and query performance
Boolean properties are indexed by default - As per create document api Booleans and nulls are automatically indexed (See dataType)

Currently the view based indexes in CouchDB are eventually consistent and all the test around queries account for that. Based on usage following indexes are configured
Subjects

/uuid/? - Authentication query
/subject/?
/namespaces/[]/uuid/? - Authentication query
/namespaces/[]/name/?
/concurrentInvocations/? - For blacklist check query
/invocationsPerMinute/? - For blacklist check query

Key points

Indexing mode is set to consistent
All indexes have precision set to -1 for now. Later based on metrics we can determine if this needs to be tweaked
key is not being indexed - Authentication query is performed on uuid and key. Of this index is only created for uuid and key check should be taken care by filtering in query evaluation. As uuid is very much unique number of rows scanned for filtering should be small
concurrentInvocations and invocationsPerMinute are used in "namespace name/limits" documents which are then used to find out blacklisted namespace if either of them is zero. We can possibly remove this by setting blocked to true even for limits docs

Whisks

/entityType/? - Hash
/namespace/? - Hash
/_c/rootns/? - Hash
/updated/? - Range

Key points

Indexing mode set to consistent

Activations

/namespace/?
/_c/nspath/?
/start/?

Key points

Indexing mode set to lazy - For activations the rate of writes would be much higher compared to those on whisks and subjects. So here it would be better to keep indexing mode lazy

Index policy updates
Current logic upon system start would query for existing indexes configured on the collection. If there is a mismatch between what is expected and what is currently found it would provision the new indexes.
CosmosDB currently shows a strange behavior where it is creating extra indexes for same path. For e.g. /namespace/? is configured with hash index. But post creation if indexes are checked its found that it has both Range and Hash index configured. To account for these extra indexes in index check this PR uses a custom IndexingPolicy type which accounts for this type of behaviour while check if indexes are same.
CosmosDB supports online index policy changes where by change in index definition would be performed asynchronously and online allowing the collection to remain available for writes while transformation is in progress. However if the application makes query relying on new indexes then those queries would fail untill indexing completes based on new index definition.
So in future if any change in index definition is done then it should be provisioned prior to deploying new code relying on that index. Current approach of checking for index definition would ensure that index definitions are consistent

This seems good to me, would like some others to chime in. My questions are mainly around:

client connection sharing across ArtifactStores (can we just create separate connections?)
multiple stores currently getting created (is there any risk?)

On this last one, I think it is a defect of WhiskStore that multiple calls to WhiskEntityStore.datastore() will return multiple unique instances - I think these should return the same instance (invoke makeStore[WhiskEntity]() once ever). I don't know if it impacts cosmos (client is shared with multiple separate instances of the same store?), but if so, maybe we can fix it here? otherwise can fix it separately.

On this last one, I think it is a defect of WhiskStore that multiple calls to WhiskEntityStore.datastore() will return multiple unique instances

One benefit of this approach is that future test using MemoryArtifactStore can work without any side effect of other tests. Caching the result of call of makeStore would make shutdown handling tricky. Currently each eventual owner of store instance is responsible for shutdown. Probably in that case we need to have a shutdown support at provider level

I think adding ArtifactStoreProvider.shutdown() seems like a fine addition, if calling client.close() multiple times is not reliable. The tests will need to know that some artifact stores have client bound to provider, and others have client bound to store, which I think is not great, but it is OK given that nothing is invoking shutdown currently, and the behavior is different amongst the stores currently.

@tysonnorris I agree current ref count based shutdown handling is not desirable and we should relook into how ArtifactStore are closed. May be we deal with this in a separate issue and use existing approach in this PR for now?

I am working on a separate PR which would add new stages to run full test suite with Cosmos. That would allow complete test coverage against this new store. Would raise it once this PR is merged

Needs a PG run

PG with commit hash 7eb9d23b8e8940b3630f98519f4cefa194983e42 is 🔵

Thanks a lot @rabbah and @tysonnorris for reviewing this big PR ✨. Opened #3872 to track documentation related work


Ability to specify custom board properties, or have the flashing process disregard 'incorrect' productIds

We're rolling our own hardware which has specs similar to the leonardo, so when flashing with the Arduino IDE we'll select that option. However we have a different productId to what's expected by the library, as specified in boards.js. Obviously with this set 'incorrectly' we'll hit Error: could not reconnect after resetting board. since the productId doesn't match what's expected.
It'd be great if we could pass an object when creating the Avrgirl instance that specifies the board properties (to pass our modified productId), or alternatively have the search process just check that it can connect to the comName specified in the 'port' property of the options object.
Or if the problem is that boards 'jump' comName's, given that we're selecting one we can search for it later with the productId, etc that it presented to us upon first list?

Hi @Mike-Dax!
Thanks for raising this issue. I totally see where you're coming from, and why things are breaking. Thanks fir the thorough explanation, it was really helpful! I think allowing a custom board object would be the most flexible way going forward to resolve this.
This is a reasonable use case, so I published a new version which allows for a custom board object to be passed in, in place of the string board name.
I haven't had time to document it yet, but there is an example in a demo file you can look at for getting started with custom boards. That Arduino micro example is a leonardo board, so your object contents would look somewhat similar.
Let me know if this works for you! 😁

closing this for now, comment if this wasn't resolved for you 😃


Rework how we do report handling

This reworks how we handle reports entirely and adds a top level Report object that we can load, validate and upgrade reports with. By default it will always validate the incoming report, upgrade it to the latest report version then validate it again. Right now we only have 1 version but this is going to change soon. Having this kind of pipelines upgrades will be handy for keeping backways compatibility is needed. It also allows us to deprecate older version of reports with a nice interface/error messaging. The ReportBuilder should always be built to output the latest report format.

I'm not making this bump the version yet as I want to try it out with the action and see if there's any other changes needed. So far it look promising

Ok this works fully with the action code (at least the unit tests). Once this is merged I can do a full test and hammer out any issues.


Pr issue 57087 - fix for relating (e.g. satisfies) function overloads to intersection specifications

Usage Scenario:
Suppose an intersection type:
interface C {
  (x:1):"1";
  (x:2):"20";
  (x:number):number | "1" | "20";
};
interface B {
  (x:2):"2"
  (x:3):"30"
  (x:number):number | "2" | "30";
};
interface A {
  (x:3):"3"
  (x:1):"10"
  (x:number):number | "3" | "10";
};

Current behavior
Without this pull, in order to write an implementation, the only reliable error-free overload implementation and declaration requires concatenating all the declarations of the intersection type as follows:
function foo2(x:1):"1";
function foo2(x:2):"20";
function foo2(x:number):number;
function foo2(x:2):"2"
function foo2(x:3):"30"
function foo2(x:number):number;
function foo2(x:3):"3"
function foo2(x:1):"10"
function foo2(x:number):number;
function foo2(x:number){
  if (x===1) return "1";
  if (x===2) return "2";
  if (x===3) return "3";
  // (*) These runtime-unused extra lines need to be added
  // to make the implementation compatible with the overload signature
  if (x===1) return "10";
  if (x===2) return "20";
  if (x===3) return "30";
  return x;
}

That has disadvantages:

The large number of overloads in the declaration imposes a workload on the ts compiler downstream.
It is not user friendly.
The compiler also requires runtime-unused extra lines need to be added to make the implementation compatible with the overload signature.

New Behavior
With this pull the following declaration and implementation are allowed instead:
function foo1(x:1):"1";
function foo1(x:2):"2";
function foo1(x:3):"3";
function foo1(x:number):number;
function foo1(x:number){
  if (x===1) return "1";
  if (x===2) return "2";
  if (x===3) return "3";
  return x;
}

Implementation
Where
The function checkOverloadsRelatedToIntersection is called only when

(target.flags & TypeFlags.Intersection) and
typeRelatedToEachType(source,target) returned Ternary.False
the number of source signatures is >1.

Therefore, it only affect cases which were previously rejected.
Algorithm
Algorithm used within checkOverloadsRelatedToIntersection:
Let s[i], i in 0...Ns-1 be the source overloads.

Let t[j], j in 0...Nt-1 be the target intersection member types.

Let t[j][k], k in 0...Ntj-1 be the target overloads in type t[j].

SourceDomainMatch(i):==
    For some j,k exists IdentitiyRelation(Parameters<s[i]>,Parameters<t[j][k]>)===true

SourceRangeMatch(i):==
    Let rt be the union of ReturnTypes<t[j][k]> for all j,k s.t. Parameters<s[i]> assignable to Parameters<t[j][k]>
    ReturnTypes<s[i]> assignable to rt

TargetDomainMatch(t[j,k]):==
    For some i exists IdentitiyRelation(Parameters<s[i]>,Parameters<t[j][k]>)===true

Passing conditions:
- For all i, SourceDomainMatch(i)===true
- For all i, SourceRangeMatch(i)===true
- For all j,k, TargetDomainMatch(t[j][k])===true

SourceDomainMatch checks that source domain maps into the target domain.
SourceRangeMatch checks that  source range maps into the target range.
TargetDomainMatch checks that target domain maps into the source domain
Diff URL

Fixes #57112

@typescript-bot test this
@typescript-bot test top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot run dt
@typescript-bot perf test this
@typescript-bot pack this

@RyanCavanaugh  - I guess this means I need to download the repos calcom/cal.com and/or backstate/backstage to confirm/recreate the problem - is that correct?

Expand out the comment a few times and you'll get exact instructions on how to reproduce each.

@RyanCavanaugh  @jakebailey
Re: the cal.com report

After setting up according to the instructions the code in .../cal.com/node_modules/typescript/lib/tsserver.js does not include the changes in this pull #57395
However, the reported error can be reliably reproduced using tsreplay as described in the instructions.
Changing the hardcoded value in tsreplay giving the memory size for tsserver from 4096 to 8192 the error reliably disappears.

This is the output from ps when --max-old-space-size=
4096, failing with final Exited unexpectedly with signal SIGABRT
node /workspaces/ts+dt/node_modules/.bin/tsreplay ./cal.com ./RepoResults7/calcom.cal.com.replay.txt cal.com/node_modules/typescript/lib/tsserver.js
usr/local/bin/node --max-old-space-size=4096 --expose-gc /workspaces/ts+dt/cal.com/node_modules/typescript/lib/tsserver.js --disableAutomaticTypingAcquisition

8192, passing with final 999999  exit
node /workspaces/ts+dt/node_modules/.bin/tsreplay ./cal.com ./RepoResults7/calcom.cal.com.replay.txt cal.com/node_modules/typescript/lib/tsserver.js
usr/local/bin/node --max-old-space-size=8192 --expose-gc /workspaces/ts+dt/cal.com/node_modules/typescript/lib/tsserver.js --disableAutomaticTypingAcquisition

In the non-failing case, looking at system memory usage, it rises nearly (but not completely) monotonically with time to rise by 4.4 GiB.
I'll follow up by checking the other report too.

Re: The backstage error report:
(1) The instruction specified path path/to/tsserver.js has some ambiguity.
There are 4 tsserver.js files in the project:
./node_modules/typescript/lib/tsserver.js
<EMAIL_ADDRESS>./microsite/node_modules/typescript/lib/tsserver.js
./storybook/node_modules/typescript/lib/tsserver.js

Of those,
<EMAIL_ADDRESS>./storybook/node_modules/typescript/lib/tsserver.js

differ from ./node_modules/typescript/lib/tsserver.js, while
./microsite/node_modules/typescript/lib/tsserver.js

is the same.
I have assumed that
./node_modules/typescript/lib/tsserver.js

is the one to use.
(2) However, none of those tsserver.js files contain the text checkOverloadsRelatedToIntersection which would be there if pull request 57395 had been merged.
(3) Just as in my previous post, increasing tsreplay memory from 4GB to 8GB prevented the error from happening.
(4) Differently from the report in the previous post, there were some ups and down in memory usage (not a monotonic increase) before the error.

@typescript-bot test top200

@jakebailey - I am grateful and confused.  I will look further into the SIGABRT examples.

Didn't realize this wasn't accepted; will look if the feature is okay'd.

This test case doesn't gets failed by the original code and doesn't get passed to the new function in the pull.:
interface A {
    f(x:"a",y:string):1;
    f(x:"b",y:string):1;
    f(x:"c",y:string):1;
    f(x:"d",y:string):1;
    f(x:"e",y:string):1;
    f(x:"f",y:string):1;
};

interface B {
    f(x:"a",y:string):2;
    f(x:"b",y:string):2;
    f(x:"c",y:string):2;
    f(x:"d",y:string):2;
    f(x:"e",y:string):2;
    f(x:"f",y:string):2;
};
(0 as any as A) satisfies A&B;

So I have to check this before proceeding further.


Running Backtest with custom ohlcv data

Dear Catalyst Maintainers,
Thank you for your work on an amazing platform!
Before I tell you about my issue, let me describe my environment:
Environment
I'm running mac osx, and installed everything with combination of pip and conda in a custom conda environment. catalyst was installe with pip. here's what my environment looks like
name: sdac
channels:

defaults
dependencies:
appnope=0.1.0=py36hf537a9a_0
bleach=2.1.2=py36_0
ca-certificates=2017.08.26=ha1e5d58_0
certifi=2018.1.18=py36_0
dbus=1.12.2=h5243cc1_1
decorator=4.2.1=py36_0
entrypoints=0.2.3=py36hd81d71f_2
expat=2.2.5=hb8e80ba_0
gettext=<IP_ADDRESS>=h15daf44_3
glib=2.53.6=h33f6a65_2
html5lib=1.0.1=py36h2f9c1c0_0
icu=58.2=h4b95b61_1
ipykernel=4.8.0=py36_0
ipython=6.2.1=py36h3dda519_1
ipython_genutils=0.2.0=py36h241746c_0
ipywidgets=7.1.1=py36_0
jedi=0.11.1=py36_0
jinja2=2.10=py36hd36f9c5_0
jpeg=9b=he5867d9_2
jsonschema=2.6.0=py36hb385e00_0
jupyter=1.0.0=py36_4
jupyter_client=5.2.2=py36_0
jupyter_console=5.2.0=py36hccf5b1c_1
jupyter_core=4.4.0=py36h79cf704_0
libcxx=4.0.1=h579ed51_0
libcxxabi=4.0.1=hebd6815_0
libedit=3.1=hb4e282d_0
libffi=3.2.1=h475c297_4
libiconv=1.15=hdd342a3_7
libpng=1.6.34=he12f830_0
libsodium=1.0.15=hd9e47c5_0
markupsafe=1.0=py36h3a1e703_1
mistune=0.8.3=py36_0
nbconvert=5.3.1=py36h810822e_0
nbformat=4.4.0=py36h827af21_0
ncurses=6.0=hd04f020_2
notebook=5.4.0=py36_0
openssl=1.0.2n=hdbc3d79_0
pandoc=<IP_ADDRESS>=ha5e8f32_1
pandocfilters=1.4.2=py36h3b0b094_1
parso=0.1.1=py36hc90e01c_0
pcre=8.41=hfb6ab37_1
pexpect=4.3.1=py36_0
pickleshare=0.7.4=py36hf512f8e_0
pip=9.0.1=py36h1555ced_4
prompt_toolkit=1.0.15=py36haeda067_0
ptyprocess=0.5.2=py36he6521c3_0
pygments=2.2.0=py36h240cd3f_0
pyqt=5.6.0=py36he5c6137_6
python=3.6.4=hc167b69_1
python-dateutil=2.6.1=py36h86d2abb_1
pyzmq=16.0.3=py36he48b5ad_0
qt=5.6.2=h9975529_14
qtconsole=4.3.1=py36hd96c0ff_0
readline=7.0=hc1231fa_4
send2trash=1.4.2=py36_0
setuptools=38.4.0=py36_0
simplegeneric=0.8.1=py36_2
sip=4.18.1=py36h2824476_2
six=1.11.0=py36h0e22d5e_1
sqlite=3.22.0=h3efe00b_0
terminado=0.8.1=py36_1
testpath=0.3.1=py36h625a49b_0
tk=8.6.7=h35a86e2_3
tornado=4.5.3=py36_0
traitlets=4.3.2=py36h65bd3ce_0
wcwidth=0.1.7=py36h8c6ec74_0
webencodings=0.5.1=py36h3b9701d_1
wheel=0.30.0=py36h5eb2c71_1
widgetsnbextension=3.1.0=py36_0
xz=5.2.3=h0278029_2
zeromq=4.2.2=ha360ad0_2
zlib=1.2.11=hf3cbc9b_2
pip:

aiodns==1.1.1
aiohttp==3.0.1
alembic==0.9.8
async-timeout==2.0.0
attrdict==2.0.0
attrs==17.4.0
bcolz==0.12.1
boto3==1.5.32
botocore==1.8.46
bottleneck==1.2.1
cchardet==2.1.1
ccxt==1.10.1166
chardet==3.0.4
click==6.7
contextlib2==0.5.5
cycler==0.10.0
cyordereddict==1.0.0
cython==0.27.3
cytoolz==0.9.0
docutils==0.14
empyrical==0.2.1
enigma-catalyst==0.5.3
eth-abi==1.0.0b0
eth-account==0.1.0a2
eth-hash==0.1.0a3
eth-keyfile==0.5.1
eth-keys==0.2.0b1
eth-rlp==0.1.0a2
eth-utils==1.0.0b2
ffn==0.3.3
fire==0.1.2
future==0.16.0
gitdb2==2.0.3
gitpython==2.1.8
hexbytes==0.1.0b0
idna==2.6
idna-ssl==1.0.0
intervaltree==2.1.0
jmespath==0.9.3
logbook==1.2.1
lru-dict==1.1.6
lxml==4.1.1
mako==1.0.7
matplotlib==2.1.2
multidict==4.1.0
multipledispatch==0.4.9
networkx==2.1
numexpr==2.6.4
numpy==1.14.0
pandas==0.19.2
pandas-datareader==0.6.0
patsy==0.5.0
pluggy==0.6.0
py==1.5.2
pycares==2.3.0
pycryptodome==3.4.11
pyfolio==0.8.0
pyparsing==2.2.0
pysha3==1.0.2
pytest==3.4.0
python-editor==1.0.3
pytz==2018.3
redo==1.6
requests==2.18.4
requests-file==1.4.3
requests-ftp==0.3.1
requests-toolbelt==0.8.0
retrying==1.3.3
rlp==0.6.0
s3transfer==0.1.13
scikit-learn==0.19.1
scipy==1.0.0
seaborn==0.8.1
smmap2==2.0.3
sortedcontainers==1.5.9
sqlalchemy==1.2.3
statsmodels==0.8.0
stopit==1.1.2
tables==3.4.2
tabulate==0.8.2
toolz==0.9.0
urllib3==1.22
web3==4.0.0b9
wrapt==1.10.11
yarl==1.1.0
zipline==1.1.1
prefix: /Users/tdrobbin/anaconda/envs/sdac

Now that you know a little about me, let me tell you about the issue I am
having:
Description of Issue

What did you expect to happen?
I'm looking to run a backtest passing in custom ohlcv data for a number of coins across time currently in the form of a panel. The panel looks as follows:

I tried running the backtest using this data by passing it to run_algorithm via the data param as follows:
# %%catalyst --start 2016-1-1 --end 2018-1-1 --capital-base 100000 -x bitfinex -c usd
from catalyst import run_algorithm

from catalyst.finance.slippage import FixedSlippage
from catalyst.finance.commission import PerDollar, PerTrade

from catalyst.api import (
    order_target_value,
    order_target_percent,
    symbol,
    record,
    cancel_order,
    get_open_orders,
    order,
    set_max_leverage,
    set_long_only,
    history,
    get_datetime,
    set_commission,
    set_slippage,
    set_cancel_policy,
    EODCancel
)

def initialize(context):
    set_max_leverage(1)
    set_long_only()
    set_cancel_policy(EODCancel())

    set_commission(PerTrade(cost=0))
    set_slippage(spread=0.00)

#    context.asset = symbol('btc_usd')
    context.asset = symbol('BTC')

    context.i = 0

def handle_data(context, data):
    if context.i == 0:
        order_target_percent(context.asset, .99)

    record(btc = data.current(context.asset, 'price'))

    context.i += 1

data = sda.zipline.get_zipline_ready_bar_data()

bt_options = dict(
    capital_base=10000,
    start=pd.Timestamp('2016-01-01', tz='UTC'),
    end=pd.Timestamp('2018-01-01', tz='UTC'),
    handle_data=handle_data,
    initialize=initialize,
    data=data,
#     exchange_name='bitfinex',
#     base_currency='usd'
)

df = run_algorithm(**bt_options)

I recieved an error that i must pass an exchange_name, so uncommented out the portion above. so my bt_options looks as follows:
bt_options = dict(
    capital_base=10000,
    start=pd.Timestamp('2016-01-01', tz='UTC'),
    end=pd.Timestamp('2018-01-01', tz='UTC'),
    handle_data=handle_data,
    initialize=initialize,
    data=data,
    exchange_name='bitfinex',
    base_currency='usd'
)

df = run_algorithm(**bt_options)

What happened instead?

I recieved the following traceback:
[2018-02-23 02:03:55.081223] WARNING: run_algo: Catalyst is currently in ALPHA. It is going through rapid development and it is subject to errors. Please use carefully. We encourage you to report any issue on GitHub: https://github.com/enigmampc/catalyst/issues
[2018-02-23 02:03:58.086530] INFO: run_algo: running algo in backtest mode
[2018-02-23 02:03:58.234683] WARNING: Loader: Refusing to download new treasury data because a download succeeded at 2018-02-21 04:59:27+00:00.
[2018-02-23 02:03:58.237140] INFO: exchange_algorithm: initialized trading algorithm in backtest mode
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-26-aabf0a13351d> in <module>()
     60 )
     61
---> 62 df = run_algorithm(**bt_options)

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/utils/run_algo.py in run_algorithm(initialize, capital_base, start, end, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, default_extension, extensions, strict_extensions, environ, live, exchange_name, base_currency, algo_namespace, live_graph, analyze_live, simulate_orders, auth_aliases, stats_output, output)
    547         simulate_orders=simulate_orders,
    548         auth_aliases=auth_aliases,
--> 549         stats_output=stats_output
    550     )

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, print_algo, local_namespace, environ, live, exchange, algo_namespace, base_currency, live_graph, analyze_live, simulate_orders, auth_aliases, stats_output)
    329     ).run(
    330         data,
--> 331         overwrite_sim_params=False,
    332     )
    333

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in run(self, data, overwrite_sim_params)
    350     def run(self, data=None, overwrite_sim_params=True):
    351         perf = super(ExchangeTradingAlgorithmBacktest, self).run(
--> 352             data, overwrite_sim_params
    353         )
    354         # Rebuilding the stats to support minute data

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in run(self, data, overwrite_sim_params)
    307         data.attempts = self.attempts
    308         return super(ExchangeTradingAlgorithmBase, self).run(
--> 309             data, overwrite_sim_params
    310         )
    311

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/algorithm.py in run(self, data, overwrite_sim_params)
    722         try:
    723             perfs = []
--> 724             for perf in self.get_generator():
    725                 perfs.append(perf)
    726

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/algorithm.py in get_generator(self)
    620         method to get a standard construction generator.
    621         """
--> 622         return self._create_generator(self.sim_params)
    623
    624     def run(self, data=None, overwrite_sim_params=True):

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/algorithm.py in _create_generator(self, sim_params)
    591
    592         if not self.initialized:
--> 593             self.initialize(*self.initialize_args, **self.initialize_kwargs)
    594             self.initialized = True
    595

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/algorithm.py in initialize(self, *args, **kwargs)
    455         """
    456         with ZiplineAPI(self):
--> 457             self._initialize(self, *args, **kwargs)
    458
    459     def before_trading_start(self, data):

<ipython-input-26-aabf0a13351d> in initialize(context)
     33
     34     context.asset = symbol('btc_usd')
---> 35     context.asset = symbol('BTC')
     36
     37     context.i = 0

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/utils/api_support.py in wrapped(*args, **kwargs)
     55                 % f.__name__
     56             )
---> 57         return getattr(algo_instance, f.__name__)(*args, **kwargs)
     58     # Add functor to catalyst.api
     59     setattr(catalyst.api, f.__name__, wrapped)

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in symbol(self, symbol_str, exchange_name)
    183     @preprocess(symbol_str=ensure_upper_case)
    184     def symbol(self, symbol_str, exchange_name=None):
--> 185         """Lookup an Equity by its ticker symbol.
    186
    187         Parameters

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py in symbol(self, symbol_str, exchange_name)
    225             exchange=exchange,
    226             data_frequency=data_frequency,
--> 227             as_of_date=_lookup_date
    228         )
    229

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange_asset_finder.py in lookup_symbol(self, symbol, exchange, data_frequency, as_of_date, fuzzy)
    114         log.debug('looking up symbol: {} {}'.format(symbol, exchange.name))
    115
--> 116         return exchange.get_asset(symbol, data_frequency)
    117
    118     def lifetimes(self, dates, include_start_date):

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/exchange.py in get_asset(self, symbol, data_frequency, is_exchange_symbol, is_local)
    238         # TODO: temp mapping, fix to use a single symbol convention
    239         og_symbol = symbol
--> 240         symbol = self.get_symbol(symbol) if not is_exchange_symbol else symbol
    241         log.debug(
    242             'searching assets for: {} {}'.format(

~/anaconda/envs/sdac/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py in get_symbol(self, asset_or_symbol, source)
    277             )
    278             quote_currency = self.substitute_currency_code(
--> 279                 parts[1], source
    280             )
    281             return '{}/{}'.format(base_currency, quote_currency)

IndexError: list index out of range

I believe there may be an issue with how I'm accessing the asset in the following:
context.asset = symbol('BTC')

Nearly identical code, however, works when using zipline.
Here is how you can reproduce this issue on your machine:
Reproduction Steps
If it would be helpful I can upload a binary of the data panel, as a pickle or zipped hdf5 or something else that would be useful.
I have also already ingested the bitfinex daily exchange data hence the btc_usd symbol lookup succeeds (when I uncomment it out).
...
What steps have you taken to resolve this already?
Looking through the stack trace and catalyst's documentation, it seems that symbol expects a target currency as well a base currency in the string with an underscore separating them. So I tried the following:
    context.asset = symbol('BTC_usd')

everything ran successfully but the resulting data (returns) were exactly the same, out to the 4th decimal place as when I used btc_usd so perhaps it is not case sensitive
...
Anything else?
I'm looking to test some idea on a large cross section using custom ohlcv data. I can get it to work with zipline, but the calendar is a bit messed up (not designed for cryptos like catalyst is!). But catalyst is designed for cryptos and in the future it would be nice to ues the live feature as well, so ideally I can do it all using catalyst.
Again, thanks a lot!
...
Sincerely,
tyler

I've managed to get custom OHLCV data into Catalyst by loading data from a CSV in the initialize function like so:
df = pandas.read_csv('minute-price-test.csv', parse_dates=['last_traded'], index_col=['last_traded'])
context.imported_prices = df

Then using the price from context.imported_prices in handle_data instead of the data that's piped in by Catalyst:
price = context.imported_prices.loc[context.datetime]['close']

You could probably skip the CSV reading part and just pass in your dataframe.
I'm looking at enhancing the exchange_bundle module to allow us to ingest data directly from a Pandas DataFrame into an exchange.
Hope the above workaround helps in the meantime!


feature(ui-ux): decentralized finance section

What this PR does / why we need it:
Design for Decentralized finance section
Additional comments?:

button is not clickable
content too wide apart when viewport is wide

seeing console errors

not sure if we can use tablet view for 1167px if theres not enuf space

check hover text color

Nice work on the transition, but need to handle on mouse leave too


thermostat fan question

Is there a way to turn the thermostat fan on/off independent of heating? The only choices appear to be AUTO, CYCLE, ON.
ON appears straight forward.
AUTO I assume means something like, "let the furnace decide" or "turn on when heating, possibly after a delay."
CYCLE I'm guessing is "run for X minutes hourly.
I haven't discovered (is it possible?) to turn the fan on/off independently of heating or cooling.
Thank you very much for your work on this project; it's great!

Hey @dhysom sorry it has taken me so long to address this.
I don't have access to a thermostat, but I'd be happy to fix this if you'd be willing to share your device with me (even just temporarily).
Let me know.

Closing due to lack of response. If a user is able to share a thermostat that is wired with fan controls, I'd be happy to look into this.


Fix/registry storage environments

@holyhope
Could you please check why only the V1.19 job is failed?
cc @bitsf


[Help] how to set gridBorderWidth

I want to hide the gridBorder,but no idea

How to set the dividing line between the header and the content section，not other griderBorder


Set AWS region in config files instead of GitHub Actions variables

Setting AWS region in config files:

Allows using different regions for different deployments, and
Prevents accidental deletion and recreation of ALL the resources if AWS_DEFAULT_REGION GitHub Actions variable is changed or deleted.

The fix:

Adds aws_region input variable to all Terraform modules and Packer templates.
In GitHub actions workflows, sets aws_region input variable to the value of AWS_DEFAULT_REGION GitHub Actions variable.
In GitHub actions workflows, sets azure_region input variable to the value of AZURE_DEFAULT_REGION GitHub Actions variable.

This supports different deployments in different  AWS/Azure regions, if required.at the same time, it does not require setting regions in each config file, when all the deployments are in the same region, which is the typical case.


Multibyte string is unreadable because of the garbled characters

Multi-byte characters may become garbled and displayed as tofu characters.
I want to specify the charset UTF-8.

I was able to reproduce the issue using plantuml.github.io/plantuml.js.
@startuml
tofu <|-- トーフ
とうふ -- 豆腐
@enduml

I think I need more additional font.
(How do I...)
font.adoc
Looks like listfonts isn't working in JS.


Animations not performing ?

Hi  johannes jo,
I set thee success icon with font awesome like "content: "\f00c";  font-family: FontAwesome;"
But i am unable to see the rotating and when blue color info came i see it is sliding bit down to up,I couldn't see these animations in my app. Is anything i need to check ?

Could you maybe provide a plunkr?


Replace IdentityServer4 with Duende.IdentityServer (#3008)

Pull Request
Fixes https://github.com/domaindrivendev/Swashbuckle.AspNetCore/issues/3008
The issue or feature being addressed
This PR replaces IdentityServer4 with Duende.IdentityServer.
Details on the issue fix or feature implementation

Replaced IdentityServer4 with Duende.IdentityServer in project dependencies.
Updated the OAuth2Integration code to be compatible with Duende.IdentityServer.
Verified that all existing integration tests are passing.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.65%. Comparing base (6b5633c) to head (8596545).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3184   +/-   ##
=======================================
  Coverage   83.65%   83.65%
=======================================
  Files          76       76
  Lines        3175     3175
  Branches      536      536
=======================================
  Hits         2656     2656
  Misses        519      519

Flag
Coverage Δ

Linux
83.65% <ø> (ø)

Windows
?

macOS
83.65% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


What about CPU usage in case of last commit?

When I create, for example, 150 instances of client.js?

It's still OK. The core module is only created once.


SEPG Automated Release - v1.4.0

SEPG Details
Release Build Link: https://jenkins.swinfra.net/job/SEPG/job/Auto-Release/863/
Build Token: github.com/UXAspects/buildenv-node18-image/main/4f0aff86a9fc8f391b8b73b0a99f271dc657b768
Versioning Behaviour: Roll on to next minor
Artifacts For Release
Docker

uxaspects/buildenv-node18:1.4.0 (Static Tag)
uxaspects/buildenv-node18:1.4 (Dynamic Tag)
uxaspects/buildenv-node18:1 (Dynamic Tag)
uxaspects/buildenv-node18:latest (Dynamic Tag)

 Build Stage
Action
Status

Release Checks
completed
:heavy_check_mark:

Gather Release Data

:white_small_square:

QA Documentation Site Deployment

:white_small_square:

QA Deployment Branch Promotion

:white_small_square:

Docker Image Promotion

:white_small_square:

Maven Artifact Promotion (Closed Source)

:white_small_square:

NuGet Artifact Promotion

:white_small_square:

Artifactory Release Artifact Tagging

:white_small_square:

Generic Artifact Promotion

:white_small_square:

Maven Artifact Promotion (Open Source)

:white_small_square:

Maven Staging Repository Removal (Open Source)

:white_small_square:

Git Tagging

:white_small_square:

Git Release

:white_small_square:

Git Release Asset Deployment

:white_small_square:

Pull Request Merge

:white_small_square:

Release Branch Deletion

:white_small_square:

view the build log here

RETRY

Retrying Auto-Release.....
Release Build Continuation: https://jenkins.swinfra.net/job/SEPG/job/Auto-Release/864/

Build Stage
Action
Status

Release Checks
in progress
:clock130:

Gather Release Data

:white_small_square:

QA Documentation Site Deployment

:white_small_square:

QA Deployment Branch Promotion

:white_small_square:

Docker Image Promotion

:white_small_square:

Maven Artifact Promotion (Closed Source)

:white_small_square:

NuGet Artifact Promotion

:white_small_square:

Artifactory Release Artifact Tagging

:white_small_square:

Generic Artifact Promotion

:white_small_square:

Maven Artifact Promotion (Open Source)

:white_small_square:

Maven Staging Repository Removal (Open Source)

:white_small_square:

Git Tagging

:white_small_square:

Git Release

:white_small_square:

Git Release Asset Deployment

:white_small_square:

Pull Request Merge

:white_small_square:

Release Branch Deletion

:white_small_square:

view the build log here


Support specifying config-file path

Kind: Feature
Description: Supports specifying path to --config-file option.
Closes: #9

@owenrumney Please review and let me know if this makes sense or needs some changes to be approved.


cannot import LLMFullResponseEndFrame

when trying v0.0.18, this error is given on loading the story page
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jwee/projects/pipecat/examples/storytelling-chatbot/src/bot.py", line 18, in <module>
    from processors import StoryProcessor, StoryImageProcessor
  File "/home/jwee/projects/pipecat/examples/storytelling-chatbot/src/processors.py", line 5, in <module>
    from pipecat.frames.frames import (
ImportError: cannot import name 'LLMFullResponseEndFrame' from 'pipecat.frames.frames' (/home/jwee/envs/alfabert/lib/python3.10/site-packages/pipecat/frames/frames.py)

Hi @janwout . This is probably because you have updated the repo (and therefore the examples) but you haven't installed latest pipecat-ai. So, from the storytelling-chatbot directory do pip install -U pipecat-ai.
Also, we just released 0.0.18 which fixes the microphone on storytelling-chatbot. I believe you were having issues before. That version should fix it.

I have also updated storytelling-chatbot README.md to make sure it's up to date. Another user on Discord just acknowledged that works for them.
Please, let us know!

ah yes, that did it, thank you!


Add a date/Timescale bar

DELETE EVERYTHING ABOVE AFTER READING - THANKS!
Sample (which sample are you talking about)
project-gantt-chart
Suggestion (the more details, the better)
Add a timescale bar at the top of the view

hi @GerryHGraham
A timescale bar itself is, is not available at this point.
Main issue is that I don't know what to supply days/weeks/months, I could make that depending on the length of the whole project.
Anyway I have added a already some dates (start, current & end) in the header as a start.
the updated code is waiting for approval in a pull request, in the meantime you can find it here https://exceltrainer.be/sharepoint/260/
Geert


Unify processing of adjacency lists across datasets

I unified how adjacency lists are processed (adding self loops, backward edges, computing type_to_num_incoming_edges) across all datasets (JsonLGraphDataset, QM9Dataset, PPIDataset). This refactoring almost doesn't impact the behavior (the only difference is that self loops are now consistently added with edge type 0, which was not the case previously for PPIDataset).
If we wish to further refactor this, it will now be very easy to extract the handling of self loops and backward edges out of datasets, and into either the model or to GraphDataset.

Thanks for this. There's a bunch of minor stylistic things, but apart from that, this looks great!

The reason we factored this out for JsonLDataset is that we wanted to use
the functionality in a different downstream dataset class (for the partial
graphs in the JsonLTraceDataset); that was occasioned by the lack of an
utility function like the one you created now. So my instinct here would be
not to unify by creating more such functions in PPI/QM9, but inline the
function calls there, and mark the existing _process_raw_adjacency_lists in
JsonLDataset as deprecated (but not remove it, to not break existing
subclasses).
On Sat, Apr 18, 2020 at 6:08 PM Krzysztof Maziarz<EMAIL_ADDRESS>wrote:

@kmaziarz commented on this pull request.
In tf2_gnn/data/ppi_dataset.py
https://github.com/microsoft/tf2-gnn/pull/18#discussion_r410723879:

             )

         )

     return final_graphs

def _process_raw_adjacency_lists(

Yes, I intentionally extracted _process_raw_adjacency_lists to be the
same in all three datasets, so that it's more unified and easier to
refactor further. I think it's used only once in QM9Dataset as well.
Would you recommend to inline it here and leave the other two?
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/microsoft/tf2-gnn/pull/18#discussion_r410723879, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKT62FPI64SYDCZY3HQFULRNHNALANCNFSM4MK5MBRA
.