Bloom

Currently still WIP, but bloom is already fully implemented and working. Currently refactoring code to align it with project standards, making small changes to make code more efficient & easier to read, and implementing possibility to really switch off bloom.

Travis doesn't complete, closing this and reopening


Extension that runs a request to a server that has a Let's Encrypt CA stopped working

Does this issue occur when all extensions are disabled?: No

VS Code Version: 1.60.2
OS Version: Windows 10 21H1

Steps to Reproduce:

Install an extension that runs a request to a server protected by a Let's Encrypt CA
An error is thrown in the VS Code developer tools indicating the certificate is expired, when it is not

I have an extension (for private use, under an NDA, so I cannot share its code) that runs a request, using axios, to a server which has HTTPS and uses a Let's Encrypt CA.
The request now fails indicating the certificate is expired, even though the certificate was renewed today and is valid for another 3 months. It started happening this afternoon after DST Root CA X3 expired.
The certificate shows as valid when browsing with Chrome, Firefox, Edge or on a smartphone.
The following error is thrown in the VS Code developer tools menu:
stack trace: Error: certificate has expired	at TLSSocket.onConnectSecure (_tls_wrap.js:1497:34)	at TLSSocket.emit (events.js:315:20)	at TLSSocket.EventEmitter.emit (domain.js:467:12)	at TLSSocket._finishInit (_tls_wrap.js:932:8)	at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:706:12)
Possibly related issue: https://github.com/electron/electron/issues/31212
Relevant PR: https://github.com/electron/electron/pull/31213

For those who use GOT this may be a temporary solution, until the VSCode update comes out: https://github.com/sindresorhus/got/issues/1148

Closing as already reported in #134244 .

This issue reoccurred in version 1.62.1.


How to use wide csv file for training and testing

I am trying to use a dataset with lots of timeseries data as specified in the:
https://ts.gluon.ai/stable/tutorials/data_manipulation/pandasdataframes.html Usecase number 3.
I am unsure on how to use this pandas dataset for this model.
Even when using a single timeseries data as mentioned in the another issue, I am unable to get the forecasts and it shows assertion error: (then its blank)

Thanks! A tutorial is coming soon about this after a short recoup! I'll post an update here once the tutorial is out.

waiting for that

Hi @TLF-1234  @loveoftheai , we uploaded a new Colab demo with a tutorial to use a CSV dataset.
Please check it and let us know if your dataset fits into one of the categories explained there.

Hi @TLF-1234 @loveoftheai , we uploaded a new Colab demo with a tutorial to use a CSV dataset.
Please check it and let us know if your dataset fits into one of the categories explained there.

It does seem similar but i am getting: TypeError: object of type 'NoneType' has no len() Type of data: float32 index: pandas._libs.tslibs.timestamps.Timestamp
I checked for duplicates in data but just can't figure out what is going wrong with this. (There are no Empty Columns or Empty data in between)

Hi. This error arises for the first column, DateTime. To avoid this, reformat this column as @turkalpmd mentioned in this link:
https://github.com/time-series-foundation-models/lag-llama/issues/10. Also, name the first column 'date,' for instance, to facilitate easy access to this column.
data['date'] = pd.to_datetime(data['date'], format='%Y-%m-%d %H:%M:%S')

Hi @TLF-1234 @loveoftheai, checking if this is resolved?

Closing as there is no response. Feel free to reopen if needed!

The Issue also accures if your data has a col named something like "date". In order to solve this simply delete the name of the date col in your lets say csv file. Specify the date col using the index in the pd.read_csv method, as its done in the goolge colab nb.


tech review for spectra_generator.md

This notebook needs a tech review.  I think it got a quick check by Troy a while back, but can either @bsipocz or @troyraen please do a tech review?
We don't intend to scale the notebook up, so functions do not need to be modified for efficiency.

When reviewing, see also: #285

Tracking what's done and still to-do from Tech Review Checklist

[x] Documentation

Done in #356

[ ] Notebook execution, error handling, etc.

Notebook runs to completion but throws one FileNotFoundError and a few warnings. I'll open a PR to address the error and maybe the warnings (need to look closer at those).

[x] Efficiency (skipping this)
[ ] Cleanup

Mostly done in #354. I'll do another small PR to get the rest.

Everything is done here except for work on the JWST function (#360). Those problems seem to be server-side and we are working with MAST to get them sorted out.

All items from the tech review checklist are done. Issues with the JWST function have been worked out, at least enough so that it returns data and doesn't raise errors or warnings. Note that some of the files that were previously available no longer exist in the MAST archive (see https://github.com/nasa-fornax/fornax-demo-notebooks/pull/360#issuecomment-2524193964).
@bsipocz Do you want this issue to stay open until the rendering issues get worked out?

No, we can handle the rendering as a separate question and anything content related coming up in it can have its own issue (e.g. the timeout issue, or anything else that comes up)


Add seq2seq notebooks and scripts

[x] zh_cn to th notebook
[ ] zh_cn to th script

Close as redundant to https://github.com/vistec-AI/thai2transformers/pull/67


Fixed class vs. interface terminology

@alfredmyers,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


[spirv] Migrate to perform vectorization on tensor ops

Previously the first thing we do after entering the SPIR-V CodeGen
backend is to bufferize. Then we deduce CodeGen configuration
and invoke the whole pipeline.
This commit moves the bufferization step downward. We first try
to deduce CodeGen configuration on tensor ops, and then invoke
the CodeGen pipeline. Notably for the vectorization pipeline,
vectorization now is done on Linalg ops with tensor semantics,
with the full flow of: tiling to invocations, vectorization,
vector optimizations, bufferizing, distributing to invocations,
and then perform loop canonicalization.

Depends on https://reviews.llvm.org/D113927 and https://github.com/google/iree/pull/7650.

Depends on https://reviews.llvm.org/D114376.


added deploy skill api call example

I added an example to deploy a skill via API. The rest of the old documentation about uploading skills may need to be updated?

I think, adding skills should be done only via the UI. Currently the UI does not support the default_skill_args field, but  it will soon. Also once the authentication works on the UI side, this API endpoint will actually only be accessible with a user access token that the UI fetches.

Since skill config is now supported by the UI, can we close this @HaritzPuerto?

agreed. closing


Mark drop_dataflow as stable

cc @teskje @antiguru @frankmcsherry
Just wanted to queue this up since MaterializeInc/materialize#18442 is inching closer to completeness. If there are remaining concerns about stabilizing drop_dataflow (beyond 2673e8c), I figured this would be a good way to draw them out. Marked as a draft, since we obviously we shouldn't merge this until all those concerns are resolved.

Between 2673e8c60 and ff516c81, drop_dataflow appears to be working well, based on testing in Materialize
(MaterializeInc/materialize#18442). So remove the "public beta" warning from the docstring.
Fix #306.

This is not going to merge without a great deal more attention, I'm afraid!

Great! :D
I am personally quite nervous about using drop_dataflow in production in Materialize without a clearer articulation of the remaining problems with drop_dataflow. I'd love to find a way to fix these problems (or justify why they won't be problems in Materialize's use of timely) that doesn't involve us discovering these problems the hard way in production!

For me, the issue is that "the remaining problems" are that things in timely (and many distributed systems) work not because they have an enumeration of remaining problems that we burn down, but because the things have specific reasons that they are meant to work. We then debug any non-adherence to those specific reasons, but the reasons need to exist first.
Here, drop_dataflow is some amount like de-allocating a Rust object behind a shared reference. Other folks in the system still have its name, still plan on doing things with it, and ... idk what sorts of assumptions get violated if it ceases to exist. The principle used to be "a dataflow is dropped once all participants have both created and eventually confirmed completion of the dataflow", with the progress tracking mechanism as the "reason" this is safe. With drop_dataflow that reason doesn't apply anymore, and .. it's unknown what class of thing might go wrong, and whether / how to fix it.

So, foremost of the remaining problems is the absence of an explanation for on what basis drop_dataflow should be safe to call, what the new dataflow lifecycle looks like (can you call drop_dataflow mid dataflow construction?), and what this means for folks who want to correctly use timely by relying on operator shutdown.

Thanks. This is really helpful.

So, foremost of the remaining problems is the absence of an explanation for on what basis drop_dataflow should be safe to call, what the new dataflow lifecycle looks like

Here is a starting point for an explanation that we can fortify or reject if it leaks from anywhere. drop_dataflow is safe to call on the basis that timely operates safely in the presence of an unwinding panics and asynchronous networks. In other words, I think an explanation for drop_dataflow can be derived be reducing its functionality to an equivalent set of events that we already deem to be safe.
To construct the equivalence of drop_dataflow to a set of already safe operations let's first consider a timely cluster running a single dataflow. This is a big assumption but we will come back to it in the end. In this setting when a worker panics (either during or after dataflow construction) all the operator state will be dropped and destructors will run. Additionally, we can hide this event from all the other workers by assuming that the network is down for an arbitrary amount of time. Therefore we can imagine a drop_dataflow implementation like so:
fn drop_dataflow_panic() {
    disable_network();
    panic!();
}

We can now split our safety argument into two subgoals.
a. That when a worker calls drop_dataflow_panic the rest of the workers do not take any action that violates safety
b. That when a worker calls drop_dataflow_panic that worker itself does not take any action that violates safety
Property a follows from the existing assumption about timely being panic safe, as already described. Importantly, property a holds regardless of whether the other worker panicked or not! In other words, for all behaviors of a worker W that are indistinguishable from drop_dataflow_panic the non panicking workers won't violate safety.
This gives us latitude to change drop_dataflow_panic as we please, and as long as its behavior is equivalent when observed over the network we won't violate property a.
Property b is trivially true for the current implementation of drop_dataflow_panic because the process halts and makes no further actions. If a process makes no actions then it is also impossible to take a safety violating action, since there are none taken.
This split is important because changing drop_dataflow_panic to an equivalent implementation has turned from a distributed system analysis problem to a single threaded program analysis problem. In other words, given an alternative implementation we only need to show that its single threaded execution produces the same observable behavior over the network to the one of disable_network(); panic!().
Timely's abstraction of inter-worker communication is provided by the Allocate trait, and each worker is aware of all the network channels allocated for a given dataflow. Therefore we can replace the drop_dataflow_panic from above to something like this (pseudocode) which I would argue behaves identically to drop_dataflow_panic when observed over the network:
/// Disables the network of the given dataflow id and immediately drops it
/// Panics if the dataflow is under construction
fn drop_dataflow_real(&mut self, dataflow_id: usize) {
    for channel in self.dataflow_channels[dataflow_id] {
        channel.disable();
    }
    if dataflow_under_construction {
        panic!();
    } else {
        self.dataflows.remove(dataflow_id);
    }
}

We have almost arrived to the conclusion that we want but we have made a huge assumption, namely that the timely cluster is running a single dataflow. If the cluster is running more than one dataflow then drop_dataflow_panic and drop_dataflow_real are no longer equivalent because in that setting drop_dataflow_real blocks only part of the network and drops only part of the state.
As far as I understand, this is the crux of the issue. If we can say that timely clusters can run multiple dataflows only as an optimization and the programmer should assume that each dataflow is its own failure domain that can crash independently then drop_dataflow is safe.
If instead we want to guarantee to timely programmers than when a dataflow panics on a worker no other dataflow gets to run on that worker, further elaboration is needed and it's unclear if drop_dataflow is safe.
On its face, having each dataflow be its own failure domain sounds like a fine model to program against. The programmer can imagine that every time they call worker.dataflow(..) this spins up a separate timely cluster that will run that singular dataflow. Admittedly this puts some amount of burden to the programmer that needs to be careful when using primitives that share state across dataflows (e.g exporting an arrangement) but the model is clear and can be reasoned about.
@frankmcsherry would love to hear your thoughts on this analysis


How i can use .net 6 in docker container?

What is tag for docker image?

You can e.g. use mcr.microsoft.com/playwright/dotnet:v1.40.0-focal which is using Ubuntu 20.04 and .NET 6.
Jammy is using Ubuntu 22.04 and .NET 8.
If you want to use a different .NET version which does not match these, we recommend creating your own Docker image.


1107 add jaro similarity

Closes #1107
Adding Jaro similarity to DuckDB and Spark cl and cll

This looks good to me. Just some minor comments above.

@ThomasHepworth  yeah moving all into their own folder makes sense to me i think,


How much time will be consumed to run the python script until the end

I can not run pyside2dash.py directly on macOS when the version of PySide6 == 6.4. and there's the wget command to download the source file.
wget --execute robots=off --wait=0.5 --force-directories --no-parent --page-requisites --mirror --convert-links --tries=3 https://doc.qt.io/qtforpython-6/
The directory size is 785.2 MegaBytes. All required python libraries have been installed. And the problem arises in the function get_function(classes: list) of the script (lines 73), no result has been found by soup = soup.find("div", class_="body"). I change it  to soup = soup.find("body"). The script runs successfully.
However, the length of the list "classes" exceeds 3000000, which makes the time consumption of running the script unacceptable.
I don't know much about BeautifulSoup and HTML, so I'd like to ask
(1) if it's my changes or the website source file changes that are causing this?
(2) How long did it take you to run the script?

My script may have bugs these days because the original https://doc.qt.io/qtforpython-6/ website changes. I'm sorry for that because I didn't use macOS anymore... Hope someone fix/fork or pull request

My script may have bugs these days because the original https://doc.qt.io/qtforpython-6/ website changes. I'm sorry for that because I didn't use macOS anymore... Hope someone fix/fork or pull request

Thanks for your reply and I will try to fix it. if the script successfully runs, I will initiate a pull request.


Problem building workspace on Eclipse+Android

The library is working fine, but, everytime the workspace is built I get this error:
Errors occurred during the build.
Errors running builder 'Android Package Builder' on project 'TestAndroid'.
Could not initialize class com.android.tools.lint.checks.BuiltinIssueRegistry
Steps to reproduce:
1º Download and unzip "Eclipse IDE for Java EE Developers" from www.eclipse.org/downloads
2º Install lombok.
3º Install ADT Plugin from https://dl-ssl.google.com/android/eclipse/
4º Configure ADT Plugin selecting Android SDK Folder
5º Create a new Android project.
6º Change any .java file and save to rebulid.
The problem occurs even though the library is not used in the project. If I remove the library from Eclipse de build has no errors.
I'm using last version of lombok and eclipse, and the Android's SDK is up to date and I'm using lombok.addGeneratedAnnotation = false in lombok.config.

i get this problem too.is it a bug?

To bypass this you can disable Android lint check when saving files. See the end of this post http://www.dosmweb.com/blog/index.php?post/2015/07/05/Project-Lombok%3A-Android


Add SonarCloud CI

Mentioned at https://github.com/adsharma/py2many/issues/191#issuecomment-866498863 , SonarQube provides quite good analysis for a lot of languages.  There is a cloud CI provider, sonarcloud.io, which only provides the default/official plugins, which sadly excludes Rust, but it is still useful.  It also provides a way to record coverage; while it isnt as mature of codecov.io, it is quite good.
As can be seen at https://sonarcloud.io/summary/new_code?id=jayvdb_py2many it would have detected the problem being fixed by https://github.com/adsharma/py2many/pull/344
The automatic analysis is enabled at Administration -> Analysis Method, ignore the fact it is not recommended. Then I suggest setting the "New Code" to Number of days : 10
We can disable code smells for test cases under Administration -> General Settings -> Analysis Scope setting "Source File Exclusions"

I haven't found a way to change the default "Cognitive Complexity" from 15 to say 20.  From my use of SonarCloud at work, this shouldnt cause a problem, as new code in PRs wont fail unless they are modifying the functions that are currently over the Cognitive Complexity threshold.

https://sonarcloud.io/summary/overall?id=jayvdb_py2many now has a C rating for reliability, and A for maintainability (165 smells, with only a few in tests/test_*.py so I haven't excluded those).
To achieve that, I set the exclusions to

tests/cases/**
tests/ext_cases/**
tests/expected/**
tests/ext_expected/**

https://sonarcloud.io/project/overview?id=jayvdb_py2many updated, with tests/dir_cases/** added to exclusions

The "C" reliability rating seems to be coming from two workarounds to silence pyflakes.
Which brings up the question of why not use flake8, which runs pyflakes internally and offers more flexible error checking (such as #noqa comments)?


Add comment system to website, disable submit button based on text box

Added a comment system to the website that lists comments as unordered list elements. The submit box is disabled on default, and enabled if any text is inside the comment box. This CL is merged into the IntroJson branch, as I needed to use the basic structure for adding  elements to the page (comments).

@bheff-work The mis-alignment was simply because I planned to include styling in another CL; however, I added a line break just to clean up the UI a bit.


[Portal] Second iteration on the component

This pull-request revamp the implementation of the Portal and the Modal component.
Those two components are core and used by many more components: Dialog, Drawer, Snackbar, Popover, Menu,  @Select.
This effort might also fix some issues with the Select component, to investigate.

Exposes and document our Portal component
Exposes and document our Modal component
Remove piece of logic for IE8
Exposes Backdrop component

Closes #9528 as it's documented
Closes #9252 as we take the owner document into account
Closes #9248 as the new portal component behave correctly during the reconciliation
Closes #9207 as we take the owner document into account
Fix one part of #9474
Breaking changes
Some properties have been renamed:
<Dialog
- ignoreBackdropClick
- ignoreEscapeKeyUp
+ disableBackdropClick
+ disableEscapeKeyDown

<Modal
- show
- disableBackdrop
- ignoreBackdropClick
- ignoreEscapeKeyUp
- modalManager
+ open
+ hideBackdrop
+ disableBackdropClick
+ disableEscapeKeyDown
+ manager

The zIndex object has been updated to match the usage.
  const zIndex = {
-  mobileStepper: 900,
-  menu: 1000,
+  mobileStepper: 1000,
   appBar: 1100,
-  drawerOverlay: 1200,
-  navDrawer: 1300,
-  dialogOverlay: 1400,
-  dialog: 1500,
-  layer: 2000,
-  popover: 2100,
-  snackbar: 2900,
-  tooltip: 3000,
+  drawer: 1200,
+  modal: 1300,
+  snackbar: 1400,
+  tooltip: 1500,
  };

<Portal container={this.container}>
 <Typography>But I actually render here!</Typography>
</Portal>

Very excited to see the ability to specify containers! Would love the ability to tap into that from the Popover component

Would love the ability to tap into that from the Popover component

@gregnb You can.

@oliviertassinari that's great! thanks for working on this

@oliviertassinari I don't think this is necessarily appropriate for Drawer.  For one, who's to say we don't want to use a Drawer within a certain component rather than at root level?
For another, you're assuming we want the backdrop to show when the drawer is open, but that's not the case when we're using the drawer for a responsive Sidebar, which shouldn't interfere with the content when it's open on a large enough display.

@jedwards1211 I'm not sure to follow. What's your conclusion?

Sorry yeah, I wasn't used to the structure of the new docs, I discovered the different types in the demos section.

@oliviertassinari Awesome! Can't wait to see this released, it's all we're waiting for to overhaul the UI on a project!

@corytheboyd We release every weekend. You can start next week :).


When a file type is removed from a document, everything else is also deleted.

There is a problem with the gridfs files deletion.
going by description in https://doctrine-mongodb-odm.readthedocs.org/en/latest/reference/storing-files-with-mongogridfs.html?highlight=file
I created a document, the problem is, most of the times, the file is just temporary in mongodb (in my app anyway), and gets deleted after. The deletion (or $document->setFile(null) ) on the @MongoDB\File also removes all other fields in that document, besides the data itself. I have a bunch of statistical information in that document, and it all gets removed when I just remove the file. What's left is just the Id. This might be the way it was intended, however doesn't not really seam like the mapping is accurate here, as it obviously removes a lot more from the database, than I from the object.
The operation should just remove the data that has been removed from the object.
Crosspost from http://www.doctrine-project.org/jira/browse/MODM-175

Closing this issue, as it's just a documentation problem. The implementation of GridFS in ODM didn't make it clear, but the file was actually the main content, with other fields mapped in the document being metadata belonging to the file. This is one of the reasons GridFS support is being rewritten for 2.0, to make this more clear.


Model's maximum context length

Hi all,
This must be a simple issue, but I can't use the chat.gpt window as it reached the maximum length.

How to reset the conversation?

You can manage multiple sessions, Ctrl+n to open new, Ctr+o to toggle sessions panel

Thanks, I couldn't find this info.


add authorization support for search queries using "include"

currently, the access rights to the model that is included in a search are not verified

moved to internal bug tracker


Refactor VocPub requirements

Several of the VocPub profile requirements are actually compounds.
This is shown by the fact that the SHACL implementation has two or more shapes for some rules, e.g. numbered Requirement-2.1.4a Requirement-2.1.4b.
Maybe refactor the requirements to reflect the way they must be implemented?

I may accept this - will review properly tomorrow - but I don't think it necissary that the Requirements:Shapes rules is 1:1 since a. you can write SHACL rules in different ways, compounding tests or not, and b. Requirements shouldn't be shackled (!) to SHACL: we don't want to force SHACL groupthink onto what we actually requirem fo our vocabs. There are also some Requirements, perhaps for other LD resources, that aren't SHACL-able e.g. things have to be online in HTML & RDF etc.

Yes I agree about SHACL lens or groupthink. But when I see '... and... respectively ...' inside a requirement, I suspect it could be usefully refactored. The SHACL solution merely corroborates this.


Fix Windows issue. Empty feature added.

Fixes Issue #15 with windows, where backslash is used instead of slash.
Also added new feature to delete all files inside the bucket before uploading (#18).

I really appreciate you taking on these two features! Not having Windows, I couldn't work on that one.
All of the formatting changes [and pointless stripping of ;] has made this PR very very noisy, and hides the actual changes.
I'll look more into it later today before merging.
Can you confirm you've tested this code in a non-trivial use? {I would normally assume that, but someone else didn't and I got bitten by that}

Thanks for getting back so quick. I've updated the PR to only reflect the real changes not the noisy formatting changes. Sorry for that.
I've tested it with windows and Linux both are working fine. (With nested files and directories)
The empty bucket code was tested with a few hundred files also inside Subdirectories and in the root folder.
I hope I've covered everything but obviously no guarantee ;)


Unit tests are visible in the project

Even though the package was imported through scoped registery the tests are still showing up in testing window which shouldn't happen. My assumption is that the assembly definition for BovineLabs.Core.Tests is missing UNITY_INCLUDE_TESTS define.

This definitely shouldn't happen. I'll take a look thanks.

This should hopefully be fixed on the 0.15.6 branch (https://gitlab.com/tertle/com.bovinelabs.core/-/tree/0.15.6) and next release:

It does work on the mentioned branch. Thanks! :)


Significant drop in NPS after upgrading from version 0.19.0 to 0.20.0

Hi 👋 We've seen a significant drop in our NPS and an increase in the number of responses right after we've updated the SDK from version 0.19.0 to 0.20.0, and there were no other changes in our app during that period that could explain this change in the NPS. We also haven't noticed a change in sentiment in the written responses that we've been receiving.
I can see that there were some "reskin" changes and also a change related the NPS slider. Could these have affected our NPS measurements? Does that mean that the current numbers are more accurate (or less accurate) than the numbers we were getting with the previous version of the SDK?

I just noticed that https://github.com/Wootric/WootricSDK-iOS/issues/90 could be related to what we're seeing.

Hi @fabio914 thanks for bringing this up. We just released version 0.20.1 which reverts the color back to the old value. Hopefully this fixes the drop in NPS 🙏
Sorry for all the trouble.
Cheers!


Check For Updates menu option missing

Type: Bug
Upon launching insiders, the check for updates menu option (in the gear bottom of the toolbar) is fully missing. You also cannot run the Code - Insiders: Check for Updates... command from the pallette, although it is still present within Keyboard Shortcuts.
Reinstalling insiders restores the function.

No, not explicitly running as administrator. Occasionally I open insiders from within a terminal running as administrator, but the issue persists regardless of permissions. I've played with changing terminal permissions as well and opening it from a non-admin terminal doesn't change anything either.
Typically, I just open insiders via the msft powertoys run module or pinned to my dock, neither of which are run as administrator either.

Can you show a screenshot of the entire VS Code window?

I've since reinstalled and resolved the issue. I can't seem to repro the issue, so more than likely opened as admin without realizing it. Not sure how, but I'll be keeping a closer eye on it. I think it's safe to close this for now, I can reopen if I end up coming across this again.

Possibly related: #178849.
Possibly related on Stack Overflow: Why doesn't my VS Code 1.77 installation on Windows with the user setup have "Check for Updates" commands / buttons?


Flavour limitations ?

Can you elaborate about the limitations and restriction of this package , (e.g) can we generate build more than 300 apps separate  android and ios build ? if  yes means, any issues may happened in future ?  can we use this package for SAAS kind of application  build

Please use the Discussions section to ask questions.


fix: use non-deprecated APIs where available in getFreeDiskStorage()

Description
This PR switches two StatFs method calls in getFreeDiskStorage() to use their newer, long, variants when available (instead of older, deprecated versions).
Fixed issue #N/A

Compatibility

OS
Implemented

iOS
✅ (n/a)

Android
✅

Windows
✅ (n/a)

Checklist

[x] I have tested this on a device/simulator for each compatible OS
[ ] ~I added the documentation in README.md~
[x] I mentioned this change in CHANGELOG.md
[ ] ~I updated the typings files (deviceinfo.d.ts, deviceinfo.js.flow)~
[ ] ~I updated the dummy web/test polyfill (default/index.js)~
[ ] ~I added a sample use of the API (example/App.js)~

LGTM - thank you!

2.0.1 is out :rocket:


🐛 BUG: Tab panel in LivestreamDetailsModal allows scroll below content

The tab panel in the LivestreamDetailsModal seems to contain a large section of whitespace at the bottom of its contents, making the panel scrollable even when it shouldn't be.
https://github.com/sugar-cat7/vspo-portal/assets/155891765/5b87e688-993c-40fa-ace4-18866f958c5d

It seems there might be unnecessary padding.
https://github.com/sugar-cat7/vspo-portal/blob/7ef734f7a472711dca20bad9d3961571304319fd/service/vspo-schedule/web/src/components/Elements/Modal/LivestreamDetailsModal.tsx#L463-L479

@sruenwg
Thank you for your help!
After checking the page post-deployment, it appears that a degradation has occurred.
https://github.com/sugar-cat7/vspo-portal/assets/69241625/04a75725-fcfa-4d85-b705-313deebed598


Module Assignments

👋 HI @SMegala2019  , please complete the module assignments according to the instructions in order to move to the next step to become a certified Campus Advisor.

Given what you have learned about Git, why would you want to use Git in your classroom?
Git is a tool, which helps to create Code Repository as well as it helps to share the knowledge for the beginners.
i wish to enhance my students programming capability so i want to use git in my classroom..
many students are having good programming knowledge, we need to kindle the interest in coding practice among students .


updating opensearch sink constructor to support loading dlq plugins

Description
Adds support for providing a plugin loader to OpenSearch sink. This will be used once #2392 is merged.
Issues Resolved
part of #2298
Check List

[x] New functionality includes testing.
[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #2415 (ada71e7) into main (7bd967a) will decrease coverage by 0.19%.
The diff coverage is n/a.

:exclamation: Current head ada71e7 differs from pull request most recent head cd50a5d. Consider uploading reports for the commit cd50a5d to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #2415      +/-   ##
============================================
- Coverage     93.86%   93.68%   -0.19%
+ Complexity     2001     1997       -4
============================================
  Files           234      234
  Lines          5546     5546
  Branches        443      443
============================================
- Hits           5206     5196      -10
- Misses          230      241      +11
+ Partials        110      109       -1

see 1 file with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Unnecessary use of generics in org.axonframework.common.caching.Cache

I would argue that using Generics for org.axonframework.common.caching.Cache methods without generifying the type (interface) itself is unnecessary and adds noise (and thus, cognitive strain) when trying to reason and use the interface.
There is no relationship created between the types of the key and value stored in the Cache (as it should, indeed, not be any). Moreover, the fact that the type names are the same, but not declared with the interface type, surely does not prevent using different types when invoking get()/set() methods.
The only reason I can think of for not generifying the interface itself is to maintain some sort of code compatibility with (older?) libraries which do not use generics.
As a solution, either remove generic types from method declarations or generify the interface itself.

Noted @octavian-nita, having the generics does not add to much for the Cache#put(K, V), Cache#putIfAbsent(K, V), Cache#remove(K) and Cache#containsKey(K).
The get method however still benefits from the generics in place, is it will ensure that an implementation has to cast to V. We thus do not view that method as unnecessary use of generics.
Regardless, you are evidently correct when it comes to the other methods.
So I'll take a stab add changing the Cache interface and the provided implementations.
Thanks for filing this concern with us @octavian-nita, any insights are always much appreciated.

Indeed, interesting motivation for maintaining Cache#get(K) generic!


add the functionality of uploading images

Describe the feature you'd like
I would like to add the functionality of uploading images to a third-party image hosting service, such as integrating with PicGo in Typora to use Gitee, Git, or OSS image hosting. This feature would be beneficial in scenarios where I need to upload large files or a large number of images, as it would help save server resources. Additionally, it would allow for easy migration to a new server while still being able to view the images properly in client programs like Typora. Please provide support for this feature
Describe the benefits this would bring to existing BookStack users
This feature would be beneficial in scenarios where I need to upload large files or a large number of images, as it would help save server resources. Additionally, it would allow for easy migration to a new server while still being able to view the images properly in client programs like Typora.
Can the goal of this request already be achieved via other means?
I currently don't have a better solution
Have you searched for an existing open/closed issue?

[X] I have searched for existing issues and none cover my fundamental request

How long have you been using BookStack?
Under 3 months
Additional context
No response

Thanks for the request @o-zero-o.
For these kinds of cases, our option to use S3 storage for images sounds like it would suit well.
Otherwise, I wouldn't be too keen to adding additional options/standards here and adding maintenance complexity since there's been little other demand for this kind of feature.


Individual clover addback correction

After discussion with @pcbend and Nikita (not sure if she is on here), we have decided to implement a clover dependent addback correction. We are still working out the details for how to put this in, but there will likely be a file that holds a correction matrix for each clover. This also means that everyone will have to figure out this correction themselves for each analysis they run because that will be much easier than keeping track of the positions of the clovers on the analysis end of things. We will likely provide a script for making such a correction.

Early morning... I meant cross-talk. Not addback.

Could you provide a few more details of the problem?
Does this tie in at all with your new Addback energy threshold at all?

Basically, the correction I put in for GRIFFIN had some assumptions. It said every clover had the same cross talk. There were two different "couplings" in that cross talk. The strong coupling was between 0-1 and 2-3 while the rest were weaker. All cross-talk lowered the energy of all of the hits. Peter and Nikita noticed that the cross talk is crystal/clover dependent, and can actually increase the energy in a few instances. The code to figure out the corrections is complete, and the code to get it in as a correction in the analysis tree will be done today.
It's slightly related to the threshold, but the low-energy bump we see in addback is likely due to positive energy hits. The lack of threshold would more likely cause a high energy tail, which I believe exists but is a very small perturbation on the peak.

So I have completed this with reasonable results.
At 1173, I have a singles FWHM of  2.03 and addback < 2.33 (I have one clover behaving a little wonky). Without the CT correction, I see multiple peaks, and a FWHM that doesn't make much sense. I will also polish up the various scripts that make the correction and complete a PR sometime soon.

I think this is more or less complete. In the future, we should take much more 60Co than we currently are in order to constrain this correction much more tightly.


Enhanceenhance case test_hdfs_json_load

Proposed changes
Issue Number: close #xxx

enhance case test_hdfs_json_load
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


Inline withTestApplication to reduce test indentation

Also add internal keyword to prevent dependencies on internal objects /classes.

Adding internal keywords is not a backwards compatible version, would recommend bumping the major version when merging this.

Adding these internal keywords is not a backwards compatible change, would recommend bumping the major version number when merging this.

Yes will release 0.2.0 when all of these PRs are merged. :)


chore: track avm proving time

Almost all that was missing was having an appCircuitName when avm-proving.

#7084  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @fcarreiro and the rest of your teammates on  Graphite

Benchmark results
Metrics with a significant change:

batch_insert_into_append_only_tree_32_depth_ms (256): 129 (-63%)
batch_insert_into_append_only_tree_32_depth_hash_ms (256): 0.444 (-63%)
batch_insert_into_indexed_tree_20_depth_ms (256): 175 (-61%)
batch_insert_into_indexed_tree_20_depth_hash_ms (256): 0.465 (-61%)
avm_simulation_time_ms (Token:transfer_public): 30.1 (-48%)
l2_block_processing_time_in_ms (4): 740 (-58%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Proof generation
Each column represents the number of threads used in proof generation.

Metric
1 threads
4 threads
16 threads
32 threads
64 threads

proof_construction_time_sha256_30_ms
11,782 (+4%)
3,175 (+3%)
1,508 (-14%)
1,442 (-2%)
1,566 (+8%)

proof_construction_time_sha256_100_ms
43,931 (-2%)
11,769 (-2%)
5,423 (-4%)
5,417 (-3%)
5,375 (-3%)

proof_construction_time_poseidon_hash_ms
79.0 (+1%)
34.0 (-3%)
34.0
59.0 (+4%)
87.0 (-2%)

proof_construction_time_poseidon_hash_30_ms
1,524
418
202
225 (-1%)
267

proof_construction_time_poseidon_hash_100_ms
5,760
1,571
724
767 (-1%)
797

L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
4 txs
8 txs
16 txs

l1_rollup_calldata_size_in_bytes
1,412
1,412
1,412

l1_rollup_calldata_gas
9,464
9,462
9,476

l1_rollup_execution_gas
610,285
610,283
610,297

l2_block_processing_time_in_ms
:warning: 740 (-58%)
1,407
2,652 (-9%)

l2_block_building_time_in_ms
25,124 (-13%)
50,679 (-4%)
97,968 (-15%)

l2_block_rollup_simulation_time_in_ms
25,029 (-13%)
50,373 (-4%)
97,616 (-15%)

l2_block_public_tx_process_time_in_ms
21,513 (-14%)
46,627 (-3%)
93,955 (-15%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 8 txs.

Metric
3 blocks
5 blocks

node_history_sync_time_in_ms
7,010
9,819 (-1%)

node_database_size_in_bytes
12,185,680
16,040,016

pxe_database_size_in_bytes
16,254
26,813

Circuits stats
Stats on running time and I/O sizes collected for every kernel circuit run across all benchmarks.

Function
time_ms
bytecode_size_in_bytes

GasToken:_increase_public_balance
69.0 (+1%)
13,873

GasToken:set_portal
16.4 (-4%)
3,495

Token:constructor
91.3 (-2%)
24,207

FPC:constructor
61.6
13,893

GasToken:mint_public
52.5 (+1%)
10,241

Token:mint_public
61.3 (-20%)
19,216

Token:assert_minter_and_mint
201 (-9%)
13,034

AuthRegistry:set_authorized
31.8
7,869

FPC:prepare_fee
80.1 (-5%)
15,187

Token:transfer_public
:warning: 30.1 (-48%)
31,425

FPC:pay_refund
66.0 (+13%)
20,080

Benchmarking:increment_balance
2,643 (-7%)
15,465

Token:_increase_public_balance
57.9
15,089

FPC:pay_refund_with_shielded_rebate
51.7
21,167

Public DB Access
Time to access various public DBs.

Function
time_ms

get-nullifier-index
0.181 (-3%)

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
256 leaves
512 leaves
1024 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.2 (-6%)
16.7 (-5%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.7
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.592 (-6%)
0.512 (-5%)
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
48.4 (-5%)
76.1 (+1%)
:warning: 129 (-63%)
246
465

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
95.9
159
287
543
1,055

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.493 (-5%)
0.468 (+1%)
:warning: 0.444 (-63%)
0.444
0.434 (-1%)

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
58.7 (-6%)
110 (-6%)
:warning: 175 (-61%)
351 (-1%)
674 (-2%)

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
109
207
355
691
1,363

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.497 (-5%)
0.492 (-7%)
:warning: 0.465 (-61%)
0.476 (-1%)
0.463 (-2%)

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
71.8 (-8%)
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
133
N/A
N/A
N/A
N/A

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
0.511 (-8%)
N/A
N/A
N/A
N/A

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
85,672
670,983

Transaction size based on fee payment method
| Metric |  |
| - |  |

Merge activity

Jun 18, 12:51 PM EDT: @fcarreiro merged this pull request with Graphite.


Cannot communicate with Arduino Micro/Leonardo

Hi, I seem to have some sort of problem with your library.
My sketch works fine while using an Arduino Uno, but on Micro and Leonardo there are some issues.
'The serial communication is connected but Sharer is not available. Ensure that Sharer.Run() is called in loop() function and that setup() function takes less than 10000ms.' is thrown.
For what I read online, DTR and RTS should be enabled on the C# side, but I can't do that since your dll does not accept that parameters (I think).

Hi,
Thanks for this relevant remark, I don't have a Micro or Leonardo to test.
Can you try this for me please :
var connection = new SharerConnection("COM3", 115200);

// Access internal SerialPort object by reflexion
var port = (System.IO.Ports.SerialPort)connection.GetType().GetField("_serialPort", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic).GetValue(connection);

// Find the right RTS / DTR configuration
port.RtsEnable = true;
port.DtrEnable = true;

// Connect now, if it doesn't work, try connect before setting Rts or Dtr
connection.Connect();

Can you find a RTS / DTR configuration that works for this type of board?
If this solves your problem, I could publicly expose the properties in SharerConnection

That does indeed work, with both RTS and DTR enables.
Thanks!


Text remains highlighted after replace

After replacing text with :%s or in a visual selection with :'<,'>s whatever was highlighted remains visibly highlighted. ESC doesn't clear this, only a dummy :%s command that I then cancel instead of executing clears the selection.

This is not an issue with this package, but with evil-traces. A recent update to evil broke some weeks ago. I am not aware of any fixes. Please raise this issue there.


[WFLY-10289] Provide a log INFO about the ee8.preview.mode status

https://issues.jboss.org/browse/WFLY-10289
Make the ee8.preview.mode function a bit less invisible.
To be reverted as part of WFLY-10288 once EE8 certification is complete.

@jamezp The double-spacing is fixed. Per the discussion above the "true".equals(previewMode) is the correct way to do it.


Problem uploading samples

Hi I have an error when uploading samples through the browser.  There does not appear to be any documentation of how to do a manual upload using scp.

Hi,
Can you describe the error? A common issue is not providing the correct full path to the files in the upload template. The full path to the location that each fastq has been scp'ed needs to be provided. Can you provide the upload template sheet you are using showing paths? (Feel free to provide a partially anonymized screenshot if you prefer).
Thanks,
Gleb

Thank you!  Here is the screenshot.  I am trying to upload a pair of files
through the web interface.  Doing New, Upload Through Browser...
I cannot really see a full path represented through this method and there
is not a sample sheet.
Maybe I should try the batch upload?  I could also try scp but I don't have
the full instructions on how to do it and not sure where the files go.
Thanks for your help!
On Mon, Nov 4, 2019 at 9:48 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi,
Can you describe the error? A common issue is not providing the correct
full path to the files in the upload template. The full path to the
location that each fastq has been scp'ed needs to be provided. Can you
provide the upload template sheet you are using showing paths? (Feel free
to provide a partially anonymized screenshot if you prefer).
Thanks,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44UMNBCYMM6XTS63OU3QSBOANA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAD2AA#issuecomment-549469440,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44SJXRN4IE4NF5M26S3QSBOANANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hello,
Looks like the attachment didn't make it through. Might have to use the Github Issues interface directly for it to go through (rather than email?) If you have any additional details about the failure, please let me know (e.g. is there a delay before the error?)
Sorry we never put up documentation for scp upload. In fact it is the method most users here in the Church Lab use today. Briefly, the steps are something like:

scp files to a location of your choice on the machine running Millstone. E.g., create a data directory /home/ubuntu/raw-data and scp there.

Through the browser Samples page, use New -> From Server Location...

Fill in the template linked in that form with each row representing a sample and the full path to the location.

Unfortunately, we didn't get around to adding scp instruction to the official docs here:
https://millstone.readthedocs.io/en/latest/user_guide/projects_alignments.html
In case it's helpful, here's a draft of more complete documentation we started writing but never got around to posting. Feel free to glance in case there is something helpful there.
https://docs.google.com/document/d/1tbPiVaaVqECliw5Eu8xBJ8OxpVHynWpo1_kFEFkoJmU/edit?usp=sharing
Thanks!
Gleb

Hi Gleb!  I just wanted to give you an update that I may have made a little
more progress.
I tried using the second option, batch upload from a template through
browser, and it seemed to get through the first file, but got hung up on
the second every time.
I then tried the scp.  It had similar issue, where file 1 says copying and
file 2 stays in a state of queued to copy.
So I tried a couple of other different files using the batch upload - it
seems to have gotten both files.
Now I'm running into some errors on fastqc and alignment step - let me try
a couple more things and then I'll let you know how it goes.  Thanks for
your help!

Shane

On Mon, Nov 4, 2019 at 9:48 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi,
Can you describe the error? A common issue is not providing the correct
full path to the files in the upload template. The full path to the
location that each fastq has been scp'ed needs to be provided. Can you
provide the upload template sheet you are using showing paths? (Feel free
to provide a partially anonymized screenshot if you prefer).
Thanks,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44UMNBCYMM6XTS63OU3QSBOANA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAD2AA#issuecomment-549469440,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44SJXRN4IE4NF5M26S3QSBOANANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Shane,
The "queued to copy" state is actually temporary and should resolve with some time. Though usually it's pretty quick.
A couple other issues to check:

what is the ec2 instnce you are using?
how much EBS space is there?

Thanks,
Gleb

Thanks Gleb!
It looks like it was stuck in queueing.  I believe there may have been
something weird about those files.
On the other files, they loaded, and then I had to make sure they were
really gzipped files, and then I got the fastqc step to work!
That is a good point, I am running on the smallest ec2 instance right now.
It does say there is about 1.7GB still free in the upper right corner.
I'll keep an eye on that and I will try the alignment step next.  Thanks
for all your help so far, I really like Millstone!  I will let you know how
it goes.  :)
On Tue, Nov 5, 2019 at 3:07 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi Shane,
The "queued to copy" state is actually temporary and should resolve with
some time. Though usually it's pretty quick.
A couple other issues to check:

what is the ec2 instnce you are using?
how much EBS space is there?

Thanks,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44U7YS6UI5BFGYDY7KTQSH4BJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEVA3Q#issuecomment-550064238,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44WALWBUGFQUUG5ASXDQSH4BJANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hey Shane,
Good to hear it's working. Sorry the UI affordances are not perfect. We're expert users here so we hack features as needed :)
As far as instance type, I'd recommend running at least m5.xlarge to make sure you have enough memory for the alignment and subsequent analysis. I'm also concerned you may run out of disk space and would recommend allocating at least 3-5x as much disk as the size of all your FASTQs.
There's a few other tricks we do here to speed things up and make efficient use of AWS. But some of them require manually modifying some of the code and config files and we haven't really documented this anywhere. For example, when aligning > 50 genomes, we'll normally change the instance type to one with many cores (e.g. c5.9xlarge) and tweak the respective config that says to use all the cores. This actually turns out to be cheaper than running a smalerl instance in a more serial model. We'll then change back to a smaller instance type to do analysis / export data.
Anyway, happy to discuss/advise further if you're interested in ramping up your microbial genome alignment/analysis pipelines.
Cheers,
Gleb

Thanks so much Gleb!
I have now gotten data in there and it runs FastQC successfully.
I then tried to do an alignment, but it seems to find no variants in the
table.  When I go to the alignment it says no data in table.  The job
status says completed.
I don't know if it's possible there are no variants - do you have a good
test dataset that you would recommend?
Or perhaps this is related to the small machine size, I could try that.
The job says it completed in 2 minutes which sounds short (this is e coli).
The log seems to go to the end, but does mention something about an output
being truncated.  I'm pasting the log output below.  Let me know what you
think!
==START OF ALIGNMENT PIPELINE FOR SAB2, (411a5775) ==
/home/ubuntu/millstone/genome_designer/conf/../tools/bwa/bwa mem -t 1
-R "@RG\tID:77e5a0a4\tPL:illumina\tPU:77e5a0a4\tLB:77e5a0a4\tSM:77e5a0a4"
-a /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/ref_genomes/d89f8976/tmpxft5Bs_MG1655_fasta
<(gzip -dc /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/samples/77e5a0a4/s1.fastq.gz)
<(gzip -dc /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/samples/77e5a0a4/s2.fastq.gz)
| /home/ubuntu/millstone/genome_designer/conf/../tools/samtools/samtools
view -bS -[M::main_mem] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48443, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and
std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.13, 49.98)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::worker2@0] performed mate-SW for 3064 reads
[samopen] SAM header is present: 1 sequences.
[M::main_mem] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48360, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and
std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.82, 49.62)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::worker2@0] performed mate-SW for 3100 reads
[M::main_mem] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48333, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and
std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (499.12, 49.84)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::worker2@0] performed mate-SW for 3324 reads
[M::main_mem] read 100000 sequences (10000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 48406, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 533)
[M::mem_pestat] low and high boundaries for computing mean and
std.dev: (329, 669)
[M::mem_pestat] mean and std.dev: (498.96, 49.80)
[M::mem_pestat] low and high boundaries for proper pairs: (261, 737)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::worker2@0] performed mate-SW for 3006 reads
[M::main_mem] read 60000 sequences (6000000 bp)...
[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (0, 29029, 0, 0)
[M::mem_pestat] skip orientation FF as there are not enough pairs
[M::mem_pestat] analyzing insert size distribution for orientation FR...
[M::mem_pestat] (25, 50, 75) percentile: (465, 499, 532)
[M::mem_pestat] low and high boundaries for computing mean and
std.dev: (331, 666)
[M::mem_pestat] mean and std.dev: (498.83, 49.63)
[M::mem_pestat] low and high boundaries for proper pairs: (264, 733)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::worker2@0] performed mate-SW for 1808 reads
[main] Version: 0.7.5a-r405
[main] CMD: /home/ubuntu/millstone/genome_designer/conf/../tools/bwa/bwa
mem -t 1 -R @RG\tID:77e5a0a4\tPL:illumina\tPU:77e5a0a4\tLB:77e5a0a4\tSM:77e5a0a4
-a /home/ubuntu/millstone/genome_designer/conf/../temp_data/projects/4064419a/ref_genomes/d89f8976/tmpxft5Bs_MG1655_fasta
/dev/fd/63 /dev/fd/62
[main] Real time: 19.494 sec; CPU: 14.312 sec
[bam_header_read] EOF marker is absent. The input is probably truncated.
[bam_rmdup_core] processing reference U00096...
[bam_rmdup_core] 43 / 230000 = 0.0002 in library '77e5a0a4'
Removed 0 outliers with isize >= 849
==END OF ALIGNMENT PIPELINE==
On Tue, Nov 5, 2019 at 6:54 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hey Shane,
Good to hear it's working. Sorry the UI affordances are not perfect. We're
expert users here so we hack features as needed :)
As far as instance type, I'd recommend running at least m5.xlarge to make
sure you have enough memory for the alignment and subsequent analysis. I'm
also concerned you may run out of disk space and would recommend allocating
at least 3-5x as much disk as the size of all your FASTQs.
There's a few other tricks we do here to speed things up and make
efficient use of AWS. But some of them require manually modifying some of
the code and config files and we haven't really documented this anywhere.
For example, when aligning > 50 genomes, we'll normally change the instance
type to one with many cores (e.g. c5.9xlarge) and tweak the respective
config that says to use all the cores. This actually turns out to be
cheaper than running a smalerl instance in a more serial model. We'll then
change back to a smaller instance type to do analysis / export data.
Anyway, happy to discuss/advise further if you're interested in ramping up
your microbial genome alignment/analysis pipelines.
Cheers,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44T4225J5QKVTLOXVH3QSIWV3A5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDFCM7I#issuecomment-550119037,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44QPMELTCXLHQXY54W3QSIWV3ANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hey Shane,
No variants sounds surprising. And I agree 2 min sounds fast for an alignment and variant calling. It's hard to tell from the logs whether anything specifically went wrong. I can't remember whether "truncated input" is bad. It is possible the small machine ran out of memory at some point and a process failed in a way that didn't disrupt the rest of the pipeline.
You can look at our unit tests for some example data. For example if you trace through this test https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138, you'll see variables pointing to an example genome and fastqs:
        self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data',
                'test_genome_known_substitutions')

        self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions.fa')

        self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions_0.snps.simLibrary.1.fq')

        self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
                'test_genome_known_substitutions_0.snps.simLibrary.2.fq')

And test_data is located here: https://github.com/churchlab/millstone/tree/master/genome_designer/test_data

Awesome, thank you Gleb!
I also looked into the Jbrowse and I saw some coverage plots and other
things, so I think it definitely did something.  I will try your test data
next.
:)
On Wed, Nov 6, 2019 at 2:23 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hey Shane,
No variants sounds surprising. And I agree 2 min sounds fast for an
alignment and variant calling. It's hard to tell from the logs whether
anything specifically went wrong. I can't remember whether "truncated
input" is bad. It is possible the small machine ran out of memory at some
point and a process failed in a way that didn't disrupt the rest of the
pipeline.
You can look at our unit tests for some example data. For example if you
trace through this test
https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138,
you'll see variables pointing to an example genome and fastqs:
    self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data',
            'test_genome_known_substitutions')

    self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions.fa')

    self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions_0.snps.simLibrary.1.fq')

    self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions_0.snps.simLibrary.2.fq')

And test_data is located here:
https://github.com/churchlab/millstone/tree/master/genome_designer/test_data
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44XJ47LUGHEM5AZF6STQSM7VJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIGJBQ#issuecomment-550528134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44QREKJHNMDEDD7TXBDQSM7VJANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Gleb!  I tried out the sample data from E coli that you pointed me to
and it looks like it mostly worked.  Thanks!
I can see SNPs in the tracks in the browser.  However, when I click on the
Variants tab on the left side, it always just says "No Data in Table".
Also, when click on the alignment and "go to variants" I get the
error message in the screenshot attached.
Thanks again for your help.

Shane

On Wed, Nov 6, 2019 at 3:43 PM Shane Brubaker<EMAIL_ADDRESS>wrote:

Awesome, thank you Gleb!
I also looked into the Jbrowse and I saw some coverage plots and other
things, so I think it definitely did something.  I will try your test data
next.
:)
On Wed, Nov 6, 2019 at 2:23 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hey Shane,
No variants sounds surprising. And I agree 2 min sounds fast for an
alignment and variant calling. It's hard to tell from the logs whether
anything specifically went wrong. I can't remember whether "truncated
input" is bad. It is possible the small machine ran out of memory at some
point and a process failed in a way that didn't disrupt the rest of the
pipeline.
You can look at our unit tests for some example data. For example if you
trace through this test
https://github.com/churchlab/millstone/blob/master/genome_designer/pipeline/tests/variant_calling/test_variant_calling.py#L138,
you'll see variables pointing to an example genome and fastqs:
    self.KNOWN_SUBSTITUTIONS_ROOT = os.path.join(settings.PWD, 'test_data',
            'test_genome_known_substitutions')

    self.TEST_GENOME_FASTA = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions.fa')

    self.FAKE_READS_FASTQ1 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions_0.snps.simLibrary.1.fq')

    self.FAKE_READS_FASTQ2 = os.path.join(self.KNOWN_SUBSTITUTIONS_ROOT,
            'test_genome_known_substitutions_0.snps.simLibrary.2.fq')

And test_data is located here:
https://github.com/churchlab/millstone/tree/master/genome_designer/test_data
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44XJ47LUGHEM5AZF6STQSM7VJA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDIGJBQ#issuecomment-550528134,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44QREKJHNMDEDD7TXBDQSM7VJANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>
--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hey Shane,
Looks like the attachment didn't make it through to Github. Maybe it's not going through by email? Can you try uploading directly at the issue URL: https://github.com/churchlab/millstone/issues/701?
Thanks,
Gleb

Here is the screenshot!

Thank you Gleb!  I have attached the screenshot in the issue.  :)
On Tue, Nov 19, 2019 at 8:42 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hey Shane,
Looks like the attachment didn't make it through to Github. Maybe it's not
going through by email? Can you try uploading directly at the issue URL:
#701 https://github.com/churchlab/millstone/issues/701?
Thanks,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44XBBQ6CXEPYU3H6TMLQUS54LA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEQV5LY#issuecomment-555835055,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44XDXWF7EHI6XRFCXELQUS54LANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Ah, interesting. It looks like it might be version issue either with Django or Postgres. Though might be the data.
Are you running Millstone on AWS using our pre-built AMI?

We are using the pre-built AMI.  I believe our IT person had to clone it
from your zone and put it onto a new image in our zone - but essentially it
should be the AMI.
Thanks for your help!  :)
On Fri, Nov 22, 2019 at 11:56 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Ah, interesting. It looks like it might be version issue either with
Django or Postgres. Though might be the data.
Are you running Millstone on AWS using our pre-built AMI?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44VYPPVSWRCSRQJORP3QVA2QDA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE6WN2Q#issuecomment-557672170,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44WLBP6FQGTBMDC2453QVA2QDANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hmmm... I tried running the test data myself on a fresh image using our AMI and it seemed to work:

One thing is to confirm the postgres on your Millstone instance is the supported version (9.3):
ubuntu@ip-172-30-0-134:~$ psql --version
psql (PostgreSQL) 9.3.15

Thanks Gleb!  That does indeed look like the screen that I am not able to
see.
My Postgres version is 9.3.15.
So you think maybe it could be something about the version of Django or
something like that?
Thanks for your help :)
On Sat, Nov 23, 2019 at 9:44 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hmmm... I tried running the test data myself on a fresh image using our
AMI and it seemed to work:
[image: image]
https://user-images.githubusercontent.com/233915/69482898-ff9bce80-0dee-11ea-92b8-93ea547e1a0b.png
One thing is to confirm the postgres on your Millstone instance is the
supported version (9.3):
ubuntu@ip-172-30-0-134:~$ psql --version
psql (PostgreSQL) 9.3.15
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44VPQ6HA5GLMPJFAY5TQVFTZBA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE72DII#issuecomment-557818273,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44UZTCUONZITUXHRVOLQVFTZBANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Shane,
Hmm.. I was more concerned the Postgres version changed (e.g. due to system update), but that appears not be the case.
I suspect what might have happened on your end is that variant calling failed, and our UI is not set up very well to reflect this. Reviewing this thread, I was reminded that you are using "the smallest ec2 instance right now. It does say there is about 1.7GB still free in the upper right corner." So I think what might be happening is alignment is failing due running short on either memory or storage.
We typically use at least an m4.2xlarge (32 GB RAM) and at least 3x as much EBS storage as the FASTQ size (or at least 100 GB). That's what I used for my test yesterday. I recall users trying to use a smaller instance having similar issues.
I think a good bet is to retry on a bigger machine (at least m4.2xlarge) with sufficient storage.
-Gleb

Thanks Gleb!  It does have some vcf file in the project export by the way.
I was about to try upgrading the size actually anyway, based on your
suggestion.  My next goal is that I want to try it on some eukaryotic
genomes.
So I will work on that and let you know how it goes.  Thanks!
On Mon, Nov 25, 2019 at 1:09 PM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi Shane,
Hmm.. I was more concerned the Postgres version changed (e.g. due to
system update), but that appears not be the case.
I suspect what might have happened on your end is that variant calling
failed, and our UI is not set up very well to reflect this. Reviewing this
thread, I was reminded that you are using "the smallest ec2 instance right
now. It does say there is about 1.7GB still free in the upper right
corner." So I think what might be happening is alignment is failing due
running short on either memory or storage.
We typically use at least an m4.2xlarge (32 GB RAM) and at least 3x as
much EBS storage as the FASTQ size (or at least 100 GB). That's what I used
for my test yesterday. I recall users trying to use a smaller instance
having similar issues.
I think a good bet is to retry on a bigger machine (at least m4.2xlarge)
with sufficient storage.
-Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44XWCQNCQC6PR6TFJPTQVQ5IXA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFDZXIY#issuecomment-558341027,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44TP5RVJ32BFLN3SPLTQVQ5IXANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Gleb, could you describe to me a little more how to add extra drive
space to the Millstone instance?  I am now using some larger data and so
have attached an EBS volume to it.  And I then use the Server location
upload for the files.  However, it seems to still fill up the slash drive.
I have tried pointing /tmp at my EBS volume too but that does not seem to
work.  Thanks!
On Mon, Nov 4, 2019 at 9:48 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi,
Can you describe the error? A common issue is not providing the correct
full path to the files in the upload template. The full path to the
location that each fastq has been scp'ed needs to be provided. Can you
provide the upload template sheet you are using showing paths? (Feel free
to provide a partially anonymized screenshot if you prefer).
Thanks,
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44UMNBCYMM6XTS63OU3QSBOANA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDAD2AA#issuecomment-549469440,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44SJXRN4IE4NF5M26S3QSBOANANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Shane,
This isn't well-documented anywhere and the process is a little messy. A better short-term solution might be to spin up a Millstone instace with a bigger disk.
However, if you'd like to try extending to the bigger EBS, I believe the rough steps are:

Mount the EBS (e.g. to /millstone_data) and make sure you can write to it, (e.g. following directions [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html, and probably have to update write permissions i.e. sudo chown -R ubuntu:ubuntu /millstone_data/
Move the Millstone files from their previous location to the EBS location:
mv /home/ubuntu/millstone/genome_designer/temp_data /millstone_data
In ~/millstone/genome_designer/conf/local_settings.py, add/update the param MEDIA_ROOT = '/millstone_data/temp_data'
Fix symlink required by Jbrowse:
rm ~/millstone/genome_designer/jbrowse/gd_data
ln -s /millstone_data/temp_data ~/millstone/genome_designer/jbrowse/gd_data
Restart Millstone server and related:
supervisorctl restart all

I might have messed up a step or two above so give that a try if spinning up a new Millstone instance isn't feasiable.

Hi Gleb!  I wanted to let you know that I got my project export working -
it was running out of disk space.  It did create a rather large file, ~50GB.
I am getting a failure when running SV.  Can you tell me how I would find
the logs with the error?
Thanks,
Shane
On Tue, Jan 28, 2020 at 10:10 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi Shane,
Got it. Indeed there might be some issue with exporting the entire project
due to the instance size. It's a not a feature we optimized.
However, what most users actually want to export is a .csv of all the
called variants and metadata. That should work for your project, and just
have to do it from the Analyze view. For example, in the public demo we
host, you'd do it from this page:
http://ec2-52-4-236-89.compute-1.amazonaws.com/projects/b4cbc454/analyze/e0f7b0c1/variants?filter=&melt=0#

Click the top checkbox that selects all.
A blue notification appears informing you that only the 100 are
selected, but you probably want to press 'Select all results that match
this filter.'
In the dropdown, select 'Export as csv', as shown in the screenshot
below:

[image: image]
https://user-images.githubusercontent.com/233915/73292000-459a5780-41cf-11ea-9e47-db37f1e8fd48.png
The rest of the data (.fastq files, generated .bam files, etc.) are
located in the millstone filesystem (temp_data folder as discussed above)
so you can browse / scp what you need there, though folder names are by
software-generated uids, so might need extra detective work, or using the
django shell to query the database for that.
Let me know if that's what you were looking for.
Thanks!
Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44QBVMITMCQXHAJIEBTRABYILA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKEKUIA#issuecomment-579381792,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44SC6F2YSEZBPIOFWE3RABYILANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>

Hi Shane,
Great to hear.
All of the Millstone related logs get written to one of the files in /var/log/supervisor. Specifically either in millstone-stdout.log or celery-stdout.log.
-Gleb

Great I will take a look, thanks Gleb!
On Sun, Feb 9, 2020 at 9:40 AM Gleb Kuznetsov<EMAIL_ADDRESS>wrote:

Hi Shane,
Great to hear.
All of the Millstone related logs get written to one of the files in
/var/log/supervisor. Specifically either in millstone-stdout.log or
celery-stdout.log.
-Gleb
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/churchlab/millstone/issues/701?email_source=notifications&email_token=AM2Y44TFIBN4AKYCILPAQM3RCA52VA5CNFSM4JHTQX62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELGS6NA#issuecomment-583872308,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM2Y44WREGFHEXXN53X7OADRCA52VANCNFSM4JHTQX6Q
.

--
Shane Brubaker
Computational Biology Architect
Zymergen, Inc,
<EMAIL_ADDRESS>


Some text with combining characters do not match

Steps to reproduce

Paste หาข้อบกพร่อง กตฺตริ กิตฺ in a buffer (note: combining characters are ◌้ ◌่ ◌ฺ ◌ิ, without ◌)
:set re=0 | /กตฺตริ กิตฺ does NOT match
but :set re=0 | /ตฺตริ กิตฺ and :set re=1 | /กตฺตริ กิตฺ match
:set re=1 ic | /ข้อบกพร่อง does NOT match
but :set re=1 ic | /ข้อ and :set re=1 ic | /หาข้อบกพร่อง and set re=1 noic | /ข้อบกพร่อง match

Expected behaviour
Above patterns should match.
Version of Vim
9.1.0672
Environment
OS: Arch Linux 6.10.4
terminal: xterm
shell: zsh
Logs and stack traces
No response

can you please provide a more simple example with non-arabic chars? I am having trouble to understand when combining chars should be matched or not

The example หาข้อบกพร่อง กตฺตริ กิตฺ is in Thai, containing:

non-combining characters: ห า ข อ บ ก พ ร อ ง  ก ต ต ร  ก ต
combining characters:  ้  ่  ฺ  ิ   ิ  ฺ (or with ◌ to be readable: ◌้ ◌่ ◌ฺ ◌ิ ◌ิ ◌ฺ)

I used that example to show multiple cases that fail to match. For example,

if set re=0 (with any value of ic), /กตฺ and /กตฺตริ match, but not /กตฺตริ กิ (note the space is followed by a normal character and a combining character)
if set re=1, /ข้อ match, but not /ข้อบ if set ic (not the first character is a normal character, the second character is a combining character)

I want set re=1 because set re=0 fails to match some Thai text (e.g. กตฺตริ กิ). I also want set ic to conveniently search English text. But the problem is the combination set re=1 ic, which used to work in older version, fails to match some Thai text (e.g. ข้อบ).
A simpler example, using only a normal character ก and a combining characteer ิ, without any meaning, might be:

กก กิ fails to match if set re=0 (but can match กก and กก )
กิกก fails to match if set re=1 ic (but can match กิ and กิก)


Fix wrong usage of multiple APIs (mainly return check)

The commits are organized reasonably IMO, so please don't squash them if possible.

I adjusted the code style and have no idea why coding style validation failed again (with no hint about illegal style).

Adding "style: waived" the style checker seems to be complaining, but without reporting any errors!

There is another controversial thing - although the EVP_KDF_* functions are documented to return negative values if a function is not implemented, it is actually not true and none of the EVP_KDF_* functions will ever return a negative value. So maybe the right fix for these would be to fix the documentation.

There is another controversial thing - although the EVP_KDF_* functions are documented to return negative values if a function is not implemented, it is actually not true and none of the EVP_KDF_* functions will ever return a negative value. So maybe the right fix for these would be to fix the documentation.

Thanks for your comments. So I should change the documents instead of checks of EVP_KDF_derive and EVP_KDF_CTX_set_params, right? @t8m

Thanks for your comments. So I should change the documents instead of checks of EVP_KDF_derive and EVP_KDF_CTX_set_params, right? @t8m

Yes, I'd suggest so.

This pull request is ready to merge


Remove delete_pr_documentation as recommended

Hello!
Pull Request overview

Remove delete_pr_documentation as recommended by internal docs

Details
This should speak for itself.

Tom Aarsen

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Test failures are related to fsspec & unrelated to this PR, it's fixed in the v1.0.0-pre branch.


ServiceBus Unsubscribe from receiver link

While fishing in the code I found this opportunity of unsubscribing. Not 100% sure if those changes are ok but I figured I send it in for a review in case you want to take it.

Can one of the admins verify this patch?

@nemakam:  The changes look reasonable to me, but I lack a depth of understanding in this area of the codebase.   Would you be so kind as to give it a look and share your thoughts?

@danielmarbach - Curios as to how did you come across this? Was this causing a problem or just coincidentally ran into it?
These changes look good, but I think it might not cover all the cases.
If one link closes due to some reason, we also recreate the links. In your PR, we are taking care of the latest link that was created and ensuring you are unsubscribing it. Alternatively, we can can remove this handler within OnSendReceiveLinkClosed which gets called every time the link is closed.

On second thoughts, is this actually a bug? We call ActiveClientLinkManager.Close() when we want to close the sender/receiver, and at that point in time, the producer(AmqpLink) of the event itself is about to be garbage collected. So there is no memory leak in this case.
There is no advantage of this PR.
Anything that you think I'm missing? Any reason you think this PR is still worth merging?
There is nothing wrong with the PR itself, but wonder if its gonna make any difference.

@nemakam I was experimenting a bit with the SDK (looking what would have to be done to do proper cancellation support). I stumbled over the handlers not being deregistered. I was aware that it probably wouldn't make a difference. Normally when I manage code bases I try to be mindful about event registration and deregistration to not fall into memory leak traps so I wondered if it would be good practice to do just do it. At the end I figured I'll send it in and see what you think.
Happy for this to be closed if no value is seen in it (it is just code after all ;) )


Is there a way to decrypt an encrypted token?

Hi,
I am trying to decrypt a token which has been encrypted using the public key. I have access access to the private key before verifying the signature.
I cannot find any function available in this package. Can anyone guide me here?

jsonwebtoken does not implement JWE (JSON Web Token Encryption), you can give the jose package a try if you need both JWT/S and JWE.

@panva it was my next to go. I was checking if this package had it.


docs: update documentation

Fix a spellchecking of frequently in the contribution section

Oh, my code formatter have format the document and remove some whitespacing 😬
I hope this is not an issue.

🥳🥳🥳 So happy to see this one get fixed cos it bugged me SO MUCH being there so long! 😂😂
I will have to check and merge tomorrow when I have time to get to a computer, but I don't think the formatting should cause any problems.
Thanks a lot!


Log in redirect sometimes does not work and fails to log-in / authenticate

Describe the bug
When logging into Spotify and staying loggin in via "Remember me", if I attempt to re-authenticate the Spotify mobile service, after restarting the program, the log in window pops up but it is a blank page and when I go back into my program manually, it does not authenticate and throws a "Unable to get any application redirect arguments when expecting them" console message.
To Reproduce:
If you can, list some of the steps you are taking

When log in window pops up to log in to Spotify, check the "Remember me" box and log in. It should work just fine. After that, either end Spotify4Unity's mobile service or restart the app/program and attempt to log in.

Expected behavior
After website pops up it should redirect back to the program and authenticate the log in.
Platform:

Android

Unity Version:

2019.3.0f.6

Found the problem, for anybody else that is having similar issues on Android:
How I fixed mine is adding "&show_dialog=true" to the end of my log-in url.
It seems like the "show_dialog" parameter is always set to false, thus, hiding the content of the webpage (the web page is a page asking the user if they want to grant the APP permission to use said user's Spotify account and info). After adding in this parameter and setting it to true, the page will no longer be blank and the user can hit "accept" to redirect back into the APP and receive the callback.


Resourcing $MYVIMRC breaks nvim-lsp

nvim --version: v0.5.0-60c581b
nvim-lsp version(commit hash): 94cce7448f34f172492d98ff6efe10986f73eb9e
:checkhealth result: Everything passes
Operating system/version: macOS 10.15.5
Terminal name/version: Terminal.app
$TERM: xterm-256color

How to reproduce the problem from neovim startup
If I source my $MYVIMRC again lsp stops working. i.e omnifunc doesn't nothing and no diagnostics pop up.
  if !exists("g:nhooyr_lsp")
    "let g:nhooyr_lsp = 1
    lua << EOF
    local lsp_loaded, lsp = pcall(require, 'nvim_lsp')
    if not lsp_loaded then
      return
    end

    lsp.gopls.setup{}
    lsp.tsserver.setup{}
    lsp.vimls.setup{}
    lsp.clangd.setup{}
EOF
  endif

So I added an if guard around it to load only once.

i think the main takeaway here is we need better logging/inspection tooling. though narrowing the particular problem here would be good meanwhile

@nhooyr I don't know if this is really a bug.

Use this minimal_init.lua
Install pyright npm i -g pyright and open a python file in a git repo with nvim -u ~/.config/nvim/miminal_init.lua
See diagnostics
Trigger reload with <space>rc
No diagnostics
Trigger attaching of the buffer with e!
Diagnostics appear

I think your issue is with step 6, but what would the alternative be?

We could forcibly restart/reattach language servers to all buffers, but that may not be desirable (as it deviates from the current behavior)
We could keep all language servers alive with the old settings, but then users might get frustrated because they wanted their settings to change.

@nhooyr I don't know if this is really a bug.

Use this minimal_init.lua
Install pyright npm i -g pyright and open a python file in a git repo with nvim -u ~/.config/nvim/miminal_init.lua
See diagnostics
Trigger reload with <space>rc
No diagnostics
Trigger attaching of the buffer with e!
Diagnostics appear

I think your issue is with step 6, but what would the alternative be?

We could forcibly restart/reattach language servers to all buffers, but that may not be desirable (as it deviates from the current behavior)
We could keep all language servers alive with the old settings, but then users might get frustrated because they wanted their settings to change.

We could forcibly restart/reattach language servers to all buffers, but that may not be desirable (as it deviates from the current behavior), and may introduce high cpu load when a user wants neovim to reflect the state it would be at start (with no language servers attached).

Doesn't have to be automatic on all buffers, only what's visible. After sourcing $MYVIMRC, any visible buffer should have the language server reattach.
If I wanted neovim to reflect the state it would be with no language servers attached, I'd close all buffers or restart it.

Just to be clear, you want the servers to shutdown and restart, with only the currently open buffer spawning/attaching to a language server, rather than the previously active language servers to remain active.


fix generated statefulset being pruned by kubectl

When the operator transfers all labels and annotations from the source
prometheus object to the generated statefulset this also kubectl managed
annotations.
These annotations will cause the generated statefulset to be pruned when
the source prometheus is applied with kubectl apply --prune.
Cleanup of the statefulset is managed by the garbage collection via
owner references of the prometheus object (which can be pruned).
To fix this do not transfer any kubectl.kubernetes.io/ annotations to
the generated statefulset.
Closes gh-2642
Signed-off-by: Julian Taylor<EMAIL_ADDRESS>

There are some formatting/generation issues. Could you run “make clean generate-in-docker” and commit the changes? Thanks! :)

Thanks! Awesome that this finally is in place! :)


Unidirectional (To JSON) mapping with immutable model object

Hi,
I recently ran into the following issue in my project: I have a request object that I need to convert to JSON, with the (self-imposed) limitations that the object should be immutable.
So far I've had no problem accomplishing this with response objects (From JSON) and ObjectMapper, but when trying to accomplish the same thing with request objects (ones that will only ever be mapped To JSON), I ran into the issue that ObjectMapper expects a Mappable object to support mapping from and to JSON. This means that properties have to be var so that they can be changed when mapping from JSON.
#42 showed me that it is possible to provide different mappings when mapping to JSON vs. from JSON, but that seems to assume that the properties will still be vars.
So far I've implemented the following workaround:
struct InviteRequest {
  let golferId: Int
  let customerId: Int
}

extension InviteRequest: Mappable {
  var mappingGolferId: Int {
    get {
      return golferId
    }

    set {}
  }

  var mappingCustomerId: Int {
    get {
      return customerId
    }

    set {}
  }

  init?(_ map: Map) {
    fatalError()
  }

  mutating func mapping(map: Map) {
    mappingGolferId    <- map["id"]
    mappingCustomerId  <- map["customerId"]
  }
}

This isn't ideal because it requires a lot of extra boilerplate code. The init method is particularly frustrating because fatalError() yields a warning that the method isn't called, while omitting it causes an error that the object's properties are not initialized.
Perhaps there is a better workaround for this, but it would be nice if ObjectMapper better supported the case where a model object is only being used for mapping to JSON - perhaps by providing separate protocols, or support for something like:
map["id"] = golferId
map["customerId"] = customerId

Thank you for your time,
Noah

@tristanhimmelman  The project I am working on is struggling with this same issue. We want to be able to map our immutable properties to a JSON but aren't able to. We're considering forking, but we would prefer not to do this. Especially if you have a potential solution or are considering making changes in the near future.

There are hidden section in README.md that was add before but was hidden afterward. Checked raw source of it for a Mapping Immutable Properties. I am actually curious why it was done like this? @tristanhimmelman

Ok, I just found that valueOrFail as mentioned in some other issues. Also causing crashes in my code, so I assume it is hidden from docs on purpose.

@nikita-leonov I removed it from the README because the feature is still experimental and not well tested. There were a lot of reported issues with it so I decided to hide it until it is ready

@nikita-leonov I've been using valueOrFail and related methods for a while and haven't add many issues. The only downside is that you have to manually map the objects. An example:
extension InviteResponse: Mappable {
  init?(_ map: Map) {
    let rawOthers: [[String: AnyObject]] = map["others"].valueOrFail()
    guard let mappedOthers = [InvitedPersonResponse](JSONArray: rawOthers) else {
      return nil
    }

    others = mappedOthers

    guard map.isValid else {
      return nil
    }
  }

  func mapping(map: Map) {
  }
}

Not sure if there is a better workaround.

I've created an issue for discussing immutable types and how we may add support for them. Please continue the conversation here #383 if you have anything to add.


Add Browserlist config

Adds the “Using Baseline to target browser support” section to the “Baseline in your project” page.

I note the plan to transition @tonypconway's work on bl2bl to under web-platform-dx. My understanding is that the Browserlist configuration is slighty more complex than a last 2.5 years so as not to miss the last version released right before that. In any case, it will be useful to link to the code from the "Baseline in your project" page.

The main idea behind this PR is to offer a simple “set it and forget it” config. This is the best chance for Baseline to get traction and usage in existing toolchains. If the config doesn’t give the true “widely available” set and could be improved, that would be preferable. If not, then contributing to Browserlist to extend its capabilities is an option, too.

Thanks for looping me in @tidoust.

a simple “set it and forget it” config. This is the best chance for Baseline to get traction and usage in existing toolchains

Totally agree @pepelsbey . As François mentioned, and I explained here and @ddbeck confirmed here, just using [Browser] last 2.5 years doesn't actually give you the right feature set for Widely Available because you need to include the last version of each core browser before that cut off date.  If you don't, you'll end up unnecessarily polyfilling, or accidentally excluding users if you're using browserslist to power a "your browser is not supported" message.  Same goes for using a particular year, you can't just have [Browser] since 2020-12-31 for the 2020 feature set - you'd only include browsers after that date, when you need to include the last one before.  I don't think there's a way to get those browser versions without either changing browserslist, or calculating your min versions and updating your browserslist config on the fly.
There was already a PR created for browserlist to support these queries but the maintainers are currently not willing to move forward with it because they have concerns around how users of browsers like UC Browser and QQ are represented.  I'm working on a few things to address this:

the bl2bl module that Francois alluded to.  As a stop gap, this should address the "set it and forget it" piece you mentioned.  Simply add the module to your project, add a config to your package.json, and add some commands to your build process.  Please do take a look at it and feel free to add issues or make PRs.  It's a bit muddled at the moment and I'd love to get it cleaned up a bit before we move it over to this org.  This already has support for including versions of browsers that implement Chromium and appear in MDN browser-compat-data's browsers data.
I'm working with the very generous @logocomune who owns useragents.io on building a best effort mapping of Chromium downstream browsers like UC, QQ and Coc Coc which aren't in MDN's BCD.  Even Yandex browser, which doesn't feature in browserslist!
We're talking to different analytics providers about making it easy to understand the coverage you get from different Baseline versions.  At the same time, I'm also trying to create a Looker Studio template that takes a browser version report from the Explore UI in Google Analytics and gives you a breakdown of which Baseline Years are represented and a percentage of users that you'll support well if you use Widely Available.

Once these things are in place - especially 1 - I would be very happy to update this to reflect bl2bl usage.  And if/when browserslist does include Baseline queries, we can revisit the guidance and deprecate bl2bl!

Thank you everyone for the discussion! Closing the PR since the suggested query doesn't accurately represent the widely available Baseline.


Use the actual lockfile name in messages instead of assuming its alwa…

…ys Gemfile.lock
Closes https://github.com/bundler/bundler/issues/3672.

This seems like a positive change and good contribution, however Travis is failing.

Travis should pass now, will merge when green

:+1: :ship:


Can't find macro defination "WIN32"

In Code::Blocks, I can't find the defination of "WIN32", thus making the whole project for Win32 not compiled.It looks like this:

Or maybe I should modify the compiling settings?

Could you replace WIN32 by _WIN32 and ensure that everything works as expected?

Sorry, I didn't try it before opening this issue.
And now I have found that the CB's parser is to blame, it didn't found this macro but it does exist.
The codes below are compiled.