Notify of validation error to user

When I'm requesting an email account in Slack,
and I'm following the instructions given to me,
and Slack tells me the request was unsuccessful,
I want to know what went wrong,
so to resubmit my command,
or ask for help with clearly identified problem,
and have it solved ASAP

I've tested this in Slack using the test command set up for that purpose. What happens is that if you return any HTTP status other than 200, Slack will output an uninformative error message, and suppress all those outputs we've worked so hard to emit. So, that's still not satisfactory...

Other than that, lots to like in the PR too! (Wish I'd taken the time to write that earlier, but we were rushing to lunch...)

@Morendil So I've changed :

201 -> 200
4xx -> 200 + code in the body

Note : rebased

Argh, I may have jumped to conclusions regarding the behaviour I saw in Slack - I didn't notice that owing to the change in PR, the command set up in Slack was pointing to an app that no longer existed.

We should distinguish between validation errors and unexpected errors. Originally "Zut, il y a une erreur" was meant to catch unexpected cases; "zut" conveys both dismay and surprise.
For instance, a missing token generates "Zut, il y a une ou plusieurs erreur(s) : 'token' n'est pas valide". This won't help the Slack user as it's a configuration error. That should say something like "Zut, j'ai un problème inattendu ('token' n'est pas valide), merci de contacter un programmeur de BetaGouvBot."
For validation errors it would be more appropriate to say something like "J'ai du mal à interpréter votre commande correctement, car X, Y, Z".
Likewise when handling an empty command (equivalent to /command help) we should not prefix the output with "zut" but something like "Cette commande attend les paramètres suivants", etc.
The fullname validation regex thinks "bob69" is invalid, but has no problem with "bob" (no last name) or with "bob.joe.bart" (three parts instead of two). Personally, I think fullname validation is superfluous, but if we're going to have it, we should ensure it matches up with the validation in beta.gouv.fr code.
We should not include debug information in the messages sent to Slack channels - at present this is what a Slack user sees:
Zut, il y a une ou plusieurs erreur(s) : je ne vois pas de qui tu veux parler
{"code":404,"errors":["je ne vois pas de qui tu veux parler"]}

I think the way to fix that is to return an empty body if a token is present. If there is a token that means we're talking to Slack, and a response body will be displayed to the Slack user. If there is no token, we're in debug mode.
If you're going to be away for some time, let me know and I can probably take it from here.

@Morendil Thanks for the feedback. Some takeaways for me :

I've tested this in Slack using the test command set up for that purpose. What happens is that if you return any HTTP status other than 200, Slack will output an uninformative error message, and suppress all those outputs we've worked so hard to emit. So, that's still not satisfactory...

We should confirm that effectively something other than 200 is skipped by Slack

We should distinguish between validation errors and unexpected errors. Originally "Zut, il y a une erreur" was meant to catch unexpected cases; "zut" conveys both dismay and surprise.

For instance, a missing token generates "Zut, il y a une ou plusieurs erreur(s) : 'token' n'est pas valide". This won't help the Slack user as it's a configuration error. That should say something like "Zut, j'ai un problème inattendu ('token' n'est pas valide), merci de contacter un programmeur de BetaGouvBot."

Yeah, I've done it like this:

400 are configuration, so should definitely said something like "merci de contacter..." : it needs to be actionable for the user !
401 reserved for auth (also configuration but specific, should give same format as 400).
404 is self explanatory
422 for validation, including empty text

Likewise when handling an empty command (equivalent to /command help) we should not prefix the output with "zut" but something like "Cette commande attend les paramètres suivants", etc.

👍  Also, not that /command doesn't get sent without params, so the empty text case is never triggered as there is no client request being made. OK to change it to /command help and rewrite the message more appropriately.

The fullname validation regex thinks "bob69" is invalid, but has no problem with "bob" (no last name) or with "bob.joe.bart" (three parts instead of two). Personally, I think fullname validation is superfluous, but if we're going to have it, we should ensure it matches up with the validation in beta.gouv.fr code.

👍 Feel free to either only check for presence, or to improve the format validation.

We should not include debug information in the messages sent to Slack channels

👍 That's unintended, we should try either halt(4xx) or halt(200) to see what happens.

I think the way to fix that is to return an empty body if a token is present. If there is a token that means we're talking to Slack, and a response body will be displayed to the Slack user. If there is no token, we're in debug mode.

I have some alternative ideas for a debug / dry-run that do not require polluting our authentication logic. Tell me if they're feasible for de /slash command and if you like them :

header (X-DRY-RUN or X-VALIDATE-ONLY)
parameter (dry-run: true or execute: false)
nested resource (my favorite) : either /compte/dry-run or /compte/validate

If you're going to be away for some time, let me know and I can probably take it from here.

Feel free to address those issues and mark me for review / ping me. I won't be off on-the-go, so a less available. Boarding now ✈ .

Another approach would be to buy (vs build): https://github.com/slack-ruby/slack-ruby-bot

Superseded by #92


Increase code coverage for pkg/aspect.

This change is

Jenkins job mixer/presubmit passed

Jenkins job manager/mixer-pr passed

Jenkins job mixer/presubmit passed

Jenkins job manager/mixer-pr passed


Use size_hint in PyArray::from_iter.

This PR changes the way the initial capacity is calculated for PyArray::from_iterator.
If size_hint gives an upper bound, that upper bound is used as the initial capacity. Else, either the minimum bound from the size hint or the original 512 / size_of::<T>() is used, whichever is larger.
Additionally, the PR fixes an off-by-one in the resize condition. The old check would resize the array one element too soon. That meant doubling the array size unnecessarily, even if the size hint gave a correct upper bound .
With these changes, from_iterator is now pretty much equally performant as from_exact_iter. The only remaining difference is that the loop still has to check if the capacity is exceeded, since we can't rely on the size hint for memory safety. Otherwise I would have considered deprecating that function entirely.

Thank you for your PR.
I don't know much about how size_hint is used, but is there no case where max_len is really large(e.g., usize::MAX, Some(10)), which is problematic?

Additionally, the PR fixes an off-by-one in the resize condition. The old check would resize the array one element too soon. That meant doubling the array size unnecessarily, even if the size hint gave a correct upper bound .

Hoops, thank you for the fix.

I don't know much about how size_hint is used, but is there no case where max_len is really large(e.g., (usize::MAX, Some(10)), which is problematic?

Theoretically, size_hint can return anything. However, an incorrect large upper or lower bound would be a buggy implementation. size_hint is intended exactly for reserving capacity. From the standard library documentation:

size_hint() is primarily intended to be used for optimizations such as reserving space for the elements of the iterator, but must not be trusted to e.g., omit bounds checks in unsafe code. An incorrect implementation of size_hint() should not lead to memory safety violations.

The default implementation of Iterator::size_hint  returns (0, None) which still results in the old initial capacity. If an implementation does return incorrect large bounds, I would consider that a bug for that Iterator, not something we should protect against here.

OK, thank you for explanation :+1:

Thanks for the review :)


Add brainhack Toronto

10/24/2023 6:33:21 |<EMAIL_ADDRESS>| 2023 | Dec 14-15 | Brainhack Toronto | Hybrid (in-person and stream of tutorials and projects) | Erin Dickie | Centre for Addiction and Mental Health |<EMAIL_ADDRESS>| @ErinWDickie | Yes | Ju-Chi Yu | Centre for Addiction and Mental Health |<EMAIL_ADDRESS>| @juchiyu | Yes | Toronto | Canada | Krembil Centre for Neuroinformatics, CAMH | 50 | 12 | Yes | https://mattermost.brainhack.org/brainhack/channels/brainhack-toronto | @BrainHackTO | https://brainhackto.github.io/global-toronto-12-2023/ | https://github.com/brainhackto/global-toronto-12-2023 | Yes | N/A | N/A | N/A | No | Panel discussion | Skating | Yes, the main organizers of the event have read the Code of Conduct and we confirm we will abide by it. | N/A |   |   |   |   |   |
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --

Hi, could you correct the address for Brainhack Toronto for us? It's not pinned at a different Krembil Centre (sorry, this is confusing). Our address is "250 College St., Toronto, Ontario, M5T 1R8" for The Krembil Centre for Neuroinformatics (12th floor).

Would this one be better?
https://maps.app.goo.gl/J2cjaxuyajCEoAnr5
lat: 43.65931007053005
lon: -79.39946103953687

Ooh that’s very close, but we still need to go slightly south. Could we use this one?
https://maps.app.goo.gl/mpVLp6azDy34Yr8f7?g_st=ic

okidoki: pushing the change

let me know if it looks better now

This is perfect! Thank you so much :)


SAGE failing in big database

Description of the bug
tail -n 500 -f /hps/nobackup/juan/pride/reanalysis/proteogenomics/PXD010154/work/b6/6e37bc1f0f149a2785a8bd4d3968ab/out_17_sage.log
Found Sage version string: Version 0.14.3
Creating temp file name...
Creating Sage config file.../tmp/20231123_073725_hl-codon-bm-10.ebi.ac.uk_43_1.json
Sage command line: /usr/local/bin/sage /tmp/20231123_073725_hl-codon-bm-10.ebi.ac.uk_43_1.json -f GRCh38r110_GCA97s_noncanonical_proteins_19Jul23-decoy.fa -o . --write-pin 01524_B03_P015424_S00_N18_R1.mzML 01277_E01_P013129_S00_N05_R1.mzML 01283_A01_P013187_S00_N01_R1.mzML 01320_E04_P013559_S00_N29_R1.mzML 01296_D03_P013201_S00_N19_R1.mzML 01093_C02_P010748_S00_N11_R1.mzML 01279_A01_P013163_B00_N01_R1.mzML 01500_H04_P015160_S00_N32_R2.mzML 01698_A04_P018021_S00_N25_R1.mzML 01077_A03_P010695_S00_N17_R1.mzML 01501_A02_P015161_S00_N09_R1.mzML 01323_C05_P013562_S00_N35_R1.mzML 01276_E04_P013128_S00_N28_R1.mzML 01293_F03_P013196_S00_N22_R1.mzML
Standard output: Running: /usr/local/bin/sage /tmp/20231123_073725_hl-codon-bm-10.ebi.ac.uk_43_1.json -f GRCh38r110_GCA97s_noncanonical_proteins_19Jul23-decoy.fa -o . --write-pin 01524_B03_P015424_S00_N18_R1.mzML 01277_E01_P013129_S00_N05_R1.mzML 01283_A01_P013187_S00_N01_R1.mzML 01320_E04_P013559_S00_N29_R1.mzML 01296_D03_P013201_S00_N19_R1.mzML 01093_C02_P010748_S00_N11_R1.mzML 01279_A01_P013163_B00_N01_R1.mzML 01500_H04_P015160_S00_N32_R2.mzML 01698_A04_P018021_S00_N25_R1.mzML 01077_A03_P010695_S00_N17_R1.mzML 01501_A02_P015161_S00_N09_R1.mzML 01323_C05_P013562_S00_N35_R1.mzML 01276_E04_P013128_S00_N28_R1.mzML 01293_F03_P013196_S00_N22_R1.mzML

Standard error: [2023-11-23T07:37:25Z WARN  sage_core::modification] Variable modifications must be specified as a list of modifications: [0.984016]. This will become a HARD ERROR by v0.15
[2023-11-23T07:37:25Z WARN  sage_core::modification] Variable modifications must be specified as a list of modifications: [0.984016]. This will become a HARD ERROR by v0.15
[2023-11-23T07:37:25Z WARN  sage_core::modification] Variable modifications must be specified as a list of modifications: [15.994915]. This will become a HARD ERROR by v0.15
[2023-11-23T07:37:25Z WARN  sage_core::modification] Variable modifications must be specified as a list of modifications: [42.010565]. This will become a HARD ERROR by v0.15
[2023-11-23T07:37:33Z TRACE sage_core::database] digesting fasta
[2023-11-23T07:37:55Z TRACE sage_core::database] modifying peptides
[2023-11-23T07:42:35Z TRACE sage_core::database] sorting and deduplicating peptides

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
Process '/usr/local/bin/sage' crashed hard (segfault-like). Please check the log.

SageAdapter took 06:50 m (wall), 0.12 s (CPU), 0.08 s (system), 0.04 s (user); Peak Memory Usage: 42 MB.```

### Command used and terminal output

_No response_

### Relevant files

_No response_

### System information

_No response_

The dataset is PXD010154 and the database is: http://ftp.pride.ebi.ac.uk/pub/databases/pride/resources/proteomes/proteogenomics/noncanonical-tissues-2023/GRCh38r110_GCA97s_noncanonical_proteins_19Jul23-decoy.fa

If the database is too big, using more jobs will be worse since sage has to build the index separately in each job. Sage is designed for extremely efficient single node, single job tasks, so splitting into smaller jobs will almost always be worse than bigger CPU jobs with more files.
The database needs to be reduced in size, and jobs run over the database in chunks, and then combine the results at the end
https://github.com/lazear/sage/issues/97

@lazear thanks for the quick reply. It will take me some time to implement a chuck solution in quantms. In the meantime, do you have an estimation (even if is far from exactly) How much memory in the node is needed for a 2GB FASTA database?

I don't have an estimation, but it's a very bad idea. Fragment indexing is largely memory bandwidth bound - larger databases cause dramatic slowdowns and the algorithm stops being able to saturate CPUs at some level (around 10x larger than human canonical uniprot sized with trypsin + 2 missed cleavages). Database splitting is absolutely necessary for good performance at a database of that size

@jpfeuffer @daichengxin @timosachsenberg How can we implement in quantms a logic to partition the database for searches where extensive databases are searched? I understand from @lazear here that it will not be possible for SAGE to have a way around large databases. However, @lazear is suggesting that instead we partitionable the database, and search all the raw files against each chuck, in this case quantms will not be partitioning/distributing files but data (fasta files). I think this is doable but we need the following:

Database creation: Make sure the target and decoy ratio in every chuck is the same, I think for MSGF and COMET we do not do any filtering with the target and decoys, it is just the id part, for SAGE internally probably used them, then we need to make sure the ratio of decoys and targets is 50/50 for every chuck. One possibility is to not allow the user to provide decoy databases, and the tool for database creation makes sure it creates the database chucks correctly. Ideas @timosachsenberg @lazear @jpfeuffer @daichengxin.

NOTE: It may be interesting to use the pypgatk tool, which is the tool I use for proteogenomics database creation, and it has an interesting way to create the decoys without redundancy using decoypyrat approach. We can easily add it as a chunk option.

After SAGE/COMET/MSGF search, it should be easy to merge all the databases chucks and continue all the steps of quantms using the big database. I have tested for the given database 2GB all the other steps and they do work perfectly well.

What do you think?

It is hard to predict from the outside which/how many sequences to put into the individual chunks. I think in Sage one could (in principle) stop generating the fragment index, perform the search, and then later merge results from individual chunks. This might lead to some peptides being searched in multiple times but well ... I think MSFragger does something similar.

Ah I think I misunderstood the issue. I thought it is just about fitting the fragment index into memory, but this is also about distributing it on several nodes?

Actually @lazear we were internally discussing the issue and trying to find out a solution and @timosachsenberg point it out that another case is semi-enzymatic analysis which also increase the size of the search space dramatically. We actually think (and we were planning to discuss with you), that it is better to have a solution within SAGE that handle both cases, big databases or semi-enzymatic searches.
The problem with splitting databases, is the target decoy approach, we will need to control the distribution of targets and decoys and also make sure other search engines supported by the tool are also handling well the chunks. We are happy to discuss here with you possible solution because as you said we are already parallelizing the searches by msrun.

Why would the distribution of targets and decoys matter? If you are performing external competition on combined search results (e.g. taking the highest scoring PSM for that spectrum) it shouldn't matter if you did one search with 100% targets and another with 100% decoys - the end result should be exactly the same. Sage, Comet, etc also will happily generate decoys for you

I think if the database is split into several parts, we need to make sure that decoys of the same peptides in different chunks are generated deterministically. Otherwise, we might end of up with more decoys per peptide.

OK, so after doing some hacking I can offer at least a partial solution here.
You'll need enough memory to fit all of the Peptide objects in memory for Sage.
I have modified the codebase so that more than 2^32 peptides can be used, and to defer fragment ion generation - this means that mzMLs can be sorted by precursor mass and processed in chunks - a new fragment ion index will be generated for each chunk that contains only peptides within the monoisotopic mass range of those MS2 precursors. This essentially defers generating the actual theoretical fragments until they are needed, and should eliminate a substantial portion of memory use, but not all of it - the Peptide objects are not terribly efficient (storing the sequence and list of protein groups).
I haven't test to see if this will work on a 2GB database... but I also can't imagine that many people are (or should be, given the impact on sensitivity and FDR control) using such large databases. It might be possible to modify Sage further to parse FASTA files in chunks on-the-fly, but that will require more work than I am willing to do right now (at the moment the program basically requires that the full set of Peptide objects are live from start to finish, so new ones cannot be created once a search has started, and there are a variety of places that assume Peptides are unique)


Refactor directories

Plugin architecture for runners
Rename mlbox to mlcommons_box

1. I looked at the MNIST example, there's a `mlbox_docker.yaml` file (along with the `mlbox.yaml`) that species how MLBox is implemented. This information is loaded by the parser, so no need for now to provide runner name. Moving forward, however, I think we need to allow users to override the default mapping between MLBox implementation and reference runners to allow custom (user/corporate) runners.

the runner spec v0.1 proposes to move this docker config into /platforms directory, and user can specify their own platform config file during runtime which may partially or fully override the default config in the box, I think that will make it possible to override or add custom platforms/runners by the user.

2. We used to talk about `configure` / `run` actions previously. The `configure` is more generic than `build`, and for runners like Cloud runners or SSH runners using that name makes more sense. What do you think?

I'd like to suggest having both configure and build and using them for different purpose: the build may be used to create a box artifact that can run on a given platform (e.g. a container or VM image), while the configure may be used to interact with the platform itself to perform things like configurations and authentications, etc.

On a separate note, I'd also suggest that we have a common or lib directory under root that can be used to put shared libraries and tools. And we might also want an examples directory unless we are going to put the examples in a separate repo.

I'd like to suggest having both configure and build and using them for different purpose: the build may be used to create a box artifact that can run on a given platform (e.g. a container or VM image), while the configure may be used to interact with the platform itself to perform things like configurations and authentications, etc.

I think it sort of makes sense. But at least now I am having difficulties in understanding if it results in fluent user experience. From a user point of view, how do they know when to use configure  and when to use build? Several examples:

Local bare metal runner.
Local docker/singularity containers.
Remote coda bare metal environment.
Remote docker/singularity containers.

According to current idea of chain of runners, supporting the above is not an issue. How do I know if the right command is configure or build if I do not know the implementation details (bare metal / docker)?

One option is to put our common code in a pip package (mlcommons_box); and we have a different package for each runner, and our top level directories are pip packages.

We would have the follow directories and each one is a pip package, which can depend on each other:
mlcommons_box (for shared dependencies and what is needed to make/vaidate a box)
mlbox_ssh_run (for running over ssh)
mlbox_docker_run (for running locally using docker)

LGTM thanks.

Thanks for the review and feedback y'all 😄


Add template file organization

I added groups and files following our discussion regarding our standard file organization. So we will stick with MVC architecture first, then what should be changed will be up to each project requirement. As @olivierobert suggested, we could also have multiple template projects for each type as well. The file organization for our current template project will look like this:

closing this one due to integration of Bakery script requiring support for project name to be included with spacing. more info in PR #5


For configuring IOPS for volumeGroups, desired: prefixMatch, current: substringMatch

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.]
If we configure iops for the volumegroupPrefix "lvmvg", and in runtime if the volumeGroup in the MountVolumeRequest is "prefx-lvmvg-suffix", the iops would be set as configured from "lvmvg".
What did you expect to happen:
This is not desired as per the definition of the configurations. It should be set only when there is a prefix match and not a substring match
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin
kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-plugin
kubectl get pods -n kube-system
kubectl get lvmvol -A -o yaml

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

LVM Driver version
Kubernetes version (use kubectl version):
Kubernetes installer & version:
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release):

/assign


copy app icon from 2020 app and use it.

Issue

close #73

Overview (Required)

copy app icon from droidkaigi 2020 app repository and use it.

Links

https://github.com/DroidKaigi/conference-app-2020/tree/master/android-base/src/main/res

Screenshot

Before
After

adaptive icon video.
https://user-images.githubusercontent.com/1423942/109424477-159f3280-7a27-11eb-9189-01c01a300a5a.mp4

👀


SSL and Basic Authentication

Hi,
Was wondering if anybody was looking at SSL and Basic Authentication ?
If no, is there any plan or idea for implementing it?
As may have some time to contribute if required
Thanks

Pull Requests welcome.


Document created via onedrive api can't be opened for editing

I am doing a POST to https://api.onedrive.com/v1.0/drive/special/approot:/Test.docx:/content to create a Test.docx file in the AppFolder. The call succeeds and I see the file being created in the correct location. However, when I visit onedrive and try to open the file in Word Online, I get

This only happens for files created via the api. Any ideas?

What content are you providing in the body of the request? It will need to be the raw bytes for a valid docx file (I.e. NOT base64 encoded)

Ah yeah. That was the issue. Thanks.

Can you provide a code snippet on how to solve this? I'm experiencing this issue right now in one of my apps. thanks


The vagga _clean --old should check version rather than symlink

I'm not sure this should be the default behavior or separate option, but this options is super-useful for some cases like "gitlab"

Implemented vagga _clean --unused


Unknown (supercontig):start-stop is not a genomic region specification

for example:
location: aesev.CIAT22838.gnm1.Unknown (supercontig):2375348-2388727 (-1)
NB: this probably is not with the dumb old web component per se but either with the graphql code or the shim. Feel free to move it to a more appropriate repo!
Linkouts would be able to handle this appropriately if the supercontig id were actually given, otherwise not.

"aesev.CIAT22838.gnm1.Unknown" is the identifier that's received from Intermine. This can be confirmed by inspecting the link in question and looking at the "data-location" attribute, which contains the identifier verbatim as it came from Intermine.

Yeah, I think the issue is that Intermine treats supercontigs as a different class than chromosomes, but I think there needs to be someplace where the genomic location is formed from a non-chromosomal position if it doesn't have one.

Actually, this one works as desired:
location: vigra.VC1973A.gnm6.scaffold_7 (supercontig):698657-700123 (1)
so I'm not sure why the example I gave above is behaving differently. Will investigate further as time allows.

The (supercontig) is being inserted in web-components: graphql-server is returning the correct location, without the (supercontig) addition, and the linkout service URLs don't have the (supercontig) in them. I'll track it down.
{
  "data": {
    "gene": {
      "identifier": "vigra.VC1973A.gnm6.ann1.Vradi0007s00010",
      "locations": [
        {
          "supercontig": {
            "identifier": "vigra.VC1973A.gnm6.scaffold_7"
          },
          "start": 698657,
          "end": 700123,
          "linkouts": [
            {
              "href": "https://gcv.legumeinfo.org/tools/gcv/search?q=vigra.VC1973A.gnm6.scaffold_7:698657-700123&sources=lis",
              "text": "View vigra.VC1973A.gnm6.scaffold_7:698657-700123 in Genome Context Viewer"
            },
            {
              "href": "https://vigna.legumeinfo.org/tools/jbrowse2?session=spec-%7B%22views%22%3A%5B%7B%22assembly%22%3A%22vigra.VC1973A.gnm6%22%2C%22loc%22%3A%22vigra.VC1973A.gnm6.scaffold_7%3A698657-700123%22%2C%22type%22%3A%20%22LGV%22%2C%22tracks%22%3A%5B%22vigra.VC1973A.gnm6.ann1%22%5D%7D%5D%7D",
              "text": "View vigra.VC1973A.gnm6.scaffold_7:698657-700123 in JBrowse2"
            }
          ]
        }
      ]
    }
  }
}

And if there's any confusion, Unknown is an actual aesev supercontig, and is correctly given by graphql:
{
  "data": {
    "gene": {
      "identifier": "aesev.CIAT22838.gnm1.ann1.Ae0001050",
      "locations": [
        {
          "supercontig": {
            "identifier": "aesev.CIAT22838.gnm1.Unknown"
          },
          "start": 2375348,
          "end": 2388727,
          "linkouts": [
            {
              "href": "https://gcv.legumeinfo.org/tools/gcv/search?q=aesev.CIAT22838.gnm1.Unknown:2375348-2388727&sources=lis",
              "text": "View aesev.CIAT22838.gnm1.Unknown:2375348-2388727 in Genome Context Viewer"
            },
            {
              "href": "https://aeschynomene.legumeinfo.org/tools/jbrowse2?session=spec-%7B%22views%22%3A%5B%7B%22assembly%22%3A%22aesev.CIAT22838.gnm1%22%2C%22loc%22%3A%22aesev.CIAT22838.gnm1.Unknown%3A2375348-2388727%22%2C%22type%22%3A%20%22LGV%22%2C%22tracks%22%3A%5B%22aesev.CIAT22838.gnm1.ann1%22%5D%7D%5D%7D",
              "text": "View aesev.CIAT22838.gnm1.Unknown:2375348-2388727 in JBrowse2"
            }
          ]
        }
      ]
    }
  }
}

Thanks, I just also realized that "Unknown" here is what they used instead of calling it "chromosome0" or similar. So this is a non-issue, I don't think there's anything else to track down, just me getting confused- closing.

Not so fast. We appended (chromosome) and (supercontig) at some point to indicate what the sequence is, but it shouldn't be shown like that, it's NOT what the location actually is and creates an incorrect location on a cut and paste operation by the user. I'll move those out of the way. I realize it's cosmetic, but it's wrong cosmetics.

Done, in adbae22 it's now after the strand indication.
aesev.CIAT22838.gnm1.Unknown:2375348-2388727 (-1) (supercontig)

(FYI fixed URL, I had copied from my local dev.)


Error: Unsupported getParameter() option

I'm trying to use node-gles with three.js and get the following error:
Error: Unsupported getParameter() option
I guess get Parameter is not implemented yet?
When do you think it's realistic for me to try my three.js experiment again?

Looks like this is resolved from https://github.com/google/node-gles/issues/34 ?

No this is something else.

OK - I'll need some more details here. Stack trace, what was the argument passed into getParameter etc.

I ran in to the same problem trying to use this project with three.js
		var maxVertexTextures = gl.getParameter( gl.MAX_VERTEX_TEXTURE_IMAGE_UNITS );
		                           ^

Error: Unsupported getParameter() option
    at new WebGLCapabilities (/home/gaz/prog/native-threejs/node_modules/three/build/three.js:15851:30)
    at initGLContext (/home/gaz/prog/native-threejs/node_modules/three/build/three.js:23321:19)
    at new WebGLRenderer (/home/gaz/prog/native-threejs/node_modules/three/build/three.js:23370:3)
    at Object.<anonymous> (/home/gaz/prog/native-threejs/main.js:16:16)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)

at this location in the library: https://github.com/mrdoob/three.js/blob/c4b7b422b3f17900e530d27e3bbf0cdd76200cc1/src/renderers/webgl/WebGLCapabilities.js#L73
I commented out some to see which would work and
gl.MAX_VERTEX_TEXTURE_IMAGE_UNITS
gl.MAX_CUBE_MAP_TEXTURE_SIZE
gl.MAX_VERTEX_ATTRIBS
gl.MAX_VERTEX_UNIFORM_VECTORS
gl.MAX_VARYING_VECTORS
gl.MAX_FRAGMENT_UNIFORM_VECTORS

threw errors when called with gl.getParameter

Guessing they need to have cases here:
https://github.com/google/node-gles/blob/f1e76c5ec756449f1cde91891b6c6c8afc2e01be/binding/webgl_rendering_context.cc#L2539

Hi @AlexVestin yeah we can change that block to not fail if the param is not implemented. It should be as easy as matching up context values.
I'll try and take a look this week at this. I'm behind on getting a new build out.

Hi @nkreeger , is it still not supported for the parameters of extensions?
const ext = gl.getExtension('WEBGL_debug_renderer_info');
if (ext != null) {
  const rendererInfo = gl.getParameter(ext.UNMASKED_RENDERER_WEBGL);
  const vendorInfo = gl.getParameter(ext.UNMASKED_VENDOR_WEBGL);
  console.log(rendererInfo, vendorInfo);
}

Params are supported - however those params are not implemented:
https://github.com/google/node-gles/blob/master/binding/webgl_rendering_context.cc#L2625
That block can probably be changed to not throw an exception. For node-gles - I'd recommend asking renderer info like so:
https://github.com/google/node-gles/blob/master/src/tests/float_texture_upload_test.ts#L11

Hi @nkreeger . Unfortunately, there are some useful extension parameters that are not supported, such as EXT_texture_filter_anisotropic.
Could you extend the getParameter method more compatibly? It is very much appreciated.

I just published 0.0.16 which has a fix for exception throwing with implemented extension support - https://github.com/google/node-gles/pull/57
Thanks @rnconrad!


fix: allow query on columns with space in name

As sql server allows empty space in column name, make it adaptable to columns with space in its name.

This looks good. Do you think this would be straightforward to check in a test?

This looks good. Do you think this would be straightforward to check in a test?

Sure! Thank you very much~

I was bitten by this issue today when attempting to do a bulk insert to a table having columns with spaces in the name. Is this just waiting on a test to be merged? I can take a crack at it if @Geo-W isn't able to. The code should probably also be updated to handle the case where the column name contains the ] character, which would need to be escaped by doubling it.


Update openssl to 3.0.8 (downstream requirement)

Update openssl to 3.0.8 which addresses a HIGH level security issue and a couple of MEDIUM security issues.
Please refer to https://www.openssl.org/news/secadv/20230207.txt for further information.

[ ] provide pristine sources in downstream before merge


brew upgraded pinned python and python3 packages

I have python3 pinned in brew, but it was upgraded today upon a brew upgrade command because it was listed as a dependency of the httpie package. Looks like python2 was also upgraded despite being pinned.
I do not think this is the intended behavior.

: %; brew list --pinned
postgresql
python
python3
: %; brew update
Updated Homebrew from 91a225e to f518f57.
 If you
were using libvo-aacenc or libaacplus, both of which have been dropped in
FFmpeg 3.0, please consider switching to the native encoder (-c:a aac),
fdk-aac (-c:a libfdk_aac, ffmpeg needs to be installed with the
--with-fdk-aac option), or faac (-c:a libfaac, ffmpeg needs to be
installed with the --with-faac option).
See the announcement
https://ffmpeg.org/index.html#removing_external_aac_encoders for details,
and https://trac.ffmpeg.org/wiki/Encode/AAC on best practices of encoding
AAC with FFmpeg.
==> Summary
🍺  /usr/local/Cellar/ffmpeg/3.1.1: 236 files, 50.1M
==> Upgrading freetds
==> Downloading https://homebrew.bintray.com/bottles/freetds-1.00.9.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring freetds-1.00.9.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/freetds/1.00.9: 2,918 files, 20.4M
==> Upgrading httpie
==> Installing dependencies for httpie: python3
==> Installing httpie dependency: python3
==> Downloading https://homebrew.bintray.com/bottles/python3-3.5.2.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring python3-3.5.2.el_capitan.bottle.tar.gz
==> /usr/local/Cellar/python3/3.5.2/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python3/3.
==> /usr/local/Cellar/python3/3.5.2/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python3/3.
==> /usr/local/Cellar/python3/3.5.2/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python3/3.
==> Caveats
Pip, setuptools, and wheel have been installed. To update them
pip3 install --upgrade pip setuptools wheel
You can install Python packages with
pip3 install
They will install into the site-package directory
/usr/local/lib/python3.5/site-packages
See: https://github.com/Homebrew/brew/blob/master/share/doc/homebrew/Homebrew-and-Python.md
.app bundles were installed.
Run brew linkapps python3 to symlink these to /Applications.
==> Summary
🍺  /usr/local/Cellar/python3/3.5.2: 3,315 files, 50.6M
==> Installing httpie
==> Downloading https://homebrew.bintray.com/bottles/httpie-0.9.4.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring httpie-0.9.4.el_capitan.bottle.tar.gz
🍺  /usr/local/Cellar/httpie/0.9.4: 314 files, 4M
==> Upgrading vim
==> Installing dependencies for vim: python
==> Installing vim dependency: python
==> Downloading https://homebrew.bintray.com/bottles/python-2.7.12.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring python-2.7.12.el_capitan.bottle.tar.gz
==> /usr/local/Cellar/python/2.7.12/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=in
==> /usr/local/Cellar/python/2.7.12/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=in
==> /usr/local/Cellar/python/2.7.12/bin/python -s setup.py --no-user-cfg install --force --verbose --single-version-externally-managed --record=in
==> Caveats
Pip and setuptools have been installed. To update them
pip install --upgrade pip setuptools
You can install Python packages with
pip install
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://github.com/Homebrew/brew/blob/master/share/doc/homebrew/Homebrew-and-Python.md
.app bundles were installed.
Run brew linkapps python to symlink these to /Applications.
==> Summary
🍺  /usr/local/Cellar/python/2.7.12: 3,293 files, 44M
==> Installing vim
==> Downloading https://github.com/vim/vim/archive/v7.4.1967.tar.gz
==> Downloading from https://codeload.github.com/vim/vim/tar.gz/v7.4.1967
######################################################################## 100.0%
==> ./configure --prefix=/usr/local --mandir=/usr/local/Cellar/vim/7.4.1967/share/man --enable-multibyte --with-tlib=ncurses --enable-cscope --wit
==> make
==> make install prefix=/usr/local/Cellar/vim/7.4.1967 STRIP=true
🍺  /usr/local/Cellar/vim/7.4.1967: 1,691 files, 22.6M, built in 9 minutes 57 seconds
==> Upgrading youtube-dl
==> Downloading https://homebrew.bintray.com/bottles/youtube-dl-2016.07.05.el_capitan.bottle.tar.gz
######################################################################## 100.0%
==> Pouring youtube-dl-2016.07.05.el_capitan.bottle.tar.gz
==> Caveats
To use post-processing options, brew install ffmpeg or brew install libav.
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
zsh completion has been installed to:
/usr/local/share/zsh/site-functions
==> Summary
🍺  /usr/local/Cellar/youtube-dl/2016.07.05: 9 files, 1.5M
Removing: /usr/local/Cellar/ffmpeg/3.0.2... (230 files, 47M)
Removing: /usr/local/Cellar/ffmpeg/3.1... (236 files, 50.1M)
Removing: /usr/local/Cellar/freetds/1.00... (2,918 files, 20.5M)
Removing: /usr/local/Cellar/httpie/0.9.3... (290 files, 3.7M)
Removing: /usr/local/Cellar/python/2.7.11... (6,206 files, 83.3M)
Removing: /usr/local/Cellar/python3/3.4.3... (4,778 files, 72.8M)
Removing: /usr/local/Cellar/vim/7.4.1952... (1,691 files, 22.6M)
Removing: /usr/local/Cellar/youtube-dl/2016.06.27... (9 files, 1.5M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/ffmpeg-3.0.2.el_capitan.bottle.tar.gz... (18.2M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/ffmpeg-3.1.el_capitan.bottle.tar.gz... (19.7M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/freetds-1.00.el_capitan.bottle.tar.gz... (3.9M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/python-2.7.11.tgz... (16M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/vim-7.4.1952.tar.gz... (12M)
Removing: /Users/jspeno200/Library/Caches/Homebrew/youtube-dl-2016.06.27.el_capitan.bottle.tar.gz... (1.2M)
==> This operation has freed approximately 372.6M of disk space.

I'm sorry, but formula pinning really only works for leaf formulae that are not dependencies of other formulae. The pinning will be ignored if they are a dependency of a formula that is to be installed, because Homebrew is designed around the assumption that all formulae have been updated to their most recent version (thus an older installed version doesn't satisfy a dependency). If a dependency was left at an older version, things could quickly break in unexpected and hard to debug ways.

Thanks for the reply. Maybe we could get a change to the brew man page in the pin section that warns that this can happen. Thanks.

The documentation change would need to be made in cmd/pin.rb and then the man page needs to be regenerated by running brew man. A pull request with this caveat added would be appreciated!


Reconsider serializing block data into shares

Description
I wanted to bring up everyone's favorite topic again. Namely serialization.
First, I want to note that the way we arrange data into shares is essentially a custom serialzation format. It's a very light-weight and smart one and additionally uses (length delimited) protobuf for the hard parts (which IMO is a good choice).
Despite the theoretical simplicity, we ourselves are not having the best time implementing the interesting parts (correctly). Namely the splitting and merging of shares. This is not optimal because every client that does DA sampling or trying to download particular Txs directly from the p2p network, will have to do the merging part, too.
A custom serialzation format is an additional burden and hurdle for writing clients in other languages (e.g., Rust, python, Javascript, etc). IMO implementing that part should not take longer than a few hours, maybe a day in any language. We now have proof that this takes (much) longer even for those who either invented the format or at least are deeply familiar with it. Of course, we could blame this on go but the reality is that every other language also has its pitfalls and tradeoffs, too.
Proposal
I'm proposing to describe shares in protobuf too. The fields of the share proto message would be the namespace, the startIndex (optional) and the total length of the serlialized message, and finally, the part of the message in that particular share (in bytes) resulting from the splitting operation.
That way the spec really would just need to describe the split and merge algorithms and nothing else. We'd need to be careful that we do the splitting in a way that still results in equally sized protobuf messages.
I know, I've been kinda advocating against going all-in on protobuf for the shares myself. At least in the sense that I wanted to keep out any knowledge of the serialization format used out of the NMT (which IMO should simply take in raw bytes and nothing else). I'm not sure yet if the implication here would be that we'd need to move away from this property.
related: https://github.com/lazyledger/lazyledger-specs/issues/35
┆Issue is synchronized with this Asana task by Unito

That's an interesting idea, and definitely needs more investigation. Provided that we could get exact sized shares, I think it could be a good simplification measure. Although, I'm still not sure how much it would simplify things, as we're already mostly working with raw bytes, and would still have to manually split/merge.
I'm not the most familiar with protobuf, how would we get the namespace before the length delimiter?
We're already using protobuf for evidence, would we first serialize evidence, then create the shares, then serialize the shares?

we could blame this on go

Agreed.

the total length of the serlialized message, and finally

The problem with this suggestion is that this length does not repeat each share. So you'd still need at least 2 serialization passes: one to length-delimit the tx/message/whatever, then another to serialize the share struct after splitting.
The issue with using protobuf for this second step is that protobuf doesn't support fixed-length arrays.

we could blame this on go

Agreed 🦀

Good that you bring this up. We should likely try to implement the current scheme in Rust as it is the most interesting language we should target after go anyways. That said, I'd be surprised if the time to finish a fully tested implementation in Rust would be less than it took us to implement this in go (maybe it will actually be faster because we've ironed out a lot of the details now - but it will be the same ballpark).
I'm happy to write up splitting and merging using protobuf messages only and we can try to compare the understandability and ease or difficultness to implement that. But that's for later. Let revisit this issue after the first test-nets.

not planned any time soon


⚠ Bofrost.at has degraded performance

In 7e3daa9, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2120 ms

Resolved: Bofrost.at performance has improved in a989fd4 after 6 minutes.


Question: basemodel DbLib.py

Consider this line in DbLib.py
self.layers = [DlibMetaData()]

What exactly is this for ? (member layers is not accessed anywhere)

In earlier versions of deepface, we were grabbing target size from .layers.input_shape. Keras models have this argument but Dlib does not have. That is why, I added it but it is not being used anymore.
Same thing is available in SFace, too: https://github.com/serengil/deepface/blob/dea697c6ec5b89375cdbc376aa680a10f4231c42/deepface/basemodels/SFace.py#L84
TLDR: not being used anymore, can be deleted.

Thank you.


Function call is messing up with intersection types

Check out line 38 of the following example: https://goo.gl/AdmME5. Does anyone know why it's failing with property 'size'. Property cannot be accessed on any member of intersection type... instead of pointing out first that the second argument passed to mapPropToClass is incorrect?

Related: https://github.com/facebook/flow/issues/4586

I think the linked issue explains this reasonably well. This is expected though confusing behavior. I’m going to close this issue since it’s not a bug, but am happy to help with adjusting that code to work.

Returning a misleading, unhelpful error message to the user does sound a bit like a bug, though !

I definitely agree with @dperetti.

This is now improved in Flow 0.55! I had to remove the JSX from the example because Try Flow currently isn't working with JSX. After doing that, the new error message is:
33:     const { size, tracking, ...otherProps } = this.props;
                ^ property `size`. Property cannot be accessed on any member of intersection type
33:     const { size, tracking, ...otherProps } = this.props;
                                                  ^ intersection
Member 1:
26: type ColProps = BoxProps & {
                    ^ BoxPropsError:
33:     const { size, tracking, ...otherProps } = this.props;
                ^ property `size`. Property not found in
26: type ColProps = BoxProps & {
                    ^ object type
Member 2:
26: type ColProps = BoxProps & {                               ^ object typeError:
38: 	mapPropToClass('size', size);
                            ^ number. This type is incompatible with the expected param type of
8: function mapPropToClass(key: string, prop: Object) {
                                              ^ object type

Which helps us reconstruct Flow's reasoning:

there is no size in BoxProps
so we instead look in the object type it intersects
and see a type error in the argument passed to the function.

I think this makes stuff a lot clearer and so this issue is fixed. WDYT, @eliseumds?

@asolove-stripe sweet! That's way more useful, thanks =)

38: 	mapPropToClass('size', size);
                            ^ number. This type is incompatible with the expected param type of

This part of the error seems bogus, right? Like the error position is way off.


fix: compatible for no config

Checklist

[x] npm test passes
[x] tests and/or benchmarks are included
[ ] documentation is changed or added
[x] commit message follows commit guidelines

Affected core subsystem(s)

Description of change

当 config 目录不存在时，scan 不报错

Codecov Report
Base: 92.42% // Head: 92.66% // Increases project coverage by +0.24% :tada:

Coverage data is based on head (3ae8ef5) compared to base (02e6a28).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #222      +/-   ##
==========================================
+ Coverage   92.42%   92.66%   +0.24%
==========================================
  Files          55       55
  Lines        1294     1296       +2
  Branches      216      218       +2
==========================================
+ Hits         1196     1201       +5
+ Misses         98       95       -3

Impacted Files
Coverage Δ

src/scanner/scan.ts
95.94% <100.00%> (+1.42%)
:arrow_up:

src/utils/fs.ts
100.00% <0.00%> (+10.00%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[FEATURE] Add interface for configurable rule

@andreaswolf Btw. that´s the reason why the guard clause https://github.com/andreaswolf/fractor/blob/main/fractor-doc-generator/src/Printer/CodeSamplePrinter.php#L54 exists

@andreaswolf Btw. that´s the reason why the guard clause https://github.com/andreaswolf/fractor/blob/main/fractor-doc-generator/src/Printer/CodeSamplePrinter.php#L54 exists

That part is still a mystery to me, but I guess it's because I've never seen an example in action. Once I (think I) understood it, I'll add documentation where appropriate :)


[CARBONDATA-2950]alter add column of hive table fails  from carbon for spark versions above 2.1

Problem:
spark does not support add columns in spark-2.1, but it is supported in 2.2 and above
when add column is fired for hive table in carbon session, for spark -version above 2.1, it throws error as unsupported operation on hive table
Solution:
when alter add columns for hive is fired for spark-2.2 and above, it should not throw any exception and it should pass
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

[x] Any interfaces changed?
NA

[x] Any backward compatibility impacted?
NA

[x] Document update required?
NA

[x] Testing done
Test case is added
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.

[x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

Build Failed  with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/362/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/363/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8610/

Build Failed with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/540/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/367/

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/372/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8614/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/544/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8619/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/549/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/394/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/571/

Build Failed  with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8641/

retest this please

Build Success with Spark 2.1.0, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder2.1/397/

Build Success with Spark 2.3.1, Please check CI http://<IP_ADDRESS>:8080/job/carbondataprbuilder2.3/8645/

Build Success with Spark 2.2.1, Please check CI http://<IP_ADDRESS>:8080/job/ApacheCarbonPRBuilder1/575/


ciphers issue with master 2019-05-07

With current master PKIX-SSH test try-ciphers pass all tests with
"test try ciphers: cipher 3des-cbc mac mac-"
, but hang on
"test try ciphers: cipher aes128-cbc mac hmac-sha1"
Hint: see issue #8565 for details how to configure and run test. Test is the same but error is new one.
10x for fixing previous one.

So uhmmm, which script is it that generates this output?

Richard Levitte wrote:

So uhmmm, which script is it that generates this output?

Test is the same as mentioned in #8565 . Script is regress/try-ciphers.sh .
It could be run with command:
make -e LTESTS=try-ciphers REGRESS_TARGETS=t-exec tests
Definitely commit
https://github.com/openssl/openssl/commit/7556b9df597ce43c1c31b294512d5146560f37c6
( issue #8604 ) resolves previous report.
So something after 3 Apr introduce new breakage.
Regards,
Roumen

Ping?

test past  with commit f2dbb71cb6bf6cd570f96e7663a22dd87854f08b / Richard<EMAIL_ADDRESS>/  "providers/common/digests/sha2.c: forward declare all dispatched functions"
then there is series of 8 commits with author Matt<EMAIL_ADDRESS>related to AES and default provider that end with commit 819a7ae9fc7721f675757c0925821f91b20dfc8f
=> no surprise "move to providers adds regression"

Is provider functionality stateless?

Two months, no fix?
Dunno what to think about quality of some "enhancements". I guess that is clear that I'm disappointed from constant degradation of project quality. It seems to me team members spend time for non significant (cosmetic) or valueless updates.
Commit like this one 718b133a5328108099ecac0bf40d8fd4886e7b64 (Implement AES CBC ciphers in the default provider) changes management of initialization vector.
There is one secure shell functionality  - privilege separation (please search internet for details). One unprivileged process start encrypted communication and when authentication process finish deserialize its "state". In mean time is started another "session" process that serialize "state" and continue with management of user session.
Actually above mentioned commit breaks "deserialization" of cipher.

Rest case - sample from https://wiki.openssl.org/index.php/EVP_Symmetric_Encryption_and_Decryption plus
--- Evp-symmetric-encrypt.c     2019-07-03 14:59:09.000000000 +0300
+++ Evp-symmetric-encrypt2.c    2019-07-08 20:24:31.686803256 +0300
@@ -107,6 +107,12 @@
         handleErrors();
     ciphertext_len += len;

+{   /* broken by https://github.com/openssl/openssl/commit/718b133a5328108099ecac0bf40d8fd4886e7b64 */
+    int iv_len = EVP_CIPHER_CTX_iv_length(ctx);
+    fprintf(stderr, "iv[%d]:\n", iv_len);
+    BIO_dump_fp (stderr, EVP_CIPHER_CTX_iv(ctx), iv_len);
+}
+
     /* Clean up */
     EVP_CIPHER_CTX_free(ctx);

Expected output:
$ ./Evp-symmetric-encrypt2
iv[16]:
0000 - 2c 8e 65 a9 92 36 ec 92-07 04 91 5c f1 a9 8a 44   ,.e..6........D
Ciphertext is:
0000 - e0 6f 63 a7 11 e8 b7 aa-9f 94 40 10 7d 46 80 a1<EMAIL_ADDRESS>0010 - 17 99 43 80 ea 31 d2 a2-99 b9 53 02 d4 39 b9 70   ..C..1....S..9.p
0020 - 2c 8e 65 a9 92 36 ec 92-07 04 91 5c f1 a9 8a 44   ,.e..6........D
Decrypted text is:
The quick brown fox jumps over the lazy dog

P.S. current for iv is :
0000 - 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00   ................

There's a lot of new development happening in master right now, so it's not a big surprise that it isn't always as stable as one might want, and that we miss corner cases.  Thanks for catching them!
I'm a little concerned, though, that we don't already catch this with the current EVP tests.  Are there interesting test cases we should add?  That could be subject for a PR.

Commit like this one 718b133 (Implement AES CBC ciphers in the default provider) changes management of initialization vector.

I'll see if I can figure that out.  The wiki example breaks, you say?

Is provider functionality stateless?

Yes and no.  It's actually up to the provider...  it can give back a provider context on init, and hold state in that context.  Every set of dispatch function also has some kind of context creator, where it can hold state for a run of the algorithm...  this is the case for ciphers, digests, etc (obviously, or the init/update/final sequence would simply be impossible).

Ah, yeah, I see what's going on there...  the changes aren't quite complete.

(digests in providers are similarly broken for the moment, I might add...  I've some work going on to clear that up, but have a few other items on my plate as well...)

Richard Levitte wrote:

[SNIP]
I'm a little concerned, though, that we don't already catch this with the current EVP tests.  Are there interesting test cases we should add?  That could be subject for a PR.
based on previous
There is test that uses test/recipes/30-test_evp_data/evpciph.txt

Commit like this one 718b133 (Implement AES CBC ciphers in the default provider) changes management of initialization vector.

I'll see if I can figure that out.  The wiki example breaks, you say?

Wiki sample work fine. It was most simple way to show issue.

Existing test could be changed to check for IV after initialization
and/or encryption.
After "move to provides" EVP_CIPHER_CTX_iv(...) always return zeroes.
Roumen

#9324 adds a very simple test for this.

Richard Levitte wrote:

#9324 adds a very simple test for this.

Yep, initialization test is enough for now.
I wonder why move to providers is activated. In the past there was
"EXPERIMENTAL" functionality. I'm concerned by TODO(3.0) comment. So it
seems to me this development fails into this category but is not marked
as experimental :(.
Roumen

Issue is now superseded from other, so closing.


Launch OvenSpace (Advanced)

Hi!
I am trying to run ovenspace with TLS certificates and Nginx but the OvenmediaEngine server returns the following error:
[[2022-03-09 17:20:30.157] E [SPAPIServer-T80:1916] OpenSSL | tls.cpp:193  | An error occurred while accept SSL connection: [OpenSSL] error:0A000418:SSL routines::tlsv1 alert unknown ca (167773208)
[2022-03-09 17:20:30.158] E [SPAPIServer-T80:1916] OpenSSL | tls_server_data.cpp:108  | An error occurred while accept TLS connection: error code: 1]
I detail below the steps followed and the configuration set up.
NGINX CONFIGURATION
First I set up the recommended NGINX configuration and restart NGINX.
server {
    listen       443 ssl;
    server_name  my_domain;

    ssl_certificate /your/path/to/my_cert_file.crt;
    ssl_certificate_key /your/path/to/my_key_file.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_pass http://<IP_ADDRESS>:5000;
    }

    location /socket.io {
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass http://<IP_ADDRESS>:5000/socket.io;
    }
}

server {
    listen       80;
    server_name  my_domain;

    rewrite ^(.*) https://my_domain$1 permanent;
}

I generated the self-signed certificates with openssl for the machine, where server_name is the name of the machine.
OVENMEDIAENGINE CONFIGURATION
After that I make OvenMediaEngine's REST API Server, WebRTC Provider, and WebRTC Publisher work with TLS and restart OvenMediaEngine.
<Managers>
		<Host>
			<Names>
				<Name>*</Name>
			</Names>

			<TLS>
				<CertPath>path/to/my_cert_file.crt</CertPath>
				<KeyPath>path/to/my_cert_file.crt</KeyPath>
				<!-- <ChainCertPath>airensoft_com_chain.crt</ChainCertPath> -->
			</TLS>
		</Host>
		<API>
			<AccessToken>my_access_token</AccessToken>
		</API>

</Managers>

<Domain>
		<Names>
			<Name>*</Name>
		</Names>
		<TLS>
			<CertPath>path/to/my_cert_file.crt</CertPath>
			<KeyPath>path/to/my_key_file.key</KeyPath>
			<!-- <ChainCertPath>path/to/file.crt</ChainCertPath> -->
		</TLS>
</Domain>

Finally, I set up the OvenSpace configuration and run the .py file.
OVENSPACE CONFIGURATION
Finally, I set up the OvenSpace configuration and run the .py file.
OME_HOST = 'machine_name'

OME_API_ENABLE_TLS = 'true'
OME_API_PORT = '8082'
OME_API_ACCESS_TOKEN = 'access-token'

OME_VHOST_NAME = 'default'
OME_APP_NAME = 'app'
OME_STREAM_NAME = 'stream'

OME_WEBRTC_PROVIDER_ENABLE_TLS = 'true'
OME_WEBRTC_PROVIDER_PORT = '8333'

OME_WEBRTC_PUBLISHER_ENABLE_TLS = 'true'
OME_WEBRTC_PUBLISHER_PORT = '8333'

BROWSER CONSOLE
When I access the address 'https://machine_name' from the browser, it returns the following error in the browser console:
[/getStreams:1          Failed to load resource: the server responded with a status of 500 (INTERNAL SERVER ERROR)
ovenspace.js:486 Could not get streams from OME.]
OVENSPACE LOGS
The OvenSpace logs are follows:

Restarting with stat
Debugger is active!
Debugger PIN: 327-571-840
(1969) wsgi starting up on http://<IP_ADDRESS>:5000
(1969) accepted ('<IP_ADDRESS>', 36050)
<IP_ADDRESS> - - [09/Mar/2022 17:19:59] "GET / HTTP/1.0" 200 7553 0.006798
(1969) accepted ('<IP_ADDRESS>', 36052)
<IP_ADDRESS> - - [09/Mar/2022 17:19:59] "GET /static/css/ovenspace.css HTTP/1.0" 30                                                     4 163 0.002619
(1969) accepted ('<IP_ADDRESS>', 36054)
<IP_ADDRESS> - - [09/Mar/2022 17:19:59] "GET /static/img/ovenspace_bi.svg HTTP/1.0"                                                      304 166 0.001761
(1969) accepted ('<IP_ADDRESS>', 36056)
<IP_ADDRESS> - - [09/Mar/2022 17:19:59] "GET /static/js/ovenspace.js HTTP/1.0" 304                                                      162 0.001767
(1969) accepted ('<IP_ADDRESS>', 36058)
<IP_ADDRESS> - - [09/Mar/2022 17:19:59] "GET /favicon.ico HTTP/1.0" 404 375 0.00051                                                     0
(1969) accepted ('<IP_ADDRESS>', 36060)
(1969) accepted ('<IP_ADDRESS>', 36062)
<IP_ADDRESS> - - [09/Mar/2022 17:20:02] "GET /getStreams HTTP/1.0" 500 443 0.027744
(1969) accepted ('<IP_ADDRESS>', 36064)
<IP_ADDRESS> - - [09/Mar/2022 17:20:02] "GET /getStreams HTTP/1.0" 500 443 0.011921
(1969) accepted ('<IP_ADDRESS>', 36066)
<IP_ADDRESS> - - [09/Mar/2022 17:20:02] "GET /static/img/ovenspace.ico HTTP/1.0" 30                                                     4 163 0.001610
(1969) accepted ('<IP_ADDRESS>', 36068)
<IP_ADDRESS> - - [09/Mar/2022 17:20:05] "GET /getStreams HTTP/1.0" 500 443 0.015352
(1969) accepted ('<IP_ADDRESS>', 36070)
<IP_ADDRESS> - - [09/Mar/2022 17:20:07] "GET /static/img/ovenspace.ico HTTP/1.0" 30                                                     4 163 0.001209
(1969) accepted ('<IP_ADDRESS>', 36072)
<IP_ADDRESS> - - [09/Mar/2022 17:20:07] "GET /getStreams HTTP/1.0" 500 443 0.012447
(1969) accepted ('<IP_ADDRESS>', 36074)
<IP_ADDRESS> - - [09/Mar/2022 17:20:10] "GET /getStreams HTTP/1.0" 500 443 0.012753
(1969) accepted ('<IP_ADDRESS>', 36076)
<IP_ADDRESS> - - [09/Mar/2022 17:20:12] "GET /getStreams HTTP/1.0" 500 443 0.015085
(1969) accepted ('<IP_ADDRESS>', 36078)
<IP_ADDRESS> - - [09/Mar/2022 17:20:14] "GET /static/css/ovenspace.css HTTP/1.0" 30                                                     4 163 0.001055
(1969) accepted ('<IP_ADDRESS>', 36080)
<IP_ADDRESS> - - [09/Mar/2022 17:20:15] "GET /getStreams HTTP/1.0" 500 443 0.012835
(1969) accepted ('<IP_ADDRESS>', 36084)
<IP_ADDRESS> - - [09/Mar/2022 17:20:17] "GET /getStreams HTTP/1.0" 500 443 0.014948
(1969) accepted ('<IP_ADDRESS>', 36086)
<IP_ADDRESS> - - [09/Mar/2022 17:20:20] "GET /getStreams HTTP/1.0" 500 443 0.011937
(1969) accepted ('<IP_ADDRESS>', 36088)
<IP_ADDRESS> - - [09/Mar/2022 17:20:22] "GET /getStreams HTTP/1.0" 500 443 0.013110
(1969) accepted ('<IP_ADDRESS>', 36090)
<IP_ADDRESS> - - [09/Mar/2022 17:20:25] "GET /getStreams HTTP/1.0" 500 443 0.012097
(1969) accepted ('<IP_ADDRESS>', 36092)
<IP_ADDRESS> - - [09/Mar/2022 17:20:27] "GET /getStreams HTTP/1.0" 500 443 0.011840
(1969) accepted ('<IP_ADDRESS>', 36094)
<IP_ADDRESS> - - [09/Mar/2022 17:20:30] "GET /getStreams HTTP/1.0" 500 443 0.015287
<IP_ADDRESS> - - [09/Mar/2022 17:20:31] "GET /socket.io/?EIO=4&transport=websocket                                                      HTTP/1.1" 200 0 28.837320

OVENMEDIAENGINE LOGS
Furthermore, in the OvenMediaEngine logs, I get the error mentioned at the beginning of this message:
[[2022-03-09 17:20:30.157] E [SPAPIServer-T80:1916] OpenSSL | tls.cpp:193  | An error occurred while accept SSL connection: [OpenSSL] error:0A000418:SSL routines::tlsv1 alert unknown ca (167773208)
[2022-03-09 17:20:30.158] E [SPAPIServer-T80:1916] OpenSSL | tls_server_data.cpp:108  | An error occurred while accept TLS connection: error code: 1]

@LydiaTM Hi. Thank you for using OvenSpace.
You need to set up chain cert in OME. Although it is not possible to confirm the structure of the certificate you have set, please set ROOT CA and Intermediate CA separately as chain cert.
If you have any problems, detailed support is available in the OvenMediaEngine repository.
<Managers>
		<Host>
			<Names>
				<Name>*</Name>
			</Names>

			<TLS>
				<CertPath>path/to/my_cert_file.crt</CertPath>
				<KeyPath>path/to/my_cert_file.crt</KeyPath>
				<!-- <ChainCertPath>airensoft_com_chain.crt</ChainCertPath> -->
			</TLS>
		</Host>
		<API>
			<AccessToken>my_access_token</AccessToken>
		</API>

</Managers>

<Domain>
		<Names>
			<Name>*</Name>
		</Names>
		<TLS>
			<CertPath>path/to/my_cert_file.crt</CertPath>
			<KeyPath>path/to/my_key_file.key</KeyPath>
			<!-- <ChainCertPath>path/to/file.crt</ChainCertPath> -->
		</TLS>
</Domain>

Thanks for your help!
I am new to the certificates issue and I don't know how to generate the intermedia certificate. Could you help me, please?
Thanks in advance.

@LydiaTM Hi. It will depend on the type of certificate issued, but this may be your case.
The certificate you set in nginx may have the following structure. (ssl_certificate /your/path/to/my_cert_file.crt;)
-----BEGIN CERTIFICATE-----
Your Domain CA
-----END CERTIFICATE-----
Intermediate CA
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
ROOT CA
-----END CERTIFICATE-----

This is chain cert. You can set this cert to <ChainCertPath>path/to/file.crt</ChainCertPath>.
And you can make a new file something like my-cert.crt and paste first section of chain cert.
-----BEGIN CERTIFICATE-----
Your Domain CA
-----END CERTIFICATE-----

And set this cert file to <CertPath>path/to/my_cert_file.crt</CertPath>
I hope this helps. As mentioned earlier, there are various methods depending on the type of certificate issued. I can only help up to the point above.


LSTM Model causing Error: "Type INT32 (2) not supported" on trying to run

Similar to this issue.
I've been trying to get an LSTM Model running on my ESP 32 (Adafruit HUZZAH32 – ESP32 Feather Board) for some time, however have been completely stumped by this issue. The complete set of errors that occurs is:
Type INT32 (2) not supported.
Node ADD (number 4) failed to invoke with status 1
Node WHILE (number 2) failed to invoke with status 1

Which happens when calling interpreter->Invoke();
The code that I am working with is my own, but heavily based off of the micro_speech example. To avoid as much fuss as possible I am using an AllOpsResolver, so everything should be loaded, it just seems to be a type issue. I do not understand why it is occurring as I have quantised my model to int8 and am providing int8 data. I have even confirmed the model's own inputs think they are receiving int8 as opposed to int32, outputting input_model->type and output_model->type = 9, which according to TfLiteType refers to int8 as opposed to the 2 of int32. Looking at the model in Netron also shows the inputs and outputs are int8 as opposed to int32.
All code, for both my model and directory are here, key files linked in .zip file.
srcFile+Model.zip
The code itself does run, however the predictions are simply the first 10 inputs. The model expects a 20x8 Input array (flattened to a 160-length 1D array), and outputs a 1x10 array of time series predictions.
I am unsure if this error is as a result of my ESP32 Code or my model conversion process, help would be appreciated.

Hi @ts532 this is not ESP32 specific. It's more to do with tflm does not support the OPs or OPs for the specific data types for which errors are produced.
Although, you have converted with options as input/output INT8, I have seen that the conversion still ends up using INT32 ops for certain OP versions, specifically when LSTMs are used.
Please find one more similar discussion here: https://github.com/espressif/tflite-micro-esp-examples/issues/30#issuecomment-1455970454
As a remedy, you can manually add specific versions as I have done in the above patch.
You may also want to raise an issue or feature request here. Hope this helps.

Hi there, thanks for the patch. Unfortunately despite applying its contents the error persists. To my understanding the Patch requires:
in tflite-lib/tensorflow/lite/micro/kernels/cast.cc

add case kTfLiteUInt8 within switch(input->type) just below case kTFLiteUInt8 on ~line 87.
in tflite-lib/tensorflow/lite/micro/kernels/floor_div.cc

add case kTfLiteInt32 within switch(input1->type) just above case kTfLiteFloat32 on ~line 113
in tflite-lib/tensorflow/lite/micro/kernels/sub.cc

add new case statement to switch(output->type) just above its default statement from ~line 129-149
case kTfLiteInt32: //patch start
 {
   if (need_broadcast) {
     tflite::reference_ops::BroadcastQuantSubSlow(
         op_params, tflite::micro::GetTensorShape(input1),
         tflite::micro::GetTensorData<int32_t>(input1),
         tflite::micro::GetTensorShape(input2),
         tflite::micro::GetTensorData<int32_t>(input2),
         tflite::micro::GetTensorShape(output),
         tflite::micro::GetTensorData<int32_t>(output));
   } else {
     tflite::reference_ops::Sub(
         op_params, tflite::micro::GetTensorShape(input1),
         tflite::micro::GetTensorData<int32_t>(input1),
         tflite::micro::GetTensorShape(input2),
         tflite::micro::GetTensorData<int32_t>(input2),
         tflite::micro::GetTensorShape(output),
         tflite::micro::GetTensorData<int32_t>(output));
   }
   break;
 } //patch end

replace if (output->type == kTfLiteFloat32) with if (output->type == kTfLiteFloat32 || output->type == kTfLiteInt32) on ~line 171

As these edits pertained to sub.cc, as opposed to add.cc, which seems to be the source of my errors, I attempted to modify the changes made to sub.cc to work with add.cc, this resulted in the program now crashing rather than brings up errors. The Changes I made were:
in tflite-lib/tensorflow/lite/micro/kernels/add.cc

add new case statement to switch (output->type) from ~line 129 to 149

case kTfLiteInt32: //patch start
    {
      if (need_broadcast) {
        tflite::reference_ops::BroadcastQuantSubSlow(
            op_params, tflite::micro::GetTensorShape(input1),
            tflite::micro::GetTensorData<int32_t>(input1),
            tflite::micro::GetTensorShape(input2),
            tflite::micro::GetTensorData<int32_t>(input2),
            tflite::micro::GetTensorShape(output),
            tflite::micro::GetTensorData<int32_t>(output));
      } else {
        tflite::reference_ops::Sub(
            op_params, tflite::micro::GetTensorShape(input1),
            tflite::micro::GetTensorData<int32_t>(input1),
            tflite::micro::GetTensorShape(input2),
            tflite::micro::GetTensorData<int32_t>(input2),
            tflite::micro::GetTensorShape(output),
            tflite::micro::GetTensorData<int32_t>(output));
      }
      break;
    } //patch end

replace if (output->type == kTfLiteFloat32) with if (output->type == kTfLiteFloat32 || output->type == kTfLiteInt32) on ~line 171.
This edit results in the following backtrace error occurring:
Backtrace:0x40082f4d:0x3ffb0e500x40087c09:0x3ffb0e70 0x4008c805:0x3ffb0e90 0x400d9e73:0x3ffb0f10 0x400da311:0x3ffb0f30 0x400da4d7:0x3ffb0f80 0x400dab8a:0x3ffb1100 0x400fc65e:0x3ffb1120 0x400f70b3:0x3ffb1160 0x400fc65e:0x3ffb1180 0x400f79b6:0x3ffb11c0 0x400d332a:0x3ffb11e0 0x400fee76:0x3ffb2820

when decoded:
0x40082f4d: panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c line 402
0x40087c09: esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_system.c line 128
0x4008c805: abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/abort.c line 46
0x400d9e53: tflite::MatchingDim(tflite::RuntimeShape const&, int, tflite::RuntimeShape const&, int) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/kernels/internal/types.h line 265
0x400da2f1: tflite::reference_ops::Concatenation (tflite::ConcatenationParams const&, tflite::RuntimeShape const* const*, signed char const* const*, tflite::RuntimeShape const&, signed char*) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/kernels/internal/reference/concatenation.h line 43
0x400da4b7: tflite::(anonymous namespace)::EvalUnquantized (TfLiteContext*, TfLiteNode*) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/kernels/kernel_util.h line 64
0x400dab6a: tflite::(anonymous namespace)::Eval(TfLiteContext*, TfLiteNode*) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/kernels/concatenation.cc line 232
0x400fc61a: tflite::MicroGraph::InvokeSubgraph(int) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/micro_graph.cc line 174
0x400f706f: tflite::(anonymous namespace)::Eval(TfLiteContext*, TfLiteNode*) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/kernels/while.cc line 107
0x400fc61a: tflite::MicroGraph::InvokeSubgraph(int) at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/micro_graph.cc line 174
0x400f7972: tflite::MicroInterpreter::Invoke() at Z:/My Files/PhDStuff/ESPArduino2/lib/tflite-lib/tensorflow/lite/micro/micro_interpreter.cc line 286
0x400d331a: setup() at Z:/My Files/PhDStuff/ESPArduino2/src/main.cpp line 88
0x400fee32: loopTask(void*) at C:/Users/tms40/.platformio/packages/framework-arduinoespressif32/cores/esp32/main.cpp line 42

Is there any advice on a next step?

Update: a potential solution to this error can be found here (TF-Micro)

Closing this as the OP Add INT32 is now available with https://github.com/tensorflow/tflite-micro/pull/1847


feat(storage-viewer): added support for local anime and manga

I see @jmir1 has already implemented this

@logickoder you can still add/fix one thing if you want. hidden categories aren't in list but when you use "ALL" category it includes hidden category stuff. so maybe it's better to hide those entries. and maybe rename it here "hide hidden categories from categories screen & all storage list"
also maybe make it remember last opened category in storage as well.
if you want I can open issue instead.

Yeah, an issue will be better


since alleles contain undefined nucleotides

singularity exec ~/software/pangenie/container/pangenie.sif  PanGenie-index -v keepA_C_0.5missing_100kbub_phased_deleteN_id_trim_filtered_SVs.vcf -r ZS11.genome.fa  -t 24  -o PanGenie
My vcf sequence contains sequences with bases N, and I get an error when building index.

PanGenie skips variants with missing sequence information, since it is based on kmers. However, it should just give a warning instead of an error and proceed running. Can you please provide more details?


Added stephlow.audio

Ahh... webrings, that takes me back, really cool idea and awesome sites!

Incomplete PR


Validate on the test set

https://github.com/Audio-WestlakeU/ATST-SED/blob/0ac8073b1419fde8cb14116c744b23a82fedff36/train/train_stage2.py#L213
I am reproducing the result of the ATST-SED. When I was doing stage 2, I realized that the test set already leaked into the valid set.
Is this designed as proposed? I cannot find this part of the explanation in the paper. Did you also keep this setting of train/valid/test split into the baseline BEATs model, otherwise I doubt whether the improvement is from it.

I see. The obj_metric is independent of the test_dataset here because of the mask. Thank you for the quickly explanation.


Feature Request: PTH

Not sure if its possible, but it would be awesome to have the ability to use NTLM hashes (PTH) instead of only plaintext passwords.

Aw man why you gotta nerd snipe me like that? Now i'm totally going to start looking up how to do PTH in Golang. No promises, but great idea

This is an AWESOME project though.  10/10 fantastic

v0.2 released with PtH support! Lmk if it’s working for you (I’ve tested in my lab) and if so I’ll close this out

This works for me!


is quorum read deprecated in etcd v3?

when a write/update op finished, we want to read the newest value for etcd immediately.
we could set quorum read option in etcd v2 client, but seems no way to set in etcd v3 client. is there any other way to do this, or is deprecated?

It's enabled by default. To get stale reads use clientv3.WithSerialize().

thanks @heyitsanthony !

It wasn't obvious from google searches and I wanted to note this here as this is where I found my answer (and maybe Google will index it better now). If you want to read from etcd when you are a node in the minority after a network partition, you will get this warning by default:
etcdserver: timed out waiting for read index response
...and your read will fail. If you don't mind getting stale read results, you can use WithSerializable and still read, even when your node is a member of the minority.

@xiang90 should we add this in faq?

@heyitsanthony Add a faq about read consistency?


chore: memory leakage for application lifecycle

alternate implementation for memory leakage.

What other benefits do you see for this approach besides using StateFlow instead of caching by ourselves?
Asking because I had considered this approach already but didn't go for it because:

Weak reference of activity doesn't guarantee that listeners will always execute their action completely because activity can be null sometimes. Even though the chance is very rare for recently emitted values, the possibility is there, making debugging difficult
None of our listeners hold activity, so they cannot leak context despite strong reference
Both our listeners today require activity to perform desired actions. Having no activity makes listeners useless despite them observing, filtering and collecting all values

My reasoning for this approach:

We are not relying on custom implementation of state flow, we are using the built in one. Which you mentioned too.
Activity should never be null it would be only null which it has been garbage collected, If the activity has been garbage collected then, it would have been null with another variable holding that value as well
Similarly, if activity is garbage collected, none of our listener would/should require it either. I can't think of a case, where activity needs to be garbage collected and our callback would need to require it. If you can think of one, please let me know.
Regardless of weak reference, like you said, SharedFlow (previous implementation) wasn't holding the strong reference to activity, if the activity was to be garbage collect, what would have happened then in case of our callback?


start of error codes

next steps:

[x] rename BaseLBRYException to something like "ErrorCodeException". call it based on what it does. don't use "LBRY" in name
[ ] get all exception codes into a single place where it's clear what they all are. maybe group them smartly. then generate exception classes from the list. do it like https://github.com/lbryio/lbry-sdk/wiki/Error-Codes but in code

replaced by #2611


CRC code calculation

Nice work, I was waiting for it :)
This is not a bug, but a (hopefully helpful) contribution.
Looking at crccalc it turns out that the CRC algorithm used might be CRC-16/AUG-CCITT. For parameters see there.
The byte sequence for turn_on, without the last 2 CRC bytes (0d07840f0201) yields 5512 !
In case you need a clean CRC lib, here is one: https://github.com/latysheff/node-polycrc .
Maybe this helps with some of your ongoing work :) :)
Regards, Steffen

Thank you so much @sfrieske this looks great! I will try this out asap 🙏

Thank you so much @sfrieske this looks great! I will try this out asap 🙏


latest PSPKI integration test request

I'm the author of PSPKI module which is used by PSPKIAudit. For past time I've actively worked on bug fixing in PSPKI and would like to execute integration tests with your module to ensure that the existing behavior is not broken. For example, I've fixed this issue: https://github.com/GhostPack/PSPKIAudit/issues/14 and other bugs related to certificate templates.

Thank you @Crypt32 and please excuse the delay getting this merged in.  Thanks to @PatrickOnGit work in #24, PSPKIAudit now pulls in PSPKI from the PSGallery. Thanks again for all your great work on PSPKI!

Actually, just to check @Crypt32, have these changes all been merged into the version of PSPKI on the PSGallery? I noticed that the latest version (3.7.2) was pushed in 2020

Actually, just to check @Crypt32, have these changes all been merged into the version of PSPKI on the PSGallery? I noticed that the latest version (3.7.2) was pushed in 2020

They are merged in sources, but not yet available in PSGallery. This year, I'm planning a major upgrade of underlying PKI API library which changes type namespaces so I would test it with PSPKIAudit before and make your tool compatible with new PSPKI version.

Initial report from my peer was they were getting a "get-certificationauthority is not recognized"

This suggests that the target machine doesn't have AD CS RSAT installed. PSPKI performs RSAT checking and if missing, dependent commands are not loaded.

There was also an issue where, if PSPKI was NOT installed, and users imported the PSPKIAudit.psm1, it would import successfully but Get-CertificationAuthority would fail since PSPKI was not installed. With the latest installation instructions specifying to import he PS manifest file (the psd1), this should no longer be an issue.
To your point @Crypt32, we should probably add a check for the RSAT modules to the psm1 still.

Thanks. I tried the new version / instructions in a lab and a production environment. I do get PSPKI 4.0.0 now, the PSPKIAudit module installs and everything I've looked at appears to be working as expected.

Great! Thank you for confirming!


Not Ok for storage

I saw that @braveKenny was successful in getting images stored and retrieved in "Best/Correct way to store images in public dirrectory and get their url in the callback?"
That's awesome!
Unfortunately, when I attempt to add the to the user's profile.profileImage I get
"Meteor.makeErrorType.errorClass {error: 409, reason: "MongoError: not okForStorage", details: undefined, message: "MongoError: not okForStorage [409]", errorType: "Meteor.Error"…}details: undefinederror: 409errorType: "Meteor.Error"message: "MongoError: not okForStorage [409]"

My fileObj look like the following. It has references to dataman and such in there. A bit more complex then I expected. I did make sure to add cfs-ejson-file

I made sure the user has a profile and profile.profileImage property.
'change .fileUploader': function(event, template) {
      FS.Utility.eachFile(event, function(file) {
        var uId = Meteor.userId();
        if (uId) {
          var fileObj = Images.insert(file);
          Meteor.users.update({
            _id: uId
          }, {
            $set: {
              'profile.profileImage': fileObj
            }
          }, function(err, result) {
            if (err) {
              console.log("Error updating user. " + err.detail);
            } else {
              console.log('set profileImage');
            }
          });
        }
      });
    }

I'm I missing something here?
Thanks
Marc

This should be working now that Meteor is using 2.6 by default.


chore: update and pin dependencies

We wait for #48 and fix linting then to not run into too many conflicts


user friendly example

Thanks a lot for contributing!
Hi @Frezyx
I added some navigation to example cause it was hard to use it manually substituting the widgets

Great !
Thank you so much for this contribution 👍🏻


FEATURE: add header to request

I'm calling
self.server_connection
           .request_with_json_result(
               Method::POST,
               "login/device/code",
               Query::<()>::None,
               Body::Json(body),
           )
           .await

and I would like to add the header Accept: application/json.

Shouldn't there already be the JSON header?

I have to add this in order to tell GitHub to send back a json.

You can also receive the response in different formats if you provide the format in the Accept header. For example, Accept: application/json or Accept: application/xml:

https://docs.github.com/en/developers/apps/building-oauth-apps/authorizing-oauth-apps#device-flow

No I mean reqwest should add it?

No I mean reqwest should add it?

Righty, but shouldn't we add a high level option in that method?


Please Update the tool

When I launch the tool and go to my game directory, it says that the pak files could not be decrypted. Please fix this!

See issue #162 for an update.

Try with the latest version please, MCDSaveEdit <IP_ADDRESS>


dump: Try compiling produced schema DDL before dumping data

Alpha 2 had a bug that leads to a broken order of CREATE/ALTER TYPE and CREATE FUNCTION. That leads to some dumps created in alpha 2 that have invalid schema DDL in them and therefore are broken.
We should attempt to compile the produced schema DDL in dump before dumping the data. That way we could at leas partially validate the correctness of the DDL.

Curious about the implementation. What we essentially want it to be able to compile a bunch of DDLs in an empty schema without actually creating a new DB.


Web-VBA-Delete the year for the incorporation date, month and day are also deleted

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Go to URL https://staging.new.expensify.com/
Login with any account
Go to settings>WS>Connect bank account

Pending state withdrawal account credentials
Click on ""Manually enter account details""
Routing Number: 011401533
Account Number:<PHONE_NUMBER>331111
Company: Alberta Bobbeth Charleson
Tax ID: 123456789
First Name: Alberta
Last Name: Charleson

Fill out the company information with an incorporation date in the future and proceed
Delete year

Expected Result:
Deleted only year
Actual Result:
Deleted Year, month and date
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] Windows / Chrome
[ ] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://github.com/Expensify/App/assets/78819774/534bf672-6558-4fe8-9b3b-a509f69db0bb
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:

This is expected behaviour IMO! The field is one singular field, not 3 individual fields. If you disagree, please bring a discussion to Slack about the ideal behaviour here.


Exercise. Contacts. Error summary being scrolled to on every change to form

Omar identified this issue - More detail needs to be added.
I found the GOV UK guidance on it - https://design-system.service.gov.uk/components/error-summary/

In hindsight some of this scrolling behaviour is probably desirable for the user so we have decided to close this issue.


[3094] Added prevention for suggesting providers with no courses

Context
prevention for suggesting providers with no courses
Changes proposed in this pull request
Added prevention for suggesting providers with no courses
Guidance to review
Provider that are either the accredited body or the provider of a findable course will only be shown
Checklist

[ ] Make sure all information from the Trello card is in here
[ ] Attach to Trello card
[ ] Rebased master
[ ] Cleaned commit history
[ ] Tested by running locally

 Location search is currently showing providers without any courses


 should allow for attributes

How can we reproduce this bug?
<button href="#" class="large" align="right">Button Label</button>

What did you expect to happen?
Since <button> generates a table, I would expect it to have the align="right" attribute, just like <columns> do.
What happened instead?
The generated table does not contained the passed attribute.
What email clients does this happen in?
Viewing HTML source, no need to email client testing.
This would be useful for aligning buttons on their own, no need to complicate things with columns. not just alignment, any HTML or custom attributes one might need to pass.

Looks like someone already created a PR, almost 3 months ago: @yvmarques, https://github.com/zurb/inky/pull/61

Merged zurb/inky#61


[spinel] expose RestoreProperties as a public method within RadioSpinel

When running border routers, users have the option to perform RCP firmware updates. After completing the update, users prefer not to restart or reinitialize the master device but only want to restart and restore the RCP device.
Restarting can be initiated by the user; for example, they can restart the RCP device by triggering the reset pin. However, after this restart is done, certain RCP information needs to be restored, such as the panid and extended address.
Typically, the master device can restore the RCP by triggering RecoverFromRcpFailure, but this behavior is not user-controllable. So there's a need to implement a callable API for RCP restoration.
If the RestoreProperties is made public, users can actively restore the RCP by calling it and some other public functions.

@abtink Could you please take a look? Thanks.


Feature Request: Open GoDoc

Hi,
Would it be possible to add a command that would open godoc at package? Eg if I had 'fmt' written on the current line and I pressed ctrl-alt-g-d, it would open 'godoc fmt' in Sublime.
Or if I had 'fmt.Printf' written on the current line and pressed the command, it would open godoc fmt and then search for 'Printf'.
This'd be a huge help.

I'm not sure of how to do with by calling the command.
godoc fmt Print

might work and is easy because I can simply gather all [\w.] before and after the cursor and split on the dot.
Needless to say that will only work sometimes(rarely). It gets interesting when the package is aliased or not on the root, e.g net/http .
The only way I can think to do it to parse the file to find out the actual variable etc. that's under the cursor, at least  then we can also give the right info. AFAICT godoc pretty much does a search which returns loads of noise.
regarding the command palette: This functionality would most likely go under GsPalette ctrl+shift+g
all this stuff is a little way off though because I'm planning to migrate all the plugin functionality into github.com/DisposaBoy/MarGo which is lagging as I wait for things to settle down so I can depend on Go1. This also means the end of calling external commands more or less so I won't have to depend on tools doing what we want or hacking around stuff.


rmfuse dosen't do anything

Hello there.
I'm running a Ubuntu 20.04.2 LTS machine, instaled the dependencies that are mencioned in the repository and the rmfuse pip package. When I try to use it it just stalls, here's my output:
$ rmfuse  -v /mnt/A/directory/remarkable_backups/
DEBUG:pyfuse3:Initializing pyfuse3
DEBUG:pyfuse3:Calling fuse_session_new
DEBUG:pyfuse3:Calling fuse_session_mount
DEBUG:pyfuse3:pyfuse-02: No tasks waiting, starting another worker (now 2 total).

Any idea what is causing this? I'm running Kubuntu.

The same happens to me on Fedora 33. It is after having succesfully logged in.#
Verbose output
rmfuse -v ~/remarkable
/home/geewee/.local/lib/python3.9/site-packages/trio/_core/_multierror.py:502: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
  warnings.warn(
DEBUG:pyfuse3:Initializing pyfuse3
DEBUG:pyfuse3:Calling fuse_session_new
DEBUG:pyfuse3:Calling fuse_session_mount
DEBUG:pyfuse3:pyfuse-02: No tasks waiting, starting another worker (now 2 total).
DEBUG:pyfuse3:pyfuse-03: No tasks waiting, starting another worker (now 3 total).
DEBUG:pyfuse3:pyfuse-04: No tasks waiting, starting another worker (now 4 total).
DEBUG:pyfuse3:pyfuse-05: No tasks waiting, starting another worker (now 5 total).
DEBUG:pyfuse3:pyfuse-06: No tasks waiting, starting another worker (now 6 total).
DEBUG:pyfuse3:pyfuse-03: too many idle tasks (6 total, 2 waiting), terminating.
DEBUG:pyfuse3:pyfuse-03: terminated
DEBUG:pyfuse3:pyfuse-05: too many idle tasks (5 total, 2 waiting), terminating.
DEBUG:pyfuse3:pyfuse-05: terminated
DEBUG:pyfuse3:pyfuse-06: too many idle tasks (4 total, 2 waiting), terminating.
DEBUG:pyfuse3:pyfuse-06: terminated
DEBUG:pyfuse3:pyfuse-07: too many idle tasks (3 total, 2 waiting), terminating.
DEBUG:pyfuse3:pyfuse-07: terminated

Very verbose output (Vv)
See gist
Other info

I have confirmed that there are files in my remarkable cloud.
I have confirmed the target directory exists
rmfuse is installed via pip install --user

Is it suppose to appear a window to input the one-time code? @GeeWee

At least on my first boot I got a 'Please enter the one-time code from (url)' which I did. After that it stalled for some minutes, i cancelled it and the re-running it afterwards gets me the same errors as you.

UPDATE:
After a reboot it aparently started working, it prompted me to the one-time code and it started downloading the files.
 ':)

RMfuse is very event-driven, and it didn't think to prompt for the one-time code until some filesystem operation required it.  On my test systems, that happened basically instantaneously.  But I gather that's not the same for everything.  The commit I just pushed will trigger the one-time code request at startup if needed.
As for the other other reports I don't see anything alarming there.  Those are standard debugging messages from FUSE.  What have you been trying to do: list files, open files, move files, add files?  What's the response: hang, error, crash?  Note that on first listing a directory, it may take a while for RMfuse to get the directory listing from the reMarkable Cloud.  (After getting the file list, it then needs to make a request for the file size of each file individually.  Urgh.)  So if you have a lot of files in a directory, it can take a while to assemble that info.  But this is cached,so future operations should be faster.

Okay so I think I probably misunderstood something - because when running rmfuse (without the v) it just output nothing.
I didn't even consider that it might be triggered by checking the directory, so I simply cancelled it, and cd'd into the directory and saw it was empty.
Perhaps it could make sense to sync the directory on the first boot, or write "Go into the directory to see your files" so others don't make the same mistake?

I've added some debug output at startup.  But I'd think that an unavoidable status message would violate the Rule of Silence.  Do any other filesystems give this sort of output?
Also, to be clear, RMfuse does not sync your local filesystem to the cloud.  Instead, it provides an interface that converts filesystem operations to API requests to the reMarkable cloud.  There's no sync to do at startup.

I think that's a fair point re: rule of silence. Also with the new part you've added in the README I think you can probably close this issue.
Thanks for your time and the software!

@mcpeixoto Feel free to reopen if you're not satisfied.  (Assuming you're allowed to do that.  If not, just comment and I'll reopen.)

I have the same problem,
➜  rpi rmfuse -vv rmk2
/home/fernand/.local/lib/python3.9/site-packages/trio/_core/_multierror.py:502: RuntimeWarning: You seem to already have a custom sys.excepthook handler installed. I'll skip installing Trio's custom handler, but this means MultiErrors will not show full tracebacks.
  warnings.warn(
DEBUG:pyfuse3:Initializing pyfuse3
DEBUG:pyfuse3:Calling fuse_session_new
FUSE library version: 3.10.4
DEBUG:pyfuse3:Calling fuse_session_mount
INFO:rmfuse.fuse:Mounting on rmk2
DEBUG:pyfuse3:pyfuse-02: No tasks waiting, starting another worker (now 2 total).
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.33
flags=0x13fffffb
max_readahead=0x00020000
   INIT: 7.31
   flags=0x0040b039
   max_readahead=0x00020000
   max_write=0x00100000
   max_background=0
   congestion_threshold=0
   time_gran=1
   unique: 2, success, outsize: 80
unique: 4, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 5012
DEBUG:rmcl.api:Renewing user token
DEBUG:pyfuse3:Calling fuse_session_unmount
DEBUG:pyfuse3:Calling fuse_session_destroy

configuration:

  :-------------------::        --------------
 :-----------/shhOHbmp---:\      OS: Fedora 34 (Thirty Four) x86_64

/-----------omMMMNNNMMD  ---:     Host: HP EliteBook 840 G5
:-----------sMMMMNMNMP.    ---:    Kernel: 5.12.15-300.fc34.x86_64
:-----------:MMMdP-------    ---\   Uptime: 7 hours, 5 mins
,------------:MMMd--------    ---:   Packages: 5111 (rpm), 12 (flatpak)
:------------:MMMd-------    .---:   Shell: zsh 5.8
:----    oNMMMMMMMMMNho     .----:   Resolution: 1920x1080
:--     .+shhhMMMmhhy++   .------/   DE: GNOME 40.2
:-    -------:MMMd--------------:    WM: Mutter
:-   --------/MMMd-------------;     WM Theme: Adwaita
:-    ------/hMMMy------------:      Theme: Adwaita-dark [GTK2/3]
:-- :dMNdhhdNMMNo------------;       Icons: breeze [GTK2/3]
:---:sdNMMMMNds:------------:        Terminal: gnome-terminal
:------:://:-------------::          CPU: Intel i7-8550U (8) @ 4.000GHz
:---------------------://            GPU: Intel UHD Graphics 620


Remade the datapack

I remade the datapack to:

Slightly optimise it (it no longer uses armor stands)

Remove some side-effects (killing all items, a "scanner" scoreboard...)

Make it easier to read (because comments).

Some tests I did show a speedup of 1 sec - 1 min.

@GDColon

Oh wow whoa I totally missed this, sounds good to me