How to distributed training in MMDetection 3.x

Hi! I need to train with two gpus in parallel but I can't find where set the parameters to achieve that. I don't know if I should set that in the cfg or as an environment variable like. Many thanks in advance!
My current environments are:
04/13 17:32:37 - mmengine - INFO -
System environment:
sys.platform: linux
Python: 3.8.16 (default, Mar  2 2023, 03:21:46) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 209652396
GPU 0,1: NVIDIA GeForce GTX 1080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.3, V11.3.58
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.12.1
PyTorch compiling details: PyTorch built with:

GCC 9.3

C++ Version: 201402

Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications

Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)

OpenMP 201511 (a.k.a. OpenMP 4.5)

LAPACK is enabled (usually provided by MKL)

NNPACK is enabled

CPU capability usage: AVX2

CUDA Runtime 11.3

NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37

CuDNN 8.3.2  (built against CUDA 11.5)

Magma 2.5.2

Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.13.1
OpenCV: 4.7.0
MMEngine: 0.7.2

Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: None
Distributed launcher: none
Distributed training: False
GPU number: 1

Up

up


chore: bump storage-incentives abi to v0.6.0-rc2

Checklist

[X] I have read the coding guide.

Description
Moving to sepolia client.

Closing this as this is part of ph4 and already done.


[Internal]Add diagnostics cfg to CosmosClientBuilder logs

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-cosmos


Global planner is not planning around obstacle

The obstacle in the simulator is not in map.
Possible solution:
Add obstacle to navigation map so it shows up in the static costmap.
Might help:
https://www.theconstructsim.com/ros-qa-136-how-to-edit-a-map-generated-with-gmapping/

Tutorial on costmaps from Robotics Knowledgebase - https://roboticsknowledgebase.com/wiki/common-platforms/ros/ros-cost-maps/

Tutorial on costmaps from Robotics Knowledgebase - https://roboticsknowledgebase.com/wiki/common-platforms/ros/ros-cost-maps/


us-va-york

{
    "coverage": {
        "country": "us",
        "state": "va",
        "county": "York"
    },
    "website": "http://www.yorkcounty.gov/CountyGovernment/FinancialMgmtServices/GISMapping.aspx",
    "note": {
        "email"<EMAIL_ADDRESS>        "phone": "757-890-3899"
    }
}

I see lots of election shapefiles but not much in the way of addresses

Closing as stale - we have a statewide layer.


get_downloads() returns data from the same site n times

When I select data thanks to get_sites() and then get_datasets(), I effectively get what I want: data for the n sites which respect my criteria. But then, when I run get_downloads() based on the same set of sites, I get n identical lists containing samples for the same site.
For instance if I run the following chunk, directly took from the page neotoma2_R_pack_doc, I get samples for the right number of sites (7) but I get 7 times the data for the same site (Lake Valencia):
brazil <- '{"type": "Polygon",
            "coordinates": [[
                [-73.125, -9.102],
                [-56.953, -33.138],
                [-36.563, -7.711],
                [-68.203, 13.923],
                [-73.125, -9.102]
              ]]}'

# We can make the geojson a spatial object if we want to use the
# functionality of the `sf` package.
brazil_sf <- geojsonsf::geojson_sf(brazil)

brazil_records <- get_datasets(loc = brazil_sf) %>%
  neotoma2::filter(datasettype == "pollen" & age_range_young <= 1000 & age_range_old >= 10000) %>%
  get_downloads(verbose = FALSE)

With get_sites() or get_datasets(), I obtain in my environment a list of lists structured this way:
-sites

-[[1]]
-[[2]]
-[[3]]
etc.

While with get_downloads(), I get:
-sites

-site
-site
-site
etc.

Hello @sedv8808,
Thank you for your answer. Unfortunately I still meet the same issue updating "neotoma2". I am using R 4.3.3 (2024-02-29 ucrt) and the latest version of neotoma2 (1.0.3).
I am working from China (Beijing). Initially, I had to deal with some problems to retrieve data from "neotoma2" and still have to work in particular conditions (using VPN, specific Wi-Fi and modified R profile on my project) but as the other functions of the "neotoma2" R package are working well, I don't understand why get_downloads is still not working.
Any suggestion is really welcome.


subscribe payload won't go after 127 chars

having a long channel name increases the chances to hit the apparent 127 chars limit.
Public channel name "my_channelmy_channelmy_channelmy_channelmy_channelmy_channelmy_channelmy_chan" works:
[[:receive_data,
  "\x81\xFE\x00\x7F0\xF9IUK\xDB,#U\x97=w\n\xDB9 C\x91,'\n\x8A<7C\x9A;<R\x9Cky\x12\x9D(!Q\xDBs.\x12\x9A!4^\x97,9\x12\xC3k8I\xA6*=Q\x97'0\\\x940\nS\x91(;^\x9C%8I\xA6*=Q\x97'0\\\x940\nS\x91(;^\x9C%8I\xA6*=Q\x97'0\\\x940\nS\x91(;^\x9C%8I\xA6*=Q\x97'0\\\x940\nS\x91(;\x12\x844"]]

[[:message_received,
  :text,
  "{\"event\":\"pusher:subscribe\",\"data\":{\"channel\":\"my_channelmy_channelmy_channelmy_channelmy_channelmy_channelmy_channelmy_chan\"}}"]]

Same channel name but with an additional char won't work:
[[:receive_data,
  "\x81\xFE\x00\x80]E\xA8*&g\xCD\\8+\xDC\bgg\xD8_.-\xCDXg6\xDDH.&\xDAC? \x8A\x06\x7F!\xC9^<g\x92Q\x7F&\xC0K3+\xCDF\x7F\x7F\x8AG$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D.g\xD5W"]]

[[:buffer_incomplete,
  "\x81\xFE\x00\x80]E\xA8*&g\xCD\\8+\xDC\bgg\xD8_.-\xCDXg6\xDDH.&\xDAC? \x8A\x06\x7F!\xC9^<g\x92Q\x7F&\xC0K3+\xCDF\x7F\x7F\x8AG$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D3 \xC4G$\x1A\xCBB<+\xC6O1(\xD1u>-\xC9D.g\xD5W"]]

browser <-> em-websocket connection remains open, though slanger won't be able to subscribe to redis.
The payload gets even worse when using private o presence channels because of additional data such as auth.

any ideas about this? it doesn't look like a slanger issue. i'm guessing em-websocket?

any ideas about this? it doesn't look like a slanger issue. i'm guessing em-websocket?

yeah I think so, are you able to reproduce this on your machine?

yeah I think so, are you able to reproduce this on your machine?

You can upgrade to use the git version of em-websocket. My patch is already merged in.

nice thanks, going to try :)

upgrading em-websocket fixes the issue, closing


Sample code: $watch firing upon page load

Using the sample code, the upload event is firing twice upon loading of the page.  I had to change it to this to prevent that.
	$scope.$watch('files', function() {
	if ($scope.files == null) return;  //<- Added this line
	$scope.upload = $upload.upload({
  url: 'server/upload/url',
  data: {myObj: $scope.myModelObj},
  file: $scope.files
}).progress(function(evt) {
  console.log('progress: ' + parseInt(100.0 * evt.loaded / evt.total) + '% file :'+ evt.config.file.name);
}).success(function(data, status, headers, config) {
  console.log('file ' + config.file.name + 'is uploaded successfully. Response: ' + data);
});
	}

Also, evt.config.file.name and config.file.name are undefined.  Using the latest Chrome and .NET Web API backend. Note: I determined that using evt.config.file[0].name worked.

Please comment this becouse it's truely important issue.

I updated the readme to have evt.config.file[0].name instead of evt.config.file.name which was wrong.
The change event is firing twice because when you click on the button it set's the value to null to handle the case when user cancels the popup (there is no easy way to detect this event in browsers other than Chrome which by default sets the value to null) and selecting the same file again.
So you are doing it right if the files are empty you don't need to upload anything but that is a correct event since the user might cancel the selected file and that's why you get that model change.

Actually the problem with file[0] was the way the file was passed in to the upload, it should've been $scope.files[0] I have updated the readme.

It says file.lenght in the sample JavaScript.
Also, I needed $scope.files = [];

I have updated the sample and included a jsfiddle to show how it works in version 3.0.0


Lerna build  (WIP PR)

This is a WIP PR to fix the issue #107  thanks 🙂

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey, thanks for the clarification. I realised that using lerna introduces a package-lock.json. We are using yarn instead of npm. Any way to integrate lerna with yarn, or should we use yarn workspaces instead?
BTW - even if #106 is not ready yet, you could start configuring the build for the core and erc20 modules.

@raulk thanks for the feedback, i'll get started then. As for the yarn integration, yes we'll use workspaces, in fact lerna already supports this integration by passing the flag --use-workspaces when bootstrapping a new lerna project.

@mul1sh any progress here?

@raulk yes, pushing in abit

Closing this PR as there has been no progress and the feedback has not been addressed yet. @mul1sh, let's continue discussing how to move forward in the bounty issue #107.


localisation support added

I've added localisation of the descriptions, added support for 9 additional languages.

This is great @holgerkrupp - thank you!


ipld plugin only works locally

I can confirm that the current implementation of the ipld plugin does not work properly with bitswap as mentioned here:
https://github.com/lazyledger/lazyledger-core/pull/152#discussion_r565689152
I've tested this by spinning up two droplets on DO that otherwise can retrieve data from each other (e.g. via dag get) if the Cid is a regular sha256 one but not if the namespaces are included in the cid.
It seems like defining one encoding for the nodes of the NMT:
https://github.com/lazyledger/lazyledger-core/blob/b83e6766973c314991ec8ba9f6f246fc422fb605/p2p/ipld/plugin/plugin.go#L26
and then simply squeezing the namespaces into the CIDs directly:
https://github.com/lazyledger/lazyledger-core/blob/b83e6766973c314991ec8ba9f6f246fc422fb605/p2p/ipld/plugin/plugin.go#L341-L345
does not work with bitswap. @Wondertan mentioned this a source for the problem (when receiving a block the cid gets recomputed locally):
https://github.com/ipfs/go-bitswap/blob/47b99b1ce34a8add8e5f38cf2eec6bea1559b035/message/message.go#L217-L222
Notice that pref.Sum seems to only accept 32 bytes:
https://github.com/ipfs/go-cid/blob/e530276a7008f5973e7da6640ed305ecc5825d27/cid.go#L563-L577

Unfortunately, adding a custom multihash isn't just doable via a plugin it seems. Several places check for particular good / valid hashes: e.g. Encode validates against a list of known codes:
https://github.com/multiformats/go-multihash/blob/6f1ea18f1da5f7735ea31b5e2011da61c409e37f/multihash.go#L269-L270
https://github.com/multiformats/go-multihash/blob/6f1ea18f1da5f7735ea31b5e2011da61c409e37f/multihash.go#L38
This small library has a stricter list of allowed hashes:
https://github.com/ipfs/go-verifcid/blob/a9d170f0d9246da3a1512396520b0d4eb7602c67/validate.go#L15-L31
and is called on pinning, garbage collection, inside the blockservice, and quite a few other places.

Unfortunately, adding a custom multihash isn't just doable via a plugin it seems. Several places check for particular good / valid hashes: e.g. Encode validates against a list of known codes:
https://github.com/multiformats/go-multihash/blob/6f1ea18f1da5f7735ea31b5e2011da61c409e37f/multihash.go#L269-L270
https://github.com/multiformats/go-multihash/blob/6f1ea18f1da5f7735ea31b5e2011da61c409e37f/multihash.go#L38
This small library has a stricter list of allowed hashes:
https://github.com/ipfs/go-verifcid/blob/a9d170f0d9246da3a1512396520b0d4eb7602c67/validate.go#L15-L31
and is called on pinning, garbage collection, inside the blockservice, and quite a few other places.

Also, I'm wondering if we need to add two custom multihashes then. The leaf and the inner nodes are handled differently in terms of the namespaces that get in front of the sha256 hash in the end. It feels a bit odd as other hashers are just this, hash functions that do know nothing about what they are hashing but here we do need to make a difference for leaves and inner nodes. Additionally, we'd need to "ignore" the max namespace as this is used for the parity shares.
Basically, we'd want these two "hashers":
https://github.com/lazyledger/nmt/blob/6e8a6a5ec3fcae4015ca5dd1923ce033a0458679/internal/hasher.go#L48-L107

Also, I'm wondering if we need to add two custom multihashes then. The leaf and the inner nodes are handled differently in terms of the namespaces that get in front of the sha256 hash in the end. It feels a bit odd as other hashers are just this, hash functions that do know nothing about what they are hashing but here we do need to make a difference for leaves and inner nodes. Additionally, we'd need to "ignore" the max namespace as this is used for the parity shares.
Basically, we'd want these two "hashers":
https://github.com/lazyledger/nmt/blob/6e8a6a5ec3fcae4015ca5dd1923ce033a0458679/internal/hasher.go#L48-L107

Can't you assign different prefixes to inner node and leaf data? Then the hash function will know what it's hashing. That's what NebulousLabs Merkle tree does.

Can't you assign different prefixes to inner node and leaf data? Then the hash function will know what it's hashing. That's what NebulousLabs Merkle tree does.

Can't you assign different prefixes to inner node and leaf data?

Yeah, we currently do this anyway (in the NMT as well as in the plugin when returning RawData).
Yes, we could read that prefix byte and hash accordingly adding only one hasher. I think it's a bit cleaner to have two different hashers instead of one that parses the input and decides how to hash it according to the embedded prefix. Also, @Wondertan suggested introducing two different Codecs for inner and leaf nodes and not make that byte part of that RawData here; in that case we'd definitely need two hashers).

Can't you assign different prefixes to inner node and leaf data?

Yeah, we currently do this anyway (in the NMT as well as in the plugin when returning RawData).
Yes, we could read that prefix byte and hash accordingly adding only one hasher. I think it's a bit cleaner to have two different hashers instead of one that parses the input and decides how to hash it according to the embedded prefix. Also, @Wondertan suggested introducing two different Codecs for inner and leaf nodes and not make that byte part of that RawData here; in that case we'd definitely need two hashers).

Here is a quick and dirty implementation of only using one hash function: https://github.com/lazyledger/go-multihash/blob/a31dec8c92fa0f526d20fbc54572953d5a08e03d/sum.go#L184
IMO, we should move most of the logic of that code to another repo and the hash function should be a few lines like the other ones in that file. That would be another good argument for  two (instead of one) mulihashers, it would make these two functions even a bit cleaner.
For now, I'll keep the above version and see if everything works as expected (over the network). We can iterate on it if it does.

Experimented with using one mh in https://github.com/lazyledger/lazyledger-core/pull/155 which additionally required these changes:
https://github.com/lazyledger/go-multihash/pull/1
https://github.com/lazyledger/go-ipfs/pull/1
https://github.com/lazyledger/go-verifcid/pull/1
I can confirm that this (almost) works: querying for namespaced CID works, retrieving the leaf data doesn't.
For example:
ipfs dag get bagao4amb5yataazahlu5shzrnd5enehjoa6mefxqvg4nav537bp6xje5ynoa3q2yjojhnumb4afjcqvs7kv6ufho3m works as expected but actually going down the path to the leaf via
ipfs dag get bagao4amb5yataazahlu5shzrnd5enehjoa6mefxqvg4nav537bp6xje5ynoa3q2yjojhnumb4afjcqvs7kv6ufho3m/0/0/0/0/0 times out (or doesn't return).

Way fork MH if you could have just register new hash fund?

Probably for no good reason. It just seemed the most consistent approach, as we definitely need to fork go-verifcid to add the custom multihash to the "good set": https://github.com/lazyledger/go-verifcid/blob/master/validate.go (note that this also means we need to fork go-ipfs as it uses go-verifcid internally).
And go-verifcid uses go-multihash to add the other hashes. So if we ever want to upstream this all (surely in a cleaner version), we'd likely want to fork go-multihash first. If we can avoid forking go-multihash for now, that would certainly be preferable as each fork comes with a non-negligible maintenance overhead... I'll spend the 2nd half of this week on this.

@liamsi, can you pls specify all the use cases for IPFS you are willing to have for lazyledger? I need that to understand the picture widely and to propose other possible workarounds not to introduce, IMO, redundant forks.

Yes, happy to summarize it here shortly. Will try do so before end of the week. In the meantime, a slightly out of data and relatively vague (and probably confusing) description can be found here: https://github.com/lazyledger/lazyledger-core/issues/35

@liamsi, the issue you’ve mentioned is probably enough, and I should have read it before, as I did some wrong assumptions regarding your use case and, therefore, gave slightly wrong suggestions. Going to expand on this soon.

To very briefly summarize, we will use IPFS in the following cases:

as a (LL) block store, which means the erasure coded version of the block (see the image below)
for Data Availability proofs: sample cells from the erasure coded version of the block (see image below), e.g. retrieve the data of CID/0/1/1/0/0 of the row or column tree and validate that it checks out with the row/column tree (with the roots in the DataAvailabilityHeader)

Ideally, we want the hashes in above data availability root to match with the CIDs (it's OK that IPFS adds a few bytes of overhead to identify the multihash etc.). That's why I think your suggestion to add one or multiple custom multihashers was exactly right. Actually, I do not see a way around this now. Looking forward to your input!
On a side note, we will also use a DHT to find nodes that stored data for a particular namespace. But I have not yet though this through but we will likely use a separate protocol on top (using libp2p).

Regarding retrieving the leaves: the bug was indeed in here (cc @musalbas): https://github.com/lazyledger/go-multihash/blob/a31dec8c92fa0f526d20fbc54572953d5a08e03d/sum.go#L202-L207
These two commits (in combination) fixed it: https://github.com/lazyledger/go-multihash/commit/1743be8d8a5bf1b66b1910780599cef5313b61d8 and https://github.com/lazyledger/go-multihash/commit/88bad1265973e3d986433b60817d2e49a72812f3

On a side note, we will also use a DHT to find nodes that stored data for a particular namespace. But I have not yet thought this through but we will likely use a separate protocol on top (using libp2p).

As @musalbas mentioned on discord, for a start we can simply use the same mechanism as for DA proofs:

The client does a "search " of tree on IPFS to find the leafs for a namespace, which should be O(log(n))
As each node has its min and max namespace id


new dat-ecosystem website project plan

@todo

[x] Study Road map guide

Explanation of this roadmap

@output :package: Screencast -explanation:

[x]  Road map
#13

@info
Summary
This directory contains a roadmap to building the the new dat-ecosystem website
Mission:
Our mission is to foster the decentralized garden by empowering initiatives and projects that have a clear positive social impact, such as civic tech, marginalized communities and science with p2p technologies which are secure, local-first, interoperable, easy-to-use, and fast.

[ ] @output :question: from new dat-ecosystem website project plan issue

feedback 2022.04.26
Heya :-)
It looks interesting, but also quite abstract.

the new dat-ecosystem website project plan is what should be the outcome of this trial phase, which should then have maybe a figma wireframe as an output

UX research sounds good, but problem definition sounds also quite abstact. I was kinda hoping a problem refinement could of course happen later, but the trial phase might already be there to figure this out ...and is there even a problem?

what exactly is a solution ? ...is that the website? ...and where was the concept that gets validated? ...and what are the validation criteria? where they worked on?
...i mean, it is dat-ecosystem and it needs a website and i feel thats quite specific and tangible already, while "solution", "validation", etc... is way to abstract for my taste

the product design sounds good, but that will give me wireframes again? ...i thought we have that earlier already. Is that again figma?

this step makes sense to me, but it turns in a single todo step the "wireframe" into the "final page" :-) ...i guess it makes sense as a first iteration of the project plan that i can give feedback on, but because not a single of the listed "resource link" or "group of resource links" has been added as an input for some of the todos, it is hard for me to give any more specific feedback

What are the parts or aspects or categories or "page" or "components" or whatever you call it that should be in the webapp/website page to solve or display or show which of the resources? ...of course getting more specific can "go wrong", but it would be a first iteration and "best guess" of what would maybe be relevant to include in the website/webapp ...and then i could give feedback and we do another iteration.
This project plan is "fail safe" or "bullet proof" in that it is definitely correct, because everyone wants their problem to be defined and wants the solution and wants it validated - it is so general, that it is always correct, but it doesn't say anything about what that means for the specific project at hand, so i would rather see a more specific and adapted version to the project at hand that might have lots of mistakes or prioritises things in the wrong way or whatever, but exactly because of that i can share some thoughts and give some feedback of how to iterate to make it better.
This *process of having something more opinionated and specific and me being able to give feeedback of how to iterate on it to improve it is what I think is the best way to over time get onto the same page - basically to slowly get a shared understanding of what the website should be, based on the professional input and work and me as a "customer" or "client" to steer that to match my or rather the "end user" needs that i can represent.
But :+1: for the format - it looks good

Oh no worries, I'll make the necessary changes and edit the Info section to include additional context.

Oh no worries, I'll make the necessary changes and edit the Info section to include additional context.

I think one goal would be to have more than one issue and link those new additional issues as todos in this main one here.

https://github.com/wizardamigos/wizardamigos.github.io/issues/38
https://github.com/datdotorg/datdot-ui/issues/1

these are two example links to show you how complex an issue structure for a real world plan can get.
maybe one goal should be to include every single resource link and make them inputs of one of those todos in this main issue or a linked sub issue, so we are sure to somehow process all of them ...and the corresponding planned outputs somehwat reflect what we try to achieve as an output when we process those resource links in the trial phase description

merged into #49


Merge lightning worker into one thread.

What problem does this PR solve?
Issue Number: close https://github.com/pingcap/tidb/issues/18025
Problem Summary:
There are a lot of threads in TiKV. But most of them are only working a while and then suspend themself and wait for waking up. If there are a log of threads waiting for waking up, system will cost much time to schedule.
What is changed and how it works?
Proposal: xxx
This PR is to merged most of worker into one or two threads so that they can run without suspend and schedule.

I refactor Worker in components/tikv_util/src/worker/mod.rs so that it could run multiple Runner in one threads.
I use  GLOBAL_TIMER_HANDLE for some timed task.
Use unified read pool for SchedPool, which is used to deal with transaction-write.

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Check List
Tests

Unit test

Release note

Reduce cost of thread scheduler so that TiKV could run in machine with fewer CPU cores.

It seems that this PR can not improve  performance for TiKV. I will split some refactor of code from it into other PR.


README Update

Add logger setting and system config section.

Looks good to me.
How about mentioning of different logdir configuration options with their priorities ?

@pkalever
Yes, I will add it now.
Thanks,

Updated, @pkalever please review.
Thanks,

@pkalever
Please review.

@pkalever Thanks.

@mikechristie
Updated

Thanks.


Cherry-pick gRPC dependencies rather than using grpc-all

This was briefly discussed internally. As per title we should drop the grcp-all dependency and favor cherry-picking gRPC dependencies.
This should help us getting rid of some dependencies as well as duplicated classes (e.g. the ones pulled by both protobuf-java and protobuf-lite).

Note: This fix should also be applied to the grpc-generated packages for each API.


[BUG]: GithubRelease@1 does not accept service connections of type "InstallationToken"

Task name
GithubRelease
Task version
1.*
Environment type (Please select at least one enviroment where you face this issue)

[X] Self-Hosted
[ ] Microsoft Hosted
[ ] VMSS Pool
[ ] Container

Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows 10
Task log
N/A

Relevant log output
Step  input gitHubConnection expects a service connection of type github with authentication scheme OAuth,OAuth2,PersonalAccessToken,Token but the provided service connection **** is of type GitHub using authentication scheme InstallationToken.

Aditional info
The pipeline won't save because the task requires a github connection of a type that is not installation token. Oddly enough when using a OAuth or PAT based connection, the pipeline overview "complains" and suggests to use the github app instead (installation token).

I would love to use the github app instead and I can use it for the GitHubComment@0 task but for some reason not for the GithubRelease task.

Error message when saving:
> Step  input gitHubConnection expects a service connection of type github with authentication scheme OAuth,OAuth2,PersonalAccessToken,Token but the provided service connection **** is of type GitHub using authentication scheme InstallationToken.

Banner suggesting to use InstallationTokens instead:
> This pipeline is using an OAuth-based connection. Switch to the Azure Pipelines GitHub App for improved reliability and performance.

[Use app connection]

Seems like quite the contradiction to me.

Bump

bump

Hey @xmdanni @julianh-nable you can expect installation token support to be released by the end of this week or start of the next week.


Fit long endos like Church Slavonic's in Details, popups

Issues resolved by this pull request
Closes #94
Summary
Kind of a gross hack but I'm okay with it since there are only 1 or 2 instances in the dataset so far. Here's the logic/check to deal with this issue:

If there are no spaces in the Endonym (aka single-word Endo)
...and it's 17 characters or more (Church Slavonic is 19 but I dropped it down a couple just in case)
...reduce the font size by 4px or so

@rperlin-ela I'm sure there's a better way to do this but let me know if you're okay with this gross "fix" mentioned above.
https://deploy-preview-110--languagemapping.netlify.app/details?id=663

Make sure it fits on mobile
Laptop/desktop/etc.
And that it's only doing it to long single-word instances
...but not impacting any other instances

Excellent, it works. Assume pop-up (at least for Church Slavonic on desktop) is a separate issue.

On Oct 14, 2020, at 12:51 AM, Jason Lampel<EMAIL_ADDRESS>wrote:
@rperlin-ela https://github.com/rperlin-ela I'm sure there's a better way to do this but let me know if you're okay with this gross "fix" mentioned above.
https://deploy-preview-110--languagemapping.netlify.app/details?id=663 https://deploy-preview-110--languagemapping.netlify.app/details?id=663
Make sure it fits on mobile
Laptop/desktop/etc.
And that it's only doing it to long single-word instances
...but not impacting any other instances
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Language-Mapping/language-map/pull/110#issuecomment-708154903, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNKB5EPA6TBFA4SCEATVHTSKUU5TANCNFSM4SQC773A.

No I fixed that too, are you not seeing what I'm seeing?

@rperlin-ela let me know about the popup thing (my prev comment w/screenshot). I think that's the only thing holding up this PR.

I'm still seeing the last two characters of the Church Slavonic pop-up going to the next line, at least on desktop (mobile looks a-ok). In the details panel it also treads a little bit close to the ? icon for Glossary. Maybe something just on my end?

Hmm and this is on the deploy not production, yeah?
https://deploy-preview-110--languagemapping.netlify.app/

Yes, on Safari — just noticed that on Firefox the pop-up is fine but still a tad crowded on the Details panel.

On Oct 16, 2020, at 12:55 PM, Jason Lampel<EMAIL_ADDRESS>wrote:
Hmm and this is on the deploy not production, yeah?
https://deploy-preview-110--languagemapping.netlify.app/ https://deploy-preview-110--languagemapping.netlify.app/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Language-Mapping/language-map/pull/110#issuecomment-710219262, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNKB5CFCHFJAV2D6UTRWBLSLB3JDANCNFSM4SQC773A.

ok hopefully resolved, let me know if not: https://deploy-preview-110--languagemapping.netlify.app/
i tested in safari and firefox on laptop so if you're not seeing them fit then try CMD + 0 (that's a zero) in the browser/s to reset your zoom.

All good!

On Oct 17, 2020, at 3:13 AM, Jason Lampel<EMAIL_ADDRESS>wrote:
ok hopefully resolved, let me know if not: https://deploy-preview-110--languagemapping.netlify.app/ https://deploy-preview-110--languagemapping.netlify.app/
i tested in safari and firefox on laptop so if you're not seeing them fit then try CMD + 0 (that's a zero) in the browser/s to reset your zoom.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Language-Mapping/language-map/pull/110#issuecomment-710764990, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNKB5DAQMJBJE5J3LR6HH3SLE7YBANCNFSM4SQC773A.


Fix failing rds test

Our travis tests are failing due to a recent RDS functional test that was added.  I added two things:

Fix the test (tests that need credentials should subclass BaseSessionTest)
Add package setup/teardown that more closely resembles the travis environment.  This likely went unnoticed because if you have credentials configured locally the test will pass.  With this change, the original test would have failed on a dev laptop.

Current coverage is 97.65% (diff: 100%)

No coverage report found for develop at 505a419.

Powered by Codecov. Last update 505a419...21a7139


small corrections for continuous-to-discrete approximations

This PR replaces #403.
From sawyerbfuller:

combined dtime/sample_system and dtime/c2d into the same function because they do the same thing. old c2d also had an unnecessary test

fixed TransferFunction.sample and StateSpace.sample to have updated docstring that adds recent methods added to scipy.signal.cont2discrete (except for 'matched', which is only available for TransferFunction and only for python-control)

Additional modifications (murrayrm):

Added c2d back into dtime.py with a PendingDeprecationWarning to fix unit tests.

Fixed up a few PEP8 problem along the way

Coverage increased (+0.01%) to 84.223% when pulling 8db007a02a018791cb69fa594bbaf8face26086d on murrayrm:sawyerbfuller-master into 03183d157a5f6cd8a9918498ab8717dc7b91c9cf on python-control:master.

Coverage increased (+0.01%) to 84.223% when pulling 8db007a02a018791cb69fa594bbaf8face26086d on murrayrm:sawyerbfuller-master into 03183d157a5f6cd8a9918498ab8717dc7b91c9cf on python-control:master.


fix orgnization webhooks

fix missing discord webhook wrong route and missing template on orgnization webhooks and fix bug on deliver webhook.

LGTM

What was the bug that led to the changes to the HookTask.deliver() method?

@ethantkoenig that should be a bug of GetWebhookByRepoID when the webhook is an orgnization webhook. Then the latest status will not be updated. And I added UpdateWebhookLastStatus for only update the last_statu but not all the columns.

LGTM


increase test timeouts

close https://github.com/binance-chain/tss-lib/issues/114

thank you @HAOYUatHZ - we fixed according to your suggestion (as part of #115 )

thank you @HAOYUatHZ - we fixed according to your suggestion (as part of #115 )

Great to know I can help!


Add async attribute to manager and stylesheet

Adds ability to set stylesheet to be loaded asynchronously.
Uses media="print" onload="this.media='all'"

hey @williamtetlow  thank you for this my friend, I really appreciate this, I'll have this merged in the morning and put out, sorry for not responding sooner,

@williamtetlow  sincerest apologies man, I totally forgotten about this, then got distracted with other stuff... Ive merged this through  will publish to npm shortly... Thank you again for the assist... 💚


Siac Absolute Filepath

Siac has an abs(dir string) function which converts a relative path into an absolute path. First, this is a bad name for a function that operates widely in a package, we should change the name of the function to be absolutePath(dir string)
Second, the abs function does not gracefully handle being run in a directory that does not exist. A terminal can end up in a directory that doesn't exist if another program deletes the directory while the terminal is cd'd into that directory. We need to clean up the abs function so that it gracefully handles being in a non-existent directory.

Could you clarify what you mean by "gracefully handle"? I think the code that uses the path should handle the invalid path because there is no guarantee that just because the path existed while calling abs(...) it will continue to exist until actually using the path.


Update to Godot 4

Hi, I updated your script to be compatible with Godot 4!
https://github.com/hiulit/Godot-3-2D-Fake-Confetti-Particles/assets/1598486/9f771c68-befa-4aa6-8cc8-16e4142fb0bd

UP

Hey @florisgravendeel, since this change is not compatible with Godot 3 and current code base is still required for Godot 3.
So, would it be better if we close this PR, and update your Repository to "Godot 4 2D Fake Confetti Particles"?

Sounds fine to me!


Don't emit Session notification even if creation fails

fixes #380
adds test simulating the behavior
moves protocol load out of try-finally block
tear down async test fixture gracefully
fix existing test in light of new behavior

🎉


catkin not required for pure cmake packages

This prevens rosdep resolving in ROS2

Can one of the admins verify this patch?

ok to test

If it speeds-up the review (or reduces risk), I can make the flag:
<run_depend condition="$ROS_VERSION == 1">catkin</run_depend>

yes, could you add that line (in libpointmatcher too). I'll try to process the PRs next week.

Hi! The change in the package.xml from this merge is causing issues when building our library COVINS: https://github.com/VIS4ROB-lab/covins
As soon as we call catkin build <xzy> (e.g. catkin build eigen_catkin), catkin throws
Error(s): The "run_depend" tag must not have the following attributes: condition, caused by libnabo/package.xml.
We found a workaround for now and removed the libnabo dependency, but maybe you want to consider whether this change is technically really necessary, since it seem like there might be some compatibility issues (we experienced this on 3 different machines with both Ubuntu 18 and Ubuntu 20).

Ow dear, that is a regression from my side. I forgot to add the new package format:
<package format="3">

Which supports the condition flag. PR incoming

Indeed, would as well expect CI catches this... Anyway, no problem, thanks for the quick reply and for taking care of this.


Review dos-unexpected-revert-with vector authorization check

Option1
Make sure that the caller is whitelisted (e.g: caller == owner).
or
The required authorization is done by someone with a priviledged role in the contract (e.g: do owner.require_auth()).
Option2
Remove altogether authorization checks in the detector.
Update error messages and documentation.

Despite ensuring that the require_auth() is effectively apolied on contract owner Address is difficult, we decided to keep the validation of require_auth in order to reduce the number of false positives that could occur.
This decision leads to some false_negatives, that can only be resolved with a better identification of the owner.


Refactor Generic Host lifetimes to work better with ServiceBase

#1347
The RegisterDelayStartCallbackAsync callback is inlined to work around the ServiceBase.Run blocking call.
RegisterStopCallback goes away, it only ever called StopApplication.
ProcessLifetime goes away, ConsoleLifetime becomes the default.

@glennc one funny thing I noticed when debugging is that OnStop gets called 3 times, but it's fine because the CTS in StopApplication prevents any recursion.

OnStop called by the service.
OnStop => StopApplication => Stop => OnStop
Host.StopAsync => Stop => OnStop

StartAsync cannot block. The usage pattern is too different from the other usage pattern. We need to pick one way and stick with it.

Easy enough, I forced ServiceBase.Run onto a background thread.

Redesigned and Updated. I got rid of all the callbacks.


Input data files for HFVRP - questions

Hi developers,
This is a great software and I am grateful to know about your excellent VRP solver!! I have some questions:
It seems that HFVRP.py file is suited to read the data files that start with "c___" (that are in folder:  VRPsolvereasy/demos/data/HFVRP), such as, for instance, "c50_15hd.txt ", correct?
Because for the input files starting with 'X____', you need to create a parser file to read the data appropriately (ie. for instance, 'X101-FSMFD.vrp' file) . Am I right?
How would you create a parser file (in python) to do that? I am not familiar to do that unfortunately.
Also, I think the output gives you the optimal route, but it does not give you any associated cost. Is that right? Am I supposed to manually calculate the cost, given the route and route distance and the 'fixed cost' and 'variable cost' assigned?
Is there any way that I can meet one of the experts or developers on zoom to go over some of this? I would really appreciate it.

Here is the data file that I'm referring to for better interpretation:
c75_18hvrp.txt

Last summer, when I got the HFVRP problem to run (for a list of cities that I put for 'Philly airport', for instance), I got the output for the python program to say:
Statistics :
best lower bound : 5.0
solution time : 4.8521057
number of nodes : 1
solution value : 5.000000764184426
root lower bound : 5.0
root root time : 4.8461617.
Status : 0.
And then it would list the optimal routes , but no costs associated with it.
Is what shows as the "solution value" (above) the cost? I assume that is the objective function value, but the cost= $5 seems so small, given that the 'fixed cost= $300' and the 'variable cost = 4.0' for this truck that I put in my problem.
Thats why I dont understand this 'solution value' that it outputs and why its so small.
How do I get it to show the cost of the optimal routes it gives me? Do I need to manually calculate it myself given the fixed and variable costs that I give it?

Dear @mventura00 ,
Indeed, there was no reader for the XH (large) instances. I've committed a change to this demo. Could you try it now? I also included two options: -f for the format (0 for classic, 1 for XH), and -u to set an upper bound. Here's an example to try:
python HFVRP.py -i X219-HD.vrp -f 1 -u 120737.1 -t 350

Dear @mventura00 ,
Indeed, there was no reader for the XH (large) instances. I've committed a change to this demo. Could you try it now? I also included two options: -f for the format (0 for classic, 1 for XH), and -u to set an upper bound. Here's an example to try: python HFVRP.py -i X219-HD.vrp -f 1 -u 120737.1 -t 350

Thank you so much for answering my question! I will check out the new change committed to code for HFVRP... and test it out. I'll comment back later! Thank you for your attention and time!

Dear @mventura00,
Can you please close this issue if it has been resolved for you.


Slow dataflow analysis

hi，i use dataflow analysis analyze AES design, it had spent one day.now, it still running. i want to know why?

I think it's because the current dataflow analyzer does not use memorization.

Thanks for your reply . so, if I have enough time and LINUX platform has enough memory, no matter how large design always can analyze? such as : c8051

I think the current analyzer does not handle a realistic size of hardware. I think it should be accelerated, but I don't have enough time for this :(

I want to extract dataflow from the verilog design (uart,AES,c8051). But I can't write the verilog compiler front end. so , I read your paper and find pyverilog. about my mission, can you give me some suggestion? thank you very much!

It would be really helpful if the dataflow analyzer could be sped up.


Small improvements around CI and tests

Fixes #1936
Fixes #1921
Tasks

[x] Ignore deps update via renovate for the performance benchmark folder
[x] Brings in GH actions
[x] Improve issue template
[x] Fixes failing index tests for MongoDB
[x] Investigate failing website build
[x] Cache dependencies
[x] IT tests against two major releases of each database we support e.g PG 9 and 12, Mongo 3 and 4
[x] Cancel Previous runs

Hope to bring the above changes in and I'll look into the below items next:
As separate manual task and required before merging this PR

[x] Disable Circle CI
[x] Generate an NPM token and add it in GH secrets

On Follow up PR

[ ] Add CI to deploy website automatically when new release is created.

@ssd71 I asked you for a review, since you might have a better idea around flasky indexes tests.

While testing for indexes themselves is a good idea, and shouldn't take an unreasonable amount of time because the test databases should be empty, do you think we're better off mocking or jest.spying the createIndexes function? We could test the function itself separately perhaps.
Feel free to ignore if that sounds like a crazy idea in this context :P

While testing for indexes themselves is a good idea, and shouldn't take an unreasonable amount of time because the test databases should be empty, do you think we're better off mocking or jest.spying the createIndexes function? We could test the function itself separately perhaps.
Feel free to ignore if that sounds like a crazy idea in this context :P

It is a good idea, but we can stick with the current tests. From my understanding, the reason they are failing is that the createIndexes function is async which probably means that in some random occasion the function has not been full resolved once the assertion is done.

@craicoverflow @wtrocki I've pushed some minor improvements here and there. Can you review please?
PS: CircleCI build is failing because I have removed the config file. The plan is to deactivate it completely once we are happy with this workflow.

I would disable circleCI before merge so it will not confuse our branch protection rules

I have created https://github.com/aerogear/graphback/issues/1945 to automate website deployment via https://v2.docusaurus.io/docs/deployment/#triggering-deployment-with-github-actions


Support ALTER SOURCE ADD COLUMN for shared source

We've implemented the feature. But may need refinement https://github.com/risingwavelabs/risingwave/issues/17340


does not compile with Visual Studio 2017 Toolset v141

query_transformation.h uses std::unary_function which got removed in C++17
https://github.com/SOCI/soci/blob/master/include/soci/query_transformation.h
http://en.cppreference.com/w/cpp/utility/functional/unary_function

This is related to #604.
I will try to get rid of the unary_function for SOCI 4

thanks :)
if anyone else has the same issue, for now I've just provided this function in the std namespace myself before including soci.h as lazy solution.
namespace std
{
	template <class Arg, class Result> struct unary_function
	{
		typedef Arg argument_type;
		typedef Result result_type;
	};
}

great thanks! 😄

Welcome. Thanks for testing.


Fix Stream::read($length) implementation

When $length value exceeds the stream eof, the remaining stream content
must be returned.
See #13

@ajgarlag thanks for the PR.
The tests are failing because of a dependency issue. I fixed it, also updated a few other dependencies in #15. Once @joelwurtz checks/merges it, you should rebase your branch. Checks should pass afterwards.

@sagikazarmark I've rebased your branch, but the build still fails with PHP5.6 and PHP7.
Once #15 is fixed and merged, I will rebase master to finish the PR.

Ok. It seems that the current failure is not related to dependency issues.

I'm not so sure about this change, IMO i prefer to have an exception when the user doesn't handle well the size of the stream.
For me this change should be in the documentation, where we should to the user to check the size of stream before reading.
The PSR7 line is only a recommendation and is related to the fread implementation in PHP which can sometimes return less bytes than asked.

In my code, I was using Http\Message\Encoding\FilteredStream to read a gzipped Http\Client\Socket\Stream
To fill the FilteredStream buffer, the inner stream is read in chunks of 8192 bytes, which is defined by a constant, so I have no control of the stream size.

I see, you make me doubting, should the filtered stream be updated or this one ?
Just mad a test on fread, and it's silent when reading more data than asked, so we should keep the same behavior, like you propose.

Can you rebase on master ?
Will certainly throw a new version once this has been merged.

@joelwurtz Rebased and squashed.
The travis-ci build was successful before squashing, but failed after it.
Reviewing the test failure, I think it was a transient error, so this could be merged.

Yeah no big deal, thanks !

@joelwurtz Can you tag a new version with this fix?

Tagged in 1.1.0

Thanks!


Adding in Adobe Flash changes

I'm sorry, I'm trying to link to the Adobe Roadmap change but I don't think I've done the URL correctly in the text for it to show up right.

Docs Build status updates of commit c73ddf5:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-edge/web-platform/site-impacting-changes.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit e583fd7:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

microsoft-edge/web-platform/site-impacting-changes.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Add Fortran keyword "data"

Add the Fortran data keyword to the .f and .for heuristics.
Fixes misclassification of https://github.com/jbuonagurio/RHTDP.

:+1:


flushChunks not returns shared css between modules

Hi!
Awesome library, SSR + Code Splitting is a very common problem and you resolved it very well here :)
I would like to integrate it to an already production project, but we have an issue with the css files shared between modules.
For example, I have a Home module, and when I navigates to it to render synchronously, the chunkNames array is
[ 'Home', 'searchForm' ],
When I execute flushChunks:
  const { scripts, stylesheets } = flushChunks(stats, {
    chunkNames,
  });

This is the result:
[ 'js/Home.717e415a5908f7599b0a.chunk.js',   'js/searchForm.876649092bef6d33b3e8.chunk.js',   'js/main.ad7ec855eb95d76053ca.js' ] [ 'css/main.9eb3c8d1.css',   'css/Home.72db2c6e.chunk.css',   'css/searchForm.6a9b3cfc.chunk.css' ]
The thing is that when the browser loads the page, it gets some others js and css files on demand, but those css files are needed in the first paint, because otherwise generates a weird user experience changing width and height of items, among others things :(.
The other files I am talking about in this case are:
['css/default~Home~addressForm~checkoutRouter~landing~myAddresses~myOrders~shopDetailsRouter~shopsList.9eb5c300.chunk.css', 'css/default~Home~addressForm~checkoutRouter~landing~myOrders~shopDetailsRouter~shopsList.857b76b4.chunk.css']
It seems that those files are shared between modules (all those are module names).
Is there any way to get those files in the flushChunks call?
If you need more details please let me know.
Thanks!

Hi @MartinGian, could you please share your webpack stats output? It seems it contains only assetsByChunkName property which is not enough to extract shared chunks.
In my case I'm using webpack-stats-plugin with fields property set to null to include all shared chunks as well, but there's a side effect - stats file is nearly 60Mb and parsing takes almost 700ms what is quite a lot..
@faceyspacey, do you have by the chance any recipe to handle dynamic chunks in more accurate and faster way?


Defer module name lowercase conversion for Windows

This update just defers lowercasing function module names in windows-metadata to the point where they actually need to be lowercase in code generation to ensure the metadata reader faithfully preserves this information.

@abbec If you get a moment, would be great to hear if this change affects you (https://github.com/microsoft/windows-rs/issues/638)


fix: check if window is defined for ssr environments

Fixes #29

I'm going to merge this PR, but the one I opened will enable all of the existing functionality to continue working as expected in SSR apps.


Fix FormPopover error display [ready]

This PR introduces an ErrorIndicator component to be used in cases such as displaying errors on the FormPopover footer. Currently, the error is being displayed through a FormFieldValidationBlock below the button components. However, the Canon classes being used in that component are meaningless outside of a control group, so the error would display as plain black text and the positioning (being rendered below the buttons) would display the error beside of the buttons, widening the popover substantially.
Current (screenshot of code I'm developing within Reach consuming the FormPopover pattern) -

With changes (rendering error through ErrorIndicator component located above buttons) -

Canon does not have any patterns surrounding error text outside of tables and validation, so I have settled on the rs-status-error class here to convey that something is borked and paint it red.
Any thoughts or feedback as to a different approach (including whether or not you think there should be a more appropriate class in the Canon repo) would be appreciated.

Screenshot?

LGTM 🎉

It'd be better if we had an approved Canon pattern for this kind of thing, but I'm not sure there are even designers working on Canon anymore. This is probably alright in the mean time.

LGTM

@willcumbie That was my thought on this, too - I was pretty stumped as to which approach to take. Also it did confuse me immensely that we didn't have any sort of basic CSS for errors, esp in popovers (again, they're all very custom to Canon tables or field validation). Ultimately I think a 'for now' fix is ok enough... It feels weird, but less weird than some of the other solutions I came up with. 🙊

looks good, squash into a single commit


chip-device-ctrl zclwrite fails for all data types except string

Problem
chip-device-ctrl zclwrite fails for all data types except string
STR:
1 - commission device using chip-device-ctrl
2 - try to write to any writable attribute whose data type is not string
The following works:
chip-device-ctrl > zclwrite Basic UserLabel 12344321 1 0 abcde
The following do not work:
chip-device-ctrl > zclwrite Basic LocalConfigDisabled 12344321 1 0 True
chip-device-ctrl > zclwrite Identify IdentifyTime 12344321 1 0 3000
The error is:
An exception occurred during processing input:
argument 4: <class 'TypeError'>: wrong type
Proposed Solution
In src/controller/python/chip/clusters/CHIPClusters.py function WriteAttribute() ensure that funcCaller gets correctly typecast value argument for the attribute.

@erjiaqing


Remove Hackflight from Nengo branch; just take off and hover

Should work with SocketModule instead of Python extension


LwM2mNodeSenMLDecoder sometimes fails trying to convert from double to long

Hi,
We are using Leshan 2.0.0-M5.
In the log files I sometimes see that when reading currentTime resource from device object (3/0/13) decoding sometimes fails with the following exception:
08:50:47 [CoapServer(main)#27] ERROR c.n.s.d.s.s.d.ManagerImpl - Endpoint 94193A0407000000: Read request for path 3 failed with exception: {}
org.eclipse.leshan.core.request.exception.InvalidResponseException: Unable to decode response payload of request [ReadRequest [path=/3 format=null]] from client [94193A0407000000]
at org.eclipse.leshan.server.californium.request.LwM2mResponseBuilder.decodeCoapResponse(LwM2mResponseBuilder.java:458)
at org.eclipse.leshan.server.californium.request.LwM2mResponseBuilder.visit(LwM2mResponseBuilder.java:122)
at org.eclipse.leshan.core.request.ReadRequest.accept(ReadRequest.java:187)
at org.eclipse.leshan.server.californium.request.RequestSender$2.buildResponse(RequestSender.java:226)
at org.eclipse.leshan.core.californium.AsyncRequestObserver$1.onResponse(AsyncRequestObserver.java:60)
at org.eclipse.leshan.core.californium.CoapAsyncRequestObserver.onResponse(CoapAsyncRequestObserver.java:99)
at org.eclipse.californium.core.coap.Request.setResponse(Request.java:931)
at org.eclipse.californium.core.server.ServerMessageDeliverer.deliverResponse(ServerMessageDeliverer.java:258)
at org.eclipse.californium.core.network.stack.BaseCoapStack$StackTopAdapter.receiveResponse(BaseCoapStack.java:217)
at org.eclipse.californium.core.network.stack.AbstractLayer.receiveResponse(AbstractLayer.java:89)
at org.eclipse.californium.core.network.stack.ExchangeCleanupLayer.receiveResponse(ExchangeCleanupLayer.java:105)
at org.eclipse.californium.core.network.stack.ObserveLayer.receiveResponse(ObserveLayer.java:139)
at org.eclipse.californium.core.network.stack.BlockwiseLayer.receiveResponse(BlockwiseLayer.java:776)
at org.eclipse.californium.core.network.stack.ReliabilityLayer.receiveResponse(ReliabilityLayer.java:308)
at org.eclipse.californium.core.network.stack.AbstractLayer.receiveResponse(AbstractLayer.java:89)
at org.eclipse.californium.core.network.stack.BaseCoapStack.receiveResponse(BaseCoapStack.java:146)
at org.eclipse.californium.core.network.CoapEndpoint$1.receiveResponse(CoapEndpoint.java:314)
at org.eclipse.californium.core.network.UdpMatcher$4.run(UdpMatcher.java:457)
at org.eclipse.californium.elements.util.SerialExecutor$1.run(SerialExecutor.java:290)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.eclipse.leshan.core.node.codec.CodecException: Invalid content [1.638435E9] for type TIME for path /3/0/13
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.parseResourceValue(LwM2mNodeSenMLDecoder.java:504)
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.extractLwM2mResources(LwM2mNodeSenMLDecoder.java:426)
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.parseRecords(LwM2mNodeSenMLDecoder.java:193)
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.decode(LwM2mNodeSenMLDecoder.java:97)
at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decode(DefaultLwM2mDecoder.java:156)
at org.eclipse.leshan.core.node.codec.DefaultLwM2mDecoder.decode(DefaultLwM2mDecoder.java:138)
at org.eclipse.leshan.server.californium.request.LwM2mResponseBuilder.decodeCoapResponse(LwM2mResponseBuilder.java:450)
... 21 common frames omitted
Caused by: java.lang.IllegalStateException: Can not convert 1.638435E9 to long safely : Unsupported number java.lang.Double
at org.eclipse.leshan.core.util.datatype.NumberUtil.numberToLong(NumberUtil.java:59)
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.numberToLong(LwM2mNodeSenMLDecoder.java:527)
at org.eclipse.leshan.core.node.codec.senml.LwM2mNodeSenMLDecoder.parseResourceValue(LwM2mNodeSenMLDecoder.java:493)
... 27 common frames omitted
Best regards,
Sonny

Are you using senml-json or senml-cbor ?

could you share the exact payload (byte for cbor / string for json)  sent by the client ?

Hmm, I don't set a specific ContentFormat in ReadRequest, so I'm not sure which one is used by default.
I'll try to capture the exact payload but the problem is this happens rarely (maybe once or several times per day).
Note: I only 4 devices for development/testing, we're not in production yet.

Hmm, I don't set a specific ContentFormat in ReadRequest, so I'm not sure which one is used by default.

You can put in the ReadRequest the format you would like to be used for the response.
If you set nothing, the client will chose its preferred format but in all case the format used to encode the response payload is in the response.
Eg. :
Frame 218: 64 bytes on wire (512 bits), 64 bytes captured (512 bits) on interface lo, id 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: <IP_ADDRESS>, Dst: <IP_ADDRESS>
User Datagram Protocol, Src Port: 52954, Dst Port: 5683
Constrained Application Protocol, Acknowledgement, 2.05 Content, MID:32713
    01.. .... = Version: 1
    ..10 .... = Type: Acknowledgement (2)
    .... 1000 = Token Length: 8
    Code: 2.05 Content (69)
    Message ID: 32713
    Token: 7c515cfbdc858367
    Opt Name: #1: Content-Format: application/vnd.oma.lwm2m+tlv
        Opt Desc: Type 12, Elective, Safe
        1100 .... = Opt Delta: 12
        .... 0010 = Opt Length: 2
        Content-type: application/vnd.oma.lwm2m+tlv
    End of options marker: 255
    Payload: Payload Content-Format: application/vnd.oma.lwm2m+tlv, Length: 6
        Payload Desc: application/vnd.oma.lwm2m+tlv
        [Payload Length: 6]
    [Uri-Path: /3/0/13]
    [Request In: 217]
    [Response Time: 0.014677148 seconds]
Lightweight M2M TLV, Device (1 element)

See in my example : Content-type: application/vnd.oma.lwm2m+tlv

Just speculation,

I'll try to capture the exact payload but the problem is this happens rarely (maybe once or several times per day).

It seems at some point we get a double (a floating point number) in Leshan where we expect a long (an integer).
In cbor number format is pretty clear but in json this is more fuzzy.
So I will bet that your application rather use senml+json ?
My guess is when the timestamp ends with several 0, your client  will use the exponential format.
E.g. in your stacktrace :  1.638435E9 for<PHONE_NUMBER> (for 2021-12-02T08:50:00.000Z)
I also guess our json library (jackson) convert a timestamp like 1.638435E9 in double and<PHONE_NUMBER> in long.
Why sometime it works ? because when timestamp does ends with many 0 maybe you client use the "normal notation" like<PHONE_NUMBER>
Anyway, this is too many guess and I would like to be sure so I wait for more input from you.

Any news about this ? I'm really curious to know what happens as maybe we should be more flexible here. :thinking:

Not yet. I tried to record the messages using Wireshark during the night to catch this error but system crashed somewhere at midnight due to a lack of memory..
I'll post Wireshark capture as soon as I'll have it.

Content format is "Content-Format":"application/senml+json".

OK so I will try to test how Leshan behave with JSON number exponential notation.

I'm able to reproduce it : https://github.com/eclipse/leshan/pull/1174.
I will try to add a safe conversion in NumberUtils.

@sonny82  #1174 should fix your issue.

Great, thanks.

This is not yet available in master
If you can test the PR and tell me if it works for you, this would be great.
If you can not test, let me know this way I don't wait for you before to merge. :wink:

I'm using senml-cbor now. In order to test this properly I would have to let it run for at least one day, but we are solving some other issues now, so I'm not sure when I'll be able to do it. Please, don't wait with the merge (regarding the fact hat you are able to reproduce this issue in a unit test). In any case, I'll do some more testing when I'll have time and I'll also test using senml-json. BR

@sonny82  could I ask how you are using Leshan ?
If it makes sense maybe you could consider to comment at : https://github.com/eclipse/leshan/issues/830

Sure.
We are working as an outsourcing partner for a company that has an existing generic backend for managing different kinds of devices (electric meters, water meters, heat meters, ect). On this specific project they have to integrate and manage around 13.000 heat meters, where each meter has an attached NB-IoT module that supports LWM2M 1.1.
Meter data will be sent using MQTT-SN protocol, LWM2M (Leshan 2.0) will be used for bootstrapping (LWM2M DM server + MQTT-SN server) and for management of the NB-IoT module (FW upgrades, tracking battery and signal strength information, configuration of readout and report intervals, etc.).
Devices will be battery powered and will use queue mode.
In the first version we will not use Leshan cluster (not sure if it's really needed as the number of devices is not that big).
We will use DTLS. Due to security reasons we are not allowed to use existing Redis or file based security stores.
We will retrieve keys from a secure key store on demand when needed and then use in-memory security and configuration store.
Communication with the backend will be done using the REST API. We developed a simple API using async servlets.
End customer doesn't need GUI (backend has it's own GUI), however, demo-GUI will also be deployed for easier admin-management.
We are not in production yet.
I'll also write a comment to #830.

hi,
now we found a similar problem using SENM_CBOR:

#1174 should fix this kind of problem for senML+cbor too. Which version are you using ?
Note that : even if I add this more flexible behavior which tolerates to convert Double in Long safely.
I feel that your device behaves strangely ... maybe you should report it ?
For JSON, there is no clear way to know the type of a number.
A common way is to consider that :

3000.0 is a floating point number (float/double in java)
3E3 is a floating point number (float/double in java)
3000 is a integer (short/integer/long in java) .
So OK maybe it's OK to return 1.638435E9 insteadof<PHONE_NUMBER> but I feel this is not the best move and so other implementation could have issue too.

For CBOR, I feel this is even more wrong as the LWM2M specify that integer representation must be used an integer.
See http://www.openmobilealliance.org/release/LightweightM2M/V1_1_1-20190617-A/HTML-Version/OMA-TS-LightweightM2M_Core-V1_1_1-20190617-A.html#Table-C-2-Data-Type-Mapping
Reading the specification again, I'm wondering if Leshan should tolerate this by default for senml+cbor. I feel he should not.

We don't use your fix yet.
Yes, I think I will contact the manufacturer regarding this issue.

I create an issue about this : https://github.com/eclipse/leshan/issues/1176

I contacted the manufacturer regarding the senml-cbor decoding issue.
They think that the problem might be on the Leshan side in the following bits of code:
https://github.com/eclipse/leshan/blob/da6d32e5079e393e58cba6379223d7d721e6e549/leshan-core/src/main/java/org/eclipse/leshan/senml/cbor/upokecenter/SenMLCborPackSerDes.java#L83
Their comment: "Regardless of BigNum being Integer, it is set as float."

I think their are wrong. :thinking:

If this was true, why sometime it works and sometime it does not.
record.setFloatValue method name is misleading but we don't set a float here but a Number which could be an integer. 'it should be renamed)
there is a simple way to verify. Instead of looking at Leshan code, just capture the traffic and look at the payload send by the device if this is a float number at cbor point of view. I feel this is an issue at client side. For now without more information, I bet on this.

Hi,
I'm attaching Wireshark captures for both senml+json and senml+cbor (I'm sorry it took so long).
It fails when the integer value I'm trying to read is a thousand-round value (in this case 3000).
If I change the value to 2999 decoding works.
Note: this has been reproduced with the 2.0.0-M5 version (#1174 not applied yet).
If I read these captures correctly I see that also if I set content format in request to senml+cbor, I'm getting senml+json response..
wireshark_captures.zip

About the float number SenML-JSON issue :
Captures confirm what I explained in : https://github.com/eclipse/leshan/issues/1170#issuecomment-988627519
3e3 notation is used and so is considered as a Float by jackson.
Even if I don't recommend to use exponential notation in JSON for Integer. We can not consider it as a bug at client side.
#1174 should fix the problem at server side. So I think we are good here.
Not related to this issue but looking your capture, I feel there is maybe some issue at client side :
1) too many answer about DTLS retransmission ?
See failure_senml-json.pcapng :
1	0.000000	2022-01-04 07:27:36,129189	<IP_ADDRESS>	<IP_ADDRESS>	5683	9180	CoAP	67	CON, MID:49225, GET, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35	Error	2022-01-04 07:27:36,129189
2	2.030676	2022-01-04 07:27:38,159865	<IP_ADDRESS>	<IP_ADDRESS>	5683	9180	CoAP	67	CON, MID:49225, GET, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35 [Retransmission]	Note	2022-01-04 07:27:38,159865
3	6.092597	2022-01-04 07:27:42,221786	<IP_ADDRESS>	<IP_ADDRESS>	5683	9180	CoAP	67	CON, MID:49225, GET, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35 [Retransmission]	Note	2022-01-04 07:27:42,221786
4	14.213182	2022-01-04 07:27:50,342371	<IP_ADDRESS>	<IP_ADDRESS>	5683	9180	CoAP	67	CON, MID:49225, GET, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35 [Retransmission]	Note	2022-01-04 07:27:50,342371
5	21.520511	2022-01-04 07:27:57,649700	<IP_ADDRESS>	<IP_ADDRESS>	9180	5683	CoAP	86	ACK, MID:49225, 2.05 Content, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35		2022-01-04 07:27:57,649700
6	21.534478	2022-01-04 07:27:57,663667	<IP_ADDRESS>	<IP_ADDRESS>	9180	5683	CoAP	86	ACK, MID:49225, 2.05 Content, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35		2022-01-04 07:27:57,663667
7	21.551604	2022-01-04 07:27:57,680793	<IP_ADDRESS>	<IP_ADDRESS>	9180	5683	CoAP	86	ACK, MID:49225, 2.05 Content, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35		2022-01-04 07:27:57,680793
8	22.800684	2022-01-04 07:27:58,929873	<IP_ADDRESS>	<IP_ADDRESS>	9180	5683	CoAP	86	ACK, MID:49225, 2.05 Content, TKN:00 fc ee 4e 86 1a cb 5c, /33906/0/35		2022-01-04 07:27:58,929873

There is a lot of retransmission because it seems that device takes long time too answer.  (1 request + 3 retransmission)
Then we get 4 answers in a short frame time.
Maybe this is a network issue but here I would expect only 1 answer from the device (maybe 2 because of network latency) but 4 sounds a bit strange.
(Anyway this is not really a bug at client side but something which could maybe be improved)
2) request where  senml-cbor is asked and senml-json is returned
I think this behavior does not follow the LWM2M specification.
The specification says :

An LwM2M Server data request MAY contain an option specifying the data formats the Server would prefer to receive for the payload; if this data format is not accepted by the LwM2M Client, the request is rejected;

And

Operations
Available CoAP Response Codes
Reason Phrase

Read
4.06 Not Acceptable
None of the preferred Content-Formats can be returned

(Table: 6.7.-3 Response Codes: Device Management and Service Enablement Interface)
So either the device should return payload using senml+cbor or it should return 4.06 error.
(At least this is my understanding)

Thank you.
I can confirm that applying #1174 solves this issue.

Thx again for taking time to reporting this. That's help a lot :pray:


remove external scrips and css

don't use external resources
include them and load them with laravel mix and call them locally
<script src="https://code.jquery.com/jquery-3.3.1.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js"></script>
<script src="https://cdn.usebootstrap.com/bootstrap/4.1.1/js/bootstrap.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/<EMAIL_ADDRESS><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.perfect-scrollbar/1.4.0/perfect-scrollbar.js"></script>

In the very next version we are going to totally remove layout and auth files from laravel generator and instead of that, we will be utilizing laravel fortify. If you quickly want to do that, I will suggest using the https://github.com/InfyOmLabs/laravel-ui-coreui package.
After generating scaffold you may need to generate auth scaffold via https://github.com/InfyOmLabs/laravel-ui-coreui steps and it should replace your existing auth files.
I'm not yet fully sure, it may just work by following the above steps. Maybe some additional steps needed.


Runtime: Refactor reconciliation

Task breakdown:

[x] #2729
[x] #2732
[x] #2733
[x] #2730
[x] https://github.com/rilldata/rill/pull/2761
[x] #2782
[x] https://github.com/rilldata/rill/pull/2813
[x] https://github.com/rilldata/rill/pull/2814
[x] Refactor connection cache for acquire-like semantics (so it doesn't close connections that are in use) (https://github.com/rilldata/rill/pull/2875)
[x] Leverage connector declarations in rill.yaml (add a default connector for stage.db) (https://github.com/rilldata/rill/pull/2875)
[x] Implement source reconciler (https://github.com/rilldata/rill/pull/2786)
[x] Implement model reconciler (https://github.com/rilldata/rill/pull/2786)
[x] Implement metrics view reconciler (https://github.com/rilldata/rill/pull/2786)
[x] Implement migration reconciler (https://github.com/rilldata/rill/pull/2786)
[x] #2904
[x] Implement controller event loop https://github.com/rilldata/rill/pull/2946
[x] Implement new catalog format and caching
[x] Merge https://github.com/rilldata/rill/pull/2889
[x] Replace FindInstance with cached lookups
[x] Simplify transporters to accept a uniform config object
[x] Controller: Before moving from pending to running, clear error
[x] Clone Get's from a reconciler if mutating the result
[x] Make calls to UpdateSpec(trigger=false) safe (in case projectparser updates at same time)
[x] Undelete if creating something deleted. Error if updating something deleted.
[x] Create probably has issue with State not being set. Should call New on reconciler.
[x] Lower name unique is enforced in parser. Also needed in catalog, controller, and in ProjectParser
[x] Rename package dag2 to dag
[x] Adds registryCache that caches instances in-memory and starts/stops a Controller for each instance
[x] Integrate into local CLI
[x] Removes old reconcile
[x] [P0] Reconcilers: Add refs checks for failed parents
[x] [P0] Limit concurrency of DuckDB source ingest/etc., so we don't block all connections
[x] [P0] Increase rill_pool_size on local app https://github.com/rilldata/rill/pull/3045
[x] [P0] Incorporate DuckDB transactions in new reconcile: https://github.com/rilldata/rill/pull/2897
[x] [P0] Fix TODOs in code
[x] https://github.com/rilldata/rill/pull/3057
[x] [P0] Move connector-specific source property parsing and backwards compatibility rules into individual connectors/transporters (such as csv.delimiter, etc.) https://github.com/rilldata/rill/pull/3072
[x] [P0] Parsing ExtractPolicy
[x] [P0] Opening srcConnector with correct variables (connectorVariables) https://github.com/rilldata/rill/pull/3076 https://github.com/rilldata/rill/pull/3072
[x] [P0] Rewrite SQL sources for s3/gcs (see mergeFromParsedQuery and https://github.com/rilldata/rill/pull/2915) https://github.com/rilldata/rill/pull/3082
[x] [P0] Enforce metrics policies in new resource APIs https://github.com/rilldata/rill/pull/3083
[x] [P0] Replace old reconcile with new reconcile https://github.com/rilldata/rill/pull/3092
[x] [P0] Use new catalog resource types in analytical APIs https://github.com/rilldata/rill/pull/3092
[x] [P0] Weird logs when opening on an uninitialized project (@begelundmuller)
[x] [P1] Fix renames not working for models (they become deletes)
[x] [P1] Deduplicate file watcher paths during batching
[x] [P1] Parser: Prevent distinct files overriding the same model (change upsertResource to insertResource?) (@begelundmuller)
[x] [P2] Don't use LongRunning for model views
[x] [P0] Implement stubbed tests in runtime/controller_test.go
[x] [P0] Integrate new reconcile with admin server (@begelundmuller)

Closed by https://github.com/rilldata/rill/pull/3092


⚠ Samsung has degraded performance

In 2209967, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 3155 ms

Resolved: Samsung performance has improved in e722c44.


Feature request. Return the content of the page if possible

I see an unrealized possibility where the get method could supply the contents of the response itself to the callback function, in addition to the response code.
Do you see the possibility of such a use of this program?

Oh! I see.
Thanks @tcort


Drop advanced configuration options

TL;DR
There are so many configuration options in release-please. It is infeasible to continue to pipe every option as an input. There are a few rare options that should be kept to make a quick start possible (release-type, path, fork).
Detailed design
Any advanced configuration options should use a release-please manifest config. This will allow users to also use the release-please CLI to debug their configuration which is not possible when configuring your repository using action inputs.
Additional information
No response

this is making our release cycle so much more difficult or even impossible :/


Timeout when trying to use

Hey, this is a awesome initiative. I thought I'd give it a spin.
I'm getting a timeout message off the bat:
portal:       "type": "ServiceBusError",
portal:       "message": "The request with message_id \"57f834fd-c7ab-b748-b045-81ca07f5446c\" to \"address\" endpoint timed out. Please try again later.",
portal:       "stack":
portal:           ServiceBusError: The request with message_id "57f834fd-c7ab-b748-b045-81ca07f5446c" to "address" endpoint timed out. Please try again later.

Inside docker have this.
2023-03-25 10:15:41 [02:15:41.747] SEND (ch=0) disposition(role:True,first:0,last:0,settled:True,state:rejected(error:error(condition:amqp:not-found,description:Not response link was found. Ensure the link is attached or reply-to is set on the request.)),batchable:False)
2023-03-25 10:15:41 [02:15:41.747] SEND (ch=0) close(error:error(condition:amqp:internal-error,description:Object reference not set to an instance of an object.))

Am on a M1 mac, happy to dig in with a bit of direction about best way to debug or where to look.

i am experiencing a similar issue (note: i am using nodejs with azure sdk for js)
info: ServiceBusEmulator.ServiceBusEmulatorWorker[0]
      Worker started at: 04/13/2023 21:56:30 +00:00
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Docker
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost/health - -
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost/health - - - 200 - text/plain 192.2564ms
warn: ServiceBusEmulator.ServiceBusEmulatorHost[0]
      AMQP SSL errors RemoteCertificateNotAvailable.
[09:56:39.850] SEND AMQP 3 1 0 0
[09:56:39.856] SEND sasl-mechanisms(sasl-server-mechanisms:[MSSBCBS,EXTERNAL,ANONYMOUS])
[09:56:39.871] RECV sasl-init(mechanism:ANONYMOUS,initial-response:...,hostname:sbemulator)
[09:56:39.874] SEND sasl-outcome(code:Ok)
[09:56:39.881] SEND AMQP 0 1.0.0
[09:56:39.883] SEND (ch=0) open(container-id:ContainerHost-867a9581d1124f6f9c088071de7ab000,host-name:localhost,max-frame-size:262144,channel-max:255,idle-time-out:2147483647)
[09:56:39.887] RECV AMQP 0 1 0 0
[09:56:39.890] RECV (ch=0) open(container-id:487d68d9-d7e5-b44d-bbed-09fba0a1aec0,host-name:sbemulator,max-frame-size:0,channel-max:0,idle-time-out:60000,properties:[product:MSJSClient,version:7.9.0,user-agent:azsdk-js-azureservicebus/7.9.0 Node/v16.19.1 OS/(x64-Linux-4.19.0-23-amd64),platform:(x64-Linux-4.19.0-23-amd64),framework:Node/v16.19.1])
[09:56:39.895] RECV (ch=0) begin(remote-channel:0,next-outgoing-id:0,incoming-window:2048,outgoing-window:4294967295,handle-max:0)
[09:56:39.899] SEND (ch=0) begin(remote-channel:0,next-outgoing-id:0,incoming-window:2048,outgoing-window:2048,handle-max:63)
[09:56:39.915] RECV (ch=0) attach(name:e7778c9c-3aef-a046-a462-3bd4229ebc99,handle:0,role:False,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(durable:0,timeout:0,dynamic:False),target:target(address:$cbs,durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:0)
[09:56:39.931] SEND (ch=0) attach(name:e7778c9c-3aef-a046-a462-3bd4229ebc99,handle:0,role:True,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(durable:0,timeout:0,dynamic:False),target:target(address:$cbs,durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:2147483647)
[09:56:39.937] SEND (ch=0) flow(next-in-id:0,in-window:2048,next-out-id:0,out-window:2048,handle:0,delivery-count:0,link-credit:100,available:0,drain:False,echo:False)
[09:56:39.941] RECV (ch=0) attach(name:cbs-de597895-e2fb-f444-a760-3f71f67a4563,handle:1,role:True,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(address:$cbs,durable:0,timeout:0,dynamic:False),target:target(durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:0)
warn: ServiceBusEmulator.ServiceBusEmulatorHost[0]
      AMQP SSL errors RemoteCertificateNotAvailable.
[09:56:39.953] SEND AMQP 3 1 0 0
[09:56:39.955] SEND sasl-mechanisms(sasl-server-mechanisms:[MSSBCBS,EXTERNAL,ANONYMOUS])
[09:56:39.956] RECV sasl-init(mechanism:ANONYMOUS,initial-response:...,hostname:sbemulator)
[09:56:39.958] SEND sasl-outcome(code:Ok)
[09:56:39.959] SEND AMQP 0 1.0.0
[09:56:39.960] SEND (ch=0) open(container-id:ContainerHost-867a9581d1124f6f9c088071de7ab000,host-name:localhost,max-frame-size:262144,channel-max:255,idle-time-out:2147483647)
[09:56:39.962] RECV AMQP 0 1 0 0
[09:56:39.963] RECV (ch=0) open(container-id:487d68d9-d7e5-b44d-bbed-09fba0a1aec0,host-name:sbemulator,max-frame-size:0,channel-max:0,idle-time-out:60000,properties:[product:MSJSClient,version:7.9.0,user-agent:azsdk-js-azureservicebus/7.9.0 Node/v16.19.1 OS/(x64-Linux-4.19.0-23-amd64),platform:(x64-Linux-4.19.0-23-amd64),framework:Node/v16.19.1])
[09:56:39.964] RECV (ch=0) begin(remote-channel:0,next-outgoing-id:0,incoming-window:2048,outgoing-window:4294967295,handle-max:0)
[09:56:39.964] SEND (ch=0) begin(remote-channel:0,next-outgoing-id:0,incoming-window:2048,outgoing-window:2048,handle-max:63)
[09:56:39.967] RECV (ch=0) attach(name:33404695-cc7e-4d42-9466-7d049479bd7a,handle:0,role:False,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(durable:0,timeout:0,dynamic:False),target:target(address:$cbs,durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:0)
[09:56:39.970] SEND (ch=0) attach(name:33404695-cc7e-4d42-9466-7d049479bd7a,handle:0,role:True,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(durable:0,timeout:0,dynamic:False),target:target(address:$cbs,durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:2147483647)
[09:56:39.972] SEND (ch=0) flow(next-in-id:0,in-window:2048,next-out-id:0,out-window:2048,handle:0,delivery-count:0,link-credit:100,available:0,drain:False,echo:False)
[09:56:39.973] SEND (ch=0) attach(name:cbs-de597895-e2fb-f444-a760-3f71f67a4563,handle:1,role:False,snd-settle-mode:Unsettled,rcv-settle-mode:First,incomplete-unsettled:False,initial-delivery-count:0,max-message-size:2147483647)
[09:56:39.977] RECV (ch=0) attach(name:cbs-5829df45-3780-f549-99bb-4d5ee235aec2,handle:1,role:True,snd-settle-mode:Unsettled,rcv-settle-mode:First,source:source(address:$cbs,durable:0,timeout:0,dynamic:False),target:target(durable:0,timeout:0,dynamic:False),incomplete-unsettled:False,initial-delivery-count:0,max-message-size:0)
[09:56:39.983] SEND (ch=0) attach(name:cbs-5829df45-3780-f549-99bb-4d5ee235aec2,handle:1,role:False,snd-settle-mode:Unsettled,rcv-settle-mode:First,incomplete-unsettled:False,initial-delivery-count:0,max-message-size:2147483647)
[09:56:39.976] SEND (ch=0) detach(handle:1,closed:True,error:error(condition:amqp:internal-error,description:Value cannot be null. (Parameter 'key')))
[09:56:39.984] SEND (ch=0) detach(handle:1,closed:True,error:error(condition:amqp:internal-error,description:Value cannot be null. (Parameter 'key')))
[09:56:39.994] RECV (ch=0) flow(next-in-id:0,in-window:2048,next-out-id:0,out-window:4294967295,handle:1,delivery-count:0,link-credit:1000,available:0,drain:False,echo:False)
[09:56:40.001] RECV (ch=0) flow(next-in-id:0,in-window:2048,next-out-id:0,out-window:4294967295,handle:1,delivery-count:0,link-credit:1000,available:0,drain:False,echo:False)
[09:56:40.022] RECV (ch=0) transfer(handle:0,delivery-id:0,delivery-tag:30,message-format:0,settled:False,more:False,rcv-settle-mode:First,resume:False,aborted:False,batchable:False) payload 450
[09:56:40.033] SEND (ch=0) disposition(role:True,first:0,last:0,settled:True,state:rejected(error:error(condition:amqp:not-found,description:Not response link was found. Ensure the link is attached or reply-to is set on the request.)),batchable:False)
[09:56:40.041] RECV (ch=0) transfer(handle:0,delivery-id:0,delivery-tag:30,message-format:0,settled:False,more:False,rcv-settle-mode:First,resume:False,aborted:False,batchable:False) payload 380
[09:56:40.043] SEND (ch=0) disposition(role:True,first:0,last:0,settled:True,state:rejected(error:error(condition:amqp:not-found,description:Not response link was found. Ensure the link is attached or reply-to is set on the request.)),batchable:False)
[09:56:40.114] SEND (ch=0) close(error:error(condition:amqp:internal-error,description:Object reference not set to an instance of an object.))
[09:56:40.117] SEND (ch=0) close(error:error(condition:amqp:internal-error,description:Object reference not set to an instance of an object.))

the Not response link... is coming from a dependency lib amqpnetlite here:
https://github.com/Azure/amqpnetlite/blob/c7596e6f039d8dee706e7d126c30e83cbfa0205e/src/Listener/ContainerHost.cs#L637
the other error: Value cannot be null. (Parameter 'key') - i have no idea what the origin is
the message being sent is very simple:
const { ServiceBusClient } = require('@azure/service-bus')

const connectionString = 'Endpoint=sb://sbemulator/;SharedAccessKeyName=all;SharedAccessKey=CLwo3FQ3S39Z4pFOQDefaiUd1dSsli4XOAj3Y9Uh1E=;EnableAmqpLinkRedirect=false'

const test = async () => {
  const sbClient = new ServiceBusClient(connectionString)
  const sender = await sbClient.createSender('some-topic', 'some-subscription')
  const msg = {
	  body: {test: 1, something: 'interesting'},
	  subject: 'test subject',
	  contentType: 'application/json',
  }

  await sender.sendMessages([msg])
}

test()

my setup is like this:

build the container: devopsifyme-sbemulator$ docker build -t sbemulator:latest -f ./src/ServiceBusEmulator.Host/Dockerfile .
run the dependencies (note: creates ado_default network): docker-compose -f .ado/docker-compose.common.yml up --force-recreate --renew-anon-volumes --remove-orphans --abort-on-container-exit
after dependencies start up - run the container  docker run --name sbemulator  --network ado_default -e Emulator__RabbitMq__Host=rabbit -e Emulator__RabbitMq__Username=guest -e Emulator__RabbitMq__Password=guest -e Emulator__QueuesAndTopics=some-topic/Subscriptions/some-subscription sbemulator

contents of the docker-compose dependencies file (.ado/docker-compose.common.yml):
version: '3.9'
services:
  rabbit:
    container_name: "rabbit"
    image: rabbitmq:3.11.6-management
    ports:
      - "15672:15672"
      - "5672:5672"
    healthcheck:
      test: rabbitmq-diagnostics check_port_connectivity
      interval: 5s
      timeout: 60s
      retries: 20

is there any special requirements that i am missing?
maybe i messed up the setup?
how likely is the problem related to the cacert.cer?
other than that i just wanted to say that this service-bus mocking opportunity is something that i am really excited about, hopefuly we can somehow pinpoint and resolve the issues so that it works out of the box. congrats on the idea.
cheers
rafal.

Hey guys, I am not a node.js wizard myself :( Would one of you be so kind and upload an archive with a ready-to-go project that I could use for troubleshooting? :)
Since it looks like a classic "Object reference not set to an instance of an object." exception, I would probably like to just run the whole thing in Visual Studio on my end, then start the sample node.js app, wait for the breakpoint and go from there ;)

@piotr-rojek i uploaded a "ready to run" package (link below).
intructions:

extract the tgz package
cd sbemu-node
if you have nodejs installed you can just run it locally or via docker

run locally:

npm install
npm start

run with docker:

npm run docker:build
npm run docker:run or npm run docker:bash and then in the container node index.js

if you do not have nodejs installed you can build a docker image (NODE:16):

docker build -f Dockerfile .
docker-compose up or `docker-compose --env-file=.env up

NOTE: you can inspect the package.json file - the scripts section has the docker build and run scripts if you run into any trouble.
the following env variables are expected or available:

required (see .env file for examples)

CONNECTION_STRING - required
TOPIC - required
SUBSCRIPTION - required

NOTE: you can change the variable values passed to the container in the .env file OR in the docker-compose.node.yml file OR by setting them via command line CONNECTION_STRING=.... TOPIC=... SUBSCRIPTION=... docker-compose up
i only tested running the scripts (using npm run ... ) and locally (running npm install and npm start)

optional:

PORT - optional (default: 3000, remember to change the ports section in docker-compose)
LOG_LEVEL - optional (default: 'info')

after running the package you will have an api endpoint exposed /publish that should work with the connection string and when you POST data to it i will try to publish the message to the topic/subscription that you point it to via mentioned env variables.
example (here i am running this on a VM via docker-compose and exposing the api via port 3000 by default):
curl --request POST \
  --url http://<IP_ADDRESS>:3000/publish \
  --header 'Content-Type: application/json' \
  --data '{
  "test": "123"
}'

NOTE: you might need to add your sbemulator service to the docker compose file or have it run in the same docker network, but let's not get ahead of ourselves: try to run this and please let me know if you run into any trouble so that i can help.
NOTE: i did not add any ENV or ARG to the Dockerfile, but maybe the mentioned env vars should be there for readability?
and now the link:
sbemu-node  (click download)
cheers, hopefuly this makes sense
rafal.

@rafak  thanks! I am getting "File not found or corrupt" when trying to download though ;(

As per our QnA discussion I thought I would post my simple set up here for you.
I followed your setup instructions to run the docker images.

make sure node is installed (I am using npm and windows)
create a client folder for the nodejs project
cd to the folder and run npm init accept all the defaults or paste the following into a package.json

{
  "name": "sbe-testclient",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@azure/service-bus": "^7.9.0",
    "dotenv": "^16.0.3"
  }
}

set the certificate so that node knows where it is, on the command prompt
set NODE_EXTRA_CA_CERTS=C:\\localpath\\devopsifyme-sbemulator\\docker\\rabbitmq-amqp1\\testca\\cacert.pem
run npm install
create a index.js file and paste the following in

const { ServiceBusClient } = require("@azure/service-bus");

// Define connection string and related Service Bus entity names here
const connectionString = "Endpoint=sb://localhost/;SharedAccessKeyName=all;SharedAccessKey=CLwo3FQ3S39Z4pFOQDefaiUd1dSsli4XOAj3Y9Uh1E=;EnableAmqpLinkRedirect=false";
const queueName = "testQ";

async function main() {
  const sbClient = new ServiceBusClient(connectionString);

  // createSender() can also be used to create a sender for a topic.
  const sender = sbClient.createSender(queueName);

  try {
    // Send a single message
    console.log(`Sending one scientists`);
    const message = {
      contentType: "application/json",
      subject: "Scientist",
      body: { firstName: "Albert", lastName: "Einstein" },
      timeToLive: 2 * 60 * 1000, // message expires in 2 minutes
    };
    await sender.sendMessages(message);

    // Close the sender
    console.log(`Done sending, closing...`);
    await sender.close();
  } finally {
    await sbClient.close();
  }
}

main().catch((err) => {
  console.log("sendMessages Sample: Error occurred: ", err);
  process.exit(1);
});

module.exports = { main };

execute the script npm run start

@piotr-rojek  sorry, trying out new stuff, seems like it disappeared,
try this one (we transfer)

Hey guys, could you please verify the just released v0.1.9? I tested it with the provided code above, so at least sending and receiving should work fine. Turned out it was missing some fields in the message that the SDK expected. Also, one expected field was not sent by node.js to the emulator on AMQP level.
Reach out if there are more issues to address! Thanks for the code snippets, they helped me a lot! Much appreciated ;) It would probably be beneficial to include integration tests for nodejs - let me know if someone is keen on helping with that :)
/Piotr

will test, we can work on some tests too.

@piotr-rojek looks much better, the errors are not showing, i am getting something different now but it must be the cacert setup
{
	"err": {
		"code": "UNABLE_TO_VERIFY_LEAF_SIGNATURE"
	}
}

sbemu-node    |     {"msg":"service bus error","data":{"error":{"code":"UNABLE_TO_VERIFY_LEAF_SIGNATURE"},"errorSource":"receive","entityPath":"test-topic/Subscriptions/test-subscription","fullyQualifiedNamespace":"sbemulator","identifier":"test-topic/Subscriptions/test-subscription-1c505ceb-8c26-6e49-8e8b-94818b6f3540"}}

i will try to work on the certs, i guess i need to add the cacert to the sbemu-node image, will let you know.

Great news ;)
I would prefer to handle "smoke test" level scenarios for integration tests. We could add a container running tests to the docker-compose.integration.yml file. Existing c# tests simply send a message and assert that it can be received.


Fix for #237

Though this fix does I believe resolve the problem, perhaps there is a more elegant fix that avoids the need to store/test the variable distanceCaller?

@Thargoids You need to do a pull on the dev branch, merge dev into your branch, fix conflicts and commit those changes


Remove SCSS and Format all CSS files

This PR removes our dependency on SASS, except for styles.scss which will be formatted and removed later.
SASS has proven not to be very useful and the dev team seems to continue to accidentally update the CSS files directly instead of relying on the SASS build.
Additionally, SASS encourages complicated nested CSS selectors that needlessly increase the file size.
Going back to basics in true Edge Deliver spirit.
Test URLs:
Before: https://main--exlm--adobe-experience-league.aem.live?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live/en/docs?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live/en/docs/analytics?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live/en/docs/analytics/analyze/admin-overview/analytics-overview?martech=off
After: https://experienceleague-dev.adobe.com/en/playlists/acrobat-sign-perform-advanced-tasks-administrators?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live/en/perspectives?martech=off
After: https://remove-sass--exlm--adobe-experience-league.aem.live/en/perspectives/building-engagement-programs?martech=off

How about removing sass compilation and other general SCSS files?
_breakpoints.scss, _mixins.scss,  _typography.scss and styles.scss under styles.
removing sass-compile.js
removing sass related scripts in package.json