More VFS Cleanup

Brings things almost entirely into alignment with the quasar FS semantics.

Lots of new integration tests enabled
Error reporting works now (btw: reading a non-existent file should be an error, not an empty dataset!!)
Manage, read and write file is all in place

One test is "pendinged" still because it's really, really slow

Executing queries (as opposed to evaluating)
Move error semantics are now in alignment with what quasar wants, rather than what precog wants
Other improvements that I've forgotten about

Fixes #2421
Fixes #2253

@djspiewak so I'm a bit concerned about how slowly some of the newly-enable file system tests are running. Do you have a sense of what's making them slow and how we can improve their performance?

@djspiewak since this enables new file system functionality for mimir, I think this should be "feature" version instead of "revision". Thoughts?

@djspiewak I'm actually confused why the mimir build got so much slower for this branch. sbt it/testOnly *FilesSpec* on this branch locally completes in a matter of seconds, so did the full stack integration tests get slower? We should "timeout" any that are timing out. I have them running locally to see if any fall under this category. But it seems that none of the new tests you have enabled are slow.

@alissapajer

since this enables new file system functionality for mimir, I think this should be "feature" version instead of "revision". Thoughts?

I'm on the fence about that.  I could see it either way.  I selected revision mostly because it's more than existing functionality is being un-errored rather than new stuff being "added" per se.  Maybe that's a meaningless distinction.  :-)

@djspiewak regarding "revision" versus "feature" version label, if some functions are un-errored then they were effectively non-existent before. So even though the signatures were there and callable, you couldn't argue that the feature was actually implemented, only that the signatures were flushed out.


Bug - Goroutine Leaks in Worker

What is the issue?
While running integration tests in resp server, I noticed goroutines leaks. As integration test progress, number of goroutines keep on increasing arbitrarily.
Steps to reproduce
Use pprof for checking goroutine traces. Add it in TestMain function like this
import (
	"net/http"
	_ "net/http/pprof"
)

func TestMain(m *testing.M) {
	go func() {
		fmt.Println(http.ListenAndServe("localhost:6060", nil))
	}()

This will now let us see profile at this dashboard http://localhost:6060/debug/pprof/ at any given point in the program. Add time.Sleep to pause the program at a particular point, preferably at the test start to see goroutines traces after previous test completion. You can run the entire test suite or a few selected tests.
If we put test to sleep at the start the first test TestAPPEND, we will notice all the initial goroutines started by TestMain. Following is the list of these 13 goroutines:
A. 8 goroutines are directly spawned from dicedb codebase.
Nested goroutines are child goroutines.

TestMain: main goroutine (--> M.Run --> T.Run)

Resp Server

Accept Connection Requests
Watch Manager

Listen For Events

Shard Manager

Shard Thread

Goroutine to handle error on global error channel gec

B. 5 other goroutines
2 goroutines related to pprof:

pprof.writeRuntimeProfile to write profile
server to access profile data ListenAndServe, that we added above

1 goroutine for time.Sleep
1 goroutine to handle low level signals - Sigqueue
1 gorutine from http/server.go, probably to support http servers
Ideally, number of goroutines should fall back to these 13 gorutines once a test is finished executing. You will notice this number will keep going up as you execute more tests.
Note: You might see 15 goroutines instead of 13 as TestMain also acquires a connection to fire ABORT command after all tests are executed. TestMain should acuqire this connection just before firing the command. I will be fixing that too as part of this issue, so I am ignore it.
Why is it happening?
Each test, gets a connection using getLocalConnection function. This request is received by AcceptConnectionRequests goroutine and it creates a new worker goroutine (G1) BaseWorker.Start. This worker goroutine creates another goroutine (G2) to read responses from the server BaseWorker.Read.
So every time a connection is requested, 2 new goroutines are created. G2 reads and G1 processes. G2 synchronises with G1 before exiting, but G1 doesn't. In fact G1 can return for various reasons and it doesn't communicate this to G2 which results in Leakages.

I will raise fix PR for this in some time.


Allow reuse of slices exposed through Ready

Solve https://github.com/etcd-io/raft/issues/15:
Use pooled slice to reduce the total bytes allocated.
Benchmark result:
 % benchdiff --old 177ef28a -r BenchmarkRawNode -c 5 .
test binaries already exist for '177ef28'; skipping build
checking out 'e5ac368'
building benchmark binaries for 'e5ac368' 1/1 -

  pkg=1/1 iter=5/5 go.etcd.io/raft/v3 |

name                    old time/op        new time/op        delta
RawNode/single-voter-8         922ns ± 3%         883ns ± 2%   -4.14%  (p=0.008 n=5+5)
RawNode/two-voters-8          1.33µs ± 1%        1.36µs ± 1%   +1.79%  (p=0.032 n=4+5)

name                    old firstIndex/op  new firstIndex/op  delta
RawNode/single-voter-8          1.00 ± 0%          1.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            3.00 ± 0%          3.00 ± 0%     ~     (all equal)

name                    old lastIndex/op   new lastIndex/op   delta
RawNode/single-voter-8          2.00 ± 0%          2.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            2.00 ± 0%          2.00 ± 0%     ~     (all equal)

name                    old ready/op       new ready/op       delta
RawNode/single-voter-8          2.00 ± 0%          2.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            2.00 ± 0%          2.00 ± 0%     ~     (all equal)

name                    old term/op        new term/op        delta
RawNode/single-voter-8          0.00 ± 8%          0.00 ± 0%     ~     (p=0.444 n=5+5)
RawNode/two-voters-8            1.00 ± 0%          1.00 ± 0%     ~     (all equal)

name                    old alloc/op       new alloc/op       delta
RawNode/single-voter-8          551B ± 0%          410B ± 1%  -25.59%  (p=0.000 n=4+5)
RawNode/two-voters-8            843B ± 1%          702B ± 0%  -16.68%  (p=0.008 n=5+5)

name                    old allocs/op      new allocs/op      delta
RawNode/single-voter-8          4.00 ± 0%          4.00 ± 0%     ~     (all equal)
RawNode/two-voters-8            6.00 ± 0%          6.00 ± 0%     ~     (all equal)

@pavelkalinnikov Thank you for your comments! I've resolved them.

we still reallocate and copy the slice.

The trick is at when we put back the slice:
rn.raft.msgsAfterAppendPool.Put(rn.raft.msgsAfterAppend[:0])

This clears the slice but reserves the underlying array, which we can reuse so that we don't need to allocate a new one each iteration.


[Comms] Drop posts/mails to other communities about sponsors

Drop a mail for communities who have mailing lists, and/or posts in communities who have other forms of forums/groups made about existing sponsors and add link to sponsorship deck to call for more.
Announcement Date: Between Aug 16-19, whichever feasible.
Tasks

[ ] Content, @Sc0rpi0n101
[ ] Posting, @WatchfulHitman

Timeline
Pre 4PM

[ ] Mail Content, @Sc0rpi0n101
[ ] Post Content, @Sc0rpi0n101 (if applicable)

Post 5PM

[ ] Post at various places, @WatchfulHitman

Communication Channel (if needed for your task)

[ ] Mailing List

[ ] All Local Python User Groups

[ ] Any other forum/group wherever applicable.

Not relevant anymore.


reCaptcha validation

Hi James,
First of all, thanks for building it. I have been using for a month and it works great. I would like to add reCaptcha function and I researched other AWS Serverless Apps Repositories but I don't get hot to do it.
I open the issue as an enhancement for someone with better programing skills than me.
Best regards.

Sorry, I only just saw this issue - I'll see if I can work a reCAPTCHA example into a future project. Thanks for the idea.


Seriously, which version should I use? Or should I just switch to https://git.io/tdl?

I'm on 2.2.2 now, been getting NETWORK_BAD_REQUEST a lot. Updating to 3.x the code just stuck after I call a telegram api.
I see @goodmind pointing to https://git.io/tdl, should we just switch over to that?

Try 3.2.7


Blur

Continuing my attempts at Issue #90,
I implemented a blur pool layer for future integration.
Although I wanted to copy Kornia's implementation  I found that they tightly coupled max pooling with the downsampling. Also it had an internal dependency on for pyramid downsampling.
As the original implementation  suggests three different strategies for antialiasing(each involving Max and average pooling and strided convolution), I had to decouple the code both from both max pooling and the internal dependency.
I ended up re-implementing the whole thing.
However, I did end up making several design choices.
First of all, I only implemented two most common binomial filters(3 and 5) for downsampling.
(Kornia seems to have implemented one while the original repo implements 7)
This is because the original paper says the most common meaning filters to be used would be the filters of 3 and 5, and since the code base I'm working up to(#90) tested only 3 and 5 (and settled for 3). It is not to hard to implement other filters should the need arise.
Secondly, I removed most options for several padding strategies since neither kornia, the original anti-aliasing paper nor #90 discusses the tradeoffs for each padding and just used the original reflective padding strategy.
I thought about the consequences of the various padding strategies but it seems that only the makes original one makes sense.
Currently the blurpool function is not used by ResNet as intended.
This is because further design choices are necessary.
Although potentially any and all maxpool, avg-pool and s=2 strided conv can be blurred with this,
the original paper and #90 show that it is both unnecessary and inefficient.
So my next step would be to implement this in the style of #90(certain downsampling strided convs only) and only exposing a boolean argument somewhere.
Next, it would be implemented as value argument to enable either Vanilla, #90 style, and the original paper style (all? strided convs and maxpools).
I guess I'd need to make some design decisions and then see how it works.
In the meantime I would like for you to look at the code and point out improvements or changes you would like to see before I move forward.

@VRandme thanks for the submission, been focused on a few other things lately so a bit slow on this right now, I don't see any issues with what's there so far but ultimately (as you allude to), the success depends on the integration in an actual network.
What sort of testing have you done so far on this? I assume you did some basic checks inputing simple patterns where an inspection would confirm appropriate blur. Have you tried integrating it into a basic network and training?
As for the combination with the other layers (let's call it integration strategies), there was some relevant twitter chatter I was involved in a while back on the topic...
https://twitter.com/ducha_aiki/status/1216859128397750278
... and don't miss this branch of the same convo,  https://twitter.com/evgeniyzhe/status/1218241324740202496

I implemented the rest of the blur.
The option is exposed through a argument on the resnet function
defaults to not applied, blur ='max' applies it in the initial maxpool of the stem, blur = 'strided' for strided convolutions(assembled cnn style), blur = 'max_strided' for both (original paper style). (My push includes the documentation in the comments)
Sample untrained networks for ImageNet are available(resnetblur18 for original style and resnetblur50 for assembled cnn style) but I don't see any use for them aside for testing for internal consistency.
I would like to find a way to import the original weights, but copyright concerns and my general inability to port weights from another code base makes this difficult for me.
I did not spend too much time on how to best incorporate the blur pool to ResNets since in my view, the original authors and the researchers from clova-ai did more than enough testing already. I merely implemented the two best methods they have already found.
As for independent testing, the BlurPool2d function has been tested both manually and against the original implementation for several random inputs.
However, the fully implemented version is almost untested.
I closely examined the code, the outputted models (by print(model)), the output from pytorch summary, and output from random input.
This is far from enough testing obviously, but since I am unable to port the original weights for various reasons at this time I think this is as far as I can go at the moment.
The fact that the original code and weights only deal with ImageNet and not CIFAR does not help.
I think testing wise the following could be done:

find a way to port original weights and test if it works on my blurpool code integrated into your resnet code.
(highly likely since the blurpool itself does not introduce independent parameters. Nonetheless whether it properly ports and runs correctly is important)

train from scratch(at least resnet18 and resnet50 should be done to exhaust major codepaths) and compare results with original paper
(I just realized that the original github provides weights for resnet18 so maybe that testing could be tractable in the near future)

The Corona epidemic has put a lot of strain on my job and time is becoming a more precious resource.
I hope you stay safe.

@VRandme thanks, I'll see if I can get a train session running at some point against a somewhat recent non blur resnet comparison point

@VRandme I fixed the bottleneck and ran a training session on resnetblur50 ... 79.17 top-1 without using the JSD loss, not bad. There is some other cleanup I want to do before merging, but definitely seems worth adding.
Also, noticed someone tweeting about this https://github.com/mrT23/TResNet yesterday, uses the AA blur, a stem modification, and the Inplace ABN for BN

I did read the paper and here are my thoughts.
Many parts are quite an invasive change. They won't be easy or even possible to add as a drop in argument option.
(for example: space to depth stem, mixed basic block and bottleneck blocks in place and fused BN and LRELU)
Most of their ideas are sound. I have much more to comment on that so I will make another issue on this repo discussing which parts might be integrated on resnet and what wouldn't be.
Getting back to the their Blur implementation, you can consider it as a further cutdown and optimized version of my code.
There are some arguments that you can put in the function, but blocked by asserts.
Only the b2(1,2,1) filter and stride of 2,reflect padding is implemented.
Cuda, half precision and JIT is enabled.
Pseudocode is in the paper as follows:

Actual code is contained here
Here is my conclusion:
The authors are aware and even mention the original assembled net paper.
Their testing and implementation is further evidence that the most basic b2 version with stride 2 (kernel size 3x3) is size and computation efficient with tangible accuracy and robustness benefits.
Unless you want to maintain your code base towards the maximally experiment-able direction,
I think just replacing my code with their implementation, or adjusting it to that effect would be the best direction for practicality considering speed and simplicity of code.
Their license is Apache 2.0 so no license concerns either.
When this is merged, I'll make a TResNet issue here discussing overall changes and low hanging fruits regarding this codebase.

@VRandme I'm in no rush to replicate TResNet, just curious if you'd seen it since it's in the same lineage of ResNet experiments. I'm not sure if JIT will actually have an impact for the blur, but that's pretty easy to test.
Agreed that some of the changes are hard to integrate nicely in a on/off fashion that won't impact backwards weight compat. I was thinking of fiddling with the ABN impl, before reading this paper, may try a BnAct factory version of ResNet and see if I can maintain weight compat.
I made some changes to the blur filter generation in this PR so it works for any filter size. Also got the filters out of the state_dict without too much of a hack.

Haven't forgotten about this, since it impacts the main resnet class, wanted to tweak a few things, run full regression test of all the other dep models. I did try a filter size 5 run, was worse than the default 3

hmm considering that the recent TResnet merge included an anti aliasing layer,
where do you want to go with this? I feel like it is ripe for closing at this moment unless there are more you want to implement.
Although I feel like there is enough evidence that filter size 3 is the best compromise for the models and datasets that are widespread, if you would like to test more here are some pointers.
There is an obvious trade off between resource use (runtime and memory use) vs accuracy with increasing filter size.
What is not apparent is the Consistency Tradeoff.
It is not clear consistency(or robustness) has a positive relationship with filter size past 3.
All in all, I personally am content with the #121 pulled antialiasing layer with the JIT optimized code.


Maid fiddler problem and won't connect

Error message
Paste the error message in the area below:
Error  :SybarisLoader] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object
  at Mono.Cecil.Inject.InjectionDefinition.VerifyInjectionDefinition (Mono.Cecil.MethodDefinition injectMethod, Mono.Cecil.MethodDefinition injectTarget, InjectFlags flags, System.Int32[] localVarIDs, Mono.Cecil.FieldDefinition[] memberReferences) [0x00000] in <filename unknown>:0
  at Mono.Cecil.Inject.InjectionDefinition..ctor (Mono.Cecil.MethodDefinition injectTarget, Mono.Cecil.MethodDefinition injectMethod, InjectFlags flags, System.Int32[] localVarIDs, Mono.Cecil.FieldDefinition[] memberReferences) [0x00000] in <filename unknown>:0
  at Mono.Cecil.Inject.MethodDefinitionExtensions.InjectWith (Mono.Cecil.MethodDefinition method, Mono.Cecil.MethodDefinition injectionMethod, Int32 codeOffset, System.Object tag, InjectFlags flags, InjectDirection dir, System.Int32[] localsID, Mono.Cecil.FieldDefinition[] typeFields) [0x00000] in <filename unknown>:0
  at COM3D2.MaidFiddler.Patcher.Patcher.FixFeelingGetter (Mono.Cecil.AssemblyDefinition ass) [0x00000] in <filename unknown>:0
  at COM3D2.MaidFiddler.Patcher.Patcher.Patch (Mono.Cecil.AssemblyDefinition ass) [0x00000] in <filename unknown>:0
  at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (object,object[],System.Exception&)
  at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0
  --- End of inner exception stack trace ---
  at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0
  at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) [0x00000] in <filename unknown>:0
  at BepInEx.SybarisLoader.Patcher.Loader.Patch (Mono.Cecil.AssemblyDefinition assembly) [0x00000] in <filename unknown>:0

To Reproduce
Write a small description of what you tried to do (in GUI or the game).
Starting the game first then maid fiddler
Version information:

Game version [e.g. 1.402] (steam)
Maid Fiddler version [the current one]

Additional context
it got so bad I reset everything as in wipe the game clean then reinstalled everything
and yes it still would not work

Greetings!
From the looks of it, either your game install or your Sybaris install is incorrect. Maid Fiddler <IP_ADDRESS> has been tested on English release of COM (ver. 1.04.2) and confirmed to work fine:

Most likely your Sybaris install is incorrect. For English COM, you can use one of these:

bare, no-plugin BepInEx pack
a basic BepInEx pack
Experimental modular installer for COM

I suggest wiping all of your COM and reinstalling it from scratch to make sure you don't have any leftover.
Finally, make sure you actually use the R18 patch. Maid Fiddler <IP_ADDRESS> does not support It's a Night Magic as it uses an outdated game build.
If you have any further issues, please create a new issue.

Thanks


Template7 It does not work in index.html

Hi!
var myApp = new Framework7({
template7Pages: true,
template7Data: {
'url:index.html' : {
test: '1111',
},
'url:action.html' : {
test: '2222',
}
}
});
When the home page is loaded, the data is output, and line output {{test}} when loaded action.html, then everything is correct.
How to do that on the main page of the data exchange work?
Thank you for the quick support!

Yes, it doesn't work on app init, because F7 doesn't load something. There are a lot of solutions on forums, check them. For example, you may just call .router.refreshPage method to reload home page and preprocess it with T7

Closing as no further response/discussion

This doesn't work. Any chance you can provide a practical example of this? I am using ajax content with preprocess.


dotnet resotore and Create Roll Exception

please check attachment  i get latest branch run yarn install and dotnet restore
after that db created but create role method throws exception.

I had this same problem. I created a SQLExpress db first, migrated to that then restored that to SQL db.
Did you follow the README?

i resolved by deleting migrations folder, bin and run dotnet run dropdb command.


[Feature Request] html link with preview, rich preview

Most modern web apps have the feature that they show a preview of a link pasted into the app, something like this

open graph protocol
 example
 example

A feature like that would greatly improve Draw.io in my opinion and is especially useful for mind maps

This isn't possible since we don't have control over the storage options and we don't store anything ourselves.


InfluxDB2 backup not working

Summary
I can not get an InfluxDB2 backup working.
Maybe I am doing something wrong, then please tell me.
I succesfully use this great tool already for backups of Postgres and MySQL.
Steps to reproduce

use latest InfluxDB2 version
use latest docker-db-backup version
create all access token via Influx UI
use config below

What is the expected correct behavior?
The backup should work.
Relevant logs and/or screenshots
[tiredofit/db-backup:3.7.5 21:19:35 /] $ backup-now
** Performing Manual Backup
2023-03-10.21:19:38 [NOTICE] ** [db-backup] Dumping Influx2 database: 'all'
2023/03/10 21:19:38 INFO: Downloading metadata snapshot
Error: failed to backup metadata: failed to download metadata snapshot: 401 Unauthorized: read:authorizations is unauthorized
2023-03-10.21:19:38 [ERROR] ** [db-backup] DB Backup of 'influx2_all_influxdb_20230310-211938' reported errors
2023-03-10.21:19:38 [ERROR] ** [db-backup] Skipping creating archive file because backup did not complete successfully
2023-03-10.21:19:38 [ERROR] ** [db-backup] Skipping Checksum creation because backup did not complete successfully
2023-03-10.21:19:38 [ERROR] ** [db-backup] Skipping moving DB Backup to final location because backup did not complete successfully
2023-03-10.21:19:38 [NOTICE] ** [db-backup] DB Backup for 'all' time taken: Hours: 0 Minutes: 00 Seconds: 00
2023-03-10.21:19:38 [INFO] ** [db-backup] Backup routines finish time: 2023-03-10 21:19:38 CET with overall exit code 1
2023-03-10.21:19:38 [NOTICE] ** [db-backup] Backup routines time taken: Hours: 0 Minutes: 00 Seconds: 00
2023-03-10.21:19:38 [ERROR] ** [db-backup] Skipping Cleaning up old backups because there were errors in backing up

Environment
DB_TYPE=influx
DB_HOST=http://influxdb
DB_NAME=ALL
DB_USER=my_organization
DB_PASS=my_all_access_pass_from_influx_ui
DB_DUMP_FREQ=1440
DB_DUMP_BEGIN=0300
DB_CLEANUP_TIME=8640
INFLUX_VERSION=2
COMPRESSION=GZ
CONTAINER_ENABLE_MONITORING=FALSE

Image version / tag: latest
Host OS: Ubuntu 22.04

docker-compose.yml
version: '3.7'

services:
  influxdb:
	image: influxdb:2.6-alpine
	container_name: influxdb
	restart: unless-stopped
	environment:
	  - DOCKER_INFLUXDB_INIT_MODE=setup
	  - DOCKER_INFLUXDB_INIT_USERNAME=my_admin_user
	  - DOCKER_INFLUXDB_INIT_PASSWORD=my_admin_password
	  - DOCKER_INFLUXDB_INIT_ORG=my_organization
	  - DOCKER_INFLUXDB_INIT_BUCKET=my_bucket
	  - INFLUXD_LOG_LEVEL=warn
	  - TZ=Europe/Berlin
	volumes:
	  - /etc/localtime:/etc/localtime:ro
	  - ./conf:/etc/influxdb2
	  - ./data:/var/lib/influxdb2
	networks:
	  - default

  influxdb-backup:
	image: tiredofit/db-backup
	container_name: influxdb-backup
	restart: unless-stopped
	environment:
	  - DB_TYPE=influx
	  - DB_HOST=http://influxdb
	  - DB_NAME=ALL
	  - DB_USER=my_organization
	  - DB_PASS=my_all_access_pass_from_influx_ui
	  - DB_DUMP_FREQ=1440
	  - DB_DUMP_BEGIN=0300
	  - DB_CLEANUP_TIME=8640
	  - INFLUX_VERSION=2
	  - COMPRESSION=GZ
	  - CONTAINER_ENABLE_MONITORING=FALSE
	volumes:
	  - /etc/localtime:/etc/localtime:ro
	  - ./backups:/backup
	depends_on:
	  - influxdb
	networks:
	  - default

Thanks for creating this tool and looking into it and helping me! :)

Hey @tiredofit, do I need to add anything so you can look further into it? :)

Yeah, well... turns out, while trying the backup of the InfluxDB in another way, that I have to use the "admin token" - which I deleted after the installation. Damn. So, after recreating the DB from scratch, now everything works as expected. Damn admin or as you called it "root token". Thanks anyway, your tool is so good!

In case anyone else runs into this issue, you can make a clone of the admin token by logging into the InfluxDB web UI and cloning the admin user's token. Then just rename it (optional) and use that token for db-backup.


Icons missing in find widget in IE

Moved from Microsoft/vscode#8384

From @isidorn
Ref: #8229

IE11, open https://microsoft.github.io/monaco-editor/index.html
Open the find widget - notice how match case and match whole world icons are missing

IE11 does not like precisely those two icons, they are svgs with inline styles...


Fixed typo

Change these to bool while you're at it?

No. Not sure about the data type and I may want bits....
uint8 BRICK1_VALID_SHIFTS=0
uint8 BRICK1_VALID_MASK=1
uint8 BRICK2_VALID_SHIFTS=1
uint8 BRICK2_VALID_MASK=2
uint8 BRICK3_VALID_SHIFTS=2
uint8 BRICK3_VALID_MASK=4
uint8 BRICK4_VALID_SHIFTS=3
uint8 BRICK4_VALID_MASK=8

Replaced with https://github.com/PX4/Firmware/pull/8031


Communication between containers

URL to the Wiki page
https://github.com/qdm12/gluetun-wiki/blob/main/setup/connect-a-container-to-gluetun.md
What's missing?
Hello,
when I do this like in the wiki, the container is online via the gluetun vpn container. But i can no longer communicate between containers by using their service name as a hostname. Is it possible to still have this while only the WAN Connection goes thru the gluetun container?

Found it here: https://github.com/qdm12/gluetun-wiki/blob/main/setup/inter-containers-networking.md


Logs conceptual docs: write 'em

It's currently a tiny paragraph today. Lots more to document.
For help: https://docs.google.com/presentation/d/1fGrdeZxNakp0ZIy17ERhu-OLns_Sm9cpFvyyi6doeCw/edit#slide=id.p

https://opentelemetry.io/docs/concepts/signals/logs/ looks much better now, so I close this one, although their is always more we can do:-)


Remove beforeLint

Fixes #35
Also played around with a solution to #36 and it should be pretty easy, and also allow a beforeLint style feature anyway.

@simonsmith looks good. You left a few
expect(stdout).to.not.contain('beforeLint ran');

assertions that we should remove.

Good spot! Removed

Feel free to Squash and merge if tests pass


feat(http): switch from trust-dns to hickory-dns

Essentially still the same crate, but it was rebranded as hickory-dns. Thus, further versions aren't released under trust-dns anymore.

I think we decided to that it was a breaking change so we did not do it on main, but next have been updated to it in https://github.com/twilight-rs/twilight/pull/2287

Ohh, I see. Didn't realize that there is a next branch for the next breaking change version. Sorry for the trouble, already changed then :sweat:


TensorRT conversion fails with 5D input data

Hi, I'm having an issue with tensorrt conversion of model which uses 3D convolutions and processes 5D input.
Code to reproduce error (I cut the model to minimal example):
import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Input,Conv3D,BatchNormalization,Activation

print("TensorFlow version: ",tf.version.VERSION)

src_dir='model_bn'
dst_dir='model_bn_conv'
batch=4
input_shape=(24,160,160,3)

def input_fn():
    yield [np.zeros(((batch,)+input_shape),dtype='float32')]

model=Sequential()
model.add(Input(shape=input_shape))
model.add(Conv3D(24,3))
model.add(BatchNormalization())
model.add(Activation('relu'))

model.save(src_dir)

params=tf.experimental.tensorrt.ConversionParams(precision_mode='FP32')
converter=tf.experimental.tensorrt.Converter(input_saved_model_dir=src_dir,conversion_params=params)

converter.convert()
print('Converted successfully!')

print('Running build()...')
converter.build(input_fn=input_fn)
print('Build completed')
converter.save(dst_dir)

The output:
TensorFlow version:  2.8.0
2022-03-31 11:44:04.711186: I tensorflow/core/platform/cpu_feature_guard.cc:152] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-03-31 11:44:05.311164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13608 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:3b:00.0, compute capability: 7.5
WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.
2022-03-31 11:44:05.990618: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2022-03-31 11:44:06.422604: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-03-31 11:44:06.422784: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-03-31 11:44:06.458670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13608 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:3b:00.0, compute capability: 7.5
2022-03-31 11:44:06.485282: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1191] Optimization results for grappler item: graph_to_optimize
  function_optimizer: Graph size after: 30 nodes (20), 41 edges (31), time = 1.05ms.
  function_optimizer: function_optimizer did nothing. time = 0.01ms.

2022-03-31 11:44:06.529996: I tensorflow/core/grappler/devices.cc:66] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-03-31 11:44:06.530104: I tensorflow/core/grappler/clusters/single_machine.cc:358] Starting new session
2022-03-31 11:44:06.539721: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13608 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:3b:00.0, compute capability: 7.5
2022-03-31 11:44:06.551640: W tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:192] Calibration with FP32 or FP16 is not implemented. Falling back to use_calibration = False.Note that the default value of use_calibration is True.
2022-03-31 11:44:06.551665: I tensorflow/compiler/tf2tensorrt/convert/trt_optimization_pass.cc:211] [TF-TRT] not using explicit QDQ mode
2022-03-31 11:44:06.552184: W tensorflow/compiler/tf2tensorrt/segment/segment.cc:954]

################################################################################
TensorRT unsupported/non-converted OP Report:
        - NoOp -> 2x
        - Identity -> 1x
        - Placeholder -> 1x
--------------------------------------------------------------------------------
        - Total nonconverted OPs: 4
        - Total nonconverted OP Types: 3
For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops.
################################################################################

2022-03-31 11:44:06.552276: W tensorflow/compiler/tf2tensorrt/segment/segment.cc:1281] The environment variable TF_TRT_MAX_ALLOWED_ENGINES=20 has no effect since there are only 1 TRT Engines with  at least minimum_segment_size=3 nodes.
2022-03-31 11:44:06.552292: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:795] Number of TensorRT candidate segments: 1
2022-03-31 11:44:06.552678: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:909] Replaced segment 0 consisting of 11 nodes by TRTEngineOp_0_0.
2022-03-31 11:44:06.556775: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1191] Optimization results for grappler item: tf_graph
  model_pruner: Graph size after: 22 nodes (-8), 33 edges (-8), time = 0.273ms.
  debug_stripper: debug_stripper did nothing. time = 0.008ms.
  layout: Graph size after: 26 nodes (4), 37 edges (4), time = 1.157ms.
  dependency_optimizer: Graph size after: 16 nodes (-10), 15 edges (-22), time = 0.228ms.
  constant_folding: Graph size after: 16 nodes (0), 15 edges (0), time = 0.564ms.
  common_subgraph_elimination: Graph size after: 13 nodes (-3), 15 edges (0), time = 0.124ms.
  TensorRTOptimizer: Graph size after: 3 nodes (-10), 2 edges (-13), time = 1.148ms.
  constant_folding: Graph size after: 3 nodes (0), 2 edges (0), time = 0.294ms.
Optimization results for grappler item: TRTEngineOp_0_0_native_segment
  model_pruner: Graph size after: 13 nodes (0), 15 edges (0), time = 0.083ms.
  debug_stripper: debug_stripper did nothing. time = 0.006ms.
  layout: Graph size after: 13 nodes (0), 15 edges (0), time = 0.4ms.
  dependency_optimizer: Graph size after: 12 nodes (-1), 13 edges (-2), time = 0.116ms.
  constant_folding: Graph size after: 12 nodes (0), 13 edges (0), time = 0.366ms.
  common_subgraph_elimination: Graph size after: 12 nodes (0), 13 edges (0), time = 0.093ms.
  TensorRTOptimizer: Graph size after: 12 nodes (0), 13 edges (0), time = 0.013ms.
  constant_folding: Graph size after: 12 nodes (0), 13 edges (0), time = 0.346ms.

Converted successfully!
Running build()...
2022-03-31 11:44:06.640977: I tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:446] TRTEngineOp not using explicit QDQ
2022-03-31 11:44:06.643431: I tensorflow/compiler/tf2tensorrt/common/utils.cc:100] Linked TensorRT version: 8.2.3
2022-03-31 11:44:06.643602: I tensorflow/compiler/tf2tensorrt/common/utils.cc:102] Loaded TensorRT version: 8.2.3
2022-03-31 11:44:07.618558: F ./tensorflow/compiler/tf2tensorrt/utils/trt_tensor_proxy.h:158] 'trt_tensor_' Must be non NULL
Aborted

Other observations:

Removing activation layer or swapping it with BatchNorm makes script to execute without errors, but doesn't produce valid trt engine. It writes the following warning: "TF-TRT Warning: Engine creation for TRTEngineOp_0_0 failed. The native segment will be used instead. Reason: INVALID_ARGUMENT: Rank of perm for transpose does not match with that of the input."
Removing Conv3D layer does not change the result.
Trying to tweak conversion parameters did not help.
Using 4D input data and 2D convolutions seem to work fine.

System information
(Using NVidia NGC container for Tensorflow, version 22.03)

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.8.0
Python version: 3.8.10
CUDA/cuDNN version: 11.6/8.3.3
GPU model and memory: NVidia Tesla T4 16Gb

@sachinprasadhs I was able to reproduce this issue on colab using TF v2.8.0 ,tf-nightly ,please find the gist here.Thanks!

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


Unbearably slow Python autocomplete (sometimes)

Lapce Version
nightly-acf1de5
System information
Debian 11
python3.9.2
lapce-python plugin
python-lsp-server 1.7.0
pyflakes 2.2.0
pycodestyle 2.6.0
mccabe 0.6.1
Describe the bug
Autocompletion sometimes takes many seconds to complete.
Additional information
I don't know (yet at least) how to force this behavior.
The file is ~1000 lines of pytest tests.
The variables I write are declared globally at the top of the file.
The same thing happens for class names imported from another module (from module import ClassName), and for some variables declared in the signature of a function (but not all of them).
However, the weird part is that pytest, imported at the top, autocompletes instantly.
I can write pytest.mark.parametrize using three completions without any problem...
https://user-images.githubusercontent.com/66979306/212218846-49be2394-5477-4b78-a2aa-5cdbfe181927.mp4
The video shows the delay.
The first time, I write press  and wait for the completion.
The second time, I wait until the messages in the bottom-left corner stop and then hit  to complete instantly.
Also the second time, you can see the red squiggle progress from character to character between each cycle of messages.
From my tests, it seems that the more I manually complete the variable name, the longer it takes for autocomplete to work.
The messages cycle:
lint: pyflakes -> lint: pycodestyle -> lint: mccabe
Both times, my CPU jumps to ~40%.
That's all I've got

After some more usage and refactor, it seems the slow down was due to a deep json-like structure of nested dicts.

This should be opened against Python plugin issue tracker


Define the scope of updates in the DCP2 MVP

Goal
User stories

As a data contributor, I want to add additional information about my samples and experiments to my contribution
As a data contributor, I want to fix errors in the information I provided about my samples and experiments
As a data consumer, I want to be confident in the accuracy of the metadata associated with HCA data in the DCP

Questions
Existing functionality

What classes of updates can the ingest importer and core infrastructure already support
What work is involved in ensuring the exporter what are the requirements for downstream DCP services to support these updates
Do any of the updates impact how analysis pipelines run?
What is the cost of that work and how does it impact the DCP delivery timelines?
How many of the DCP1 dataset updates can be supported by existing functionality https://docs.google.com/spreadsheets/d/1B52QjioJ7CzOLXFIcB-Q8mfjiFSH0jzbfdNs_nce3P4/edit#gid=1663653290
Have all the errors identified in the experimental design validation been added to the above list?

New update features

What classes of updates do we not support currently?
What value to data consumers does delivering these updates provide?
What is the cost of implementing the functionality to ingest
What requirements would supporting these types of updates have for the TDR teams or the Data Browser team?
How does that implementation impact MVP delivery timelines?

New data not in DCP1

Have we identified any updates for new datasets which aren't in DCP1?
Where are those updates recorded?

Other questions

Can we/should we include the INSDC study/project id switch as part of this effort?
Can we/should we include review/update of schema regexes as part of this effort?
Does the scope of this effort suggest we need to relax the view that no scientific/user supplied schema changes should be allowed for MVP
Do any in-scope updates require us to develop a schema migration solution for ingest during MVP phase
What process is envisioned to propagate these updates through various pipes for DCP2 MVP release?

An outcome that updates are not possible for MVP is acceptable but we would need to understand why and what the consequences for consumers for not making these updates
Acceptance criteria

[ ] Scope definition for (meta)data updates in DCP2 MVP
[ ] Definition of new features that need to be implemented to meet the defined scope
[ ] List of updates which can be propagated and a timeline for those updates
[ ] Confirmation from TDR import team they are happy with a defined scope and understand the consequences for their own development plans
[ ] Confirmation from Data browser team they are happy with the defined scope and understand the consequences for their own development plans
[ ] Confirmation from pipelines team they are happy with the defined scope and understand the consequences for their own development plans

So I think when in my original statement I said we would like to be able to add new fields to the schema no one had pointed out that our no additional attributes design choice (which I support) would mean even new optional fields still require migration of old data to the new schema.
@rolando-ebi @mshadbolt @ESapenaVentura can you comment here? Is there any way to legitimately add new optional fields to our existing schema without having a migration strategy?

Tagging myself on this ticket to make some task tickets to put in the sprint backlog for myself

Not sure who wrote them but can either @clairerye or @lauraclarke clarify some of the questions above
Existing functionality
Question 1 - do you mean whether we can do the update within ingest rather than whether we can propagate that update downstream to other components?
Question 2 doesn't make sense, not sure if it needs to be split into two questions or what it is actually getting at
Question 3 - still a bit confused on this as we have been told that analysis will be doing a manual check to see that pipelines run correctly?
Question 4 - The cost of what work?
New update features
Question 3 - do you mean the cost of implementing all possible classes of updates into ingest?

@ESapenaVentura @rays22 I believe you both did work on running datasets through the experimental graph validator (#14 ), were you recording any errors in the spreadsheet linked above  in question 6 of 'Existing functionality'

@ESapenaVentura @rays22 I believe you both did work on running datasets through the experimental graph validator (#14 ), were you recording any errors in the spreadsheet linked above in question 6 of 'Existing functionality'

I have not recorded any errors in the spreadsheet https://docs.google.com/spreadsheets/d/1B52QjioJ7CzOLXFIcB-Q8mfjiFSH0jzbfdNs_nce3P4/edit#gid=1663653290 .
I have recorded the cross-entity validation errors in the relevant AUDR tickets and
https://github.com/ebi-ait/hca-ebi-wrangler-central/issues/70 .

good questions @mshadbolt I will try to answer, but maybe best if @lauraclarke also comments.

Question 1 - do you mean whether we can do the update within ingest rather than whether we can propagate that update downstream to other components?
This Possibly needs to be 2 or 3 questions

Can we do the updates in ingest?
Can we get the updated submission to the GCP? - Not looking at archives yet.
Can anyone downstream do anything with the updated submission after its in the GCP?

Question 2 doesn't make sense, not sure if it needs to be split into two questions or what it is actually getting at
I think this is actually asking point 2 and 3 above.

Question 3 - still a bit confused on this as we have been told that analysis will be doing a manual check to see that pipelines run correctly?
In that case, perhaps its more of letting them know the decision we make rather than trying to include this initially.

Question 4 - The cost of what work?
How much effort will it take to implement updates in ingest (if not possible) and get the updated submissions to GCP (if not possible).
I think downstream costs should also be considered, but separately to our own.

Okay to add some clarification given I wrote the questions

What classes of updates can the ingest importer and core infrastructure already support

Ie, if a wrangler was to try and resubmit an already ingested spreadsheet with some changes to the metadata, what type of updates would be applied to ingests existing records about a dataset and what type of updates would cause ingest to either barf or make a mess of our internal records?

What work is involved in ensuring the exporter what are the requirements for downstream DCP services to support these updates

This is about once the ingest database contains the updated metadata (or data), what changes are needed in the exporter and what if any changes are needed in the Terra importer or other downstream services so those updates are also propagated to these other services

Do any of the updates impact how analysis pipelines run?

This is actually an independent question from the other two and I possibly shouldn't have put it in the same list. This question is getting at if updates we can do would need pipelines to be retriggered because then we have to figure out the right way to communicate those changes and inform the pipelines time of when they should or shouldn't run those pipelines (they might want to avoid running at all in the first instance given it is all manual if an update is coming they would require them to rerun)

What is the cost of that work and how does it impact the DCP delivery timelines?

I should have been specific to that, I meant any changes we need to make to the exporter and any changes anyone else needs to make to downstream services

And for
New update features

What classes of updates do we not support currently?
What value to data consumers does delivering these updates provide?
What is the cost of implementing the functionality to ingest

question 3 refers to any new class of update that we want to implement so we can decide if the value scientists will get during the MVP for supporting this type of update is worth the effort involves so perhaps it should be updated with "For each class of update"
This does need to be considered by both us and downstream components as updates can't just stop at the ingest importer

I have started on a document here: https://docs.google.com/document/d/11x8RDBmB2un1RTAXF3HKgcx-uUBD-Bkrl3QycrcnHc8/edit?usp=sharing


deadbeef 1.8.4 -> 1.9.1

Description of changes
This updates Deadbeef to 1.9.1 (in the hope of solving the problem of playback/shuffle mode sporadically resetting itself to random values, for one, and also 1.8.4 is old).
Had to package libdispatch in order to enable that (it's sort of present in Nixkpgs already as part of (at least) Swift and Zig, but not usable as a build input).
Also upgraded deabeefPlugins.mpris2 while at it.
Things done

Built on platform(s)

[v ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[v ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[ ] Fits CONTRIBUTING.md.

Thanks.


slight sdk schema change

@danmarshall,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Fixed docker target branch and binary name

Now it simply uses the default branch (in our case rust).

@rnestler review plz :)

I can't get it to run:
% docker build -t  coredump/spaceapi:latest .
Sending build context to Docker daemon
Post http:///var/run/docker.sock/v1.19/build?cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=coredump%2Fspaceapi%3Alatest: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

Do you have any suggestions?

Never mind I figured it out, was rather easy: https://wiki.archlinux.org/index.php/Docker#Installation


Added decorator exclude_xframe_options_header

Done

Added decorator exclude_xframe_options_header to avoid the security header X-Frame-Options with the value SAMEORIGIN

QA
QA this change with https://github.com/canonical-web-and-design/snapcraft.io/pull/3337

So the interface for using this feature is:
@exclude_xframe_options_header
def my_view():
    # ...

Another option would be something like:
def my_view():
    # ...
    response = flask.render_template("template")
    response.headers["x-frame-options"] = "allowall"
    return response

And then we would write the after_request function to only define x-frame-options if it didn't already exist. This feels a little more intuitive to me, but it has the downside that allowall is in fact not a valid option for x-frame-options - it only works because setting x-frame-options to an invalid option will cause browsers to simply ignore the header, which has the same effect.
So I can't quite decide what is best here.

So the interface for using this feature is:
@exclude_xframe_options_header
def my_view():
    # ...

Another option would be something like:
def my_view():
    # ...
    response = flask.render_template("template")
    response.headers["x-frame-options"] = "allowall"
    return response

And then we would write the after_request function to only define x-frame-options if it didn't already exist. This feels a little more intuitive to me, but it has the downside that allowall is in fact not a valid option for x-frame-options - it only works because setting x-frame-options to an invalid option will cause browsers to simply ignore the header, which has the same effect.
So I can't quite decide what is best here.

Another option would be something like:
def my_view():
    # ...
    response = flask.render_template("template")
    response.headers["x-frame-options"] = "allowall"
    return response

This was my first choice until I searched what is the default value for x-frame-options and discovering that there isn't and the header shouldn't be set then. I don't like setting an invalid value just to avoid the issue, I like the decorator, I'm also open to new ideas.
As an improvement of the current change, we can mix these two options so if we need to set the values DENY or ALLOW-FROM ... this decorator is not needed because it's already defined.

Another option would be something like:
def my_view():
    # ...
    response = flask.render_template("template")
    response.headers["x-frame-options"] = "allowall"
    return response

This was my first choice until I searched what is the default value for x-frame-options and discovering that there isn't and the header shouldn't be set then. I don't like setting an invalid value just to avoid the issue, I like the decorator, I'm also open to new ideas.
As an improvement of the current change, we can mix these two options so if we need to set the values DENY or ALLOW-FROM ... this decorator is not needed because it's already defined.

Yes I agree with all that. Let's do both =)

Yes I agree with all that. Let's do both =)

Approved, assuming you put in the conditional =)

I did! 😃

Approved, assuming you put in the conditional =)

I did! 😃


Refactored BSP module and use isolated classloading for implementations

I also excluded bsp4j from transitive Ammonite dependencies, as we don't use Ammonites BSP support and it brought older versions into the tree.
I removed any use of bsp4j API in scalalib, which was only there previously by accident, as I overlooked it's presence through Ammonites transitive dependencies. scalalib now does not depend on bsp4j. I added more case classes (equivalents to the bsp4j classes) if the BSP support required it.
The bsp module is now only a thin module, which provided the entry point to BSP and also manages the classloading of the implementation, which is now contained in bsp.worker module.
I also minimized the use of reflection in MillMain to start the BSP server, which should reduce the potential for errors which can only be catched in (manual) tests.

Nice, from a pure code point of view this looks good! I like the separation it introduces although I'm not super familiar with workers yet.

"Worker" is a somewhat overloaded term. I don't use a T.worker here. I'd like to, but as we start the BSP as early as possible (and therefore can't wait for Mill to resolve and initialize), we need to do it without Mill's task support.

I'll try to use this locally a bit today to see if I hit on anything from a usability standpoint.

That would be nice. We still have no proper automated test setup that can guarantee a workable server. I typically just delete all .bsp and .metals dirs and fire up a vscode to check whether Mill BSP somehow ends up with a reasonable modules list in vscode. As I don't use it daily, if find it hard to idicators for a working/non-working integration. It would be nice, if we had some server-independent test suite, which only requires a BSP connection file and allows to assert on the outcome.

if we had some server-independent test suite, which only requires a BSP connection file and allows to assert on the outcome.

Sort of an aside, but we talked about this a bit at the Center. It's on our radar and something we'd like to explore and potentially work on to better ensure the BSP ecosystem as a whole is more tested and less error-prone. I'll loop you in when I have more information there.

Just a head up, using this locally for a bit, everything seems to work expected when using Metals.


bench exits with error status 1 (libjudy related?)

Trying to compile simhash-cpp, I am running into a couple of issues. The first one is that I am unable to compile test.cpp. The issue is described already in #2.
The other issue is related to bench.cpp. It compiles, but when I am trying to run the executable, I get the following error message:
Inserting 10000000 hashes...
Running 40000000 queries...
Queries complete with 0 errors
There are 9997291 items in the table
File 'simhash.h', line 137: Judy1FreeArray(), JU_ERRNO_* == 9, ID == 113
zsh: exit 1     ./bench 10000000

I have used the following packages for compilation:
judy 1.0.5
g++ 4.9.2
Catch build 53

Any idea what is happening?
EDIT: The JU_ERRNO above corresponds to (from July.h):
// JU_ERRNO_CORRUPT occurs when Judy detects an impossible value in a Judy data
// structure:
//
// Note:  The Judy data structure contains some redundant elements that support
// this type of checking.

        JU_ERRNO_CORRUPT        = 9

We've not encountered that, but we wouldn't be heartbroken to ditch libJudy. What OS are you running

Er. Nevermind. We've been able to repro that locally.

Any news on this one?

Hi, I encountered the same problem in my project and found out it happens when freeing the Judy array after storing too many elements in it. Here's a simple test: http://dpaste.com/34QBQ41
It fails on 10000 elements in my environment -- judy-1.0.5 and
$ uname -a
Linux localhost 3.13.0-46-generic #75-Ubuntu SMP Tue Feb 10 15:24:04 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Have you guys made any progress at your end? Thanks.

On my Macbook the ./test reports failures as well.  I am using all the latest things.
Catch 1.1, Build 1.
Judy-1.0.5
G++ for Mac, which is
Target: x86_64-apple-darwin14.1.0
Thread model: posix
Here are the outputs from terminal.
PVGM50844878A:src i064503$ make clean
rm -rf driver test *.o
PVGM50844878A:src i064503$ make
g++ -g -O3 -Wall -o simhash.o -c simhash.cpp
g++ -g -O3 -Wall -o bench bench.cpp simhash.o -lJudy
g++ -g -O3 -Wall -o test test.cpp -Ideps/Catch/single_include
PVGM50844878A:src i064503$ ./test
test is a Catch v1.1 b1 host application.
Run with -? for options

-------------------------------------------------------------------------------
cyclic
  multiple rotation
-------------------------------------------------------------------------------
test.cpp:14
...............................................................................

test.cpp:39: FAILED:
  REQUIRE( Simhash::Cyclic<uint64_t>::rotate(0xABCDEFFFABCDEFFF, 64) == 0xABCDEFFFABCDEFFF )
with expansion:
<PHONE_NUMBER>3709551615 == 0xabcdefffabcdefff

-------------------------------------------------------------------------------
tokenizers
  strspn
-------------------------------------------------------------------------------
test.cpp:85
...............................................................................

test.cpp:120: FAILED:
  REQUIRE( (next == __null) )
with expansion:
  false

===============================================================================
test cases:  3 |  1 passed | 2 failed
assertions: 30 | 28 passed | 2 failed

Due to @janpom's mention, I checked bench.cpp with different values of uint64_t limit. It turns out, that with a value of 4148 or higher, I get the libjudy error:
% ./bench
Inserting 4148 hashes...
Running 16592 queries...
Queries complete with 0 errors
There are 4123 items in the table
File 'simhash.h', line 137: Judy1FreeArray(), JU_ERRNO_* == 9, ID == 113
zsh: exit 1     ./bench

At 4147, the queries complete with errors, but the program hangs without completing (needed to sigterm it):
% ./bench
Inserting 4147 hashes...
Running 16588 queries...
Queries complete with 3 errors
^C
zsh: interrupt  ./bench

Below that (up until 4100, it sometimes passes with 0 errors, sometimes doesn't with 3 errors. But anytime it just hangs.
At 4099 and below, everything seems to work nicely:

% ./bench
Inserting 4099 hashes...
Running 16396 queries...
Queries complete with 0 errors
There are 4098 items in the table

Odd stuff. :)

This appears to be fallout from libjudy, and its brokenness on newer versions of GCC. seomoz/simhash-py#8 discusses a workaround.

Since with https://github.com/seomoz/simhash-cpp/commit/922891f05c3c43cad23ea7cb374ecb23f3f7e281 judy arrays are not used anymore, this should be closed IMHO

Agreed. I lack permissions on this repo to do that anymore. @lindseyreno - do you know who (if anyone) at Moz is looking after this one these days?


The GitHub workflow for publishing a release does not work

The Github workflow for publishing a release (.github/workflows/publish-a-release.yml ) does not work.

It triggers when PR is merged to main, but does not run the release step as those start like this:

  publish-to-github-releases:
    name: Publish wakepy to GitHub
    # only publish to GitHub Releases on tag pushes
    if: startsWith(github.ref, 'refs/tags/')

It can be triggered manually, but does not run the release step for the same above mentioned reason.
It will not trigger on tag push. The yaml starts with:

'on':
  workflow_dispatch:
  push:
    branches:
      - main

Going to change this so that it's only triggered with workflow_dispatch. The workflow_dispatch action is allowed by only the persons with write access to the repo (ref: https://github.com/orgs/community/discussions/26622).


Cw 801

My cw801 can be track in my traccar with port 5040 but online device is little second and offline longtime and online little second again...
Like that countinue...
Can anyone help me...?
https://www.traccar.org/forums/
-->

Use forum for questions.


how to use dark mode support

Hello,
I notice that one of the Mac build settings is "darkModeSupport" (https://www.electron.build/configuration/mac)
How can the Meteor desktop app tell if the device is in dark mode so that it can set its styles to follow the system-wide dark mode setting?

It seems as though we'd need to do something like this:
const { nativeTheme } = require('electron')

systemPreferences.subscribeNotification(
  'AppleInterfaceThemeChangedNotification',
  function theThemeHasChanged () {
    updateMyAppTheme(nativeTheme.shouldUseDarkColors)
  }
)

How can we access the systemPreferences object?

@LDubya
in settings.json:
      "mac": {
          "darkModeSupport": true,
          "extendInfo": {
            "NSRequiresAquaSystemAppearance": false
          }
      }

in desktop.js
eventsBus.on('windowCreated', (window) => {
  const setTheme = function () {
    const darkMode = systemPreferences.isDarkMode();
    desktop.send("darkMode",  darkMode)
  }

  systemPreferences.subscribeNotification('AppleInterfaceThemeChangedNotification', function () {
    setTheme()
  })

  setTheme()
});

This will only work on the build Electron application, not the one in development. It looks like meteor-desktop needs support for enableDarkMode and/or 'extendInfo'. cc @wojtkowiak @darqs


GridLayoutManager with Full-Span Group View, AbstractExpandableItemAdapter

Using AbstractExpandableItemAdapter with GridLayoutManger, groups are next-to-next child, like normal recycler view.
How can I make groups extend to full-span and items normally next-to-next?
Or how do i manually set span for groups Childs? ,
                                GridLayoutManager layoutManager = new GridLayoutManager(this, 2);
                                layoutManager.setSpanSizeLookup(new GridLayoutManager.SpanSizeLookup() {
                                    @Override
                                    public int getSpanSize(int position) {
                                        return 2;
                                    }
                                });

https://github.com/h6ah4i/android-advancedrecyclerview/issues/294 solves this!


Remove pyinotify dependency

Resolves #3934

@enykeev @armab @Kami Can you guys resolve your differences here?

Superceded by #4145.

:+1:

And #4145 was superseded by #4191.


CI Failures Associated with Pytest v8.0.0 Release

What is your issue?
A recent release of pytest (v8.0.0) appears to have broken our CI.
pytest                    8.0.0              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-env                1.1.3              pyhd8ed1ab_0    conda-forge
pytest-github-actions-annotate-failures 0.2.0                    pypi_0    pypi
pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge

Strangely, the issue doesn't seem to occur when using previous versions (e.g. v7.4.4). our last successful CI run used pytest v7.4.4
pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
pytest-env                1.1.3              pyhd8ed1ab_0    conda-forge
pytest-github-actions-annotate-failures 0.2.0                    pypi_0    pypi
pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge
pytest-xdist              3.5.0              pyhd8ed1ab_0    conda-forge

i recreated the environment and successfully ran tests locally. the CI failures appear to be connected to the latest release of pytest. i haven't had a chance to do an in-depth exploration of the changes from pytest which could be influencing this disruption. so, i wanted to open an issue to track what is going on. in the meantime, i'm going to pin pytest to an earlier version.
any insights, especially from those familiar with changes in the pytest v8.0.0 update, are warmly welcomed.

The biggest change which seems to be affecting the tests here is that now pytest.warns does not capture all warnings, where previously it did.
We had similar changes needed in mpl: https://github.com/matplotlib/matplotlib/pull/27624
In particular, nesting pytest.raises and pytest.warns in pytest 7 would cause things to not fail even if one of those conditions were not met (e.g. the exception is raised before the warning would have been issued) so in pytest 8 this now (correctly I think) tells you when you thought you were testing that a warning was issued that is not, in fact, actually issued.
A quick scan of the failing tests looked like all of them had that pattern of nested raises and warns. The fix is either to ensure that expected warnings are issued before exceptions interrupt control flow or to not test for warnings that are not issued.

I'm going to try making a pull request for this.


Add Bicycle and Parking Area Support

Wish flow enable bike sharing simulation.

http://sumo.dlr.de/wiki/Simulation/Bicycles
https://github.com/lcodeca/PyPML

hi @harryprince sorry I missed this! I've been looking through the github repository you shared, and it seems that this could probably be coupled with the kernel abstractions in Flow to support the relevant features, although it would not be trivial as there is some overlap between our API and the one presented here. If you are interested in discussing how this could be done, I can share my thoughts with you. Let me know!

Thanks, I found Lara's job, which builds some citywide vehicle parking scenarios.

https://github.com/lcodeca/MoSTScenario
https://github.com/lcodeca/LuSTScenario

hi @harryprince, interestingly enough we had been working on these (at least LuST) in the past. To my recollection, the integration was very close to complete. I will talk to some people and see what the progress was on this. More very soon!


Deprecate old versions' docs

Partial fix for both https://github.com/Leaflet/Leaflet/issues/7712, and https://github.com/Leaflet/Leaflet/issues/8228:

All reference versions use the same title

[ ] Add unique titles for the old versions (e.g. "Documentation - v0.7.7" etc.) or remove old docs per https://github.com/Leaflet/Leaflet/issues/7712?

Obviously this change will make it a bit harder for people to find the docs for older Leaflet versions.

@Malvoz let's rebase and get this merged!


Added Byteman to thirdparty libraries.

This adds Byteman. Necessary for the runtime to load the javaagent with Byteman.
This is necessary for setting j.s.Statement.setQueryTimeout method as a workround.

CC @ruhan1  Looks like an error resolving a SNAPSHOT version of galley-transport-httpclient using the Gateway. Looks like the host cannot be resolved. Can this be looked at ?

@whitingjr This is caused by a missing snapshots of galley. I've just rebuilt that snapshots and restart the test. Let's wait for the result.

@whitingjr can this be re-started again. looks like the job was cancelled


Added OG-USA_logo.png and OG-USA_logo_long.png to the docs folder

This PR adds the OG-USA_logo.png and OG-USA_logo_long.png logo files to the docs/ folder. This will allow the logo to have a permanent URL that does not change when the logo is updated.

Codecov Report

Merging #523 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #523   +/-   ##
=======================================
  Coverage   67.84%   67.84%
=======================================
  Files          45       45
  Lines        5828     5828
=======================================
  Hits         3954     3954
  Misses       1874     1874

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4d6ea72...49d16a4. Read the comment docs.

@jdebacker I am waiting to merge this until we make a decision on the newest suggestion for the logo. I reopened Issue #520 with a mock-up of @kerkphil 's suggestion of putting the flag in one of the middle-bar circles.


Remove Duplicate Definitions

Removes duplicate definitions from the completer and applies Clippy fixes to the same file where the changes were made. Needed by the Ion shell so that builtin commands like true aren't duplicated as they are also picked up by the filename completers.

Changes will be merged in #36


Installer et customiser Material-UI

linked to #142

I remove the RFC label, it's been added as part of the Theodo stack.
https://m33.gitbook.io/standards/technical-gesture/theodo-stack
cc @ThierryCols @jbeurel @benja-M-1 @tcheymol


Update npm version

Please, update npm version.
When I'm installing react-components through npm i've got this code
var $ = require('jquery'); in timeout-transition-group

@jeresig please, look on this

+1; pointing to github #master for now

I just pushed a new version 0.3.0 to npm, hopefully that works for you! I'm also going to update react-components to use react 0.14.0 in a bit, and release that as 0.4.0.


Can you help fix my corrupt exe.xml file?

Software version
0.6.0
Problem
BetterBravoLights refused to install because my exe.xml file is corrupt.
My exe.xml file is located at
C:\Users\aless\AppData\Roaming\Microsoft Flight Simulator\exe.xml
The error reported was
Unexpected end tag. Line 7, position 3.

The contents of the exe.xml file are as follows:
<?xml version="1.0" encoding="Windows-1252"?>
  <Launch.Addon>
    <Name>AFCBridge</Name>
    <Disabled>False</Disabled>
    <Path>E:\Steam\steamapps\common\MicrosoftFlightSimulator\community\AFC_Bridge\bin\AFC_Bridge.exe</Path>
   </Launch.Addon>
</SimBase.Document>

I understand that BetterBravoLights didn't corrupt the file and that it's simply warning
about the corrupted file. But could you help me fix it?

Hiya @macinati,
This seems to be a common problem - this is the second of these just today in fact!
Your exe.xml file is missing the opening <SimBase.Document line near the top.
Edit your exe.xml file and add in the missing second line, making your file:
<?xml version="1.0" encoding="Windows-1252"?>
<SimBase.Document Type="SimConnect" version="1,0">
  <Launch.Addon>
    <Name>AFCBridge</Name>
    <Disabled>False</Disabled>
    <Path>E:\Steam\steamapps\common\MicrosoftFlightSimulator\community\AFC_Bridge\bin\AFC_Bridge.exe</Path>
   </Launch.Addon>
</SimBase.Document>

Hi @macinati,
I'm going to assume that's fixed it and close this. I'm about to release 0.7.0 of BetterBravoLights which itself is able to auto-fix this exact common problem.


⚠ ClamAV has degraded performance

In 20e09ee, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 68 ms

Resolved: ClamAV performance has improved in af45977.


embark run issue

when i exe both "embark run" and "embark blockchain" i will got this msg:
Local Npm module "grunt-cli" not found. Is it installed?
actually i do install this module . no matter -g or not.i will get previous msg.
and then i found embark run can't step into a  infinite loop ,but got a msg: Done, without errors.
so how can i visit http://localhost:8000?

i found if i set console_toggle to true geth will enter console ,then it won't return immediately. so this issue don't exist.

another issue.
if (config.console_toggle) {
cmd += "console";     //there shouble a space   cmd += "console        ";
}
if not the cmdline could be "   .......consolejs ..../.../mine.js.

@leochan007 do you still need help with this?


SPIKE - Log retention (incident investigations)

As an engineer I maintain a log of an incident to use in understanding how the incident came about and any steps that need to be taken to avoid it happening again.
As as engineer I need the logs to be kept long enough to allow me to carry out incident investigations further and to a level that can really understand what happened.
I need to review the length of time our logs are kept to see if they should be kept longer to allow the above to work better.
Possibility of extending Cloud Watch logs to a month (at least for core services.)
Small cost implication of carrying this out (probably no more than a few hundred a month)
Definitions of Done:

Cost of carrying this out
Work for each repo - can we use the same approach for each?

Repos using the cloudwatch logs
network-access-control-infrastructure
nvvs-devops-monitor
staff-device-dns-dhcp-infrastructure
staff-device-logging-infrastructure
staff-device-shared-services-infrastructure
staff-infrastructure-network-services

A new ticket raised to update the code after some times .. https://app.zenhub.com/workspaces/nvvs-devops-622a0b371800e400133bb924/issues/gh/ministryofjustice/nvvs-devops/465

https://dsdmoj.atlassian.net/wiki/spaces/MOJO/pages/4541448241/Update+Log+retention+for+Core+services


Make another client get token method that returns the full oauth2 tok…

…en struct
Also made the non-cached version of get oauth2 token method public.

[ ] @johnny-lai

lgtm

[lockit v4.9.1] merged via Lockit CLI on 2019-08-28 14:56:23.750427 -0700 PDT m=+3.721475803


kernel/work: Fix race under with delayed work item cancellation

The call to unschedule_locked() would return true ("successfully unscheduled") even in the case where the underlying z_abort_timeout() failed (because the callback was already unpended and in-progress/complete/about-to-be-run, remember that timeout callbacks are unsynchronized), leading to state bugs and races against the callback behavior.
Correctly detect that case and propagate the error to the caller.
Fixes #51872
Signed-off-by: Andy Ross<EMAIL_ADDRESS>

Twister failure (qemu_arc_em and tests/kernel/mem_protect/syscalls) is reproducible, but seems to be present upstream too.  Is there a bug tracking that?

Makes sense.  There are additional instances of unchecked z_abort_timeout in kernel/poll.c and subsys/canbus/isotp, might be worth reviewing them.

Any plans for this PR?
#51872 is already closed but still no real fix in main is present.


Update complexMultilineImports.sql

Now it works for all the databases

Heh, was just about to mention this.

Yes, sorry, there were a couple of errors that I missed because they were hidden by others. I'm working on making the build stable again

This should fix the main build. I've disabled the test for Db2 because I'm unable to start it and I don't know how to update the query. I will get back to it in the future


batch allocate empty items

Batch allocates empty items ahead of time; and batch grows them. Introduces a new argument "capacity" which sets the initial capacity. Unsure about the ergonomics of this, but think generally the right idea. Benches now seem to pull ahead consistently.
The only thing I'm confused about is that the regular vec test slows down. Unsure why tbf.
Future directions
We should have a way to shrink the pool too if we consistently dip under 25% utilization we should halve the pool size.
Results
new
test tests::bench_remem ... bench:   1,323,002 ns/iter (+/- 249,490)
test tests::bench_vec   ... bench:   1,419,678 ns/iter (+/- 10,898)

before
test tests::bench_remem ... bench:   1,316,925 ns/iter (+/- 201,543)
test tests::bench_vec   ... bench:   1,287,742 ns/iter (+/- 223,129)

Just pushed a patch that changes the benchmark. I think we can do better, but I think the original bench wasn't measuring the right thing. Namely: we were creating a single vec, then filling it. What we want to be testing is creating lots of vecs, and then dropping them. The point is to showcase what vec reuse is like.
Even with a mutex the numbers show a bigger delta with the new bench, and much less variance:
test tests::bench_remem ... bench:     725,083 ns/iter (+/- 10,443)
test tests::bench_vec   ... bench:   1,118,801 ns/iter (+/- 93,288)


Import simple telemetry implementation without native_deps and language interface implementation

I can't accept it while it's still in draft though


Code Quality: Refactor Link_Controller class

Both these classes use a Document_Parser trait. I think that can be easily turned into a class, with some public methods to get the document title, site icon, and a representative image.
Also, the html_to_xpath() method is superfluous too and can be replaced by using $document = Google\Web_Stories_Dependencies\AmpProject\Dom\Document::fromHtml( $markup )