Out of memory issue with lru_maintainer on

I tried to turned on slab_automove=1 in version 1.4.29. But it seems that the automove will only be executed when lru_maintainer is on. I checked the code and the logic has been move to lru_maintainer thread. But I did not see (maybe I missed) that it is mentioned in the protocol. So I turned on the lru_maintainer but I found tons of out of memory error when the memory is fully used. It seems that the lru tail eviction is not happening when there is not free chunks available. I did not see the issue when lru_maintainer is off. Could anyone help me take a look?

Can you include stats settings dump? You also need -o slab_reassign or it won't be able to move pages.
I'll take a closer look soon. That might be a glitch if automove is on but reassign is off (should be a startup error). Can you also try -o modern instead of specifying sub-options? Though you'll need 1.4.31 for that.
thanks

I'm pretty sure I turned on the slab_ressign.
Here are the related logic for slab_automove I found in the code.
When I set slab_automove to 1, only the first code will be executed which stays in lru_maintainer_juggle function.
    if (settings.slab_automove > 0 && chunks_free > (chunks_perslab * 2.5)) {         slabs_reassign(slabs_clsid, SLAB_GLOBAL_PAGE_POOL);     }
                    if (settings.slab_automove == 2) {                         slabs_reassign(-1, orig_id);                     }

Could I get the stats settings output anyway? It would help a lot.
automove == 2 works out pretty badly in practice. it might get removed.
thanks!

I have killed the process last week. But here is my -o params I used to start the process:  -o slab_reassign,slab_automove,maxconns_fast,lru_maintainer,hash_algorithm=murmur3
In my case, I want to use the slab_automove feature but not necessary lru_maintainer. But it would be great if lru_maintainer can give some performance improvement.
I tested the slab move with mcperf and found that it only worked when lru_maintainer is on.
./mcperf --num-conns=100 --conn-rate=1000 --sizes=0.01 --num-calls=10000 -s infra-cache-zen-dev-39c869c0   ./mcperf --num-conns=100 --conn-rate=1000 --sizes=0.001 --num-calls=10000 -s infra-cache-zen-dev-39c869c0 ./mcperf --num-conns=100 --conn-rate=1000 --sizes=0.001 --num-calls=10000 -s infra-cache-zen-dev-39c869c0  -m get
Then I tested with -o slab_reassign,slab_automove,maxconns_fast,lru_maintainer,hash_algorithm=murmur3 with real traffic and tons of
SERVER_ERROR out of memory storing object came into log when there is no free chunks any more.

Where'd you get mcperf from? I'll try to reproduce.
Enabling LRU maintainer does help memory efficiency in a lot of ways. It roughly breaks even CPU wise in my testing. It defers LRU shuffling to when the background thread gets to it, so fetches don't hold the LRU lock. It might look slightly worse in total CPU usage at the low end but scales much better with traffic and the latency is the same all along.
Either way, I'll take a look, thanks.

You can find the mcperf here: https://github.com/twitter/twemperf

Are you using this with the default memory limits, btw? -m 64, you've only listed the -o related options.
thanks

Any chance I could get stats settings from your prod test? This problem isn't obvious yet.
To fully answer your initial question: Yes the slab automove only works with lru maintainer now. I'm not sure how clearly that's documented in the doc/protocol.txt file, and will double check.
The old automove code was a separate thread that very slowly made decisions... it was too slow to ever be effective, and LRU maintainer has a lot of information about what's going on so it works a lot better from there. There is still a hole in the automover if memory is full at all times in all slab classes. It does the movement decision by simply "reclaiming" pages from classes with enough free memory back into a global pool, then they get re-assigned when another class is under pressure.
So if every entry in your systems has a 0 TTL, it can't do anything. The LRU crawler (hopefully less intrusive after 1.4.31!) compliments the automover by clearing out expired memory, which then gets reclaimed to global, then etc. That's why -o modern is what it is.
That all said, with your test you shouldn't be getting OOM errors. It should simply cause evictions. Since you haven't shared the full startup options I can't tell if your memory limit is very low, if -I has been used to increase the memory limit, etc. When I fill memory with those startup options it just evicts a lot.
There are a few conditions which can cause OOM's, but I really need as much information as you can share. You may contact me privately if that helps as well.
thanks for your patience.

For the automove issue, is there any case that user only need the automove but not the lru_maintainer feature? Or should we at least give an error when user only turn on slab_auotmove without lru_maintainer?
For the test environment,  here is my full command to start memcached.
./memcached -m 26500 -c 32768 -o slab_reassign,slab_automove,maxconns_fast,encoded_value,lru_maintainer,hash_algorithm=murmur3 -d
I use 26.5G for the memory. But I remember the OOM issue also came out when I set it to 2.6G.
Ideally, the eviction should happen when there is no free chunks. And it works well when I turn off the lru_maintainer. From the log I printed, it seems that for most of times, it failed to lock any item from the last of 5 of tail of COLD_LRU.
I dug into  a little bit more and did some code changes to find the issue. I think the main cause for the OOM is the new split LRU mechanism. If I always assign the new allocate item to COLD_LRU, turned off the COLD to WARM code for active item, and do unlink and link for item update. The issue is gone.

What's "encoded _value" in there?

automove without lru_maintainer would work for automove=2. I should make it a startup error under any other condition, yes.

I believe that you're seeing something but it doesn't happen under normal conditions, so it's definitely not related to simply having the lru_maintainer on or off. I can start up memcached with those options, flood it with 200 million sets, and it just does a ton of evictions, zero OOM's.

I'm not sure how your code change affected that. Do you have any other patches to the system, or is this just your debug code?
Do you have full stats output from after a run causes OOM's? It would be greatly helpful to get the full output of stats, stats items, stats slabs from an instance after it's been causing OOM's. there're a lot of stats I can comb through for information.
The "failed to lock item", means that the item is in active use for some reason. The weird thing is that evictions only ever happen from COLD_LRU in both cases. With lru_maintainer items drop into COLD_LRU, then evict from the bottom. without lru_maintainer there's only a COLD_LRU, and items evict from that.
So simply moving items between HOT|WARM|COLD wouldn't cause item locks to be held at the bottom of COLD_LRU for no reason. It'd be helpful to see the stats dump since that could help confirm what I'm suspecting.
thanks

(also: are you able to reproduce this at all with mcperf? or were you using mcperf to illustrate how automove worked? I haven't been able to repro yet).

just another general poke on this thread, thanks.

You can ignore the encoded_value. I actually see the issue in the non-changed codes.
I only use mcperf to check if automove works or not.
For OOM issue, my traffic pattern is that it is with around 5200 qps get requests and around 1200 set requests. When you tested with 200 million set test, did you also do the get requests? Since without get traffic, eviction should always happen successfully.
I did not find any logic issue from the code. I also believe ideally there should not be too much OOMs happened. But I feel that the new lru mechanism is less aggressive and take longer time to move a non-popular item to the end of cold lru.

Yes. Code releases are validated by running 12+ hour torture tests with mixed gets/sets totalling in the millions of ops/sec, while randomly shuffling pages between slab classes and running the LRU crawler. It doesn't cause OOM's.
Yes the background process is slightly less aggressive, but given 1200 sets/sec there's no way the background thread would get behind. You need to do 300,000+ generally. Even if it does get behind, it still advances each of the queues in a direct-reclaim mode.
I really honestly need the stats output I asked for before; it has a lot of counters related to reflocks/etc being found while trying to do evictions. I'm completely confident that this feature isn't broken as badly as it would seem; enabling lru maintainer doesn't just break evictions in all cases. There are regression tests, burn-in tests, and at minimum thousands of active users running with the feature.
You gave a hint that the items at the bottom of the LRU were locked for some reason, which is very difficult to cause. Either some (new?) code is leaking locks or something you're doing is holding locks. It's not clear if you meant reflocks or item mutex locks as well. Given your workload and how the mutex locks are used, I'd be very surprised if that were the case.
If items are reflocked, there are some defenses against that but that did change with the LRU maintainer. If reflocked items are frequently landing in your tail, it means your have client which are dying off or hanging without fully reading the data it requested (you can see clients which may be in this mode via stats conns).
The things I really need to know:

ideally full stats, stats items, stats slabs, dumps from a running instance which has exhibited the behavior. stats conns would be good too, but you should sanitize its output.
at very least the ratios of sets to OOM's and what slab classes they were happening on. as well as the lrutail_reflocked and lru_maintainer_juggles counters.

To rule out the more recent changes, you could try 1.4.28. I'm also not sure if you've tried 1.4.31 with these features enabled, as there have been some bugfixes since 1.4.29.
thanks!

Here are the  stats you mentioned you the last post.
stats_conns.txt
stats_items.txt
stats_slabs.txt
stats.txt

Thank you! This helps narrow down what I need to test quite a lot.

Interesting:
STAT items:3:outofmemory 0
STAT items:4:outofmemory 21789845
STAT items:5:outofmemory 119629616
STAT items:6:outofmemory 16695361
STAT items:7:outofmemory 770911
STAT items:8:outofmemory 164951
STAT items:9:outofmemory 125178
STAT items:10:outofmemory 0
STAT items:11:outofmemory 0
STAT items:12:outofmemory 0
[rest are zero]

STAT items:3:moves_to_cold 0
STAT items:3:moves_to_warm 0
STAT items:4:moves_to_cold 0
STAT items:4:moves_to_warm 0
STAT items:5:moves_to_cold 0
STAT items:5:moves_to_warm 0
STAT items:6:moves_to_cold 0
STAT items:6:moves_to_warm 0
STAT items:7:moves_to_cold 23637766
STAT items:7:moves_to_warm 1972675
STAT items:8:moves_to_cold 17614437
STAT items:8:moves_to_warm 2263486
STAT items:9:moves_to_cold 12072967
STAT items:9:moves_to_warm 1953559
STAT items:10:moves_to_cold 12724624
STAT items:10:moves_to_warm 2345664
STAT items:11:moves_to_cold 10267559
STAT items:11:moves_to_warm 2629313
[rest are non-zero, similarly sized]

This is pretty odd. You were telling me that it never worked, but it looks like for low slab classes moves haven't happened (everything sitting in HOT)... however for higher classes the moves work fine and there are never OOM's. lrutail_reflocked isn't high enough to account for this (it would be > than the OOM errors).
Can you please explain what you did to see that was: "From the log I printed, it seems that for most of times, it failed to lock any item from the last of 5 of tail of COLD_LRU" ? exactly what lines did you inspect and what exactly did you see?
I don't believe it's an item lock since that wouldn't hang just some of the slab classes.
Also, any chance of seeing stats settings as well?
thanks!

Here is the settings.
stats_settings.txt
Sorry for the unclear sentence before. "From the log I printed, it seems that for most of times, it failed to lock any item from the last of 5 of tail of COLD_LRU" I actually means that it failed to get the possible eviction item candidate in the lru_pull_tail function. I think in lru_pull_tail, it will try to lock any item in the tail and evict it if evict is set. I added a log after
        if ((hold_lock = item_trylock(hv)) == NULL)             continue;
But it did not print.

you mean like this?
        if ((hold_lock = item_trylock(hv)) == NULL)
            continue;
        printf("got this far\n");

For the instance the stats you gave me are from, are all of the slab classes used at all times, or only some some of the time?
Not sure how I can be super clear: Since the high slab classes have zero OOM's but a lot of moves, and the small slab classes had zero moves, but a lot of OOM's.. Were the small slab classes unused and then suddenly used, or is traffic uniform all the time?

Yes. I added the log like that.
I believe all the slab classes are used all times and the traffic is uniform.
The weird thing is that for example slab 6, there is no free chunks left but there is no items moves between lrus.

Ok, thank you. Well the weird thing is that some slab classes work and some don't... but I'm a little lost as to how this could happen.
Need to think and run more tests. I might ask you to try a patch with some extra counters. I noticed the version listed isn't vanilla 1.4.29 for the instance you gave me. Since I've not seen your changes I have to go on faith that this isn't from a change on your part, is that safe to assume? I'm a little worried about spending a lot of time on this if it is.
Getting a lot closer though. thanks for your patience.

This should be exactly the same codes as vanilla 1.4.29. I only added a default config file to make my packing script work. I did not change any code in this version.

I ask because in the version line:
STAT version 1.4.29_1_gf332f28
It would just be 1.4.29 if it were built from the tarball.
I'm looking into if the internal sizing of the sub-LRU's can get out of sync somehow. That would prevent anything from overflowing from HOT into COLD, and since evictions only come from COLD you'd see this scenario. I can't think of any other way it could happen.
Nothing obvious yet but I haven't had more than a few minutes to look into it. I'll add some asserts and run some load tests, but I might write a patch to expose the values so we can get a sample from your end. if I can't reproduce.

I did some test and print some info and found an issue that might be the root cause.
Please point it out if I'm wrong with anything.
In "do_item_alloc", the slab matching is based on total size of item (header + item).
`    size_t ntotal = item_make_header(nkey + 1, flags, nbytes, suffix, &nsuffix);
if (settings.use_cas) {
ntotal += sizeof(uint64_t);
}
unsigned int id = slabs_clsid(ntotal);`

But in "lru_pull_tail", the limit comparison is based on sizes_bytes
case WARM_LRU:                 if (limit == 0)                     limit = total_bytes * settings.warm_lru_pct / 100;                 if (sizes_bytes[id] > limit) {
And sizes_bytes(in do_item_link_q function) is the total size of all the items, but not counting for the header size.
So let's say if our item size is 100 and the header size is 200, in the data I sent you before, it will be assigned to slab 6 which chunk size is 304. But the sizes_bytes of its hot lru will never exceed 33%. And no movement will happen inter-lru. That's also why we only see this issue in smaller size slab.
From the data I print in my test, we definitely have header which has size over 200. I think it will also be the same problem for other users.
To fix this issue, we should probably count the total size in sizes_bytes rather than item only size.

Was suspicious about that last night. Need to triple confirm and run more tests, but it does look like I intended to do ITEM_ntotal() instead of it->nbytes. Previous to 1.4.29 this would just count the number of items, but I had to switch it so large items could be proportioned correctly.
Would've been easier to just keep counts of active chunks but I wasn't able to make that working before release...
Thanks for sticking with this. Being able to finally see the full stats output made the area of interest obvious and made the rest easy to do, if this indeed the bug :)

https://github.com/dormando/memcached/tree/sizes_bytes_fix can you try this tree, please?
I need to head out shortly but this passes tests. Need to spend some more time with it and add proper regression tests and re-audit for any similar bugs.. At a glance it does look like other parts of the new cod uses ITEM_ntotal() properly.
In the meantime it would be helpful to see if this fixes your problem.

I changed to compare the item number in each lru (if hot lru use more than 33% of chunks, move it) and it worked pretty well. I'll test your fix soon.

Hi,
I think I have the same problem as described here:
Current configuration:
memcached -u nobody -m88064 -c9216 -p11000 -n 5 -f 1.05 -o modern,slab_chunk_max=8192

Memcached stats:

stats.conns
stats.items
stats.settings
stats.slabs

Eviction graph - https://www.dropbox.com/s/yjfv0wh2eupaveo/Screenshot 2016-09-07 15.19.49.png?dl=0 almost instantly when we see evictions application starts reporting out of memory errors.
It started with upgrade to 1.4.29 (previously it was some default ubuntu version 1.4.14), as soon as cache was full we started noticing errors like: MemcachedClient: store failed - SERVER_ERROR out of memory storing object. After upgrade to 1.4.31 the problem is still the same, as soon as cache is full we have problems with out of memory.

@ljagiello any chance you could try this branch as well?

@dormando sure I will try that branch.

Either of you have any luck with the patch?

@dormando so far it's running I need ~2 weeks before my cache will be full, I will try on weekend to reproduce the issue in some controlled environment.

I have tested it through the weekend and it works well.

thank you!

@dormando we have tested it over past 3 days it works well, we don't see no memory , When do plan of releasing with this fix? Thank you !

Same here, looks stable.

Very soon! Was sick last week, apologies

released now in 1.4.32, sorry for the very long delay :(


CPP macros break locations

MACRO(foo) is treated as whatever it expands too. So, in this example:
#define MACRO(x) x
foo x = MACRO(foo) (x+1)
[MAC] and [(] would be highlighted, since the grapher outputs locations for this instead:
#define MACRO(x)
foo x = foo (x+1)

One example where this comes up is the containers library.
<EMAIL_ADDRESS>352
353          STRICT_1_OF_2(go)
354          go m Nil = m


Solr as backend support

Please let me know , if there is a proposal to treat Solr as backend store besides elasticsearch. I would like to contribute in that area.

Hello, @number9code. At the moment, no one working on Ground is planning on supporting Solr as a backend. If you're interested in contributing that code, I'd be happy to talk more and discuss architecture, etc.

Hello @vsreekanti , Sure please let me know how we can talk.

Please email me at vikrams AT berkeley DOT edu, and we can set up some time.


[Bug]: is it a bug or a feature

What happened?
A bug happened!
Something is not adding up.
see the details

closing this ticket, now.
thanks,

ooooops closed by mistake, reopening this ticket now
please disregard previous comment


Docker for stern

Note: this issue was imported from https://github.com/wercker/stern/issues/114, but it was originally created by karancode...
I am using stern on my macbook and I just realized that while switching to window/ubuntu I have to either build it from source or download the binary.
But I cannot use it as a docker container.
This issue is to for an enhancement request to have a docker image for stern.

Note: this comment was imported, but it was originally made by karancode...
I think that this isn’t enough or it’s a complete no go as stern in docker will not be able access to access the logs from the pods.
Still would like to know your views on this!

Note: this comment was imported, but it was originally made by bergerx...
Seems like when combined with #84 it will likely work in cluster with a serviceaccount.


fix microservices base image version

What is the current behavior? (You can also link to an open issue here)
Docker base image has not fixed version, so latest image is used up to production
What is the new behavior?
Docker base image will have a fixed version coming from the result of microservices-base CI build job.
Please check if the PR fulfills these requirements

[ ] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[X] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[X] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Pull request test result: failure :x:


fix: Golang/timestamp

Issue #, if available:
Description of changes:

Aws sdk timestamp fix
Required timestamp in model
Formatting
Java and Rust CI would fail due to Positional (known Issue)

aws-sdk snippet:
func com_amazonaws_dynamodb_RestoreSummary_RestoreDateTime_FromDafny(input interface{}) *time.Time {
	return func() *time.Time {
		var s string

		for i := dafny.Iterate(input.(dafny.Sequence)); ; {
			val, ok := i()
			if !ok {
				break
			} else {
				s = s + string(val.(dafny.Char))
			}
		}
		if len(s) == 0 {
			panic("timestamp string is empty")
		} else {
			t, err := time.Parse("2006-01-02T15:04:05.999999Z", s)
			if err != nil {
				panic(err)
			}
			return &t
		}
	}()
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Raised a PR for the fix expect dafnyTimestamp[..|dafnyTimestamp| - 1] <= retTimestamp[..|retTimestamp| - 1]; https://github.com/smithy-lang/smithy-dafny/pull/691/files


Notebook outline should only list the title, not all the markdown contents

Originally posted here by @MuRongKanKan https://github.com/microsoft/vscode-jupyter/discussions/9426

I can't agree more! The Outline mixes the titles and the contents, it make me mad!

Totally agree

I hope this feature coming soon!!!


Can not disable requests

Hi,
After I import Offline.js, I am setting Offline.options = {requests: false} and it doesn't seem to disable requests (i.e. an ajax request made while offline is still being resent when back online).
It looks like this option is referenced only once, when requests.js is first initialised.
As I am including the .min version of Offline.js (includes requests), I don't understand how it's possible to set Offline.options before requests is loaded and have this option take effect?
Is this a bug, or am I doing something wrong?
Cheers,
Steven

+1, though I did find a workaround:
Offline.on('up', function() {
  Offline.requests.clear();
})

It does seem to be broken, though, as the events are properly being honored, but not the options.


Add a Links section to the readme

Add a Links section like the journey template.  Could be the ones mentioned in the wiki.  Maybe add one specific to hangul.

Added in b5d234301f3514d88d705a59fee8caf878659ef2


No info about linter and render in docs

Dapp has linter and render commands, but documentation doesn't describe them.

Fixed in #712


adding dataprep support for CLIP based models for VideoRAGQnA example for v1.0

Description
Add support for CLIP based multimodal dataprep for VideoRAG usecase.
Issues
The RFC is under review : https://github.com/opea-project/docs/pull/49
Type of change
List the type of change like below. Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds new functionality)
[ ] Breaking change (fix or feature that would break existing design and interface)
[ ] Others (enhancement, documentation, validation, etc.)

Dependencies
List the newly introduced 3rd party dependency if exists.
Tests
Describe the tests that you ran to verify your changes.

@chensuyue can you please tag this PR with 'V1.0' Milestone

@chensuyue  hello Suyue, the File change warning outputs below:

find /home/runner/work/GenAIComps/GenAIComps/.github/workflows/docker/compose/ -name 'dataprep'
grep -q .
'[' -n 'comps/dataprep/vdms/langchain/docker/Dockerfile
comps/dataprep/vdms/multimodal_langchain/docker/Dockerfile' ']'
grep -q dataprep-compose-cd.yaml
echo 'Please check if the added comps/dataprep/vdms/multimodal_langchain/docker/Dockerfile is included in the yaml under path /home/runner/work/GenAIComps/GenAIComps/.github/workflows/docker/compose/dataprep-compose-cd.yaml.'
Please check if the added comps/dataprep/vdms/multimodal_langchain/docker/Dockerfile is included in the yaml under path /home/runner/work/GenAIComps/GenAIComps/.github/workflows/docker/compose/dataprep-compose-cd.yaml.
no_add=TRUE
[[ TRUE == \T\R\U\E ]]
exit 1
Error: Process completed with exit code 1.

Do we need to add this PR into the /workflows/docker/compose/dataprep-compose-cd.yaml file? It seems not all dataprep microsevices are in it.

@lvliang-intel  @Spycsh @XinyuYe-Intel We see the repo is been reorged, pls, help review this PR so we could be merged today

@chensuyue hello Suyue, the File change warning outputs below:

echo 'Please check if the added comps/dataprep/vdms/multimodal_langchain/docker/Dockerfile is included in the yaml under path /home/runner/work/GenAIComps/GenAIComps/.github/workflows/docker/compose/dataprep-compose-cd.yaml.'
Please check if the added comps/dataprep/vdms/multimodal_langchain/docker/Dockerfile is included in the yaml under path /home/runner/work/GenAIComps/GenAIComps/.github/workflows/docker/compose/dataprep-compose-cd.yaml.
no_add=TRUE
[[ TRUE == \T\R\U\E ]]
exit 1
Error: Process completed with exit code 1.

I have added VDMS into the /workflows/docker/compose/dataprep-compose-cd.yaml file, but what is this used for?

This is new for release image build/scan/publish.


Removes commented out codes

Coverage remained the same at 100.0% when pulling faebb322008e6ef8df49090781f889b653ff9c5b on ch-testing-130169407 into 836eb8c6642a9128597c1b42a6a3501899705121 on develop.


[Core]aaz: Improve generic update arguments to consist with previous behaviors

Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

[ ] I adhere to the Error Handling Guidelines.

Core


Compatibility request for Create

Using the non-standard placements for slabs on moving constructs causes the slabs to lose their texture and become pink/black checkered blocks. This isn't an issue with Quark vertical slabs, so maybe find pointers there? Here's a reference to the issue on Create's page. Also I mixed this and Klee slabs up and made a couple reports with those. These are the real ones.
Forge: 31.2.23
MC: 1.15.2
Create: 0.2.4d
Double Slabs: 2.7.6

The top block next to the wool is a normal slab, the rest are Quark vertical slabs.
https://github.com/Creators-of-Create/Create/issues/409
Thanks!

Thanks for the report, I have a suspicion it may be due to something on the other mods side and how they render the constructs as the vertical slabs use something known as model data which allows any slab to be a vertical slab. When there is no model data, it reverts to the missing model texture hence the appearance shown. I'll take a look into this though and get back to you!

Huh. That sounds super cool. I figured you used something like that since there aren't extra blocks and it's a lightweight mod. Thanks for the quick response!

Hi there! Sorry for the extreme delay, I believe Create has fixed this :)


Updating link to Cypress documentation from main README

Summary of changes
Please provide the following information:

Clarifying text that links to documentation from main README in the branch.

[x] I confirm this contribution is my own and I agree to license it with Apache 2.0.
[x] I confirm the moderators may change the PR before merging it in.

Maintainers of this repository update the Mbed OS and Client hashes.
Please report any issues through issues in relevant repositories.

Thanks for your changes.
@ARMmbed/pdmc-integrators please review.

Jenia Kogan, can you please review this PR?
Thanks!

approved from my side
@yogpan01 @teetak01 can this be merged?


Simplify BigQuery Service onboarding

Is your feature request related to a problem? Please describe.
Creating a new bigquery service is not user friendly.
Describe the solution you'd like
A lot of fields (project id, private key, client email ..etc) could be extracted from a GCP Service Account Key JSON file.
Describe alternatives you've considered
N/A
Additional context

Hey @haithem-souala, The bigquery service there are two ways you can pass the credentials, one is through passing the direct values like project id, private key, client email etc or one is through passing the file path of GCP Service Account Key JSON file.
select the GCS Credentials Configuration value as GCS Credentials Path to input the Service Account Key JSON file path.

You can select

Thanks for your quick reply.
GCS Credentials Path, that mean i should download the Key from GCP console, then upload it somewhere, and use the SA key json file URL to configure the BQ Service, i mean, i think that too much.
What about just a field where we can put the content of the SA key file or upload the key.

ok So in the GCS Credentials Path we are expecting a local file path so if you have downloaded in your Downloads folder then you can provide value as <Path of Downloads Dir>/<Filename>.json, I think we can improve the description here to make sure this confusion does not happen.
Also the DATABASE and USERNAME field has been removed, this will be reflected from 0.11.0 release.

@ulixius9 did we handle adding clear description?


PKS on Azure docs clarifications

From https://docs.pivotal.io/pks/1-6/installing-pks-azure.html

"enter the FQDN that you registered to point to the PKS API load balancer, such as api.pks.example.com. To retrieve the public IP address or FQDN of the PKS API load balancer, see the terraform.tfstate file."
Q: what value from terraform.tfstate?
Default Security Group :"The name of the security group attached to the cluster's subnet"
Q: Which security group created by Terraform? There are 5:
-bosh-deployed-vms-security-group
-ops-manager-security-group
-pks-api-sg
-pks-internal-sg
-pks-master-sg
In Enter Primary Availability Set. "erraform creates this availability set and its name is YOUR-ENVIRONMENT-NAME-pks-as"
Q. The actual name created is ENVIRONMENT-NAME-availability-set so is doc incorrect?

Hello @sudhakarau1 .  Thank you.
We're revisiting this request as part of removing all terraform references. You can see the changes for these particular terraform references here:
Known Issue: Azure Default Security Group Is Not Automatically Assigned to Cluster VMs:

TKGI v1.10: https://docs-pcf-staging.cfapps.io/tkgi/1-10/release-notes.html#security-group

Kubernetes Cloud Provider: Default Security Group and Primary Availability Set:

TKGI v1.10: https://docs-pcf-staging.cfapps.io/tkgi/1-10/installing-azure.html#cloud-provider

This rewording has been revised in TKGI v1.8 - TKGI v1.10 documentation.
Other terraform references will be removed on other tickets.


[frontend, dev_compiler] Make sure that expression compilation worker reuses the language settings from dill files

Expression compiler worker in ddc reuses the same options for all module dill files that are loaded into it. The experimental flags might differ from module to module - find a way to transfer the effects of those settings from the original library to the debug library.
https://github.com/dart-lang/sdk/blob/8030a3f68ecb38cbf45223023794ccce1bedd4cd/pkg/front_end/lib/src/fasta/incremental_compiler.dart#L1590
Update: looks like the language mode should be the same for all libraries, see https://github.com/dart-lang/sdk/issues/43307

This was resolved by adding a null safety setting to module metadata in https://github.com/dart-lang/sdk/issues/43307


actions.invoke returning a 405 (method POST not supported)

I was getting errors trying actions.invoke but not getting any error stack back (using .then().catch(err)). actions.list works fine which means my base setup of API key etc was fine. I added a line of instrumentation into base_operation.js handle_errors() which showed that OW was returning a 405 method not allowed.

Hello Fintan,
Can you do a few things to help me diagnose this...

Are you able to provide me the output from the following command?

wsk action get ACTION_NAME

Ensure you redact any confidential information!

Could you provide me the source code for the client library invocation?

It looks like you are on Windows, what version are you using?

We took this offline to resolve.


Remove a space in the  tag

I think that corrects this thing: #463

?

Hey, add space ☝🏼
This work for me: document.body.className = document.body.className.replace(/ pace-done/g, '');

This works always for me, whether the body has class or not
.replace(/\s?pace-done/g,"")


Remove duplicated text

"of bike paths" appeared twice in the sentence.

Thanks!

@all-contributors please add @BioGeek for doc


Draw marking points at <from, to> in path.

Draw marking points at <from, to> in path.
It must be anything different with basic NAVER mark not to be confused.

Draw marking points at <from, to> in path.
It must be anything different with basic Daum mark not to be confused.


collarbora support

hi do you plan to add collabora docker container support with your nextcloud container, and your docker-compose file?

Collabora works fine with the "collabora/code" image,
docker-compoes.yml:
version: "2"

services:
  collabora:
     restart: always
     image: collabora/code
     tty: true
     cap_add:
     - MKNOD
     environment:
     - "domain=cloud\\.nextcloud\\.com|cloud\\.example\\.org"
     ports:
     - "9980:9980"

NGINX Config:
  location ^~ /loleaflet {
    proxy_pass https://docker.lan:9980;
    proxy_set_header Host $http_host;
  }

  location ^~ /hosting/discovery {
    proxy_pass https://docker.lan:9980;
    proxy_set_header Host $http_host;
  }

  location ^~ /lool {
    proxy_pass https://docker.lan:9980;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
  }


(TypeScript) Generics with extends broken from 10.3.0

Describe the issue

Generic types using the extends keyword break subsequent highlighting.

I bisected a bit and tracked it down to a change between 10.2.1 and 10.3.0
Could be this commit? (only because It's the only one in there marked typescript that I could see.)
Which language seems to have the issue?
TypeScript
Are you using highlight or highlightAuto?
I've definitely seen it with hljs.highlightBlock and hljs.highlight in my code.
Sample Code to Reproduce

http://jsbin.com/tutirup/edit?html,output
interface Prefixer<Something extends string> {
  (): `other__${Something}`;

  parse: <From extends string>(
    value: From
  ) => number;
}

const cloneWith = <T, A extends keyof T, V>(
  i: T,
  a: A,
  value: V
): Omit<T, A> & {[K in A]: V} => ({
  ...i,
  [a]: value,
});

const source = { name: 'bob', age: 90 };
const result = cloneWith(source, 'age', 'old!');
type TestSource = typeof source.age // number
type TestResult = typeof result.age // string

Expected behavior

Switching back to 10.2.1 gives this:

Additional context

I've seen this in a couple of places... my blog, VSCode's markdown preview and found an issue in SO meta (i stole some code from the question linked there for this repro).

Just to be clear is this an issue with latest v11 also then?

Ahh sorry, yes the wasn't very clear.
I found the problem in 11.1.0.
10.3.0 only came up because that was the first version I could find which showed the problem.

Isn't <> typecasting deprecated in favor of as?
  parse: <From extends string>(
    value: From
  ) => number;

I'm not sure I'm even following what this is doing sytantically.

:) it is pretty contrived for sure. looking now I'm not sure I can even make up a reason someone would want a type like that. but the syntax is valid. It's not a typecast:, it's a generic type.
Thanks for looking at the problem! If there's any thing i can do to help get that PR over the line then please let me know.

@jaybeeuu
If we deal with <Something extends ... as a special case, will that solve things or are there other variants of types that will also look like html? (I mean technically it could still be HTML, but that's a smaller edge case IMHO)
<T, A extends keyof T, V>
This is easy to handle because , is not valid HTML - so we can immediately bail.

Hey, yeah I think even for a proper parser its a headache to tell the difference. I'm pretty sure that's why why typescript requires the use of the .tsx extension to enable jsx. I think it puts the parser into a different mode. Certainly on things like arrow functions in .tsx files it seems to assume jsx before generic so you need to do things like add commas to tell it to stop.
To my knowledge the only things that can follow the name of a type argument in a to generic are

,
extends

I dont think there's any other keywords or operators, and a quick scroll through the docs confirms that.
I reckon you're right that handling extends as a special case would be ok.
(Next issue is bound to be someone saying their jsx with extends boolean prop has stopped highlighting... but at least i'll be happy.... :D )


[SqlCall.js] Remove .connect and .setOptions #56

Resolves #56

Closing this PR and reopening it to target master branch instead of v1.0-dev


Poor error message for unsupported operations

CertificateClient supports Key Vault version 2016-10-01, but that service version doesn't support the backup_certificate operation added in 7.0. So this code raises an error:
from azure.keyvault.certificates import ApiVersion, CertificateClient

client = CertificateClient(VAULT_URL, object(), api_version=ApiVersion.V2016_10_01)
client.backup_certificate("...")

The error suggests the client or service doesn't support this particular API version, when in fact it's only this particular operation which is not supported:
  File "...\azure-keyvault-certificates\azure\keyvault\certificates\_generated\_operations_mixin.py", line 54, in backup_certificate
    raise NotImplementedError("APIVersion {} is not available".format(api_version))
NotImplementedError: APIVersion 2016-10-01 is not available

Have opened an issue for this in autorest: https://github.com/Azure/azure-sdk-for-python/blob/master/sdk/textanalytics/azure-ai-textanalytics/azure/ai/textanalytics/_text_analytics_client.py#L504

Closing this because the autorest issue is closed.


i sent my serum to my Upbit address

i sent my serum to my Upbit address but i can't recieved yet

Can you share the transaction? This is likely an upbit problem.

it's right?

@lemon1825 you might want to give a URL link to the transaction instead


Foreign Key Popup not opening in a new window

Hi,
When I've got a foreign key field, it's possible to "add" a new foreign key value. Normally with Django, it opens a new browser (target=blank).
However, with Material, it just opens a new page- meaning you leave the original model you were trying to edit.
I made a screenshot which should make more sense.
Not sure if this is a one-off bug for me, or a project wide-issue...

Can you reproduce it here http://forms.viewflow.io/admin/integration/continent/add/ ? With largest Country FK ?

Any comments?

Closed, no activity, can't reproduce.

Sorry- I've now subscribed.
No- not able to reproduce it.
My setup is django 1.9 with virtualenv on Python 3.4
I just set up a fresh repo, and same issue is reproduced. The console shows no error.

yep, it reproduced on dj19

So downgrade to 1.8 or wait for a fix? Thanks

Pull requests are welcome ;)

Sorry for not mentioning earlier- this seems to affect Safari (NOT Google Chrome).

No, it should works without refresh. Any js errors in console?

Nope, no js errors in the console.
I do notice a jQuery error when a model has a list filter on it (only seems to be in Chrome though)-  "(index):655 Uncaught TypeError: $(...).actions is not a function".
I'm downgrading my django to 1.8, will report back

I noted that occurence only in django-material 0.8.0. I tested with verstion 0.7.0 and the branch master worked fine.

This isn't correct for me. Both 0.8.0 and 0.7.0 have the same issue (as above in screenshots) with django 1.9.
Has anyone made progress on this?


[✨] qwik-auth release cycle is not in sync with the monorepo

Is your feature request related to a problem?
As it's clear in this commit a bug in qwik-auth is solved by a team member of the auth.js.
It's integrated into the qwik-auth.
Yet there is no new publish for it. The last publish belongs to 4 months ago (0.1.3).
Since it's part the of qwik ecosystem, why not publishing it with the monorepo?
Describe the solution you'd like
We would like to receive bug fixes as soon as they are available. We don't know how much should we stay behind until you publish the @builder.io/qwik-auth package again.
Describe alternatives you've considered
For now, our only option is to downgrade everything and we are very upset doing it, as it's time-consuming and requires a lot of extra work from our side for many projects.
Additional context
No response

The issue might not be coming from qwik-auth. I was just upgrading the X stack https://github.com/x-ploration-of-mars/x-stack and as you'll see in the repo and the live demo, it's working fine up to @auth/core 0.21.0 and vite 4.5.2. I found out that upgrading to vite 5 causes an issue with qwik-auth, same for upgrading to @auth/core 0.22.0 and later... I haven't been able to make a reproduction for it yet, but this might also be because it's deployed on the edge.
So apart from keeping @auth/core at 0.21.0 and vite at 4.5.2, you shouldn't have to stay behind. Feel free to make a reproduction and describe your issue with more details if you want further help.

Ah also, qwik-auth isn't sufficiently related to qwik/core or qwik-city to have them in the same release cycle. As per my comment in https://github.com/BuilderIO/qwik/pull/5790#issuecomment-1925187201, we'll likely migrate it to @auth/qwik eventually (might be soon 👀).

This is solved, we recently release a new version.
Spoiler: we are working to move the integration in the official Next-Auth repo :tada:


Add type to represent a loaded configuration file

Changes
Export type for a loaded configuration file such that downstream applications can read its contents in the same way the SDK does. Rename the loader instance to ConfigFile to make it symmetric with ConfigAttributes.
Tests
This doesn't change functionality.

[x] make test passing
[x] make fmt applied
[x] relevant integration tests applied

Integration tests all pass.


Write metadata

Google document for metadata:
https://docs.google.com/document/d/150e2fCIOoHsMMcVoC4GilHRYDj0rzwNbAjvE-i9Xk5I/edit#

Metadata are written and published.


fix(engine-core):  tell webpack to ignore require of napi library

We use prisma in AWS lambda functions that are packaged via webpack.
When trying out the napi preview, webpack tried to resolve the following require at build time and failed:
https://github.com/prisma/prisma/blob/1fcdcf55ed51219a408eaf37ecdf1614f5c7b770/src/packages/engine-core/src/NAPIEngine.ts#L240
When adding the "magic comment" /* webpackIgnore: true */ to this require, webpack will ignore this require and it will be resolved at runtime (see https://webpack.js.org/api/module-methods/#magic-comments).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks Manuel for this PR 💚


HaLVM build relocation should be possible

Problem is equivalent to this MinGHC issue.
Alex and Happy data-dir paths are being hardcoded relative to the build root, and the data is not copied into the install prefix. So, if you happen to be trying to distribute your built HaLVM artefacts, alex and happy will crash with IO errors:
alex: /home/tim/HaLVM/platform_ghc/share/x86_64-linux-ghc-7.8.4/alex-3.1.4/AlexTemplate-ghc: openFile: does not exist (No such file or directory)

Mostly done, with one problem remaining: hsc2hs. It's dynamically linked by default, so the resulting binary requires the precise versions of array etc that were present on the builder's system.
Building it statically would fix


CI: Deduplicate PR workflows

The "tests on Ubuntu" step fails but appears unrelated.

The "tests on Ubuntu" step fails but appears unrelated.

Yep, re-ran and all good. Looks like it was just a temporary mirror failure.


Added integration tests between sendResetPwd and resetPwdLong/Short

As a demonstration of for https://github.com/feathers-plus/feathers-authentication-management/issues/110

Added by https://github.com/feathers-plus/feathers-authentication-management/commit/65a0e0493943722e982da4c4c6b5042344c2e057.


Ruby user-agent grows longer with each new stub instantiated

What version of gRPC and what language are you using?
grpc-ruby/1.9.1
What operating system (Linux, Windows, …) and version?
Linux 3.16.0-5-amd64 Debian 3.16.51-3+deb8u1 (2018-01-08) x86_64 GNU/Linux
What runtime / compiler are you using (e.g. python version or version of gcc)
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
What did you do?
We have a Ruby gRPC client test script that starts getting an error we'd never seen before after running for a while: #<GRPC::ResourceExhausted: 8:to-be-sent initial metadata size exceeds peer limit>. If we keep running it, it goes on to fail differently: #<GRPC::Cancelled: 1:unknown cause> with trailing metadata of :status 431. HTTP 431 is "Request Header Fields Too Large."
The test creates a new stub on each iteration. We tracked the issue down to the user-agent header growing by one "grpc-ruby/1.9.1 " prefix with each new stub. So requests from the first stub have user-agent=grpc-ruby/1.9.1 grpc-c/5.0.0 (linux; chttp2; glossy). Requests from the second stub have user-agent=grpc-ruby/1.9.1 grpc-ruby/1.9.1 grpc-c/5.0.0 (linux; chttp2; glossy), and so on.
What did you expect to see?
user-agent=grpc-ruby/1.9.1 grpc-c/5.0.0 (linux; chttp2; glossy) all the way.
What did you see instead?
Requests from the second stub have user-agent=grpc-ruby/1.9.1 grpc-ruby/1.9.1 grpc-c/5.0.0 (linux; chttp2; glossy), and so on.

@duelinmarkers this is a guess: are you able to see if this leak still occurs on the 1.10.pre1 on pre2 releases?
I am wondering if this was possibly fixed by a PR that made it into the 1.10 branch (https://github.com/grpc/grpc/pull/14501).
If not, do you have any tips to reproduce beyond creating multiple stubs from the same process?
I am trying to reproduce this with the greeter server and client but it is not easily reproducing.

@apolcyn (I'm working with @duelinmarkers on this) We still see the issue on 1.10.pre1 and pre2. I've narrowed it down to an issue when we've specified channel_credentials. So, with a ruby client and java server, we've found that we only observe the user-agent growth issue when using SSL and goes away when SSL is removed, everything else the same. This is why the greeter server and client example isn't able to reproduce it.

I tracked this down to (1) the fact that we're reusing the same channel_args Hash to create each stub and (2) this code that mutates the input channel_args each time a new stub is created.
https://github.com/grpc/grpc/blob/59ea0ae3ebcca0aef5a15c5aa5b4d27b7f3fc9c4/src/ruby/lib/grpc/generic/client_stub.rb#L38-L43
I think it would be best for the grpc library to channel_args.dup before mutating it in setup_channel. If others agree we can submit a PR for this. For now we'll work around it by .duping the channel_args from our config when creating new stubs.

@duelinmarkers thanks for tracking this down. I agree that this API could be made safer by duping the channel args that are passed into it, but I'm not jumping on it just yet since I'm wondering if this kind of thing could break anyone.
However I think that duping the channel args internally is non-breaking, because it looks like the only thing it would prevent is the case when someone wanted their using string to get a " grpc-ruby/#{VERSION}" added to it on each new stub, which is unlikely.


update  deprecated instrumented registry

Pull Request Description:

Updated imports to use androidx.test.platform.app.InstrumentationRegistry for improved compatibility with modern Android testing practices.
Refactored code to use InstrumentationRegistry.getInstrumentation().getTargetContext() for obtaining the application context instead of deprecated methods.
-This pull request solved the issuse no. 435  on  tool-suite-X
All Test are passed after description :

Notified @Lamouresparus  about @wbrunette requesting a review


[MXNET-319] Reduce Scala Warning messages

Description
The previous JavaDoc generation is causing too much warning messages. This PR is intend to reduce the numbers of issues we have as indicated in https://github.com/apache/incubator-mxnet/issues/11603.
There are three fixs:

Seq[NDArray] erasure problem, same explained in here
Add <pre> tag on JavaDoc to reduce warnings
Minor fixing on the other files

@nswamy @yzhliu @andrewfayres
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Have you checked the javadoc generated?

#11603


map_fn can't process sparse tensor in graph mode

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.5.0
Python version: 3.8.5
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

v2.5.0-rc3-213-ga4dfb8d1a71 2.5.0
Describe the current behavior
the following customized keras layer can't be executed in graph mode.
#!/usr/bin/python3

import numpy as np;
import tensorflow as tf;

# please ignore this function, it is just for generating a sparse tensor
def Dense2Sparse():
  dense = tf.keras.Input((None, None, None)); # dense.shape = (batch, num_heads, query_length, key_length)
  mask = tf.keras.Input((1, None, None)); # mask.shape = (batch, 1, query_length or 1, key_length)
  reshaped_mask = tf.keras.layers.Lambda(lambda x: tf.cond(tf.math.not_equal(tf.shape(x[0])[2], tf.shape(x[1])[2]), lambda: tf.tile(x[0], [1,1,tf.shape(x[1])[2],1,]), lambda: x[0]))([mask, dense]); # mask.shape = (batch, 1, query_length, key_length)
  reshaped_mask = tf.keras.layers.Lambda(lambda x: tf.tile(x[0],[1,tf.shape(x[1])[1],1,1]))([reshaped_mask, dense]); # mask.shape = (batch, num_heads, query_length, key_length)
  indices = tf.keras.layers.Lambda(lambda x: tf.where(tf.cast(x, dtype = tf.int32)))(reshaped_mask); # indices.shape = (num non zero values, 4)
  values = tf.keras.layers.Lambda(lambda x: tf.gather_nd(x[0], x[1]))([dense, indices]); # values.shape = (num non zero values)
  sparse = tf.keras.layers.Lambda(lambda x: tf.sparse.SparseTensor(x[0], values = x[1], dense_shape = tf.cast(tf.shape(x[2]), dtype = tf.int64)))([indices, values, dense]);
  return tf.keras.Model(inputs = (dense, mask), outputs = sparse);

# NOTE: ***this layer cannot be executed in graph mode***
class SparseDenseMatMul(tf.keras.layers.Layer):
  def __init__(self, **kwargs):
    super(SparseDenseMatMul, self).__init__(**kwargs);
  def call(self, inputs):
    a = inputs[0]; # a.shape = (batch, heads, query_length, key_length)
    b = inputs[1]; # b.shape = (batch, heads, key_length, value_dim)
    reshaped_a = tf.sparse.reshape(a, (-1, tf.shape(a)[-2], tf.shape(a)[-1])); # reshaped_a.shape = (batch * heads, query_length, key_length)
    reshaped_b = tf.reshape(b, (-1, tf.shape(b)[-2], tf.shape(b)[-1])); # reshaped_b.shape = (batch * heads, key_length, value_dim)
    def dot(x):
      a = x[0];
      b = x[1];
      c = tf.sparse.sparse_dense_matmul(a,b);
      return c; # c.shape = (query_length, value_dim)
    results = tf.map_fn(dot, (reshaped_a, reshaped_b), fn_output_signature = tf.TensorSpec((tf.shape(reshaped_a)[-2], tf.shape(reshaped_b)[-1]), dtype = tf.float32));
    results = tf.reshape(results, (tf.shape(a)[0], tf.shape(a)[1], tf.shape(results)[-2], tf.shape(results)[-1]));
    return results;

a = np.random.normal(size = (4,3,10,40));
mask = np.random.randint(low = 0, high = 2, size = (4,1,10,40));
a = Dense2Sparse()([a, mask]);
b = np.random.normal(size = (4,3,40,20)).astype(np.float32);

print(SparseDenseMatMul()([a,b])); # eager mode is OK

inputs_a = tf.keras.Input((None, None, None), sparse = True);
inputs_b = tf.keras.Input((None, None, None));
results_c = SparseDenseMatMul()([inputs_a,inputs_b]);
model = tf.keras.Model(inputs = (inputs_a, inputs_b), outputs = results_c);
print(model([a,b])); # graph mode is failed

Describe the expected behavior
map_fn should process sparse tensor in both eager and graph mode.
Contributing

Do you want to contribute a PR? (yes/no): no
Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
#!/usr/bin/python3

import numpy as np;
import tensorflow as tf;

# please ignore this function, it is just for generating a sparse tensor
def Dense2Sparse():
  dense = tf.keras.Input((None, None, None)); # dense.shape = (batch, num_heads, query_length, key_length)
  mask = tf.keras.Input((1, None, None)); # mask.shape = (batch, 1, query_length or 1, key_length)
  reshaped_mask = tf.keras.layers.Lambda(lambda x: tf.cond(tf.math.not_equal(tf.shape(x[0])[2], tf.shape(x[1])[2]), lambda: tf.tile(x[0], [1,1,tf.shape(x[1])[2],1,]), lambda: x[0]))([mask, dense]); # mask.shape = (batch, 1, query_length, key_length)
  reshaped_mask = tf.keras.layers.Lambda(lambda x: tf.tile(x[0],[1,tf.shape(x[1])[1],1,1]))([reshaped_mask, dense]); # mask.shape = (batch, num_heads, query_length, key_length)
  indices = tf.keras.layers.Lambda(lambda x: tf.where(tf.cast(x, dtype = tf.int32)))(reshaped_mask); # indices.shape = (num non zero values, 4)
  values = tf.keras.layers.Lambda(lambda x: tf.gather_nd(x[0], x[1]))([dense, indices]); # values.shape = (num non zero values)
  sparse = tf.keras.layers.Lambda(lambda x: tf.sparse.SparseTensor(x[0], values = x[1], dense_shape = tf.cast(tf.shape(x[2]), dtype = tf.int64)))([indices, values, dense]);
  return tf.keras.Model(inputs = (dense, mask), outputs = sparse);

# NOTE: ***this layer cannot be executed in graph mode***
class SparseDenseMatMul(tf.keras.layers.Layer):
  def __init__(self, **kwargs):
    super(SparseDenseMatMul, self).__init__(**kwargs);
  def call(self, inputs):
    a = inputs[0]; # a.shape = (batch, heads, query_length, key_length)
    b = inputs[1]; # b.shape = (batch, heads, key_length, value_dim)
    reshaped_a = tf.sparse.reshape(a, (-1, tf.shape(a)[-2], tf.shape(a)[-1])); # reshaped_a.shape = (batch * heads, query_length, key_length)
    reshaped_b = tf.reshape(b, (-1, tf.shape(b)[-2], tf.shape(b)[-1])); # reshaped_b.shape = (batch * heads, key_length, value_dim)
    def dot(x):
      a = x[0];
      b = x[1];
      c = tf.sparse.sparse_dense_matmul(a,b);
      return c; # c.shape = (query_length, value_dim)
    results = tf.map_fn(dot, (reshaped_a, reshaped_b), fn_output_signature = tf.TensorSpec((tf.shape(reshaped_a)[-2], tf.shape(reshaped_b)[-1]), dtype = tf.float32));
    results = tf.reshape(results, (tf.shape(a)[0], tf.shape(a)[1], tf.shape(results)[-2], tf.shape(results)[-1]));
    return results;

a = np.random.normal(size = (4,3,10,40));
mask = np.random.randint(low = 0, high = 2, size = (4,1,10,40));
a = Dense2Sparse()([a, mask]);
b = np.random.normal(size = (4,3,40,20)).astype(np.float32);

print(SparseDenseMatMul()([a,b])); # eager mode is OK

inputs_a = tf.keras.Input((None, None, None), sparse = True);
inputs_b = tf.keras.Input((None, None, None));
results_c = SparseDenseMatMul()([inputs_a,inputs_b]);
model = tf.keras.Model(inputs = (inputs_a, inputs_b), outputs = results_c);
print(model([a,b])); # graph mode is failed

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
Traceback (most recent call last):
  File "test.py", line 44, in <module>
    results_c = SparseDenseMatMul()([inputs_a,inputs_b]);
  File "/home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 969, in __call__
    return self._functional_construction_call(inputs, args, kwargs,
  File "/home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1107, in _functional_construction_call
    outputs = self._keras_tensor_symbolic_call(
  File "/home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 840, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py", line 880, in _infer_output_signature
    outputs = call_fn(inputs, *args, **kwargs)
  File "/home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 695, in wrapper
    raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

    test.py:32 call  *
        results = tf.map_fn(dot, (reshaped_a, reshaped_b), fn_output_signature = tf.TensorSpec((tf.shape(reshaped_a)[-2], tf.shape(reshaped_b)[-1]), dtype = tf.float32));
    /home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:51 __init__  **
        self._shape = tensor_shape.TensorShape(shape)
    /home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:765 __init__
        self._dims = [Dimension(d) for d in dims]
    /home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:765 <listcomp>
        self._dims = [Dimension(d) for d in dims]
    /home/xieyi/.local/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:206 __init__
        six.raise_from(
    <string>:3 raise_from

    TypeError: Dimension value must be integer or None or have an __index__ method, got value '<tf.Tensor 'sparse_dense_mat_mul_1/strided_slice_4:0' shape=() dtype=int32>' with type '<class 'tensorflow.python.framework.ops.Tensor'>'

@breadbread1984
Some Arguments are deprecated in map_fn like dtype in future versions,for this refer  the document,might be that is causing TypeError.
Looking at the error log at is looks like this #169 and let us know if it helps.Thanks

I am not using dtype of map_fn, the dtype is argument of TensorSpec.

@Saduf2019
I was able to replicate the issue reported.Please find the gist here.Thanks

@breadbread1984
Please post this issue on keras-team/keras repo.
To know more refer to:
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999

To make your example work, you could simply replace
    results = tf.map_fn(dot, (reshaped_a, reshaped_b), fn_output_signature = tf.TensorSpec((tf.shape(reshaped_a)[-2], tf.shape(reshaped_b)[-1]), dtype = tf.float32));

with
    results = tf.map_fn(dot, (reshaped_a, reshaped_b), fn_output_signature = tf.TensorSpec((reshaped_a.shape[-2], reshaped_b.shape[-1]), dtype = tf.float32));

In graph mode, tf.shape returns a tf.Tensor which is not statically known in graph mode. As the error message suggested, Dimension value must be integer or None or have an __index__ method and cannot be a tensor. The shape attribute of a SparseTensor is a TensorShape which is statically known.


discharge value NULL in output streams layer when above spawning threshold

eg, on North Thompson just downstream of Clearwater.
Likely a join issue, discharge seems fine where there is spawning.

Not a join issue, this is by design. Generating discharge on 8th and greater streams takes too much time to be worth the processing effort:
https://github.com/smnorris/bcfishpass/blob/main/scripts/discharge/sql/discharge03_wsd.sql#L13

Confirmed that we do not currently need to model discharge on major rivers (stream_order >=8).


1.20.2 Port

@Gaming32 can we not replace nullables with optionals without reason? I see a one or two instances where you replaced nullables with optionals without it being required for a codec.

It's not without reason. Mojang prefers Optionals here, and I would rather keep it consistent than keep it as nullables for no reason.


Unable to run the attacks and defenses?

Whenever I try to run the run_attacks_and_defenses.py or the run_attacks_and_defenses.sh, I end up getting stuck at "Restoring parameters from inception_v3.ckpt" -- would anyone know how I can get past this point?
Thanks

@AlexeyKurakin ?

Could you provide some additional information about the issue.

Did you modify the code of the attack? I yes - does it work fine with unmodified version?

How long did you wait? It looks like you running it on CPU, and it's quite slow to run inception model on CPU. So it might be the case that your model is just running and you need to wait longer.
Restoring parameters from the checkpoint may take up to a few minutes (depeding on you CPU and disk speed). After parameters are restored model starts generating adversarial examples, which may take some time on CPU
To see if your model actually doing anything, you can check output folder to see if there is any files saved there. Also you can add additional printing/logging before and after session run.

Thanks Alex, feel free to reopen the issue @rittikaadhikari if that did not solve it for you.


Hosted agent missing Anaconda2

Agent version and platform
Working 2.119.1
Failing  2.123.0
VSTS type and version
VSTS Hosted 2017
What's not working?
We are compiling a python program into an executable, first step of this is to copy Anaconda2 to a directory without spaces, however it doesn't seem to be installed anymore or has been moved...
Don't know when the change occurred because this repository is not changed very often
Agent and Worker's diag log
Working
2017-06-23T15:37:26.6114216Z ##[section]Starting: Build
2017-06-23T15:37:26.6144216Z Current agent version: '2.119.1'
....
2017-06-23T15:37:35.5266443Z ==============================================================================
2017-06-23T15:37:35.5266443Z Task         : Command Line
2017-06-23T15:37:35.5266443Z Description  : Run a command line with arguments
2017-06-23T15:37:35.5266443Z Version      : 1.1.3
2017-06-23T15:37:35.5266443Z Author       : Microsoft Corporation
2017-06-23T15:37:35.5266443Z Help         : More Information
2017-06-23T15:37:35.5266443Z ==============================================================================
2017-06-23T15:37:35.5706446Z ##[command]xcopy "C:\Program Files (x86)\Anaconda2" "C:\Anaconda2_32" /s /e /y
2017-06-23T15:37:35.6519640Z C:\Program Files (x86)\Anaconda2\cwp.py
Current
2017-10-02T16:29:31.0026934Z ##[section]Starting: Build
2017-10-02T16:29:31.0056944Z Current agent version: '2.123.0'
.....
2017-10-02T16:29:43.6687365Z ==============================================================================
2017-10-02T16:29:43.6687365Z Task         : Command Line
2017-10-02T16:29:43.6687365Z Description  : Run a command line with arguments
2017-10-02T16:29:43.6687365Z Version      : 1.1.3
2017-10-02T16:29:43.6687365Z Author       : Microsoft Corporation
2017-10-02T16:29:43.6687365Z Help         : More Information
2017-10-02T16:29:43.6687365Z ==============================================================================
2017-10-02T16:29:43.7264176Z ##[command]xcopy "C:\Program Files (x86)\Anaconda2" "C:\Anaconda2_32" /s /e /y
2017-10-02T16:29:43.8046568Z File not found - Anaconda2

@phatcher I see that the last successful run was 6/23. We have had many changes to the image for the hosted 2017 build pool that remove things that are installed. I have reached out to someone to see if Anaconda2 is one of those things. I am guessing it's not due to the difference in agent version (doesn't correlate to what's installed on hosted build agents).

@stephenmichaelf Yes it was a little while ago, software was stable but someone found a bug so the CI build triggered.
We had to change to the VS 2017 Host to get the python tools in the first place, so I'd be a bit surprised if you removed them
If it's gone what's the alternatives apart from flipping to a local agent?

@phatcher You can take a look here to see what's installed on the build machines. We do have Python 2.7.13 and 3.6.2 on the machines but not Anaconda 2.
As of today you can't do anything that needs to run as Admin(this obviously limits what can be installed), we are in progress on adding that feature, should be available in the next couple months.
For your specific case, I would recommend going to a local agent. Even when we have the ability to run as Admin you would still have to wait for any additional software to be installed for each build. The reason is that when running hosted agents it's a clean run each time.
Hope that helps! Let me know if you have other questions or concerns.

@stephenmichaelf Thanks for the link to the build machines, I'd missed that looking at the build agent page.
Basically I need to run pip to install some packages, then pyinstaller to compile the python - are these part of the standard python install or Anaconda?  Sorry for the dumb questions but I'm generally a C# guy and support the VSTS scripts and have never written any python.

@phatcher No problem!
If you use the Task named "Command Line" then you can do what you want. It's hard for me to say if it will fully work but you can try the following:

PowerShell Script Task

Add a file to your source code with the following content:
$output = $Env:BUILD_REPOSITORY_LOCALPATH + "\get-pip.py"

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
Invoke-WebRequest -Uri https://bootstrap.pypa.io/get-pip.py -OutFile $output

Then point to it with this PowerShell Script Task.

Install pip

python get-pip.py

Run any pip install commands you need

python -m pip install SQLAlchemy

Use pip to install pyinstaller:

python -m pip install pyinstaller

I just created this task and ran on the hosted 2017 agent and it worked:

After these steps are done you will have pip and pyinstaller installed and can use them as you need.

Going to close this, please reopen if you have further questions :)


Update the docstring of read_json and concat

Remove "pandas objects" from docstrings of read_json and concat.

While we're here, I think we can change Concatenate pandas objects -> Concatenate Koalas objects at concat.

@HyukjinKwon Thank you for reviewing! Updated.


arm-frc-linux-gnueabi-gcc: init at 4.9.4

Motivation for this change
This adds support for FIRST Robotics Competition compiling to nix by adding a cross compiler and all necessary packages. Normally the compiler will take a lot of work to build in the userspace, but with this package it is all automated and compatible with nix.
This package is based on the arch version of the package: https://aur.archlinux.org/packages/arm-frc-linux-gnueabi-gcc/
DOES NOT SUPPORT MACOS
Support for macOS can be found here: http://first.wpi.edu/FRC/roborio/toolchains/
Things done

[x] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[x] NixOS
[x] ~macOS~
[x] Linux

[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Travis build failed due to 4MB log limit >.<

yes, just keep the log as it is. Travis is only used here for pull requests.

Could a macOS person test this please? I don't have access to osx atm

macOS support removed because toolchain is available for OSX at http://first.wpi.edu/FRC/roborio/toolchains/
ar and tar were a bit of a pain here.
I think it is ready to be merged

@Mic92 It won't pass Travis due to log size but it is ready to be merged.

I rebased the patches here: https://github.com/Mic92/nixpkgs/commits/frc
If everything is still working for you, I would push this to master.

Is there a way we can do a compatibility alias for this? Not sure what the "equivalent" would be for arm-frc-linux-gnueabi-gcc

@matthewbauer Yes, we can do it like pkgsi686Linux. See https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L20-L28 and the use of nixpkgsFun to recur. Also see https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L5640-L5665 for the one for cross in particular. I was hoping to get rid of nixpkgsFun eventually, but I suppose the compat alias will only be temporary.

Hehe it's like exactly a year so maybe that competition is about to begin again anew!

Lol- I think FIRST starts in January though


update libgrape-lite's commit id in Flex dockerfile

What do these changes do?

Related issue number

Fixes #3015

It would be better if you can depend on a specific release/tag on hiactor/libgrape-lite.


rkt

It would be nice to include rkt in this docs as an alternative to docker

I might consider rkt in a future guide, but for now I'm going to stick with Docker.


Is there a 64bit version of the Utymap.Share library?

The one from the release doesn't work on my machine I think it's x86? Unity does not recognize it no matter what

I'm using only x64 version, so it is included in release artifacts. If you have problems, you can try to build your own version

Hmm, still I keep on getting in Unity:
Failed to load 'Assets/Plugins/UtyMap.Shared.dll' with error 'The specified module could not be found.
Even though as you can see the dll is at the proper place:

I tried building myself but run into boost cannot find certain libraries errors. I'm stuck

Shared lib on windows depends on zlib.dll, have you copied it too? You might get such error if dependency could not be found.


Checking release version of extractor tool

Hi Team,
Currently i have three project in which i am using extractor tool to extract API's from APIM.
Is there a way to check which release of Extractor tool i am using in those projects. As there are differences in features available in two projects. I tried looking for it but couldn't find it.
Thanks,
Vibhu Kaushik

currently there is no way to check which release the extractor tool is


Add ALB attributes annotation usage

Add usage and values for the attributes annotation
Credits to @masterzen who had added this to his PR but this feature had already been merged without this doc

@argeualcantara Would you mind signing the CNCF CLA?

Thanks!


web: redirects

For now, to be able to use this module on web I'm using universal_io, but it lacks support of http redirects (due to XHR).
We can use fetch API with manual redirects to get this functionality, but it requires either new dedicated package or find existing with such functionality.
The pitfall here is that we have to maintain ability to use custom HttpClient on native platforms.

We have migrated to http and have fetch_client, it still need some finish touches regarding redirect, to make it a drop in solution.

fetch_client is now mature enough, added readme note.


MINOR: update JavaDocs for Kafka Streams DSL helpers

also deprecate ZK config for Streams

@dguy @enothereska @guozhangwang

Updated this.

Merged to trunk and 0.10.2.


New StorageDriver for Google Drive

Currently we have storage drivers which are IaaS based like OpenStack, AWS etc., and it involves setting up the environment to use it. But most of the users use Google Drive like storage for storing the files.
So we are proposing a new storage driver using Google Drive as backend.

@manoj535 and @bharaththiruveedula worked on this and will send the PR for this issue.

Closing, as we are not considering adding any new storage driver to registry anytime soon. Feel free to create an implementation of the storage driver in a dedicated repo, however, as I said, unfortunately we don't have the bandwidth to maintain them. In the future we might make the drivers pluggable but that future is very far now.


Uploads to Amazon Cloud Drive restarting at 99%

Using rclone 1.32 on Windows Server 2012 R2
Connecting to Amazon Cloud Drive
Command used:rclone sync I:\MOVIES ACD:SERVERNAME\MOVIES --bwlimit 24M --max-size=50G --transfers=6 --exclude ".*/"
For the past ten days to two weeks, many of my uploads to Amazon hit 99% and then restart at 0%. When I run the command with the verbose command, I regularly see one of the following error messages (in order of frequency):
error Post https://content-na.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: write tcp <IP_ADDRESS>:52277->xx.xxx.xxx.xxx:443: wsasend: An existing connection was forcibly closed by the remote host
Received error: HTTP code 504: "504 GATEWAY_TIMEOUT", no response body - low level retry
(these low level retries sometimes get up to four or five before finally succeeding)
Received error: HTTP code 500: "500 Internal Server Error", reponse body: {"message":"Internal failure"}
Is there something happening on Amazon's end? I upped the transfer number to six in the hopes that more attempts would lead to more successful uploads and it seems to be working, but I am still seeing regular resets.

I think this is a problem at Amazon's end...
If you check out the developer forum you'll see various relevant threads (one by me!).

Yeah, I guess so. The problem has definitely gotten worse lately. Thanks for pointing me to the forum.

@ncw Looks like your thread was deleted. Amazon mentions a new API in their forums
https://forums.developer.amazon.com/questions/33186/the-new-amazon-drive-api-and-sdks-for-android-and-1.html
maybe this will fix the problem with files larger than 10 GB?

Here is the thread I missed the last character off the URL when pasting it!
I can't decide if it is a new API or just a wrap of the existing API for android & IOS but I sent off a request for an invite and I'll let you know!

I posted a potential fix for this in #606 if you fancy giving it a go.

I'll try this over the weekend and see how it works!

Sorry, but would it be possible to build a version for Windows x64? My home server runs Windows Server 2012 R2.

See #606

Note to self - maybe combine this fix with #559

Looks like the problems on Amazon's end are getting worse. From looking at the log, there have been several 504 errors mid-transfer, as opposed to at the end.
The patch seems to have mostly worked on the occasion where the error came at the end. It waited, then looked again and found the file with the correct size and indicated that it was returning with no error. However then it threw this error
'Received error: %!v(PANIC=runtime error: invalid memory address or nil pointer dereference)'
And started the file all over again.
I'll upload the entire log on Monday so that there's an entire weekend's worth of data to pore through, but as of this point I've uploaded 175 GB worth of data and managed to successfully complete one (1) file. Something is very wrong on Amazon's end.

I think there might be a bug in the patch.
Here's what happens after any file reaches 100% but then gets a timeout error:
Timeout error detected - waiting to see if object was uploaded correctly: HTTP code 504: "504 GATEWAY_TIMEOUT", no response body
Object found with correct size - returning with no error
pacer: Rate limited, sleeping for 150.761883ms (1 consecutive low level retries)
pacer: low level retry 1/1 (error <nil>)
Received error: %!v(PANIC=runtime error: invalid memory address or nil pointer dereference) - low level retry 2/10
Removing failed copy
Resetting sleep to minimum 20ms on success

I think it's seeing that it's uploaded correctly and is the correct size, but then it throws the PANIC=runtime error and decides it's a bad copy, deletes it, and starts over from the beginning again.

Here's the full log from this weekend's attempt.
test.txt

Very useful log, thanks.
I can see that if I hadn't made a mistake this would have worked 15 times and not worked 23 times which is not bad.
The time for a file to appear after the TIMEOUT error was from 0-15 seconds which is useful information.
At 2016/08/06 04:04:3 something interesting happened.  Your token expired and at the same time all 6 of your uploads died, none of them near 100%.  Then for 10 minutes or so ACD timedout all your uploads - in fact most of the errors seemed to happen about here.
I see what happened there - because all 6 of the upload threads were busy, the token didn't get renewed in a timely fashion. The previous token renewal was at 2016/08/06 01:55:51 so 2 hours and 10 minutes before. The tokens normally only last for one hour.
I'll have a go at fixing that too and that should make things much more reliable.
I've fixed the mistake in the original patch (hopefully!) - here is a new one with the fix for the token timeout also (not very well tested!).
http://pub.rclone.org/v1.32-18-gaffc483β/

OK here is another attempt, a bit better tested:
http://pub.rclone.org/v1.32-18-gaffc483β/

I'll give this latest build a spin throughout the week and let you know the results!

With correct URL this time: http://pub.rclone.org/v1.32-18-g29ab7eaβ/

I've reworked the patch again with the following changes.  On a failed upload we

detect if we uploaded the file to 100% - if we didn't do nothing
do this for any upload error (so might get round the 500 error)
increase timeout to 2 minutes - timeout might need to vary with file size?
made timeout configurable with --acd-upload-wait-time - perhaps set it large when uploading huge files?

Here is the beta: http://pub.rclone.org/v1.32-18-gf7562dcβ/

Hi, what do you mean by 'large', I have ~25G file, what should I specify? :)

Sorry missed your message - you could try --acd-upload-wait-time 10m to wait 10 minutes. I don't know the optimum size though.

I've merged this to master and it will be in the v1.33 release - thanks for your help testing.
Please re-open if you see any problems.
Here is a beta with the latest code in: http://pub.rclone.org/v1.32-20-g2ebeed6β/

Thanks again for addressing this issue so promptly and effectively!


Own training results in good accuracy on test set but poor results on real data compared to TotalSegmentator

Hello,
I followed step by step your guide on how to train our own model and I used your public dataset. I have a decent accuracy on the test set (around 92% of dice) but once I feed a real life example into the model which is a scanner of the whole body, the results are quite poor compared to what TotalSegmentator gives when we use it. I thought it was only because the scan was too big so I split it into subparts but even with it, the results are nowhere close to those from TotalSegmentator.
Is there anything I could have missed to reproduce a worse version of yours but still decent on a real life example ?
Thank you in advance for your response.
Nils Cercariolo

I found the issue, it's because I didn't apply the as_closest_canonical function from nibabel and therefore, the results were not very good.


Feat: empty theme

Requested feature
There should be a clean way to set up an "empty" theme, i.e. the one that defines all the segment-specific highlight groups, but links them from StatusLine and StatusLineNC by default and does not override the colors when a custom colorscheme is loaded.
Motivation
I am working on a custom theme (yet another solarized port) and I would like to support lualine out of the box. However, the way lualine is currently written, it overrides all the highlight groups defined by the colorscheme by the colors from the current lualine theme on every theme load and every background change.
I find this intrusive and I would prefer if there was a way to disable this behavior. Something like an empty theme, that defines all the highlight groups that link to StatusLine and StatusLineNC and that does not overwrite the colors defined by the user-selected colorscheme. I find this approach much more convenient than the currently suggested way of adding your own custom theme to lualine (even through the options).

Statusline & StatuslineNC is just just two hl groups . I can't see how I'll map all the colors lualine uses to just these two.

currently suggested way of adding your own custom theme to lualine

Currently suggested way is actually not to add theme here . Rather have them with colorscheme for example like other solarized colorscheme here .
If you want to you just use a dummy theme file that just maps lualines colors to highlight groups and define those hl groups as part of your colorscheme
For example:
-- file: lua/lualine/themes/solarized.lua
return {
      normal = { a = 'statusOuter', b = 'statusMiddle', c = 'statusInner' },
      insert = { a = 'statusInsert' },
      visual = { a = 'statusVisual' },
      replace = { a = 'statusReplace' },
      inactive = { a = 'statusInactive', b = 'statusInactive', c = 'statusInactive' },
}

Then define statusOuter, statusMiddle, statusInner, statusInsert,.... highlight groups as part of your colorscheme.


Create AudioEffectEQCustom; docs included

An audio equalizer with customizable band frequencies is added and the documentation has been updated. Closes #27585.
I believe this cannot be worked around by a few lines of code and there are plenty of use cases (a parametric EQ can make up a large part of audio effects), so I made it.
Other changes (none should introduce compatibility breakage):

Existing preset equalizers (6, 10, 21 bands) are made subclasses of a new class AudioEffectEQPreset which inherits AudioEffectEQ
Prevented AudioEffectEQ from being created directly on the bus
Minor changes in logic of class EQ

I tried my best to comply with existing conventions, but in case anything is to be changed or discussed please kindly specify! ^ ^

@kawa-yoiko Is this still desired? If so, it needs to be rebased on the latest master branch. Also, the above feedback should be addressed. Also, since this PR is a feature proposal, you should consider opening a proposal which explains example use cases and how this approach will solve the problem.
Otherwise, abandoned pull requests will be closed in the future as announced here.

@aaronfranke Thanks, I will open a proposal and rebase afterwards. Do I force push here or open a new PR?

@kawa-yoiko You force-push here.

Hey, is this a set customizable of parametric filters (freq, gain, Q)?
Are there shelveing filters available too?
Are there lowpass/highpass filters available as well?
I think the description could use some work, maybe I can help?

Hi @unfa, thanks for your interest! If you are interested in improving the documentation please kindly point out where text can be improved (did I understand correctly?). But as godotengine/godot-proposals#935 has received no feedback, I don't have a clear idea on opinions from the Godot team and community. May I slightly disturb @aaronfranke for a clarification on what to do next?
For details, it's actually semi-parametric (tunable frequency and gain, with fixed Q). The equalizer algorithm is implemented in eq.h. Shelving/high-pass/low-pass filters are implemented by variants of AudioEffectFilter, unrelated to this change.

At the moment, we don't have a formal approval process for the proposals repo. However, I can tell you that because https://github.com/godotengine/godot-proposals/issues/935 has received no feedback from users, this PR is unlikely to be accepted. #27585 shows that you are not the only person interested, but it's not enough. If you want this feature, feel free to share a link to the proposal on social media and have people come in and give feedback (in the form of a comment or just :+1: and :-1:).
In the meantime, I'll mark this PR as a draft because it's not in a mergeable state (in terms of needing feedback from users, needing approval from the Godot devs, and needing a rebase).

@kawa-yoiko @unfa I saw this PR was stuck
Are we still interested in this feature?

Massive yes!

@aaronp64 I suppose the current obstacle is to collect opinions from the community (since godotengine/godot-proposals#935 has received only a few votes and no comment, so the general interest remains unclear). I'm not active on social media, but I think the forum is a good option. I (or someone else) can make a post to introduce this proposal.
I'd be very glad to help push this forward (rebasing shouldn't be much of an issue) or have someone interested base their work on this (as I might be occupied in the next couple of weeks).

@adamscott Is this feature enhancement desired or not?


good job

I was about to write a rough version of it myself, and there were other people working on projects

Thanks! I'm going to close this issue as it doesn't really describe a possible improvement. Please don't hesitate to get involved. We're in discord at #langchain-go


Download Latest arbitrum data from Dune

Describe the feature you'd like to request
The process is not currently automated. We will need to download the latest data and load into the warehouse.
Describe the solution you'd like
Download the latest dune data and load it into the data warehouse
Describe alternatives you've considered
N/A. This is just a manual task until we have an alternative data source for blockchain data.

Finished!


updated references to msysgit and GitHub for Windows

These projects have been renamed in recent times, so the locations and
details are slightly different but the instructions remain largely the
same.

Hi @shiftkey, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

This seems like a small (but important) contribution, so no contribution license agreement is required at this point. Real humans will now evaluate your PR.

TTYL, AZPRBOT;

Thanks for this contribution - sending to the author for review.

Resent request for review to @szarkos

Sorry for the delay (traveling).  This looks good to me, thanks for updating those links!


Bucket API getFiles returns huge amount of unnecessary data

Fetching a list of files from my bucket produces 271K of JSON output from the server. And I just have 4 files in the bucket.
The data includes things like the complete description of the bucket itself (per every file) or the description of the googlecloud storage library (authors, dependencies, etc). Some of this is probably not loaded from the server but is added by the node api library. Even if that doesn't lead to network traffic it's still unnecessary and inconvenient when I try to find the relevant data. The screenshot below show the data structure with most elements collapsed. None of those elements are relevant for listing files in the bucket. I only want to see the data that is printed by gsutil when doing ls.
https://www.dropbox.com/s/4dgpu7jh8z6lb9h/Screen%20Shot%202017-04-13%20at%2013.29.58.png?dl=0
I might be missing some option or parameter that would limit the output from getFiles.
Environment details

OS: OSX
Node.js version: 7.2.0
npm version: 3.10.9
google-cloud-node version: 1.0.0

Steps to reproduce
var storage = require('@google-cloud/storage');
var fs = require('fs');

var gcs = storage({
    projectId: 'MYPROJECT'
});

var bucket = gcs.bucket('MYBUCKET');

bucket.getFiles(function(err, files) {
    if (!err) {
        fs.writeFileSync("files.json", JSON.stringify(files))
    } else {
        console.error(err)
    }
});

There is a lot of stuff on the object-- some is plumbing that enables our API to function. The only thing that's really meant for the user is the metadata property, which is the API's raw record of the resource.
In the context of getFiles(), the API response returns all of the metadata for each file. If we didn't include it on the object, it would require further API calls to download that information. To find the data you need, you should look into to the metadata object. In the case that you just want names, it's a simple map over the response:
bucket.getFiles(function(err, files) {
    if (!err) {
        var fileMetadataObjects = files.map(function(file) {
            return file.metadata
        })
        fs.writeFileSync("files.json", JSON.stringify(fileMetadataObjects))
    } else {
        console.error(err)
    }
})

And of course, if you just want the name, you can use file.metadata.name or any other properties that are relevant for your use.

Just to address the 271,000 lines of JSON, though, I'm not particularly thrilled about that. From a quick test, the output of one file for me was about 1,200 lines. I'll re-open so we can define those huge objects as non-enumerable, so they're hidden from the output.

Ok so do I understand correctly that only the metadata is actually downloaded from the server?

Yes, everything else is state and internal data.

Cool, then it was just me panicking :)
Still would be user friendlier to not have that metadata in the request I think.

This issue was moved to googleapis/nodejs-storage#19

This would be a change in @google-cloud/common that would affect all of the modules, so I'll re-open this issue here.

This issue was moved to https://github.com/googleapis/nodejs-common/issues/13.


233 spatyper tool on galaxy

Wrapper for the simplest use case of spaTyper. Takes a query .fasta file (SPA region of staphylococcus aureus) + a set of defined repeats for that region (typically downloaded) and outputs text similar to the following.
Start the identification of repeats in Spa protein:
+ Repeats fasta file provided:  ~/galaxy_tools/tools/spaTyper/test-data/sparepeats.fasta
+ Check or download repeats types file in folder:  ~/planemo/bin
A previous download created spatypes.txt on: 2023-03-15 17:43:58

-----------------
Results:
Sequence name Repeats Type
AJ829762.1 Staphylococcus aureus subsp. aureus partial spa gene for protein A 99-20-69-31-70-13-17-16-16-16-16 t11215
-----------------

Currently passing all tests (input query fasta and repeats fasta call type t11215 based on text search)

Aha! Thanks, just got back to this

Hmmm - well it looks like it failed CI, but it works locally (sharing my local test-log)
planemo_spaTyper_test_2023-03-16.log


CLOUD-47073 Not usable service links are removed, Ranger web access l…

…ink added
@martonsereg pls review

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/cloudbreak-pull-request/1757/


Record market price data in settings cache

Write market, price and timestamp data to settings json. Migrate to another storage mechanism for this data when it becomes too slow. Related to issue #12

This is a duplicate of #7 ...


Add clear_children

Clearing the children of an element implies looping over them in reverse order, add a method to clear them that takes care of it.

Added in next release


Missing put / delay imports in README examples

In the README, put and delay are missing from import. Is this intentional? Being new to this library, I was immediately thrown off, and went searching where they came from..
import { take, call } from 'redux-saga'
// sagas/index.js
function* incrementAsync() {

  while(true) {

    // wait for each INCREMENT_ASYNC action
    const nextAction = yield take(INCREMENT_ASYNC)

    // call delay : Number -> Promise
    yield delay(1000)

    // dispatch INCREMENT_COUNTER
    yield put( increment() )
  }

}

export default [incrementAsync]

I see that delay is just an example function, but put is part of the API. I'll whip up a PR to adjust the readme


Multi column Filter with dynamic data

What feature are you proposing?
Is it possible can we do the multi column filter based on the dynamic table data available on one column filter.
To make it clear kindly refer this library which have that functionality.
https://github.com/cheekujha/react-table-filter
Thanks,
Gnanavel N

This feature/issue has been tagged as one that will likely be fixed or improved in the next major release of React-Table. This release is expected to be completed in the next 6 - 8 months. Please see #1143 for more detail. If you believe this feature can be easily fixed, we invite you to fork the project, attempt the fix, and use it until the latest version is available. If the fix is simple enough, we will also consider it as a PR to the current version.


SW_ Constants type-error

I was using the windows_sys crate, version "0.45.0".
While looking at the constants associated with the Win32 ShowWindow function, I noticed what seems to be an error in the type annotations.
All the ShowWindow constants (such as SW_HIDE) start with the SW_ prefix and have the type SHOW_WINDOW_CMD.
However, there other constants (such as SW_ERASE) which have nothing to do with ShowWindow, and instead are associated with ScrollWindowEx. These constants still have the type SHOW_WINDOW_CMD, which seems to be an error.
There are also constants (such as SW_PARENTCLOSING) which, while related to ShowWindow, are not commands to be passed to the ShowWindow function as arguments, but possible values received in the lParam of the WM_SHOWWINDOW message. Whether or not it is an error that these constants also have the type SHOW_WINDOW_CMD, I am unsure of.
Here are some links to the official documentation:
ShowWindow: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-showwindow
ScrollWindowEx: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-scrollwindowex
WM_SHOWWINDOW: https://learn.microsoft.com/en-us/windows/win32/winmsg/wm-showwindow
SW_HIDE: https://microsoft.github.io/windows-docs-rs/doc/windows/Win32/UI/WindowsAndMessaging/constant.SW_HIDE.html
SW_ERASE: https://microsoft.github.io/windows-docs-rs/doc/windows/Win32/UI/WindowsAndMessaging/constant.SW_ERASE.html
SW_PARENTCLOSING: https://microsoft.github.io/windows-docs-rs/doc/windows/Win32/UI/WindowsAndMessaging/constant.SW_PARENTCLOSING.html
It is possible that this is a widespread issue to other unrelated constants that share the same prefix.
These are just the ones that I was able to find.
Are these errors, or am I misunderstanding something?

The windows and windows-sys crates are generated from metadata describing the Windows API. I'll transfer this issue to the Win32 metadata repo for verification.