Lag on refocus when using external monitor

Hyprland Version
Hyprland, built from branch main at commit 1b99a69dc11c3c8266559c3c20b2f4dac6621dcc dirty ([gha] Nix: update wlroots).
Bug or Regression?
Bug
Description
When I am using an external monitor on my laptop (using USB-C wired to iGPU, as HDMI wired to dGPU still does not work on Hyprland), the mouse becomes really laggy for 1-2s every time I change which window is focused (only affects the laptop monitor). This effect is very minimal but does appear to happen on the external monitor. It does not happen at all when the external monitor is not connected.
How to reproduce

Plug an external monitor through USB-C into a laptop, wired to the iGPU
Enable follow_mouse=1 for triggering this easier
Open two windows side by side on the laptop built-in monitor
Move the mouse between them
Observe insanely choppy mouse for 1-2s after each change in focus, and then it becomes smooth.

Crash reports, logs, images, videos
I cannot record a video due to this. Here's an unhelpful log though:
[LOG] Set keyboard focus to surface 55aa61a65280, with [Window 55aa61a6a1a0: title: "nheko"]
[LOG] Set keyboard focus to surface 55aa618e8e00, with [Window 55aa61a62650: title: "DocViewer — Mozilla Firefox"]

Above is printed when I'm clicking on nheko, observing the choppy mouse, then clicking on Firefox and seeing the same behavior (both are on laptop display).
If there is anything I can provide please let me know. This issue has been happening on and off for the duration of me using Hyprland, but it's gotten a lot worse recently to the point of being pretty much unable to use my laptop screen when docked.
Related to #2669

This can sometimes happen without needing a refocus, but the lag does get worse when you refocus. Moving the mouse on the laptop screen in general is insanely laggy.

It appears to have gotten worse over time. This used to be a less pronounced effect, but the lag has gotten worse recently. Is there anything I can do to help get this fixed?

are you >= 0.32.3?

Is there anything I can do to help get this fixed?

Yea, profile with tracy. I'd do that myself but recently tracy has stopped cooperating with me.

Notes from Discord:
Yes I am on Hyprland from the latest git.
This is reproducible by having an external monitor plugged in, and causing a tiled client to grow in size on the laptop monitor. I do this by:

Open two foot windows side-by-side
Close one
Observe lag being triggered for the rest of the duration of the Hyprland session

vaxry helped me set up tracy on my machine, and I found that several lines in CAnimationManager::scheduleTick() and CAnimationManager::wlTick() have 100% cache misses, and sometimes cause the function to take 100-600ms to execute. It looks like it may be the cause of the lag, but the execution time makes no sense since those lines are super cheap.
The only way to get tracy to detect the Hyprland it is running inside of, a nested session must be started and exited.
Some images:

There are no other Hyprland functions that even show up in the trace (is tracy set up correctly at all? would explain the bizarre data collected). I'm not sure whether publicly sharing a trace is ok, but I have it saved in case it's helpful and can provide it on request.
I seem to be the only person affected by this. If @foi or @EinarArnason or anyone else does encounter this, please comment here with your system setup! Else, it seems this will not be fixable.

I'm running into this on a Lenovo P1 Gen3 laptop + thunderbolt 4 dock using hybrid graphics. This is only an issue on intel+nvidia graphics, if I use only nvidia then it's not an issue. When the lag happens the load on the GPU and the CPU go up to 30%.

Me (Nvidia GTX 1060) and a friend (Intel IGPU) are both running into this issue with the added oddity that the entire display buffer will sometimes flicker a black screen with garbled pixels resembling the desktop for about a frame. I would also like to add that it is more visible on a high refresh rate monitor. All of it gets much worse at high CPU usage and this didn't happen before I updated.
Here is a video about the frame buffer issue:
https://github.com/hyprwm/Hyprland/assets/44762123/2152f046-28af-4f0b-acf8-485ded559bbc
I had to record it at 1/8th the speed so it be visible in reality it a very quick flash

I'm experiencing something similar on GTX3090 with a single monitor.
When I switch window focus my fps drops (usually to around 50 or so according to the UFO test https://www.testufo.com/) for under a second. If both windows (old-focus and new-focus) are terminals there doesn't appear to be any drop.
I'm on 535 driver, but tried 545 and the experimental vulkan driver.
For reference, I do not get these fps drops on sway - at least with WLR_RENDERER=VULKAN and XWAYLAND_NO_GLAMOR=1. When I switch focus in sway I get visual artifacts and slight flickering instead. Maybe a coincidence, but I thought it was interesting.

Hmm, I'm not sure whether it's the same bug as, in my case, the running apps are still smoothly running, only the cursor itself lags. It's likely related though. Are you using hardware cursors?

Ah ok. No, I'm using a software cursor. My fps drop seems to be more systemic and the mouse lag is just one consequence.

Well still, if refocusing causes it, I think it is probably related. Something goes really wrong in Hyprland when you change focus, only for a few users (including me, maybe you, and everyone else who posted here).
Maybe someone can get a valid profile with Tracy? I have documented how I got Tracy to cooperate, but could not get any useful data myself.

I think I actually just managed to get rid of the issue I described. I did three things:

Video card was running in x4 mode, so I moved it to a slot where it can run in x16
Ram was running at 2333mhz or so, which I changed to 3666
I enabled CONFIG_ACPI_IPMI in my kernel. This option is mentioned on the Gentoo wiki page for nvidia drivers and I somehow overlooked it.

If someone thinks it would be useful for me to figure out which of these changes did it I can, but if your issue is hyprland-related then this might not be helpful.
I still seem to get significant fps drops when changing workspaces, however, and when opening up a bunch of new gui windows in quick succession. The latter might be normal, not sure. These drops seem to happen regardless of whether I have animations enabled or not.

Same thing gets triggered on my system (AMD+AMD laptop with hybrid graphics) when starting some particular games. It starts happening when I start Disco Elysium or Magicraft and it only stops after a full restart.

Have you tried setting a minimum gpu clock speed?
I'm actually still facing this issue but less so than before due to the changes I made. It turns out I had powersaving stuff turned off (Powermizer options mentioned in the wiki) when I made my post 2 days ago which improved performance a lot at the cost of ~100W increase in power draw.
Today I set a minimum gpu clock speed of 1000hz fps drops during window focus change they are pretty minimal now. I still detect a few dropped during workspace transition animations but I'll take the improvement.
I also have run hyprland on a 9th gen Intel laptop. It's unusably laggy by default, but if I max gpu clock speed it becomes nice and smooth. I'm not sure how related the experience with my intel and nvidia gpus are, but in both cases it seems the clocks can go fast enough to run hyprland well, but don't for whatever reason.

I'm jumping the gun a bit here (as we don't know if maxing the GPU clocks helps OP), but maybe OP experiences this problem only when they have an external monitor connected because of the additional load the extra display adds to the relatively weak iGPU. My laptop display resolution is 4k, but if I drop down to 1080p the experience is much more smooth. And from hanging the discord lately my feeling is that people on intel iGPU with >1080p displays are disproportionately represented in the group of lag complainers (though this is super informal). This group has weak gpus and heavy loads, so it would make sense.
If the same problem is indeed causing fps drops on strong nvidia gpus then that is a bit more mysterious. Maybe it's due to generally bad wayland support by nvidia? Should add that my desktop also has a 4k display, but when I drop down to 1080p it's a lot smoother.
I've done more testing in sway, and I do get much better performance there, though it's not perfect, and in fact on my laptop I drop frames quite often as well--unless I set the gpu clocks high, in which case it's super smooth. This too makes sense due to sway's lighter load: Moving the cursor around within a window and changing window focus both use significantly fewer resources.

I ran a quick survey with Hyprland users trying to evaluate the hunch I sketch above, and just thought I should share the results here. It's got some sample size issues (Only 9 and 8 respondents out of 43 reported having animation lag and mouse lag, respectively), but I feel like it was enough to answer my basic questions.
https://rpubs.com/gentoobtw/hypr
Regarding the issue of mouse lag when refocusing windows, it looks to mostly a problem for NVIDIA users. Incidentally I tried out KDE Plasma 6 the other night (on my desktop w NVIDIA GPU) and found just about exactly the same performance issues: mouse lags when switching window focus, especially when inactive for a bit beforehand, and similar animation lag issues.
Regarding the animation lag issue I mention in the above post (which is less relevant to the issue in OP), it looks like things get worse the more pixels you have to render. People with 1080p displays get smooth animations almost no matter their GPU, and around the 2x1440p mark animation lags starts to get more common.
Anyway next I plan on trying Gnome, which has triple-buffering, on my laptop and desktop.

My laptop has TigerLake-H GT1 (primary gpu) and GTX 3070 GPUs with 2560x1400@165 screen. It seems I have solved the fps drops issue by setting intel GPU frequency manually: sudo intel_gpu_frequency --set 1400, sudo intel_gpu_frequency --max 1450

If someone thinks it would be useful for me to figure out which of these changes did it I can, but if your issue is strictly hyprland-caused then this might not be helpful.

@espro1 but does it happen on sway? I had no issues like this on sway with a x4 eGPU

In my case I get cursor and apps lagging 1-2s when I move the cursor between monitors (and windows due to this). I have an AMD RX 590 eGPU via Thunderbolt 4 cable. Did not have this issue on sway. My system is hybrid (Intel iGPU and AMD eGPU).
2 external 4K displays, laptop monitor disabled. 1.5 scaling for both monitors.

Same here. Framework 16 with dedicated AMD graphics and an external 3440x1440@144 monitor. Scaling 1.6 on Laptop Screen, no scaling on external. If I move mouse cursor from one display to another whole system is kinda freezing for a half a second or so. You can also hear music stuttering.
hyprland-git on current Arch:
Hyprland, built from branch main at commit 8f9887b0c9443d6c2559feeec411daecb9780a97  (Nix: remove unused dependencies).
Date: Mon Sep 2 22:48:58 2024
Tag: v0.42.0-112-g8f9887b0, commits: 5181

flags: (if any)

This issue can be observed more easily by running a cpu-heavy process like compiling (the linux kernel) or video encoding. Smooth mouse movement when staying on the same application, choppy and delayed mouse movement when crossing a window border. I also tested this with a single monitor and the issue was far less pronounced. Must use multiple monitors (not necessarily laptop) to reproduce, possibly more is worse.

I'm seeing this problem when using two 4k monitors (60hz and 240hz):

Framework Laptop w/ i5-1240P (docked, iGPU)
Desktop w/ 7950x3d using iGPU

Interestingly enough if I use my 7900xtx with my desktop I don't see the same stutter when switching between windows. It may still be there but not to the degree that I see with the iGPUs. I also don't really notice it when using the framework without external monitors and just the built in screen.

I'm experiencing the same issue. I'm using an ASUS ROG Strix 16 laptop with an RTX 3070 and Ryzen 5900HX. My main monitor (1440p) runs correctly at 240Hz, and the laptop's internal display works fine at 165Hz. However, my third screen drops to 13 FPS even with just VSCode open and nothing else running.


Call dropped by load balancing policy

What version of gRPC and what language are you using?
gRPC 1.6.0 installed by pip
What operating system (Linux, Windows, …) and version?
CentOS Linux release 7.3.1611 (Core)
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.4.5 (default, Nov  9 2016, 16:24:59)
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
What did you do?
I have a series of py.test suites that check GRPC API of a particular service. Channel and client for a service is periodically recreated in a such manner:
class DummyGRPCChannel(object):
    def __init__(self, host, port, root_certificates=None, private_key=None, certificate_chain=None, logger=logging):
        self.host = host
        self.port = port
        self.endpoint = join_host_port(self.host, self.port)
        self.credentials = grpc.ssl_channel_credentials(root_certificates, private_key, certificate_chain)
        self.channel = self._connect(logger=logger)
        self.stubs = dict()

    def _create_channel(self):
        raise Exception('Not implemented')

    def _connect(self, logger=logging):
        logger.info('Trying to connect to {}'.format(self.endpoint))
        with step('[PRECONDITION] Waiting for grpc channel at {}'.format(self.endpoint)):
            wait_for_connection(host=self.host, port=self.port)
        return self._create_channel()

    def create_stub(self, stub_cls):
        _stub = stub_cls(self.channel)
        _stub_name = stub_cls.__name__
        if _stub_name not in self.stubs:
            self.stubs[_stub_name] = list()
        self.stubs[_stub_name].append(_stub)
        return _stub

class GRPCChannel(DummyGRPCChannel):
    def _create_channel(self):
        channel = grpc.insecure_channel(
            self.endpoint,
            options=[('grpc.max_message_length', Constants.MAX_MESSAGE_LEN), ('grpc.min_reconnect_backoff_ms', 100)])
        return channel

class DummyEndpoint(object):
    def __init__(self, host, port, use_tls=False, tls_creds=None):
        self.host = host
        self.port = port
        self._channel = None
        self.clients = dict()
        self.use_tls = use_tls
        self.tls_creds = tls_creds

class Endpoint(DummyEndpoint):
    def _connect(self, logger=logging):
        if self.use_tls:
            self._channel = SSLGRPCChannel(self.host, self.port, logger=logger, **self.tls_creds)
        else:
            self._channel = GRPCChannel(self.host, self.port, logger=logger)
    # Method for create client
    def create_client(self, stub, name=None, logger=logging):
        if self._channel is None:
            self._connect(logger=logger)
        name = name if name is not None else rand_str()
        with step('Creating {} client'.format(stub.__name__)):
            _client = self._channel.create_stub(stub)
            self.clients[name] = _client
            return _client

Using:
from component import component_pb2

service = Endpoint(HOST, PORT, False)
client = service.create_client(component_pb2.ComponentStub, name='component', logger=logger)
client.DoSomething(component_pb2.DoSomethingRequest())

Requests to the service are performed in a default manner (according to the grpcio documentation). There are a lot of tests within a single testing session, and after a while, requests start to fail with a strange error: "Call dropped by load balancing policy". This problem is floating, so the moment when error start to occur is changes from time to time.
What did you expect to see?
Successful response from the server
What did you see instead?
state = <grpc._channel._RPCState object at 0x7fd982bfbda0>, call = <grpc._cython.cygrpc.Call object at 0x7fd982bfb9e8>, with_call = False, deadline = None
    def _end_unary_response_blocking(state, call, with_call, deadline):
        if state.code is grpc.StatusCode.OK:
            if with_call:
                rendezvous = _Rendezvous(state, call, None, deadline)
                return state.response, rendezvous
            else:
                return state.response
        else:
>           raise _Rendezvous(state, None, None, deadline)
E           grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Call dropped by load balancing policy)>
/usr/lib64/python3.4/site-packages/grpc/_channel.py:440: _Rendezvous

The thing that confuses me a lot is that every GRPC_ERROR_NONE is treated as load balancing error in GRPC core:
    calld->error = error == GRPC_ERROR_NONE
                       ? GRPC_ERROR_CREATE_FROM_STATIC_STRING(
                             "Call dropped by load balancing policy")
                       : GRPC_ERROR_CREATE_REFERENCING_FROM_STATIC_STRING(
                             "Failed to create subchannel", &error, 1);

See https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/client_channel.c#L1027

I had the same issue; downgrading to 1.4.0 fixed it for me.

I had the same issue. Downgrading to 1.3.9 is working for me. (for npm).

Having the same issue doing automated test cases in node.js.  My set-up and tear-down methods rapidly create and destroy same-process client/server instances and eventually at some point I get this error.  Same as authury147: downgrading to 1.3.9 (npm) resolves this issue.  This solution does not make me feel comfortable and it's unclear how I could remove the restrictive policy for automated tests.

See the same issue running automated tests in python

I am getting the same issue. Not sure if its related to #11015

I am getting the same issue on my unit tests in nodejs.

Since it hasn't been said, seeing this in the Ruby gem v1.6.2

Since it hasn't been said, seeing this in the Ruby gem v1.6.2
Same issue here, downgraded to 1.4.5

Can someone who's affected by this please try running with #12850 and let me know if it solves the problem?  Thanks!

Applied the change from #12850, and still getting the issue. Waiting to hear from others too.

Hmm, okay, then there must be something else going on here.  Can you run with the following environment variables and share the log?
GRPC_TRACE=client_channel,pick_first,round_robin,glb
GRPC_VERBOSITY=DEBUG

Thanks!

Also getting this issue. We are using gRPC 1.6.6.

@markdroth
Please find the logs
 D1010 10:46:19.951687754   12280 pick_first.c:469]           Pick First 0x3292dd0 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 4;  D1010 10:46:19.951719419   12280 client_channel.c:262]       chand=0x342da10: lb_policy=0x3292dd0 state changed to TRANSIENT_FAILURE D1010 10:46:19.951725386   12280 client_channel.c:248]       chand=0x342da10: setting connectivity state to TRANSIENT_FAILURE D1010 10:46:19.951730104   12280 client_channel.c:262]       chand=0x342da10: lb_policy=0x3292dd0 state changed to SHUTDOWN D1010 10:46:19.951741121   12280 client_channel.c:248]       chand=0x342da10: setting connectivity state to TRANSIENT_FAILURE D1010 10:46:19.951747539   12280 pick_first.c:100]           Pick First 0x3292dd0 destroyed. D1010 10:46:19.952038752   12280 client_channel.c:1421]      chand=0x342da10 calld=0x33cadd8: entering combiner D1010 10:46:19.952044039   12280 client_channel.c:1108]      chand=0x342da10 calld=0x33cadd8: deferring pick pending resolver result D1010 10:46:19.955187366   12280 client_channel.c:370]       chand=0x342da10: got resolver result: error="No Error" D1010 10:46:19.955207306   12280 pick_first.c:687]           Pick First 0x32e0720 created. I1010 10:46:19.955212976   12280 pick_first.c:335]           Pick First 0x32e0720 received update with 2 addresses I1010 10:46:19.955230865   12280 pick_first.c:416]           Pick First 0x32e0720 created subchannel 0x32c0a80 for address uri ipv4:<IP_ADDRESS>:50057 I1010 10:46:19.955239873   12280 pick_first.c:416]           Pick First 0x32e0720 created subchannel 0x33316b0 for address uri ipv4:<IP_ADDRESS>:50057 D1010 10:46:19.955248052   12280 client_channel.c:477]       chand=0x342da10: resolver result: lb_policy_name="pick_first", service_config="(null)" D1010 10:46:19.955252948   12280 client_channel.c:559]       chand=0x342da10: initializing new LB policy D1010 10:46:19.955258047   12280 client_channel.c:248]       chand=0x342da10: setting connectivity state to IDLE D1010 10:46:19.955262544   12280 client_channel.c:1092]      chand=0x342da10 calld=0x33cadd8: resolver returned, doing pick D1010 10:46:19.955266605   12280 client_channel.c:940]       chand=0x342da10 calld=0x33cadd8: applying service config to call D1010 10:46:19.955270317   12280 client_channel.c:1178]      chand=0x342da10 calld=0x33cadd8: starting pick on lb_policy=0x32e0720 D1010 10:46:19.955276330   12280 pick_first.c:469]           Pick First 0x32e0720 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 4;  D1010 10:46:19.955281710   12280 client_channel.c:262]       chand=0x342da10: lb_policy=0x32e0720 state changed to TRANSIENT_FAILURE D1010 10:46:19.955285334   12280 client_channel.c:248]       chand=0x342da10: setting connectivity state to TRANSIENT_FAILURE D1010 10:46:19.955296491   12280 client_channel.c:1160]      chand=0x342da10 calld=0x33cadd8: pick completed asynchronously D1010 10:46:19.955307747   12280 client_channel.c:1015]      chand=0x342da10 calld=0x33cadd8: failed to create subchannel: error={"created":"@1507657579.955300672","description":"Call dropped by load balancing policy","file":"../src/core/ext/filters/client_channel/client_channel.c","file_line":1011} D1010 10:46:19.955312154   12280 client_channel.c:897]       chand=0x342da10 calld=0x33cadd8: failing 1 pending batches: {"created":"@1507657579.955300672","description":"Call dropped by load balancing policy","file":"../src/core/ext/filters/client_channel/client_channel.c","file_line":1011} I1010 10:46:19.955320963   12280 client_channel.c:1383]      OP[client-channel:0x33cadb8]:  CANCEL:{"created":"@1507657579.955300672","description":"Call dropped by load balancing policy","file":"../src/core/ext/filters/client_channel/client_channel.c","file_line":1011} D1010 10:46:19.955326324   12280 client_channel.c:1403]      chand=0x342da10 calld=0x33cadd8: failing batch with error: {"created":"@1507657579.955300672","description":"Call dropped by load balancing policy","file":"../src/core/ext/filters/client_channel/client_channel.c","file_line":1011} D1010 10:46:19.955334960   12280 client_channel.c:262]       chand=0x342da10: lb_policy=0x32e0720 state changed to SHUTDOWN D1010 10:46:19.955423580   12280 client_channel.c:248]       chand=0x342da10: setting connectivity state to TRANSIENT_FAILURE D1010 10:46:19.955436127   12280 pick_first.c:100]           Pick First 0x32e0720 destroyed.

Any chance we have a rough estimate on this fix? We are currently being blocked by this issue (A new feature we need is only in gRPC 1.6.6 yet this bug broke half of our tests). An ETA would be appreciated so that we can decide whether to wait for the fix or try to look for some workaround.

In case it might be helpful, we're seeing this error in the gRPC ruby gem 1.6.6:
GRPC::Unknown: 2:Call dropped by load balancing policy
~/.rvm/gems/ruby-2.1.10/gems/grpc-1.6.6-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:31:in `check_status'
~/.rvm/gems/ruby-2.1.10/gems/grpc-1.6.6-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:171:in `attach_status_results_and_complete_call'
~/.rvm/gems/ruby-2.1.10/gems/grpc-1.6.6-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:363:in `request_response'
~/.rvm/gems/ruby-2.1.10/gems/grpc-1.6.6-x86_64-linux/src/ruby/lib/grpc/generic/client_stub.rb:152:in `request_response'
~/.rvm/gems/ruby-2.1.10/gems/grpc-1.6.6-x86_64-linux/src/ruby/lib/grpc/generic/service.rb:170:in `block (3 levels) in rpc_stub_class'

@ravikamaraj Thanks for the log output.  The log indicates that the pick_first LB policy is not able to connect to the subchannel, so it puts itself into state SHUTDOWN, which is exactly what I suspected was happening.  But in that case, I don't understand why #12850 didn't fix it.  Is the log output the same when running with #12850?
@qingling128 It's hard to provide an ETA when I don't fully understand the problem.  I suspect it's on the order of days or maybe a week or two, probably not longer.

@markdroth
For changes to take effective, looks like classes has to be compiled, and included as part of grpc_node.node. Updating the classes directly under node_modules is not taking effect. Can you help with the next steps
Thanks
Thanks

@markdroth
I was able to run our app using grpc code from #12850, and was able to reproduce the issue. But this time, got an different error Failed to create subchannel. Please find the logs below
D1011 04:43:59.398873141   10305 pick_first.cc:472]          Pick First 0x31f4180 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 1;
D1011 04:43:59.398896640   10305 client_channel.cc:268]      chand=0x3154278: lb_policy=0x31f4180 state changed to CONNECTING
D1011 04:43:59.398902325   10305 client_channel.cc:254]      chand=0x3154278: Setting connectivity state to CONNECTING
D1011 04:43:59.400546494   10305 client_channel.cc:1396]     chand=0x3154278 calld=0x31cdcd0: entering client_channel combiner
D1011 04:43:59.400569959   10305 client_channel.cc:1110]     chand=0x3154278 calld=0x31cdcd0: starting pick on lb_policy=0x31f4180
D1011 04:43:59.400575701   10305 client_channel.cc:969]      chand=0x3154278 calld=0x31cdcd0: applying service config to call
D1011 04:43:59.400973399   10305 pick_first.cc:472]          Pick First 0x31f4180 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 2;
I1011 04:43:59.400990443   10305 pick_first.cc:576]          Pick First 0x31f4180 selected subchannel 0x323c110 (connected 0x30b9770)
I1011 04:43:59.400995994   10305 pick_first.cc:589]          Servicing pending pick with selected subchannel 0x30b9770
D1011 04:43:59.401002861   10305 client_channel.cc:268]      chand=0x3154278: lb_policy=0x31f4180 state changed to READY
D1011 04:43:59.401009388   10305 client_channel.cc:254]      chand=0x3154278: Setting connectivity state to READY
D1011 04:43:59.401023627   10305 client_channel.cc:1093]     chand=0x3154278 calld=0x31cdcd0: pick completed asynchronously
D1011 04:43:59.401030752   10305 client_channel.cc:1015]     chand=0x3154278 calld=0x31cdcd0: create subchannel_call=0x31ce120: error="No Error"
D1011 04:43:59.401036464   10305 client_channel.cc:943]      chand=0x3154278 calld=0x31cdcd0: sending 0 pending batches to subchannel_call=0x31ce120
D1011 04:43:59.404948073   10305 pick_first.cc:472]          Pick First 0x31f38f0 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 4;
D1011 04:43:59.404963671   10305 client_channel.cc:268]      chand=0x30b6588: lb_policy=0x31f38f0 state changed to TRANSIENT_FAILURE
D1011 04:43:59.404968707   10305 client_channel.cc:254]      chand=0x30b6588: Setting connectivity state to TRANSIENT_FAILURE
D1011 04:43:59.404975008   10305 client_channel.cc:268]      chand=0x30b6588: lb_policy=0x31f38f0 state changed to SHUTDOWN
D1011 04:43:59.404984597   10305 client_channel.cc:254]      chand=0x30b6588: Setting connectivity state to TRANSIENT_FAILURE
D1011 04:43:59.404991284   10305 pick_first.cc:101]          Pick First 0x31f38f0 destroyed.
D1011 04:43:59.405368758   10305 client_channel.cc:1396]     chand=0x30b6588 calld=0x32e03a0: entering client_channel combiner
D1011 04:43:59.405388943   10305 client_channel.cc:1252]     chand=0x30b6588 calld=0x32e03a0: deferring pick pending resolver result
D1011 04:43:59.451300528   10305 client_channel.cc:378]      chand=0x30b6588: got resolver result: error="No Error"
D1011 04:43:59.451336777   10305 pick_first.cc:685]          Pick First 0x32d9520 created.
I1011 04:43:59.451342766   10305 pick_first.cc:335]          Pick First 0x32d9520 received update with 2 addresses
I1011 04:43:59.451365513   10305 pick_first.cc:419]          Pick First 0x32d9520 created subchannel 0x30df9f0 for address uri ipv4:xx.xx.xxx.xxx:50057
I1011 04:43:59.451377807   10305 pick_first.cc:419]          Pick First 0x32d9520 created subchannel 0x3111c20 for address uri ipv4:xx.xx.xxx.xx:50057
D1011 04:43:59.451385161   10305 client_channel.cc:487]      chand=0x30b6588: resolver result: lb_policy_name="pick_first", service_config="(null)"
D1011 04:43:59.451390054   10305 client_channel.cc:569]      chand=0x30b6588: initializing new LB policy
D1011 04:43:59.451396176   10305 client_channel.cc:254]      chand=0x30b6588: Setting connectivity state to IDLE
D1011 04:43:59.451401208   10305 client_channel.cc:1233]     chand=0x30b6588 calld=0x32e03a0: resolver returned, doing pick
D1011 04:43:59.451405618   10305 client_channel.cc:1110]     chand=0x30b6588 calld=0x32e03a0: starting pick on lb_policy=0x32d9520
D1011 04:43:59.451409171   10305 client_channel.cc:969]      chand=0x30b6588 calld=0x32e03a0: applying service config to call
D1011 04:43:59.451416796   10305 pick_first.cc:472]          Pick First 0x32d9520 connectivity changed. Updating selected: 0; Updating subchannels: 0; Checking 0 index (2 total); State: 4;
D1011 04:43:59.451423011   10305 client_channel.cc:268]      chand=0x30b6588: lb_policy=0x32d9520 state changed to TRANSIENT_FAILURE
D1011 04:43:59.451428472   10305 client_channel.cc:254]      chand=0x30b6588: Setting connectivity state to TRANSIENT_FAILURE
D1011 04:43:59.451432697   10305 client_channel.cc:1093]     chand=0x30b6588 calld=0x32e03a0: pick completed asynchronously
D1011 04:43:59.451450456   10305 client_channel.cc:1041]     chand=0x30b6588 calld=0x32e03a0: failed to create subchannel: error={"created":"@1507722239.451436683","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1039,"referenced_errors":[{"created":"@1507722239.451421876","description":"Channel Shutdown","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":113}]}
D1011 04:43:59.451472794   10305 client_channel.cc:900]      chand=0x30b6588 calld=0x32e03a0: failing 0 pending batches: {"created":"@1507722239.451436683","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1039,"referenced_errors":[{"created":"@1507722239.451421876","description":"Channel Shutdown","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":113}]}
D1011 04:43:59.451506415   10305 client_channel.cc:1337]     chand=0x30b6588 calld=0x32e03a0: failing batch with error: {"created":"@1507722239.451436683","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1039,"referenced_errors":[{"created":"@1507722239.451421876","description":"Channel Shutdown","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":113}]}
D1011 04:43:59.451516950   10305 client_channel.cc:1337]     chand=0x30b6588 calld=0x32e03a0: failing batch with error: {"created":"@1507722239.451436683","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1039,"referenced_errors":[{"created":"@1507722239.451421876","description":"Channel Shutdown","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":113}]}
D1011 04:43:59.451521804   10305 client_channel.cc:268]      chand=0x30b6588: lb_policy=0x32d9520 state changed to SHUTDOWN
D1011 04:43:59.451532024   10305 client_channel.cc:254]      chand=0x30b6588: Setting connectivity state to TRANSIENT_FAILURE
D1011 04:43:59.451540077   10305 pick_first.cc:101]          Pick First 0x32d9520 destroyed.

lsof o/p
node      10305           user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305           user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10306     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10306     user   17u     sock                0,7      0t0 249259644 can't identify protocol
V8        10305 10307     user   16u     sock                0,7      0t0 249248843 can't identify protocol
V8        10305 10307     user   17u     sock                0,7      0t0 249259644 can't identify protocol
V8        10305 10308     user   16u     sock                0,7      0t0 249248843 can't identify protocol
V8        10305 10308     user   17u     sock                0,7      0t0 249259644 can't identify protocol
V8        10305 10309     user   16u     sock                0,7      0t0 249248843 can't identify protocol
V8        10305 10309     user   17u     sock                0,7      0t0 249259644 can't identify protocol
V8        10305 10310     user   16u     sock                0,7      0t0 249248843 can't identify protocol
V8        10305 10310     user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10311     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10311     user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10312     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10312     user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10313     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10313     user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10314     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10314     user   17u     sock                0,7      0t0 249259644 can't identify protocol
node      10305 10316     user   16u     sock                0,7      0t0 249248843 can't identify protocol
node      10305 10316     user   17u     sock                0,7      0t0 249259644 can't identify protocol

@ravikamaraj Thanks, that looks much better.  So I think #12850 is working as I expected -- we're now returning an error that indicates that the channel is shutting down, rather than the incorrect error about the LB policy dropping the call.  (Note that the error we're returning contains another error within it; the outer error says "failed to create subchannel", but the inner error says "Channel Shutdown", which is what's being returned by the LB policy.  It may be that the wrapped languages should do a better job of exposing the referenced errors as well as the outer one, but that's a separate issue.)
Note that the thing that's triggering this problem in the first place is that the client is losing connectivity to the server.  This can happen because (e.g.) the server goes down or because there's some network problem.  Is that expected in this environment?  Do you know how long the connectivity problem is lasting?
To be clear, I do think there's still another bug here -- even if there's a legitimate interruption of connectivity, we should be handling it more gracefully.  Looking at this more closely, it seems odd that after we destroy and recreate the LB policy, the new instance is immediately seeing the subchannel in state shutdown.  Even if the server went down and stayed down for the rest of the test, the new LB policy instance should start in state CONNECTING, not state SHUTDOWN.  I wonder if we're somehow failing to remove the subchannel from the subchannel index before we shutdown the old LB policy, so we're getting the same subchannel back in the new LB policy instead of creating a new subchannel...
Can you try running with #12878 instead of #12850?  I don't think it will solve the problem, but it will provide better logging that should help me see what's actually going on here.
I'm also a little mystified as to why this would be different in 1.6.x than in 1.4.x.  The handling of the LB policy restart case didn't really change between those two versions.  Can you also send me the log of running the same test with 1.4.x?
Thanks for all the info!

@markdroth - Thanks for the info!

Note that the thing that's triggering this problem in the first place is that the client is losing connectivity to the server. This can happen because (e.g.) the server goes down or because there's some network problem. Is that expected in this environment?

Services run behind AWS ELB, and idle connections are closed upon idle timeout.

Do you know how long the connectivity problem is lasting?

The issue last as long as the node service is restarted.

Can you try running with #12878 instead of #12850?

Sure, I will run sometime this week, and post the results

Can you also send me the log of running the same test with 1.4.x?

I was not able to reproduce the issue. I will run more test this week.

Can I temporarily disable the loading balancing feature of gRPC? I have a RoundRobin Polly policy so the LB is not a key feature to me. Or can I downgrade gRPC to a specific version to avoid this issue? Which version should I use? Thanks.

Same here. Any potential workaround?

Until someone can provide me trace logs of this problem with #12878 patched in, I don't have enough information to understand what's triggering the problem, so I can't provide a work-around.
From earlier in this issue, it seems that people are not having this problem with gRPC 1.4.x, only with 1.6.x, so you can try downgrading to that until we get this resolved.

example project to easily reproduce this in nodejs : https://github.com/subk/grpc-issue-12506
It looks like reusing the same host/port on each test run will always trigger the "load balacing policy" error at some point.
If the server binds on a random port on each run, everything is fine.
My 2 cts

Thanks a lot! I tried @subk's suggestion and modified our test mocks to use a different port every time. Now we no longer get this error.
The potential impact of this issue is still a bit concerning though. We'll stay tuned for a fix while developing with this workaround.

I've asked David to look into trying to reproduce this locally so that we can try to figure out what's causing the problem.

@markdroth
Please find trace logs for patch #12878 with 1.6.1
D1021 05:20:57.577039924   17249 client_channel.cc:1394]     chand=0x338bb98 calld=0x334c7c0: entering client_channel combiner
D1021 05:20:57.577068211   17249 client_channel.cc:303]      chand=0x338bb98: starting name resolution
D1021 05:20:57.577215873   17249 client_channel.cc:1250]     chand=0x338bb98 calld=0x334c7c0: deferring pick pending resolver result
D1021 05:20:57.623056859   17249 client_channel.cc:376]      chand=0x338bb98: got resolver result: error="No Error"
D1021 05:20:57.623093926   17249 pick_first.cc:596]          Pick First 0x338c1f0 created.
I1021 05:20:57.623100761   17249 pick_first.cc:264]          Pick First 0x338c1f0 received update with 2 addresses
D1021 05:20:57.623105894   17249 subchannel_list.cc:103]     [LB 0x338c1f0] Creating subchannel list 0x331fd10 for 2 subchannels
D1021 05:20:57.623128958   17249 subchannel_list.cc:157]     [LB 0x338c1f0] subchannel for address uri ipv4:<IP_ADDRESS>:50057 shutting down, ignoring
D1021 05:20:57.623139765   17249 subchannel_list.cc:157]     [LB 0x338c1f0] subchannel for address uri ipv4:<IP_ADDRESS>:50057 shutting down, ignoring
D1021 05:20:57.623147420   17249 client_channel.cc:485]      chand=0x338bb98: resolver result: lb_policy_name="pick_first" (changed), service_config="(null)"
D1021 05:20:57.623155261   17249 client_channel.cc:567]      chand=0x338bb98: initializing new LB policy
D1021 05:20:57.623161501   17249 client_channel.cc:252]      chand=0x338bb98: setting connectivity state to TRANSIENT_FAILURE
D1021 05:20:57.623166146   17249 client_channel.cc:1231]     chand=0x338bb98 calld=0x334c7c0: resolver returned, doing pick
D1021 05:20:57.623170571   17249 client_channel.cc:1108]     chand=0x338bb98 calld=0x334c7c0: starting pick on lb_policy=0x338c1f0
D1021 05:20:57.623175542   17249 client_channel.cc:967]      chand=0x338bb98 calld=0x334c7c0: applying service config to call
D1021 05:21:00.197542657   17249 pick_first.cc:366]          Pick First 0x330a7d0 connectivity changed for subchannel 0x3364200 (0 of 2), subchannel_list 0x33199e0: state=SHUTDOWN p->shutdown=0 sd->subchannel_list->shutting_down=0 error={"created":"@1508588460.197499274","description":"Endpoint read failed","file":"../src/core/ext/transport/chttp2/transport/chttp2_transport.cc","file_line":2520,"grpc_status":14,"occurred_during_write":0,"referenced_errors":[{"created":"@1508588460.197495772","description":"EOF","file":"../src/core/lib/iomgr/tcp_uv.cc","file_line":154}]}
D1021 05:21:00.197591482   17249 subchannel_list.cc:84]      [LB 0x330a7d0] subchannel list 0x33199e0 index 0 of 2 (subchannel 0x3364200): stopping connectivity watch
D1021 05:21:00.197598929   17249 subchannel_list.cc:235]     [LB 0x330a7d0] subchannel_list 0x33199e0 UNREF 2->1 (pf_selected_shutdown)
D1021 05:21:00.197603759   17249 pick_first.cc:77]           Pick First 0x330a7d0 Shutting down
D1021 05:21:00.197607867   17249 subchannel_list.cc:277]     [LB 0x330a7d0] Shutting down subchannel_list 0x33199e0 (pf_shutdown)
D1021 05:21:00.197612136   17249 subchannel_list.cc:40]      [LB 0x330a7d0] subchannel list 0x33199e0 index 0 of 2 (subchannel 0x3364200): unreffing subchannel
D1021 05:21:00.197635089   17249 subchannel_list.cc:235]     [LB 0x330a7d0] subchannel_list 0x33199e0 UNREF 1->0 (pf_shutdown)
D1021 05:21:00.197639650   17249 subchannel_list.cc:204]     [LB 0x330a7d0] Destroying subchannel_list 0x33199e0
D1021 05:21:00.197650357   17249 client_channel.cc:266]      chand=0x3240088: lb_policy=0x330a7d0 state changed to SHUTDOWN
D1021 05:21:00.197681244   17249 client_channel.cc:252]      chand=0x3240088: setting connectivity state to TRANSIENT_FAILURE
D1021 05:21:00.197694471   17249 pick_first.cc:77]           Pick First 0x330a7d0 Shutting down
D1021 05:21:00.197705581   17249 pick_first.cc:70]           Pick First 0x330a7d0 destroyed.
D1021 05:21:00.198302048   17249 client_channel.cc:376]      chand=0x3240088: got resolver result: error="No Error"
D1021 05:21:00.198322908   17249 pick_first.cc:596]          Pick First 0x330a7d0 created.
I1021 05:21:00.198328767   17249 pick_first.cc:264]          Pick First 0x330a7d0 received update with 2 addresses
D1021 05:21:00.198333395   17249 subchannel_list.cc:103]     [LB 0x330a7d0] Creating subchannel list 0x33199e0 for 2 subchannels
D1021 05:21:00.198372962   17249 subchannel_list.cc:170]     [LB 0x330a7d0] subchannel list 0x33199e0 index 0: Created subchannel 0x3364200 for address uri ipv4:<IP_ADDRESS>:50055; initial connectivity state: IDLE
D1021 05:21:00.198405591   17249 subchannel_list.cc:170]     [LB 0x330a7d0] subchannel list 0x33199e0 index 1: Created subchannel 0x3232650 for address uri ipv4:<IP_ADDRESS>:50055; initial connectivity state: IDLE
D1021 05:21:00.198416758   17249 client_channel.cc:485]      chand=0x3240088: resolver result: lb_policy_name="pick_first", service_config="(null)"
D1021 05:21:00.198421846   17249 client_channel.cc:567]      chand=0x3240088: initializing new LB policy
D1021 05:21:00.198426960   17249 client_channel.cc:252]      chand=0x3240088: setting connectivity state to IDLE
D1021 05:21:02.576758902   17249 client_channel.cc:1073]     chand=0x338bb98 calld=0x334c7c0: cancelling pick from LB policy 0x338c1f0
D1021 05:21:02.576828162   17249 client_channel.cc:1091]     chand=0x338bb98 calld=0x334c7c0: pick completed asynchronously
D1021 05:21:02.577576542   17249 client_channel.cc:1039]     chand=0x338bb98 calld=0x334c7c0: failed to create subchannel: error={"created":"@1508588462.577543524","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1037,"referenced_errors":[{"created":"@1508588462.576825647","description":"Pick Cancelled","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":120,"referenced_errors":[{"created":"@1508588462.576747733","description":"Deadline Exceeded","file":"../src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":4}]}]}
D1021 05:21:02.577612732   17249 client_channel.cc:898]      chand=0x338bb98 calld=0x334c7c0: failing 0 pending batches: {"created":"@1508588462.577543524","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1037,"referenced_errors":[{"created":"@1508588462.576825647","description":"Pick Cancelled","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":120,"referenced_errors":[{"created":"@1508588462.576747733","description":"Deadline Exceeded","file":"../src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":4}]}]}
D1021 05:21:02.577652441   17249 client_channel.cc:1335]     chand=0x338bb98 calld=0x334c7c0: failing batch with error: {"created":"@1508588462.577543524","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1037,"referenced_errors":[{"created":"@1508588462.576825647","description":"Pick Cancelled","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":120,"referenced_errors":[{"created":"@1508588462.576747733","description":"Deadline Exceeded","file":"../src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":4}]}]}
D1021 05:21:02.577674579   17249 client_channel.cc:1335]     chand=0x338bb98 calld=0x334c7c0: failing batch with error: {"created":"@1508588462.577543524","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1037,"referenced_errors":[{"created":"@1508588462.576825647","description":"Pick Cancelled","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":120,"referenced_errors":[{"created":"@1508588462.576747733","description":"Deadline Exceeded","file":"../src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":4}]}]}
D1021 05:21:02.577685059   17249 client_channel.cc:1335]     chand=0x338bb98 calld=0x334c7c0: failing batch with error: {"created":"@1508588462.577543524","description":"Failed to create subchannel","file":"../src/core/ext/filters/client_channel/client_channel.cc","file_line":1037,"referenced_errors":[{"created":"@1508588462.576825647","description":"Pick Cancelled","file":"../src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":120,"referenced_errors":[{"created":"@1508588462.576747733","description":"Deadline Exceeded","file":"../src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":4}]}]}

This happened for us in grpc 1.6.6 (in Node). We had issues with the remote grpc service, which were recovered, but one of our servers got stuck in some kind of bad state and continue to throw this error for several hours until the client instance was manually restarted. Other clients were able to contact the grpc service without issues during this time.

Can anyone please confirm is this is still an issue at current head? The code has gone through significant changes lately.

@dgquintas still having this issue just downgraded from 1.7.0 to 1.4.0 this fixed the issue for me

@dgquintas, the only significant change lately was #12878, and the last comment from @ravikamaraj above shows that there is still a bug with that PR (although I think the PR did change a bit between when he ran that test and when it was merged, so I suppose anything's possible).
If the problem is still occurring at head, I would be interested to see if #12829 fixes it.

This still occurs in 1.7.1

@adematte would you have a way to reproduce it? Or some logs obtained as described above? Thanks!

@ravikamaraj thanks for the logs! What I can see in them is that the pick first policy goes into TRANSIENT_FAILURE right away, due to:
D1021 08:12:25.896662833   26436 subchannel_list.cc:103]     [LB 0x3076b70] Creating subchannel list 0x2f709c0 for 2 subchannels
D1021 08:12:25.896710056   26436 subchannel_list.cc:157]     [LB 0x3076b70] subchannel for address uri ipv4:xx.xx.xxx.xxx:50057 shutting down, ignoring
D1021 08:12:25.896754559   26436 subchannel_list.cc:157]     [LB 0x3076b70] subchannel for address uri ipv4:xx.xx.xxx.xx:50057 shutting down, ignoring

This is surprising. It'd happen, for example, if the servers aren't accessible. It may be a silly question, but were there healthy, accessible servers at those ip:ports?

Assuming the root cause is the same in both places, in our case the servers were accessible and healthy, and able to be contacted by other grpc clients (most of which used the Go grpc implementation)

@subk Thanks for the repro example! After talking to @murgatroid99 (from grpc node), it seems the failure may not be due to grpc itself, but rather to the way the testing is done: by repeatedly bringing up and down the server on the same port (also the use of forceShutdown on it), it seems the scheduling of events behind the scenes results in trying to start a server on a port that's still being used by the previous instance of the server (this port-reusing issue may even be happening at the operating system level). That'd explain why a random port every time fixes it. Another way to verify this would be to launch a single server on a separate process, thus preventing the continuous up-and-down over the same port.

@connor4312 , any chance of getting the client logs (with environment variables described here set) when the issue happens?

@dgquintas

This is surprising. It'd happen, for example, if the servers aren't accessible. It may be a silly question, but were there healthy, accessible servers at those ip:ports?

Yes, the servers are healthy running behind AWS ELB

@ravikamaraj , I'm trying to verify the hypothesis from this comment: are the servers continuously up or are they brought down and up repeatedly?

@dgquintas we still haven't found a way to reproduce this error. We used to encounter it during testing because we were mistakenly bringing up and down clients excessively. However this error now occurs in our staging environment and we are holding off production due to this issue. Please note that the hosts are healthy when the problem occurs and when it does the only fix is to bring down the impacted client.
We've activated GRPC traces, i ll send them next time we encounter that error.

Whatever this is causing the issue, we are hopeful the changes coming up in https://github.com/grpc/grpc/pull/12829 will fix it. We are trying to get it merged this week.

@dgquintas

are the servers continuously up or are they brought down and up repeatedly?

servers are continuously up

Same problem here, version 1.7.2
During tests, it starts and stops server on each test, after 2 successful tests I got this issue.
First I thought it was something specific to the third test I'm working on and I once I found this issue, I just copied and pasted the ones that are working and I got the same error.
So is nothing related to my third test but it seems starting and stoping the server a lot generates this issue

We're experiencing this issue in production with GRPC Ruby. We're tracking the issue on GitLab: https://gitlab.com/gitlab-org/gitaly/issues/810
Our unicorn servers are generally recycled before we experience the issue, but we see it a lot in our long running Sidekiq (job) servers.
The only fix is to restart the services, but we're also planning on downgrading back to GRPC 1.4.x as the problem occurs exclusively on GRPC 1.6.x.
Unfortunately the volumes on the affected servers is too high to enable tracing, but if we manage to find a way to recreate it, I'll add some more traces.

I experienced this issue in my demo project with GRPC Ruby (grpc gem is 1.7.3). I used haproxy as load balancer. It will raise "Call dropped by load balancing policy" when I stop and stop the grpc server.
I upgrade grpc gem to 1.8.0 today. Now the issue is disappeared.

Getting this on 1.10.1 through Firebase UI (their auth thing).

I am getting this error as well. The server side is still running GRPC 1.8.6 while the client side (Python) 1.11.0 (we are evaluating upgrading to 1.11.0 and production server will be running 1.8.6 before the upgrading). The error message is exactly Call dropped by load balancing policy.

Can one of you try running with #14886 and see if the problem is still occurring?  I think I may have fixed one possible cause of this as part of that PR, which may have fixed the problem.  If not, we'll have to investigate further.  Thanks!

@markdroth is there C# NuGet build ?

Not yet.  But the PR will be included in the upcoming 1.12 release.

@markdroth After confirming the PR was merged into 1.12.0-pre1, I tested this version and the "Call dropped by load balancing policy" exception has not appeared so far. I will continue to pay attention to it.


Pull Request to Update Student Bio Page

The portfolio link of the student bio page was updated with this PR.

The changes have been reviewed and is now being approved for merging into the main.


Not working on Graphene OS

I tried everything to get it working, but it when I start TakoStats it just won't show the overlay. I tested other things that rely on Shizuku and they work just fine (I tested App Ops & Shizuku app installer).
In addition to that, whenever I click on the "Overlay" button, the app crashes. I attached a screen recording and a log of that happening.
Error in TakoStats 7c01fa473dc8.txt
https://github.com/user-attachments/assets/a5487ec1-52e0-4e57-9a9a-03e9f83bf300

Same issue here.

It doesn’t work on stock android too


Revert "ROX-11335: Move Scanner DB secret out of main Scanner DB cont…

…ainer (#2039)"
This reverts commit 3026244df2756148636c14ade87607cf7565daff.
Description
This breaks upgrade tests at the moment during the downgrade step. This is because the Upgrade tests don't seem to account for Scanner's init container
Checklist

[x] Investigated and inspected CI test results
~[ ] Unit test and regression tests added~
~[ ] Evaluated and added CHANGELOG entry if required~
~[ ] Determined and documented upgrade steps~
~[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)~

Testing Performed
CI

Closing in favor of https://github.com/stackrox/stackrox/pull/2307


Example with interpolation not working

Describe the bug
Interpolation with relative levels seems not to be working.
To Reproduce
from omegaconf import OmegaConf

s = """
server:
  host: localhost
  port: 80

client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}
  # relative interpolation
  description: Client of ${.url}
"""

conf = OmegaConf.create(s)
print(conf.client.url)  # Works
print(conf.client.description)  # Error

Expected behavior
Expected to be printed Client of http://localhost:80/, instead I get:
omegaconf.errors.ConfigKeyError: str interpolation key '.url' not found
        full_key: client.description
        reference_type=Any
        object_type=dict

Additional context

[X] OmegaConf version: 2.0.6
[X] Python version: 3.8.8
[X] Operating system : Windows 10
[X] Please provide a minimal repro

Relative interpolation is not supported in OmegaConf 2.0.
The correct doc for 2.0 is https://omegaconf.readthedocs.io/en/2.0_branch/.
If you want to try relative interpolations, you can use the dev build (pip install --pre --upgrade omegaconf)


Support secret parameters featrue

Fluentd 0.12.13 or later supports secret parameter feature.
This feature works as below:
  <match mysql.input>
    type mysql_bulk
    host localhost
    database test_app_development
    username root
    password xxxxxx
    column_names id,user_name,created_at,updated_at
    table users
    flush_interval 10s
  </match>
</ROOT>

Hmm..What does work this feature with old version?

When using older version of fluentd, this secret parameter causes no effect. It will be simply ignored.

Ah, that's right.
No problem!
:+1:

Thanks!

update version 0.0.7.
pushed rubygems.
https://rubygems.org/gems/fluent-plugin-mysql-bulk


Update Export Map example to handle WebGL

This is the example update extracted from #13000 as separate PR.

Thanks, @mike-000


ThingInterface does not have contentUrl

In the modelling discussion, for AudioObject, MediaObject and VideoObject, the URI of the object is contained in the contentURL feld.
However, for requesting a ControlAction, it is impossible to access this field, because the the PropertyValue nodes do not support this.
PropertyValue --> NodeValue implements ThingInterface. ThingInterface does not have contentURL field.
Could be a solution to just add the contentUrl field in ThingInterface??

works. thanks


add support for HTTPS RR

This is a fresh PR with a single commit. There was a strange issue with the previous PR.

has doctest strings
test suite passes
there is some difference in behavior from dig, which uses quotes for the alpn field even when not necessary
note that cloudflare.com has HTTPS records for test purposes

Thanks very much for this - mow merged and new release uploaded to PyPi (0.9.20). Apologies for the delay getting to this.


fix: Datasource navigation causing infinite loop

Description
We were seeing an infinite loop of navigation between datasources when there was are more than one page in an app.
The problem was because the datasource navigation url (pageId) would change every time which caused a bug in context switching. The root cause of this issue was the urls not being built correctly because of a useCallback hook.
This PR fixes the issue to ensure the url for datasources is always based on the current page id and not a "cached" pageId
PR fixes following issue(s)
Fixes #25356
Media
Before
https://github.com/appsmithorg/appsmith/assets/12022471/a50e7540-e25a-4cf8-a432-48e0b16ce7b1
After
https://github.com/appsmithorg/appsmith/assets/12022471/c70ebd1a-6f7a-40f9-87bc-b09358dba871
Type of change

Bug fix (non-breaking change which fixes an issue)

Testing
How Has This Been Tested?

[x] Manual

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

/ok-to-test

/build-deploy-preview

/ok-to-test


[bug] Fibaro Smart Implant (FGBS-222) IN2 not working

Version
core-2021.4.3
Checklist:

[ ] I am not using HomeAssistant. Or: a developer has told me to come here.
[X] I have checked the troubleshooting section and my problem is not described there.
[X] I have read the changelog problem was not mentioned there.

Build/Run method

[X] Docker
[ ] PKG
[ ] Snap package
[ ] Manually built (git clone - npm install - npm run build )

zwavejs2mqtt version: 0.11.0? <version here, you can find it on control panel (not anymore?)>**
zwave-js version:  3.2.1? <version here, you can find it on control panel (not anymore?)>**
Describe the bug
Fibaro Smart Implant (FGBS-222) with Device ID 145 in my case, has no working Input 2. Input 1 and Output 1&2 are working fine. Maybe an incorrect mapping?

When Input 1 is closed, 145-113-0 (red circle) is switching accordingly to the configuration (normally open or normally closed) between [1] Idle and [2] Intrusion.
When Input 2 is closed, nothing happens. There is also no entry in the zwave log.
145-113-1 and 145-113-2 are staying on undefined the whole time. So my guessing is, that maybe the mapping is wrong? IN1 only switches the global sensor 145-113-0 and IN2 is switching nothing. I think it should be like IN1 switching 145-113-1 & 145-113-0 and IN2 switching 145-113-2 & 145-113-0
There are some threads about this problem in the home assistant forums. I seems like this bug was also persistent in OZW.
To Reproduce
Include Fibaro Smart Implant and change the configuration to normally closed or open alarm input for both inputs. Close and open Input 2 to ground.

@LeoGr13 Please attach logs. cc @AlCalzone

Sorry, ofc I forgot about that.
zwavejs2mqtt-store (2).zip
What I also forgot to mention is, that there is a configuration option to reverse operation of IN1 and IN2. If you configure it as reverse operating, only IN2 works. Just to clarify, that I dont believe in a hardware issue or something like this.

Just to be sure, does the log include a re-interview? If not, please do that and post another one. I can take a look later.

Log did not include a re-interview, so I re-interviewed the device and now everything works.
I feel so sorry for wasting your time, I should have just tried the re-interview before.
Just searched the internet for the problem and found the posts on the forums, where many people had the same problem for a long time and never came up with a solution. So I decided to open an issue..
Maybe this device just needs a re-interview after changing configuration parameters?
IN1 and IN2 are now properly reporting their state, also changing parameters in the protection class works perfectly (to use inputs independently from outputs)
Thank you
These logs include the re-interview and the working inputs:
Logs with reinterview and working.zip

If a config parameter changes device functionality like endpoints, at least a re-interview is necessary.
Usually for these params, the manual indicates that the device needs to be excluded and re-included. This can be a tad annoying but is required by the Z-Wave specifications and sometimes even necessary for the devices to fully accept the changes.


Application lifecycle properties with default values when privacy mode set in OPTIN after EXEMPT

HI !
Something going wrong with application lifecycle properties.
I have to use a consent SDK which is enable or not your analytics SDK.
The user can change the privacy mode EXEMPT to OPTIN and OPTIN to EXEMPT.
And the issue is when the mode change from EXEMPT to OPTIN, all lifecycle properties have default values.
For exemple app_fsd is equals to 19700101
You can reproduce it with little changes in MainActivity is your sample.
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding.recyclerview.apply {
            layoutManager = LinearLayoutManager(this@MainActivity)
            adapter = MainAdapter(animals, this@MainActivity::onItemClick)
        }

        lifecycleScope.launch {
            delay(10_000)
            PianoAnalytics.getInstance().privacyModesStorage.currentMode = PrivacyMode.EXEMPT
        }
    }

    override fun onResume() {
        super.onResume()
        PianoAnalytics.getInstance().privacyModesStorage.currentMode = PrivacyMode.OPTIN

        PianoAnalytics.getInstance().sendEvents(
            Event.Builder(Event.PAGE_DISPLAY)
                .properties(
                    Property(PropertyName.PAGE_FULL_NAME, "main"),
                )
                .build()
        )
    }

Start the app, wait for 10 secondes and click on an animal. Then come back to the list and you can see in logs, default values.

Hi @axel-lefaux,
As I see the issue appears only after changing PrivacyMode to a less strict mode. It persists until app's restart for app_fsd, and for other properties until "session expiration" (configuration.sessionBackgroundDuration seconds not in foreground mode) or app's restart.
Thanks for finding it.

Released in version 3.3.3


Beets Crashes on Import

Every time I run beets import it crashes and gives me this error
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 114                                                       0, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 113                                                       0, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 881                                                       , in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 858                                                       , in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 316, i                                                       n run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 3                                                       47, in run
    self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 1                                                       60, in coro
    task = func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 1380,                                                        in manipulate_files
    task.finalize(session)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 537, i                                                       n finalize
    self._emit_imported(session.lib)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 563, i                                                       n _emit_imported
    plugins.send('album_imported', lib=lib, album=self.album)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 458, in                                                        send
    result = handler(**arguments)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 123, in                                                        wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beetsplug/importadded.py", line                                                        109, in update_album_times
    album.added = min(album_mtimes)
ValueError: min() arg is an empty sequence

Here's the config.yaml
directory: /media/media/music/New/Imported
library: /media/media/music/Music/musiclibrary.blb

import:
    copy: yes
    move: no
    log: /home/media/apps/beets/beets.log

plugins: lyrics chroma lastgenre lastimport fetchart ftintitle fromfilename inline web copyartifacts rewrite discogs importadded replaygain rewrite importfeeds mpdupdate plexupdate smartplaylist duplicates fuzzy info mbcollection missing types check follow

paths:
    default: $albumartist_sort/($original_year) $album%aunique{} ($year) [$format]/%if{$multidisc,Disc $disc/}$track. $title
    singleton: Non-Album/$artist/$title
    comp: Compilations/$album%aunique{}/$track $title
    albumtype:soundtrack: Soundtracks/$album/$track $title
    ext:log: $albumartist - $album
    ext:cue: $albumartist - $album

item_fields:
    multidisc: 1 if disctotal > 1 else 0

original_date: no

match:
    preferred:
        countries: ['US', 'GB|UK']
        media: ['CD', 'Digital Media|File']
        original_year: yes

lyrics:
    auto: yes
    fallback: 'No lyrics found'
    force: no
    google_API_key: ********
    google_engine_ID:<PHONE_NUMBER>23014548361:lndtuqkycfu
    sources: google lyricwiki lyrics.com musixmatch

chroma:
    auto: yes

acoustid:
    apikey: ********

lastgenre:
    auto: yes
    canonical: yes
    count: 1
    fallback: ''
    force: yes
    min_weight: 10
    source: artist
    separator: ', '
    whitelist: yes

copyartifacts:
    extensions: .*
    print_ignored: yes

echonest:
    apikey: ********
    auto: yes
    upload: yes
    convert: yes
    truncate: yes

fetchart:
    auto: yes
    cautious: yes
    cover_names: cover front art album folder
    google_search: no
    minwidth: 0
    enforce_ratio: no
    remote_priority: no
    sources: coverart itunes albumart amazon google wikipedia

ftintitle:
    auto: yes
    drop: no
    format: feat. {0}

importadded:
    preserve_mtimes: no

lastimport:
    user: ********
    per_page: 500
    retry_limit: 3

mpd:
    host: localhost
    port: 6600
    password: None
    rating: yes
    rating_mix: 0.75

replaygain:
    auto: yes
    backend: gstreamer
    overwrite: yes
    targetlevel: 89

importfeeds:
    absolute_path: yes
    dir: /media/media/music/Playlists/
    fortmats: m3u_multi
    m3u_name: New\ Music.m3u
    relative_to: /media/media/music/Music/

plex:
    host: localhost
    port: ********

duplicates:
    album: no
    checksum: ffmpeg -i {file} -f crc -
    copy: none
    count: no
    delete: no
    format: $albumartist - $album - $title
    full: no
    strict: no
    keys: [mb_trackid, mb_albumid]
    move: none
    path: no
    tag: no

fuzzy:
    threshold: 0.7
    prefix: '~'

musicbrainz:
    user: ********
    pass:********

mbcollection:
    auto: yes

missing:
    count: yes
    format: $albumartist - $album - $title
    total: yes

types:
    rating: int

web:
    host: <IP_ADDRESS>
    port: 8337

check:
  import: yes
  write-check: yes
  write-update: yes
  convert-update: yes
  threads: 4

follow:
    email: ********
    password: ********
    userid: ********
    auto: yes

Ubuntu 14.04

Thanks for the report, and sorry for the frustrating crash.
The error is coming from the importadded plugin. Can you check (on a lark) whether flipping that preserve_mtimes option has any effect?
Also, does this happen regardless of which album you import? And does it happen before it prompts you for any input?

It would prompt me for input for several albums when I was importing new things, but in this specific case that I posted the log from i was using
beet import -AC /path/to/library

to import my current library so it wasn't prompting me at all. It didn't happen on the same album but usually about 40 albums into an import -AC and only 5 or 6 in on an auto tagging import, so it may be a time thing.
I flipped the preserve_mtimes to yes, and it seemed to get a little further. but got the same error.
Then I removed importadded from the plugins line and got this;
/media/media/music/Music/Beastie Boys/Paul's Boutique Demos (Bootleg) (2005) [FL                                                                     AC]
/media/media/music/Music/Beastie Boys/Paul’s Boutique (1989) [FLAC]
Ignored files:
    1 - Cover.jpg
    2 - Back.jpg
    3 - Disc.jpg
    Matched Artwork.jpg
/media/media/music/Music/Beastie Boys/Root Down (EP) (1995) [FLAC]
/media/media/music/Music/Beastie Boys/So What'cha Want (Single) (1992) [FLAC]
/media/media/music/Music/Beastie Boys/The In Sound From Way Out! (Insrumental An                                                                     thology) (1996) [FLAC]
Followed A Silent Film
/media/media/music/Music/Beastie Boys/The Mix-Up (2007) [FLAC]
/media/media/music/Music/Beastie Boys/To the 5 Boroughs (2004) [FLAC]
/media/media/music/Music/Beastie Boys/Tour Shot! (EP) (1994) [FLAC]
/media/media/music/Music/Beastie Boys/Ultra Fly Tracks (Bootleg) (1994) [FLAC]
/media/media/music/Music/Beat, The/I Just Can’t Stop It (1980) [FLAC]
/media/media/music/Music/Beat, The/Special Beat Service (1982) [FLAC]
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/_                                                                     _init__.py", line 1140, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/_                                                                     _init__.py", line 1130, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/c                                                                     ommands.py", line 881, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/c                                                                     ommands.py", line 858, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/impo                                                                     rter.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util                                                                     /pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util                                                                     /pipeline.py", line 183, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/impo                                                                     rter.py", line 1353, in plugin_stage
    func(session, task)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plug                                                                     ins.py", line 117, in wrapper
    assert self._log.level == logging.NOTSET
AssertionError

Actually I added importadded back to the plugins line and ran import -AC and got that same AssertionError, so I don't know what's going on.

Does this seem to be associated with a specific album? Can you reproduce it by importing the same album again?

No, but it does do it after a consistent average of albums number, Beastie Boys, Beck, etc. around 40-45 albums for an import -AC import.
But then it crashes after only seven or eight albums during a normal autotagging import as seen here
media@mediaserver:~/scripts$ beet -c ~/.config/beets/import_config.yaml import -q /media/media/music/New/Manual

/media/media/music/New/Manual (1 items)
Skipping.

/media/media/music/New/Manual/Black Rebel Motorcycle Club - 2001 - Sampler (4 items)
Skipping.

/media/media/music/New/Manual/Black Rebel Motorcycle Club - 2001 - Screaming Gun [FLAC] (5 items)
Tagging:
    Black Rebel Motorcycle Club - Screaming Gun
URL:
    http://musicbrainz.org/release/0505714a-c90f-4501-8339-1db9e7aff9d9
(Similarity: 96.9%) (tracks) (2001, US)
 * Failsafe                 -> Fail-Safe
 * TV Loop (Deep Down) (#5) -> TV Loop (Down Deep) (#4) (index, title)
 * Screaming Gun (#4)       -> Screaming Gun (#5) (index)

/media/media/music/New/Manual/Black Rebel Motorcycle Club - Beat The Devil's Tattoo (2010) [FLAC] (VVR730519) (15 items)
Correcting tags from:
    Black Rebel Motorcycle Club - Beat The Devil's Tattoo
To:
    Black Rebel Motorcycle Club - Beat the Devil’s Tattoo
URL:
    http://musicbrainz.org/release/2be4297a-4afd-436b-9afd-95cc326ab950
(Similarity: 99.4%) (country) (CD, 2010, GB, Abstract Dragon)
 * Beat The Devil's Tattoo -> Beat the Devil’s Tattoo
 * Shadow's Keeper         -> Shadow’s Keeper
 * Half-State              -> Half‐State
 * 1-51                    -> 1:51
Moving artifact: Black Rebel Motorcycle Club - Screaming Gun.log
Moving artifact: Black Rebel Motorcycle Club - Screaming Gun.m3u
Moving artifact: Black Rebel Motorcycle Club - Screaming Gun.cue
Followed Black Rebel Motorcycle Club

/media/media/music/New/Manual/Black Rebel Motorcycle Club - Beat The Devil's Tattoo - 2010 (CD - FLAC - Lossless) (13 items)
Correcting tags from:
    Black Rebel Motorcycle Club - Beat The Devil's Tattoo
To:
    Black Rebel Motorcycle Club - Beat the Devil’s Tattoo
URL:
    http://musicbrainz.org/release/fdf5cdda-a94f-49f6-9f8a-80f56f3e7c59
(Similarity: 100.0%) (CD, 2010, US, Vagrant Records)
 * Beat The Devil's Tattoo -> Beat the Devil’s Tattoo
 * Shadow's Keeper         -> Shadow’s Keeper
 * Half State              -> Half-State
This album is already in the library!
Skipping.

/media/media/music/New/Manual/Black Rebel Motorcycle Club - iTunes Session (2010) [iTunes Plus] (7 items)
Skipping.

/media/media/music/New/Manual/Black Rebel Motorcycle Club - Specter at the Feast (12 items)
Tagging:
    Black Rebel Motorcycle Club - Specter at the Feast
URL:
    http://musicbrainz.org/release/aca722b8-341c-4227-a169-7cad4b40da0d
(Similarity: 100.0%) (CD, 2013, US)
 * Let The Day Begin   -> Let the Day Begin
 * Hate The Taste      -> Hate the Taste
 * Sometimes The Light -> Sometimes the Light

/media/media/music/New/Manual/Black Rebel Motorcycle Club - The Effects of 333 - 2008 (WEB FLAC) (10 items)
Correcting tags from:
    Black Rebel Motorcycle Club - The Effects Of 333
To:
    Black Rebel Motorcycle Club - The Effects of 333
URL:
    http://musicbrainz.org/release/12129186-c087-4a5c-aafe-ef016e5789b2
(Similarity: 98.2%) (media) (Digital Media, 2008, US, Abstract Dragon)
 * The Effects Of 333     -> The Effects of 333
 * I Know You're In There -> I Know You’re in There
Moving artifact: Black Rebel Motorcycle Club - Beat the Devil’s Tattoo.cue
Moving artifact: Black Rebel Motorcycle Club - Beat the Devil’s Tattoo.log
Moving artifact: Black Rebel Motorcycle Club - Beat The Devil's Tattoo.m3u
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1140, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1130, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 881, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 858, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 160, in coro
    task = func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 1268, in user_query
    plugins.send('import_task_choice', session=session, task=task)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 458, in send
    result = handler(**arguments)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 117, in wrapper
    assert self._log.level == logging.NOTSET
AssertionError

It's happening even without the importadded plugin enabled on the plugins line.
Sorry my config.yaml is so large, I like to type out all the options so I can remember what they are easily.

FYI, also that import_config.yaml used on that last log I posted is exactly the same except for the directories and libraries.

This is actually a different error (see the bottom of the traceback blob). Has the original one recurred?
Have you tried narrowing down which plugin is causing the new error? We've had reports that the third-party copyartifacts can cause crashes like this.

OK, I'll try to narrow it down.
I disabled copyartifacts and got this
/media/media/music/Music/Big Pink, The/A Brief History of Love (2009) [FLAC]
/media/media/music/Music/Billy Boy on Poison/Drama Junkie Queen (2009) [FLAC]
/media/media/music/Music/BIlly Bragg/Back to Basics (Anthology) (1987) [FLAC]
/media/media/music/Music/BIlly Bragg/Don't Try This at Home (Bonus Disc) (1991) [FLAC]; /media/media/music/Music/BIlly Bragg/Don't Try This at Home (Bonus Disc) (1991) [FLAC]/Disc 01; /media/media/music/Music/BIlly Bragg/Don't Try This at Home (Bonus Disc) (1991) [FLAC]/Disc 02
/media/media/music/Music/BIlly Bragg/Talking With the Taxman About Poetry [1986] (1986) [FLAC]; /media/media/music/Music/BIlly Bragg/Talking With the Taxman About Poetry [1986] (1986) [FLAC]/Disc 01; /media/media/music/Music/BIlly Bragg/Talking With the Taxman About Poetry [1986] (1986) [FLAC]/Disc 02
/media/media/music/Music/BIlly Bragg/William Bloke (1996) [FLAC]; /media/media/music/Music/BIlly Bragg/William Bloke (1996) [FLAC]/Disc 01; /media/media/music/Music/BIlly Bragg/William Bloke (1996) [FLAC]/Disc 02
/media/media/music/Music/BIlly Bragg/Workers Playtime (1988) [FLAC]; /media/media/music/Music/BIlly Bragg/Workers Playtime (1988) [FLAC]/Disc 01; /media/media/music/Music/BIlly Bragg/Workers Playtime (1988) [FLAC]/Disc 02
/media/media/music/Music/Billy Bragg and The Blokes/England, Half English (Bonus Disc) (2002) [FLAC]; /media/media/music/Music/Billy Bragg and The Blokes/England, Half English (Bonus Disc) (2002) [FLAC]/Disc 01; /media/media/music/Music/Billy Bragg and The Blokes/England, Half English (Bonus Disc) (2002) [FLAC]/Disc 02
Followed A Tribe Called Quest
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1140, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1130, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 881, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 858, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 183, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 1353, in plugin_stage
    func(session, task)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 117, in wrapper
    assert self._log.level == logging.NOTSET
AssertionError

With copyartifacts enabled, importadded enabled, preserve_mtimes set to yes
/media/media/music/Music/Big D and the Kids Table/How It Goes (2004) [FLAC]
/media/media/music/Music/Big D and the Kids Table/Strictly Rude (2007) [FLAC]
/media/media/music/Music/Big Data/1.0 (EP) (2013) [MP3]
/media/media/music/Music/Big Pink, The/A Brief History of Love (2009) [FLAC]
/media/media/music/Music/Billy Boy on Poison/Drama Junkie Queen (2009) [FLAC]
Ignored files:
    1 - Cover.jpg
    10 - Booklet 8.jpg
    11 - Booklet 9.jpg
    12 - Booklet 10.jpg
    13 - Alternate Back.jpg
    2 - Back.jpg
    3 - Booklet 1.jpg
    4 - Booklet 2.jpg
    5 - Booklet 3.jpg
    6 - Booklet 4.jpg
    7 - Booklet 5.jpg
    8 - Booklet 6.jpg
    9 - Booklet 7.jpg
    Unmatched Artwork.jpg
/media/media/music/Music/BIlly Bragg/Back to Basics (Anthology) (1987) [FLAC]
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1140, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1130, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 881, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 858, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 183, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 1353, in plugin_stage
    func(session, task)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 117, in wrapper
    assert self._log.level == logging.NOTSET
AssertionError

copyartifacts enabled, importadded enabled, preserve_mtimes no
/media/media/music/Music/Blur/For Tomorrow (1993) [FLAC]
/media/media/music/Music/Blur/Leisure (1991) [FLAC]
/media/media/music/Music/Blur/M.O.R. (1997) [FLAC]
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.14', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1140, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/__init__.py", line 1130, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 881, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/ui/commands.py", line 858, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 316, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/util/pipeline.py", line 183, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/importer.py", line 1353, in plugin_stage
    func(session, task)
  File "/usr/local/lib/python2.7/dist-packages/beets-1.3.14-py2.7.egg/beets/plugins.py", line 117, in wrapper
    assert self._log.level == logging.NOTSET
AssertionError

It seems I'm not even getting that original ValueError: min() arg is an empty sequence

Anything else I should try? I might have time a little later to try toggling off a bunch of plugins one at a time, but if you have suggestions to narrow it down that'd be great.

OK, thanks for looking into it. It sounds like copyartifacts is not the cause then—but I would still try fiddling with third-party plugins first. For example, my first stop would be check, then follow.

Just to bump this issue, i got the exact same traceback. For info I also have the check plugin enabled, it may be that (besides that i only have discogs and info).

@Ho33e5, can you help by narrowing it down to be certain that it's check? Or otherwise help us reproduce the problem?

Sure, I disabled the check plugin and it seems that this crash doesn't occur anymore. But I can't really reproduce the crash reliably, it is not on particular albums, it occurs on big import, something like every 20-30 imports.

Strange. Maybe it would be a good idea to report the bug to the beets-check project.

It looks like this was the beets-check bug reported elsewhere.


Slow subscribe ?

Hi,
I've a project with thousands of subscribers, the more I add the slower it becomes to add new ones.
I think the main reason is here
        if let index = subscriptions.index(where: { $0.subscriber === subscriber }) {
            subscriptions.remove(at: index)
        }

unfortunately this is going to be a major issue for my project; would you consider a PR for a different approach like an hashtable or whatever ?
thanks in advance

Thousands (plural!!) sounds crazy. Maybe it's just me and my approach is old-fashioned :)
Using a Set (and thus Hashable) should indeed improve performance. We have to be careful to not just provide a quick fix that makes the public API worse. It's fairly simple to wrap subscription blocks in StoreSubscriber at the moment, but if someone changed their subscription to block-based this way, how should they provide the hash value? An auto-incrementing integer would be a start, but also very odd.
I cannot say I like the change much, but I also cannot come up with good reasons to keep the subscription/removal algorithm as slow as it is now. Even though it sounds weird to me, why not support thousands of subscribers? I cannot come up with a real reason to not support this besides "breaking API change".
Although I wonder how the performance of propagating state updates is for you :)

Thanks for starting a discussion :)
I don't have much to add to my bug report, unfortunately I lack the confidence to talk about a solution yet but I can point out that the main problem for me is the price to pay for indexing the array of subscriptions when I can guarantee my subscribers are all unique. A "patch" for my case on my local fork is to have a "bulk creation" flag to toggle the index test and live with that until an improvement is done.
Still, I've a question, why do you think this requires a breaking API change? I'm new to the project and just started exploring the code base but I believe it would be better to keep the same generic api but also support something like Equatable StoreSubscribers with a new class of subscribe functions
func subscribe<S: StoreSubscriber & Hashable> ...
I understand it would split the code path to support hashable and non hashable subscribers but this would leave the problem of "what is the proper hash for your subscriber?" in the hand of the user where it should actually be.

Although I wonder how the performance of propagating state updates is for you :)

it's a linear time and that is ok :)

Since we restrict subscriptions to class types, we might be able to do some object identifier magic for a hash table. Let me see if I can whip something up.

Yeah I think that might work a little better :)
Opening a PR in a moment

Still need to understand a pattern where you have thousands of subscribers. I have a game which manage thousands of units using ReSwift, subscribing and dispatching to all the units would be indeed slow. (Didn't even tried, but I guess that dispatching to thousand of subscriber would be even more slower than subscribing). I decided first to do a controller around units of the same type, so this controller actually subscribe to the state, and dispatch the change to the units which are owned by this controller.
I preferred to do a controller around anyway, because there was much more to handle than state changes.
In an iOS app (not a game), I would love to see a use case using a lot of subscribers, I need to try it myself.
I wonder how React coupled with redux handle that, as the recommended design pattern is to not do your state connect not too deep.

After a bit more testing it appears that changing subscriptions to a set causes a significant (46%) regression to state update performance, due to the increased cost of filtering & enumerating a set. I'm taking a look to see if theres anything I can quickly do to reduce that cost, but I'll be PR'ing this change either way to allow others to change things as well, however I recommend we don't merge the PR unless we can resolve the state update performance regression, as for most users that is a much more important metric.

Managed to get both the subscribe & notify performance better.
See the PR for details, but in summary, when subscribing and notifying 3000 subscribers:

testNotify(): 0.0046359 -> 0.0031482 (32% faster)
testSubscribe(): 0.1861 -> 0.0087853 (95% faster)

guys you are just awesome! thanks for the top class support 👍

👍

Amazing work @mjarvis


Update oauth.py

check it - https://api.vk.com/oauth/authorize?client_id=5823310&redirect_uri=http://aiohttp-login.imbolc.name/auth/login/vkontakte&v=5.60&scope=email

You will update your site?

Yep, i'll update it and publish new version to pypi in a few days, thanks for PR

done


Last activity date (UTC) does not reflect Teams channel conversations?

The documentation describes the following:
Last activity date (UTC) | The latest date a message was received by the group. - This is the latest date an activity happened in an email conversation, Yammer, or the Site.
Is it so that the last activity date field is NOT updated for example if somebody posts a channel message to the Teams workspace of that group? We are utilizing this info in our custom archival scripts and I assume there can be various Teams that only get activity in the messaging side on channels, but not at all in email/yammer (alternative communication platforms of the m365 group) or necessarily the SharePoint site (file edits?).
As general feedback, hopefully this can be improved.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4d43b01f-5c1e-515a-900a-7aab927c294e
Version Independent ID: 23b1fc80-9d25-c9fb-2600-3d91b11b9063
Content: Microsoft 365 admin center groups reports - Microsoft 365 admin
Content Source: microsoft-365/admin/activity-reports/office-365-groups-ww.md
Service: o365-administration
GitHub Login: @KwekuA
Microsoft Alias: kwekua

@TuomasPakkanen Thank you for your feedback. That is correct, it only captures the latest date an activity happened in an email conversation, Yammer, or the Site. I'd recommend you leave the feedback on our main feedback portal - https://feedbackportal.microsoft.com/ so that our dev/product team can review and provide the reason why this is not available or if it is in the roadmap.
Hope this helps!
Thanks
Sri

@TuomasPakkanen Thank you very much for your kind feedback. Considering your valuable feedback, please log this issue as advised. Here we are limited to documentation fixes and improvements only. Concerning team will evaluate and take the appropriate action, as per their schedule. We proceed here to close it. Thanks again for taking out some time and sharing your feedback.


feat: Adding support to Athena ClientRequestToken

Feature or Bugfix

Feature

Detail
Adding support to Athena query reuse. Athena has a option to reuse queries, which basically the same thing that the aws-sdk-pandas athena cache option. Later we can deprecate the aws-sdk-pandas cache.
Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: e825f9f4c1301af1ec7d9e5bdea6fe1fec19ea83
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: c59cd1eafd3050c8df22eccc8b335dff141f8540
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: c59cd1eafd3050c8df22eccc8b335dff141f8540
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 100fbf0a8c1bc32ecfa64fce79925b87f5930f8b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 100fbf0a8c1bc32ecfa64fce79925b87f5930f8b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: b6c4e8cc0b6e9dcb62b87346fde5cc8e846f1fbe
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: b6c4e8cc0b6e9dcb62b87346fde5cc8e846f1fbe
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


    docs: update proxy option link

New updates on head repo.
https://github.com/vitejs/vite/commit/734a9e3a4b9a0824a5ba4a5420f9e1176ce74093

resolve at #603


How to open specific routes as modals.

Is there a way to open a specific route within a nested stack with the mode of modal.

Made a pr with my workaround fix https://github.com/react-community/react-navigation/pull/1378

How do I use this? :)
Where do I need to put mode:'modal' to have it open for specific view and not whole stacknavigator?

Hi! In an effort to get the hundreds of issues on this repo under control I'm closing issues tagged as questions. Please don't take this personally - it's simply something we need to do to get the issues to a manageable point. If you still have a question I encourage you to re-read the docs, ask on StackOverflow, or ask on the #react-navigation Reactiflux channel. Thank you!