Philips Hue changed to unavailable

Home Assistant release with the issue:

Last working Home Assistant release (if known):
unsure.. but upgraded to 0.78 and issue still persists.
Operating environment (Hass.io/Docker/Windows/etc.):

Docker running on Mint.
Component/platform:

Description of problem:
Philips Hue hub becomes unavailable when it is actually online and responsive. Therefore all lights switch to "changed to unavailable". Then switch back to the previous state.
Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
hue:
  bridges:
    - host: <IP_ADDRESS>
      allow_unreachable: true
      allow_hue_groups: true

tried both to false.. same problem

Traceback (if applicable):

Additional information:
thread on the forum
https://community.home-assistant.io/t/philips-hue-changed-to-unavailable/
errors from the logs
2018-09-18 12:46:01 ERROR (MainThread) [homeassistant.components.light.hue] Unable to reach bridge <IP_ADDRESS>
2018-09-18 12:46:06 INFO (MainThread) [homeassistant.components.light.hue] Reconnected to bridge <IP_ADDRESS>
All custom components were removed, all additions components were removed and it still happened. Sometimes every 5-10 minutes.

Home Assistant release with the issue: 77.x
Last working Home Assistant release (if known):
Cannot remember for sure, but sometime in the 60's, it's been several months.
Operating environment (Hass.io/Docker/Windows/etc.):
HA in python VENV, ubuntu 16.04
Component/platform:
Hue intergration
Description of problem:
I have 10 hue lights, and every 1-3 hours, some of them change to unavailable.  Not always the same lights,  changed to unavailable for 1-3 minutes and back on.  Does not seem to be tied to any specific automation, script or HA control.  Lights will switch to unavailable in the middle of the day when no one is home.  I have noticed that usually when one light is unavailble, if I try to use HA to control other HA lights, they change to unavailable as well.  This leads me to beleive that it's the hub that's unavailable, and the lights just have not updated thier status.  While they are "unavailable" i'm able to control them via Alexa (natively, I don'd expose them with HA to alexa), or the Hue App itself.  I saw that if HA thinks the Hub is unavailable it will set lights to unavalable, so I moved the my hub so it is plugged into the router directly, the same place where my HA server is plugged into, just in case it was an issue with switches, etc.  It did not make any difference.

I saw that if HA thinks the Hub is unavailable it will set lights to unavailable

It's exactly that that is causing the issue here, hence my link to the https://community.home-assistant.io/t/philips-hue-changed-to-unavailable/65556/19?u=mariusthvdb change in setup.
Though that change might have its use, it seems to cause the unavailability of all light while in fact they are not. ( In my setup, the hub isnt even unavailable, at east not visible in the frontend.) A very unfortunate side-effect, which I hope can be reverted?

Adding to that, they are not only marked as unavailable, but the state machine also sees them as unavailable making logic based on timed attributes (last_changed) and state useless/ impossible.

I've also been having the same problem for the last couple of months.
I'm running HA via Docker on an Intel NUC.
I started a fresh Docker container of HA and after adding my Hue bridge have been gradually adding other things to it. It was all fine until I added automations that turned on Hue lights. It then stared showing the Hue lights as unavailable again.
Hoping the problem can be found, and this can be fixed soon.

well, purely coincidental I seem to have stumbled on a 100% related event:
whenever I click on an entity to display a more info card, and see the loading state history, my Hue lights go Unavailable. NO exception..
never noticed this before, and kept searching the origin in automations and (python)-scripts related to the lights.
I have set the history and recorder to record only a few components (in fact because of this Hue issue and trying to prevent as much timing related  and processor heavy events from happening).

" It was all fine until I added automations that turned on Hue lights. It then stared showing the Hue lights as unavailable again."
Even with my automations removed 100% it still happens to me.

well. Ive just installed a brand new Hassos version and let discover add the Hue . NI automatons yet, nothing else of any significance.
Hue is constantly switching in the backend:

That can't be right, proof:

The lights though shouldn't have to be shown as Unavailable because of this, making the whole integration useless. Which it is, cause we cant reach the lights when Unavailable (while they fact are just fine) and automations become unreliable or impossible.
Hope we hear from the dev's soon and be able to debug this further, and solve..

Mine are now showing as unavailable again, even with all Hue related automations removed.
I'm giving up on (trying to use) Hue lights in Home Assistant for now. I'll just use HomeKit apps/automations until this is fixed.

Added a suggestion for a possible mitigation in the forum thread mentioned above, if anyone wants to try it out.

I've been unable to repro this issue.
I suggest someone that has this issue to start adding some timestamps to see if it's really waiting 4 seconds.
https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/light/hue.py#L167
Change this to:
    from datetime import datetime
    try:
        start = datetime.now()
        _LOGGER.info("Updating start: %s (is_group: %s)", start.isoformat(), is_group)
        with async_timeout.timeout(4):
            await api.update()
        _LOGGER.info("Updating done in %s (is_group: %s)", start-datetime.now(), is_group)
    except (asyncio.TimeoutError, aiohue.AiohueException):
        _LOGGER.exception("failed to update in time")

This will start logging some timestamps.

where would I change that on a docker install please?

Create file in config custom_components/light/hue.py and copy the contents of the light/hue.py file of your release. Note that you will have to remove these files when your'e done testing to avoid having your platform go out of sync with rest of Hue component.

Is there any way a Hassio install can do this too, in an effort to assist? Or Hassos for that matter?

I made the changes and the hue wouldn't start. Can someone provide the full code please? I am on the latest version of HA

I've opened a PR to add the logging to the Hue component. You will be able to configure the logger component to show these logs in the next release:
logger:
  logs:
    homeassistant.components.light.hue: debug

This logic is now live in the beta and will be part of 0.80.

thanks, and no idea what I should be expecting, but this is 1 minute of the log, now on HA 0.80.0b0:
2018-10-08 14:45:01 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.029 seconds
2018-10-08 14:45:01 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.048 seconds
2018-10-08 14:45:07 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.163 seconds
2018-10-08 14:45:07 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.181 seconds
2018-10-08 14:45:13 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.054 seconds
2018-10-08 14:45:13 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.071 seconds
2018-10-08 14:45:19 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.243 seconds
2018-10-08 14:45:19 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.251 seconds
2018-10-08 14:45:25 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.198 seconds
2018-10-08 14:45:25 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.237 seconds
2018-10-08 14:45:31 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.314 seconds
2018-10-08 14:45:31 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.331 seconds
2018-10-08 14:45:36 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.202 seconds
2018-10-08 14:45:36 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.215 seconds
2018-10-08 14:45:42 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.056 seconds
2018-10-08 14:45:42 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.088 seconds
2018-10-08 14:45:48 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.031 seconds
2018-10-08 14:45:48 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.097 seconds
2018-10-08 14:45:54 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.177 seconds
2018-10-08 14:45:54 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.183 seconds
2018-10-08 14:46:00 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.224 seconds
2018-10-08 14:46:00 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.244 seconds

fyi, ive disabled all automations that listen to the lights, and no custom component hue.py is active.
hope this helps, please let us know if we can anything else to give you more info.

Yeah, these are the response times that I expect. As you can see, it never takes more than 0.5 seconds to finish a request. It goes to unavailable if it takes over 4 seconds.

Ok thanks, give it some time ;-)
Almost got that already , but, indeed no Unavailable seen yet.
Should it update this frequently ? That is expected behavior too?

Tradfri has a complete different implementation. It pushes changes to us instead of us having to fetch it every 5 seconds.
Timer out of sync is a big red flag that something on your system is doing I/O and holding up the event loop. Are you running any custom components?

On dev-info page, you can find a list of all loaded integrations, could you copy paste that ?

You can also DM it to me on Discord if you feel uncomfortable putting it in public.

On dev-info page, you can find a list of all loaded integrations, could you copy paste that ?

you mean this:

Yeah that page. To DM me on Discord, find me in the HA chat, click on my name… 😉

I am getting this even more consistently after the upgrade to 0.81.2. Anything I can do to help dubug?

+1 so bad in my 081.2 setup now the lights most of the time wont even show up. Horrible lag in the system.
No other errors than before to report though, even stripped down the full instance to be as stand alone as possible. Integration doesn’t feel platinum at all, even if the error is caused by something else in the setup and Hue not connecting is only the messenger....

What is your CPU usage?

can now positively report that killing the mqtt on my Pi enables the Hue lights almost without hiccups. Ive moved the Mqtt broker to a second en brand new install on another Pi, have the main setup read that Mqtt stream, and immediately the lights are gone again on.

What MQTT broker and how is it installed?

mosquitto add-on
{
"plain": true,
"plain_websockets": false,
"ssl": true,
"ssl_websockets": false,
"anonymous": true,
"logins": [
{
"username": "username",
"password": "password"
}
],
"customize": {
"active": false,
"folder": "mosquitto"
},
"certfile": "fullchain.pem",
"keyfile": "privkey.pem"
}

do you have a lot of traffic on MQTT?

well, I guess, it is a constant stream of my energy meters and attached switches.

Are you using MQTT state stream ?

no, this is the mqtt setting:
broker: !secret mqtt_base_url
port: 1883
username: !secret mqtt_username
password: !secret mqtt_password
and of mqtt_base_url is pointing to the dedicated Hassio instance with the mosquitto broker.

updated to 0.81.4, but no relieve:
2018-11-03 18:54:53 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:54:53 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 4 seconds
2018-11-03 18:55:16 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:55:16 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 8 seconds
2018-11-03 18:55:23 INFO (MainThread) [homeassistant.setup] Setting up apple_tv
2018-11-03 18:55:23 INFO (MainThread) [homeassistant.setup] Setup of domain apple_tv took 0.0 seconds.
2018-11-03 18:55:44 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:55:44 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 16 seconds
2018-11-03 18:55:53 INFO (MainThread) [homeassistant.setup] Setting up remote
2018-11-03 18:55:53 INFO (MainThread) [homeassistant.setup] Setup of domain remote took 0.0 seconds.
2018-11-03 18:56:38 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:56:38 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 32 seconds
2018-11-03 18:57:28 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:57:28 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 64 seconds
2018-11-03 18:58:53 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 18:58:53 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 128 seconds
2018-11-03 19:01:10 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 19:01:10 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 256 seconds
2018-11-03 19:05:37 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-03 19:05:37 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 512 seconds

only thing that stops this, is stop subscribing to the mqtt brokers events. which is not an option....
this behavior is really deteriorated since 0.81.1 so please have a look what was changed that could cause this.

What do you mean with "mqtt brokers events" ?
Can you enable the Hue debug logging for more info.

brokers events: What I meant was that I have mosquitto addon installed, and disabling that made all lights available, practically without issues.
Meanwhile I am on 0.81.5, (also have Mqtt configured via integration now, on both systems, still version 2) and have rebuilt the 2 systems from scratch, with remarkable results. It seems that the mqtt binary_sensors of this format:
- platform: mqtt
  name: 'Netwerk Auditorium state bin'
  state_topic: 'redacted_mac_address/powerswitch-zwave/id/state'
  payload_on: 'on'
  payload_off: 'off'
  qos: 0
  device_class: plug

caused the issues of the Hue lights being unavailable at random and frequent times. Ive completely tokens these out (some 20 of them) and this has made my main system (without the broker, reading mqtt events from the hub system) show the Hue lights permanently (1 condition excluded) and the Hub system, on which the broker resides, show the lights better than before, albeit not as permanent as the may system yet.
The 1 condition I refer to with still causes Hue lights to not show at all, is a front-end issue. Whenever a customization fails, eg not being able to read a state while the customization needs that state, the system errors out with the frontend error and system write/log. This causes the lights to become unavailable, or, worse, a startup not to show at all.
Ive set debug, but this doesn't give much more info than before: lights being unavailable if it takes longer than 4 seconds, other than that continuous stream of connecting succes.
1 thing always remains in the startup log:
2018-11-09 10:55:06 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>
2018-11-09 10:55:06 ERROR (MainThread) [homeassistant.components.hue] Error connecting to the Hue bridge at <IP_ADDRESS>. Retrying in 2 seconds

No matter how dismantled the system is, this always happens on startup without exception.

I'm struggling with this too. I have enabled debugging in configuration.yaml, but don't see any messages in the System Log under Hassio's system tab. Is this where I would expect to see them?
I'm on Hassio version 0.81.1.

you need to enable logger: in configuration.yaml, and add this
logs:
homeassistant.components.light.hue: debug
see:https://www.home-assistant.io/components/logger/

I finally got the logging turned on for the hue lights.  I am running 81.6.  I am not running MQTT, just straight up HA on Ubuntu in a VENV.  I'm not seeing anything specific, other than hue is updating every 6 seconds.  Not sure if that's supposed to happen that often.  Anyway, Last night 2:31-2:33am, several lights went unavailable.  No automations running at that time.  Here is the log, but again, looks pretty uneventful.
2018-11-15 02:31:58 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.052 seconds
2018-11-15 02:32:04 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.022 seconds
2018-11-15 02:32:04 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.037 seconds
2018-11-15 02:32:10 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.023 seconds
2018-11-15 02:32:10 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.042 seconds
2018-11-15 02:32:16 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.017 seconds
2018-11-15 02:32:16 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.052 seconds
2018-11-15 02:32:22 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.018 seconds
2018-11-15 02:32:22 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.055 seconds
2018-11-15 02:32:28 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.036 seconds
2018-11-15 02:32:28 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.051 seconds
2018-11-15 02:32:34 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.031 seconds
2018-11-15 02:32:34 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.052 seconds
2018-11-15 02:32:40 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.018 seconds
2018-11-15 02:32:40 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.053 seconds
2018-11-15 02:32:46 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.017 seconds
2018-11-15 02:32:46 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.038 seconds
2018-11-15 02:32:52 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.025 seconds
2018-11-15 02:32:52 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.052 seconds
2018-11-15 02:32:58 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.039 seconds
2018-11-15 02:32:58 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.054 seconds
2018-11-15 02:33:04 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.036 seconds
2018-11-15 02:33:04 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.051 seconds
2018-11-15 02:33:10 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.036 seconds
2018-11-15 02:33:10 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.051 seconds
2018-11-15 02:33:16 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.040 seconds
2018-11-15 02:33:16 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.053 seconds
2018-11-15 02:33:22 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.037 seconds
2018-11-15 02:33:22 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.052 seconds
2018-11-15 02:33:28 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 0.022 seconds
2018-11-15 02:33:28 DEBUG (MainThread) [homeassistant.components.light.hue] Finished group request in 0.035 seconds

So I'm getting the same sort of thing with logging - no errors and they all finish within 4 seconds, but still the lights are going unavailable. It tends to happen in clusters - all good for a few hours, then available/unavailable every couple minutes for about 5 minutes. Nothing unusual happening at the same time. Any suggestions on where to go from here?

Anything happening on your system (outside HASS) during those 5 minutes? Maybe a backup service? Docker update?

No, I log everything and nothing significant is happening. I moved the Hue hub to see if it was a problem with interference, but no obvious change. There are a number of people with the same issue, so there must be some commonality.
I have a v1 bridge (the round one) and no other zigbee hubs/devices. I have Lifx lights and tp-link switches with no similar problems. I have various other devices (Apple TV, Harmony Hub, etc) and have not had similar problems. I do have connectivity drop-out issues with two old smart switches since moving to mesh wifi, but the Hue problems pre-dated this.

I just installed Hass.io today and I have the same problem. Here are the steps to reproduce my Hue problem.
Open Hue android app.
All lights show available.
Open hassio.local in a tab on a different device
Wait
Hue android app shows more and more lights becoming unavailable.
All my lights are connected through the Philips Hue hub. Could aggressive polling by Hass.io be the cause?

Ok, next step:
further testing shows that also binary_sensor ping causes instability in the Hue platform. Had already rebuilt one system and left these sensors (for my 6 household phones) out with remarkable results. No more Unavailable for the lights.
My secondary system, which holds the mosquitto broker, and was broken down to a minimal setup with mostly sensors and devices, still had frequent Unavailable Hue lights. It also still had the binary_sensors ping.
Although I don't want to miss these, I've taken them out now on the second system too. Havent seen a Unavailable yet since...
Must conclude that binary_sensors cause Hue lights (and groups) to go Unavailable intermittently. How can this be amended, so we can use the binary_sensors without issue?

I don't have any binary_sensor Pings, but I do have quite a few binary sensors,  many are templates, some are cameras motion/line crossing, etc.

For people reporting problems, can you please report how many lights you have connected to your bridge?
I might have found the issue with Home Assistant not runnnig the updates in parallel. Would love some testing, add PARALLEL_UPDATES = 0 to your light/hue.py file and see if that resolves it.

I have 19 lights connected to the Hue hub.
I'm running HA via Docker, where/how do I access light/hue.py?

14 for me and I'm also in a docker..

I have 6 lights and also 3 motion sensors on my hue hub, which might be increasing the load as I'm using rest sensors to poll the state. (PR #15121 for native sensor support died out a while ago)
Running via hassio on a rasp3

Sounds like this might not be the issue after all, but I have 5 light strips, 5 color bulbs, and 2 white bulbs.  I also have 2 dimmer control switches.

17 lights of various types (lights strips, white, white ambiance, white&color ambiance, hue-go)
2 dimmer switches, 1 switch, 8 motion sensors
1 hub.

Also having this problem, 0.83.2, docker on Ubuntu. Two bulbs, one motion sensor, one led strip. Static IP on the Hue hub, still log filling up with lines of hue unavailable.

Same thing - static IP on the hue, 0.84.3 hassio on RPi

Same thing - only one hue light, 0.84.3 hassio on RPi

So I recently upgraded my RPI3 to Hassio 0.84.5 and noticed a few glitches with my hue lights. When I've checked I am also impacted by this. I was confident it was only since the 0.84.5 upgrade, however looking back through device history I now see I've been affected for some time without knowing about it. Removed MQTT and ping binary sensors but still suffering the issue.

Same issue with pi3b+, new install hassio os 0.84.5, 3 light strips, 8 colour bulbs. Noticed as automations involving hue not working but other lights in same script ie. nanoleaf work.. Manually toggle of switch in hassio iOS app brings hue light on, no problem.

I migrated off of my pi3b onto a server i have running in the hopes that the problem was the processing power of the pi, but i'm still seeing lights becoming unavailable.  Its usually the same bulb that errors but i've seen it happen to all of them.

I'm seeing this on 0.84.6 on an older machine, but it's got 32GB of RAM and 4 cores. I tried upping the timeout to 5 seconds and still saw this:
2019-01-01 18:57:51 DEBUG (MainThread) [homeassistant.components.light.hue] Failed to fetch light:
2019-01-01 18:57:51 ERROR (MainThread) [homeassistant.components.light.hue] Unable to reach bridge <IP_ADDRESS> ()
2019-01-01 18:57:51 DEBUG (MainThread) [homeassistant.components.light.hue] Finished light request in 5.005 seconds

I'm suspect it might be the async_timeout package, so I'm going to try to run without it. I'm removing line 171 from the hue light component found here.

please see https://community.home-assistant.io/t/hue-motion-sensors-remotes-custom-component/27176/614?u=mariusthvdb for what seems to be a solution (albeit upping the timeout in the components)
all errors and 'flapping' gone....

Mariusthvdb: Does not work for me.. :(

I upped the timeout a while back, it did not make a difference for me either.  I don't get the messages about taking longer than X seconds, or hub disconnected though.  I only am getting the hue light NAME is unavailable.  and then usually back to available a few minutes later

@ptdalen
did you set the timeout in both files?
increasing the timeout in the homeassistant/components/light/hue.py from 4 to 10:

171:        with async_timeout.timeout(4):
And in the homeassistant/components/hue/bridge.py from 5 to 10:

164:        with async_timeout.timeout(5):

also, I use this setup, and do configure the Hue bridge ip for now as follows:
device_tracker:
  - platform: hue
sensor:
  - platform: hue
hue:
  bridges:
    - host: !secret hue_ip

using the CC version on @yottatsa s repo, which is an adapted version, or at least was, of the official CC by @robmarkcole .

@Mariusthvdb I had not made the changes in both locations.  I just did.  I'll keep an eye on it in HA and report back later.

I made the changes in both files.  Changed to 10 and restarted HA.  I've had 2 different hue bulbs go unavailable in the past 3-4 hours.

@ptdalen
Are these true unavailables, also in the Hue App ( lights do show unavailable now and then) or HA issues you think.
Is it as frequent as before or do you notice an improvement ?

It's a bit hard to track down to be honest.  It's only been a few hours so I'm not sure if its less frequent quite yet.  As far as if they are truly unavailble, thats the hard thing so see.  95% of the time it actually has no real effect on my so I dont notice.
  They go from unavailable back to available in logbook (or entity history) so quickly, that it usually does not cause an issue.  Once in a while, I go to bed, and wake in the morning and find a light that did not turn off like it was supposed to, then checking the logs, I see that it happened to be unavailable at the time it was supposed to turn off.  I almost never look at the lights in the Hue App, but if I happen to get lucky enough to catch one at the exact time its unavailable I defintiely will report back.  I think this is exactly what has made it so hard to narrow down.
I think the issue with constant flapping or losing connection to the hub is not the exact same issue.  Mine just seen to go unavailable off and on throughout the day and night.  Almost like they go into a "sleep" mode.  Sometime if I notice a light is unavailable, and I turn on/off another hue light, they come back immediately as available.

Every time home assistant restarts my hue lights switch to unavailable. The only way I have found to fix this is to delete the integration and re-configure. I'm still using the original hub (the circular one) and am thinking about getting the latest version to hopefully get this fixed.
For those who are experiencing this, do you have the older hub as well or if this is still present with the version 2 hub?

I am using version 2 of the hub and still have issues.

I have tried absolutely everything to debug this fault without any success.
I'm not a programmer at all so this has been very frustrating. Everything is working well but I have random and repeated 'Unable to reach bridge' errors. in the logs.
I'm running HassOS 2.8, hass.io supervisor 145, Home Assistant 0.88.0 on a RPi 3+
I have tried all of the following over several weeks: -
Stripping out all components in configuration.yaml
Stripping out all add-ons
Deleting and re-configuring the hue component
Un-linking the bridge and re-linking from the hue app
Setting a static IP address and changing the IP address of the bridge
In addition I'm now getting intermittent comms errors to my tp-link switches.
This bug spoils an otherwise very good home automation system. Hoping the developers sort this soon.

I used to get those errors too.
My Raspberry PI with Hassio was behind a powerline adapter (TPLink).
Decided to move it closer to my router, and plugged it in to my bridge - never received any errors again so far and it's been almost a month now...
Might help some of you ?

Hub v2  here and all cat6 direct into routers

Hub v2 also. Just rearranged my network so I’m direct into router too. Thanks for the advice.
Running out of things to try after this.

Is anybody else with problems using monitor? The website warns of wifi interference and when I reconfigured mine to be on demand every 15 minutes, I can see almost all the changes to unavailable occur within 1 minute of a scan.

I removed discovery from my config and my hue lights come up in my interface more often than not now. Every once in a while a restart will make the unavailable.

I've done some investigation with my hue bridge.  It turns out that it is a bit unreliable on the network level, and sometimes needs to do tcp retransmissions.  I've done a packet capture and can see that it clearly takes longer; https://i.imgur.com/CVo3pR0.png
So it would be really good to be able to set a higher timeout to avoid things flicker to unavailable all the time.

I've done some investigation with my hue bridge. It turns out that it is a bit unreliable on the network level, and sometimes needs to do tcp retransmissions. I've done a packet capture and can see that it clearly takes longer; https://i.imgur.com/CVo3pR0.png
So it would be really good to be able to set a higher timeout to avoid things flicker to unavailable all the time.

I've been having progressively worse bridge connectivity issues as I've added Hue devices and tried pretty much all of the workarounds detailed in this ticket. This morning I updated my switch to disable auto negotiation for the port the bridge is on and have been error-free for the past few hours. I'm betting I either have a marginal bridge or the Hue kit doesn't always handle negotiation properly.

https://community.home-assistant.io/t/all-my-hue-lights-randomly-becoming-unavailable/58296/65

Active thread here: https://github.com/home-assistant/home-assistant/issues/23796

I know there's an active thread at #23796 but this one is older and it's still occurring on my 0.100.2 installation. I tried disabling discovery but that's not helping at all.

I have problems too: every time I restart my Raspberry Pi 4 (where ha is running on a docker) i get all my hue lights unavailable. The hue bridge is online, the movement sensor connected to the hue bridge are online, but all lights unavailable. The only thing i can do is to unlink and relink the hue integration (tapping on the hue switch physical button).

I'm seeing what I believe is this issue as well.  I have hassio installed on a rpi3b+, basically a fresh install.  Out of 4 reboots, 2 the hue lights disappeared and they show up as unavailable.  Home Assistant version 0.101.3.
Happy to provide any logs / other information if helpful.

Experiencing similar issues. Running on a rpi4 in a Docker image.
First all hue lights show as unavailable and after a short time they show as available again in the UI. Never had this before.

I put this in that forum thread, but maybe it will help diagnose...
Upon boot the entire hue system was unavailable. The logs would say, 'hue system taking longer than 10 seconds'. I run everything on copper , no wifi used (I have WiFi, but the only things that connect to it are some phones and ESP32(s). Also no zigbee (other than Hue).
I rebooted many times over multiple days, tried many, many different possible fixes, none of which worked. What finally fixed it for me was:
discovery:
  ignore:
    - philips_hue


Fixed the Cloud Shell login flow

@JasonRShaver,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

@JasonRShaver : Thanks for your contribution to the Azure documentation! The author, @anhowe, has been notified to review your proposed change.

#please-close
Original author has not responded. This doc has been significantly updated and this text is no longer present.


Compatibility with Ad Astra

Hi!
Ad Astra adds several planets to explore via rockets, and they have new monsters and villagers.
It would be amazing if the Lunarians would be bucketable.
They are basically reskinned Villagers that function the same way on the Moon as the usual villagers do on the regular overworld.
Due to the villager thing, they CAN technically be picked up already, but they are converted to Plains Villagers, which means I don't actually get the Lunarian, aaaaand they would fit so well with the theme I'm building on a server with some friends where I play as sort of an "alien visitor" in my builds, so having Lunarians instead of Villagers would fit perfectly.

I shall have a look into it, what Minecraft version specifically does this occur in?

I shall have a look into it, what Minecraft version specifically does this occur in?

Amazing.
I am currently running 1.20.4, but we will eventually update to 1.21 once some more mods that we use catch up, so I guess a bit of both. :P

Alright, the latest update should fix this issue. Versions 1.20.1-1.3.0 and above and 1.20.2-1.2.0 and above support it. As of writing this, curseforge is processing the files but they are available on Modrinth now (https://modrinth.com/mod/villager-in-a-bucket/versions).
If there are any problems please open a new issue. I did testing so there shouldn't be any but please do some testing to double check.

Can confirm. Seems to work now, thank you! :D


Viewport

Adding the ability to use hotkeys with:
Default Tool
Zoom
Pan
Angle measurement
Scroll stack
Length measurement
Flip Horizontally
Flip Vertically
Rotate Right
Rotate Left
Invert
Zoom In
Zoom Out
Zoom to Fit
Reset
Clear Tools
Scroll Down
Scroll Up
Scroll to the Last Image
Scroll to First Image

@biharck, can you remove the package-lock.json and add it to your .gitignore?
The main OHIF projects use yarn and commit yarn.lock

You added a new Action, which broke a unit test. Can you update the unit test to include the new Action? You can see the error by clicking "Details" next to the integration check.

It should take you here:
https://circleci.com/gh/OHIF/ohif-core/70?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

@biharck if you resolve @evren217's issue and the conflict, then I think we can merge this and your other PR. I do believe we will want to change these actions in the (very) near future to use the exposed cornerstone DOM element instead of reacting to state changes though.


feat: Adding configurable description template for reporting issues

Signed-off-by: Kristen Armes<EMAIL_ADDRESS>Summary of Changes
This change adds the option to configure a template string that will be inserted in the description of the Report an issue modal when creating a ticket from a table view. If defined, the template string will prepopulate the issue description text field and can be used to suggest more detailed information to be provided by the user when an issue is reported.
Tests
frontend/amundsen_application/static/js/config/index.spec.ts was modified to test the retrieval of the config, and frontend/amundsen_application/static/js/pages/TableDetailPage/ReportTableIssue/index.spec.tsx was modified to test that the description textarea is populated with the expected config, or nothing if the config is a blank string.
Documentation
A description of the config has been added to frontend/docs/application_config.md
CheckList
Make sure you have checked all steps below to ensure a timely review.

[X] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"

In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

[X] PR includes a summary of changes.
[X] PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
[X] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain docstrings that explain what it does

Looks good to me, might want to announce that this change is being merged on slack so people can add the field to their custom config when they move to this version of frontend.


cast mjpeg data pointer to unsigned char. some compilers don't accept…

A void pointer is not accepted by e.g. msvc14 to set data buffer. Use cast to unsigned char pointer.
Add pre-processor definition so that mjpeg conversion function is actually declared. A cmake set won't influence the ifdef used to declare the uvc_mjpeg2rgb function in libuvc.h
change the uvc_get_device_list function return value to the actual uvc device count. This enables direct list iteration. The error return code is also preserved.

change the uvc_get_device_list function return value to the actual uvc device count. This enables direct list iteration. The error return code is also preserved.

@Rhodanos LIBUVC_HAS_JPEG should be exposed to the preprocessor at https://github.com/ktossell/libuvc/blob/master/include/libuvc/libuvc_config.h.in#L20 , which gets loaded prior to the uvc_mjpeg2rgb ifdef. Is that not getting applied?

No it is not. At least not with the build I did on windows. I've defined the flag in CmakeLists.txt using the command
ADD_DEFINITIONS(-DLIBUVC_HAS_JPEG) instead. Anyway it does not make too much sense for me to add a definition for jpeg code if jpeg found is not checked so I additionally added
if(JPEG_FOUND)ADD_DEFINITIONS(-DLIBUVC_HAS_JPEG)


Change 'listX' methods to 'getX' or 'getAllX'

Also see https://github.com/itslanguage/itslanguage-js/issues/232. This should reflect here to.

No where in the examples are those methods used :+1:


Robot e2e updates

updated robot e2e based on proposed automation starter kit

corrected display of user's first and last names: auth-passport.js and app.js
added new data-testid attribute where needed
added new structure for e2e/robot
added robot test cases for (based on fake user stories): login, navigator bar, counter
modified circle.yml to run smoke test or all enabled cases

@SethDavenport would you kindly review this pr when you have the time.

@SethDavenport please review. should be good to merge.

:shipit:


add partner name for create, update and show commands for managementp…

…artner

This checklist is used to make sure that common guidelines for a pull request are followed.

For new extensions:

[x] My extension description/summary conforms to the Extension Summary Guidelines.

If this PR is for a new extension or change to an existing extension, use the following to try out the changes in this PR:
docker run -it microsoft/azure-cli:latest
export EXT=<NAME>
pip install --upgrade --target ~/.azure/cliextensions/$EXT "git+https://github.com/jeffrey-ACE/azure-cli-extensions.git@master#subdirectory=src/$EXT&egg=$EXT"

Please change the description in setup.py to conform to https://github.com/Azure/azure-cli-extensions/blob/master/docs/extension_summary_guidelines.md
e.g. "Support for Management Partner preview." would be sufficient.

@derekbekoe Fixed the comment, can you please approve?


[Relay][Topi] Conv NHWC pack int8 kernel layout

Relavant PR - https://github.com/dmlc/tvm/pull/4038
Currently, _declaration_conv_nhwc_pack expects (NHWC, HWOI). However. TOPI expects the kernel layout to be HWIO whenever the input data layout is NHWC. There is no easy way to pass around the kernel_layout to TOPI.
Currently, we are planning to comment out the code to achieve correct functionality. This issue is to record this activity, and track the progress of fixing the schedule.
@llyfacebook plans to take a look for now.

close for now as the relevant PR get merged. Let us open a new thread for new items


Section navigation mode

Hi, thanks once more for this fantastic add-on.
If I'm not mistaken, then "section navigation mode" hasn't been working for a week or so. I may look into this myself later, but also wanted to leave this report to confirm that it's not an issue on my end. Thanks.

Guessing you had the new UI enabled?  Recent versions of the extension fix this.  Feel free to reopen if still broken.

I'm not sure if I'm on the new UI or not 🤔 (I don't remember enabling anything). I'm on version 185 of the extension, which looks like the latest version (and just tried restarting Firefox) -- I'm still seeing that it doesn't work. (Could I be screwing something up?)

Ahh, just tested in firefox and looks like the el.computedStyleMap() function isn't supported by firefox.
For now version 187 includes a partial fix.  Navigation mode will work, but the sidebar menu won't automatically open if closed

(Thank you!!!!!)


ocamlPackages.getopt: init at 20120615

Motivation for this change
This is a continuation of PR #129444 and #129509 which are prerequisites to make LIGO build under nix.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
21.11 Release Notes (or backporting 21.05 Relase notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Baked into #134424


Refactor installation and upgrade docs

I started looking at getting rid of the "Upgrade from 2 to 3" docs, and then started thinking about promoting the installer sections out of Agent (note the formatting is off due to the JS path detection)

I think we could go further and remove the v2/v3 selector from prime page real estate, and just leave a tiny 2 page with links in the menu, untill we can deprecate v2 altogether.

Slight caveat around removing the selector code if we're in any danger of introducing a v4
This work isn't actually done, I'm just looking at low effort ways of streamlining the navigation.

@pda Is there any timeline on releasing v4 of the agent? Is it likely to be this year?

Lets say that it's not going to be this year. Could we keep the selector commented out? So remove it visually, but not remove the ability to do the thing?

@pda Is there any timeline on releasing v4 of the agent? Is it likely to be this year?

Sorry, I missed this earlier. Nope! No major version bump planned at this point, very unlikely for this year.


Request "raw" Body Content-Type always application/json

Hello!
When I set the body type to Raw (json, text, etc), I'm presented with another dropdown in the bottom right of that input box. From there I can choose JSON, HTML or Text - but regardless of which I choose, the content-type that is being sent is application/json.
Shouldn't HTML yield text/html, and Text => text/plain ?
I find that if I manually set the header in the Headers tab, then it's sent correctly - but it seems like it should be done automatically...

Yup, you're right. Should be fixed in v2 soon.


fixed "".truncated crashed

fixed Fatal error: Can't form Range with upperBound < lowerBound

          1 Warning

:warning:
The source files have been modified. Please consider adding a CHANGELOG entry if necessary.

          4 Messages

:book:
Thank you for submitting a pull request to SwifterSwift. The team will review your submission as soon as possible.

:book:
iOS: Executed 679 tests, with 0 failures (0 unexpected) in 16.700 (17.633) seconds

:book:
tvOS: Executed 644 tests, with 0 failures (0 unexpected) in 4.118 (4.491) seconds

:book:
macOS: Executed 511 tests, with 0 failures (0 unexpected) in 6.166 (6.325) seconds

SwiftLint found issues
Warnings

File
Line
Reason

StringExtensions.swift
960
Lines should not have trailing whitespace. (trailing_whitespace)

StringExtensionsTests.swift
580
Lines should not have trailing whitespace. (trailing_whitespace)

  Generated by :no_entry_sign: Danger

@philCc Can you add a changelog entry for this as well? :)

@philCc Can you add a changelog entry for this as well? :)

I'm sorry, I don't know where to add.

@philCc Can you add a changelog entry for this as well? :)

I'm sorry, I don't know where to add.

You can take a look at this :)
https://github.com/SwifterSwift/SwifterSwift/blob/master/CONTRIBUTING.md#adding-changelog-entries
Or check-in previous merged PRs to see how others have done it :)

It has been added and the changelog format error has been modified

Thank you for contributing to SwifterSwift! I've invited you to join the SwifterSwift GitHub organization - no pressure to accept! If you'd like more information on what that means, check out our contributing guidelines. Feel free to reach out if you have any questions! 😃


Unpin openstack-k8s-operators

We pinned these to the released versions for the dev preview release but we should resume tracking the latest code.

If we use a specific release version then renovate does not attempt to bump it unless a new release (tag) is created. To track the latest commit of the main branch we have to bump it to the latest commit of main (or at least non-released version).

If we use a specific release version then renovate does not attempt to bump it unless a new release (tag) is created. To track the latest commit of the main branch we have to bump it to the latest commit of main (or at least non-released version).

Oh, I did not know you've to force bump manually after tagged version use. This is weird.


Multi draggable items feature

This PR provides multiple selection of draggable items.
Instead of relying on draggableId which on allows  to cope for single selection, I created a new variable selectedDraggedIds in the context API to hold all the items selected in different droppableId.
I've modified the display of ghost components during the drag so that we can visualise all items being dragged.
https://user-images.githubusercontent.com/3605098/165356985-e7be3e03-a76f-4d94-90fd-3fcf80e3d9c4.mov

Nice man, it didn't occur to me at first


No FB0?

/dev/fb0 is not created when the ISO is run under VirtualBox. I've tried the different options for VB graphics controller without success. FB0 is created under a default Debian 10 install in VirtualBox, so I don't think it's a VB conflict. I have not yet tried in on physical hardware.

@rnsi Added Boschs display DRM for qemu. I tried with VirtualBox drivers but it did not render fbDoom correctly. Please try qemu with the command qemu-system-x86_64 DoomLinux.iso. And thanks for your continuous feedback.

I tried the ISO on physical hardware:
On a Dell i5 Gen2 laptop with Intel graphics, fbdoom loads, but the screen is scrambled.
On a HP desktop with NVIDIA Quadro K2200, it fails to create FB0.
It boots fast, though!

@rnsi Did the "Linux logo" load correctly on Dell i5 Gen2 laptop with Intel graphics? Or was it scrambled as well?


Up method in the Migrations/_InitialCreate.cs

Maybe it's not an issue or it's least priority but when I came to the section where it says there's an Up method in the Migrations/_InitialCreate.cs I found that the file created in my project Migrations folder has the suffix name _Initial.cs not _InitialCreate.cs so I thought of reporting it for consistency although maybe I'm having an issue.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6719f08e-3bd7-dc1a-71df-f2ef9fbca9d8
Version Independent ID: 7096fdb3-612e-9e00-bd0b-8ea4886a09ce
Content: Add a model to a Razor Pages app in ASP.NET Core
Content Source: aspnetcore/tutorials/razor-pages/model.md
Product: aspnet-core
Technology: aspnetcore-tutorials
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

@arseyam  it's worth reporting. See my PR changes https://github.com/aspnet/AspNetCore.Docs/pull/13687/files

@arseyam thanks for bringing this to our attention.


No src directory in merged header placement mode?

Thanks for creating PFL! The merged header placement rules seem to overlap quite well with build2's canonical project layout. However, I notice two differences, in build2:

the src project is omitted and the name of the project is used directly (skips one level of nesting).
The tests directory is only for functional/integration tests since tests are placed next to headers/sources in the merged mode.

Any thoughts on these?

This does seem reasonable, but I don't think pitchfork should allow for it. Pitchfork should not be a perfect fit for all existing layouts, but should be a single, mostly prescriptive layout. There is some variety in arrangement of headers and sources, but AFAICT, this is because there's really no consensus in the community on how to arrange them, and the different styles are fundamentally different. Having or not having a top level src/ are practically the same, so allowing for both makes the project structure more complicated for (IMO) little benefit.
Yes, this is an excellent idea, which I believe is currently allowed, albeit somewhat underspecified. The merged test placement asks for unit tests to be in the same directory as normal source code. It's not 100% clear that integration or other kinds of tests should still be under the tests/ directory, though.


Clean up multitude of print outs

There are a lot of printouts that are useful for debugging, but generate a lot of noise during actual usage. Make a pass at cleaning them up.
Consider adding a "verbosity" level that controls what is printed.

Still a variety of std::cout messages that need to e cleaned up.


Provide PostGRES support for storing large and geospatial data

We should investigate other databases as well.
Need to figure our relationship to Girder

+1 to adding pgsql support to Girder as an alternate model layer. This
would be non-trivial to implement and maintain, but would lower the overall
burden on projects that use Girder and would benefit from pgsql support.
-Patrick
On Fri, Oct 2, 2015 at 7:53 PM, Aashish Chaudhary<EMAIL_ADDRESS>wrote:

We should investigate other databases as well.
Need to figure our relationship to Girder

—
Reply to this email directly or view it on GitHub
https://github.com/Kitware/minerva/issues/135.

The use case need definition.
We already have geospatial search in Girder items in Mongo through the geospatial plugin.  We also have the dataservices work in minerva which include a postgis DB that we can interact with through a geoserver search api.
What specific use case do we have that can not be supported by one of these?
Is the large geospatial data expected to be a source that is external to a Minerva source, similar to our current Mongo collection source?  I.e., we have a Minerva source which is a Girder item, and that item has metadata about how to connect to the datastore, but we do not track all of the data in the source within Girder.  If this is the case, what operations do we want to support, and why can't this be done via geoserver?

I've been tasked with adding this feature, and I want to make sure that whatever I pick will satisfy various people's requirements.  There have been discussions via email and meatspace, but I'd like to move that here so we have a good record of what gets decided.  People involved include @aashish24, @zachmullen, @danlamanna, @mgrauer, @mbertrand, @kotfic.
I don't think this is a Minerva-specific task, so it would be good if we can satisfy wider needs.
I think this should be a girder plugin.  I'm open to suggestions about which repo it should reside in (probably its own under an existing github organization).
@aashish24 said he would like to see a postgres database as a collection with each table as an item (presumably within a folder within that collection).  Since a folder is necessary to contain items in a collection, perhaps a better model would be to configure a postgres database into a folder (which could be in a collection or not).  When would we check for new tables or dropped tables?  Or is this only done at configuration time?
In the geoapp, I create an end point per table with standard database options: limit, offset, sort direction, sort direction, returned fields.  A query can produce a function where clause matching any number of restrictions (anded together).  Each column in the table can be matched to an exact value.  Numeric and dates can be restricted to one or both ends of a range.  Strings can be matched using stemming and lexical grouping.
This would be easy to extend to read the table's format and autogenerate rest endpoint based on that.  If we want to include special queries for PostGIS datatypes we can do that.
The virtue of this technique is that it is easy to extend to access other databases (Mongo and elasticsearch would need more configuration, as there is no distinct concept of columns).
Alternately, if the goal is have full ability to do anything that Postgresql supports, we should either just use PostREST or something similar.
There could also be a new model (a superset of the item model) that would expose python functions for database access.
I could see a girder plugin with configuration options allowing a list, each with the following:

Database Name: (user defined name to reference this database)
Database Type: (only Postgres for now)
Database Location: (<IP_ADDRESS>:5432 by default)
Show database in folder: [select or create a folder here]
Show tables: [blank for all tables in the default schema, or a json list]

There would be new endpoints:
GET database/(item id)/find       -- get data from a table
GET database/(database name)   -- get a list of items/tables associated with the db
We might also want
PUT database/(database name)/refresh  -- refresh the tables associated with the db
I could see it being useful to have the end points under the item (for instance at item/(item id)/database/find) instead.
Feedback is welcome.

In the geoapp, I create an end point per table with standard database options: limit, offset, sort direction, sort direction, returned fields. A query can produce a function where clause matching any number of restrictions (anded together). Each column in the table can be matched to an exact value. Numeric and dates can be restricted to one or both ends of a range. Strings can be matched using stemming and lexical grouping.

I certainly prefer this approach as a general solution. It might not meet @aashish24 's needs for GUMBO in particular, so we should consider that, but in general we don't want to expose arbitrary database connections to end users as girder would become a botnet very quickly. With your approach, we at least control the database connections, though malicious users could still construct queries designed to cripple the availability of the backend.
To me, the truly generally useful (and secure!) part of this is simply in exposing utilities for girder+pgsql integration. That ends up being a pretty minimal layer, but I could see us reusing such utilities across many projects.

+1 to adding pgsql support to Girder as an alternate model layer. This
would be non-trivial to implement and maintain, but would lower the overall
burden on projects that use Girder and would benefit from pgsql support.

@cpatrick I've actually done a little bit of hacking with SQLAlchemy on the side to support this. I didn't go any farther than creating an alternate model layer and changing the AssetStore model to use Postgres instead of Mongo - but I agree it would be non-trivial to do it on the entire system.
I think the larger question though, is what is Mongo bringing Girder that Postgres couldn't? It seems like people want to use features like PostGIS and FTS to which Mongo's implementations just aren't as good.
On task, it seems like @mgrauer is right that this needs quite a bit of fleshing out in order to provide legitimate feedback. Some research time might be good to see if we can use logic and/or ideas from PostgREST and others.

I think the larger question though, is what is Mongo bringing Girder that Postgres couldn't?

Complete schemalessness. We don't even have to declare what tables (collections) we are using until runtime. It just alleviates the need for database scripts and schema upgrade/downgrade infrastructure. It also makes it extremely easy for plugins to modify core collections since indexes can be declared at runtime.

I included these in an earlier thread,  just for completeness I am going to
put them back here.  Please see:
https://wiki.postgresql.org/wiki/HTTP_API  for a proposal that illustrates
some of the complexity of this problem.  Also,  while we probably can't use
PostgREST (https://github.com/begriffs/postgrest) because it is written in
Haskell -  It is probably worth taking a look before we dive straight into
designing another tool from scratch.
/Chris
On Tue, Feb 16, 2016 at 11:53 AM, Zach Mullen<EMAIL_ADDRESS>wrote:

I think the larger question though, is what is Mongo bringing Girder that
Postgres couldn't?
Complete schemalessness. We don't even have to declare what tables
(collections) we are using until runtime. It just alleviates the need for
database scripts and schema upgrade/downgrade infrastructure. It also makes
it extremely easy for plugins to modify core collections since indexes can
be declared at runtime.
—
Reply to this email directly or view it on GitHub
https://github.com/Kitware/minerva/issues/135#issuecomment-184766933.

--
Christopher Kotfila
R&D Engineer
Kitware Inc.
http://www.kitware.com/company/team/kotfila.html

I certainly prefer this approach as a general solution. It might not meet @aashish24 's needs for GUMBO in particular, so we should consider that, but in general we don't want to expose arbitrary database connections to end users as girder would become a botnet very quickly. With your approach, we at least control the database connections, though malicious users could still construct queries designed to cripple the availability of the backend.

Does it sounds plausible to expose information on what backend storage (mongo, postgres) is used to store the data? If we know that the server side implementation can provide specific queries in native backend sql language for some complex geospatial queries on a folder or on an item. I understand that it will security concerns but I think unless you have some sort of layer like oracle has (views on db) I don't know if we can prevent bad code that is deployed on server to prevent running.

For a Postgres database using psycopg2, we know the data type of each column.  Simple data types lend themselves to generalized queries very easily.  For instance, GET /item/{id}/database/select?distance_min=4&distance_max=8&caption_regex=(fast|slow).
If is less obvious to me how we would want to expose PostGIS data types (and even the built in geometric datatypes).  For instance, if we call GET /item/{id}/database/select and specify a column that is of the geometry datatype, it is probably more useful to apply some ST_* function to that column then to perform a direct comparison.
For someone who is actively using the PostGIS functions, can you think of how you would like this exposed via a REST call?  For instance, I could easily see wanting to make a query like SELECT town FROM towns WHERE ST_Intersects(ST_SetSRID(ST_MakePoint(-72, 42.3601), 4326), ST_Transform(geom, 4326));, where town and geom are columns in the table.  It seems like this isn't particularly generalizable, as you need knowledge of PostGIS to construct such a query, and allowing general SQL makes it very hard to prevent injection attacks.
Thoughts, anyone?

For a query like:
SELECT town FROM towns WHERE ST_Intersects(ST_SetSRID(ST_MakePoint(-72, 42.3601), 4326), ST_Transform(geom, 4326));

How about something like GET /item{id}/database/select?intersects=Point(-72 42.3601)&columns=town
And the code would be responsible for forming the correct query, something like this:
def create_intersect_query(table, column, intersects_feature, epsg=4326)
	intersect_query = "SELECT %s from %s where ST_Intersects(ST_SetSRID(ST_GeomFromText(%s), {epsg}), ST_Transform(the_geom, {epsg}));".format(epsg=epsg)
	params = (table, columns, intersects_feature)
        conn = psycopg2.connection(....)
	try:
	c = conn.cursor()
		c.execute(intersect_query, params)
		c.fetchall()
	finally:
		conn.close()

This approach won't be able to handle every possible query that a user might want to make, but it helps avoid injection attacks and doesn't require the user to have deep knowledge of SQL.
I'm thinking of using a similar approach for Gaia.
-Matt

Here's a sample of the approach I've been taking with gaia, interested to hear what you think of it:
class PostgisIO(GaiaIO):

    def get_table_info(self):
        query = 'SELECT f_geometry_column, srid from geometry_columns ' \
                'WHERE f_table_name = %s'
        return self.pg_query(query, [self.table])[0]

    def get_query(self):
        self.geom_column, self.epsg = self.get_table_info()
        if self.columns[0] == '*':
            columns = self.columns[0]
        else:
            if self.geom_column not in self.columns:
                self.columns.append(self.geom_column)
            columns =  ','.join(
                ['"{}"'.format(x) for x in self.columns])
        query = 'SELECT {} from "{}"'.format(columns, self.table)
        if self.filter:
            filter_sql, filter_params = filter_postgis(self.filter)
            query += ' WHERE {}'.format(filter_sql)
        query += ';'
        return query, filter_params

def filter_postgis(filters):
    """
    Generate a SQL statement to be used as a WHERE clause
    :param filters: list of filters in the form of (attribute, operator, values)
    for example [('city', 'in', ['Boston', 'New York']), ('id', '>', 10)]
    :return: SQL string and list of parameters
    """
    sql_filters = None
    sql_params = []
    for filter in filters:
        attribute = filter[0]
        operator = filter[1]
        values = filter[2]
        if type(values) in (list, tuple):
            sql_filter = '"{}" {} ('.format(attribute, operator) + ','.join(
                ['%s' for x in values]) + ')'
            sql_params.extend(values)
        else:
            sql_filter = '"{}" {} %s'.format(attribute, operator)
            sql_params.append(values)
        if not sql_filters:
            sql_filters = sql_filter
        else:
            sql_filters = sql_filters + ' AND ' + sql_filter
    return sql_filters, sql_params

pgio = PostgisIO(
    table='towns',
    columns='town,pop2010',
    filter=[
        ('type', 'in', ['C','TC']),
        ('town','=','NORTH ADAMS')
    ]
)
query, params = pgio.get_query()
print query, params
conn = psycopg2.connect(pgio.get_connection_string())
try:
    cur = conn.cursor()
    cur.execute(query, params)
    print cur.fetchall()
finally:
    conn.close()

I've created a new repository here with my initial work on this.  It doesn't (yet) have any sort of UI, and doesn't (yet) support using functions as part of the queries.  There are probably other issues still that will be resolved as it gets developed.
This inspects the table to determine the available columns and their basic datatypes.  I've based some of the filtering from @mbertrand's example.
It includes some code from geoapp which can do polling on changing databases, though I don't think there is any application for that here yet.

Thanks @manthey we will have a look at it.

In terms of the database introspection,  is there a reason to implement (and eventually support) this ourselves rather than relying on several of the other hardened implementations that exist (e.g.  SQLAlchemy)?

I'm not sure what SQLAlchemy would gain us in this case.
Postgres reports whether, for instance, columns are fundamentally numeric or string or something else.  SQLAlchemy doesn't expose this -- you get the data type (such as "VARCHAR()" or "INTEGER()" or "NUMERIC(precision=10, scale=0)"), but then we'd have to figure out what the underlying datatype of those would be if we wish to handle them differently.  If we were using something other than Postgres, you'd have to know what that db's datatypes are separately.

SQLAlchemy gains us well documented well tested database reflection and querying that we're not obligated to maintain.  Can you expand on what you mean by "if we wish to handle them differently"?  In what cases would knowing the postgres specific datatypes be necessary?

If you know a datatype is numeric, then you know that comparisons should be done on numeric data types (4 < 10, but "4" > "10").  Depending on how datatypes are exposed, it might be nice if a comparison value is converted appropriately.
I'm not opposed to using SQLAlchemy, though I find it somewhat cumbersome.  I can't tell from the documentation, for instance, if it will cast a numeric column to match a text comparison value, or the text value to match the numeric column, or just pass both to the underlying sql engine and hope for the best.

Maybe this answers your questions?

@manthey the API looks great for the first part we discussed. How should I try it out? The part2 of this would be to support custom functions / queries as discussed.

@aashish24 sure, give it a try.  There is no UI for setting up the database access yet -- you have to set the appropriate data on the POST item/{id}/database endpoint.  Also, there are no tests (or, at least, no useful tests).
I could see, if we were ambitious, having a UI section on the item's page that would let you construct a query via a web form and get back a table of data.
Regarding custom functions: generically, it would be nice if we can support all non-volatile functions.  I know how to query Postres to get such a list (but I haven't found where it is exposed in SQLAlchemy).  As an alternative to limiting functions to those that are non-volatile (or, perhaps, in addition to), we can make sure all sessions with the database are read-only (but this isn't the same as not calling volatile functions, since things like setseed() can be called which change the state of the random number generator).

@manthey probably you have already seen this  but you can query the database directly as needed.

@aashish24 the lastest check-in supports functions.  I'll add tests next.  @kotfic I switched to using SQLAlchemy.  I still have to reach into postgres's internal tables to determine which functions should be allowed, as SQLAlchemy doesn't expose that, but otherwise everything is going through SQLAlchemy's orm methods.  This means that we can use any database SQLAlchemy supports (though we only support functions in the query in Postgres at the moment).

The feature has been implemented similarly as @manthey said above. Now girder server talks to  Postgres database defined with database_assetstore through database_assetstore and visualizes geospatial or tabular data from the Postgres. https://github.com/Kitware/minerva/pull/410


Angular 2 css not loaded

Hello,
I'm using the latest version of angular & formio,
I did all the instructions and even have look at the source code.
I am NOT using bootstrap, but I am using angular matrial.
The formbuilder component (I didn't try the rest, yet) loads without any css.
as any other library - I thought maby I need to import to the angular.json styles array costum css,
I found some issued about things like that but it seems to be very old and not relevant to the current version..
How am I suppposed to add the css?
thanks!

We currently do not support angular material, but just through the "materia" theme in bootstrap.

I understand, I must import bootstrap to the main project? There's any trick to import it only for this component?


Made Links prettier

hide github link behind text

Suggestion to change the link as suggested by @MikeMcC399 in https://github.com/corona-warn-app/cwa-website/pull/2206#issuecomment-996564271

@browkin
Thank you for picking up my suggestion to make the link appear prettier!
I did also suggest using a modified search string
https://github.com/corona-warn-app/cwa-server/pulls?q=is%3Apr+log4j+2.16.0+in%3Atitle
to narrow down the results. Unfortunately whether the original search string or the one above is used, there are several PRs which were closed (not merged) so it's more difficult to find out which ones are really relevant.
I wonder even if it is helpful to publish that search string. The person who suggested it may be able to explain why it was thought to be necessary instead of just stating that the issue has been remediated through a version update.

The search string above is not such a good suggestion because it misses one PR.
Stand by for an alternative.

Try this search string instead:
is:pr log4j in:title merged:>2021-12-01
https://github.com/corona-warn-app/cwa-server/pulls?q=is%3Apr+log4j+in%3Atitle+merged%3A>2021-12-01

@Ein-Tim
Good catch!
According to https://logging.apache.org/log4j/2.x/security.html there was a DoS vulnerability in Log4j 2.16.0.
Probably @ioangut should be consulted about the external message in https://www.coronawarn.app/en/faq/#log4j.
The search string I proposed
https://github.com/corona-warn-app/cwa-server/pulls?q=is%3Apr+log4j+in%3Atitle+merged%3A>2021-12-01
also pulls up and displays the 2.17 updates.

"Log4J" is incorrect capitalized in the title. It should be "Log4j" (lower case "j").
See https://logging.apache.org/log4j/2.x/

continues here https://github.com/corona-warn-app/cwa-website/pull/2214

@browkin
Please close this PR, since it has been replaced by

#2214.


Example gallery-like structure to see all plots at a glance

e.g. https://texample.net does this, where you can see a preview of each example on a single page, then click on it.  We used to have that in MakieGallery as well, which is still up at https://juliaplots.github.io/MakieReferenceImages/gallery !
It would be great to have something like that here, as a welcoming front page.  We could also possibly incorporate the documentation example plots in here, by parsing the docs!

Here's a list of todos which should encapsulate exactly what has to happen here:

adding the reference tests as their own category, so they're still integrated and searchable but somewhat separate.
enforcing the BeautifulMakie Literate.jl programming style for longer examples, which is really good.
allowing multiple tags.
creating the front page as a searchable, scrollable gallery, which has elements that redirect you to the appropriate page when clicked on.

In general, examples should:

Work when copy-pasted.  That means that most assets should be located in the Makie monorepo, within reason, or accessed through Downloads.download.
Be (mostly) backend-independent, i.e. using Makie.* instead of GLMakie.*.
Not visibly call functions which affect global state, i.e., activate!, unless necessary (these should be hidden)
There should be an interface where examples can declare their tags, and some autogeneration should be done, for example to show the package versions.

For the gallery overview I think we should follow the idea from here. Here is the issue I opened but never acted on it 😄 .  As for the other points, most are already covered, I think.

DemoCards.jl should be able to do that with a few tweaks...

mmm... now I do have a vue component https://github.com/MakieOrg/BeautifulMakie/blob/main/docs/src/components/Gallery.vue that handles the into gallery. My issue has been to get all paths and structure programmatically, maybe this could help. Or, replace it.

Oh yeah, we could definitely add a vue component style to DemoCards (it could even be done on our end so we don't have to wait for a PR there)!


Create Selemela07 devcontainer.json

Patch diff

thank you for helping to contribute!
Possible to help me understand what this PR does? I'll close it for now.


RestTemplate equivalent in jQuery/Thymeleaf for Consul/Ribbon exposed service

Hi,
I know in Java, we can use @LoadBalanced annotation for getting the Consul/Ribbon exposed services via rest template. Say I have a Spring Data Rest service registered in Consul. Is there any way I can call the service directly (using jQuery Ajax) via Consul/Ribbon url.
Thanks
Bharath

This is not the appropriate issue tracker for Spring Cloud questions. Looks like a good question for http://stackoverflow.com/questions/tagged/spring-cloud


Sheen values look too dark

Three GPU PathTracer
Model Viewer
DSPRT

cc @richardassar - I know it's been a bit since this was added but any thoughts on the discrepancies compared to the Model Viewer models? The car and couch look particularly dark.

I'll take a look, very busy at the moment with a few things but I will get to this when I have the oppotunity.
I do recall the kitchen towel model in Three GPU PathTracer looked more like what is currently in the DSPRT screenshot above, it had that purple sheen to it, so something might have changed since.
I'll let you know what I find.

Of course! Whenever you have a moment or interest would be helpful if you can. I may take a deeper look when I get a chance, as well. I did a quick survey of how that cloth example changed over the releases since sheen was added (and notes on why the changes happened) and it looks like v0.0.6 -> v0.0.7 is the meaningful release. v0.0.7 fixed diffuse lighting contribution from being too dark which meant the sheen contribution did not stand out as much as it did in the previous release - perhaps the sheen value is also being scaled down somewhere? Or diffuse contribution needs to be scaled down where sheen contribution is high?

version
screenshot
note

v0.0.6

v0.0.7

Base diffuse brightening correction likely from #255.

v0.0.8

Roughness and metalness textures broken by #298 causing incorrect parameters and base contribution darkness.

v0.0.9 to v0.0.12

Texture indexing fixed in https://github.com/gkjohnson/three-gpu-pathtracer/commit/2067567c346b2c250a40ee920119e4af39cacfa5.

DSPRT

Two thoughts after taking a deeper look in #382:

It seems possible that sheen roughness should not have the "perceptual roughness" squaring applied (see here).
The MeshPhysicalMaterial.sheen factor is not respected meaning you cannot turn sheen off at the moment.

Here's a comparison with and without the perceptual roughness applied on models where it makes a significant difference. The cloth model looks generally unchanged from this:

sheenRoughness ** 2
sheenRoughness
model-viewer

Note the weird hard-edged dark patches on the sofa are from a directional light casting hard shadows.

It seems possible that sheen roughness should not have the "perceptual roughness" squaring applied (see here).

It looks like three.js squares the sheen roughness but I'm also seeing that the sheen roughness value is being squared inside the sheen functions already here and here - so we're currently squaring the roughness twice.
And here's the couch comparison without the directional light for a better comparison:

Path Tracer
Three.js


a finally clause is allowed to do a function return

int test( int input )
{
    try {
        print("You gave me " + input);
    } finally {
        if (input == 100) {
            int output = input + 1;
            print("I prefer " + output);
            return output;
        }
    }
    print("Good enough!");
    return input;
}

print("I received " + test(50));
print("I received " + test(100));

yields
> finally.ash

You gave me 50
Good enough!
I received 50
You gave me 100
I prefer 101
Good enough!
I received 100

I don't like that. Now it yields
> finally.ash

You gave me 50
Good enough!
I received 50
You gave me 100
I prefer 101
I received 101

which is what I expected, when I wrote that code.

Could you add that example as a test? I'm pretty sure we already have (ash, output) tests somewhere.

I slightly augmented that test to include a test of the "abort" code path in a "finally" clause.
That gives me 100% code coverage of the code I changed in this PR. A first for me!


Projection on view does not have correct fields: Unit tests

This PR does not contain a solution yet, but contains testcases for issue #167.

@nils just to make sure: are you planning on implementing the fix for this on top, or is the scope for this PR just the tests? In the latter case, is it ready for review or still in draft?

@nils no worries, navigating through CSN can be quite the task and wrt projections the solution was not exactly straight-forward either. :)
I decided to extend a bit on the test suite you offered to cover some additional tests, like renaming and only selecting a subset of properties. If you find that these tests cover your case we can supersede your original PR (regretfully, as the provision of runnable test cases was highly appreciated!).

Thanks @daogrady for the efforts you put into this!
I have not tested your changes yet, but looking at the test suite the solution should also cover our issue.
I don't mind my testing code becoming obsolete, it served its purpose nevertheless 🙂 Your extended tests look great to me!


how to get the thumbnail ?

The read function will return an instance of SimplePie.  All you need to do is use the same commands you would use on that to retrieve the thumbnail.
Info


[BUG] Don't support the disk more than<PHONE_NUMBER> section

Describe the bug

Install failed with a 9.5T nvme disk
To Reproduce
Steps to reproduce the behavior:

Go to '...'

Expected behavior

Support bundle

Environment:

Harvester ISO version: harvester0.3 pre1
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630):  Baremetal

Additional context

May need to switch to GTP partitions.

@johnliu55tw  Is this tested on legacy BIOS or UEFI mode? For large disks, we might want to use UEFI mode (or turn on the COS_INSTALL_FORCE_EFI flag) because the installer defaults to msdos partition in legacy mode.

@yaocw2020 Is this tested on legacy BIOS or UEFI mode? For large disks, we might want to use UEFI mode (or turn on the COS_INSTALL_FORCE_EFI flag) because the installer defaults to msdos partition in legacy mode.

I don't know. It's the problem from a community user.

Most likely happens on BIOS. With the 2^32*512 bytes = 2TB disk size limit of MBR

Tested on a VM with UEFI firmware (ovmf-x86_64-code.bin), the partition table automatically becomes GPT.

changed to Unified Extensible Firmware Interface (UEFI)，still have issue.
it is me who raise this issue.

@tscswcn what is your Harvester ISO image version?

@tscswcn The error message is different from what's in the issue description. line 103 is:
parted -s ${DEVICE} mklabel ${PARTTABLE}, looks like the disk can't be written. Dumping your dmesg output might help.

@tscswcn what is your Harvester ISO image version, can u try the v0.3.0-rc1 if u are not using this one, thanks?

harvester 0.3- prew

@tscswcn The error message is different from what's in the issue description. line 103 is: parted -s ${DEVICE} mklabel ${PARTTABLE}, looks like the disk can't be written. Dumping your dmesg output might help.

how to do this for just install?

@tscswcn you can switch to another virtual terminal by Ctrl + Alt + F2 and log in with rancher/rancher to execute some commands during installation.

I'm on v0.3.0-rc1 and running into this, but my server has no UEFI support (this is with a 2.2TB disk

)
Most other operating systems I install seem to partition the disks as GPT by default. Is there a way to achieve this via some kind of preseed/config file or manual partitioning?

We need to fix this in v0.3.0. It's common for users to have a hard drive bigger than 2T (which around<PHONE_NUMBER> * 512).

We could force using GPT partition scheme on legacy BIOS system, as GRUB supports this. I've submitted a PR for cOS-tookit for this usage: https://github.com/rancher-sandbox/cOS-toolkit/pull/759


[Bug]: The koodo-reader app manifest is outdated

Prerequisites

[X] I have written a descriptive issue title.
[X] I have searched all issues/PRs to ensure it has not already been reported or fixed.
[X] I have verified that I am using the latest version of Scoop and corresponding bucket.

Package Name
koodo-reader
Expected/Current Behaviour
koodo-reader is not updated to the latest
Steps to Reproduce
# Not important

Possible Solution
Update the app manifest koodo-reader to the latest version, which version 1.6.5
Scoop and Buckets Version
Current Scoop version:
v0.3.1 - Released at 2022-11-15

'extras' bucket:
9c1f21c30 (HEAD -> master, origin/master, origin/HEAD<EMAIL_ADDRESS>Fix hash (Closes #12848)

'java' bucket:
0b128307 (HEAD -> master, origin/master, origin/HEAD) temurin8-nightly-jre: Update to version 8.0.412-4.0.202402231854

'main' bucket:
e4b19cc02 (HEAD -> master, origin/master, origin/HEAD) xmake: Update to version 2.8.7

'MyScoop' bucket:
eae3cc2b2 (HEAD -> master, origin/master, origin/HEAD) dopamine3: Update to version 3.0.0-preview.24

'nerd-fonts' bucket:
60b9f27 (HEAD -> master, origin/master, origin/HEAD) smiley-sans-dev: Update to version 20240219.b31b81a

'nirsoft' bucket:
860f46c (HEAD -> master, origin/master, origin/HEAD) Update url list [ci skip]

'nonportable' bucket:
f5f908e (HEAD -> master, origin/master, origin/HEAD) textpad8-np: Update to version 8.18.1

'versions' bucket:
6470dd869 (HEAD -> master, origin/master, origin/HEAD) ffmpeg-yt-dlp: Update to version 6.1.1-7-20240225

Scoop Config
last_update           : 2024-02-25 12:44:23 PM
aria2-warning-enabled : False
aria2-enabled         : False
cat_style             : auto
alias                 : @{reinstall=scoop-reinstall}
scoop_branch          : master
scoop_repo            : https://github.com/ScoopInstaller/Scoop

PowerShell Version
Name                           Value
----                           -----
PSVersion                      7.4.1
PSEdition                      Core
GitCommitId                    7.4.1
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

Additional Softwares
No response

Updated automatically in e730cb6.


hacky untyped fix for the duplicate answers problem

This has no tests and isn't typed, but it might be worth merging since it seems to fix a critical bug when demoing this stuff.
I'll leave it up to you to decide.
I think I problem is described pretty well in the story:
https://www.pivotaltracker.com/story/show/177333639
The technical issue is that the host code using the the storage system is assuming that the watchAnswer callback will be called after the createOrUpdateAnswer is complete.

I think @knowuh should do the review on this.


Added Infrared methods

[x] Issue #15 by @mahdoosh1

I have added methods for using infrared:
# Query the frequencies supported by the infrared transmitter
getfrequency()

# Transmit an infrared frequency pattern
transmit(frequency: int, pattern: str)

Thank you!


Asio, Libuv and Epoll are now inside uS namespace

Basically everything that is normally not namespaced in libuv should not be namespaced here either (UV_READABLE, uv_os_sock_t, etc). Wrappers like Poll are correctly uS-namespaced.

Done. Now uv_sock_t, UV_READABLE and UV_WRITABLE in global namespace again.


Fix gradle deprecation warning. Use doLast intead of <<.

This PR fix #758 issue.
Please take a look when you have a time.

I close this PR for rerun CI.

@hiroyuki-sato Thank you for taking. The change looks good to me. Do you have any concerns to merge this into master? Or do you have any remaining tasks for this?

@muga Thank you for the reply.
I have no remaining tasks. Please merge if you ok with that.

Thank you. Looks good. Merging.

Thank you!


Build error related to Page Bundles images after commit 27be1f6

An issue has arisen following the commit 27be1f6. Since this commit, I've been encountering a specific build error related to template rendering in my Hugo site that wasn't present before. The error occurs during the site rendering process, particularly when processing the list.searchindex.json template. Here's the error message:
Error: error building site: render: failed to render pages: render of "home" failed: "hugo-modules/search/layouts/_default/list.searchindex.json:25:67": execute of template failed: template: _default/list.searchindex.json:25:67: executing "_default/list.searchindex.json" at <.Page>: can't evaluate field Page in type string

Before this commit, my site was built without any issues. However, there was a pre-existing problem where images were not displaying in the search results for entries using Page Bundles. The front matter for these entries is typically as follows:
---
title: 'Some Title'
images: ['image.png']
...
---

For items where the front matter is formatted like this:
---
title: 'Some Title'
images: ['/images/image.png']
...
---

images display correctly in the search results, provided that the list.searchindex.json code is as it was before the commit 27be1f6.
It seems that the recent changes might have affected how Page Bundles and their associated images are handled or rendered. I would appreciate any assistance or guidance on resolving this error to restore the site's functionality and ensure images are displayed correctly in the search results.

Hey @alex-feel, thank you.


Shorthand for setting number of tasks

Hi,
I really like the macro addition and was wondering if it would make sense for You to include a shorthand for number of spawned tasks?
So having e.g. @set chunks=8 instead of @set scheduler = DynamicScheduler(; nchunks=8)?
It feels like an option that users would frequently want to change and this would make the code more clear and readable.
On a separate note, not sure how useful it would be in general, but I found myself frequently testing longer pipelines with many functions that have multithreading and having a global setting that changes them all at once makes the work much easier.
I have implemented an "config" struct in my packages that gets queried about parameters to run the code with, but maybe it would be useful to have something similar already here?

So having e.g. @set chunks=8 instead of @set scheduler = DynamicScheduler(; nchunks=8)?

Would you want a shortcut for setting the number of tasks or the number of chunks? Note that for DynamicScheduler, that's the same thing but for, say, GreedyScheduler it isn't. (In fact, as of now, GreedyScheduler doesn't even have nchunks but only ntasks.)
Assuming you want a shortcut for the number of tasks, let me first say that I've also had this though before. Given that we know call the loop macro @tasks it doesn't seem unnatural to have a ntasks option. However, at the time, I couldn't make up my mind whether and how we should add this.
What's your reason for requesting it? Are you trying to set ntasks<nthread() to, effectively, control the number of threads used? Or are you trying to set ntasks>>nthreads() to tune for load balancing?
(Note that with #60, I proposed a (kind of) alternative for the latter.)
As for your other idea, at least for now I don't plan to add a "global config" feature. A workaround is to just to instantiate a "global" Scheduler and use it in all @tasks blocks.

Right, I had ntasks in mind.
In my particular use case I rather go for ntasks < thread() - my workers almost always do identical work, so no need for load balancing. So I usually want control over the number of threads - mostly to test if multithreading works as expected or to find the optimum number above which adding more threads gives no benefit.
But yeah, as you point out, there might not be a single nthreads-like abstraction that could be used with all schedulers that you offer.

Assuming that make the following work
tmapreduce(sin, +, 1:10; ntasks=2)

what should happen if both ntasks and a scheduler is provided?
What would we expect to happen in the following cases?

tmapreduce(sin, +, 1:10; ntasks=2, scheduler=StaticScheduler())

Should this amount to StaticScheduler(; nchunks=2)? Or DynamicScheduler(; nchunks=2) (ignoring scheduler entirely)?  Or DynamicScheduler(; nchunks=2) (ignoring scheduler entirely)?  Or DynamicScheduler(; nchunks=2) (ignoring scheduler entirely)? Or throw an error?

My first naive thought was to keep the kind of scheduling (static, dynamic, ...) when scheduler is provided but simply ignore all the options set for the scheduler. I.e.
julia> tmapreduce(sin, +, 1:10; ntasks=2) # meaning DynamicScheduler(; nchunks=2)
_scheduler = DynamicScheduler{OhMyThreads.Schedulers.FixedCount}(:default, 2, -1, :batch)
1.4111883712180107

julia> tmapreduce(sin, +, 1:10; ntasks=2, scheduler=StaticScheduler())
_scheduler = StaticScheduler{OhMyThreads.Schedulers.FixedCount}(2, -1, :batch)
1.4111883712180107

julia> tmapreduce(sin, +, 1:10; ntasks=2, scheduler=StaticScheduler(; nchunks=5))
_scheduler = StaticScheduler{OhMyThreads.Schedulers.FixedCount}(2, -1, :batch)
1.4111883712180107

julia> tmapreduce(sin, +, 1:10; ntasks=2, scheduler=StaticScheduler(; nchunks=2))
_scheduler = StaticScheduler{OhMyThreads.Schedulers.FixedCount}(2, -1, :batch)
1.4111883712180107

My first naive thought was to respect the kind of scheduling (static, dynamic, ...) when scheduler is provided but simply ignore all the scheduler options. I.e.

I think I like this option, but it's hard to say. I could also imagine making it go the other way such that scheduler overrides the kwargs.

But maybe that's a cleaner API (error when we get ntasks + scheduler isa Scheduler and work if we get scheduler isa (valid) Symbol)?

I think this one is maybe best, that way there's no guessing about which one takes priority.

FYI: I made a (rather big) PR (https://github.com/JuliaFolds2/OhMyThreads.jl/pull/81) where I implement the no guessing strategy from above.

PR looks great, will definitely make it work for my use case, and I agree that no-guessing strategy won't make anyone upset but will prevent mistakes.
I think it's safe to close this issue now, thanks!


More fixes to show

It should

[x] not materialize the geometry column
[x] always show the units, even if NoUnits
[x] remove Missing with nonmissingtype
[x] show in html with a different method
[x] adjust styling options

Hello all , I would be happy to start with this issue , can you please provide sort of hint to how to resolve this issue and what it is related to .

Hi @Jay-sanjay we are already tackling the issue. Thank you for your interest. If you want to contribute with other issues in the stack, you are welcome.

@juliohm  Yeah sure , thanks for informing :)

Fixed on master.


Dependency hyprwayland-scanner missing from installation guide (Arch)

Total noob trying to give the project a try for the first time on a minimal Arch install (using KVM).
Here's the link to the section of the documentation where it shows the command:
yay -S gdb ninja gcc cmake meson libxcb xcb-proto xcb-util xcb-util-keysyms libxfixes libx11 libxcomposite xorg-xinput libxrender pixman wayland-protocols cairo pango seatd libxkbcommon xcb-util-wm xorg-xwayland libinput libliftoff libdisplay-info cpio tomlplusplus hyprlang hyprcursor

When I run make all it crashes because or missing dependency hyprwayland-scanner.
Hope this is helpful.

I'm dumb, wrong repo.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:30: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:30: host_section_needs_exact_pinnings: Packages in host must have exact version pinnings, except python build tools.
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:48: missing_pip_check: For pypi packages, pip check should be present in the test commands
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:54: missing_license_family: The recipe is missing the about/license_family key.
WARNING: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:54: missing_description: The recipe is missing a description
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:54: missing_documentation: The recipe is missing a doc_url or doc_source_url
ERROR: /tmp/abs_1aznyg22ou/clone/recipe/meta.yaml:54: missing_dev_url: The recipe is missing a dev_url
Errors were found


InputEventGesture blocks InputEventScreenDrag

Tested versions
4.2 stable
System information
Windows 10, Godot 4.2 Stable, Forward Mobile.
Issue description
InputEventFromWindow Inherited by : InputEventScreenDrag and InputEventWithModifiers(InputEventGesture).
Two of these events are located in different branches of InputEventFromWindow. Consequently, at the moment of finger touch on the screen, only one of these events can be active.
I cannot track the positions of two fingers on the screen if these two fingers are engaged in an InputEventGesture. Similarly, I cannot receive an InputEventGesture if two fingers are used in an InputEventScreenDrag. I cannot create a custom InputEventGesture based on the positions and distance between fingers obtained from InputEventScreenDrag. Is this just my misunderstanding, or is this really a limitation?
Using the built-in functionality of InputEventGesture and its Inherited by: InputEventMagnifyGesture, InputEventPanGesture is impossible, as each event has only one property. I can only obtain the average point between the fingers and a factor, which is unclear how to relate to anything.
For example, InputEventMagnifyGesture returns a factor that logically should be used for zoom. And indeed, this is possible, for instance: zoom *= zoom_factor. However, the result is very poor. No matter how much I try to find a way to tie it to camera2d to achieve a good zoom and simulate the effect of fingers sticking to the image, I haven't succeeded. This could be achievable by writing a custom script that uses DragEvent.
I'm not knowledgeable in these matters, but it seems to me that this problem could be solved either by adding more properties to Gesture events or by changing the inheritance and rewriting the code somehow. As I mentioned, I'm not adept at this.
Perhaps I'm missing something, but after a couple of days of testing, I still haven't found a way to achieve a good zoom using two fingers with the built-in functionality of InputEvents. Are there any solutions available to accomplish this now without delving into C++ or reverting to older versions when Gesture didn't work? Thank you.
A good zoom refers to zooming with two fingers, with the camera's position changing towards the center between the two fingers, and with the effect of fingers sticking to the image. I can achieve all of this using only InputEventScreenDrag, InputEventScreenTouch, and their properties. However, this is hindered by InputEventGesture. At some point, the engine may decide that I'm using two fingers not in InputEventScreenDrag, but in InputEventGesture, and everything breaks.
P.S. I've tried using InputEventGesture, but it provides too few possibilities, and as I've already mentioned, using it doesn't yield the desired result.
Steps to reproduce
n/a
Minimal reproduction project (MRP)
n/a

Sorry, nevermind, set setting to false helps:
input_devices/pointing/android/enable_pan_and_scale_gestures