Roadmap

I think the easiest solution is to consider these changes to be for a 1.0.0 release

[ ] Change all environments to use mujoco (and the gymnasium MujocoEnv)
[ ] Change to use Gymnasium rather than Gym
[ ] Add pypi releases
[ ] Fix the CI testing
[ ] Add environments to gymnasium.make

Proposed idea for environments
These ideas might be wrong but I would be interested in feedback on them
For the multi-task environments, we can consider there existing two "types" of environments.

A task environment where the agent has to solve a task associated with just that environment. On reset, the environment is reset for just this environment.
A *multi-*task environment that contains many task environments where the agent has to solve any of the sub-tasks. On reset, the multi-task environment will randomly select one of the task environments and use this task for the episode until the next reset.

I would propose that users use the multi-task environment from make, i.e., gymnasium.make("metaworld/multi-task-50"). Additionally, I think we should include the one-hot vector info either inside the multi-task environment that can be enabled or as a wrapper within metaworld such that users don't need to copy the previous implementation (this can allow the wrapper to be updated as API is).
This multi-task environment should be a very generic environment such that custom task environments can be used with it.
For meta-environments, we can consider there existing two "modes" of environments or even two different environments.

In "training" mode, the environment has variance in the goal space but it always has the same goal type.
In "testing" mode, the environment has a new goal type that is different

Similar to multi-task I think we can use make again with a parameter to specify the environment's mode, gymnasium.make("metaworld/meta-1", mode="training"). Then this environment can be passed any training library returning the trained agent. Then we can have an evaluate_meta_agent function that takes the environment and a function for the evaluating the agent policy given an observation and info.

There's a lot to unpack in that roadmap, and I definitely think we should take things step by step.

Fixing the CI should be relatively easy. It was working before it was moved to Farama, it's just broken now because it depends on some secrets that aren't set up yet.
Moving to the mujoco package would be a massive user improvement, and something I've been already looking at doing.
PyPI releases would be good to have as well. There's already a package, but it's never been updated.
Currently, Meta-World doesn't really use that much of gym either (the environments are not present in gym.make). It makes sense to move it to gymnasium. However, we should probably make sure that garage has gymnasium support before we do that change, otherwise no one doing Meta-RL or multi-task RL will want to use the new version.
For a set of environments to add to gymnasium.make, I think it would be best to add all of the environments listed in metaworld.envs.ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE and metaworld.envs.ALL_V2_ENVIRONMENTS_GOAL_HIDDEN (e.g. gymnasium.make('metaworld/pick-place-v2-goal-hidden')). We should probably set the seeded_rand_vec flag by default, which samples new goals on reset.

The rest of the proposed roadmap I'm not so confident in. For example, some multi-task RL algorithms (e.g. task embeddings) require the one-hot vector to not be present, so adding it by default seems unwise. There's also the complication that the meaning of "task" in MT1 and the meaning of "task" in MT50 is quite different, and I don't understand how you're proposing to handle those differences.
I very much think that it's a bit too early to be proposing entirely new ways of using the Meta-World benchmark, as the last part of this roadmap describes. There's already an evaluation protocol for Meta-World, and proposing a new one without clarity on how it improves on the previous one is likely to cause confusion.

Thanks for the feedback.
At least in my head, I would have thought we could create a super class for Meta and Multi-task envs.
This might not be possible but I think it would be a very helpful thing for the field to encourage other environment implementation and training algorithms that work for these environments

KR could you provide us with some insight into the correct evaluation protocol of Meta-World? For multi-task learning algorithms the evaluation should be on the training tasks, and for meta-learning algorithms the evaluation is on the held out test tasks? Both of these evaluations repeated a number of times to generate a success rate.
Could you also provide us with the distinction between task in MT1 vs task in MT50? The way that I understood it was that each environment is a task, and then each environment has 50 different initializations to simulate the initial state distribution of a singular environment. Therefore MT1 is a singular task, and MT50 contains 50 tasks, each with 50 different initializations per task.
I agree that enabling the seeded_rand_vec flag by default is a good idea. Better to have 1 environment per task where different initializations can be sampled by doing env.reset() rather than 50 environments per task where an initialization must be sampled from somewhere else.

I wrote a description about tasks, and how Meta-RL and Multi-Task RL differ in #393, that hopefully is more clear than prior explanations. Feel free to ask any followup questions you have.
The Meta-RL evaluation procedure is basically the following:
for task in test_tasks:
    policy = meta_algo.get_policy()
    for adaptation_steps in range(n_adaptation_steps):
        data = rollout(policy, task)
        meta_algo.adapt_policy(policy, data)
    task_performance = evaluate(policy)

The implementation used in the original Meta-World paper is here. However, it only supports one adaptation step (which is fine for RL^2 and PEARL, but means that MAML only runs one gradient step.


messagePump - ClientConnectionErrors Gone Wild

Apparently I had some sort of LAN network issue on remote vacation home that sorted itself out.   There were HASS communication errors to several other IoT devices but they were not nearly as pronounced as the comm errors from the S30 custom integration.   This is similar to issue #123 requesting improved error handling.   Not a big deal, just a lot of log spam.
3:50:47 PM – (ERROR) Lennox S30/E30 (custom integration) -
message first occurred at January 6, 2022, 5:39:39 PM and shows up 256 times

MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]

2:54:47 PM – (ERROR) /usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py -
message first occurred at January 6, 2022, 5:47:36 PM and shows up 112 times

2:54:07 PM – (ERROR) Lennox S30/E30 (custom integration) -
message first occurred at January 6, 2022, 5:39:19 PM and shows up 48 times

messagePump exception
2:54:07 PM – (ERROR) /usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py -
 message first occurred at January 6, 2022, 5:39:19 PM and shows up 48 times

2022-01-09 14:54:07 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - general error host [<IP_ADDRESS>] Code [5] Reference [5] [messagePump exception]
2022-01-09 14:54:09 ERROR (MainThread) [lennoxs30api.s30api_async] MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]
2022-01-09 14:54:09 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
2022-01-09 14:54:10 ERROR (MainThread) [lennoxs30api.s30api_async] MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]
2022-01-09 14:54:10 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
2022-01-09 14:54:20 ERROR (MainThread) [lennoxs30api.s30api_async] MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]
2022-01-09 14:54:20 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
2022-01-09 14:54:39 ERROR (MainThread) [lennoxs30api.s30api_async] MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]
2022-01-09 14:54:39 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
2022-01-09 14:54:47 ERROR (MainThread) [lennoxs30api.s30api_async] MessagePump clientResponse please report this information as an Issue - Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]
2022-01-09 14:54:47 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
2022-01-09 14:58:54 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]
2022-01-09 15:03:20 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]
2022-01-09 15:10:46 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]
2022-01-09 15:28:25 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]
2022-01-09 15:37:07 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]

Thanks for reporting.  There’s some diagnostic info we’ve been attempting to capture on some intermittent errors so this should help that. And yes, I agree it should log a bit less!

Just updated to  0.2.2/ 0.1.5 and checked logs to make sure all was well.    I was seeing a lot of the same errors as reported above indicating a communication error
2022-01-31 08:31:05 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [2] [ClientResponseError Error while executing request: [Content-Length can't be present with Transfer-Encoding]: error=<class 'aiohttp.client_exceptions.ClientResponseError'>, resp.status=[400], resp.request_info=[RequestInfo(url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'), method='GET', headers=<CIMultiDictProxy('Host': '<IP_ADDRESS>', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Python/3.9 aiohttp/3.8.1')>, real_url=URL('https://<IP_ADDRESS>/Messages/hass-lake/Retrieve?Direction=Oldest-to-Newest&MessageCount=10&StartTime=1&LongPollingTimeout=15'))], resp.headers=[None]]
I then pinged the device and confirmed that there was indeed a comm/network issue that was causing a intermitten high latency and packet loss...
--- <IP_ADDRESS> ping statistics ---
28 packets transmitted, 16 packets received, 42% packet loss
round-trip min/avg/max = 2.777/910.956/2077.017 ms

--- <IP_ADDRESS> ping statistics ---
37 packets transmitted, 8 packets received, 78% packet loss
round-trip min/avg/max = 10.243/1000.740/2001.655 ms

No other devices on the internal network were experiencing comm errors - just the S30.   I was just about to reboot the router then I noticed a new message in the log...
2022-01-31 08:31:08 WARNING (MainThread) [lennoxs30api.s30api_async] S30 has rebooted sysId [LCC] old uptime [1903320] new uptime [60]
Huh, the S30 apparently detected a problem and spontaneously rebooted?!   Indeed, all the comm errors cleared up and 0 packet loss now.  I assume the lennoxs30api did not force the reboot.    I'm wondering now if I need to contact service to have the S30 replaced?

Yes, I believe the S30 has some sort of built in health check / hardware watchdog - so that If the software crashes / runs out of memory, etc. in the S30 - it will reboot automatically and go back to controlling the HVAC system.  I believe the crash logs get delivered to Lennox via an api and hopefully they review and fix stuff.  There fix may be to turn that local port off.  I’ve disabled automatic updates on my S30, as I don’t want to lose local control.
We put that warning in to detect when it does this as we though it was related to enabling diagnostic mode to get the power inverter data.
The ping issues could be a network problem or it could have been the S30 getting hosed up.
Good information.  These also seem to manifest on restarts.

There is some diagnostic data that the S30 sends regarding the WiFi connection.  I have seen complaints on the Lennox forum about misbehavior when the WiFi connection is not strong enough.  I don't know your setup.  The Wifi Connection is to the box near the HVAC system (mine has an antennae on it) not to the panel.
If you enable message logging to a file.  Periodically they'll be messages output like this.  I believe the rssi is the signal strength.  Mine looks to be -60; which according to this is OK - https://www.metageek.com/training/resources/understanding-rssi/
My wifi is in the basement and the S30 in the sttic.  I'm in a quiet Wifi location (no neighboring houses. etc.)
I'll look at putting this info into a sensor as I work on handling these connection errors better.  In the meantime, if you have the disk space consider enabling message logging, to look at the rssi over time and maybe we'll see other diagnostic info from the S30.
    "messages": [
        {
            "MessageId": 0,
            "SenderID": "**redacted**",
            "TargetID": "ha_prod",
            "MessageType": "PropertyChange",
            "Data": {
                "interfaces": [
                    {
                        "publisher": {
                            "publisherName": "lcc"
                        },
                        "Info": {
                            "status": {
                                "macAddr": "**redacted**",
                                "ssid": "**redacted**",
                                "ip": "**redacted**",
                                "router": "**redacted**",
                                "networkStatus": "enabled",
                                "channel": 3,
                                "dns": "**redacted**",
                                "dns2": "",
                                "subnetMask": "**redacted**",
                                "bitRate": 72200000,
                                "speed": " 72.2 MBit per second",
                                "rssi": -60
                            },
                            "APDetails": {
                                "rssi": -60,
                                "security": "WPA-Personal",
                                "password": "**redacted**",
                                "ssid": "**redacted**",
                                "bssid": "58:6d:8f:ed:ea:be"
                            },
                            "diagnostics": {
                                "ssid": "**redacted**",
                                "bssid": "58:6d:8f:ed:ea:be",
                                "txByteCount": 10775785,
                                "rxByteCount": 0,
                                "ip4addr": "**redacted**",
                                "txPacketCount": 38219,
                                "rxPacketCount": 0
                            }
                        },
                        "id": 0
                    }
                ]
            }
        }
    ]

Thanks Pete for the help.    I had  "Debug logging logs messages" already enabled, but have now just enabled "Log S30 message to seperate file"  and entered just "S30.log".    Not sure if that will write to the root or to /config - but no matter, I have plenty of SSD space.
This S30 is at a vacation home with little to no neighboring WiFi interference.    The router is on the same floor as the iComfort hub just 15' away on a dedicated  2.4Ghz SSID with only 5 wireless devices.  I expect the signal strength to be quite good.
I'll get back to you as soon as I get a substantial amount of rssi data over the next 24 hours.

Debug running for nearly 8 hours now - log contains 660 instances of "rssi" - and all are -25, supposedly proving no variation in WiFi signal.
No lennoxs30api errors in the HASS log for over 6 hours, and then at 14:55 the comm errors started.  ping was showing very high latency and major packet loss. No watchdog reboot has occurred yet.
I just pinged all the other WiFi devices on this same SSID and all are responding very quickly, so there is no general network issue.
The S30 debug log does not show any errors that I saw.  It just shows gaps in the log when the comm errors are occurring.
At the moment - the S30 is not responding to the integration, to the mobile app, or to the iComfort portal.  Worst possible time for it to be acting up.   Severe cold snap approaching tonight.  Expecting a low of 4F and ice buildup.
Sure hope the TStat manages to keep the heat on independently of whatever this comm error is.   I do have other temperature sensors around the house that are monitored.

That does not sound like a good situation.  I do not know how to perform a remote reset of the S30, though if we can’t talk to it wouldn’t help anyway.   There have been no changes to the comms layer in 0.2.2 - we are just grabbing a couple additional fields out of the messages related to smart away.  But, the situation has gotten worse for you since that upgrade.
You may want to try disabling the integration and let it sit for a bit?
Sounds like you are able to VPN into that network.  The iOS phone app uses a direct connection to the S30 that is different than the path the integration uses.  If you can get it on that LAN, that is worth a try.
Any special firewalls, dns, VLANs that could prevent the S30 from communicating to the cloud?
What is the S30 firmware version?  That info is in the s30 state entity
What do you have app_id set to?  I have seen changing this can sometimes fix issues.  Try something like ha_prod
We finally got a warmup here (upstate NY) been killer cold.  -20 the other night.  I get the stress.

When you have time, please post the S30 integration settings (screen shots or the entry from .storage/core.config_entries)

Pete,  I seriously doubt this comm problem has anything to do with the latest S30 update, but I will disable it overnight.    Thanks so much for taking the time to review the issue and offering suggestions to try.
My S30 is also 3.81.207 and the hub is 3.81.777.  The app_id is HASS-Lake, and I just changed it to HASS-Lake2 and restarted.  (didn't help)
The S30 is able to connect to the cloud.  I can log into the iComfort portal and it initially shows offline, then briefly online with the temp gauge, then seconds later offline again.   This sort of matches what I see with the pings that extremely slow or not responding at all.
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
64 bytes from <IP_ADDRESS>: seq=187 ttl=64 time=2072.262 ms
64 bytes from <IP_ADDRESS>: seq=188 ttl=64 time=1072.217 ms
64 bytes from <IP_ADDRESS>: seq=189 ttl=64 time=72.141 ms
64 bytes from <IP_ADDRESS>: seq=190 ttl=64 time=204.287 ms
64 bytes from <IP_ADDRESS>: seq=191 ttl=64 time=57.101 ms
64 bytes from <IP_ADDRESS>: seq=192 ttl=64 time=336.933 ms

808 packets transmitted, 6 packets received, 99% packet loss
round-trip min/avg/max = 57.101/635.823/2072.262 ms

Config:
"entry_id": "82dff3c16f29d2668e3d890704deea18",
                "version": 1,
                "domain": "lennoxs30",
                "title": "Main Floor T-Stat",
                "data": {
                    "app_id": "hass-lake2",
                    "create_sensors": true,
                    "allergen_defender_switch": true,
                    "create_inverter_power": true,
                    "scan_interval": 1,
                    "init_wait_time": 30,
                    "fast_scan_interval": 0.75,
                    "protocol": "https",
                    "pii_in_message_logs": false,
                    "message_debug_logging": true,
                    "log_messages_to_file": true.
                    "message_debug_file": "S30.log",
                    "host": "<IP_ADDRESS>",
                    "cloud_connection": false
                },
                "options": {},
                "pref_disable_new_entities": false,
                "pref_disable_polling": false,
                "source": "user",
                "unique_id": "lennoxs30_<IP_ADDRESS>",
                "disabled_by": null

Glad it appears to be working and heating your house.
it does appear that the S30 is having problems.  As with any electronic gadget power cycling it would be a good idea.  It’s software reboot probably doesn’t do this and it looks like it’s a WiFi radio / chip set issue.
I see you have inverter power enabled, did you also set the diagnostic level on the S30 to obtain the power inverter data?  Thereis a number entity to do this.  If so I’d try to reset it to 0.

Yes, as long as the S30 is still functioning as a normal T-Stat and operating the Heat Pump at the last temp I had set (60F)  - I'm not too concerned about it for now.   I really had wanted to monitor the emergency heat and all.
Yes, I had created a number switch to enable the inverter.   It was set to "1".   I never used 2.
Interesting - the S30 is responding to pings now quickly.   Working as it should from the iComfort app.   I haven't re-enabled the integration yet.    I'll go ahead and re-enable it, but keep the inverter info off for now.

Yes, I would try restarting the integration and return diagnostic level to 0 as soon as possible and leaving it like that an assess stability.

I've done exactly that.    All is well for now.   I can monitor whole-house energy usage so will be able to derive emergency electrical heat spikes without the inverter-specific data.  I had forgotten this inverter option was an "alpha" feature.    So there is a chance that was affecting my connectivity to the S30 somehow (at level 1, not 2)?

Now seeing a new error...
2022-02-02 09:31:56 ERROR (MainThread) [lennoxs30api.s30api_async] executeOnUpdateCallback - failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py", line 992, in executeOnUpdateCallbacks
callbackfunc()
File "/config/custom_components/lennoxs30/sensor.py", line 92, in update_callback
self.schedule_update_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 643, in schedule_update_ha_state
self.hass.add_job(self.async_update_ha_state(force_refresh)) # type: ignore
AttributeError: 'NoneType' object has no attribute 'add_job'
2022-02-02 09:33:56 ERROR (MainThread) [lennoxs30api.s30api_async] executeOnUpdateCallback - failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py", line 992, in executeOnUpdateCallbacks
callbackfunc()
File "/config/custom_components/lennoxs30/sensor.py", line 92, in update_callback
self.schedule_update_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 643, in schedule_update_ha_state
self.hass.add_job(self.async_update_ha_state(force_refresh)) # type: ignore
AttributeError: 'NoneType' object has no attribute 'add_job'
2022-02-02 09:34:57 ERROR (MainThread) [lennoxs30api.s30api_async] executeOnUpdateCallback - failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py", line 992, in executeOnUpdateCallbacks
callbackfunc()
File "/config/custom_components/lennoxs30/sensor.py", line 92, in update_callback
self.schedule_update_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 643, in schedule_update_ha_state
self.hass.add_job(self.async_update_ha_state(force_refresh)) # type: ignore
AttributeError: 'NoneType' object has no attribute 'add_job'
2022-02-02 09:36:55 ERROR (MainThread) [lennoxs30api.s30api_async] executeOnUpdateCallback - failed
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/lennoxs30api/s30api_async.py", line 992, in executeOnUpdateCallbacks
callbackfunc()
File "/config/custom_components/lennoxs30/sensor.py", line 92, in update_callback
self.schedule_update_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 643, in schedule_update_ha_state
self.hass.add_job(self.async_update_ha_state(force_refresh)) # type: ignore
AttributeError: 'NoneType' object has no attribute 'add_job'

Re-enabling the integration was not as straightforward as it should have been as all the entities were still showing "disabled by device" after the S30 integration was enabled.   Had to manually enable the main climate entity and tried to enable the other sensors/switches but they were stuck disabled.    This may be the cause of the above errors.
I think the best thing now is to delete it and reinstall fresh.

Yes, that sounds like an issue.  Looks like that path needs more testing by me 😣
yes, I think delete and reinstall is the right approach.   We’ll want the power inverter sensor enabled so we get the diagnostic level number.  Once it is up set to zero.

update - reinstalled S30 integration from scratch - that went without a hitch.
Exactly 11 minutes after the HASS restart, connection errors started again....
2022-02-02 11:30:44 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration lennoxs30 which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant

2022-02-02 11:41:44 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [4] [messagePump - Server Disconnected]
2022-02-02 11:41:45 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [3] [messagePump - ClientConnectionError Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]
2022-02-02 11:41:46 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [3] [messagePump - ClientConnectionError Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]
2022-02-02 11:41:47 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [3] [messagePump - ClientConnectionError Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]
2022-02-02 11:41:48 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [3] [messagePump - ClientConnectionError Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]
2022-02-02 11:41:49 ERROR (MainThread) [custom_components.lennoxs30] messagePump_task - communication error to host [<IP_ADDRESS>] Code [11] Reference [3] [messagePump - ClientConnectionError Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]
2022-02-02 11:41:51 ERROR (MainThread) [custom_components.lennoxs30] reinitialize_task host [<IP_ADDRESS>] Code [11] Reference [3] [login failed due host not reachable url [https://<IP_ADDRESS>/Endpoints/HASS-Lake/Connect] Cannot connect to host <IP_ADDRESS>:443 ssl:False [Connect call failed ('<IP_ADDRESS>', 443)]]

I found the inverter number switch was reset to "1" after the restart so I changed this to 0, and haven't seen the connection errors for the past 15 minutes.   If this holds up, it likely implicates the S30 integration inverter sensor triggering a connection problem.   And after several hours of this - the S30 eventually goes haywire/offline?
BTW - the inverter sensor is still showing a value of 1,967 Watts which must have been the last reading.   Obviously this value needs to be ignored once the inverter/number switch is zero.

Good, sounds like we may be getting back to a point of stability.  Could just be luck..
Let’s run it in diagnostic mode zero for a while (week or so) which will give us a good assessment.
At level 0, the data needed by that sensor does not get delivered hence it will hold that value.  So yeah, it’s not going to be very useful right now.
I had observed instability on my S30 when using diagnostic mode.  The code was a contribution and appears to be working fine on his system.  I believe his system does not have a Lennox account and hence does not send data to the cloud.  Hence I’ve marked it as alpha until I get feedback that it is stable in more than one system.
I believe that every client has an independent subscription - e.g. the integration, each mobile app,… Whenever data changes it is sent / queued to each client.  Clients do not always disconnect well, so for example if the Mobile app is closed or the integration is stopped, messages for these clients remain in the queue for some amount of time.
When diagnostic mode is enabled much more data is generated and in your system the production of data may be exceeding the consumption and then we see this cascading set of failures.
Lennox has a diagnostic app that runs on tablets. The way it is supposed to be used is this.  Technician shows up at your house, they press a button on the S30 controller that enables a temporary WiFi network (and hence disconnects from your wifi.). The app connects to the 443 port, send command for diag level 2 and then the data streams into his tablet for display.

@GaryOkie - has the system remained stable since the change to diagnostic level?

sounds great - thanks for reducing the error messages, especially those that trigger python tracebacks upon a timeout.
I don't have my S30 blocked from the Internet at this time, but I do have it configured for  local only mode.

How do you configure it for local only mode?
If you want to try to get that power inverter working, I think we can say diag level 1 is problematic.  You could try diag level 2 or not…

I just set up the integration using an IP address and did not provide a Lennox ID/password.   That's what I meant by "local only" from a HASS/S30 integration perspective.   Of course, since it is not blocked from the Internet, the S30 will phone home to provide stats and for remote portal and app access, so in that sense, it sure isn't local only.   Sorry I wasn't clearer.
This reminds me - during troubleshooting I wanted to try a different subnet, but it sure wasn't obvious how to change the IP without reinstalling from scratch.  I did find a way by editing the .storage core entity file, but shouldn't the IP address (and optional user/password) be part of the config flow UI settings?

Yeah, I think we can allow that.  Let’s add an issue.  I edit the storage file, no issues with doing that.

That would be helpful updating the config UI, as editing .storage files is similar to editing MS registry.  No issue if you know where to look!
I'll leave my system at diag 0.   I need stability for now while away.

This issue should be fully resolved in the  0.2.4 release; which will be released by tomorrow.


Using negative examples

Manually annotated hard negative examples are used in training. Actual sampling from the image is done batch for batch to avoid excessive memory usage and allow for data augmentation in the future.

Please squash 6923740 and e04cec1

main.ipynb:

The visualization figure would be clearer with a legend
Run the last cell so the results are visible when you open up the figure on GitHub

With the modification it might be a cleaner design to have generate_training_data return the minibatch generator; but this would make the visualization plot a bit messier, it is up to you.

I can update the image in readme.md  (the source file is not checked in).

With the modification the parameter names positive_sample_radius and negative_sample_radius are a bit confusing as positive_sample_radius is used for the hard-negatives.
An idea is to rename

positive_sample_radius -> sample_radius
negative_sample_radius ->  edge_radius,
or something in that spirit

Great additions, thanks!

Superseded by PR #3


Any plan to support vmess?

thanks to your contribution.

vmess is complicated with a user management system.
It doesn't have priority now.


Setting SerializerSettings.ContractResolver to DefaultContractResolver, fails to load swagger UI

Following the announcement for breaking change in ASP.Net JSON serialization behavior, I updated my startup.cs to use -
 services
        .AddMvc()
        .AddJsonOptions(options => options.SerializerSettings.ContractResolver = new DefaultContractResolver());

After this change, the swagger UI fails(at /swagger endpoint) to load in the browser. If I revert back the above change, the swagger UI is displayed properly.
What is the correct way to let swagger know that we want to use DefaultContractResolver, and not camel case one?
My project.json is using - "Swashbuckle.AspNetCore": "1.0.0-rc1"

@abhisek-sinha - this doesn't repro with any of my sample applications. There must be something specific to your setup. Can you enable the dev. exception page (i.e. UseDeveloperExceptionPage) and then navigate directly to the Swagger JSON at /swagger/v1/swagger.json. This should provide details of the server-side exception and might help troubleshoot.
With just the limited info you've given above and without a repro, there's really not much else I can do to assist without the additional exception info.

Thanks for your reply. I did not try to see what happened at server, but in browser, IIRC, I think the swagger endpoint array was empty, and so it couldn't load the UI.
But I'll enable the exception page, and provide you details on what happens there.

I am disappointed that I couldn't get the problem to show up again. There had been some other changes to the project, so I am not sure, if there is anything else that's affecting the behavior. And I wasn't able to restore the project to the date when it was showing this problem.
I think it is OK to close this issue now. I would revert back if the problem shows up again.


Support building chip-tool with python3.12

This updates 2 python packages to versions compatible with python3.12, fixing the following errors when running
./scripts/examples/gn_build_example.sh examples/chip-tool BUILDFOLDER:

pyyaml 6.0 build error: AttributeError: cython_sources, fixed here

construct import error:
File "/tmp/pip-install-ifgk4tul/construct_6d60304f85e249759f6c184ea89f1317/construct/core.py", line 3, in <module> import struct, io, binascii, itertools, collections, pickle, sys, os, tempfile, hashlib, importlib, imp ModuleNotFoundError: No module named 'imp'
this was fixed here, in version v2.10.57 of construct. I'm just updating to latest though.

Now running ./scripts/examples/gn_build_example.sh examples/chip-tool BUILDFOLDER succeeds when the host python is python3.12.

[!NOTE]
SiLabs patched their fork for prompt-toolkit (already upstreamed though):
https://github.com/SiliconLabs/matter/pull/147/files
Nordic's fork needs that patch too, they'll get it after upmerging.

Fixes #32589.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Can specify which database connection to use

Bump on this one!, I need the specify-db-connection capability

Great @ian-nisbet


Added Kubernetes Contexts page docs.

Added steps to verify Kubernetes Contexts page functionality, that has already been implemented.

@rostalan Thank you


Add log rotation

The logfiles can get large, especially if there's a lot of fingerprints and/or -debug has been turned on. (I'm thinking of main.stdout).
It may be worth looking into automatically rotating this.
Maybe something like lumberjack? Seems like an easy swap.
Alternatively, if we want to push this onto the user, we should mention this explicitly in README.md.

Seelog does this: https://github.com/cihub/seelog/wiki/Receiver-reference#rolling-file-writer-or-rotation-file-writer
This will be easy to implement.


install without composer

Hi guys, is there guide on how to install to my php without using composer?
Thanks a lot.

This package has extra dependencies that need composer. Run composer require [package_name] and require the vendor/autoload.php in your project.

@rennokki  Got it. Thanks a lot for your reply.


Stuck after Loading window finishes

Im able to get the loading window on macos, and it finishes correctly, but then when minecraft actually starts, its stuck and doesnt open, the icon stays saying "not responding".  The weird thing is that without the loading window mod the modpack actually loads, and testing the window loading mod alone makes minecraft open, so I bet its a mod that causes a conflict somehow.  I will leave the logs just in case.
latest.log
It looks as if it was halfways there but the problem is, that the cpu usage is usually high during the startup, with window loading mod, the cpu stays high for a bit and then drastically drops and now minecraft is using almost no cpu and cannot continue initializing the game.  Remember that without this mod the game opens perfectly, so it may be a mod compatibility or the mod itself.

Can you upload the ipc-client-log.txt file from the .cache/mod-loading-screen folder inside your instance folder?

fixed, removing craft-pressence solves the issue

btw I renamed background.gif to background.png and I all see is a gray background when opening the game, am I doing something wrong?

Can you upload a log with your launcher's built-in upload log functionality (if it has one)?

ipc-client-log.txt

The game loads fine with the cpu being high, but when getting to this point, the game refuses to fully start.

What modpack are you using? I want to try to repro.

Modrinth doesnt let me upload .mrpack so I zipped it
CoreCraft Reimagined 1.0.0.mrpack.zip](https://github.com/user-attachments/files/16769653/CoreCraft.Reimagined.1.0.0.mrpack.zip)

Apparently I cant enter to the game's main loop.
This is the log without the mod

Do you get the same output?


testing jth#635

Test the Jenkins test harness
testing https://github.com/jenkinsci/jenkins-test-harness/pull/635
(abusing a plugin that is using CI with a number of forks != 1)

Closing after two weeks with the assumption that @jtnord can reopen if it is needed.

Sorry @MarkEWaite I should have closed this earlier.


Final tensorflow loss for results reported in paper

Hi @abisee ,
I'm training the network and the tensorflow loss is going down, but really really slow. Frankly, I have always worked with CNN's and they're much faster to train. I just want to get a sense for what is a good value for training loss at which point I can say that the network has learnt something meaningful. I read the issues and you mentioned your model had 230k iterations to reach the results of the paper, and that you had a learning rate schedule where you changed the size of hyperparameters to speed up initial iterations. I'm going for more of a brute force approach right now as I'm trying to get a proof of concept ready for my project, and not final results.
I'm currently at tensorflow loss of about 2.2 after 2-3 days of training. It started originally at over 6. What is a good value to stop, or what was your model's value for the results in paper?
Thanks :)

Yes, RNNs are very slow to train, especially for long sequences (such as in this project), due to the sequential nature of the recurrent connections. I assume by "brute force", you mean you're not using smaller values of max_enc_steps and max_dec_steps. If you're just looking for a "proof of concept" and slowness is your main problem, I'd advise you to try the smaller values of max_enc_steps and max_dec_steps.
The best way to gauge the quality of your learned model is to run decode mode and look at the output. You can compare it qualitatively to the output provided in the paper and/or the complete output on the test set linked in the README.

Thanks for the prompt reply @abisee!
Yes that's what I meant, I am not using smaller values for max_enc_steps and max_dec_steps for now. I am qualitatively tracking progress by decoding as you suggested. But it would be great to know what your value of tensorflow training loss was when you finally stopped training, if you can somehow find the number in your training log files. Will help me out significantly :)
Thanks a lot :)

@Spandan-Madan looking at the smoothed loss curve according to Tensorboard, the training loss was about 2.8 after about 230k iterations, before we turned on coverage.

Hi @abisee, Thanks a lot! This will be quite helpful. By the way, what do you mean by turning on "coverage"?

@Spandan-Madan "coverage" is one of the main ideas of the paper. See also these flags.

@abisee Oh Damn I remember now, It's been over a month and a half since I read it so it totally slipped my mind sorry! Thanks a lot for pointing to the code :)

@abisee can you let us know what was the final running_avg_loss and validation loss for your model? For me, it seems that validation loss stops decreasing after a while and becomes almost flat while the training loss keeps decreasing (overfitting?). Was wondering if you experienced the same trend.


unify .register syntax, add actionable date example in JS

FWIW I think all the quotes in the documentation / comments should be double (") not single (') - do you think it would be a lot of work to change all the instances of this? might be nice to have some consistency

at least let's make your change use double qs

You should use single quotes since a majority of the code utilizes single quotes. Consistency is the primary purpose of choosing between single vs. double in JS.

overruled by SDK owner.... so LGTM!

LGTM, though I'm holding off in case you want to change your whitespace here before I merge. If you do, that'd be a great time to practice your rebase-and-squash git-fu!

merged as squashed 0c02bff


no mention of submariner engine pod in the docs here , should be though since its a crucial component...

I would excpect it to be explained in :
https://submariner.io/architecture/

so apparently it's synonymous to the deamonSet pod running on the designated gateway node
specified here : https://submariner.io/architecture/gateway-engine/

It is explained here


CAMEL-13316: Olingo components handle value/property results

When split, both olingo components handle results which are values or
properties rather than entities.

Provides tests confirming the fix.

Thanks, merged on master


MemoryError : Dask DataFrame .apply

I am running .apply on a dask dataframe with 8 partitions and observe the following.
Does this indicate I am out of disk space, the systems (Windows10) shows a python out of memory error and the systems is not functional with a reboot.
C:\p\x1\lib\dask\async.py in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
482                 _execute_task(task, data)  # Re-execute locally
483             else:
--> 484                 raise(remote_exception(res, tb))
485         state['cache'][key] = res
486         finish_task(dsk, key, state, results, keyorder.get)
MemoryError:
Traceback
File "C:\p\x1\lib\dask\async.py", line 267, in execute_task
result = _execute_task(task, data)
File "C:\p\x1\lib\dask\async.py", line 248, in _execute_task
args2 = [_execute_task(a, cache) for a in args]

Yeah, probably you have too much data in RAM.  You could try using smaller/more partitions.

Thank you. Where does DASK store temporary and intermediate files?
On Mon, Jul 18, 2016 at 8:53 AM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

Yeah, probably you have too much data in RAM. You could try using
smaller/more partitions.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1385#issuecomment-233319705, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXVTXj8MhXbQTosEgJK2y3zOgYighsiks5qW3c2gaJpZM4JOi5w
.

Dask only stores intermediate data in memory.  It manages larger-than-memory computation by streaming through the input intelligently and removing intermediate data as quickly as possible.  For computations that require shuffles it does use a temporary file, but this is uncommon.
Perhaps if you include a minimal failing example that fails for you then we can help to identify what your bottleneck is.

The entire dataframe fits in memory, ~5GB, after adding the two additional
columns it becomes a 7.9GB pkl file. The system has 32GB memory, about 27GB
free
On Mon, Jul 18, 2016 at 8:53 AM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

Yeah, probably you have too much data in RAM. You could try using
smaller/more partitions.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1385#issuecomment-233319705, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXVTXj8MhXbQTosEgJK2y3zOgYighsiks5qW3c2gaJpZM4JOi5w
.

Without more information about what you're doing I'm not sure how I can help you

import dask.dataframe as dd
import numpy as np
import pandas as pd
from pyproj import Proj, transform
inProj = Proj(init='epsg:4326')
outProj = Proj(init='epsg:3857')
def wgs84_2_webmercator2(row):
xwm,ywm = transform(inProj,outProj,row['lon'],row['lat'])
return pd.Series([xwm,ywm])
%time df=pd.read_pickle(r'D:\nyc_taxi\trip_data_pickup_pd.pkl')
mask=((df['lat']>-90.0) & (df['lat']<90.0) & (df['lon']>-180.0) &
(df['lon']<180.0) )
df=df[mask]
df['xwm']=1.0
df['ywm']=1.0
df['idx']=df.index
ddf = dd.from_pandas(df, npartitions=8)
ddf[['xwm','ywm']]=ddf.apply(wgs84_2_webmercator2,columns=['lon','lat'],axis=1)
%time ddf.compute()
On Mon, Jul 18, 2016 at 3:08 PM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

Without more information about what you're doing I'm not sure how I can
help you
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1385#issuecomment-233427405, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXVTd4Z0--vodS3gnMWhGYEGTqvg7zyks5qW89BgaJpZM4JOi5w
.

And does that work with just pandas if you don't use dask.dataframe?

Ping.

The call I used was:
%time df[['xwm','ywm']]=df.apply(wgs84_2_webmercator,axis=1)
On Tue, Jul 26, 2016 at 12:03 PM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

Ping.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1385#issuecomment-235316696, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXVTU7_ehkGw4L9Ej2f6H_E2VoeOhkvks5qZi_SgaJpZM4JOi5w
.

And that works fine in pandas, yes?

yes, it is working, what I am using for now.
On Tue, Jul 26, 2016 at 2:03 PM, Matthew Rocklin<EMAIL_ADDRESS>wrote:

And that works fine in pandas, yes?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/1385#issuecomment-235353171, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXVTSf0R3vjOpsnjLuYcYPZ52q6v7DOks5qZkvqgaJpZM4JOi5w
.

Closing for now as stale. Feel free to reopen if the issue persists and you can provide a reproducible example.


Notes aren't properly escaped in Python, security concern

Among the active forks of ulogme, yours is the only one that claims ulogme is secure.
Please see: https://github.com/karpathy/ulogme/issues/48
Sidenote: @karpathy released this under MIT first, so I'd suggest you add credit to him wherever you have credited yourself as well? Unless you have an agreement with him?

Hi,
Thanks for your remarks.
I forgot to mention any other contributors, sorry (I copy-pasted a piece of Markdown in the README.md, without remembering it was a fork)... That's fixed (97e3135).
About the security, I changed my claimed, and I tested and merged your suggestion (https://github.com/karpathy/ulogme/issues/48), cf. (c7cca4f). Thanks!


feat: Anvil Widget Name Component

Description

Widget name component and associated utilities. Feature reference: https://ce-32868.dp.appsmith.com/

Product and engineering requirements are listed in the issue description.

Fixes #33074

Automation
/ok-to-test<EMAIL_ADDRESS>:mag: Cypress test results

[!IMPORTANT]
🟣 🟣 🟣 Your tests are running.
Tests running at: https://github.com/appsmithorg/appsmith/actions/runs/8922613170
Commit: cc70b7d8cd35f66810eb9a9ce7bb51d2871cc945
Workflow: PR Automation test suite
Tags: ``

Communication
Should the DevRel and Marketing teams inform users about this change?

[ ] Yes
[x] No

/build-deplooy-preview skip-tests=true

/build-deploy-preview skip-tests=true


It needs to be handled in this way, otherwise an error will be reported

It needs to be handled in this way, otherwise an error will be reported:

VS 2019

Please explain why?

Please explain why?

can try this on

can try this on

This is because you are including Windows.h before HTTPRequest.hpp without defining WIN32_LEAN_AND_MEAN first.

This is because you are including Windows.h before HTTPRequest.hpp without defining WIN32_LEAN_AND_MEAN first.

Ok, added parentheses in https://github.com/elnormous/HTTPRequest/commit/bc1301f731306276af47ad9f78081a0fd504520c

Ok, added parentheses in https://github.com/elnormous/HTTPRequest/commit/bc1301f731306276af47ad9f78081a0fd504520c


Multiple item (entity) types

Is it possible to have multiple item (entity) types, for example article and event, and for an article to get similar articles and for an event to get similar events?

yes. you can use different entityTypes and modify engine data source to read the corresponding entityTypes for training (https://github.com/PredictionIO/template-scala-parallel-similarproduct/blob/v0.3.2/src/main/scala/DataSource.scala#L68)
Or you can set those as "categories" of item and specify categories you want in the query.

Thank you for the answer!


Dependabot cannot parse rust-toolchain file

This month I didn't get as many crate upgrade PRs as I thought I should have. The only ones were in the spec-runner workspace.
Checking the "Dependency Graph" settings on this repository brings me to https://github.com/artichoke/artichoke/network/updates/671999075 and this error message:

Dependabot can't parse your rust-toolchain
Dependabot failed to update your dependencies because there was an error parsing the rust-toolchain found at /rust-toolchain.
Dependabot encountered the following error:
only rust-toolchain files formatted as TOML are supported, the non-TOML format was deprecated by Rust

Learn more

Update Logs

 Log output
  proxy | 2023/06/02 20:15:33 proxy starting, commit: d719b9b9871e853c5fd63bf3552f3f857d979f3c
  proxy | 2023/06/02 20:15:33 Listening (:1080)
updater | 2023-06-02T20:15:34.222397645 [671999075:main:WARN:src/devices/src/legacy/serial.rs:222] Detached the serial input due to peer close/error.
updater | time="2023-06-02T20:15:37Z" level=info msg="guest starting" commit=8ab4a20db815b67034070152643b9878c12b051d
updater | time="2023-06-02T20:15:37Z" level=info msg="starting job..." fetcher_timeout=10m0s job_id=671999075 updater_timeout=45m0s updater_version=49704c16cb0893c0ab8c5f884471c324baf39b83-cargo
updater | 2023/06/02 20:15:40 INFO Raven 3.1.2 ready to catch errors
updater | 2023/06/02 20:15:42 INFO <job_671999075> Starting job processing
  proxy | 2023/06/02 20:15:42 [002] GET https://api.github.com:443/repos/artichoke/artichoke
  proxy | 2023/06/02 20:15:42 [002] * authenticating github api request with token for api.github.com
  proxy | 2023/06/02 20:15:43 [002] 200 https://api.github.com:443/repos/artichoke/artichoke
  proxy | 2023/06/02 20:15:43 [004] GET https://api.github.com:443/repos/artichoke/artichoke/git/refs/heads/trunk
  proxy | 2023/06/02 20:15:43 [004] * authenticating github api request with token for api.github.com
  proxy | 2023/06/02 20:15:43 [004] 200 https://api.github.com:443/repos/artichoke/artichoke/git/refs/heads/trunk
  proxy | 2023/06/02 20:15:43 [006] GET https://api.github.com:443/repos/artichoke/artichoke/contents/?ref=b3499bc1e377d3acf7ae3eb1f62ebda620674ede
  proxy | 2023/06/02 20:15:43 [006] * authenticating github api request with token for api.github.com
  proxy | 2023/06/02 20:15:43 [006] 200 https://api.github.com:443/repos/artichoke/artichoke/contents/?ref=b3499bc1e377d3acf7ae3eb1f62ebda620674ede
  proxy | 2023/06/02 20:15:43 [008] GET https://api.github.com:443/repos/artichoke/artichoke/contents/rust-toolchain?ref=b3499bc1e377d3acf7ae3eb1f62ebda620674ede
  proxy | 2023/06/02 20:15:43 [008] * authenticating github api request with token for api.github.com
  proxy | 2023/06/02 20:15:43 [008] 200 https://api.github.com:443/repos/artichoke/artichoke/contents/rust-toolchain?ref=b3499bc1e377d3acf7ae3eb1f62ebda620674ede
updater | 2023/06/02 20:15:43 ERROR <job_671999075> Error during file fetching; aborting
updater | 2023/06/02 20:15:44 INFO <job_671999075> Finished job processing
updater | 2023/06/02 20:15:44 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +-------------------------------+
updater | |            Errors             |
updater | +-------------------------------+
updater | | dependency_file_not_parseable |
updater | +-------------------------------+
updater | time="2023-06-02T20:15:44Z" level=info msg="task complete" container_id=job-671999075-file-fetcher exit_code=0 job_id=671999075 step=fetcher
updater | time="2023-06-02T20:15:44Z" level=warning msg="failed during fetch, skipping updater" job_id=671999075

The playground is also impacted:

https://github.com/artichoke/playground/issues/1065

I poked dependabot to check for updates and it looks like it is happy again:


Use table visitor when inserting data

fixes issue with insert statments not using schema/catalog

This bug was already fixed by #209.

Should have updated master :facepalm: - thanks

My bad, I should have checked the other pull requests.


Use Case: Post to & fully populate a repository record

As a data creator, I want to be able to post/upload an RO-Crate to a repository and have it populate the repository metadata, so that I can automate my processes and don't have to re-key slight variations of the same information again, and again, and again.
This could be accomplished in a number of ways, e.g.:

A given repository supports a deposit protocol like Sword (v1, v2, v3), AND accepts/understands RO-Crate format so that the repository can parse and map the package to its own internal representation (requires adoption by the repository)
A given repository adds support for RO-Crate to its bespoke REST API, (e.g. Zenodo, Figshare) so that the repository can parse and map the package to its own internal representation (requires adoption by the repository)
A given repository adds support for RO-Crate when integrating with other platforms, e.g. the Zenodo and Figshare integration with Github hooks, but without having to manually populate the metadata in the repository (requires adoption by the repository)
A separate tool/library which can parse an RO-Crate and map the RO-Crate representation either to a generic Sword package or to the specific API of a given repository (does not require adoption by the repository, but the tool(s)/libraries must be maintained etc.)

Note, there is a whole other discussion about how the repository treats the upload, i.e. does it maintain the original RO-Crate as the Archival Information Package, or does it simple parse & unpack what it needs and discard the rest of the RO-Crate structure, or both. Sword v3 is a useful starting point thinking through these scenarios, but they are beyond scope of the RO-Crate spec itself

These look good, but the question is how can we support this as specification designers/writers? The answer is probably in providing examples of how data can be extracted from an RO-Crate. Eg at UTS we are working on an indexer that builds a Solr based discovery portal for data - we can link to that code which will demonstrate how to traverse the JSON-LD @graph and extract info.


Colors not working correctly on Windows

OS: Windows 11 Pro
OS Build: 22000.434
Sample output from testing Colorful.Console.WriteLine(string, Color) gives me:

That dark blue is always the first color, no matter what. After cloning the repo to debug, I found that GetConsoleColorNative is the part that's returning incorrect results. Manually changing it to take the color.ToNearestConsoleColor() path works fine, of course.
Just wanted to report for visibility. Thanks for all your hard work on this library!

Can you please elaborate, please? I just downloaded it to implement it in a version of Mastermind I'm programming and I encountered the same issue.
Having this code...

... gets this result:

Can you please explain what you did?

Since my use case is trivial, I decided to just create a utility method instead of using this library.
using System;

public class Program
{
    public static void Main()
    {
        WriteInColor("Hello World", ConsoleColor.Green);
    }

    private static void WriteInColor<T>(T value, ConsoleColor color)
    {
        var oldColor = System.Console.ForegroundColor;
        System.Console.ForegroundColor = color;
        Console.WriteLine(value);
        System.Console.ForegroundColor = oldColor;
    }
}

I'd be happy to help the author create a fix in the future, but I think it requires discussion.


I wonder if we should add an EAP version to the build matrix

Could be an effective way to catch breaking changes in new releases. It's a bit expensive though to do it on a per PR basis.

This is done.
https://cloud-tools-for-java-testing.appspot.com/job/google-cloud-intellij master daily build orb (IDEA EAP)/


Align the logo on the heading with the text on README.md

Might just be OCD, but I don't like how it currently is where the logo's position is higher than the text. Figured this might fix it. Written in a bit of a hurry, better fixes are appreciated.
A view of the change if commit can't show it: https://github.com/PlayerValley/bloxstrap/tree/patch-1

I think I tried doing this a while ago, but didn't like how close it was to the horizontal rule line. I'd probably be better off making an actual full logo with the icon and text together in one image.


invokeMethodAsync doesn't work on methods of inner classes.

Also XClass/XMethod on inner classes doesn't work!

Both scenarios cannot work since non-static inner classes don't have a public no-argument constructor and no object can be created for them for injection. Added improved error message to injection code.


New Sentry logic fails if replay log hits EOFError

start prevalyer
log a transaction (Sentry.serialId = 1)
restart prevalyer, hit EOF error (Log.serialId incremented b/f exception thrown)
log a transaction (Sentry.serialId = 3)
restart prevalyer (BOOM!   assertion failure)

In Log.loadInitState(), don't increment serialId until after transaction loads successfully from pickle file.

So, this is interesting.
How should system behave if there is an exception loading the Sentry transaction from the Log?
Crash immediately?  (My fix does not deal with this case.)

i see three cases:

both trx and sentry trx load
eoferror loading trx
other error loading trx
eoferror loading sentry trx
5 other error loading sentry trx

cases 3, 4 and 5 are currently ignored.

// I am sorry for not answering earlier.
If there's any error reading transaction log one should not continue - the data is corrupted. You may opt to resume working from last known good state or just stop with error immediately. One more option might be to use a snapshot that is later than the failed log entry but if it existed prevayler would not read the log anyway.
If you decide to ignore broken tail of log then there we need some logic to fix that up somehow - say truncate that log at last known position, erase or quarantine all subsequent logs, make snapshot immediately and start new log from the next transaction.
Maybe adding record boundaries is not the best idea but at least it helps to ensure that only whole transaction is read and verify transactions order so one does not rely on file names only.


Implement client_id/client_secret flow

https://github.com/googlemaps/google-maps-services-python/blob/master/googlemaps/client.py#L315

Done in #11


[BUG] Mobs can't do damage while I'm holding an unblocked item

https://github.com/user-attachments/assets/473525f0-018d-4ce5-bfab-40547654625c

Fixed on latest version.


Drop the Registered Validator, if Possible

Synopsis
Since we changed our strategy regarding the validation exceptions, we may do not need to register an instance of Validator in our auto-configuration.
Steps

[x] Investigate the Possibility
[x] Drop the Validator
[x] Drop the Post Processor.

No more Bean Overriding!


Paginate list conversation

Description

Added Conversation.list_conversations_by_account_paginated/2 function.

Issue
related #451

Generated SQL
SELECT
c0."id",
c0."status",
c0."priority",
c0."source",
c0."read",
c0."archived_at",
c0."first_replied_at",
c0."closed_at",
c0."metadata",
c0."assignee_id",
c0."account_id",
c0."customer_id",
c0."inserted_at",
c0."updated_at"
FROM
"conversations" AS c0
LEFT OUTER JOIN LATERAL (
  SELECT
    inserted_at
  FROM
    messages
  WHERE
    conversation_id = c0."id"
  ORDER BY
    inserted_at DESC
  LIMIT
    1
) AS f1 ON TRUE
WHERE
(c0."account_id" = $1)
AND (c0."archived_at" IS NULL)
AND (
  (
    (f1."inserted_at" = $2)
    AND (
      (c0."id" < $3)
      AND $4
    )
  )
  OR (
    (
      (f1."inserted_at" < $5)
      AND $6
    )
    AND $7
  )
)
ORDER BY
f1 DESC,
c0."id" DESC
LIMIT
$8

Checklist

[x] Everything passes when running mix test
[x] Ran mix format
[x] No frontend compilation warnings

@fmterrorf looks great! i need to take a little more time to review but wanted to get your thoughts on something... after working on this, do you think it would make our lives a lot easier to just add a new field to conversations that holds the last_message_sent_at/last_activity_at, and make sure to update that whenever a message is created?

@reichert621 yeah I believe so. I think that will make ordering better

Som maybe we could do

Add the last_activity_at field. Then we would use latest messages.inserted_at value for the conversations.last_activity_at for existing conversations. This will be a migration
Then implement the pagination

Yup that sounds very reasonable to me
On Thu, Mar 11, 2021 at 9:51 PM Daven @.***> wrote:

So maybe we could do

Add the last_activity_at field. Then we would use latest
messages.inserted_at value for the conversations.last_activity_at for
existing conversations. This will be a migration
Then implement the pagination

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/papercups-io/papercups/pull/631#issuecomment-797197471,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABIFHF5J5YVWAX752VOLZC3TDF6SLANCNFSM4ZBH7D2Q
.

feel free to handle those two steps you listed in separate PRs if that's easier btw!

Going to add a separate PR for implementing last_activity and pagination


[iOS] Fix Makefile

Issue

N/A

Overview (Required)

make init command

Create shell script.

Install all dependencies(e.g. rbenv, mint) if not installed.

This may resolve #667

xcode may need swiftgen.
make to run xcode target

Links

no link

Thank you as always 😄


Address "Dashboard state transition panel inconsistent representation" and "Dashboard measure reaction execution time (reactor-c)"

This addresses the two items which we agreed on Tuesday that we should solve during this sprint.
See CHANGELOG.md for details, including screenshots showing what changed.
Pinging @elgeeko1 since this unblocks the task "Release Dashboard fixes to Customer"

@elgeeko1 Dashboard JSON and CHANGELOG.md are both updated to include the improvement that we discussed this morning. I think that that improvement turned out as intended and that all three improvements/bugfixes are good to release.
Minor detail: After merging, it may be helpful to update the commit hash and link appearing on line 3 of CHANGELOG.md. This would be similar to the change made and discussed in https://github.com/xronos-inc/xronos-dashboard/pull/5/commits/d7c4b7b4c1c816fc716386982a0cfd22b891693a.


Syntax highlighting of the second half of the sftp configuration file fails

Syntax highlighting of the second half of the sftp configuration file fails
Screenshots

Desktop (please complete the following information):

OS: [Windows]
OS Version [22H2]
NxShell Version [1.91]

Can you provide your documents

Can you provide your documents

cassandra.zip

The file is quite large. Please wait patiently. It can be highlighted normally


add nozzle upgrade to release notes

Forgot to add the nozzle upgrade to the release notes

@zippolyte Thanks!


Fix redirects and placeholder on qrcodes page

📝 Changed routes:

/[redirect]
/admin/qrcodes

(view full size)
(view full size)

Commit a2556e555fe60cb0db05fc2f60c03d0d9f3a6c5d (https://cesium-link-jnmy7lkso-cesiuminho.vercel.app).


Is boost test thread safe?

I see no documentation about threads anywhere in https://www.boost.org/doc/libs/1_75_0/libs/test/doc/html/index.html
I am also failing to find references to thread safety by searching the code here in github.
Should I expect this code to work?
BOOST_TEXT_CASE(test_parallel)
{
    std::vector<std::string> vect(1024u);
    std::for_each(
        std::execution::par_unseq,
        vect.begin(),
        vect.end(),
        [](auto const& str)
        {
            BOOST_CHECK_EQUAL(str, "");
        });
}

?

Hi. This is hopefully a simple question to answer.
Is BOOST_TEST thread safe or not?

This is a very simple question.
Can no project member answer it?

Boost.Test is currently not thread safe.

Can that be clearly documented in the manual then?
https://www.boost.org/doc/libs/1_76_0/libs/test/doc/html/boost_test/section_faq.html <-

In branch topic/doc-update

Merged in https://github.com/boostorg/test/pull/337 rev https://github.com/boostorg/test/commit/b52b0ea6429c393485f8ea27ab5edd678823e65a.


document looses drag-features

I tested with the demo page: I set the header-elements directive to "draggable='true'" just for testing the drag features. But if there is a widget inside of the grid and [resizeable] is true, I cannot drag the header-element anymore until I remove that widget.

@Draftscode let me understand this a bit better,
are you trying to drag and drop from another component outside of the widget-grid area into the grid?
If yes, then I must say that I have not tried that out yet.
Drag and resizing works for widgets already added to the grid.
And addition of widgets is through a callback (mostly a click callback on a button outside the grid)
In any case, if you can reproduce your problem on a plnkr, i can investigate further.

@Draftscode pls fork http://plnkr.co/pvA8jx and update with your changes so that the issue can be investgated fuurther.

Made some changes to your plnkr: http://plnkr.co/edit/pWDcFbujOGE0MDJjOZST
Have fun

Made some changes to your plnkr: http://plnkr.co/edit/pWDcFbujOGE0MDJjOZST
Have fun
Gesendet: Montag, 18. September 2017 um 16:50 Uhr
Von: "Naveen Adarsh"<EMAIL_ADDRESS>An: envyN/ngx-widget-grid<EMAIL_ADDRESS>Cc: Draftscode<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [envyN/ngx-widget-grid] document looses drag-features (#3)
@Draftscode pls fork http://plnkr.co/pvA8jx and update with your changes so that the issue can be investgated fuurther.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@Draftscode Please check the plnkr http://plnkr.co/edit/KXSaDCrCE7UfJcGvQuDL?p=preview.
Seems to have solved the problem with 2.0.3
Please reopen if you find the issue again.
🥂

@envyN Looks fine. Thanks for your efforts.


If ColdStarted node is down, new leader is not elected

Steps:
Start 1st node in coldstart mode (no knowledge of other members) - this will become a Leader node
Start a 2nd node which sends request to 1st node to join cluster. 1st Node executes AddMembersAsync - this works, node joins cluster. membership count=2
Start 3rd node, 4th, 5th nodes using step 2. membership count = 5
Leader is 1st Node
Kill 1st Node - expecting other nodes to take over as Leader
What I see if Leader is still shown as 1st Node (after 1st node is already offline) - no election takes place
It\s better to create a new issue.
Originally posted by @sakno in https://github.com/dotnet/dotNext/discussions/62#discussioncomment-4290857

Test ClusterRecovery checks exactly the same scenario and appropriate fix 390e5d1b


vpn_openvpn_server.php: netbios options prettify

Hi!
NetBIOS options in server config looks a bit ugly (especially in dark themes).
Can we add a couple of breaks to prettify it?
before:

after:

Thanks!

hmm, I think officially it should have been a label for the input tag, if I'm not mistaken these are formatted properly (have tried yet)

hmm, I think officially it should have been a label for the input tag, if I'm not mistaken these are formatted properly (have tried yet)

in my opinion this is CSS: select's is display: inline-block; inputs is display: block;
so it doesn't look very good with current formatting (input and select in same div)

in my opinion this is CSS: select's is display: inline-block; inputs is display: block;
so it doesn't look very good with current formatting (input and select in same div)

let's use your fix, the rest of the page handles these issues in similar matter and label has the same issue indeed. thanks!

let's use your fix, the rest of the page handles these issues in similar matter and label has the same issue indeed. thanks!

Thanks!
fast as always )

Thanks!
fast as always )


Paste in Find doesn't work

When copied a text and then pressed cmd+v on mac, the copied text is not pasted on the Find bar. whereas when pasted using right click it does paste.

One more thing noticed here is when I moved the application to Application folder, it worked


Added PvP Heaven

Added my PvP Heaven Network

Logo:

Logo:

You need to upload that image to the logos directory.

Oh god it’s the skid himself! Spoony! Doesn’t know how to even upload a logo LOOOL

Should stick to skidding instead spoony, look like this is too complicated for your brain to handle.

oops mb

I didn't even realize it needed a logo till I checked out other people's pull requests

Oh god it’s the skid himself! Spoony! Doesn’t know how to even upload a logo LOOOL

Didn't even realize it needed a logo till I check out other people's pull requests's go back to skidding yang

Please stop using this as a place to fight.


Prepare for v0.16.21 release

Testing CI but wait for Cilium v1.17.0-pre.3 to be released before reviewing/merging this.

Most (all?) of the test failures are due to warning logs in the Cilium and Hubble UI logs. These get treated as errors since https://github.com/cilium/cilium/pull/35723. To allow skipping this check temporarily (and not block the release for too long to wait for all fixes to be done and merged), I've added a flag to specify log levels to check: https://github.com/cilium/cilium/pull/36231

Please consider this issue before doing a new release.
https://github.com/cilium/cilium-cli/issues/2870

thanks for the heads up @Foxboron i think i'm the one for suggested curling stable.txt from cilium/cilium repo, and i acknowledge that it's not great for downstream package managers to have releases that are not reproducible.
@tklauser what do you think about getting the latest release from the vendored cilium/chart repo? that way the default stable cilium version won't move around. something like:
% ls vendor/github.com/cilium/charts/ | grep -E "cilium-[0-9]+\.[0-9]+\.[0-9]+\.tgz" | sort -V | tail -n 1 | sed -E 's/cilium-([0-9]+\.[0-9]+\.[0-9]+).tgz/\1/'
1.16.3

Currently blocked on https://github.com/cilium/cilium/pull/36358 getting merged to cilium/cilium. Once it is, I'll update this PR.


Sanger Imputation Service issue

Dear xcltk developers,
First of all thank you for developing such an amazing tool along with XClone. The documentation is great!
I have encountered an issue while trying to upload the VCF file generated from the baf_pre_phase.sh script. I am unsure of how to resolve this problem since I used the reference genome you mentioned that is compatible with Sanger. Could you please suggest some solutions?
--- Aborted Job ---
  The genotype probability distribution in the input file does not match the reference
  panel frequencies well. The number of genotypes expected with low frequencies under HWE
  (with P<=0.1) is too big in the user data: 0.32 whereas the threshold is 0.26. For comparison,
  the number of these genotypes in 1000Genomes data is 0.17, the attached plot shows typical
  GT distributions

  This is usually an indicator of REF,ALT alleles being on incorrect strand. Another frequent
  problem is the VCF using a different reference sequence, for example GRCh38 instead of GRCh37.

I also have another question regarding xcltk basefc parameter -r  <region file>, would it be possible to provide some example of what this file should be?

Hi, thanks for your feedback. For the Sanger phasing issue, could you share detailed information of your single cell dataset (e.g., is it 10x scRNA-seq or SMART-seq; aligned to hg19 or hg38; etc...) and the cmdline options you used when running baf_pre_phase.sh.
For basefc, the -r <region file> could be either a BED file (set -T bed) or a TSV file (set -T tsv). The TSV file should contain three tab-delimed columns: <chrom> <start> <end>, and both <start> and <end> are 1-based genomic positions and included. Note that files of both formats should not contain header.
An example of BED file:
chr1	0	1000
chr1	1000	2000
chr1	2000	3000

Its equivalent TSV file is:
chr1	1	1000
chr1	1001	2000
chr1	2001	3000

BTW, the latest manual of xcltk preprocessing is at xcltk/preprocess.

Thank you for your response. My dataset was aligned to hg38 and I used the following command to run baf_pre_phase.sh:
baf_pre_phase.sh \
    -N H08 \
    -s H08.bam \
    -b H08_barcodes.tsv \
    -F human_g1k_v37.fasta \
    -O H08_output \
    -g 38 \
    -C CB \
    -u UB \
    -p 16

Thank you for providing the format of region file in basefc. Could you also please clarify what this file represents? Does it contain the regions of entire genes? Do you have any recommendation for what to provide as a default?

The basefc input is indeed the whole list of features (typically genes). You may use your own gene list file or our pre-compiled file annotate_genes_hg19_update_20230126.txt and annotate_genes_hg38_update_20230126.txt at xcltk/preprocess/data/. Select the right genomic version and set basefc -T tsv as the file is in TSV format.
For the phasing issue, the baf_pre_phase.sh cmdline looks fine, assuming the BAM file was generated by typical CellRanger pipeline. Does your VCF file submitted to the phasing server have a filename suffix "*.fixref.sort.vcf.gz"? (file with that suffix is the final output of the pre_phase script).

Thanks for sharing the example VCF, it looks fine.
Could you also share the logging file for double checking whether every step in the script worked well, especially the "fixref" step indicated in the sanger report.
"This is usually an indicator of REF,ALT alleles being on incorrect strand. Another frequent problem is the VCF using a different reference sequence, for example GRCh38 instead of GRCh37."
You may also run "fixref" on your submitted VCF by yourself (see the "Check and fix the REF allele" paragraph on Sanger phasing resources page for details).

Thank you so much for your response Xianjie. I had to rerun the command again since I didn't keep the log but now it can be downloaded from log.txt. It seem sthe fixref step was successfully done, what do you think?

Thanks for your efforts to generate the log file. It seems every step works well.
Then I would suggest you checking "fixref" status (bcftools +fixref) and "genotype frequencies" (bcftools +af-dist) of the VCF, following the official instructions (see the "Check and fix the REF allele" paragraph on Sanger phasing resources page for details).
On the other hand, there may be some mismatches between the sequenced data and sanger refernece panel, as indicated on this Biostars thread. You may try selecting different reference panels on sanger service page.

Thanks Xianjie for your suggestions.

I did run bcftools +fixref but everything appeared to be ok.
Unfortunately, I encountered an issue while attempting to run bcftools +af-dist. The URL for the 1000 Genomes allele frequency annotations is broken and cannot be downloaded.
I also looked into other reference panels on Sanger, but encountered the same error.

Is there anyway to run xcltk and XClone using a common SNP VCF file on my local machine instead of the Sanger Server? Something similar to Numbat?

Thanks for your updates.

We are testing performing reference phasing on local machine, similar to Numbat as you mentioned, to make the phasing process smoother. However, evaluating the new strategy may take some time. For now, you may want to use an alternative method for phasing, given the results you listed.
Since xcltk preprocessing framework is designed to be divided into three parts (pre-phase, phase, and post-phase), it should be flexible to use a different phasing method, such as Eagle2 used in Numbat, before the post-phase step. For example, if you have results of Numbat preprocessing pipeline, you may try merging all the phased VCF files (in the <output_dir_of_Numbat_preprocess>/phasing dir) into one new VCF as input of baf_post_phase.sh.
(Optional) you may phase the VCF submitted to Sanger server with Eagle2 on local machine, to check whether the VCF could be phased smoothly.

Thank you Xianjie. I do already have the phasing files according to hg38 from Numbat and now attempting to use the merged VCF as input for baf_post_phase.sh. However, I'm unsure whether I need to modify the baf_post_phase.sh script and ignore lines 170 to 184 (which are for liftover). Can you advise me on this matter?

Hi, you may set -g 38 and -G 38 for the updated baf_post_phase.sh to skip liftover (commit ec1ebe1). For details of each option, you can type ./baf_post_phase.sh -h.

Hi Xianjie, I just wanted to let you know that after switching to python 3.7 my initial issue with sanger imputation server was miraculously solved! It seems both XClone and xcltk must be run using only python 3.7. Thank you for your help anyway.

Thanks for the updates. It is very helpful.


Ninja suit not properly using batteries

Description
while playing as ninja, if you swap a low battery with a fresh battery, it does not show the correct percentage on the suit and you still have to charge with the ninja gloves.
Reproduction

be ninja
swap low battery for full battery
suit power does not increase, despite having a fully charged battery

EDIT**
new info, apparently ninja suit doesn't consume batteries at all

Battery swapping was removed to enforce people to use the actual charging interaction of the gloves.
Is it actually taking the new battery? Can you provide a video of it swapping a small to a medium etc.
Because that shouldn't be possible.
My assumption is its just a ui mispredict showing the full battery that isn't actually inserted.

Battery swapping works, at very least you can upgrade them from the default small.

battery swapping was never allowed, need actual repro steps

Battery swapping was removed to enforce people to use the actual charging interaction of the gloves.
TIL lol

Is it actually taking the new battery? Can you provide a video of it swapping a small to a medium etc.
Because that shouldn't be possible.
I will do that as soon as I get a chance :)

From my experience, you can change the battery ONCE, for example from the small to the medium one, after that you're fucked. It starts bugging out, you cant recharge sometimes, sometimes it tells you full battery but it doesnt let you use abilities.
I'll try to recreate it and post a video of it

If you try to swap for a downgrade or same tier of cell the issue happens, it gets fixed once you upgrade the cell.
https://github.com/space-wizards/space-station-14/assets/170836000/6937b9a8-59ea-4a1b-9ffc-13255eed13b8

either mispredict or battery swap somehow isnt updating the battery drainer uid will need to vv to be sure


Allow skipping of message serialization

According to
https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage
modern browsers allow messages other than strings and have a built in
serialization algorith.
Especially when working with JSON messages it is very convenient to
simply use the browsers built-in serialization.

Good point, I missed the fact that modern browsers will automatically serialize the data.

Thanks for merging!


Installing on M1 Mac (arm64)

If I try to install using pip install pyquadkey2 on an M1 Mac I get this error:
clang -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/coryisaacson/miniconda3/lib -L/Users/coryisaacson/miniconda3/lib -Wl,-rpath,/Users/coryisaacson/miniconda3/lib -L/Users/coryisaacson/miniconda3/lib -L/opt/homebrew/opt/libffi/lib -I/opt/homebrew/opt/libffi/include build/temp.macosx-11.1-arm64-cpython-310/src/pyquadkey2/quadkey/tilesystem/tilesystem.o -o build/lib.macosx-11.1-arm64-cpython-310/pyquadkey2/tilesystem.cpython-310-darwin.so
      ld: library not found for -lSystem

My LD_LIBRARY_PATH is set correctly, are there prerequisites for the System lib to be built?

I resolved this, it was an incorrect path, I needed to put /usr/bin at the front of the path as I am using brew llvm also.


Doxygen: need from cpp files throw error in needflow

Follow-up of #193
If the need defined in doxygen is used in a needflow it throws this error. The same need block copied to rst worked fine. I can create an example for this if you wish.
Extension error (sphinxcontrib.needs.directives.needflow):
Handler <function process_needflow at 0x7f3e3c92c5e0> for event 'doctree-resolved' threw an exception (exception: 'refid')
make: *** [Makefile:29: html] Fehler 2

Fixed by PR #233


CLIRMatrix

For cross-lingual IR.
CLIRMatrix: A massively large collection of bilingual & multilingual datasets for Cross-Lingual Information Retrieval. Shuo Sun and Kevin Duh.
Appearing at EMNLP'20. Doesn't seem like full text or data are available yet.

Here's the paper: https://www.aclweb.org/anthology/2020.emnlp-main.340.pdf

The data are available here: http://www.cs.jhu.edu/~shuosun/clirmatrix/.
It looks like the format should be easy to parse: documents are in TSV and queries/qrels/scoreddocs are in jsonl.
The big design decision here is how to deal with the large number of language combinations (18k+).

Could do it as something like "clirmatrix/bi139/en/es" (where the first language code is either the query or document language). But this will greatly inflate the namespace.
An alternative is to provide all combinations via "clirmatrix/bi139" and provide the language codes via properties of the namedtuples, but there may be a lot of overhead doing this.
Or use #13 to specify the language pairs when loading the dataset like ir_datasets.load("clirmatrix/bi139", doc_lang="en", query_lang="es").

I think I prefer option (3) above, but I still need to figure out what a good mechanism is for handling #13.

It's appealing to always be able to specify a specific dataset via a string so it can be passed around easily. So maybe option 3(b) could take a cue from the URL convention like this: ir_datasets.load("clirmatrix/bi139?doc_lang=en&query_lang=es"). Would be a shortcut for option 3 above (parameters would be first pared like URL parameters), but allow stuff to be specified as a string. ping #13.

Thanks for the great contribution @ssun32!


Hotfix

merging fixes from master
note: unit tests is not working in release branch, just for note..
@tommikas @kjbracey-arm @yogpan01 @teetak01

Should ARMmbed/mbed-trace#31 perhaps get in too? Although I'm not sure which builds it fixes exactly.

Does it fix the library order problem?

Hard to tell - no commit message.

It doesn't seem to. I'm not sure what application and problem it fixes exactly. Asked Jaakko for elaboration but he's probably left the office by now.

I merged that #31 to master, but could it cause problems I'm not sure. Perhaps we shouldn't merge it to release branch

maybe we should merge this and update version #37 and do yotta publish, ok?

Jaakko's fix would seem to work. Tested quickly.

I take that back. It does seem to fix it.

This is still missing Jaakko's fix?

As in this alone will not fix the build issues.

now it's included in this PR -> going to release branch

ready to merge ?

builds linux-example + linux-tests work fine with this.


Request foreach block for dictionary

Iterate over key-value pair, as foreach for List do.

Note that you can technically do this as dictionaries will coerce into an associative list when used in a list operation. For example:

However, a version of the foreach that has two variables (key, value) might be more intuitive than unpacking the item into key and value.

Thanks for your advice..There are many ways to iterate over dictionary. As you said, there are two variables that are more intuitive.BTW: It would be better if the foreach had 2 variables, one is item and other is the index of the current iteration.

I'd like to give this a shot. My idea is to make a "foreach key,value" block:

That generates the same code as this combination of blocks:

Does that seem like the correct strategy?

@BeksOmega that sounds reasonable to me, although we might want to have the text be for each [key] with [value] in dictionary
The block definition should be straightforward for you given your familiarity with Blockly. If you’re not familiar with the Scheme language, defining the behavior will be a bit more complicated. Take a look at the existing definitions in runtime.scm for the other for each blocks to get a sense of how they work. We can help you out if you get stuck.
Sent from my iPhone

On Feb 8, 2020, at 15:52, Beka Westberg<EMAIL_ADDRESS>wrote:

I'd like to give this a shot. My idea is to make a "foreach key,value" block:
That generates the same code as this combination of blocks:
Does that seem like the correct strategy?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

@BeksOmega Please note that need to support the "break" block.

@ewpatton How about adding the current iteration index to foreach? When using foreach in java, in order to get the serial number, you have to define a count variable or use for.It is more tedious in ai that requires at least 6 code blocks.

@ewpatton How about adding the current iteration index to foreach? When using foreach in java, in order to get the index, you have to define a counter variable or use for.It is more tedious in ai that requires at least 6 code blocks.


[PE-D][Tester D] Delete Command confirmation message disappears after typing in another command, leading to confusion

Step 1

Step 2

Step 3

Step 4 (1 possible way: user types in y)

Step 4 (another possible way: user types in another command)

Delete Command confirmation message disappears after typing in another command, leading to confusion.
Explanation: After step 3, the user does not know that they are still under the delete command execution, because the result display does not show it anymore. The user will only find out after typing in another valid command, which could be a long command, and thus result in a huge waste of time.
Reproduce: type delete 1, then type add n/John Doe p/98765432<EMAIL_ADDRESS>a/311, Clementi Ave 2, #02-25 t/friends t/, notice that the confirmation message disappears.
Consider changing the error in Step 3 to tell user that they can only type y or n.

Labels: severity.Medium type.FeatureFlaw
original: leroytan/ped#18

will add under planned enhancements


Contributing guide add

Fixed issue #2

LGTM
p.s. if anything needs to be added or removed, suggestions are always welcome

Hey @shivam-sharma7 , this looks good! I do think we can make this a bit more informative for beginner folks out there, by adding more information like how to commit, how to make a change, how to push etc.
You can take reference from the guide here (from another repo) & see if you can add some more steps to our guide!
Let me know your thoughts on this or any questions :)
cc @AvineshTripathi @anubha-v-ardhan @saiyam1814

do you want to add in this PR or have a seperate one

Hey @anubha-v-ardhan , is this PR good to go or have any other changes needs to be done?
cc @shivam-sharma7 @AvineshTripathi

Hey @anubha-v-ardhan , is this PR good to go or have any other changes needs to be done? cc @shivam-sharma7 @AvineshTripathi

I was thinking if we can do a more detailed contributing guide diviiding it into two sections for core website work and notes work since notes will also be on website repo. But since it's not clear yet how the notes work will proceed, I think we can merge this once https://github.com/kubesimplify/website/pull/26#pullrequestreview-972992673 is resolved and make issue for enhancing this guide later.

@shivam-sharma7 Please comment on #2 and get it assigned to yourself to avoid confusion. Always notify everyone by commenting on the issue before you start working on it.


Add Secure parameters to cChocoPackageInstaller

It would be great if we could add the Secure Parameters to the cChocoPackageInstaller

Can you be specific about what you are looking for? Use case?

No update on this for some time. Closing the issue but feel free to come back and reopen if this is still an issue for you.


Analyze component frontmatter now markdown

fix for #178 & #177
PR for content branch name analyze-badges https://github.com/HumanCellAtlas/data-portal-content/pull/122

this is merged


Remove buildToolsVersion from build.gradle or provide default

Description
Remove buildToolsVersion from the build.gradle. According to this documentation, after 3.1.0 the plugin automatically picks a version.
However, if its necessary to keep, then you should provide a default value as well as the extension. Here is an example of how you would do that:

This package will break builds on projects that do not specify a buildToolsVersion in their build.gradle ext section.
How to repeat issue and example

Install package in react native project
Don't specify buildToolsVersion in the build.gradle ext section of that project
Build fails

Solution
Remove buildToolsVersion or specify a default fallback value.
Additional Information

Image Picker version: 3.0.0-vnext.2
React Native version: 0.62.2
Platform: Android
Development Operating System: Windows

It looks like in version 2.3.4, this default functionality was in place.

@scousino can you test directly from GH using the vnext branch?

@Johan-dutoit tested and works

Thanks @scousino.

:tada: This issue has been resolved in version 2.4.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket: