decrease reconnection time

I seem to observe that it takes quiet a long time for the connection to be restored after a network change (currently I have already been waiting for half an hour and still nothing). Is there anything fowl can do to decrease that delay? Or is it something that is done internally by the wormhole library?

Hmm, interesting.
That's basically "Dilation code, inside magic-wormhole" .. but I also maintain that library so definitely interested, although probably a tracking this via an issue on https://github.com/magic-wormhole/magic-wormhole makes more sense.

How long were you disconnected for?

Hmm, interesting.
That's basically "Dilation code, inside magic-wormhole" .. but I also maintain that library so definitely interested, although probably a tracking this via an issue on https://github.com/magic-wormhole/magic-wormhole makes more sense.

Yes, I was going to add that you should feel free to transfer the issue
to the other repo if appropriate, but I was thinking that the dilation
could have some reconnect timer which could be set via an argument from
the client, so it made sense to first open it here. I was not able to
find anything like it from a quick look at the dilation documentation
and even some bits of source code, though.
Could you please point me to the relevant code or at least explain how
this portion of the protocol/implementation works? I understand that the
leader of the connection should determine when the subchannel is dead,
but I was not able to find any details.
What I would naively imagine is something along the lines of SSH's
ServerAliveInterval. When the leader would deem the subchannel dead, it
would try to negotiate a new connection via the mailbox server, after
the other side has reconnected there (which I assume does happen, as
it's the central point of communication between the two clients). Or
even perhaps both sides could agree on a subchannel being dead after
they reconnect via the mailbox server.

How long were you disconnected for?

 I did exactly what you write in your
blog post: close the laptop, change train and reopen :-P
I have since then changed the network once more (and was offline for
even longer than before) and now I'm waiting to see if it reconnects (it
never did on the second network). This is not the first time I'm seeing
this. When I first tried fowl, I was going to raise an issue asking
about the durability of the connection as there seemed to be none and
then it just suddenly reconnected after some time. So I waited until I
test it again and since there was no improvement, I thought it better to
ask.
By the way, the other side should have remained connected the whole
time, so if I understood correctly, the mailbox server should not free
the session - right?

All the dilation code is here: https://github.com/magic-wormhole/magic-wormhole/tree/master/src/wormhole/_dilation
I'll take a look; it's been a bit since I've delved into the Dilation implementation.
Overall, your expectations are correct. And yes, the mailbox would typically be used to negotiate a new "generation" (I believe is the term used) of the wormhole. The Leader always decides this.

By the way, the other side should have remained connected the whole time, so if I understood correctly, the mailbox server should not free the session - right?

Correct. BOTH sides have to be disconnected for 10 minutes before the mailbox is freed (with the default server, other deployments can choose their own timeouts).

All the dilation code is here: https://github.com/magic-wormhole/magic-wormhole/tree/master/src/wormhole/_dilation

Yes, that's where I was looking briefly, but with no luck. I was hoping
you could be more specific, but I understand if you don't know either
off the top of your head.

I'll take a look; it's been a bit since I've delved into the Dilation implementation.

I will try to take another look when I have the time, perhaps you will
beat me to it. But at least you confirm, that it has nothing to do with
fowl and is really an internal wormhole thing, if I understand
correctly.

By the way, the other side should have remained connected the whole time, so if I understood correctly, the mailbox server should not free the session - right?

Correct. BOTH sides have to be disconnected for 10 minutes before the mailbox is freed (with the default server, other deployments can choose their own timeouts).

It's been twelve hours since my last network change and still nothing,
so I guess it's not coming back this time.
I have let both sides still running for now in case you have any
suggestions for what I should try or even if you wanted to inspect the
mailbox server to see whether it still considers the connection alive,
provided that you have access to it. But I will have to turn it off
eventually.

All we'd be able to see on the mailbox server is if the mailbox itself is still allocated (but I think the interesting things are all in the content of the messages).

That's what I originally meant - seeing whether the mailbox server still
thinks the connection is active. Of course the messages themselves would
be even more valuable.
To test the mailbox server, I tried allocating a code with the same
nameplate (by issuing wormhole receive ...) and got an error. On a
second attempt, it allocated it. So either I interrupted someone's
transfer by an accident, or the connection was indeed still there and I
probably broke it. But actually, I'm not sure whether this works like
this - is it possible to interrupt a connection by using the wrong code
after the connection has already been established? Shouldn't the
nameplate be freed after the sides verify each other using the code? (I
am not saying that they should not use the mailbox server, of course
they will still need it to arrange for the transit connection, however I
would maybe expect that the nameplate would only be relevant so that
they can find each other initially and then the mailbox server could
assign them some less human friendly identifier. But maybe that doesn't
really make sense, to use two different things for session
identification, although it could prevent these interruptions of
established connections by a MITM attacker, if those are really
possible.)

wormhole send --debug-state M would be the way to dump state-transitions for the Manager machine...would probably be good to add a similar option in fowl for at least us to use :)

That would be really useful for debugging this.
By the way, I just unintentionally tested it again - had two devices
connected directly via a cable, both connected to the same wireless
network. I created the fowl session and then uplugged them and the
connection died and did not reconnect within maybe 20 minutes. While
those previous cases I considered rather tough (quite long and frequent
disconnects), this I would expect to reconnect.
Two questions (the first one I already kind of implicitly asked so it's
possible we won't know until one of us reads the dilation code
properly): what is the expected time to reconnect? To me those 10 to 15
minutes would seem plausible, but I'm not really sure whether there is
some reason for the leader to wait longer to see if the original
connection might come back alive (although since both sides should still
be able to communicate via the mailbox server, it would seem that there
is no reason that they can't just agree to drop the old connection (or
generation as you said)). In an optimal case, I would probably want a
minute or two at the maximum (but again, there may be a good reason to
wait longer before giving up on the older connection, I don't know).
Second question: were you actually able to reproduce this yourself? Or
does the durability work as expected for you?
Thank you for your patience and interest :-)

But actually, I'm not sure whether this works like this - is it possible to interrupt a connection by using the wrong code after the connection has already been established?

No.
The nameplates are ephemeral; the mailboxes have 256-bit IDs and are the long-lived part. So by re-using e.g. 6-*-* you've just re-allocated the "6" nameplate (but it will now point at a different mailbox).

Yes, I agree from a usability standpoint something less than 10 minutes is desirable.
I have not repeated this issue myself (see ticket on magic-wormhole, I tried an overnight experiment), but haven't spent a ton of time yet -- and my familiarity with the deep aspects of Dilation has waned a little :)

Note that with fowl there's another aspect here: since Dilation is invisible to the streams being forwarded, it's possible that client software itself has timeouts. I've been mostly experimenting with telnet and nc ... but the "higher level" protocol may play a role here too :/

The nameplates are ephemeral; the mailboxes have 256-bit IDs and are the long-lived part. So by re-using e.g. 6-*-* you've just re-allocated the "6" nameplate (but it will now point at a different mailbox).

Exactly what I would expect, thanks.

I have not repeated this issue myself (see ticket on magic-wormhole, I tried an overnight experiment), but haven't spent a ton of time yet -- and my familiarity with the deep aspects of Dilation has waned a little :)

I find it strange that it worked for you on a local network while it did
not reconnect for me on the cable either after I plugged it back in
(although I did not wait for too long). It occurred to me that it could
have something to do with the fact that in your case it might not have
been the leader who was put to sleep, however in my case both sides
still had access to the mailbox server, so that should not be the
reason. Also, you write about only seeing -2 messages - that denotes
the generation? In that case, could it be possible that your setup
really reused the old connection, because putting the laptop to sleep
did not close it (because the laptop had no time to handle it before
sleeping, then it woke up and the connection was still active and so it
got reused, while in my case, the network was interrupted for some time
while the device was still active, so it had time to handle that and
close the connection on its side after some time)? I'm just speculating
though.
How long did it take to reconnect for you?

Note that with fowl there's another aspect here: since Dilation is invisible to the streams being forwarded, it's possible that client software itself has timeouts. I've been mostly experimenting with telnet and nc ... but the "higher level" protocol may play a role here too :/

That should not be my case - I am experimenting with SSH and I'm
starting a new session every time. That's not a problem, right, the
forward should be durable even if nothing is going through the channel
at the moment?

That's not a problem, right, the forward should be durable even if nothing is going through the channel at the moment?

Yes, that should work.
Also yeah I didn't try to figure out which side was Leader (which could indeed make a difference).
Yes, I was talking about "generations" with the -2 thing.

Yes, that should work.

...welllll, so there are network devices like NATs that like to cancel TCP connections that are "inactive". We could send pings (but I think the application has to decide to do that, not Dilation itself).

...welllll, so there are network devices like NATs that like to cancel TCP connections that are "inactive". We could send pings (but I think the application has to decide to do that, not Dilation itself).

Well OK, but I assume the mailbox connection would be considered active
and so even if the subchannels got cut off, the wormhole should be able
to reconnect them, at least on demand, no?

Well OK, but I assume the mailbox connection would be considered active and so even if the subchannels got cut off, the wormhole should be able to reconnect them, at least on demand, no?

Yes.
There's some complexity in here, though: often you don't learn that some NAT "disconnected" the stream until you try to send more bytes (hence the ubiquity of sending pings). But, overall, it should work.

The server/client distinction is arbitrary and has no intended
correspondence to which side was the leader.

In fact, the "client" is the laptop on the "flaky" network, where I have
requested listening on local port and forwarding of the traffic to a
port on the "server" device. "server"'s connection should be quite
stable.

Some more annec-data: I just did an experiment, consisting of the following:

connect laptop to desktop
forward ssh ports
connect ssh between desktop / laptop (over fowl / wormhole)
above is all on a LAN / home network
laptop sleeps
relocate (took about 30 minutes before I regained network, on a new network)
tried "doing a thing" in ssh (nothing)
after about 10 minutes, i started to see some action in the fowl console (it claimed to be forwarding 36 bytes at a time, occasionally -- assuming keepalive)
after about 10-15 additional minutes, ssh began working (I'm not sure precisely when)

So, it did eventually re-negotiate and forward all the missing SSH bytes, but "half an hour" is of course way too long.
I don't know what affects the above (e.g. is this something a program like "fowl" can control, or is it ultimately a magic-wormhole problem?)

The connection survived the return trip too, but I didn't make a good note of times...

So, from my ad-hoc real-world testing, there is definitely at least two problems here:

reconnection takes "too long" sometimes
there isn't enough feedback (up to the user) about what Dilation may (or may not) be doing (e.g. "re-connecting" vs. "thinks it is connected right now" etc).

Even if the re-connection still takes as long as it does right now, having some feedback about the process would greatly help the user-experience (and presumably help debug why it takes a long time sometimes).

Agreed. I will try to remember to test this again with changing networks
and see if I observe the same behaviour as before (that is, no
reconnection after hours) or rather reproduce what you write (so,
reconnection within some (still-not-so-)reasonable amount of time).

It might be related to how long you're offline too ... my next step here will be to add more instrumentation to magic-wormhole so that fowl can produce some output to the user indicating what's going on (it might be as simple as "exponential backoff, with no upper bound" or something, for example)

That would sound plausible, but see what I wrote above about my flaky
network. Plus, if I recall correctly, you put the laptop to sleep over
night and it reconnected within 30 minutes.
Either way, adding some verbose connection status reporting to
magic-wormhole seems like the best next step.

I tested the "get on a train" scenario again and something somewhat
interesting happened. I started two fowl sessions on the same device
(within about fifteen minutes, simply because I forgot to specify some
-L/-R connections on the first one) and connected from a second device.
The second device was subsequently offline for about an hour and then
connected to a different network. One of the two fowl sessions
reconnected in 20 minutes, the other still has not (it's been many
hours).
Do you have any explanation for this? It doesn't sound much like the
exponential backoff, does it? The only thing that comes to mind which
could be different between the two sessions is which device was the
leader for each. Or is there something else too?

I wouldn't expect leader/follower from purely the fowl level to have any effect (there is a Leader / Follower at the Dilation level, but it's random depending on the keys).

Is the session that didn't connect also the session with no subchannels? With no traffic, the OS can often pretend that a TCP stream is alive (until it tries to send a byte, and then discovers that it is not).

I was not using either connection at the time when I lost the first
network and I only attempted to use it once I connected to the other
network. So no subchannels should have been open for either connection.

I wouldn't expect leader/follower from purely the fowl level to have any effect (there is a Leader / Follower at the Dilation level, but it's random depending on the keys).

I meant Leader/Follower at the Dilation level.


Fixes for C3 LTE connection being deleted

Description
Partially resolve #22115 by fixing two wifiManager.cc code paths that could delete the LTE connection.
The first fix will make it so we don't wrongly try to forget connections outside of ui's new connection process. It was possible to trigger this if wpa_supplicant tried to re-authenticate to an existing Wifi network while driving out of range. In that case, it didn't even try to delete the current Wifi connection; instead it was trying to delete an empty SSID connection.
The second fix will make sure the LTE connection can't possibly make it onto the knownConnections SSID list. We prevent that at init time, but there was a possible path if NetworkManager ever sent a DBus NewConnection event for the LTE. Unsure if this would do it, but it's possible that ui could come up before the LTE modem finishes booting and enumerating. If that ever happened, the first bug that was trying to forget empty SSID connections could actually match the empty SSID LTE connection.
Verification
None. The associated bug is difficult to reproduce on-demand; I've made no attempt. Please feel 150% free to check my logic.
Additional context
As already noted in the bug, we should also make an AGNOS-side fix to always un-delete the LTE connection at boot-up. If nothing else, we need to recover LTE on any devices that got stuck in this state before receiving the fix.
Related to the current issue, I think the current ui assumption that lteConnection is the only non-802-11-wireless connection is asking for trouble when we can hook up USB Ethernet devices. I don't have a USB Ethernet interface handy to write code for handling that case tonight.

For S&Gs I instrumented wifiManager.cc to show which connections it was adding at init time, and show any new connections that came in via DBus events. Then I forced a situation where the LTE modem started well after regular power-on, after UI.
LTE is still picked up as a defined connection at init time even when the hardware isn't present, and powering on LTE well after boot/UI startup doesn't kick off the NewConnection event I thought possible.
This PR is still valid and applicable, it still fixes bugs, and I think the first fix will almost certainly mitigate the original problem. I just wish I could see the complete chain of events.
diff --git a/selfdrive/ui/qt/offroad/wifiManager.cc b/selfdrive/ui/qt/offroad/wifiManager.cc
index 81da7bd..72854ee 100644
--- a/selfdrive/ui/qt/offroad/wifiManager.cc
+++ b/selfdrive/ui/qt/offroad/wifiManager.cc
@@ -3,6 +3,7 @@
 #include <algorithm>
 #include <set>
 #include <cstdlib>
+#include <cstdio>

 #include "selfdrive/common/params.h"
 #include "selfdrive/common/swaglog.h"
@@ -341,6 +342,7 @@ void WifiManager::connectionRemoved(const QDBusObjectPath &path) {
 }

 void WifiManager::newConnection(const QDBusObjectPath &path) {
+  printf("UI debug: newConnection called for connection %s\n", qPrintable(path.path()));
   knownConnections[path] = getConnectionSettings(path).value("802-11-wireless").value("ssid").toString();
   if (knownConnections[path] != tethering_ssid) {
     activateWifiConnection(knownConnections[path]);
@@ -376,8 +378,10 @@ void WifiManager::initConnections() {
   for (const QDBusObjectPath &path : response.value()) {
     const Connection &settings = getConnectionSettings(path);
     if (settings.value("connection").value("type") == "802-11-wireless") {
+      printf("UI debug: init adding Wifi connection %s\n", qPrintable(path.path()));
       knownConnections[path] = settings.value("802-11-wireless").value("ssid").toString();
     } else if (path.path() != "/") {
+      printf("UI debug: init adding LTE connection %s\n", qPrintable(path.path()));
       lteConnectionPath = path.path();
     }
   }

Started UI:
comma@tici:/data/openpilot/selfdrive/ui$ ./ui
GBM_INFO::msmgbm_mapper(125)::gbm mapper instantiated

gbm_create_device(156): Info: backend name is: msm_drm
UI debug: init adding Wifi connection /org/freedesktop/NetworkManager/Settings/1
UI debug: init adding LTE connection /org/freedesktop/NetworkManager/Settings/2
: "wl-shell" is a deprecated shell extension, prefer using "xdg-shell-v6" or "xdg-shell" if supported by the compositor by setting the environment variable QT_WAYLAND_SHELL_INTEGRATION
: Wayland does not support QWindow::requestActivate()

Started LTE:
comma@tici:/data/openpilot$ mmcli -m 0
error: couldn't find modem
comma@tici:/data/openpilot$ nmcli conn
NAME                        UUID                                  TYPE  DEVICE
openpilot connection young  67419148-9117-4993-bd7f-a5b65a957ebd  wifi  wlan0
lte                         6586fb1f-4a66-46d4-9164-0100b73750ee  gsm   --
comma@tici:/data/openpilot$ time /usr/comma/lte/lte.sh start
Starting LTE...
 Resetting...
 Pulsing power button...
 Waiting until the modem comes online...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Waiting...
  Modem is online!

real    0m15.080s
user    0m0.020s
sys     0m0.000s
comma@tici:/data/openpilot$ nmcli conn
NAME                        UUID                                  TYPE  DEVICE
openpilot connection young  67419148-9117-4993-bd7f-a5b65a957ebd  wifi  wlan0
lte                         6586fb1f-4a66-46d4-9164-0100b73750ee  gsm   cdc-wdm0

No further output from UI.


CHAOS-6506 Add note

Add note regarding HTTP, HTTPs, and gRPC support for pod HTTP faults

@SmritiSatya pls close this PR and open a new PR with changes. If you're comfortable, you can also rebase.


AttributeError: 'lxml.etree._Element' object has no attribute 'fldCharType'

when I open my .docx file , which is saved from a .doc file use python-docx,  it cames out this Error,
C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py:194: UnexpectedElementWarning: Skipping unexpected tag: {http://schemas.openxmlformats.org/wordprocessingml/2006/main}background   UnexpectedElementWarning) C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py:194: UnexpectedElementWarning: Skipping unexpected tag: {http://schemas.openxmlformats.org/wordprocessingml/2006/main}pict   UnexpectedElementWarning) Traceback (most recent call last):   File "E:/_master/硕士论文/data/data_preprocess/temp.py", line 27, in <module>     db_json = simplify(db)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\__init__.py", line 33, in simplify     out = document(doc.element).to_json(doc, _options)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 106, in to_json     "VALUE": [ elt.to_json(doc, options) for elt in self],   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 106, in <listcomp>     "VALUE": [ elt.to_json(doc, options) for elt in self],   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\body.py", line 25, in to_json     JSON = elt.to_json(doc, options, iter_me)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\paragraph.py", line 142, in to_json     out: Dict[str, Any] = super(paragraph, self).to_json(doc, options, super_iter)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\paragraph.py", line 27, in to_json     for elt in run_iterator:   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 62, in __iter__     self.__iter_name__ if self.__iter_name__ else self.__type__):   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py", line 167, in xml_iter     for elt in xml_iter(current, handlers.TAGS_TO_NEST[current.tag], _msg):   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py", line 156, in xml_iter     yield handlers.TAGS_TO_YIELD[current.tag](current)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\form.py", line 106, in __init__     super(fldChar, self).__init__(x)   File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 36, in __init__     self.props[prop] = getattr(x, prop) AttributeError: 'lxml.etree._Element' object has no attribute 'fldCharType'
what should I do to solve it?

Thank you for your question!   This error indicates that the .docx file was not valid.  Specifically, the fldCharType attribute is required on fldChar elements, by the open office specification, and the error indicates that this required attribute is missing.
Since you saved this file with python-docx (from the original .doc file), you might want to raise an issue with python-docx.

I had the same error with some tables in the docx. Is there any docx format for simplify_docx? In other words, what attribute in docx files that simplify_docx can't deal with?
Thanks.

when I open my .docx file , which is saved from a .doc file use python-docx, it cames out this Error,
C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py:194: UnexpectedElementWarning: Skipping unexpected tag: {http://schemas.openxmlformats.org/wordprocessingml/2006/main}background UnexpectedElementWarning) C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py:194: UnexpectedElementWarning: Skipping unexpected tag: {http://schemas.openxmlformats.org/wordprocessingml/2006/main}pict UnexpectedElementWarning) Traceback (most recent call last): File "E:/_master/硕士论文/data/data_preprocess/temp.py", line 27, in <module> db_json = simplify(db) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\__init__.py", line 33, in simplify out = document(doc.element).to_json(doc, _options) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 106, in to_json "VALUE": [ elt.to_json(doc, options) for elt in self], File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 106, in <listcomp> "VALUE": [ elt.to_json(doc, options) for elt in self], File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\body.py", line 25, in to_json JSON = elt.to_json(doc, options, iter_me) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\paragraph.py", line 142, in to_json out: Dict[str, Any] = super(paragraph, self).to_json(doc, options, super_iter) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\paragraph.py", line 27, in to_json for elt in run_iterator: File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 62, in __iter__ self.__iter_name__ if self.__iter_name__ else self.__type__): File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py", line 167, in xml_iter for elt in xml_iter(current, handlers.TAGS_TO_NEST[current.tag], _msg): File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\iterators\generic.py", line 156, in xml_iter yield handlers.TAGS_TO_YIELD[current.tag](current) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\form.py", line 106, in __init__ super(fldChar, self).__init__(x) File "C:\Users\Luke\Anaconda3\lib\site-packages\simplify_docx\elements\base.py", line 36, in __init__ self.props[prop] = getattr(x, prop) AttributeError: 'lxml.etree._Element' object has no attribute 'fldCharType'
what should I do to solve it?

请问，这个问题解决了吗？

according to the README

This project relies on the python-docx package which can be installed via
pip install python-docx. However, as of this writing, if you wish to
scrape documents which contain (A) form fields such as drop down lists,
checkboxes and text inputs or (B) nested documents (subdocs, altChunks,
etc.), you'll need to clone this fork of the python-docx package

You need a modified version of python-docx to support parsing forms, otherwise python-docx will not parse fldChar and cause this issue.

I created a new fork moving over the changes made in this (stale) python-docx fork to the latest python-docx repo and this error was resolved.
You can install the new fork:
pip install git+https://github.com/dalmia/python-docx.git

Hope this helps!


The cause of reproducibility problem of bazel build in jaxlib

A discussion issue for reproducibility issue with bazel that causes build failures in jaxlib package.
I created #296737 when I first noticed this problem. After it was fixed (by #291705), another person opened it again because the issue occurred again. As per https://github.com/NixOS/nixpkgs/pull/321559#pullrequestreview-2134078491, I am opening a new issue for discussing what are possible causes of this. Hopefully the original issue or a similar issue won't need to be opened again.
CC: @ndl @samuela @natsukium @GaetanLepage

Add a :+1: reaction to issues you find important.

For #296737 , when it first happened, @ConnorBaker and I managed to reproduce it and point out that the difference in hash was due to:
Files this/external/go_sdk/versions.json and other/external/go_sdk/versions.json differ

This was the diff in the tar file:
339C 1240: 7B 0A 20 20 20 20 22 66  69 6C 65 6E 61 6D 65 22  {.    "f ilename"
339C 1250: 3A 20 22 67 6F 31 2E 32  32 2E 31 2E 73 72 63 2E  : "go1.2 2.1.src.
339C 1260: 74 61 72 2E 67 7A 22 2C  0A 20 20 20 20 22 6F 73  tar.gz", .    "os
339C 1270: 22 3A 20 22 22 2C 0A 20  20 20 20 22 61 72 63 68  ": "",.     "arch
339C 1280: 22 3A 20 22 22 2C 0A 20  20 20 20 22 76 65 72 73  ": "",.     "vers
339C 1290: 69 6F 6E 22 3A 20 22 67  6F 31 2E 32 32 2E 31 22  ion": "g o1.22.1"

vs
339C 1250: 3A 20 22 67 6F 31 2E 32  32 2E 30 2E 73 72 63 2E  : "go1.2 2.0.src.
339C 1260: 74 61 72 2E 67 7A 22 2C  0A 20 20 20 20 22 6F 73  tar.gz", .    "os
339C 1270: 22 3A 20 22 22 2C 0A 20  20 20 20 22 61 72 63 68  ": "",.     "arch
339C 1280: 22 3A 20 22 22 2C 0A 20  20 20 20 22 76 65 72 73  ": "",.     "vers
339C 1290: 69 6F 6E 22 3A 20 22 67  6F 31 2E 32 32 2E 30 22  ion": "g o1.22.0"
339C 12A0: 2C 0A 20 20 20 20 22 73  68 61 32 35 36 22 3A 20  ,.    "s ha256":

That file is generated via: https://github.com/bazelbuild/rules_go/blob/master/go/private/sdk.bzl#L86-L87
I'm not sure why that file would change, but maybe the sdk versions released across the globe are not identical all at the same time. A solution could be to make it more reproducible by storing a fixed version of that file.

That file is fetched from an endpoint via: https://github.com/bazelbuild/rules_go/blob/master/go/private/sdk.bzl#L86-L87
I'm not sure why that file would change, but maybe the sdk versions released across the globe are not identical all at the same time. A solution could be to make it more reproducible by storing a fixed version of that file.

Do we know how exactly jaxlib depends on rules_go? It's must be a deep transitive dependency...

Likely: https://github.com/bazelbuild/rules_python/blob/49c4dc8239d8d1f8dfab765f6c65750786fd32f5/internal_deps.bzl#L77

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-buildproxy-reproducible-http-https-responder-in-sandboxed-nix-builds/40081/18

any news on that ?

any news on that ?

Is it currently failing ?
The plan anyway is to update the jax package to the latest version.
Unfortunately, they have changed a lot of things and I have not managed to account for them yet.
I don't know if the update will change anything related to this specific issue though.
draft PR: https://github.com/NixOS/nixpkgs/pull/318995

Is it currently failing ?

Yep we currently have a hash mismatch, in 24.11 or 24.05

Yep we currently have a hash mismatch, in 24.11 or 24.05

Oh! I didn't backport https://github.com/NixOS/nixpkgs/pull/323681

Yep we currently have a hash mismatch, in 24.11 or 24.05

Oh! I didn't backport #323681

in 24.11

You mean unstable? Seems to work in hydra:
https://hydra.nix-community.org/job/nixpkgs/cuda/python3Packages.jaxlib.x86_64-linux https://hydra.nixos.org/job/nixpkgs/trunk/python312Packages.jaxlib.x86_64-linux

Indeed, hydra prints no error.
If i have some free time ahead, i'll launch another try in unstable, i'll keep you in touch when i'll have the results.

Has the issue never happened since #323681 is merged (though the root cause still seems unclear)?


Misleading information on pkg.go.dev

In the readme section you can see that v2 is recommended as the current stable version. However, it's no longer true, according to your github repo.

Thanks, that is indeed outdated.

3.0.2 has been tagged without that statement.  pkg.go.dev hasn't got it yet, but it should be there soon.

pkg.go.dev has picked up 3.0.2


updating to latest ofnet for bgp refactoring

The PR contains fixes for:

concurrent map access issues
minor fixes for tests
minor feature fixes
cleanups

LGTM


provide a better error when the user is not logged in

Running ziti edge list ers (or anything really) does not provide a CLEAR error to the user indicating they just aren't logged in. The feedback should be much more obvious than what is currently output.
See:
$ ziti edge list ers
error: no identity 'default' found in cli config /home/cd/.config/ziti/ziti-cli.json

☝ not exactly obvious that they just haven't logged in yet...

I would like to work on this issue.
The codebase is quite big, please help me find files related to this issue!

I would like to work on this issue. The codebase is quite big, please help me find files related to this issue!

That errors comes from here: https://github.com/openziti/ziti/blob/v1.0.0/ziti/util/identities.go#L328
A basic fix would maybe check the identity count and return an error saying you need to login if there are 0 saved logins. Above and beyond that, if the selected saved login isn't found you could output the list of saved logins and let the user know they could either login  or use ziti edge use to select an existing saved login.

Thank you so much @plorenz for the jumpstart.


Revisit docs for --static-swift-stdlib

The main README.md talks about --static-swift-stdlib and maybe some of the other pages.  They likely need a revisit given Swift 5 removing some of the static linkage support.

Just saw https://github.com/apple/swift-protobuf/pull/852 landed the other day, but I thought from other threads I'd read the static library trick wasn't going to generally work?

With https://github.com/apple/swift-protobuf/pull/852 static linking still doesn't work but SwiftPM just warns and doesn't pass the flag through to swiftc


[new feature request] support configuring API URL/DataCentre

This feature is needed as currently conversations, calls, users are tied to a geographic data centre.
In particular, this is most needed when creating an app user, ie. [vonage apps:users:create [NAME]"
About DataCentre, please check https://developer.nexmo.com/client-sdk/configuration/configure-data-center

We should be able to add this into vonage config to allow users to specify.  I'll get this started in the next couple of weeks - thanks!


Submit a Cloud Definition

Do NOT copy/paste a definition from somewhere else. Read about the word you want to define and come up with your own definition. Copy/Paste submissions will be closed and not added.
Fill out the JSON with your submission:
    {
    "word": "Fault Domain",
    "content": "A 'Fault Domain' can be defined as the scope of resources/services sharing a single point of failure that would become unavailable if a fault occurred and is typically used in reference to physical equipment, for example, if the power went down in a datacentre rack resulting in loss of service, this rack and its associated devices is your fault domain. - See also 'Availability Sets' which are available to help to reduce this risk by spreading your resources across multiple 'Fault Domains' or physical spaces with their own power source and networking to allow for redundancy/high availability by removing the single point of failure.",
    "learn_more_URL":"https://learn.microsoft.com/en-us/windows-server/failover-clustering/fault-domains",
    "tag":"disaster recovery, compute, general",
    "abbreviation": "",
    "author_name":"Daryl",
    "author_link": "https://howardlabs.co.uk"
    }

Fill out the JSON below with the following.
Word (REQUIRED)
The word you are defining. Check this URL for all words we currently have.
Content (REQUIRED)
The definition. No more than 3 sentences.
learn more URL (REQUIRED)
Website where people can visit to learn more about the word.
tag (REQUIRED and select one)
Tech category the word fits in. Options:

compute
security
service
general
analytics
developer tool
web
networking
database
storage
devops
ai/ml
identity
iot
monitoring
cost management
disaster recovery

abbreviation (OPTIONAL)
If the word is commonly abbreviated, please provide it. For example, command line interface is often abbreviated as CLI.
author name (REQUIRED)
Your name.
author link (REQUIRED)
The URL you want your name to link to.

Added 👍🏽


Feature request: Support Deno

Which Cloudflare product(s) does this pertain to?
Wrangler core
What version(s) of the tool(s) are you using?
3.60.3 [wrangler]
What version of Node are you using?
Deno 1.44.2 (latest)
What operating system and version are you using?
macOS Sonoma 14.5 (latest)
Describe the Bug
Observed behavior
$ deno run npm:wrangler
error: unexpected argument '--no-warnings' found

  tip: a similar argument exists: '--no-config'

Usage: deno run --allow-all --no-config [SCRIPT_ARG]...

For more information, try '--help'.

Expected behavior
Same as npx wrangler
Steps to reproduce
Install Deno:
$ curl -fsSL https://deno.land/install.sh | sh
Run:
$ deno run npm:wrangler
Cause
https://github.com/cloudflare/workers-sdk/blob/c643a8193a3c0739b33d3c0072ae716bc8f1565b/packages/wrangler/bin/wrangler.js#L34
Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs
No response

We don't officially support deno (or any other runtime) as a runner for wrangler (and I don't think there are immediate plans to do so), but we'd be happy to discuss. Is there a reason you cannot use node?

@threepointone
Thank you for your response.
While it is possible for us to use Node.js, there are several compelling reasons why supporting Deno would be highly beneficial:
Deno offers several advantages over Node.js, including enhanced security, native TypeScript support, simpler module management, built-in standard modules, a modern API design, and a streamlined distribution model. These features make Deno particularly attractive for new projects or those that prioritize security.
1. Security
Deno is designed with security in mind. By default, it restricts access to the file system, network, and environment variables. This minimizes the risk of unintended side effects on the system by running scripts. Access permissions must be explicitly granted via flags, allowing for fine-grained control.
2. TypeScript Support
Deno natively supports TypeScript without the need for additional configuration or tools. This allows developers to write type-safe code easily, improving code quality and reducing errors.
3. Module Management
Deno does not rely on a package.json file or a centralized repository like npm. Instead, it uses a simple mechanism for importing modules via URLs. This simplifies dependency management and allows for explicit version specification.
4. Standard Modules
Deno includes a set of standard modules that provide common functionalities out of the box. This reduces the reliance on external libraries for basic tasks. Moreover, standard modules are released alongside Deno, ensuring compatibility and reducing issues.
5. Simple and Modern API
Deno's API is designed to be simple and modern, utilizing async/await for asynchronous operations. Additionally, it aims to have the same code run both in the browser and on the server, promoting code reuse.
6. Single Executable
Deno is distributed as a single executable file, which eliminates the need for dependency management. This makes installation and distribution straightforward.
7. Adoption of Best Practices
Deno incorporates many best practices learned from Node.js. This includes the removal of deprecated features and minimizing global objects, leading to a cleaner and more predictable environment.
Given these reasons, we strongly hope you will consider supporting Deno. This support would not only enhance the development experience but also bring substantial benefits to the broader developer community.
Thank you for your consideration.

None of those benefits apply to using a tool like wrangler as a CLI for local development; particularly for development/deployment for another runtime (Workers). I'm going to close this issue, but we'll be happy to revisit it in the future if the need becomes more common.

I"m happy to reopen the issue, and I'll leave it to the team to decide, but I strongly suspect they won't. It becomes another target to maintain, and test against, and guarantee compliance with. It's non-trivial.
In your specific case, you can get around the need for a package.json / npx by installing it globally (with npm install wrangler -g), a bunch of our users already do that.


installation problem

I am on debian and have llibxml2 package (2.9.4) installed. Still this fails
$ sudo gem install libxml-ruby
Building native extensions.  This could take a while...
ERROR:  Error installing libxml-ruby:
ERROR: Failed to build gem native extension.
current directory: /var/lib/gems/2.3.0/gems/libxml-ruby-3.0.0/ext/libxml

/usr/bin/ruby2.3 -r ./siteconf20170908-17659-mcmmoq.rb extconf.rb
checking for libxml/xmlversion.h in /opt/include/libxml2,/opt/local/include/libxml2,/usr/local/include/libxml2,/usr/include/libxml2... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/usr/bin/$(RUBY_BASE_NAME)2.3
--with-xml2-config
--without-xml2-config
--with-xml2-dir
--without-xml2-dir
--with-xml2-include
--without-xml2-include=${xml2-dir}/include
--with-xml2-lib
--without-xml2-lib=${xml2-dir}/lib
extconf failure: need libxml2.
Install the library or try one of the following options to extconf.rb:

  --with-xml2-config=/path/to/xml2-config
  --with-xml2-dir=/path/to/libxml2
  --with-xml2-lib=/path/to/libxml2/lib
  --with-xml2-include=/path/to/libxml2/include

To see why this extension failed to compile, please check the mkmf.log which can be found here:
/var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/libxml-ruby-3.0.0/mkmf.log
extconf failed, exit code 1
Gem files will remain installed in /var/lib/gems/2.3.0/gems/libxml-ruby-3.0.0 for inspection.
Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/libxml-ruby-3.0.0/gem_make.out

Ok, I have resolved it by install ing libxml2-dev package :))
Sorry

@gorn thanks for posting your fix, found this via Google and it solved it for me as well. :smile:


Social media preview image

@tetogomez please have a look at this tool that tests social media images , it seems we have several things to fix
https://socialsharepreview.com/?url=https://mainnet.eosio.online

@tetogomez it seems the social media preview image is not working


[INCR 75] only require futures for python 2.7

Thanks for the pull request, @singuliere! I've created OSPR-3064 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.
Feel free to add as much of the following information to the ticket:

supporting documentation
edx-code email threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.

The CI errors are unrelated to this PR
line 2411, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages/toxbat/requirements.py", line 38, in <module>
    from pip.download import PipSession
ImportError: No module named download
The command "tox" exited with 1.

Is it a known problem?

@singuliere Thank you for your contribution. Can you let me know when tests pass?

blocked by https://github.com/edx/xblock-sdk/pull/146

@singuliere Thanks for the update!

rebased on top of #146 to verify it improves the situation

@natabene this PR still contains all commits from #146 (required for tests to pass) and is therefore not yet fit for review. It will be rebased once #146 is merged. Thank you for the kind attention on each and every contribution: it is very reassuring to know you're around to make sure nothing is left unattended!

@singuliere Thanks for the update and your kind words, let me know once it is ready, so we can start reviewing.

#146 has been merged, you should be clear to rebase now.

rebased & repushed :-)

@singuliere 🎉 Your pull request was merged!
Please take a moment to answer a two question survey so we can improve your experience in the future.


New sparklines use unintuitive scaling

I'm a recent (but big) fan of macmon. Had set out to build something like this myself a few months ago and discovered this excellent project.
I recently updated to v0.3.0 and was very confused, but realised it's because the sparklines now seem to be doing some relative scaling instead of representing the absolute percentage. I can see how that might be better for processors, but I find that deeply unintuitive and unhelpful for RAM:

The above looks to my eye like I'm using nearly 100%, but I'm actually only using about 10%. Memory usage, in particular, doesn't spike as much as processors and GPU, so this ultimately just looks like a big block most of the time.

I forked and cloned to run a git bisect. It seems like the behaviour change is from feat: switch chart type

In 6cbcd72

In 54aed4d

@vladkens it seems like it was just a simple issue in which value was being giving as the max for the RAM sparkline specifically. Please see PR #8

@gianlucatruda thank you for find this and for PR


Explore how to show infinite progress in reduced motion mode

In reduced motion mode today, we still use animations to show infinite progress bars and spinners. We should figure out a different way to indicate this to users  in reduced motion without such a distracting animation

Possible references:

https://github.com/microsoft/fluentui/pull/25561
https://github.com/twbs/bootstrap/pull/31882

fyi @meganrogge

Is there a reason we need animation when in reduced motion mode? If users find it distracting to see an animation, not sure changing the speed will help?

For @hbons - here's an example of the progress bar.
We also occasionally show spinners in the status bar " Building..." which may also need to be looked at.
https://user-images.githubusercontent.com/25163139/226985600-916ca5e6-7f4e-4e0e-930a-b1582b43f857.mp4

Here are a few experiments with reduced motion: https://jsfiddle.net/vjfou76c/

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


⚠ Bofrost.lu has degraded performance

In f4c49fb, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1424 ms

Resolved: Bofrost.lu performance has improved in 9b0f166 after 8 minutes.


How to map NS430 commands in FC3 modules?

Gudday,
I'm one of those weirdos who enjoys playing with the NS430 in DCS, and I noticed that I can't add the NS430 to a basic key emulator profile, which is how I thought I could get around the FC3 modules not being clickable.. How can I create a profile that will allow me to use the flight panels to manipulate the NS430 in FC3 modules?
Thanks!
Nick

Regarding NS430 and FC3. I really do not know anything about NS430 or FC3. I just implemented the DCS-BIOS part of NS430. I guess it would be better to ask on Discord of users experiences. BadCrC for example.

i think Nich means that it would nice if NS430 can activated also in keyemulator mod.
atm its grayed out.

To offer further clarification;

In the early days of DCSFP, NS430 was offered as it's own DCS-BIOS module,
After a request from a user in the forum, NS430 was removed as it's own module and integrated as an option for other modules, so that you could control the NS430 and your choice of DCS-BIOS aircraft.
I know that FC3 aircraft don't have DCS-BIOS modules as they don't have clickable cockpits.
I'd like to control the NS430 in FC3 aircraft.
I think the only way I could do this would be if you re-added the NS430 module as a standalone DCS-BIOS module.

Would it be possible to re-add a standalone NS430 module while keeping the NS430 also integrated to the other modules?

@Nick-Thain, could you DM on our Discord?

I'm there now, I assume I DM ArturDCS?

Seems like it is not possible at the moment. For some reason there aren't any devices in the lua environment corresponding to the devices that are there whilst having a non-FC3 module.

Gudday mate,
Yeah I'm finding the same thing here, I'm just grateful that the NS430
works with the other DCS-BIOS modules now! Really appreciate you taking the
time to take a deeper look at it for me mate.
In the interim, I found a great mod on the DCS User Files that fixes the
rotary dial issue where each key press used to equal several 'clicks' of a
rotary dial; with the mod, each key press turns exactly one 'step'/click,
so I've now mapped that function to an MFD button pair.
Thanks again!
Nick
On Thu, Mar 18, 2021 at 9:13 AM Jerker Dahlblom @.***>
wrote:

Seems like it is not possible at the moment. For some reason there aren't
any devices in the lua environment corresponding to the devices that are
there whilst having a non-FC3 module.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/DCSFlightpanels/DCSFlightpanels/issues/109#issuecomment-802057139,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ARET4GHTNHFR6IJUTQCVAELTEIRDLANCNFSM4YXJGNLA
.


Adding Support for PortinsModel for TargetRespOrgId

Super Simple -- adding support for TargetRespOrgId in the PortinsModel
Changed locally and tested working, submitted via API, Port Order ID 1eb76805-21e7-4ac0-9b18-6d35425aac34
Haven't added the other yet, but I figure since I'm not getting paid for it, the BW team can!

Sorry, that was awful, I'll rebase


Add com.google.cloud.sql postgres-socket-factory java library

Description
This library enables JDBC connections to GCP Cloud SQL postgres databases using IAM (RBAC) rather than username and password security. This change is a no-op for most deployments except for making the prod Docker image slightly larger.
Checklist
General
Read the full guidelines for PRs here

[ ] Added the correct label: < feature | enhancement | bug | under-development | dependencies | infrastructure | ignore-for-release | database >
[ ] Assigned to a specific person, civiform/developers, or a more specific round-robin list
[ ] Added an additional reviewer from outside your organization as FYI (if the primary reviewer is in the same organization as you)
[x] Removed the release notes section if the title is sufficient for the release notes description, or put more details in that section.
[x] Performed manual testing (Chrome and Firefox if it includes front-end changes)

Instructions for manual testing

Follow instructions here for creating a DB and a service account with access to it: https://cloud.google.com/sql/docs/postgres/connect-overview
CiviForm JDBC string should look like: jdbc:postgresql:///${DB_NAME}?cloudSqlInstance=${DB_CONNECTION_NAME}&socketFactory=com.google.cloud.sql.postgres.SocketFactory

I followed these steps and the connection succeeded.
Issue(s) this completes
Fixes #<issue_number>; Fixes #<issue_number>...

reviewers please self-assign, add labels, and run PR workflows

Thanks for reviewing @nb1701 and @dkatzz! I don't have a merge button so if it looks good please merge.


cmd/link: missing arg stack map for functions defined in different package

What version of Go are you using (go version)?
tip (86a338960d)
Does this issue reproduce with the latest release?
yes
What did you do?
https://play.golang.org/p/4-OeiC68PO4
package main

import "bytes"

func growstack(n int) {
	if n > 0 {
		growstack(n - 1)
	}
}

func main() {
	var x, y []byte
	defer bytes.Compare(x, y) // on AMD64 bytes.Compare is implemented in internal/bytealg
	growstack(10000)          // trigger a stack grow
}

$ go run b.go
runtime: frame bytes.Compare untyped args<PHONE_NUMBER>36+56
fatal error: missing stackmap

runtime stack:
runtime.throw(0x1073aff, 0x10)
	/Users/cherryyz/src/go-tip/src/runtime/panic.go:598 +0x72
runtime.adjustframe(0x7ffeefbff518, 0x7ffeefbff630, 0x10b3be0)
	/Users/cherryyz/src/go-tip/src/runtime/stack.go:698 +0x578
runtime.tracebackdefers(0xc000000180, 0x1077300, 0x7ffeefbff630)
	/Users/cherryyz/src/go-tip/src/runtime/traceback.go:73 +0xda
runtime.adjustdefers(0xc000000180, 0x7ffeefbff630)
	/Users/cherryyz/src/go-tip/src/runtime/stack.go:733 +0x45
runtime.copystack(0xc000000180, 0x4000, 0x10c5f01)
	/Users/cherryyz/src/go-tip/src/runtime/stack.go:878 +0x17e
runtime.newstack()
	/Users/cherryyz/src/go-tip/src/runtime/stack.go:1063 +0x30e
runtime.morestack()
	/Users/cherryyz/src/go-tip/src/runtime/asm_amd64.s:481 +0x8f

The stack map of bytes.Compare is missing.
The compiler does generate the stack map
$ go build -gcflags -S bytes
...
"".Compare.args_stackmap SRODATA size=12
        0x0000 02 00 00 00 0e 00 00 00 09 00 09 00              ............
...

But it doesn't make into the binary. Apparently the linker dropped it on the floor?
This doesn't seem to be the case for assembly function defined in the same package.

Ok, for normal assembly functions, the obj package will insert a FUNCDATA which refers to the args_stackmap generated by the compiler from the Go declaration.
https://go.googlesource.com/go/+/master/src/cmd/internal/obj/plist.go#78
But this is only done for functions whose name starts with ""., which means, defined in the same package.
Maybe we can make the obj package also insert FUNCDATA for functions not defined in the same package as the declaration? Or the assembly writer is expected to write a GO_ARGS pseudo-instruction (https://go.googlesource.com/go/+/master/src/runtime/funcdata.h#34) for this kind of functions?

CC @aclements @randall77

I'd rather avoid the GO_ARGS solution.  We can certainly do that for internal/bytealg, but this problem would occur for anyone writing an assembly routine that is exported.  I don't want them writing a GO_ARGS declaration when we have all the information we need to construct it for them.
What happens if we just remove the "". restriction?  Does everything still work?  How much extra linker work is it?  Extra binary size?
Maybe we could restrict to functions which have a declaration but no body.  I'm not sure whether we have that bit in the data exported from a package, though.

What happens if we just remove the "". restriction? Does everything still work?

This doesn't work, since not all assembly functions have Go declarations. For them, it will result in undefined relocation targets at link time. I tried restricting to only functions whose name has a ".", but still some functions don't have Go declarations. Maybe we can establish a convention that if a function doesn't have Go declaration (therefore cannot be called from Go), it should not have "." (written as "·" in assembly source code) in its name? We could fix up the exceptions in std, but not sure what to do with code outside our repo.

Can we restrict to exported functions?  I think it would be reasonable to require that exported assembly functions have Go declarations.  In fact, I don't think it would work otherwise (unless assembly was calling assembly?).

I'd rather avoid the GO_ARGS solution. We can certainly do that for internal/bytealg, but this problem would occur for anyone writing an assembly routine that is exported.

I may be misunderstanding, but doesn't this also require that the assembly routine be implemented in a different package than its Go declaration? I'm okay with saying that end users don't get to do that.

@aclements Yes, I guess that's true.
In any case, I have a patch that restricts to exported functions, and it seems to work.  I'll mail it in a sec.

My patch didn't quite work, due to shared libraries.
When building runtime as a shared library, that shared library actually includes the implementations of bytes.IndexByte, for example.  That implementation now references bytes.IndexByte.args_stackmap as its args pointer map funcdata.  But that symbol doesn't exist when compiling runtime, and won't exist if you never use the bytes package.
Adding something like:
FUNCDATA $0, ·IndexByte·args_stackmap(SB)

to the bytes.IndexByte assembly fixes things.  Of course, that's probably the original GO_ARGS fix Cherry suggested.
I guess that's good enough to fix this bug, I'll whip up a CL.
There's the remaining question of whether my CL 122519 should be submitted.  I think it might still be a good fix for people deferring exported assembly functions which don't have package "". (Presumably people use pkg·Func as well as ·Func to declare their assembly functions.)


Remove > from iui-browse-input

Fixes #146

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


AnimationEditor keys button fix

The animation key buttons (Add key, frame mode, interpolation mode, loop interpolation mode) all had a slight offset, meaning the mouse click wasn't detected in the right area. It was most noticeable on the add key button since it's a much smaller area than the others.
Fixes one of the issues in #11359

adding a key should be more consistent (sometimes pressing the green + does nothing)

Changes look fine. Please squash commits together. (PR workflow)

Okay, working on this, my tab key just decided to die, so it's making this more of a pain than it needs to be ;)
Should be done in a moment hopefully

Okay, I think that's sorted

I've fixed this with https://github.com/godotengine/godot/pull/13039
Just need to get it apporved


Vaadin UI Enhancement

Overview of the issue
Is it possible with a little effort to adapt Vaadin CRUD-UI Addon (Apache 2.0) to Jeddict?
https://vaadin.com/directory/component/crud-ui-add-on
https://github.com/alejandro-du/crudui
Expected Outcome

Current Outcome

Steps to Reproduce(Only for bug reports)

Jeddict Version(s)

Ofcourse, Can you create the sample CRUD application with One (or more) entity.


Estimate probability of symptomatic and severe disease given risk budget

Hi!
This is a lovely tool to build intuition around risk and I can't believe I didn't know about this before.
I was wondering if it wouldn't be possible to, aside from going above / below a risk budget, also estimate probability of symptomatic disease, severe disease and perhaps death? It seems like the current model is considering any infection at all, which seems to ignore one of the best benefits of getting vaccinated: less severe course of disease. (Right? Please correct me if I'm wrong here)
E.g. Given a 1%-per-year risk budget of disease, does this imply 1 * (1-0.8) = 0.2% yearly chance of symptomatic illness and 1 * (1 - 0.95) = 0.05% against severe disease? (given e.g. ~80% efficacy against symptomatic disease with 2 doses, ~95% against severe) You could make this more complex with risk profile details: age group and immune-affecting conditions.
It would be one interesting way to demonstrate another effect of the vaccines: the second you hit the vaccination button you'd see that risk of death and severe disease gets much closer to 0, but perhaps risk of infection still stays fairly high.
It would also probably affect people's risk calculus a bit: e.g. for someone who lives and works alone (low propensity to spread), but is vaccinated, a 1-2% per year chance of asymptomatic or mild symptomatic infection (thus larger than 1% chance of any infection) could be more acceptable than prolonged social isolation, as long as the chance of moderate+ disease remains low. For a person without a vaccine, or that has an immune condition or a housemate with one, the math is of course different.
What do you think about this? Thank you!

I personally love this idea.
The two big question marks we'd need to think through (that I currently see) are:

The big lift it would be for our research time to nail down these numbers in ways we are confident about.
How to present even more information for folks without overwhelming them. (I hear @jy2wong asking many follow up questions in my mind right now.)

I'd also want to see "chance of getting long COVID" on that list of stats (once we have the data)
And we'd also need to figure out a way to display those % chances. Because they are very very small %s. But I doubt we would want to use a "multiply by 1 million" approach because then people will start to get confused about "which number is actually a microCOVID (chance of getting COVID)" 🤔

The big lift it would be for our research time to nail down these numbers in ways we are confident about.

I've already seen discussions in this bug tracker around which study / numbers to use for various vaccine efficacy multipliers, perhaps this is an excuse to formalize that process. Not every study has every efficacy number but a lot of them have symptomatic / hospitalization / ICU etc split out, and some sources collating various results (https://www.ft.com/content/5a24d39a-a702-40d2-876d-b12a524dc9a5). But I can't deny that it's probably quite a bit more work :-)

How to present even more information for folks without overwhelming them. (I hear @jy2wong asking many related follow up questions in my mind right now.)

One way to think about it is that it could be done as a few new caution budget / risk tolerance options, like "1% yearly risk of mild to moderate disease given vaccine X" or somesuch.

I'd also want to see "chance of getting long COVID" on that list of stats (once we have the data)

Gosh, yes.

Hi! I hope you don't mind a comment from a lurker. I use microcovid.org lots, and really appreciate it. (Bias disclosure: my household health objective is "avoid infection" mainly due to our concerns about long covid.)
What I don't love about trying to model severity is that you probably need new inputs in order to quantify it responsibly. Based on the research IIUC it doesn't seem right to tell someone their risk of hospitalization is low without asking their age, gender, whether they're immune compromised, have asthma, or other risk factors not currently modeled. The tool encapsulates these factors in the user's choice of their budget, but it leaves it to them.
I could imagine a second tool that's like "budget calculator" that helps you decide what your overall microcovid budget should be, and let that be a separate decision from the day-to-day microcovids?

my household health objective is "avoid infection" mainly due to our concerns about long covid.

We're on the same page. I have been extremely careful so far in avoiding even relatively minor risk because of concerns around long term effects (I think being a user of this tool self-selects for that to some degree).
I'm just about due my second shot and was excited for it, and so I was trying to see how my risk assessments change based on that. What drove me to ask this was that, well, they didn't seem to change all that much. This surprised me and I was trying to figure out why, which led me to realize that the tool is somehow not reflecting benefits associated with vaccines in the case of getting infected, according to one study (https://www.ama-assn.org/delivering-care/public-health/what-doctors-wish-patients-knew-about-breakthrough-covid-infections, which I realize yet isn't a scientific consensus) it seems that even in the case of a breakthrough infection, 25-30% is completely asymptomatic, a further 50-60% have mild symptoms.  Another one has results in a similar vein, with a lot of the infected only realizing because they got tested for some other reason (https://www.nejm.org/doi/full/10.1056/NEJMoa2109072). AFAIK there's also concern that we undercount breakthrough infections in self-reported population studies because such a large portion is asymptomatic. Then again, I'm not sure if that implies that that's "built in" to current efficacy numbers.
I think the reason this feels a bit icky (at least to me) is because it seems like you'd be somehow encouraging people to get infected by relaxing the boundaries of what's acceptable. In reality we're already playing this game: just because the tool marks something "low risk" doesn't mean you can't get sick from it - it doesn't make it automatically safe, just safer, and not doing it it always less risky than doing it. Similarly with the 1% per year target - 1% is a number chosen arbitrarily because we use the decimal system and 1 is a small number, but I'm not certain most people can really quantify the difference between 1% and 2% a year risk. I certainly can't. The point is that we want to do better than a generic "this is safe" and "this is not" and limit the risk at a reasonable level, and targeting e.g. risk of getting long covid or a symptomatic infection might be a much more achievable target for some than risk of any infection.

Based on the research IIUC it doesn't seem right to tell someone their risk of hospitalization is low without asking their age, gender, whether they're immune compromised, have asthma, or other risk factors not currently modeled.

This kind of averages papering over over population variance issue I think is true for many other things that the tool provides: e.g. the 2 dose Pfizer multiplier is 0.17. Is it the same for elderly or immunocompromised people? Is it the same for every variant? Certainly not. Similarly with risk of infection - I don't want to perpetuate the "kids don't get sick from covid" myth but below age 15, they do seem to get ill at much lower rates, and the tool ignores that too. I'm not sure how one decides which detail to include and which to ignore, but it's probably limited by available research.
But also, IMHO the entire point of this tool is to be able to get /specific/ assessments considering specific factors in specific situations, and not rely on simple averages. I'd say that asking some of these risk factors would be entirely within the spirit of the tool, if the model / information to then do estimates from that was available.

I could imagine a second tool that's like "budget calculator" that helps you decide what your overall microcovid budget should be, and let that be a separate decision from the day-to-day microcovids?

That's a perfectly reasonable solution I think!
Sorry for the long post, part of this is also me working this through in my head to see what I think. I'd love for someone actually knowledgable to chime in.


Develop a true Web Component version of time-elements

I'm trying to use time-elements in an application built on top of Polymer (2.0) and Web Components and I can't seem to get this to work since there is no real web component defined for these custom HTML tags. I know this is added to the Web Components website but it only has the .js file and no HTML files to import. My request is two-fold:

Make full-blown Web Components for each custom HTML tag (eg. <relative-time>, <local-time>, etc.) with HTML imports.
Is there a way to properly use the current implementation in a Polymer 2.0 Web Component?

Thank you for your work!
Edit: I forgot to mention that I'm relatively new to Web Components and Polymer so it's very possible that I'm missing something and time-elements in its current form is already a real web component and I just don't realize it.

After some searching I found out what my problem was from Issue #54 . Apparently Polymer was not binding my variable to the datetime attribute because I wasn't using the attribute binding syntax of $= instead of just the =. So, in my case I needed to use something like datetime$=[[bindingVariable]] and that worked.
As for where to reference the time-elements.js, I was able to include it in my top-level Web Component component and it found my <relative-time> tag inside a child Web Component. I wasn't sure that would work, but that's because I'm still learning Web Components.
So, I've resolved point 2 above, but it would still be really helpful to create real HTML imports instead of having to import the script directly.


Wrong Checkmark Icon

I get the wrong icon for checked checklist items.

I am on Mac 14.6.1
Plugin Version v1.5.7
I was looking into it myself a little. In my editor all the icons are not displayed.

I am curious why that is? Is it because the font is not installed on my computer?

Thanks for providing a PR! It was just merged and will be available as an update in Joplin once @joplinbot has updated https://github.com/joplin/plugins (runs every 30 mins.).


Update appveyor.yml configuration

[ ] Only include master and develop branches

See here:
http://www.appveyor.com/docs/appveyor-yml


1.0.0-beta.8 Release

Versioning
Which Crates to Increment
To check if any of the top-level crates need version increments, go to the zebra GitHub code page: https://github.com/ZcashFoundation/zebra and use the last modified dates of each crate.

[x] Increment the crates that have new commits since the last version update
[x] Increment any crates that depend on crates that have changed
[x] Keep a list of the crates that haven't been incremented, to include in the PR

Version Locations
Once you know which versions you want to increment, you can find them in the:

[x] zebra* Cargo.tomls
[x] tower-* Cargo.tomls
[x] zebra-network protocol user agent: https://github.com/ZcashFoundation/zebra/blob/main/zebra-network/src/constants.rs
[x] README.md
[x] book/src/user/install.md
[x] Cargo.lock: automatically generated by cargo build

Reviewing Version Bumps
Check for missed changes by going to:
https://github.com/ZcashFoundation/zebra/tree/<commit-hash>/
Where <commit-hash> is the hash of the last commit in the version bump PR.
If any Zebra or Tower crates have commit messages that are not a version bump, we have missed an update.
Also check for crates that depend on crates that have changed. They should get a version bump as well.
README
As we resolve various outstanding known issues and implement new functionality with each release, we should double check the README for any necessary updates.
We should check and update if necessary:

[x] The "Beta Releases" section
[x] The "Known Issues" section

to ensure that any items that are resolved in the latest release are no longer listed in the README.
Change Log
We follow the Keep a Changelog format.
We use the Release Drafter workflow to automatically create a draft changelog.
To create the final change log:

[x] Copy the draft changelog into CHANGELOG.md
[x] Delete any trivial changes. Keep the list of those, to include in the PR
[x] Combine duplicate changes
[x] Edit change descriptions so they are consistent, and make sense to non-developers
[x] Check the category for each change

prefer the "Fix" category if you're not sure

Change Categories
From "Keep a Changelog":

Added for new features.
Changed for changes in existing functionality.
Deprecated for soon-to-be removed features.
Removed for now removed features.
Fixed for any bug fixes.
Security in case of vulnerabilities.

Create the Release
Create the Release PR
After you have the version increments and the updated changelog:

[x] Push the version increments and the updated changelog into a branch
(name suggestion, example: v1.0.0-alpha.0-release)
[x] Create a release PR by adding &template=release-checklist.md to the
comparing url (Example).

[x] Add the list of deleted changelog entries as a comment to make reviewing easier.
[x] Also add the list of not-bumped crates as a comment (can use the same comment as the previous one).

[x] While the PR is being reviewed, turn on Merge Freeze to stop other PRs merging

Create the Release

[ ] Once the release PR has been approved and merged, create a new release
using the draft release as a base, by clicking the Edit icon
in the draft release.
[ ] Set the release title to Zebra  followed by the version tag, for example:
Zebra 1.0.0-alpha.0
[ ] Copy the final changelog of this release to the release description
(starting just after the title ## [Zebra ...)
[ ] Set the tag name to the version tag, for example: v1.0.0-alpha.0
[ ] Set the release to target the main branch
[ ] Mark the release as 'pre-release' (until we are no longer alpha/beta)
[ ] Publish the release

Final Testing

[ ] After tagging the release, test that the exact cargo install command in
README.md works (--git behaves a bit differently to --path)
[ ] Turn off Merge Freeze to start merging PRs again

If the build fails after tagging:

fix the build
increment versions again, following these instructions from the start
update README.md with a new git tag
update CHANGELOG.md with details about the fix
tag a new release

@mergifyio update

I didn't bump the tower crates, because they hadn't changed, and their dependencies hadn't changed.
I bumped all the Zebra crates to keep their versions consistent.
I did not delete any changelog entries.


Question: Am i doing this correctly?

sparsestructures.txt
Can you tell me if im adding these structures correctly? Thank you!

well the file has to be named sparsestructures.json5 and placed in the config folder, but other than that, if the namespaces are correct, then it should be fine.


Boot time database init to deprecate nv migrate

There are a handful of values that must exist in the database in order for the server to work as expected. These include the openid, profile, and realm scopes and the authority role, along with default permissions.
At present, these values are created by running nv migrate before first running the server. This is not ideal for a variety of reasons. First, if overlooked, auth flows won't work correctly. Second, it's a pain to run this in the context of a Docker based deployment.
I propose that as part of the bootstrapping process, the server should check the database for the existence of these values and create them if they're missing. This also gives us a convenient place in the code the throw an error if the database is unreachable, resolving #39.

A draft of this behavior (sans unreachable Redis error) now lives in the bootdb branch: https://github.com/anvilresearch/connect/commit/904d7447cc31a6c7184d477b82cd6113b0feb4b4

Merged #142


Issue in generate_asd_dataset when f_s is different than channel sampling frequency

Using dingo_generate_asd_dataset fails when the sampling frequency f_S specified in the asd_dataset_settings.yaml file does not correspond to the sampling frequency of the requested channel. I think this could be fixed adding a resampling step to https://github.com/dingo-gw/dingo/blob/8c68a7c4e3ea75e6aade9027eea4bef331fa4a1c/dingo/gw/download_strain_data.py#L54 to resample to the requested sampling frequency in the settings file.

As hector mentioned this could be fixed by adding a resampling step to gwpy.TimeSeries. If we want to use the gwpy resampling method this would involve running t.resample(4096) and if we want to use the lal method would need to modify the following code snippet
# Resampling to 2048 Hz
lal_timeseries = {}
for ifo in ["H1", "L1"]:
    lal_timeseries[ifo] = raw_data["strain"][ifo].to_lal()
    lal.ResampleREAL8TimeSeries(lal_timeseries[ifo], float(1 / settings_raw_data["f_s"]))
    raw_data["strain"][ifo] = gwpy.timeseries.TimeSeries(
        lal_timeseries[ifo].data.data,
        epoch=lal_timeseries[ifo].epoch,
        dt=lal_timeseries[ifo].deltaT,
    )


[0.8.x] GUVNOR-2464: Move StyleHelper from kie-wb-common to uberfire-extensions

See https://issues.jboss.org/browse/GUVNOR-2464
Corollary of https://github.com/uberfire/uberfire-extensions/pull/238 for 0.8.x
Branch name consistent with kie-wb-common to keep CI sweet.
(cherry picked from commit<PHONE_NUMBER>5dc56c7ddbdea162d8f280aff97181)

See also https://github.com/droolsjbpm/kie-wb-common/pull/313

@porcelli ...

0.8.x: a4d427e3999adc93df009a1b2fc8515fa6ee2820


Use old base image for aarch64

The latest image is based on Alpine 3.9, which has issues with python3
segfaulting.
This fixes #8
A better long-term fix is probably to switch to using homeassistant/{arch}-base-python for all platforms, since it apparently has a working python3 pre-installed (it doesn't use the version from the alpine repos), but that's a bit more work (the Dockerfile in this project would need some changes).

See https://hub.docker.com/r/campbellr/aarch64-googlebackup/ which was built with this change:
 $ uname -a
Linux a0d7b954-ssh 4.14.81-v8 #1 SMP PREEMPT Sun Feb 3 21:35:18 UTC 2019 aarch64 Linux
$  ~ docker run -it campbellr/aarch64-googlebackup:1.5.2 python3 -c 'print("it works")'
it works

See https://hub.docker.com/r/campbellr/aarch64-googlebackup/ which was built with this change:
 $ uname -a
Linux a0d7b954-ssh 4.14.81-v8 #1 SMP PREEMPT Sun Feb 3 21:35:18 UTC 2019 aarch64 Linux
$  ~ docker run -it campbellr/aarch64-googlebackup:1.5.2 python3 -c 'print("it works")'
it works

@campbellr Thanks for digging into this. I'll look into re-working my Dockerfile to try to use their python base instead.

See https://hub.docker.com/r/campbellr/aarch64-googlebackup/ which was built with this change:
 $ uname -a
Linux a0d7b954-ssh 4.14.81-v8 #1 SMP PREEMPT Sun Feb 3 21:35:18 UTC 2019 aarch64 Linux
$  ~ docker run -it campbellr/aarch64-googlebackup:1.5.2 python3 -c 'print("it works")'
it works

@campbellr I guess I'm going to need a little more help on how to build with that python base. I've been using their build script: https://github.com/home-assistant/hassio-build/tree/master/builder
I can't relate what you're saying to the process I use with their builder.

Hey @samccauley no problem.
I used that same script to build my image too.
It appears that by default that script just sets the $BUILD_FROM build arg (from your Dockerfile) tohomeassistant/{arch}-base. If instead we override that to use homeassistant/{arch}-base for all architectures (by specifying it in build.json) then we should end up with a working version of python in your image on aarch64 (since that python base image appears to build its own version of python instead of pulling the broken version from the alpine repos).
Take a look at https://github.com/campbellr/addon-hassiogooglebackup/commit/bbcf7fbcbd22ce9028623fbf105c38c0d3e64fe3 which does does this.
(I just created that as an example, i haven't fully tested it because that hassio builder script takes hours on my laptop... It seems to work fine for amd64 though)
Let me know if you want me to close this PR and open up a new one with the above commit.

@campbellr I was able to use the python base per your explanation. Thank you. Please give version 1.5.2 of my add-on a try. Let me know how if that works for you on 64 bit hassos.

Seems to be working. Thanks @samccauley


How to paginate the test result ?

Hi there,
Would you kindly clarify how to paginate the result of the filtered data in this form?
Thanks in advance

This is quite old course, but if you'd like to apply pagination to it, you can use this package for instance: https://packagist.org/packages/sebastiansulinski/php-paginator


Vendor buildifier

It would be awesome if the buildifier hook could download buildifier instead of assuming that it is system installed. Buildifier is a go program, so it can simply be wgetted from the github repository with a simple check to determine whether the os is windows, mac, or linux.

Agree that would be way better.
I'd be happy to take a patch, now that I'm subscribed to updates to this repo.  :)


Reference data storage API proposal

This codebase could potentially be used for reference data storage, allowing us to keep a standardized, centralized place to upload & download RE import data, SDK app data, etc. I propose adding an endpoint that provides a JSON-RPC 2.0 api for uploading files:
Methods:

upload_file - Upload a new file and get its ID. Returns a hex-encoded content hash of the file, which represents its ID. If the file already exists, then the user is given the file ID and this is a no-op. The user can provide some optional additional metadata which is stored in Minio's object metadata.
download_file - Download a file from the store using its ID, which is its content hash.

The server can gzip files if they have not already been compressed. We can also have a Python client module that takes care of creating tarballs, gzip compression/extraction, proper low-memory streaming, and checking downloaded data against its content hash.
Optionally, we could have a policy to expire files if they haven't been accessed after a certain time period, such as 500 days.
Blake2b hashing should be used.
Users can upload directories if they pack them into a tarball, and the client can help with this. The server can detect if uploaded files are already gzipped using this

Blake2b hashing should be used.

Why do you recommend it? Speed of hashing?
Have you considered BLAKE3?

Yeah, blake2b is a lot faster, I would use blake3, but the python package seems to be in development, and they warn you against using it in production without extra work. Blake2b is builtin to hashlib in py 3.6+ so very stable

A few contextual questions:

What are the requirements?
What file caching options exist?
How do they stack up?

Good questions..

Requirements: ability for many users to access the same set of reference data in order to run programs such as importers, bioinformatics tools, etc. This is non-workspace data. My major use case is various importers for ArangoDB.

I'm realizing we may need auth for private research data. I can think of a couple use cases. I think we could implement this without much trouble using KBase auth. I will add to the spec above, but this feature could be added later

Other options:

We could use a hosted service like GCS or S3. I really doubt KBase would go this route, and I have no lbl access to anything like this.
We could use the existing file cache server API. However, I think it uses concepts that aren't relevant here, and we could reuse a lot of the code for a different API.
We could use blobstore that Gavin wrote relatively recently. I'm not sure if it has extra API baggage we don't want, and I'm not sure if others don't want this store used for non-workspace data. But this is worth considering.

In Slack, we tentatively decided to use the blobstore API for this:
https://github.com/kbase/blobstore


finish first exercise

Because

This PR

Issue

Closes #XXXXX
Additional Information

Pull Request Requirements

[x] I have thoroughly read and understand The Odin Project Contributing Guide
[x] The title of this PR follows the location of change: brief description of change format, e.g. 01-flex-center: Update self check
[x] The Because section summarizes the reason for this PR
[x] The This PR section has a bullet point list describing the changes in this PR
[x] If this PR addresses an open issue, it is linked in the Issue section
[x] If applicable, I have ensured that the TOP solution files match the Desired Outcome image

Finish exercise 02-flex-header

finished exercise 04-flex-information

finished exercise 07-flex-layout-2


.env.example file and the guide to get environment variable

Is your feature request related to a problem? Please describe.
This is a feature request,
Describe the solution you'd like to see
However, when using t3 stack. I found after running the CLI, the App is not successfully launched because of lacking environment variable.
Describe alternate solutions
I think adding guide on docs about how to environment variable would be super helpful.
Or just have a .env.example file and a link to how to get those environment variable.
Additional information
Hi Theo. Big fan of you, appreciate your work and tech choice.

I found after running the CLI, the App is not successfully launched because of lacking environment variable.

That shouldn't the the case. We generate a gitignored .env as well as a checked in .env.example.
If you didn't get that, please submit a bug report with reproduction steps


[SPARK-18196] [CORE] Optimise CompactBuffer implementation

What changes were proposed in this pull request?
See the JIRA for details - summary is slightly increased footprint in the class but 4% performance increase observed on PageRank with HiBench large
How was this patch tested?
Existing unit tests and HiBench large

Test build #67898 has started for PR 15713 at commit cdf8b8d.

It leads to a bigger memory footprint because of the extra elements that may be allocated but unused in the array? and the performance win is probably avoiding those branches? I could believe it.
But then there's little point to this class. Reading the original problem statement, I wonder why it wasn't easier to just allocate an ArrayBuffer with a small initial capacity?

Test build #67898 has finished for PR 15713 at commit cdf8b8d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67898/
Test FAILed.

+CC @rxin who can elaborate better.
The reason iirc why the first two variables are inline is to do with usual low cardinality of the buffer - it effectively comes for "free" due to padding overheads of an object : lowering number of objects created and so gc impact.
For specific workloads, ofcourse this might not be the case.
Having said that, this was done in 1.0-1.1 timeframe, and perhaps not applicable anymore ?
A more direct optimization could be to have specialized versions of the class for the primitive types (or rely on something like trove or apache collections). We used to do that for some pretty remarkable when working with primitives - for both AppendOnlyMap and CompactBuffer

Thanks for the prompt feedback, we found this opportunity when profiling Spark 1.6.2 with HiBench large and again this showed up as a hot method with the PageRank benchmark, we can gather data to see if it's still hot with Spark 2 also and I'm planning to contribute lots of similar improvements
Paraphrasing from a colleague:

This data structure is the backing data structure used by RDDs that are doing group by operations (we saw it from a PairRDD doing a groupByKey in PageRank)
The downside of the existing implementation is that every method in this class has an if ... else ... if ... else ... which handles element 0, element 1 and then everything else respectively
We found that on PageRank this change provides a throughput boost of around 5% and costs us about 1 MB of estimated RDD size (86.5 MB to just under 88 MB)_

Note that with our testing using OpenJDK 8 we didn't see a noticeable performance improvement (nor a regression) despite the very slight footprint increase (an increase of 2 MB instead of 1.5 MB), ideally we'll improve the performance for everybody so there may be scope for optimisations here that'll be of use to OpenJDK users too

I believe the benchmark, in which case it seems like this class could just be removed at this point, and replaced with an ArrayBuffer that sets an appropriately small initial size. Is that possible to easily benchmark too while you have the machinery in place?

Good point, I'll first see if we can build and pass the unit tests by replacing CompactBuffer with ArrayBuffer[2] and we'd also repeat the profiling (this would verify if groupByKey is still hot for PageRank).
I'll keep the CompactBufferSuite for our experiment and replace the CompactBuffer references with ArrayBuffer to verify we can still perform all of the operations that the tests exercise.
My concern is that in comment for the CompactBuffer class we see the downsides of said ArrayBuffer (the 16 entry Object array we'd get, so we'd indeed want a default two element ArrayBuffer that only grows if needed -- which sounds like what we currently have but encapsulated within this class so we can override two operators and grow the backing array as required)

Having worked with @a-roberts to help develop this change I can see that the type itself may not be required any more with the change proposed, but keeping it around would allow us to distinguish between ArrayBuffers which are supposed to be small and ArrayBuffers that are general purpose so that if the best trade-off for the case of small buffers changes again in the future we still have the single point at which to tune it. I would be interested to hear @srowen thoughts on future tuneability vs maintenance.

@andrewcraik If we are simply delegating to ArrayBuffer, it makes sense to remove the class - it is an additional object created for each key being aggregated.
Alternative would be to implement the array management within CompactBuffer and specialize array for primitives - reducing overhead if relevant.
Having said that, I do think the original intention for the class continues to hold - depends on cardination of values per key. For larger cardinality, it might be as simple changing order of conditions to improve performance: newIndex > 1 followed by 0, 1.

I do agree we should save the extra object allocation, but again whether we should flatten the array into the class or remove the class and use ArrayBuffer directly is the question - do we want to preserve the semantic distinction of the type intents or loose it for simplicity? I'm not sure what the right trade-off is, but preserving the class would provide future ability to tune that would be lost by removal of the type.
While reordering branches might help a bit, the size of the method in bytecodes is inflated by having the primitive fields for 0 and 1 leading to reduced icache locality, longer distance branches and more complexity for the JIT compiler to handle. Part of the performance gain from removing the field is just from making the code smaller and simpler for the Java JIT.
Perhaps if the feeling is that there is value in the semantic meaning of the type due to the trade-offs it implies finding some way to not just delegate might be the best way forward?

What is this benchmark? Can you show the query and the explain plan, if it is in Spark SQL?
The current change doesn't make a lot of sense. If the findings are correct, we should just use ArrayBuffer (or java ArrayList) directly.

The benchmark used is HiBench 5 which works with Spark 1.6, I expect there will be HiBench 6 officially available soon with Spark 2 support
We can see here that the Spark PageRank example is used and that the main functions, in order of their occurrence, are textFile, map, split, slice, groupByKey, mapValues, join, flatMap, reduceByKey

The benchmark sounds fine. I think it would indeed be interesting to see how a simple ArrayBuffer works, then.

So it seems like the benchmark is the worst case for the current implementation -- it's doing a big groupByKey to potentially add a lot of items to the array buffer.
Do we have cases that only add one or two items?

@a-roberts I'm interested in your thoughts on that last point which is a fair one. If it's as-good for small cases and better for large cases to just use ArrayBuffer, that's a great win.

I'll add a print to keep track of how big the CompactBuffer actually gets when used with, say, SparkSqlPerf (1 scale factor then 2 scale factor) and HiBench.
Once this is complete and we have an idea of how big it gets for these workloads, I'll be running the unit tests with an ArrayBuffer(2) replacing all CompactBuffer instances, I have the code ready and hoping to soon finish up the Partitioned* class PR to speed up the PageRank implementation and to contribute the first phase of improvements to the SizeEstimator.

In response to @rxin's question, for HiBench CompactBuffers are used only on PageRank (none of the other 11) and these buffers mainly have between 3 and 40 elements, no more than 60, never with only two elements. The PageRank workload processes 500k pages (large profile), we have 500k CompactBuffer constructor calls and 500k prints in the += method when curSize <= 2, indicating they're always expanding.
I don't know of any cases where we're adding only a couple of elements, I also ran SparkSqlPerf, all 100 queries, again we have no output indicating that we use this class (no prints from the constructor, the growToSize or the += methods).
Here's a breakdown of growBySize invocations (prints the curSize variable) with PageRank so we have an idea of how big the CompactBuffers actually become.
I used the Spark WordCount example on the 677mb stdout file containing my prints to generate this data and we have a total of 18,762,361 growth events.
(3,500000), (4,500000), (5,500000), (6,500000), (7,500000), (8,500000), (9,500000), (10,500000), (11,500000), (12,500000), (13,500000), (14,500000), (15,500000), (16,500000), (17,500000), (18,500000), (19,500000), (20,500000), (21,499998), (22,499995), (23,499992), (24,499978), (25,499951), (26,499879), (27,499729), (28,499321), (29,498517), (30,496984), (31,494114), (32,488878), (33,480328), (34,467214), (35,447829), (36,421619), (37,387790), (38,346826), (39,300660), (40,251266), (41,201702), (42,155372) (43,114024), (44,79886), (45,53196), (46,33580), (47,20146), (48,11569), (49,6222), (50,3143), (51,1491), (52,684), (53,289), (54,126), (55,39), (56,15), (57,6), (58,1), (59,1), (60,1)

On the left we have the CompactBuffer size in elements and on the right we have a number representing how many times this appeared in the output file (therefore the CompactBuffer has grown to have this many elements that many times).
If there are better ways to figure this out or other workloads to suggest do let me know, I've got the code ready that replaces CompactBuffer with ArrayBuffer(2) for profiling and testing.

Performance results against the Spark master branch on a 48 core machine running PageRank with 500k pages follow
Vanilla CompactBuffer, no changes, run time and throughput (bytes per second) provided
ScalaSparkPagerank 2016-12-01 13:16:09 259928115            47.933               5422738
ScalaSparkPagerank 2016-12-01 13:22:41 259928115            45.551               5706309
ScalaSparkPagerank 2016-12-01 13:26:31 259928115            46.745               5560554
ScalaSparkPagerank 2016-12-01 13:28:58 259928115            51.699               5027720
ScalaSparkPagerank 2016-12-01 13:33:26 259928115            48.415               5368751
240.343s / 5 = 48.068s avg

The commit here
ScalaSparkPagerank 2016-12-01 10:26:12 259928115            48.706               5336675
ScalaSparkPagerank 2016-12-01 10:37:30 259928115            48.947               5310399
ScalaSparkPagerank 2016-12-01 10:40:16 259928115            49.768               5222796
ScalaSparkPagerank 2016-12-01 12:55:37 259928115            48.873               5318439
ScalaSparkPagerank 2016-12-01 12:58:12 259928115            47.535               5468141
243.829 / 5 = 48.7658s avg

Way too similar so attributing this to benchmark noise, without the 51s run this would be a few percentage points worse though
Use an ArrayBuffer (initial capacity of 16, default) instead of CompactBuffer
ScalaSparkPagerank 2016-12-01 13:42:45 259928115            62.190               4179580
ScalaSparkPagerank 2016-12-01 13:55:20 259928115            54.112               4803520
ScalaSparkPagerank 2016-12-01 13:59:06 259928115            60.818               4273868
ScalaSparkPagerank 2016-12-01 14:06:26 259928115            57.428               4526156
ScalaSparkPagerank 2016-12-01 14:35:01 259928115            58.218               4464737
292.766 / 5 = 58.5532s avg

Use an ArrayBuffer (initial capacity of 2) instead of CompactBuffer
ScalaSparkPagerank 2016-12-01 15:31:16 259928115            53.544               4854476
ScalaSparkPagerank 2016-12-01 15:36:32 259928115            58.105               4473420
ScalaSparkPagerank 2016-12-01 15:38:45 259928115            53.976               4815623
ScalaSparkPagerank 2016-12-01 15:44:09 259928115            55.174               4711061
ScalaSparkPagerank 2016-12-01 15:50:01 259928115            55.084               4718758
275.883 / 5 = 55.1766s avg

With my tests I see that using an ArrayBuffer is noticeably worse, so I'll continue to look into what's going on to see if we can improve performance here as this is definitely a hot codepath for this particular algorithm

@a-roberts feel free to continue investigating, but at some point it's also important to ask the question: is this micro-optimization worth it? It only improves 4% for a very specific workload based on your measurement, and might regress other workloads but we are not sure about that. It also takes time from a lot of people to look at ...

Although I'd love to find we can get rid of a custom collections class, it seems like we can't do that and that the optimization here doesn't result in a win. Let's close this for now.


Creating road layout programmatically using Matlab functions

Proposal Submission
Proposed title of article
Creating road layout programmatically using Matlab functions.
Proposed article introduction
Road layout can be designed using Matlab codes. Matlab supports a variety of functions that can be used to define road layouts programmatically. Through these functions, different road layouts like straight roads, junctions, roundabouts, and road lanes can be achieved. These structures can be joined to form a road network.
In Matlab, road layout created programmatically has points defined as coordinates in a 2D graph or 3D graph for a complex road network. Also, a driving scenario can be created by inserting blocks in the road layout to simulate traffic flow.
In this article, we are going to discuss how to create different road layouts using codes and then graphically displaying the plotted roads. This is helpful in road designs simulation andtraffic simulations to help solve traffic problems.
Key takeaways

Defining and plotting coordinates in 2D and 3D graphs.
Designing road layout using codes.
Using driving scenario functions.
Basic understanding of traffic simulation.

Article quality
In this article, we implement road layout design and driving scenario  programmatically  using codes other than using the driving scenario designer app.
References
N/A

Seems like a helpful topic. Let's ensure that it adds value to the community. Thanks