local IPv6 address not detected automatically

baresip Android app user reported that baresip is not automatically detecting global IPv6 address of the device although it has two of them:
https://github.com/juha-h/baresip-studio/issues/56#issuecomment-546453814
https://github.com/juha-h/baresip-studio/issues/56#issuecomment-546464204
The only way he can get registration and calls to work over IPv6 is to manually set bind_address to a global IPv6 address of the device. Config has 'prefer_ipv6=yes'.
What could cause this issue?  Anything to do about it?

Maybe the only choice is to override baresip/re network auto-configuration altogether if it has not been designed to work in Android environment?
Something like this:

get prefer_ipv6 from config
get ifname of default network and ip addresses of DNS servers dynamically from Android
with that info, create new network object using net_alloc() baresip API call
using that object, call net_force_change() baresip API function

Would that work?

In baresip API, there is function net_use_nameserver that can be used to dynamically update DNS servers, but I have not found a corresponding function net_use_interface.
Would this kind of function do it provided that ifname is not longer that 63 chars?
void net_use_interface(char *ifname)
{
    struct config *conf = conf_config();
    strcpy(conf->net.ifname, ifname);
    net_force_change(baresip_network());
}

Variable name ifname is a bit misleading, since if I have understood correctly, it can contain either interface name is its IP address.

In order to facilitate IPv6 address use without need to give it in net_interface config variable, I obtained a public IP address from Android OS and then tried to set the address like this:
    struct config_net *nc = net_config(baresip_network());
    LOGD("setting address '%s'\n", native_addr);
    strcpy(nc->ifname, native_addr);
    nc->prefer_ipv6 = strchr(native_addr, ':') != NULL;
    net_force_change(baresip_network());

net_config() is a new API function that just returns &net->cfg.
I tested it with an IPv4 address, since I don't have IPv6 network available. It produced to log:
10-27 18:16:22.524 16681 16681 D Baresip Lib: setting address '<IP_ADDRESS>'
10-27 18:16:22.524 16681 16717 D Baresip Lib: IP-address changed: <IP_ADDRESS>

I got the same result when I tried with the function in the previous comment.
The device has these addresses:
10-27 18:15:43.711 16681 16716 D Baresip Lib: net interfaces:
10-27 18:15:43.711 16681 16716 D Baresip Lib:          lo:  <IP_ADDRESS>
10-27 18:15:43.711 16681 16716 D Baresip Lib:      ccmni0:  <IP_ADDRESS>
10-27 18:15:43.711 16681 16716 D Baresip Lib:         ap0:  <IP_ADDRESS>

and it successfully uses both of the non-lo interface addresses, if I give the address as net_interface config var value and restart baresip.
This is getting desperate. How can the net_interface address be changed on the fly?

I wrote a new API function net_use_interface (below) and with that I'm able to change bind address on the fly.   Would it make sense to create a pull request?
/**
 * Bind to specific interface or IP address and preferred address family
 *
 * @param net         Network instance
 * @param ifname      Interface name or IP address
 * @param prefer_ipv6 Prefer IPv6 address family
 *
 * @return 0 if success, otherwise errorcode
 */

int net_use_interface(struct network *net, const char *ifname,
		      bool prefer_ipv6)
{
	struct sa temp_sa;
	bool got_it = false;
	char buf4[128] = "", buf6[128] = "";
	int err;

	info("Binding to interface or IP address '%s'\n", ifname);

	/* check for valid IP-address */
	if (0 == sa_set_str(&temp_sa, ifname, 0)) {

		switch (sa_af(&temp_sa)) {

		case AF_INET:
			net->laddr = temp_sa;
			break;

#ifdef HAVE_INET6
		case AF_INET6:
			net->laddr6 = temp_sa;
			break;
#endif

		default:
			return EAFNOSUPPORT;
		}

		goto print_network_data;
	}

	err = net_if_getaddr(ifname, AF_INET, &net->laddr);
	if (err) {
		info("net: %s: could not get IPv4 address (%m)\n",
		     ifname, err);
	}
	else
		got_it = true;

#ifdef HAVE_INET6
	err = net_if_getaddr(ifname, AF_INET6, &net->laddr6);
	if (err) {
		info("net: %s: could not get IPv6 address (%m)\n",
		     ifname, err);
	}
	else
		got_it = true;
#endif
	if (!got_it) {
		warning("net: %s: could not get network address\n",
			ifname);
		return EADDRNOTAVAIL;
	}

print_network_data:

	if (sa_isset(&net->laddr, SA_ADDR)) {
		re_snprintf(buf4, sizeof(buf4), " IPv4=%H",
			    print_addr, &net->laddr);
	}
#ifdef HAVE_INET6
	if (sa_isset(&net->laddr6, SA_ADDR)) {
		re_snprintf(buf6, sizeof(buf6), " IPv6=%H",
			    print_addr, &net->laddr6);
	}
#endif

	info("Local network address: %s %s\n", buf4, buf6);

	net->af = prefer_ipv6 ? AF_INET6 : AF_INET;

	if (net->ch) {
		net->ch(net->arg);
	}

	return 0;
}

Related to #834, it might be better to set net->af to AF_INET6 only if net->laddr6 was set.  Also, if net->laddr or net->laddr6 was not set, should it be set to null?

Below is new version of  net_use_interface API function that initializes laddr and ladd6 and (due to net_prefer_ipv6 bug or feature) does not set af to AF_INET6 if IPv6 address is not set.
int net_use_interface(struct network *net, const char *ifname,
		      bool prefer_ipv6)
{
	struct sa temp_sa;
	char buf4[128] = "", buf6[128] = "";
	int err;

	info("Binding to interface or IP address '%s'\n", ifname);

	sa_init(&net->laddr, AF_INET);
	sa_init(&net->laddr6, AF_INET6);

	/* check for valid IP-address */
	if (0 == sa_set_str(&temp_sa, ifname, 0)) {

		switch (sa_af(&temp_sa)) {

		case AF_INET:
			net->laddr = temp_sa;
			break;

#ifdef HAVE_INET6
		case AF_INET6:
			net->laddr6 = temp_sa;
			break;
#endif

		default:
			return EAFNOSUPPORT;
		}

		goto print_network_data;
	}

	err = net_if_getaddr(ifname, AF_INET, &net->laddr);
	if (err) {
		info("net: %s: could not get IPv4 address (%m)\n",
		     ifname, err);
	}

#ifdef HAVE_INET6
	err = net_if_getaddr(ifname, AF_INET6, &net->laddr6);
	if (err) {
		info("net: %s: could not get IPv6 address (%m)\n",
		     ifname, err);
	}

#endif
	if (!sa_isset(&net->laddr, SA_ADDR) &&
	    !sa_isset(&net->laddr6, SA_ADDR)) {
		warning("net: %s: could not get network address\n",
			ifname);
		return EADDRNOTAVAIL;
	}

print_network_data:

	if (sa_isset(&net->laddr, SA_ADDR)) {
		re_snprintf(buf4, sizeof(buf4), " IPv4=%H",
			    print_addr, &net->laddr);
	}
#ifdef HAVE_INET6
	if (sa_isset(&net->laddr6, SA_ADDR)) {
		re_snprintf(buf6, sizeof(buf6), " IPv6=%H",
			    print_addr, &net->laddr6);
	}
#endif

	info("Local network address: %s %s\n", buf4, buf6);

	net->af = prefer_ipv6 && sa_isset(&net->laddr6, SA_ADDR) ?
		AF_INET6 : AF_INET;

	if (net->ch) {
		net->ch(net->arg);
	}

	return 0;
}

baresip's net.c is calling this:
		(void)net_default_source_addr_get(AF_INET6, &net->laddr6);
```

and here is the code (in libre):

```c
int net_default_source_addr_get(int af, struct sa *ip)
{
#if defined(WIN32)
	return net_hostaddr(af, ip);
#else
	char ifname[64] = "";

#ifdef HAVE_ROUTE_LIST
	/* Get interface with default route */
	(void)net_rt_default_get(af, ifname, sizeof(ifname));
#endif

	/* First try with default interface */
	if (0 == net_if_getaddr(ifname, af, ip))
		return 0;

	/* Then try first real IP */
	if (0 == net_if_getaddr(NULL, af, ip))
		return 0;

	return net_if_getaddr4(ifname, af, ip);
#endif
}
```

1. Does your build of libre have HAVE_ROUTE_LIST enabled ?

here is net_if_getaddr:

```c
/**
 * Get IP address for a given network interface
 *
 * @param ifname  Network interface name (optional)
 * @param af      Address Family
 * @param ip      Returned IP address
 *
 * @return 0 if success, otherwise errorcode
 *
 * @deprecated Works for IPv4 only
 */
int net_if_getaddr(const char *ifname, int af, struct sa *ip)
{
	struct ifentry ife;
	int err;

	if (!ip)
		return EINVAL;

	ife.af     = af;
	ife.ifname = (char *)ifname;
	ife.ip     = ip;
	ife.sz     = 0;
	ife.found  = false;

#ifdef HAVE_GETIFADDRS
	err = net_getifaddrs(if_getaddr_handler, &ife);
#else
	err = net_if_list(if_getaddr_handler, &ife);
#endif

	return ife.found ? err : ENODEV;
}
```

Please notice the comment: "Works for IPv4 only"

2. Does your build of libre have HAVE_GETIFADDRS enabled ?

The function `net_getifaddrs()` should support both v4+v6 but
the function `net_if_list` supports only v4.

3. my advice is to add print debug code to libre, re-compile
  and try to figure out which path it takes and where it terminated.

4. also try to use `net_if_debug` to see the real status of the networking side.

Here is an example from a Linux host, default build of libre
and both IPv4 and IPv6 address:

```
/netstat
--- Network debug ---
 Preferred AF:  AF_INET
 Local IPv4:  ens3|<IP_ADDRESS>
 Local IPv6:  ens3|2a01:4f8:c0c:34f9::2
 Domain:
net interfaces:
         lo:  <IP_ADDRESS>
       ens3:  <IP_ADDRESS>
         lo:  ::1
       ens3:  2a01:4f8:c0c:34f9::2
       ens3:  fe80::5054:a2ff:fe02:1840
net routes:
 Destination                                 Next Hop                                 Iface           Type
 <IP_ADDRESS>/0                                   <IP_ADDRESS>                               ens3
 <IP_ADDRESS>/24                               ?                                        ens3
 2a01:4f8:c0c:34f9::/64                      ?                                        ens3
 fe80::/64                                   ?                                        ens3             LINKLOCAL
 DNS Servers from System: (3)
   0: <IP_ADDRESS>:53
   1: <IP_ADDRESS>:53
   2: <IP_ADDRESS>:53
```

As you can see, the "Local IPv6" also appears in the list of net interfaces.

in this build, HAVE_GETIFADDRS is disabled:
https://github.com/alfredh/baresip-android/blob/master/Makefile#L99
try to enable this flag and see if it works.
you might need to provide a copy of ifaddrs.h if it is missing from Android NDK.

Thanks for the tips.
It turned out that Android has getifaddrs starting from API level 24, but source of ifaddrs-android.h plus two other .h files could be included in apps targeting earlier APIs.  The include alternative sounds too complicated to me.  So the practical options are (1) drop IPv6 support from my Android app or (2) increase its minimum API level from 21 to 24.
On the other hand, my app is able to learn all IP addresses (IPv4 and IPv6) the active interface of the Android device has.  So how about creating a new API function that can be used to set net->laddr and net->laddr6?

on a high level I think the policy should be something like:

libre/baresip provides network addresses and dns nameservers for
common platforms.

it is possible to override via baresip.h API for certain platforms that
wants to do that (e.g. Android).

it would be great if you can add such a function to set the IP-address.
struct network {
	struct config_net cfg;
	struct sa laddr;
#ifdef HAVE_INET6
	struct sa laddr6;
#endif

the function can be as simple as:
void net_set_address(struct network *net, const struct sa *ip)
{
    // if IPv4:
    net->laddr = *ip;

    // if IPv6:
    net->laddr6 = *ip;
}

and the function can be called directly after net_alloc.

OK, I could try that.  How about  net->af?

if you need to be able to change this on the fly, you can add an extra setter function for that.
this is orthogonal to setting the IP-address itself.

Alfred E. Heggestad writes:

if you need to be able to change this on the fly, you can add an extra
setter function for that.
this is orthogonal to setting the IP-address itself.

I wrote the function like this:

void net_set_address(struct network *net, const struct sa *ip,
		     bool preferred)
{
	switch (sa_af(ip)) {

	case AF_INET:
		sa_cpy(&net->laddr, ip);
		if (preferred)
			net->af = AF_INET;
		return;

#ifdef HAVE_INET6
	case AF_INET6:
		sa_cpy(&net->laddr6, ip);
		if (preferred)
			net->af = AF_INET6;
		return;
#endif
	}
}

Is is OK or is a separate function needed?

great, this looks good to me.
you can send me a PR or just push the code to master.

Alfred E. Heggestad writes:

great, this looks good to me.
you can send me a PR or just push the code to master.

Perhaps still better with separate setter:
int  net_af(const struct network *net);
int  net_set_af(const struct network *net, int af);

Will work on it.

great, thanks for the code.

Right after baresip_init(), I call net_set_addr() with IPv6 address provided that the active Android
network has an IPv6 address, and then call net_set_af() with AF_INET6 if the user wants to prefer IPv6. I do the same if a new network becomes available after network loss.
I have not been able to test any of that yet, because my mobile operator does not support IPv6 and my Android device is not able to get IPv6 address from my WiFi router using DHCP.
Once some testing gets done, I'll report what happened.


Docker database size

Update the documentation about docker to include this issue, reported through the mailing list. The solution is:
service docker stop
dockerd --storage-opt dm.basesize=30G

I assume this is an issue when the working directory is not mounted in?

Probably yes. Sorry I read this after merging.


DNM sse s3 changes

These are changes for the java s3 tests for sse-s3.  This is a companion to https://github.com/ceph/ceph/pull/44494
There's probably a problem with this in its current form - it still fails teuthology - so will need more fixing. testSSEKMSNotDeclared ObjectTest.java:908

I've pushed out an update to this which disables testSSEKMSPresent for now.
I'm hoping we can find somebody who knows more about gradle/testng to put plumbing in for enabling sse-s3 tests.  Once that logic is present, it will also make sense to implement additional sse-s3 java tests for the rest of the functionality that's also being tested in the python s3 tests.


trained model not found

Hello dear,
first, thank you to share your code with us
the problem is I don't find the page of  trained model when I try to download it.

do you download the code ?can you share it to me ?i want to know more details.Thank you for your attention to this matter.


[HIGH] Add SQLite support to web browser and fallback to IndexedDB

Placeholder tracking issue for adding SQLite support to Chrome, Safari, Edge and FireFox (Not IE)
Also for adding a fallback mechanism to retrieve data from IndexedDB when SQLite fails or has a problem
cc @roryabraham

Hey @MIAhmed! What's your Slack handle? Would you be interested in creating a proposal for this?

Thanks for running with this @muttmuure, I think generally there was consensus that this is a good idea, but let me post a formal proposal first just to get everyone on board. Going to slap a planning label on this and put it on HOLD

Chatting in this thread

Does this still rely on the deprecated WebSQL standard? As far as I'm aware only WASM is now a viable option to use SQL on this platform.

Hey @MIAhmed! What's your Slack handle? Would you be interested in creating a proposal for this?
@muttmuure Yes, I am interested in implementing fallback to the indexedDB as I have already implemented that.
I do not have Slack access

Does this still rely on the deprecated WebSQL standard? As far as I'm aware only WASM is now a viable option to use SQL on this platform.

No, I'm referring to the official SQLite wasm build

Sounds like we got consensus to do some exploratory research here and investigate the potential performance benefits.

For transparency to our open-source community, we've asked Margelo to begin researching this since they're experienced with working in Onyx and maintain https://github.com/Margelo/react-native-quick-sqlite

commenting here for assignment! @roryabraham

I did some research today and here's are my initial thoughts: cc @roryabraham @muttmuure @adhorodyski
(Lmk if you have any comments/critic)
Which SQLite/WASM approach
As suggested in the documentation of the official library (SQLite WASM) i would go for the approach where we use the extra wrapped worker thread. This way, additionally to (potentially) improved speed we have the benefit of offloading much of the load in Onyx to a different thread.
Reference: Worker + Promise approach

@chrispader having set up wasm support for Encryptify, I wonder if this is something we can implement at the react-native-quick-sqlite layer?

I would definitely add this functionality to react-native-quick-sqlite directly. The library is not yet web-compatible anyway, so this would simply extend the library by adding the whole web part and wouldn't interfere with the native implementation at all.
Onyx integration

That way, we would be able to leave Onyx mostly unchanged – just check if OPFS is supported, and if it is use the existing quick-sqlite storage provider, if not fall back on IndexedDB.

Exactly! As you've suggested, we would then check for availability of OPFS in Onyx and use IndexedDB (idb-keyval provider) as fallback.
Onyx operates async anyways, so this approach wouldn't require any changes in the library.
From https://expensify.slack.com/archives/C05LX9D6E07/p1704972138886149?thread_ts=1704921473.038539&cid=C05LX9D6E07 by @adhorodyski

one of the benefits is the unification of the query layer - this can potentially really simplify the architecture which is now limited to the least capable platform (key-value on web). Think indexes for example, running migrations, or bringing in an ORM for a type-safe database access.

I'm not sure if i understood your idea 100%, but if it refers to re-designing Onyx to support relational data, i don't think that would be a good idea atm. I wouldn't suggest to rely on any specific storage provider implementation too much, like SQLite on both web and native. Or was the idea to drop Onyx and create a completely re-designed (relational data) storage solution?
Imo Onyx is exclusively designed as a key-value store and its big advantage is that it's not dependent on any specific storage layer, instead it can use any underlying storage solution underneath... and this will potentially even improve even more.
I also agree with @tgolen, that as long as we don't completely re-structure our approach to querying and persisting data, i don't think we need react-query right now.
(Correct me if i got any aspect of your suggestion wrong though)
Performance
As @roryabraham already mentioned in this Slack thread the main benefit would definitely be faster speeds and to offload a lot of work to a separate worker thread and therefore unblock the main thread and improve app's performance.
I haven't created any real benchmarks and performance comparisons with IndexedDB yet, but from what i've read it's definitely faster than IndexedDB.
Analytics

We could also create a log of grafana graph to see how much the IndexedDB provider is actually used in the wild. That way a year from now there's no guesswork – we can look at how much it's actually used by users and consider dropping it from Onyx to simplify the codebase.

That definitely makes sense! Do we already use Grafana and is there an account/token to use it?
I could also imagine adding analytics for web and native performance of SQLite. This way we could analyze which kind of  devices profit the most from this change and which might have problems.
Going forward...
@roryabraham @tgolen just to make sure i'm not putting too much work into this straight ahead, was the idea to create a P/S for this with all the actual problems solved in the app first?
Or do we want to dive straight into the actual implementation in react-native-quick-sqlite? After my initial research, i don't see any functionality of the library that couldn't be realized with the WASM implementation of SQLite.

As for the implementation:
react-native-quick-sqlite is currently only synchronous. Also, the WASM API is limited, though totally sufficient for the use in Onyx/Expensify.
My idea would be to create a separate version inside the react-native-quick-sqlite library, which is

asynchronous and
limited in functionality, based on what the WASM implementation allows

This way, our other users of the library aren't affected and if the WASM implementation of SQLite get's more comprehensive in the future, we can extend the scope of the new version and also provide an additional async version.

Thanks @chrispader, here are my thoughts:

Was the idea to create a P/S for this with all the actual problems solved in the app first? ... Or do we want to dive straight into the actual implementation in react-native-quick-sqlite?

Basically, I think we should try to do the minimum amount of work possible to get real benchmarks that show the difference in performance between IndexedDB and SQLite-wasm. It would also be interesting to see how offloading Onyx work to a separate worker thread would help speed up the app, but I don't see how we could investigate that without building out a complete solution.
Once we have benchmarks that show the performance change, we can evaluate whether it's worth pursuing further. I admit it's difficult to write a problem statement for performance improvements because "how fast is fast enough"? I think a major selling point for us though is that the amount of work in Onyx and E/App to make this change is very small – all the work prettymuch is in react-native-quick-sqlite.

react-native-quick-sqlite is currently only synchronous

Maybe for simplicity start with the synchronous SQLite-wasm implementation then? I'd think that the way to approach this would be:

Start with the easiest thing, which is to create a synchronous implementation for web to match the synchronous implementation on iOS and Android.
Then build an async implementation for web, iOS, and Android
Then, if it's beneficial, update Onyx to use the async implementation.

That way there's no breaking changes, and all platforms have both sync and async capabilities, without any confusing configuration, sub-packages, or patches

Just confirming that we have @danielk1977 and @drhsqlite in this discussion, as they are the literal experts!

I'm here. But @sgbeal is the sqlite/wasm expert.

As suggested in the documentation of the official library (SQLite WASM) i would go for the approach where we use the extra wrapped worker thread. This way, additionally to (potentially) improved speed we have the benefit of offloading much of the load in Onyx to a different thread.
Reference: Worker + Promise approach

If i may interject here, as the person who wrote that documentation and code: The worker1 API is not recommended for any serious work. It has some severe limitations, noted in the above link, e.g. making it work properly with transactions is difficult and it cannot handle nested loops of queries because JS's postMessage() model simply can't support it.
The sqlite/wasm library can, and arguably should, be run in a separate thread without the worker1 API. The most flexible way to use it in a separate thread is to load the library and any directly-db-using client code into the same worker. That gives the client code full access to the sqlite API without the limitations of the worker1 API. It's not only much more comfortable to use that way, but also much more powerful. The one down-side is that the client-side code on the other side of the worker will have to coordinate its own communication do the db-using worker. That also has advantages, however, such as being able to encapsulate all of the SQL in the db-using thread and accessing it via a higher-level postMessage() API from the main thread.

I haven't created any real benchmarks and performance comparisons with IndexedDB yet, but from what i've read it's definitely faster than IndexedDB.

We have never worked with IndexedDB and have no benchmarks comparing it. We do have a wasm port of sqlite3's native-level benchmarking tool, speedtest1, which can be used to get a fairly good idea of its overall speed:
The baseline is an in-memory db:
https://wasm-testing.sqlite.org/speedtest1-worker.html?size=15
Then we have two alternatives for persistence via OPFS, each with their own trade-offs, as documented here:

https://wasm-testing.sqlite.org/speedtest1-worker.html?vfs=opfs&size=15
https://wasm-testing.sqlite.org/speedtest1-worker.html?vfs=opfs-sahpool&size=15

(That "size" URL parameter can be used to adjust the workload size - note the query counts in the output.)
I/O of the former is significantly slower than the latter but it offers a moderate level of concurrency (multi-tab use), whereas the latter is blazing fast but cannot be accessed from more than one tab/window at a time. Note that "slow" and "fast" are always relative, however, and the former is "fast enough" for many types of applications.
Note that concurrency in OPFS is a pain point - it does not natively support any concurrency without the use of experimental new APIs which were only recently released and are not available outside of Chrome. Our first VFS works around those limitations somewhat but it cannot reliably support more than a few browser tabs visiting the same db without locking-related issues. Our second VFS sacrifices all concurrency support for speed, and a given pool of dbs can only be accessed by a single tab concurrently.
For the sake of transparency, i feel compelled to point out that our wasm build  is not the only option. Roy Hashimoto's wa-sqlite project is actively developed, supports multiple storage back-ends (including IndexedDB) via the sqlite VFS API, and is specifically developed for the cutting-edge JS ecosystem. In contrast, we publish plain-vanilla JS and make no use of node.js-based toolchains, which has been a point of contention with some potential users but it's a stance we must stick to for the sake of developer bandwidth and sanity.

If i may interject here, as the person who wrote that documentation and code: The worker1 API is not recommended for any serious work. It has some severe limitations, noted in the above link, e.g. making it work properly with transactions is difficult and it cannot handle nested loops of queries because JS's postMessage() model simply can't support it.

Ah ok, thanks for the notice! By the "suggested approach in the official documentation" i was mostly referring to the GH Readme where it says "preferred option 🏆). If you suggest to use a differnet, i'd be glad to go with that instead!

The sqlite/wasm library can, and arguably should, be run in a separate thread without the worker1 API. The most flexible way to use it in a separate thread is to load the library and any directly-db-using client code into the same worker.
...
The one down-side is that the client-side code on the other side of the worker will have to coordinate its own communication do the db-using worker. That also has advantages, however, such as being able to encapsulate all of the SQL in the db-using thread and accessing it via a higher-level postMessage() API from the main thread.

If i understand you correctly here, this would mean that we can't or shouldn't implement this directly as-planned into react-native-quick-sqlite and indirectly into react-native-onyx, because we also need client-code from Expensify/App to run in the worker thread and directly communicate with the database.
I mean there is probably some hacky way to achieve this with all sorts of callbacks, but that already sounds like callback hell...
(@sgbeal please correct me if anything in this statement is incorrect or if i am mistaken somehow)
cc @roryabraham

Also JFYI @roryabraham
I'm going to be 100% OOO from 31/01/2024 - 17/03/2024. From what i can see right now, the work on this feature will either take longer than anticipated (from my side) and/or involve much more than just a change to react-native-quick-sqlite.
Therefore, i can finish my research on this and we can figure out the best approach to achieve this feature and then i'd have to hand this task over to another Margelo engineer. Maybe @Szymon20000 or @janicduplessis, since they also both have experience with react-native-onyx and/or react-native-quick-sqlite.
I'm happy to put more work into research, benchmarking and design of implementation until then, but i think the comment from @sgbeal changes a lot about my initial proposal and our initial idea of implementing this.

If i understand you correctly here, this would mean that we can't or shouldn't implement this directly as-planned into react-native-quick-sqlite and indirectly into react-native-onyx, because we also need client-code from Expensify/App to run in the worker thread and directly communicate with the database.

My experience with React is literally zero, so i can't speculate on what is or is not feasible with it, but loading both sqlite and your db-side client code into a single thread should not limit what you can do, beyond any limitations which you build in to that abstraction.
Let's say you've got a primary thread (be it the main window thread or a worker - that's largely irrelevant for sqlite), then load the library and your db-side code into its own shared thread. That gives your db-side code full access to the sqlite API (both the C-style API and a higher-level OO APIs), with no limitations on what it can do vis a vis the sqlite API. Your primary thread could then postMessage() requests to your client-level code in the db thread, and you can react to that however you like, within the limitations of the postMessage() protocol (e.g., iterating over nested loops of queries is not possible with that approach, and transactions are tricky to get right).
Your primary thread can post a message like:
postMessage({type: 'getDataForCustomer', customerId: 123456 })

and your db thread will have full access to the sqlite API, including custom SQL functions, to fetch that data, reformat it into whatever form you want the primary thread to have, and return it via a response postMessage(). The supplied worker1 API and its "promiser" proxy simplify that interaction somewhat, but its model of working introduces several significant limitations, such as:

Inability to use nested query loops. That is, nested for(...) loops which each run a query are not possible in that model. Subselects within a single query are unaffected.
Inability (or nearly so) to use transactions unless they can be packed into a single SQL string (containing an arbitrary number of statements), noting that the large-blob-of-SQL approach precludes the use of bind() for binding SQL parameters.
Inability to install custom SQL functions.

Plus the SQL will all have to be part of your primary thread, rather than separated neatly into the db-using thread.
Even so, it would not require a significant development effort to write your own custom worker1-like proxy which is specific to your APIs and moves all of the db-related logic into the db worker, which your primary thread then remote-controls via a custom-fit API. That would enable you to eliminate the above limitations because they would not apply to that hypothetical new remote-controlled worker.

I mean there is probably some hacky way to achieve this with all sorts of callbacks, but that already sounds like callback hell...

Working via postMessage() always has a bit of callback hell involved, even through the callbacks are effectively encapsulated in the onmessage handler. The worker1 promiser proxy effectively uplifts those callbacks from onmessage into something more recognizable as callback hell, but it's still cross-thread callback hell, and the cross-thread aspect of it introduces the above-mentioned significant usage limitations. Those APIs were created as proxies for simple use cases, and suffice for certain types of applications, but your apps would almost certainly outgrow those capabilities within days.
You won't be able to avoid postMessage() altogether unless you have sqlite in your main app thread, but OPFS is not available when sqlite is run that way. We can, however, simplify your use of postMessage() by housing your db-using code and sqlite in the same worker thread and remote-controlling that combination from your primary thread.

I'm happy to put more work into research, benchmarking and design of implementation until then, but i think the comment from @sgbeal changes a lot about my initial proposal and our initial idea of implementing this.

Likewise, we're happy to work with you to recommend a best-fit approach, with the caveat that our collective React know-how and ambitions are literally zero. Since the JS ecosystems evolves so rapidly, beyond our tiny team's ability to keep up with or sensibly claim to support, we very specifically provide only plain-vanilla JS which we then expect interested parties to integrate into their favorite framework as they see fit. We can't assist much with the framework-specific details but are happy to assist with both the integration and how to make use of the JS APIs.
We're available just about any time for video calls, if you'd like to talk through this.

Relational data approach / Onyx re-design?
From https://expensify.slack.com/archives/C05LX9D6E07/p1704972138886149?thread_ts=1704921473.038539&cid=C05LX9D6E07 by @adhorodyski

one of the benefits is the unification of the query layer - this can potentially really simplify the architecture which is now limited to the least capable platform (key-value on web). Think indexes for example, running migrations, or bringing in an ORM for a type-safe database access.

I'm not sure if i understood your idea 100%, but if it refers to re-designing Onyx to support relational data, i don't think that would be a good idea atm. I wouldn't suggest to rely on any specific storage provider implementation too much, like SQLite on both web and native. Or was the idea to drop Onyx and create a completely re-designed (relational data) storage solution?
I like the idea of improving the overall performance by not trying to convert relational data into key-value data and therefore losing potential of indexing and efficient querying. I just think it wouldn't be a good idea to try achieve this in Onyx...
Imo Onyx is exclusively designed as a key-value store and its big advantage is that it's not dependent on any specific storage layer, instead it can use any underlying storage solution... and we will potentially improve this aspect even more.
I also agree with @tgolen, that as long as we don't completely re-structure our approach to querying and persisting data, i don't think we need react-query right now.
(Correct me if i got any aspect of your suggestion wrong though)

also @cc @dsilva as you might be interested in this:
Hey @chrispader, you totally got this right and I wanted to shed some light onto the React side of things and how SQLite in the long run can be integrated with the UI layer, not Onyx-related. My idea is definitely not restructuring Onyx and I agree, react-query doesn't fit nowhere in there currently :)
What I rather wanted to propose is establishing a long-term vision for a new solution that is really scalable and future proof for the app's performance at any scale. This includes relying on the relational data structures and the power of SQL as well as the latest concurrency features available in React.
The problem
Currently, we rely on the key-value (KV) architecture even for mobile devices (running SQLite) which forces us to calculate all the bits of information on the same thread that is responsible for handling user interactions. No matter where we got the data from (be it indexedb or an sqlite table), as the scale increases, we have to add up another iterations and use more of the device resources to handle the workload.
For reference, please see the below screenshot showing how this looks like for an Android device on the JS thread, with Heavy account when sending a message (one of the core metrics):

As we can see, most of the time (here ~15 seconds) is being spent on resorting list items, completely blocking user interactions, and it’s like this by design - with KV we can do all the tricks, but every filter operation, fields recalculation and all the heavy lifting still sits on the JS engine.
This can be optimised, but only to a certain point at which we won't be able to squeeze anything more out of it without sacrificing a few seconds for the most basic operations. I’m sure this is nowhere we want to be a few years from now.
What's the most important, this pattern spreads across our core metrics (App startup, Opening up a report, Sending a message), making it really hard (if not impossible at some point) to hit the performance targets as we scale.
Solution
Even though we have SQLite, we don’t currently use nearly any of its features. We utilise it as a KV.
For the long term, as SQLite becomes available also for the web platform, I propose utilising the tool at hand to its full potential, enabling us to offload all the data related calculations in the App to the database through SQL.
Efficient querying with filters, joining tables, sorting (ordering), indexing or running migrations can all run outside of the JS thread within milliseconds.
An example from before running on the database through ORDER BY should take only some milliseconds and be instantly available to the UI.
Moreover, all of this can operate with some really minimal tooling around as React aims to fully support Promises with the concurrent features - and even the most complex query running through a database driver is just a Promise. This means that we can replace Onyx with a really thin and efficient communication layer between the UI and DB and also provide a good developer experience - will less code to write, easy to follow patterns and minimalistic abstractions.
There are also a few projects which currently try to address these problems in a similar way:
https://www.evolu.dev/
I really wanted to open this up for some feedback, as I’ve heard there’s an interest in utilising SQLite more as we get the web support in place.
An offline-first application should be no slower than tens of milliseconds per operation basically no matter the scale - and that’s what we can definitely aim for.

We are huge huge fans of SQLite.  Can you outline a strategy for how to incrementally migrate Onyx to use more SQLite functionality, without needing to rewrite everything or lose momentum?  You've mentioned that grouping, sorting, and filtering are all great features of a relational database, and that doing these after the fact in the application layer is super slow.  Maybe you could pick a couple scenarios that are particularly slow and suggest how in theory we could move those slow operations into the SQLite layer, without fundamentally breaking the Onyx API layer?

Escalating this conversation to Slack to make it go faster: https://expensify.slack.com/archives/C01GTK53T8Q/p1706933222963169

Left a proposal in the slack thread to try and gain consensus on next steps here.

Ah, @drhsqlite that's a link to our #expensify-open-source slack room. We can invite you if you'd like, but the proposal is not specific with regards to the implementation, just high-level next steps

Most of the links are internal, but you can see some previous benchmark results comparing two competing IndexedDB storage solutions here

Yes, happy to add it to the existing performance benchmark tool and get us some numbers!

Hey, I am helping with implementing the performance benchmark, please assign me as well. Thanks!

Hello everyone 👋
These are preliminary benchmark results:

Operation
Baseline (mock storage)
IDB (current)
Local forge
SQLite (main)
SQLite (worker)

Single set
115.9
157.8
170.1
155.4
287.6

Get collection
10.39
9.46
10.66
9.55
10.39

Merge collection
365.7
619.8
61338.8
542.7
600.8

Single merge
114.7
185.6
192
192.4
311.2

Clear
99.08
982.7
681
166.4
612

The agenda is:

Baseline (mock storage) - is not a storage at all. All functions are immediately returns Promise.resolve. Added it to see how much time is needed for Onyx to actually call necessary functions.
IDB (current) - current storage that used in Expensify
Local forge - previous storage that was replaced by IDB
SQLite (main) - SQLiteWASM running on a main thread (🚨 note, that in this case it will be in-memory DB, and all data will be lost after page reload - it happens because OPFS is not available on a main thread 🚨).
SQLite (worker) - SQLiteWASM running on a worker thread (with OPFS)

All metrics were captured with depth=1, length=35, items=10_000 and all operations were performed 10x times and then average value was calculated (sum / 10).
The working branch is: https://github.com/margelo/react-native-onyx/tree/feat/sqlite-wasm
As I can see single merge and single set operations are slower comparing to IDB - I think it's mainly due to the fact that we are passing large JSON objects into postMessage and data serialization steals some CPU time. I'll investigate it further whether it's possible to optimize it or not 👀

Thanks. Are those values in milliseconds?
From those preliminary results, it doesn't look like there are any significant gains, and maybe even some losses. JSON serialization is the worst :rage4:
Hopefully, there is a way to optimize it.

Are those values in milliseconds?

Yes, in milliseconds.

Hopefully, there is a way to optimize it.

I'll try to find possible ways 👀

SQLite (worker) - SQLiteWASM running on a worker thread (with OPFS)

Note that there are two distinct OPFS VFSes. One of them trades some level of concurrency (multi-tab access) for a significant cost in speed and the second forgoes all concurrency for corresponding speed gains (it's typically 2-3x faster than the other one, depending on the workload, and is never slower). Those VFSes can be run through a wasm build of sqlite's "speedtest1" benchmark, the project's own I/O-centric benchmarking tool, via:
https://wasm-testing.sqlite.org/
See the "speedtest1" links around the center of the page for the various options, including in-memory, "opfs" (the first OPFS VFS), "opfs-sahpool" (the second OPFS VFS), and "kvvfs" (which can use sessionStorage and localStorage for small amounts of persistence in the main thread).
Chrome v121 has a new, experimental, still-up-for-change feature which should permit us to add concurrency support to the second (faster) VFS, but (A) that's still a big question mark and (B) it's currently only in bleeding-edge Chrome versions (and subject to change).
The two current OPFS VFSes are covered at:
https://sqlite.org/wasm/doc/trunk/persistence.md#opfs

Thanks for pointing out @sgbeal
I used new sqlite3.oo1.OpfsDb(/${DB_NAME}.sqlite3); construction, I think by default it resolves to vfs=opfs (i. e. lower speed, concurrency support), right?
Anyway I updated table (I added benchmark only for SQL queries execution) and in this case SQLite shows the fastest results. So I think the problem here is in data serialization between threads. I'm investigating whether it can be improved somehow 👀

Quick noob question:
based on the speed test in https://wasm-testing.sqlite.org/speedtest1.html, if instead of storing JSONs we had a few structured tables for data that will be abundant and we have a format that is mostly defined (reportActions, transactions) would those operations be faster?

if instead of storing JSONs we had a few structured tables for data that will be abundant and we have a format that is mostly defined (reportActions, transactions) would those operations be faster?

@danieldoglas SQLite is also fast with plain JSON messages. The main bottleneck right now is a communication between threads (main <--> worker).

Regarding the byte-level serialization: rather than postMessage() data back and forth between the VFS-specific thread and the library thread, the VFS thread communicates with the library's thread over a SharedArrayBuffer, so there is no extraneous copying going on between those threads. The two threads use Atomics.notify() to make each other aware of new data in the buffer. The communication between the library and the VFS always happens in chunks equal to the db page size, so the VFS can (and does) reserve a fixed-sized SharedArrayBuffer for its communication with the library. The major performance hit in all of that is us waiting on the cross-thread communication between Atomics.notify() and Atomics.wait(), all of which happens in black-box browser internals.

@sgbeal so you are saying, that usage SharedArrayBuffer should make communication faster? I thought about experimenting with it, but still kind of afraid, that writing/reading from SharedArrayBuffer can be a bottleneck 🤔 But I'll try to see how it goes! 👀

based on the speed test in https://wasm-testing.sqlite.org/speedtest1.html, if instead of storing JSONs we had a few structured tables for data that will be abundant and we have a format that is mostly defined (reportActions, transactions) would those operations be faster?

That's really anyone's guess and mine would be mere speculation. That will, at least in part, depend on whether those objects span db pages, because...

@sgbeal so you are saying, that usage SharedArrayBuffer should make communication faster?

Faster compared to (say) postMessage() because it eliminates an extra copy which might otherwise be necessary[^1]. For example, when the library reads from storage, it passes a destination byte array to the VFS. In this particular case, the library and the VFS-related I/O live in different threads but they communicate via the same SharedArrayBuffer, so the I/O itself is performed directly on that shared buffer. The coordination/synchronization of the communication between those two threads, irrespective of the actual I/O, is where the biggest single performance hit comes from. The process goes something like:

The library tells the VFS "go fetch this byte range while I wait on you," at which point the library uses Atomics.wait() to wait on completion.
The I/O thread, which has been using Atomics.wait() to wait for a task request, wakes up, reads those bytes into the SharedArrayBuffer, reports "work is done - you may continue now" using Atomics.notify(). It then calls Atomics.wait() to wait on the next request.
The library thread receives that Atomics.notify() notification via its own blocking Atomics.wait() call and can then consume the bytes from the SharedArrayBuffer.

The same happens for writing: the library thread populates the SAB, tells the I/O thread that data is available, and then waits on the I/O thread's response to that write. Thanks to the SAB, the I/O thread does not need to make a separate copy of that data before passing it on to the OPFS layer.
Does that answer your question?
[^1]: postMessage() can move, rather than copy, buffers between threads, but in our case doing so is more expensive/slow than an Atomics.wait/notify()-based construct because we have to either create a new buffer for each operation or coordinate moving of a single buffer back and forth between threads. Our very first prototype used postMessage() for the cross-thread communication but it didn't perform as well.

Does that answer your question?

Yes, thank you 🙏
We've investigated SharedArrayBuffer usage, but since it operates on bytes level, we anyway need to serialize/deserialize a data and here we will loose our performance. I. e. if we do JSON.stringify to pass a data to worker, then here we are wasting a lot of CPU resources (but if there is another optimal way of converting JS object into SharedArrayBuffer - let me know).
We also looked at the performance when we have no serialisation/deserialisation on the worker thread API, but assuming that all the data layer code would operate in the worker thread directly (ie. API calls happening there and onyx code lives there as well). For that case we have these results:

Operation
SQLite (worker) - sql queries execution time

Single set
21.9

Get collection
9.21

Merge collection
229.6

Single merge
124.1

Clear
506.2

I. e. if there is no communication between threads, then SQLiteWASM is an absolute winner 💪 (I also added these numbers to main comparison table).
Let me know what do you about the results and further direction 😊

We've investigated SharedArrayBuffer usage, but since it operates on bytes level, we anyway need to serialize/deserialize a data and here we will loose our performance. I. e. if we do JSON.stringify to pass a data to worker, then here we are wasting a lot of CPU resources (but if there is another optimal way of converting JS object into SharedArrayBuffer - let me know).

AFAIK there is no more optimal approach for general-purpose or high-level software. We internally use a fit-to-purpose serialization approach which has proven to be faster, but it's in no way flexible. It depends, for example, on having a fixed number of fixed-size values, which isn't practical for anything but the most specialized of use cases.

Let me know what do you about the results and further direction

Unfortunately, my lack of background with your apps, infrastructure, and use cases leaves me unable to say terribly much which is likely to be of use.
If you like, we can do a video call and, with some hand-holding from you, perhaps i'll be able to offer some concrete suggestions and/or provide guidance on getting the most out of sqlite's JS/WASM APIs. Just send a meeting link and time to support at sqlite org and i can make the time slot work.

I sent an invitation to your email (took it from github profile,

Got it. See you on Friday!

VFS-2 can be used with concurrency, but it's supported only in latest Chrome browser versions

To clarify: that's capability is currently hypothetical. The new Chrome feature might allow us to add concurrency to that VFS, but we won't know for sure until we're able to experiment with it. Even if we do that, however, that OPFS feature is experimental and currently only in Chrome, so it may still change in incompatible ways. We have no information about whether the other browsers plan to adopt that feature and we are not keen on the idea of releasing a Chrome-only VFS.

most recent discussion here

conversation is ongoing, with some recent highlights being:
@quinthar:

So there are kind of three different optimizations being discussed, and they are entirely independent (ie, we can do any of them by themselves), but we are talking about them as fi they are somehow the same:

Query speed: Use on-disk imaging to improve disk read times
Worker offload: Add APIs to postprocess (ie, sort/filter) results on the worker thread to improve main thread responsiveness
Worker overhead: Reduce the amount of data going over postMessage to improve overall command performance

Of all those, I think the most important goal should be offload both querying and postprocessing to the worker thread to keep the main thread continuously responsive.  Improving the query speed is nice, but kind of marginal.

@adhorodyski:

On the 3 extracted topics, worker offload, worker overhead, query speed (thanks for this clarification!):
I gave it another spin and and as crazy as it gets, I think I've managed to solve all of them at once with the current key-value data structure for SQLite (amazes my every day how powerful it is :stuck_out_tongue:). You can think of it as an API for heavy processing on the other thread with very little adjustments to what we have right now :slightly_smiling_face:
I'll do my best to share a POC shortly with you but need to run some more tests.
Without moving anything around I'm able to query the KV structure using SQL and get the UI reactivity based on the database updates. So far it looks like it can be both performant and require little to changes, just adding the new queries on top and syncing with database hooks

excited to hear more about @adhorodyski's proposal.

I’ve been deeply invested in ongoing discussions about performance and have observed that our primary challenge doesn't seem to stem from the storage database itself but rather from extensive processing times on the main thread. This challenge manifests in two main areas:

Substantial computations within client code, notably in components like LHN.
Onyx’s architecture, which processes everything on the main thread, contributing to bottlenecks.

It’s clear these issues are significant and not directly related to the choice of storage provider. I’m curious about the native side - do we face similar challenges there, or is the conversation about moving storage handling off the main thread mainly focused on web implementations?
Given my history with Onyx, I have immense respect for what we've built and its evolution from a simple pub-sub model to a sophisticated key/value storage solution. As we navigate these performance challenges, I believe it's beneficial to revisit the foundational goals of Onyx:

What needs did Onyx initially aim to fulfill?
Could exploring a facade or middleware approach with another state management solution, like Redux, benefit us?

Reflections on Onyx’s Purpose:

Onyx’s simplicity and the ease of its pub-sub interface for components have been its strengths, enabling quick onboarding and straightforward usage.
It was designed to function outside of React, which, while initially seen as an advantage, has led to challenges with static connections and the inability to lazy load resources as one might with React components.
It aimed to reduce boilerplate code.

Considering Alternatives:

Before Onyx, had we explored using existing solutions like Redux? Redux’s adaptability with various storage backends could offer valuable insights into alternative architectures.
The interface Onyx provides for component subscriptions isn’t vastly different from Redux’s. However, the approach to state updates and management diverges significantly, without the structured flow of reducers and actions seen in Redux.

I once hesitated to use Redux, favoring more familiar patterns like MVC with MobX. Yet, the potential for integrating existing solutions through middleware, possibly enhancing Onyx or even adopting new approaches, is worth considering. This doesn’t diminish Onyx’s value but opens a dialogue on leveraging collective wisdom and advancements from the wider community.
As we ponder the future of Onyx, I see an opportunity for evolution rather than abandonment. Could we envision a redesign or integration that honors Onyx's original principles while addressing our current challenges? I’m eager to contribute to a path forward that builds on our collective achievements and addresses the needs of our evolving technology landscape.

An interesting library we could consider (or pull from for inspiration) is legend-state. Reviewing the design goals for Onyx written by @tgolen, and adding this one:

it should be as performant as possible

legend-state seems to check all the boxes. I believe it's quite similar to Onyx in its design, with even less boilerplate. I also think it would be feasible to leverage it to simplify Onyx internals while keeping the existing interface for Onyx largely unchanged, avoiding the need for a major app-wide refactoring effort.
I've also met the maintainer in person a few times at React Native conferences, and I believe he'd be open to working with us. A direct line to support could alleviate concerns with lack of control over the library.

@roryabraham I'm going to need some more time to prepare a draft of how I think this could look like, but to give you all more context I genuinely think nearly all we need is the right design and composition, not much of the new tools - this might be the 10% but mostly due to the ergonomics of not rewriting community libraries.
I basically work with SQL queries & JS Promises, binding them to the UI.
I'll do my best to share a WIP proposal of this design next week.

It seems like conversation has stalled for the last few weeks, and the initial investigations we did have prettymuch concluded. It sounds like @adhorodyski has an idea he'd like to propose for which SQLite on the web would be a prerequisite, so I'm going to assign this issue to him. I'm also going to unassign @hannojg and @kirillzyusko because their explorations are complete.

@roryabraham still working on it - correct with the web support being a prerequisite, but after a few discussions with @jbroma together we decided to slightly pivot as IndexedDB is for now the only option which fully supports this platform. I'm aiming for a design where once web catches up, we can easily migrate just this very piece with a very little effort. Will update within days!

Hey @adhorodyski, we discussed this internally, and since we're not working from a clear problem statement here and the ROI of any changes is still very much up-in-the-air, we've agreed that we're going to close this issue for now.
We advise that you shift your focus into other areas of the performance audit for the time being. We love SQLite and may want to pick this discussion back up someday, but for now we're going to cap it here for #focus. Thanks everyone for all your participation so far!

@roryabraham I agree this was fine to be closed - I'll post my research on Slack just so we don't loose it later on, I think this is still valuable and holds/maybe we'll continue the discussion from there/kill it with fire for now :D

cc @roryabraham posted internally under this thread, please let me know if there's a better place for this :)

thanks, I think that's an appropriate place for that


Add backup solution for etcd

e.g. Velereo
Backup Idea:
Cluster-API resources in tenant-namespaces (playground0-9), automatic periodic etcd snapshots to S3 (regiocloud)

implemented in main-volatile

main-volatile has been merged to main


feat: add redis storage

feat: add redis storage
closes https://github.com/jina-ai/internal-tasks/issues/117

good job!


Python 3.10

Hello,
In the very last version, on the main branch, you have added some python 3.10 code but the Dockerfile is still in 3.9.

#70 Should fix this

Thanks for the issue! We just merged the fix on the main branch, should be included in the next release


Roll engine ef2bed65db22..39a950e5d1d1 (5 commits)

https://github.com/flutter/engine/compare/ef2bed65db22...39a950e5d1d1
git log ef2bed65db22b1ce3d05947d5cfa3927eb6fa975..39a950e5d1d1cbeb3d70334c6275d224e843b30e --no-merges --oneline
39a950e5d remove legacy build deps (flutter/engine#8820)
c0be4e277 Fix api conformance check (flutter/engine#8817)
30639ee7b Allow FlutterEngine to be used on back-to-back screens (#31264). (flutter/engine#8808)
45d0c4d11 Roll src/third_party/skia cd5d14619fe2..68eb8c276355 (4 commits) (flutter/engine#8815)
9856479ec Roll src/third_party/dart b6997deb3e..1577b95c93 (18 commits)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff (amirha@google.com), and stop
the roller if necessary.

Auto-roller completed checks. About to merge.


NPM code and GitHub are out of sync

NPM code error

Hi @zzz0908, thank you for reaching out! Do you mean that the link on npm is to github.com/bjnortier/dxf while the repository now is located at https://github.com/skymakerolof/dxf? Seems like github handles this redirect gracefully, but I will update package.json to reflect this change as well, so it should be resolved with the next release to npm.


The Identity.logout function takes a redirectUri although it's not used

This is confusing to our users. logout() does XHR calls, and so redirecting users doesn't make sense either.

I'm one of said confused users :D


queryCache is empty when using ReactQueryCacheProvider

Describe the bug
When I'm trying to queryCache.invalidateQueries(keyGoesHere) and I'm using the new hydration API, the queryCache object is empty, so I cannot invalidate or override any of the data.
Expected behavior
queryCache should contain the queries that are prefetched on the "Hydrate" component or another way to invalidate or set new data for a specific query.
Screenshots

**Desktop **
-macOs

For someone else having the same issue
instead of import { queryCache } from "react-query" you should use const querycache = useQueryCache()

I had this issue as well where the 'import' technique was not clearing the cache setup by ReactQueryCacheProvider.  Using the hook resolved the issue


feat: add id method

This PR adds the id() method that is present in the official client libraries.

@dyc3 released in 0.6.0+0.16.17.


go/types,cmd/compile/internal/types2: shared type expressions are evaluated once per variable

If a VarDecl has an explicit type expression, it will get evaluated once for each declared variable. So if you have a VarDecl like var x, y interface { M() } you end up with two different *types2.Interface objects.
I ran into this because I wanted to map all types2.Type objects (that appeared in source) back to some syntactic element, so my initial approach was to just walk the AST looking for type expressions and then using Info.Types[expr].Type to get the types2.Type object and insert them into a map[types2.Type]syntax.Expr map.
However, this approach missed some types, because of VarDecls with multiple variables and an explicit type expression.
I don't think this is strictly a violation of the types2 API (you're supposed to use types2.Identical), but I think it would be nice and less surprising if there was a 1:1 correspondance.
I assume this also affects go/types.
/cc @griesemer @findleyr

Indeed this affects both go/types and types2.  Here's the relevant TODO:
https://cs.opensource.google/go/go/+/master:src/cmd/compile/internal/types2/decl.go;l=464;drc=a9705e157beb51574233e23cc2e2a412d4681a15
I think this could be done, but we should probably handle decls with init exprs as well, which IIRC would be a non-trivial refactoring.

I discovered this issue also affects struct fields (struct { a, b T }) and function signatures (func(a, b T)).
It presumably in theory also affects constant declarations, but if a constant declaration includes an explicit type expression, then it must be a defined type. So they'll always have a unique types.Type instance (right?).

 (To be fair to gccgo, I'm running it from a network file system, and it takes a few seconds to compile even a file with just package p.)
I tried a few more minor variations on the idea, but couldn't figure out a way to make go/types go exponential.

@mdempsky This is in the 1.18 milestone; time to move to 1.19?  Thanks.

@mdempsky This is in the 1.19 milestone; time to move to 1.20? Or Backlog?  Thanks.

Too late for 1.19. But we should probably look into this. Moving to 1.20.

Too late for 1.22.

Moving to 1.24. We didn't get to this.


the-sz.Lacey version 2.95

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] This PR only modifies one (1) manifest
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.6 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Hi @the-sz,
The package didn't pass a Defender or similar security scan. This might be a false positive and we can rescan tomorrow.
(Automated response - build 788.)

Automatic Validation ended with:

Installation failed with exit code -1978335187 2024-02-18 14:07:00.622 [FAIL] Installer failed security check. Url: https://the-sz.com/common/get.php?product=lacey Result: 0x80004005 2024-02-18 14:07:00.622 [CLI ] Terminating context: 0x8a15002d at D:\a_work\1\s\external\pkg\src\AppInstallerCLICore\Workflows\DownloadFlow.cpp:1e6

(Automated response - build 788.)

@wingetbot run


pgupgrade: reverting postgresql version to 13 for quay (PROJQUAY-7253)

This reverts the quay pg upgrade changes in commit 673eff30cc03e590539fca12bc2b1721d68ff4f1
Refer https://github.com/quay/quay-operator/pull/959#issuecomment-2364286444 for details.
Testing

Tested push-pull
Ran e2e test

/ok-to-test

/test ocp-latest-e2e

/retest-required

The tests are failing for timeouts while getting waiting for quay to get ready. Checking if the test environment is changed. Did not notice any error while deploying locally:
https://shudeshp-registry-quay-quay-enterprise.apps.jonathan-ocp.gcp.quaydev.org/


gain unit work using autobase changes

Rebased version of #87, otherwise the same design so probably doesn't need review.
Closes #74

now rebased again onto 2076-2


Search does not work

Describe the Issue
Searching "circuit" does nothing.
Expected Solution
Search works
Anything else we should know?

Seems to be working for me. How are you viewing the textbook?

I use Firefox on OSX and search doesn't work for me either. I enter the search term, press return and nothing happens.

Yes same. Also search on iphone does nothing.


Build with sandbox sources

Build the platform from sources using the more robust strategy I mentioned in https://github.com/elm-lang/elm-compiler/issues/995.
(Side note)
The only trouble I seem to be having is that I don't have elm-make in my $PATH by the time that I need to build elm-reactor:
Build log ( /home/rehno/projects/development/elm-platform/installers/Elm-Platform/0.15.1/.cabal-sandbox/logs/elm-reactor-0.3.2.log ):
Configuring elm-reactor-0.3.2...
Building elm-reactor-0.3.2...
Preprocessing executable 'elm-reactor' for elm-reactor-0.3.2...
Linking dist/dist-sandbox-f78d21b8/build/elm-reactor/elm-reactor ...
Custom build step: creating and collecting all static resources
setup: elm-make: does not exist
cabal: Error: some packages failed to install:
elm-reactor-0.3.2 failed during the building phase. The exception was:
ExitFailure 1

I'm not sure if this is actually a problem, could you try this install on your local machine @evancz ? I also created a patch to make sure that elm-reactor is built last if necessary: https://github.com/rehno-lindeque/elm-platform/compare/build-with-sandboxsources...build-elm-reactor
I was hoping that I could just temporarily change my PATH by running the command as
 PATH=./Elm-Platform/0.15.1/bin:$PATH runhaskell BuildFromSource.hs 0.15.1

but it seems as though Setup.hs must start on a fresh environment because that didn't do the trick for me.
Anyway, I'm about 95% sure this PR should do the trick (perhaps with https://github.com/rehno-lindeque/elm-platform/compare/build-with-sandboxsources...build-elm-reactor added on).

Oops, just noticed that I added the --quiet flag on a different git command than I was intending.

Ok, that's it - please let me know if it looks ok.

LGTM :+1:
I noticed only a few days ago that doing something like this made builds easier. I did it like the following, but I didn't run into the elm-reactor post-build dependency issue because didn't try to compiler elm-reactor. If it wasn't for that, you could do it like I did and not have to fiddle with source-directories.
cabal --ghc-options=-XFlexibleContexts\
      --sandbox-config-file=cabal.sandbox.config\
      install\
        elm-compiler/elm-compiler.cabal\
        elm-make/elm-make.cabal\
        elm-package/elm-package.cabal

Oh interesting! I didn't know you could just list cabal files that way. I'll see if I have time to update this PR tonight, that should make for a pleasingly tiny amount of code.

Sorry, I don't think I'm going to get to this soon, I have to prioritize a bit. However, the current PR is almost certainly an improvement over the previous state of this file (since the current strategy doesn't build at the moment I believe).
I believe it is ok to merge as is.

What do you think of the ideas listed in #60 about using cabal freeze everywhere once we have discovered a set of packages that works?

@jvoigtlaender, is your comment https://github.com/elm-lang/elm-platform/issues/41#issuecomment-132195920 suggesting that this will close #41? If so, I will work on getting this plus cabal freeze in place for folks who are building from source.

I cannot say for certain whether https://github.com/elm-lang/elm-platform/issues/41#issuecomment-132195920 will close that issue. On my machine, I succeeded installing all the Elm tools using the fixed version numbers recorded in https://www.stackage.org/lts-3.1/cabal.config (by dropping that file into the project directories).
Fixing version numbers is generally the way to go, I think. Whether you do that by relying on the https://www.stackage.org/lts/cabal.config files or by trying to work out compatible versions yourself on your own machine and then do a cabal freeze is a question that I think will mainly affect your own comfort, not that of others ultimately installing from the repositories.

I messed around with the add-source thing, but it seemed to mean that build times are always maximally bad. Like, if I just want to run cabal build on something, I need to install everything it depends on every time.
I don't think it's worth it to introduce such large delays. I am working on using a makefile to document all the tricks I do in my workflow and make it easier for others to do.

I am going with the cabal.config approach where we just lock down all the dependencies for each release. I made a trial of this approach in https://github.com/elm-lang/elm-platform/tree/master/versions/0.15.1 and I think it'll do better than this PR in avoiding dependency stuff.
It's not yet clear to me if it will build with GHC 7.10 but I think I am okay saying to folks "You must use GHC 7.8" as long as we know that is true.

Thanks everyone for working on getting this more reliable! We'll see if the cabal.config thing works out :)

No problem! I should probably mention @Apanatshka's technique would work without the cabal build issue. It doesn't matter though, anything that solves it for people just starting out with the source will be perfect (cabal.config ought to do the job).

Actually, I suspect the workflow from the build script (without cabal add-source) is not going to work for people developing. For people who only want to install Elm once from source, it should work, but for people trying to set up an environment for developing the tools, it probably won't (as has been observed in other issues).
Here is why:

Say I set this up once. This will, among other things, build some version of elm-compiler and some version of elm-package. Say elm-compiler-0.15.1 and elm-package-0.5.1. The latter needs the former, so when it is built, it is *linked against the elm-compiler-0.15.1 package built and installed into the sandbox previously`.
Say I now, as a developer, change something in the elm-compiler code. I rebuild the package after the change. That installs a new version (but with same version number 0.15.1) of elm-compiler into the sandbox. This breaks the elm-package built, because that one was linked against a now not anymore existing elm-compiler install. Crucially, cabal does not allow two different incarnations (compiled from different code and/or against different installs of dependencies) of the same package with the same version number.
At this point, I can only throw away the sandbox.

As I understand, the cabal add-source is exactly for dealing with such situations, because it will not install some package version into the sandbox.
The build/rebuild times that @evancz experienced, I can only explain to myself with that something changed in the source files in the dependency directories. Maybe they were touched by some git command in the meantime?
Having to throw away the sandbox whenever something has changed in one of the tool directories will of course lead to the same recompilations.
In summary, the approach without cabal add-source may work fine for people just wanting to install Elm from source, not touching the source in any way. But for developers, I doubt it will work. (But maybe I'm wrong. Evan must have some workflow on his own machine that works for him.)

Actually, my previous comment was maybe too pessimistic. Instead of having to go with cabal add-source, it should be possible to go with what build.sh does, including my fixes from https://github.com/elm-lang/elm-platform/pull/65, but then when a source change in one of the project directories was made, always make sure that cabal is called with all projects at once. Something like cabal install ./elm-compiler ./elm-package ./elm-make ./elm-reactor ./elm-repl. That will only do new compilation for stuff that has changed, but if something has changed in the elm-compiler subdirectory, it will make sure that the other stuff that depends on elm-compiler (which is all of them) will be relinked as well, thus avoiding that the sandbox gets into a broken stage. (Actually, the packages that produce only executables could be left out from that command, as long as one knows that one doesn't need those executables to be rebuilt right now.)
(All this here concerns "developer mode" only. For doing-an-install-just-once the linked script along with my linked fixes should just work.)

Something like cabal install ./elm-compiler ./elm-package ./elm-make ./elm-reactor ./elm-repl.

@jvoigtlaender unless I'm missing something this is the suggestion Apanatshka made above. (It would be very easy to amend this PR to do that.)

I have a workflow that works well with --sandbox. I never have to throw things away. My builds are as small as possible. I'll share that in the makefile I create.

Alright, I have convinced myself that this is probably the best way to go right now. I don't think I'll develop this way, but if it works better for linux people who just want to use Elm, I am down.
Once the --bindir and FlexibleInstances questions are sorted, I'll merge this in and let people know.

@rehno-lindeque, yes, what I suggested above was what @Apanatshka suggested.

Ok, I rebased this and fixed bindir for elm-reactor. The improvement above would be very easy to apply if someone wants to take a shot at it (I mostly don't want to rebuild the sandbox from scratch again because that's the time-consuming part :))

Cool, let's see what happens! Thanks for sticking with this. This is some of the hardest stuff because everything takes like an hour to even try out and it mostly breaks, so I really appreciate your work here!

I'm glad I could help!


refactor of RequireResolveContextDependency to es6

What kind of change does this PR introduce?
refactor
Did you add tests for your changes?
current tests should pass
If relevant, link to documentation update:
N/A
Summary
upgrade RequireResolveContextDependency to es6
Does this PR introduce a breaking change?
No

trigger rebuild for appveyor


Yasnippet changes possibly disallow completion popups

It seems after the recent yasnippet integration, company popups are disabled once I add the company-dict backend. The popup dialog is created only on the <tab> key. This is in spite of (setq company-dict-enable-yasnippet nil).

Odd, I can't reproduce this. With company-dict-enable-yasnippet set to nil, company-dict should completely ignore the yasnippet bit.
I took a peek at your emacs.d and didn't see anything that would cause this. What version of Emacs are you running?
I'll do some more testing and see if anything turns up.

I am using both Emacs 25.1 and 24.5 on two different machines. I see that enabling company-dict disables company popup and I have to press <tab> to start the completion (possibly via yasnippet?). Just disabling company-dict allows the popup to appear.
This could also be because of my yasnippet setup. I will try and look into it more tomorrow.

I tried with emacs -Q and it is reproducible. Here is the error that I get:
Company: An error occurred in auto-begin
Company: backend company-dict error "Symbol's function definition is void: remove-if-not" with args (candidates com)

Do you need (require 'cl)?

I digged a bit more, and I tried evaluating the following code.
(require 'package)
(setq package-user-dir (expand-file-name "~/.emacs.d/elpa/")
      package-enable-at-startup nil)
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/") t)
(package-initialize)

(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))
(eval-when-compile
  (require 'use-package))

(use-package company
  :ensure t
  :config
  (setq company-global-modes t ; Turn on company-mode for all major modes
        company-show-numbers t ; Quick-access numbers for the first ten candidates
        company-minimum-prefix-length 3
        company-tooltip-flip-when-above t ; Invert the navigation direction if the completion popup is displayed on top
        company-tooltip-align-annotations t
        company-tooltip-limit 20
        company-selection-wrap-around t
        company-dabbrev-downcase 'case-replace
        company-dabbrev-ignore-case 'keep-prefix
        company-dabbrev-code-ignore-case t
        ;; Offer completions in comments and strings
        company-dabbrev-code-everywhere t)
  (global-company-mode 1))

(defcustom dotemacs-temp-directory (concat user-emacs-directory "tmp/")
  "Storage location for various configuration files."
  :group 'dotemacs)

(use-package company-dict ; FIXME: yasnippet support disables company popup
  :ensure t
  :after company
  :config
  (setq company-dict-dir (concat dotemacs-temp-directory "dict/")
	company-dict-enable-fuzzy t
        company-dict-enable-yasnippet nil)
  (add-to-list 'company-backends 'company-dict))

It seems the problem arises when trying to customize company-dict-dir. The problem does not arise after evaluating the sexp with the line commented.

Alright, I pushed a fix for the remove-if-not issue. I don't see anything suspect with custom directories (I use one myself), but I'll do some more experimenting.

Is the above snippet working correctly for you? Do  you get to see company popups even after evaluating company-dict-dir?

You're right. That snippet didn't work. It complained first about remove-if-not and then about parent-mode-list — I've fixed these two in the latest commit.

Aah, I overlooked the fact that you expected a reply. Yes, the problem persists.

This is very odd. I can no longer reproduce it. What else is in your company-backends? Is it possible another backend is interfering?

I am not an elisp expert, so cannot help you much. Adding dictionary files such as all under company-dict-dir seems to create a problem for me. The popup works with an empty directory. Does this help?

Oh, perhaps it's a permissions or file encoding issue then? I just pushed another update. I changed the way it reads dict files and optimized the parsing process. Please try that and let me know if it fixes it.

No, the problem still persists for me. Here is a screenshot: Typing com just shows entries from the dictionary all, it does not show the other candidates.

The snippet that I am evaluating with emacs -Q is the one that I have shared earlier.

Sorry for the tremendously late response, but are you still having this issue (if you're even using the plugin anymore, haha).

I am using it, but things are okay right now. I do not remember what resolved it or whatever. I will close it for now, thanks for the follow up.


look into switching from tarpaulin to grcov for code coverage support

Once support for it enters a stable rust release.

Support for tarpaulin was dropped in https://github.com/pkgcraft/pkgcraft/commit/6aa958cfc977cf5b0316de6c4126de4760ef34ae mostly due to CI hangs.

It may be possible to re-enable code coverage soon since it looks like -C instrument-coverage support is trending towards stabilization. Beyond that, grcov support can be enabled by following instructions similar to those outlined at https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html.

It's now possible to use llvm-tools instrument-coverage support in stable rust along with llvm-cov and nextest via cargo llvm-cov nextest. The final step for CI would be to generate some coverage data file that codecov can consume.

Coverage is now done via cargo llvm-cov in https://github.com/pkgcraft/pkgcraft/commit/29432c25061029d154fb49bc8eb8e696a170c8fb.


Too many Tokens call

Package
I’m not sure
Ask the question
Hello,
I am reaching out to seek assistance with an issue I am experiencing in my Drupal Next setup with a NextJS frontend.
I have configured Drupal Next with a frontend in NextJS for an informational and public website. There is no authenticated section on the site. Upon analyzing the NewRelic statistics, I have observed an unusually high rate of 1000 calls per minute to the token retrieval functionality.
To provide some context, I have set up authentication between Drupal and NextJS using OAuth2 and a NextJS user. Despite this configuration, I am struggling to comprehend the reason behind these 1000 calls per minute. I am wondering if you or your team have encountered similar challenges in your applications.
If you have any insights into this matter or can recommend a specific configuration for authentication that may address this issue, I would greatly appreciate your guidance. Please let me know if additional information or logs are needed to better diagnose the problem.
Thank you in advance for your time and assistance.
Best regards,

Try increasing token expiration time.


Ledger Live mobile, calling getSigner() after switching accounts results in wrong signer address

Currently when using built-in wagmi hooks on ledger live mobile we get an unique issue where the data is being fetched for the correct account, but the signer it uses is wrong.
Expected behaviour:

When switching from main account to a secondary one and then back to the main one, the expected signer is the main one

Actual behaviour:

The signer is the secondary one instead of the main one

Steps to reproduce:
Open Ledger Live mobile
Switch current account to any other
Switch active account back to the original one
getSigner() returns the secondary account instead of the primary resulting in failing transactions

Hey @Tdrachev, thanks for the feedback 🙌
cc. @adrienlacombe-ledger for triage / prioritization.
In the meantime @Tdrachev, if it's urgent / blocking you, know that the project is open source and contributions are welcome 👍

isn't it weird that issue is with LLM and not LLD @chabroA ?

isn't it weird that issue is with LLM and not LLD @chabroA ?

Nobody said it was only on LLM from what I understand, but that it was experienced on LLM (maybe @Tdrachev didn't test on LLD)
If this indeed only happen on LLM and not LLD, then it might not necessarily be linked to the connector directly.
Also @Tdrachev, could you test with the previous version (0.1.1) see if this changes anything.

@chabroA,
On desktop the issue is not present indeed, using the getSigner hook from wagmi returns the correct signer account. Issue is only happening on mobile.
Was curious wether ledger live mobile works differently than desktop in switching accounts?
I can try and check the repo when there is more time to see if I can debug  (not necessary blocking at this point).


:sparkles: Add examples

I added  reverse and comp functions, as well as if-let macro experimentally. If you think it is appropriate, you can move them to boot.dst.
I will also implement let as monads (as it is originally is). Tell me your opinion.
I like the iter approach you  implemented there :)

I don't see the if-let, but thanks for more examples! I have merged but have some suggestions.
Lastly, tuple-prepend is an O(n) operation that copies the whole tuple, unlike lisp cons, which is why I really only use it in macros. The tuple reversing function would work best by simply using the array-reverse function and converting to a tuple.
I'm not sure why you would want to have let as a monad; it would be much less efficient. It would be an interesting exercise, though. I admit I am a bit fuzzy on the formalities of monads, though.

Sorry, I mispelled, i meant if-not
I was under the impression that let was a monand due to https://github.com/khinsen/monads-in-clojure/blob/master/PART1.md i read  a cuple of years ago. Search the source code of clojure and let expands to an expression of let*.
Thanks for the review.


Fix mangling of generic and instantiated methods

Methods on instantiated types were taking the same path as generic
methods unnecessarily (this will be a small improvement in terms of
string allocations)
Generic method definitions were attempting to include mangled names of
their generic parameters. (This is the main reason why I looked into it - I hit this while running tests for #3152.)

@jkotas or @nattress PTAL


Added velocity to swiping event callback

Dunno if anyone else will find this useful, but I found myself needing a reference to the current velocity for my page swipe indicator.
Might be better to just expose velocity on props object...
(Sorry for the dupe commit messages, was trying to rebase/squash but it didn't do what I expected.)

Can you rebase this down to a single commit?

@goatslacker I like this addition but I don't know if @jakedowns is still around to rebase. We can just add these changes in the next couple of updates with the build changes.

Closing (superseded by #33 )


Import error when adding additional labels

I have the following issue. I have an existing task with labels A and B. Now, I want to add labels C and D that appear in a file with annotations. After adding the labels in the task interface, uploading the annotations produces the following error:
TypeError: Cannot read property 'attributes' of undefined
If I instead create a new task with labels C and D, the import works fine.
Server version: 1.0rc0
Core version: 2.0.0
Canvas version: 1.0.0
UI version: 1.1.

@manybodycpa  Hi, thanks for the report!

@manybodycpa
I am unable to reproduce the issue.
Steps I tried:

Created a task with two labels
Created four objects there
Dumped annotations as CVAT Images
Changed labels in the dumped annotation file for two objects (doesn't exist in the task)
Added these labels on the task page
Uploaded annotations successfully

Please, update your CVAT and the issue probably will go away.
I will close the issue for now, if you still experience it, please reopen


Update extlinks to comply with Sphinx 5 deprecation warnings

See https://github.com/ome/bio-formats-documentation/pull/250 for background

the duplication is probably left over from the split "doc/contributing"


Update to Office Hours

Updated Office Hours text

Coverage increased (+0.04%) to 92.239% when pulling 59ae788854ecd585ed34d69272d0ae455cb02d0f on update-office-hours into da89364d9d438024bc95555b2d25fdbeff273d44 on master.

This change is already live? - see https://www.nats.io/community/


Feature: using lipgloss style table instead of homemade

Codecov Report
Attention: 14 lines in your changes are missing coverage. Please review.

Comparison is base (af1295d) 70.23% compared to head (ae9ce8f) 69.71%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #26      +/-   ##
==========================================
- Coverage   70.23%   69.71%   -0.53%
==========================================
  Files          11       11
  Lines        1344     1334      -10
==========================================
- Hits          944      930      -14
- Misses        329      334       +5
+ Partials       71       70       -1

Files
Coverage Δ

taskpoet/poet.go
57.39% <93.10%> (-5.55%)
:arrow_down:

taskpoet/durations.go
83.33% <73.91%> (-9.85%)
:arrow_down:

... and 1 file with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


XASX closing auction print

Normal trading ceases at 1600 for the ASX, after which there is a dutch auction at 1610 for the final day-close print. Would it be possible to account for this somehow. Perhaps a 'match' session?
Cheers

XTAE has similar auction.

Hi @j7zAhU.
exchange_calendars has historically only been concerned with regular trading times. Although, that's not to say that it couldn't incorporate additional timings. I suspect we'd be open to a PR suggesting a framework to register / provide such information (@gerrymanoim - ?).
I believe pandas_market_calendars now provides for something along these lines - perhaps start by having a look at how it's been implemented there?
Cheers

I've implemented a solution to this, but in another code so cannot comment on feasibility of modifying trading_index(). But happy to discuss the framework.
I add an "auction" datetime column to calendar, that's only change to schedule. Most work is in creating the trading intervals:

assume auction consumes 1 minute. Nothing official on end time to confirm this, maybe it's 1 second
map auction time to a trading interval as if it was during the session, then adjust the 'auction interval' close to match auction end time. If 'auction interval' overlaps the actual last session interval, merge them.

Merging should only happen with large periods e.g. 1h. With small periods e.g. 5m there should be a gap between close and auction.

The final print happens at a random time within that minute


As an advanced user, I would like to have shortcuts to type my command more quickly.

Create case-insensitive aliases for commands.

for this i was thinking of doing like the first letters of each command word so for example find becomes f and findRelation becomes fr


Support for discriminated uniones with callable discriminator

The latest pydantic release has added support for Union discriminators that can be implemented via a custom function: https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions-with-callable-discriminator
I recently attempted to utilise this with pydantic-xml but ran into the following error raised by pydantic-xml:
E   pydantic_xml.errors.ModelFieldError: .... field type error: only string discriminators are supported
I'd be willing to contribute a PR if you're able to point me in the right direction of how to add support for this to pydantic-xml

I had a quick poke around in the code and I'm not familiar enough the internals of pydantic to make an obvious patch (the current tagged_union.py logic assumes that the fields will all be available as literals when the model is created. However I did end up whipping up a really simple unit test that hopefully highlights the issue.
https://github.com/dapper91/pydantic-xml/compare/master...joshvote:pydantic-xml:master


fix CustomFocusTraversalTest

FocusOrderToProperties changes in this commit
Fixed broken test

Why do we need this change (useFocusOrderModifier)?

Because correct parameter name is useFocusOrderModifier.
It's mistake in very beginning of this code.

Ah, understand, thanks.


Build - $inject:nomod

I'm getting this error after building or with serve:dist
vendor.js:9279Uncaught Error: [$injector:nomod] Module 'dmrApp' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.
http://errors.angularjs.org/1.5.8/$injector/nomod?p0=dmrApp(anonymous function) @ vendor.js:9279(anonymous function) @ vendor.js:11293n @ vendor.js:11217(anonymous function) @ vendor.js:11291(anonymous function) @ app.js:1
vendor.js:9279 Uncaught Error: [$injector:modulerr] Failed to instantiate module dmrApp due to:
Error: [$injector:nomod] Module 'dmrApp' is not available! You either misspelled the module name or forgot to load it. If registering a module ensure that you specify the dependencies as the second argument.
http://errors.angularjs.org/1.5.8/$injector/nomod?p0=dmrApp
    at http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:376
    at http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:10542
    at n (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:10015)
    at http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:10327
    at http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:19680
    at r (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:824)
    at f (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:19528)
    at rt (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:21453)
    at a (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:8085)
    at ie (http://localhost:3002/scripts/vendor-7dc26fb0a5.js:31:8395)
http://errors.angularjs.org/1.5.8/$injector/modulerr?p0=dmrApp&p1=Error%3A%…http%3A%2F%2Flocalhost%3A3002%2Fscripts%2Fvendor-7dc26fb0a5.js%3A31%3A8395)

It' very weird and I cannot get rid of it.
Here's the modules
angular
.module('dmrApp', [
'ngAnimate',
'ngCookies',
'ngTouch',
'ngSanitize',
'ngMessages',
'ngAria',
'ui.router',
'ui.bootstrap',
'ngStorage'
]);
Here's the structure :

Same problem here, is there a solution ?

This is the same issue as #102, the version of ngAnnotate used by the cg-angular generator is outdated. To update it, in the root of your application there should be a package.json file. You need to change version on the line "grunt-ng-annotate" to "~0.10.0", and then save the file. Then from the command line and in the root directory of your app, run 'npm update grunt-ng-annotate'. Then try your build process again.


Feature Request: Sub-Tabs

With the large number of vertical tabs, it would be helpful to consolidate them and then have horizontal sub-tabs for the related functionality. For example...
General: Welcome, Settings
Pre-Filter (for quality)
Crop: Automatic, Manual
Resize: Automatic, Manual
Tag/Caption: Generate, Process Tags, Process Captions, Edit
Post-Filter (for tags/captions)
Generation: Generate Prompts, View Metadata

Yeah, sounds like something that I'll eventually have to add if I keep adding new features but for now its not in my plans.


Custom application icon not showing on uploaded images

Describe the Bug
I set a custom application icon with the new setting. For most sites the icon is used as the favicon. But when I open an image in an new tab it will use the default bookstack favicon.
It seems like that /uploads doesn't use the custom application icon.
Steps to Reproduce

Set custom application icon
Open any site with a image in the text
Click on that image
In the new tab the favicon should be the default bookstack icon

Expected Behaviour
The custom application icon should be used as the favicon.
Screenshots or Additional Context
No response
Browser Details
No response
Exact BookStack Version
v23.01.1
PHP Version
8.2
Hosting Environment
Installed on AlmaLinux 9, Apache httpd as webserver

Thanks @JK1198 for reporting.
BookStack sets the icon via meta tags, so it will be only used on BookStack-rendered HTML pages.
Images won't go through BookStack at all in most cases.
We'd need to set the public/favicon.ico file directly for that but that's a little bit of a pain since .ico format support is limited and that's not a directory we advise to be writeable, so would need some extra file permission checks around that.

Have started on this in #4032, having some fun byte-level hacking the fairly-well supported bmp image format into ico format. Seems to work from a quick test.

Now addressed as part of  #4032, to be part of the next feature release.


Open Fyde r120 does not boot on Orange Pi 5b emmc

I follow the instructions for using the .run file and select the Pi 5b.  After some minutes, an .img file is created.  I flash this to my emmc. There are no issues or problems reported.  Upon rebooting the Orange Pi 5b, the red power led and green activity led are visible.  There is, however, no video output or anything that comes across the HDMI.
I have tried several times, and each time with r120 the image written to the emmc does not appear to boot.

by instruction you shall flash it to sdcard  boot sd and install from there to emmc, but it not boots from sd on opi 5 plus for me too.

I don't get the green activity light. Just the red power light