libs/libc: correct the getrandom(2) prototype

Summary
libs/libc: correct the getrandom(2) prototype
libs/libc/getrandom(2): add ORed bit mask definition
Impact
getrandom(2) prototype
Reference here:
https://man7.org/linux/man-pages/man2/getrandom.2.html
Testing
getrandom test

I'm opposed to this as the new flags don't do anything, they are just cruft copied over from Linux.
NuttX is a new OS, it does not have to copy every non-standard feature from Linux.

This is another hack to facilitate using third party Linix code that is not really supported. Is this what we as a group?
I would suggest that if you add the interface you have to add the code to supported and add a Kconfig to support the legacy interface.

David, getrandom() itself is not a hack, as it adds feature that is not possible to implement with POSIX alone. The reasons why use getrandom() instead of open("/dev/urandom") have been discussed in the mailing list in past. (low max number of fd's configured, attacker can try to exhaust fds on purpose, need for slightly more error handling)

I think there is a real problem with getrandom(), not caused by this PR but introduced by Issue 619 https://github.com/apache/incubator-nuttx/issues/619
There is a comment in getrandom() function description: "Note that this function cannot fail, other than by asserting." But this is no longer true, if nxsem_wait_uninterruptible() return ECANCELLED because some other task called pthread_cancel() for the thread calling getrandom(). This is very bad, as caller might consume the buffer before next cancellation point. I think solution is to make this function a cancellation point? Any thoughts about this? I think non-standard functions can be cancellation points, nothing in POSIX is prohibiting that.
The wrong solution would be to change getrandom() return value to int and force every caller to deal with this error. It it bad for security if every caller needs to deal with cancelled or otherwise failing getrandom(). What to do when it fails? Default to some insecure mode like calling rand()? (NuttX is on purpose different than Linux in this regard.)

Recently, it seems that the objectives of the OS have shifted.  Historically, for a dozen years, the objective has been to be a Unix-work alike and to conform strictly to the requirements of a Unix system as specified at OpenGroup.org.  This is still stated pretty clearly in the INVIOLABLES.md:
## Strict POSIX compliance

  - Strict conformance to the portable standard OS interface as defined at
    OpenGroup.org.
  - A deeply embedded system requires some special support.  Special
    support must be minimized.
  - The portable interface must never be compromised only for the sake of
    expediency.
  - Expediency or even improved performance are not justifications for
    violation of the strict POSIX interface.

Today, I think that objective has morphed; the OS is now treated like a Linux work-alike system.  That is not necessarily bad, it is just a different objective.  The only "bad" thing about this is that it happened with no conscious consent of the community.
As a standard, Unix-like OS, inclusion of non-standard Linux interfaces is inappropriate; as a Linux work-alike OS, it is almost mandatory to include these non-standard interfaces.  Being another Linux clone is not a very lofty objective for an OS although it many be practical for businesses who want to host Linux applications on NuttX.

I'm opposed to this as the new flags don't do anything, they are just cruft copied over from Linux.
NuttX is a new OS, it does not have to copy every non-standard feature from Linux.

But why do you have the right to add a new function which doesn't follow OpenGroup or other Unix variant? If you have this right, does it mean I also can add any funcitons I like?
getrandom has the same prototype in both FreeBSD and Linux:
https://www.freebsd.org/cgi/man.cgi?query=getrandom&sektion=2&manpath=freebsd-release-ports
https://man7.org/linux/man-pages/man2/getrandom.2.html
I don't think that there is any reason why we don't follow them.

Recently, it seems that the objectives of the OS have shifted. Historically, for a dozen years, the objective has been to be a Unix-work alike and to conform strictly to the requirements of a Unix system as specified at OpenGroup.org. This is still stated pretty clearly in the INVIOLABLES.md:
## Strict POSIX compliance

  - Strict conformance to the portable standard OS interface as defined at
    OpenGroup.org.
  - A deeply embedded system requires some special support.  Special
    support must be minimized.
  - The portable interface must never be compromised only for the sake of
    expediency.
  - Expediency or even improved performance are not justifications for
    violation of the strict POSIX interface.

Today, I think that objective has morphed; the OS is now treated like a Linux work-alike system. That is not necessarily bad, it is just a different objective. The only "bad" thing about this is that it happened with no conscious consent of the community.
As a standard, Unix-like OS, inclusion of non-standard Linux interfaces is inappropriate; as a Linux work-alike OS, it is almost mandatory to include these non-standard interfaces. Being another Linux clone is not a very lofty objective for an OS although it many be practical for businesses who want to host Linux applications on NuttX.

I need said that there are many features in NuttX which not defined by POSIX:

bluetooth, can, ieee802154 and netlink socket
Many libc function only define in FreeBSD or Linux(e.g. bzero...)
nx graphics stack
So my question is that should we remove all these features from code base to confirm the pure POSIX compliance or extend the rule to cover the functions defined by FreeBSD or Linux but not in POSIX.

getrandom has the same prototype in both FreeBSD and Linux:
https://www.freebsd.org/cgi/man.cgi?query=getrandom&sektion=2&manpath=freebsd-release-ports
https://man7.org/linux/man-pages/man2/getrandom.2.html
I don't think that there is any reason why we don't follow them.

NuttX getrandom() interface is modeled after OpenBSD's getentropy() and arc4random_buf():
https://man.openbsd.org/getentropy
https://man.openbsd.org/arc4random.3
It was originally called speckrandom_buf() but was changed later to getrandom() by my colleague when the underlying cipher was changed to BLAKE2Xs. Today, I would create it as nx_getrandom() to be clear that it is a NuttX interface (but I don't propose this change/rename now, after several years of use).
Linux is not the pinnacle of OS development. I'm opposed to idea that new operating systems mindlessly imitate Linux even when things can be improved with little effort. Linux people have the idea that no user-space callers every have to change due to kernel changes, so they are stuck with bad interfaces forever. That is a good rule for OS with huge installed space, but not for us. Interfaces that have two parameters are easier to use than ones with three. Interfaces that cannot ever fail are more reliable, secure and easier to use than ones that can.

getrandom has the same prototype in both FreeBSD and Linux:
https://www.freebsd.org/cgi/man.cgi?query=getrandom&sektion=2&manpath=freebsd-release-ports
https://man7.org/linux/man-pages/man2/getrandom.2.html
I don't think that there is any reason why we don't follow them.

NuttX getrandom() interface is modeled after OpenBSD's getentropy() and arc4random_buf():
https://man.openbsd.org/getentropy
https://man.openbsd.org/arc4random.3

It's fine that you model either getentropy, arc4random or getrandom.

It was originally called speckrandom_buf()

it's fine if speckrandom_buf is just called by kernel function, but not good for userspace.

but was changed later to getrandom() by my colleague when the underlying cipher was changed to BLAKE2Xs. Today, I would create it as nx_getrandom() to be clear that it is a NuttX interface (but I don't propose this change/rename now, after several years of use).

No, nx_xxx is just for OS internal function. If the function plan to expose to userspace(part of libc), it isn't a good idea to add nx_ prefix.

Linux is not the pinnacle of OS development. I'm opposed to idea that new operating systems mindlessly imitate Linux even when things can be improved with little effort. Linux people have the idea that no user-space callers every have to change due to kernel changes, so they are stuck with bad interfaces forever.

POSIX compliance is NuttX's most important unique feature compared with other RTOS. There are many API design isn't perfect, does it mean we should change API prototype? For example, many socket API accept only IPv4 address:
int bind(int sockfd, FAR const struct sockaddr *addr, socklen_t addrlen);
int connect(int sockfd, FAR const struct sockaddr *addr, socklen_t addrlen);
int accept(int sockfd, FAR struct sockaddr *addr, FAR socklen_t *addrlen);
ssize_t sendto(int sockfd, FAR const void *buf, size_t len, int flags,
               FAR const struct sockaddr *to, socklen_t tolen);
ssize_t recvfrom(int sockfd, FAR void *buf, size_t len, int flags,
                 FAR struct sockaddr *from, FAR socklen_t *fromlen);

But now we support many different transport layer(IPv6, Unix Domain, Bluetooth, CAN) which represent addresss in the different way(sockaddr_in6, sockaddr_un, sockaddr_can). Should we change addr type from struct sockaddr * to void *? I don't think so.

That is a good rule for OS with huge installed space, but not for us.

Do you know why @anchao find the wrong getrandom prototype? Because he blindly believe that NuttX implement getrandom as man described and then follow the documentation to write the code, but finally it fail to compile. I think nobody like this experience especially we always emphasize NuttX standard compliance.

Interfaces that have two parameters are easier to use than ones with three. Interfaces that cannot ever fail are more reliable, secure and easier to use than ones that can.

getrandom is designed to can be implemented by the hardware crypto accelerator, nobody can make 100% guarantee that the hardware will always work as expect(especially some accelerator connect to CPU through SPI, SDIO, USB bus). Now we have no way to report the error with your prototype?

But now we support many different transport layer(IPv6, Unix Domain, Bluetooth, CAN) which represent addresss in the different way(sockaddr_in6, sockaddr_un, sockaddr_can). Should we change addr type from struct sockaddr * to void *? I don't think so.

Nobody is suggesting we should change well entrenched APIs like socket functions. They are also in OpenGroup specs of course. I think simple library wrappers like bzero() are fine as well, as there is widespread existing practise to have such compatibility wrappers, they are trivial to implement, harmless to those who don't need them, and everyone agrees how they work.

Do you know why @anchao find the wrong getrandom prototype? Because he blindly believe that NuttX implement getrandom as man described and then follow the documentation to write the code, but finally it fail to compile. I think nobody like this experience especially we always emphasize NuttX standard compliance.

It is good it failed to compile. That allows programmer to notice, that maybe something is different for NuttX. Users should not expect that Linux man pages describe NuttX APIs. This is especially important for security code. I have seen at least one TLS implementation that resorts back to using rand() if getrandom() fails. It is very good that that kind of insecure code fails to compile for NuttX!
It is unfortunate the NuttX function is called getrandom(). I originally wanted it to have a distinctive name, but it was changed to getrandom() by other people.
This commit just introduces more confusion to system as it adds things that do not work or are contradicting existing code, like this piece in comments:
GRND_NONBLOCK  Don't block and return EAGAIN instead
NuttX getrandom() cannot return EGAIN and it cannot block (for an indefinite time at least, as the semaphore there can not really cause starvation). Comment is incorrect, because it hints that getrandom() blocks by default. On Linux it does, which is just another design failure of the Linux version. Why should we ever block for random numbers, or even have a blocking API, since we already have the ability to generate nearly infinite sequences of them without blocking?

getrandom is designed to can be implemented by the hardware crypto accelerator, nobody can make 100% guarantee that the hardware will always work as expect(especially some accelerator connect to CPU through SPI, SDIO, USB bus). Now we have no way to report the error with your prototype?

getrandom() is not designed specifically for hardware crypto accelerators. We use it in real products with MCUs that lack hardware RNG. It only needs enough initial entropy for one cryptographical key (unless generate huge amounts, several Gb, of output, then it must re-key itself) and it can get that from sensors, interrupt timings, factory production line etc.
To better understand these issues, please try this exercise: 1) implement Linux getrandom() using OpenBSD's arc4random() function 2) implement arc4random() using Linux's getrandom(). Part 1) is trivial, but for 2) one has an insurmountable problem, what to do if getrandom() fails (for any reason)?

It is good it failed to compile. That allows programmer to notice, that maybe something is different for NuttX. Users should not expect that Linux man pages describe NuttX APIs. This is especially important for security code. I have seen at least one TLS implementation that resorts back to using rand() if getrandom() fails. It is very good that that kind of insecure code fails to compile for NuttX!

But now you directly ignore the error at all, could you tell me how can the caller handle these error case securely?

nxsem_wait_uninterruptible return fail
user call getrandom too early and then no enough entropy is collected yet

It is unfortunate the NuttX function is called getrandom(). I originally wanted it to have a distinctive name, but it was changed to getrandom() by other people.
This commit just introduces more confusion to system as it adds things that do not work or are contradicting existing code, like this piece in comments:
GRND_NONBLOCK Don't block and return EAGAIN instead

If some flags don't make sense, we can remove it, but the prototype should keep consistently with the practice.

NuttX getrandom() cannot return EGAIN and it cannot block (for an indefinite time at least, as the semaphore there can not really cause starvation). Comment is incorrect, because it hints that getrandom() blocks by default. On Linux it does, which is just another design failure of the Linux version. Why should we ever block for random numbers, or even have a blocking API, since we already have the ability to generate nearly infinite sequences of them without blocking?

getrandom is designed to can be implemented by the hardware crypto accelerator, nobody can make 100% guarantee that the hardware will always work as expect(especially some accelerator connect to CPU through SPI, SDIO, USB bus). Now we have no way to report the error with your prototype?

getrandom() is not designed specifically for hardware crypto accelerators. We use it in real products with MCUs that lack hardware RNG. It only needs enough initial entropy for one cryptographical key (unless generate huge amounts, several Gb, of output, then it must re-key itself) and it can get that from sensors, interrupt timings, factory production line etc.

But how you tell caller if there isn't enough entropy has been collected yet?

To better understand these issues, please try this exercise: 1) implement Linux getrandom() using OpenBSD's arc4random_buf() function 2) implement arc4random_buf() using Linux's getrandom(). Part 1) is trivial, but for 2) one has an insurmountable problem, what to do if getrandom() fails (for any reason)?

It's arc4random_buf design problem: it doesn't return the error code. We should avoid to use this function, it is wrong to drop getrandom return value. If you look at how NuttX implementation now, there are many places the random generatation isn't really random:
void getrandom(FAR void *bytes, size_t nbytes)
{
  int ret;

  ret = nxsem_wait_uninterruptible(&g_rng.rd_sem);
  if (ret >= 0)
    {
      rng_buf_internal(bytes, nbytes);
      nxsem_post(&g_rng.rd_sem);
    }
}

static void rng_buf_internal(FAR void *bytes, size_t nbytes)
{
  if (!g_rng.output_initialized)
    {
      if (g_rng.rd_newentr < MIN_SEED_NEW_ENTROPY_WORDS)
        {
          cryptwarn("Entropy pool RNG initialized with very low entropy. "
                    " Consider implementing CONFIG_BOARD_INITRNGSEED!\n");
        }
      rng_reseed();
    }
  else if (g_rng.rd_newentr >= MAX_SEED_NEW_ENTROPY_WORDS)
    {
      /* Initial entropy is low. Reseed when we have accumulated more. */
      rng_reseed();
    }
  else if (g_rng.blake2xs.out_node_offset == UINT32_MAX)
    {
      /* Maximum BLAKE2Xs output reached (2^32-1 output blocks, maximum 128
       * GiB bytes), reseed.
       */
      rng_reseed();
    }

Do you think it's more secure to blindly ignore the error?

But now you directly ignore the error at all, could you tell me how can the caller handle these error case securely?
1. nxsem_wait_uninterruptible return fail

...

From #2497
getrandom() is not a cancellation point and must not implement any cancellation point logic. It should not use nxsem_wait_uninterruptible() but should use something like nxsem_wait_noncancelable(). nxsem_wait-noncancelable() has not been formalized but would simply ignore the ECANCELED error (as well as the EINTR error).
In that case, getrandom would correctly ignore the cancellation request and simply call nxsem_wait() again.
ECANCELED should only occur in the deferred cancellation mode. In other modes, the task calling getrandom() will be terminated immediately with no return value. (I think anyway. I haven't looked at the code in a long time). In the deferred cancellation mode, cancellation can only occur at a few cancellation points; getrandom() is not a cancellatoin point.

But now you directly ignore the error at all, could you tell me how can the caller handle these error case securely?
1. nxsem_wait_uninterruptible return fail

Implementation bug, corrected in https://github.com/apache/incubator-nuttx/pull/2497

2. user call getrandom too early and then no enough entropy is collected yet

Not a problem in practise. We fill most of otherwise empty internal flash space with good quality random bytes during device production and then can activate flash read-out-protection. It is also easy to store gathered entropy across resets (internal flash, BBSRAM) and reseed back to entropy pool during board initialization phase.

If the entropy isn't collected enough, we should block and wait the enough entropy by default, or return -1 with errno set to EAGAIN if user pass GRND_NONBLOCK. But your implementation just log a warning:
static void rng_buf_internal(FAR void *bytes, size_t nbytes)
{
  if (!g_rng.output_initialized)
    {
      if (g_rng.rd_newentr < MIN_SEED_NEW_ENTROPY_WORDS)
        {
          cryptwarn("Entropy pool RNG initialized with very low entropy. "
                    " Consider implementing CONFIG_BOARD_INITRNGSEED!\n");

Better to do as the warning says :) You can always change this to PANIC() if want to be really sure we never run past this point with insufficient entropy.

But how you tell caller if there isn't enough entropy has been collected yet? or the external hardware RNG can't access because I2C/SPI/USB bus is broken.

Can any real device do anything other than PANIC() if some of buses or other HW it really needs are broken?

It's arc4random_buf design problem: it doesn't return the error code. We should avoid to use this function, but it is wrong to drop getrandom return value.

arc4random_buf() is very carefully designed by people who know how to design cryptographical software. Linux and Glibc maintainers botched this with their overly complex and error prone interface. arc4random functions were deliberately designed to be easy to use to avoid crypto failures due to misuse by their caller. Of course this places burden to implementator to get it right. Note that we have other APIs that can never fail, getpid(), getuid(), longjmp(), pthread_getspecific() etc.
It seems Austin Group commenters got confused about this, interesting discussion though:  https://www.austingroupbugs.net/view.php?id=859

But now you directly ignore the error at all, could you tell me how can the caller handle these error case securely?
1. nxsem_wait_uninterruptible return fail

Implementation bug, corrected in #2497

2. user call getrandom too early and then no enough entropy is collected yet

Not a problem in practise. We fill most of otherwise empty internal flash space with good quality random bytes during device production and then can activate flash read-out-protection. It is also easy to store gathered entropy across resets (internal flash, BBSRAM) and reseed back to entropy pool during board initialization phase.

This is your implementation detail, but our factory process is totally different.

If the entropy isn't collected enough, we should block and wait the enough entropy by default, or return -1 with errno set to EAGAIN if user pass GRND_NONBLOCK. But your implementation just log a warning:
static void rng_buf_internal(FAR void *bytes, size_t nbytes)
{
  if (!g_rng.output_initialized)
    {
      if (g_rng.rd_newentr < MIN_SEED_NEW_ENTROPY_WORDS)
        {
          cryptwarn("Entropy pool RNG initialized with very low entropy. "
                    " Consider implementing CONFIG_BOARD_INITRNGSEED!\n");

Better to do as the warning says :) You can always change this to PANIC() if want to be really sure we never run past this point with insufficient entropy.

It's a runtime error, not a programming error at all, we should return errror here and let the caller decide what best action should to take.

But how you tell caller if there isn't enough entropy has been collected yet? or the external hardware RNG can't access because I2C/SPI/USB bus is broken.

Can any real device do anything other than PANIC() if some of buses or other HW it really needs are broken?

I don't think so, it's depends on the usecase. The caller may has more reasonable choice than PANIC(e.g. refuse the security connection).

It's arc4random_buf design problem: it doesn't return the error code. We should avoid to use this function, but it is wrong to drop getrandom return value.

arc4random_buf() is very carefully designed by people who know how to design cryptographical software. Linux and Glibc maintainers botched this with their overly complex and error prone interface. arc4random functions were deliberately designed to be easy to use to avoid crypto failures due to misuse by their caller. Of course this places burden to implementator to get it right. Note that we have other APIs that can never fail, getpid(), getuid(), longjmp(), pthread_getspecific() etc.
It seems Austin Group commenters got confused about this, interesting discussion though: https://www.austingroupbugs.net/view.php?id=859

If you think arc4random_buf is better design, please remove getrandom and implement arc4random_buf instead. But since it's impossible to return error code for arc4random_buf, how to handle the low entropy case? PANIC, I don't think it's good choice as note before.

If you think arc4random_buf is better design, please remove getrandom(wrong prototype) and implement arc4random_buf instead.

The title of this PR it incorrect as well, as it implies that some other prototype is somehow more correct than the existing one, which is not the case as Linux getrandom() is a non-standard, non-portable interface that exists on Linux alone.

But since it's impossible to return error code for arc4random_buf, how to handle the low entropy case? PANIC, I don't think it's good choice as note before.

Low entropy case is mostly theoretical. If your MCU has a built-in RNG (most do these days), then seed the entropy pool from it during board init, before rest of the system is running. Most designs read ADC voltages, temperatures etc. early on so don't have to rely solely on HW RNG, which should be just one of the inputs to the entropy pool, not the only one. I have never had this supposed low entopy problem in practise.

Recently, it seems that the objectives of the OS have shifted. Historically, for a dozen years, the objective has been to be a Unix-work alike and to conform strictly to the requirements of a Unix system as specified at OpenGroup.org. This is still stated pretty clearly in the INVIOLABLES.md:
## Strict POSIX compliance

  - Strict conformance to the portable standard OS interface as defined at
    OpenGroup.org.
  - A deeply embedded system requires some special support.  Special
    support must be minimized.
  - The portable interface must never be compromised only for the sake of
    expediency.
  - Expediency or even improved performance are not justifications for
    violation of the strict POSIX interface.

Today, I think that objective has morphed; the OS is now treated like a Linux work-alike system. That is not necessarily bad, it is just a different objective. The only "bad" thing about this is that it happened with no conscious consent of the community.
As a standard, Unix-like OS, inclusion of non-standard Linux interfaces is inappropriate; as a Linux work-alike OS, it is almost mandatory to include these non-standard interfaces. Being another Linux clone is not a very lofty objective for an OS although it many be practical for businesses who want to host Linux applications on NuttX.

Thinking out loud here: What about a compromise: by default NuttX is a Unix-like OS with POSIX interface as originally intended. In addition, if the user wants certain Linux-like APIs, the user could activate a new Kconfig option, say, CONFIG_LINUX_COMPAT. Likewise, if the user wants certain BSD-like APIs, a similar Kconfig option CONFIG_BSD_COMPAT could exist... The idea being to give people a way to add Linux/BSD APIs they need/want, but keeping them optional so that other users could exclude them.

Hi @juniskane
I don't understand what we're arguing about, Whatever the name you like to call it getentropy，arc4random_ buf, it's up to you.
but could you please bring the naming getrandom() back to POSIX ?
Seriously, no one cares about your implementation，I only care about whether the definition of user interface can be easily accepted by everyone.
Application developers only need to check the man page from POSIX !!!
https://www.unix.com/man-page/posix/2/getrandom/
You prefer to define user interfaces in reliable, secure and easier to use, this is very commendable, the goal is consistent with us,
but the prototypes is a legacy of history, if we don't fix it soon, more developers will question it in the future，
Q："why the getrandom(2) does not strictly follow the POSIX compliance ?"
Juniskane： "Interfaces that have two parameters are easier to use than ones with three. Interfaces that cannot ever fail are more reliable...."
Q："why the getrandom(2) does not strictly follow the POSIX compliance? NuttX should be Strict POSIX compliance!"
Juniskane： "Oh, Dear developer, Interfaces that have two parameters are easier to use than ones with three. Interfaces that cannot ever fail are more reliable...."
What should we do like this "tomorrow"？
I think the best solution is to accept the legacy, accept the history, there is no absolute correctness or wrongness.
this is just because different people may view the same thing from different perspectives,
A standard interface will make it easier for application developers to feel the perfection of NuttX,
what an amazing thing to let a FreeBSD or Linux application run in NuttX without any changes,
I don't quite understand what will affect you if change to a compliance prototype?
getranom() is not in your openBSD page,  Could you please change the name back to arc4random_buf(), thank you very much !!

I don't understand what we're arguing about, Whatever the name you like to call it getentropy，arc4random_ buf, it's up to you.
but could you please bring the naming getrandom() back to POSIX ?
Seriously, no one cares about your implementation，I only care about whether the definition of user interface can be easily accepted by everyone.
Application developers only need to check the man page from POSIX !!!
https://www.unix.com/man-page/posix/2/getrandom/

That is just the Linux man page for getrandom(). I am well familiar with Linux getrandom(2) as my colleagues and I studied it back in 2015 when we decided to improve the interface instead of mindlessly copying Linux. You seem to confuse POSIX with Linux manpages? getrandom() has never been in any standards, be they formal international standards or industry group de-facto standards.
Please see my reply to Xiao Xiang in https://github.com/apache/incubator-nuttx/pull/2497, I suggest you read in detail everything referenced there and in this thread as there is more to this topic than just the prototype. LKML mailing list has also had extensive discussions about getrandom().

Solaris 11.3 man page says for getrandom(): "Otherwise, it returns 0 and sets errno to indicate the error." Linux/FreeBSD error return value is -1. This may be just an error in Solaris man page though.

On Solaris if GRND_RANDOM is not specified then getrandom() never blocks. On Linux, GRND_NONBLOCK is needed for this even for urandom pool as random source (at least with some kernel versions). Which way should NuttX behave?
Solaris getrandom() will either fail completely or will return a buffer filled with the requested size, Linux version can return partial buffers if interrupted by signal. Which way should NuttX behave?

But, all these aren't the justification to change the function prototype. Because the current implementation only support the nonblocking behaviour, the right thing is checking GRND_NONBLOCK flag and return return error directly if user forget to set it.

@anchao per the dissussion here #2497, please rename getrandom to arc4random_buf instead.

Because the current implementation only support the nonblocking behaviour, the right thing is checking GRND_NONBLOCK flag and return return error directly if user forget to set it.

No, the correct thing is to ignore GRND_NONBLOCK as it is not needed in this implementation. getrandom() with flags == 0 should not return any spurious errors if it can complete the request. I think with current implementation, flags can be ignore completely for now. But since you and anchao want to bring the badly designed Linux version to NuttX, I'd expect you to be mindful about little details like this. Have you checked, if the FreeBSD and Linux versions are different from each other in some subtle way? We cannot completely ignore Solaris precedent (and others who might have implemented getrandom()), as people might port Solaris code to NuttX as well.

So the plan is rename current function as arc4random_buf and also add the three argument Linux version? I guess I have to live with that. That way we are compatible with both Linux and BSD. arc4random_int(), arch4random_uniform() and the blocking behavior in (mostly theoretical if system designer anticipated it) low entropy case can be added in later commits, if someone wants to do that. The latter must not cause arc4random() functions to block in any case.

Because the current implementation only support the nonblocking behaviour, the right thing is checking GRND_NONBLOCK flag and return return error directly if user forget to set it.

No, the correct thing is to ignore GRND_NONBLOCK as it is not needed in this implementation. getrandom() with flags == 0 should not return any spurious errors if it can complete the request. I think with current implementation, flags can be ignore completely for now.

If the entropy is low, the currrent NuttX implementation isn't safe, because the returned value isn't really random in this case. It's a secure hole to use this value blindly without any caution.

But since you and anchao want to bring the badly designed Linux version to NuttX, I'd expect you to be mindful about little details like this. Have you checked, if the FreeBSD and Linux versions are different from each other in some subtle way? We cannot completely ignore Solaris precedent (and others who might have implemented getrandom()), as people might port Solaris code to NuttX as well.

NuttX implement many POSIX API, it isn't impossible that NuttX has the same behaviour in all case as FreeBSD or Linux. But anyway, we need try the best to ensure the compability.

So the plan is rename current function as arc4random_buf and also add the three argument Linux version of getrandom? I guess I have to live with that. That way we are compatible with both Linux and various BSD Unixes. arc4random_int(), arch4random_uniform() and the blocking behavior in (mostly theoretical if system designer anticipated it) low entropy case can be added in later commits, if someone wants to do that. The latter must not cause arc4random*() functions to block in any case.

getrandom will be removed, because the prototype is wrong. Only arc4random_buf exist in the future.

Have you checked, if the FreeBSD and Linux versions are different from each other in some subtle way? We cannot completely ignore Solaris precedent (and others who might have implemented getrandom()), as people might port Solaris code to NuttX as well.

See my thoughts above about introducing some kind of CONFIG_COMPAT_LINUX, CONFIG_COMPAT_BSD options. The idea is: if none of these options are activated, NuttX will only be Posix. If CONFIG_COMPAT_LINUX, that activates some linux compatibility interfaces. If there is a question which version of an interface should be used, the CONFIG_COMPAT_* options can be used for disambiguation. The disadvantage of this idea is more difficult maintenance. The advantage is that if people want some Linux or BSD interfaces to make it easier to port programs from those OS, the interfaces can be added, but are optional. So we don't violate the Inviolables :-)

Have you checked, if the FreeBSD and Linux versions are different from each other in some subtle way? We cannot completely ignore Solaris precedent (and others who might have implemented getrandom()), as people might port Solaris code to NuttX as well.

See my thoughts above about introducing some kind of CONFIG_COMPAT_LINUX, CONFIG_COMPAT_BSD options. The idea is: if none of these options are activated, NuttX will only be Posix. If CONFIG_COMPAT_LINUX, that activates some linux compatibility interfaces. If there is a question which version of an interface should be used, the CONFIG_COMPAT_* options can be used for disambiguation. The disadvantage of this idea is more difficult maintenance. The advantage is that if people want some Linux or BSD interfaces to make it easier to port programs from those OS, the interfaces can be added, but are optional. So we don't violate the Inviolables :-)

libc implementer already use _XXX_SOURCE to enable the non standard functions:
https://ftp.gnu.org/old-gnu/Manuals/glibc-2.2.3/html_node/libc_13.html

@juniskane please review the update.

@juniskane please review the update.

Rename commits look OK to me.

@juniskane thanks, I will merge it now.


To Use OpenCC in Java

I browsed through past issues and know that there are two ways to use OpenCC in Java. 1) To integrate with jni, and 2) to invoke the command line. However, I didn't see how to integrate them together exactly.
Is there any example about using OpenCC in Java with jni?
Thank you!

Is there any example about using OpenCC in Java with jni?

Perhaps https://github.com/osfans/trime?

https://github.com/qichuan/android-opencc


Bump package version to 0.45.1

@fedorov I need you or someone else to review/approve this PR to bump the project's version.


Interactive Learning: terminal text cutting off if prompt is larger than terminal window

Rasa Core version: 0.11.12
Python version: 3.6.5
Operating system (windows, osx, ...): macOS
Issue: When doing interactive learning, the bolded text the system sends back cuts off if the terminal window is too small. It doesn't wrap to the following line, even if you try to expand the window after. Sometimes it cuts off so much that you have to go back and reset the previous action because you can't tell what intent was triggered. Screenshots below.

Thanks for raising this issue, @MetcalfeTom  will get back to you about it soon.

Yup, it's a bug!  Seems to be a problem with PyInquirer's prompt.  Thanks for bringing this to our attention @cjcelotto, I'll add it to the backlog
Related to this is that textwrap will terminate the script with ValueError: invalid width if the window width is too narrow.  Not sure if this is fixable but worth a look into as well.

Awesome! Thank you @akelad and @MetcalfeTom .

I opened the issue in PyInquirer repo: https://github.com/CITGuru/PyInquirer/issues/21

@Ghostvv Sorry for late response, this should be fix soon and @cjcelotto

@CITGuru thank you

@Ghostvv This has been fixed here https://github.com/CITGuru/PyInquirer/releases/tag/1.0.3

@CITGuru thank you, we'll update our requirements


Ability to send automated monthly email newsletter to showcase new Latinas added

Automate email newsletter so if there's even just 1 new Latina added, it will trigger a monthly email newsletter and be sent out to all users who opt into email notifs

Since we're hoping that our platform takes off, I'm highly considering integrating a transactional email service as opposed to using Python's integrated SMTP module.
A few pros to using SMTP:

Easy to set up
Cost effective
Platform agnostic

Cons to SMTP:

Less secure
No built-in analytics
Longer send times
Long-term maintenance and uptime burden

Pros of transactional service:

Analytics
Higher email delivery rates
Faster email delivery speeds
Scalability
Better for large volume of emails

Cons of transactional service:

Slight learning curve potentially
Third-party dependent
Higher growth == higher cost

Now, I've been doing a little bit of research and I'm a little familiar with AWS and they seem to offer the best "bang for your buck" in the sense that they offer a free plan where you get up to 62,000 emails/month. I already have an AWS account and could set it up and start integrating the API into my code, but I wanted to check in and see if anyone has any thoughts, comments, questions, opinions?
¡Gracias chicas!

Yep I agree a transactional email service makes sense
I've heard good things about mailgun and sendgrid but if you're familiar with the aws one, I think that's okay that we set it up there!

Yes, transactional! 👌🏽👩🏽💻
On Sat, Apr 1, 2023 at 7:31 PM Frances Coronel @.***>
wrote:

Yep I agree a transactional email service makes sense
I've heard good things about mailgun and sendgrid but if you're familiar
with the aws one, I think that's okay that we set it up there!
—
Reply to this email directly, view it on GitHub
https://github.com/Latina-Dev/latina-dev/issues/12#issuecomment-1493183471,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AZFXIF4WQVYCPYT6R6FY7ILW7DCFLANCNFSM6AAAAAAWN3SG7M
.
You are receiving this because you were mentioned.Message ID:
@.***>

AWS SES use case email sent at 3:39PM CST. Email details can be found below:
Hello Amazon AWS Team,
We appreciate you taking the time to review my request for production access. Only registered users will receive emails from us. Our website’s address is https://latina.dev/ and will be an open source website created with the following objective in mind:
Latina Dev is a free, open source website that will provide a number of different resources to its subscribers such as free resources for learning software development, a place for other Latina developers to connect and network, and monthly newsletters detailing new resources added and/or possible future events.
We currently are in the production stage. This means that we do not yet have emails, but are in the process of creating our platform and implementing a transactional email service. We intend to use AWS SES to provide weekly/monthly newsletters detailing free new resources for Latina developers to notably spread awareness of Latinas in tech, but to also give more Latinas the opportunity to become involved in software development through a supportive open source community platform. These emails will be obtained through LinkedIn connections, existing LinkedIn groups for Latinas in STEM, and through contacting organizations and communities focusing on uplifting and raising awareness of underrepresented peoples in tech.
In accordance with the AWS documentation, we will use Amazon SNS to enable bounce, complaint, and unsubscribe notifications. Once notified of any bounced, complaint, or unsubscribe email action, invalid emails will be placed in the Account-Level Suppression List. This avoids delivery of newsletter emails to invalid email addresses or recipients no longer interested in receiving newsletter emails.
We will also reference the “Amazon SES using an AWS SDK” along with “Email format in Amazon SES” documentation to ensure the compliance of email content and format (as advised by AWS) to help us maintain a good email sending reputation.
Finally, if you require any additional information in order to grant us production access, please do not hesitate to contact me.
Best,
Fernanda Pérez Gutiérrez—Latina Dev Contributor

AWS SES use case accepted with a sending quota of 50,000 messages per day and a maximum send rate of 14 messages per second.


crystal deps install should do a shallow clone

It doesn't seem that dependencies are cloned using shallow clones (at least the github ones). Unless there is a good reason not to, we don't need the full history of the repo when getting dependencies, do we?

With Git 1.9 this is indeed feasible. However prior to that shallow clones do not support fetch or pull, making updating them impossible.

We can always clone again in the case of an update if shallow clones don't support fetch or pull.

Yes, I was thinking the same (but forgot to reply, it seems), although that
poses a problem if the repo itself is very big, but I'm guessing libraries
are going to be small and not full of resources?
On Thu, Apr 23, 2015 at 8:39 AM, Ary Borenszweig<EMAIL_ADDRESS>wrote:

We can always clone again in the case of an update if shallow clones don't
support fetch or pull.
—
Reply to this email directly or view it on GitHub
https://github.com/manastech/crystal/issues/555#issuecomment-95357939.

I recently saw this commit for Bower that is related: https://github.com/bower/bower/pull/1628

I think we can close this in favour of #1357 and/or a specific issue at https://github.com/ysbaddaden/shards too.

@jhass Definitely! Feel free to close any issues related to the package manager, the discussion can continue in #1357 or in shards itself.


Fix for spaces in app name. Issue #2

Changed absolute import to a relative one. This makes the working of the app independent from the package name.

@freakboy3742 With the new changes it is possible to run the app also as a module as well as a package. I think this gives people the choice to run the app in both ways. I also had problems to debug a app that runs as a package.

I think it'd be stronger to prevent modules with spaces outright. Cookiecutter now by default includes a pre-build hook to prevent such names, which I've backported for our uses in #8.

I agree with @craiglabenz. Although this approach makes the immediate problem go away, an invalid app_name is going to cause problems elsewhere; better to catch it early.


How to check whether client id and client secret are correct before actually calling the API?

In my program I would like to check whether the supplied client id and client secret are correct before actually using them to add songs to playlists, etc.
Is there are special function to check whether stuff is correct?
My issue is that I would like to check the credentials at the start of the program and not when it is running and trying to call the API with incorrect credentials.
My code looks like this:
auth_manager = spotipy.oauth2.SpotifyOAuth(scope="playlist-modify-public playlist-modify-private "
                                                     "playlist-read-private",
                                               client_id='incorrect_id',
                                               client_secret='false_secret',
                                               redirect_uri='http://localhost:8888/callback')
sp = spotipy.Spotify(auth_manager=auth_manager)

# Here I would like to check whether client_id and client_secret are correct
# my current workaround is to just use me()
    try:
        spotipy_client.me()
    except SpotifyOauthError:
        print('Ya dun goofed')

This feels very hacky and I would like to know whether I can simply use a function to check.

The only way to check if the client id and secret are correct is to attempt to go through the authentication process. The web API is designed with the assumption that the client id and secret are hardcoded into the program (or, at least, they don't change at runtime) by the developer of that program, who knows if the values are correct.

Thanks for the reply. So my somewhat 'hacky' solution is the best one under the circumstances, right?

Yes.


Bugfixing

Made a couple of tweaks to get this working.

Errors on lambda:
START RequestId: 841ab4b8-33a2-11e6-af0b-3b2581f58095 Version: $LATEST
Unable to import module 'lambda_function': No module named dns.resolver

END RequestId: 841ab4b8-33a2-11e6-af0b-3b2581f58095
REPORT RequestId: 841ab4b8-33a2-11e6-af0b-3b2581f58095	Duration: 0.33 ms	Billed Duration: 100 ms

This error can be fixed by running this code in the lambda folder before zipping/uploading:
pip install dnspython -t .

Can you split this into two pull requests? The first commit "Bugfixing" looks good, but the second isn't ready to merge and changes things that shouldn't be changed(or that need more justification than "bugfixing")


Update vcpkg to 2024.06.15

Currently at 2024.02.14, which is 7 releases behind. Latest release is 2024.08.23
https://github.com/microsoft/vcpkg/releases

Original plan to update to 2024.08.23. However, that version has an issue on Mac that the install name of curl is not properly set. It is possible to use install_name_tool to correct the problem. But for now we can update to 2024.06.15 instead, and wait and see the next time when we update, whether the problem  has been fixed.


constant chunk size?

Regarding
https://github.com/openid/openid4vp_ble/blob/e5370d5b7a2fa82b627b67bda6858ff81a64114d/main.md?plain=1#L332
Are all chunks (except the last one) of the same size?
Example:
1000 Bytes payload split to 400 + 400 + 200 Bytes
Is it also allowed to send:
1000 Bytes payload split to 400 + 100 + 200 + 300 Bytes ?
I must admit that I have no real use case for this but the specification should be clear on that. Otherwise some verifiers might implement it that way and break compatibility.

As the MTU is negotiated, the chunk size is expected to follow the MTU size except the last one. The total content size is known so the last chunk can be less.
so 1000 Bytes payload split to 400 + 400 + 200  Bytes is correct if the negotiated MTU is ~400


[Feature Request] Not to show files listed in .git/info/exclude

Just as the title, maybe gitu could respect .git/info/exclude and not to show them in the file tree. Since I'm trying to use gitu to manage my dotfiles, listing every file in $HOME is not a good choice: have to wait for a couple of seconds that gitu could respond to user input.
Lazygit has this feature, but it seems to use the git command directly under the hood (I haven't read its code though.), so it might not be a proper reference.

Hi!
I think it already does this. Could you provide some steps to reproduce the problem?
I might've misunderstood, or else I suspect there's an issue regarding submodules or something.
gitu on  altsem/value-arguments ❯ cat .git/info/exclude
# git ls-files --others --exclude-from=.git/info/exclude
# Lines that start with '#' are comments.
# For a project mostly in C, the following would be a good set of
# exclude patterns (uncomment them if you want to use them):
# *.[oa]
# *~
invisible.txt

gitu on  altsem/value-arguments ❯ git status
On branch altsem/value-arguments
Your branch is up to date with 'origin/altsem/value-arguments'.

nothing to commit, working tree clean

gitu on  altsem/value-arguments ❯ touch visible.txt invisible.txt
gitu on  altsem/value-arguments [?] ❯ git status
On branch altsem/value-arguments
Your branch is up to date with 'origin/altsem/value-arguments'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        visible.txt

nothing added to commit but untracked files present (use "git add" to track)

And in gitu:
▌On branch altsem/value-arguments
▌Your branch is up to date with 'origin/altsem/value-arguments'.

 Untracked files
 visible.txt

 Recent commits
 99a75bb altsem/value-arguments origin/altsem/value-arguments refactor: explicitly close `pending_menu`, access args from prompts
 4cb8b25 test: refactor src/tests/log, test grep/limit "other"
 eb26e0a Add more tests
 691b815 fix snapshots
 1d1d7c5 Fix value_as::<T>()
 994cfd3 rename get_args to init_args
 c70ebc3 refactor: extract type alies for set_prompt's `default_fn`
 439474c refactor: run `cargo clippy --fix`
 44d0f48 refactor: remove requirement of `clone` for Args
 5a61344 Add tests

Sorry for the late reply! Quite busy these days.

I think it already does this.

I didn't notice it, sorry.
However, there is an issue of lagging when starting gitu. I've recorded a video.
https://github.com/altsem/gitu/assets/40446983/dec68589-b676-42e8-8c90-fdb1444d6bc9
Take my dotfiles repo for an example, there is ~150 modified files, and ~40 untracked files, Starting Lazygit is fast, but when it comes to gitu, have to wait for 1~2s. As you can see, there is about 100 lines of exclude in my .git/info/exclude. So probably there is some performance related issues?

@chenrry666 No worries! Hmmm. I tried just creating a bunch of random files and listing them doesn't seem to take too long time. It's hard to tell from the video, but I'm guessing that filtering actually does fail.
Gitu is using libgit2 to retrieve the status. I found this open issue about the .gitignore (could be related?): https://github.com/libgit2/libgit2/issues/6250
It'd be interesting to know more about your dotfiles setup.
Which OS are you on?
Do you use worktrees or is the repo a submodule?
Are the files not being ignored symlinks?

I am using Arch Linux. The dotfiles repo isn't bare repo, so it is using worktree. The files are correctly ignored, sorry I didn't mention that.
The issue is that the lagging before the TUI being drawn may relate to some performance issues. Maybe the filtering process causes the lagging, IMO.

Alright. It could be like you say. And the code is indeed not quite optimized and will try render everything (even though they're not shown).
How many files is it? I think git status --porcelain | wc -l would answer it :)

I couldn't reach my laptop in two weeks at the moment. 🫠  It was sent for repair just this afternoon. Hence I could't provide more details in a short time.
I do remember there is more than 40k files in my $HOME (including yarn cache, crates, etc.) so it is probably where the lagging comes from. 🥲

Luckily I am able to come back. Here are the results:
❯ git status --porcelain | wc -l
70
❯ find . -type f |wc -l
163871

Much files here :cold_sweat: but git has correctly ignored it.


Dao UT coverage and map support

[x] Increase Dao UT coverage
[x] Add map support from Dao.

Fixed. Close.


Ability to disable attribute policy checks at runtime via config

Policy checks on large sets of model instance can get slow; this switch may help.

The difference seems to be so small for my use cases. Only couple milliseconds for relatively large set (10 k) of models.

The idea was to be able to turn off the functionality in testing or from the console at will.


[Bug Report]: deploying vnet getting - AggregateTypeValidation / TemplateViolation

Describe the bug
Getting started using Resource Modules, trying to create a vnet using the min configuration.
getting the error -
{"code": "InvalidTemplate", "target": "/subscriptions/b69f2302-5e6b-4d23-97f6-622c9d99a762/resourceGroups/ms.network.virtualnetworks-learn-rg/providers/Microsoft.Resources/deployments/pd4iykfzvvxku-test-learn", "message": "Deployment template validation failed: 'Template language version must be specified in order to use feature 'AggregateTypeValidation'.'.", "additionalInfo": [{"type": "TemplateViolation", "info": {"lineNumber": 0, "linePosition": 0, "path": ""}}]}
using az bicep -
ubuntu@DevBox:/data/lz-kube$ az bicep version
Bicep CLI version 0.12.1 (e43d137c2d)

This appears to be caused by this issue in Bicep
To reproduce
using the bison template file with <<namePrefix>> replaced.
deploy using -
az deployment sub create -l uksouth -f main.bicep -n main0026
Code snippet
No response
Relevant log output
`{"code": "InvalidTemplate", "target": "/subscriptions/b69f2302-5e6b-4d23-97f6-622c9d99a762/resourceGroups/ms.network.virtualnetworks-learn-rg/providers/Microsoft.Resources/deployments/pd4iykfzvvxku-test-learn", "message": "Deployment template validation failed: 'Template language version must be specified in order to use feature 'AggregateTypeValidation'.'.", "additionalInfo": [{"type": "TemplateViolation", "info": {"lineNumber": 0, "linePosition": 0, "path": ""}}]}`

This is my first run at using ResourceModules - I may be doing something stupid!

By the way, you can see the unforunate effect in our pipeline overview:

which is also still struggling with the second introduced Bicep bug: https://github.com/Azure/bicep/issues/8890

Another note: To test deploy, I recommend using the script utilities\tools\Test-ModuleLocally.ps1 as it automatically replaces tokens for you. You just have to setup you first invocation once. For example like this:
# #########[ Function Test-ModulesLocally.ps1 ]#############
. 'C:\dev\ip\Azure-ResourceModules\ResourceModules\utilities\tools\Test-ModuleLocally.ps1'

$TestModuleLocallyInput = @{
    TemplateFilePath           = 'C:\dev\ip\Azure-ResourceModules\ResourceModules\modules\Microsoft.RecoveryServices\vaults\deploy.bicep'
    ModuleTestFilePath         = 'C:\dev\ip\Azure-ResourceModules\ResourceModules\modules\Microsoft.RecoveryServices\vaults\.test\common\deploy.test.bicep'
    PesterTest                 = $false
    ValidationTest             = $false
    DeploymentTest             = $true
    ValidateOrDeployParameters = @{
        Location          = 'westeurope'
        ResourceGroupName = 'validation-rg'
        SubscriptionId    = '11111111-1111-1111-1111-111111111111'
        ManagementGroupId = '11111111-1111-1111-1111-111111111111'
        RemoveDeployment  = $false
    }
    AdditionalTokens           = @{
        deploymentSpId = '11111111-1111-1111-1111-111111111111'
        tenantId       = '11111111-1111-1111-1111-111111111111'
        namePrefix     = 'randm'
    }
}
Test-ModuleLocally @TestModuleLocallyInput

Alright, the latest release is out an the issue seems to be gone. @theMichaelB, please update and test again. I'll close the issue now :)


libmanette: 0.2.2 -> 0.2.3

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/libmanette/versions.
meta.description for libmanette is: '"A simple GObject game controller library"'.
meta.homepage for libmanette is: '"https://wiki.gnome.org/Apps/Builder"

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.2.3 with grep in /nix/store/3082398br3x0n9dngnl5x9jzyhz1qypp-libmanette-0.2.3

directory tree listing: https://gist.github.com/e2129448618f6a10ebc3d8c7998096cd

du listing: https://gist.github.com/8e62090f0faa91952847e1a0f3ef3bf8

Rebuild report (if merged into master) (click to expand)

7 total rebuild path(s)
2 package rebuild(s)
2 x86_64-linux rebuild(s)
2 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
2 aarch64-linux rebuild(s)
First fifty rebuilds by attrpath
gnome3.quadrapassel
libmanette

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/3082398br3x0n9dngnl5x9jzyhz1qypp-libmanette-0.2.3 \
  --option binary-caches 'https://cache.nixos.org/ https://r-ryantm.cachix.org/' \
  --option trusted-public-keys '
  r-ryantm.cachix.org-1:gkUbLkouDAyvBdpBX0JOdIiD2/DP1ldF3Z3Y6Gqcc4c=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(r-ryantm's Cachix cache is only trusted for this store-path realization.)
Or, build yourself:
nix-build -A libmanette https://github.com/r-ryantm/nixpkgs/archive/0c15f0d0ca08a0d7db9b9393083681f57e94a242.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/3082398br3x0n9dngnl5x9jzyhz1qypp-libmanette-0.2.3
ls -la /nix/store/3082398br3x0n9dngnl5x9jzyhz1qypp-libmanette-0.2.3/bin

cc @lethalman @jtojnar @hedning @worldofpeace for testing.

$ ~/Projects/what-changed/commits.py HEAD
0c15f0d0ca08a0d7db9b9393083681f57e94a242: libmanette: 0.2.2 -> 0.2.3
	=============
	Version 0.2.3
	=============

	* Build system changes:
	 - Add 'gudev' option, allowing to explicitly build without gudev.
	* Fix build on Fedora 30.


Backup using Checkpointing

Possible API

Keyspace::backup_to<P: AsRef>(path: P) -> crate::Result<()>;
TxKeyspace::backup_to<P: AsRef>(path: P) -> crate::Result<()>;

(When we have https://github.com/fjall-rs/fjall/issues/70 we could also provide a offline backup method that takes an exclusive temporary lock of an unopened Keyspace and clones it file-by-file (ideally using hard links))
Steps

[ ] Implement a copy method in value-log and lsm-tree to copy an LSM-tree in its current form - hard linking if possible, make sure every file is synced, including metadata files (manifest, ...) - this should be easy to unit test
[ ] In fjall, lock the journal and a global lock (let's call it BackupLock) to prevent garbage collection and compactions from submitting results (which would change disk segments)
[ ] Create the new folder
[ ] Fsync & copy all journal files
[ ] Drop journal lock (write ops can now proceed)
[ ] For each partition, call partition.tree.copy to the appropiate position in the new folder
[ ] Copy over keyspace-level metadata files (actually it's just version)
[ ] Finish backup (drops BackupLock, everything else can now proceed)
[ ] Using the global lock, prevent any changes to all LSM-tree levels (no segments are allowed to disappear through compaction, flushes can actually proceed technically, as long as they are not picked up by the backup)

Unsure
How to prevent compactions from applying their result? There may be compactions going on currently. We may need to pass the BackupLock to all trees, and make compactors take the lock? Can this be tested reliably? Same goes for blob file rewrites.

Also discussed in https://github.com/fjall-rs/fjall/discussions/50

Originally posted by jeromegn May 19, 2024
If one wanted to do a backup of the database, what's the best practice here? Is there a way to do an "online" (not shutting down the process using the database) backup?
Since there are many directories and files, I assume the safest way is to persist from the process, exit and then create a tarball of the whole directory.
With SQLite, for example, it's possible to run a VACUUM INTO that will create a space-efficient snapshot of the database. SQLite can support multiple processes though, so it's a different ball game.

Apache Paimon is an LSM format designed for writing to S3 compatible object storage while it's running. I really liked this author's articles on it.
https://jack-vanlightly.com/analyses/2024/7/3/understanding-apache-paimon-consistency-model-part-1

Has there been any more thought on this? I'm looking at using this crate but backups are necessary for my use case. I wouldn't mind trying out an implementation if theres been a general decision on the right approach.

The general idea is to use RocksDB-like Checkpoints for online backups, I didn't want to add it in 1.x because I knew 2.x would add and change some stuff anyway. But to fully implement it, there will need to be some synchronization, because the journal(s) need to be copied, then all the disk segments (hard-linked if possible) and metadata files, for all partitions. Once the backup starts, no journal GC, journal writes, blob file rewrites, memtable flushes or compactions are allowed to complete until the backup is done, but reads should not be blocked. So I think there needs to be a keyspace-level RwLock...
For offline backups, simply using cp -R will work.

I have added a todo list to the OP. I still have some stuff coming up for 2.4.0, so this isn't gonna make it in there, but I want to take a look at it in the near future... unless someone wants to take a look into it and contribute a possible solution, even if just a draft.

@Svenskunganka I think it can be more granular. Copying the active journal should be quick, so we just need to lock it shortly. Then, we can already unlock the active journal - and then, we copy the sealed journals, they are just not allowed to be dropped by journal GC during all this time for consistency.


using in Windows system get error "cannot load module 'libsqlite3'"

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

Ah okay found a solution

replace the tami5/sqlite.lua with kkharji/sqlite.lua
follow the instruction in kkhariji/sqlite, Download sqlite3 precompiled and set let g:sqlite_clib_path = path/to/sqlite3.dll (note: /)

for me I put vim.g.sqlite_clib_path = "C:\\xxxxxx\\sqlite3\\sqlite3.dll" in my user init.lua file

Ah okay found a solution

replace the tami5/sqlite.lua with kkharji/sqlite.lua
follow the instruction in kkhariji/sqlite, Download sqlite3 precompiled and set let g:sqlite_clib_path = path/to/sqlite3.dll (note: /)

for me I put vim.g.sqlite_clib_path = "C:\\xxxxxx\\sqlite3\\sqlite3.dll" in my user init.lua file

It also works for me.Thanks a lot!😘

I have encountered similar cases.
I am using an M2 MacbookAir and was getting an error trying to read the files under /opt/homebrew in sqlite.
So I used homebrew and did the following, which solved the problem.
$ which sqlite3
/usr/bin/sqlite3

$ brew install sqlite
$ echo 'export PATH="/opt/homebrew/opt/sqlite/bin:$PATH"' >> ~/.zshrc
$ source ~/.zshrc
$ brew link sqlite3 --force
$ which sqlite3
/opt/homebrew/opt/sqlite/bin/sqlite3

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

for

I tried running  "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info?
For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info? For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

OS: Ubuntu 22.04 lts
Neovim version: 0.7.2 (nix pkg)
Configuration: I am using nvchad vanilla without any modifications https://nvchad.com/
Error message : "packer.nvim: Error running config for telescope-frecency.nvim: ...im/site/pack/packer/
start/sqlite.lua/lua/sqlite/defs.lua:31: libsqlite3.so: cannot open shared object file
: No such file or directory"
I also installed sqlite3 but the error still persisted. BTW thanks for clarification

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info? For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

OS: Ubuntu 22.04 lts Neovim version: 0.7.2 (nix pkg) Configuration: I am using nvchad vanilla without any modifications https://nvchad.com/ Error message : "packer.nvim: Error running config for telescope-frecency.nvim: ...im/site/pack/packer/ start/sqlite.lua/lua/sqlite/defs.lua:31: libsqlite3.so: cannot open shared object file : No such file or directory"
I also installed sqlite3 but the error still persisted. BTW thanks for clarification

I'm sorry I made a mistake in my previous reply,this plugin need sudo apt-get install sqlite3 libsqlite3-dev # Ubuntu,not only the sudo apt-get install sqlite3.Which you can refer his doc in Github.When I replied you I was having my breakfast and I reply with mobile Github so I didn't pay much caution to my answer.It's my fault.
If this didn't work,can you post your plugin's specification?That means the code you write in your packer.For instance,in my ubuntu nvim's config,it is the code like this:
    use {
        "nvim-telescope/telescope-frecency.nvim",
        config = function()
            require"telescope".load_extension("frecency")
        end,
        requires = {"kkharji/sqlite.lua"}
    }

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info? For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

OS: Ubuntu 22.04 lts Neovim version: 0.7.2 (nix pkg) Configuration: I am using nvchad vanilla without any modifications https://nvchad.com/ Error message : "packer.nvim: Error running config for telescope-frecency.nvim: ...im/site/pack/packer/ start/sqlite.lua/lua/sqlite/defs.lua:31: libsqlite3.so: cannot open shared object file : No such file or directory"
I also installed sqlite3 but the error still persisted. BTW thanks for clarification

I'm sorry I made a mistake in my previous reply,this plugin need sudo apt-get install sqlite3 libsqlite3-dev # Ubuntu,not only the sudo apt-get install sqlite3.Which you can refer his doc in Github.When I replied you I was having my breakfast and I reply with mobile Github so I didn't pay much caution to my answer.It's my fault.
If this didn't work,can you post your plugin's specification?That means the code you write in your packer.For instance,in my ubuntu nvim's config,it is the code like this:
    use {
        "nvim-telescope/telescope-frecency.nvim",
        config = function()
            require"telescope".load_extension("frecency")
        end,
        requires = {"kkharji/sqlite.lua"}
    }

Thank you very much for your help it work but I had to do some tweaks
I installed libsqlite3-dev as you said but the problem still persisted but then I ran
this command
let g:sqlite_clib_path = "/usr/lib/x86_64-linux-gnu/libsqlite3.so"
inside vim then it could detect the file
I think the issue is not in the frequency  but the "sqlite.lua" itself
I added this to my config and now it runs as expected

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info? For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

OS: Ubuntu 22.04 lts Neovim version: 0.7.2 (nix pkg) Configuration: I am using nvchad vanilla without any modifications https://nvchad.com/ Error message : "packer.nvim: Error running config for telescope-frecency.nvim: ...im/site/pack/packer/ start/sqlite.lua/lua/sqlite/defs.lua:31: libsqlite3.so: cannot open shared object file : No such file or directory"
I also installed sqlite3 but the error still persisted. BTW thanks for clarification

I'm sorry I made a mistake in my previous reply,this plugin need sudo apt-get install sqlite3 libsqlite3-dev # Ubuntu,not only the sudo apt-get install sqlite3.Which you can refer his doc in Github.When I replied you I was having my breakfast and I reply with mobile Github so I didn't pay much caution to my answer.It's my fault.
If this didn't work,can you post your plugin's specification?That means the code you write in your packer.For instance,in my ubuntu nvim's config,it is the code like this:
    use {
        "nvim-telescope/telescope-frecency.nvim",
        config = function()
            require"telescope".load_extension("frecency")
        end,
        requires = {"kkharji/sqlite.lua"}
    }

Thank you very much for your help it work but I had to do some tweaks I installed libsqlite3-dev as you said but the problem still persisted but then I ran this command let g:sqlite_clib_path = "/usr/lib/x86_64-linux-gnu/libsqlite3.so"
inside vim then it could detect the file I think the issue is not in the frequency but the "sqlite.lua" itself I added this to my config and now it runs as expected

My pull request  on kkharji/sqlite.lua has been merged thus there is no need of this work around on Linux as I have fixed this issue on linux

Hello
I am using neovim 0.7.2 in Windows 10, sqlite installed via scoop
Installed via
use {
  "nvim-telescope/telescope-frecency.nvim",
  config = function()
    require"telescope".load_extension("frecency")
  end,
  requires = {"tami5/sqlite.lua"}
}

But when I run, I always got
E5108: Error executing lua ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: cannot load module 'libsqlite3': <d5>Ҳ<bb><b5><bd><b6><a8><b5><c4>ģ<bf>顣

stack traceback:

        [builtin#203]: at 0x7ff9657a6340
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/defs.lua:31: in main chunk
        [C]: in function 'require'
        ...a\site\pack\packer\start\sqlite.lua/lua/sqlite/utils.lua:252: in function '__index'
        ...data\site\pack\packer\start\sqlite.lua/lua/sqlite/db.lua:662: in main chunk
        [C]: in function 'require'
        ...ta\site\pack\packer\start\sqlite.lua/lua/sqlite/init.lua:66: in function '__index'
        ....nvim/lua/telescope/_extensions/frecency/sql_wrapper.lua:46: in function 'bootstrap'
        ...cy.nvim/lua/telescope/_extensions/frecency/db_client.lua:92: in function 'init'
        ...ope-frecency.nvim/lua/telescope/_extensions/frecency.lua:285: in function 'setup'
        ...\start\telescope.nvim/lua/telescope/_extensions/init.lua:20: in function '__index'
        [string ":lua"]:1: in main chunk

How can I fix this ?

The Exact same error happened to me also. I am on Ubuntu 22.04 LTS ,neovim installed through nix

Hello,have you installed sqlite3 in your ubuntu? Please find the README of your plugin's 'requires' to check the environment require.As for me,I use '        requires = {"kkharji/sqlite.lua"}',and in its document,I must install sqlite in my device before,using 'sudo apt-get install sqlite3'.

I tried running "sudo apt-get install sqlite3" but the problem still persisted but strangely Telescope.nvim still kinda works

Can you post your specification here,with your error info? For your confuse, the Telescope isn't a plugin rely on sqlite3,but its extension frequency need.

OS: Ubuntu 22.04 lts Neovim version: 0.7.2 (nix pkg) Configuration: I am using nvchad vanilla without any modifications https://nvchad.com/ Error message : "packer.nvim: Error running config for telescope-frecency.nvim: ...im/site/pack/packer/ start/sqlite.lua/lua/sqlite/defs.lua:31: libsqlite3.so: cannot open shared object file : No such file or directory"
I also installed sqlite3 but the error still persisted. BTW thanks for clarification

I'm sorry I made a mistake in my previous reply,this plugin need sudo apt-get install sqlite3 libsqlite3-dev # Ubuntu,not only the sudo apt-get install sqlite3.Which you can refer his doc in Github.When I replied you I was having my breakfast and I reply with mobile Github so I didn't pay much caution to my answer.It's my fault.
If this didn't work,can you post your plugin's specification?That means the code you write in your packer.For instance,in my ubuntu nvim's config,it is the code like this:
    use {
        "nvim-telescope/telescope-frecency.nvim",
        config = function()
            require"telescope".load_extension("frecency")
        end,
        requires = {"kkharji/sqlite.lua"}
    }

Thanks very much, this fixes my problem

Now the master branch does not use SQLite3 library. See #91, #148.


prep release

Codecov Report

Merging #38 (3d86f6d) into main (3afb66d) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 3d86f6d differs from pull request most recent head 6cb8378. Consider uploading reports for the commit 6cb8378 to get more accurate results

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@           Coverage Diff           @@
##             main      #38   +/-   ##
=======================================
  Coverage   26.77%   26.77%
=======================================
  Files          42       42
  Lines        1535     1535
=======================================
  Hits          411      411
  Misses       1124     1124

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Update chainlink tables

What?
Update chainlink tables with more fields (making them identical to tables for other blockchains in https://github.com/nansen-ai/evmchain-etl-table-definitions) and pulling the current set of feeds from https://data.chain.link/
How?
Event feeds pulled from https://data.chain.link/
Event structure parsed with https://nansen-contract-parser-prod.web.app/
Generated with:
https://gist.github.com/bschreck/dcbe55848dbac19609841c5bab65e88e
Related PRs (optional)
Similar PR for mainnet ethereum: https://github.com/blockchain-etl/ethereum-etl-airflow/pull/501
Similar PR for polygon: https://github.com/blockchain-etl/polygon-etl/pull/114

@bschreck some addresses were removed, e.g. 0xfd001350eac4bed88dde16d0770d12e8e4d9508b. Just want to make sure it's intentional as those contract events will be removed from the tables

Closing this as no new feeds were actually found.


Restoring parameters  hungs

I have the same problem with blstm,
with --schedule 'train'

INFO:tensorflow:Global model_fn finished.
INFO:tensorflow:Create CheckpointSaverHook.
2017-12-29 07:38:44.128222: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-12-29 07:38:44.851032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:05:00.0
totalMemory: 11.90GiB freeMemory: 11.74GiB
2017-12-29 07:38:44.851088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:05:00.0, compute capability: 6.1)

, --schedule 'evalution'

2017-12-29 07:43:23.631197: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-12-29 07:43:24.383548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:05:00.0
totalMemory: 11.90GiB freeMemory: 11.74GiB
2017-12-29 07:43:24.383603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:05:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from d/model.ckpt-1

or '--schedule 'test''

INFO:tensorflow:Create CheckpointSaverHook.
2017-12-29 07:52:49.675929: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2017-12-29 07:52:50.477620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:05:00.0
totalMemory: 11.90GiB freeMemory: 11.74GiB
2017-12-29 07:52:50.477673: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:05:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from d/model.ckpt-1

 --schedule 'continuous_train_and_eval' seems work,

totalMemory: 11.90GiB freeMemory: 11.74GiB
2017-12-29 08:15:39.798320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: TITAN Xp, pci bus id: 0000:05:00.0, compute capability: 6.1)
INFO:tensorflow:Restoring parameters from d/model.ckpt-1
INFO:tensorflow:Saving checkpoints for 2 into d/model.ckpt.
INFO:tensorflow:loss = 5.00657, step = 2

but --schedule 'train_and_evaluate'
The problm seems like to https://github.com/tensorflow/tensor2tensor/issues/353, and i use tf.nn.bidirectional_dynamic_rnn without  'orthogonal initializer`

https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/contrib/learn/python/learn/learn_runner.py#L272
  if config.task_type == run_config_lib.TaskType.MASTER:
    # TODO(rhaertel): handle the case where there is more than one master
    # or explicitly disallow such a case.
    return 'train_and_evaluate'
  elif config.task_type == run_config_lib.TaskType.PS:
    return 'run_std_server'
  elif config.task_type == run_config_lib.TaskType.WORKER:
    return 'train'

may be 'train_and_evaluate' is used for local trainning and run_config_lib.TaskType.MASTER: train is only for WORKER.
But I think this is caused by prams.

Closing for now. If you decide to reopen or rereport, please include T2T version, command lines, and stack trace.


Example using 3p npm dependencies

Could you please add an example of using 3p npm dependencies.
For eg. something similar to https://github.com/bazelbuild/rules_nodejs/tree/stable/e2e/typescript

Thanks for the example.


Generator optimization

Now we have two kinds of iterators - lua-like (functions) and js-like (objects with next method). The idea is to make generated js-like iterators contain data to make lua iteration faster. Additionally, all iterator functions are extracted to helpers to avoid extra allocations.
Input on Playground
local ____symbolMetatable = {__tostring = function(self)
  if self.description == nil then
    return "Symbol()"
  else
    return "Symbol(" .. tostring(self.description) .. ")"
  end
end}
function __TS__Symbol(self, description)
  return setmetatable({description = description}, ____symbolMetatable)
end
Symbol = {iterator = __TS__Symbol(_G, "Symbol.iterator")}

-- New helpers:

local __TS__IterableCoroutine = {}
local __TS__IterableArguments = {}

function __TS__IterateStep(self)
  local co = self[__TS__IterableCoroutine]
  local args = self[__TS__IterableArguments]
  local status, value
  if args then
    status, value = coroutine.resume(co, unpack(args))
    self[__TS__IterableArguments] = nil
  else
    status, value = coroutine.resume(co, self)
  end

  if coroutine.status(co) == "dead" then
    return
  end

  if not status then
    error(value)
  end

  -- Fix for #600
  return true, value
end

function __TS__Iterate(result)
  if not result[__TS__IterableCoroutine] then
    -- TODO: Fallback for manually constructed iterators
  end

  return __TS__IterateStep, result
end

function __TS__GeneratorNext(self, ...)
  local co = self[__TS__IterableCoroutine]
  local args = self[__TS__IterableArguments]
  local status, value
  if args then
    status, value = coroutine.resume(co, unpack(args))
    self[__TS__IterableArguments] = nil
  else
    status, value = coroutine.resume(co, ...)
  end

  return {
    done = coroutine.status(co) == "dead",
    value = value,
  }
end

function __TS__GeneratorIterator(self)
  return self
end

function __TS__Generator(fn)
  return function (...)
    local args = {...}
    return {
      [__TS__IterableCoroutine] = coroutine.create(fn),
      [__TS__IterableArguments] = args,
      next = __TS__GeneratorNext,
      [Symbol.iterator] = __TS__GeneratorIterator,
    }
  end
end

-------------------------------------------------------------------------------

local foo = __TS__Generator(function(self, argument)
  local ____ = coroutine.yield(1)
  local ____ = coroutine.yield(argument)

  return 3
end)

local it = foo(_G, 2)
local x1 = it:next()
print(x1.value, x1.done) -- 1 false
local x2 = it:next()
print(x2.value, x2.done) -- 2 false
local x3 = it:next()
print(x3.value, x3.done) -- 3 true

for ____, value in __TS__Iterate(foo(_G, 2)) do
  print(value) -- 1, 2
end

This looks like a really good direction for these. A couple thoughts:
Handling the initial arguments could be simplified a lot by wrapping the function passed to coroutine.create:
[__TS__IterableCoroutine] = coroutine.create(function() return fn(unpack(args)) end),

Then, you won't need to track it and check for it every iteration. Of course, the trade-off is an extra piece of garbage.
When transforming for (const v of foo(2)) to for ____, value in __TS__Iterate(foo(_G, 2)), how will you know to add the initial dummy parameter? In this particular example, you could probably check that foo is a generator, but what if foo is called outside of the for...of loop?
const bar = foo(2);
for (const v of bar) {
    ...
}

Handling the initial arguments could be simplified a lot by wrapping the function passed to coroutine.create

Yeah, I have thought about that, but assumed it would be more performant that way. I probably should benchmark both solutions.

how will you know to add the initial dummy parameter

Well, when __TS__Iterate is used we always can safely assume it.

Ok, so I assume the standard iterator fallback will return this initial value as well.
Regarding performance, I suspect the answer will be: it depends. Functions that only yield a couple times like the example will benefit more from avoiding the extra lambda. But heavy functions that yield a lot will perform better without the per-iteration checks.