Cannot set key again if having removed key

I am unable to set key again if I Set Key then remove key. Then open the DB again with no key, then neither SetKey or ReKey will work again. (Key is successfully removed as DB Browser for Sqlite can open it at that time)

SetKey on a non-empty database is not supposed to succeed, but ReKey should.
I'm unable to reproduce. Are you using the latest version? Release or git master? Do you set & remove the key using DB Browser, shell or C interface? In addition, .dbinfo output of the database before setting the key first time and after removing the key could be useful.
$ rm -f db
$ ./sqleet 'file:db?key=swordfish' "CREATE TABLE f(x,y)"
$ ./sqleet 'file:db?key=swordfish' "INSERT INTO f VALUES(4,2)"
$ xxd db | head -n1
00000000: 0a98 87dd ae61 7903 e4a8 5d99 c557 0b64  .....ay...]..W.d
$ ./sqleet 'file:db?key=swordfish' "PRAGMA rekey=''" && xxd db | head -n1
00000000: 5351 4c69 7465 2066 6f72 6d61 7420 3300  SQLite format 3.
$ ./sqleet db "PRAGMA rekey='GOD'"
ok
$ xxd db | head -n1
00000000: af20 0557 5c86 297d af85 ed08 3e75 4983  . .W\.)}....>uI.
$ ./sqleet 'file:db?key=GOD' .dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE f(x,y);
INSERT INTO f VALUES(4,2);
COMMIT;

I was using latest release, and the C interface.  ReKey works fine, I only get the issue when having done ReKey that removes the key and then put key again on it. I will play a bit more with it and try not using SetKey again in between and see what happens. Thanks.

I am having a similar problem using PRAGMA statements.
I have a db that was previously encrypted. I rekeyed it to remove the key. Now when I try to rekey again to add the password its not encrypting.
I've stepped through the code and it appears that when sqlite3_rekey_v2 is calling codec_set_to the codec->pagesize is bunk. it contains a value of : -842150451
Any thoughts?

Hi @resilar thanks for this. It appears to have solved the problem we reported.

Great!

Hi Resliar
Are there any known issues with using sqleet in a multi threaded app?
I’ve got an app that has multiple threads trying to read/write to db. It appears that even when I use the KEY pragma after opening the db, some of the threads will get the not a db error.
Just wondering if it could be some sort of read/write collision between threads. They are not sharing a db handle. Each thread opens the db separately.
From: resilar<EMAIL_ADDRESS>Reply-To: resilar/sqleet<EMAIL_ADDRESS>Date: Tuesday, September 17, 2019 at 9:13 AM
To: resilar/sqleet<EMAIL_ADDRESS>Cc: Developer<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [resilar/sqleet] Cannot set key again if having removed key (#26)
Thanks for the details @rocketcyberdevhttps://github.com/rocketcyberdev
I'm still unable to reproduce the actual sqlite3_rekey_v2() error, but I succeeded to trigger the following (presumably related) valgrind error:
sqlite> PRAGMA rekey='';
sqlite> PRAGMA rekey='GOD';
==55348== Conditional jump or move depends on uninitialised value(s)
==55348==    at 0x1D97FE: codec_set_to (sqleet.c:451)
==55348==    by 0x1D9E89: sqlite3_rekey_v2 (sqleet.c:613)
==55348==    by 0x19303F: sqlite3Pragma (sqlite3.c:123854)
==55348==    by 0x1C577F: yy_reduce (sqlite3.c:152913)
==55348==    by 0x1C6B5C: sqlite3Parser (sqlite3.c:153461)
==55348==    by 0x1C7DCC: sqlite3RunParser (sqlite3.c:154597)
==55348==    by 0x194E0B: sqlite3Prepare (sqlite3.c:124831)
==55348==    by 0x195161: sqlite3LockAndPrepare (sqlite3.c:124924)
==55348==    by 0x19535A: sqlite3_prepare_v2 (sqlite3.c:125008)
==55348==    by 0x1EB488: shell_exec (shell.c:11585)
==55348==    by 0x1F952B: runOneSqlLine (shell.c:18110)
==55348==    by 0x1F9A44: process_input (shell.c:18210)
==55348==
445 static int codec_set_to(Codec *codec, Btree *pBt)
446 ...
451         if (!codec->pagesize)
452             codec->pagesize = sqlite3BtreeGetPageSize(pBt);
Indeed, codec->pagesize is uninitialized since commit ff10548https://github.com/resilar/sqleet/commit/ff10548206cd945fc22d51b683908b20b8c6e1a1 (Implement URI API). Commit 4a6f062https://github.com/resilar/sqleet/commit/4a6f0625cd11d9b615073565cb84e7b5da1b4b2c adds the missing initialization of codec->pagesize, likely fixing this rekeying issue. Closing, please reopen if the commit does not resolve this issue.
Highly appreciated if @einhugurhttps://github.com/einhugur or @rocketcyberdevhttps://github.com/rocketcyberdev could test the fix and report back.

I also updated the v0.29.0 release/tag of sqleet to include the fix given the seriousness of the bug. Now thinking about it, this was a little unprofessional because tags should not abruptly change. In future, a better protocol is needed for releasing bugfix releases without "rewriting history".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/resilar/sqleet/issues/26?email_source=notifications&email_token=AJD55TRNCPAECRK44OYSAPTQKDQWRA5CNFSM4ISEG4AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD64VJTA#issuecomment-532239564, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJD55TURMVYGBQITVXRCCNLQKDQWRANCNFSM4ISEG4AA.

Interesting. To be honest, I have not tested multi-threaded setup very much. However, there should be no problem since sqleet's sqlite3_key_v2 & sqlite3_rekey_v2 are protected by the database mutex. SQLITE_NOTADB (assuming it's not coming from sqlite3 code) is returned by verify_page1() which is called holding the database mutex in addition to the pager shared lock, so there should be no synchronization issues.
Have you tried compiling sqleet explicitly with -DSQLITE_THREADSAFE=1 or -DSQLITE_THREADSAFE=2? -DSQLITE_THREADSAFE=0 disables mutexes and makes concurrent use of SQLite3 (and sqleet) unsafe.

They are not sharing a db handle. Each thread opens the db separately.

This might be the culprit. https://www.sqlite.org/threadsafe.html says:

SQLite supports three different threading modes:

Single-thread. In this mode, all mutexes are disabled and SQLite is unsafe to use in more than a single thread at once.
Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.
Serialized. In serialized mode, SQLite can be safely used by multiple threads with no restriction.

Any idea which mode you are using?

Thanks Resilar,
Yes we have SQLITE_THREADSAFE=1 (Serialized) set at compile time.
I will go back and do some additional testing to make sure that we’re not somehow accidentally opening the db without the key pragma somewhere in the code path or that some runtime db open is overriding the threadsafe setting.
From: resilar<EMAIL_ADDRESS>Reply-To: resilar/sqleet<EMAIL_ADDRESS>Date: Thursday, September 19, 2019 at 2:42 AM
To: resilar/sqleet<EMAIL_ADDRESS>Cc: Developer<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [resilar/sqleet] Cannot set key again if having removed key (#26)
Interesting. To be honest, I have not tested multi-threaded setup very much. However, there should be no problem since sqleet's sqlite3_key_v2 & sqlite3_rekey_v2 are protected by the database mutex. SQLITE_NOTADB (assuming it's not coming from sqlite3 code) is returned by verify_page1() which is called holding the database mutex in addition to the pager shared lock, so there should be no synchronization issues.
Have you tried compiling sqleet explicitly with -DSQLITE_THREADSAFE=1 or -DSQLITE_THREADSAFE=2? -DSQLITE_THREADSAFE=0 disables mutexes and makes concurrent use of SQLite3 (and sqleet) unsafe.
They are not sharing a db handle. Each thread opens the db separately.
This might be the culprit. https://www.sqlite.org/threadsafe.html says:

SQLite supports three different threading modes:

Single-thread. In this mode, all mutexes are disabled and SQLite is unsafe to use in more than a single thread at once.
Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads.
Serialized. In serialized mode, SQLite can be safely used by multiple threads with no restriction.

Any idea which mode you are using?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/resilar/sqleet/issues/26?email_source=notifications&email_token=AJD55TX4AIVZ2Y32JP2ORNDQKMUPBA5CNFSM4ISEG4AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7CRIIY#issuecomment-533009443, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJD55TSUHRYEIFWJHZ33J3LQKMUPBANCNFSM4ISEG4AA.

Now thinking about it, a shared pager lock in verify_page1() might not be enough because the function clears the page cache to force reading the first page from the disk.   sqlite3PagerExclusiveLock(pager) is maybe worth trying.
I'll look into this and write some concurrency tests when I have time.

Sounds good thanks.
Sent from my iPhone
On Sep 20, 2019, at 2:16 AM, resilar <notifications<EMAIL_ADDRESS>wrote:
Now thinking about it, a shared pager lock in verify_page1() might not be enough because the function clears the page cache to force reading the first page from the disk. sqlite3PagerExclusiveLock(pager) is maybe worth trying.
I'll look into this and write some concurrency tests when I have time.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/resilar/sqleet/issues/26?email_source=notifications&email_token=AJD55TTB5PCSVDO3SBUMII3QKR2FJA5CNFSM4ISEG4AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7F2ETI#issuecomment-533439053, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJD55TXCVZC2AY4RQYLHKMTQKR2FJANCNFSM4ISEG4AA.

So far I have written a multi-threaded concurrency smoke test program that continuously (re)opens and reads an encrypted sqleet database from dozens of threads. This has not reproduced the SQLITE_NOTADB error, contrary to my expectations, assuming a concurrency bug in verify_page1(). I'll add writer threads probably later this week to see if concurrent writes triggers the error.

@rocketcyberdev Finally some progress! Adding concurrent writers did the trick and caused sqlite3_key() (as well as sqlite3_open() with key embedded in URI)  to fail in my test setup - hopefully for the same reason you are experiencing in your app.
Concurrent writes to a non-WAL-journaled SQLite3 database can block readers in addition to other writers. As a result, opening an encrypted sqleet database can fail because the password/key is verified in verify_page1() by reading the first page of the database. Now, if a writer blocks the read, then sqlite3PagerSharedLock() on sqleet.c:371 fails with error code SQLITE_BUSY. The error is unhandled by sqleet, causing sqlite3PagerPagecount() to return 0 as the database page count. As a result, sqleet thinks the database is empty and generates a new random salt that makes the key derivation function to return wrong encryption key. This ultimately explains the SQLITE_NOTADB error.
Whew, sorry for the crazy amount of details. The take home message is that sqleet ignores SQLITE_BUSY error in verify_page1() which should be returned to the caller or handled appropriately. I still need to do further research to determine the expected sqleet behavior in this situation, but my current understanding is that SQLite3 generally requires the user to take care of SQLITE_BUSY and SQLITE_LOCKEDerrors themselves by implementing retrying logic, or using the built-in SQLite3 busy_handler & busy_timeout mechanisms to do that.
There are quite a few approaches to choose from:

Return SQLITE_BUSY directly to the user.
Implement retrying logic (obeying busy_timeout and busy_handler?) and return SQLITE_BUSY if retries fail.
Do not call verify_page1() when an encryption key is set in sqlite3_open() or sqlite3_key(). I do not like this approach because a wrong key is then not immediately detected (a database read or write is required to see if that fails due to an invalid encryption key). However, I kinda like the idea of optionally skipping the verification, because it could be useful in some (multi-threaded) use-cases where the key is known to be correct beforehand; thus, allowing skipping the heavy key derivation process.
Something else that I cannot think of at the moment.

Anyway, give me a week or two to address this issue in sqleet. The issue is not critical since it only causes sqlite3_open() or sqlite3_key() to fail in a multi-threaded  setup with a low probability. Retrying the function call can be used as a temporary workaround until then.

Ping @utelle just in case you are interested or have some ideas. I have not yet checked if wxSQLite3 handles SQLITE_BUSY errors, but if not, you might want to do so if my analysis is correct. I can release my pthreads-based concurrency test program if you are unsure about the multi-threaded usage of wxSQLite3

Awesome. Thanks for all the details and your efforts. We do use the busy_timeout to have sql retry automatically when a busy error occurs. Right now I think we set at 1000ms. So option 2 sounds like it would work out of the box for us.

From: resilar<EMAIL_ADDRESS>Sent: Tuesday, October 1, 2019 8:51:47 AM
To: resilar/sqleet<EMAIL_ADDRESS>Cc: Developer<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [resilar/sqleet] Cannot set key again if having removed key (#26)
@rocketcyberdevhttps://github.com/rocketcyberdev Finally some progress! Adding concurrent writers did the trick and caused sqlite3_key() (as well as sqlite3_open() with key embedded in URI) to fail in my test setup - hopefully for the same reason you are experiencing in your app.
Concurrent writes to a non-WAL-journaled SQLite3 database can block readers in addition to other writers. As a result, opening an encrypted sqleet database can fail because the password/key is verified in verify_page1() by reading the first page of the database. Now, if a writer blocks the read, then sqlite3PagerSharedLock() on sqleet.c:371https://github.com/resilar/sqleet/blob/06e922a71dda1ab7ee5c16c8f9e0fa2924b433ac/sqleet.c#L371 fails with error code SQLITE_BUSY. The error is unhandled by sqleet, causing sqlite3PagerPagecount() to return 0 as the database page count. As a result, sqleet thinks the database is empty and generates a new random salt that makes the key derivation function to return wrong encryption key. This ultimately explains the SQLITE_NOTADB error.
Whew, sorry for the crazy amount of details. The take home message is that sqleet ignores SQLITE_BUSY error in verify_page1() which should be returned to the caller or handled appropriately. I still need to do further research to determine the expected sqleet behavior in this situation, but my current understanding is that SQLite3 generally requires the user to take care of SQLITE_BUSY and SQLITE_LOCKEDerrors themselves by implementing retrying logic, or using the built-in SQLite3 busy_handlerhttps://www.sqlite.org/c3ref/busy_handler.html & busy_timeouthttps://www.sqlite.org/c3ref/busy_timeout.html mechanisms to do that.
There are quite a few approaches to choose from:

Return SQLITE_BUSY directly to the user.
Implement retrying logic (obeying busy_timeout and busy_handler?) and return SQLITE_BUSY if retries fail.
Do not call verify_page1() when an encryption key is set in sqlite3_open() or sqlite3_key(). I do not like this approach because a wrong key is then not immediately detected (a database read or write is required to see if that fails due to an invalid encryption key). However, I kinda like the idea of optionally skipping the verification, because it could be useful in some (multi-threaded) use-cases where the key is known to be correct beforehand; thus, allowing skipping the heavy key derivation process.
Something else that I cannot think of at the moment.

Anyway, give me a week or two to address this issue in sqleet. The issue is not critical since it only causes sqlite3_open() or sqlite3_key() to fail in a multi-threaded setup with a low probability. Retrying the function call can be used as a temporary workaround until then.

Ping @utellehttps://github.com/utelle just in case you are interested or have some ideas. I have not yet checked if wxSQLite3 handles SQLITE_BUSY errors, but if not, you might want to do so if my analysis is correct. I can release my pthreads-based concurrency test program if you are unsure about the multi-threaded usage of wxSQLite3
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/resilar/sqleet/issues/26?email_source=notifications&email_token=AJD55TWHDYLRSNORUHXALC3QMNIXHA5CNFSM4ISEG4AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEABKZCQ#issuecomment-537046154, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJD55TTHQ4MRCB4JVVSPIWTQMNIXHANCNFSM4ISEG4AA.

Ping utelle just in case you are interested or have some ideas. I have not yet checked if wxSQLite3 handles SQLITE_BUSY errors, but if not, you might want to do so if my analysis is correct. I can release my pthreads-based concurrency test program if you are unsure about the multi-threaded usage of wxSQLite3

This is definitely an interesting issue. At the moment I don't know yet if and how wxSQLite3 is affected by this problem. However, wxSQLite3 does not perform a check like verify_page1 as sqleet does. So maybe wxSQLite3 is not affected, but to be sure I would certainly like to get your concurrency test program. TIA.

@rocketcyberdev Preliminary fix 7dc8126 merged to master. sqleet now handles SQLITE_BUSY appropriately by invoking the handler set by sqlite3_busy_handler() or sqlite3_busy_timeout(). Highly appreciated if you could test the fix and tell whether it solves your issue. If not, try also disabling SQLite3 shared-cache mode (disabled by default).

Indeed, wxSQLite3 does not seem to have verify_page1()-style verification of the encryption key when setting a codec. This is probably common behavior as SQLCipher neither automatically verifies the key according to documentation SQLCipher API: Testing the key. Issue #30 tracks optional key verification skipping in sqleet.
In my opinion, immediate feedback from sqlite3_key_v2() about the correctness of the passed-in key is worth the struggle to read the 1st database page for key verification. However, reading the page 1 from the database file on disk has turned out to be a surprisingly difficult task because of B-trees, page caches and other layers of abstractions on top of the disk storage. I just noticed that the current version of sqleet still does not get the synchronization right with shared-cache mode enabled (issue #31).
Private gist of my test program. Unfinished and ugly, but better than nothing (polished final version will hopefully be merged into master in the future). In addition, SQLite3 source's test/threadtest[1-4].c are also useful! The official thread tests, however, do not test encryption codecs at all. Basic encryption functionality (such as key() and rekey()) can be somewhat tested with small code patches, though.

Thanks will give it a try and report back.

From: resilar<EMAIL_ADDRESS>Sent: Thursday, October 3, 2019 5:50:33 PM
To: resilar/sqleet<EMAIL_ADDRESS>Cc: Developer<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [resilar/sqleet] Cannot set key again if having removed key (#26)
@rocketcyberdevhttps://github.com/rocketcyberdev Preliminary fix 7dc8126https://github.com/resilar/sqleet/commit/7dc8126f58156bf411166b695fb68502b2676475 merged to master. sqleet now handles SQLITE_BUSY appropriately by invoking the handler set by sqlite3_busy_handler() or sqlite3_busy_timeout(). Highly appreciated if you could test the fix and tell whether it solves your issue. If not, try also disabling SQLite3 shared-cache modehttps://www.sqlite.org/sharedcache.html (disabled by default).

Indeed, wxSQLite3 does not seem to have verify_page1()-style verification of the encryption key when setting a codec. This is probably common behavior as SQLCipher neither automatically verifies the key according to documentation SQLCipher API: Testing the keyhttps://www.zetetic.net/sqlcipher/sqlcipher-api/#testing-the-key. Issue #30https://github.com/resilar/sqleet/issues/30 tracks optional key verification skipping in sqleet.
In my opinion, immediate feedback from sqlite3_key_v2() about the correctness of the passed-in key is worth the struggle to read the 1st database page for key verification. However, reading the page 1 from the database file on disk has turned out to be a surprisingly difficult task because of B-trees, page caches and other layers of abstractions on top of the disk storage. I just noticed that the current version of sqleet still does not get the synchronization right with shared-cache modehttps://www.sqlite.org/sharedcache.html enabled (issue #31https://github.com/resilar/sqleet/issues/31).
Private gist of my test programhttps://gist.github.com/resilar/20cbe17b6a7a0a4c4eece6dd823d5180. Unfinished and ugly, but better than nothing (polished final version will hopefully be merged into master in the future). In addition, SQLite3 source's test/threadtest[1-4].c are also useful! The official thread tests, however, do not test encryption codecs at all. Basic encryption functionality (such as key() and rekey()) can be somewhat tested with small code patches, though.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/resilar/sqleet/issues/26?email_source=notifications&email_token=AJD55TRYWX7NS2C6PYDMXF3QMZZLTA5CNFSM4ISEG4AKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAJ2OZI#issuecomment-538158949, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJD55TQGPKPT2CJ6MO3OX4LQMZZLTANCNFSM4ISEG4AA.

Thanks for providing your thread test code.

Indeed, wxSQLite3 does not seem to have verify_page1()-style verification of the encryption key when setting a codec. This is probably common behavior as SQLCipher neither automatically verifies the key according to [the SQLCipher documentation].

It is true that neither wxSQLite3 nor SQLCipher (nor any other encryption extension I tested,  except sqleet) perform an immediate verification of the encryption key.
However, this follows the general behaviour of SQLite itself, as for example SQLite itself does not check the validity of a database file until an SQL command is executed that actually requires to read from (or write to) the database file. That is, a call to sqlite3_open will usually succeed, even if you try to open a non-database file.
One could argue that on opening an existing file SQLite should check immediately whether the file is really a valid database file, but the SQLite developers chose a different approach. And for the time being wxSQLite3 will stick to this "official" approach.

Very good points! I'll consider making the immediate codec verification optional (instead of the other way around), especially if reading the page 1 becomes too much hassle with shared cache mode support. Breaking sqlite3_open() behavior does not feel the right thing to do.
I do not really know if multi-thread shared-cache with encryption is reasonable (or secure) in the first place. SQLCipher segfaults in threadtest3.c tests which utilize shared cache (even if shared cache mode is disabled), while sqleet fails the same tests only with shared cache enabled. In threadtest3.c, I have patched opendb_x() to perform keying (and testing the key if using SQLCipher).
It looks like the page 1 verification of sqleet might actually not be the cause of the failing shared-cache tests (performing the verification later as in SQLCipher/wxSQLite3 does not make the tests pass). More research is needed...

I'll consider making the immediate codec verification optional (instead of the other way around) ...

That is certainly a good approach. For sure there are scenarios where verifying the codec immediately is preferable, and your approach allows to do that without forcing the "typical" sqleet user to modify build options.

I do not really know if multi-thread shared-cache with encryption is reasonable (or secure) in the first place.

I have to admit that I haven't spent much time yet on analyzing this kind of use case. However, I don't see a largely increased security risk in using shared-cache, since it is restricted to threads belonging to a single process.

SQLCipher segfaults in threadtest3.c tests which utilize shared cache (even if shared cache mode is disabled), while sqleet fails the same tests only with shared cache enabled. [...]
It looks like the page 1 verification of sqleet might actually not be the cause of the failing shared-cache tests (performing the verification later as in SQLCipher/wxSQLite3 does not make the tests pass). More research is needed...

I will try to dedicate some time to further analyze this shared cache issue.

I will try to dedicate some time to further analyze this shared cache issue.

Highly appreciated if you could take a look! To quickly reproduce, use my patched version of opendb_x() below to run threadtest3.c test cases with encrypted databases. I also highly recommend compiling with -DSQLITE_DEBUG to enable SQLite3 assert()s for detecting subtle bugs/errors that do not cause crash or test case failure.
// threadtest3.c:519
static void opendb_x(
  Error *pErr,                    /* IN/OUT: Error code */
  Sqlite *pDb,                    /* OUT: Database handle */
  const char *zFile,              /* Database file name */
  int bDelete                     /* True to delete db file before opening */
){
  if (pErr->rc == SQLITE_OK) {
    int rc;
    int flags = SQLITE_OPEN_CREATE | SQLITE_OPEN_READWRITE | SQLITE_OPEN_URI
#if 1
            | SQLITE_OPEN_SHAREDCACHE
#else
            | SQLITE_OPEN_PRIVATECACHE
#endif
            ;
    if (bDelete) unlink(zFile);
    if ((rc = sqlite3_open_v2(zFile, &pDb->db, flags, 0)) != SQLITE_OK) {
      sqlite_error(pErr, pDb, "open");
      goto fail;
    }

    sqlite3_busy_handler(pDb->db, busyhandler, 0);

#if defined(SQLITE_HAS_CODEC)
    if ((rc = sqlite3_key(pDb->db, "test",  4)) != SQLITE_OK) {
      printf("[ERROR] sqlite3_key('%s', 'test'): %s (%s) %d\n",
             zFile, sqlite3_errmsg(pDb->db), sqlite3_errstr(rc), rc);
      sqlite_error(pErr, pDb, "key");
      goto fail;
    }

    // Validate key (SQLCipher/wxSQLite3)
    rc = sqlite3_exec(pDb->db, "SELECT count(*) FROM sqlite_master;",
                      NULL, NULL, NULL);
    if (rc != SQLITE_OK) {
      printf("[ERROR] sqlite3_verify('%s', 'test'): %s (%s) %d\n",
             zFile, sqlite3_errmsg(pDb->db), sqlite3_errstr(rc), rc);
      sqlite_error(pErr, pDb, "verify");
      goto fail;
    }
#endif

    sqlite3_create_function(
        pDb->db, "md5sum", -1, SQLITE_UTF8, 0, 0, md5step, md5finalize
    );
    sqlite3_exec(pDb->db, "PRAGMA synchronous=OFF", 0, 0, 0);
  }
  return;

fail:
  sqlite3_close(pDb->db);
  pDb->db = 0;
  return;
}

In the meantime I used your variant of threadtest3.c to perform some tests.
If option SQLITE_OPEN_PRIVATECACHE is used threadtest3 runs without crashing producing the following test output. However, 2 of 14 tests fail.
Running walthread1 for 20 seconds...
Thread 11 says: 36 checkpoints
Thread 10 says: 95 iterations
Thread 9 says: 192 iterations
Thread 8 says: 167 iterations
Thread 7 says: 125 iterations
Thread 6 says: 107 iterations
Thread 5 says: 112 iterations
Thread 4 says: 197 iterations
Thread 3 says: 124 iterations
Thread 2 says: 104 iterations
Thread 1 says: 197 iterations
Running walthread2 for 20 seconds...
Thread 4 says: W 22 R 52
Thread 3 says: W 16 R 55
Thread 2 says: W 71 R 1
Thread 1 says: W 70 R 1
Running walthread3 for 20 seconds...
Thread 6 says: ...
Thread 5 says: ...
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running walthread4 for 20 seconds...
Thread 2 says: ...
Thread 1 says: ...
Running walthread5 for 1 seconds...
Error: (22) "Invalid argument" at line 1281
Running cgt_pager_1 for 0 seconds...
Running dynamic_triggers for 20 seconds...

 Checkpoint mode  : PASSIVE
 Peak WAL         : 73302 frames
 Transaction count: 22256 transactions
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running checkpoint_starvation_2 for 10 seconds...
 Checkpoint mode  : RESTART
 Peak WAL         : 52 frames
 Transaction count: 590 transactions
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running create_drop_index_1 for 10 seconds...
Warning: (17) "database schema has changed" at line 39
Thread 5 says: ok
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running lookaside1 for 10 seconds...
Thread 6 says: ok
Thread 5 says: ok
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running vacuum1 for 10 seconds...
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running stress1 for 10 seconds...
Warning: (17) "no such table: t1" at line 64
Warning: (17) "sqlite3_prepare_v2() - no such table: t1 (17)" at line 123
Thread 10 says: deleted from t1 1/1 attempts
Warning: (17) "no such table: t1" at line 64
Warning: (17) "no such table: t1" at line 94
Warning: (1) "sqlite3_reset() - no such table: t1 (1)" at line 123
Thread 9 says: deleted from t1 1/1 attempts
Warning: (17) "no such table: t1" at line 94
Thread 8 says: wrote t1 8/8 attempts
Thread 7 says: wrote t1 47/47 attempts
Thread 6 says: read t1 3686/3687 attempts
Thread 5 says: read t1 2089/2089 attempts
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running stress2 for 60 seconds...
Error: (5) "database is locked" at line 260
Thread 15 says: ok
Thread 14 says: ok
Thread 13 says: ok 5/6
Thread 12 says: ok 320/320
Thread 11 says: ok 250/250
Thread 10 says: ok 300/300
Thread 9 says: ok 285/285
Thread 8 says: ok 280/280
Thread 7 says: ok 295/295
Thread 6 says: ok 275/275
Thread 5 says: ok 285/285
Thread 4 says: ok 345/345
Thread 3 says: ok 340/340
Thread 2 says: ok 305/305
Thread 1 says: ok 330/330
2 errors out of 14 tests

If option SQLITE_OPEN_SHAREDCACHE is used threadtest3 outputs
Running walthread1 for 20 seconds...
Error: (6) "database table is locked: t1" at line 1011

and then crashes due to an access violation on trying to encrypt page 2 of the main database. Following the call stack:
sqlite3BtreeGetPageSize(Btree * p) Line 66277
sqlite3Codec(void * pCodecArg, void * data, unsigned int nPageNum, int nMode) Line 84
sqlite3PagerCodec(PgHdr * pPg) Line 57856
walWriteOneFrame(WalWriter * p, PgHdr * pPage, int nTruncate, __int64 iOffset) Line 61761
sqlite3WalFrames(Wal * pWal, int szPage, PgHdr * pList, unsigned int nTruncate, int isCommit, int sync_flags) Line 61962
pagerWalFrames(Pager * pPager, PgHdr * pList, unsigned int nTruncate, int isCommit) Line 54013
sqlite3PagerCommitPhaseOne(Pager * pPager, const char * zMaster, int noSync) Line 57208
sqlite3BtreeCommitPhaseOne(Btree * p, const char * zMaster) Line 67386
vdbeCommit(sqlite3 * db, Vdbe * p) Line 79092
sqlite3VdbeHalt(Vdbe * p) Line 79493
sqlite3VdbeExec(Vdbe * p) Line 87312
sqlite3Step(Vdbe * p) Line 82296
sqlite3_step(sqlite3_stmt * pStmt) Line 82362
sqlite3_exec(sqlite3 * db, const char * zSql, int(*)(void *, int, char * *, char * *) xCallback, void * pArg, char * * pzErrMsg) Line 119923
sql_script_x(Error * pErr, Sqlite * pDb, const char * zSql) Line 631
walthread1_thread(int iTid, void * pArg) Line 1005
launch_thread_main(void * pArg) Line 811

The cause of the problem is that the codec structure holds a pointer to a SQLite3 database structure and a pointer to a SQLite3 btree structure. Both structures do not contain valid data, but seem to be uninitialized.
My guess is that special measures would be required to determine the database and btree associated with a codec structure in shared cache mode. However, at the moment I have no idea which steps exactly would be required to accomplish proper operation in shared cache mode.

Thanks for your input. I have not yet taken a deep dive into the shared cache concurrency stuff, so please take my misguided thoughts (in this and earlier comments) as a grain of salt.
Feel free goto TLDR. The verbosity level of this post is over 9000. Sorry, personal(ity) issues.

I just noticed that the official SQLite3 source code does not pass threadtest3 with the opendb_x() patch I gave above when enabling SQLITE_OPEN_SHAREDCACHE for all databases.
Therefore, my opendb_x() patch is not a reasonable reference point with SQLITE_OPEN_SHAREDCACHE unconditionally enabled. My bad, I thought SQLITE_OPEN_SHAREDCACHE would merely highlight implementation errors of encryption extensions in shared cache mode. This is not true.

The primary focus should be on threadtest3 tests run with neither SQLITE_OPEN_SHAREDCACHE nor SQLITE_OPEN_PRIVATECACHE set in opendb_x() to match the original test code. This way most test cases use private cache, while shared cache mode is explicitly enabled by some test cases such as dynamic_triggers, stress1, stress2 and maybe others.
With neither cache flag force-enabled by opendb_x(), the official SQLite3 code without encryption passes all test cases with 0 errors (and usually at most a couple "no such table: t1" warnings from stress1). In comparison, sqleet passes tests walthread[1-5] and checkpoint_starvation_[1-2] succesfully with no errors, but fails all subsequent test cases with multiple SQLITE_NOATDB errors per each test (in addition to a few SQLITE_AUTH and SQLITE_NOMEM errors, supposedly caused by the same root issue). sqleet finishes threadtest3 with the final result  "36 errors out of 14 tests". In private cache mode, all errors vanish implying sqleet handles shared pager cache improperly.

If option SQLITE_OPEN_PRIVATECACHE is used threadtest3 runs without crashing producing the following test output. However, 2 of 14 tests fail.

In my test environment, sqleet as well as SQLite3 (without codecs) finishes threadtest3 with "1 errors out of 14 tests" when using private cache. The only error being "database is locked" (SQLITE_LOCKED) in the beginning of stress2 test case. Additionally, sqleet gets "database schema has changed" warning right after the database locked error (no idea if this is meaningful or not).
The SQLITE_LOCKED error I'm experiencing appears to be a side effect of SQLITE_OPEN_PRIVATECACHE force-enabled by the patched version of opendb_x(). Disabling private cache by either explicitly setting SQLITE_OPEN_SHAREDCACHE or removingSQLITE_OPEN_PRIVATECACHE (i.e., both cache flags unset) makes sqleet and SQLite3 pass stress2 with 0 errors and 0 warnings.
@utelle Nevertheless, I'd suggest investigating further that walthread5 "invalid argument" error because it is observed only with wxSQLite3. However, the sample size here is only 2 so the error may very well be caused by differences in our test setups rather than an actual bug in wxSQLite3 code.

If option SQLITE_OPEN_SHAREDCACHE is used threadtest3 outputs ... and then crashes due to an access violation on trying to encrypt page 2 of the main database

The official SQLite3 code (without encryption) also frequently crashes during the walthread1 test case in shared cache mode. However, just to be specific, the exact time of the crash varies a lot, and about every fifth test run passes walthread1 without crashing and, after that, finishes the rest of the test cases with a total of "18 errors out of 14 tests"  (10 walthread1 errors, 3 walthread2 errors, 1 walthread3 errors, 1 walthread4 error, 2 checkpoint_starvation_1 errors,  1 checkpoint_starvation_2 error, 1 stress1 warning, and no errors from the remaining test cases). Out of these 18 errors, 17 are locked database errors (SQLITE_LOCKED), and the only different one is  "WAL failed to grow - 0 frames" in checkpoint_starvation_1 immediately after (and probably caused by) a database locking error.
In comparison, sqleet in shared cache mode always crashes during walthread1 due to a NULL-pointer dereference in threadtest3.c when comparing SQL execution results. The result strings, in this case, are NULL pointers apparently because of SQLITE_NOTADB error(s) encountered while keying the database. I'm unsure about whether the SQLITE_NOTADB originates from sqleet or SQLite3 internals, but the root cause is most likely improper handling of shared cache mode in sqleet.

TLDR:

My opendb_x() patch sucks, remove the SQLITE_OPEN_(SHARED|PRIVATE)CACHE flags to match the original threadtest3.c code

SQLite3 without encryption finishes with 18 errors when shared caching is forcibly enabled in the patched opendb_x() function.

Error "invalid argument" (walthread5) is unique to wxSQLite3 and not observed with sqleet or SQLite3 (@utelle). Otherwise sqleet and wxSQLite3 show similar threadtest3 results.
We can probably all agree that encryption extensions should ideally be transparent, that is, threadtest3 should produce identical results with and without encryption codec. Status:

Private cache mode is OKayish: Only 1 to 2 additional errors observed with sqleet or wxSQLite3 encryption enabled. These are possibly explained other factors than buggy encryption codec, for example, the threadtest3 patch for testing encryption extensions may suck. More research needed.
Shared cache mode is completely broken (for unknown reason). Requies further investigation.

It must be noted that SQLCipher does not pass the full set of threadtest3 tests without crashing regardless of caching mode used.

Sorry for the wall of text. Writing helps me to think about complex topics, which sometimes results in insane posts like this. I'll try to keep the verbosity level within sane limits in the future.

I will have to digest all the information you gave. However, I really appreciate that you share your thoughts in full. 😄
Therefore, just a quick comments about the current status of my tests. More to come later.

Shared cache mode and encryption
Item 16 in the SQLite change log for release of SQLite3 version 3.7.15 indicates that using shared cache and encryption together should work. And this means that, if wxSQLite3, sqleet, and SQLCipher don't work properly in shared cache mode, then they are flawed in this aspect. And we should try to fix this for wxSQLite3 and sqleet.

Private vs shared cache mode

Private cache mode: AFAICT at the moment the threadtest3 seems to work properly for wxSQLite3 and sqleet most of the time, but a few errors pop up which might be related to other issues which require further investigation.
Shared cache mode: yes, currently completely broken. According to my current findings a main reason seems to be, how the Btree structure is referenced from the codec. The Btree structure is not part of the shared cache; only the subcomponent BtShared is. I modified the wxSQLite3 code to reference BtShared instead of Btree. After this modification threadtest3 at least doesn't crash anymore - however, many more "locked" errors pop up, and sometimes execution stops without displaying the full set of expected messages.

Error "Invalid argument" (walthread5) in wxSQLite3
I have not yet found out, what causes this error, unfortunately.

For reliable results we will need a test application that works at least for standard SQLite3 without errors. Otherwise it will be very hard to verify which issues are caused by the encryption extension and which not.

For reliable results we will need a test application that works at least for standard SQLite3 without errors.

Standard SQLite3 without encryption passes threadtest3 with 0 errors when using unmodified opendb_x() or my patched version with the added cache open flags removed (test cases designed for shared cache mode enable it manually at runtime). In this default setting, sqleet finishes threadtest3 with a dozen shared-cache related errors. Thus, I have something to compare against.
Remove the setting of SQLITE_OPEN_SHAREDCACHE and SQLITE_OPEN_PRIVATECACHE flags from the patched opendb_x() and check how many threadtest3 errors wxSQLite3 gets. If none, then shared cache support of wxSQLite3 is at least acceptable.

With my modified wxSQLite3 version (referencing BtShared structure instead of Btree structure) I get the following results with your patched version of threadtest3 (with cache open flags removed):
Running walthread1 for 20 seconds...
Thread 11 says: 37 checkpoints
Thread 10 says: 115 iterations
Thread 9 says: 109 iterations
Thread 8 says: 108 iterations
Thread 7 says: 118 iterations
Thread 6 says: 93 iterations
Thread 5 says: 130 iterations
Thread 4 says: 98 iterations
Thread 3 says: 106 iterations
Thread 2 says: 67 iterations
Thread 1 says: 136 iterations
Running walthread2 for 20 seconds...
Thread 4 says: W 14 R 52
Thread 3 says: W 12 R 54
Thread 2 says: W 65 R 1
Thread 1 says: W 64 R 0
Running walthread3 for 20 seconds...
Thread 6 says: ...
Thread 5 says: ...
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running walthread4 for 20 seconds...
Thread 2 says: ...
Thread 1 says: ...
Running walthread5 for 1 seconds...
Error: (22) "Invalid argument" at line 1286
Running cgt_pager_1 for 0 seconds...
Running dynamic_triggers for 20 seconds...

 Checkpoint mode  : PASSIVE
 Peak WAL         : 60792 frames
 Transaction count: 18508 transactions
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running checkpoint_starvation_2 for 10 seconds...
 Checkpoint mode  : RESTART
 Peak WAL         : 52 frames
 Transaction count: 781 transactions
Thread 4 says: ...
Thread 3 says: ...
Thread 2 says: ...
Thread 1 says: ...
Running create_drop_index_1 for 10 seconds...
Thread 5 says: ok
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running lookaside1 for 10 seconds...
Thread 6 says: ok
Thread 5 says: ok
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running vacuum1 for 10 seconds...
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running stress1 for 10 seconds...
Warning: (1) "sqlite3_reset() - no such table: t1 (1)" at line 123
Warning: (1) "sqlite3_prepare_v2() - no such table: t1 (1)" at line 123
Thread 10 says: deleted from t1 1/1 attempts
Thread 9 says: deleted from t1 11/18 attempts
Thread 8 says: wrote t1 4335/4855 attempts
Thread 7 says: wrote t1 4361/4844 attempts
Thread 6 says: read t1 2597/8122 attempts
Thread 5 says: read t1 6546/10575 attempts
Thread 4 says: ok
Thread 3 says: ok
Thread 2 says: ok
Thread 1 says: ok
Running stress2 for 60 seconds...
[ERROR] sqlite3_verify('test.db', 'test'): database schema is locked: main (database table is locked) 6
Thread 15 says: ok
Thread 14 says: ok
Thread 13 says: ok 228/270
Thread 12 says: ok 265/265
Thread 11 says: ok 155/245
Thread 10 says: ok 189/275
Thread 9 says: ok 225/295
Thread 8 says: ok 183/280
Thread 7 says: ok 200/290
Thread 6 says: ok 195/275
Thread 5 says: ok 227/285
Thread 4 says: ok 280/280
Thread 3 says: ok 280/280
Thread 2 says: ok 205/280
Thread 1 says: ok 226/290
1 errors out of 14 tests

That is, I got 1 error (and 2 warnings) from walthread5 (Error: (22) "Invalid argument" at line 1286) - I still have to find out, what's causing the error "Invalid argument". Additionally, I got 1 error from stress2 ([ERROR] sqlite3_verify('test.db', 'test'): database schema is locked: main (database table is locked) 6) - however, this is no error from the original version of threadtest3 ... and it does not occur every time I run the test.

Regarding the error from walthread5 [Error: (22) "Invalid argument" at line 1286] I'm at a loss. The error is produced in function filecopy_x of threadtest3 - in the line if( nCopy!= read(fd1, aBuf, nCopy) ){ invoked from filecopy(&err, "test.db", "test_sv.db");.
That is, the file test.db should simply be copied to file test_sv.db. File test.db exists (and had a size of 1024 bytes in one of my tests). However, the function read returned 56 as the number of bytes read. Since 56 != 1024 function system_error is invoked and returns error code 22 (= ERROR_BAD_COMMAND - The device does not recognize the command.).
I don't see where wxSQLite3 would be to blame as these are C library functions producing the error.

After inspecting the database file at the position where the read stopped, I found out that the next byte in the file was a 0x1A (= ^Z) - and that character is a "end of file" marker for text files.
I modified the code of threadtest3, so that it uses the function _open instead of open, and I added the flag for binary files. Thereafter the error vanished.
Note: my test environment is Windows 10 with Visual Studio.

Okay, that explains the invalid argument error. Good work! Nasty bug though.

I get the following results with your patched version of threadtest3 (with cache open flags removed) ... That is, I got 1 error (and 2 warnings) [out of 14 tests]

That is an exceptionally good result in comparison to the "36 errors out of 14 tests" result by sqleet. However, I managed to reduce the number of errors from 36 down to 0 by skipping the immediate verify_page1() verification using raw keys that do not need the database salt for key derivation. Clearly reading the database page 1 to initialize the codec with the salt & KDF is the buggy part in sqleet when using shared cache mode concurrently from multiple threads.
I'm starting to feel much more hopeful now that the root cause of shared-cache issues has been identified somewhat accurately. :) Well, at least in sqleet. wxSQLite3 also seems to be getting very close to a stable concurrent shared-cache mode with encryption (I do not know if the few remaining errors/warnings are just false positivies or non-issues in practice).

Additionally, I got 1 error from stress2 ([ERROR] sqlite3_verify('test.db', 'test'): database schema is locked: main (database table is locked) 6) - however, this is no error from the original version of threadtest3 ... and it does not occur every time I run the test.

IIRC I have seen this error also with the standard SQLite3 (no encryption). However, I cannot reproduce it now successfully, so I might be recalling incorrectly. Better to investigate the error thoroughly, just in case. I'll look into it if the error starts occurring in sqleet after fixing the more severe bugs in the database locking & access.

In the meantime I somewhat cleaned up my modifications of the wxSQLite3 encryption extension and committed the code to the wxSQLite3 GitHub repository. This code seems to work now sufficiently well in shared cache mode - no errors, only warnings in stress1 test of threadtest3.
Additionally, I adjusted the test applications threadtest2 and threadtest4 to use encrypted databases. Both run without errors for wxSQLite3. I have not yet tested sqleet with those new tests.

Congratulations!
Late reply because I had to suddenly drop this project for a couple months due to being busy with other things in life. However, today I further investigated sqleet's 33 threadtest3 errors and finally got the perfect score "0 errors out of 14 tests" with threadtest3 patched to use encryption by opening URI file:test.db?key=swordfish instead of the filename test.db. The core problem was presumably somewhere in verify_page1() because all 33 errors disappeared after bypassing the pager and page cache by re-implementing the key verification using low-level sqlite3Os*() file & lock functions.
I was unable to determine the exact problem accurately, but sqlite3PcacheClear() seemed to produce (indirect) crashes and memory errors in other threads. Moreover, I have not yet tried to force-enable shared-cache or private-cache in the whole threadtest3 test suite. Such test configuration may be unreasonable or even impossible to pass (with official SQLite3 and no encryption), but the test results could be interesting nonetheless.
Anyway, sqleet with robust (shared-cache) concurrency support is still coming. The current "fixed" development version needs refactoring and preferably more research to find better solutions than using sqlite3Os*() low-level functions to verify the key during opening the database. Obviously, there is also the option of verifying the key "lazily" when the database is accessed (read from/written to) the first time. AFAIK wxSQLite3 and SQLCipher works like this (although I'm pretty sure that the latter does not pass threadtest3 without crashing and dozens of errors).
Issue #30 tracks the implementation of optional "lazy key verification" feature. It might become the default if key verification immediately in sqlite3_open() (or equivalent) turns out to be too difficult.


pip installation

please make btchip available throug pip, or even better an apt-get ppa.

BTW, the deb package is in debian testing
https://tracker.debian.org/pkg/btchip-python


Feature request: Add haptic support for all buttons

Hi,
I found how to add haptic feedback for the non-roku buttons (power, volume control...), but I can't find how to add it for the Roku buttons.
I tried the below which didn't produce haptic feedback (but did work as a home button), and couldn't find how to do it for the Apps.
            home:
              tap_action:
              haptic: medium
              action: call-service
              service: remote.send_command
              service_data:
                entity_id: remote.roku_bedroom
                command: home

Am I missing something or is there just no support for this yet?
Thanks!

I'll add an option

Excellent, thanks!

I don't see myself doing this anytime soon; need to replace launchApp callservice with a call to handleClick with the formatted action wanted and a haptic parameter
https://github.com/custom-cards/roku-card/blob/master/src/roku-card.ts#L498

Did a rewrite and while do so realized this was already possible, it was just an indentation error in your configuration
            home:
              tap_action:
                haptic: medium
                action: call-service
                service: remote.send_command
                service_data:
                  entity_id: remote.roku_bedroom
                  command: home


[SPARK-4017] show progress bar in console

The progress bar will look like this:

In the right corner, the numbers are: finished tasks, running tasks, total tasks.
After the stage has finished, it will disappear.
The progress bar is only showed if logging level is WARN or higher (but progress in title is still showed), it can be turned off by spark.driver.showConsoleProgress.

How to turn it off on earth? Below does not work.
log4j.rootCategory=ERROR, console
spark.driver.showConsoleProgress=false
spark.ui.showConsoleProgress=false

spark.ui.showConsoleProgress=false works for me. I pass it via --conf to spark-submit. Try that if you haven't already.

I run it in eclispe. Every interval I see 200+ lines of "===>".


[1.13] Automated Bump DC/OS UI v2.82.7

High-level description
What features does this change enable? What bugs does this change fix?
Corresponding DC/OS tickets (required)
Related tickets (optional)

Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain.
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

Please update CHANGES.md file and apply Ready for Review Label.
* Updated DC/OS UI to [1.13+v2.82.7](https://github.com/dcos/dcos-ui/releases/tag/1.13+v2.82.7)

You can edit CHANGES.md directly here: https://github.com/mesosphere/dcos/edit/dcos-ui/bump/2019-07-09-10-10-53/1.13+v2.82.7/CHANGES.md?pr=%2Fdcos%2Fdcos%2Fpull%2F5830

@mesosphere-mergebot label [Ready for Review]

this bump is empty, but EE has changes.


Vuex warning when commit mutation if data is array of objects

Description
I'm creating a Vue component with the Vue Wrapper of HoT and making use of Vuex for handling the state (based on the example of https://handsontable.com/docs/7.0.0/frameworks-wrapper-for-vue-vuex-example.html). When I enable the strict mode on the vuex (https://vuex.vuejs.org/guide/strict.html) and my data is an array of objects (and not an array of arrays like in the example), I get the following Vuex warning:
[vuex] do not mutate vuex store state outside mutation handlers.
Steps to reproduce

Set up an example like ttps://handsontable.com/docs/7.0.0/frameworks-wrapper-for-vue-vuex-example.html
Create a data object which is an array of objects
Make sure that Vuex is in strict mode
Open console window of browser
Modify the content of the cells to commit a mutation
Notice the warning in the console log

Demo

The first 3 steps are already setup in this jsfiddle:
https://jsfiddle.net/x4ngL6rp/4/
Your environment

Vue wrapper version: 4.0.0
Handsontable version: 7.1.0
Browser Name and version: Chrome 75
Operating System: Windows 10

You're right.
Once we turn on the strict mode we get the error https://jsfiddle.net/handsoncode/g24ztrjw/
I will leave it for investigation.

this.$store.commit('updateData', this.hotRef.getSourceData()); commits to Vuex reference to inner Handsontable data property. So each time Handsontable tries to modify data internally the change is propagated to Vuex storage through reference.
You need to break the reference between those objects ie. by creating a deep copy.

Gotcha.
So it is basically a trade-off between the traceability of the mutations vs the performance hit of the deep copy.


Foxx: JWT token iat key

The iat (issued at) generated by the jwtStorage is of type NumericDate, and it's specified in the JWT RFC that "A JSON numeric value representing the number of seconds" and not milliseconds :) !

@pluma: can you have a look at this?

Okay, that is correct and the PR is a good start.
However this is a breaking change because it would break existing JWTs in a surprising way (they would probably not time out correctly as existing milliseconds values would be read as seconds values instead).
@fceller Can you make a decision about this? My recommendation would be to ship this in a minor release and document the changed behaviour. The old logic is definitely non-compliant with the specification.

@pluma yes, do not change this in 3.1. Change it in the upcoming 3.2.


iptables: command not found

Hello, the container does not work anymore since alpine 3.19:
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add <IP_ADDRESS>/24 dev wg0
[#] ip link set mtu 1380 up dev wg0
[#] ip -4 route add <IP_ADDRESS>/24 dev wg0
[#] wg set wg0 private-key /etc/wireguard/privatekey && iptables -t nat -A POSTROUTING  -o eth0 -j MASQUERADE
/usr/bin/wg-quick: line 295: iptables: command not found

The last working Tag is: 20231207-bcfa2df

Thanks for reporting this!


kubeflow-roles failed to upgrade 1.6 to 1.7

Failed to reach active/idle:
kubeflow-roles stuck in maintenance
This could be easy check, because charm does not report errors.
Jira

Hi @i-chvets, I was not able to reproduce this issue by deploying kubeflow-roles 1.6/stable and refreshing to kubeflow-roles latest/stable. Here is the output of juju status after refreshing:
ubuntu@ip-172-31-13-175:~$ juju status kubeflow-roles
Model     Controller  Cloud/Region        Version  SLA          Timestamp
kubeflow  uk8s        microk8s/localhost  2.9.34   unsupported  11:16:49Z

App             Version  Status  Scale  Charm           Channel      Rev  Address         Exposed  Message
kubeflow-roles           active      1  kubeflow-roles  latest/edge   97  <IP_ADDRESS>  no

Unit               Workload  Agent  Address      Ports  Message
kubeflow-roles/0*  active    idle   <IP_ADDRESS>

If you provide steps to reproduce and some logs, we can dig deeper into this.

Works. Verified in latest upgrade.


Add explicit Python 3.12 support

Basically just need to update unit tests to run for 3.12 once dependencies are updated

Testing this in https://github.com/ortk95/planetmapper/tree/233-python3.12

Pylint action is currently crashing for 3.12: https://github.com/pylint-dev/pylint/issues/9091


Add tsx support

This PR adds typescript support for TSX. It mostly just leverages the typescript libray for this, but ensures the compile gets the --jsx preserve args for tsx files.

Thanks!


Update protect-your-global-administrator-accounts.md

changed from company admin to global admin (tested and this was changed recently)

@JohanFreelancer9 Please copy review

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/4132

@JoeDavies-MSFT Please merge this PR. Thanks


image_url helper not working?

I'm trying to serve a full URL of an image in my middleman app.
when I call image_url in a view I receive this error.
undefined method `image_url' for #<#<Class:0x007ff580fbce80>:0x007ff58699ab50> Did you mean? image_alt image_tag

nvm, I realize now there is no way for middleman to know that information

@nedzen You can simply use the built in image_path(img)


Policy 'Enforce Subscription budget' is missing in azopsreference

The referenced policy 'Enforce Subscription budget' is missing in the azopsreference folder (referenced for example in the Contoso reference implementation).
--

This has been updated and added to the reference implementations.

This has been updated and added to the reference implementations.


Empty struct being returned from LEFT LATERAL JOIN where I'd expect a nil

Environment

Elixir version (elixir -v): 1.10.4
Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.): Postgres 12.4
Ecto version (mix deps):

ecto: 3.4.6
ecto_sql: 3.4.5

Database adapter and version (mix deps): postgrex 0.15.6
Operating system: MacOS 10.15.6

Current behavior
I'm performing this query:
    rules_sq =
      from(r in Rule,
        where:
          (r.match_type == "exact" and r.match_string == parent_as(:job_titles).job_title) or
            (r.match_type == "broad" and
               like(parent_as(:job_titles).job_title, fragment("'%' || ? || '%'", r.match_string))),
        order_by: [
          fragment(
            "CASE WHEN ? = ? THEN 0 ELSE levenshtein(?, ?) END ASC",
            r.match_type,
            "exact",
            r.match_string,
            parent_as(:job_titles).job_title
          )
        ],
        limit: 1
      )

    "job_titles"
    |> with_cte("job_titles",
      as:
        fragment(
          "SELECT job_title FROM unnest(?) job_title",
          type(^job_titles, {:array, :string})
        )
    )
    |> from(as: :job_titles)
    |> join(:left_lateral, [jt], subquery(rules_sq))
    |> join(:left, [jt, r], o in Occupation, on: r.occupation_id == o.id)
    |> select([jt, r, o], {jt.job_title, o, r})
    |> Repo.all()

The Rule schema looks like this:
  schema "occupation_matching_rules" do
    field :match_string, :string
    field :occupation_id, :id
    field :match_type, :string
  end

The details of the Occupation schema are probably irrelevant for this query.
This query works fine when one of the rules matches one of the job titles passed in, resulting in:
[{"customer assistant", %Occupation{id: 1, name: "foo", etc...}, %Rule{id: 1, etc...}}]

But when one of the job titles doesn't match a rule I get:
[{"customer assistant", nil, %Rule{id: nil}}]

An empty Rule struct is returned.
Expected behavior
I'd expect to see something more like the below, with a nil in place of the empty Rule struct.
[{"customer assistant", nil, nil}]

Thank you for the report! Can you please provide a small application or a failing test case that reproduces the error? We have limited time and I want to make sure our time is spent on fixing the bug rather than reproducing it. Thanks!

Sure thing, here's a patch file with a failing integration test attached:
https://gist.github.com/tomtaylor/0f6f18225b518bb2c60b2f10666681cf

Those are the best, thank you! <3

Thanks @josevalim!


ENH: add result_type_false: check the minimum promotion rules according to the spec

https://data-apis.org/array-api/latest/API_specification/type_promotion.html#type-promotion specifies the minimum required promotion rules, and states that

A conforming implementation of the array API standard may support additional type promotion rules beyond those described in this specification.

array_api_strict only allows this minimal promotion rules. So, add a test that promotions beyond this minimum set are not allowed. The test is mostly informational: an array provider may want signal that it allows extra promotion paths by xfailing the test.

So, add a test that promotions beyond this minimum set are not allowed.

This feels wrong to me. This test suite should not start including tests for behavior that is allowed by the standard (and common in practice), but just not standardized. That just leads to a lot of busywork I think?

I agree this test doesn't make sense. We could use something like this in array_api_strict when testing strictness. Currently those tests don't use hypothesis, but rather just test each combination.
As far as the strategy, it looks like it would do what it says it does. I don't understand the max_size parameter that does nothing. Also, to be fully complete, it would need to include uint64. Maybe a cleaner way to implement it would be to take all possible combinations of dtypes, remove the ones that are promotable, and sample_from that. (but again, I don't know that we actually need this strategy)

I still think it's a not totally invalid approach, the decision is only where the documentation is, here or in the array libraries. But okay, the reaction is pretty clear :-). Let's close this. Thanks for the review!

the decision is only where the documentation is

It's more than that. Note that "should" and "must" have precise meanings, see the discussion at https://github.com/data-apis/array-api/issues/796 (we should really close that issue by clarifying the docs). The test suite by design has to verify all the "must" requirements, but not enforce "should" recommendations like they are "must" requirements, nor enforce things that aren't specified.
In the case of type promotion rules, the spec lays out what must be implemented and nothing else. It says:
"Mixed integer and floating-point type promotion rules are not specified because behavior varies between implementations."
If it had said something like "mixed integer and floating-point ... must raise" or "...must not be implemented" then it needs a test like this. In the absence of that language, it doesn't.


[Bug]: Dev mode: workspace management is confusing

Is there an existing issue for the same bug?

[X] I have checked the troubleshooting document at https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/Troubleshooting.md
[X] I have checked the existing issues.

Describe the bug
It's always using a docker volume mount instead of the local directory, so the agent can't see the files I put in the workspace directory.
Also, that volume mount doesn't have permissions for the opendevin user so it can't create new files in it. Using WSL2 and starting using  "make run".
Edit: update from comment below: There appears to be a divergence between container vs local host filesystems of the FileWriteAction and read action vs the CmdRunAction.
Current Version
d692a72bf3809df35d

Installation and Configuration
config.toml:

LLM_API_KEY="na"
LLM_BASE_URL="http://<IP_ADDRESS>:5000/v1" #
LLM_MODEL="na"
LLM_EMBEDDING_MODEL="lokal"
MAX_ITERATIONS=10000 #set max steps opendevin will run
WORKSPACE_BASE="./workspace"

Model and Agent
Monologue and local model
Reproduction Steps
Using windows WSL2.
in config.toml set:
WORKSPACE_BASE="./workspace"
make build
make run
observe with "docker inspect" that the volume shows it's ignoring the local directory
  "HostConfig": {
            "Binds": [
                "None:/workspace:rw"
            ],

Use the logged ssh command to get into the container as the opendevin user and observe:
Connecting to opendevin@localhost via ssh. If you encounter any issues, you can try `ssh -v -p 55917 opendevin@localhost` with the password '1ac90739-a5d0-4325-bc4c-59d4515b3766' and report the issue on GitHub.

$ ssh -v -p 55917 opendevin@localhost
$ cd /workspace

observe that any existing files you put in workspace are not in there.
opendevin@opendevin_sandbox:/workspace$ touch afile
touch: cannot touch 'afile': Permission denied

The opendevin user can't make a file and it is owned by root instead of opendevin
opendevin@opendevin_sandbox:/workspace$ ls -la
total 12
drwxr-xr-x 2 root root 4096 Apr 21 15:02 .
drwxr-xr-x 1 root root 4096 Apr 21 15:18 ..

Logs, Errors, Screenshots, and Additional Context
I'm trying to give the agent a simple task by adding some words to words.txt and asking it to use the terminal to sort the words into a new file. It correctly thinks to try 'cat words.txt | sort > newfile.txt' but the permissions and the lack of local directory mount is preventing it from working.
The front end does show my local workspace directory and the files I put there, but inside the sandbox is not the actual workspace.

Actually there seems to be a divergence going on.
The FileWriteAction and FileReadAction appears to sync with my local directory fine.
But CmdRunAction appears to be stuck inside the sandbox container which does not sync with my local directory.
In this run I manually put the list "balloon\r\ngiraffe\r\nmouse\r\nzerba" into the container and fixed the permissions manually as a workaround, but the agent cound't see it with the FileReadAction so it through to create it's own words.txt using FileWriteAction. But it could see the original words.txt using CmdRunAction, but the CmdRunAction didn't see the words.txt that was created with FileWriteAction.
 {
    "action": "think",
    "args": {
      "thought": "It seems like there might be an existing project here. I should probably start by running `ls` to see what's here."
    }
  },
  {
    "action": "read",
    "args": {
      "path": "/workspace/words.txt",
      "start": 0,
      "end": -1,
      "thoughts": ""
    }
  },
  {
    "observation": "error",
    "content": "File not found: /workspace/words.txt",
    "extras": {},
    "message": "Oops. Something went wrong: File not found: /workspace/words.txt"
  },
  {
    "action": "write",
    "args": {
      "path": "words.txt",
      "content": "apple orange banana",
      "start": 0,
      "end": -1,
      "thoughts": "Wrote words.txt with funny names"
    }
  },
  {
    "observation": "write",
    "content": "",
    "extras": {
      "path": "words.txt"
    },
    "message": "I wrote to the file words.txt."
  },
  {
    "action": "read",
    "args": {
      "path": "/workspace/words.txt",
      "start": 0,
      "end": -1,
      "thoughts": ""
    }
  },
  {
    "observation": "read",
    "content": "apple orange banana\n",
    "extras": {
      "path": "/workspace/words.txt"
    },
    "message": "I read the file /workspace/words.txt."
  },
  {
    "action": "read",
    "args": {
      "path": "words.txt",
      "start": 0,
      "end": -1,
      "thoughts": "Read words.txt"
    }
  },
  {
    "observation": "read",
    "content": "apple orange banana\n",
    "extras": {
      "path": "words.txt"
    },
    "message": "I read the file words.txt."
  },
  {
    "action": "think",
    "args": {
      "thought": "Trying to get to the correct location for words.txt"
    }
  },
  {
    "observation": "null",
    "content": "",
    "extras": {},
    "message": ""
  },
  {
    "action": "run",
    "args": {
      "command": "ls",
      "background": false
    }
  },
  {
    "observation": "run",
    "content": "words.txt",
    "extras": {
      "command_id": -1,
      "command": "ls",
      "exit_code": 0
    },
    "message": "Command `ls` executed with exit code 0."
  },
  {
    "action": "read",
    "args": {
      "path": "/workspace/words.txt",
      "start": 0,
      "end": -1,
      "thoughts": "Trying to get to the correct location for words.txt"
    }
  },
  {
    "observation": "read",
    "content": "apple orange banana\n",
    "extras": {
      "path": "/workspace/words.txt"
    },
    "message": "I read the file /workspace/words.txt."
  },
  {
    "action": "run",
    "args": {
      "command": "sort words.txt",
      "background": false
    }
  },
  {
    "observation": "run",
    "content": "balloon\r\ngiraffe\r\nmouse\r\nzerba",
    "extras": {
      "command_id": -1,
      "command": "sort words.txt",
      "exit_code": 0
    },
    "message": "Command `sort words.txt` executed with exit code 0."
  },
  {
    "action": "run",
    "args": {
      "command": "sort words.txt",
      "background": false
    }
  },
  {
    "observation": "run",
    "content": "balloon\r\ngiraffe\r\nmouse\r\nzerba",
    "extras": {
      "command_id": -1,
      "command": "sort words.txt",
      "exit_code": 0
    },
    "message": "Command `sort words.txt` executed with exit code 0."
  },
  {
    "action": "run",
    "args": {
      "command": "sort words.txt > output.txt",
      "background": true
    }
  },
  {
    "observation": "run",
    "content": "Background command started. To stop it, send a `kill` action with id None",
    "extras": {
      "command_id": null,
      "command": "sort words.txt > output.txt",
      "exit_code": 0
    },
    "message": "Command `sort words.txt > output.txt` executed with exit code 0."
  },
  {
    "action": "think",
    "args": {
      "thought": "Running again works."
    }
  },
  {
    "observation": "null",
    "content": "",
    "extras": {},
    "message": ""
  },
  {
    "action": "run",
    "args": {
      "command": "cp output.txt sorted_words.txt",
      "background": false
    }
  },
  {
    "observation": "run",
    "content": "",
    "extras": {
      "command_id": -1,
      "command": "cp output.txt sorted_words.txt",
      "exit_code": 0
    },
    "message": "Command `cp output.txt sorted_words.txt` executed with exit code 0."
  },
  {
    "action": "run",
    "args": {
      "command": "ls",
      "background": false
    }
  },
  {
    "observation": "run",
    "content": "output.txt  sorted_words.txt  words.txt",
    "extras": {
      "command_id": -1,
      "command": "ls",
      "exit_code": 0
    },
    "message": "Command `ls` executed with exit code 0."
  },

@tpsjr7 I believe you need to set WORKSPACE_MOUNT_PATH when running in development. We probably need to add this to make setup-config

Thank you, yes that resolved my issue. Yes, fresh eyes, I simply was following Development.md to setup.

I believe this has been made better by a recent PR which defaults one of the variables to the other's value


infinite "error: failed to fill whole buffer" messages when openocd is suddenly closed

Fixed in #5


Page number automatically goes back to page 1

Bug description
When you go to the next page the number above directly resets itself to page 1
Steps to reproduce
First search up results with more than 1 page
Now click on next
When you now click on next you will stay on page 2 and you will see that the page number above is 1
Expected behavior
Go to the next page without a issue
Windows 10 with the latest nightly version

On which source/search did this occur? I just tried on a few sources with no issues.

Hmm I can't reproduce it after reopening grabber. Will comment here again if I find a way to reproduce it

@Bionus When you use for example the sources sankaku complex, gelbooru, yande.re, konachan, rule34, danbooru and anime-pictures you will get this bug. When you take either danbooru away or anime-pictures the bug will not appear anymore. So for some reason those 2 sources can't be used together now

@Bionus When you use for example the sources sankaku complex, gelbooru, yande.re, konachan, rule34 and anime-pictures you will get this bug. When you take away anime-pictures the bug will not appear anymore. (You will have to click 2 times on next most of the time before you see the bug)

@Bionus When you use for example the sources sankaku complex, gelbooru, yande.re, konachan, rule34 and anime-pictures you will get this bug. When you take away anime-pictures the bug will not appear anymore. (You will have to click 2 times on next most of the time before you see the bug)

Indeed, this only happened with multiple sources selected, and when the number of pages was not the same between them (especially when one source didn't have any result at all).
This should have been fixed by commit b63910b3024b80a655d4942167436e468de255b6.


pass GB sheets through pygrambank

I would suggest that changes to bedlan/uratyp-1/raw/GB/language-tables should be passed through pygrambank describe and sourcelookup. See #3 for examples of warnings.
If UraTyp team thinks this is a good idea, we may need to modify pygrambank commands somewhat to account for placement of files.

The "UraTyp team" - as far as converting to CLDF goes - is me.

I don't understand. Is this not also the repos for UraTyp raw curation?

Yes, unfortunately this is true that converting to CLDF = Robert. I do not have these skills...

Yes, unfortunately this is true that converting to CLDF = Robert. I do not have these skills...

That's alright. What I'm suggesting  is not related to the conversion to CLDF. I am suggesting that sheets in  bedlan/uratyp-1/raw/GB/language-tables are passed through some checks, like these:  https://github.com/grambank/grambank/wiki/Automatic-quality-checks.
At Glottobank/Grambank, every time we edit a file we run it through these checks to make sure that the sources line up, that the values make sense etc. We do this little by little each time, it saves us getting a long list of things to see to when we render the CLDF-version.
If you did this here at UraTyp, you could either use pygrambank (we'd have to modify it a little but not much) or you could have your own. Then you'd get warnings like in #4 and #3 right away when edits are made.

I appreciate that you like pygrambank so much. UraTyp is very different to Grambank, though. At this point, operationally, UraTyp is just Miina and me - and we get by. As creator of pygrambank I know what it can do and what not. And just like Johannes in the case of Grambank, it would be me in the case of UraTyp who'd be running the checks. As far as I can tell, pygrambank isn't necessary for the regular operation of UraTyp updates.

Okay. I thought that for checking linking to sources and invalid values, pygrambank or something like it might be useful in the day-to-day.

There really is no "day-to-day". UraTyp is not something - as far as I can tell - that's continuously worked on. It's rather Miina and me (with hours, rather than days or weeks of our time) putting the next release together.

There really is no "day-to-day". UraTyp is not something - as far as I can tell - that's continuously worked on. It's rather Miina and me (with hours, rather than days or weeks of our time) putting the next release together.

Okay. I don't know your set-up, it just seemed  useful to me. If you rather handle invalid values, linking references etc some other way then that's alright by me. For the GB sheets specifically, it seemed to me that pygrambank would be handy but please disregard if it isn't.

It sounds like the suggestion is turned down, if so if @MiinaNo or @xrotwang could please close the issue to confirm.


Memex Domain Export.

We need to be able to export a domain for use by the crawling teams. The domain should be the aggregate of all of the trails within a domain (At some point we may want to be able to choose specific trails.)
The format should be similar to
{
  "urls": ["http://la.backpage.com/1234", "http://la.backpage.com/1234", "http://www.wikipedia.com"], //All visited urls
  "topLevelDomains": ["http://la.backpage.com"], //common top level domains
  "searchTerms": ["escorts","las angeles","massage"], //all search terms
  "domainEntities": ["cherry","mimi","pasadena"], //entities added by a user.
  "domainEntityTypes": ["person","place","bitcoin address"],  //entity types added by a user.
  "commonEntities": ["massage", "parlor","las angeles","pasadena"]  //top 20 most extracted entities.
}

We should include the domain Id.
I'd rename "topLevelDomains" to "rootUrls".  We'll need to do a search for unique ones which should be easy enough.
I understand SearchTerms, Domain Entity Types, and Domain Entities.  What are commonEntities?
We'll need to add the url relevance to the urls array.  I do think we should include ids on each appropriate item (overall domain,urls, domain entities) in case we want to retrieve results from a third party who uses this export

rootURLS is fine.
commonEntities would be the most extracted entities. I'm not sure how useful this will be but we can see.
I agree everything should probably be an object array of {"id": "1223", "value": "value"}

since we don't currently have commonEntites, can we just include ExtractedEntities (with id, occurences, url extracted from, etc)?

That's what I meant, just pick an arbitrary number like the top 20 extracted entities with the highest counts.

Looks like it's working as described.  JSON exports and has all the sections with appropriate content.  Couldn't get domain items manually added because that panel stuff was in a different branch, but the section was there in the JSON.

@bmcdougald @michaelsframe When I try this on a larger domain, I only get a small amount of the domain, with no logging.

Fixed in 100/90


Switching breakpoint condition should preserve the condition in breakpoint widget

Originaly from @rkeithhill

For me, the only issue that exists here is that the dropdown UI does not save the current value when switching between Expression and Hit Count. I know the watermark text says to hit press Enter but I think a fair number or folks will type an Expression, hit the drop down to set the Hit Count, then they press Enter. In this scenario only the Hit Count is accepted. The condition expression is not.

To verify: enter both condition and hit condition for a breakpoint and make sure both get applied.
fyi @rkeithhill this is now fixed


Handling of edoc files

Is there any way to include edoc files and generate doc through rebar3_ex_doc? For example, edoc files within this repo, https://github.com/massemanet/redbug/tree/master/doc.

@kianmeng, when I first ran rebar3_ex_doc it seemed to have cleared doc (I was also using overview.edoc and had a .gitignore in the folder), so I imagined with was explicitly made so, but am also curious regarding the reply to your question, since it would be neat to at least have a way to smoothly transition (I ended up taking the edoc files and converting them to Markdown, outside of doc).

@kianmeng  If I understand your question correctly, you're wanting to know if we can convert edoc output to ex_doc style and such? If so, there is currently no way to do this, the docs generated by ex_doc come from BEAM chunks vs edoc output.
@paulo-ferraz-oliveira Anything in doc shouldn't be blown away, is that what happened to you?

@starbelly Thanks for the answer. It seemed no other ways except what @paulo-ferraz-oliveira suggested, manually convert the edoc file to md file. Since it involved some manual tweaking, and I only have basic knowledge of Erlang and EDoc, I'll ask the Erlang library owner whether they want to migrate to ex_doc first before submitting a PR.
By the way, do you guys know which Erlang libraries to explore first for edoc to ex_doc migration, so we've a standard documentation layout for both Elixir and Erlang libraries?
Or do we wait until the Erlang support for ex_doc, https://github.com/elixir-lang/ex_doc/issues/1333 have completed, or EEP 059, https://www.erlang.org/eeps/eep-0059, have been accepted and implemented?

@paulo-ferraz-oliveira Anything in doc shouldn't be blown away, is that what happened to you?

Yeah, I believe the whole folder was just deleted. I had, in the past, overview.edoc there, so am pretty certain my procedures wouldn't do that.
...
I just tried it again, and the files were removed. It's not a big deal, for me, since I do not want to mix edoc and ex_doc but it might be for some (especially if you run this without eg. having staged your files).

Yeah, it shouldn't be so destructive. I'll have to track down where that's happening. It's been a minute.

Close as the question have been answered, https://github.com/starbelly/rebar3_ex_doc/issues/19#issuecomment-1100908017.

@paulo-ferraz-oliveira This happens here https://github.com/elixir-lang/ex_doc/blob/935f2dac019d0bdcb470e68fc8b077e4e27bfe15/lib/ex_doc/formatter/html.ex#L149
If ex_doc has already been run once, your edoc output won't be purged, otherwise it will.  That behaviour would have to be changed in ex_doc of course, thus I will close the issue here.

Sure. Not a big problem to me, though, as stated before. I had rebar3 edoc stuff, not rebar3_ex_doc stuff 👍


Make DGS compatible with Spring Boot 2 and 3

Pull request checklist

[x] Please read our contributor guide
[x] Consider creating a discussion on the discussion forum
first
[x] Make sure the PR doesn't introduce backward compatibility issues
[x] Make sure to have sufficient test cases

Pull Request type

[ ] Bugfix
[x] Feature
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Other (please describe):

Changes in this PR
Updated to Spring Boot 2.7.6
Changed the auto configuration to be compatible with Spring 2 and 3

Hi @esfomeado,
I'm trying to understand the motivation for these changes outside of a Spring Boot 3.0 migration.

By removing spring.factories, this would break backwards compatibility with existing projects using a version of Spring Boot prior to 2.7.
Alone, these changes aren't enough to facilitate using DGS in a Spring Boot 3.0 project. You'd still experience compatibility issues, such as the dependencies conflicts between javax and jakarta.


Integer overflow (32-bit) when saving large tensor on Windows

System Info
Windows 10 Pro 22H2 64-bit (also confirmed on Windows 11)
Python 3.11.4
safetensors 0.4.0
numpy 1.26.2
torch 2.1.1+cu118
Information

[ ] The official example scripts
[X] My own modified scripts

Reproduction
import torch
from safetensors.torch import save_file
filename = "E:\Temp\test.safetensors"
test_tensor = torch.randn((32, 2048, 64000), dtype=torch.half, device="cuda:0")  # device="cpu" also fails
save_file({ "test_tensor": test_tensor }, filename)

Expected behavior
Expected to create .safetensors file with a size of about 8GB, containing a single tensor. Instead, fails with the error:
Traceback (most recent call last):
  File "C:\Sync\Dev2\test.py", line 6, in <module>
    save_file({ "test_tensor": test_tensor }, filename)
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\safetensors\torch.py", line 281, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
                   ^^^^^^^^^^^^^^^^^
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\safetensors\torch.py", line 475, in _flatten
    return {
           ^
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\safetensors\torch.py", line 479, in <dictcomp>
    "data": _tobytes(v, k),
            ^^^^^^^^^^^^^^
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\safetensors\torch.py", line 421, in _tobytes
    data = np.ctypeslib.as_array(newptr, (total_bytes,))  # no internal copy
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\ctypeslib.py", line 521, in as_array
    p_arr_type = ctypes.POINTER(_ctype_ndarray(obj._type_, shape))
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\xx\AppData\Local\Programs\Python\Python311\Lib\site-packages\numpy\ctypeslib.py", line 354, in _ctype_ndarray
    element_type = dim * element_type
                   ~~~~^~~~~~~~~~~~~~
ValueError: Array length must be >= 0, not -201326592

The reported array length is exactly 2 * (test_tensor.numel() - 2**32), leading me to suspect this is an integer overflow issue. The issue only seems to manifest on Windows, with the code above performing as expected on Linux.

Definitely looks like an overflow for Windows.
Can you make sure you're running on 64bit execution mode and not 32 ? (For instance isolating the python code that's being called and checking total_bytes value.

So I've investigated it a bit. As far as I can tell, I'm running in 64-bit mode, and there should be no way for Python to load a 32-bit version of numpy.
import sys
import platform
import numpy.distutils.system_info as sysinfo
print(sys.maxsize)
print(platform.architecture())
print(sysinfo.platform_bits)

Yields<PHONE_NUMBER>854775807, ('64bit', 'WindowsPE') and 64, respectively.
I can narrow the error down to the following line in torch.py:
length = int(np.prod(tensor.shape).item())

Specifically it's the np.prod function that returns a negative value. According to the numpy documentation, the accumulator for the prod function should use the same datatype as the input array, but maybe that's not entirely accurate? In any case, this is one way to fix it:
length = int(np.prod(tensor.shape, dtype=np.uint64).item())

It does look like something somewhere is defaulting to a 32-bit mode (although you'd think then it would overflow to a negative value at 231, or to a small positive value at 232), but I'm not sure what other tests I could run.


Store cookies in XDG_DATA_HOME instead of XDG_CONFIG_HOME

1000th pull request !

@swift-ci test

@swift-ci test

@swift-ci please test

@pushkarnk Could you please run the CI Build again. I have pushed in the necessary changes to fix the test-failures.

@swift-ci please test

@swift-ci please test

@pushkarnk @parkera The test-failure is because of a bug in Cookie Storage where it fails to create non-existing parent directories. A fix for this is already available in the open PR: https://github.com/apple/swift-corelibs-foundation/pull/1003

We have #1003 merged now. Lets try to merge this one as well.
@swift-ci test and merge

@swift-ci test and merge


Fix shellcheck error / warnings

Please run shellcheck against this -- there are many warnings and an error:
 validate_parameters $@ ${piped_value}

Should be: "$@".

Thanks!

Please consider re-opening - there are many more issues than the one I pointed out.

I'd highly recommend adding shellcheck to your list of linters to prevent ongoing issues, but it's also available for copy-paste usage online.

Yes, I will try to fix other warning/infos stated by shellcheck separately.
The one (& only) error it showed & you mentioned, I have fixed it.

Thanks for getting the warnings, too!


No values found in Gymfile when running on CircleCI (works locally)

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
When I try to run gym on CircleCI Gymfile is not loaded and the build fails later on.
Screenshot from CircleCI:

As you can see on the screenshot above, looks like the Gymfile is empty.
I know for sure that Gymfile contains information as you can see on the next screenshot (circleci command):

The same lane works correctly locally with the same fastlane version:

Fastfile on deploy_crashlytics lane:
  gym(
    scheme: "FieldMargin",
    export_method: "ad-hoc"
  )

Gymfile
export_options(
	method: "ad-hoc",
	provisioningProfiles: {
		"com.fielmdargin.notesdev" => "match AdHoc com.fielmdargin.notesdev"
	}
)

So far I tried all sorts of things like deleting the file, creating it again and avoiding copy/paste just in case some weird character was causing the issue, but I fail to see why CircleCI is not reading the Gymfile file. Any ideas? Thanks!
 Just to give it a shot?

Interesting, CircleCI does read the scheme but not the export_options.
I added the stack information from both sides because this sounds more like a Ruby issue when parsing the file (just a guess).
CircleCI:
[09:55:52]: Successfully loaded '/Users/distiller/notes-ios/fastlane/Gymfile' 📄

+---------------------+---------------------+
| Detected Values from './fastlane/Gymfile' |
+---------------------+---------------------+
| scheme              | FieldMargin         |
| export_options      |                     |
+---------------------+---------------------+

Stack:
### Stack

| Key                         | Value                                         |
| --------------------------- | --------------------------------------------- |
| OS                          | 10.12.6                                       |
| Ruby                        | 2.3.5                                         |
| Bundler?                    | false                                         |
| Git                         | git version 2.13.5 (Apple Git-94)             |
| Installation Source         | ~/.gem/ruby/2.3.5/bin/fastlane                |
| Host                        | Mac OS X 10.12.6 (16G29)                      |
| Ruby Lib Dir                | ~/.rubies/ruby-2.3.5/lib                      |
| OpenSSL Version             | OpenSSL 1.0.2l  25 May 2017                   |
| Is contained                | false                                         |
| Is homebrew                 | false                                         |
| Is installed via Fabric.app | false                                         |
| Xcode Path                  | /Applications/Xcode-9.app/Contents/Developer/ |
| Xcode Version               | 9.0                                           |

### System Locale

| Error                       |
| --------------------------- |
| No Locale with UTF8 found 🚫 |

Locally:
[09:52:40]: Successfully loaded '/Users/lluisgerard/Xcode/apps/notes-ios Xcode 9 GM working frameworks/fastlane/Gymfile' 📄

+----------------+--------------------------------------+
|       Detected Values from './fastlane/Gymfile'       |
+----------------+--------------------------------------+
| scheme         | FieldMargin                          |
| export_options | {:method=>"ad-hoc",                  |
|                | :provisioningProfiles=>{"com.fielmd  |
|                | argin.notesdev"=>"match AdHoc        |
|                | com.fielmdargin.notesdev"}}          |
+----------------+--------------------------------------+

### Stack

| Key                         | Value                                        |
| --------------------------- | -------------------------------------------- |
| OS                          | 10.12.6                                      |
| Ruby                        | 2.2.4                                        |
| Bundler?                    | false                                        |
| Git                         | git version 2.14.2                           |
| Installation Source         | ~/.fastlane/bin/bundle/bin/fastlane          |
| Host                        | Mac OS X 10.12.6 (16G29)                     |
| Ruby Lib Dir                | ~/.fastlane/bin/bundle/lib                   |
| OpenSSL Version             | OpenSSL 1.0.2g  1 Mar 2016                   |
| Is contained                | false                                        |
| Is homebrew                 | true                                         |
| Is installed via Fabric.app | false                                        |
| Xcode Path                  | /Applications/Xcode9.app/Contents/Developer/ |
| Xcode Version               | 9.0                                          |

### System Locale

| Variable | Value       |   |
| -------- | ----------- | - |
| LANG     | en_US.UTF-8 | ✅ |
| LC_ALL   | en_US.UTF-8 | ✅ |
| LANGUAGE | en_US.UTF-8 | ✅ |

Some context about this issue:
Now I'm trying to figure out how to use the right provisioning profile without using Gymfile so I can solve this asap. To add a little bit of context, we've moved to match and  Xcode 9 recently. Until now I was using my identity to sign on circleci with a development cert, it's been working for months (because fastlane is awesome), but we moved to ad-hoc after some issues, it seems to me that fastlane gym is trying to build the app with development and then sign it with ad-hoc? that's why I had to specify adhoc profile locally to make it work.

I'm closing this issue because I'm having multiple issues at the same time and I'm waiting for a new update. Also I'm not using Gymfile at the moment.
Will reopen when I have more info or try this again.


bspwm: 0.9.4 -> 0.9.5

Motivation for this change
Upstream update
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build bspwm

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: bspwm
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: bspwm
Partial log (click to expand)
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/loop/wm: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/overlapping_borders/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/external_rules: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/pseudo_automatic_mode/external_rules: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/panel/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/panel/panel: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5/share/doc/bspwm/examples/panel/panel_bar: interpreter directive changed from " /bin/sh" to "/nix/store/xn5gv3lpfy91yvfy9b0i7klfcxh9xskz-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5...
/nix/store/rfwbkqld0kzgimsh6pnd4kblj1kpabdg-bspwm-0.9.5

Success on aarch64-linux (full log)
Attempted: bspwm
Partial log (click to expand)
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/panel/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/overlapping_borders/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/loop/wm: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/loop/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/pseudo_automatic_mode/external_rules: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/external_rules: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/external_rules/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5/share/doc/bspwm/examples/bspwmrc: interpreter directive changed from " /bin/sh" to "/nix/store/adw9jx59wnrh5659wz43nbjya3m4b3gl-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5...
/nix/store/yi7h4gwvhy5xpr5f02i1q30an40qz63p-bspwm-0.9.5


Angular template: Dependency injection fails for lazy loaded module

Moved from https://github.com/aspnet/Docs/issues/7826. Originally opened by @tedishero.
Lazy loading itself works, however as soon as I introduce a dependency to a component or service in the lazy loaded module, it starts to complain about not able to resolve dependencies.

Thanks for contacting us, @tedishero.
Thank you for filing this issue. In order for us to investigate this issue, please provide a minimalistic repro project that illustrates the problem.

Hi. We're closing this issue as no response or updates have been provided in a timely manner and we have been unable to reproduce it. If you have more details and are encountering this issue please add a new reply and re-open the issue.


Allow trailing whitespace in blockquote in edit mode

This PR:

adds new shouldKeepWhitespace flag to the parser.replace function in ExpensiMark, which allows trailing whitespace in blockquotes
changes quote rule logic when shouldKeepWhitespace is true

https://github.com/Expensify/expensify-common/assets/39538890/8a4de1fe-c2c4-423f-929e-57728961c01c
Fixed Issues
$ GH_LINK
Tests

What unit/integration tests cover your change? What autoQA tests cover your change?
What tests did you perform that validates your changed worked?

QA

What does QA need to do to validate your changes?
What areas to they need to test for regressions?

I have read the CLA Document and I hereby sign the CLA