Remote CouchDB is not encrypted

When using crypto-pouch in an ionic 2 App and replicating to a remote CouchDB database the CouchDB documents are not encypted.
The code I am using looks like this:
`  init(localDBName, remoteDBName)
{
this._DB = new PouchDB(localDBName);
PouchDB.plugin(CryptoPouch);
this._DB.crypto('a password');

this.remote = remoteDBName;

console.log('Data: init(): PouchDB database opened for localDBName = ' + localDBName + ' : ' + this._DB + ', remoteDBName = ' + this.remote);

// Insert the url for the CouchDB server into the remoteDBName
var realRemoteDB = this.remote.replace("localhost:5984", COUCHDB_SERVER_URL);

console.log('Data: init(): remoteDB path being used is: ' + realRemoteDB);

this._remoteDB = new PouchDB(realRemoteDB);
this._remoteDB.crypto('a password');

let options = {
  live: true,
  retry: true,
  continuous: true
};

return this._DB.sync(realRemoteDB) //No options here -> One time sync
  .on('complete', (info) => {
    console.log('Data: init(): first one time sync has completed about to do live syncing now');
    this.events.publish('SYNC_FINISHED', true); // Let login know we have synced the data
    return this._DB.sync(realRemoteDB, options) //Continous sync with options
      .on('complete', (info) => {
        console.log('***** DATA: init() Complete: Handling syncing complete');
        console.dir(info);
      })
      .on('change', (info) => {
        console.log('***** DATA: init() Change: Handling syncing change');
        console.dir(info);
      })
      .on('paused', (info) => {
        console.log('***** DATA: init() Paused: Handling syncing pause');
        console.dir(info);
      })
      .on('active', (info) => {
        console.log('***** DATA: init() Active: Handling syncing resumption');
        console.dir(info);
      })
      .on('error', (err) => {
        console.log('***** DATA: init() Error: Handling syncing error');
        console.dir(err);
      })
      .on('denied', (err) => {
        console.log('***** DATA: init() Denied: Handling syncing denied');
        console.dir(err);
      });
  });

}`

likely an issue with transform pouch which this is based on

Hi Calvin,
Is there anything you can suggest? Should I hunt for the bug in the transform pouch code and if so can you point me in the right direction?
Very Best
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Wednesday, 5 April 2017 at 23:23
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
likely an issue with transform pouch which this is based on
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I can take a deeper look today, what versions of everything are you using?
On Thu, Apr 6, 2017, 5:04 AM Bill Noble<EMAIL_ADDRESS>wrote:

Hi Calvin,
Is there anything you can suggest? Should I hunt for the bug in the
transform pouch code and if so can you point me in the right direction?
Very Best
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Wednesday, 5 April 2017 at 23:23
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
likely an issue with transform pouch which this is based on
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292113173,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4ny5gSpewFzKceUjoMn9hFPeS85Eyks5rtKp_gaJpZM4M0w0i
.

Thanks Calvin that is greatly appreciated.
The versions I am using are:
"pouchdb": "^6.1.2",
"crypto-pouch": "^3.1.1",
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Thursday, 6 April 2017 at 12:34
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
I can take a deeper look today, what versions of everything are you using?
On Thu, Apr 6, 2017, 5:04 AM Bill Noble<EMAIL_ADDRESS>wrote:

Hi Calvin,
Is there anything you can suggest? Should I hunt for the bug in the
transform pouch code and if so can you point me in the right direction?
Very Best
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Wednesday, 5 April 2017 at 23:23
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
likely an issue with transform pouch which this is based on
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292113173,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4ny5gSpewFzKceUjoMn9hFPeS85Eyks5rtKp_gaJpZM4M0w0i
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

in both calls to sync you need to use the remote pouchdb instance, not just the name of the remote database,
this._DB.sync(this._remote)

you may need to save this._remote in a variable for use in the second call to sync but the gist of it is that since encryption is done locally, you can't refer to the database by name, because then pouchdb will internally create a new instance that won't have encryption associated with it.

Thanks Calvin!
I will try that out now and let you know how I get on.
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
in both calls to sync you need to use the remote pouchdb instance, not just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call to sync but the gist of it is that since encryption is done locally, you can't refer to the database by name, because then pouchdb will internally create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Yaay! That seems to have worked.
I assume that all of the encrypted data is in the ‘data’ field of the document?
Massive thanks for helping me sort this it is greatly appreciated.
Very Best,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
in both calls to sync you need to use the remote pouchdb instance, not just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call to sync but the gist of it is that since encryption is done locally, you can't refer to the database by name, because then pouchdb will internally create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

yup thats the encrypted data

Hi Calvin,
I have tested the revised sync calls running my ionic 2 app in the browser and it seems to work great.
However, when I run my app on my iPhone the sync times out with an ETIMEDOUT error:
{"line":3,"column":4869,"sourceURL":"file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js","originalStack":"v@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:4869\nonreadystatechange@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/main.js:173087:24\nn@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:2:27345\ninvokeTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10289\nrunTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:7647\ninvoke@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:11404\nabort@[native code]\ncancelTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10646\ncancelTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:8206\nfile:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:3203\nabort\ntimeoutReq@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/main.js:172991:14\ninvokeTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10289\nrunTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:7647\ninvoke@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:11404\nfile:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:2:30198","zoneAwareStack":"v@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:4869\nonreadystatechange@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/main.js:173087:24\nn@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:2:27345\ninvokeTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10289\nrunTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:7647\ninvoke@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:11404\nabort@[native code]\ncancelTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10646\ncancelTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:8206\nfile:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:3203\nabort\ntimeoutReq@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/main.js:172991:14\ninvokeTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:10289\nrunTask@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:7647\ninvoke@file:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:3:11404\nfile:///var/containers/Bundle/Application/CD1C2BBE-1425-4E54-AA21-BA1518B7BFBD/vanilla%20app.app/www/build/polyfills.js:2:30198","code":"ETIMEDOUT","status":0,"result":{"ok":false,"start_time":"2017-04-10T08:09:58.265Z","docs_read":0,"docs_written":0,"doc_write_failures":0,"errors":[],"status":"aborting","end_time":"2017-04-10T08:12:49.175Z","last_seq":0}}
Before I started using the pouchdb instance in the sync call (when I was using the name of the remote database) the sync did not timeout on my iPhone.
Do you have any thoughts on why this might be happening? The iPhone is connecting to the server ok and successfully logging in with SuperLogin it is just the PouchDB sync that is causing problems (but not when running on the browser when it runs ok).
One thing that may or may not be making a difference is my setup (but again this worked ok when using the remote database name)
·         SuperLogin and CouchDB are running on my laptop
·         I use ngrok to tunnel back to my laptop from the iPhone
Any help you can give would be hugely appreciated.
Many thanks,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
in both calls to sync you need to use the remote pouchdb instance, not just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call to sync but the gist of it is that since encryption is done locally, you can't refer to the database by name, because then pouchdb will internally create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi Calvin,
I forgot to add the CouchDB log to my previous email:
[info] 2017-04-10T08:55:34.993290Z couchdb@localhost <0.18233.0> -------- Starting index update for db: shards/80000000-ffffffff/sl-users.1491582997 idx: _design/auth
[info] 2017-04-10T08:55:35.009309Z couchdb@localhost <0.18233.0> -------- Index update finished for db: shards/80000000-ffffffff/sl-users.1491582997 idx: _design/auth
[notice] 2017-04-10T08:55:35.010439Z couchdb@localhost <0.27087.27> e3747ff4ce localhost:5984 <IP_ADDRESS> undefined GET /sl-users/_design/auth/_view/emailUsername?include_docs=true&key=%22a%40a.com%22 200 ok 19
[notice] 2017-04-10T08:55:35.014278Z couchdb@localhost <0.27086.27> b30df4c41a localhost:5984 <IP_ADDRESS> undefined GET /sl-users/a%40a.com? 200 ok 1
[notice] 2017-04-10T08:55:35.047596Z couchdb@localhost <0.27074.27> 4f0fa3bcdb localhost:5984 <IP_ADDRESS> undefined POST /_users/_bulk_docs 201 ok 30
[notice] 2017-04-10T08:55:35.054389Z couchdb@localhost <0.27067.27> 4665acf960 localhost:5984 <IP_ADDRESS> undefined GET /vanilla%24a(40)a(2e)com/ 200 ok 1
[notice] 2017-04-10T08:55:35.054410Z couchdb@localhost <0.27085.27> 9543e4ba02 localhost:5984 <IP_ADDRESS> undefined GET /product/ 200 ok 1
[notice] 2017-04-10T08:55:35.060044Z couchdb@localhost <0.27084.27> c4175a0c37 localhost:5984 <IP_ADDRESS> undefined GET /vanilla%24a(40)a(2e)com/_security? 200 ok 1
[notice] 2017-04-10T08:55:35.060851Z couchdb@localhost <0.27097.27> 90e12ea4ae localhost:5984 <IP_ADDRESS> undefined GET /product/_security? 200 ok 1
[notice] 2017-04-10T08:55:35.141181Z couchdb@localhost <0.27083.27> f3e2b2d819 localhost:5984 <IP_ADDRESS> undefined PUT /vanilla%24a(40)a(2e)com/_security 200 ok 77
[notice] 2017-04-10T08:55:35.150234Z couchdb@localhost <0.27096.27> 508acbb022 localhost:5984 <IP_ADDRESS> undefined PUT /product/_security 200 ok 85
[notice] 2017-04-10T08:55:35.185608Z couchdb@localhost <0.27082.27> 5451a658ae localhost:5984 <IP_ADDRESS> undefined POST /sl-users/_bulk_docs 201 ok 32
[notice] 2017-04-10T08:55:36.272372Z couchdb@localhost <0.27066.27> e464eac4a8 087deddc.ngrok.io <IP_ADDRESS> aQD-rrhATXC3kbeTcQ6iXA GET /vanilla%24a(40)a(2e)com/ 200 ok 2
Thanks,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
in both calls to sync you need to use the remote pouchdb instance, not just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call to sync but the gist of it is that since encryption is done locally, you can't refer to the database by name, because then pouchdb will internally create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Safari runs all the encryption in JavaScript in the main thread which is
probably what's so slow, if you can out all the database stuff in a web
worker that would probably be your best bet
On Mon, Apr 10, 2017, 5:57 AM Bill Noble<EMAIL_ADDRESS>wrote:

Hi Calvin,
I forgot to add the CouchDB log to my previous email:
[info] 2017-04-10T08:55:34.993290Z couchdb@localhost <0.18233.0> --------
Starting index update for db: shards/80000000-ffffffff/sl-users.1491582997
idx: _design/auth
[info] 2017-04-10T08:55:35.009309Z couchdb@localhost <0.18233.0> --------
Index update finished for db: shards/80000000-ffffffff/sl-users.1491582997
idx: _design/auth
[notice] 2017-04-10T08:55:35.010439Z couchdb@localhost <0.27087.27>
e3747ff4ce localhost:5984 <IP_ADDRESS> undefined GET
/sl-users/_design/auth/_view/emailUsername?include_docs=true&key=%22a%
40a.com%22 200 ok 19
[notice] 2017-04-10T08:55:35.014278Z couchdb@localhost <0.27086.27>
b30df4c41a localhost:5984 <IP_ADDRESS> undefined GET /sl-users/a%40a.com?
200 ok 1
[notice] 2017-04-10T08:55:35.047596Z couchdb@localhost <0.27074.27>
4f0fa3bcdb localhost:5984 <IP_ADDRESS> undefined POST /_users/_bulk_docs 201
ok 30
[notice] 2017-04-10T08:55:35.054389Z couchdb@localhost <0.27067.27>
4665acf960 localhost:5984 <IP_ADDRESS> undefined GET /vanilla%24a(40)a(2e)com/
200 ok 1
[notice] 2017-04-10T08:55:35.054410Z couchdb@localhost <0.27085.27>
9543e4ba02 localhost:5984 <IP_ADDRESS> undefined GET /product/ 200 ok 1
[notice] 2017-04-10T08:55:35.060044Z couchdb@localhost <0.27084.27>
c4175a0c37 localhost:5984 <IP_ADDRESS> undefined GET
/vanilla%24a(40)a(2e)com/_security? 200 ok 1
[notice] 2017-04-10T08:55:35.060851Z couchdb@localhost <0.27097.27>
90e12ea4ae localhost:5984 <IP_ADDRESS> undefined GET /product/_security? 200
ok 1
[notice] 2017-04-10T08:55:35.141181Z couchdb@localhost <0.27083.27>
f3e2b2d819 localhost:5984 <IP_ADDRESS> undefined PUT
/vanilla%24a(40)a(2e)com/_security 200 ok 77
[notice] 2017-04-10T08:55:35.150234Z couchdb@localhost <0.27096.27>
508acbb022 localhost:5984 <IP_ADDRESS> undefined PUT /product/_security 200 ok
85
[notice] 2017-04-10T08:55:35.185608Z couchdb@localhost <0.27082.27>
5451a658ae localhost:5984 <IP_ADDRESS> undefined POST /sl-users/_bulk_docs 201
ok 32
[notice] 2017-04-10T08:55:36.272372Z couchdb@localhost <0.27066.27>
e464eac4a8 087deddc.ngrok.io <IP_ADDRESS> aQD-rrhATXC3kbeTcQ6iXA GET
/vanilla%24a(40)a(2e)com/ 200 ok 2
Thanks,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
in both calls to sync you need to use the remote pouchdb instance, not
just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call
to sync but the gist of it is that since encryption is done locally, you
can't refer to the database by name, because then pouchdb will internally
create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292903962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4n6VAII95DcyBJKArkXE_Wkh-7Jx7ks5ruf0cgaJpZM4M0w0i
.

Many thanks for the speedy response Calvin!
For my test case I only have two small documents that I need to sync and the current timeout is @10mins which seems rather long for what is being encrypted/decrypted?
My App needs to wait for the PouchDB to finish syncing before it can proceed as it needs to show the data. If I use a web worker will this significantly speed things up? I am wondering why this takes seconds when running the App in a web browser and over 10 mins, and timing out, when running on an iPhone?
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 11:40
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
Safari runs all the encryption in JavaScript in the main thread which is
probably what's so slow, if you can out all the database stuff in a web
worker that would probably be your best bet
On Mon, Apr 10, 2017, 5:57 AM Bill Noble<EMAIL_ADDRESS>wrote:

Hi Calvin,
I forgot to add the CouchDB log to my previous email:
[info] 2017-04-10T08:55:34.993290Z couchdb@localhost <0.18233.0> --------
Starting index update for db: shards/80000000-ffffffff/sl-users.1491582997
idx: _design/auth
[info] 2017-04-10T08:55:35.009309Z couchdb@localhost <0.18233.0> --------
Index update finished for db: shards/80000000-ffffffff/sl-users.1491582997
idx: _design/auth
[notice] 2017-04-10T08:55:35.010439Z couchdb@localhost <0.27087.27>
e3747ff4ce localhost:5984 <IP_ADDRESS> undefined GET
/sl-users/_design/auth/_view/emailUsername?include_docs=true&key=%22a%
40a.com%22 200 ok 19
[notice] 2017-04-10T08:55:35.014278Z couchdb@localhost <0.27086.27>
b30df4c41a localhost:5984 <IP_ADDRESS> undefined GET /sl-users/a%40a.com?
200 ok 1
[notice] 2017-04-10T08:55:35.047596Z couchdb@localhost <0.27074.27>
4f0fa3bcdb localhost:5984 <IP_ADDRESS> undefined POST /_users/_bulk_docs 201
ok 30
[notice] 2017-04-10T08:55:35.054389Z couchdb@localhost <0.27067.27>
4665acf960 localhost:5984 <IP_ADDRESS> undefined GET /vanilla%24a(40)a(2e)com/
200 ok 1
[notice] 2017-04-10T08:55:35.054410Z couchdb@localhost <0.27085.27>
9543e4ba02 localhost:5984 <IP_ADDRESS> undefined GET /product/ 200 ok 1
[notice] 2017-04-10T08:55:35.060044Z couchdb@localhost <0.27084.27>
c4175a0c37 localhost:5984 <IP_ADDRESS> undefined GET
/vanilla%24a(40)a(2e)com/_security? 200 ok 1
[notice] 2017-04-10T08:55:35.060851Z couchdb@localhost <0.27097.27>
90e12ea4ae localhost:5984 <IP_ADDRESS> undefined GET /product/_security? 200
ok 1
[notice] 2017-04-10T08:55:35.141181Z couchdb@localhost <0.27083.27>
f3e2b2d819 localhost:5984 <IP_ADDRESS> undefined PUT
/vanilla%24a(40)a(2e)com/_security 200 ok 77
[notice] 2017-04-10T08:55:35.150234Z couchdb@localhost <0.27096.27>
508acbb022 localhost:5984 <IP_ADDRESS> undefined PUT /product/_security 200 ok
85
[notice] 2017-04-10T08:55:35.185608Z couchdb@localhost <0.27082.27>
5451a658ae localhost:5984 <IP_ADDRESS> undefined POST /sl-users/_bulk_docs 201
ok 32
[notice] 2017-04-10T08:55:36.272372Z couchdb@localhost <0.27066.27>
e464eac4a8 087deddc.ngrok.io <IP_ADDRESS> aQD-rrhATXC3kbeTcQ6iXA GET
/vanilla%24a(40)a(2e)com/ 200 ok 2
Thanks,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Friday, 7 April 2017 at 15:15
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
in both calls to sync you need to use the remote pouchdb instance, not
just the name of the remote database,
this._DB.sync(this._remote)
you may need to save this._remote in a variable for use in the second call
to sync but the gist of it is that since encryption is done locally, you
can't refer to the database by name, because then pouchdb will internally
create a new instance that won't have encryption associated with it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292903962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4n6VAII95DcyBJKArkXE_Wkh-7Jx7ks5ruf0cgaJpZM4M0w0i
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

you should check for errors on the outer sync, or try to read a document, it could be something is wrong there, otherwise it could be similar to #53 where it takes a long time to run the key derivation (this is where it turns the human readable password into the machine readable encryption key, this intentionally takes some time in order to prevent brute force hacking the key)

Thanks Calvin.
I am still not sure why it is just seconds when running the App on a browser and timesout at 10 minutes when running on an iPhone. It is on the outer sync that the timeout occurs.
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 13:36
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
you should check for errors on the outer sync, or try to read a document, it could be something is wrong there, otherwise it could be similar to #53 where it takes a long time to run the key derivation (this is where it turns the human readable password into the machine readable encryption key, this intentionally takes some time in order to prevent brute force hacking the key)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

in chrome and firefox there are built in encryption functions that are very
efficient, in safari, both desktop and mobile there are not so any
encryption is done in javascript which is much less efficient, in addition,
your desktop computer is probably more powerful then your phone.  Lastly
there is a chance that it isn't actually taking 10 minutes but that there
is an error and it's not taking ten minutes, it's just errored out and will
never finish, you just don't notice until it times out
On Mon, Apr 10, 2017 at 8:42 AM Bill Noble<EMAIL_ADDRESS>wrote:

Thanks Calvin.
I am still not sure why it is just seconds when running the App on a
browser and timesout at 10 minutes when running on an iPhone. It is on the
outer sync that the timeout occurs.
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 13:36
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
you should check for errors on the outer sync, or try to read a document,
it could be something is wrong there, otherwise it could be similar to #53
where it takes a long time to run the key derivation (this is where it
turns the human readable password into the machine readable encryption key,
this intentionally takes some time in order to prevent brute force hacking
the key)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292937834,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4nxTNxSNFXm_63aVgeGXKCOYHaXCoks5ruiOkgaJpZM4M0w0i
.

That makes perfect sense Calvin thanks. I think for my needs I need to use a faster encryption option. I really appreciate you taking the time to help with this though thanks.
Cheers,
Bill
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 13:53
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
in chrome and firefox there are built in encryption functions that are very
efficient, in safari, both desktop and mobile there are not so any
encryption is done in javascript which is much less efficient, in addition,
your desktop computer is probably more powerful then your phone. Lastly
there is a chance that it isn't actually taking 10 minutes but that there
is an error and it's not taking ten minutes, it's just errored out and will
never finish, you just don't notice until it times out
On Mon, Apr 10, 2017 at 8:42 AM Bill Noble<EMAIL_ADDRESS>wrote:

Thanks Calvin.
I am still not sure why it is just seconds when running the App on a
browser and timesout at 10 minutes when running on an iPhone. It is on the
outer sync that the timeout occurs.
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 13:36
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted
(#54)
you should check for errors on the outer sync, or try to read a document,
it could be something is wrong there, otherwise it could be similar to #53
where it takes a long time to run the key derivation (this is where it
turns the human readable password into the machine readable encryption key,
this intentionally takes some time in order to prevent brute force hacking
the key)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/calvinmetcalf/crypto-pouch/issues/54#issuecomment-292937834,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABE4nxTNxSNFXm_63aVgeGXKCOYHaXCoks5ruiOkgaJpZM4M0w0i
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

so a couple things you can try:

deriving the keys ahead of time like mentioned #53
pass the iterations option to reduce how long it takes to derive the keys, the default is 100000 but you can change that to something smaller, can also be used to test to make sure that's actually whats slow, if you change it to 5 and it's still slow then that's probably not the issue.
you can pass the algorithm option set to chacha20, this will use a different encryption scheme that should be faster then the default when they both run in JavaScript (though it's slower then the default when the default runs natively).

Thanks Calvin I will give those a try.
From: Calvin Metcalf<EMAIL_ADDRESS>Reply-To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Date: Monday, 10 April 2017 at 14:06
To: calvinmetcalf/crypto-pouch<EMAIL_ADDRESS>Cc: Bill Noble<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [calvinmetcalf/crypto-pouch] Remote CouchDB is not encrypted (#54)
so a couple things you can try:
deriving the keys ahead of time like mentioned #53
pass the iterations option to reduce how long it takes to derive the keys, the default is 100000 but you can change that to something smaller, can also be used to test to make sure that's actually whats slow, if you change it to 5 and it's still slow then that's probably not the issue.
you can pass the algorithm option set to chacha20, this will use a different encryption scheme that should be faster then the default when they both run in JavaScript (though it's slower then the default when the default runs natively).
—You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Add requirements to imageSpec

TL;DR
Add support passing requirements.txt to image spec.
Type

[ ] Bug Fix
[x] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
Example
from flytekit import task, workflow, ImageSpec

new_flytekit = "git+https://github.com/flyteorg/flytekit@01f2d63221467face8d35ed0af5308bbb3e66eaa"
image = ImageSpec(registry="pingsutw", packages=["pandas", new_flytekit], apt_packages=["git"], requirements="requirement.txt")

@task(container_image=image)
def bar():
    print("hello")
    print("hello")

@workflow
def wf() -> str:
    bar()
    return "hi"

if __name__ == '__main__':
    wf()

Tracking Issue
NA
Follow-up issue
NA

is this ready to go @pingsutw ?


Update get started developing guide

Jira link
hmrc-335
What?
I have added/removed/altered:

[x] Guide on getting up and running with pgadmin
[x] Guidance on downloading packages from homebrew
[x] Directing to docker docs for docker-compose up flags

Why?
I am doing this because:

I think these are important additions to the get started developing guide for new joiners, so they're onboarding is easier

Have you? (optional)

[ ] Pair reviewed this change

Agree with the pgadmin comment

LGTM although I'd remove the requirement for pgadmin - psql is a tool that everyone has and does what you need. If people want to use pgadmin then they can

I have a note there saying you can use whatever database management tool you'd like but that's the one we mostly use. I reckon using some UI is quite standard for most devs

Regardless, I don't want new developers to think we are forcing them down one particular route.

Regardless, I don't want new developers to think we are forcing them down one particular route.

Ok, should we have a call to decide then?


Iterate/improve ValueSets table UI

UX feedback:

[ ] The many-to-one relationship of data needs to be reflected in UI.
[ ] If there is some change history that's available surface it in the UI.

[Direct edit this ticket to clarify]

@jimduff-usds @bgantick @sethdarragile6

For reference, we already have some work planned on this feature: https://app.zenhub.com/workspaces/experience-607d9d5e68b95200150fec37/issues/cdcgov/prime-reportstream/3489. The current version is very much an MVP and isn't really in use yet (we had other priorities come up). cc @RakshindaAslam

@fmccaf1 @drew-usds @CnoteNava For consideration in upcoming sprints, and requires mockups and UX decisions :)

This is old ticket and needs refinement. O+O will contact us about improvements like these in the future if they are high priority.


Port HTML report to chart.js

Originally intended to close #213, has turned into a rewrite using chart.js.
fibber.html
report.html
Feedback, criticism, feature requests welcome!
To do

[x] Report/template

[x] better error bars
[x] Colophon
[x] Explanation
[x] Outlier info
[x] test with different data sets
[x] ~(optional?) legends~
[x] strut 0-width bars in overview
[x] fix tooltip on scatter plot
[x] ~(collapsible) table of contents?~
[x] general code cleanup
[x] git history cleanup
[x] make titles links
[x] things are too bold
[x] link style more subtle
[x] overview fixed height, responsive width
[x] stack KDE and scatter at small widths
[x] make tick display more robust
[x] move example chart to www/report.html
[x] match precision with old report
[x] change cursor when hovering y-axis
[x] change sort order

[x] interface?
[x] report order
[x] alphabetical order
[x] lacitebahpla order
[x] time order

[x] Haskell+packaging

[x] Drop js-flot, js-jquery
[x] Include chart.js

[x] make js-chart?
[x] EMBED flag

[x] split out custom JS+CSS?

@considerate @cartazio

Hooray! It looks like this is still a draft, so let me know when this is ready for a full review.

This is ready for review. The fact that it's one commit is a bit unfortunate, but since most work happened in files that have since disappeared there is not really a nice way to make it more atomic.

Awesome! I'll do a more thorough review of this patch over the weekend, but a quick before-and-after comparison makes me notice two things right off the bat:

The legends appear to be missing after your changes. Is this intentional?
After your changes, it is no longer possible to manually zoom in/out or pan horizontally on the overview chart. I'm guessing this is because this functionality has been replaced by the ability to click on an individual benchmark and have the chart automatically zoom in/out accordingly? If so, perhaps that's a reasonable justification for removing this functionality, although I wonder if certain people relied on this.

Some other random thoughts:

There are some not-so-obviously discoverable features in the new chart engine, such as the ability to toggle between linear and log scales by clicking on the x-axis. I wonder if it would be worth describing these features in prose somewhere on the HTML page itself?
I'm also not clear on what some of the new drop-down menu options (e.g., colex) mean. These might be worth spelling out.
I'm not sure if it's just my web browser, but trying to hover my mouse over the yellow trendlines in the per-benchmark reports feels a lot more finicky after your changes, in the sense that the timing information in a tooltip won't appear unless my mouse is directly over the line. In contrast, with today's criterion, your mouse doesn't have to be directly over the trendline in order for the tooltip to appear; as long as it is close enough to the trendline, the tooltip will surface. Is it possible to increase the "hitbox" of the trendline so that it mimics the behavior of today's criterion?

The legends appear to be missing after your changes. Is this intentional?

This was mostly because they seemed to duplicate information already present on the same screen. It's fairly trivial to add them back in, would you like me to?

After your changes, it is no longer possible to manually zoom in/out or pan horizontally on the overview chart.

Yeah, zooming was left out intentionally. I figured the primary use case was to see bars that got dwarfed by others, which I felt the click-to-focus was a better interface for. It's possible to do zooming, but it also requires a disproportionate amount of code to get right. What do you think?

There are some not-so-obviously discoverable features in the new chart engine, such as the ability to toggle between linear and log scales by clicking on the x-axis.

Makes sense, I added a note.

I'm also not clear on what some of the new drop-down menu options (e.g., colex) mean. These might be worth spelling out.

Both lexical and colex sort by group, with lexical the head is most significant and with colex the tail is most significant (i.e. big-endian vs little-endian). I suppose "by group" would be a better name for lexical, do you know a good, self-evident name for colex?

trying to hover my mouse over the yellow trendlines in the per-benchmark reports feels a lot more finicky after your changes

414911b expanded the radius, do you think it's OK now?

@RyanGlScott Note that report.html haven’t been regenerated with the updated radius for the hit boxes yet so to test the tooltips you would have to generate a new report.

@jonascarpay @RyanGlScott I added titles to the kernel density estimates and scatter plots to provide the same amount of context as the previous version for those plots.
I think it might somewhat redundant and would personally like to keep the output as free from noise as possible. What do you think?

I've timed out for now, but this is looking very promising. More review to come later!

Is there a preview link that works on mobile? :))

Alright, that should do it for the feedback up to now.
@cartazio Refined Github gives me this preview link, I think that should work. It does look like on mobile part of the overview chart gets eaten, will look into that.

Hrmm. One good thing to test is having an example plot where there’s a huge
range of magnitudes of values.
Def like how this one looks so far.
One issue with the previous was some sort of huge algorithmic penalty for
actually rendering some of the plots. Was that part of the issues with the
old flot code?
On Fri, Nov 13, 2020 at 9:16 AM Jonas Carpay<EMAIL_ADDRESS>wrote:

Alright, that should do it for the feedback up to now.
@cartazio https://github.com/cartazio Refined Github gives me this
https://refined-github-html-preview.kidonng.workers.dev/jonascarpay/criterion/raw/master/www/report.html
preview link, I think that should work. It does look like on mobile part of
the overview chart gets eaten, will look into that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bos/criterion/pull/229#issuecomment-726821227, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABBQWG7D5GEHPXBYTXJRLSPVEUJANCNFSM4TNDRVBQ
.

One issue with the previous was some sort of huge algorithmic penalty for
actually rendering some of the plots. Was that part of the issues with the
old flot code?

Without running any proper benchmarks (ironic, isn't it?) generating the same charts took much longer with flot than with chart.js. That said, it still takes a non-trivial amount of time to render the benchmark link posted above on mobile.

colexical order sorts groups in descending alphabetical order

I'm not sure I agree that this description is correct, the groups are sorted in ascending alphabetical order but we split the report name on '/' and compare the two right-most group elements first with  (the default) ascending alphabetical string comparison, then if equal the group elements to the left of those are compared until one of the elements are unequal. If all elements are equal the shorter array is sorted before the longer one.

This image provides a visualization of the lexical and colexical orderings. The right-most 1s are placed before the right-most 5s in colexical order.

OK. My attempts at succinctly characterizing this have failed, so perhaps it's best that I stop trying. :)
In that case, I'm content with the current description, but include a link to the Wikipedia article on lexicographic ordering you mentioned above, since that would be valuable context to have.

One last request: do you mind writing up a description of the changes from this patch in the changelog.md file? There have been quite a few changes (both in terms of new functionality and UI differences), and it would be good to advertise them prominently. Thanks!

One last request: do you mind writing up a description of the changes from this patch in the changelog.md file? There have been quite a few changes (both in terms of new functionality and UI differences), and it would be good to advertise them prominently. Thanks!

Yes, of course, I'll co-ordinate with @jonascarpay and describe the changes in the changelog either today or tomorrow.

Alright, that's the Changelog. Do you want me to squash the history?

Alright, that's the Changelog. Do you want me to squash the history?

Please do. And thanks again for all of your combined work on this!

Done, thank you as well!

I’m so excited about this. Thank you all so much!
On Sat, Nov 14, 2020 at 12:03 PM Jonas Carpay<EMAIL_ADDRESS>wrote:

Done, thank you as well!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bos/criterion/pull/229#issuecomment-727243944, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAABBQVEFUDV7CTZKXR72U3SP3BA5ANCNFSM4TNDRVBQ
.

I've uploaded criterion-<IP_ADDRESS> to Hackage with these changes.


investigate better method for calculating % completion

Multiple requests can set the progress bar to go backwards due to the number of requests without responses being added to the total.  Perhaps an easier way to deal with this is by simply not allowing the progress bar to go backwards

I've having a hard time getting this to work for slow uploads, without the bar jumping backwards multiple times. Is there a way we can disable the automatic calls to cfpLoadingBar.inc() on calling cfpLoadingBar.start();? Or at least be able to specify how frequently it should be called?

@kahboom I created pull request #209 to make auto-increment optional. This should allow you to increment at your own interval.


Bring back success and failure

Hey,
i notices that you changes the function signature of .POST, .GET, ... to only only have completionHandler instead of two seperate params. In my opinion this makes it harder to write large apps because in my projects i have a global failure function that is always passed in as a reference from my base controller. I used to only specify the success function in my view controller.
Now i have to always check if its successfull and call the failure function manually. If you have 14 parts in your app that do HTTP requests, you have to write a lot of dublicated code.
Greetings
Leo

Hey Leo,
I thought this might be a more controversial change :smile:. I believe the completionHandler is a better setup for a few reasons.

It is more idiomatic Swift. Swift favors a single trailing closure on the methods and allows for a bit simpler code.
It encourages handling the error case. It is simple to just ignore the failure closure and thus that code path. Using the single closure gives a single non optional HTTPResponse type and allows you to just check the error to see if the request failed, then handle the response object.
Did I mention it cleaned up the syntax :smile:

For the issue you mentioned with a global failure function, I think that might be able to be accomplished with something like so:
var request = HTTPTask()
request.GET("http://vluxe.io", parameters: nil, completionHandler: {(response: HTTPResponse) in
	if failureHandler(response) { //failureHandler is your global closure to handle failure processing
		//if failure handler returns true, all is good and do your success work
	}
})

//example global failure closure
func failureHandler(response: HTTPResponse) -> Bool {
	if let err = response.error {
        println("error: \(err.localizedDescription)")
        //.... show UI alert or whatever the default handling is
        return false //failed we got an error
    }
    return true
}

Any questions, let me know.

Hmm, i still dont think it's a change for the better, i am going to run with it. Thanks for explaining anyway :)


DRAFT ONLY collect kernel stderr

For #

[ ] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR).
[ ] Title summarizes what is changing.
[ ] Has a news entry file (remember to thank yourself!).
[ ] Appropriate comments and documentation strings in the code.
[ ] Has sufficient logging.
[ ] Has telemetry for enhancements.
[ ] Unit tests & system/integration tests are added/updated.
[ ] Test plan is updated as appropriate.
[ ] package-lock.json has been regenerated by running npm install (if dependencies have changed).
[ ] The wiki is updated with any design decisions/details.

Codecov Report

Merging #13443 into master will decrease coverage by 30.56%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           master   #13443       +/-   ##
===========================================
- Coverage   59.92%   29.36%   -30.57%
===========================================
  Files         670      341      -329
  Lines       37211    16637    -20574
  Branches     5311     2285     -3026
===========================================
- Hits        22300     4885    -17415
+ Misses      13774    11715     -2059
+ Partials     1137       37     -1100

Impacted Files
Coverage Δ

src/client/common/process/baseDaemon.ts
6.62% <0.00%> (-15.67%)
:arrow_down:

src/client/debugger/constants.ts
0.00% <0.00%> (-100.00%)
:arrow_down:

src/client/datascience/notebook/constants.ts
0.00% <0.00%> (-100.00%)
:arrow_down:

src/client/interpreter/autoSelection/constants.ts
0.00% <0.00%> (-100.00%)
:arrow_down:

src/client/language/braceCounter.ts
5.40% <0.00%> (-94.60%)
:arrow_down:

src/client/language/textBuilder.ts
8.69% <0.00%> (-91.31%)
:arrow_down:

src/client/language/characterStream.ts
4.41% <0.00%> (-89.71%)
:arrow_down:

src/client/language/tokenizer.ts
4.08% <0.00%> (-89.22%)
:arrow_down:

...onments/discovery/locators/services/condaHelper.ts
7.69% <0.00%> (-87.18%)
:arrow_down:

src/datascience-ui/common/index.ts
9.40% <0.00%> (-86.33%)
:arrow_down:

... and 554 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f47defc...5466f9f. Read the comment docs.


Not sure which json feed file errored out

I have 80K records which are in several files (20 records/ file). I have created FeedMe url like this
http://www.example.com/1.json
I have written a shell script which copies the file from a folder puts in the root of the site, lets FeedMe process the file and the moves that file to a different folder. The issue is sometimes I see in TaskManager plugin (tasks table in craft database), few files failed and I have no idea which failed because every file that is being processed is 1.json.
Is there a way to modify the logging so I know which file failed?

It really depends on what the error being thrown is, but I suppose it could record the node that its trying to parse. Would that be helpful? Not being familiar with your setup, and whether they contain the same nodes, etc.

I see there are about 5 feedme.log files in runtime folder. I opened up first file and the first line in that file is
<date/ time> [info] [plugin] [Forced] FeedMeError - .
This is followed by content blocks. I don't know if these content blocks are the ones that failed or not.

I recon they are. Currently (depending on the error), it'll either provide the error message returned (entry couldn't be saved due to required missing fields, etc), or report the entry it failed on. This all depends on what duplication handling you're doing.
Whats one of the content blocks in the error for example?

[   {     "ContentID": 110924,     "Id": 73518,     "Title": "Despite being technically impossible, underwater breathing ‘device’ receives $830,000 in crowdsourced funding",     "CorrectedTitle": "Despite being technically impossible, underwater breathing ‘device’ receives $830,000 in crowdsourced funding",     "Slug": "despite-being-technically-impossible-underwater-breathing-device-receives-830-000-in-crowdsourced-funding",     "Parent": "",     "Ancestors": "",     "Author": "bauschje01",     "Post Date": "2016-03-30 09:30:00",     "Expiry Date": null,     "Enabled": "Yes",     "Status": "live",     "Summary": "Despite mounting evidence that this device is not feasible, its online funding campaign is closing in on the one million-dollar mark",     "Sub Title": "Gadget is being lauded by some, criticized by many",     "Contents": "
An underwater breathing device, widely criticized by experts in both the scientific and technical communities, has just surpassed the $830,000 benchmark in contributions made via crowdsourcing site Indiegogo.

Referred to as “Triton”, the mouth-based device is marketed as having artificial gills capable of providing its user with enough oxygen for up to 45 minutes of underwater swim time, at a maximum dive depth of 15 feet. It is able to accomplish this by way of a microporous filter that has tiny holes in it, which are small enough to stop water from getting in, but large enough to allow for the dissolved, “free-floating” oxygen to enter.

The oxygen is next compressed and transferred over to an on-board storage tank, a process powered by a supercharged lithium-ion battery.

Now for the reality: this device, and all of its claims, is impossible for several reasons. For one, it’s small — Triton measures just 11.5 inches in length. If it’s capable of filtering out oxygen from the water — enough for a diver to not drown — it would have to be filtering approximately 24 gallons of water every minute. What’s more, it would need to be operating at 100% efficiency to avoid any risk of accidental choking and / or shortness of breath.

Additionally, this much water coming through the filter system is equivalent to, more or less, six garden hoses’ worth of pressure. To sustain this constant level of flow, the swimmer would have to be moving through the water at exceptionally fast speeds.

So, that’s just the filtering process. In terms of the actual technology, being able to compress oxygen requires a lot of energy. Andrew David Thaler, a deep-sea ecologist, spoke to Business Insider about it and explained that a battery capable of supporting this type of system would need to be so powerful, it isn’t actually available on the market yet.

Also, breathing in pure, pressurized oxygen is hazardous to one’s health — too much of it can cause severe damage to the user’s respiratory and central nervous systems.

There has been a theory floated out there that perhaps the device is actually electrolyzing the water to produce usable oxygen. This technology already exists, so it’s plausible — but only as a theory. You see, to support a system that electrolyzes that much water to produce enough oxygen to support a diver underwater for 45 minutes would require a ton of electrical energy — about a nuclear reactor’s worth (or so).

All of this is nothing new to the Triton crew — they’ve been hearing these criticisms for the last two years since the technology was first introduced. None of it has deterred them, though, as they went ahead and built a prototype and quickly put it on Indiegogo to see if they could win the public’s support.

And you know what? They got it, having already reached the team’s funding goal of $50,000, and in fact, already surpassing the $830,000 mark at the time this article’s being written.

That’s 2,200+ people who’ve gotten behind this idea, all scientific and technical evidence aside.

To check out the campaign for yourself, head to the Triton page .

Via iflscience.com
",     "Part Numbers": "",     "Print": "0",     "Print Issue Month Year": "03/2016",     "Meta Tags": "biotechnology-research,triton-underwater-breathing-device",     "ArticleType": "Featured Articles",     "Primary Article Classification": "Wearables",     "Companies": "Electronic Products Magazine",     "Primary End Markets": "Science/ Research",     "Image Thumbnail": "fajb triton 02 mar2016",     "HotTopic": "0",     "DownloadList": [       {         "DownloadType": "",         "DownloadValue": ""       }     ]   }
This is one from feedme.log file

Although that shows the node that an error was thrown on, its quite odd that I can't find the code that outputs that. I wonder if it was generated by an older version of Feed Me?

What version of FeedMe would that be? Currently I am using 1.4.11. I still haven't updated it to the latest one

Yeah, thats pretty much the latest. Anyway - maybe you could add an item to your feed thats a unique identifier (number, date, etc) that you could map to keep track of what file its processing on?

I could add a field, just file number would do fine as all the files are unique anyway.

Closing for the moment due to inactivity. Re-open if you have more issues.


Add keyboard support

Please add the support for [Enter] key while navigating the left tree like panel with classes (= double click by mouse)
While cursor is in Code panel (java source) please add support for CTRL + F4 key combination (to close the current tab

Thanks

https://github.com/deathmarine/Luyten/blob/master/src/us/deathmarine/luyten/MainMenuBar.java#L121-L232
I have accelerators for most keyboard commands Close is Ctrl + W closes the open doc as opposed to the reverse which in most cases Ctrl + F4 is normally close and Ctrl + W is close tab. I will most likely take that route.
Ref: http://en.wikipedia.org/wiki/Table_of_keyboard_shortcuts
As for [Enter] to open a selected tree item, I concur.

Hi
Thanks, agree with 1st part, just need a fast combination to close 10-th, 100th of opened tabs...
on 2) The idea was to have a fast mean to navigation 100-th of java files without clicking the mouse button, so using [Enter] on selected tree item come as the 1st idea to my mind
If there are other combinations- that will be fine as well, the final goal is user convenience of the Luyten tool usage by keyboard only!

Menu item File - Recent files is always disabled (grayed out) - is it a feature to be implemented?
usually thats also a level of user-friendliness

Another idea, maybe worth tracking by a separate  issue - Advance the classes navigation, yeah. I'll open up a new one right now...

#62 Closes this.


Add enable_model_warmup flag for AOT compilation at model server start

Add the enable_model_warmup flag at model server start
Associated PR: https://github.com/google/JetStream/pull/92
 - model_name=gemma-7b
  - tokenizer_path=assets/tokenizer.gemma
  - per_device_batch_size=1
  - max_prefill_predict_length=1024
  - max_target_length=2048
  - async_checkpointing=false
  - ici_fsdp_parallelism=1
  - ici_autoregressive_parallelism=-1
  - ici_tensor_parallelism=1
  - scan_layers=false
  - weight_dtype=bfloat16
  - load_parameters_path=<ckpt_path>
  - enable_model_warmup=true

curl --request POST --header "Content-type: application/json" -s localhost:8000/generate --data '{
    "prompt": "What are the top 5 programming languages",
    "max_tokens": 200
}'
{
    "response": " for data science in 2023?\n\n1. Python\n2. R\n3. SQL\n4. Java\n5. Scala\n\n**Note:** The order is based on popularity and demand in the data science industry in 2023."
}

PR is dupe of https://github.com/google/maxtext/pull/764, which failed checks because it was a PR from a forked repo

Hey @gobbleturk , this PR is a dupe of https://github.com/google/maxtext/pull/764 which failed checks because it was a PR from a forked repo, once this PR is in I will close the other one, thanks!

Hey @gobbleturk , this PR is a dupe of #764 which failed checks because it was a PR from a forked repo, once this PR is in I will close the other one, thanks!

Sorry for not seeing earlier, please add me as an assignee which is what I check https://screenshot.googleplex.com/5nTrBzS4v4hfpgq


grpc transport is failing

Few issues I have found :

import snappi v0.7.7  is failing
ImportError while loading conftest '/home/keysight/athena/tests/py/conftest.py'.
py/conftest.py:2: in
import snappi
/usr/local/lib/python3.8/dist-packages/snappi/init.py:1: in
from .snappi import Config
/usr/local/lib/python3.8/dist-packages/snappi/snappi.py:13: in
import grpc
workaround I did, locally I have installed grpcio, grpcio-tools which should be inside the dependency of snappy
all set & control operations which returns warning failing:
res = api.set_config(cfg)
res.warnings is NoneType, if there is no warning it should return empty list
mismatch of return type for get_capture api
for http_transport its returning _io.BytesIO but for grpc_transport its returning string which is not expected.

fixed in 0.7.13


Error escaping message unpacking

What is wrong?
   ERROR  09-05 09:51:49               asyncio  Exception in callback UDPTransport._on_read_ready
handle: <Handle UDPTransport._on_read_ready>
Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 69, in uvloop.loop.Handle._run
  File "uvloop/handles/udp.pyx", line 64, in uvloop.loop.UDPTransport._on_read_ready
  File "/home/ubuntu/trinity/p2p/discovery.py", line 448, in datagram_received
    self.receive(address, cast(bytes, data))
  File "/home/ubuntu/trinity/p2p/discovery.py", line 463, in receive
    remote_pubkey, cmd_id, payload, message_hash = _unpack_v4(message)
  File "/home/ubuntu/trinity/p2p/discovery.py", line 1261, in _unpack_v4
    cmd = CMD_ID_MAP[cmd_id]
KeyError: 9

How can it be fixed
:man_shrugging:

This got fixed with #1063


Update editor.md

Added alert to warn against multiple instances of an editor open at once.
Description of the change:  Added alert to warn against multiple instances of an editor open at once.
Reason for the change:  Improve customer experience
Link to original source:  https://sendgrid.com/docs/ui/sending-email/editor/

Closes #

Hello @drespinoza,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


docs: add a Semver document

Resolved issues:
resolves #434
Description of changes:
We're mostly doing semantic versioning already, put a document in the repo to make it clear.
Call-outs:
Copied from another Crypto Tools project.  The comments in this document around API changes in beta releases need review.
Testing:
We increment the version number in our weekly releases.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #2268 into main will increase coverage by 42.13%.
The diff coverage is 83.33%.

@@             Coverage Diff             @@
##             main    #2268       +/-   ##
===========================================
+ Coverage   38.57%   80.71%   +42.13%
===========================================
  Files         164      265      +101
  Lines       12626    17874     +5248
===========================================
+ Hits         4871    14427     +9556
+ Misses       7755     3447     -4308


Cargo.lock package order is unstable across different cargo versions

Problem
When a lockfile contains multiple entries with the same package name and version, but different sources, the order of these packages in the lockfile changes between builds. (Even though the dependencies are exactly the same.)
Since our project commits the lockfile to git, these redundant changes produce commit noise, checkout conflicts, and merge conflicts.
Steps

Stable to nightly:

Build https://github.com/ZcashFoundation/zebra/commit/6bb5220b13d5dce210045bdafdb7ac9e7db7c369 with nightly cargo
Run git diff

Nightly to stable:

Build https://github.com/ZcashFoundation/zebra/commit/64662a758db5bfe0edb86a751aafd04aa0081faa with stable cargo
Run git diff

Building on a different version changes the lockfile, even though there haven't been any dependency changes.
Note: cargo just needs to do dependency resolution, so a cargo check should work just as well as a build.
Possible Solution(s)

Sort the lockfile by name, version, and source
Workaround: warn when a build uses packages with the same name and version from different sources

Notes
Output of cargo version:
cargo +nightly --version
cargo 1.52.0-nightly (90691f2bf 2021-03-16)
cargo +stable --version
cargo 1.51.0 (43b129a20 2021-03-16)

Thanks for the report! Can you clarify a bit, though, to help reproduce this?
I am unable to get Cargo to nondeterministically generate a particular lock file, the ordering of crates in a lock file is always stable given a singular version of Cargo. What I do see, however, is that stable cargo and nightly cargo are producing the different orderings. Is that the behavior you're seeing as well? Or are you seeing the same Cargo produce different versions of a lock file?

I am unable to get Cargo to nondeterministically generate a particular lock file, the ordering of crates in a lock file is always stable given a singular version of Cargo. What I do see, however, is that stable cargo and nightly cargo are producing the different orderings. Is that the behavior you're seeing as well? Or are you seeing the same Cargo produce different versions of a lock file?

I think you're right, I was confused because:

the commits and cargo both changed the lockfile, and
different developers on our team use different cargo versions.

It would be great to have an explicit, deterministic order across cargo versions. But perhaps it's not such a priority if the order only changes between releases, and only for a few crates.

Ok thanks for the info! This is definitely a bug in Cargo and something we strive to not have happen. This is caught up in the larger change for switching the defaults for git dependencies to the HEAD branch instead of a hardcoded master branch.
The PR at fault here is https://github.com/rust-lang/cargo/pull/9133. The change there is that Ord for SourceId changed from being hand-rolled to being derived, and the derived variant is different from the hand-rolled variant. I didn't realize that it would affect situations like this and the lock file.
I'm unfortunately not sure what to do about this change. If we caught this soon after landing I'd say we should revert and fix this before re-landing, but this change has been out for long enough that I don't think it's as feasible to revert any more. At best all I can think of is to hopefully learn from this and ensure that this doesn't happen again in Cargo :(

I didn't realize that it would affect situations like this and the lock file.

To be fair, this situation should be rare - it was actually a bug in our dependencies where we changed one crate in our workspace, rather than patching them all.

I don't think it's as feasible to revert any more

I think a revert would just create a bunch of new ordering issues.
And I can understand switching to derived Ord - that way, you never forget a new field. (Forgetting fields would definitely cause you ordering issues.)
Thinking about this a bit more...
How can you make sure the ordering doesn't change in future?

new fields need to be added to the end of any types with derived Ord impls

add a comment to those types
how do you make sure you dependencies don't change their orders?

how do you test that encoding/decoding/type changes don't change the order?

add some fixed test cases
how do you do randomised property testing?

It's really tricky, we had a similar issue recently where we left a new field out of a manual PartialEq impl. proptest didn't pick it up, because the edge case was very low probability. You'd need coverage-guided fuzzing or similar techniques to find it.
It's probably not worth the effort to write a big test harness for these kinds of low-impact, low-probability bugs. But maybe a crater run with a diff would help detect future issues like this? (You might already do that!)

Discussion at today's Cargo meeting concluded that we should:

Revert #9133 on beta, restoring the order between stable/beta
Investigate a fix (now posted as https://github.com/rust-lang/cargo/pull/9384) and land on nightly to have all three channels agree again

This will mean that if projects have switched to using the beta/nightly lock file format they'll need to switch back (sorry about that!)  but hopefully the transition period won't be too large. I've tried to add at least a small test to catch changes to this in the future as well.


Add CI that does deployments

Right now we're just relying on cargo test, but we aren't covering container image deploy or upgrades.
https://github.com/ostreedev/ostree-rs-ext/pull/82
is starting this, but we also need to hook this repo up to Prow.
Another alternative is to reuse rpm-ostree's CI.

This was fixed in https://github.com/ostreedev/ostree-rs-ext/pull/281


[CSL-1867] Describe update system's consensus rules

I want to notice that you should also describe checks which are part of internal invariants of the involved types (e. g. see mkUpdateProposal).

:cho: how is it related to consensus rules? you can notice I didn't write the UpdateProposal type here and related, internal invariants forces to do it. and again, this docs doesn't describe everything about US just consensus rules

:cho: how is it related to consensus rules? you can notice I didn't define the UpdateProposal type here and related, internal invariants force to do it. and again, this docs doesn't describe everything about US just consensus rules

Because if a block contains a proposal with invalid signature, this block is invalid. This rule is not described anywhere. If you know better place for it, let me know, but it must be described somewhere (otherwise people will think we are silly and forgot this check).

Why doesn't 'txp.md' describe similar invariants? These invariants is more related to block-processing rather than consensus rules. I would describe them into block-processing/overall.md

There is one https://github.com/input-output-hk/cardano-sl/blob/a43e18d58f345e302729f0b0c5491ab97946b723/docs/txp.md#general-checks
Also there is a check which applies to whole TxPayload:

    Number of transactions must be the same as number of TxWitnesses.

But I've forgotten about invariants of Tx (also we should mention invariant of Coin somewhere).

Isn't better to introduce new doc with invariants description?

Isn't better to introduce new doc with invariants description?

I think it would be good to have a document where all types are described along with their invariants.
Probably it's ok to do it in another issue.

I've rebased on master


add podman option for image build and oc for apply

Makefile: Add option to use podman or docker (CTR_CMD)
Makefile: Add option to use oc or kubectl (KS_CMD)
make docker-build -> make image-build
make docker-push -> make image-push
Add target for make image-build-amd64 -> convenience for those devs working on MacOS who are testing in OCP (I'm open to leaving this out of this PR if others don't think it's useful enough)

/cc @bouskaJ
/cc @cooktheryan

closing, will open from this repo branch


Some examples publishing to local DB

Some examples publishing to local DB would be helpful like using AAD auth.
Also I don't know from where to get sqlpackage.exe.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ac30c0d5-990b-1114-cb66-d6bbb789b698
Version Independent ID: 9432dace-7676-2a86-485a-dd5bc6568e86
Content: SqlPackage.exe - SQL Server
Content Source: docs/tools/sqlpackage.md
Product: sql
Technology: tools-other
GitHub Login: @pensivebrian
Microsoft Alias: broneill

The documentation on this page is true for SqlPackage.exe downloaded from
https://docs.microsoft.com/en-us/sql/tools/sqlpackage?view=sql-server-ver15
and it is version 15.0.4897.1
but the one that ships with VS2019 (Version 16.7.5) does not support the /Version command.
I found it by searching my entire HDD and it was located at
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150
If I look at the version on the file properties it says version 15.0.4841.1

What is the command to publish using SqlPackage.exe using AAD MFA authentication? How do I specify the user? /TargetUser is only for SQL Auth.

I got AAD MFA to work by installing from the MSI package located here
https://docs.microsoft.com/en-us/sql/tools/sqlpackage-download?view=sql-server-ver15
I was trying out the Windows.NET core package and it was not popping up the user auth dialog.

@xavierjohn -- Xavier, thank you for your feedback and researches.  Please consider these resources:

SQL Server on Microsoft Q&A
DBA Stack Exchange
Stack Overflow

@pensivebrian -- Brian, please look into this potential documentation issue.

#assign:dzsquared

@xavierjohn  - from your comments it sounds like you were able to determine the command you needed and downloaded the latest version.
#please-close

The command to use MFA is
\ProgramFiles\Microsoft SQL Server\150\DAC\bin\SqlPackage.exe
/Action:Publish /TargetServerName:$TargetServerName /TargetDatabaseName:$TargetDatabaseName $SourceFile /UniversalAuthentication /TenantId:$TenantId

@xavierjohn -- Xavier, thank you for clarifying.
cc @dzsquared for awareness


Update zio-nio to 1.0.0-RC10

Updates dev.zio:zio-nio from 1.0.0-RC9 to 1.0.0-RC10.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "dev.zio", artifactId = "zio-nio" } ]

labels: library-update, semver-pre-release

Superseded by #379.


Can you tell me the training requirements?

Hello, is the segment.py in the segment folder used to train the model? If I want to achieve the effect of the pre-trained model, do I need to use the cityscape and idd datasets? I would like to ask how much GPU memory is needed to do this. Now the GPU is 2080ti and the memory is 11g. Is it impossible to train or will it take a week or a few weeks to train a good model? I would be very grateful if you could answer.

Yes. segment.py is used for training. Mapillary and IDD datasets were used for training. Mapillary and Cityscapes would also work. Two/four 2080ti would be better. You can reduce the batch size or the resolution to facilitate training on a single 1080 GPU. Besides, you can consider our newer trans4pass models for training. https://github.com/jamycheung/Trans4PASS


CC capacity config fields does not gets updated with new broker fields

Describe the bug
CC capacity config fields does not gets updated with new broker fields. The Operator will proceed thanks to the default value populated to the capacity.json but does not get updated when a new broker is added to the cluster. It means CC using the same resource values for all the brokers except for the initial cluster.
Steps to reproduce the issue:
Create a KafkaCluster using SimpleKafkaCluster
Add a broker to the cluster
Expected behavior
Capacity config should be up to date containing info from all brokers

Could not reproduce the issue, please reopen if needed.


Certain artifact sets stack when they shouldn't

Hello, I noticed that certain artifact sets will duplicate their effects in the optimizer when multiple characters with that set are in the same team, examples are Deepwood Memories and Viridescent Venerer. The optimizer assumes that, since two characters with e.g. Deepwood are in the team, the 30% bonus is doubled to 60%.

Dupe of #361


Ideas for future commands to add (for discussion)

Not intended to be implemented, just interested in hearing what commands people would find useful in the toolbox. Then some of us could volunteer to try implementing some of them, or prototype them at least....contributing to grow our own toolbox...
The idea is that these would be useful for us and also for providers.
"3scale login" (like heroku toolbelt). It takes provider_key/token and domain, and stores (in ENV? in CWD?) who you are as a provider, and you don't need to send provider key and domain with each command?
"3scale account" Dumps information about the account identified via key/domain. When created, admin and member users, etc.
"3scale whois" Takes an identified (any of user_key?, email? name?) and uses the API to tell you who that is...User info, on what account, a provider user or developer etc... May involve searching using multiple methods, depending if it can distringuish what type of key was provided.
"3scale invoices" Return a set of invoices for a specified time period, in cvs/json/xml format.
"3scale stats" Return data on defined metrics for a given service & time period in csv/json/xml format.

Closing old issues


A generic error occurred. Now will retry again

The error "A generic error occurred. Now will retry again" is displayed just after set the video URL on the option 2 "Get followers" the error is displayed all the time in that specific option.

Hello! It's just probably that the followers option is not available on zefoy right now. Under the followers button in the browser you would see in red "soon will be update", which means that this option is unavailable right now. You can run the script maybe after a few hours or a day to see if the followers option is available again.

You're welcome, I will keep this issue open until I get a confirmation from you.


Added FReMP Stack

For all PRs

[x] I read the Contribution Guide

For new stacks

[x] I followed the do's and don'ts
[x] This stack does not already exist

Looks great, thanks for the submission @kouul! 🙇

Hi @dzello, thanks for the merge.
Just one thing, the URL of the stack was made on https://awesomestacks.dev/f-re-mp-flask-react-js-mongo-db-python rather than https://awesomestacks.dev/fremp-flask-react-js-mongo-db-python/. Is this the normal behaviour?


Apple Silicon support

Veldrid currently does not work natively in macOS on Apple Silicon devices. The first problem that occurs is missing libSDL.dylib for target architecture. This can be easily fixed with simple hack:

Install sdl2 using homebrew brew install sdl2.
Run application passing DYLD_LIBRARY_PATH=/opt/homebrew/lib.

After that there's a next problem when using Metal renderer:
Unhandled exception. System.EntryPointNotFoundException: Unable to find an entry point named 'objc_msgSend_stret' in shared library '/usr/lib/libobjc.A.dylib'.
   at Veldrid.MetalBindings.ObjectiveCRuntime.objc_msgSend_stret(Void* retPtr, IntPtr receiver, Selector selector)
   at Veldrid.MetalBindings.ObjectiveCRuntime.objc_msgSend_stret[T](IntPtr receiver, Selector selector) in Veldrid.MetalBindings.dll:token 0x60001e0+0x8
   at Veldrid.MetalBindings.NSView.get_frame() in Veldrid.MetalBindings.dll:token 0x6000191+0x0
   at Veldrid.MTL.MTLSwapchain..ctor(MTLGraphicsDevice gd, SwapchainDescription& description) in Veldrid.dll:token 0x6000794+0x4b
   at Veldrid.MTL.MTLGraphicsDevice..ctor(GraphicsDeviceOptions options, Nullable`1 swapchainDesc) in Veldrid.dll:token 0x6000712+0x23f
   at Veldrid.GraphicsDevice.CreateMetal(GraphicsDeviceOptions options, SwapchainDescription swapchainDescription) in Veldrid.dll:token 0x6000110+0x7
   at Veldrid.StartupUtilities.VeldridStartup.CreateMetalGraphicsDevice(GraphicsDeviceOptions options, Sdl2Window window, Boolean colorSrgb) in Veldrid.StartupUtilities.dll:token 0x600000e+0x28
   at Veldrid.StartupUtilities.VeldridStartup.CreateMetalGraphicsDevice(GraphicsDeviceOptions options, Sdl2Window window) in Veldrid.StartupUtilities.dll:token 0x600000d+0x0
   at Veldrid.StartupUtilities.VeldridStartup.CreateGraphicsDevice(Sdl2Window window, GraphicsDeviceOptions options, GraphicsBackend preferredBackend) in Veldrid.StartupUtilities.dll:token 0x600000b+0x35
   at Veldrid.StartupUtilities.VeldridStartup.CreateWindowAndGraphicsDevice(WindowCreateInfo windowCI, GraphicsDeviceOptions deviceOptions, GraphicsBackend preferredBackend, Sdl2Window& window, GraphicsDevice& gd) in Veldrid.StartupUtilities.dll:token 0x6000004+0x20

It seems like Metal bridge is not ABI compatible with ARM64 Darwin.
Trying to run with OpenGL renderer is also not successful:
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'libveldrid-spirv' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(liblibveldrid-spirv, 1): image not found
   at Veldrid.SPIRV.VeldridSpirvNative.CrossCompile(CrossCompileInfo* info)
   at Veldrid.SPIRV.SpirvCompilation.CompileVertexFragment(Byte[] vsBytes, Byte[] fsBytes, CrossCompileTarget target, CrossCompileOptions options) in Veldrid.SPIRV.dll:token 0x6000032+0x1bc

It can't load libveldrid-spirv since it is not compiled to ARM64 architecture (or it is just absent in this case like libsdl2).

https://github.com/mellinoe/ImGui.NET-nativebuild/pull/4
https://github.com/mellinoe/ImGui.NET/pull/265

All of the native libraries should be a simple recompile, I'll try to work through those and get them shipped. As for Veldrid's ObjC layer, there's very likely some shortcuts that were taken because it only supported x86 macOS. There's at least one TODO related to this; https://github.com/mellinoe/veldrid/blob/master/src/Veldrid.MetalBindings/ObjectiveCRuntime.cs#L244-L247, and probably some other places that made assumptions that are broken on Arm64.

That said, I don't have an ARM64 macOS device to test this on.

All of the native libraries should be a simple recompile, I'll try to work through those and get them shipped. As for Veldrid's ObjC layer, there's very likely some shortcuts that were taken because it only supported x86 macOS. There's at least one TODO related to this; https://github.com/mellinoe/veldrid/blob/master/src/Veldrid.MetalBindings/ObjectiveCRuntime.cs#L244-L247, and probably some other places that made assumptions that are broken on Arm64.

Actually I could get simple example working, but not with Metal. The only thing I had to patch is to replace objc_msgSend_stret<CGRect> with CGRect_objc_msgSend in NSView.frame. With OpenGL I got it working, but with Metal I only had pink screen.
Also, there is something wrong with textures:
Unhandled exception. SixLabors.ImageSharp.ImageFormatException: Bad method for ZLIB header: cmf=234
   at SixLabors.ImageSharp.Formats.Png.Zlib.ZlibInflateStream.InitializeInflateStream(Boolean isCriticalChunk) in SixLabors.ImageSharp.dll:token 0x600143e+0x78
   at SixLabors.ImageSharp.Formats.Png.Zlib.ZlibInflateStream.AllocateNewBytes(Int32 bytes, Boolean isCriticalChunk) in SixLabors.ImageSharp.dll:token 0x6001436+0xf
   at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ReadScanlines[TPixel](PngChunk chunk, ImageFrame`1 image, PngMetadata pngMetadata) in SixLabors.ImageSharp.dll:token 0x600130d+0x18
   at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.Decode[TPixel](Stream stream) in SixLabors.ImageSharp.dll:token 0x6001303+0x0
   at SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode[TPixel](Configuration configuration, Stream stream) in SixLabors.ImageSharp.dll:token 0x60012fd+0x8
   at SixLabors.ImageSharp.Image.Decode[TPixel](Stream stream, Configuration config) in SixLabors.ImageSharp.dll:token 0x6000147+0x15
   at SixLabors.ImageSharp.Image.<>c__DisplayClass98_0`1.<Load>b__0(Stream s) in SixLabors.ImageSharp.dll:token 0x6001a33+0x0
   at SixLabors.ImageSharp.Image.WithSeekableStream[T](Configuration configuration, Stream stream, Func`2 action) in SixLabors.ImageSharp.dll:token 0x600018b+0x41
   at SixLabors.ImageSharp.Image.Load[TPixel](Configuration configuration, Stream stream, IImageFormat& format) in SixLabors.ImageSharp.dll:token 0x6000189+0x20
   at SixLabors.ImageSharp.Image.Load[TPixel](Configuration configuration, Stream stream) in SixLabors.ImageSharp.dll:token 0x6000188+0x0
   at SixLabors.ImageSharp.Image.Load[TPixel](Configuration configuration, String path) in SixLabors.ImageSharp.dll:token 0x6000175+0x23
   at SixLabors.ImageSharp.Image.Load[TPixel](String path) in SixLabors.ImageSharp.dll:token 0x6000173+0x0
   at Veldrid.NeoDemo.Objects.Skybox.LoadDefaultSkybox() in PATH/src/NeoDemo/Objects/Skybox.cs:line 104
   at Veldrid.NeoDemo.NeoDemo..ctor() in PATH/src/NeoDemo/NeoDemo.cs:line 82
   at Veldrid.NeoDemo.Program.Main(String[] args) in PATH/src/NeoDemo/Program.cs:line 9

Also, I'm going to submit a pull request that adds support for SDL2 native support.

That said, I don't have an ARM64 macOS device to test this on.

Actually I compiled and tested this on ARM64 device. It's important to check compatibility especially with older macOS version (10.13 was the default OS on Travis CI before). If there will be some problems, it is possible to build separate libraries on old x86 macOS and arm64 version on new macOS. I'm not sure if we can merge them into one universal library, if not I'm not sure how to tell .NET to use different libraries for different architectures.

I found some strange behaviour when trying to make your veldrid-samples repository working: first (getting started) example worked flawlessly, but the second (cube) didn't work with Metal (I got black screen).
After playing out with shaders (I removed any MVP matrix calculations and left vertex shader with output = input without any calculations) I came to the conclusion that there are some problem around uniforms passing.
Any ideas?

Yes, seems that I was right. After hardcoding all uniforms in shader code, I got something working:

I used Xcode Metal debugger to gain more information. It works for me only when only first uniform is used. No matter when it set, it just has to be in 0th set be 0th binding. Trying to use separate resource set does not help either.

Buffer actually has expected value, shader tries to use right value:

Geometry gets unpredictable shape:

Something super strange is happening, in order to make it working (besides changing fragment shader to not use texture, but just set color to uv coordinate) enough to make this change in vertex shader:
     vec4 worldPosition = World * vec4(Position, 1);
     vec4 viewPosition = View * worldPosition;
     vec4 clipPosition = Projection * viewPosition;
-    //gl_Position = clipPosition;
+    gl_Position = View * (Projection * (World * vec4(Position, 1)));

While writing this I noticed one strange thing. I have this shader code:
void main()
{
    mat4 _Projection = mat4(
        vec4(1.0296494, 0.0, 0.0, 0.0),
        vec4(0.0, 1.8304877, 0.0, 0.0),
        vec4(0.0, 0.0, -1.0050251, -1.0),
        vec4(0.0, 0.0, -0.5025126, 0.0)
    );
    mat4 _View = mat4(
        vec4(1.0, 0.0, 0.0, 0.0),
        vec4(0.0, 1.0, 0.0, 0.0),
        vec4(0.0, 0.0, 1.0, 0.0),
        vec4(-0.0, -0.0, -2.5, 1.0)
    );
    mat4 _World = mat4(
        vec4(-0.9335928, 0.28618205, -0.21564847, 0.0),
        vec4(0.0, 0.6018058, 0.79864246, 0.0),
        vec4(0.35833564, 0.74560684, -0.56184155, 0.0),
        vec4(0.0, 0.0, 0.0, 1.0)
    );
    vec4 worldPosition = _World * vec4(Position, 1);
    vec4 viewPosition = View * worldPosition;
    vec4 clipPosition = Projection * viewPosition;
    //gl_Position = clipPosition;
    gl_Position = View * (Projection * (World * vec4(Position, 1)));
    fsin_texCoords = TexCoords;
}

It generates the following vertex shader:
#include <metal_stdlib>
#include <simd/simd.h>

using namespace metal;

struct _47
{
    float4x4 _m0;
};

struct _65
{
    float4x4 _m0;
};

struct _73
{
    float4x4 _m0;
};

struct main0_out
{
    float2 m_104 [[user(locn0)]];
    float4 gl_Position [[position]];
};

struct main0_in
{
    float3 m_57 [[attribute(0)]];
    float2 m_106 [[attribute(1)]];
};

vertex main0_out main0(main0_in in [[stage_in]], constant _47& _49 [[buffer(0)]], constant _65& _67 [[buffer(1)]], constant _73& _75 [[buffer(2)]])
{
    main0_out out = {};
    out.gl_Position = _67._m0 * (_75._m0 * (_49._m0 * float4(in.m_57, 1.0)));
    out.m_104 = in.m_106;
    return out;
}

When I do this change:
-    vec4 worldPosition = _World * vec4(Position, 1);
+    vec4 worldPosition = World * vec4(Position, 1);

Note: this variable is used in next lines of code, but actually its final result is discarded.
It generates the following vertex shader code:
#include <metal_stdlib>
#include <simd/simd.h>

using namespace metal;

struct _58
{
    float4x4 _m0;
};

struct _69
{
    float4x4 _m0;
};

struct _86
{
    float4x4 _m0;
};

struct main0_out
{
    float2 m_103 [[user(locn0)]];
    float4 gl_Position [[position]];
};

struct main0_in
{
    float3 m_50 [[attribute(0)]];
    float2 m_105 [[attribute(1)]];
};

vertex main0_out main0(main0_in in [[stage_in]], constant _58& _60 [[buffer(0)]], constant _69& _71 [[buffer(1)]], constant _86& _88 [[buffer(2)]])
{
    main0_out out = {};
    out.gl_Position = _60._m0 * (_71._m0 * (_88._m0 * float4(in.m_50, 1.0)));
    out.m_103 = in.m_105;
    return out;
}

The first thing I note is that it changes the order of matrix multiplication. It this SPIRV bug?

Seems to be this https://github.com/mellinoe/veldrid/issues/289

Veldrid.SPIRV version 1.0.11 does not have such a problem.

Looks like there is some problem with SixLabors.ImageSharp, it parses texture wrong way:

Metal debugger shows texture with all black pixels but first 3 rows. Looking at spnza_bricks_a_diff.binary shows a lot of zero bytes.

Got NeoDemo working on Metal by using Veldrid.SPIRV 1.0.11. There is some problem with colors:

Checking "Render with multiple threads" fixes the problem.

I'm trying to get the NeoDemo running on an M1 Mac and I'm running into an issue with the Nerdbank.GitVersioning package.
It is throwing the following error:
Nerdbank.GitVersioning.Inner.targets(17, 5): [MSB4018] The "Nerdbank.GitVersioning.Tasks.GetBuildVersion" task failed unexpectedly.
System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception.
---> System.PlatformNotSupportedException: osx.10.17-arm64
Did you any of you run into this, or perhaps this is a new issue with a later version of OSX?

@Tarik02 Thanks for the response!  I started with a fresh pull of the veldrid repository and upgraded to .NET 6.0 and upgraded the Nerdbank.GitVersioning.  This time it worked without issue, so I must have originally not upgraded all references to Nerdbank.
Doing that (and doing steps 1 and 2 from your original post) gets me to having this error (the same one you had originally) when I run NeoDemo:
Unhandled exception. System.EntryPointNotFoundException: Unable to find an entry point named 'objc_msgSend_stret' in shared library '/usr/lib/libobjc.A.dylib'.
at Veldrid.MetalBindings.ObjectiveCRuntime.objc_msgSend_stret(Void* retPtr, IntPtr receiver, Selector selector)
I get this same error when using Veldrid.SPIRV 1.0.11 and 1.0.14.
Was there anything else I needed to do to get the NeoDemo running?

@adamthole

The only thing I had to patch is to replace objc_msgSend_stret with CGRect_objc_msgSend in NSView.frame.

@Tarik02 Thank you.  I saw that above but I misunderstood, thinking it was only needed for OpenGL.
I've made that update, and now I'm getting an error about not being able to find libveldrid-spirv:
Unhandled exception. System.DllNotFoundException: Unable to load shared library 'libveldrid-spirv' or one of its dependencies.
Looking around in the Nuget folder it looks like the nuget package doesn't come with an osx-arm64 version, only the osx-64.  Did you have to manually compile libveldrid-spirv, or am I missing something else?
This error is being thrown in ShaderHelper.cs, line 21:
shader[] shaders = factory.CreateFromSpirv(
                new ShaderDescription(ShaderStages.Vertex, vsBytes, "main", debug),
                new ShaderDescription(ShaderStages.Fragment, fsBytes, "main", debug),
                GetOptions(gd));

Thanks again for your help!

Looking around in the Nuget folder it looks like the nuget package doesn't come with an osx-arm64 version, only the osx-64. Did you have to manually compile libveldrid-spirv, or am I missing something else?

year, as for all native dependencies


AXIS: overlap incorrect

Higher values of overlap should make more overlap between images. They appear to make less... i.e. "80" appears to make the images have 20% overlap, etc.

Fixed with @300b348


Adds UUI-Copy

Description

Adds a button to trigger copying content to the clipboard, by two approaches:

Setting the value property on uui-copy to contain the text content you want to copy to the clipboard
Setting the copy-from property will look for HTML element with that ID and use the text content

If the selected element has a value property then assume we want to get the value of an input field such as uui-input
By setting copy-from it overrides any value set in the value property

[!NOTE]
This is heavily inspired by Shoelace.style copy button
https://github.com/shoelace-style/shoelace/blob/next/src/components/copy-button/copy-button.component.ts

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Chore (minor updates related to the tooling or maintenance of the repository, does not impact compiled assets)

Motivation and context

How to test?
I believe the stories should cover some scenarios
Screenshots (if appropriate)
Checklist

[ ] If my change requires a change to the documentation, I have updated the documentation in this pull request.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.

I am open for discussion and feedback to improve this
/cc @iOvergaard & @nielslyngsoe


NotificationView warning/danger colors update to meet accessibility standards.

Platforms Impacted

[x] iOS
[ ] macOS

Description of changes
The design team updated the colors used for Warning and Danger toast scenarios for the NotificationView in order to meet accessibility standards.
Reflecting this into code.
Verification
Validated the demo app in both dark and light modes:

Before
After

Pull request checklist
This PR has considered:

[x] Light and Dark appearances
[ ] VoiceOver and Keyboard Accessibility
[ ] Internationalization and Right to Left layouts
[ ] Different resolutions (1x, 2x, 3x)
[ ] Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)

Microsoft Reviewers: Open in CodeFlow

Closing as per offline conversation with @victa.
This work will not be only a matter of updating the colors. Shadows and other design aspects need to be reviewed and approved.


use norm_mode 1 as default

Not sure if this is related to #53: why does the current default NORM_MODE set 2 for non-Indic, non-RTL languages? Shouldn't this be 1?
Also, the decision tree looks quite different than the corresponding one in tesseract/src/training/language-specific.sh. Does anyone know how that came to be?

Plus (just to be sure): Am I correct in assuming that under 2, combining characters get recoded as extra symbol, whereas under 1 they are merged with the base character?

Decision seems to derive from here: c90cd3f27acbacc8d30db1b44d1c017aecc7bf20
@wrznr could you please elaborate on the kind  of feedback you gave (or link to it)?

@wrznr could you please elaborate on the kind of feedback you gave (or link to it)?

answer (on other channel): here – a simple question.
IMHO the response should have been to rethink the old default in tesstrain, not immediately revert to it.
As stated above, Tesseract's own default used to be 1 in src/training/language-specific.sh. But that file (along with all other shell scripts) has very recently been removed from tesseract by @stweil. It now resides here:
https://github.com/tesseract-ocr/tesstrain/blob/1d8238684fe81e600431e5bdfe7dd24fbeaaf9f9/src/training/language_specific.py#L1373
So, again, why not 1 by default, and is my interpretation regarding combining characters correct?

@Shreeshrii it seems the original deviation regarding --norm_mode default came from changes proposed by you (introducing finetuning here). Could you please elaborate on your choice?

@bertsky It looks like the initial PR by @kba and myself was not prepared carefully enough. We took norm_mode for granted where we should have inspected its semantics in greater detail. When @Shreeshrii later tried to correct this with the option for setting the norm_mode in a sensible way, some misunderstandig occurred leading to the current suboptimal setting. It would be great if we could fix this now that you had have the chance to dive deeper into the consequences of this parameter setting.

Sure, I'll prepare a PR, but will first do some (training and) testing.

Sure, I'll prepare a PR, but will first do some (training and) testing.

@bertsky Did you find anything different in testing? I see that the Norm Mode is still 2 for non-indic and non-RTL languages one year after this conversation.

@giri-kum sry, I don't remember anymore. I think I had some results, but inconclusive due to other problems. Isolated experiment should not be difficult to set up (run 10-20 trainings with mode 1 and mode 2, evaluate valset with external, true CER measurement, compare averages, perhaps repeat with different GT sets or languages), but I don't have the time right now.


Customizing location of flag files in the results_pipeline dir

Right now, project.results_pipeline is used as the subdir within the project.output_dir. Looper then adds the sample name on to the end of this folder automatically, and then looks there for flags and summarizer files
The refgenie pipeline shows that we need more flexibility. We're setting up the pipeline to place flags and outputs in a much deeper folder (so we can exclude it from archive checksumming, etc). But there's no way to tell looper about this, and hence, flag checks and summarizers will never work with this pipeline.
This could be solved by making the 'results_pipeline' + {sample.name} hard-coding into a template that can be populated with sample properties, which should point directly to the output folder.
Related to pepkit/looper#242

I think this might make sense to add as an attribute in the pipestat schema, maybe under the status components?
Or wait, would it make more sense to add to the looper pipeline interface (maybe I moved it from looper to pipestat prematurely...)

Currently, the user can pass flag_file_dir to the PipestatManager class during initialization or provide the directory in the config file.
Then, pipestat determines:
  flag_file_dir = self[CONFIG_KEY].priority_get(
      "flag_file_dir", override=flag_file_dir, default=os.path.dirname(self.file)
  )
  self[STATUS_FILE_DIR] = mk_abs_via_cfg(flag_file_dir, self.config_path)


Change to gratis license

As per discussion with @Amorymeltzer, this is the more appropriate license -- the binary is still free.

:+1: