Not able to run samples as per user guide

Hi, I am trying to get started with Hydra, I am running Hydra in local configuration for testing. Where can I get sample job configurations and sample data? Where should the data be placed if I have a csv file? I managed to configure a job, however not able to provide input data. Please help. Your sample job with automatic data inject runs correctly.

Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running.  When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data.  I've
attached two files to this email.  job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams' sub
directory of your local stack.  The second file is called http.log.gz which
has some dummy apache log lines.  If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn to
generate a simple tree.
Let me know how it goes.  We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS>wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data. Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams' sub
directory of your local stack. The second file is called http.log.gz which
has some dummy apache log lines. If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data. Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub https://github.com/addthis/hydra/issues/194#issuecomment-162320822 .  https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif

I see the attachments in the original email, can you double check?  Maybe
github is removing the attachments.  Send me your direct email.
On Sun, Dec 6, 2015 at 7:35 AM, Suresh Rao<EMAIL_ADDRESS>wrote:

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please
resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams' sub
directory of your local stack. The second file is called http.log.gz which
has some dummy apache log lines. If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data.
Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162320822> . <
https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif>
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162321274.

Hi Matt,
I rechecked, the attachment is missing. Can you please forward it to my email address<EMAIL_ADDRESS><EMAIL_ADDRESS>Thanks and Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:26
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
I see the attachments in the original email, can you double check? Maybe
github is removing the attachments. Send me your direct email.
On Sun, Dec 6, 2015 at 7:35 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please
resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams' sub
directory of your local stack. The second file is called http.log.gz which
has some dummy apache log lines. If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data.
Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162320822> . <
https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif>
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162321274.

—
Reply to this email directly or view it on GitHub https://github.com/addthis/hydra/issues/194#issuecomment-162322202 .  https://github.com/notifications/beacon/APJocXNg8w_bLZ4O33T-6jKyYNm682Dcks5pNEQQgaJpZM4Gvb8s.gif

Here it is.  I've included plain text of job as well in the email.
""
source.mesh2:{
mesh:{
dateFormat:constant
files:"/data/*"
}
processAllData:true
format:{
type:"column",
columns:["IP","IDENTD","USERID","TIMESTAMP","REQUEST","CODE","XFER"],
tokens:{
separator:" ",
group:[""", "[]"],
pack:false,
},
lineFilter:{
op:"exclude",
match:["#.*"],
},
},
}
map.fields:["IP","TIMESTAMP","REQUEST","CODE", "XFER"]
map.filterOut:[
{from:"TIMESTAMP", filter:[{split:" "}, {index:0}]}
{op:"time", src:{field:"TIMESTAMP", format:"dd/MMM/yyyy:HH:mm:ss"},
dst:{field:"TIME", format:"native"}},
{op:"time", src:{field:"TIME", format:"native"}, dst:{field:"DATE",
format:"yyMMdd-HHmmss", timeZone:"EST"}},
{op:"field", from:"DATE", to:"DATE_YMD", filter:{op:"slice",to:6}},
{debug {every:5}}
]
output.tree:{
timeField:{field:"TIME", format:"native"}
root:[
{const:"root"}
{field:"DATE_YMD"}
{field:"CODE"}
{field:"IP"}
]
}
""
On Sun, Dec 6, 2015 at 8:01 AM, Suresh Rao<EMAIL_ADDRESS>wrote:

Hi Matt,
I rechecked, the attachment is missing. Can you please forward it to my
email address<EMAIL_ADDRESS><EMAIL_ADDRESS>Thanks and Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:26
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
I see the attachments in the original email, can you double check? Maybe
github is removing the attachments. Send me your direct email.
On Sun, Dec 6, 2015 at 7:35 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please
resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS><mailto:
<EMAIL_ADDRESS>>
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams'
sub
directory of your local stack. The second file is called http.log.gz
which
has some dummy apache log lines. If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn
to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations
and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data.
Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162320822> . <

https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif

—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162321274.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162322202> . <
https://github.com/notifications/beacon/APJocXNg8w_bLZ4O33T-6jKyYNm682Dcks5pNEQQgaJpZM4Gvb8s.gif>
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162322440.

Thanks Matt. But I have not got the sample data still.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:34
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Here it is. I've included plain text of job as well in the email.
""
source.mesh2:{
mesh:{
dateFormat:constant
files:"/data/*"
}
processAllData:true
format:{
type:"column",
columns:["IP","IDENTD","USERID","TIMESTAMP","REQUEST","CODE","XFER"],
tokens:{
separator:" ",
group:[""", "[]"],
pack:false,
},
lineFilter:{
op:"exclude",
match:["#.*"],
},
},
}
map.fields:["IP","TIMESTAMP","REQUEST","CODE", "XFER"]
map.filterOut:[
{from:"TIMESTAMP", filter:[{split:" "}, {index:0}]}
{op:"time", src:{field:"TIMESTAMP", format:"dd/MMM/yyyy:HH:mm:ss"},
dst:{field:"TIME", format:"native"}},
{op:"time", src:{field:"TIME", format:"native"}, dst:{field:"DATE",
format:"yyMMdd-HHmmss", timeZone:"EST"}},
{op:"field", from:"DATE", to:"DATE_YMD", filter:{op:"slice",to:6}},
{debug {every:5}}
]
output.tree:{
timeField:{field:"TIME", format:"native"}
root:[
{const:"root"}
{field:"DATE_YMD"}
{field:"CODE"}
{field:"IP"}
]
}
""
On Sun, Dec 6, 2015 at 8:01 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
I rechecked, the attachment is missing. Can you please forward it to my
email address<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Thanks and Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:26
To: addthis/hydra<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
I see the attachments in the original email, can you double check? Maybe
github is removing the attachments. Send me your direct email.
On Sun, Dec 6, 2015 at 7:35 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please
resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS><EMAIL_ADDRESS><mailto: <mailto:
<EMAIL_ADDRESS><EMAIL_ADDRESS>> >
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams'
sub
directory of your local stack. The second file is called http.log.gz
which
has some dummy apache log lines. If you create a directory called 'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via spawn
to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar) example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in local
configuration for testing. Where can I get sample job configurations
and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data.
Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162320822> . <

https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif

—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162321274.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162322202> . <
https://github.com/notifications/beacon/APJocXNg8w_bLZ4O33T-6jKyYNm682Dcks5pNEQQgaJpZM4Gvb8s.gif>
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162322440.

—
Reply to this email directly or view it on GitHub https://github.com/addthis/hydra/issues/194#issuecomment-162322554 .  https://github.com/notifications/beacon/APJocRNYJFq_EvTcZnJgNrhqAtq5yHS8ks5pNEXQgaJpZM4Gvb8s.gif

I think your email is blocking things (or github).   But I have not got the sample data still.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:34
To: addthis/hydra<EMAIL_ADDRESS>Cc: Suresh Rao<EMAIL_ADDRESS>Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Here it is. I've included plain text of job as well in the email.
""
source.mesh2:{
mesh:{
dateFormat:constant
files:"/data/*"
}
processAllData:true
format:{
type:"column",
columns:["IP","IDENTD","USERID","TIMESTAMP","REQUEST","CODE","XFER"],
tokens:{
separator:" ",
group:[""", "[]"],
pack:false,
},
lineFilter:{
op:"exclude",
match:["#.*"],
},
},
}
map.fields:["IP","TIMESTAMP","REQUEST","CODE", "XFER"]
map.filterOut:[
{from:"TIMESTAMP", filter:[{split:" "}, {index:0}]}
{op:"time", src:{field:"TIMESTAMP", format:"dd/MMM/yyyy:HH:mm:ss"},
dst:{field:"TIME", format:"native"}},
{op:"time", src:{field:"TIME", format:"native"}, dst:{field:"DATE",
format:"yyMMdd-HHmmss", timeZone:"EST"}},
{op:"field", from:"DATE", to:"DATE_YMD", filter:{op:"slice",to:6}},
{debug {every:5}}
]
output.tree:{
timeField:{field:"TIME", format:"native"}
root:[
{const:"root"}
{field:"DATE_YMD"}
{field:"CODE"}
{field:"IP"}
]
}
""
On Sun, Dec 6, 2015 at 8:01 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
I rechecked, the attachment is missing. Can you please forward it to my
email address<EMAIL_ADDRESS><EMAIL_ADDRESS><mailto:
<EMAIL_ADDRESS>Thanks and Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 20:26
To: addthis/hydra<EMAIL_ADDRESS><mailto:
<EMAIL_ADDRESS>>
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
I see the attachments in the original email, can you double check? Maybe
github is removing the attachments. Send me your direct email.
On Sun, Dec 6, 2015 at 7:35 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi Matt,
Thanks for the quick response. You missed to attach the files. Please
resend.
Regards
Suresh Rao
From: Matt Abrams<EMAIL_ADDRESS>Sent: 06 December 2015 19:54
To: addthis/hydra<EMAIL_ADDRESS><mailto:
hydra@noreply.github.com%20%3cmailto:%0b> <mailto:
<EMAIL_ADDRESS><EMAIL_ADDRESS>> >
Cc: Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>>
Subject: Re: [hydra] Not able to run samples as per user guide (#194)
Hello -
LocalStack sets up a mesh server to stream data from the machine where
local stack is running. When you created your local stack instance you
should get a directory:
hydra/hydra-local/streams
This is the default directory where hydra sources look for data. I've
attached two files to this email. job.conf is a sample job config that
reads data from a directory called 'data' which lives in the 'streams'
sub
directory of your local stack. The second file is called http.log.gz
which
has some dummy apache log lines. If you create a directory called
'data'
in the 'streams' directory and then copy the http.log.gz file into that
directory you should be able to run the attached job.conf file via
spawn
to
generate a simple tree.
Let me know how it goes. We should probably add this (or similar)
example
to the default localstack.
Thanks,
Matt
On Sat, Dec 5, 2015 at 7:36 AM, Suresh Rao<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

Hi, I am trying to get started with Hydra, I am running Hydra in
local
configuration for testing. Where can I get sample job configurations
and
sample data? Where should the data be placed if I have a csv file? I
managed to configure a job, however not able to provide input data.
Please
help. Your sample job with automatic data inject runs correctly.
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162320822> .
<

https://github.com/notifications/beacon/APJocYsVqFUiqJjH7K7UyDpdNVg9jSkxks5pNDybgaJpZM4Gvb8s.gif

—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162321274.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162322202> . <

https://github.com/notifications/beacon/APJocXNg8w_bLZ4O33T-6jKyYNm682Dcks5pNEQQgaJpZM4Gvb8s.gif

—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162322440.

—
Reply to this email directly or view it on GitHub <
https://github.com/addthis/hydra/issues/194#issuecomment-162322554> . <
https://github.com/notifications/beacon/APJocRNYJFq_EvTcZnJgNrhqAtq5yHS8ks5pNEXQgaJpZM4Gvb8s.gif>
—
Reply to this email directly or view it on GitHub
https://github.com/addthis/hydra/issues/194#issuecomment-162323833.

Thanks a lot. I am  able to run the job now.


Make webworker-threads optional

This is intended to resolve #345.
webworker-threads is now an optionalDependency, meaning npm will attempt to install it, but not fail if it does not succeed.
The trainParallel,trainParallelBatches and functions that call them (only retrainParallel in this lib) will now throw a runtime error prompting the user to install webworker-threads.
The tests add proxyquire as a devDependency to simulate the absence of the webworker-threads module.

awesome thanks you beat me to it :-) (which isnt too hard these days unfortunately)


Updates packaging script for HSDK 2.7 using new --add option

HSDK 2.7 introduces a new --add option for the Holoscan Pacakger, which enables adding additional files and dependencies to the HAP container image.
This PR updates the Endoscopy Tool Tracking packaging scripts to use the new --add option.
Please merge after HSDK 2.7 release.

@mocsharp does this need re-review / merge?

@mocsharp does this need re-review / merge?

Yes, please merge. I have verified the CPP app is working. The Python will not work until 2.8 with the fix to relax pip version pinning.


New Fleet Version - Fleet configuration needs to be updated

Hi,
This issue related to fleet update 4.0.0-rc1. They deprecated auth_jwt_key and auth_jwt_key_file. Also, I don't think that will be an issue but now they are using "fleet" name instead of "kolide" i.e. database name.
Here is more detail about the new configuration,
https://github.com/fleetdm/fleet/blob/v4.0.0-rc1/docs/3-Deployment/2-Configuration.md
Changelog,
https://github.com/fleetdm/fleet/releases/tag/v4.0.0-rc1

Here is my change/fix to this issue,
`#logger_bootstrap.sh

mysql -uroot -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'kolide';"
mysql -uroot -pkolide -e "create database kolide;"

mysql -uroot -e "ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'fleet';"
mysql -uroot -pfleet -e "create database fleet;"

fleet prepare db --mysql_address=<IP_ADDRESS>:3306 --mysql_database=kolide --mysql_username=root --mysql_password=kolide

fleet prepare db --mysql_address=<IP_ADDRESS>:3306 --mysql_database=fleet --mysql_username=root --mysql_password=fleet

mysql -uroot --password=kolide -e 'use kolide; update enroll_secrets set secret = "enrollmentsecret" where active=1;'

mysql -uroot --password=fleet -e 'use fleet; update enroll_secrets set secret = "enrollmentsecret" where active=1;'
`

`#../resources/fleet/fleet.service
-ExecStart=/usr/local/bin/fleet serve --mysql_address=<IP_ADDRESS>:3306 --mysql_database=kolide --mysql_username=root --mysql_password=kolide --redis_address=<IP_ADDRESS>:6379  --server_cert=/opt/fleet/server.crt --server_key=/opt/fleet/server.key --logging_json --osquery_result_log_file=/var/log/fleet/osquery_result  --osquery_status_log_file=/var/log/fleet/osquery_status --auth_jwt_key=dasdsadasdjhshgfhdfb --server_address <IP_ADDRESS>:8412 --osquery_enable_log_rotation
+ExecStart=/usr/local/bin/fleet serve --mysql_address=<IP_ADDRESS>:3306 --mysql_database=fleet --mysql_username=root --mysql_password=fleet --redis_address=<IP_ADDRESS>:6379  --server_cert=/opt/fleet/server.crt --server_key=/opt/fleet/server.key --logging_json --osquery_result_log_file=/var/log/fleet/osquery_result  --osquery_status_log_file=/var/log/fleet/osquery_status --server_address <IP_ADDRESS>:8412 --osquery_enable_log_rotation
`

Ok, I found the issue about osquery configuration, at line 305 in logger_bootstrap.sh you need to delete "where active=1" part from the SQL query. It seems "active" column doesn't exist.

Will fix this tonight!

@clong could you also update on ESXi ansible logger tasks because its still uses the old configuration

Fixed here: https://github.com/clong/DetectionLab/commit/45986d583a31d7dffaf395f552085d18c66f5fe5


Copy tokens to other masters

Before this PR, in multi-masters case, only first master(groups['masters'][0]) got the tokens.
This PR will copy tokens to all masters. It's helpful for setup a HA kubernetes cluster.

This change is

/approve
but I'll let @ingvagabund do the review

but I'll let @ingvagabund do the review

Thanks Eric, you are the best friend.

@redhat-k8s-bot test please

@xialonglee again, the current CI does not cover HA master so as long as it does not break the single master deployment I am merging.

@redhat-k8s-bot test please

/lgtm


Syntax error: syntax error at or near "order" - PostgreSQL & Backticks

Using PostgreSQL i'm getting an error when sorting SQL query by the order column because of the backticks that are used in the query.
PostgreSQL against MySQL uses double quotes for quoting identifiers, so PostgreSQL does not recognize it.
Syntax error           Query response

ORDER BY
  `ORDER` ASC,   ==>   "ORDER" ASC,
  id ASC

What do you think would be the best solution for this problem?

This has been fixed by the auther of the package laravel-process-approval!


Can this run on railway or other service?

If it can run Docker containers in one of the supported image architectures then it should work.


Expand device list

We need to figure out as comprehensive a list of devices as we can, and then add a bunch of misc and test driver IDs

include IDs for SPOT and APRS per https://github.com/VIP-LES/EosGround/issues/8

APRS makes some sense, but what is the use case for a Spot device ID?

Spot is a valid telemetry data source that we can integrate into the ground station

Should ground station specific components have a Device ID?

Could be useful for storing the api results in a DB so we can track the source

That's a good enough reason for me


autoPatchelfHook: Fixes/improvements for Android SDK emulator

This is related to #50596, which refactors the androidenv implementation.
For patching various ELF files, a similar implementation to autoPatchelfHook was introduced there and after talking with @svanderburg, I went on to improve the hook in a way so that it's possible to be also used for androidenv, which also makes it less confusing for users having two things doing roughly the same but are named similarly.
Essentially, these changes make it easier to use only the autoPatchelf command without automatically patching all the build outputs.
For example:
with import <nixpkgs> {}; stdenv.mkDerivation {
  name = "foo";
  ...
  nativeBuildInputs = [ autoPatchelfHook ];
  dontAutoPatchelf = true;
  postPatch = ''
    autoPatchelf foo
    autoPatchelf --no-recurse bar
  '';
}

The --no-recurse flag here only patches files within the bar directory but not files in subdirectories like bar/a/b/c.
In order to make sure these changes do not break existing users of this hook, I've tested it using the following Nix expression and looking for symbol lookup errors:
with import ./. { config.allowUnfree = true; };

runCommand "test-executables" {
  drvs = [
    anydesk cups-kyodialog3 elasticsearch franz gurobi
    masterpdfeditor oracle-instantclient powershell reaper
    sourcetrail teamviewer unixODBCDrivers.msodbcsql17 virtlyst
    vk-messenger wavebox zoom-us nomachine-client polar-bookshelf
  ];
} ("for i in $drvs; do for b in $i/bin/*; do " +
   "[ -x \"$b\" ] && timeout 10 \"$b\" || :; done; done")

Success on x86_64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
  /nix/store/kyclvsckyzsf85m8pq1y68ci1gr158a0-auto-patchelf-hook.drv
these paths will be fetched (0.06 MiB download, 0.17 MiB unpacked):
  /nix/store/3h4j4qk4996cch5xsxldng4r6a5i4cwb-stdenv-linux
  /nix/store/l2i9gmhz0rsi91mz4pvvvnvzkvi5wl6b-bzip2-<IP_ADDRESS>.1-bin
  /nix/store/pmvnf35kg6k2p0z91ac663akazlf7wz6-bzip2-<IP_ADDRESS>.1
copying path '/nix/store/pmvnf35kg6k2p0z91ac663akazlf7wz6-bzip2-<IP_ADDRESS>.1' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/l2i9gmhz0rsi91mz4pvvvnvzkvi5wl6b-bzip2-<IP_ADDRESS>.1-bin' from 'https://nix-cache.s3.amazonaws.com'...
copying path '/nix/store/3h4j4qk4996cch5xsxldng4r6a5i4cwb-stdenv-linux' from 'https://nix-cache.s3.amazonaws.com'...
building '/nix/store/kyclvsckyzsf85m8pq1y68ci1gr158a0-auto-patchelf-hook.drv'...
/nix/store/b7aky5y002ipvj34nv6k3dv40z1hlryp-auto-patchelf-hook

Success on x86_64-darwin (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/lm8kfbw38i9m5w0byh2fp3750yjxdpca-auto-patchelf-hook.drv
building '/nix/store/lm8kfbw38i9m5w0byh2fp3750yjxdpca-auto-patchelf-hook.drv'...
/nix/store/0hjvkix1nvp6slr0r9j37sw5lsshrj7f-auto-patchelf-hook

Success on aarch64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/303i8gzpa9128a85a2bwahi8cg914kii-auto-patchelf-hook.drv
building '/nix/store/303i8gzpa9128a85a2bwahi8cg914kii-auto-patchelf-hook.drv'...
/nix/store/80jmgld7lkbi1mg5ms3c7hky2f697l4y-auto-patchelf-hook

Success on x86_64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
/nix/store/b7aky5y002ipvj34nv6k3dv40z1hlryp-auto-patchelf-hook

Success on x86_64-darwin (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
/nix/store/0hjvkix1nvp6slr0r9j37sw5lsshrj7f-auto-patchelf-hook

Success on aarch64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
/nix/store/80jmgld7lkbi1mg5ms3c7hky2f697l4y-auto-patchelf-hook

Great! I'll try to adjust my androidenv pull request later today and see if it works with the new implementation. I'll report back asap!

I believe I am still having a practical issue while trying to migrate the Android SDK expressions to the autopatchelf hook implementation.
Apparently, to use autoPatchelf, you must refer to the libraries that you want to use (through buildInputs or runtimeDependencies) by providing the Nix store paths of the packages that you want to use. The autopatchelf hook seems to scan the lib/ sub folders of every Nix store path for the libraries that it needs.
I don't think there is a way to directly refer to library paths (without making an assumption that there is a lib/ sub folder)? The problem is that the Android SDK also bundles shared libraries residing in weird sub folders inside the same base directory as the executables, e.g. <basedir>/lib64, /lib/qt` etc.
The autopatchelf implementation that I created uses colon separated library paths (as opposed to white space delimited Nix store paths) and can also inspect library sub folders in the same package. Is there also a way to add this kind of flexibility to the autopatchelf hook?
(As a sidenote: my original motivation to use colon separated library paths is that you can use LD_LIBRARY_PATH outside a Nix builder environment to automatically make dynamic library references static).
My biggest struggle is to port the expression that patches the Android emulator. It needs external packages, as well as libraries that are bundled with the emulator package itself, as you may probably see while inspecting contents of this file: https://github.com/svanderburg/nix-androidenvtests/blob/nextgen/androidenv/emulator.nix
In the expression, I refer to Nix packages (and use lib.makeLibraryPath to compose the paths), but I also specify paths referring to sub directories inside the same package. autopatchelf will also inspect these folders for libraries that an executable may need.
Would it also be possible to do something similar with the autoPatchelf command or adjust it in such a way that it can?

The autopatchelf implementation that I created uses colon separated library paths (as opposed to white space delimited Nix store paths) and can also inspect library sub folders in the same package. Is there also a way to add this kind of flexibility to the autopatchelf hook?

The library subdirectories for the same package are already inspected by autoPatchelfHook:
https://github.com/NixOS/nixpkgs/blob/503b41b9b2d726f10372e3daac713cc2042b8cf0/pkgs/build-support/setup-hooks/auto-patchelf.sh#L174-L177
However, when running autoPatchelf manually without the postFixup hook, it will only work on the path you've specified.
Maybe it makes sense to add shell function like addAutoPatchelfSearchPath (maybe with a shorter), which allows to add the patch to the dependency cache.
Does that sound good for you?
And would such a function then also recurse into subdirectories or even have the same --no-recurse flag?

My biggest struggle is to port the expression that patches the Android emulator. It needs external packages, as well as libraries that are bundled with the emulator package itself, as you may probably see while inspecting contents of this file: https://github.com/svanderburg/nix-androidenvtests/blob/nextgen/androidenv/emulator.nix

Seperately specifying libs_i386 and libs_x86_64 here shouldn't be necessary because autoPatchelf will only find compatible libraries or do you see a problem in that approach?

Seperately specifying libs_i386 and libs_x86_64 here shouldn't be necessary because autoPatchelf will only find compatible libraries or do you see a problem in that approach?

I don't need the separation, although I would find it nicer to write it separately, but that's just a matter of taste. I can live without it.
Basically, the only problem I have is converting this line in the emulator expression:
export libs_x86_64=$packageBaseDir/lib64:$packageBaseDir/lib64/qt/lib:$libs_x86_64

I want autopatchelf to inspect these two directories shown above in addition to the packages that I provide as buildInputs. These two paths translate to: $out/libexec/android-sdk/emulator/lib64 and $out/libexec/android-sdk/emulator/lib64/qt/lib (as may see, these paths are quite unconventional).
If there's any environment variable (or function) allowing me to provide these paths, so that they get added to the RPATH of the right executables, that will basically fix my problem.

The example you mentioned can be added via:
cachedDependencies+=(
  $(find "$out/libexec/android-sdk/emulator/lib64" \
    \! -type d \( -name '*.so' -o -name '*.so.*' \))
)

That's a lot of churn, so that's why I mentioned factoring this out into a dedicated function, but this should work (haven't tested it though).

@aszlig Would be nice if you could implement a function wrapper so that I can conveniently add these paths

Success on x86_64-darwin (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/g3axkjvf3yhl39qj7n7h3cii37kgdbsf-auto-patchelf-hook.drv
building '/nix/store/g3axkjvf3yhl39qj7n7h3cii37kgdbsf-auto-patchelf-hook.drv'...
/nix/store/n5wgv0z2kq6p6c7myn3inr66pa6qvx1m-auto-patchelf-hook

Success on aarch64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/hsc3qgyxax4pkxcb3lvq1y4n9xndsipl-auto-patchelf-hook.drv
building '/nix/store/hsc3qgyxax4pkxcb3lvq1y4n9xndsipl-auto-patchelf-hook.drv'...
/nix/store/dwamgiyrw9b0pj9qdrqjfnmwdkh5yhp4-auto-patchelf-hook

Success on x86_64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/ksvgg8h435yfbxbb446v8382rsbxy8v6-auto-patchelf-hook.drv
building '/nix/store/ksvgg8h435yfbxbb446v8382rsbxy8v6-auto-patchelf-hook.drv'...
/nix/store/m2rrwzyd3nz21avx5l40953yzzvxx2lp-auto-patchelf-hook

Great now it seems to work a lot better. I have managed to make the basic test cases work (deploying an SDK installation with some plugins, building an APK, and launching the emulator).
Unfortunately, when try to compile the hello-jni example (that depends on the Android NDK), I'm still running into a problem:
setting RPATH to: /nix/store/1fd7ybkplw7cjx6hyiiyvb03abx0l0iz-ncurses-6.1-abi5-compat/lib:/nix/store/1fd7ybkplw7cjx6hyiiyvb03abx0l0iz-ncurses-6.1-abi5-compat/lib
searching for dependencies of prebuilt/linux-x86_64/bin/yasm
searching for dependencies of prebuilt/linux-x86_64/bin/python2
searching for dependencies of prebuilt/linux-x86_64/lib/python2.7/config/python.o
wrong ELF type
builder for '/nix/store/9zvwb63r9pcv4az6q1fsvlj8s5hymv26-ndk-bundle-18.1.5063045.drv' failed with exit code 1

Apparently, it recurses into a directory that provides an object file (python.o). autoPatchelf tries to patch it. With my autopatchelf implementation this file is skipped because I use the file command to determine whether a file is an appropriate ELF binary or not. Yours apparently works different. Would it be possible to exclude ordinary .o files so that the patch process succeeds?
(As a sidenote: it's really stupid of the Android package maintainers to include this file, but this something we have to live with)

autoPatchelfHook was using file as well, see comments in #47222, the problem with file however is that it is very slow if you have a large number of files other than ELF files. I'll fix the object files case in a minute.

@svanderburg: Can you provide an output of readelf -h -l prebuilt/linux-x86_64/lib/python2.7/config/python.o? I tested with a bunch of object files but couldn't reproduce it.

Success on aarch64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/31yxzbhl3sjbmjymrzyrfqsqd2k51b29-auto-patchelf-hook.drv
building '/nix/store/31yxzbhl3sjbmjymrzyrfqsqd2k51b29-auto-patchelf-hook.drv'...
/nix/store/a3priyk68x48s55q7b81s8mmrmh36s9v-auto-patchelf-hook

Success on x86_64-linux (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/5v95psss0kbm65zkiajmwp3sxwhx2i6h-auto-patchelf-hook.drv
building '/nix/store/5v95psss0kbm65zkiajmwp3sxwhx2i6h-auto-patchelf-hook.drv'...
/nix/store/w5wy21k8c6ncchlrpikhxmf3ms697n8f-auto-patchelf-hook

Success on x86_64-darwin (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
these derivations will be built:
  /nix/store/dwz8z4clzb5f37arljbhd34ifmn9010z-clang-5.0.2.drv
  /nix/store/q6iw6cww7baxq43zmwkxj491k8a07s3l-stdenv-darwin.drv
  /nix/store/m04xsjnajcngzy93svpnkvqqxp2qnkyl-auto-patchelf-hook.drv
waiting for locks or build slots...
building '/nix/store/q6iw6cww7baxq43zmwkxj491k8a07s3l-stdenv-darwin.drv'...
building '/nix/store/m04xsjnajcngzy93svpnkvqqxp2qnkyl-auto-patchelf-hook.drv'...
/nix/store/larwcz9jdjriaj9q3drwc5q0k1fb5sii-auto-patchelf-hook

@svanderburg: Fixed, thanks.

Timed out, unknown build status on x86_64-darwin (full log)
Attempted: autoPatchelfHook
Partial log (click to expand)
[ 68%] Updating PPCGenFastISel.inc...
building of '/nix/store/plp1iag0842vjjzlzv7ynn6xahrapvx1-llvm-5.0.2.drv' timed out after 3600 seconds
cannot build derivation '/nix/store/hv8vrp6shlb2lnmg8s9jfgdq56gp61pq-cctools-port-895.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/dwz8z4clzb5f37arljbhd34ifmn9010z-clang-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/gakqwwlvhgnb6y7xp5qxvrjczz7ql4yy-compiler-rt-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/y01mnj37vk51my6yxyjdrzgv2wb00sfx-cctools-binutils-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/gjgmbdh04sfv3kmhdlajm23vsgm0six7-cctools-binutils-darwin-wrapper.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/q6iw6cww7baxq43zmwkxj491k8a07s3l-stdenv-darwin.drv': 6 dependencies couldn't be built
cannot build derivation '/nix/store/0i897mrh9syp4k2qjh2mimnzvbjzk340-auto-patchelf-hook.drv': 1 dependencies couldn't be built
error: build of '/nix/store/0i897mrh9syp4k2qjh2mimnzvbjzk340-auto-patchelf-hook.drv' failed


Run "emqttd start" with default config but default ssl port is not up

Environment

OS: Ubuntu 18.04
Erlang/OTP: 20 [erts-9.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:10] [kernel-poll:false]
EMQ: 2.3.11

Description
A description of the issue
I installed latest emqtt for ubuntu via this link (http://emqtt.io/downloads/latest/ubuntu16_04-deb) in ubuntu 18.04 laptop.
After start emqtt (emqqtd), I got only default tcp port (1883) up. Could not connect ssl port (8883).
Im quite new with mqtt and emqtt also. Pls tell me what I did wrong? I use default config, and though that the ssl config would be applied with default cert files.
Thanks alot. Im do a POC and trying to apply emqtt to our production system.

I test it on my test-server which installed ubuntu 18.04. It has no problem.

Hmmm.
This is what i got

Thanks for your support. Sorry this is my fault. There was the haproxy open port 8883.


Service pipeline stops serving static files after new data committed

What happened?:
I'm using a pipeline that generates artifacts which are then served by a Flask app embedded in a service pipeline. Some of these artifacts are png images that need to be served from static. To achieve this, I use the following service pipeline:
service.json
{
    "pipeline": {
      "name": "dummy_service"
    },
    "input": {
        "pfs": {
            "glob": "/",
            "repo": "do_stuff"
        }
    },
    "service": {
        "external_port": 30887,
        "internal_port": 8887
    },
    "transform": {
        "cmd": [ "/bin/bash" ],
        "stdin": [
            "ln -s /pfs/do_stuff static",
            "python app.py"
        ],
        "image": "dummy:0.0.1"
    }
}

where dummy:0.0.1 is created from the following Dockerfile:
Dockerfile
FROM python
RUN pip install flask flask-cors --no-cache-dir
WORKDIR /app/
COPY app.py .

and app.py is as follows:
app.py
from flask import Flask, url_for, redirect
from flask_cors import CORS

def create_app():
    app = Flask(__name__)

    CORS(app)

    @app.route("/")
    def home():
        return "home"

    @app.route("/<file>")
    def view_file(file):
        return redirect(url_for("static", filename=f"{file}.png"))

    return app

if __name__ == "__main__":
    app = create_app()
    app.run(host="<IP_ADDRESS>", port=8887)

Let's say the do_stuff pipeline processes a datum and produces the output image1.png. Then, I can deploy the above pipeline using:
pachctl create pipeline -f service.json
I can then navigate to <IP_ADDRESS>:30887/image1 and it will successfully display <IP_ADDRESS>:30887/static/image1.png. However, if do_stuff processes a new datum and produces image2.png, then all static locations end up breaking (including image1). Running
pachctl update pipeline -f service.json --reprocess
fixes the problem and allows both files to be served from static.
While broken, the logs show (let pachderm=image1 and elephant=image2):
log
 * Serving Flask app 'app'
 * Debug mode: off
Address already in use
Port 8887 is in use by another program. Either identify and stop that program, or start the server with a different port.
Address already in use
 * Serving Flask app 'app'
 * Debug mode: off
Port 8887 is in use by another program. Either identify and stop that program, or start the server with a different port.
 * Serving Flask app 'app'
 * Debug mode: off
Address already in use
Port 8887 is in use by another program. Either identify and stop that program, or start the server with a different port.
 * Serving Flask app 'app'
 * Debug mode: off
Address already in use
Port 8887 is in use by another program. Either identify and stop that program, or start the server with a different port.
<IP_ADDRESS> - - [16/Nov/2022 22:27:14] "GET / HTTP/1.1" 200 -
<IP_ADDRESS> - - [16/Nov/2022 22:27:19] "GET /pachyderm HTTP/1.1" 302 -
<IP_ADDRESS> - - [16/Nov/2022 22:27:19] "GET /static/pachyderm.png HTTP/1.1" 404 -
<IP_ADDRESS> - - [16/Nov/2022 22:27:36] "GET /elephant HTTP/1.1" 302 -
<IP_ADDRESS> - - [16/Nov/2022 22:27:36] "GET /static/elephant.png HTTP/1.1" 404 -

Note that I am using a symlink in service.json to treat /do_stuff as static. Perhaps this is not the best way to serve files from static? Do you have any other suggestions?
What you expected to happen?:
I should be able to process new datums and serve created artifacts from a downstream static without restarting the service pipeline.
How to reproduce it (as minimally and precisely as possible)?:
Start by downloading some images:
curl https://images.g2crowd.com/uploads/product/image/social_landscape/social_landscape_3cecb217ddd404cf2e71d2612ea5d37f/pachyderm.png --output pachyderm.png
curl https://e7.pngegg.com/pngimages/1001/89/png-clipart-walking-elephant-like-elephant.png --output elephant.png

Then, create a new repo and add one of the png files:
pachctl create repo dummy_data
pachctl put file dummy_data@master -f pachyderm.png

Then, use the following pipeline to do some "processing":
pipeline.json
{
    "pipeline": {
      "name": "do_stuff"
    },
    "input": {
        "pfs": {
            "glob": "/*",
            "repo": "dummy_data"
        }
    },
    "transform": {
        "cmd": ["/bin/bash"],
        "stdin": [
            "cp /pfs/dummy_data/* /pfs/out/"
        ],
        "image": "dummy:0.0.1"
    }
}

Create the two pipelines:
pachctl create pipeline -f pipeline.json
pachctl create pipeline -f service.json

Navigate to <IP_ADDRESS>:30887 and check that the service is running. "hello" should print at the top of the screen. Then, navigate to <IP_ADDRESS>:30887/pachyderm, which should redirect to <IP_ADDRESS>:30887/static/pachyderm.png and the Pachyderm logo should appear.
Add the next datum:
pachctl put file dummy_data@master -f elephant.png
After the file has been processed, go to either <IP_ADDRESS>:30887/pachyderm or <IP_ADDRESS>:30887/elephant and both links will be broken.
Running
pachctl update pipeline -f service.json --reprocess
fixes both <IP_ADDRESS>:30887/pachyderm or <IP_ADDRESS>:30887/elephant.
Anything else we need to know?:
Environment?:

Kubernetes version (use kubectl version): 1.23
Pachyderm CLI and pachd server version (use pachctl version): pachctl 2.3.5, pachd 2.3.5
Cloud provider (e.g. aws, azure, gke) or local deployment (e.g. minikube vs dockerized k8s): Local k3s
If you deployed with helm, the values you used (helm get values pachyderm):
OS (e.g. from /etc/os-release): Ubuntu 18.04.6 LTS
Others:

Thanks @dsgibbons for submitting this. I'll talk to the team and we will get back to you shortly.

An immediate workaround would be to exec python app.py, though we tested that internally and didn't get that to work in every case.  It worked with my random version of python3 and flask, but not with someone else's.
The problem here is that we send SIGKILL to bash, not python, and bash just leaves python running when it gets SIGKILL.    (But if python spawns additional processes to handle HTTP requests, we can't kill those.)
Longer term, PR #8385 fixes this problem in general.  I'll let you know when that's available to test in a nightly release.

Great, thank you very much for that @jrockway. I'll try the short-term fix and see if that helps. I look forward to testing your PR.

Looks good. I'll raise this with my team and see if we can test this fix next week. Thank you!

That is good to hear.  Feel free to @jrockway if you have anything to report!

@jrockway We tried pulling the latest helm chart version, but it only seems to go up to 2.4.1 (which we currently have installed). Is your public helm chart updated to include nightly builds? Are the pachd container images for nightly builds public? If so, where can we find them?

Ok I've finally gotten around to testing this fix. It works in the sense that the service pipeline doesn't go down if a new datum is added. Unfortunately, the new datum is not recognised by the service pipeline - a restart of the service pipeline is required to locate the new datum in the static/ folder. Perhaps symlinking to static/ is not the best way to have the service pipeline update to serve new artefacts? Do you have any suggestions?


Update README.md

Add React-Native Init Example
Add iOS10 Privacy Settings Guide
Fix Issue with #4

I've updated the README with the relevant changes to address the issue introduced in iOS 10. The changes are in version 0.0.7 of the package, thanks for bringing up the issue!
I'm going to close this PR for now, feel free to reopen if you think there are still outstanding concerns. Cheers!


I want to add Dijkstra’s shortest path algorithm in c and c++

Need to add a description and your approach. Thanks, Closing as of now.


Backs up only outgoing texts - Samsung Android 2.1

I use a Samsung Galaxy 5 with a Eclair (2.1) installed.  All messages sent and received until April have been backed up perfectly in a separate Gmail folder. But thereafter, only outgoing messages would back up, it hasn't backed up any incoming texts since then. I need over a 1000 very important texts need to back up asap!
Would appreciate some help here.
Thanks!

If it's really important use another app to backup to SD card so long...


Do .dump_tree() in JSON format as well

This feature request is posted on StackOverflow: https://stackoverflow.com/q/66239653/3648361
It sounds like a good idea. The suggested returned value is not exactly JSON string, but a serializable structure of dict and list that can be saved to JSON file by one liner.

Thanks, @vasily-v-ryabov  ! Any high-level thoughts on how to do this?
I am thinking to implement it by doing string manipulation after dumping the print_control_identifier into file.
If any other thoughts if you have please let me know, I can contribute to this. As I need it urgent

Thanks for looking into this enhancement @vasily-v-ryabov ! . Do you have any high-level idea on how to do this?
At this point what I am thinking is to do string manipulation to get the desired output. Let me know if you have other thoughts.
I can think about it . !

First we review only pull requests to branch "atspi" which contains some re-factoring. So .wrapper_object() is renamed to .find(), some keywords in .child_window(...) were changed, child_window(...) was renamed to .by(...) and so on. We made detailed error messages so it should be easy to adapt your old code to the new interface. I just want to let you know what you may face on your way.
JSON-nizable data can be prepared inside .dump_tree() method without any string manipulation. You need to append all the data to return value object using list and dict methods. Just take a look into .dump_tree() implementation in "atspi" branch and ask questions.
I'd suggest to make it usable this way: output_obj = dlg.dump_tree(jsonize=True). Then you can dump it to plain JSON string by json.dumps(output_obj).

@eltimen FYI.

@sreevathsabr you can install "atspi" version from the branch this way (just in case):
pip uninstall -y pywinauto
pip install https://github.com/pywinauto/pywinauto/archive/atspi.zip

@vasily-v-ryabov Thanks for your suggestion. I will look at it and work on it.  Once I achieve the desired output, sure try to push the code to here.

@vasily-v-ryabov  I have implemented this feature locally and have done testing on it. It works pretty well . I am following the instruction of contribution to add my code to repo . If there any specific process to follow let me know .

We have only this draft guide: https://github.com/pywinauto/pywinauto/wiki/Contribution-Guide-(draft)
Local run of unit tests may be problematic in some cases. Don't hesitate to ask us if you see some issues not connected with your code. Maybe set up AppVeyor for your fork is more reliable than local test runs.
BTW, have you synced with upstream/atspi branch after we merged PR #1049 ? I don't see any pushed commits in your fork.

If there is any commit, I can help to integrate it into upstream.

I have a code to which I have made changes, Can I commit right away? Do I need to run any tests around it?  I went through the guide I am confused with it.  I can push the code , if it's ok to commit .

Sure. You're welcome to push a commit to your fork. If you have any feedback about the guide, you're welcome to provide it. We're happy to improve contributor experience.


[#3936] resource version fix for discover/index

Pull Request Checklist:

[ ] Positive Test Case Written by Dev

[ ] Automated Testing

[ ] Sufficient User and Developer Documentation

[ ] Passing Jenkins Build

[ ] Peer Code review and approval

Positive Test Case

[Enter positive test case here]

METRIC
VALUE

https://sonarqube.cuahsi-workstation.com:9000/dashboard?id=hydroshare-4902


MongoDB auth error

Environment

VerneMQ Version: 1.5.0
OS: Centos 7 (Docker 18.06)
Erlang/OTP version (if building from source): 20

Preface
We are added some required for us attributes into document to solve autodelete document issue
{
	"_id": ObjectId("5ba8d21797f1f8adc3311654"),
	"client_id": "90019b20-d7a3-42df-9bdf-81670a34de20",
	"created": ISODate("2018-09-24T11:40:02.806Z"),
	"mountpoint": "",
	"passhash": "$2a$10$Ddo3V6cq5j4adJdjhDssGej4GVyjomCe6fD6RDt3vqQiCeMQfdvmW",
	"publish_acl": [{
		"pattern": "test"
	}],
	"subscribe_acl": [{
		"pattern": "test"
	}],
	"system": false,
	"username": "test",
	"expired": ISODate("2018-09-24T12:40:02.906Z")
}

Expected behavior
Normal authentication
Actual behaviour
Authentication error
12:48:15.326 [error] can't call function auth_on_register with args [{addr,<<"<IP_ADDRESS>">>},{port,41380},{mountpoint,<<>>},{client_id,<<"90019b20-d7a3-42df-9bdf-81670a34de20">>},{username,<<"user">>},{password,<<"password">>},{clean_session,true}] in "/opt/vernemq/share/lua/auth/mongodb.lua" due to {error,badarg}
12:48:15.326 [warning] can't authenticate client {[],<<"90019b20-d7a3-42df-9bdf-81670a34de20">>} due to error

lua script cat parse value of created attribute but its regular Mongo func
Removing this attribute solves ther problem but its not right solution for us

Hi, it looks like the LUA script doesn't really care about the extra items - it looks up the document by the client_id, username and mountpoint - and then retrieves the publish_acl and subscribe_acl files. Are you using the stock lua script?
Are you sure it's because of the ISODate function? Could you experiment with removing them again and adding raw "created" and "expired" fields?
I'll have to test this out to see why this fails - I probably won't have time to look into this today, though.

Yes we are using stock lua scripts

Are you sure it's because of the ISODate function?

If change value of this fields to string - everything all right.
But we need to use ISOSate func

It seems the mongodb driver returns the mongodb Date type as an erlang timestamp tuple, which the function used to map mongodb results into lua doesn't understand returns as is - which makes lua throw a badarg error

There's a fix ready in #875 - it would be great if you could test it out.

Yes, it works


API to generate a graph to visualize the update to tasks and results

Can you provide the link to the hackmd @aneojgurhem to know intents and purposes?


 My suspicion is folks wouldn't invoke that with the intention of deleting all your history. I was sort of expecting it to let me choose (kind of like #854) suggests. To prevent future people from repeating my Friday afternoon misfortune, can we at least warn folks about what will be deleted before deleting it?

Oh yikes, sorry about that! 😭

My suspicion is folks wouldn't invoke that with the intention of deleting all your history

I know of several instances where people have done that with the explicit intent of deleting all their history, but this does seem way too easy to get wrong 🙃
The original intent was for it to work similarly to the --delete flag to find, but I'll see about getting a confirmation dialogue in.
Otherwise, we will be adding deletion to the interactive UI, but will be doing it as part of a larger UI change!

I've just opened this to try and make it harder to accidentally delete everything!
https://github.com/ellie/atuin/pull/878
We might be able to recover your history, if you haven't already managed to somehow. Would you like me to see if it can be sorted?

another idea, is to list out what (or how much) will be deleted and ask for confirmation

If you're using sync, we might be able to recover your history. If you haven't already managed to somehow. Would you like me to see if it can be sorted?

I'm not unfortunately, I was just running it locally. It's all good, not the end of the world. A simple option here may be just to have an empty argument to fail. I haven't tried wildcards but I could see atuin search --delete * deleting everything. Ultimately confirming destructive actions would probably be a better longer term UX (if for no other reason than to save me from myself lol).

I think an interesting question here is whether or not this CLI is conceptually more of a UI (for humans) or an API (for programs).
Obviously this isn't mutually exclusive, since humans are the ones who end up programming use of the APIs. But I maybe if we imagine a variety of programmatic uses of this interface, we'll notice that unqualified delete hitting everything is super useful somehow.
(Like how at the human level people are sometimes tempted to return a single thing instead of a list of one element when there's only one result, but for programmatic use, the regularity of always getting a list even if it has one or zero elements ends up being more useful.)
Having said that though, I think a good case could be made that unlike regular search, delete should be an additive interface rather than a subtractive one: where the default is to delete nothing, and further parameters add things to that empty to-be-deleted set.

If you're using sync, we might be able to recover your history. If you haven't already managed to somehow. Would you like me to see if it can be sorted?

I'm not unfortunately, I was just running it locally. It's all good, not the end of the world. A simple option here may be just to have an empty argument to fail. I haven't tried wildcards but I could see atuin search --delete * deleting everything. Ultimately confirming destructive actions would probably be a better longer term UX (if for no other reason than to save me from myself lol).

Ahh ok, no worries. I've merged the PR I mentioned - it was more or less what you suggest there, so this should not happen again. Sorry about the not-so-great experience you had here!


[Documentation] Improve infrastructure documentation and lings

Objective
Make infrastructure-related documentation accessible and increase visibility into DevNet
Origin Document

Links:

https://www.notion.so/pocketnetwork/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd
https://github.com/pokt-network/protocol-infra
https://www.notion.so/pocketnetwork/V1-networks-7d95c10c930c45c3823c871f21d44fca?pvs=4

Goals

Increase visibility into the ongoing work in DevNet
Make infrastructure-related dependencies and documentation as accessible as that related to protocol development

Deliverable

[ ] Make sure all the links above are our main DEVELOPMENT readme
[ ] Leave a TODO to productionize them as READMEs in the future (add a comment in the docs)
[ ] Point this out explicitly in the internal new-hire guide
[ ] Link to our DevNet grafana dashboard directly in the main README of pocket-core/pocket
[ ] Bonus/optional: productionizing the notion docs to markdown in the repo

Non-goals / Non-deliverables

Changing the existing notion docs

General issue deliverables

[ ] Update the appropriate CHANGELOG(s)
[ ] Update any relevant local/global README(s)

Creator: @Olshansk
Co-Owners: @okdas @dylanlott

I'm going to use this ticket to attribute the work related to the Devnet workshop preparations. By the end of it, I should have thoroughly documented guides on how to use Devnets.


SWARM-1629: Update copyrights

Please retest, I updated wildfly-swarm-docker-test-linux to not run on Windows, which I think will help with the Linux failure

retest this please

restest this please


Nuxt compiled in Development mode

Hi!
For some reason the 0.16.x versions of this builder make it so that my Nuxt project is deployed in development mode (with Vue's productionTip in the console). Unless I use v < 0.16.x of this builder, I always get dev build on Now
Here's my project repo:
https://github.com/msurguy/nuxt-express-now

Thanks for the bug report @msurguy. I've put forward a PR which you'd be welcome to test with nuxt/now-builder#fix-development-node-env (although note https://github.com/nuxt/now-builder/issues/64#issuecomment-505351231).

@danielroe sorry for the question, how would I test this dev version? I've tried specifying it like this without luck:
"builds": [
    {
      "src": "nuxt.config.js",
      "use": "@nuxtjs/now-builder#fix-development-node-env",
...

@msurguy Specify it like this:
"builds": [
    {
      "src": "nuxt.config.js",
      "use": "nuxt/now-builder#fix-development-node-env",

Thanks @danielroe! It worked well and was correctly built and deployed. PS, I didn't have to move any modules to devModules.

Fixed with<EMAIL_ADDRESS>


Memory limitation in netcdf export step

The read_from_npy function in particlefilesoa.py generates a 64-bit array with dimension (number_of_trajectories, number_of_unique_time_recods) so the capacity to store this array in memory is a requirement to export the npy files dumped by parcels to netcdf. For long runs/frequent output, this restriction is much stricter than the memory limit on the number of active particles/field size. Furthermore, if there is insufficient memory to store this array, this will only be flagged as an OOM memory during export to netcdf (i.e. after the simulation has completed), which may result in a lot of time being wasted.
I have two suggestions:

It is not possible to predict the size of this array before particle tracking has completed, but I think the minimum possible size is (number_of_trajectories, run_time/output_dt), and all of these variables are available prior to execution. So whilst this would not guarantee that there is sufficient memory, throwing a warning if 8*number_of_trajectories*run_time/output_dt > available_memory would at least catch runs that are guaranteed to fail at the export step (assuming that BG memory use does not vary by much).
Evaluate whether 8*(self.maxid_written+1)*len(self.time_written) >= psutil.virtual_memory()[1] in the early stages of the export function and, if true, read the numpy files and write to the netcdf file in chunks, rather than trying to store all data from a particular variable in memory at once. I've written up a naive modification of the relevant functions in particlefilesoa.py below (which splits particle IDs into chunks with limits in id_range, and only loads particle IDs satisfying id_range[0] <= id < id_range[1] at once). This is not a great implementation and I have only carried out basic tests, but I would be happy to work on this further if you think this method is reasonable and useful.

def read_from_npy(self, file_list, time_steps, var, id_range):
    """
    Read NPY-files for one variable using a loop over all files.

    Attention:
    For ParticleSet structures other than SoA, and structures where ID != index, this has to be overridden.

    :param file_list: List that  contains all file names in the output directory
    :param time_steps: Number of time steps that were written in out directory
    :param var: name of the variable to read
    """

    valid_id = np.arange(id_range[0], id_range[1])
    n_ids = len(valid_id)

    data = np.nan * np.zeros((n_ids, time_steps))
    time_index = np.zeros(n_ids, dtype=np.int64)
    t_ind_used = np.zeros(time_steps, dtype=np.int64)

    # loop over all files
    for npyfile in file_list:
        try:
            data_dict = np.load(npyfile, allow_pickle=True).item()
        except NameError:
            raise RuntimeError('Cannot combine npy files into netcdf file because your ParticleFile is '
                               'still open on interpreter shutdown.\nYou can use '
                               '"parcels_convert_npydir_to_netcdf %s" to convert these to '
                               'a NetCDF file yourself.\nTo avoid this error, make sure you '
                               'close() your ParticleFile at the end of your script.' % self.tempwritedir)

        id_avail = np.array(data_dict["id"], dtype=np.int64)
        id_mask_full = np.in1d(id_avail, valid_id) # which ids in data are present in this chunk
        id_mask_chunk = np.in1d(valid_id, id_avail) # which ids in this chunk are present in data
        t_ind = time_index[id_mask_chunk] if 'once' not in file_list[0] else 0
        t_ind_used[t_ind] = 1
        data[id_mask_chunk, t_ind] = data_dict[var][id_mask_full]
        time_index[id_mask_chunk] = time_index[id_mask_chunk] + 1

    # remove rows and columns that are completely filled with nan values
    tmp = data[time_index > 0, :]
    return tmp[:, t_ind_used == 1]

def export(self):
    """
    Exports outputs in temporary NPY-files to NetCDF file

    Attention:
    For ParticleSet structures other than SoA, and structures where ID != index, this has to be overridden.
    """

    if MPI:
        # The export can only start when all threads are done.
        MPI.COMM_WORLD.Barrier()
        if MPI.COMM_WORLD.Get_rank() > 0:
            return  # export only on threat 0

    # Retrieve all temporary writing directories and sort them in numerical order
    temp_names = sorted(glob(os.path.join("%s" % self.tempwritedir_base, "*")),
                        key=lambda x: int(os.path.basename(x)))

    if len(temp_names) == 0:
        raise RuntimeError("No npy files found in %s" % self.tempwritedir_base)

    global_maxid_written = -1
    global_time_written = []
    global_file_list = []
    if len(self.var_names_once) > 0:
        global_file_list_once = []
    for tempwritedir in temp_names:
        if os.path.exists(tempwritedir):
            pset_info_local = np.load(os.path.join(tempwritedir, 'pset_info.npy'), allow_pickle=True).item()
            global_maxid_written = np.max([global_maxid_written, pset_info_local['maxid_written']])
            for npyfile in pset_info_local['file_list']:
                tmp_dict = np.load(npyfile, allow_pickle=True).item()
                global_time_written.append([t for t in tmp_dict['time']])
            global_file_list += pset_info_local['file_list']
            if len(self.var_names_once) > 0:
                global_file_list_once += pset_info_local['file_list_once']
    self.maxid_written = global_maxid_written
    self.time_written = np.unique(global_time_written)

    for var in self.var_names:
        # Find available memory to check if output file is too large
        avail_mem = psutil.virtual_memory()[1]
        req_mem   = (self.maxid_written+1)*len(self.time_written)*8*1.2

        if req_mem > avail_mem:
            # Read id_per_chunk ids at a time to keep memory use down
            total_chunks = int(np.ceil(req_mem/avail_mem))
            id_per_chunk = int(np.ceil((self.maxid_written+1)/total_chunks))
        else:
            total_chunks = 1
            id_per_chunk = self.maxid_written+1

        for chunk in range(total_chunks):
            # Minimum and maximum id for this chunk
            id_range = [chunk*id_per_chunk,
                        np.min(((chunk+1)*id_per_chunk, self.maxid_written+1))]

            # Read chunk-sized data from NPY-files
            data = self.read_from_npy(global_file_list, len(self.time_written), var, id_range)
            if var == self.var_names[0]:
                self.open_netcdf_file((self.maxid_written+1, data.shape[1]))
            varout = 'z' if var == 'depth' else var
            getattr(self, varout)[id_range[0]:id_range[1], :] = data

    if len(self.var_names_once) > 0:
        for var in self.var_names_once:
            getattr(self, var)[:] = self.read_from_npy(global_file_list_once, 1, var, [0, self.maxid_written+1])

    self.close_netcdf_file()

Thanks for raising this Issue and possible workaround, @nvogtvincent! As a pure coincidence, @SLYpma, @quintenbohte and myself were discussing how to circumvent these memory issues too at the time you raised this Issue!
I think your suggestion of working with chunks could indeed work. I don't entirely understand whether the different chunks all end up in the same NetCDF file (and if so, how can you keep appending data?), or whether you get one NetCDF file per chunk. Feel free to put this into a fork and create a Pull Request, then we can work on it together (and you get the credit for this idea). Creating a Pull Request also better highlights exactly which changes you made to the code
More generally, I think the problem is that NetCDF just isn't very suited for trajectory data, especially when the trajectories have variable length. I've been discussing this before with @CKehl, but maybe we should think of other datastructures (databases??) to store the data, that are more flexible than the array-based NetCDF format. Although of course it would mean that users also need to learn new tools. Any thoughts welcome!

Just wanted to link a discussion I saw earlier over at https://github.com/MITgcm/MITgcm/issues/291 and https://github.com/ocean-transport/floater/issues/78, about Lagrangian data structures.

@nvogtvincent Thank you. In the meantime, I will try to convert the output to a netCDF file with ragged rows, to see how practical that is to use. I shall share results on a new thread once I have them. I will write a standalone utility to convert the existing netcdf to ragged rows until people think the format is useful enough to include.

I'm thinking that implementing zarr could be very useful. I don't have much experience with it, but as I understand it, zarr arrays can be (dynamically) written in chunks, much in the same way as Parcels now spits out numpy arrays. The added benefit is that the zarr output can be read directly with xarray, without the need for any netCDF conversion. Moreover, zarr also supports ragged arrays, which is useful for simulations similar to @JamiePringle's.

@daanreijnders See my netCDF code in "ragged netCDF output #1157"
I am intrigued by Zarr, but do not have experience with it yet. One comment -- even if it can be written in chunks, parcels may not produce the data in the right order. Parcels produces lots of particle data for a given time; often the best way to write chunks is as sets of all time data for a single drifter...
But I don't really know -- it would be nice if someone could re-write my "ragged netCDF" utility to write Zarr, and then they could be inter compared. It should not be that hard (TM).

@erikvansebille, @nvogtvincent   and @CKehl are there any plans to merge the chunking patch ( #1092 ) into the development branch, or is there an issue with it that I can help fix?  As far as I can tell the unit test output, it is just failing a lint check, but there is no output to figure out why it is failing.
Even with a 128Gb memory machine, ocean parcels is failing save its output due to memory limitations for my dense connectivity runs. I am having to save data with an ad-hoc script from the temporary *.npy files.
If y'all do not want to pursue #1092 I am happy to try to contribute a code to solve this problem. Perhaps something along the ragged netCDF output I shared in discussion would make it easier to write the output without having to have all times and trajectories in memory at once.
I will contribute a standalone code to go from *.npy temporary files to ragged netcdf (I have to do so to get my work done), but I don't want to duplicate the effort on #1092 unless there is some reason y'all think this patch won't go forward.
Jamie

Hi @JamiePringle, I don't have any immediate plans to finish #1092 because (1) it wasn't a very elegant solution and (2) I remember that I had problems using this code on HPCs because psutil.virtual_memory()[1] wasn't returning the correct amount of available memory on our cluster.
I'm also occasionally running into OOMs during particlefile export so I'm not surprised that you're also having problems, but I've just been working around this because unfortunately I haven't got the time to write a general fix for this. So if you're willing to try to implement a solution, at least from my perspective, that would be great.
But FYI, whilst ragged netCDF output could be a really useful addition to Parcels and could partially mitigate this problem, it isn't a complete fix (unlike some method involving chunking) because I've had OOMs with extremely large particle numbers even using write_ondelete=True, i.e. the smallest possible array that includes all particles. I'm wondering if Dask or something similar has a chunked netcdf export option?

@nvogtvincent @erikvansebille I have created a stand alone python code that takes the output in the out-XXXXX directory and converts it into a netCDF file which is, as far as I can tell, exactly the same as the default output from parcels.  The only feature I have not implemented is the creation of variables that are only written once.
The maximum memory usage of this code is (maximum number of time records for a single drifter)*(number of drifters in a single MPI rank). So in the case where you use MPI to parallelize the runs, or when drifters only last a portion of the parcels run, this memory usage is much less than in the existing code.  I convert a run where each drifter exists for 64 time records and there are 47 million particles using about 12Gb of memory.
The downside is that all of the *.npy files are read twice. I don't think it is much slower than the existing code, but then again my filesystem is heavily tuned to cache often read files.
This code could easily be tuned to use even less memory by only caching the data for a single variable at a time, but at the expense of reading the *.npy files more times. I do not think worth it. As it is I can use it to process runs with 10s of millions of drifters without memory issues.
I have not yet tried to integrate it into parcels. There are decisions @erikvansebille and @CKehl need to make about tradeoffs between memory usage an IO. ALSO, there are many different things that can effect the *.npy files. Before thinking of incorporating this code into parcels, I think the following should happen:

This code should be tested on other folks' output records. There are lots of potential edge cases. I have taken care to process user defined variables in the particle set properly, but, for example, I have assumed the user defined variables map onto netCDF types
I have not implemented the *_once variables, since I have no examples of runs that produce them
I don't know what to do if 'lonlatdepth_dtype' is ever inconsistent with 'var_dtypes'
I have only tested this on SOA particles... I think...
I have assumed that particle ID is unique to each drifter
The output is not sorted by particle ID. Not a problem for me, but is different from existing output.
I assume dictionaries are ordered, so this limits me to python versions >= 3.7. If this is an issue I can specify orderedDicts explicitly.

To run the code, one must keep the temporary output file. I am not sure how to do this properly, but I do it by not calling .close() on the pset. I am not sure this works for non-MPI runs, but could be made to by changing how the temporary directory is created. I always use MPI.
The code is then called with: "python convertToNetcdf_from_npy_lowMem.py outDirectoryName outputFileName"  As written it is rather chatty, but if you tire of that you can set "printDebug=False".
The code is in the attached zip file.
convertToNetcdf_from_npy_lowMem.zip

This has been fixed now with the implementation of native zarr output in #1199 and v2.4.0


Please do Add Notification Image Support

Fix the issue of showing only the same notification icon to all the Media Notifications, Please add an option to pass the Path of the image file to the MediaNotification.showNotification so that we could pass the image to be displayed on the notification.
Also, Thank You For Creating This.

That would be great indeed :-)


⚠ GitLab Server has degraded performance

In 20aabb2, GitLab Server (https://gitlab01.its-telekom.eu) experienced degraded performance:

HTTP code: 200
Response time: 6628 ms

Resolved: GitLab Server performance has improved in 2227c30.