Cloudflare D1 10gb size limit

The strict database size cap of 10gb for Cloudflare D1 is quite limiting for large projects.
Drizzle currently supports many database drivers, some of which are edge-friendly.
Perhaps something like a Planetscale integration would be useful for larger projects to get around these limits.
Being able to use the NuxtHub's tasty tooling with other relational data sources would be a big DX win.
Additional context
Cloudflare D1 limits documentation
Rumors that the size cap will probably remain

Hey @birdlavv
What other database are you thinking about to use instead of D1?
I will also ask the Cloudflare team about those limits.

Thanks for looking into it, @atinux!
Currently, we are using Planetscale until D1 can scale to a larger size.

Then I guess the solution might be #208

Being able to use any relational DB (via #208) in the Drizzle UI in the NuxtHub dashboard would be 🔥
According to the Hyperdrive docs, MySQL support (Planetscale/Vitess) is not yet available. There does seem to be some movement on the Planetscale end.
D1 with a bigger size cap would be ideal due to the incredible Cloudflare pricing.

I didn't know of the 10GB Cloudflare limitation! This is a serious limitation for real world production apps...
Turso is a think a closer alternative to D1. I've used it before and its very good, ultra nice DX.
If Cloudflare is reluctant to increase that 10GB limit, I'm afraid making NuxtHub database agnostic is very important, maybe unjs/d0 can help in this regard?

TLRD: In general, I love the idea of NuxtHub wiring-up modules on Cloudflare by default (e.g. D1 for database, Workers KV for KV-storage, R2 for blob, ...) but being open to wire-up other providers for these modules by leveraging Nitro/UnJS.
I'm pretty exited for first party Nitro Database, and in general, IMO one of the greatest strengths of NuxtHub is leveraging nitro and maintaining its provider-agnostic spirit (powered also by the great UnJS ecosystem).
I definitively agree, it would be great if the admin UI of NuxtHub uses useDatabase() in order to connect to the databases defined in the project's nitro.config/nuxt.config. Then the user should be able to select which database he wants to view/edit, much like the "DevTools > Storage" UI allows selecting which drive from multiple.
Allowing this, despite having to manually create these resources externally on the corresponding providers without an automation would probably already be good enough for more users (I speak at least for myself), and could probably be come faster. The automations could be introduced later if the team decides it, I understand this might be out of scope for NuxtHub itself, which is built (at least for now) primarily on Cloudflare.


[DOCS] Write a prompt engineering tutorial

After a discussion related to this recent tutorial, I suggested that a similar tutorial showcasing a GPT3 prompt engineering workflow with argilla would be exciting for a lot of people. This paper from the ACL 2023 industry track found that prompt engineering - if done well - can be both cheaper and lead to better performance (in terms of saving time and money) than fine-tuning for a use case in customer service. This would serve as a terrific motivation for such a tutorial.

Hi @mnschmit @davidberenstein1957 I have read the paper and found it very practical and resourceful. Would love your insights on it and start a tutorial with relevant use-case and evaluation metrics. I am not clear on the dataset and evaluation metrics. Would love to connect on this.

@mnschmit @vasudevanubrolu are you still working on aligning this?

Hi @davidberenstein1957 I started my work on argilla this week, will keep you posted on the progress on the items assigned to me

hi @vasudevanubrolu, that would be great. Thank you 👍  Let me know if you need help.

Likewise, let me know if you need help :)


Asp Provider does not load deeper nested objects correctly

Key prefix is lost when processing deeper nodes.
Also empty entries are added too the provider dictionary that should not be there.

Released with 0.11.0

Released with 0.11.0

Released with 0.11.0


fix typo in comment

Coverage remained the same at 76.586% when pulling 28a60153a72503b409041467b6d62f07a1ae48bc on fstab:patch-1 into 2df2a8983e35dff23c4fb894e5f5ae3f98f1cf7b on opentracing:master.

Thanks!


Filter box overflowing

Our filter box doesn't currently add a scrollbar when the number of errors overflow - we need to add this.

Is this the issue illustrated in the attached image? If so then I agree!
Also, can these errors be sorted (probably by number) in this list to make them easier to find?

Really helpful - yes we will do both of those things

On the sorting - currently they are sorted by the error with the highest number of occurences - not sure if sorting by number is better?

I don't know how everyone else works - but when I'm addressing errors in my 903 data I try to deal with all of the similar errors at the same time - so if I've summoned up the courage to tidy up all of the postcode problems then I want to identify them - and to do that I'd currently have to find out how many of them there are and then scroll down to find them. I already know the error IDs, so if they were sorted in that order I could scroll straight to them. Also, if I fix half of them and then re-filter I have to go and look up how many there are now before I can find them again.
Obviously others may work differently though...

That makes a lot of sense - I think there are a few cases like this where we might want to allow for different sorting. I'll have a think about how we can implement it.

Yeah, I think this is something that has multiple use-cases:

load data and get a quick sense of where the errors are (sorted by number of errors)
load data and review/work on a specific error (sorted by error code for ease of finding)

To be honest though any form of sorting would be an improvement over the reporting DFE provide in the submission portal!
Stretch goal: I'm not sure whether DFEs concept of Stage 1 and Stage 2 errors has come up in this project yet but being able to sort based on that might be something worth considering too, or even just a way to flag that an error is Stage 1.  Less directly important if this tool doesn't suppress S2 errors where S1 errors also exist but helpful for context and in comparison with what we'll be seeing at year-end.

closed by https://github.com/SocialFinanceDigitalLabs/quality-lac-data-beta/commit/0e62996b2a3d99c3c1b384bcec0cb8c1601bac69


Setup taskfile

Add Taskfile.yml with a minimal set of common development commands.

@Alex-Izquierdo @ddonahue007 please review


Unless you have a flux capacitor, the date of the most recent release is incorrect

A minor OCD change.

Might be the best commit message ever... Thanks!

GREAT SCOTT!


Fixes #3679: Issues with new media details UI

Issues with new media details UI
Fixes #3679
Tests performed (required)
Tested prodDebug on Honor Play with API level 29.
GIF of what's changed

Codecov Report

Merging #3697 into master will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##             master   #3697      +/-   ##
===========================================
- Coverage      7.35%   7.33%   -0.02%
  Complexity      335     335
===========================================
  Files           292     292
  Lines         12896   12916      +20
  Branches       1020    1022       +2
===========================================
  Hits            948     948
- Misses        11871   11891      +20
  Partials         77      77

Impacted Files
Coverage Δ
Complexity Δ

...fr/free/nrw/commons/media/MediaDetailFragment.java
0.00% <0.00%> (ø)
0.00 <0.00> (ø)

...n/java/fr/free/nrw/commons/upload/UploadModel.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

...e/nrw/commons/upload/UploadMediaDetailAdapter.java
0.00% <0.00%> (ø)
0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 840e126...d58b38b. Read the comment docs.

It works as intended, solves the resizing issue. But I pointed some code style issues, after they are fixed we can merge this one.

@neslihanturan I've made the changes :)

Thanks @kbhardwaj123 , interesting why I can't see your commit for code style changes? Code looks okay though.

I can see them when pulled to local just the web interface doesn't display them. Thanks for the fix @kbhardwaj123 !


GREL script not adding DaSSCo to Project Number field for NHMA exports

The GREL script for OpenRefine seems to be correctly formatting all data for the NHMA exports, except for the Project Number column. It creates this column but does not fill it with "DaSSCo", instead it just creates a blank column.

I've updated the script and now just need to update github

The script has been updated on github


add some visual effects to where undo/redo occurs (feature request)

Operating system or device - Godot version:
Official 2.0.2_stable win64
Issue description (what happened, and what was expected):
When I work on several scenes at the same time, the undo/redo stack apply to all the scenes.
Having one undo/redo stack per scene would be less confusing, but as @reduz  stated into #3511, it is not possible to change this behavior.
So, to make it less confusing, what about adding some visual effects to where a undo/redo is applied ?
For instance, when a property or a node is undone/redone, something like a red or yellow circle could brievely appear over the concerned item to attract eyes.
I think it would make easier to follow the undo/redo flow.
Steps to reproduce:
Link to minimal example project (optional but very welcome):

I whole-heartedly agree, I find undo and redo to currently be only dangerous things that do mostly invisible confusing changes to my stuff, breaking things, and leaving me lost. :-(

Sounds good, though the UndoRedo class would be painfully changed if we need to accommodate for this..

Bugsquad note: This feature is still not implemented as of commit https://github.com/godotengine/godot/commit/4d787869e189b670d609f33c308991741592e40a.

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.
The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.
If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!


Entire URL cannot be extracted in handler, only path is provided

Expected Behavior
uri: &Uri - Entire request url is provided
Actual Behavior
Only path is provided
Steps to Reproduce the Problem
My handler:
#[handler]
pub async fn test_get_url(uri: &Uri) -> Response {
    println!("Request uri: {}", uri);
    Response::builder()
        .status(StatusCode::OK)
        .header("Content-Type", "text/plain")
        .body("OK".to_string())
}

Request:
curl --request GET \
  --url http://<IP_ADDRESS>:8000/xd/

print result:
Request uri: /xd/
Specifications

Version: poem version = "3.1.1"
Platform: MacOS

Maybe we don't have the information of the base url?
How about set to the server then the poem will auto combine them?


Random walks II | MIT Computational Thinking Spring | Lecture 13

Random walks II | MIT Computational Thinking Spring | Lecture 13
7:20 Introduction
8:45 Julia concepts
10:18 Pascal's triangle
23:00 Convolutions build Pascal's triangle!
26:20 Random walks: Independent and identically-distributed random variables
29:30 Random walks as a cumulative sum
31:38 Cumulative sum
42:29 Evolving probability distributions in time
55:25 Implementing time evolution of probabilities

sweet, thank you!


User-supplied certs should be mounted even if TLS is disabled

Right now, no certs are mounted to the Cryostat pod when TLS is disabled in the operator. Certificates specified with TrustedCertSecrets should still be added to the truststore.

This branch should be moved outside of the outer if tls != nil branch:
https://github.com/cryostatio/cryostat-operator/blob/90c3363e6f7ae448809db812ea16565ede27dce7/internal/controllers/common/resource_definitions/resource_definitions.go#L263-L286
I think this can be refactored as follows:

Initialize volSources before if tls != nil branch
Create projections for each TrustedCertSecret, if any. (This is the branch referenced above)
Inside if tls != nil branch, add the self-signed CA projection.
After the if tls != nil branch, create the Projected Volume using volSources accumulated in previous steps.
Append the Projected Volume to the volumes slice.

We will then need to add a test case for a Cryostat CR with enableCertManager: false and one or more trustedCertSecrets results in a Deployment with the certificates mounted in the projected volume.
It's also worth checking in an actual cluster that a Projected Volume with zero sources (no cert-manager, and no trusted cert secrets) does not result in an error. I would expect this to just mount an empty directory, but it would be good to be sure.


Automatic conversion between embedded rid bag and bonsai rid bag is broken in the tx mode

Seem like there is a special update handling logic applied in the tx mode to lower the overhead on remote commit, as a side effect the conversion is broken.  com.orientechnologies.orient.test.database.auto.ORidBagTest#testFromEmbeddedToSBTreeAndBack will be disabled.

After discussion with Andrey we have decided to modify the test to support only upgrades to bonsai rid bag version, since downgrades are disabled by default.

What happens if a database goes distributed? Are the non embedded ridbag transformed in embedded again? This is because distributed bonsai ridbags are not supported yet.

@lvca embedded ridbag never converted we never supported that Sergey opened that bug by mistake not functionality but test was incorrect.

OK I will close it then


fix prefix not being validated for other slashes

:loudspeaker: Type of change

[ ] Bugfix
[x] New feature
[ ] Enhancement
[ ] Refactoring

:scroll: Description
FIxes #146
:bulb: Motivation and Context

:green_heart: How did you test it?
:pencil: Checklist

[x] I reviewed submitted code
[x] I added tests to verify changes
[ ] I updated reference documentation to reflect the change
[x] All tests passing
[x] No breaking changes

:crystal_ball: Next steps

@MatejNedic can you fix the build?


Add Highway Support

A follow-up will be needed to structurally solve roundtrip information retention

Oops, there you go, sorry :(

LGTM! Big step forwards to not saying a building has 2 lanes ;)


Needs GATT characteristic for remote firing

There needs to be a GATT characteristic added which supports read, write, and notify when the blaster is being fired and revved.
This will allow remotely firing the blaster from secondary positions on the battlefield.

If this is to be done, the security would need to be shored up before enabling this. To prevent unintentional discharge.


ARM template deployment error

I used the Deploy to Azure option to deploy the baseline template and get the following error message joining Azure Files to domain, even using the correct credentials.
VM has reported a failure when processing extension 'AzureFilesDomainJoin' (publisher 'Microsoft.Compute' and type 'CustomScriptExtension'). Error message: "Command execution finished, but failed because it returned a non-zero exit code of: '1'. The command had an error output of: 'The user name or password is incorrect
+ CategoryInfo          : AuthenticationError: (:) [], CimException
+ FullyQualifiedErrorId : Win32Error:1326
+ PSComputerName        : localhost

@eversonamancio thanks for reporting the issue, we are running tests and will get back to you.

@eversonamancio please take a look at the storage domain join script inside your management VM at: C:\Windows\Temp\ManualDscStorageScriptsLog.log
If using custom DNS servers and/or private endpoints, please make sure that the management VM can resolve your domain name and the storage account FQDN to the private IP on the private endpoint.

Hi Dany,
I just clicked on Deploy to Azure button and followed the process to deploy AVD so I couldn't identify the management VM you mentioned.
In my test I didn't deploy a private endpoint.
[A screenshot of a computer  Description automatically generated]
Everson Amancio
Celular: +55 11 976959913
Twitter/Skype/LinkedIn: eversonamancio
From: Dany Contreras @.>
Sent: Wednesday, August 23, 2023 10:59 AM
To: Azure/avdaccelerator @.>
Cc: Everson @.>; Mention @.>
Subject: Re: [Azure/avdaccelerator] ARM template deployment error (Issue #459)
@eversonamanciohttps://github.com/eversonamancio please take a look at the storage domain join script inside your management VM at: C:\Windows\Temp\ManualDscStorageScriptsLog.log
If using custom DNS servers and/or private endpoints, please make sure that the management VM can resolve your domain name and the storage account FQDN to the private IP on the private endpoint.

Reply to this email directly, view it on GitHubhttps://github.com/Azure/avdaccelerator/issues/459#issuecomment-1690016725, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFCJLVJCU5IJ7WPEWBZIAJTXWYEANANCNFSM6AAAAAA3YJ7HWM.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

@eversonamancio the deployment process creates a service objects resource group and a management VM, so even though the deployment didn't complete, the management resources were created, you can connect to the VM to look for the logs i referenced.
The resources names are:

Resource group: rg-avd----service-objects
Management VM: vmmgmt

The log is located inside the management VM at: C:\Windows\Temp\ManualDscStorageScriptsLog.log

Excellent! I'm gonna check it out.
Thanks.
Everson Amancio
Celular: +55 11 976959913
Twitter/Skype/LinkedIn: eversonamancio
From: Dany Contreras @.>
Sent: Wednesday, August 23, 2023 4:41 PM
To: Azure/avdaccelerator @.>
Cc: Everson @.>; Mention @.>
Subject: Re: [Azure/avdaccelerator] ARM template deployment error (Issue #459)
@eversonamanciohttps://github.com/eversonamancio the deployment process creates a service objects resource group and a management VM, so even though the deployment didn't complete, the management resources were created, you can connect to the VM to look for the logs i referenced.
The resources names are:

Resource group: rg-avd----service-objects
Management VM: vmmgmt

The log is located inside the management VM at: C:\Windows\Temp\ManualDscStorageScriptsLog.log

Reply to this email directly, view it on GitHubhttps://github.com/Azure/avdaccelerator/issues/459#issuecomment-1690535452, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFCJLVJMWVLZYJZHAXWME2TXWZMD7ANCNFSM6AAAAAA3YJ7HWM.
You are receiving this because you were mentioned.Message ID<EMAIL_ADDRESS>

@eversonamancio the deployment process creates a service objects resource group and a management VM, so even though the deployment didn't complete, the management resources were created, you can connect to the VM to look for the logs i referenced.
The resources names are:

Resource group: rg-avd----service-objects
Management VM: vmmgmt

The log is located inside the management VM at: C:\Windows\Temp\ManualDscStorageScriptsLog.log

@danycontre I connected to the management VM, but there is no log file named ManualDscStorageScriptsLog.log in this VM.

@eversonamancio what is the identity service provider you are using (AAD, AADDS, ADDS)?
If using ADDS or AADDS, w=once you log into the management VM please make sure the VM can resolve your domain name and that it has line of site to your domain controllers.

Closing this item, no reply from requestor.


Empty files after version processing

Hi.
Basically, I come up with empty files after processing the file and returning versions by the example. I think the issue is that the files are not rewinded after being passed to MiniMagick. Since it works if I rewind the file cursor, or pass a new tempfile to one of the MiniMagick processing methods. I guess adding a rewind call for the versioned files when storing should do the trick?

I see, image_processing isn't rewinding the file after doing the processing, strange that I've missed that. I'll push a fix shortly.

Nice, thanks!


[Feature request] Use shell for meson invocation

Feature description: use default shell for all meson invocations (the same way as ninja task are currently invoked).
Reasoning:
v1.2.0 added an awesome IDE integration, but it works only with native meson. Which means, it can't utilize meson that is present only in default shell (e.g. WSL shell).

I second this. At work my meson projects run in Cygwin. Currently this extension can correctly detect all meson targets (when I set the "mesonPath" option to cygwins bin dir) in this case, but is unable to run any tasks.
An option to execute inside the default shell would be great!


Bwockner/project strings

Documentation for udSDKProject.py

I ignored anything that wasn't a docstring in this PR.
I'm not sure how much it matters, but there is inconsistency's with capital letter usage for these docstrings?


Connection never finishes loading

Azure Data Studio Version: 1.49.1 (newest)
OS Version: Apple M3 Max, Sonoma 14.3.1

Steps to Reproduce:

Successfully connect to a database using Data Studio
Wait 3+ hours
Attempt to reconnect when prompted to do so

I am able to re-verify my credentials using Microsoft Entra ID but the connection never finishes loading. I have tried restarting my laptop, quitting and reopening the app itself multiple times, and updating Data Studio but nothing works. It seems to just not work about 80% of the time and it's very frustrating.

Does this issue occur when all extensions are disabled?: Yes, I am not using any extensions

Testing that the file successfully uploaded


Support for array "items" to be array (valid JSON Schema, but not OpenAPI)?

Hi,
Will there be support forthcoming for items to be an array, as discussed in https://github.com/OAI/OpenAPI-Specification/pull/609?
My use case is, I want to send data in a request and/or response, where my data consists of a header section plus a body and where the latter could be somewhat large in size-- often 1k times larger, sometimes 1 million times larger, than the header size. For performance purposes, the header must come first, which allows the body to then be processed with fixed space regardless of the body size (a hypothetical example would be a header that contained some arbitrary string you need to count occurrences of, plus a body to search for that string in-- as long as you know what you are searching for before you receive the body, you can process the body in chunks and not have to remember what you already processed. In that simple example, obviously you could send the search string in some http header, but the header in my use case is more complex than just a string to search for, and it will be encoded as JSON, so not really viable to send as an http header).
Therefore, I'd like the specification to also reflect the fact that the header must come before the body.
Using JSON Schema, this is easy. I can specify that the data must be an array of two items, where the first item conforms to the header schema, and the second conforms to the payload schema. So my data may look like:
[
	{ "type": "header", ... },
	{ "type": "body", ... }
]

and my spec would be something like:
{
	"$schema": "https://json-schema.org/draft/2019-09/schema",
	"type": "array",
	"items": [
		{
			"type": "object",
			"properties": {
				"type": {
					"enum":  ["header"]
				}
			}
		},
		{
			"type": "object",
			"properties": {
				"type": {
					"enum":  ["body"]
				}
			}
		}
	]
}

However, openAPI does not allow items to be an array. So the best I can do, which isn't very helpful, is to use a merged schema (e.g. oneOf or anyOf) to allow any array element to be a header or body-- which means that any validation using this specification will be useless for the purposes of ensuring that the header comes before the body.
Is there a better solution to this problem (whether currently available or planned to be available)?
Thank you

@MikeRalphson just wanted to check in to see if the aforementioned validator happens to be available yet-- I checked on https://github.com/Mermade/oas-kit and only saw 3.0.0, but please lmk if I missed. No rush if not as we've implemented a workaround for now


Code merge between MSD/CFD personals with 9.0 Marshal Branch

Title of ticket:
Description
The purpose of this task is to Sync the Code FE,BE , migration scripts between 8.4.0(planned for 17 OCT) & 9.0 on an intermediate marshal branch and functionally test everything is in tact between the releases went to PROD till that point and 9.0 Marshal code.
Once this exercise is over , we are good to merge with regular DEV, TEST Branches.
Note #1 : This task can be only be instantiated after 8.4.0 release , which is planned on OCT-17-2023
Note#2 : There is a small/minor release planned after 8.4.0 , which is 8.4.5 - so that need to be covered as well before closing this task
Dependencies
Are there any dependencies?
DOD

[ ]  Migration Scripts Ordered with PROD scripts on the bottom and new scripts on top
[ ]  Any code merge issues, code conflicts fixed - consulted with corresponding devs
[ ]  Functionality Testing of CFD/MSD personals - Sections etc. Admin app. - Smoke testing
[ ]  Functionality Testing of Gathering Records  - Smoke testing
[ ]   Functionality of  Document reviewer -  - Smoke testing

test

[Uploading... image.png]


Detecting MacRoman file as ISO-8859-1

Hey,
I'm trying to transcode some CSV files but it ends up being gobbly gook. Charlock is detecting the file as ISO-8859-1, whereas BBEdit says the same file is MacRoman (Western), as exported from the Mac version of Excel.
    def transcode(filename)
      require 'charlock_holmes'

      content = File.read(filename)
      detection = ::CharlockHolmes::EncodingDetector.detect(content)
      puts "Unknown encoding for #{filename}" and return if detection.nil?

      encoding = detection[:encoding]
      return if encoding == 'UTF-8' # already what we want

      puts "Transcoding from #{encoding} to UTF-8 for #{filename}"
      utf8_encoded_content = CharlockHolmes::Converter.convert(content, encoding, 'UTF-8')
      File.open(filename, 'w:utf-8') {|f| f.write(utf8_encoded_content) }
    end

Is there a way to tell if this is an issue with ICU's detection vs. something I'm doing wrong?
Thanks, Nathan.

I tried iconv on the command line with ISO-8859-1 and MacRoman. MacRoman works, ISO-8859-1 doesn't as it's the wrong encoding. So it definitely seems like ICU is either missing or improperly detecting MacRoman.
iconv -f ISO-8859-1 -t UTF8 < myfile.csv > test.txt
iconv -f MacRoman -t UTF8 < myfile.csv > test.txt

So I read somewhere that MacRoman and Windows-1251 are both variations of ISO-8859-1? Perhaps it's not possible for ICU to detect the difference. I played around with rchardet and python's chardet as well, for the same results.
I imagine BBEdit is taking advantage of extended attributes on the Mac xattr -l com.apple.TextEncoding to get some extra info.
Nathan.

Interesting...
One thing to keep in mind for the detection stuff is that the more data it has to work with, the more accurate detection will be. Is this string of text very large? I've seen this before as well but usually only with very small chunks of text that aren't really large enough to make any sort of accurate detection.
If the text is larger than 1kb or so I'd like to think that's probably enough to make a somewhat accurate detection. Where is the CSV file coming from?

Oh, the CSV file was a very large data file that I was importing into MySQL. In the end, I just handled the encoding manually instead of trying to script the detection and conversion.

This may help:
http://stackoverflow.com/questions/4198804/how-to-reliably-guess-the-encoding-between-macroman-cp1252-latin1-utf-8-and


process cell tags remove-input and remove-output

This PR adds removing cell input and output by using the remove-input and remove-output tags.
My understanding from #436 is that the behaviour (remove output rather than hide with css) is consistent with Jupyter Book use of the same tags.
This is complimentary to #436 in that it deals with remove- tags as opposed to hide-*.
Thank you for looking at this, and making nbsphinx in the first place!

@mgeier  thank you for making nbsphinx it is just what I need to keep a large part of my documentation in notebooks.
So my use-case is that I have (graphviz) graphs in the documentation that I want to generate from the notebook but I don't want to include the source code in the HTML because it adds nothing in my case (and people can always jump to the notebook if they want).
I hope this can fill a gap, but if this is totally different from what you would have in mind for handling these tags, don't hesitate to reject the PR, too.

Thanks for this PR!
I'm open for this feature, but I think there are some details still to be ironed out.
Can you please remove the outputs from the example notebook?
In your example, the padding after an input cell without output is missing.
So is the padding before an output cell without input.
The implementation is fine, but did you think about re-using the tag-based filtering from nbconvert?
Would this be possible?
I think  remove-input and remove-output are good defaults, but what about allowing configuration options for specifying other tag names (like IIRC it is possible in nbconvert)?

My understanding from #434 is that the behaviour (remove output rather than hide with css) is consistent with Jupyter Book use of the same tags.

Is there a link to some documentation?

This is complimentary to #434

I don't see the connection.
Do you happen to mean #436?
Or #131?
BTW, did you have a look at  #185 and  #86?

I don't see the connection.
Do you happen to mean #436?

Oh sorry. Yes, I changed the text to #436.

BTW, did you have a look at #185 and #86?

I didn't. So I am aware of hide_input but ended up implementing remove-cell as a tag (by adding or "remove-cell" not in ...tags) in my local branch (which I'd happily add here, too). I must admit that tags are about my limit w.r.t. discoverability (it's bad, but it's the best we have, and standardizing across multiple implementations might be best, so I took the JupyterBook tags based on my impression that it seemed reasonable in #436) the custom metadata is more a way to sell headache pills to me than a user interface (and this is not to take a dig at nbsphinx using them when they were the best thing we had, but just that I had a bad time with them when using them for global things and I think they're best avoided for per-cell things due to the complexity of the UI).

Can you please remove the outputs from the example notebook?
Will do.

In your example, the padding after an input cell without output is missing.
So is the padding before an output cell without input.

The implementation is fine, but did you think about re-using the tag-based filtering from nbconvert?
Would this be possible?

This would be very elegant.
So I found removing cells in the nbconvert docs. I tried to adapt the eample there to the style of defining config used by nbsphinx, but it seems to not work for me.
            config=traitlets.config.Config({
                'HighlightMagicsPreprocessor': {'enabled': True},
                # Work around https://github.com/jupyter/nbconvert/issues/720:
                'RegexRemovePreprocessor': {'enabled': False},
                'TagRemovePreprocessor': {
                    'enabled': True,
                    'remove_cell_tags': ('remove-cell',),
                    'remove_all_output_tags': ('remove-output',),
                    'remove_input_tags': ('remove-input',),
                },
            }),

If you have ideas, going via nbconvert config would seem much more elegant to me.
P.S.: Our project using nbsphinx and also your insipid theme went public today. Thank you for covering us for our documentation.

I think remove-input and remove-output are good defaults, but what about allowing configuration options for specifying other tag
names (like IIRC it is possible in nbconvert)?

Personally, I don't really see the value.

I don't have a concrete use case myself, but I could imagine that somebody wants different versions of their docs, once with removed cells and another one where all cells are visible.
Or they are already using a set of tags for something else, and suddenly they want to hide all those tagged cells.
If you look at it from a cost/benefit perspective: It costs basically nothing to expose the configuration options, therefore it doesn't matter if the benefit isn't huge.
Having the tags as literal strings littered around the source code is bad anyway, so at least I would make global variables for them. But making them Sphinx configuration options is just as easy, and to me it is quite an obvious thing to do (even if I don't have a use for that myself).

Jupyter seems to be unable to specify common tags,

Yeah, that's even more reason to make them configurable in nbsphinx!

but in the end I see more value with having some sort of standardization and so aligning with Jupyter Book + Hardcoding might be a feature.

Sure, standardization would be great, but as you said, this didn't happen (except for metadata.jupyter.source_hidden and metadata.jupyter.outputs_hidden: https://nbformat.readthedocs.io/en/latest/format_description.html#cell-metadata).
Jupyter Book is just another third-party package, it isn't in the position to standardize anything.
But if there are no better suggestions, I think it's reasonable to use their settings as our defaults.

That said, if you intended the question to express a preference for configuration, I'll add that.

Yeah, I guess I did.
Yes, please!

My understanding from #434 is that the behaviour (remove output rather than hide with css) is consistent with Jupyter Book use of the same tags.

Is there a link to some documentation?

https://jupyterbook.org/interactive/hiding.html#removing-code-cell-content
Would you add that link to the documentation? I can do that when I flip the order of the hide methods in the documentation.

Thanks for the link.
I think that adds no value to our documentation. We have to mention the default tag names anyway.

I think I have the other questions, but I'd appreciate a hint if not.

I think those were indeed all the questions, but I think this comment hasn't been addressed yet:
"In your example, the padding after an input cell without output is missing.
So is the padding before an output cell without input."

If you have suggestions for improving the theme, please let me know!

The most annoying thing was that Jupyter Notebooks don't need a blank line but nbsphinx generates funny stuff without (seems to be jupyter/nbconvert#917, so not a nbsphinx bug).

Yeah, the problem is that there are different Markdown implementation with slightly different behavior.
For now, nbsphinx is using pandoc (which itself supports multiple Markdown dialects), but in the future I'd like to use a Python library (which probably doesn't exist yet) for this.

Other than that I probably haven't used half the customization that is possible. One thing I thought about is whether to put the logo in the topbar instead of the sidebar.

There is no specific setting for putting the logo there, but you can try to use the left_buttons option for that.
But I think in general it's a waste of space to put the logo into the topbar.
You should think about users with small screens (like smartphones). Putting the logo there will make the topbar actually less useful for them.
You could of course add some CSS to avoid showing the logo on small screens, but if the logo is so important to you, you probably also want it to be shown on those small screens, right?
What about putting the logo at the top of each page (using the page.html template)?

That likely is already possible but I took your advice and focus on the content. I really like how nbsphinx and your insipid theme made that possible!

That's great to hear!

The PDF output seems to have the same problem, there is no space between cells: https://648-46379698-gh.circle-artifacts.com/0/nbsphinx.pdf#section.6

The PDF output seems to have the same problem, there is no space between cells: https://648-46379698-gh.circle-artifacts.com/0/nbsphinx.pdf#section.6

Yes, but so this happens when I use the nbconvert facility, so it would appear to be a bug there.

The PDF output seems to have the same problem, there is no space between cells: https://648-46379698-gh.circle-artifacts.com/0/nbsphinx.pdf#section.6

Yes, but so this happens when I use the nbconvert facility, so it would appear to be a bug there.

It very likely is a bug there, but nbsphinx doesn't use the LaTeX/PDF workflow that nbconvert uses (it uses the reST template which is later converted to LaTeX by Sphinx).
Either way, I would like it to work correctly in nbsphinx, but if it gets fixed in nbconvert as well, that's of course also good.

Either way, I would like it to work correctly in nbsphinx, but if it gets fixed in nbconvert as well, that's of course also good.

I'm afraid my notebook foo isn't enough to implement these tags after all.


failed to reproduce the results

Thanks for the code! I failed to reproduce the results in Table 3. 1) the results from follower+naive CL and follower+self-paced Cl are the same; 2) the reproduced results are generally worse than the reported ones, and sometimes conflict; 3) the results do not change with random seeds. Do you have any suggestions?

val_seen

val_unseen

reported
NE↓
SR
OSR
SPL
CLS
NE↓
SR
OSR
SPL
CLS

follower
4.85
52.3
65.3
44.3
58.2
7.12
28.6
40.9
20.3
35.0

+naïve CL
5.03
48.6
62.0
40.4
55.9
7.13
31.1
42.8
21.2
34.3

+self-paced CL
4.23
58.7
69.2
51.1
63.3
6.69
32.2
44.2
24.5
38.9

reproduced

follower
4.40
56.4
68.2
47.6
60.9
7.23
26.2
38.7
17.8
31.9

+naïve CL
4.45
55.2
67.4
44.4
56.4
6.80
30.0
43.1
20.2
34.3

+self-paced CL
4.44
58.2
69.3
48.3
58.8
7.05
28.3
39.2
19.8
34.1

reported

self-monitoring
4.27
58.4
67.0
51.9
64.1
6.42
38.4
48.1
28.3
41.5

+naïve CL
4.08
61.0
69.8
54.6
64.9
6.30
40.0
51.7
28.6
41.1

+self-paced CL
4.19
58.8
68.2
53.3
65.4
5.98
41.0
52.4
30.8
43.9

reproduced

self-monitoring
4.17
58.2
67.4
52.9
65.3
6.45
35.6
46.0
27.2
41.9

+naïve CL
3.98
61.9
70.2
55.5
66.2
6.20
39.9
50.2
29.5
41.7

+self-paced CL
4.02
59.9
69.3
54.8
66.6
6.28
40.1
51.2
31.2
44.7

reported

envdrop
4.55
57.7
65.6
54.4
67.2
5.92
45.7
54.2
41.8
57.0

+naïve CL
4.49
57.8
63.1
54.8
67.2
5.93
44.3
50.5
41.3
57.6

+self-paced CL
4.42
58.1
65.6
54.8
67.4
5.48
47.6
54.3
44.1
59.1

reproduced

envdrop
4.57
57.3
65.7
54.0
67.1
5.86
45.6
53.6
41.8
57.3

+naïve CL
4.89
56.4
63.5
53.3
65.7
5.91
43.9
51.9
40.4
57.4

+self-paced CL
4.44
57.8
65.4
54.5
67.0
5.94
44.7
53.7
41.5
57.3

Could you please give me more details about your training settings and local machine information?
Moreover, it is strange that the results do not change with random seeds. Our observation is the results will fluctuate with random seed. We have reported the results from repeating experiments in Table 7.

Thanks for the reply!
I build a singularity image to run the code, with ubuntu16.04 torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 python3.8.5.

Hi Qi Zheng,
My suggestion is that you should switch your python and torch version as the same in paper. However, I will also try your settings. I will contact you when the results come out.
With regard to your reproduced experiments, although the results from follower and self-monitoring are slightly lower than that in paper, they do show that curriculum learning is useful. For envdrop agent, maybe you should try again. Since I have repeated the experiments many times in my machine, the performance changes with the random seed, but exhibit a similar trend that SPCL > NCL > ML.

Thanks! I will try the setting and let you know in case of any changes.

Hi, I use the same version as suggested. The result of follower and self-monitoring is consistent with that in the paper, however, the result envdrop trained by SPCL is worse than the baseline.


[BUG] Adding 'sample tag' in UI causes image viewer to close.

Problem
When using the UI, (both the desktop app and the web browser) the viewer closes when attempting to add a 'sample tag' to an image. Images can only be tagged from the 'home' view.
Below, I use screenshots from the 'quickstart' dataset to demonstrate, but the bug does not occur with this dataset.
When selecting 'sample' under 'labels' within the image viewer, the image closes.

Images can be tagged from this view only.

System information

OS Platform and Distribution (Linux Ubuntu 20.04, Windows 11 Pro):
Python version (python --version): 3.11.5, 3.11.8
FiftyOne version (fiftyone --version): v0.24.1
FiftyOne installed from (pip or source): pip

Willingness to contribute
The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

[ ] Yes. I can contribute a fix for this bug independently
[ ] Yes. I would be willing to contribute a fix for this bug with guidance
from the FiftyOne community
[X] No. I cannot contribute a bug fix at this time (I don't know enough yet, sorry!)

Hi @oliviajbreen. Do you see any errors in the develop tools console when this issue occurs?

Resolved by https://github.com/voxel51/fiftyone/pull/4621


[monkeys-audio] Add header file and remove the backup URL

Fix https://github.com/microsoft/vcpkg/issues/29592

Note: Based on the update frequency of the upstream and all URL in the vcpkg_download_distfile must be valid, the role of backup cannot be realized.

[x] Changes comply with the maintainer guide
[ ] SHA512s are updated for each updated download
[ ] The "supports" clause reflects platforms that may be fixed by this new version
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

Thanks for the fix!


Remove FixedSizeNginxNetwork to demonstrate a simpler test setup

This gives us an example we can point to of using Test.Setup without the custom Network interface in between

Merging since it's very unobjectionable


Bump Go in github action to 1.20

This patch includes https://github.com/knative-extensions/net-istio/pull/1147 and Bump Go version to 1.20 in github action.

/cc @ReToCode @kvmware


vue  project with vue cli 3 gives SCRIPT1003: Expected ':' error with blank screen. other than ie application works fine with other browsers.

Version
3.0.0-rc.5
Reproduction link
https://stackoverflow.com/questions/53516050/vue-cli-3-0-project-not-working-on-ie11-script1003-expected-error-with-bla
Node and OS info
Node 8.9.1 windows 10
Steps to reproduce
create project using vue cli 2
create component using es6 shorthand and arrow function syntax
run npm run dev and open application in ie11
What is expected?
application should run
What is actually happening?
gives blank screen with SCRIPT1003: Expected ':' error in ie11

That link is not a valid reproduction. Please provide a github repository.

I can not upload my project code on open repository.

i can provide you the other required detail any time please.

You can follow the steps that you yourself described, make sure the error is reproducible with that fresh project, and then upload that project to github.
Then we can look at that project and be sure we are talking about the same thing.

I found the solution for this. putting here if someone face the same issue.
Issue was, i have installed "vue-offline": "^1.0.204", plugin that was creating issue. this is not supporting in vue cli 3.0. you need to upgrade this plugin with the latest one to fix this.
with vue cli 3.0 it is not required to install any polyfil externally, vue cli is taking care of all this.
Thanks for your support.

I have same problem :SCRIPT1003:Expected':' error with blank screen. and I have tried to upgrade "vue-offline" ,plugin latest version ,but it still not working ;


add a mantle preview command to create a preview deployment

use case: create a preview deployment from all PRs
dependent on #40 for post-merge cleanup
also dependent on #21 for making previews private access

Running mantle preview create will:

Create a new temporary environment (not saved back to config file) with the configuration:

name: preview-<branch-name>-<pr-number> # is this sufficiently unique/understandable?
overrides:
  places:
    *:
      configuration:
        name: Preview (<pr-number>): <name>

Deploy to the temporary environment
Add the temporary environment to the state file

Running mantle preview destroy will:

Look for a temporary environment in the state file with the name format described above
Destroy the environment if it exists
Remove the temporary environment from the state file


Can the Flutter team make the keyboard stick to the bottom?

Can the Flutter team make the keyboard stick to the bottom?
I hope the Flutter team can do this, because my keyboard disappears and pops up again.
When I finish typing and click the button, my keyboard disappears and pops up again.
I wish the Flutter team made the keyboard always stay at the bottom.

Hi @mycar98765. please provide a complete minimal reproducible code sample so that we can investigate this issue. The code sample you provided is not complete.
Thank you

@danagbemava-nc  what kind of code do you want? By the way, it is the chat message bar. What I want to do is like WhatsApp chat screen. When the send message button is clicked, the keyboard doesn't disappear and pop up again.

A runnable code sample that shows the issue you're facing. The code sample that you provided above is not runnable. It doesn't have to be your project code, just a small sample that reproduces the bug will do.
Thank you

@danagbemava-nc ok, here's the code you should be able to run:
Scaffold(
   appBar: AppBar(
    …
  ),
  body: Column(
    mainAxisSize: MainAxisSize.min,
     children: [
      Expanded(
        child: Stack(
          children: [
            …
             Align(
               alignment: Alignment.bottomCenter,
               child: Row(
               children: [
                 Expanded(
                    child: Column(
                      mainAxisSize: MainAxisSize.min,
                      children: [
                        …
                        Row(
                          children: [
                            Expanded(
                              child: TextField(
                                …
                              ),
                            ),
                            isLoading
                                ? const CircularProgressIndicator()
                                : Container(
                                    decoration: BoxDecoration(
                                      color: …,
                                      shape: BoxShape.circle,
                                    ),
                                    child: IconButton(
                                      onPressed: …,
                                      icon: …,
                                  ),
                          ],
                        ),
                      ],
                    ),
                  ),
                ],
              ),
            ),
          ],
        ),
      ),
      …
    ],
  ),
),

I used some part of your code sample and when I tap the send button, the keyboard doesn't go anywhere. See the recording below.
Without a complete code sample that reproduces the bug we won't be able to do anymore here.
https://user-images.githubusercontent.com/88313112/230391834-c6bdd102-1c63-4776-b0ea-31828801235c.MP4

sample used
import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatefulWidget {
  const MyApp({
    super.key,
  });

  @override
  State<MyApp> createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  @override
  Widget build(BuildContext context) {
    return const MaterialApp(
      home: HomeScreen(),
    );
  }
}

class HomeScreen extends StatefulWidget {
  const HomeScreen({super.key});

  @override
  State<HomeScreen> createState() => _HomeScreenState();
}

class _HomeScreenState extends State<HomeScreen> {
  bool isLoading = false;

  @override
  void initState() {
    super.initState();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(),
      body: Column(
        mainAxisSize: MainAxisSize.min,
        children: [
          Expanded(
            child: Stack(
              children: [
                Align(
                  alignment: Alignment.bottomCenter,
                  child: Row(
                    children: [
                      Expanded(
                        child: Column(
                          mainAxisSize: MainAxisSize.min,
                          children: [
                            Row(
                              children: [
                                const Expanded(
                                  child: TextField(),
                                ),
                                isLoading
                                    ? const CircularProgressIndicator()
                                    : IconButton(
                                        onPressed: () {
                                          setState(() {
                                            isLoading = true;
                                          });

                                          Future.delayed(const Duration(seconds: 3),
                                              () {
                                            setState(() {
                                              isLoading = false;
                                            });
                                          });
                                        },
                                        icon: const Icon(Icons.send),
                                      ),
                              ],
                            ),
                          ],
                        ),
                      ),
                    ],
                  ),
                )
              ],
            ),
          )
        ],
      ),
    );
  }
}

Is this because I added this in my StatefulWidget?:
@override
void initState() {
  super.initState();
  focusNode.requestFocus();
}

That is unlikely to be issue since that line of code should only run when the widget is first initialized.
In its current state, this issue is not actionable for us since no valid repro has been provided. As such I'm going to close this issue.
Please file a new issue when you're able to come up with a reduced test case that we can use to reproduce the issue so that we can investigate it.
Thank you


Fix clouds visually resetting every 100 seconds

Closes #5.
The mod(TIME, 100.0) call causes the clouds to do a very noticeable jump every 100 seconds. Since the textures are all set to repeat_enable, this modulo operation can be removed with (as far as I can tell) no negative side-effects. The clouds now continuously animate smoothly.

Seems great, thank you! I can't remember why I was using mod there in the first place


ks param set when used with boolean does not create string value

/kind bug
In prototype
@optionalParam withRbac string false Whether to include RBAC setup

creates params.libsonnet with
withRbac: "false",

What happened:
ks param set seldon-core withRbac false

creates
withRbac: false,

which is a JSON boolean rather than a string.
What you expected to happen:
Consistent behaviour. Should set "false" in params file
How to reproduce it (as minimally and precisely as possible):
As above with bool-like parameter
Anything else we need to know?:
Environment:

ksonnet version (use ks version):
ksonnet version: 0.8.0
jsonnet version: v0.9.5
client-go version: v1.6.8-beta.0+$Format:%h$

Kubernetes version (use kubectl version):

N/A

I'm thinking about how we can tackle this issue. ksonnet-lib/jsonet are not typed, so we can't easily tell what type an item should be. The easiest solution is to introduce a flag to ks param set to enforce the type. Jsonnet does provide a std.type method so we can guess based on an existing value, but that would assume we have a good initial value to work from.
Here is where this is happening in the code.
https://github.com/bryanl/ksonnet/blob/use-app-targets-by-default/pkg/kubecfg/param.go#L96

I think the problem here is mainly the inconsistencies between the prototype create / generate and the ks param set behavior. Either we decide to set booleans as strings in ks param set or set the proper boolean param in generate. I'm favoring the latter.

This still seems to happen in:
ks version
ksonnet version: 0.11.0
jsonnet version: v0.10.0
client-go version:


TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Version:
torch==1.13.0
torch-geometric==2.3.0
torchdrug==0.2.0

Code:
import torch_geometric

Output:
---------------------------------------------------------------------------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 import torch_geometric

File ~/env/lib/python3.8/site-packages/torch_geometric/__init__.py:2
      1 import torch_geometric.utils
----> 2 import torch_geometric.data
      3 import torch_geometric.sampler
      4 import torch_geometric.loader

File ~/env/lib/python3.8/site-packages/torch_geometric/data/__init__.py:7
      5 from .data import Data
      6 from .hetero_data import HeteroData
----> 7 from .batch import Batch
      8 from .temporal import TemporalData
      9 from .dataset import Dataset

File ~/env/lib/python3.8/site-packages/torch_geometric/data/batch.py:11
      9 from torch_geometric.data.collate import collate
     10 from torch_geometric.data.data import BaseData, Data
---> 11 from torch_geometric.data.dataset import IndexType
     12 from torch_geometric.data.separate import separate
     15 class DynamicInheritance(type):
     16     # A meta class that sets the base class of a `Batch` object, e.g.:
     17     # * `Batch(Data)` in case `Data` objects are batched together
     18     # * `Batch(HeteroData)` in case `HeteroData` objects are batched together

File ~/env/lib/python3.8/site-packages/torch_geometric/data/dataset.py:20
     15 from torch_geometric.data.makedirs import makedirs
     17 IndexType = Union[slice, Tensor, np.ndarray, Sequence]
---> 20 class Dataset(torch.utils.data.Dataset, ABC):
     21     r"""Dataset base class for creating graph datasets.
     22     See `here <https://pytorch-geometric.readthedocs.io/en/latest/tutorial/
     23     create_dataset.html>`__ for the accompanying tutorial.
   (...)
     41             downloading and processing the dataset. (default: :obj:`True`)
     42     """
     43     @property
     44     def raw_file_names(self) -> Union[str, List[str], Tuple]:

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Bumping this up -- also experiencing this issue.  No problem if loading torchdrug after PyG.

the same issue raised.
python script/downstream.py -c config/downstream/EC/gearnet.yaml --gpus [0]
Traceback (most recent call last):
  File "/home/chu/github/GearNet/script/downstream.py", line 12, in <module>
    from torchdrug import core, models, tasks, datasets, utils
  File "/opt/envs/ml/lib/python3.9/site-packages/torchdrug/models/__init__.py", line 10, in <module>
    from .esm import EvolutionaryScaleModeling
  File "/opt/envs/ml/lib/python3.9/site-packages/torchdrug/models/esm.py", line 6, in <module>
    import esm
  File "/opt/envs/ml/lib/python3.9/site-packages/esm/__init__.py", line 8, in <module>
    from .data import Alphabet, RobertaAlphabet, BatchConverter, FastaBatchedDataset  # noqa
  File "/opt/envs/ml/lib/python3.9/site-packages/esm/data.py", line 11, in <module>
    from torchvision.datasets.utils import download_url
  File "/opt/envs/ml/lib/python3.9/site-packages/torchvision/__init__.py", line 6, in <module>
    from torchvision import datasets, io, models, ops, transforms, utils
  File "/opt/envs/ml/lib/python3.9/site-packages/torchvision/datasets/__init__.py", line 1, in <module>
    from ._optical_flow import FlyingChairs, FlyingThings3D, HD1K, KittiFlow, Sintel
  File "/opt/envs/ml/lib/python3.9/site-packages/torchvision/datasets/_optical_flow.py", line 30, in <module>
    class FlowDataset(ABC, VisionDataset):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Also seeing this issue with esm:
`TypeError                                 Traceback (most recent call last)
Cell In[6], line 1
----> 1 from torchdrug import core, models, tasks, utils
3 model = models.GIN(input_dim=dataset.node_feature_dim,
4                    hidden_dims=[256, 256, 256, 256],
5                    short_cut=True, batch_norm=True, concat_hidden=True)
6 task = tasks.PropertyPrediction(model, task=dataset.tasks,
7                                 criterion="bce", metric=("auprc", "auroc"))
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/torchdrug/models/init.py:10
8 from .infograph import InfoGraph, MultiviewContrast
9 from .flow import GraphAutoregressiveFlow
---> 10 from .esm import EvolutionaryScaleModeling
11 from .embedding import TransE, DistMult, ComplEx, RotatE, SimplE
12 from .neurallp import NeuralLogicProgramming
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/torchdrug/models/esm.py:6
4 import torch
5 from torch import nn
----> 6 import esm
8 from torchdrug import core, layers, utils, data
9 from torchdrug.layers import functional
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/esm/init.py:8
...
28     # and it's up to whatever consumes the dataset to decide what valid_flow_mask should be.
29     _has_builtin_flow_mask = False
31     def init(self, root, transforms=None):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases`

Also seeing this issue with esm: `TypeError Traceback (most recent call last) Cell In[6], line 1 ----> 1 from torchdrug import core, models, tasks, utils 3 model = models.GIN(input_dim=dataset.node_feature_dim, 4 hidden_dims=[256, 256, 256, 256], 5 short_cut=True, batch_norm=True, concat_hidden=True) 6 task = tasks.PropertyPrediction(model, task=dataset.tasks, 7 criterion="bce", metric=("auprc", "auroc"))
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/torchdrug/models/init.py:10 8 from .infograph import InfoGraph, MultiviewContrast 9 from .flow import GraphAutoregressiveFlow ---> 10 from .esm import EvolutionaryScaleModeling 11 from .embedding import TransE, DistMult, ComplEx, RotatE, SimplE 12 from .neurallp import NeuralLogicProgramming
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/torchdrug/models/esm.py:6 4 import torch 5 from torch import nn ----> 6 import esm 8 from torchdrug import core, layers, utils, data 9 from torchdrug.layers import functional
File ~/anaconda3/envs/linea/lib/python3.9/site-packages/esm/init.py:8 ... 28 # and it's up to whatever consumes the dataset to decide what valid_flow_mask should be. 29 _has_builtin_flow_mask = False 31 def init(self, root, transforms=None):
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases`

Have you been able to solve this issue for esm?


SUggestion

Hi - Interesting project.  One suggestion I would make is that if you used a WEMOS microcontroller - i.e. a WiFi enabled Arduino then you could get the time from NTP, and could report results in real time using BLYNK - https://blynk.io/
I set up a temperature and humidity monitor this way, and it will alert me if the temperature exceeds a certain set point.
Good Luck....Mark

WiFi-enabled! I've been wanting to get into microcontrollers and blynk.io looks great. Consider it done!

Great idea!

Yes - Blynk is pretty easy to use and free as long as you keep things simple.  If you are also thinking of doing some Android Apps, check out the MIT App Inventor - also very easy to use.
http://appinventor.mit.edu/


How to create dbffile field type date (D (date))

I want create dbf file but error..
let fieldDesc = [
{ name: 'date', type: 'D', size: 8 }
];
and set value
let records = [
{ date: '12/12/19' }
];

Help me.

@sahaphon you just need to pass a Date object instead of a string. Eg:
let fieldDesc = [
    { name: 'date', type: 'D', size: 8 }
];
...
let records = [
    { date: new Date('2019-12-12') }
];


Language and script in metadata sheets

@alevivier : I understand that you are or will soon be away on fieldwork. This is not urgent and can be addressed once you are back and have time. The matter came up in another discussion (see #250 for precedents) and we wanted to keep it in sights. Also, please assign anyone else who should be involved in this. I also have a number of other concerns about our metadata sheets, which I believe more and more people are using, yet I know of no instructions for the proper way of filling many of the cells, and several details don't work out as well as they could. I think a guide should be drafted for the metadata sheets (this one seems very much out of date), and the fields and vocabularies need to be finalised before too many people start filling them out. I'm happy to be involved in that discussion and just as happy to be left out of it so long as it takes place.
The question now is: what purpose does it serve to have entries for language and script in the metadata spreadsheets? In their current state, the fields only contain information that is also encoded in the XML files and can be pulled from there (#250), so recording the same information redundantly in the metadata seems to offer no advantage while creating space for human error.

Some relevant bits from #250:
Michaël said it is not a problem to generate a smart display of the language information extracted from the XML, such as "Text in Tamil, with parts in Sanskrit. Translations in English and in French"
Arlo said, "I am fairly sure the matter of redundant representatiion of language and script metadata was discussed by Adeline, manu and myself when we were working on the template and guide for the metadata spreadsheet, but I don't remember why we accepted/required the redundancy."
Dan said, "I also think that the redundancy was discussed and also have no clear recollection of the details. But I think that back then the idea was to record something slightly different in the metadata, perhaps by allowing a freetext description of the language of the inscription (e.g. "non-Standard Sanskrit" or "Sanskrit with boundary descriptions in Telugu"). That way, the redundancy is only partial. But what we have in the sheets now can be matched 100% to the data encoded in the XML, so the redundancy is a bad thing."

Dear Dan,
Here the last version of the metadata guide: [https://docs.google.com/document/d/1RqePCIm7SOBGl0M_V_q95TU87ogJR-NsZsdRrS0YnGE/edit?usp=sharing]
It has been written according to the last version of the template

About the question of redundancy of languages and scripts in metadata:
I don't remember why we chose to keep the mention of languages and scripts both in the metadata and in the edition. If you feel this information is redundant, of course it can been deleted in one part. But we need to be sure that this information could be searched in the database. I don't know how the database will be searchable (by fields or free search)


Laggy window resizing of Files.

Description
When you are on the home page of Files, the window resizes smoothly. But when you open a directory that has a lot of Files, like Downloads, and / or Documents. The window resizing becomes extremely laggy.
Steps To Reproduce

Try resizing the Window with Files on the homepage
Enter the Documents / Downloads directory or any directory with a lot of files,  and try to resize the window.

Expected behavior
I expected the Files window to resize smoothly.
Files Version
<IP_ADDRESS>
Windows Version
Windows 11 Beta 10.0.22000.132
Relevant Assets
No response

Thanks for the report. Is it laggy only in Details view or also in Tiles / Grid view?

Thanks for the report. Is it laggy only in Details view or also in Tiles / Grid view?

It is also lagging in Tiles / Grid view.

@RickeyHo this issue should be resolved in the preview version. http://files.community/download/preview


Add help copy to settings

As an Admin I want to see help copy in the date settings, so it is clear what all of the date's do and how they affect users

 [QAE17IMP] As an Admin I want to see help copy in the date settings, so it is clear what all of the date's do and how they affect users


E5 and M5

To use Privileged Identity Management, your directory must have one of the following paid or trial licenses:
Azure AD Premium P2
Enterprise Mobility + Security (EMS) E5
Microsoft 365 M5

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 66c20f5e-ca50-337e-8fc1-1d3d434178a7
Version Independent ID: 4f7c30ee-a49c-a79e-e06d-2c788ae91e39
Content: Deploy Privileged Identity Management (PIM) - Azure AD
Content Source: articles/active-directory/privileged-identity-management/pim-deployment-plan.md
Service: active-directory
Sub-service: pim
GitHub Login: @curtand
Microsoft Alias: curtand

Hi @bobnerland Thanks for reaching out. We will review and update as appropriate.

@bobnerland
The same has been mentioned in this document in the Licensing Section as well as this other detailed document.
Let us know if you need further assistance.

@bobnerland
We will be closing this thread now. Let us know if you need further help on the same by re-opening the thread.

Hi
You have not corrected the text. It`s nothing called Microsoft 365 M5.
R
Bob
Fra: neeleshray-msft<EMAIL_ADDRESS>Svar til: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Dato: onsdag 8. januar 2020 05:27
Til: MicrosoftDocs/azure-docs<EMAIL_ADDRESS>Kopi: Bob Nerland<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Emne: Re: [MicrosoftDocs/azure-docs] E5 and M5 (#45648)
@bobnerlandhttps://github.com/bobnerland
We will be closing this thread now. Let us know if you need further help on the same by re-opening the thread.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/issues/45648?email_source=notifications&email_token=ADZUJKBFFMAJ3WLQF437J3DQ4VIZLA5CNFSM4KDA4MZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEILEUUA#issuecomment-571886160, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADZUJKE2TMZESGKYAXOHASLQ4VIZLANCNFSM4KDA4MZQ.


User is always plaintiff

SME question - Is the user always the individual who is seeking to file for divorce and thus the plaintiff starting a new case?
Additionally, is this form used ONLY for divorce? can it be used for other means (i.e. minor changing their name) and are we restricting it for that purpose

Kate replied to questions:
This could be used in divorces or custody cases between unmarried parents. There might be other appropriate uses, but none jump out.  Definitely not used in name change, which has very different service rules
Restrictions - have filed or intend to file this concurrently with a family law case; have done a "diligent search" to locate defendant


No integration for the Suffocating/Drowning effect

I came across this while testing Foundry out with a friend. He rolled a crit with a perforating weapon, and used the rolled crit card from the crit deck to add this effect to his attack. (Critical Hit Deck 22)
The entry specified the opponent would have perforated lungs, and would enter a suffocating effect, with only a Core rulebook page to refer to, no effect entry, nor automation like another persistent effect like poison.
Looking into it, there was no suffocating effect compendium entry when looking at the "Suffocate" spell either, even though it was one of its listed effects.
Is the Suffocating effect an edge case and not considered ? Or do all persistent effects with additional rules (opposed to poison) still need to be implemented ?
With so many automation already in places for additional and persistent effects, it is hard to balance having a few effects being discovered as we play, and then applying them accordingly without forgetting it or making a calculation mistake, especially when being new to Foundry.

The rules for drowning and suffocation for reference: https://2e.aonprd.com/Rules.aspx?ID=468
There is nothing we can currently automate for this beyond just having a tracking effect which we don't generally add to the system.


[Regression] Shell command not working

  - name: Remove Support Folders
    shell: cmd="rm -rf {{ vars.install_dir }}/{var,etc,data*}"
    ignore_errors: true

This is working fine on build 245 & 253. But not on latest. 255+ / 266..

This should be resolved when the go shell module finishes

Issue resolved when shell module was ported to go


Doesn't serialize Maps and Sets

Parses to an empty object for both.

Serializes and deserializes otherwise valid JSON objects containing circular references into and from a specialized JSON format.

What is serialized by JSON, is serialized in  here. What is not, won't be serialized here.
Maps and Sets are not serialized in  JSON, won't be serialized here.
This is just to avoid circular references, it doesn't add or change anything else to the standard JSON protocol.


Remove c3_qasm_perfect_simulator and provide dry_run option

Describe the missing feature
The c3_qasm_physics_simulator is now in pretty decent shape with increasing support for most of the native features of the c3 simulator and integrates seamlessly into workflows. The c3_qasm_perfect_simulator is no longer necessary, which was a stopgap solution while building the backend is no longer necessary and should be removed. Keeping it around leads to unmaintainable duplicated code. Instead, providing a dry_run option to the backend.run() call would be useful for the user to check that the code runs all fine, and everything has been correctly configured without having to perform the compute intensive full physics simulation. This dry_run path should have a complete API coverage with the standard interface and check as much of the code as possible without running the actual simulation.
Describe the solution you'd like

A backend.run(qc, dry_run = True) interface
Remove entire C3QasmPerfectSimulator code
Remove entire get_sequence() and make_gate_str() from c3_backend_utils
Restructure and streamline qiskit tests
compute_propagators(), evaluate() and process() methods in Experiment should have dry run equivalents

Describe alternatives you've considered
How have you tried to resolve it in the current version of the code? Have you considered non c3-toolset alternative solutions? Please elaborate.
Additional context
Add any other context or screenshots about the feature request here. Please feel free to suggest details about how you believe this feature request might be best implemented. If you have used similar features in other libraries, consider providing links or resources to example implementations.

It would be nice if there is a way of mapping / distilling the QPU
structure defined in C3 to a QISKIT QPU structure, so that gate-only
circuits (i.e. no pulse gates) can be quickly checked using the QISKIT Aer
native simulators
https://qiskit.org/documentation/tutorials/simulators/1_aer_provider.html.
Optional extra bonus points if we can bootstrap a C3 QPU definition from a
QISKIT QPU definition and a specification of the type of qubits and
couplers.
On Tue, 4 Jan 2022 at 21:11, Anurag Saha Roy @.***>
wrote:

Describe the missing feature
The c3_qasm_physics_simulator is now in pretty decent shape with
increasing support for most of the native features of the c3 simulator and
integrates seamlessly into workflows. The c3_qasm_perfect_simulator is no
longer necessary, which was a stopgap solution while building the backend
is no longer necessary and should be removed. Keeping it around leads to
unmaintainable duplicated code. Instead, providing a dry_run option to
the backend.run() call would be useful for the user to check that the
code runs all fine, and everything has been correctly configured without
having to perform the compute intensive full physics simulation. This
dry_run path should have a complete API coverage with the standard
interface and check as much of the code as possible without running the
actual simulation.
Describe the solution you'd like

A backend.run(qc, dry_run = True) interface
Remove entire C3QasmPerfectSimulator code
Remove entire get_sequence() and make_gate_str() from
c3_backend_utils
Restructure and streamline qiskit tests
compute_propagators(), evaluate() and process() methods in Experiment
should have dry run equivalents

Describe alternatives you've considered
How have you tried to resolve it in the current version of the code? Have
you considered non c3-toolset alternative solutions? Please elaborate.
Additional context
Add any other context or screenshots about the feature request here.
Please feel free to suggest details about how you believe this feature
request might be best implemented. If you have used similar features in
other libraries, consider providing links or resources to example
implementations.
—
Reply to this email directly, view it on GitHub
https://github.com/q-optimize/c3/issues/172, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAH3Q4MWIH376SEKSK4MFMTUUNA5XANCNFSM5LIFEFQA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Minimal change: "firefox" to "chrome" in README.md.

Google Chrome is the quickest, smartest browser, and it's the main alternative to Firefox and Safari. What's more, the number of Chrome users is much bigger, isn't it?

How about open /tmp/termtosvg_exp5nsr4.svg || xdg-open $_ to use BROWSER

@Anti-Li I don't think the name of the browser matters for this precise example. Chrome being the most used browser is an excellent reason to mention Firefox instead.
@jedahan Thanks, but it looks too complicated to be included in a "Basic usage" section. It could also trigger issue #4 which is precisely why the example suggests running a browser from the command line.


Importing breaks when a package has a same relative import path as the app

I wasn't sure how to phrase this, but I'll try and explain our problem here.
Say we have a project with a bunch of components and one of those components has the same relative import URL as a component in a different package. For example our project import index file has the following imports;
@import '{mk:packagename}/all';
@import '{}/modules/orders/orders';

The _orders.scss file has a relative import within the module's folder:
@import 'components/card/card';

The _all.scss file is the index file of the package and has the following import in it;
@import 'components/card/card';

For some reason the second import is ignored or at least it looks that way. The class is not outputted in the final CSS used by the application. When we rename our 'card' partial in the projects component folder everything works.
Is this expected behavior?

Yeah, this is known.
Basically in order to allow cross package imports (the {mk:packagename} syntax), I had to write a custom importer for node-sass. However, in the case of relative file paths, node-sass doesn't give me enough information about the file for me to know exactly what the full file path is (and it's not likely to change: https://github.com/sass/node-sass/issues/1188 ).
This means that the code more or less has to guess what the path is, but this breaks if you have duplicates. I didn't find any good way to work around this issue.
What you could do is change the relative path into an absolute one (start with {} or {mk:packagename})

Alright, thanks, good to know.
We'll make sure to use more absolute paths in our packages then.
Thanks for the quick response!


Fix accidental URI path decoding in uri-with-query

This changes URI getPath to getRawPath to prevent accidental decoding.
Please answer the following questions and leave the below in as part of your PR.

[x] I have read the developer documentation.

[x] This PR corresponds to an issue with a clear problem statement.

[x] This PR contains a test to prevent against future regressions

[x] I have updated the CHANGELOG.md file with a description of the addressed issue.

Thanks! Can you also create an issue like the PR template suggests and mention the issue number in the changelog?

Thanks! Can you also create an issue like the PR template suggests and mention the issue number in the changelog?

Sure, I noticed the behavior and was prodding around trying to find out if it's maybe my own fault when I already had the fix. Wasn't sure if you really preferred an issue in that case and took the template question as really just a question. Will open one directly next time 😉

Thanks a lot!


Improve performance and accessibility for dropdowns (on hover)

https://www.w3.org/WAI/tutorials/menus/flyout

The NavMenu.js component addresses this.


Add reference to dua-cli in the README as similar tool

As dua is providing both a CLI mode as well as an interactive mode via dua i I placed it into both categories.
Disclaimer: I am the author of this tool and have adapted this paragraph for my own README.
Edit: Now that I got to use pdu a little I finally get to appreciate the way the data is presented. Whereas dua gives a high-level overview and pdu dives in to reveal exactly where the main offenders in terms of disk space usage are. It took me a while and I even wrote my own tool to solve this problem, but finally I can see the benefits of this kind of visualization.
dust never worked for me as it was too slow and…used too much memory, so pdu truly makes a difference here.
Lastly I encourage you to build a TUI which allows the safe deletion of picked items to support the entire workflow people are usually using pdu for.

Thanks for your compliment btw :smile:. Though I would imagine that pdu uses even more memory than dust (threads and all). Of course, I'm no expert in parallel computing.

Thanks for your compliment btw 😄. Though I would imagine that pdu uses even more memory than dust (threads and all). Of course, I'm no expert in parallel computing.

I compared to dua, here is the result of pdu:
real        10.25
user         2.20
sys         56.06
           146489344  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               13239  page reclaims
                  50  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
              160643  voluntary context switches
              313338  involuntary context switches
<PHONE_NUMBER>77  instructions retired
<PHONE_NUMBER>15  cycles elapsed
           146114624  peak memory footprint

And here is the one of dua.
real        10.56
user         5.42
sys         26.00
           185794560  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               15260  page reclaims
                  50  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
              207419  voluntary context switches
              355053  involuntary context switches
<PHONE_NUMBER>97  instructions retired
<PHONE_NUMBER>1  cycles elapsed
           179029640  peak memory footprint

So, erm, it uses less than a dua which certainly does less, too 🤦♂. It's a bit surprising to me, as dua doesn't have any state on its own in this mode (using jwalk.

Regarding this line, I see that you intend to push items to aggregates? My advice is "beware of reallocation (vec resizing)", I would attempt to give a size hint (Vec::with_capacity) or switch to another container format.

Lastly I encourage you to build a TUI which allows the safe deletion of picked items to support the entire workflow people are usually using pdu for.

I'm not sure if lazy me is willing to do that.  Testing a TUI would be nigh-impossible. How could one guarantee the stability of the software without tests?
Finally, would you mind if I add dua to the list of benchmark?

Regarding this line, I see that you intend to push items to aggregates?

I believe this is just the top-level that will be listed later - it's not huge and won't show up in any profile.

I'm not sure if lazy me is willing to do that.

Indeed it would be quite some work. If lines-of-code count is any measure, the 4k lines of PDU would certainly go up quite a bit. dua clocks in at 3.4k LOCs for everything Rust, so the ways of pdu seem more complex which probably translates to the TUI as well.
I am lazy too, but once I want something bad enough, I will do it :D. With dua this problem I was having of clearing disk space is solved and it isn't clear that a pdu TUI would do so much better to make it worth the effort.

How could one guarantee the stability of the software without tests?

It's actually straightforward. dua itself is an event-driven engine that has events manipulate its state. This is perfectly testable. The rendering is done with tui which supports a test-backend that can be used with snapshot testing. That way one can unit-test both the application state and the looks perfectly. dua doesn't test the looks, it's something I do visually only. Furthermore it only tests the most important happy-paths of typical user journeys. The latter were written after I had something working which was easy enough to not require unit tests solely to protect against regression.

Finally, would you mind if I add dua to the list of benchmark?

Not at all - it would be nice if you could ping me here as I am curious about the findings. Basically you would be comparing the pdu engine against jwalk which dua relies on.

So…I couldn't help myself but to imagine how a TUI could work. But let's start with a question: Is there a way to increase the contrast of these percentage lines? It's so hard to make out on certain levels - probably my main sore point with pdu right now.
The reason I keep thinking about workflow here is what I am usually doing with that data: I want to delete some of it. Even though a full-blown TUI with selection and subsequent (potentially parallel) deletion would be great, maybe there is a way to output the list of paths that it displayed before in a format that makes copy-pasting for deletion easier.
In the end, the user needs complete paths and pdu could provide them. I could imagine running it once for visualization, and another time to get a flat list of paths for removing manually copied ones.

You see, I really like pdu because its automating a part of what dua i allows doing: 1) find a list of candidates for deletion, and 2) delete them. 1) is done better in pdu and I have a feeling that with a little tuning it could replace some uses of dua for me.
Right now I would probably run it before dua i to get an idea, and then use dua i to queue the offenders for deletion. The suggestion above with outputting a flat (and parseable) list of items would definitely help automating these tasks.
Maybe something like pdu --list --level 2 | dua i --from-pdu-list could use the existing TUI of dua to schedule folders on level 2 of the directory tree for deletion. The user would then have to dequeue the ones they don't want to delete.

it isn't clear that a pdu TUI would do so much better to make it worth the effort

Correction: pdu's UI isn't original, I stole it from dust, which in turn stole it from dutree.
I'm probably not going to implement an interactive TUI for deleting files in the near future. However, parallel-disk-usage is also a library crate, the data structures and algorithms for aggregating and visualizing the directory tree are already there. Anyone who wants this feature bad enough could build a tool on top of this library. I am also interested in seeing how it could be done.

Is there a way to increase the contrast of these percentage lines? It's so hard to make out on certain levels - probably my main sore point with pdu right now.

That's the problem of your terminal and/or your fonts.
I use Tilix (which uses VTE under the hood) as my main terminal, and Hack Nerd Font as my font. Here is the screnshot:

I also test the same command on Alacritty, and it looks not as good:

Right now I would probably run it before dua i to get an idea, and then use dua i to queue the offenders for deletion. The suggestion above with outputting a flat (and parseable) list of items would definitely help automating these tasks.
Maybe something like pdu --list-level 2 | dua i --queue-pdu-list could use the existing TUI of dua to schedule folders on level 2 of the directory tree for deletion. The user would then have to dequeue the ones they don't want to delete.

GNU's du can already create a flat, machine-readable list of items. I also plan to add JSON input/output to pdu in the future (it wouldn't be flat however).

That's the problem of your terminal and/or your fonts.

Thanks for the hint, I will see how I can get alacritty to display this better then and change the font.

I also plan to add JSON input/output to pdu in the future (it wouldn't be flat however).

Neat, that would be working fine as well as I would implement this specifically to be able to use pdu as part of the processing pipeline.

I also plan to add JSON input/output to pdu in the future (it wouldn't be flat however).

Neat, that would be working fine as well as I would implement this specifically to be able to use pdu as part of the processing pipeline. I subscribed to releases to stay informed :).

Subscribe to https://github.com/KSXGitHub/parallel-disk-usage/issues/17 and https://github.com/KSXGitHub/parallel-disk-usage/issues/18 as well.

Finally, would you mind if I add dua to the list of benchmark?

Not at all - it would be nice if you could ping me here as I am curious about the findings. Basically you would be comparing the pdu engine with jwalk which dua relies on.

I am happy to inform you that the benchmark reports is now available at a release artifact

My benchmark seems to suggest that you should stop using jwalk.

Congratulations, it's amazing to see there is still performance to be gained in this field.
To me the sub-second runs don't matter that much, but for bigger trees this really starts to show and a couple of milliseconds become seconds.
On my test-set with 1.44 million files dua takes 12.2s whereas pdu takes 10.3, quite significant.
I took a look at what it would mean to use pdu as engine and noticed this would pull in additional dependencies like clap and thus increase the compile time. To fix this in the current setup, cargo features could be used.
In the meantime I will be waiting for the JSON export feature to land which would allow me to use the greater speeds of figuring out good candidates for deletion with the actual deletion TUI of dua :D.

I took a look at what it would mean to use pdu as engine and noticed this would pull in additional dependencies like clap and thus increase the compile time. To fix this in the current setup, cargo features could be used.

Sound advice. I will be implementing this soon.

I took a look at what it would mean to use pdu as engine and noticed this would pull in additional dependencies like clap and thus increase the compile time. To fix this in the current setup, cargo features could be used.

The CLI part of parallel-disk-usage library and its dependencies (clap, structopt, etc.) can now be disabled by disabling default features.

Version 0.3.0 has been released. It can now print disk usage data as JSON to stdout as well as visualizing input JSON from stdin. A new benchmark reports with the latest version of dua has also been produced.

Thanks a lot! I have added an issue to hopefully one day implement a pdu integration.
On another note, can it be that the picture in the README right below the list of program versions used to create it is out of date? The benchmark has run multiple times by now yet the last modified date of it appears as  Sun, 30 May 2021 04:24:53 GMT (produced with xh HEAD https://camo.githubusercontent.com/8a2a9497f22a5d1879128c069cfdb8c1679a7f8b620f5077d602b0170c7b5d11/68747470733a2f2f6b73786769746875622e6769746875622e696f2f706172616c6c656c2d6469736b2d75736167652d302e322e342d62656e63686d61726b732f746d702e62656e63686d61726b2d7265706f72742e636f6d706574696e672e626c6b73697a652e737667).
By now I reproduced part of the benchmark run and am curious about what's happening on the CI runner.
Note that both pdu and dua are extremely close and in theory, pdu  could be a little faster but now suffers from the same M1 problem that dua would suffer from had I not compiled in a different amount of default threads on Apple Silicon. So dua uses 4 threads whereas pdu uses 8.
hyperfine 'dua --apparent-size tmp.sample' 'pdu tmp.sample' 'du tmp.sample'

Benchmark #1: dua --apparent-size tmp.sample
  Time (mean ± σ):      76.4 ms ±   7.1 ms    [User: 105.4 ms, System: 257.2 ms]
  Range (min … max):    73.0 ms … 111.1 ms    26 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (111.1 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark #2: pdu tmp.sample
  Time (mean ± σ):      83.5 ms ±   3.9 ms    [User: 81.3 ms, System: 515.8 ms]
  Range (min … max):    81.2 ms … 103.0 ms    28 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (103.0 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Benchmark #3: du tmp.sample
  Time (mean ± σ):     152.9 ms ±   1.5 ms    [User: 11.6 ms, System: 140.7 ms]
  Range (min … max):   150.0 ms … 156.1 ms    19 runs

Summary
  'dua --apparent-size tmp.sample' ran
    1.09 ± 0.11 times faster than 'pdu tmp.sample'
    2.00 ± 0.19 times faster than 'du tmp.sample'

It's interesting how fast du is given that it uses way less system resources, making it the definitive winner per Watt :D.
As for the reason that on CI the world looks different, the only explanation I could pull out of thin air is that hyperfine is run individually on each program whereas it could possibly also be used to produce all output to generate the report from.

As for the reason that on CI the world looks different, the only explanation I could pull out of thin air is that hyperfine is run individually on each program whereas it could possibly also be used to produce all output to generate the report from.

I think it's actually about the way you invoke hyperfine:

hyperfine 'dua --apparent-size tmp.sample' 'pdu tmp.sample' 'du tmp.sample'

There are also warnings:

Warning: The first benchmarking run for this command was significantly slower than the rest (111.1 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Warning: The first benchmarking run for this command was significantly slower than the rest (103.0 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

In the GitHub Workflow files, I always add --warmup=3 to every hyperfine command.
If you also want to also measure cold start, I suggest rebooting after each benchmark.

On another note, can it be that the picture in the README right below the list of program versions used to create it is out of date?

Yes, I have yet to update the benchmark section of the README. But it doesn't actually matter, because the dua's performance doesn't change much in the benchmark performed by pdu 0.3.0 (direct link to the benchmark reports).
There's also 0.4.0, which I have yet checked out.

I think it's actually about the way you invoke hyperfine:

It's the lazy way of invoking it, admittedly. Ultimately it's hyperfine who runs the programs hundreds of times to get comparable values still.

If you also want to also measure cold start, I suggest rebooting after each benchmark.

With prepare it's possible to purge the fs cache, on MacOS it would be --prepare purge.

[…] because the dua's performance doesn't change much […]

And that's the last unresolved riddle here. Thus far the arrival of pdu already uncovered a lot of interesting knowledge and as far as I can see also helped fixing a synchronization issue in the pdu progress reporting. From my tests I now that both tools very similar regarding performance and it comes down to milliseconds. dua being consistently slower than single-threaded progams, however, makes no sense to me and I am sure there is more interesting knowledge to be uncovered here.
Please don't get me wrong, to me it matters not who is 'the fastest', but I want to understand what's going on as the benchmark contradicts both my experience and measurements alike.

being consistently slower than single-threaded progams, however, makes no sense to me and I am sure there is more interesting knowledge to be uncovered here.

I am still in disbelief that these fast programs are actually single-threaded.

Please don't get me wrong, to me it matters not who is 'the fastest', but I want to understand what's going on as the benchmark contradicts both my experience and measurements alike.

I didn't intend to make pdu the fastest either. I only wanted a dust with acceptable performance, the fact that it becomes the fastest is unintentional.

I am still in disbelief that these fast programs are actually single-threaded.

That's a good point. Last time I tested them on MacOS they were. Maybe that changed. What matters is the version the CI system is using and their threaded-ness should be easy to observe with time.
Just to try one, I downloaded the latest source of ncdu, built it and ran it like this:
➜  ncdu-1.15.1 time ./ncdu  ~/dev
./ncdu ~/dev  0.83s user 17.37s system 56% cpu 32.105 total

It doesn't even saturate a single CPU core, which is quite typical on my system when doing single-threaded filesystem traversals. What matters is the system time (it was stuck in GUI mode for a while longer) so it takes about 18s to traverse what takes dua at its best configuration 10.5s and pdu currently 11.5s due to using 8 threads.
Grepping through the code to look for threading didn't yield results either so I doubt there is a compile flag to turn that on.


feat(logstash): improve logstash provisioning

passing env vars to logstash from .env
using this vars in logstash.yml
added docker-compose for dev setup so it'd work
removed output block that wasn't used

@linuxbandit the difference is that in the previous version it used cloud.id and cloud.yml, doesn't work this way apparently. This works.


Clean up .github directory

Coverage remained the same at 63.554% when pulling 16165cb18c4ca3792c48cbde3068645dcd5d4130 on corinagum:issues into d4ddd6e18afd86c47aa2328ee8cbe972f2bae54a on microsoft:master.