Uploading "raw" data bag items fails, and we now write raw items.

Description
Uploading an encrypted data bag to Chef server fails with this error:
ERROR: Chef::Exceptions::InvalidDataBagItemID: Data Bag items must have an id matching /^[\.\-[:alnum:]_]+$/, you gave: nil

This issue blocks our Learn Chef tutorials. Users reported the issue on the RHEL and Ubuntu flavors of the same tutorial:
https://learn.chef.io/manage-a-web-app/rhel/run-your-web-application-on-a-node/#comment-2597692112
https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2609047548
The user in the second thread worked around the issue by manually adding the id field to the encrypted data bag item's .json file.
I can repo this using Chef DK 0.12.0. I'm almost certain that it does not repro from 0.10.0.
Chef Version
$ chef --version
Chef Development Kit Version: 0.12.0
chef-client version: 12.6.0
berks version: 4.3.0
kitchen version: ERROR

Platform Version
Applies to all platforms. Can repro from Windows 8.1 and Mac OS workstations.
Replication Case
I believe this bug would apply to uploading any encrypted data bag item to the Chef server, but here's a quick way to repo. knife needs to be configured to communicate with a Chef server.
Perform step 2 from the tutorial's quick start page to set up the encrypted data bag locally.
Perform step 3 on this page to upload the data bag to the Chef server.
Client Output
$ knife data bag from file database_passwords mysql_customers.json
ERROR: Chef::Exceptions::InvalidDataBagItemID: Data Bag items must have an id matching /^[\.\-[:alnum:]_]+$/, you gave: nil

Stacktrace
$ knife data bag from file database_passwords mysql_customers.json -VV
INFO: Using configuration from /Users/thomaspetchel/learn-chef/.chef/knife.rb
/Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/data_bag_item.rb:44:in `validate_id!': Data Bag items must have an id matching /^[\.\-[:alnum:]_]+$/, you gave: nil (Chef::Exceptions::InvalidDataBagItemID)
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/data_bag_item.rb:73:in `validate_id!'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/data_bag_item.rb:80:in `raw_data='
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife/data_bag_from_file.rb:94:in `block in load_data_bag_items'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife/data_bag_from_file.rb:85:in `each'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife/data_bag_from_file.rb:85:in `load_data_bag_items'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife/data_bag_from_file.rb:58:in `run'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife.rb:419:in `block in run_with_pretty_exceptions'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/local_mode.rb:44:in `with_server_connectivity'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife.rb:418:in `run_with_pretty_exceptions'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/knife.rb:212:in `run'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/lib/chef/application/knife.rb:142:in `run'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/chef-12.6.0/bin/knife:25:in `<top (required)>'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/bin/knife:23:in `load'
    from /Users/thomaspetchel/.rbenv/versions/2.1.1/bin/knife:23:in `<main>'

I can't reproduce this with a current version of chef:
ubuntu@ip-172-31-37-200:~/learn-chef$ chef-client -v
Chef: 12.9.27
ubuntu@ip-172-31-37-200:~/learn-chef$ knife data bag from file database_passwords mysql_customers.json --secret-file .chef/encrypted_data_bag_secret --local-mode
WARNING: No knife configuration file found
Updated data_bag_item[database_passwords::mysql_customers]

yeah 12.7 should have fixed this.

A user reported that this continues to fail using chef-client 12.9.36.
https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2623267662
One thing to note is that the tutorial has the user encrypt the data bags locally so they can be run through Test Kitchen. (@thommay's example is a bit different than that.) The user later uploads the already-encrypted data bags to the Chef server (e.g. knife data bag from file and not knife data bag from file --secret-file. Users report that while the upload succeeds, the node fails to decrypt the data.
I feel that I have a valid use case, but if this is not the recommended pattern, please let me know. I could imagine doing this differently by having the user encrypt a second copy of the data bags for Test Kitchen to use. However:

it's more steps, more files to keep track of.
the source data bags are less source control friendly because you essentially have unencrypted data under your chef-repo/data_bags directory.

As a second alternative, we could have the user validate the cookbook using unencrypted data bags, and later encrypt & upload them per @thommay's example. However, it seems to me that you're no longer faithfully testing what would happen on your node. I can imagine that if the decryption process fails on the node, it would be frustrating for the user (who's just learning about all of this) to debug the issue on a node. The method we use on Learn Chef enables the user to troubleshoot any issues on local test instances before applying the config to something real.
That said, I would love to ensure this is working if my use case is valid.

I am able to reproduce this error simply by attempting to upload a data_bag.  With or without encryption.  Here are my current versions:
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2
Also note that when I created the data bag via Knife, I had to log out of the Chef Server and back in for the data bag to even show up in the GUI.  Knife however was able to list it immediately.

@tpetchel note that you're using 12.6.0 on rbenv and not omnibus-chefdk which isn't what learnchef users should be doing...

for me with master:
% knife --version
Chef: 12.10.11
% knife data bag from file encrypted ./creds.json
Updated data_bag_item[encrypted::creds]

that is an already-encrypted databag saved on disk that i am re-uploading

So, if people want to fix this then we need a better repro fix.  You should validate that you are using chef-client 12.8.1 or better since we know fixes went into that version.  We also need to see the full contents of the data bag item that you are trying to upload, because guessing what you all are doing does not seem to be working...     reminder on minimal, complete, verifiable example:  http://stackoverflow.com/help/mcve

I am encountering this issue whether I am attempting to upload an encrypted data bag or not encrypted.
I am using a Windows 7 Workstation with WinRM 5.0, latest Chef DK (0.12.0) and Chef Server(12.5.0).
Chef Development Kit Version: 0.12.0
chef-client version: 12.8.1
berks version: 4.3.0
kitchen version: 1.6.0
Chef Server: 12.0.5
I have validated the JSON to be good format.
Here is an sample of trying it without encrypting it:
knife data bag from file serverDataBag .\serverDataBag\bsl.json
ERROR: Chef::Exceptions::InvalidDataBagItemID: Data Bag items must have an id matching /^[.-[:alnum:]_]+$/, you gave: nil
Here is an example of the new data bag I am attempting to upload.  I have removed the specific user/pass info:
{
"BizAccount": {
"phypath":"Jha.Account.WebHost",
"user":"xxxxx",
"pass":"xxxxx"
},
"BizAuthentication":{
"phypath":"Jha.Authentication.WebHost",
"user":"xxxxx",
"pass":"xxxxx"
},
"BslMfa":{
"phypath":"Jha.Authentication.WebApi",
"user":"xxxxx",
"pass":"xxxxx"
},
"Jha.Account.WebApi":{
"phypath":"Jha.Account.WebApi",
"user":"xxxxx",
"pass":"xxxxx"
}
}
I had an existing encrypted data_bag on the server that I am no longer able to update with the same command, which results in the same error.
The existing databag that I have had on the server for months has not had any changes to the JSON.  So I do not anticipate this is an issue related to the JSON.
Thanks

So to state the obvious here if you're getting this error message:
ERROR: Chef::Exceptions::InvalidDataBagItemID: Data Bag items must have an id matching /^[.-[:alnum:]_]+$/, you gave: nil

Check your json and make sure that you have an "id" field at the top level of the json which identifies the data bag item.  When you are doing knife data bag from file <databag> there is no 'ITEM' argument passed on the command line and that value has to be read from the JSON.
If you don't have that field in your JSON the error message is simply 100% accurate -- you haven't given us enough information in the JSON for us to know what to do.
It may be possible that data bags which were knife download'ed before 12.8.1 were having the id field deleted from them, and may be corrupted on disk.  There's not a whole lot we can do[*] about that either, the easiest solution would be to re-run knife download /data_bags again.
[*] We could start to assume the id from the filename and fatal if there is a different id in the file, but I really don't want to go down that route and add more confusion going forwards.

@lamont-granquist: It's not so much that I'm using 12.6.0, the sample output just wasn't updated in awhile. I'll take a look tomorrow and build a better repo case if the issue persists.
That said, this all used to work for the longest time. Were there any major changes made in recent releases?

It was definitely buggy around 12.6.0-era but that should have been fixed in 12.8.1
And like I speculated its possible that people are hitting this who have downloaded data bag items using 12.6.0 and then trying to upload with 12.8.1 and the id field got stripped by the bugginess in 12.6.0
Still need a replication case that /has/ a valid id field in it and then fails on 12.8.x/12.9.x/12.10.x

the output of knife data bag from file encrypted mysql_customers.json --secret-file MYSECRETFILE --local-mode is:
% ruby -rjson -e 'puts JSON.pretty_generate(JSON.parse(STDIN.read))' < mysql_customers.json
{
  "name": "data_bag_item_encrypted_mysql_customers",
  "json_class": "Chef::DataBagItem",
  "chef_type": "data_bag_item",
  "data_bag": "encrypted",
  "raw_data": {
    "id": "mysql_customers",
    "root_password": {
      "encrypted_data": "noDOsTpsTAZlTU5sprhmYZzUDfr8du7hH/zRDOjRAmoTJHTZyfYoR221EOOW\nXJ1D\n",
      "iv": "Bnqhfy6n0Hx1wCe9pxHLoA==\n",
      "version": 1,
      "cipher": "aes-256-cbc"
    },
    "admin_password": {
      "encrypted_data": "TcC7dU1gx6OnE5Ab4i/k42UEf0Nnr7cAyuTHId/LNjNOwpNf7XZc27DQSjuy\nHPlt\n",
      "iv": "+TAWJuPWCI2+WB8lGJAyvw==\n",
      "version": 1,
      "cipher": "aes-256-cbc"
    }
  }
}

Which clearly lacks the 'id' field.  Then I can replicate:
% knife data bag from file encrypted mysql_customers.json
ERROR: Chef::Exceptions::InvalidDataBagItemID: Data Bag items must have an id matching /^[\.\-[:alnum:]_]+$/, you gave: nil

A properly formatted encrypted data bag item downloaded from the chef-server looks like this:
% cat mysql_customers.json
{
  "id": "mysql_customers",
  "root_password": {
    "encrypted_data": "8R966HFobcC+m+woksdtTbC/fWqHy2KovSAkr71zwBAt2CBLKamXcdS/jSrA\nQl9w\n",
    "iv": "04i16AAnWRCWgbx58j539w==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  },
  "admin_password": {
    "encrypted_data": "u8l3yczEBzD+yhD77CL+XTIjW03KaBC8O8QyG3pAML2/GkCFyfhW40sWvEVK\nNMWW\n",
    "iv": "nliIDZy12ypYGXhSxRrdPQ==\n",
    "version": 1,
    "cipher": "aes-256-cbc"
  }
}

You're getting the on-disk format for chef-zero by using --local-mode.  I don't know if that changed or what, but that smells really incorrect.
Maybe @jkeiser or @thommay has more information on if that might have changed recently.  I'm somewhat skeptical that ever worked right.  And definitely feel like that is the wrong API to be using...
I'm leaning towards this being a doc-bug in learnchef...

So this bug may actually be "we need a supported API to encrypt a data bag on disk with the user's secret-key (without involving bizzare hacks that try to round trip through chef-zero)"

Here's a more minimal repro case. I'm chatting with @jkeiser offline.
#!/bin/bash

mkdir -p data_bags/passwords

(
cat <<'EOF'
{
  "id": "my",
  "password": "ABCD123!"
}
EOF
) > data_bags/passwords/my.json

knife data bag from file passwords my.json --local-mode --secret-file .chef/encrypted_data_bag_secret
knife data bag from file passwords my.json --local-mode

This happens regardless of whether the data bag is encrypted or not.
Either chef-zero needs to write the format expected of knife data bag from file or knife needs to be smart enough to do the translation. There are lots of use cases where you'd need to upload a data bag JSON from chef-zero, such as when you're bringing up an entire infrastructure in AWS, including a chef server, and you then need to make that new chef server aware of the AWS environment.
Other examples include encryption, as indicated here, as well as writing out a data bag from a local recipe for later upload elsewhere.
This has worked in the past but I'm unsure what changed. It sorta looks like chef-zero started writing the on-disk version or something.

Also, if this helps anyone, I've used this script to translate my data bags en-masse... just use find to locate the JSON and shove them through this script into a new one.
#!/usr/bin/env ruby

require 'json'

data = JSON.parse(ARGF.read)

if data.include?('raw_data')
  puts JSON.pretty_generate(data['raw_data'])
else
  puts JSON.pretty_generate(data)
end

So @tpetchel the "workaround" here is the fix you should use:
https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2623267662
Simply keep the on-disk format unencrypted, and don't try to solve that problem in the learnchef tutorial.
We may wind up patching this to reinstate the old behavior to address @ChadScott's use-cases, but just because the --local-mode hack worked doesn't mean that anyone should be documenting that -- and particularly we should not be teaching chef newbies to use that as an API, ever, even if its fixed.  That is well down the "one strange trick" path and not canonical chef, and the reasons why it worked are way too obscure to put in front of newbies.
We probably do need to support encrypting data-bags in the chef-repo with the secret-key but that API should not be identical.  The output should be the same format as one gets from knife download /data_bags and not the on-wire format that chef-zero is saving to disk.

we probably broke this with some combination of

removing json_class
the PR for #4664 which fixed #4392

before anyone suggests reverting #4664, we can't do that because that just pushes the problem around, and #4664 affects many more users.  we're also not reverting the removal of json_class, we just need to fix bugs here.
we apparently have a requirement to support both pretty 'user' format and on-wire 'chef-server' format that we weren't ever aware of (and had zero tests around, etc).
what i'd really like to see is a mixin for all our object models which supported reading from a file and decoding either format, for all our object models.  along with tests so that we don't break it in the future.  along with nuking some of the code in knife's object loader which seems to be getting into a fight with core-chef code.
at the very least i'd like to see that mixin created and the data bag item object patched to support it along with sorting out the conflicting logic in knife.  we can be iterative about fixing our entire object model, but i'd like to see some progress in the direction of making things sane.

we also need to support a better knife API for encrypting an on-disk data bag item without round-tripping through chef-zero and writing out a wire-format encrypted data bag -- that is terrible for many different reasons, not the least of which is that it'll inherently and by-design break knife diff and that 'bug' will never be fixable because that API should never have been used.

Sorry for not closing this out earlier. I got some more information offline and understand how to make my scenario work now using the correct APIs/patterns.

not closed because we still need to do the work

I arrived at this issue from #4892 and I'm not sure how related this issue exactly to the one just mentioned, but the comments section referred here. I had the exact same error message, namely:
Chef::Exceptions::ValidationFailed: Option data_bag's value {"encrypted_data"=>"6zYvton8uCvsJfTbeuf5huwxWzsZ5TZoF6oJ1jsFb+s=\n", "iv"=>"FrlwhrRL3A/KGsyATYtsLw==\n", "version"=>1, "cipher"=>"aes-256-cbc"} does not match regular expression /^[\-[:alnum:]_]+$/

I resolved this by switching my default EDITOR to Notepad instead of Sublime Text as Sublime Text didn't keep the file open for the duration of the edit, which caused a mismatch with the regular expression as just the id key and value made it to the Chef Server during the creation of the data bag item. Again, not quite sure if this is helpful, but just throwing this out there for anyone who might be running into the same issue.

I came here the same way that usrme did.  Running  a "knife data bag edit  bag item  --encrypt"  and exiting the editor with or without changes  causes the data bag to become corrupted  with any further attempts to edit the data bag (with or without --encrypt) impossible.

Saved data_bag_item[bofh]
$ knife data bag show user_password bofh --secret-file ../../../chef-data-bag-shared-secret.txt
ERROR: Chef::Exceptions::ValidationFailed: Option data_bag's value {"encrypted_data"=>"4Pc1wRINte5r/jw5wf6GvpCh/61pU5IkNu6fULPVioK5hHZ1Y6aXV/lthZUl\nNCob\n", "iv"=>"q96kYsHC8rX/9WfgZTxZlA==\n", "version"=>1, "cipher"=>"aes-256-cbc"} does not match regular expression /^[-[:alnum:]_]+$/
The workaround appears to be ensuring that the data bag is encrypted on creation then I don't get the error the 2nd on 'knife data bag show' command.

The original bug seems to be around using the on-disk data bag format from something like chef-solo, as opposed the the format that we use against the Chef Server API. Which perhaps worked at one point but was never intended to work. Something like knife data bag show test_data_bag bob -F json (without --secret-file) would give you the JSON to save on disk that would be compatible with knife data bag from file test_data_bag bob.
I believe the issue @jdblack and @usrme had is a different bug that the OP that results when using knife node edit --secret-file with an editor.
I reproduced @GreyArea765's issue on 14.8.12, which isn't latest but is what happens to be on my laptop. I'm don't think that's the same as @tpetchel's original bug though. The bug there is probably to have knife data bag edit --secret-file fail if it is editing an unencrypted data bag.

Issues around using knife data bag edit --encrypt against an unencryped data bag and having it corrupted have been moved to #8282, and distracting comments in this PR have been cleaned up.


Add SystemPacketBuffer to support pbuf_type PBUF_RAM

Current Matter has pbuf_type PBUF_POOL support by default. Bouffalol Lab platform needs use another pbuf_type PBUF_RAM.
PBUF_RAM has three kinds of memory management, this PR is to support ram_heap with LWIP macro options: MEM_LIBC_MALLOC = 0 and MEM_USE_POOLS = 0.
This PR only contains SystemPacketBuffer changes for review:

Add method MemorySize() to calculate actual memory size which allocated from ram_heap
Update method AllocSize() to call MemorySize() for which is using ram_heap
By default, Matter is still using PBUF_POOL for LWIP. Add macro defines as below in platform to enable this feature
#define LWIP_PBUF_FROM_CUSTOM_RAM_HEAP  1
#define CHIP_SYSTEM_CONFIG_PACKETBUFFER_LWIP_PBUF_TYPE PBUF_RAM

Discussed with @tcarmelveilleux, @tcarmelveilleux thinks better to do

1- Remove dependency on pbuf entirely
2- Keep track of length always

Thanks for reviews and inputs from @tcarmelveilleux @bzbarsky-apple @pidarped.


Added support for ocs provider server to fetch noobaa client resources

Continuation PR for #2680

/hold

Continuation PR for #2680

should #2680 be also in for this PR?

/lgtm

/unhold

/cherry-pick release-4.17

/cherry-pick release-4.17


GPLv3 License violation (readme)

Same thing as in the other repo, you say in readme:
You are not allowed to copy or use this app code to create similar application to distribute it on Google Play or any other store.
while the GPLv3 license allows this completely, but requires the other app to release it's own source code if it is distributed (as it is a copyleft).

Same thing as in the other repo, you say in readme: You are not allowed to copy or use this app code to create similar application to distribute it on Google Play or any other store. while the GPLv3 license allows this completely, but requires the other app to release it's own source code if it is distributed (as it is a copyleft).

So same case here as you said in other repo to maybe use this https://commonsclause.com/

@ristomatti do you want to give your opinion to this? (original story here https://github.com/norkator/open-intelligence/issues/191, how this got started)

@raikasdev BTW this repo you've contributed to https://github.com/Testausserveri/TestausFood is also suffering from a similar issue. It says the code is FOSS but since there's no explicit license, AFAIK this prevents using it as such. :slightly_smiling_face:


Rolling buffer size / duration in fine-tune segmentation model

Hi! @juanmc2005  sorry for bothering you! It's me again!
I learned and understand a lot from your reaching in #58 before!
Recently I thought of a question, please forgive my bad memory if I asked
In your awesome work, I know that there's the rolling buffer in 5s, and here are two tunable parameters related to "step" and "latency".
SO~Does the rolling buffer size can be changed?
if i fine-tune the segmentation model like
pretrained = Model.from_pretrained("pyannote/segmentation")
seg_task = Segmentation(callhomeSPK2, duration=10.0, max_num_speakers=2)
Does the duration here mean the rolling buffer or something else?
Cause, after I fine-tune them and store them in ckpt file
it seems that it can be loaded with  Model.from_pretrained(ckpt_path, strict=False)
But as far as I know, it doesn't feel like it's possible to adjust simply by doing this.
That's why I came to ask for your teaching :)
And why i want to change the rolling buffer size is just because I suddenly thought of the paper I saw half a year ago,  Online end-to-end neural diarization with speaker-tracing buffer and Online streaming end-to-end neural diarization handling overlapping speech and flexible numbers of speakers, from Xue, Yawen.
So I'm wondering if it's possible to improve performance by upgrading the rolling buffer size.
Looking forward to your reply~~
Best Regards

Hi @Shoawen0213,
Definitely! The duration of the rolling buffer can be changed in diart. If you fine-tune a pyannote/segmentation model with duration=10.0 and you don't specify the duration in PipelineConfig, it should automatically choose the rolling buffer size to match the training duration.
Remember that step <= latency <= duration, so if you increase the duration you can also increase the latency.
However, I can think of two disadvantages of doing this:

You'll increase the real-time factor because getting an output will now be more costly
You'll probably have to use a coarser resolution to avoid having extremely long input sequences to the LSTM

I hope this makes sense, let me know what you think.

Hi! @juanmc2005, Thanks for your answer!!
I did some simple experiments increasing the buffer size, like 10 seconds and 20 seconds.
The RTF did not increase so significantly actually, but the result was indeed not as good as the original.
This makes me really curious why those two papers can be good with buffer size in the 50s, 100s
Again!
Thanks for your awesome work!!! It really helps me a lot indeed!


Multi-module projects

This proposal is to have projects that consist of more than one module, where the main module imports others.  The UI, I suppose, would look like tabs within the editor area.  The goal would be to keep this as simple as possible for users who don't want it.  I am not interested in making the UI more complex for beginners in order to support a few advanced students; but it would be good to have a reasonable path for those advanced students.
This also makes Haskell-mode much more appealing as a general-purpose tool for learning Haskell.

Related to this, there's also #12, which proposes a whole project type that's just a library.  That's a different approach, where students who want to split up their code would create more than one project.  At a first glance, that seems far too complicated for just solving the organization problem.  It could be done in addition to this one, at some point.  Just like normal Haskell packages can contain multiple source files by themselves, but can also depend on other packages.


[AutoPR network/resource-manager/dns/resource-manager] Added ref to CreateOrUpdatePrivateZone in /examples

Created to sync https://github.com/Azure/azure-rest-api-specs/pull/3485

(message created by the CI based on PR content)

Codecov Report

Merging #2989 into master will decrease coverage by 0.05%.
The diff coverage is 24.7%.

@@            Coverage Diff             @@
##           master    #2989      +/-   ##
==========================================
- Coverage   56.35%    56.3%   -0.06%
==========================================
  Files        8124     8124
  Lines      166280   166468     +188
==========================================
+ Hits        93713    93723      +10
- Misses      72567    72745     +178

Impacted Files
Coverage Δ

.../operations/virtual_network_peerings_operations.py
17.27% <ø> (ø)
:arrow_up:

...ions/express_route_service_providers_operations.py
26.47% <ø> (ø)
:arrow_up:

...ons/load_balancer_network_interfaces_operations.py
26.47% <ø> (ø)
:arrow_up:

...01/operations/express_route_circuits_operations.py
12.8% <ø> (-0.05%)
:arrow_down:

...erations/application_security_groups_operations.py
15.9% <ø> (ø)
:arrow_up:

...etwork/v2017_10_01/operations/usages_operations.py
26.47% <ø> (ø)
:arrow_up:

...0_01/operations/load_balancer_probes_operations.py
20.75% <ø> (ø)
:arrow_up:

.../load_balancer_backend_address_pools_operations.py
20.75% <ø> (ø)
:arrow_up:

...1/operations/network_security_groups_operations.py
14.45% <ø> (-0.09%)
:arrow_down:

...etwork/v2017_10_01/operations/routes_operations.py
17.27% <ø> (ø)
:arrow_up:

... and 109 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9230842...6a74b1e. Read the comment docs.


Add Kubernetes Instructions to the Documentation App

We currently have experimental K8s instructions in the repo. They should be labeled "experimental" as more customers offer feedback. These instructions should also be included in the Docs app because some people will find it easier to navigate and parse.

I believe the reason we flagged the Terraform examples as experimental was because we didn't have CI to prevent regressions. The K8s deployment runs in CI and is regression tested, so it would seem somewhat confusing to me if we marked it as experimental.
The docs certainly should contain all deployment readmes, regardless of stability. I'm not sure whether adding things to the docs manually all the time is the way to go though. We need infra to handle this automatically. I've created #594 to track this.


getInternalRouterError for missing POST endpoint instead of 405 Method Not Allowed

Reproduction

Default Vite project, without action defined in root.tsx
Hit POST at / emulating malicious attack
Get getInternalRouterError which is a server error, not a client error

System Info
System:
    OS: macOS 14.5
    CPU: (8) arm64 Apple M1 Pro
    Memory: 117.66 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.11.0 - ~/.nvm/versions/node/v20.11.0/bin/node
    Yarn: 1.22.22 - ~/.nvm/versions/node/v20.11.0/bin/yarn
    npm: 10.2.4 - ~/.nvm/versions/node/v20.11.0/bin/npm
    pnpm: 9.0.2 - ~/.nvm/versions/node/v20.11.0/bin/pnpm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 127.0.6533.119
    Safari: 17.5
  npmPackages:
    @remix-run/dev: ^2.11.1 => 2.11.1
    @remix-run/express: ^2.11.1 => 2.11.1
    @remix-run/node: ^2.11.1 => 2.11.1
    @remix-run/react: ^2.11.1 => 2.11.1
    vite: ^5.3.5 => 5.3.5

Used Package Manager
pnpm
Expected Behavior
Silently return 405 Method Not Allowed to the client without triggering error reporting such as Sentry.
POST was made by a malicious attack attempt from a random client, thinking it's a PHP server.
Actual Behavior
A server error thrown:
Error: You made a POST request to "/" but did not provide an `action` for route "root", so there is no way to handle the request.
    at getInternalRouterError

without triggering error reporting such as Sentr

I'm not sure what you are asking for here.  Do you want to filter this from going to Sentry?  You can control server-side error handling via handleError

My point is that it's not a server error. Client (status code 400-499) errors should be reported to the client, not to the server owner. (minimal access log would suffice, definitely not the backtraces)
For instance, you could hit a random URL such as https://my-remix-site.org/non-existent-path, and it should just respond 404 Not Found to the client. You don't want to get notified for it, and frameworks should handle those client errors.

Client (status code 400-499) errors should be reported to the client, not to the server owner

Strong disagree :).  "Client error" means that the request was in error (as opposed to the server code having a bug), but that doesn't mean the server owner should not be notified.  These 4xx errors may be a sign of something gone wrong - there's a bunch of good examples in https://httptoolkit.com/blog/how-and-why-to-monitor-http-client-errors.

it should just respond 404 Not Found to the client. You don't want to get notified for it

Why not? Your application may not - but what if I want to know about a spike in 404s on the server to see if I published a dead link somewhere?  It's not Remix's call to make as to whether or not to log 4xx errors - we can only give applications the information needed to decide what's best for their use case.

I also feel there should be a catch-all method to ignore all client errors

There is:
export function handleError(error) {
  if (isRouteErrorResponse(error) && error.status < 500) {
    return;
  }
  // ...
}

Thanks! I see you have a good counter point of view — I've updated my code to handle specific client errors rather than blanket catch-all.
export function handleError(error, { request }) {
  if (request.signal.aborted) return
  if (isProd && isRouteErrorResponse(error) && [404, 405].includes(error.status)) return

  Sentry.captureException(error)
  console.error(error)
}

Ah, I just got another error report, and realized that it's considered as 400 Bad Request, not 405 Method Not Allowed — that's why it slipped through. (405 is safer to ignore than 400)
I think this is a bug — should we reopen this issue? What do you think?
{
  data: Error: You made a GET request to "/api/xxx" but did not provide a `loader` for route "routes/api+/xxx", so there is no way to handle the request.,
  error: {
    message: You made a GET request to "/api/xxx" but did not provide a `loader` for route "routes/api+/xxx", so there is no way to handle the request.,
    name: Error,
    stack: ...
  },
  internal: true,
  status: 400,
  statusText: Bad Request
}

I think these are ok as-is - plus it would potentially be a breaking change to alter the status.
We use 405 for the action scenario because without an action, in no-case is a POST request acceptable by the path.
But in the loader scenario, it's more nuanced.  If the routes/_index route doesn't have a loader, then GET http://localhost:5173/?_data=routes%2F_index is invalid because it's a targeted request to a specific route without a loader, but GET http://localhost:5173/ is valid to load the HTML document.
405 is defined as:

The request method is known by the server but is not supported by the target resource

So IMO the GET method is supported by the target resource / but just not in this specific use-case driven by the query params.

So IMO the GET method is supported by the target resource / but just not in this specific use-case driven by the query params.

Fair, but my second case is at a resource route POST /api/xxx, not /. Also FWIW, I set unstable_singleFetch turned on.
In my case, it's intentional to not have a GET method for that — it's an action that receives application/json and causes mutation in the database, so it makes sense to return 405 for GET.

Hmm, I don't think it's a good idea to add such a boilerplate code to all API resource routes, just because some random bots might hit the system with non-existent GET requests.


Added feature to flatten ndjson

closes: #73
Provided users with options to flatten NDJSON file when the values are nested.
Options to flatten NDJson:

provide path to the attribute
Use lambda function to create a custom logic. Here Lambda function have access to complete row as it first param.

Example File:

{
 "name": "dave",
 "address" : {
    "city" : "landon",
    "country": "UK",
    "street": "221B Bakers Street"
  }
}
{
 "name": "jane",
 "address" : {
    "city" : "Landon",
    "country": "UK",
    "street": "221B Bakers Street"
  }
}

API
load_file(
   "path/to/file/somefile.ndjson",
   "output_table": <astro table object>,
   "flatten_ndjson": {
      "address":  "address.street",
      "name": lambda x: x.toUpper()
   }
)

Results:
Name   Address
DAVE    221B Bakers Street
JANE    221B Bakers Street

@tatiana, I like the idea of using pandas json_normalize, it has much richer interface. Maybe we can have a param like 'normalize' which we can pass to json_normalize.
json_normalize(data, **normalize)

@dimberman @tatiana this condition can occur for json file type as well, do we need to handle it?

Can you fix merge conflicts please @utkarsharma2


XML Validator not working

Description
XML Validator doesn't work (xml is always valid) if you don't add the path to the xsd schema in the xml file.
A program like XMLSpy when validating against a schema asks if it should modify the xml file and adds the path to the xsd schema in the xml file itself.
Steps To Reproduce
Go to the XML Validator
When the check doesn't work

adding the attached 'Shema.xsd'
adding the attached 'not-working.xml'
When the check works
adding the attached 'working.xml'. In the file is added the following change in the line cpFile. Need to change the path to the schema
<cpFile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://nhif.bg/xsd/hosp/cp file:///E:\TestXML/Schema.xsd" xmlns="http://nhif.bg/xsd/hosp/cp">

It is now apparent that the xml is not validating according to the schema.
Expected behavior
Changing the xml file is done automatically for the check to work properly.
DevToys Version
Version <IP_ADDRESS> | X64 | RELEASE | 8bcfa60 | 8bcfa60
Relevant Assets/Logs
TestXML.zip

@JustAGabriel FYI

@JustAGabriel @veler is a correction planned in the near future?

Hi @NikolayEKolev I will look into this!

PR will be created mid/end of next week.

I am currently working on a solution. Though the solution does not aim to implement a option to automatically adjust/add namespaces within the validation scope.

Is there an option to implement an automatic mechanism for namespace inconsistencies with a warning that the original file will be changed?

In general for sure.
But I simply don't see the need nor the demand for it.
Create a feature request, get some up votes and prove me wrong 🙂

In general for sure. But I simply don't see the need nor the demand for it.
Create a feature request, get some up votes and prove me wrong 🙂

I guess the question is whether to ask to modify the file. If this is what you are talking about, there is no need to create a new FR. I think in XMLSpy it asks that's why I wrote it :)
I can prepare another schematic and file for testing.


Update releases page

This PR adds notes to the releases page for preview release 11

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Will mark as ready for review once versions are finalized

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/

Preview is available here:http://soroban-docs-pr566.previews.kube001.services.stellar-ops.com/


Changing the point cloud range

Am trying to change the point cloud range from [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0] to [-115.0, -74.0, 0.0, 230.0, 320.0, 30.0], it fails with the below error
return forward_call(*input, **kwargs)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/models/backbones/sparse_encoder.py", line 119, in forward
x = self.conv_input(input_sp_tensor)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/ops/spconv/modules.py", line 132, in forward
input = module(input)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(input, **kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/ops/spconv/conv.py", line 197, in forward
out_features = Fsp.indice_subm_conv(
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 92, in decorate_fwd
return fwd(_cast(args, cast_inputs), **_cast(kwargs, cast_inputs))
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 35, in _cast
return type(value)(iterable)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 33, in
iterable = map(lambda v: _cast(v, dtype), value)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 25, in _cast
return value.to(dtype) if is_eligible else value
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[hykeserver:688058] *** Process received signal ***
[hykeserver:688058] Signal: Aborted (6)
[hykeserver:688058] Signal code:  (-6)
[hykeserver:688058] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f0655bf1420]
[hykeserver:688058] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f0655a2e00b]
[hykeserver:688058] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f0655a0d859]
[hykeserver:688058] [ 3] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/../../../../libstdc++.so.6(+0xb135a)[0x7f06510cf35a]
[hykeserver:688058] [ 4] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/../../../../libstdc++.so.6(+0xb03b9)[0x7f06510ce3b9]
[hykeserver:688058] [ 5] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/../../../../libstdc++.so.6(__gxx_personality_v0+0x87)[0x7f06510ceae7]
[hykeserver:688058] [ 6] /home/hykeserver/anaconda3/envs/bevf_ptt19/bin/../lib/libgcc_s.so.1(+0x111e4)[0x7f065273b1e4]
[hykeserver:688058] [ 7] /home/hykeserver/anaconda3/envs/bevf_ptt19/bin/../lib/libgcc_s.so.1(_Unwind_Resume+0x12e)[0x7f065273bc1e]
[hykeserver:688058] [ 8] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/libc10_cuda.so(+0x1d2fb)[0x7f064badf2fb]
[hykeserver:688058] [ 9] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/libc10.so(_ZN3c1010TensorImpl17release_resourcesEv+0xa4)[0x7f064b865314]
[hykeserver:688058] [10] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/libtorch_python.so(+0x295359)[0x7f05e913b359]
[hykeserver:688058] [11] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/libtorch_python.so(+0xadb231)[0x7f05e9981231]
[hykeserver:688058] [12] /home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/lib/libtorch_python.so(_Z28THPVariable_subclass_deallocP7_object+0x292)[0x7f05e9981532]
[hykeserver:688058] [13] python(+0x11cb86)[0x563e9b7b7b86]
[hykeserver:688058] [14] python(+0x11cf76)[0x563e9b7b7f76]
[hykeserver:688058] [15] python(+0x11cf76)[0x563e9b7b7f76]
[hykeserver:688058] [16] python(+0x1109a0)[0x563e9b7ab9a0]
[hykeserver:688058] [17] python(+0x11d346)[0x563e9b7b8346]
[hykeserver:688058] [18] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [19] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [20] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [21] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [22] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [23] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [24] python(+0x11d2fc)[0x563e9b7b82fc]
[hykeserver:688058] [25] python(PyDict_SetItem+0x2ac)[0x563e9b80025c]
[hykeserver:688058] [26] python(PyDict_SetItemString+0x4f)[0x563e9b80094f]
[hykeserver:688058] [27] python(PyImport_Cleanup+0x9b)[0x563e9b8e546b]
[hykeserver:688058] [28] python(Py_FinalizeEx+0x83)[0x563e9b8e5823]
[hykeserver:688058] [29] python(Py_RunMain+0x110)[0x563e9b8e8780]
[hykeserver:688058] *** End of error message ***
Aborted (core dumped)
I noticed that my GPU memory is not used completely, anybody faced this issue?

Hi @VeeranjaneyuluToka,
One potential reason why you met with this error is that there exists a parity between the point_cloud_range and grid_size (here) in the head. For example, if we have xy range = [-51.2m, 51.2m] and voxel_size=0.1m, the grid_size should be [1024, 1024] because (51.2-(-51.2)) / 0.1 = 1024. Therefore, you may want to adjust these grid_size parameters accordingly. These parameters are used in SparseTensor construction and a too small value may lead to negative coordinates, which breaks the neighborhood query kernel in the SpConv backend.
Best,
Haotian

@kentang-mit , Thanks for your reply!
I changed as below
voxel_size = [0.075, 0.075, 0.2] - no change
point_cloud_range = [-400.0, -400.0, -78.04, 400.0, 400.0, 78.04]
grid_size = [ 10667, 10667, 1]
with the above settings, am getting the error as below
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/models/fusion_models/bevfusion.py", line 251, in forward_single
feature = self.extract_lidar_features(points)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/models/fusion_models/bevfusion.py", line 139, in extract_lidar_features
x = self.encoders["lidar"]["backbone"](feats, coords, batch_size, sizes=sizes)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/models/backbones/sparse_encoder.py", line 119, in forward
x = self.conv_input(input_sp_tensor)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/ops/spconv/modules.py", line 132, in forward
input = module(input)
File "/home/hykeserver/anaconda3/envs/bevf_ptt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/ops/spconv/conv.py", line 163, in forward
outids, indice_pairs, indice_pair_num = ops.get_indice_pairs(
File "/home/hykeserver/Downloads/git_repos/object_detection_3d/bevfusion/mmdet3d/ops/spconv/ops.py", line 92, in get_indice_pairs
return get_indice_pairs_func(
RuntimeError: mmdet3d/ops/spconv/src/indice_cuda.cu 124
cuda execution failed with error 2
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
Any inputs on this?

@kentang-mit , i experimented a bit more on the valid range, changed accordingly and it solved this issue.  Thanks!


Remove MetricResult.config.

Related to #491
This change removes MetricResult.config. This member is not used at this point, and we basically don't need to store the same config in this struct because we can obtain the same information directly from Metric.

I think MetricConfig (or Metric) should be serialized separately as well. I also think parallel structure is more maintainable than nested structure.
"metric_configs": {
  "foo": <foo_config>,
  "bar": <bar_config>
},
"metric_results": {
  "foo": <foo_result>,
  "bar": <bar_result>
}

OK, sounds good. I'll review this.


reapply my changes

Let's talk about it if you want to change anything in this PR

OK bartek take a look now. I split the federation handler and the reverse federation handler apart rather than have these conditionals within the unified function.

I put the common error handling code in a separate function.


Unlisted Youtube Video Results in "FFmpegPostProcessorError: Conversion failed!"

Checklist

[X ] I'm reporting a broken site support issue
[ X] I've verified that I'm running youtube-dl version 2020.03.24
[ X] I've checked that all provided URLs are alive and playable in a browser
[ X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ X] I've searched the bugtracker for similar bug reports including closed ones
[ X] I've read bugs section in FAQ

Verbose log

kevin@linuxserver:~/youtube-dl/scripts$ ./ytdl-kidsyoutube-singlevideo-verbose https://youtu.be/VZu7iFWrzs8
[debug] System config: []
[debug] User config: []
[debug] Custom config: [u'--format', u'(bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best', u'--merge-output-format', u'mkv', u'--write-info-json', u'--write-all-thumbnails', u'--add-metadata', u'--embed-subs', u'--ignore-errors', u'--proxy', u'', u'--download-archive', u'/home/kevin/scripts/youtube/downloaded.txt']
[debug] Command-line args: [u'--config-location', u'/home/kevin/youtube-dl/config2.txt', u'-v', u'-o', u'/home/kevin/youtube-dl/kidstmp/%(title)s.%(ext)s', u'https://youtu.be/VZu7iFWrzs8']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.24
[debug] Python version 2.7.17 (CPython) - Linux-5.3.0-51-generic-x86_64-with-Ubuntu-19.10-eoan
[debug] exe versions: ffmpeg 4.2.1-2, ffprobe 4.2.1-2
[debug] Proxy map: {}
[youtube] VZu7iFWrzs8: Downloading webpage
[youtube] VZu7iFWrzs8: Downloading MPD manifest
[info] Writing video description metadata as JSON to: /home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.info.json
[youtube] VZu7iFWrzs8: Downloading thumbnail ...
[youtube] VZu7iFWrzs8: Writing thumbnail to: /home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.jpg
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/dash/expire/1588275172/ei/hNOqXp2eHIH3hwbI6YmgDg/ip/<IP_ADDRESS>/id/559bbb8855abcecf/source/youtube/requiressl/yes/playback_host/r3---sn-8xgp1vo-ab5s.googlevideo.com/mh/4O/mm/31%2C29/mn/sn-8xgp1vo-ab5s%2Csn-ab5szn7z/ms/au%2Crdu/mv/m/mvi/2/pl/16/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1190000/vprv/1/mt/1588253494/fvip/3/keepalive/yes/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AJpPlLswRgIhANdJMl_r6pAtXfGTl-jTmDXeFBiv-axEivq1-nmpXLTNAiEA2PUQ6iRaJM2UBv7lapGrvrh_veTkQbcxWq0fj5ww20s%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/ALrAebAwRAIgSq0Yat4Yl9fidwprUPVto-akBuGAo1NozRAiD2Z-KskCIFQLty-asb1eQOkBgaQuRiLly9Nd0OLErvEUuX9zGbUf'
[dashsegments] Total fragments: 305
[download] Destination: /home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.f303.webm
[download] 100% of 541.05MiB in 02:15
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/dash/expire/1588275172/ei/hNOqXp2eHIH3hwbI6YmgDg/ip/<IP_ADDRESS>/id/559bbb8855abcecf/source/youtube/requiressl/yes/playback_host/r3---sn-8xgp1vo-ab5s.googlevideo.com/mh/4O/mm/31%2C29/mn/sn-8xgp1vo-ab5s%2Csn-ab5szn7z/ms/au%2Crdu/mv/m/mvi/2/pl/16/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1190000/vprv/1/mt/1588253494/fvip/3/keepalive/yes/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AJpPlLswRgIhANdJMl_r6pAtXfGTl-jTmDXeFBiv-axEivq1-nmpXLTNAiEA2PUQ6iRaJM2UBv7lapGrvrh_veTkQbcxWq0fj5ww20s%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/ALrAebAwRAIgSq0Yat4Yl9fidwprUPVto-akBuGAo1NozRAiD2Z-KskCIFQLty-asb1eQOkBgaQuRiLly9Nd0OLErvEUuX9zGbUf'
[dashsegments] Total fragments: 164
[download] Destination: /home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.f251.webm
[download] 100% of 20.12MiB in 00:27
[ffmpeg] Merging formats into "/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel 'repeat+info' -i 'file:/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.f303.webm' -i 'file:/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.f251.webm' -c copy -map '0:v:0' -map '1:a:0' 'file:/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.temp.mkv'
ERROR: Conversion failed!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 2065, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 512, in run
    self.run_ffmpeg_multiple_files(info['__files_to_merge'], temp_filename, args)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 235, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError: Conversion failed!

[ffmpeg] Adding metadata to '/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.mkv'
ERROR: [Errno 2] No such file or directory: '/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.mkv'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 808, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 863, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1644, in process_video_result
    self.process_info(new_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 2000, in process_info
    self.post_process(filename, info_dict)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 2065, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 498, in run
    self.run_ffmpeg_multiple_files(in_filenames, temp_filename, options)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 210, in run_ffmpeg_multiple_files
    os.stat(encodeFilename(path)).st_mtime for path in input_paths)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 210, in <genexpr>
    os.stat(encodeFilename(path)).st_mtime for path in input_paths)
OSError: [Errno 2] No such file or directory: '/home/kevin/youtube-dl/kidstmp/Hickory Dickory Dock + More Kids Songs & Music Classes _ The Jam Cats Music.mkv'

Description

Every week my daughter's music class posts 2 unlisted youtube videos while we are all in quarantine. I download these videos for her normally without problem.
This week, I was able to download the first video, but the second video keeps throwing the above error. As far as I can tell, there is nothing different about this video than the 10 others that they've posted. Additionally, I am using the same script/options on this video as I am on the others (with the exception of adding the -v flag for debug purposes).
I searched the issue tracker, and it seems that this error can mostly be related to DRM, or needing an account to access the video, but neither of those things seem to apply to this video, as I am able to play it in an in-private window without logging in on a completely separate network (in addition to being able to play it on my own account on my own LAN).
Any help would be greatly appreciated as to what I may be missing. Thanks

https://github.com/ytdl-org/youtube-dl/issues/25687


old projects (community examples) crash the app

When one want to open an ancient Community Edition example project, the app crashes... looks like the json format is not compatible since a recent update...

Yes, but more than that, most of the community project available are in an old json format and not readable with latest version of the app


Repository showing "excluded" on Hacktoberfest offical website

After I made a commit on a WebDev Level-1 project, the status for this repository is showing as "excluded" on the official Hacktoberfest 24 website. Wasted all my time contributing in this repo

Thank you for bringing this to our attention, and I’m really sorry to hear about the frustration you’re experiencing. We understand how disheartening it can be to put in the effort and not see your contribution counted, and we truly appreciate your hard work on the WebDev Level-1 project.
We’re currently investigating why the repository was marked as excluded on the official Hacktoberfest 24 website, and we’ve already reached out to the Hacktoberfest organizers to resolve the issue. We're doing everything we can to get this sorted out as quickly as possible.
In the meantime, please don’t feel that your contribution was wasted—it’s valuable to us and the community. We’ll keep you updated on the situation, and we hope to have a solution soon.
Thanks for your understanding and patience.


Create Kashyap.md

added profile

Closing as two weeks with no response; feel free to reopen when you sign the CLA.


WIP - DON'T MERGE - Introduce Maven based build

This PR aims to introduce Maven based build for Chirper. This change is required as part of the O'Reilly book "Deploying Reactive Microservices". The subsequent PR after this change will attempt to introduce native Maven plugin to deploy Lagom based services into ConductR.
Structure of Changes
The pom.xml files for this build are retrofitted into Chirper from the Lagom 1.3.2 Maven project generator.
The structure of the change is done based on the following Online Auction PR:
https://github.com/lagom/online-auction-java/pull/78
Removal of SBT build files
Chirper example includes the front-end module which is essentially a Play app. Due to the different structure between Maven based vs SBT based Play app, it is not possible for SBT and Maven build to co-exist. Because of this the SBT related files are now removed.
Travis Build
Travis build has now been converted to use Maven with this PR.
The assembly/conductr-bundle.xml are introduced to the modules which are meant to be packaged as an application. This is required since Travis will invoke maven install prior executing the maven tasks defined in the .travis.yml - this is described in https://docs.travis-ci.com/user/languages/java/#Dependency-Management. Without the presence of assembly/conductr-bundle.xml, the maven install task will fail causing Travis build to fail.
The generated .zip file configured by assembly/conductr-bundle.xml only has lib directory which contains all the dependent jar files. It is expected that native ConductR Maven plugin to generate the required directories, bundle.conf, and executable file.
Lagom based service changes
The Lagom based service can be converted with minimal change.
front-end module changes
The largest change is done on the front-end module.
The controller, assets, and views are moved to match the directory structure expected for Maven-based Play project. The routes are now described with Routes.java.
Compilation of the .jsx files requires node and npm which are installed through the frontend-maven-plugin:
https://github.com/eirslett/frontend-maven-plugin
The frontend-maven-plugin provides the means to invoke npm to install the react-tools which contains the jsx executable to compile the .jsx files. The resulting .js files are then served via Assets controller declared in Routes.java.
The use of the frontend-maven-plugin also means that hot reloading also works with .jsx file changes. The compilation of the .jsx file is done at the generate-sources Maven phase, and the .jsx files will be recompiled with every hot reload.
Note on react-tools npm module
The react-tools is deprecated in lieu of the babeljs compiler: http://babeljs.io/
The babeljs compiler is able to compile many variants of Javascripts (i.e. ES5, ES6, JSX) into browser compatible Javascript.
babeljs itself pulls quite a large number of dependencies, causing the npm install and .jsx compilation step to take a significantly longer time compared to the use of react-tools.
Because of this reason the react-tools is chosen as it doen't impede the time it takes for the hot reload to complete.

I have performed a manual test on the Chirper frontend using a browser and everything seems to be in working order.
Scenario: Frontend Usage

Start all service using mvn lagom:runAll.
Create a user called steve.
Login as steve
Create a chirp from steve: i am steve.
Logout
Create a user called jim.
Login as jim.
Add steve as a friend.
The chirp i am steve is visible for jim.
Logout
Login as steve
Create a chirp from steve: i like donuts.
Logout
Login as jim.
The chirp i like donuts is visible for jim.

Throughout the steps above, the page looks correct (i.e. CSS is displayed correctly) and operations are working as expected.
Scenario: Hot reload JSX
Modify the main.jsx to add an <h1> element randomly somewhere on the login page, and it's displayed as expected. When changes are reverted, the page reflects this change accordingly.
The JSX compilation steps adds 2-3 seconds to the hot reloading time.
Scenario: Hot reload CSS
Modify the CSS to change the login page banner background to red, and it's working as expected. When changes are reverted, the page reflects this change accordingly.

Due to the different structure between Maven based vs SBT based Play app, it is not possible for SBT and Maven build to co-exist. Because of this the SBT related files are now removed.

@fsat were you thinking this would merge to master and replace the SBT version entirely, or remain on a separate branch?
This repo is still an Activator template, and I know Activator is deprecated, but there might still be some people trying to use it that way... plus the name of the repo would be misleading. If we want to have a pure-Maven version, maybe we should push it to a new, separate repo?

@jroper and @huntc - I think @TimMoore raised a point about Chirper being part of Activator despite Activator being deprecated, and I'd like to confirm if you think it's better for this PR lives in a separate repo.
Perhaps another alternative is to have a separate branch for maven along side master within lagom/activator-lagom-java-chirper
I also have a question about which bundles we should publish (generated by sbt vs maven) given the bundles will have the same name.
Can I ask for your input in this matter please?

Marked as wip - waiting for the answer to https://github.com/lagom/activator-lagom-java-chirper/pull/52#issuecomment-297638682

@fsat I think there's a trick to not discard sbt entirely. I learnt from https://github.com/lagom/online-auction-java/pull/78 that disabling PlayLayoutPlugin in the build.sbt definition of a Play project will make the project fallback to maven-friendly folder structure. See https://github.com/lagom/online-auction-java/pull/78/files#diff-fdc3abdfd754eeb24090dbd90aeec2ceR167
Maybe with a similar trick we can have mvn and sbt coexisting?
But I'd like @jroper's comment on that, I'm probably missing something.  I'm not sure what are the build requirements for the actual web site (JS compilation, webjars?, etc...) and sbt plugins doing that might not be ready to operate when disabling PlayLayoutPlugin.

@ignasi35 - thanks for telling me about PlayLayoutPlugin. I'll try this out and put the findings here.
The only thing that's special with Chirper is the compilation of .jsx files which is supported by sbt-web on the sbt side of the build, and while on the maven side I had to use the frontend-maven-plugin which supports node + npm so I can get node-based jsx compiler installed via `npm.

To answer the questions:

We should remove all mentions of Activator from this repo, including changing its name.
What do you mean which bundles we should publish? A user is either going to use maven or sbt, not both, so it doesn't matter if they both produce bundles with the same name.  And even if they do, the bundles should essentially be equivalent, drop in replacements for each other.

@jroper - I've reinstated the SBT support by disabling the PlayLayoutPlugin as both @ignasi35 and yourself suggested.
I kept the assets and public web files on src/main/resources/assets and src/main/resources/public and pointed sbt-web to these directories instead. I know this is different than what you've suggested, but hopefully this is ok.
I've tested the hot reload for both SBT and Maven, and that seems to work with this setup.
I've also reinstated the sbt test on Travis similar to what you suggested, and that seemed to work ok.
I'd like to continue with our discussion as well.

We should remove all mentions of Activator from this repo, including changing its name.

This is removed from the pom.xml. Perhaps we can change the name after Activator is removed? I don't think I have permission to change the project.
There's no mention of Activator left in these files:
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "pom.xml" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.sbt" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.html" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.css" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.js*" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.java" | grep -v "target/" | xargs grep -i "activator"
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.conf" | grep -v "target/" | xargs grep -i "activator"

The only thing left is the activator.properties. I think perhaps we should wait until Activator is removed before removing this file.
Felixs-MBP-2:activator-lagom-java-chirper felixsatyaputra$ find . -iname "*.prop*" | grep -v "target/" | xargs grep -i "activator"
./activator.properties:authorLogo=http://www.lightbend.com/assets/images/activator/logo-typesafe-activator.png
./activator.properties:sourceLink=https://github.com/lagom/activator-lagom-java-chirper/tree/master

What do you mean which bundles we should publish? A user is either going to use maven or sbt, not both, so it doesn't matter if they both produce bundles with the same name. And even if they do use both, the bundles should essentially be equivalent, drop in replacements for each other.

I agree, they should be equivalent. I guess my question was about whether we prefer sbt vs maven bundle to be published.
But then again after I checked, Chirper bundles are not published to Bintray anyway, so I guess we shouldn't worry about bundles from which build are to be published.

Sounds good, @jroper and @fsat. I think we can take care of the repo rename and deletion of activator.properties after the deactivation.
Agree that we shouldn't be publishing any artifacts to bintray, so I don't think it's an issue.


Navbar Brand and Home page link

Describe the bug
On clicking the 'Home' link in the navbar after clicking any other link like the 'Projects' link. The image on the home page is touching the navbar.
To Reproduce
Steps to reproduce the behavior:

Open index.html and click on 'Projects' in the navbar
Now click on either 'Home' or 'Awesome Javascript' in the navbar
We can see an error that the home page image is touching the navbar which is not good-looking.

Expected behavior
On clicking on either the 'Home' or 'Awesome Javascript' link =, the page should be the same as it is while opening the first-time index.html.
Logs

Error: Home page image is touching navbar.

On clicking either the 'Home' or Awesome Javascript' link, it should be like the above image.
Environment (please complete the following information):

OS: Windows
Browser: Chrome

Additional context
I am a participant in GSSoC'21. I would like to work on this issue. May I work on it under GSSoC'21?

@urvashi-code1255 Kindly assign me this issue.

Everyone UI of the website has been updated. Feel free to find bug there and create a new issue for that will be happy to assign.
As this is old I'm closing this issue for now


Doom startup errors prevent loading doom defaults.

What did you expect to happen?
When I open Emacs.app I expect it to load the dashboard and all the doom defaults such as the space key as the leader etc.
What actually happened?
Some doom defaults are not loaded (the space key being the most noticable one) and gives me this huge error message instead of the doom dashboard.
No keybindings involving space work.
Debugger entered--Lisp error: (file-error "Cannot bind server socket" "Address already in use")
  make-network-process(:name "websocket server on port 35903" :server t :family ipv4 :noquery t :filter websocket-server-filter :log websocket-server-accept :filter-multibyte nil :plist (:host local :on-open org-roam-ui--ws-on-open :on-message org-roam-ui--ws-on-message :on-close org-roam-ui--ws-on-close) :host local :service 35903)
  websocket-server(35903 :host local :on-open org-roam-ui--ws-on-open :on-message org-roam-ui--ws-on-message :on-close org-roam-ui--ws-on-close)
  org-roam-ui-mode(1)
  custom-set-minor-mode(org-roam-ui-mode t)
  custom-theme-recalc-variable(org-roam-ui-mode)
  enable-theme(user)
  enable-theme(doom-nova)
  #f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d4840af5>)) #<bytecode 0x424ad7bf>)(doom-nova t nil)
  apply(#f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d48423f1>)) #<bytecode 0x424ad7bf>) (doom-nova t nil))
  #f(advice-wrapper :after #f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d4845c81>)) #<bytecode 0x424ad7bf>) solaire-mode--prepare-for-theme-a)(doom-nova t nil)
  funcall(#f(advice-wrapper :after #f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d4848f45>)) #<bytecode 0x424ad7bf>) solaire-mode--prepare-for-theme-a) doom-nova t nil)
  (prog1 (funcall fn theme no-confirm no-enable) (if (and (not no-enable) (custom-theme-enabled-p theme)) (progn (setq doom-theme theme) (put 'doom-theme 'previous-themes (or last-themes 'none)) (doom-run-hooks 'doom-load-theme-hook))))
  (let ((last-themes (copy-sequence custom-enabled-themes))) (mapc #'disable-theme custom-enabled-themes) (prog1 (funcall fn theme no-confirm no-enable) (if (and (not no-enable) (custom-theme-enabled-p theme)) (progn (setq doom-theme theme) (put 'doom-theme 'previous-themes (or last-themes 'none)) (doom-run-hooks 'doom-load-theme-hook)))))
  (progn (let ((last-themes (copy-sequence custom-enabled-themes))) (mapc #'disable-theme custom-enabled-themes) (prog1 (funcall fn theme no-confirm no-enable) (if (and (not no-enable) (custom-theme-enabled-p theme)) (progn (setq doom-theme theme) (put 'doom-theme 'previous-themes (or last-themes 'none)) (doom-run-hooks 'doom-load-theme-hook))))))
  (unwind-protect (progn (let ((last-themes (copy-sequence custom-enabled-themes))) (mapc #'disable-theme custom-enabled-themes) (prog1 (funcall fn theme no-confirm no-enable) (if (and (not no-enable) (custom-theme-enabled-p theme)) (progn (setq doom-theme theme) (put 'doom-theme 'previous-themes (or last-themes ...)) (doom-run-hooks 'doom-load-theme-hook)))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((last-themes (copy-sequence custom-enabled-themes))) (mapc #'disable-theme custom-enabled-themes) (prog1 (funcall fn theme no-confirm no-enable) (if (and (not no-enable) (custom-theme-enabled-p theme)) (progn (setq doom-theme theme) (put ... ... ...) (doom-run-hooks ...)))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((last-themes (copy-sequence custom-enabled-themes))) (mapc #'disable-theme custom-enabled-themes) (prog1 (funcall fn theme no-confirm no-enable) (if (and ... ...) (progn ... ... ...))))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  doom--load-theme-a(#f(advice-wrapper :after #f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d4855f11>)) #<bytecode 0x424ad7bf>) solaire-mode--prepare-for-theme-a) doom-nova t)
  apply(doom--load-theme-a #f(advice-wrapper :after #f(compiled-function (theme &optional no-confirm no-enable) "Load Custom theme named THEME from its file and possibly enable it.\nThe theme file is named THEME-theme.el, in one of the directories\nspecified by `custom-theme-load-path'.\n\nIf the theme is not considered safe by `custom-safe-themes',\nprompt the user for confirmation before loading it.  But if\noptional arg NO-CONFIRM is non-nil, load the theme without\nprompting.\n\nNormally, this function also enables THEME.  If optional arg\nNO-ENABLE is non-nil, load the theme but don't enable it, unless\nthe theme was already enabled.\n\nNote that enabling THEME does not disable any other\nalready-enabled themes.  If THEME is enabled, it has the highest\nprecedence (after `user') among enabled themes.  To disable other\nthemes, use `disable-theme'.\n\nThis function is normally called through Customize when setting\n`custom-enabled-themes'.  If used directly in your init file, it\nshould be called with a non-nil NO-CONFIRM argument, or after\n`custom-safe-themes' has been loaded.\n\nReturn t if THEME was successfully loaded, nil otherwise." (interactive #f(compiled-function () #<bytecode 0x1fe1d48582a5>)) #<bytecode 0x424ad7bf>) solaire-mode--prepare-for-theme-a) (doom-nova t))
  load-theme(doom-nova t)
  (progn (load-theme doom-theme t))
  (if (and doom-theme (not (custom-theme-enabled-p doom-theme))) (progn (load-theme doom-theme t)))
  doom-init-theme-h()
  funcall(doom-init-theme-h)
  (condition-case e (funcall hook) ((debug error) (signal 'doom-hook-error (list hook e))))
  doom-run-hook(doom-init-theme-h)
  run-hook-wrapped(doom-run-hook doom-init-theme-h)
  (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car (cdr e))) (symbol-name (car (cdr e))) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e)))))
  (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car ...)) (symbol-name (car ...)) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp ...) (symbol-name ...) (car ...)) (car (cdr ...)))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- hooks)) (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if ... ... ...) (car ...))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  doom-run-hooks(after-init-hook delayed-warnings-hook)
  apply(doom-run-hooks (after-init-hook delayed-warnings-hook))
  run-hooks(after-init-hook delayed-warnings-hook)
  command-line()
  normal-top-level()

Here are the error messages without my private config (sandbox C-c C-p)
Debugger entered--Lisp error: (file-error "Searching for program" "emacs" "no such file")
  signal(file-error ("Searching for program" "emacs" "no such file"))
  xclip-mode(1)
  (condition-case err (xclip-mode 1) ((debug error) (message "%s" err) nil))
  (and (require 'xclip nil t) (condition-case err (xclip-mode 1) ((debug error) (message "%s" err) nil)))
  (if nil (and (require 'clipetty nil t) (global-clipetty-mode 1)) (and (require 'xclip nil t) (condition-case err (xclip-mode 1) ((debug error) (message "%s" err) nil))))
  doom-init-clipboard-in-tty-emacs-h()
  funcall(doom-init-clipboard-in-tty-emacs-h)
  (condition-case e (funcall hook) ((debug error) (signal 'doom-hook-error (list hook e))))
  doom-run-hook(doom-init-clipboard-in-tty-emacs-h)
  run-hook-wrapped(doom-run-hook doom-init-clipboard-in-tty-emacs-h)
  (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car (cdr e))) (symbol-name (car (cdr e))) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e)))))
  (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car ...)) (symbol-name (car ...)) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp ...) (symbol-name ...) (car ...)) (car (cdr ...)))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- hooks)) (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if ... ... ...) (car ...))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  doom-run-hooks(after-init-hook delayed-warnings-hook emacs-startup-hook tty-setup-hook window-setup-hook)
  (let ((inhibit-startup-hooks nil)) (doom-run-hooks 'after-init-hook 'delayed-warnings-hook 'emacs-startup-hook 'tty-setup-hook 'window-setup-hook))
  doom-run-all-startup-hooks-h()
  eval-buffer(#<buffer  *load*> nil "/var/folders/b5/qqjwmwfn7lg8h9nkfgrmw7_00000gp/T/d..." nil t)  ; Reading at buffer position 25215
  load-with-code-conversion("/var/folders/b5/qqjwmwfn7lg8h9nkfgrmw7_00000gp/T/d..." "/var/folders/b5/qqjwmwfn7lg8h9nkfgrmw7_00000gp/T/d..." nil t)
  load("/var/folders/b5/qqjwmwfn7lg8h9nkfgrmw7_00000gp/T/d..." nil t)
  command-line-1(("-l" "/var/folders/b5/qqjwmwfn7lg8h9nkfgrmw7_00000gp/T/d..."))
  command-line()
  normal-top-level()

Heres my Doom private config
Describe your attempts to resolve the issue
I have re-installed both emacs-mac and doom twice but it still results in the same issue.
Steps to reproduce

Uninstall emacs, doom.
Reinstall emacs-mac (homebrew railwaycat formula)
Install doom
Open Emacs

System Information
https://pastebin.com/N0mzzK7a

That error is actually caused by org-roam-ui. It has existed for quite some time and never stopped my entire config from loading. I still took the time to fix it and now almost everything works except my spell checker.
In init.el
(spell +aspell)

(I already have aspell via brew)
And in config.el
(after! ispell
  (setq ispell-program-name "/usr/local/bin/aspell"
        ispell-dictionary "en_US"
        ispell-extra-args '("--sug-mode=ultra")))

Everything used to work but now I'm unable to add to my dictionary using +spell/add-word
Thanks!

That error is actually caused by org-roam-ui.

Same difference then, there's no issue with doom so this can be closed.

Everything used to work but now I'm unable to add to my dictionary using +spell/add-word

Raise another ticket then.


Load Filter Criteria for Method Data

Create views that load the data for each dropdown in the module

[ ] Parameter Code

[ ] Method Code

[ ] Subsitute Data Code

[ ] Bypass Approach Code

create view camdecmpswks.vw_methods_master_data_relationships


Gradle 6.5‘ s

......
<=============> 100% CONFIGURING [2h 51m 43s]

Resolve dependencies of :ImTenon:releaseCompileClasspath > Resolve dependencies of :ImTenon:releaseRuntimeClasspath
.....

What's going on and what can I do?
And there are not any useful hints here.
Another more, What is wrong when "SSL peer shut down incorrectly" happened? Please give out more useful details. thx a lot.
This things stuck me wast me  few days.


POST, PUT fail with 409 confilct; PATCH: 405 method not allowed

NOTE: GitHub issues are reserved for bug reports only.
Please read the CONTRIBUTING.md guidelines to learn on which channels you can
seek for help and ask general questions:
https://github.com/Kong/kong/blob/master/CONTRIBUTING.md#where-to-seek-for-help
Summary
Kong fails when attempting to update APIs regardless of what HTTP very is utilized.  The only solution is to stop traffic to that API, delete it, and it back in, and re-enable traffic.
The documentation states that you can use name or id with PATCH verb :
https://getkong.org/docs/0.12.x/admin-api/#update-api
Steps To Reproduce

Add an API using POST call to 8001 Admin port.
Attempt to update by changing API settings and reposting: 409 error.
Attempt to use PUT: 409 error.  Attempt to use PATCH: 405 error: method not allowed

Additional Details & Logs

Kong version 0.12.1

Operating System: CENTOS 7

Let's declare this as working as designed, and simply blame incorrect, missing and wholly misleading descriptions in the official documentation for 0.12.

PATCH should work to either to /apis/<id> or /apis/<named>


Higher than expected random write IOPS

I created a script that tests the random read/write performance of SSDs, after after sequential and random preconditioning. The test basically runs these functions in order:
Create_NTFS_Volume
Sequential_Precondition
Random_Precondition
Random_Workload
Sequential_Precondition
Sequential_Workload
Process_XML
Delete_NTFS_Volume
The sequential precondtion flags:
$sp = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b128k -d9000 -o128 -t1 -Suw -w100 -L  -c$FileSize $DataFile" -PassThru
$sp.WaitForExit()
The random precondition flags:
$rp = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b4k -d9000 -o32 -t4 -Suw -r -w100 -L  -c$FileSize $DataFile" -PassThru
 $rp.WaitForExit()

The random workload flags:
$p = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b$bs -d$Time -o$qdepth -t$thread -Suw -r -Rxml -w$wp -L -c$FileSize $DataFile" -RedirectStandardOutput $stdOutLog -RedirectStandardError $stdErrLog -PassThru
$p.WaitForExit()
However I am getting way higher than expected random write IOPS, way higher than the SSD is capable of. Is there something I am missing in the switches?

Not specifying noncached IO would be the usual error, and you're doing that correctly with -Suw.
A possible problem is defeating your preconditioning by recreating the loadfile on each step. On deletion, unless you've disabled delete notification in the filesystem, this will issue a TRIM/UNMAP to the SSD which will clean out its LBA/PBA mapping. Don't do that.
Other than that, if you could mention the result you're getting and the device, I could help reason about whether the result really is unreasonable. The specific numbers might suggest other causes.

From: hazemawadalla<EMAIL_ADDRESS>Sent: Tuesday, February 11, 2020 3:14 PM
To: microsoft/diskspd<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: [microsoft/diskspd] Higher than expected random write IOPS (#129)
I created a script that tests the random read/write performance of SSDs, after after sequential and random preconditioning. The test basically runs these functions in order:
Create_NTFS_Volume
Sequential_Precondition
Random_Precondition
Random_Workload
Sequential_Precondition
Sequential_Workload
Process_XML
Delete_NTFS_Volume
The sequential precondtion flags:
$sp = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b128k -d9000 -o128 -t1 -Suw -w100 -L -c$FileSize $DataFile" -PassThru
$sp.WaitForExit()
The random precondition flags:
$rp = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b4k -d9000 -o32 -t4 -Suw -r -w100 -L  -c$FileSize $DataFile" -PassThru
$rp.WaitForExit()
The random workload flags:
$p = Start-Process -NoNewWindow -FilePath "$Diskspdpath" -ArgumentList "-b$bs -d$Time -o$qdepth -t$thread -Suw -r -Rxml -w$wp -L -c$FileSize $DataFile" -RedirectStandardOutput $stdOutLog -RedirectStandardError $stdErrLog -PassThru
$p.WaitForExit()
However I am getting way higher than expected random write IOPS. Is there something I am missing in the switches?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/diskspd/issues/129?email_source=notifications&email_token=ACCWSKKPV5XB4JXNZZJAQRTRCMWNLA5CNFSM4KTML7GKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IMXVCZQ, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACCWSKKFKH2MFOJYJW74XKDRCMWNLANCNFSM4KTML7GA.

Hi Don,
I have tested not deleting the file on each step after you mentioned it, and I'm still getting high random write numbers. I work for a drive manufacturer and we are using fio for QoS testing. I will attach the fio results (which the drive was speced at), compared to the latest run of my script.
The 4K random write QD 32 with fio is 25400, compared to 81841 IOPS with diskspd. Can you help me understand the discrepancy?
Kingston_SEDC500R7680_7680G_J2.9 2-sample comparison_fiov317.xlsx
diskspdresults.zip

I’ll take a look at what you attached shortly.
However: as a device maker I’d expect you have access to device statistics to confirm whether the number of operations or bytes transferred matches (roughly) what DISKSPD reports.
What is your expectation for the device capability?
Get Outlook for iOShttps://aka.ms/o0ukef

From: hazemawadalla<EMAIL_ADDRESS>Sent: Friday, February 21, 2020 10:16:38 AM
To: microsoft/diskspd<EMAIL_ADDRESS>Cc: Dan Lovinger<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [microsoft/diskspd] Higher than expected random write IOPS (#129)
Hi Don,
I have tested not deleting the file on each step after you mentioned it, and I'm still getting high random write numbers. I work for a drive manufacturer and we are using fio for QoS testing. I will attach the fio results (which the drive was speced at), compared to the latest run of my script.
The 4K random write QD 32 with fio is 25400, compared to 81841 IOPS with diskspd. Can you help me understand the discrepancy?
Kingston_SEDC500R7680_7680G_J2.9 2-sample comparison_fiov317.xlsxhttps://github.com/microsoft/diskspd/files/4237432/Kingston_SEDC500R7680_7680G_J2.9.2-sample.comparison_fiov317.xlsx
diskspdresults.ziphttps://github.com/microsoft/diskspd/files/4237436/diskspdresults.zip
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/diskspd/issues/129?email_source=notifications&email_token=ACCWSKJ2WJTZ5R7DLXTYTSTREALANA5CNFSM4KTML7GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTTEUY#issuecomment-589771347, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACCWSKMR7274SUKCGULG65DREALANANCNFSM4KTML7GA.

Hey Dan,
We designed the drive to be a read intensive SSD capable of 99,000 Read/26000 Write IOPS, @ 4k QD32 1 thread.

I replied with this by the email connector on the 21st, but it appears not to have made it.
I pulled up all of the results (thanks for including the full sweep in XML form!).
Focusing on diskspeedrandom_1_100_4k_32, it looks OK: single thread QD32 random 4KB/4KB 100% unbuffered writethrough write. Load is to a ~7.5TB file which I assume is your device ~fully allocated.
The one thing that occurs to me is that you're using the default non-zero but constant fill pattern for the write buffer source (bytes are 0 - 1 - 2 - 3 .... - 255, repeating). Does your device have intelligence to detect constant buffer fill and optimize/dedup the operations? I'm not sure what FIO's default is, but if it is random or at least a pattern your device may not recognize in the IO path, that may be the difference.
Iff your device does have this intelligence, try this to overcome it: DISKSPD supports creating a random write source buffer with the -Z switch. A size on the order of a few 10's of MiB is usually plenty. In DISKSPD 2.0.17a the write source will be chosen at 4-byte aligned offsets within the buffer, 512-byte aligned in the most recent release to avoid processor architectural effects of sub-cacheline aligned buffers (several % overhead in certain cases).
Last, if you can get the host interface statistics that should strongly narrow down where the disconnect is.

I also ran into the same problem, the results are way higher than what iometer and fio shows.
diskspd.exe -b4K -t16 -r -o16 -d39 -Sh E:\iobw.tst

@hazemawadalla have likely root caused his issue offline. It has to do with differences in SSD preconditioning methodology between the two specific devices/platforms he was making his comparative runs on. It will take about a week to make the runs to confirm but I suspect that will close his specific issue.
If you open a separate issue, we can see about root causing yours.

@dl2n Thx, I've posted a new issue.

There is not an issue with diskspd per se, but a limitation. SSD preconditioning by capacity is an essential enhancement to diskspd. It is difficult to know by "how much" your SSD performs at steady state vs FOB, especially with larger SSD capacities. With fio (and even other tools, like iometer) you can specify --loops=$ which ensures all LBAs are written to more than once. The SNIA spec recommends 2x capacity, and diskspd has no graceful way of doing that.
There is a workaround, but it is very tedious coding. Keep track of the total bytes written using windows perfmon and stop the random/sequential precondition process once you hit 2x capacity. Another easier workaround, is if you know your device's approximate datarate when performing random or sequential workloads, you can calculate the approximate amount of time it would take to precondition the device.


What to do with TaggedUsers?

Describe your issue

src/controllers/taggedusers/TaggedUser.{cpp,hpp}, src/controllers/taggedusers/TaggedUserModel.{cpp,hpp} and some remnants (more links: 3/5 4/5 5/5) have been in Chatterino's code since (at least) 2018 if not before. None of this code is used AND even when I uncommented it. It's broken because src/controllers/taggedusers/TaggedUserController.{cpp,hpp} files are missing. What should we do with this code?
@fourtf, since you wrote the code, I believe you should get the final say in this.
OS and Chatterino Version

current master cc9d44a0bbb57d2e52093dac37d4506c376170f1

delete it?

Closed by #3208