hook manager mutex and global variable enhancement

This patch aims to fix issue [MESOS-9337]

add missing mutex acquisition for the potential race
change the global mutex and map variable to POD type

Signed-off-by: Dong Zhu<EMAIL_ADDRESS>

@bmahler Thanks for your info, I will update the patch on my own and submit later.

@bmahler I have re-submitted the changes. Could you please review it again ?

@asekretenko these are good questions to be asking, as I also felt this looked a bit strange. I suspect it's just modeled in the same way that ModuleManager is written which has quite a bit more to it but is also an all static class with similar static non-POD issues. For unload, it looks like it is used for testing so I'm not sure how easily it could be removed. At this point I haven't read through all the module related code enough to be contemplating changing the design, so I can't write a useful TODO or ticket (but please feel free to if you're interested in pitching in to clean this up!)
Since @bluezd is a new contributor and was interested in hooks/modules, I looked for some small tickets for him to get some experience with the contribution process, and so this patch is just focusing on the trivial work of removing static non-PODs (which we ban) and fixing some unsafe locking in this code.


map.get_waypoint (project_to_road=False) does not result in waypoints at expected locations

Carla 0.9.11 (package install using apt-get)
MacOS 11.2
Custom Map

API Call: map.get_waypoint(carla.Location(x, y, z), project_to_road=False, lane_type=carla.LaneType.Any)
Issue: Attempting to generate waypoints off of lane center, at location (x, y, z) results in waypoints that are sometimes at the lane center, sometimes at location point, and sometimes somewhere in between. I have removed all None type waypoints from the resulting loop. LaneType has no effect on results.
Number of input locations: 2203
Number of Waypoints (excluding NoneType): 1124

API Call: map.get_waypoint(carla.Location(x, y, z), project_to_road=True, lane_type=carla.LaneType.Any)
Result: Works as intended, waypoints are generated on lane center, nearest to input location.
Number of input locations: 2203
Number of Waypoints (excluding NoneType): 2203

API Call: map.generate_waypoints(10)
Result: Works as intended, waypoints are generated on lane center, equally spaced.
Number of Waypoints: 444

Image below shows plots of
blue: map.generate_waypoints(10)
red: map.get_waypoint(carla.Location(x, y, z), project_to_road=False, lane_type=carla.LaneType.Any)
green: carla.Location(x, y, z)

Hi @amzoo, I think there is a misconception on how the get_waypointfunction works. Waypoints are always objects with the position in the center of the lane.
When you call the function map.get_waypoint(carla.Location(x, y, z), project_to_road=False, lane_type=carla.LaneType.Any) you are getting a waypoint corresponding to the lane (x, y, z) is in. If (x, y, z) is outside the lane the function will return None.
Finally, generate_waypoints() creates waypoints only in lanes of type driving, while your function may return a waypoint of any type because of the carla.LaneType.Any. So probably you are getting waypoints of sidewalks or other lane types. I suggest you try lane_type=carla.LaneType.Driving to get all the waypoints of the same type.

Understood! I realize now that those rogue waypoints I was seeing are actually on the center of the sidewalk. I misunderstood that the (x,y,z) was only being used to choose a specific lane.
Is there anyway to generate a waypoint not on the center of a lane? My use case is in the racing world, in which vehicles may travel anywhere on the drivable lane, and not stick to center.

@amzoo Waypoints are representations of the road so their transform only points towards the center of the lane. You can however obtain displaced locations in the road using the transform.get_right_vector() to obtain a location laterally displaced from the center of the lane.

I can use carla.Locations fine, but I was hoping there were carla.Waypoints that I could move from the center of the lane in order to use the methods in the Waypoint class

@amzoo, waypoints represent the center of the lane only. You need to need to do the displacement yourself


Prepare sodiumoxide for release

Fixed, thanks.

Update libsodium-sys/Cargo.toml too

Pull Request Test Coverage Report for Build 1342

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.001%) to 95.234%

Totals

Change from base Build 1334:
0.001%

Covered Lines:
3117

Relevant Lines:
3273

💛  - Coveralls

Done.

I’ll try to make the release on Tuesday/Wednesday.

@dnaq, great thanks a lot.
@kpp, obviously thanks to you too!

bors r+

@tasn can you investigate why the tests fail on Windows?

Hey, I unfortunately can't. I'm not a rust expert, nor do I know this library very well... I have no idea where to start.
All I changed in this PR is the version info and the changelog, so it's unlikely it introduced it.

Np. @kpp could you take a look at it. I don't have access to a windows system to reproduce.

I don't know why it's not working...

Seems a bit weird since all tests have passed on master. Might be some issue with either the appveyor conf or appveyor itself. I would like to see the tests run on a windows machine before merging though.

So, I did some experimentation with the appveyor configuration here: #429 and saw that if we switched to using the prebuilt libraries on 64 bit msvc the tests passed. However there should be no reason for the tests failing when not using prebuilt libraries as well, so something weird is going on.
Also when looking through the tests it seems strange that 32-bit msvc and mingw used the prebuilt libraries, and so did 64-bit mingw, but 64-bit msvc wanted to use vcpkg. Seems like those tests have gone a bit stale. We should of course run all tests on all variants. So if someone wants to take care of that I would be glad. Or should we remove the vcpkg support completely since we haven't tested it for most build variants?, that should simplify the build.rs as well.
@kpp: what do you think?
@tasn: would you update the pull request with the changes in #429?

Oh, and @kpp, maybe it would be easier to switch to azure pipelines in the future so we can run all tests there instead of supporting both travis and appveyor?

Done.

@dnaq yeah that's the plan just that none of us have sat to do it

Or should we remove the vcpkg support completely since we haven't tested it for most build variants?, that should simplify the build.rs as well.

I think we should remove the vcpkg support. IIRC it came out of box from some PR. But unfortunately it is broken, so we'd better get rid of it until somebody requests for its support.

I added some more changes to #429, we can either merge that one or this one if you add the changes to it. Would feel more fair to merge this one and close #429

I updated this PR with your changes.

Would you mind rebasing to clean up the history a bit? One commit for the vcpkg changes and one for the change log/version number bump?

Done.

I still see four different commits, could you squash them into two?

Oh. Maybe lets merge a separate PR with vcpkg removal and then rebase this PR onto latest master?

Done.

@tasn can you change the title and commit message of the vcpkg commit. @kpp I'm OK with both changes being in the same pull request as long as it's two different commits.

@dnaq, I can't guess what you have in mind... Could you please just do your changes in a branch (like the draft PR you did) and let me know once ready so I can update this PR?
Thanks!

Pushed, thanks!

bors r+


adding print button

This uses PrintJS to add a simple print button to each page to print to PDF. It also makes the "download" button a dropdown with a symbol so that people can choose to download the ipynb file or the PDF.
Still needs a bit of tweaking...
Todo

[ ] Figure out how to get the right page width to trigger on the PDF printing punting this to #281

Merging this because a class at Berkeley is having an issue with printing pages and this would help them!


error: package com.himelbrand.forward calls does not exist

when i'm trying to run my project, I'm getting this error.

@ashryOlarm  did you fins solution for that?


Use of informative git prompt

I suggest to use zsh-git-prompt plugin to show git status in Spaceship prompt.

We've made OMZ as dependency since v2.0.0 and we're regretting about it. There is a problem related to dependence management in shell scripting: that's not so easy to keep dependencies in sync with primary project. For now our goal is to make spaceship as standalone as possible, without any dependency.
By now, we don't plan to include this plugins. Anyway, thanks for link. We will keep it in mind.


Identity should not be added if it has error in admin dashboard

Version information

OS: docker Ubuntu 20.04
Branch: development
commit: c3fe796303c612a8c035009ddbc1e8130e67a672
Browser: Version 84.0.4147.125 (Official Build) (64-bit)
explorer: testnet

Steps to reproduce
1- Go to settings in admin dashboard.
2- Add identity with wrong details.

3- Then close it.
4- I found that the the identity is added, although it has a problem.

Verified
commit : 881f1904e00ba187ecf254cd8ba4857949767d38
branch: development


PublishDiagnostics Notification and File Deletions

if the client deletes a file is the server then required to send a empty PublishDiagnostics Notification to clear the issues? at least vscode does not seem to clear the issues for deleted resources.

sample from vscode after deleting c.mydsl (server receives did change watched files and removed the file from its structure but did not send and empty notification.
=> Bug in vscode or should the server have sent an empty notification for the deleted file

According to the specification...

Diagnostics are “owned” by the server so it is the server’s responsibility to clear them if necessary.

This has been brought up in other VS Code extensions in the past. See Microsoft/vscode-docker#237 and ionide/ionide-vscode-fsharp#730.
Although it does seem to me like clients should just wipe these things out themselves. Especially in the case where the client initiated the deletion (and you didn't popup an external terminal and called rm Errors.java yourself externally).
What does everyone else think?

no i used the normal delete file context menu in vscode.
@mickaelistria how does lsp4e treat this situation? at least in eclipse you cannot show resource markers for files that are not there anymore

For LSP4E and Eclipse IDE in general, it's just impossible by design of the marker API to create or keep a marker for a file that doesn't exist, so removing a file does remove automatically all markers.
This behavior is deep in the roots of markers in Eclipse Platform, and even if server sends different signals, it's not something we can implement a-la VSCode. The closest thing for LSP4E to replicate a similar behavior would be to placed diagnostics for deleted files as markers on the closest parent folder, but I don't think it'd be a good thing as no-one complains about it in LSP4E while 3 people already complained about VSCode appraoch.
I believe the issue you report is more specific to VSCode allowing such inconsistent state of markers, not something that's a bug in LSP.

no i mean it the other way round: what happens if in lsp4e the server reports issues for files that do no exist: does it give a framework error in lsp4e

If LSP4E receives diagnostics on files that don't exist, those are just
ignored:
http://git.eclipse.org/c/lsp4e/lsp4e.git/tree/org.eclipse.lsp4e/src/org/eclipse/lsp4e/operations/diagnostics/LSPDiagnosticsToMarkers.java#n66
. I don't think we'll change this behavior in LSP4E unless there are valid
user-stories to suppor it.

To make things consistent in all cases we decided to make the creator of markers responsible for their deletions. Since the marker were created my the server he owns them and needs to wipe them.
Unless I see a compelling case to change this I opt to close this issue. The current design is easy to communicate since it has no exceptions.


Transition Animations

Adds fun transitions to most VC presentations involving series and episodes

Add to listening history as well

Try to add to search

Some of the transitions are a bit funky ... Really try to play with the app on the simulator on multiple phone types to fix some of these bugs. Also I am still getting a crash when going from an EpisodeDetailViewController to aSeriesDetailViewController via the title button.


[Bug]: INP Problem

Expected Behavior
In Google Web Core Vitals

Problem with INP when closes cookie consent
Current Behavior
In Google Web Core Vitals

Problem with INP when closes cookie consent
Steps to reproduce
In Google Web Core Vitals

Problem with INP when closes cookie consent
Proposed fix or additional info.
No response
Version
3.0.1
On which browser do you see the issue?
No response

Unless you provide a demo I won't be able to help you.

@agracia-foticos did you manage to solve this problem?
Google Search Console reports this issue as you described.

@bennetsadyba actually we dont use this module, we prefer a custom implementation of consent


A0-1004 Crash recovery for ordering example

A0-1004

@DamianStraszak - I've added a general warning to the run.sh script, didn't use if because not sure how to properly state the condition...


[SPARK-17244] Catalyst should not pushdown non-deterministic join conditions

What changes were proposed in this pull request?
Given that non-deterministic expressions can be stateful, pushing them down the query plan during the optimization phase can cause incorrect behavior. This patch fixes that issue by explicitly disabling that.
How was this patch tested?
A new test in FilterPushdownSuite that checks catalyst behavior for both deterministic and non-deterministic join conditions.

cc @hvanhovell @gatorsmile

cc @brkyvz who found this bug

Test build #64436 has started for PR 14815 at commit 6728fc3.

Test build #64437 has started for PR 14815 at commit d0b1009.

Test build #64436 has finished for PR 14815 at commit 6728fc3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64436/
Test FAILed.

Merged build finished. Test FAILed.

Test build #64437 has finished for PR 14815 at commit d0b1009.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64437/
Test PASSed.

Merged build finished. Test PASSed.

Thanks @gatorsmile, I address your comments.

Test build #64496 has started for PR 14815 at commit f86f5ce.

Looks pretty good to me!
Could you also add one more test case for testing the following scenario:
only consider pushing down those expressions that precede the first non-deterministic expression in the condition.

I am just afraid the others might break it in the future. Thanks!

That's a good idea. More generally, I think the best way to fix these class of problems is to unify all predicate pushdown logic at a common place.

Test build #64496 has finished for PR 14815 at commit f86f5ce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64496/
Test PASSed.

In 2.0, we combined the PPD rules. Now, only two PPD rules left, PushPredicateThroughJoin and PushDownPredicate. IMO, the major reason why we did not combine them into the same one is because PPD for Join is a little bit special.

Test build #64500 has started for PR 14815 at commit b326cd8.

LGTM pending testing

Test build #64500 has finished for PR 14815 at commit b326cd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64500/
Test PASSed.

LGTM. Merging to master and branch 2.0.


DocsArchive-<IP_ADDRESS>Auto archive content

Auto archive content from https://github.com/OPS-E2E-PPE/docs-archive-test-source

Docs Build status updates of commit 538af38:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error]

Cannot sync git repo to specified commit because branch Release_Archive_master_2020-03-30-03-50-43 has been deleted or has been force pushed: fatal: couldn't find remote ref refs/heads/Release_Archive_master_2020-03-30-03-50-43

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Support drawer position, left or right

I think I've never seen an app with a drawer on the right though?

@mkonicek it'll be useful for RTL languages.

True, OK 👍

yes support.
const DrawerOptions = DrawerNavigator(   { 'Dashboard View': { screen: DashboardScreen } },   { drawerPosition: 'right' }, );


[enet, libsodium, miniupnpc] Lack .pc files

Problem, described
These projects all support pkg-config upstream (in that they have the ability to generate a .pc file), but vcpkg's port of them does not export the .pc files

enet
libsodium
miniupnpc

Solution, proposed
Have the above project ports export the generated .pc files to where such things typically go when vcpkg exports .pc files.

If the ports are switched to their native buildsystem they most likely will automatically generate the .pc files so this should be the task.


feat-request(nominal-typebox): branded regexp fields

Is your feature request related to a problem? Please describe.

Support Typebox' RegExp branded alternative in @coderspirit/nominal-typebox.

Describe the solution you'd like
Same as for brandedString, but with RegExp.
Describe alternatives you've considered
Not applicable
Additional context
Not relevant.

Fixed by #115


New menu with support for voice commands

This commit adds a new menu and support for voice commands.

So... looks good!  Sorry to be a pain, but could you also:
a) squash into a single commit
b) add the 'signoff' stuff as described in https://github.com/openhab/openhab.pebble/blob/master/CONTRIBUTING.md
Otherwise, Kai will be cross. ;-)
Thanks!

Otherwise, Kai will be cross. ;-)

Yeah, yeah, blame me :-P
I don't like putting high bars for contributions either, but this is something that a properly managed open-source project simply has to ask for...

No problem, I've just squashed the commits and added the signoff line.


Implement mask and return_mask option to F.dropout

Refs https://github.com/chainer/chainer/pull/4421#issuecomment-395377917

Jenkins CI test succeeded without errors!

Although these PRs won't break backward compatibility, they change the interface. Just wondering if they need to be backported.

I think it's better to backport these non-breaking changes, because it will make migration to v5  smooth for users.
Maybe it's better to add notes like this to class document?:
Direct access of mask attribute will be deprecated in Chainer v5.  Use  return_mask/mask  option of F.dropout to get/set mask used for dropout.

I see you point regarding the backport. Let's do so.
As for the the documentation you suggest, it could help users but if this wasn't a public API before I don't think we need to document it that way.

As for the the documentation you suggest, it could help users but if this wasn't a public API before I don't think we need to document it that way.

Ok, I agree with it.

Jenkins CI test failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

jenkins, test this please.

jenkins, test this please.

Jenkins CI test (for commit e077d80afe2ea1cdd7e78ab7c712f5c274c27f7f) succeeded without errors!

jenkins, test this please.

Jenkins, test this please.

Jenkins CI test (for commit 826770d43496a3f6ca24fa446f5a16bfd6f3877e) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)

LGTM!

Jenkins CI test (for commit 826770d43496a3f6ca24fa446f5a16bfd6f3877e) failed with status FAILURE.
(For contributors, please wait until the reviewer confirms the details of the error.)


Multiplexer feature

Description of the new feature/enhancement

A multiplexer feature would be very useful and cool!
By having multiple panes in a single windows:

You can easily copy/paste output from one pane to another.
You can work on multiple tasks at once.
You can monitor multiple commands at once.
You can compose a command in one pane while looking at the 'help' page for that command in another pane.

What's wrong with tmux or screen?

Whats the point of your question @nofunatall ?
I didn't ask for opinions about what application should I use if I need a multiplexer console!
Here we discuss about windows terminal!

Let me rephrase that.
Are you after just window / tab splitting via hotkeys or are you after all the features of a fully fledged multiplexer?

I imagine it's best to start small.
At first it`s enough to have the splitting thing. Then more functionality can be added.

This is, broadly, a /dupe of #1000. Thanks.
We actually have split panes today!

Perhaps he had in mind here is a multiplexer

I completely agree that this is not the main feature, since soon it will not need to be implemented inside the Windows Terminal at all, as modern tmuxes will perform this functionality, and this will be cross-platform.
https://youtu.be/5h1fJ-es8kQ


issue in MUI textfield

I have an issue with customization of MUI Textfield

Closing since no reproduction was provided within 14 days.


Fix #1635 - Add springdoc button

Fixes #1635

Cypress test must be updated, with the new button => Generator.spec.ts

Regarding this I have already added test in Generator.spec.ts to check if spring doc button is getting generated or not, anything else needs to be added too ?
https://github.com/jhipster/jhipster-lite/pull/1733/files#diff-858debb55e6d85be1d68c1e215357e371882b664a2ee7ae3da0245341728825dR41

Cypress test must be updated, with the new button => Generator.spec.ts

Regarding this I have already added test in Generator.spec.ts to check if spring doc button is getting generated or not, anything else needs to be added too ?
https://github.com/jhipster/jhipster-lite/pull/1733/files#diff-858debb55e6d85be1d68c1e215357e371882b664a2ee7ae3da0245341728825dR41

No, I've done à mistake, sorry :/


Include in ultils/lang Portuguese (ptBR).

I started the translation into Brazilian Portuguese. I'm still checking to see if there's anything else to do. Initially I only translated utils/lang.ts.

Perfect, changing utils/lang.ts should be sufficient.

@all-contributors please add @vitoropereira for code


chore: Update v2 deployment manifest to K8s release API and default K8s version to 1.18

What type of PR is this?
/kind cleanup
/kind failing-test

What this PR does / why we need it:
Updates the v2 deployment manifests for the K8s release API version corresponding to PR kubernetes-sigs/azuredisk-csi-driver#836. This also requires that the minimum version of K8s supported by the driver is now 1.18. Updates the azure-cluster-up.sh script accordingly.
Which issue(s) this PR fixes:

Requirements:

[x] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] tested upgrade from previous version

Special notes for your reviewer:
Release note:
none

Pull Request Test Coverage Report for Build 879557853

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 36.128%

Totals

Change from base Build 877492366:
0.0%

Covered Lines:
1099

Relevant Lines:
3042

💛  - Coveralls


GSB: Watch out for a concrete ResolvedType in ArchetypeType::resolveNestedType

Resolves SR-13519
The regression was caused by 653fa07260497ab736c7667158e9e59226723af4, apparently. Under certain conditions, maybeResolveEquivalenceClass would look into the superclass of the base equiv. class to find a nested type, resulting in a concrete ResolvedType that is not associated with an equivalence class. resolveEquivalenceClass and at least one of its clients were not expecting this to happen.

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

How to change the base branch: (Link)
More detail about the branch update: (Link)

@slavapestov ping

@slavapestov ping

I'll take a look at this shortly. I think another dupe of this came in recently.

Sorry for the pinging, I somehow missed you self-assigned this.


Train recognition model for Arabic line text.

I have a large dataset with Arabic line text. I want to train/ finetune the OCR model using this data.
The data structure is *.png *.gt.txt. I can not fine a prober guide to transform the data to the correct format and perform the training.
Note: The data have the same font, and each line is a mix of numbers and text.
@mittagessen

Contact me, might help add you to a group of people who are interested in creating Arabic OCR tool that works for real.


Rubocop - Style/PercentLiteralDelimiters conflict

$ rubocop -V
0.52.0 (running on ruby 2.3.2p217 (2016-11-15 revision 56796) [x86_64-darwin17])

Rubocop notifies me to:
only: [:index, :create] -> only: %i[index create] for the auto-correct
And it did the auto-correction for me:
[Corrected] Style/SymbolArray: Use %i or %I for an array of symbols.
But then it raises error when I want to commit the change:
C: Style/PercentLiteralDelimiters: %i-literals should be delimited by ( and )
I should change only: %i[index create] to only: %i(index create) manually.
The auto-correct and the error does not match.

Not reproducible in 0.57.1.


A general simulation tutorial

Summary
Still missing some references at the bottom.
The call to plot_bloch_vector() from qiskit visualization does not show anything.
Details and comments

This simulation is really cool. I feel though like this may be more of an "advanced" tutorial - the model and dynamics are very non-trivial. Is this what you were referring to originally about having two simulation tutorials? Maybe we can have this and a more basic one, e.g. observing rabi oscillations of a single qubit driven on resonance?
Regardless of the above,  I think this tutorial is good but just needs to be edited down a bit to be simpler:

Given how advanced the model is, it might be worth giving more explanation of what it is at the beginning of the tutorial, so that when you get to section 2 setting up the Solver, there isn't as much text before getting into the code.

The Hamiltonian and dissipator information in section 2 can go at the beginning.

Section 1

I think we can simplify this by taking advantage of Pauli in quantum info (this is suggested by @chriseclectic ). Setting z = Operator(np.zeros((dim, dim))), the Pauli X operator on qubit with index 3 for example, can be created via: z + Pauli('X')(3). This can maybe even be incorporated directly into the Hamiltonian generation code for section 2. Two qubit operations can be created in the same way, e.g. z + Pauli('XY')(i, j). The addition of z is a bit of a hack but I feel like it's more readable than the string manipulation and ''.join that is currently being used.

Section 2

Can be simplified according to the above two comments.

Section 3

Less explanation in the text is required - I think you can get rid of everything but the first sentence.
DensityMatrix also has a from_label constructor, you can do: DensityMatrix.from_label('0' * N)
Use np.linspace to generate the time points. This will automatically include the endpoints and avoid requiring to check if the endpoint is in the list.
As the model is constant in time, you could use the scipy_expm method, though it will require setting up the solver to be in a vectorized evaluation mode. E.g. at construction add the initial kwarg: evaluation_mode='dense_vectorized', then when calling solve do method='scipy_expm', max_dt = ..., where ... is something at least as large as the size of the steps between time points. Alternatively, if using a standard ODE solver I think you will want to go into the frame of the drift.

Section 4

Interesting comments about the mean bloch vector v.s. individual qubits bloch vector.

Section 5

Cool!

Section 6

I think this should be made way shorter and maybe just be moved after the plot of Section 5. This stuff is interesting but the tutorial should be more concerned with demonstrating how the code can be used to perform a task, rather than interpretation of the results of this specific simulation.

No need to have print statements at the end of code blocks - we already know what the code block is for
Since you're not using JAX there's no need to use the Array class, you can just use standard numpy arrays.

Thank you @DanPuzzuoli for the review, I'll apply it. I have two questions - I did not know how to use np.linspace since it accepts a number of steps and not the time step. Passing the number of steps would require a computation since IMO it should be determined by the time step and the total simulation time span. What do you suggest? About Array - without using it I got type hint warnings, what do you suggest? Thanks.

Where are the type-hint warnings appearing? Depending on where it is we should create an issue for it, but whatever the case you should avoid using Array in this notebook.
For using np.linspace can you just choose N so you get a suitable dt for the time span you're using? I mainly just don't like the if statement adding a time value if necessary, and would prefer you just setting N directly to whatever value works.

@DanPuzzuoli I applied your corrections except two that I didn't know how:

I didn't know how to use the construct z + Pauli('X')(3). I found the Pauli constructor Pauli(label='X'), but then I got an error that the Pauli class is not callable and didn't find a syntax that achieves it. What should it be?
About vectorization, I wasn't sure if you want me to use it. Since I expect it allows simulating only about half the number of qubits, I didn't apply this modification. About going to the drift frame - I tried adding to the Solver constructor the parameter drift=H, but that seemed to work slower (for 7 qubits on my laptop it took ~18 seconds instead of ~9 without this parameter), so I thought it may not be what you meant. Please let me know what you suggest to do.
I removed the Array usages. There are type warnings in the Solver constructor (with hamiltonian_operators and dissipator_operators) and the solve() method (with the t_span parameter).
I can work on a simpler tutorial following this one.

The "Simulating rabi oscillations..." tutorial looks really good. I think the one main comment is that it'd be good to establish some notational consistency between tutorials:

The pulse tutorial, and the one I'm writing for JAX, use X, Y, Z for Pauli operators instead of s_x, s_y, s_z (and \sigma_j in latex). I prefer this as it corresponds directly to the symbol used to construct it (e.g. Operator.from_label('Z')).
Frequencies are usually written as either \omega (rad/s) or \nu (1/s). Can you use this instead of f?
I'm wondering if you need to write the form of the lindblad equation, or if you should just state the jump operators?

Aside from this, some minor preferences:

I know this is contradicting a bit what I said in the previous tutorial, but can you put the Hamiltonian description into the first section, and the dissipator description into the last section when you add dissipation to the model? In contrast to the other tutorial, I think the model here is "standard" enough to not require explicitly stating what it is in the intro.
Can you change the title of section 1 to something like: "Setup the solver with the Hamiltonian model". I feel like the term "hamiltonian" should be used in it.
Change the title of section 2 to just: "Solve the system"

Other than this looks good to me! (Will comment on the other tutorial in a separate comment.)

For the spin chain tutorial:

Same general notational convention comments from the previous comment apply.
Similar to previous comment: Do we need to write the Lindblad equation here or can we just state what the dissipators are?
Can you just put 2 * np.pi everywhere instead of using the _2_pi.
There's still some print statements to be eliminated.
Can you put a comment line above the DensityMatrix('0' * N) saying: "# density matrix with all qubits in ground state" or something.
For the issue with constructing the Pauli's and getting the "no callable" error, do you need to just update terra? It works fine for me when I try it.

@DanPuzzuoli I applied the changes in your two comments above.
For the question, I think that it is a bit better to write down the Lindblad equation explicitly with the full expressions for the dissipators, to avoid ambiguity that comes from different possible notations (where is the dagger - on the left or right jump operator, and is there a factor of 1/2 or a factor of 2 on the other term in the dissipator), and also just for completeness. Let me know whether to remove it nevertheless. Thanks.


"Gateway Services publish failed" message when using gwa apply

Even when only publishing a Product or DraftDataset, gwa makes a PUT call to api.gov.bc.cagw/api/v2/namespaces/gw-abc123/gateway which will return an error message:

This is unnecessary and a nuisance for users.

Verified - looks good

testing modified Jira automation - this should not change ticket status to In Progress


[AppConfig] az appconfig kv export/import: Add a new parameters to support exporting and importing all key values from a snapshot of the source configuration

Related command

az appconfig kv export
az appconfig kv import
Description

This change enables users to export key-values from a configuration snapshot to other destinations.
Testing Guide

History Notes

[AppConfig] az appconfig kv export/import: Add a new parameter --snapshot to support exporting and importing all key values from a snapshot of the source configuration

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

AppConfig

Could you please add a test case for this new parameter?

Could you please add a test case for this new parameter?
@zhoxing-ms Please the test cases have been updated to test the 2 new parameters.

Could you please add a test case for this new parameter?
@zhoxing-ms Please the test cases have been updated to test the new parameters.

@albertofori Please resolve this conflict

@albertofori Please resolve this conflict

@zhoxing-ms Please conflicts have been resolved, however, could you hold on from merging for now as this needs another review for functionality that was added for import.


Conversion fails on certain video where videostream has missing frame

Checklist

[ ] I'm reporting a broken site support issue
[x] I've verified that I'm running youtube-dl version 2020.09.20
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

.\external\youtube-dl.exe -C --cookies cookies.txt --no-mtime -o "downloads\%(upload_date)s [%(uploader)s] %(title)s (%(id)s).%(ext)s" --rm-cache-dir -4 --add-metadata --embed-subs --all-subs -f "(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best" -v --merge-output-format mkv https://www.youtube.com/watch?v=YV_tHOrufkw
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-C', '--cookies', 'cookies.txt', '--no-mtime', '-o', 'downloads\\%(upload_date)s [%(uploader)s] %(title)s (%(id)s).%(ext)s', '--rm-cache-dir', '-4', '--add-metadata', '--embed-subs', '--all-subs', '-f', '(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best', '-v', '--merge-output-format', 'mkv', 'https://www.youtube.com/watch?v=YV_tHOrufkw']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Public IP address: 95.217.XXX.XXX
Removing cache dir C:\Users\Malte/.cache\youtube-dl ..
[youtube] YV_tHOrufkw: Downloading webpage
WARNING: video doesn't have subtitles
[youtube] YV_tHOrufkw: Downloading MPD manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1602383828/ei/dBuCX6e8I5H_yAWH7rrwBQ/ip/95.217.XXX.XXX/id/615fed1ceaee7e4c/source/youtube/requiressl/yes/playback_host/r6---sn-5goeen7d.googlevideo.com/mh/uo/mm/31%2C26/mn/sn-5goeen7d%2Csn-f5f7lnee/ms/au%2Conr/mv/u/mvi/6/pl/24/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/mt/1602361563/fvip/6/keepalive/yes/fexp/23915654/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRQIgdSXiQBazULin60luzEDqJCnWL530kKzG3kr0e04Oa5oCIQCOnl2DIgJsoXOoTndW-76pMw4Zv8OzENKOIlheh-DaQA%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl/lsig/AG3C_xAwRgIhALdst4d3X4A3KaM4QRg7n0B_E2dDOQ0qKYLRyPSeFEziAiEA5WZM0QrIu5XnPArEVvaqjSaIx7q76MeUJjgHbxwlkvU%3D'
[dashsegments] Total fragments: 5
[download] Destination: downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm
[download] 100% of 4.92MiB in 00:01
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1602383828/ei/dBuCX6e8I5H_yAWH7rrwBQ/ip/95.217.XXX.XXX/id/615fed1ceaee7e4c/source/youtube/requiressl/yes/playback_host/r6---sn-5goeen7d.googlevideo.com/mh/uo/mm/31%2C26/mn/sn-5goeen7d%2Csn-f5f7lnee/ms/au%2Conr/mv/u/mvi/6/pl/24/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/mt/1602361563/fvip/6/keepalive/yes/fexp/23915654/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRQIgdSXiQBazULin60luzEDqJCnWL530kKzG3kr0e04Oa5oCIQCOnl2DIgJsoXOoTndW-76pMw4Zv8OzENKOIlheh-DaQA%3D%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl/lsig/AG3C_xAwRgIhALdst4d3X4A3KaM4QRg7n0B_E2dDOQ0qKYLRyPSeFEziAiEA5WZM0QrIu5XnPArEVvaqjSaIx7q76MeUJjgHbxwlkvU%3D'
[dashsegments] Total fragments: 4
[download] Destination: downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm
[download] 100% of 262.52KiB in 00:00
[ffmpeg] Merging formats into "downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm" -c copy -map "0:v:0" -map "1:a:0" "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv"
ERROR: Conversion failed!
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\YoutubeDL.py", line 2065, in post_process
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\postprocessor\ffmpeg.py", line 523, in run
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpksi3o1r1\build\youtube_dl\postprocessor\ffmpeg.py", line 235, in run_ffmpeg_multiple_files
youtube_dl.postprocessor.ffmpeg.FFmpegPostProcessorError: Conversion failed!

Description
When trying to download the video https://www.youtube.com/watch?v=YV_tHOrufkw as seen in the log above, the conversion fails after the download of the audio and video stream has completed. When running the ffmpeg command manually, the following output is generated:
.\external\ffmpeg -y -loglevel "repeat+info" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm" -c copy -map "0:v:0" -map "1:a:0" "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv" -loglevel verbose
ffmpeg version 4.3 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200621
  configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, matroska,webm, from 'file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm':
  Metadata:
    encoder         : google/video-file
    http://youtube.com/streaming/otf/durations/112015: Segment-Count: 4
                    : Segment-Durations-Ms: 5333,5334,5333,4850,
                    :
                    :
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0(eng): Video: vp9 (Profile 0), 1 reference frame, yuv420p(tv, bt709/unknown/unknown), 1280x720, SAR 1:1 DAR 16:9, 60 fps, 60 tbr, 1k tbn, 1k tbc (default)
Input #1, matroska,webm, from 'file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm':
  Metadata:
    encoder         : google/video-file
  Duration: 00:00:20.90, start: -0.007000, bitrate: 102 kb/s
    Stream #1:0(eng): Audio: opus, 48000 Hz, stereo, fltp, delay 312 (default)
Output #0, matroska, to 'file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv':
  Metadata:
    http://youtube.com/streaming/otf/durations/112015: Segment-Count: 4
                    : Segment-Durations-Ms: 5333,5334,5333,4850,
                    :
                    :
    encoder         : Lavf58.45.100
    Stream #0:0(eng): Video: vp9 (Profile 0), 1 reference frame (VP90 / 0x30395056), yuv420p(tv, bt709/unknown/unknown), 1280x720 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 60 fps, 60 tbr, 1k tbn, 1k tbc (default)
    Stream #0:1(eng): Audio: opus ([255][255][255][255] / 0xFFFFFFFF), 48000 Hz, stereo, fltp, delay 312 (default)
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
Automatically inserted bitstream filter 'vp9_superframe'; args=''
[matroska @ 000001aba3cf6180] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 5350 >= 5333
av_interleaved_write_frame(): Invalid argument
No more output streams to write to, finishing.
frame=  323 fps=0.0 q=-1.0 Lsize=    1378kB time=00:00:05.36 bitrate=2105.5kbits/s speed= 536x
video:1336kB audio:67kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Input file #0 (file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm):
  Input stream #0:0 (video): 323 packets read (1368175 bytes);
  Total: 323 packets (1368175 bytes) demuxed
Input file #1 (file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm):
  Input stream #1:0 (audio): 269 packets read (68512 bytes);
  Total: 269 packets (68512 bytes) demuxed
Output file #0 (file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv):
  Output stream #0:0 (video): 323 packets muxed (1368175 bytes);
  Output stream #0:1 (audio): 269 packets muxed (68512 bytes);
  Total: 592 packets (1436687 bytes) muxed
[AVIOContext @ 000001aba434c000] Statistics: 2 seeks, 7 writeouts
[AVIOContext @ 000001aba3ced080] Statistics: 1409024 bytes read, 0 seeks
[AVIOContext @ 000001aba3cec700] Statistics: 98304 bytes read, 0 seeks
Conversion failed!

The error hints that something is wrong with the frames of the video stream being processed. And indeed if you play the video stream webm in VLC media player, you can see a lost frame after approximately 320 frames in the statistics tab of the media information window. So it seems that the file fetched by youtube-dl is somehow corrupted.
I can't tell if this is an issue in youtube-dl with the fragments not being fetched correctly and stitched together or if wrong data is sent by the YouTube servers, which is why I'm raising this issue.
Hope my description is clear enough. Thanks.

I think it must be something with your selected formats. When just doing the exe with the URL without all the options, I get a complete file without errors.
But I also getting streams 298 and 140, not 302 and 251 like yours, thus other codecs.
The file in the end has the following streams:
``
Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile<EMAIL_ADDRESS>Format settings                          : CABAC / 1 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 1 frame
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 20 s 849 ms
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Title                                    : ISO Media file produced by Google Inc. Created on: 10/08/2020.
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : A_AAC-2
Duration                                 : 20 s 944 ms
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Title                                    : ISO Media file produced by Google Inc.
Language                                 : English
Default                                  : Yes
Forced                                   : No
``

True, the default format selection of youtube-dl does not always download the best quality available, so I took that format code which I found on this reddit post: https://www.reddit.com/r/DataHoarder/comments/c6fh4x/after_hoarding_over_50k_youtube_videos_here_is/
What you are getting is 720p H.264 HFR with 128 Kbps AAC audio while my download gets 720p VP9 HFR with VBR ~160 Kbps OPUS audio. Similar codecs with similar or better quality at similar or smaller file sizes.
See this table for the different format codes: https://gist.github.com/AgentOak/34d47c65b1d28829bb17c24c04a0096f
That format selection has worked very reliably so far for me and others, and the different codec should in theory not produce a broken video file.

Hmm but wouldn't that mean, that ffmpeg has a bug and not the downloader? Because it only downloads the corresponding files.
mkvmerge doesn't have any problems joining the two temp files into a MKV file, which can be played without problems afterwards.

It's not a bug in ffmpeg but normal behavior when it tries to parse a file that is considered corrupt or malformed. It doesn't really help that makemkv seems to work, since youtube-dl uses ffmpeg for that and I'm not aware that makemkv can be used natively instead.
The issue with the conversion is caused by the file in the first place having that one "broken" or "missing" frame as it is also recognized by VLC when playing back the webm file.
Since the frame seems to be at the exact point where the second DASH frame would begin (Segment-Durations-Ms: 5333, [...]), this leads me to believe that this could be an issue with how youtube-dl downloads and saves the frames. But this could also be YouTube delivering the broken file. I'm gonna research and try some things tomorrow, but for now I can't tell.

Without making any changes the download now seems to work fine for this video. Seems to be an issue on YouTube's end, but I wonder if a workaround can be added to youtube-dl for this issue. I'll report back if I find another video with the same issue.
.\external\youtube-dl.exe -C --cookies cookies.txt --no-mtime -o "downloads\%(upload_date)s [%(uploader)s] %(title)s (%(id)s).%(ext)s" --rm-cache-dir -4 --add-metadata --embed-subs --all-subs -f "(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best" -v --merge-output-format mkv https://www.youtube.com/watch?v=YV_tHOrufkw
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-C', '--cookies', 'cookies.txt', '--no-mtime', '-o', 'downloads\\%(upload_date)s [%(uploader)s] %(title)s (%(id)s).%(ext)s', '--rm-cache-dir', '-4', '--add-metadata', '--embed-subs', '--all-subs', '-f', '(bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec=vp9.2][height>=1080][fps>30]/bestvideo[vcodec=vp9][height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec=vp9.2][height>=1080]/bestvideo[vcodec=vp9][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec=vp9.2][height>=720][fps>30]/bestvideo[vcodec=vp9][height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec=vp9.2][height>=720]/bestvideo[vcodec=vp9][height>=720]/bestvideo[height>=720]/bestvideo)+(bestaudio[acodec=opus]/bestaudio)/best', '-v', '--merge-output-format', 'mkv', 'https://www.youtube.com/watch?v=YV_tHOrufkw']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Public IP address: 95.217.XXX.XXX
Removing cache dir C:\Users\Malte/.cache\youtube-dl ..
[youtube] YV_tHOrufkw: Downloading webpage
WARNING: video doesn't have subtitles
[youtube] YV_tHOrufkw: Downloading MPD manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1602622764/ei/zMCFX4OrGYLyyAX0t6SIBg/ip/95.217.XXX.XXX/id/615fed1ceaee7e4c/source/youtube/requiressl/yes/playback_host/r3---sn-4g5e6ns6.googlevideo.com/mh/uo/mm/31%2C29/mn/sn-4g5e6ns6%2Csn-axq7sn7e/ms/au%2Crdu/mv/m/mvi/3/pl/24/nh/%2CIgpwcjAzLmxlZDAzKgkxMjcuMC4wLjE/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1536250/vprv/1/mt/1602601015/fvip/10/keepalive/yes/fexp/23915654/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgORxG0cxsJKUEFVOqe185NLvqZ--z5MB6Fs9un9h_j7MCIAF3zGA7rLJe5e2KcfdIufJ5g7T3YH5P3vGPSkEEfs8B/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cnh%2Cinitcwndbps/lsig/AG3C_xAwRgIhAP9WIW00DVW5uUyh8b1ITFCbHgfNp0fNl_KeZGL8rf29AiEAvgSvEMys9awogOsVY4aOwKWB0VJlBx4X07iNvD34JBs%3D'
[dashsegments] Total fragments: 5
[download] Destination: downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm
[download] 100% of 4.94MiB in 00:06
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/dash/expire/1602622764/ei/zMCFX4OrGYLyyAX0t6SIBg/ip/95.217.XXX.XXX/id/615fed1ceaee7e4c/source/youtube/requiressl/yes/playback_host/r3---sn-4g5e6ns6.googlevideo.com/mh/uo/mm/31%2C29/mn/sn-4g5e6ns6%2Csn-axq7sn7e/ms/au%2Crdu/mv/m/mvi/3/pl/24/nh/%2CIgpwcjAzLmxlZDAzKgkxMjcuMC4wLjE/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1536250/vprv/1/mt/1602601015/fvip/10/keepalive/yes/fexp/23915654/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgORxG0cxsJKUEFVOqe185NLvqZ--z5MB6Fs9un9h_j7MCIAF3zGA7rLJe5e2KcfdIufJ5g7T3YH5P3vGPSkEEfs8B/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cnh%2Cinitcwndbps/lsig/AG3C_xAwRgIhAP9WIW00DVW5uUyh8b1ITFCbHgfNp0fNl_KeZGL8rf29AiEAvgSvEMys9awogOsVY4aOwKWB0VJlBx4X07iNvD34JBs%3D'
[dashsegments] Total fragments: 4
[download] Destination: downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm
[download] 100% of 262.52KiB in 00:01
[ffmpeg] Merging formats into "downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm" -c copy -map "0:v:0" -map "1:a:0" "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv"
Deleting original file downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f302.webm (pass -k to keep)
Deleting original file downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).f251.webm (pass -k to keep)
[ffmpeg] Adding metadata to 'downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).mkv'
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).mkv" -c copy -metadata "date=20201008" -metadata "description=" -metadata "purl=https://www.youtube.com/watch?v=YV_tHOrufkw" -metadata "comment=" -metadata "title=BUT I DID KIDNAP HIS WIFE" -metadata "artist=Posy" "file:downloads\20201008 [Posy] BUT I DID KIDNAP HIS WIFE (YV_tHOrufkw).temp.mkv"
[ffmpeg] There aren't any subtitles to embed

I have had the same issue very recently with quite a lot of videos. This only happens with VP9 streams. It seems that the files downloaded from YouTube are somewhat corrupted. When played with SMPlayer (mpv), there is no duration displayed, and the videos are unseekable.
Here are two examples from today.
https://youtube.com/watch?v=sH-efMgF2K8
https://youtube.com/watch?v=65w5RdydvJc
This command will lead to ERROR: Conversion failed!.
youtube-dl youtube.com/watch?v=sH-efMgF2K8 --format bestvideo[ext=webm]+bestaudio[ext=webm]
What is interesting is the fact that I downloaded the same videos a few months ago, and the streams were merged correctly then.

Are you really unable to reproduce?
youtube-dl https://youtube.com/watch?v=sH-efMgF2K8 --format bestvideo[ext=webm]+bestaudio[ext=webm]
[youtube] sH-efMgF2K8: Downloading webpage
[youtube] sH-efMgF2K8: Downloading MPD manifest
[download] [BG2 -EE #4] Baldur's Gate Saga SCS Completionist Playthrough - Freedom Regained... Again-sH-efMgF2K8.f303.webm has already been downloaded
[download] 100% of 271.89MiB
[dashsegments] Total fragments: 219
[download] Destination: [BG2 -EE #4] Baldur's Gate Saga SCS Completionist Playthrough - Freedom Regained... Again-sH-efMgF2K8.f251.webm
[download] 100% of 30.98MiB in 00:22
[ffmpeg] Merging formats into "[BG2 -EE #4] Baldur's Gate Saga SCS Completionist Playthrough - Freedom Regained... Again-sH-efMgF2K8.webm"
ERROR: Conversion failed!

Is this working at your end?


Add information received in response to prepare to the trace

The PR introduces parsing operation type received in custom payload in response to prepare and setting it as a tag of bound statement. The incoming payload of prepared statement is propagated as outgoing payload if no outgoing payload has been explicitly set but then operation type is removed from the payload that is propagated.

@smaczliwka Please rebase on top of latest master.

and also modify the parsing code to work correctly with https://github.com/zpp-2021-telemetry/scylla/pull/10

Now, when testing with https://github.com/zpp-2021-telemetry/scylla/pull/10, there is one problem driver-side: opentelemetry custom payload now contains both: list of replicas, operation type. However, the parsing code doesn't correctly parse those values separately - the operation type string contains the bytes of the list of replicas.
The way I reproduced it:
2 Scylla nodes started (code from https://github.com/zpp-2021-telemetry/scylla/pull/10).
CREATE KEYSPACE ks2 WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 2};
CREATE TABLE ks2.t(pk int, ck int, v int, PRIMARY KEY(pk, ck));

        PreparedStatement ps = s.prepare("INSERT INTO ks2.t(pk, ck, v) VALUES (?, ?, ?)");
        System.out.println("Operation type is: " + ps.getOperationType());

which now prints:

those first 4 characters are probably the bytes of replica list.


Make autocomplete schema-aware

This PR would fix #22, and also I think #39 as a side-effect
Issues:

I used pandas DataFrames for working with database metadata -- I found them a lot more flexible than a pure standard lib dict/list based approach. But I'm not sure how you'd feel about adding pandas as a requirement.
We don't have any tests for "DoubleQuoted" identifiers -- I'm not confident I didn't break that.

Taking a look now. It might take me a while to fully review.
I'm a bit hesitant to add Pandas as a requirement since it's not quite straight-forward to install. But if it really does make it easy to do certain things, then I'm open to adding it.
I've been meaning to learn Pandas for a while now, this will be a great opportunity. :)

I'm really sorry about that. Right now I only know how to squash commits in
git, I haven't figured out how to split them apart yet, haha. Mostly I
wanted to find out what you thought about the overall approach before I
spent too much more time on it. What I should've had as the first commit
was refactoring packages.sqlcompletion.suggest_type to return an array of
suggestion objects (right now they're just dicts, but they could be
namedtuples or actual custom classes or whatever) instead of the type as a
string like 'columns-and-functions'  or whatever, since that doesn't scale
too well as the number of different possibilities expands.
I'm totally cool with not merging right away, since I think this would
provide the foundation for more complicated autocompletion (CTEs, user
defined functions (especially set-returning functions, which can act as
tables), arbitrary-depth-nested composite types...) so we might as well get
it right.
I also totally understand your hesitations regarding pandas (even if we
don't end up using it, you should still learn it, it's awesome!) it's
totally feasible to rewrite in a more straightforward way -- it just saved
me some time in development.
On Sun, Jan 18, 2015 at 6:19 PM, Amjith Ramanujam<EMAIL_ADDRESS>wrote:

Taking a look now. It might take me a while to fully review.
I'm a bit hesitant to add Pandas as a requirement since it's not quite
straight-forward to install. But if it really does make it easy to do
certain things, then I'm open to adding it.
I've been meaning to learn Pandas for a while now, this will be a great
opportunity. :)
—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-70431614.

No worries!!
I like the refactoring done to suggest_type method.
I'm pulling down your repo to play around a little. I might send PRs to
your repo to make some corrections. Then we can finally merge it back into
mainline.
On Sun Jan 18 2015 at 3:59:40 PM darikg<EMAIL_ADDRESS>wrote:

I'm really sorry about that. Right now I only know how to squash commits
in
git, I haven't figured out how to split them apart yet, haha. Mostly I
wanted to find out what you thought about the overall approach before I
spent too much more time on it. What I should've had as the first commit
was refactoring packages.sqlcompletion.suggest_type to return an array of
suggestion objects (right now they're just dicts, but they could be
namedtuples or actual custom classes or whatever) instead of the type as a
string like 'columns-and-functions' or whatever, since that doesn't scale
too well as the number of different possibilities expands.
I'm totally cool with not merging right away, since I think this would
provide the foundation for more complicated autocompletion (CTEs, user
defined functions (especially set-returning functions, which can act as
tables), arbitrary-depth-nested composite types...) so we might as well
get
it right.
I also totally understand your hesitations regarding pandas (even if we
don't end up using it, you should still learn it, it's awesome!) it's
totally feasible to rewrite in a more straightforward way -- it just saved
me some time in development.
On Sun, Jan 18, 2015 at 6:19 PM, Amjith Ramanujam <
<EMAIL_ADDRESS>wrote:

Taking a look now. It might take me a while to fully review.
I'm a bit hesitant to add Pandas as a requirement since it's not quite
straight-forward to install. But if it really does make it easy to do
certain things, then I'm open to adding it.
I've been meaning to learn Pandas for a while now, this will be a great
opportunity. :)
—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-70431614.

—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-70433123.

@darikg You're right about the auto-escaped tables and columns being broken.
I've added new tests to the master branch to properly test the auto-escaped identifiers.
I also pulled in your PR into a branch in the mainline. https://github.com/amjith/pgcli/tree/darikg/schema-completion and merged it in with the master.
The tests are failing.
My attempts to fix them have been unsuccessful because of my inexperience with pandas. But I've narrowed it down to this line. https://github.com/amjith/pgcli/blob/darikg/schema-completion/pgcli/pgcompleter.py#L94
I think your intention is to apply the escaping to all the relevant values in the DataFrame, but it doesn't apply the transformation in place, it actually applies the transformation and returns the transformed copy.
So I tried changing it to this:
      # data is a DataFrame with columns [schema, table, is_visible]
        tmp = data[['schema', 'table']].apply(self.escaped_names)
        self.tables = self.tables.append(tmp)

But that didn't work because we want tmp to have the is_visible col but that's not present in the transformed tmp var.
I'm also confused by how the apply() method works. Because I would have expected it to work with apply(self.escape_name) but that gives an error.
I might start updating that branch https://github.com/amjith/pgcli/tree/darikg/schema-completion with a non-pandas version of the change you've made, if you don't mind.

Thanks for the auto-escaped identifiers tests, they made it a lot easier to find all my mistakes (there were a lot).
You're right about DataFrame.apply operating in-place, so it should be
data[['schema', 'table']] = data[['schema', 'table']].apply(self.escaped_names)
Looks like the tests are passing in python 2, but failing in 3 because of the dict-sorting in the test files. I'll have to figure out a better way to do that. I should probably write a lot more schema-autocompletion tests in general too -- I assume there's a whole bunch of weird edge cases

@darikg Thanks for fixing that. I'll take a look at the failing Py3 tests.
I'll dig into this a little more tonight. It's been fun seeing how simplified the code can be when pandas is used. But I'm still not sure why apply() didn't take self.escape_name as an arg.
Also if you have any docs for quick start on Pandas, I'd appreciate some pointers. :)

In a DataFrame, each column is a Series object, and DataFrame.apply(func)
calls func on each Series in turn. Series are duck-typed like lists, so you
need basically func to accept a list-like object. You can do
DataFrame.applymap() instead, which applies func on individual elements in
the frame.
The pandas docs are actually really good. Also the guy who wrote pandas has
a book (python for data analysis) with a bunch of chapters on pandas.
On Mon, Jan 19, 2015 at 4:19 PM, Amjith Ramanujam<EMAIL_ADDRESS>wrote:

@darikg https://github.com/darikg Thanks for fixing that. I'll take a
look at the failing Py3 tests.
I'll dig into this a little more tonight. It's been fun seeing how
simplified the code can be when pandas is used. But I'm still not sure why
apply() didn't take self.escape_name as an arg.
Also if you have any docs for quick start on Pandas, I'd appreciate some
pointers. :)
—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-70562712.

I tried it out last night. It works well.
I did run into one issue. When you type:
SELECT * FROM django_migration WHERE <tab>

The column names are listed twice. I was debugging it for a while and I think it's coming from populate_scoped_cols() when you do an inner join merge, I haven't confirmed it yet.
But the weird thing is, I can't reproduce it in the unit-tests. At first I thought I couldn't reproduce it in tests because the tests call set() on the list of suggestions which removes duplicates, but even if I replaced the set() with list() it still doesn't trigger that behavior in tests. Which makes it harder to debug.
I've been adding a ton of _logger.debug() statements to debug this  (caveman-style) while running the repl.

Wow, nice find. I was able to replicate something like that by creating two identical tables, one in the public schema and one inside another (non-visible) schema. Then get_scoped_cols was ultimately combining rows from unqualed_tables and columns dataframes and on just the table column, instead of schema and table. So two different tables of the same name ended up contributing columns to the suggestion list. I just pushed a commit that should fix that. Is this the case in your database by any chance?
I was going to add a test for this issue as well, but I was thinking, does it make sense to make the mock database structure in test_smart_completion more and more complicated, or should it be split up into a simple mock database with only a public schema, and then a separate set of tests for a more complicated db design?

Nope. That wasn't it.
It was actually this line:
return list(qualed['column']) + list(unqualed['column'])

Screenshot:

It's picking the columns from a qualed table public.django_migrations and an unqualed table django_migrations and appending them together and returning it.
But both of them are the same table so the columns appear twice.
I think the simple solution is make those lists into sets an do a union operation. That will get rid of duplicates, but that's really treating the symptoms. The root cause fix would be excluding the qual table name from the unqualed table name list.
BTW, I'm totally fine with separating the more complicated tests to a different file.

Hmm... if the table in the query isn't qualifed, then the corresponding value of scoped_tbls['schema'] should be null, and django_migrations shouldn't show up in qualed (because of the missing inner join condition on schema. Right?
Anyway, I was thinking about how to do this without pandas, and I think I would define namedtuples Table('schema_name', 'table_name', 'is_visible') and `Column('schema_name', 'table_name', 'column_name')' and store the database metadata as lists of namedtuples instead of dataframes.
I think I should also have suggest_type return tuples of namedtuple suggestion types, instead of my current dict-based approach.
What do you think?

Hmm... if the table in the query isn't qualifed, then the corresponding value of scoped_tbls['schema'] should be null, and django_migrations shouldn't show up in qualed (because of the missing inner join condition on schema. Right?

It doesn't happen on an unqualed table name, only on qualed table name and the schema has to be 'public' and the table needs to be visible.
So try this:
SELECT * FROM public.users WHERE

The table 'users' has to be visible and it should be qualed with a schema name.
I have some thoughts on the panda-less implementation. I'll write up my thoughts later tonight.

About the data-structures to use in place of Panda.
I don't think using a namedtuple for Tables and Columns is going to work for a couple of reasons.
Having Table(schema, table_name, is_visible) will result in a lot of duplication of data. For eg: All the tables in public schema will have ('public', 'table1', True), ('public', 'table2', True), ('public', 'table3', True) etc. Where the schema and is_visible columns are redundant. When you have to lookup all the tables in the 'custom' schema, you have to iterate over a list of namedtuples to filter them by the schema column that matches 'custom'.
So here's my proposal.
The parseutils.extract_tables() should return a tuple (or a namedtuple) with (schema, table, alias).

'schemata' should be a list of schemas.
'schema_tables' should be a dictionary with schema as key and a list of table names in the schema. eg: {'public' : ['table1', 'table2'], 'custom' :['table1', 'table2']}
'visible_tables' should be a set() of all the visible table names from schemas. This is used for unqaled completion. If we don't have this, then we would have to iterate through the schema_tables dictionary to gather all the values and build a set of all the table name on the fly (which is inefficient).
'columns' should be a dictionary with keys as 'tables' and values as a list of column names that are in that table.

What do you think?

Good call on setting up the metadata storage for finding stuff more efficiently.

'columns' should be a dictionary with keys as 'tables' and values as a list of column names that are in that table.

What about when two tables in different schemas have the same name? I guess we could have two (redundant) dictionaries, one indexed by 'table' for unqualed references, and another indexed by the tuple ('schema', 'table') for qualified references?

Yes. There are two structures with tables in them. One is schema_tables for qualified references and the other is visible_tables which is just a set() of all tables from all the schemas without a qualified reference.
So when someone types SELECT * FROM custom.u, we can look up all the tables in the schema named custom from the schema_tables dictionary and find the tables that start with 'u'. Or if the user types SELECT * FROM u without a schema name, we'll look up all the tables that start with 'u' in the visible_tables + all the schemas.
I'm not sure if I'm making this clear.
If you're constrained for time or have other priorities, let me know and I can build on top of the work you've done. I have touched it so far (apart from testing it locally) because I didn't want to duplicate any work you might have done.
BTW, thank you for talking through the design with me. It's been very helpful. :)

I'm thinking of a case like this:
> CREATE SCHEMA s1;
> CREATE TABLE s1.tbl (a INT, b INT);
> CREATE SCHEMA s2;
> CREATE TABLE s2.tbl (c INT, d INT);
> SELECT <autocomplete> FROM s1.tbl;

Now, you need to look up the columns so you can suggest (a, b) and not (c, d). There's a columns dictionary, but how is it indexed? It can't just be columns['tbl'], but needs to be something like columns[('s1', 'tbl')]... right?

Oh good catch.
Here's my current thinking. Instead of having multiple data-structures for schemas, tables and columns we have one data-structure. Let's call it metadata.
metadata is a nested dictionary like this:
{'schema': {'table': [columns,....] } }

We always maintain a state called default schema (initially set to 'public').
Now
> SELECT * FROM <tab>            # will autocomplete all the tables from the default schema.
> SELECT * FROM custom.          # will autocomplete all tables from 'custom' schema and columns from 'public.custom' table if it exists.
> SELECT <tab> FROM table        # will autocomplete all columns from 'table' in the default schema.
> SELECT <tab> FROM custom.table # all columns from 'custom.table'
> SELECT * FROM blah WHERE <tab> # all columns from a table 'blah' in the default schema.

The way to change the default schema is to use set search_path TO custom. That will make 'custom' to be the new default.
What do you think?

That would definitely work for the way I use postgresql :). The question is how many other people's weird usage patterns do you want to support?

search_path can be a list of schemas, not just one.

set search_path to custom, public, other_schema

Given (1), tables of the same name in different schemas on the search_path can shadow each other.

E.g. given tables s1.tbl and s2.tbl, SET search_path TO s1, s2; SELECT * FROM tbl will select all from s1.tbl, not s2.tbl. (Or maybe the other way around? I can't remember)

The default search_path can be set at the database or user level, and doesn't have to default to public.

Actually, at least on my local server, it defaults to "$user", public

I think these issues won't be a problem as long as we grab the search path as a list of schema names while grabbing the rest of the metadata, and search for unqualified tables by iterating over the search_path one schema at a time (so correct shadowing behavior is exhibited.)
One extra thing to keep in mind for the future is eventually I'd like to be able to support user-defined functions, especially table-returning functions, and even more especially, somehow hinting function argument names. I think that's a bunch of extra metadata that would probably be stored in separately from the table and column metadata, but I thought it was at least worth bringing up.

I don't use schemas very much. So I'm glad you're thinking of all these possibilities.
If I understand your correctly, if we make the default schema as a list, then we can stick with the nested-dict as the data-structure for metadata, right?
About the default search_path, do you happen to know how it's set at the user level? If that comes from the database server then can we read that in while starting up?
I think adding support for user-defined functions could be handled separately. We currently have built-in functions and keywords that are statically initialized, similarly we can populate a user-defined functions at startup time.
Hinting the args is possible but that'll require a bit more plumbing. We'll let future @darikg and @amjith  worry about that. :P

You do have a point about various ways of setting the default, but I can't think of another way to handle it. Even if we decide to go with a different data-structure we'd still need to support a way to complete unqualed references, which means we'll either show way more table names for auto-completion or none at all.
I'm sure there is a way to get the search_path from the server itself, so we can merely refresh the default path whenever we encounter a set command instead of parsing the path from the sql statement itself.

If I understand your correctly, if we make the default schema as a list, then we can stick with the nested-dict as the data-structure for metadata, right?

Yeah, exactly.

About the default search_path, do you happen to know how it's set at the user level? If that comes from the database server then can we read that in while starting up?

I know the query SHOW search_path returns a string containing a comma separated list of schema names. We could parse that (I guess taking special care for the edge case where a schema is double-quoted and contains a comma) in to a list of schema names. There might be a function in postgresql somewhere that returns a column of schema names instead, which would be nicer.

You do have a point about various ways of setting the default, but I can't think of another way to handle it. Even if we decide to go with a different data-structure we'd still need to support a way to complete unqualed references

Well, it's handled currently with the pg_catalog.pg_table_is_visible function, which does all the work for us. But then you'd either need to store the is_visible tag for each table (as in the current PR), or keep two redundant data structures (as we discussed upthread).

I think we can settle for using SHOW search_path and parsing the output right now using a new method called refresh_search_path().
We can trigger a refresh_search_path() when a set statement is entered. Let's punt on maintaining two different data-structures unless we run into a situation that demands it.

Sounds good. Turns out there's a slightly better way to get the search path:
select unnest(current_schemas(false)) which returns a column of schema names, instead of a single string.
Anyway, I can start implementing this, or if you'd prefer to, let me know.

If you can implement it, I'd very much appreciate it. But if you're swamped
let me know.
On Sat, Jan 24, 2015, 06:17 darikg<EMAIL_ADDRESS>wrote:

Sounds good. Turns out there's a slightly better way to get the search
path:
select unnest(current_schemas(false)) which returns a column of schema
names, instead of a single string.
Anyway, I can start implementing this, or if you'd prefer to, let me know.
—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-71318802.

Sure, I can start working on it. Does it make sense to keep working on this branch, or make a new pr from master?

Same branch. But merge in master to bring this branch up to date.
On Sat, Jan 24, 2015, 08:19 darikg<EMAIL_ADDRESS>wrote:

Sure, I can start working on it. Does it make sense to keep working on
this branch, or make a new pr from master?
—
Reply to this email directly or view it on GitHub
https://github.com/amjith/pgcli/pull/127#issuecomment-71324956.

Done!
todo:
[ ] Figure out a way to compare lists of dictionaries in test_sqlcompletion so the tests stop failing in python 3
[ ] Use itertools.groupby to avoid unnecessary name escaping on the same schema & table names over and over
[ ] Split test_smart_completion into two files, on for simpler tests on a public-schema-only database, and another for a more complicated structure
[ ] More schema-related tests in general

Done!
todo:

[ ] Figure out a way to compare lists of dictionaries in test_sqlcompletion so the tests stop failing in python 3
[ ] Use itertools.groupby to avoid unnecessary name escaping on the same schema & table names over and over
[ ] Split test_smart_completion into two files, on for simpler tests on a public-schema-only database, and another for a more complicated structure
[ ] More schema-related tests in general

Just tried it out. The completion works great. I haven't tested the path stuff yet, but I'll check it soon.
One tiny issue found so far is the completion for special commands need to be adjusted.
\d custom.<tab>

That should show only tables from the custom schema. But that could be a future enhancement.
Thank you for working on this feature. I'll read through the code later today.
Cheers!

The way I handled the \d command looks a little fragile, have a look and tell what you think

You've been very productive!
I pulled the changes and tried it out it does work as expected.
I looked at the code and you're right it's a wee bit fragile. I'm wondering why it wasn't handled by the elif condition below elif token_v.endswith('.'):.
I still haven't had time (stuck on a unicode issue #134) to review the full change-set but the initial perusal looks good. I'll give it a good look tonight.
Thanks!

I think it's something wonky with sending non-sql through sqlparse:
In [1]: from sqlparse import parse

In [2]: p = parse(r'\d xxx')[0]

In [3]: p.tokens
Out[3]: [<Error '\' at 0x345ec28>, <Identifier 'd xxx' at 0x4457f48>]

In [4]: p.tokens[-1].value
Out[4]: u'd'

This is quite the PR. :sparkles:
I appreciate the time you've put into this. Thank you very much!!
I've finally given it a thorough reading and I'm quite satisfied with all the changes. I'll come back to the suggestion for '.' at a later time.
This is good for merge. :shaved_ice:


Typecheck do_bench function in stable and beta

We cannot do bench in stable or beta because Bencher is available
only in nightly, but still enable do_bench function in stable or
beta to be able to do meaningful cargo check in stable or beta:

to typecheck the function
to mute unused imports warnings

(Had to switch to beta because intellij-rust is broken with nightly cargo)


Explicit publish publishes when its not supposed to

Describe the bug
Explicit publish will publish the page if there is anything in the publish portion in the frontmatter. The only way it will not publish is if the field is blank. Given the private pages of the documentation, I believe that it should only publish the page when publish is set to true.
To Reproduce

Changed config file to filters: [Plugin.ExplicitPublish()],
Added publish property to docs.
Added false to the field (since I thought it was a true/false thing)
Started a local server to test.
Found page as published even with a false property.
Changed publish to 9.
Refreshed the local website.
Page was still published.
Deleted the contents of publish.
Refreshed website.
Page was not published.

Expected behavior
The page should only publish if the publish property is set to true.
Desktop (please complete the following information):

Quartz Version: v4.3.1
node Version: v20.17.0
npm version: 10.8.2
OS: Win 11
Browser Chrome/Firefox

Additional context
N/A

Can you check that the publish frontmatter in Obsidian is a boolean(checkbox) field? "false" is truthy in JavaScript.


Configure Azure Pipelines

This configuration builds the application and produces builds for both Debug and Release configurations.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: eyeki sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


CORTX QUICK START GUIDE item 9 link is not valid

Submitting contributions link to demo video
404 NOT FOUND

Hey @justinzw, gentle reminder, you need to add the videos on vimeo to the quickstart guide.

This is automatically resolved here https://github.com/Seagate/cortx/blob/main/CONTRIBUTING.md as @johnbent edited the Contributing guide and merged it with Quickstart.md


add option to use index first in table scan

for #180

This change is

I will close this first.


Lecture 10 - missing data

links don't work - upon checking on server directory doesn't exist. Is it possible to put data in github?
In [4]
datapath = "http://thredds.atmos.albany.edu:8080/thredds/dodsC/cesm/"
ctrl = xr.open_dataset(datapath + 'som_control/som_control.cam.h0.clim.nc', decode_times=False)
co2 = xr.open_dataset(datapath + 'som_2xCO2/som_2xCO2.cam.h0.clim.nc', decode_times=False)
same for
In [15]:
ozone = xr.open_dataset( datapath + 'som_input/ozone_1.9x2.5_L26_2000clim_c091112.nc')

Thanks, I'm not maintaining this repo anymore because all the content has been repackaged in The Climate Laboratory, which will be kept up-to-date.
However I think the same path errors exist in the book version of those notes, which stems from some reorganization of our data servers. I will fix the book version.

HI Brian,thanks for the prompt response. Yes its in the book  as well.
Great set of lectures by the way - I've loved going through them
regards
Geoff
On Sunday, 17 May 2020, 05:53:11 pm GMT+1, Brian Rose<EMAIL_ADDRESS>wrote:
Thanks, I'm not maintaining this repo anymore because all the content has been repackaged in The Climate Laboratory, which will be kept up-to-date.
However I think the same path errors exist in the book version of those notes, which stems from some reorganization of our data servers. I will fix the book version.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

See the updated version in the book at https://brian-rose.github.io/ClimateLaboratoryBook/courseware/spectral-bands.html. All links now working.

Yes - its now working and will allow me to play with the inputs
many thanksGeoff
On Monday, 18 May 2020, 03:52:15 pm GMT+1, Brian Rose<EMAIL_ADDRESS>wrote:
See the updated version in the book at https://brian-rose.github.io/ClimateLaboratoryBook/courseware/spectral-bands.html. All links now working.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.


0.3.2 contains a breaking change

We're seeing breakage on Rust's CI for a breaking change in the 0.3.2 release:
[00:03:50] error[E0308]: mismatched types
[00:03:50]  --> /cargo/registry/src/github.com-1ecc6299db9ec823/cargo-0.32.0/src/cargo/util/sha256.rs:9:34
[00:03:50]   |
[00:03:50] 9 |         let hasher = Hasher::new(Algorithm::SHA256);
[00:03:50]   |                                  ^^^^^^^^^^^^^^^^^
[00:03:50]   |                                  |
[00:03:50]   |                                  expected reference, found enum `util::sha256::crypto_hash::Algorithm`
[00:03:50]   |                                  help: consider borrowing here: `&Algorithm::SHA256`
[00:03:50]   |
[00:03:50]   = note: expected type `&util::sha256::crypto_hash::Algorithm`
[00:03:50]              found type `util::sha256::crypto_hash::Algorithm`
[00:03:50]
[00:03:50] error: aborting due to previous error
[00:03:50]
[00:03:50] For more information about this error, try `rustc --explain E0308`.

It looks like a reference wasn't previously needed but now it is? Would it be possible to yank 0.3.2 and publish 0.3.3 with a fix?

0.3.3 has been released, without the API change.

Thanks so much!


chainer.links.ResNetXXLayers fails to init

I have found that chainer.links.ResNetXXLayers raises error when loading a caffemodel file.
Reproduction environment
This environment is built from scratch using vagrant to confirm my error.
However I get the same error also in my actual environment.

Ubuntu 16.04 x64 (vagrant)
Python 3.6.2 (pyenv)
Chainer 2.0.2
cupy not used
CUDA not used

Reproduction procedure
First prepare caffemodel.
# download ResNet-xx-model.caffemodel from github:KaimingHe/deep-residual-networks
$ md5sum ResNet-50-model.caffemodel
44b20660c5948391734036963e855dd2  ResNet-50-model.caffemodel
$ mv ResNet-50-model.caffemodel $HOME/.chainer/dataset/pfnet/chainer/models/ResNet-50-model.caffemodel

Then try to load the caffemodel via chainer.links.ResNet50Layers, you'll get the following error.
>> import chainer.links
>> chainer.links.ResNet50Layers()
Now loading caffemodel (usually it may take few minutes)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 331, in __init__
    super(ResNet50Layers, self).__init__(pretrained_model, 50)
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 112, in __init__
    pretrained_model, self)
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 717, in _retrieve
    lambda path: npz.load_npz(path, model))
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/dataset/download.py", line 151, in cache_or_load_file
    content = creator(temp_path)
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 716, in <lambda>
    path, lambda path: _make_npz(path, path_caffemodel, model, n_layers),
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 700, in _make_npz
    ResNet50Layers.convert_caffemodel_to_npz(path_caffemodel, path_npz, 50)
  File "/home/ubuntu/.pyenv/versions/3.6.2/lib/python3.6/site-packages/chainer/links/model/vision/resnet.py", line 147, in convert_caffemodel_to_npz
    chainermodel = cls(pretrained_model=None, n_layers=n_layers)
TypeError: __init__() got an unexpected keyword argument 'n_layers'

I got exactly the same error with both ResNet101Layers and ResNet152Layers as well as ResNet50Layers.

I think this is fixed in #3088.
Can you test with latest master?

Oh, sorry I haven't checked the latest code. One day late.
I confirmed now it's working.
Thanks


Test ReactNative

Currently we have some e2e tests (and some unit tests, though not enough yet) in place to ensure nothing breaks unexpectedly with changes.
These only test the web version of react though, we somehow need to make sure the changes between StyledNativeComponent and StyledComponent (and any other possible components) in sync. Any ideas how we could do that?

I could pick this up. Apart from unit tests (we'll probably want to use jest—#325), we might want to do device snapshot tests.
For snapshot tests, we'll need a RN project—should that be in the core styled-components repo, or should we make a new repo for it? I'm inclined to say new repo because it's easier, and because css-to-react-native will also affect the snapshot tests.
I have no idea how to make the CI work either way—if it'll even work with RN 😂

We test native now! :tada: https://github.com/styled-components/styled-components/blob/master/src/native/test/native.test.js


fix: add missing include path ../../**/* in generate tsconfig.json

🔗 Linked issue
fix #204

❓ Type of change

[ ] 📖 Documentation (updates to the documentation or readme)
[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
the generated tsconfig.json file in .nitro/types is missing "include path" of "../..//*", causing typescript unable to grab all global types defined in .nitro/types/nitro.d.ts.  adding "../..//*" to tsConfig.include array in nitro\src\build.ts should fix this issue.

📝 Checklist

[x] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

Thanks!


Helm "include calicoctl pod" boolean

Expected Behavior
When installing calico via helm, it would be handy if I could "flip a bit" to include calicoctl as a Kubernetes pod.  This would provide a clean way to bundle up a known calicoctl within the cluster and make getting started easier.
Current Behavior
After installing the helm chart, I've got to go fetch calicoctl.
Possible Solution
Add a boolean to values.yml that will load the "calicoctl as a Kubernetes pod" workflow
Steps to Reproduce (for bugs)

Install helm chart
notice a lack of calicoctl

Context

I'm trying to make it easier to get up and running from scratch with the calico operators
Your Environment

Calico version: 3.24.2
Orchestrator version (e.g. kubernetes, mesos, rkt): 1.25.2
Operating System and version: CentOS Stream 9
Link to your project (optional):

We want to move away from bundling calicoctl as a pod. There is a Calico API server that performs the same function as calicoctl as a pod and is the recommended way to interact with Calico APIs, and is installed by default when using the helm chart already.
What is your use-case for installing calicoctl on the cluster?

Mostly for looking at the IPAM.  I can live with my current workflow.


:bug: DepReport aggregated by name.

Update /analyses/report/dependencies to group by (provider, name) instead of (name, version, sha). Currently, provider is always "".
Removes DepReport fields:

version
sha

Also fixes DepAppReport.SHA JSON field name.
fixes: #557

Successfully tested with UI on https://github.com/konveyor/tackle2-ui/pull/1536.


Search box - default language is blank

The default language box is blank when clicking on Advanced Search, https://www.webonary.org/cerma/?lang=en

It looks like this one crashed during the import. The status box says it stopped before importing reversals, but the Browse Views tabs shows 2 reversal indexes, both of them French.

The Browse tab has the vernacular - Cerma and one reversal - French.

This dictionary was uploaded in 2014, so I'm not sure the compiler is available to upload again. Can you just fix this in this dictionary?

Looks good.


Enable OTEL tracing to capture diagnostics activity info from OPCUA server

Purpose
Enable OTEL support on Opc-plc to capture Tracing information from OPC UA Server.
Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

Test the code

What to Check
Verify that the following are valid

...

Other Information

This pull request is awaiting opc ua stack update nuget to be available.