[Please read the message below]

Please do not submit this issue.

[X] I understand

Hello, I am getting a security warning from Snyk because of the version of the ansi-regex library pretty-format uses.
Snyk Link
My pretty-format version "27.4.2" requires  "ansi-regex": "^5.0.1",
When can you fix this security issue.


Change Validator trait to use UpgradedContext

This is a minor change that does not affect a lot of code. The reason for making it is to make trait bounds easier for the upcoming PRF eval upgrade where the validator bounds need to be expressed in a form of upgraded context.
It is also easier to reason about because at the time when validator is created, the UpgradableContext is gone

I did something slightly different in #1208 for the DZKP types -- I removed the DZKPUpgradedContext associated type on trait UpgradableContext completely and changed the type parameter on trait DZKPValidator to an associated type.
I'm not sure if there's a reason to prefer one or the other, but it's probably worth making the two variants the same unless there's a good reason they need to be different.


Fixed Dropdown item icon spacing issue.

Fixed Dropdown item icon spacing issue.
Removed useless class name.

@joshsylvester , please help review.

You've resolved the problem for Chrome, Safari and Firefox, but IE and Edge need to be treated as well. Here's how they look like on your PR at the moment:
IE11:

Edge:

Note that both browsers above look normal on v4.9.0.
Sorry if I'm butting it and there's already plans to resolve this, I wanted to bring it up just in case.

From the comment(https://servicemax.atlassian.net/browse/UI-3074?focusedCommentId=176098&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-176098), it seems now we just deferred IE and Edge browsers fix.

@joshsylvester , addressed the comments, please help review again.

@benjaminliugang We discussed on our team and thought, the best way is to add a margin on the right as well, so all icons will render with space based on the direction of page rendering. I have another PR and have tagged you as well for reference

@benjaminliugang We discussed on our team and thought, the best way is to add a margin on the right as well, so all icons will render with space based on the direction of page rendering. I have another PR and have tagged you as well for reference

@benjaminliugang We discussed on our team and thought, the best way is to add a margin on the right as well, so all icons will render with space based on the direction of page rendering. I have another PR and have tagged you as well for reference

Added comment, please check.

Close this PR, the issue will be fixed by https://github.com/predixdesignsystem/px-dropdown/pull/104.


Monocular on EuRoc.

Is Kimera monocular supposed to give good results for the EuRoc dataset ? I ran "roslaunch chimera_vio_ros chimera_vio_ros_euroc_mono.launch" and the odometer just gave the drone going at one direction with big speed. It also took a while to initialise after the images were already being feed. I ran the stereo version with Kimera semantics and the results were very good. Is this result expected or it there any bug ?

I found the Issue for the Euroc Mono.. Maybe was an VIO initialisation problem so I set autoInitialise on BackEnd parm yaml to 1. Otherwise there is a waiting time fot the gt odom and maybe this was the problem


[ENH] handle panics in query service

This PR adds a test to confirm Tonic handles panics in the simplest cases for the query node.
Panics within components are untested (I suspect they wouldn't be caught by this).

Are we adding the panic hook in another PR?

yep 👍


octodns-providers.bind: 0.0.5 -> 0.0.6

Diff: https://github.com/octodns/octodns-bind/compare/v0.0.5...v0.0.6
Changelog: https://github.com/octodns/octodns-bind/blob/v0.0.6/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[X] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 288340 run on x86_64-linux 1

2 packages built:

octodns-providers.bind
octodns-providers.bind.dist


serialization updates

Added support to not send default values while sending the request.
Added support to populate entities with it's default value if it is present in the mapper while deserializing the response.
During deserialization, if the service does not provide the discriminator property then we set it. While setting the discriminator property, we compare model property name and the clientName of the polymorphicDiscriminator instead of the serializedName of the polymorphicDiscriminator.
Added tests for serializing and deserializing additional properties.

@bterlson, @daviwil - Please migrate the changes made in this PR to @azure/core-http as it fixes some important [de]serialization bugs.

Was able to catch some bugs that were then fixed in the second last commit in this PR, because we run autorest.typescript tests with the changes in this PR. This gives us confidence that we shouldn't be breaking anything by mistake. Please make sure to run autorest.typescript tests whenever changes are made to @azure/core-http once autorest.typescript moves to @azure/core-http.

Thanks!  I'll get core-http up to date with the latest changes of ms-rest-js soon.  I agree, it'd be good to run autorest.typescript's tests when core-http changes, I'll talk to our engineering systems team about it.

@daviwil - FYI, this azure devops config file for running autorest.typescript tests against the changes made in the runtime can be helpful.


render fails when trying to render youtube with easylist filter

http://localhost:8050/render.html?url=http://youtube.com&filters=easylist
{"info": {"text": "Protocol \"\" is unknown", "url": "", "type": "Network", "code": 301}, "type": "RenderError", "error": 502, "description": "Error rendering page"}``
rendering reddit.com works and easylist does apply to it.
http://localhost:8050/render.html?url=http://reddit.com&filters=easylist
easylist.txt

There are many easylist filters, which one are you using? Are you sure it doesn't filter out youtube? Because it looks like this is the problem in your case - filter contains youtube URL, so request to youtube.com fails.

(I use aquarium right now)
I got my easylist from
https://github.com/TeamHG-Memex/aquarium/tree/master/{{cookiecutter.folder_name}}/filters
then i changed it to
https://easylist-downloads.adblockplus.org/easylist.txt
both don't seem to render youtube for me.

I tried to render this page  with easylist filter, it fails with exact the same error.
2017-09-28 09:06:08.419028 [render] [140513949654992] loadFinished: RenderErrorInfo(type='Network', code=301, text='Protocol "" is unknown', url='')

The easylist I use is https://easylist.to/easylist/easylist.txt
If I don't use request filter, the above error would not happen (though there are still some js execution problem, but that should be another story).

@kmike  - I am also facing this issue. We are using Splash version 2.3.3
Kindly look into it.

I think the way to go is to find an offending rule in easylist.txt list (or in whatevel list you're using), and either remove it, or fix https://github.com/scrapinghub/adblockparser to handle it properly (if it is an issue with parsing).

@kmike  - I am confirming that this is not a case where an entry in the filters file is blocking the root URL.
 Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand.  Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%7D&v=2.8.1&o=12
2017-11-14 11:30:04.410672 [network-manager] Finished downloading https://images.images4us.com/888poker/en/600x338-RoadToRewards_PromoImage-1466416307425_tcm1488-286519.jpg
2017-11-14 11:30:04.413254 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-black/12.svg
2017-11-14 11:30:04.415644 [network-manager] Headers received for https://tr.outbrain.com/pixel?marketerId=004f3710fc709473dcf4ee98873de9a264&obApiVersion=1.0.4&name=888Poker-Site-Visit&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=04710129427257925
2017-11-14 11:30:04.418439 [network-manager] Headers received for https://sp.analytics.yahoo.com/sp.pl?a=10000&jsonp=YAHOO.ywa.I13N.handleJSONResponse&d=Tue%2C 14 Nov 2017 11%3A30%3A03 GMT&n=0&b=Poker Calculator \u2013 Texas Holdem Poker Odds Calculator&.yp=435730&f=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&enc=UTF-8
2017-11-14 11:30:04.419553 [network-manager] Headers received for https://sp.analytics.yahoo.com/sp.pl?a=10000&jsonp=YAHOO.ywa.I13N.handleJSONResponse&b=Poker Calculator \u2013 Texas Holdem Poker Odds Calculator&.yp=10018688&f=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&enc=UTF-8
2017-11-14 11:30:04.420442 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-black/12.svg
2017-11-14 11:30:04.422194 [network-manager] Finished downloading https://tr.outbrain.com/pixel?marketerId=004f3710fc709473dcf4ee98873de9a264&obApiVersion=1.0.4&name=888Poker-Site-Visit&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=04710129427257925
2017-11-14 11:30:04.422866 [network-manager] Finished downloading https://sp.analytics.yahoo.com/sp.pl?a=10000&jsonp=YAHOO.ywa.I13N.handleJSONResponse&d=Tue%2C 14 Nov 2017 11%3A30%3A03 GMT&n=0&b=Poker Calculator \u2013 Texas Holdem Poker Odds Calculator&.yp=435730&f=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&enc=UTF-8
2017-11-14 11:30:04.431970 [network-manager] Finished downloading https://sp.analytics.yahoo.com/sp.pl?a=10000&jsonp=YAHOO.ywa.I13N.handleJSONResponse&b=Poker Calculator \u2013 Texas Holdem Poker Odds Calculator&.yp=10018688&f=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&enc=UTF-8
2017-11-14 11:30:04.433165 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/0.svg
2017-11-14 11:30:04.434531 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/0.svg
2017-11-14 11:30:04.801206 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-black/8.svg
2017-11-14 11:30:04.803054 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-black/8.svg
2017-11-14 11:30:04.804960 [network-manager] Finished downloading https://www.888poker.com/poker/sites/default/files/inline-images/poker-pot-odds.jpg
2017-11-14 11:30:04.805924 [network-manager] Headers received for https://pixel.sitescout.com/iap/1d0d3c6f92ed977c?cookieQ=1
2017-11-14 11:30:04.806551 [network-manager] Finished downloading https://pixel.sitescout.com/iap/1d0d3c6f92ed977c?cookieQ=1
2017-11-14 11:30:04.940703 [network-manager] Headers received for https://www.facebook.com/tr/?id=399939243484685&ev=PageView&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003378&v=2.8.1&ec=0&o=12&it=1510659003151
2017-11-14 11:30:04.942510 [network-manager] Finished downloading https://www.facebook.com/tr/?id=399939243484685&ev=PageView&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003378&v=2.8.1&ec=0&o=12&it=1510659003151
2017-11-14 11:30:05.009467 [network-manager] Finished downloading https://www.888poker.com/poker/sites/default/files/inline-images/texas holdem.jpg
2017-11-14 11:30:05.068248 [network-manager] Headers received for https://www.facebook.com/tr/?id=510495282489251&ev=PageView&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003684&v=2.8.1&ec=0&o=12&it=1510659003151
2017-11-14 11:30:05.069836 [network-manager] Finished downloading https://www.facebook.com/tr/?id=510495282489251&ev=PageView&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003684&v=2.8.1&ec=0&o=12&it=1510659003151
2017-11-14 11:30:05.089225 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/1.svg
2017-11-14 11:30:05.090699 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/1.svg
2017-11-14 11:30:05.201951 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/4.svg
2017-11-14 11:30:05.203853 [network-manager] Headers received for https://www.facebook.com/tr/?id=399939243484685&ev=Microdata&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003883&cd[Schema.org]=%5B%5D&cd[OpenGraph]=%7B%22twitter%3Acard%22%3A%22summary%22%2C%22og%3Asite_name%22%3A%22www.888poker.com%22%2C%22og%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%2C%22og%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22og%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%7D&v=2.8.1&o=12
2017-11-14 11:30:05.205838 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/4.svg
2017-11-14 11:30:05.207903 [network-manager] Finished downloading https://www.facebook.com/tr/?id=399939243484685&ev=Microdata&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659003883&cd[Schema.org]=%5B%5D&cd[OpenGraph]=%7B%22twitter%3Acard%22%3A%22summary%22%2C%22og%3Asite_name%22%3A%22www.888poker.com%22%2C%22og%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%2C%22og%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22og%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%7D&v=2.8.1&o=12
2017-11-14 11:30:05.208769 [network-manager] Headers received for https://www.facebook.com/tr/?id=510495282489251&ev=Microdata&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659004194&cd[Schema.org]=%5B%5D&cd[OpenGraph]=%7B%22twitter%3Acard%22%3A%22summary%22%2C%22og%3Asite_name%22%3A%22www.888poker.com%22%2C%22og%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%2C%22og%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22og%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%7D&v=2.8.1&o=12
2017-11-14 11:30:05.210277 [network-manager] Finished downloading https://www.facebook.com/tr/?id=510495282489251&ev=Microdata&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&rl=&if=false&ts=1510659004194&cd[Schema.org]=%5B%5D&cd[OpenGraph]=%7B%22twitter%3Acard%22%3A%22summary%22%2C%22og%3Asite_name%22%3A%22www.888poker.com%22%2C%22og%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%2C%22og%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22og%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%2C%22twitter%3Atitle%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22twitter%3Aurl%22%3A%22https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator %22%7D&cd[Meta]=%7B%22title%22%3A%22Poker Calculator \u2013 Texas Holdem Poker Odds Calculator%22%2C%22meta%3Adescription%22%3A%22888poker\u2019s Poker odds Calculator is perfect for finding out where you stand in a hand. Learn when you\u2019re ahead or behind \u2013 with this poker hands calculator.%22%7D&v=2.8.1&o=12
2017-11-14 11:30:05.313880 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/5.svg
2017-11-14 11:30:05.316655 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/5.svg
2017-11-14 11:30:05.425621 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/6.svg
2017-11-14 11:30:05.427132 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/6.svg
2017-11-14 11:30:05.603875 [network-manager] Headers received for https://pixel.tapad.com/idsync/ex/receive?partner_id=2499&partner_device_id=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.604707 [network-manager] Finished downloading https://pixel.tapad.com/idsync/ex/receive?partner_id=2499&partner_device_id=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.605470 [network] [139674464718920] GET https://pixel.tapad.com/idsync/ex/receive/check?partner_id=2499&partner_device_id=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.607921 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/2.svg
2017-11-14 11:30:05.609245 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/2.svg
2017-11-14 11:30:05.611214 [network-manager] Headers received for https://bcp.crwdcntrl.net/map/c=1389/tp=STSC/tpid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.611913 [network-manager] Finished downloading https://bcp.crwdcntrl.net/map/c=1389/tp=STSC/tpid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.612408 [network] [139674464718920] GET https://bcp.crwdcntrl.net/map/ct=y/c=1389/tp=STSC/tpid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.620531 [network-manager] Headers received for https://dpm.demdex.net/ibs:dpid=82530&dpuuid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.621220 [network-manager] Finished downloading https://dpm.demdex.net/ibs:dpid=82530&dpuuid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.621748 [network] [139674464718920] GET https://dpm.demdex.net/demconf.jpg?et:ibs%7Cdata:dpid=82530&dpuuid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.715030 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/8.svg
2017-11-14 11:30:05.717149 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/8.svg
2017-11-14 11:30:05.725788 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/9.svg
2017-11-14 11:30:05.726946 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/9.svg
2017-11-14 11:30:05.760163 [network-manager] Headers received for https://pixel.tapad.com/idsync/ex/receive/check?partner_id=2499&partner_device_id=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.760951 [network-manager] Finished downloading https://pixel.tapad.com/idsync/ex/receive/check?partner_id=2499&partner_device_id=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.761503 [network] [139674464718920] GET https://match.adsrvr.org/track/cmf/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:05.901965 [network-manager] Headers received for https://dpm.demdex.net/demconf.jpg?et:ibs%7Cdata:dpid=82530&dpuuid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.904853 [network-manager] Headers received for https://bcp.crwdcntrl.net/map/ct=y/c=1389/tp=STSC/tpid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.906726 [network-manager] Finished downloading https://dpm.demdex.net/demconf.jpg?et:ibs%7Cdata:dpid=82530&dpuuid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.907573 [network-manager] Finished downloading https://bcp.crwdcntrl.net/map/ct=y/c=1389/tp=STSC/tpid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.915722 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/10.svg
2017-11-14 11:30:05.917184 [network-manager] Headers received for https://sync.teads.tv/um?eid=73&uid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:05.918635 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/10.svg
2017-11-14 11:30:05.920404 [network-manager] Finished downloading https://sync.teads.tv/um?eid=73&uid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:06.156245 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/3.svg
2017-11-14 11:30:06.157924 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/7.svg
2017-11-14 11:30:06.159154 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/12.svg
2017-11-14 11:30:06.160405 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/11.svg
2017-11-14 11:30:06.161393 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/3.svg
2017-11-14 11:30:06.163012 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/7.svg
2017-11-14 11:30:06.164779 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/12.svg
2017-11-14 11:30:06.166500 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/numbers-red/11.svg
2017-11-14 11:30:06.666873 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/css/888font/888-Regular.woff
2017-11-14 11:30:06.667733 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/poker-background-mobile.jpg
2017-11-14 11:30:06.867005 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/default.svg
2017-11-14 11:30:06.868490 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/default.svg
2017-11-14 11:30:06.878881 [network-manager] Headers received for https://match.adsrvr.org/track/cmf/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:06.880041 [network] [139674464718920] GET https://match.adsrvr.org/track/cmb/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:06.881756 [network-manager] Finished downloading https://match.adsrvr.org/track/cmf/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:06.891374 [network-manager] Headers received for https://loadm.exelator.com/load/?p=204&g=700&buid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:06.893738 [network-manager] Finished downloading https://loadm.exelator.com/load/?p=204&g=700&buid=f3ee5152-59d4-4711-ae75-7061aa393c6a
2017-11-14 11:30:07.043557 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/css/888font/888-Regular.woff
2017-11-14 11:30:07.045130 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/images/footer/PClogo-68x55-1487251650244_tcm1488-343920.png
2017-11-14 11:30:07.047198 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/images/footer/PClogo-68x55-1487251650244_tcm1488-343920.png
2017-11-14 11:30:07.219439 [network-manager] Headers received for https://match.adsrvr.org/track/cmb/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:07.220620 [network] [139674464718920] GET https://pixel.tapad.com/idsync/ex/receive?partner_id=1830&partner_device_id=94e7147c-7060-4e28-a4ac-70496d4ac74b&ttd_puid=%2C
2017-11-14 11:30:07.222387 [network-manager] Finished downloading https://match.adsrvr.org/track/cmb/generic?ttd_pid=tapad&ttd_tpi=1&ttd_puid=%252C
2017-11-14 11:30:07.398897 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/suit-sprite.svg?v=2
2017-11-14 11:30:07.401221 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/suit-sprite.svg?v=2
2017-11-14 11:30:07.403295 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/poker-background-mobile.jpg
2017-11-14 11:30:07.404074 [network-manager] Headers received for https://pixel.tapad.com/idsync/ex/receive?partner_id=1830&partner_device_id=94e7147c-7060-4e28-a4ac-70496d4ac74b&ttd_puid=%2C
2017-11-14 11:30:07.405199 [network-manager] Finished downloading https://pixel.tapad.com/idsync/ex/receive?partner_id=1830&partner_device_id=94e7147c-7060-4e28-a4ac-70496d4ac74b&ttd_puid=%2C
2017-11-14 11:30:07.546676 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/pattern.png
2017-11-14 11:30:07.548748 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/pattern.png
2017-11-14 11:30:07.735437 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/images/footer/NewSocialSprite-472xpc-1470052461630.png
2017-11-14 11:30:07.887208 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/question-mark.svg
2017-11-14 11:30:07.888520 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/question-mark.svg
2017-11-14 11:30:08.062386 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/images/footer/NewSocialSprite-472xpc-1470052461630.png
2017-11-14 11:30:08.238471 [network-manager] Finished downloading https://www.888poker.com/poker/sites/default/files/inline-images/2._10_steps_to_beating_freerolls-image-2-01-1473764199888_tcm1488-322185.jpg
2017-11-14 11:30:08.246144 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/images/footer/footer-changes-4-1472726752547.jpg
2017-11-14 11:30:08.247385 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/images/footer/footer-changes-4-1472726752547.jpg
2017-11-14 11:30:08.417832 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/css/888font/888-Black.woff
2017-11-14 11:30:10.135635 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/css/888font/888-Black.woff
2017-11-14 11:30:10.144979 [network] [139674464718920] GET https://cdn.inspectlet.com/inspectlet.js
2017-11-14 11:30:10.149847 [network] [139674464718920] GET https://tr.outbrain.com/pixel?marketerId=004f3710fc709473dcf4ee98873de9a264&obApiVersion=1.0.4&name=PAGE_VIEW&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:10.151604 [network] [139674464718920] GET https://amplifypixel.outbrain.com/pixel?mid=004f3710fc709473dcf4ee98873de9a264&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:10.220369 [network] [139674464718920] GET https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/reset-icon.png
2017-11-14 11:30:10.500074 [network-manager] Headers received for https://tr.outbrain.com/pixel?marketerId=004f3710fc709473dcf4ee98873de9a264&obApiVersion=1.0.4&name=PAGE_VIEW&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:10.506389 [network-manager] Finished downloading https://tr.outbrain.com/pixel?marketerId=004f3710fc709473dcf4ee98873de9a264&obApiVersion=1.0.4&name=PAGE_VIEW&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:10.569395 [network-manager] Headers received for https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/reset-icon.png
2017-11-14 11:30:10.574548 [network-manager] Finished downloading https://www.888poker.com/poker/themes/pokerlobby/poker-calc-assets/images/reset-icon.png
2017-11-14 11:30:10.738253 [network-manager] Headers received for https://cdn.inspectlet.com/inspectlet.js
2017-11-14 11:30:11.166772 [network-manager] Finished downloading https://cdn.inspectlet.com/inspectlet.js
2017-11-14 11:30:11.200627 [network] [139674464718920] POST https://hn.inspectlet.com/ginit/867654156
2017-11-14 11:30:11.212401 [network-manager] Headers received for https://amplifypixel.outbrain.com/pixel?mid=004f3710fc709473dcf4ee98873de9a264&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:11.214406 [network-manager] Finished downloading https://amplifypixel.outbrain.com/pixel?mid=004f3710fc709473dcf4ee98873de9a264&dl=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&bust=031644981005229056
2017-11-14 11:30:11.215669 [render] [139674464718920] loadFinished: RenderErrorInfo(type='Network', code=301, text='Protocol "" is unknown', url='')
2017-11-14 11:30:11.215884 [render] [139674464718920] loadFinished: disconnecting callback 0
2017-11-14 11:30:11.216115 [render] [139674464718920] [lua_runner] dispatch cmd_id=0
2017-11-14 11:30:11.216296 [render] [139674464718920] [lua_runner] arguments are for command 0, waiting for result of 0
2017-11-14 11:30:11.216485 [render] [139674464718920] [lua_runner] entering dispatch/loop body, args=(PyResult('return', None, 'network301'),)
2017-11-14 11:30:11.216655 [render] [139674464718920] [lua_runner] send PyResult('return', None, 'network301')
2017-11-14 11:30:11.216871 [render] [139674464718920] [lua_runner] send (lua) (b'return', None, b'network301')
2017-11-14 11:30:11.217172 [render] [139674464718920] [lua_runner] instructions used: 94
2017-11-14 11:30:11.217362 [render] [139674464718920] [lua_runner] caught LuaError LuaError('[string "function main(splash)\\r..."]:3: network301',)
2017-11-14 11:30:11.217776 [pool] [139674464718920] SLOT 0 finished with an error <splash.qtrender_lua.LuaRender object at 0x7f087ea69550>: [Failure instance: Traceback: <class 'splash.exceptions.ScriptError'>: {'source': '[string "function main(splash)\r..."]', 'error': 'network301', 'message': 'Lua error: [string "function main(splash)\r..."]:3: network301', 'type': 'LUA_ERROR', 'line_number': 3}
	/app/splash/browser_tab.py:519:_on_content_ready
	/app/splash/qtrender_lua.py:694:error
	/app/splash/lua_runner.py:29:return_result
	/app/splash/qtrender.py:19:stop_on_error_wrapper
	--- <exception caught here> ---
	/app/splash/qtrender.py:17:stop_on_error_wrapper
	/app/splash/qtrender_lua.py:2180:dispatch
	/app/splash/lua_runner.py:192:dispatch
	]
2017-11-14 11:30:11.217957 [pool] [139674464718920] SLOT 0 is closing <splash.qtrender_lua.LuaRender object at 0x7f087ea69550>
2017-11-14 11:30:11.218097 [render] [139674464718920] [splash] clearing 0 objects
2017-11-14 11:30:11.218289 [render] [139674464718920] close is requested by a script
2017-11-14 11:30:11.218958 [network-manager] Finished downloading https://hn.inspectlet.com/ginit/867654156
2017-11-14 11:30:11.220371 [render] [139674464718920] cancelling 0 remaining timers
2017-11-14 11:30:11.220579 [pool] [139674464718920] SLOT 0 done with <splash.qtrender_lua.LuaRender object at 0x7f087ea69550>
2017-11-14 11:30:11.222220 [events] {"error": {"error": 400, "description": "Error happened while executing Lua script", "type": "ScriptError", "info": {"source": "[string \"function main(splash)\r...\"]", "error": "network301", "message": "Lua error: [string \"function main(splash)\r...\"]:3: network301", "type": "LUA_ERROR", "line_number": 3}}, "active": 0, "_id":<PHONE_NUMBER>18920, "fds": 63, "load": [0.02, 0.04, 0.03], "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36", "path": "/execute", "args": {"png": 1, "har": 1, "load_args": {}, "resource_timeout": 0, "images": 1, "url": "https://www.888poker.com/poker/poker-odds-calculator", "wait": 0.5, "save_args": [], "uid":<PHONE_NUMBER>18920, "html": 1, "http_method": "GET", "timeout": 60, "filters": "hotjar", "lua_source": "function main(splash)\r\n  local url = splash.args.url\r\n  assert(splash:go(url))\r\n  assert(splash:wait(0.5))\r\n  return {\r\n    html = splash:html(),\r\n    png = splash:png(),\r\n    har = splash:har(),\r\n  }\r\nend", "viewport": "1024x768", "render_all": false, "response_body": false}, "maxrss": 232720, "timestamp":<PHONE_NUMBER>, "rendertime": 18.535261154174805, "qsize": 0, "status_code": 400, "client_ip": "<IP_ADDRESS>", "method": "POST"}
2017-11-14 11:30:11.222540 [-] "<IP_ADDRESS>" - - [14/Nov/2017:11:30:10 +0000] "POST /execute HTTP/1.1" 400 301 "http://localhost:8050/info?wait=0.5&filters=hotjar&images=1&debug=1&expand=1&timeout=60.0&url=https%3A%2F%2Fwww.888poker.com%2Fpoker%2Fpoker-odds-calculator&lua_source=function+main%28splash%29%0D%0A++local+url+%3D+splash.args.url%0D%0A++assert%28splash%3Ago%28url%29%29%0D%0A++assert%28splash%3Await%280.5%29%29%0D%0A++return+%7B%0D%0A++++html+%3D+splash%3Ahtml%28%29%2C%0D%0A++++png+%3D+splash%3Apng%28%29%2C%0D%0A++++har+%3D+splash%3Ahar%28%29%2C%0D%0A++%7D%0D%0Aend" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.89 Safari/537.36"
2017-11-14 11:30:11.222809 [pool] SLOT 0 is available

@kmike - As you can see from the debug log that the issue is coming way after the initial URL's are parsed and downloaded. So I dont think this is a case where filters are blocking the main URL

@kmike  - On further investigation , its found that there is a iframe inside  tag , which again calls an html page which has 2 iframe , 2 of which had src pointing to the links we have blacklisted
<!-- Google Tag Manager Start -->
  | <noscript><iframe src="//www.googletagmanager.com/ns.html?id=GTM-QSGL"
  | height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
  | <script>(function(w,d,s,l,i){w[l]=w[l]\|\|[];w[l].push({'gtm.start':
  | new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
  | j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
  | '//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
  | })(window,document,'script','dataLayer','GTM-QSGL');</script>
  | <!-- End Google Tag Manager -->
 ```

Content of www.googletagmanager.com/ns.html?id=GTM-QSGL

ns

```
We had blocked doubleclick.net in the filters.
Two question here -

Event with iframe disabled by default ( And manually disabling iframe from render.json API ) Its rendering iframe
If the Iframe is evaluated  and its src is filtered out , should we be seeing complete page blackout ?

@kmike  - So the Issue have narrowed down to the fact that if there is a iframe which has a src , which is filtered out by the the filter list , then the entire site render gives failure.
Is there a way to disable rendering of iframes ?

+1 on this issue,   It's causing my team a bunch of pain currently.    Wading through each of 4mb of easylist filters to find which specific one is the cause for every given case where this occurs (and it occurs alot when dealing with heterogeneous page sources) isn't a fair response, especially for what seems clearly an failure in Splash to not handle errors gracefully.
It's a bug in the filter handling, and should be addressed.

I found a workaround for the issue.   The trick is to change the url of the filtered request from being '' (an empty string) to be a valid url instead -- one that returns no data.
Specifically, I created an empty file and put it up at a public s3 url.
touch /tmp/empty.txt
aws s3 cp /tmp/empty.txt s3://static.[REDACTED].com/dev/null --acl public-read
# just to make make sure it's actually accessible
curl https://s3.amazonaws.com/static.[REDACTED].com/dev/null

And then in my /execute lua script,  I did the equivalent of the following:
 splash:on_request(function (request)
      if request.url == '' then
	 request.url = 'https://s3.amazonaws.com/static.[REDACTED].com/dev/null'
      end
end)

After which the network301 on the specific page I was fighting with that failed consistently when easylist filters were enabled cleared up completely.   I suspect it'll be true for the others that have been problematic.

@mlwelles Good trick! Maybe we can simply do a request.abort if url is empty?

@mlwelles Thanks for the solution! For anyone else trying out the solution, I found that request.url can only be used for reading the URL attribute but not setting. To set the URL I think you need the line:
request:set_url()

Is anyone working on fixing this, or could offer some tips on where one might start in the source to fix it? @mlwelles  approach works, but if you're paying for a proxy you end up paying (and waiting) for quite a lot of requests that are just getting an empty file. There doesn't seem to be a way to disable the proxy on individual sub requests via the splash:on_request handler either.

is there any updates about this issue?
rendering doesnt block ads while Splash loaded the filters files


[Bug][dataproxy-sdk-cpp]Data sending fails but dataproxy-sdk-cpp returns success

What happened
When reporting data to DataProxy through dataproxy-sdk-cpp, after DataProxy returns an error, dataproxy-sdk-cpp still counts successfully
What you expected to happen
counts failure
How to reproduce
sdk send a message,  dataproxy  force return a error, check result
Environment
Linux
InLong version
master
InLong Component
InLong SDK
Are you willing to submit PR?

[X] Yes, I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I'll handle it

Thanks


Support for mouse wheel left & right clicks

Is it possible to implement mouse wheel left/right click?

Maybe? I don't have a mouse that supports mouse wheel left and right clicks to do any testing. We'd just have to see what the OS exposes this as.

My bad, I'm closing this issue; my mouse is a logitech g502 and it actually emulates Scroll Left and Scroll Right with the mouse wheel side buttons.
But if ever this is implemented, here's a reading list:
https://www.autohotkey.com/docs/KeyList.htm
https://www.autohotkey.com/docs/Hotkeys.htm#HWheel
https://docs.microsoft.com/en-us/windows/win32/controls/wm-hscroll


Segfault on startJVM - Django 2.0 - Celery 4.2

Hey. I keep running into a segfault error using a Django application with Celery. The Celery process fires off a task using Jpype and hits a segfault attempting to start the JVM. I can run this process successfully outside of celery.
Here are the import versions:
Jpype1 == 0.6.3
Django == 2.0.7
Celery == 4.2.1
System: MacOS High Sierra
Firing off a task in the web application with the following code
print(jpype.getDefaultJVMPath())
print(jpype.isJVMStarted())
if not jpype.isJVMStarted():
    print("Starting JVM...")
    jpype.startJVM(
        jpype.getDefaultJVMPath(), '-ea'
        )

Gives me this output:
[2018-08-15 09:56:31,139: WARNING/ForkPoolWorker-8] /Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home/lib/jli/libjli.dylib
[2018-08-15 09:56:31,140: WARNING/ForkPoolWorker-8] False
[2018-08-15 09:56:31,141: WARNING/ForkPoolWorker-8] Starting JVM...
A fatal error has been detected by the Java Runtime Environment:
SIGSEGV (0xb) at pc=0x00007fff50c619d7, pid=36049, tid=515
... (There's a massive stack track that I'll avoid putting in for now).
Like I said earlier, outside of Celery, I can run JPype fine. If I enter the console separately and enter the above commands, the JVM works without issue.
I also created a separate test using only Celery and a task containing the code with JPype and that worked successfully. Somehow when it runs in the Django instance, I'm running into this issue.
I don't know if anyone else has encountered an issue similar to this. If I could get some pointers on what causes the seg fault issue generally, that may help me identify what might be going on. I'm betting it's some type of environment issue.
I appreciate any help / guidance you can give on this issue.

Barring attempting to run with tracing turned on I can't see much reason for a fail.   Perhaps there is an incompatible jvm being called.

I'll try using Java 8.0 and update with the results. Currently I'm on Java 10.

Java 10 is not currently covered by our test matrix.  Thus you may be encountering a new issue.

Just solved it by creating a docker container (ubuntu image) containing Java 8, celery and Django. So it was indeed an environment issue. Thanks for the help!


fix common attribute translation migration helper

escape the sql insert statements

Description
Please describe your pull request:

Why is it necessary? On the Showpare update, it caused an mysql error, because the sql query is not escaped.
What does it improve? The migriation prozess.
Does it have side effects? No.

Questions
Answers

BC breaks?
no

Tests pass?
yes

Related tickets?
http://issues.shopware.com/issues/SW-17440.

How to test?
Migrate translations with ' in column objectdata.

Hi @k-30, thanks for your PR and for submitting the issue. We will escape the values and are not going to use prepared statements in this migration for performance reasons. The multi-insert is much faster then separate prepared statements.
            $values = array_map(function($value) {
               return $this->connection->quote($value);
            }, $values);

Can you confirm that the above lines fix the issue for you?

Hi @bcremer ,
it doesn't work well:
Could not apply migration (Migrations_Migration799). Error: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ''('article', 'a:6:{s:10:"txtArtikel";s:50:"Blaklader work pants trousers c' at line 1

Please try this new function (replaces the old one):
private function getUpdatedTranslations($rows, $columns)
  {
      $values = [];

      foreach ($rows as $row) {
          try {
              $updated = $this->filter(unserialize($row['objectdata']), $columns);
          } catch (Exception $e) {
              //serialize error - continue with next translation
              continue;
          }

          if ($updated === null) {
              continue;
          }

          $row = array_map(function ($value) {
              return $this->connection->quote($value);
          }, $row);

          $updated = $this->connection->quote(serialize($updated));

          $values[$row['id']] = "({$row['objecttype']}, {$updated}, {$row['objectkey']}, {$row['objectlanguage']})";
      }

      return $values;
  }

The same problem...

That's interesting, that function works for our tests. Could you please make a dump of your s_core_translations-table and link it here so we can analyse the problem?

Hi @soebbing ,
the problem was solved, something was wrong in the dev environment. after a backup, your fix works now.

@k-30  Thanks again for helping us fixing this issue :+1:


Use configured TTL on keepAlive

This is necessary to avoid resetting the TTL on a lock record to the table default TTL.
For example, if the table has a default TTL of 60s and the LockFactory is configured with a default TTL of 30s, and you do:
lock.tryLock();
lock.keepAlive();
The TTL on the record will be 60s rather than 30s.
This patch fixes that by using the TTL passed into CassandraLock in the update.

@chrisribble Nice fix, thank you!


Add LOC/sLOC metrics

[ ] @projectwallace/css-analyzer needs to be updated to 2.5.0 for this
[ ] Add them to the Complexity section

Updating css-analyzer in https://github.com/bartveneman/wallace-cli/pull/57


Use modern dlpack interface in torch interoperability document

Because old interface is removed in numpy and sometime breaks compatibility of numpy and cupy
>>> t = torch.rand(10, 10)
>>> numpy.from_dlpack(to_dlpack(t))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'PyCapsule' has no attribute '__dlpack__'

Nice catch! New DLPack support has been in PyTorch for almost two years (https://github.com/pytorch/pytorch/pull/57110) so I think it's ok to drop usage of torch.utils.dlpack.to_dlpack.

/test mini


Network metadata updates on chainid.network

Fixes #979  .
Changes proposed in this PR:

networks metadata parameters fix
sorting between main and test networks fix

There is a typo for the GAIA-X Testnet

There is a typo for the GAIA-X Testnet
missed it, thanks!


RichTextLabel paragraph line separation messing up when resizing window due to oversampling

Godot Version: 4.1.2 Stable
OS: Mac OS Sonoma
Hello, I found this same error while using Godot 4.1.2 Stable, I don't know if I'm doing something wrong on my part but I wanted to share it in case this is an actual bug.
I'm using RichTextLabel
Oversapling is on, stretch mode set to canvas_items
I'm running Godot on Mac OS Sonoma, I thought the error could've been related to the HiDPI monitor but turning Allow HiDPI off in project settings didn't do nothing
I tried messing with most of the RichTextLabel settings, but nothing worked either
The only workarounds I've found are setting the stretch_mode to Viewport or disabling oversampling.
I'm using a .ttf font. Please let me know if more info is needed in order to have a clearer idea of what might be causing this.
It can be reproduced using a default RichTextLabel. I realized that using bigger fonts kinda masks the error, but with smaller fonts like this pixel art one you can really see it
https://github.com/godotengine/godot/assets/81658580/421305a7-ee12-4581-8bd5-cadc2528fc6c
Here the default font is at is default size (idk what that is lol) but if I change the default font size to something smaller like 8px it becomes as noticeable as the pixel art font:
https://github.com/godotengine/godot/assets/81658580/70812428-9493-4890-bec6-8684b03d356a

Same problem here.
https://github.com/user-attachments/assets/91d25e56-1cc0-4be3-8365-48e95dce4ffd
Disabling font oversampling solves the issue (but results in blurry text when displaying on higher resolution than the base project resolution)

FWIW this doesn't appear to be specific to RichTextLabel, as I'm seeing the same thing happen with plain Labels that just have multiple lines of text.


The mod doesn't work

I wrote the rules according to the documentation, but after disabling the mods and restarting the game, the mods still worked.I've tried a lot of things and it didn't work, maybe it's a bug.
Below is a screenshot.

发个加载了不该加载的模组的日志看看

暂时不支持 FabricLoader 0.16.1 及以上

https://modrinth.com/mod/mod-sets/version/1.8.0 应该修复了这个问题


fixing sidebar

🛠 Fixes Issue
Fixes: #768
👨💻 Description
What does this PR do?
fixing items alignement given in sidebar

Briefly describe the feature or bug fix you've implemented.
Highlight any major changes or added functionality.
Mention how it enhances the project.

📄 Type of Change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update (adds or updates related documentation)

📷 Screenshots/GIFs (if any)
Include screenshots or GIFs to demonstrate your changes

✅ Checklist

[ ] I am a participant of GSSoC-ext.
[ ] I have followed the contribution guidelines of this project.
[ ] I have made this change from my own.
[ ] I have taken help from some online resources.
[ ] My code follows the style guidelines of this project.
[ ] I have performed a self-review of my own code.
[ ] I have added documentation to explain my changes.

Mandatory Tasks

[ ] Make sure you have self-reviewed the code. A decent size PR without self-review might be rejected.

🤝 GSSoC Participation

[ ] This PR is submitted under the GSSoC program.
[ ] I have taken prior approval for this feature/fix.

@YadavAkhileshh no labels?

@YadavAkhileshh pls add the labels


[Feature Request]: VsCode: Go to definition for models

Feature Description
With VSCODE and itellephense not found definitions for models, for example:
//app/http/controllers/Api/ProcessController.php
namespace App\Http\Controllers\Api;
use App\Models\Center\Center;
$center = Center::where("uid", 1)->first();
$mySome = $center->getSome(); //No definition found for 'getSome()'

//app/models/Center/Center.php
namespace App\Models\Center;
class Center extends Model {
    public function getSome()
    {
        return "some";
    }
}

Go to definition "getSome()" on Controller not go

The feature you requested is already in progress, this will be available in the newer extension named Laraphense.
For now you can use PHP Intelephense extension and ide-helper package as dev dependency


Unable to synchronize wallet to chain: missing parent of first header

getbalance and gettransaction not work,,
hcd,  txindex  ,,from 1height

i guess，the wallet you using is too later。
attempt to use the latest code，delete old wallet and recover，


TRex doesn't do anything after a screen capture

I just got TRex from the app store and enabled screen capture rights for it. When I do a screen capture by clicking TRex in the menubar > "Capture Text", nothing happens, no window pops up or such. Did I misunderstand sth regarding the usage or is there a bug? I'm on macOS 11.3.1, MacBook Pro (13 Zoll, 2019, 4 Thunderbolt 3).

Hm, I just did a clean install from the App Store and it works as expected.
After you click on Capture Text you should see a crosshair, like one you see in macOS screenshot tool. You don't see it?
Can you check if you have anything in Console.app output relevant to TRex?

Sorry, I wasn't precise enough. I get the crosshair and I make a selection of text in an image but afterwards nothing happens.

Ah, that's the point, recognized text is copied to the clipboard.
In settings you can configure additional behavior, for example you can automatically open a link in Safari.
But by default it just puts the text into system clipboard.

OK, I'm embarrassed. Clearly just PEBCAC. Sorry for bothering you!

Perhaps you could add a word about this behaviour (the recognized text is only copied in the clipboard) in the README.md file.
Because when I see https://github.com/amebalabs/TRex/raw/main/Resources/demo.gif I expected a window with the recognized text as in the gif.
Thanks.

@quark67 Yes, I will clarify this in the Readme.

I expected a window with the recognized text as in the gif.

You are not the first one to assume that, In my nerdy head I just assumed that everybody knows this app(Tot)

You're right, I don't know this app (Tot).


Repository settings

Current version of doc settings does not mention that you can create so-called "repository settings" (source).
By repository settings, I mean .vscode/settings.json. This way you can override settings specific to that folder/repository.
Case regarding workspaces: I know you can use workspaces for this, but they have one disadvantage - you need to open workspace for settings to apply; in case of .vscode/settings.json, you can run code folder and settings.json is automatically applied.
Is this an actual omission in the official docs or am I misunderstanding something here?

These are also workspace settings. See https://github.com/microsoft/vscode/issues/77718#issuecomment-748807315 by @bpasero in a recent thread about ongoing confusion around the term "workspace".

@gjsjohnmurray Seems true but I have noticed the following difference in regards to workspaces:

if I save .code-workspace, I need to open .code-workspace for the workspace settings to be applied
if I have setttings.json to .vscode, the settings are applied automatically when I open the folder via code from command line

As a concrete example:

in the first case, I would do code .vscode/test.code-workspace
in the second case, I would do code .

Yes, see the linked issue for a very long discussion about how the term "workspace" is overloaded. When you have just opened a folder, "workspace settings" is the settings.json file that you have identified. Can also be called "folder settings"


The problem of the accuracy

Hello! I have tried to run the command line "sh run_vqa_PathVQA.sh" and then run "sh run_test_PathVQA.sh", but the total accuracy is 46.6%. I have tried it for many times, but the total accuracy I get is always around 46.6%.Can you please tell me how can I get the total accuracy on 'MMQ + MEVF' which is 49.0%?

Dear PengPeixi,
After reading your concern, I have cloned and re-run the codes by following all instructions in our git repo on two different hardware setups.
(1) Intel Xeon E5-2690 v4 CPU and Nvidia GeForce GTX 1080 Ti GPU 11Gb RAM.
(2) AMD Ryzen Threadripper 1950X 16-Core Processor CPU and Nvidia GeForce TITAN V GPU 12 Gb RAM.
The evaluation logs in the train set and the validate set, the pretrained models on PathVQA of both mentioned setups can be found in: (link)
The testing result in the PathVQA test set is 48.96% in setup (1) and 48.23% in setup (2), respectively.
The results in the test set still meet the claim in our previous issue response: the experimental error is approximately 1.58%.
We hope that some comments of us in this issue can help: https://github.com/aioz-ai/MICCAI21_MMQ/issues/2

Thank you very much for your reply! I will try to change the initial seed. And when I unzip or open the zip file which is downloaded from the link you supply, some errors occur. Could you please upload the zip file again?

Thank you very much for your reply! I will try to change the initial seed. And when I unzip or open the zip file which is downloaded from the link you supply, some errors occur. Could you please upload the zip file again?

I have updated the zip file. You can download and use them.

Thank you very much for your generous help!


how to use color plugin

I'm submitting a ...  (check one with "x")
[x ] Other, please describe

How do I use plugins, I need to use color plugin but I can't do this:
https://alex-d.github.io/Trumbowyg/documentation/plugins/#plugin-colors
<script src="node_modules/trumbowyg/dist/plugins/colors/ui/trumbowyg.colors.css"></script>
<script src="node_modules/trumbowyg/dist/plugins/colors/trumbowyg.colors.min.js"></script>

since node_modules is not distributed with my project. How can I make the color plugin available to the thrumbowyg editor withouth doing the above? Can I import it somehow inside my vue app?

Color Plugin example can be found here
https://github.com/ankurk91/vue-trumbowyg/blob/master/examples/App.vue

import 'trumbowyg/dist/plugins/colors/trumbowyg.colors';

  import 'trumbowyg/dist/plugins/colors/ui/trumbowyg.colors.css'

Yes you can import any plugin.


Fix subtable break

The PairPos subtable break was being forced after each 'pos' statement following a 'subtable' statement, it should only be forced the time.

Thanks Denis! You're right, the state of the builder needed to be reset after building each subtables!


Automated cherry pick of #10438

Cherry pick of #10438 on release-7.0.
/cc  @JulienTant
NONE

Will try to auto merge this PR once all tests and checks are passing. This might take up to an hour.


Memory leak in Internet Explorer after creating and destroying component

Version
2.6.11
Reproduction link
https://github.com/Feofilakt/vue-issue-repro
Steps to reproduce

Load reproduction project
Install cli-service-global (https://cli.vuejs.org/guide/prototyping.html) and run "vue serve"
Open site in Internet Explorer of latest version, open development tools and run memory profiler
Make heap snapshot
Check the checkbox
Uncheck the checkbox
Make heap snapshot

What is expected?
Memory of two snapshots must be about equal
What is actually happening?
Memory of two snapshots varies significantly

I'm afraid I didn't yet determine all the reproduction conditions for sure. If you cannot reproduce the bug, try increase the count in "for" loop, please.

When reporting memory leaks, please make sure to boil it down to the bare minimum in an HTML file (no cli, it also makes it easy to switch between dev and prod). Also, make sure it hasn't been reported before by searching for the term leak in issues.
If the leak can only be reproduced in one browser, it might be a browser bug too.  In the case of IE, they are known cases of memory leaks that haven't been addressed. It's better to show the problem in a modern browser such as Chrome, Firefox or Edge

Seems that Vue caches last created component so it still remains in memory after destroyed.

@vue boss, do we should give up Internet Explorer?


Node State Animator

Take Two nodes and can animate properties between
Can point required field or basically - all fields?

No useful


Write README

Update the README to reflect the new format

Updated now

Check over contents/chapter titles

Moved to in progress, update as we need to with links


Name of an Instagram user

Proposed Method
.name()
Description
Get the name of an Instagram user with the username provided.
To what class it belongs?
Users
Note

Make all the changes in the src/scrape_up/instagram/users.py file within the User() class.
Use a private method  __scrape_page to scrape the page.
Include the method in the documentation.md file.
Upload a screenshot with the PR, demonstrating the method.

Hi @nikhil25803 I am Gssoc 2023 contributor and would like to work on this issue please do assign it to me .

Go ahead @kruti002

Isn't it a violation to web scrape a name according to Instagram?

Hi @nikhil25803 I would like to work on this issue please do assign it to me .

Hi ,
Aditya Bagla ( GSSOC'23 contributor )
I would like to contribute on this issue . Can you please assign me ?


Link styling in FAQ is off

Link styling in FAQ should look like inline clickable text.
Currently instead it looks like this:

might be related to #159 ?

Maybe. It will need fixing anyway.


Layer controls missing on File -> New -> Ansimation

Layer UI controls are missing when using File -> New -> Ansimation.

Also missing preview, char table and channels:

Closing the Ansimation document brings back the controls properly with the other file remaining open, etc.

Setting to tabs should work.

Should be fixed.


feat: update fee calculation

Description
Current fees for a TX are set too low. Tune the costs.

Transfer costs 0.0047 $ (the same as in Basilisk)
Full block: 395 $ (393 in Basilisk)
Fee multiplier after one hour of operation with fully loaded blocks: 1.0135 (the same as in Basilisk)
After one day: 1.3826 (the same as in Basilisk)


HTTP serving implementation addressing #26

This is the HTTP implementation of the serving side outlined in #26. The updating example that goes with this PR uses this capability and does simplify the creation of HTTP services.

Closing this one, to be replaced by a common HTTP and gRPC interface


Metadata about artworks belonging to series is missing

On MoMA website I see that artworks can belong to a series. But this metadata is not available in the data dump here. Could it be added?

Great question! Please email<EMAIL_ADDRESS>to request, and we can look at including. It's a bit more complex data since artworks can belong to multiple series (and for various reasons we only show one on the website).


Using dbgeng the Debugger launches the program with unexpected current directory causing EXE launch to fail

Is your feature request related to a problem? Please describe.
Using python + dbgeng as the debugging setup, the .exe is launched with Ghidra/Debug/Debugger-agent-dbgeng/data/debugger-launchers as the current directory for the process. This is unexpected, as typically, the current directory for a process is either the path to the folder containing the .exe, or the path of the calling process. Since the program is launched within debugger-launchers, an .exe fails to launch if it expects files and folders relative to the .exe location.
Describe the solution you'd like
I would like the solution that we can specify which directory the program will be run in.
Describe alternatives you've considered
Manipulating the exe to set the current directory to the folder the .exe is in. Hacky at best.
Additional context
I am on 11.1, not sure if it has been addressed since then, I can't find any mention of it on GitHub. I searched using Debugger Current Directory

@gynt If you use the extended version of the launcher, you can set the “Initial Directory”.

@gynt If you use the extended version of the launcher, you can set the “Initial Directory”.

@d-millar How can I use the extended version? Where is it located? Or is that an 11.2 feature?

Yes, sorry - 11.2.  The launcher is listed as “dbgeng-ext”.


[Python][Rust] Create extension point in python for Dataset/Scanner

Describe the enhancement requested
As the Arrow ecosystem grows ever richer, desire paths emerge :)
Integrating Arrow based projects written in Rust works great across the C data interface. But it doesn't allow lazy execution or pushdowns in the same way that pyarrow Dataset/Scanner's do.
My proposal here is to expose Dataset/Scanner python abc's with s.t. rust libraries can extend via pyo3+python so higher level tooling (like duckdb for example, can query these without having to transfer the whole Table into memory first).
In keeping with the same principles as the C data interface, I think it would be sufficient for this python interface to be very minimal: Dataset with schema and scanner methods, Scanner with projected_schema and to_reader methods. The to_reader should return a RecordBatchReader which would then link pyo3 datasets into the Arrow C data interface.
Thanks for your consideration!
Component(s)
Python

to be precise, this is already doable today as a workaround, what i'm proposing here is:

formalize this extension point
if possible, make the top level abstraction pure python, so subclasses doesn't need to deal with cython etc if coming from Rust

From a quick read, is this about formalizing an extension point per se or actually just an interface? Some way to say, "given a filter/column selection, please try to push these down and give me a C Data Interface stream"

I've had similar challenges with supporting datasets in delta-rs. Another aspect you'll need to think about is supporting Filesystems. In Rust, that means calling into Python functions, which I fear can be sub-optimal because of the GIL. I don't think there's a practical way to directly access the underlying C++ implemented FS unless we made the ABI stable (which I don't see us doing in the foreseeable future).
A route I'm exploring right now is using the ADBC as a stable ABI for pushing down scan queries to storage formats and systems. It probably make more sense for table formats like Delta Lake, which have database-like semantics, than file formats like Lance (which I assume is the projection with the use case you are discussing).

if possible, make the top level abstraction pure python, so subclasses doesn't need to deal with cython etc if coming from Rust

It's harder, but part of me would prefer a stable C ABI, because it would mean the extension could be used in any language, not just Python.

From a quick read, is this about formalizing an extension point per se or actually just an interface? Some way to say, "given a filter/column selection, please try to push these down and give me a C Data Interface stream"

You're right. Good question. I think just the interface actually, so this issue is incorrectly named. Ie Rust packages could implement the same interface that higher level tooling will call, without necessarily needing to inherit from the cython classes defining Dataset/Scanner today.

I've had similar challenges with supporting datasets in delta-rs. Another aspect you'll need to think about is supporting Filesystems. In Rust, that means calling into Python functions, which I fear can be sub-optimal because of the GIL. I don't think there's a practical way to directly access the underlying C++ implemented FS unless we made the ABI stable (which I don't see us doing in the foreseeable future).

Does object store rs work for this?

A route I'm exploring right now is using the ADBC as a stable ABI for pushing down scan queries to storage formats and systems. It probably make more sense for table formats like Delta Lake, which have database-like semantics, than file formats like Lance (which I assume is the projection with the use case you are discussing).

if possible, make the top level abstraction pure python, so subclasses doesn't need to deal with cython etc if coming from Rust

Yeah most def pure python

It's harder, but part of me would prefer a stable C ABI, because it would mean the extension could be used in any language, not just Python.

It sounds like what you basically want is a
struct DataFrameProducer {
    int read(struct DataFrameProducer* self, ??? filters, ??? selection, struct ArrowArrayStream* out);
};

with corresponding wrappers/carriers in Python, Rust, Go, etc. Then this can be fed into DuckDB, Ballista, Acero, etc. and can be produced by ADBC, Acero, DuckDB, etc.
You can have nearly both "pure Python" and "C ABI", I think. A Python-level interface could be transformed by PyArrow into the C ABI and vice versa. Where possible, the Python-level interface should let you 'extract' the underlying C ABI, if it exists, but otherwise we can push the responsibility of the GIL and such into PyArrow (or something like that). (So basically, shove all the non-Python code into PyArrow.)
The question is what the filter/selection format should be; ideally it would be language agnostic and implementation agnostic and so Dataset's expressions aren't great there.

The question is what the filter/selection format should be; ideally it would be language agnostic and implementation agnostic and so Dataset's expressions aren't great there.

Substrait recently merged a message type for standalone expression lists. :)
https://github.com/substrait-io/substrait/pull/405

What if we created a read type in substrait for datasets?  It would require encoding file system options in Substrait as well.  However, this seems like a valuable feature.
Then the C API / interface is just the typical substrait consumer API of  "substrait plan in, record batch reader out"

I have been recently thinking about this as well, triggered by our work to support the DataFrame Interchange protocol in pyarrow (https://github.com/apache/arrow/pull/14804, https://data-apis.org/dataframe-protocol/latest/purpose_and_scope.html).
For some context, this "interchange protocol" is a python specific one to allow interchanging dataframe-data, typically between different dataframe libraries (eg convert a dataframe of library1 into a dataframe of library2 without both libraries having to know about each other). Essentially, this is very similar to our C Data Interface (it specifies how to share the memory, and in the end also has pointers to buffers of a certain size, just like the C struct), but because it is a python API, it gives you some functionality on top of those raw buffers (or, before you get to the buffers).
As illustration, using latest pyarrow (using a pyarrow.Table, so all data is already in memory, but we could add the same to something backed by a not-yet materialized stream or data on disk):
>>> table = pa.table({'a': [1, 2, 3], 'b': [4, 5, 6]})
# accessing this interchange object
>>> interchange_object = table.__dataframe__()
>>> interchange_object
<pyarrow.interchange.dataframe._PyArrowDataFrame at 0x7ffb64f45420>
# this doesn't yet need to materialize all the buffers, but you can inspect metadata
>>> interchange_object.num_columns()
2
>>> interchange_object.column_names()
['a', 'b']
# you can select a subset (i.e. simple projection)
>>> subset = interchange_object.select_columns_by_name(['a'])
>>> subset.num_columns()
1
# only when actually asking for the buffers of one chunk of a column, the data needs
# to be in memory (to pass a pointer to the buffers)
>>> subset.get_column_by_name("a").get_buffers()
{'data': (PyArrowBuffer({'bufsize': 24, 'ptr':<PHONE_NUMBER>75104, 'device': 'CPU'}),
  (<DtypeKind.INT: 0>, 64, 'l', '=')),
 'validity': None,
 'offsets': None}

To be clear, I don't propose we do something exactly like that (and personally, I think it's also a missed opportunity for the DataFrame Interchange protocol to not use Arrow for the memory layout specification, and not give access to data as arrow data or using the Arrow C Interface).
But it has some nice properties compared to our "raw" C Data Interface. It allows to first inspect the data (columns, data types) before you actually start materializing/consuming the data chunk by chunk, and allows you to get the data of only a subset of the columns. While when using the C Stream for a Table or generically RecordBatchReader, you directly get the struct with all buffers for all columns. So if you don't need all columns, you need to ensure to first subset the data before accessing the C interface data, but this means you need to know the API of the object you are getting the data from (partly removing the benefit of the generality of the C Interface).
So having some standard Python interface that would give you delayed and queryable (filter/simple projection) access to Arrow C Interface data sounds really interesting, or having this as an additional C ABI (for which we can still provide such a Python interface as well) like David sketched above.

From an implementation perspective I suspect we can satisfy any of these proposed APIs.  If we need to come up with a new API then my preference is Substrait, but if the consensus heads in some other direction I'm fine with that too.
@jorisvandenbossche , your proposal seems fine, but I don't see anything in there for filesystems.  I think this is for on-disk data moreso than purely in-memory data.  Though I believe your approach could be adapted to include filesystems.

Does object store rs work for this?

Yes, I would assume that object store rs would be able to satisfy this but I'm not familiar with the capabilities.  For example, my idea of how this would work in Substrait would be:
# This would be usable as ReadRel::read_type
message Dataset {

  # This is already definedin ReadRel and is basically a list of files
  # and a format object which defines things like delimiter (for CSV)
  LocalFiles files = 0;
  oneof filesystem {
    LocalFilesystem = 1;
    S3Filesystem = 2;
    ExtensionFilesystem = 3;
  }

  message LocalFilesystem {}
  message S3Filesystem {
    string region;
    string client_id;
    string client_secret; // could be omitted if credentials negotiated elsewhere
  }
  message ExtensionFilesystem {
    google.protobuf.Any details;
  }

}

The equivalent C interface would just be structifying those messages.

What's the reason that the potential interface would need to be aware of filesystems?

I think the goal is to create a C API capable of representing datasets?  I think that would require filesystems.  But I may be mistaken on either of these points.

The interface is for scanning the dataset, which is after the filesystems have been passed. So it's a separate concern. Yet it is still relevant to "how do I extend dataset" because your scanning implementation needs to use some filesystem. And that means the user needs to configure and pass one in.
The easiest for users is to take fsspec / PyArrow filesystems as Python interfaces, although performance may be impacted by the GIL. (I have started, but not finished, an implementation of ObjectStore in Rust that wraps fsspec filesystems here.) Or you can allow configuring a native filesystem, but then it's another API users have to learn.

wants to take advantage of the API of the former to get access to things like DuckDB integration

IMO the current DuckDB integration feels a little silly. It manipulated Python objects until it can get a RBR and the exports that through the C data interface. The same code is duplicated in the R package, except it manipulates R objects. And nothing is available in other languages. So part of me thinks it would be cleaner to replace that integration with this kind of C API, but that's for the DuckDB devs to decide :)
So there's sort of two questions:

How do I extend Dataset from a separate package, particularly if implemented in Rust? This is where the filesystem API / configuration stuff comes in.
How do I consume Dataset from a separate package? This is where the DuckDB integration comes in.

It's unclear to me rn whether we just want to create a C API you can use instead of Dataset to solve this, or make a C API on Dataset to solve these. The former is less complicated for sure, but not sure we want to sidestep the Dataset API like that.

I don't think it's either-or, I think we're tangling two concerns up here.

My proposal here is to expose Dataset/Scanner python abc's with s.t. rust libraries can extend via pyo3+python so higher level tooling (like duckdb for example, can query these without having to transfer the whole Table into memory first).

Dataset is sort of an API standard for this, or at least you can press it into service. But Dataset is also useful in its own right and a meaningful abstraction, so people want to extend it.
If we define a new API, of course Dataset should implement it!

How do I extend Dataset from a separate package

Dataset is already extensible. The real problem is the Python integration, and wheels/packaging questions on top of that. The toplevel proposal sounds like sidestepping that entirely by introducing a separate abstraction layer at the Python level (hence, exposing ABCs in Python).

IMO the current DuckDB integration feels a little silly.

I don't think they had a choice, because there's not really a formal API for what they really want :)

The toplevel proposal sounds like sidestepping that entirely by introducing a separate abstraction layer at the Python level (hence, exposing ABCs in Python).

Yup, that's exactly the proposal here.

I don't think they had a choice, because there's not really a formal API for what they really want :)

The main blocker in the current version of DuckDB is using the static methods in Scanner.from_dataset and similar. I made a PR to change that to use the instance method dataset.to_scanner. So next release it will be possible for Rust packages to disguise themselves as pyarrow datasets to DuckDB.
The issue is that you have to be really careful to override all of the methods in Dataset, or else it'll try to unwrap the non-existent CDataset and crash python. This is the main motivation for me proposing a pure python abstraction on top.
Would y'all be open to accepting a PR for this? or is there a more formal process to propose some details?

Others should chime in, but I think the best way to move forward is if you could open a draft PR with the proposed ADCs. Then we could discuss on the PR. (You could also create a design doc in a Google doc, but I think seeing the actual classes might be more useful in this case.)

ok sounds great. I should have time tmr to make the PR and that should give others time to chime in too. Thanks!

I'm suddenly rather interested in seeing this through. Also have had a change of heart and think either what Chang is proposing (an ABC) or Joris (a protocol) is the way to go. ABC seems straightforward, but I'm eager to chat with Joris if he has ideas on why a protocol like the DataFrame protocol makes more sense. (Or maybe both could be combined? The protocol returns something that subclasses the ABC?)
Starting thinking about this in a Google doc: Making Arrow dataset into a protocol
Also wrote up another doc to share the perspective of delta-rs/deltalake on PyArrow Datasets: PyArrow Datasets and Python deltalake

Thanks for reviving this and the write-up, Will!
(coincidentally, I was finishing up an earlier write up yesterday about a protocol for just the C Data Interface (so without more dataset-like capabilities like predicate/projection pushdown), and while there is potential overlap, opened the separate issue I was writing anyway: https://github.com/apache/arrow/issues/35531)
What is still not clear to me (also after reading the doc) is the exact scope or goal of a potential protocol (where is exactly the extension point?) Some possible ways I could interpret it:

An interface for consuming data from a dataset-like object, without having to be a pyarrow.dataset.Dataset (or Scanner) instance.

This is closer to "just" exposing an ArrowArrayStream, but with additional capabilities to inspect the schema and predicate/projection pushdown (before actually consuming the stream)

An interface to describe a dataset source (fragment file paths, filesystem, ..) such that any dataset implementation can read data from a source specified with this interface

Would this essentially be like a substrait ReadRel ? (in terms of information that it would need to capture, maybe with additional things like fragment guarantees/statistics).

An extension point (or ABC) specificially for the pyarrow.dataset implementation such that you can define/implement a dataset source without having to extend Arrow Dataset at the C++ level, but still plug it into a pyarrow Dataset/Scanner, such that you can make use of some aspects of the arrow implementation (for example, if you attach guarantees to your fragments, then you can rely on the Arrow Dataset implementation to handle the predicate pushdown) or use consumers that already have support for pyarrow Datasets. This is basically trying to make Arrow C++ Datasets more easily extensible.

Using the example of deltalake to duckdb, the three options would like like:

The user creates a deltalake.DeltaTable object, which exposes a stream of data through some protocol. This table object is passed to a duckdb method, which no longer checks hardcoded for a pyarrow.Table/Dataset/Scanner, but checks for the generic protocol method being available. From that method, it can basically get the equivalent of a pyarrow Scanner(filter, projection) -> RecordBatchReader -> C Stream (but without hardcoding for the pyarrow APIs). The actual reading of the data itself is fully done by the deltalake implementation.
The user creates a deltalake.DeltaTable object, which is passed to duckdb. Duckdb gets the information from this object about what to read (which files to read from where), but then reads it themselves. The actual reading here is done by a different library than the one that specified the source.
The user lets deltalake create a pyarrow Dataset object on top of a deltalake fragment/scanner implementation. This object is passed to duckdb, and duckdb uses its current pyarrow integration to consume this data. The actual reading of the data is done by both deltalake and pyarrow.dataset (Arrow coordinating things for reading fragments and streaming that data, but the actual reading is done by the deltalake fragment implementation)

What is other's people understanding of what we are discussing / are looking for?
I suppose the two questions from https://github.com/apache/arrow/issues/33986#issuecomment-1416418936 overlap with this, and essentially ask the same.
But for example, the google doc currently mentions "filesystems" as one of the challenges, but if it's option 1 that we are interested in, I still don't fully understand how filesystems are involved (the filesystem (interaction) is fully defined by the producer (deltalake, or user of deltalake that created the deltalake Table object), and once you are reading data from that source (duckdb), you don't have to be aware of the filesystem details?)

Yes, and to be clear: I think what I would find most interesting is (1).

Thanks, this is all helpful feedback. I'm going to spend some time filling out that document more. I'll ping this thread when it deserves some more attention.

Nitpicking here but in python terms we are interested in Protocols and not ABC, right? For a generic lib a static typing interface sounds better than an abstract class (which is a runtime dependency).
This explains it much better than me: https://jellis18.github.io/post/2022-01-11-abc-vs-protocol/

I have updated the document and created a rough sketch. I've also notified some devs from other projects, such as PyIceberg and dask-deltatable, to get more feedback.
Basically, I think the API that we have now for Datasets is actually very good. So doing as Chang originally suggested and just making a typing.Protocol out of it seems like it would be sufficient. I think that's what we want, but I'm honestly not 100% sure the best way to expose / publish this, so I would welcome feedback on that.
There are some possible extensions of it that could be made in the future, but I don't think they should block us from defining a protocol now.
IMO, this is a good opportunity to define something that will work well enough for now. I don't think it will be something that will last the next 5-10 years. But what we learn from pushing this API to it's limits may inform us on the design of something that's more robust and includes input from a much wider part of the PyData ecosystem.

I think @westonpace has already mentioned the high level steps that we have to take.
The key is to find out a way to access the table defined in the namedTable and query the required metadata from the APIs provided by the consumers. We should also assume that the expected access is handled by each consumer itself. That is the only concerning part as far as I understand.

@westonpace would it make sense to add a to_substrait() function to the DataSet API?

@wjones127 do you think it's time to start an ML discussion?
I wonder if we could publish a base API in the Nanoarrow Python code, based on PyCapsule, and also a higher-level version in PyArrow, based on the PyArrow objects.

@lidavidm Thanks for the nudge. ML thread here: https://lists.apache.org/thread/ko0j6pk86p5rt24w6s3m40h68r6lcqrr


Bad (or good) behaviour by first parties

Section 2.13 “How does this specification distinguish between behaviour in first-party and third-party contexts?” does not consider that first parties can be a threat to privacy and security. It assumes only third parties can be bad.
Section 3.4 “3.4. Third-Party Tracking” only considers tracking associated with third parties. It should consider tracking by any party.
First parties have the potential to perform harms just as much as third parties. The document should be expanded to reflect this.

Hello,
Thanks for the feedback.

Section 2.13 “How does this specification distinguish between behaviour in first-party and third-party contexts?” does not consider that first parties can be a threat to privacy and security. It assumes only third parties can be bad.

This question is very specific and tries to understand how features distinguish between behavior in different modes of operation. It does not imply that "first parties" cannot be a threat.

Section 3.4 “3.4. Third-Party Tracking” only considers tracking associated with third parties. It should consider tracking by any party.
First parties have the potential to perform harms just as much as third parties. The document should be expanded to reflect this.

I always thought this is implicitly understood, after all - when a user grants e.g. permission to use a specific feature, the first party is in control of something powerful.

I tend to agree with @jwrosewell , the way the document is laid out there is an implication that only 3rd party tracking is nefarious.  A more balanced document would have a heading "Tracking" with two sub sections, "third party tracking" & "first party tracking".  Each section should contain a description of the limits or rules each party operates under.

I tend to agree with @jwrosewell , the way the document is laid out there is an implication that only 3rd party tracking is nefarious. A more balanced document would have a heading "Tracking" with two sub sections, "third party tracking" & "first party tracking". Each section should contain a description of the limits or rules each party operates under.

Balanced? Rather than arguing for a specific point, the document's core is not about tracking, but web features and the risks they can bring. Risks can obviously exist both in first and third party context, though.

Specifically regarding tracking, I think there is a substantial difference between first-party and third-party tracking because they differ substantially in:

how much they differ from user expectations
how visible they are to the user.

For example, if somebody goes to facebook.com, they probably expect that facebook.com knows something about their visit, what they click on on the page, etc.  It's also relatively clear that facebook.com is what they're interacting with.
On the other hand, if they go to read a news article on reuters.com, they are much likely to have these expectations about what facebook does to track what they're doing.  A user doing so does not naturally expect facebook to track what they're doing on reuters.com, nor is it clearly visible to the user that facebook is doing so.

If we're designing for a world where someone can visit reuters.com (other publishers are available) and trusts Thomson Reuters then it is up to Thomson Reuters to decide who Thomson Reuters do and do not trust. If Thomas Reuters abuse the trust someone placed in them because a third party committed a bad act there will be sanctions for Thomas Reuters. This is a concept encapsulated in GDPR very effectively.
I don't believe the W3C should be designing standards or assessing proposals which remove these trust choices from people or publishers. Therefore the differentiation between first parties, third parties, and browser vendors implementing W3C standards needs to be considered further.
The document could be expanded to consider the identification and reporting of bad acts to support the sanctions provided for under law no matter the role of the parties.

If we're designing for a world where someone can visit reuters.com (other publishers are available) and trusts Thomson Reuters then it is up to Thomson Reuters to decide who Thomson Reuters do and do not trust. If Thomas Reuters abuse the trust someone placed in them because a third party committed a bad act there will be sanctions for Thomas Reuters. This is a concept encapsulated in GDPR very effectively.

While an interesting concept and discussion, I'm not quite sure such discussion is at the core use goal of the document.

I don't believe the W3C should be designing standards or assessing proposals which remove these trust choices from people or publishers. Therefore the differentiation between first parties, third parties, and browser vendors implementing W3C standards needs to be considered further.

This document does not "remove trust choices from people or publishers". It's a questionnnaire intended for feature developers. That said, no doubt that first parties can be involved in abusive behavior, of course.

The document could be expanded to consider the identification and reporting of bad acts to support the sanctions provided for under law no matter the role of the parties.

That's an important comment so thank you. Testing implementations and actual use is a role for research (which is keenly advised and welcomed). This is also clear from the many references the questionnaire uses. For sure an important point, but this is not the key goal of the questionnaire - auditing the abuse of features should be important though, and I believe if such potential risk of abuses are found they can be documented already in the questionnaire and in the respective privacy considerations of each spec.

Legal Framework
Whilst the W3C lacks a policy in relation to matters of regulation each participant and document contributors form their own view. In my case, as a European, I’m skewed towards the GDPR. Speaking with a lot of other participants from other geographies I understand that GDPR is held in high regard and is used as a foundation for other legal frameworks. I therefore used the GPDR illustratively in the example. It would be helpful if the W3C adopted an official baseline policy on all matters including privacy.
Trust Choices
In the case of a change to a de facto standard which I believe has been implemented by Apple, Mozilla and, shortly Google around third-party cookies, trust choices have been removed from people and publishers. This is not in debated.
Document such as this provide legitimacy for such changes to de facto standards. Where proposals are presented to TAG for review this document along with Ethical Web Principles are most often cited.
In practice these matters are related and do result in a removal of trust choices for some stakeholders.
Transparency
Within the IAB Tech Lab Project Rearc there is a dedicated work stream focused on Accountability. IAB Europe have Transparency and Control Framework (TCF). The Partnership for Responsible Addressable Media (PRAM) principle 1 states “[provide] consumers with meaningful transparency and controls, giving the marketplace the tools to understand consumer preferences and the ability to abide by those preferences.”
The W3C could follow these organisations models and embrace the need for transparency and accountability as part of the work.
To conclude this specific issue the document should be modified in the following way.

Reference the need for a W3C policy on all matters including privacy.
In the absence of such a policy list its assumption referencing reasonable legal frameworks such as GDPR. Alternatively pause the use of the document until a W3C wide policy on all matters is available.
Redefine parties to reflect the policy or legal framework chosen.
Also consider matters of antitrust raised in this related issue.

Hello,
Thanks for your input, which is highly appreciated. Please see my remarks below.

Reference the need for a W3C policy on all matters including privacy.

I'm not sure if this is the role of this document, which is meant for feature developers.

In the absence of such a policy list its assumption referencing reasonable legal frameworks such as GDPR. Alternatively pause the use of the document until a W3C wide policy on all matters is available.

This questionnaire is not a legal document, and as such does not need a legal justification nor grounds. I do not feel the need to "pause the use of the document", which was basically used for many years now and is an integral part of the W3C spec review process. I feel it would be unhelpful and stall the privacy review aspect of new web features.

Redefine parties to reflect the policy or legal framework chosen.

This is not a legal document and it does not require any legal backing.

@jwrosewell wrote:

Alternatively[,] pause the use of [this] document until a W3C wide policy on all matters is available.

We're not going to stop important technical work to wait on a theoretical policy change that may or may not happen.

Redefine parties to reflect the policy or legal framework chosen.

When and if the W3C's policies change, all W3C documents should of course be updated where appropriate to reflect the new policy. We don't need open issues to track this until such policy changes actually happen, though.

Regarding the distinction between first and third party tracking captured in this document (and many others), I think @dbaron already addressed the user-centered reason for this distinction:

I think there is a substantial difference between first-party and third-party tracking because they differ substantially in:

how much they differ from user expectations
how visible they are to the user.

For example, if somebody goes to facebook.com, they probably expect that facebook.com knows something about their visit, what they click on on the page, etc. It's also relatively clear that facebook.com is what they're interacting with.
On the other hand, if they go to read a news article on reuters.com, they are much likely to have these expectations about what facebook does to track what they're doing. A user doing so does not naturally expect facebook to track what they're doing on reuters.com, nor is it clearly visible to the user that facebook is doing so.