How do I get the JSON from the provider?

It seems that this project requires some kind of connection to the block list you shared (yellow pages, etc)
How do I get my twilo account to send my endpoint this json data?

Hi, @makerofthings7!
You have to enable to install add-ons, so you can use them to detect spam calls. I would recommend you to take a look at this guide, it will give you a better insight about how to use it. https://www.twilio.com/docs/guides/how-to-secure-your-csharp-aspnet-app-by-validating-incoming-twilio-requests
Feel free to ping us if you have more questions.


Maximum zoom scale & photo/video orientation set from the device

Hi Andrew,
Thanks for the great work.
I would like to submit the following enhancements:

Allow to set a maximum zoom scale using the new public variable 'maxZoomScale'
Allow to chose if the orientation of a photo or a video being taken should be set to the orientation of the device even if the view controller should not rotate. This is similar to the behavior of the native iPhone camera app and can be done using the new public 'shouldUseDeviceOrientation' variable.

Cheers,
Oleg

Thanks for the pull request Oleg,
Really like these changes. Will merge these and update the README tonight
Andrew


OSCP how can i access https://dos-m0nk3y.github.io/offsec/

https://dos-m0nk3y.github.io/offsec/

This repository has nothing to do with OSCP I'm not sure what your question is.


Install script for Macbook users

There is a problem with initializing nvm (nvm loading and bash-completion initialization scripts are missing from zshrc)

Set up script to install golang in $HOME so that users are not asked to give elevated permissions

Issue with setting nvm environment variables


Android not installed feedback

Requires https://github.com/EstebanFuentealba/react-native-share/pull/220

@EstebanFuentealba please can you review this and merge.

@RalfNieuwenhuizen looks good!
could you please resolve conflicts?

@jgcmarins I tried but something is broken in the eslint for the example app. I cannot commit.

@RalfNieuwenhuizen what error?
Could you provide stack trace?

$ git commit
 ❯ Running tasks for *.js
   ✔ yarn prettier
   ✖ eslint --fix
     git add
✖ "eslint --fix" found some errors. Please fix them and try committing again.

/Users/user/react-native-share/example/App.js
 9:19  error  Unable to resolve path to module 'react-native-share'  import/no-unresolved
12:7   error  'instructions' is assigned a value but never used      no-unused-vars
58:3   error  Unused style detected: styles.instructions             react-native/no-unused-styles

✖ 3 problems (3 errors, 0 warnings)

pre-commit:
pre-commit: We've failed to pass the specified git pre-commit hooks as the `lint:staged`
pre-commit: hook returned an exit code (1). If you're feeling adventurous you can
pre-commit: skip the git pre-commit hooks by adding the following flags to your commit:
pre-commit:
pre-commit:   git commit -n (or --no-verify)
pre-commit:
pre-commit: This is ill-advised since the commit is broken.
pre-commit:

I think for this case you can use -n on commit.
We need to improve this later on, we should use lint separately.

@jgcmarins in that case, pushed!


SRIOV VirtualMachineInstance with sriov plugin interface [test_id:3957]should connect to another machine with sriov interface over IPv6

/kind bug
/triage build-officer
What happened:
Failed test due to timeout:
SRIOV VirtualMachineInstance with sriov plugin interface [test_id:3957]should connect to another machine with sriov interface over IPv6
/root/go/src/kubevirt.io/kubevirt/tests/vmi_multus_test.go:874
Failed to configure address fc00::2/64 for interface eth1 on VMI testvmi68rgzcj6sk8zp2x5xmnqtqgn2dd78nvpf79xc789mq9cnqpz
Unexpected error:
    <expect.TimeoutError>:<PHONE_NUMBER>0
    expect: timer expired after 15 seconds
occurred
/root/go/src/kubevirt.io/kubevirt/tests/vmi_multus_test.go:896

Prow Job Link
Lane:
pull-kubevirt-e2e-kind-1.17-sriov

@AlonaKaplan could this be possibly caused by the expecter bug?

/assign @Alona

@AlonaKaplan could this be possibly caused by the expecter bug?

No, it's not the expecter bug.
ip addr add fc00::2/64 dev eth1 gets an error - Cannot find device "eth1

hopefully #4001 will solve it.

More details at #4432

More details at #4432

4432 is not related, error here is eth1 not found, on 4432 its the ping that doesnt work (when i logged in the machine i saw the device is there but ip was gone, reasons on 4432)
There is one more problem that i know of, but its not the right issue to speak about.

This issue is user report for a flaky test.
This test is flaky for two reasons, one is resolved and described by https://github.com/kubevirt/kubevirt/pull/4001
And the second is describe by https://github.com/kubevirt/kubevirt/issues/4432

Fixed by #4448 by @qinqon
/close


how does getColor(int seed) work?

How does the seed work in getColor(int seed)? I'm sure whether it affects the index from which a color is picked or whether it affect the number of times a color is picked.

It's been a good while since I looked at this class and this method, but if memory serves, it allows you to define a seed int in your sketch so that each time you run the sketch you would get the same sequence of random colors. Let me see if I can knock a quick example together.

Yup, seeding is a way of controlling randomness.  Most random functions are actually pseudo-random and quietly seeded by something like timestamp.  Supplying a seed value is helpful if you want to reproduce a  random result, ie you could run your application multiple times and get the same color picks.  Here's the relevant bit from HColorPool.
	public int getColor() {
		if(colorList.size() <= 0) return 0;

		int index = (int) Math.floor(H.app().random(colorList.size()));
		return colorList.get(index);
	}

	public int getColor(int seed) {
		HMath.tempSeed(seed);
		int clr = getColor();
		HMath.removeTempSeed();
		return clr;
	}

Thanks for the insight, gents!

@choiway this example shows how seed works: https://github.com/hype/HYPE_Processing/blob/lib_staging/examples/HColorPool/HColorPool_004/HColorPool_004.pde
line 33 getColor(i3) play with that a bit and you'll see how it works. Take out the i3 and it's a random color every sketch load. Put it back in, you'll get the same color way each time.


Use ASM9

What is the purpose of this commit?

First step to allow building with JDK 17

But why did you replace all these Opcodes.ASM7? Did you have any troubles with ASM API compatibility?


⚠ gorzdrav.org has degraded performance

In 072d19f, gorzdrav.org (https://gorzdrav.org) experienced degraded performance:

HTTP code: 200
Response time: 1396 ms

Resolved: gorzdrav.org performance has improved in eb245f5.


Update Wrapper

This update has breaking changes.

What changed

Migrated project to TypeScript to support typings and have stricter code.
Added a linter for better code visualization and organising when working as a team.
Added a test file to test the API against.
API structure changes, including categories (exported module is different).
Changed response of API functions.

Why should this be merged

Previous version used deprecated endpoints, and repeated code.
Both JavaScript and TypeScript users can use this now.
Contributing made easier.

Todos

[x] Categorise endpoints
[x] Return full response in API functions
[x] Add util functions
[x] Document wrapper using JSDocs

Reviews, comments and contributions are much appreciated.
I'm open to any discussions on the Discord server. My tag is GamesProSeif#0310.

Alright, this should be ready for review now.

The package can be tested using:
$ npm i GamesProSeif/some-random-api#patch-1
# or
$ yarn add GamesProSeif/some-random-api#patch-1


Pointers Don't Show

On the stable build of Reclass, pointers are displayed like this:

But the current branch looks like this:

Thank you for the report. Will be fixed in a few seconds.


Add getting started guide in README

@gabrielPeart @thii I have a lot of ideas for Puma, but that will come after Christmas. God jul @gabrielPeart and メリークリスマス @thii


Insertion sequence reporting

Report the inserted sequence for insertion SVs if the length can be determined.

@sbandara and @msahraeian please review.

+1.


docs: addTypenameDocumentTransform does not exist

🚨 IMPORTANT: Please do not create a Pull Request without creating an issue first.
Any change needs to be discussed before proceeding. Failure to do so may result in the rejection of the pull request.
Description
https://github.com/dotansimha/graphql-code-generator/issues/9763
The code example in the following document is incorrect.
https://the-guild.dev/graphql/codegen/plugins/presets/preset-client#normalized-caches-urql-and-apollo-client
addTypenameDocumentTransform does not exist.
addTypenameSelectionDocumentTransform is correct
Related # (issue)
https://github.com/dotansimha/graphql-code-generator/issues/9763

Type of change
Please delete options that are not relevant.

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Screenshots/Sandbox (if appropriate/relevant):
Adding links to sandbox or providing screenshots can help us understand more about this PR and take action on it as appropriate
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Test A
[ ] Test B

Test Environment:

OS:
@graphql-codegen/...:
NodeJS:

Checklist:

[x] I have followed the CONTRIBUTING doc and the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[x] Any dependent changes have been merged and published in downstream modules

Further comments
If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

@n1ru4l
Hello
Thank you for your review.
Is it possible to merge this PR?
Please let me know if I need anything.


Enhancement: add skip-blanks option to tile generation

Hi,
With the upcoming version of libvips dzsave will support to skip blank tile during the tile generation.
I was thinking add this option :
tile.skipBlanks [Number] threshold to skip tile generation, a value  0 - 255 for 8-bit images or 0 - 65535 for 16-bit images (optional, default -1)

I will try make an PR  as soon as I can.
(Here is the upstream issue : https://github.com/libvips/libvips/issues/1185)

sharp v0.23.0 / libvips v8.8.1 now available. Thanks for the PR!


RACK_ENV and RAILS_ENV values are always 'production'

Hi, I set up dokku on vagrant and deployed there fresh rails app. I tried to run the app in 'staging' environment, but it's always run in 'production' environment.
vagrant@vagrant-ubuntu-precise-64:/var/lib/dokku/plugins$ sudo dokku config cat
=== cat config vars ===
RACK_ENV:         staging
RAILS_ENV:        staging
WAT:              YAY

I've checked ENV constant inside the app and it has a pair ENV['WAT']='YAY'. Is there any way to run rails app in 'staging' environment?
ENV of the app: https://gist.github.com/Zhomart/9700653

We now have a build-env plugin which should handle this issue. If you are still bumping up against not being able to change it, please comment on this issue and I'll take a second look.


wip infinispan entries

https://issues.redhat.com/browse/ISPN-12444
Needs to include

[ ] List with a limit (limit works, unmarshalling entries (key and value) does not seem to work easily. Something missing somewhere)
[ ] Stream (not possible from the console IMO, I don't see how to use it properly only for keys)
[ ] Documentation

Closing for now


resources/WORKSHOP.md file has incorrect example in Projectors section

In resources/WORKSHOP.md, B is defined as the "bounce" command, which is deprecated (now the subtract command)

Thanks :) Will fix


Peer dep missing issue in React - SPFx

Category

[ ] Question
[ ] Typo
[x] Bug
[ ] Additional article idea

When i execute "npm --save install office-ui-fabric-react" command it gives below warning:
Warning<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS><17.0.0 but none was installed.

and when i get version of solution by npm by command "npm list --depth 0" i face with below error:

I want to use fabric react component i.e.: https://developer.microsoft.com/en-us/fabric#/components/panel
And when i use in tsx file it gives an error:

Package JSON:
{
  "name": "navigation-poc",
  "version": "0.0.1",
  "private": true,
  "engines": {
    "node": ">=0.10.0"
  },
  "scripts": {
    "build": "gulp bundle",
    "clean": "gulp clean",
    "test": "gulp test"
  },
  "dependencies": {
    "@microsoft/sp-core-library": "~1.1.0",
    "@microsoft/sp-lodash-subset": "~1.1.0",
    "@microsoft/sp-webpart-base": "~1.1.0",
    "@types/react": "^0.14.46",
    "@types/react-addons-shallow-compare": "0.14.17",
    "@types/react-addons-test-utils": "0.14.15",
    "@types/react-addons-update": "0.14.14",
    "@types/react-dom": "0.14.18",
    "@types/webpack-env": ">=1.12.1 <1.14.0",
    "office-ui-fabric-react": "^6.20.0",
    "react": "^15.4.2",
    "react-dom": "^15.4.2",
    "typescript": "^2.9.2"
  },
  "devDependencies": {
    "@microsoft/sp-build-web": "~1.1.0",
    "@microsoft/sp-module-interfaces": "~1.1.0",
    "@microsoft/sp-webpart-workbench": "~1.1.0",
    "gulp": "~3.9.1",
    "@types/chai": ">=3.4.34 <3.6.0",
    "@types/mocha": ">=2.2.33 <2.6.0"
  }
}

How to solve Peer dep missing issue in React?

Can you update your version of react and react-dom to 16+? I suspect that would help.
If you can't, you can loader your fabric dependency to 5.x. We are continuing to maintain the 5.x release, which supports React 15, but going forward, React 16 has fundamental changes that we can't utilize and exposes deprecation we can't fix without making 16 the minbar.

We do not support React 16 yet in SPFx, so moving back to 5.x dependency should be the solution. React 16 version support is in pipeline for SPFx, but not there yet, so that's not an option.


It would be great to see an full example on solidstart

It would be great to see an example of a project where everything is included:

authorization
working with the sqlite database,
application context where certain functions are described
storing an authorization token

I agree. While I don't think it belongs as a template in this repo I've been thinking it would be cool to create an updated version of the Realworld Demo App (https://github.com/solidjs/solid-realworld)

yes, I'm really looking forward to the updated version, it's very interesting for beginners

i would like to see the use of trcp in this project


OFplayer findFrameId

Describe the bug

https://imgur.com/a/9dmGTI8
To Reproduce

Expected behavior
Initialize frameId before call function

Screenshots

Additional context

Wait I think I find another bug in the binary search part of LEDplayer...
Fixing that is simple and is already addressed in OFplayer though, IDK why it's not fixed in the LED counterpart


Gitlab Scaffolder: TriggerPipelineAction - Add missing ability to pass variables to the pipelines

Hey, I just made a Pull Request!
I've added the ability to pass environment variables into the TriggerPipelineAction using Record<string,string>
:heavy_check_mark: Checklist

[X] A changeset describing the change and affected packages. (more info)
[X] Added or updated documentation
[X] Tests for new functionality and regression tests for bug fixes
[ N/A] Screenshots attached (for UI changes)
[X] All your commits have a Signed-off-by line in the message. (more info)

See Issue: #25461

#25466

Closed for #25480


Please provide the package for the .AddConsole and .AddDebug.

It's a general problem in docs that a package is not referred to in the docs. I usually have to find some sample project to locate what package is used.
Am I missing something here or does everyone do the same?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7184a308-a254-9350-a5cd-5bced1f369ae
Version Independent ID: 726e3bf1-f367-d733-8933-bccc04da0e16
Content: Logging in .NET Core and ASP.NET Core
Content Source: aspnetcore/fundamentals/logging/index.md
Product: aspnet-core
Technology: aspnetcore-fundamentals
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Hello @floppydisken ... Use the API docs at :point_right: https://docs.microsoft.com/en-us/dotnet/api/


Fatal error BuildOrgConnectUri CoreClass ()

Hi,
I use 'Microsoft.Extensions.Logging' with Serilog logger. After passing logger to a ServiceClient I always see some fatal errors during call to crm. Everything seems to work, so I can't understand what is the reason of these errors:
[17:04:59 [Microsoft.PowerPlatform.Dataverse.Client.ServiceClient] FTL] BuildOrgConnectUri CoreClass ()
[17:04:59 [Microsoft.PowerPlatform.Dataverse.Client.ServiceClient] FTL] BuildOrgConnectUri CoreClass ()
My connection string format:
"AuthType=ClientSecret;url=https://xxx.dynamics.com;ClientId=yyy;ClientSecret=zzz",
Azure function V4, .Net 6

Hi @cbbuild ,  Not a lot to go on there.
is there a way you can provide a bit more from the log?
and to be clear,
the client is connecting, and your able to transact requests without issue...  However, your telemetry logs are reporting errors.  Is that correct?

Hello @MattB-msft
Yes everything works as expected. I see these errors after passing ILogger to the ServiceClient class.
On Verbose level I see (I masked urls):
2023-04-14 08:38:30.120 +02:00 [DBG] Initialize Dataverse connection Started - AuthType: ClientSecret
2023-04-14 08:38:30.142 +02:00 [DBG] Direct Login Process Started
2023-04-14 08:38:30.142 +02:00 [INF] Attempting to Connect to Uri https://xxx.crm4.dynamics.com/XRMServices/2011/Organization.svc
2023-04-14 08:38:30.158 +02:00 [FTL] BuildOrgConnectUri CoreClass ()
2023-04-14 08:38:30.159 +02:00 [DBG] DiscoveryServer indicated organization service location = https://xxx.crm4.dynamics.com/XRMServices/2011/Organization.svc
2023-04-14 08:38:30.159 +02:00 [FTL] BuildOrgConnectUri CoreClass ()
2023-04-14 08:38:30.159 +02:00 [INF] Organization Service URI is 'https://xxx.crm4.dynamics.com/XRMServices/2011/Organization.svc'
2023-04-14 08:38:30.160 +02:00 [DBG] ConnectAndInitService - Initializing Organization Service Object
2023-04-14 08:38:30.160 +02:00 [INF] ConnectAndInitService - Requesting connection to Organization with Dataverse Version: No organization data available
2023-04-14 08:38:30.570 +02:00 [INF] AuthenticateService - found authority with name https://login.microsoftonline.com/xxx/
2023-04-14 08:38:30.571 +02:00 [INF] AuthenticateService - found resource with name https://xxx.crm4.dynamics.com/
2023-04-14 08:38:30.576 +02:00 [DBG] Initial ObtainAccessToken - Client Secret
2023-04-14 08:38:31.370 +02:00 [DBG] Added WebClient Header Hooks to the Request object.
2023-04-14 08:38:31.458 +02:00 [INF] ConnectAndInitService - Proxy created, total elapsed time: 00:00:01.3026290
2023-04-14 08:38:31.459 +02:00 [INF] Direct Login Process Succeeded - duration 00:00:01.3213199
2023-04-14 08:38:31.460 +02:00 [DBG] New Batch Manager Created, Max #of Batches:50000, Max #of RequestsPerBatch:5000
2023-04-14 08:38:31.465 +02:00 [DBG] Execute Command - RetrieveMultiple : RetrieveMultiple to Dataverse via IOrganizationService : RequestID=f435e31d-8d3f-4c27-98b3-555320cd0213
2023-04-14 08:38:33.617 +02:00 [DBG] Executed Command - RetrieveMultiple : RetrieveMultiple to Dataverse via IOrganizationService : RequestID=f435e31d-8d3f-4c27-98b3-555320cd0213 : duration=00:00:02.1596567

Ok, now I understand the reason.
But can we avoid these fatal errors in logs? They all have the same category as the logger passed to the ServiceClient, so I can't filter by category.
Maybe they should have the lower log level?
Is there a way to log only a 'real' errors?

Its not acutally throwing and error, as a 401 is an allowed and handled type.  see: https://github.com/microsoft/PowerPlatform-DataverseServiceClient/blob/44a1af6fca8e9b3ec806d2322fbceab332b3164e/src/GeneralTools/DataverseClient/Client/Auth/AuthorityResolver.cs#L67
I suspect your picking up the http log stream as well as ours and it's surfacing there.

Closing this at this time as we believe it to be resolved.


Improve Raspberry Pi support.

This PR improves raspberry pi support by:

making the code more consistent with the rest of the codebase
making the pi specific packages more consistent among themselves
removing unused functions or duplicate constants
adding missing documentation
fixing broken linkx
adding required copyright and licensing notes

@kenbell can you take a look?

@abarisani - sure.  I didn't get a chance today.  I'll take a look over at the weekend.

Thanks for updates.  Pi2 seems to be working fine.  PiZero I'm seeing corrupt output on the UART - I'll take a look.

Thanks for updates. Pi2 seems to be working fine. PiZero I'm seeing corrupt output on the UART - I'll take a look.

Ok, I'll wait for the update on the PiZero before merging then. Thanks!


Move to requirement file for pip

References #96

[ ] Move dependency list to a proper requirements file
[ ] Manually test the build and test scripts to ensure functionality

Got it transitioned and working, PR up.
[t:120]


Interaction bug in Map mode

When in Map mode (in Analysis for sure, probably in Routing too), if I want to pan I press my finger or mouse button and move the map. Except than when I my finger or mouse pointer is too close to a node, the app switches to the page of said node even before I release my finger/button.
The normal behaviour, in most UIs, is to do that only when:

the finger/button is released
there has been no significant movement between press and release (any significant move would engage the pan mode)

This makes navigating in the map somewhat frustrating, especially since pan/zoom parameters are lost when coming back to the page.

@StephaneChatty thanks for reporting this problem, I will try to fix this

@StephaneChatty thanks for reporting this problem, I will try to fix this

The map interaction has been changed in version 3.1.2.
Selecting a node/route/flag/poi by single-click now feels a bit slower than it used to be. Perhaps the cursor should change upon single-click to give faster feedback to user that item has been clicked and request to server has been issued.

The map interaction has been changed in version 3.1.2.
Selecting a node/route/flag/poi by single-click now feels a bit slower than it used to be. Perhaps the cursor should change upon single-click to give faster feedback to user that item has been clicked and request to server has been issued.

Le 24 janv. 2021 à 11:05, vmarc<EMAIL_ADDRESS>a écrit :
The map interaction has been changed in version 3.1.2.

Much better, thanks!

Selecting a node/route/flag/poi by single-click now feels a bit slower than it used to be. Perhaps the cursor should change upon single-click to give faster feedback to user that item has been clicked and request to server has been issued.

Either that or a color change for the clicked object.
Cheers,
St.

Le 24 janv. 2021 à 11:05, vmarc<EMAIL_ADDRESS>a écrit :
The map interaction has been changed in version 3.1.2.

Much better, thanks!

Selecting a node/route/flag/poi by single-click now feels a bit slower than it used to be. Perhaps the cursor should change upon single-click to give faster feedback to user that item has been clicked and request to server has been issued.

Either that or a color change for the clicked object.
Cheers,
St.

Re-opened to improve user feedback upon single-click: either cursor change for the duration of the round-trip to the server, or a color change for the clicked object.

Re-opened to improve user feedback upon single-click: either cursor change for the duration of the round-trip to the server, or a color change for the clicked object.


[CMake] Add minimal cmake file

Supports the use of boost logic as part of a
parent cmake project via add_subdirectory( libs/logic )
Some more information can be found on the ML: https://groups.google.com/forum/#!topic/boost-developers-archive/kM4JRmyVl3M[1-25]

Do we need CI build jobs on this?

I don't know: different maintainers had very varied opinion on that (some didn't care, some wouldn't merge it unless there is a test for the cmake file and some wouldn't merge it when I added a test).
An example of where I did add tests is here: https://github.com/boostorg/integer/pull/18/files. If you want I can add something similar to this PR (after the holidays ;)).

@jeking3: So, how should I proceed?

How are we going to ensure the CMakeLists.txt file list of primary dependencies is always accurate going forward?  I think that part of the build process should be to analyze the primary dependencies (using the tool @pdimov created?) and fail if they don't match the CMakeLists.txt file.

These dependencies change all the time... any manual solution here will not be sustainable long-term.  Even if there were a build-time check that b2 does it would be good enough to just catch it:

A dependency that no longer exists but is in CMakeLists.txt
A new dependency not in CMakeLists.txt

As it is, I can almost guarantee it will rot and be incorrect before 1.70.0 releases.
@pdimov any thoughts on how you would consider solving this as a build-time failure (b2 preferred; I suppose it could be a CI job each repo inherits though).

As Mike said, the dependency list can reside in a separate file (I used cmake/dependencies.cmake in the past) generated by boostdep --cmake logic. This is what it outputs today:
# Generated by boostdep --cmake logic

boost_declare_dependency(boost_config INTERFACE boost::config)
boost_declare_dependency(boost_core INTERFACE boost::core)

I'll need to change it to use Boost::config instead of boost::config as we're using the former now.
This file could then be included by CMakeLists.txt to replace the current target_link_libraries call, if an appropriate definition of boost_declare_dependency is defined beforehand.
In a Travis job you would then diff the output of boostdep --cmake logic with the contents of cmake/dependencies.cmake and fail on mismatch, as I do f.ex. here:
https://github.com/boostorg/boostdep/blob/develop/.travis.yml#L132
to check whether another boostdep-generated file is up to date.
I, personally, prefer to keep the dependency list manual, especially when it consists of nothing more than config and core, but automating it has its benefits.

As long as a build fails either locally or in CI that's sufficient to make sure it doesn't rot.  Thanks.  Once you have the output on boostdep modified to use the correct (capitalized) version let me know.

@pdimov: How about instead of changing the spelling, add a new output that just creates a global variable with the dependent library names as a list of strings. The main cmake file can then just loop over that list and create the target names via concatenation.

You'd need three variables, INTERFACE, PUBLIC, PRIVATE.

Although there's indeed no reason to prefix with boost_ and boost::, I can just output the name, once. I probably had some scenarios in mind where these two differ, don't remember now what they were, if any.


Add Various 'Cookbook/Blog' style docs

Thoughts

How to have multiple machines run different bits of code
Basic setup tutorial
Doc on how it all works in auto-scaling mode

Seems like we should use github pages for this this, so that there is a proper website?

@jsuresh @timbod7
https://help.github.com/en/articles/about-wikis seems like a suitable feature?
also useful for a lot of the doco that I'm ... highlighting is needed.
then apart from the README, the wiki can be used for end user doco, and the repo doco can be focused on contributors.

if you enable it, I will start committing to it  :-)

fixed by #60


update PER support

Hi,
Thanks for your information!
For the 'wrap' mode in numpy.put(), you are correct. I fixed this line in this commit.
For the mean, I don't have a reference. Since we are first sampling games, and then sampling transitions from games, we need two prioritized sampling process, which is different from Prioritized Experience Replay paper. An alternative to the mean could be max, since after averaging, the transitions with high priorities may be affected. I'm still testing these two schemes.

I also added an option to choose whether or not to use the priority replay (self.PER in the config file). If set to False, the algorithm will never update the priorities, thus all transitions will always have equal priorities (currently 1.0), which is equivalent to uniform sampling.

I made alpha in PER algorithm configurable by adding self.PER_alpha in the config file.

For the remaining items on the list, here are my thoughts:

Add the loss scaling using the importance sampling ratio. (I'm having trouble figuring out how to do this without turning the buffer into a very long list with all the steps of each game)

        I think to calculate the IS weights for each sample, we need $N$ and $P(i)$ in this formula:

        where $N$ is the total number of transitions in the buffer, and $P(i)$ is the probability of transition $i$ get sampled, which can be calculated from game priorities and transition priorities.

Maybe assign an initial value of probabilities based on the loss of root.value and the predicted value in MCTS (or 1 as you did, could be a parameter).

        As suggested by the Distributed Prioritized Experience Replay paper (Page 4, paragraph 2), the initial priority can be either set to the "maximum priority seen so far" (which performs well when replay buffer is small but do NOT scale to cases where there are a lot of actors and replay buffer is large) or the "current n-step TD error" (as you suggested, which requires an additional prediction step), we can make this configurable.

Let me know if my points make sense to you!

Hi,
Thank you for the update and the answers.
It seems clear to me. I just have two questions that came to me:
• Why do you use a root for TD error?
• Should we scale the IS weights by the maximum one as suggested in the Prioritized Experience Replay paper page 5?

For stability reasons, we always normalize weights by 1/ maxi wi so
that they only scale the update downwards

Hi,
Thank you for the update and the answers.
It seems clear to me.
Should we scale the IS weights by the maximum one as suggested in the Prioritized Experience Replay paper page 5?

For stability reasons, we always normalize weights by 1/ maxi wi so
that they only scale the update downwards


tracing: debug: Link Segger RTT/SystemView sections in DTCM if chosen

Configures Segger RTT and SystemView data linker sections to DTCM by
default if there is a zephyr,dtcm chosen node in devicetree. This fixes
a build warning in samples/subsys/shell/shell_module for the
mimxrt1170_evk_cm7 and mimxrt1170_evk_cm4 platforms, which don't
currently have a zephyr,dtcm chosen node.
Note that there are runtime issues with Segger RTT and SystemView on
this board that need further debug, but submitting this patch now to
address nightly CI failures.
Signed-off-by: Maureen Helm<EMAIL_ADDRESS>Fixes #36646
@erwango This change affects some ST boards. I think this is the right thing to do in general, but if it's not then I can limit the scope to NXP boards.

@erwango are you ok with this change?

@erwango are you ok with this change?

Thanks for caring.
I'm not able to test this atm, so maybe the safest option is to limit to NXP boards.

@erwango are you ok with this change?

Thanks for caring.
I'm not able to test this atm, so maybe the safest option is to limit to NXP boards.

Updated to reduce the scope to NXP boards only.


Is it possible to add options to force all sub-models to not be defined in definitions？

Feature Request
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":
pydantic version: 1.3
pydantic compiled: False
python version: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
platform: Windows-10-10.0.18362-SP0
optional deps. installed: ['typing-extensions']

Users don't always like definitions, just like I don't like it. Maybe adding an option to model config is an effective way to not break existing default behavior.

I don't know what you mean. Can you give an example?


Non-base58 character thrown when creating a new DB eg feed/log/etc

I want to use a CID as part of the database name, eg
await this.orbitdb.feed('PR-' + url.repoCid + '-' + url.pullCid)

This causes Non-base58 character to be thrown, because of this line in orbit-db-address.js:
const accessControllerHash = parts[0].indexOf('Qm') > -1 ? multihash.fromB58String(parts[0]) : null

Should it be
const accessControllerHash = address.toString().indexOf('/orbit') === 0 && parts[0].indexOf('Qm') === 0 ? multihash.fromB58String(parts[0]) : null

This should be fixed in 1.0.0. Note the new package name.
If the issue persists, feel free to re-open the issue.


rave_flutter: 0.0.9-alpha+1 , always show error phonenumber is required

RaveResult{status: RaveStatus.error, rawResponse: null, message: phonenumber is required}
there is no any field to pass phonenumber in RavePayInitializer, then how can i pass phonenumber to that

Please, can you paste the construction of the RavePayInitializer?

var initializer = RavePayInitializer(
amount: double.parse(mainamt),
encryptionKey: Constant.FlutterwaveEncKey,
publicKey: Constant.FlutterwavePubKey,
subAccounts: null)
//subAccounts: subAccounts.isEmpty ? null : null)
//..country = country = country != null && country.isNotEmpty ? country : "NG"
//..currency = currency != null && currency.isNotEmpty ? currency : "NGN"
..currency = Constant.CURRENCYCODE
..country = Constant.COUNTRY
..email = uemail
..fName = uname
//..lName = lastName
//..narration = narration ?? ''
..txRef = "${uuserid}${pkgid}${new DateTime.now().millisecond}"
..orderRef = "${uuserid}${pkgid}${new DateTime.now().millisecond}"
..acceptMpesaPayments = true
..acceptAccountPayments = true
..acceptCardPayments = true
..acceptAchPayments = true
//..acceptGHMobileMoneyPayments = acceptGhMMPayments
//..acceptUgMobileMoneyPayments = acceptUgMMPayments
//..acceptMobileMoneyFrancophoneAfricaPayments = acceptMMFrancophonePayments
..displayEmail = true
..displayAmount = true;
  //..staging = false;

  //..staging = !live
  //..isPreAuth = preAuthCharge
  //..displayFee = shouldDisplayFee;

var response = await RavePayManager().prompt(context: context, initializer: initializer);
print(response);
scaffoldKey.currentState.showSnackBar(SnackBar(content: Text(response?.message)));

@wilburt sir..please help me for that error.
i am stuck on this issue from last 2 days

Please, at what point does it display this error? A video will be appreciated.

sir any update on this..i am also getting phone number is required after i enter pin then this message is appear no otp screen is showing

I am getting the same . This is my constructor
  var initializer = RavePayInitializer(             amount: 10,             publicKey: secret.firstKey,             encryptionKey: secret.secondKey)           ..country = "KE"           ..currency = "KES"           ..email = busines.billerEmail           ..fName = busines.billerName           ..txRef =               "subscription renewal of ${_getPlanName(busines.package.toLowerCase())}"           ..acceptMpesaPayments = true           ..acceptAccountPayments = true           ..acceptCardPayments = true           ..acceptGHMobileMoneyPayments = false           ..acceptUgMobileMoneyPayments = false           ..staging = false           ..isPreAuth = true           ..displayFee = true;
When I select pay with card and follow all the steps until : Enter your billing details, I get the error
RaveResult{status: RaveStatus.error, rawResponse: null, message: phonenumber is required}
However there is no input for phone number in the RavePayInitializer


Entity Extraction extracts entities twice for each entity

Rasa version: 1.10.10
Rasa SDK version (if used & relevant): 1.10.1
Rasa X version (if used & relevant): 0.31.3
Python version: 3.7
Operating system (windows, osx, ...): Ubuntu 20.04
Issue: Entities when being extracted, always identify twice for all the entities encountered. (please see attached images).
This issue is happening with all the entities. Here's the reference Issue from the RASA Form (https://forum.rasa.com/t/rasa-slot-filling-error-values-being-populated-twice/32204).
Here's how it looks via RASAX Local:

Here's how it looks in the CLI:

Error (including full traceback):

Command or request that led to error:

Content of configuration file (config.yml) (if relevant):
language: en
pipeline:
- name: WhitespaceTokenizer
- name: CRFEntityExtractor
- name: RegexFeaturizer
- name: LexicalSyntacticFeaturizer
- name: CountVectorsFeaturizer
  token_pattern: (?u)\b\w+\b
- name: DucklingHTTPExtractor
  url: http://localhost:8000
  dimensions:
  - number
- name: DIETClassifier
  epochs: 100
- name: EntitySynonymMapper
policies:
- name: KerasPolicy
  featurizer:
  - name: MaxHistoryTrackerFeaturizer
    max_history: 3
    state_featurizer:
    - name: BinarySingleStateFeaturizer
- name: FallbackPolicy
- name: MemoizationPolicy
- name: FormPolicy
- name: MappingPolicy
```**yml_finish**

Content of domain file (domain.yml) (if relevant):
session_config:
  session_expiration_time: 60
  carry_over_slots_to_new_session: true
intents:
- greet:
    triggers: utter_greet
- affirm
- request_asset:
    use_entities: []
- inform
- request_sr:
    use_entities: []
- deny
- refine_sr_priority:
    triggers: utter_ask_sr_priority
- refine_sr_id:
    triggers: utter_ask_sr_id
- refine_sr_assigned_to:
    triggers: utter_ask_assigned_to
- refine_assettype:
    triggers: utter_ask_asset_type
- refine_location:
    triggers: utter_ask_location
- refine_sr_category:
    triggers: utter_ask_sr_category
- refine_sr_type:
    triggers: utter_ask_sr_type
- refine_assetname:
    triggers: utter_ask_asset_name
- restart
- choose
- stop
- chitchat:
    use_entities: []
- thankyou
- bot_challenge
- goodbye
entities:
- activity
- asset_name
- asset_type
- location
- module
- object_type
- sr_category
- sr_id
- sr_name
- sr_priority
- sr_sub_category
- sr_type
- number
slots:
  activity:
    type: unfeaturized
  asset_name:
    type: unfeaturized
  asset_type:
    type: unfeaturized
  location:
    type: unfeaturized
  module:
    type: unfeaturized
  requested_slot:
    type: unfeaturized
  sr_category:
    type: unfeaturized
  sr_id:
    type: unfeaturized
  sr_name:
    type: unfeaturized
  sr_priority:
    type: unfeaturized
  sr_sub_category:
    type: unfeaturized
  sr_type:
    type: unfeaturized
responses:
  utter_ask_activity:
  - buttons:
    - payload: Search
      title: Search
    - payload: Create
      title: Create
    - payload: Update
      title: Update
    - payload: Delete
      title: Delete
    text: What activity would you want to perform in the Asset module?
  utter_ask_all_assets:
  - buttons:
    - payload: /affirm
      title: Yes
    - payload: /deny
      title: No
    text: Would you want to see all assets?
  utter_ask_asset_name:
  - text: "Please enter the name of the Asset.\nWe will get better results if you\
      \ enter exact asset name.e.g.\n Private Stormwater-0013 or idde-0218 etc."
  utter_ask_asset_type:
  - text: Please enter the Asset Type
  utter_ask_continue:
  - buttons:
    - payload: /affirm
      title: Yes
    - payload: /restart
      title: No
    text: Are you satisfied with the result? If not, press No button to restart..
  utter_ask_location:
  - text: Please provide location of the Asset
  utter_ask_module:
  - buttons:
    - payload: /inform{"module":"asset"}
      title: Asset
    - payload: /inform{"module":"work_order"}
      title: Work Orders
    - payload: /inform{"module":"service_request"}
      title: Service Requests
    text: Please select the module
  utter_ask_sr_category:
  - text: Please enter the category of Service Request
  utter_ask_assigned_to:
  - text: Please enter the name of person to whom the SR is assigned to
  utter_ask_sr_id:
  - text: Please enter the id of Service Request
  utter_ask_sr_name:
  - text: Please enter the name of Service Request
  utter_ask_sr_priority:
  - text: Please enter the priority of Service Request
  utter_ask_sr_sub_category:
  - text: Please enter the sub category of Service Request
  utter_ask_sr_type:
  - text: Please enter the Service Request Type
  utter_chitchat:
  - text: Shall we get back to task?
  utter_confirm_sr_search:
  - buttons:
    - payload: /affirm
      title: Yes
    - payload: /deny
      title: No
    text: "Should I search a SR with the following details?\nSR Name: {sr_name}\n\
      SR ID: {sr_id}\nCategory: {sr_category}\nSR Type: {sr_type}\nPriority: {sr_priority}"
  utter_default:
  - text: Sorry, I was not able understand you. Please input something else.
  utter_goodbye:
  - text: Bye. Have a Nice Day!
  utter_greet:
  - buttons:
    - payload: /inform{"module":"asset"}
      title: Asset
    - payload: /inform{"module":"work_order"}
      title: Work Orders
    - payload: /inform{"module":"service_request"}
      title: Service Requests
    text: Hello! I am EPIC's Search Assistant! Please select a module to proceed.
  utter_iamabot:
  - text: I am a bot, powered by Rasa.
  utter_initiate_search_asset:
  - buttons:
    - payload: /refine_assettype
      title: Asset Type
    - payload: /refine_assetname
      title: Asset Name
    - payload: /refine_location
      title: Location
    text: 'To begin with search, please select one of the following:'
  utter_message:
  - text: ''
  utter_noworries:
  - text: You are welcome :)
  utter_refine_search_asset:
  - buttons:
    - payload: /refine_assettype
      title: Asset Type
    - payload: /refine_assetname
      title: Asset Name
    - payload: /refine_location
      title: Location
    - payload: deny
      title: No refinement required
    text: 'To get better search results, please select one of the following:'
  utter_refine_search_sr:
  - buttons:
    - payload: /refine_sr_category
      title: SR Category
    - payload: /refine_sr_type
      title: SR Type
    - payload: /refine_sr_priority
      title: SR Priority
    - payload: deny
      title: No refinement required
    text: 'To get better search results, please select one of the following:'
  utter_restart:
  - text: restarted
  utter_slots_values:
  - text: "I am going to run the search using the entered details:\n - **Activity**:\
      \ {activity}\n - **Asset Type**: {asset_type}\n - **Location**: {location}\n\
      \ - **Asset Name**: {asset_name}"
actions:
- action_hit_db
- action_spelling_check
- utter_ask_activity
- utter_ask_all_assets
- utter_ask_asset_name
- utter_ask_asset_type
- utter_ask_assigned_to
- utter_ask_continue
- utter_ask_location
- utter_ask_module
- utter_ask_reported_by
- utter_ask_sr_category
- utter_ask_sr_id
- utter_ask_sr_name
- utter_ask_sr_priority
- utter_ask_sr_sub_category
- utter_ask_sr_type
- utter_ask_status
- utter_chitchat
- utter_confirm_sr_search
- utter_default
- utter_goodbye
- utter_greet
- utter_iamabot
- utter_initiate_search_asset
- utter_message
- utter_noworries
- utter_refine_search_asset
- utter_refine_search_sr
- utter_restart
- utter_slots_values
forms:
- asset_form
- sr_form

```**yml_finish**

Thanks for raising this issue, @rctatman will get back to you about it soon✨
Please also check out the docs and the forum in case your issue was raised there too 🤗

You have two independent EntityExtractors:
language: en
pipeline:
- name: WhitespaceTokenizer

### The first Entity Extractor ###
- name: CRFEntityExtractor

- name: RegexFeaturizer
....

### The second Entity Extractor ###
- name: DIETClassifier
  epochs: 100

- name: EntitySynonymMapper
policies:
...

Try only using the DIETClassifier and check if the results are reasonable.

Hi Team,
Thank you for suggestion.
Removing the extra classifier solved the issue.
Best Regards,
Ankit Pachauri


Add boolean support

Support for 8-bit (1 byte) booleans via a new q_compress::types::boolean8::B8DataType type
Some references:

https://users.rust-lang.org/t/solved-what-is-byte-value-for-boolean/47347
https://doc.rust-lang.org/std/primitive.bool.html

We really want single-bit booleans. We could already cast booleans to u32 and use an existing compressor, but we should make it convenient to work with boolean data

We really want single-bit booleans. We could already cast booleans to u32 and use an existing compressor, but we should make it convenient to work with boolean data

Got it. Should we also add support for u8 then in case someone has booleans stored as 1 byte? Looking at most languages, this seems to be the most common option. This is the case for Rust — https://users.rust-lang.org/t/solved-what-is-byte-value-for-boolean/47347.

@Kommandat we'll be compressing whatever the user in memory, which is the bool type. I don't think we need a specific u8 data type because the user can cast to u32 and back trivially. The reason I want to do bool is that it's a very common type, and people might not think to cast it to u32 or use this library for it otherwise.


Move goauth2/oauth dependency to golang.org/x/oauth2

https://code.google.com/p/goauth2/ is depreciated (along with all of code.google.com), and is being moved to read-only mode on August 25 in preparation for removal. We need to move this dependency to https://godoc.org/golang.org/x/oauth2, the suggested alternative.

This was done on the google driver.
Digital ocean driver is still using it though.


Not working upon Recent upgrade of chrome in windows, but IE works

Hi Team,
       Good Evening. The existing code, which was working previously was not working after upgrade of chrome but the same code works fine for the IE. Here is the sample code.

Kindly suggest where it is going wrong?? ..
if (this.oClipboard)
this.oClipboard.destroy();
	this.oClipboard = new ZeroClipboard(this.oCopyLinkButton.getDomRef());

	this.oClipboard.on("ready", function(readyEvent)
	{
		that.oClipboard.on("error", function(e)
		{
			alert(e);
		});

		that.oClipboard.on("copy", function(e)
		{
			alert("copy called");
			that.oClipboard.setText(that.currentURL);
		});

		that.oClipboard.on("aftercopy", function(event)
		{
			alert("aftercopy called");
			that.oCopyLinkDoneNotif.open();
			setTimeout(function()
			{
				that.oCopyLinkDoneNotif.close(false);
			}, 2000);

		});
	});

Thanks,
Geetha Rani.

+1

+1

+1

+1

Hi,  my team identified a browser workaround for Chrome users:

On impacted web page, click the site info box to the left of the URL
For Flash, select 'Always allow on this site', or in Site Settings confirm 'Allow' is selected (see screenshot)

The default 'Ask' option does not work as expected.
Chrome version: 62.0.3202.94 (Official Build) (64-bit)

the workaround works on chrome
this problem also appears on firefox 56.2.0 or 57.0

This software uses flash, but flash is a security risk and shouldn't be used anymore! You could use the clipboard apis (https://www.w3.org/TR/clipboard-apis/)


An issue with the gitops-system reconsiliation

All the resources in gitops-system namespace are unexpectedly deleted when I add an additional YAML file into fleet-infra\staging-cluster\ created by the bootstrap process.
Steps to reproduce:

Deploy resources using the bootstrap gotk bootstrap github --owner=$GITHUB_USER --repository=fleet-infra --path=staging-cluster --personal --arch=arm64. I'm using --arch=arm64 but I don't think it should affect anything
Pull the created repository and add a resource description into fleet-infra\staging-cluster, a namespace definition for example. I'm using 01-namespace.yml containing

apiVersion: v1
kind: Namespace
metadata:
  name: lens-metrics-2

So that the structure looks like this:
├── staging-cluster
│   └── gitops-system
│   │  ├── toolkit-components.yaml
│   │  ├── toolkit-kustomization.yaml
│   │  └── toolkit-source.yaml
│   └── lens-metrics-2
│       └── 01-namespace.yml

Commit the changes and push.
Observe that pods and all other resources in gitops-system (apart of GitRepository and Kustomization definitions) are deleted and never come back again.

When I delete finalizers from the remaining GitRepository and Kustomization CRDs and they finally go away I'm able to execute the bootstrap command from step two and all the resources come back including the namespace I added at step 3.

@moikot this was a bug in GC after the last refactoring, thanks for reporting it. You can run now gotk bootstrap and it will pull kustomize-controller v0.0.10 that has the fix for this.


Information on which protocols are being used? (e.g. references to scientific papers)

Hey, this looks like a nice project!
I've been wondering: Is there an easy way to find out which protocols are being implemented?
E.g. which paper is followed for active security?
Which bit decomposition protocol is used?
Would be nice to include references to the respective scientific papers in the documentation and/or the code.

Thanks for your interest!  When we remember, we include paper references in our API reference documentation - for example, the active security paper is described in the documentation for ActiveProtocolSuite.  I'll go find the bit decomposition reference; in the meantime, don't hesitate to ask for what you need!
Cheers,
Tim

Thanks for the pointer!

Had some time to look into this and searched for the work referenced in the ActiveProtocolSuite. I realized then that this points to a PhD thesis which does not seem to be available to the public. Is the protocol described somewhere else? Did it undergo peer-review?

Regarding the protocol for active security, it did receive review by the board of advisors on the committee referenced in the document. The protocol was not published in an academic conference or journal since before it could be published another was published which had a slightly better asymptotic complexity and interest in the protocol in the thesis was no longer sufficient. It does include complexity analysis and a security proof which you can study and judge for yourself. We have considered adding other protocols for active security but have not had the bandwidth yet to be able to do so.
On the source of the protocol for bit decomposition, it is a straightforward application of the least-significant-bit extraction protocol by Nishide and Ohta (https://iacr.org/archive/pkc2007/44500343/44500343.pdf) in an iterative manner to extract all the bits. This is not the most efficient means by which this operation can be done, but since we were using the least significant bit extraction elsewhere, bit decomposition in this manner came largely for "free" in terms of development time. Adding a more efficient bit decomposition protocol is possible but would likely require heavy use of this primitive to motivate given development time constraints at present.

I understand. Maybe it would still be worthwhile publishing on eprint (or similar) for easier access?

I did not realize that it was very difficult to access. That sounds reasonable to me. Thanks for the suggestion.

The access of the dissertation/thesis has been opened up. It was restricted as a part of some error. Here is a link where it is hosted at the original university. I have not yet got it on Arxiv and the cryptology eprint archive doesn't accept dissertations or the like. I am closing this issue for now as it seems resolved to me.

Great, that helps!

It is now also on Arxiv: https://arxiv.org/abs/2311.08934
That one is actually slightly updated with some typos and the like fixed.

It wound up making it into the thesis which is now on both the originating university and Arxiv at the provided link in chapter 2. It just never made it into a conference or journal outside the thesis. I agree with you in principle that alternative protocols may be of interest even though their complexity may not be optimal, however asymptotic complexity reigns supreme it would seem to me for the purposes of publishing in many of these venues.


SAK-33102 copy sort order in GBNG

I feel like this PR could be simplified by creating a comparator and sort using that vs creating a temporary collection.

Hi all,
To sort the grade matrix for the table, we push this to the GradebookService by passing a SortType to GradebookNgBusinessService.getGradebookAssignments(final String siteId, final SortType sortBy).  This saves doing any extra sorting in the tool.  I think you could use this to save any extra sorting headaches.
Originally, the instructor view had slightly different sorting requirements to the big table where instead we sorted by due date and then the assignment name (perhaps to match up with the Gradebook Classic?).  This is largely facilitated by JavaScript via the tablesorter plugin.
So to get things working, we'll need to disable those tablesorter settings and pass a SortType through to the GradebookService.  I just had a quick play and it looks like this is possible by introducing a new GradebookNgBusinessService method getGradebookAssignmentsForStudent(final String studentUuid, final SortType sortedBy): https://github.com/payten/sakai/commit/cdfb9508aaba3151b626ea85442d5066622bfd28.
Hopefully that's helpful.
Thanks,
Payten

Maybe I'm missing something with this feature, but the commit @payten linked to seems to work fine. I've reverted Ryan's commit and cherry-picked Payten's. Let me know if you want me to do anything with this one.


add location to worker model

track location along with hostname. also hardware id or version?

Name is used to identify webcam. May be enough.


⚠ Bofrost.be has degraded performance

In c5455ac, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2866 ms

Resolved: Bofrost.be performance has improved in 0da721d.


Attempting to write read-only database

Sometime ago I was working with label-studio locally and it was working fine. Now I entered again and I still can see my data, but I cannot modify it. I'm having the following error:
django.db.utils.OperationalError: attempt to write a readonly database
I looked for the sql.lite file and I change its permissions to 777 just in case (~/.local/share/label-studio/label_studio.sqlite3). However, the error is still there. What can I do?

What is your backend filesystem?

Sometimes it is also required writing rights to the directory in which your sqlite database is stored
Does the user that you running label-studio changed?

It is a local hard drive. Someone changed permissions, and I did a 777 chmod on sqlite file but wasn't working. Now I did it to the entire folder ~/.local/share/label-studio and it's working. Thanks!


How to modify the location of python files installed by default in ads

Azure Data Studio Version:

Steps to Reproduce:
1.I want to change the default installation python by ADS ,for exampel :d:\python ，because the c disk space  is not enough

For a new python installation, you can specify where Python is installed (including another drive letter).
For existing python installations, that Python can be installed into the folder of your choosing.


[SPARK-21870][SQL] Split aggregation code into small functions

What changes were proposed in this pull request?
This pr proposes to split aggregation code into pieces in HashAggregateExec. In #18810, we got performance regression if JVMs didn't compile too long functions (the limit is 8000 in bytecode size). I checked and I found the codegen of HashAggregateExec frequently goes over the limit, for example:
scala> spark.range(1).selectExpr("id % 1024 AS a", "id AS b").write.saveAsTable("t")
scala> sql("SELECT a, KURTOSIS(b)FROM t GROUP BY a")

This query goes over the limit and the actual bytecode size is 12356.
This pr split the aggregation code into small separate functions and, in a simple example;
scala> sql("SELECT SUM(a), AVG(a) FROM VALUES(1) t(a)").debugCodegen

generated  code with this pr:

/* 083 */   private void agg_doAggregateWithoutKey() throws java.io.IOException {
/* 084 */     // initialize aggregation buffer
/* 085 */     final long agg_value = -1L;
/* 086 */     agg_bufIsNull = true;
/* 087 */     agg_bufValue = agg_value;
/* 088 */     boolean agg_isNull1 = false;
/* 089 */     double agg_value1 = -1.0;
/* 090 */     if (!false) {
/* 091 */       agg_value1 = (double) 0;
/* 092 */     }
/* 093 */     agg_bufIsNull1 = agg_isNull1;
/* 094 */     agg_bufValue1 = agg_value1;
/* 095 */     agg_bufIsNull2 = false;
/* 096 */     agg_bufValue2 = 0L;
/* 097 */
/* 098 */     while (inputadapter_input.hasNext() && !stopEarly()) {
/* 099 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 100 */       boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 101 */       long inputadapter_value = inputadapter_isNull ? -1L : (inputadapter_row.getLong(0));
/* 102 */       boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);
/* 103 */       double inputadapter_value1 = inputadapter_isNull1 ? -1.0 : (inputadapter_row.getDouble(1));
/* 104 */       boolean inputadapter_isNull2 = inputadapter_row.isNullAt(2);
/* 105 */       long inputadapter_value2 = inputadapter_isNull2 ? -1L : (inputadapter_row.getLong(2));
/* 106 */
/* 107 */       // do aggregate
/* 108 */       // copy aggregation buffer to the local
/* 109 */       boolean agg_localBufIsNull = agg_bufIsNull;
/* 110 */       long agg_localBufValue = agg_bufValue;
/* 111 */       boolean agg_localBufIsNull1 = agg_bufIsNull1;
/* 112 */       double agg_localBufValue1 = agg_bufValue1;
/* 113 */       boolean agg_localBufIsNull2 = agg_bufIsNull2;
/* 114 */       long agg_localBufValue2 = agg_bufValue2;
/* 115 */       // common sub-expressions
/* 116 */
/* 117 */       // process aggregate functions to update aggregation buffer
/* 118 */       agg_doAggregateVal_coalesce(agg_localBufIsNull, agg_localBufValue, inputadapter_value, inputadapter_isNull);
/* 119 */       agg_doAggregateVal_add(agg_localBufValue1, inputadapter_isNull1, inputadapter_value1, agg_localBufIsNull1);
/* 120 */       agg_doAggregateVal_add1(inputadapter_isNull2, inputadapter_value2, agg_localBufIsNull2, agg_localBufValue2);
/* 121 */       if (shouldStop()) return;
/* 122 */     }

generated code in the current master

/* 083 */   private void agg_doAggregateWithoutKey() throws java.io.IOException {
/* 084 */     // initialize aggregation buffer
/* 085 */     final long agg_value = -1L;
/* 086 */     agg_bufIsNull = true;
/* 087 */     agg_bufValue = agg_value;
/* 088 */     boolean agg_isNull1 = false;
/* 089 */     double agg_value1 = -1.0;
/* 090 */     if (!false) {
/* 091 */       agg_value1 = (double) 0;
/* 092 */     }
/* 093 */     agg_bufIsNull1 = agg_isNull1;
/* 094 */     agg_bufValue1 = agg_value1;
/* 095 */     agg_bufIsNull2 = false;
/* 096 */     agg_bufValue2 = 0L;
/* 097 */
/* 098 */     while (inputadapter_input.hasNext() && !stopEarly()) {
/* 099 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 100 */       boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 101 */       long inputadapter_value = inputadapter_isNull ? -1L : (inputadapter_row.getLong(0));
/* 102 */       boolean inputadapter_isNull1 = inputadapter_row.isNullAt(1);
/* 103 */       double inputadapter_value1 = inputadapter_isNull1 ? -1.0 : (inputadapter_row.getDouble(1));
/* 104 */       boolean inputadapter_isNull2 = inputadapter_row.isNullAt(2);
/* 105 */       long inputadapter_value2 = inputadapter_isNull2 ? -1L : (inputadapter_row.getLong(2));
/* 106 */
/* 107 */       // do aggregate
/* 108 */       // common sub-expressions
/* 109 */
/* 110 */       // evaluate aggregate function
/* 111 */       boolean agg_isNull12 = true;
/* 112 */       long agg_value12 = -1L;
/* 113 */
/* 114 */       boolean agg_isNull13 = agg_bufIsNull;
/* 115 */       long agg_value13 = agg_bufValue;
/* 116 */       if (agg_isNull13) {
/* 117 */         boolean agg_isNull15 = false;
/* 118 */         long agg_value15 = -1L;
/* 119 */         if (!false) {
/* 120 */           agg_value15 = (long) 0;
/* 121 */         }
/* 122 */         if (!agg_isNull15) {
/* 123 */           agg_isNull13 = false;
/* 124 */           agg_value13 = agg_value15;
/* 125 */         }
/* 126 */       }
/* 127 */
/* 128 */       if (!inputadapter_isNull) {
/* 129 */         agg_isNull12 = false; // resultCode could change nullability.
/* 130 */         agg_value12 = agg_value13 + inputadapter_value;
/* 131 */
/* 132 */       }
/* 133 */       boolean agg_isNull11 = agg_isNull12;
/* 134 */       long agg_value11 = agg_value12;
/* 135 */       if (agg_isNull11) {
/* 136 */         if (!agg_bufIsNull) {
/* 137 */           agg_isNull11 = false;
/* 138 */           agg_value11 = agg_bufValue;
/* 139 */         }
/* 140 */       }
/* 141 */       boolean agg_isNull19 = true;
/* 142 */       double agg_value19 = -1.0;
/* 143 */
/* 144 */       if (!agg_bufIsNull1) {
/* 145 */         if (!inputadapter_isNull1) {
/* 146 */           agg_isNull19 = false; // resultCode could change nullability.
/* 147 */           agg_value19 = agg_bufValue1 + inputadapter_value1;
/* 148 */
/* 149 */         }
/* 150 */
/* 151 */       }
/* 152 */       boolean agg_isNull22 = true;
/* 153 */       long agg_value22 = -1L;
/* 154 */
/* 155 */       if (!agg_bufIsNull2) {
/* 156 */         if (!inputadapter_isNull2) {
/* 157 */           agg_isNull22 = false; // resultCode could change nullability.
/* 158 */           agg_value22 = agg_bufValue2 + inputadapter_value2;
/* 159 */
/* 160 */         }
/* 161 */
/* 162 */       }
/* 163 */       // update aggregation buffer
/* 164 */       agg_bufIsNull = agg_isNull11;
/* 165 */       agg_bufValue = agg_value11;
/* 166 */
/* 167 */       agg_bufIsNull1 = agg_isNull19;
/* 168 */       agg_bufValue1 = agg_value19;
/* 169 */
/* 170 */       agg_bufIsNull2 = agg_isNull22;
/* 171 */       agg_bufValue2 = agg_value22;
/* 172 */       if (shouldStop()) return;
/* 173 */     }
/* 174 */
/* 175 */   }
/* 176 */
/* 177 */ }

I did also performance checks;
$ ./bin/spark --master=local[1]
scala> sql("SET spark.sql.shuffle.partitions=4")
scala> spark.range(10000000).selectExpr("id % 1024 AS a", "id AS b").write.saveAsTable("t")
scala> timer { sql("SELECT a, KURTOSIS(b)FROM t GROUP BY a").collect }

master w/ this pr, Avg. Elapsed Time: 2.520551837s
master, Avg. Elapsed Time: 54.029893146199996s

How was this patch tested?
Added tests in WholeStageCodegenSuite.

Test build #81235 has started for PR 19082 at commit 6dacad4.

I checked the TPCDS performance;
https://docs.google.com/spreadsheets/d/1DJP8BC8aOE4Fp8t9XAYki0vauCTowt2D1n7lmNXI7xw/edit?usp=sharing
Q66 is explicitly faster and the other queries are almost the same with the previous ones.

Test build #81235 has finished for PR 19082 at commit 6dacad4.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81235/
Test FAILed.

Jenkins, retest this please.

Test build #81248 has started for PR 19082 at commit 6dacad4.

Test build #81248 has finished for PR 19082 at commit 6dacad4.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81248/
Test FAILed.

Merged build finished. Test FAILed.

Aggregation code is not the only too long gen'd codes. Any big enough queries under current whole-stage codegen can produce too long gen'd codes.
Compared with special split for aggregation code like this, I'd like to push #18931 which is a more general fix.

I didn't look into the code yet, #18931 also could solve a query below?
 sql("SELECT a, KURTOSIS(b)FROM t GROUP BY a")

What it means to solve the query? Is it too long gen'd codes?

yes

After quickly scanning this change, I found actually this and #18931 are orthogonal.
This PR goes to split the aggregation expressions into functions. So it prevents long aggregation code caused by long gen'd codes expressions like Kurtosis.
#18931 goes to break the long gen'd codes caused by chain of execution operators by splitting gen'd codes of operators into functions.

yea, I'm with you; it seems they are orthogonal..., but If we have a more general way to solve both, it'd be the best.

Test build #81328 has started for PR 19082 at commit 78d0a00.

Test build #81329 has started for PR 19082 at commit abc5450.

Test build #81330 has started for PR 19082 at commit e432136.

Test build #81330 has finished for PR 19082 at commit e432136.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81330/
Test FAILed.

Test build #81331 has started for PR 19082 at commit 57c8280.

Test build #81331 has finished for PR 19082 at commit 57c8280.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81331/
Test FAILed.

Test build #81332 has started for PR 19082 at commit e3722a0.

Test build #81329 has finished for PR 19082 at commit abc5450.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81329/
Test FAILed.

Test build #81328 has finished for PR 19082 at commit 78d0a00.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81328/
Test FAILed.

Test build #81332 has finished for PR 19082 at commit e3722a0.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81332/
Test FAILed.

Test build #81351 has started for PR 19082 at commit a2b3dd3.

Test build #81351 has finished for PR 19082 at commit a2b3dd3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81351/
Test FAILed.

Jenkins, retest this please.

Test build #81354 has started for PR 19082 at commit a2b3dd3.

Test build #81354 has finished for PR 19082 at commit a2b3dd3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81354/
Test PASSed.

Merged build finished. Test PASSed.

@viirya @rednaxelafx @kiszk okay, could you check again?

Test build #81400 has started for PR 19082 at commit 6c502a9.

Test build #81400 has finished for PR 19082 at commit 6c502a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81400/
Test PASSed.

@gatorsmile could you check? Thanks!

I found q66 was over 2-times slower by recent commits (probably, bytecode size checks might affect this number..) https://docs.google.com/spreadsheets/d/1V8xoKR9ElU-rOXMH84gb5BbLEw0XAPTJY8c8aZeIqus/edit#gid=2074944948

Test build #82498 has started for PR 19082 at commit a95816c.

Could you rerun Q66 by increasing spark.sql.codegen.hugeMethodLimit?

sure, I will look into this.

Test build #82498 has finished for PR 19082 at commit a95816c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82498/
Test FAILed.

Merged build finished. Test FAILed.

retest this please.

Test build #82499 has started for PR 19082 at commit a95816c.

I checked q66 has two warnings on the current master(83488cc3180ca18f829516f550766efb3095881e);
17/10/06 07:32:26 INFO WholeStageCodegenExec: Found too long generated codes and JIT optimization might not work: the bytecode size (13420) is above the limit 8000, and the whole-stage codegen was disabled for this plan. To avoid this, you can raise the limit `spark.sql.codegen.hugeMethodLimit`:
17/10/06 07:37:51 INFO WholeStageCodegenExec: Found too long generated codes and JIT optimization might not work: the bytecode size (16634) is above the limit 15000, and the whole-stage codegen was disabled for this plan. To avoid this, you can raise the limit `spark.sql.codegen.hugeMethodLimit`:

So, I run bench with hugeMethodLimit=8000 or 20000;
OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q66 (hugeMethodLimit=8000)                  15905 / 15990          0.7        1452.0       1.0X

OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q66 (hugeMethodLimit=20000)                   7524 / 7638          1.5         686.9       1.0X

Test build #82499 has finished for PR 19082 at commit a95816c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82499/
Test FAILed.

Test build #82502 has started for PR 19082 at commit 1880dfd.

Test build #82502 has finished for PR 19082 at commit 1880dfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82502/
Test PASSed.

Also, I checked the performance of the master + this pr;
OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q66 (hugeMethodLimit=8000, w/this pr)         2107 / 2217          5.2         192.3       1.0X

I'm also curious the performance of the master + #18931 on q66. Based on @maropu's benchmark before, q66 has much performance improvement.

sure! I'll check two more;

master + #18931
master + this pr + #18931

TPC-DS benchmark just shows the perf of one of the Spark workloads. To avoid introducing the unexpected regressions, we have to be very careful when merging this PR and https://github.com/apache/spark/pull/18931, unless we fully understand all the impacts inside JVM.
In the community, we have two JVM experts @kiszk @rednaxelafx who developed features/fixes in JVM.
Q66 is a good example for us to do more investigation.

Sure. I'm totally agreed. We need to know the advantages and possible impacts if any when merging this PR and #18931,. It is good @kiszk and @rednaxelafx can help review this PR and #18931.

I checked the three pattens on q66;
                                   q66
master                           15960
master + pr18931                 14226
master + pr19082 + pr18931        1960

You can all the result here.

Thanks @maropu.
After counting accurate bytecode size, there seems a bottleneck in generated codes in aggregation, so this can improve q66 a lot.
Overall, the numbers looks great, except for there is a significant regression in q94 when applying this change.

just a sec, I'll re-run q94 (sometimes, numbers fluctuate).

ok, done (welcome any re-run requests);
OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q94(master)                                 25107 / 25322          0.2        6591.8       1.0X

OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q94(pr18931)                                25099 / 25383          0.2        6589.7       1.0X

OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q94(pr19082)                                25140 / 25361          0.2        6600.4       1.0X

OpenJDK 64-Bit Server VM 1.8.0_141-b16 on Linux 4.9.38-16.35.amzn1.x86_64
Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
TPCDS Snappy:                            Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
q94(pr18931+pr19082                         25211 / 25532          0.2        6618.9       1.0X

@maropu Thanks. Then looks there isn't any significant regression brought by this or #18931. We need to be careful but this numbers give more confidence.

The latest regression (introduced by spark.sql.codegen.hugeMethodLimit) clearly shows the ring/onion/horizontal cut (https://github.com/apache/spark/pull/18931) could introduce a performance regression.
Vertical cuts like this PR is more promising. However, it still does not resolve all the performance regressions introduced by spark.sql.codegen.hugeMethodLimit

@gatorsmile hmm, I don't know how you get to the conclusion. Is spark.sql.codegen.hugeMethodLimit any related to codegen cut? I think it is just a threshold used to determine whether to enable wholestage or not.
Can you also explain the horizontal/vertical cuts?
Btw, we didn't see any performance regression based on the benchmark numbers.

The current spark.sql.codegen.hugeMethodLimit shows an extreme case.
Just imagine we have two nodes, we want to do a horizontal/ring cut. Basically, in this scenario, horizontal/ring cutting means whether we do a whole-stage codegen or not.

I don't think spark.sql.codegen.hugeMethodLimit is the same level thing as #18931 or this PR.
hugeMethodLimit didn't do anything to affect how generated codes are split.

The regression caused by spark.sql.codegen.hugeMethodLimit shows the potential regression caused by horizontal cuts, although spark.sql.codegen.hugeMethodLimit does nothing.

Btw, I'd like to know what the horizontal/vertical cuts you meant.

The regression caused by spark.sql.codegen.hugeMethodLimit shows the potential regression caused by horizontal cuts, although spark.sql.codegen.hugeMethodLimit does nothing.

hugeMethodLimit just affects if wholestage codegen is enabled. The regression is possibly caused by we completely disable wholestage codegen when detecting maybe one huge method. The regression of disabled wholestage codegen is larger than the affect of huge method.
I didn't know how you link it to so-called "horizontal cuts" or split generated code.

Basically, cutting is to decide the boundaries of blocking loop.
@kiszk and @rednaxelafx can explain what I said above better. This is related to how JVM works and how whole-stage codegen works

@gatorsmile Are you talking about this: https://en.wikipedia.org/wiki/Loop_nest_optimization?

Nope. You can wait for the answers from the JVM experts. : )

Ok. Any comments from @kiszk and @rednaxelafx on this topic? :)

The relation is very clear. Let me copy what I said above

Just imagine we have two nodes, we want to do a horizontal/ring cut. Basically, in this scenario, horizontal/ring cutting means whether we do a whole-stage codegen or not.

After horizontal/ring cutting, each operator will be in a single blocking loop. That means, whole-stage codegen is off.

Yeah, once we exceed hugeMethodLimit, wholestage codegen is disabled. Can you tell me how hugeMethodLimit makes each operator to a single blocking loop? We didn't run codegen now.

After horizontal/ring cutting, each operator will be in a single blocking loop. That means, whole-stage codegen is off.

And do we have anything to cut the operators to individual single blocking loop? I didn't see we have anything to do this.

Sorry, it is really hard to explain it to you. Maybe you can first think about why the whole-stage codegen works better the previous solution?

No. I think you should explain it because this is your concern. You just need to point out, where is the single blocking loop for the two operators?

Just try to help you understand it. If you don't want it, I can keep quiet.

@gatorsmile You can't just drop few words without explanation and simply say that others works don't work. I do respect your work/comments and you should respect my works too.

If you really can't or don't willing to explain, we can wait for @kiszk or @rednaxelafx to explain it.

OK. Based on your attitudes, I do not care it.

@gatorsmile If my words make you upset, I'm sorry. It's you right to raise suspicion against any PRs. I do respect this right.
Maybe I'm wrong and there actually is a possible regression. Appreciate if anyone has enough insights on this topic can share their thoughts. Thank you.

Yes no need for a back-and-forth. @gatorsmile I think it's reasonable to ask for a little more detail on your comment.

Either way, I think we first need to know why the regression on q66 happens when turning off wholestage codegen. We first thought turning off too-long functions had better performance, but it is not always true. Also, we better know if this regression happens on other JVM impls, too. Next, I'll look into these issues.

@maropu The codes to do aggregation are actually wrapped in a function doAggregateWithKeys/doAggregateWithoutKey. This is also the part of generated codes this PR improves by extracting functions.
My initial thought is, during the processing of the query, this function doAggregateWithKeys/doAggregateWithoutKey actually only runs once to aggregate on all rows. No matter it is a long function or not, we don't have chance for JIT to step in. That said the length of this function doesn't impact too much in JIT issue.
The long function issue affects the performance of wholestage codegen, because it is run many times in non-compiled way. It drags the speed of other generated codes. Since doAggregateWithKeys/doAggregateWithoutKey only run once, it doesn't impact much. So wholestage codegen query is still faster than non-wholestage codegen one.
This PR improves the aggregation because it extracts small functions from doAggregateWithKeys/doAggregateWithoutKey. Those functions will be run many times in the wrapping function. So JIT has room to step in now.

The above reasoning also explains the motivation and the effect of #18931 too.
The generated codes of query operators are extracted to individual smaller functions. It is beneficial to step in by JIT.

Aha, I feel fair enough. Based the insight, there is one of solutions to make the wholestage codegen consider #calls of gen'd functions though, it seems the approach is not simple. So, splitting functions step-by-step is a preferred approach now...

Let me summarize recent interesting PRs for code generation regarding JVM bytecode limit for JIT compilation. These PRs encourages to apply JIT compilation to more methods since most of JIT compilers stop performing JIT compilation for a method with larger size (e.g. 8000 byte in HotSpot compiler). There are two categories for PRs.

limit the total JVM bytecode size of the generated Java method (#18810, #19083)
Generate a Java method with smaller size (#18931, #19082).

I think that both categories are complementary. I like these activities.
In category 1., it is to disable a whole-stage codegen for a large method (i.e. more than 8000 JVM byte code) that will not be JIT-compiled.
#18810 tries to estimate whether JVM bytecode size is less than 8000 or not by using the number of lines of a method. The threshold of the line is 2667. If estimated bytecode size more than 8000, whole-stage codegen is disabled. This threshold worked well for most of programs. However, as @maropu summarized here, it did not work for some program (e.g. q66).
Then, #19083 checks actual JVM bytecode size by using the compiled JVM bytecode. This PR can precisely avoids not to perform JIT compilation.
Category 1. cannot encourage JIT compilation to the whole-staged method.
In category 2., code generation in each part tries to smaller methods (i.e. 8000 JVM byte codes per method) to apply JIT compilations to more methods or to avoid JVM byte code generation failure (beyond 64KB per methods). This will not dis
One of activities is to use CodeGenerator.splitExpressions().
#18931 splits a set of comsume() functions in a physical plan into multiple methods instead of embedding into one method (e.g. processNext()).
#19082 splits operations in aggregation into multiple methods instead of embedding into one method (e.g. agg_doAggregateWithoutKey()).
Even if these PRs create smaller methods, JIT compiler can make compilation unit larger by applying method inlining. To make compilation unit larger encourages to apply more optimizations in the compilation. For example, in HotSpot, a method whose JVM bytecode size is up to 325 (frequently executed) or 35 (normal) will be inlined. Thus, I think that we will rarely see performance regression.
Category 2. tries to encourage JIT compilation to the whole-staged method by making its method size smaller.
@gatorsmile, @viirya, @maropu, @rednaxelafx what do you think? Do you have any comments or questions?

@kiszk Thanks for summarizing the PRs.
I just have a question about inlining method by JIT compiler. So you mean JIT compiler will inline methods into larger unit and then do JIT compilation on it? Will JIT inlining method make a huge method that fails JIT? Although it sounds nonsense, I think I'd like to make sure it won't happen.

To my best guess, trying to make huge method by method inlining will not fail JIT compilation in Hotspot. It may fail method inlining.
According to these blog entries 1, 2, we can see the following annotations for method inlining (explanations from [1])

inline (hot): the method was determined hot and inlined
too big: the method was not inlined as the generated code was getting too big (but the method was not hot)
hot method too big: the method was determined hot but not inlined because the resulting code was getting too big.

Annotation 1. shows that inlining are suceeded. Annotation 2. or 3. shows that callee is too large. While 2. or 3. is shown in the compilation log [2], it still seems to succeed to compile a method. Of course, some method inling is are disabled.
Here is an interesting experiments. While they forced to inlining by increasing threshold, it is not easy to see performance improvement.
@viirya, is it answer to your question?

@kiszk Thanks for your summary. I had a few related discussions with @rednaxelafx and @liancheng in the recent weeks. Vertical cuts like https://github.com/apache/spark/pull/19082 is pretty promising for improving the performance. Thus, I plan to review the implementation in the recent days.
At the beginning, I did not notice https://github.com/apache/spark/pull/18931 is using private or final attributes; otherwise, inlining does not benefit.  The current proposal in https://github.com/apache/spark/pull/18931 is pretty limited due to the parameter number limits and the others. Thus, we need a better design for horizontal/ring cuts. The design is highly dependent on JVM internals and whole-stage codegen algorithms. So far, we do not have anything significant to share yet.

The whole-stage codegen has two advantages according to this paper.

enable compiler optimizations among operations (3. in page 2)
pass data in a row using a set of scalar values instead of using an iterator (1. and 2. in page 2)

When method inlining is disabled (I assume each method is compiled), we will miss advantage 1. We can still enjoy advantage 2 by passing data without creating an iterator. IMHO, This is better than disabling whole-stage codegen.

You know, when we disable the whole-stage codegen, we still do the expression codegen, which byte code size is smaller than 8K at most cases. Thus, it could be faster than whole-stage codegen. That is why we saw the performance difference in the PR https://github.com/apache/spark/pull/18810
If we can cut the boundary of operators better, we still can make the byte code size smaller than 8K. That means, we are not completely disabling the whole-stage codegen.

I agree with you. #18810 compares the following two code.

Interpreter execution of Java code by whole-stage codegen with passing row data in scalar values
JITted execution of Java code by expression codegen with passing row data using iterator

In the above case, 1. is slower than 2. due to the advantage of JIT compilation.
#18931 can cut the boundary of whole-stage operators. Hopefully, this cutting makes the byte code size smaller. #19082 can encourage this assumption.
Do you want add a feature to disable whole-stage codegen if JVM bytecode size of one of cut methods is larger than 8KB after applying cutting at the boundary of whole-stage operators?

https://github.com/apache/spark/pull/18931 is not what we want, although it can partially resolves some issues. Simply disabling the whole-stage codegen after the cut might still trigger the regression like Q66. The general goal is to maximize the advantage of both whole-stage codegen and JIT optimization. Now, the point is how to cut the boundaries without introducing the extra IR.

@gatorsmile let me clarify. In other words, as one of possible approaches, do you want to find how to cut the boundary to ensure enabling method inlining for callee methods? It could maximize advantage of both whole-stage codegen and JIT optimization in a compilation unit.

@kiszk This is one of the solutions if we can remove the limit of num of parameters. However, this does not resolve all the issues. For example, the method becomes too big to inline.

@kiszk Thanks for explanation! It sounds reasonable behavior by JIT compiler.

At the beginning, I did not notice #18931 is using private or final attributes; otherwise, inlining does not benefit.  The current proposal in #18931 is pretty limited due to the parameter number limits and the others.

@gatorsmile Sorry, but may I ask what is the "others"?

You know, when we disable the whole-stage codegen, we still do the expression codegen, whose byte code size is smaller than 8K at most cases. Thus, it could be faster than whole-stage codegen. That is why we saw the performance difference in the PR #18810

I think the case disabled wholestage codegen is faster than wholestage codegen is only happened when the methods of wholestage codegen are not compiled, right? Once the methods are compiled, I'm doubt that we will see performance regression when wholestage codegen enabled.

In the above case, 1. is slower than 2. due to the advantage of JIT compilation.
#18931 can cut the boundary of whole-stage operators. Hopefully, this cutting makes the byte code size smaller than current aggressive embedding...

That is correct to me.

#18931 is not what we want, although it can partially resolves some issues. Simply disabling the whole-stage codegen after the cut might still trigger the regression like Q66.

The regression of Q66 is explained in previous comment: https://github.com/apache/spark/pull/19082#issuecomment-335340888. A hot method which can't be compiled can cause regression. I think the split operator functions are hot methods for sure, if they can't be compiled, disabling wholestage codegen should not bring regreesion IMO.

If we can cut the boundary of whole-stage operators better, we still can make the byte code size smaller than 8K. That means, we are not completely disabling the whole-stage codegen.

@gatorsmile So sounds like your point is to find proper point to split (cut) operator functions out but not split all possible operators? Is it correct?

Test build #82615 has started for PR 19082 at commit 5584bdb.

@kiszk @gatorsmile @rednaxelafx Based on the following reasoning, I don't think the cutting of whole-stage operators you may propose would be much beneficial than the approach did in #18931:

If #18931 splits smaller functions. For the smaller operator functions, JIT will try to inline it into larger compilation unit. This is based on the information from @kiszk. Please correct me if it's not correct. So I don't think we will loss the advantage of wholestage codegen.  I'm doubt that it is any different than not split it, if under your proposal.
For the bigger operator functions, it can't be inlined. But based on the info provided by @kiszk, it won't get performance improvement to inline such big functions.

Thus, even considering you can provide better cutting points, I don't think it can get much benefit but incur much complexity into wholestage codegen.

Test build #82615 has finished for PR 19082 at commit 5584bdb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82615/
Test PASSed.

Test build #83981 has started for PR 19082 at commit 28b47e5.

Test build #83981 has finished for PR 19082 at commit 28b47e5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83981/
Test FAILed.

Test build #83982 has started for PR 19082 at commit 4c2816b.

Test build #83982 has finished for PR 19082 at commit 4c2816b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83982/
Test PASSed.

ping

Will review it carefully after I finish my vacation. Thanks!

ya, enjoy!

Test build #84126 has started for PR 19082 at commit 8bae975.

Test build #84126 has finished for PR 19082 at commit 8bae975.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84126/
Test PASSed.

@maropu could you please resolve conflicts?

Ya, I was just trying to do that tonight! Thanks for pinging me.

Test build #84711 has started for PR 19082 at commit d2dc025.

Test build #84713 has started for PR 19082 at commit f866b11.

Test build #84717 has started for PR 19082 at commit 1ed010f.

Test build #84711 has finished for PR 19082 at commit d2dc025.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84711/
Test FAILed.

Test build #84713 has finished for PR 19082 at commit f866b11.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84713/
Test FAILed.

Test build #84717 has finished for PR 19082 at commit 1ed010f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84717/
Test FAILed.

Jenkins, retest this please

Test build #84724 has started for PR 19082 at commit 1ed010f.

Test build #84724 has finished for PR 19082 at commit 1ed010f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84724/
Test FAILed.

Just FYI, the test failure is not related to this PR.

Test build #84734 has started for PR 19082 at commit 0e5d366.

Test build #84734 has finished for PR 19082 at commit 0e5d366.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84734/
Test FAILed.

retest this please

Test build #84770 has started for PR 19082 at commit 0e5d366.

Test build #84774 has started for PR 19082 at commit 4077608.

Test build #84770 has finished for PR 19082 at commit 0e5d366.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84770/
Test PASSed.

Test build #84774 has finished for PR 19082 at commit 4077608.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84774/
Test PASSed.

Test build #84807 has started for PR 19082 at commit 2555e5a.

Test build #84807 has finished for PR 19082 at commit 2555e5a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84807/
Test PASSed.

Since https://github.com/apache/spark/pull/19813 fixes this issue, so I'll close this. Thanks!

@gatorsmile can you close the jira ticket with @viirya assigned? Thanks!

Since #19813 fixes this issue, so I'll close this. Thanks!

I track to here from jira ticket SPARK-23791, which contains a test case.
#19813 is merged and then reverted. From the discussion under #19813, #21193 is raised.
But when I run the testcase on spark 2.4, I don't think the issue is fix.
Test result of 3 rounds : (coden on, codegen off)
(80.807065794,13.359716698),
(73.170440475,10.728186286),
(72.61997856,10.436293345))

Yea, this issue is not fixed now. As you know, it's a long story for this issue...
I closed this pr because this is just a band-aid fix. I'm not 100% sure the current status, but IIUC we still don't have a global consensu about how to fix this in the community.
Probably, @gatorsmile and @cloud-fan might know more.

Thanks for your reply.
@gatorsmile @cloud-fan, is there any update about the root cause and solution for this problem?