[SPARK-15043] [MLLIB] Fix and re-enable flaky test: mllib.stat.JavaStatisticsSuite.testCorr

What changes were proposed in this pull request?
Following https://github.com/apache/spark/pull/12779 this test became flaky. The issue is that the mean, computed for a covariance calculation, is now calculated with the standard and slightly more accurate MultivariateOnlineSummarizer. However I think the fact that it uses treeAggregate internally can lead to a different order of summation and very very slightly different results on different runs.
The immediate fix for the test, which asserts equality, is to use 1 partition. We can find out if that appears to be robust.
More generally, it's an interesting question whether we want MultivariateOnlineSummarizer to be deterministic. I'm not sure if another aggregation method would provide more guarantees of this, in theory or practice.
How was this patch tested?
Existing Java stats suite.
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Test build #57475 has started for PR 12821 at commit f755850.

Test build #57475 has finished for PR 12821 at commit f755850.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57475/
Test PASSed.

FWIW you can consistently reproduce the differing results with ..
val x = sc.parallelize(Seq(1.0, 2.0, 3.0, 4.0))
val y = sc.parallelize(Seq(1.1, 2.2, 3.1, 4.3))
(0 to 10).map(i => Statistics.corr(x,y)).distinct

With 1 partition the result is always the same. The result is the same if I use aggregate instead of treeAggregate.

/cc @mengxr for review, since @jkbradley mentioned that you might have fixed the flakiness via a separate patch.

This patch changed the test to use approx equality: https://github.com/apache/spark/commit/19a6d192d53ce6dffe998ce110adab1f2efcb23e

It would be great to have it be deterministic, but it sounds hard or impossible to ensure beyond numerical precision in general.

@srowen If we only use 1 partition, it doesn't touch the code path of treeAggregation, which means the coverage is not sufficient. I think we should use multiple partitions and test equality with a small tolerance.

Sounds good, it's either make it so simple that it's deterministic or tolerate tiny variation.

Oh already addressed in https://github.com/apache/spark/commit/19a6d192d53ce6dffe998ce110adab1f2efcb23e


Issue mounting partition using the toolkit

I'm trying to mount my project source code but I'm running into the following error:
~/Repositories/app $ docker run -it --rm -v $(pwd):/srv ubuntu bash
Error response from daemon: Cannot start container d229edb17428971b4f690f97d050eb62960a53ee3fae3815da241d4219063783: mkdir /Users/Samuel/Repositories: file exists

I managed to narrow it down with @rszgrz to the fact that I am using a combination of symlinks and partitions to get around the default case insensitive file system on OSX.
I have a partition at /Volumes/Storage (Mac OS Extended Case-sensitive, Journaled). I point to /Volumes/Storage/Repositories with a symlink at /Users/Samuel/Repositories.
I've also tried docker run -it --rm -v /Volumes/Storage/Repositories:/srv ubuntu bash which mounts one folder, but it's empty.
It seems like there's an issue mounting volumes on a partition with docker/docker-machine.
docker version: Docker version 1.8.1, build d12ea79
uname -a: Darwin Samuels-MacBook-Pro.local 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64
docker info:
Containers: 5
Images: 77
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 87
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.0.9-boot2docker
Operating System: Boot2Docker 1.8.1 (TCL 6.3); master : 7f12e95 - Thu Aug 13 03:24:56 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: RLP7:JEBP:DGOH:H4QP:X4GL:ROAF:TT2A:FMLQ:HUDB:E7MK:U7GL:V4IK
Debug mode (server): true
File Descriptors: 10
Goroutines: 20
System Time: 2015-09-28T11:54:47.904056777Z
EventsListeners: 0
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

Environment details (AWS, VirtualBox, physical, etc.): OSX toolkit

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

docker-machine automatically mounts /Users into the VM, which makes it seamless to use with your local user files... anything other than that and you are on your own to set this up manually.
If you look at the VM details you will see the shared folder setup for /Users.
Additionally, you can't mount a file onto a dir, or a dir onto a file, which is where the "file exists" error would be coming from.
Closing as this is a non-issue.
Thanks!


Avoid use of Spark leaf, unary and binary expressions and exec nodes

Fixes #2052.
Spark 3.2 changed the class hierarchy for leaf, unary and binary expressions and exec nodes, and the plugin was deriving many GPU classes from these abstract Spark classes.  This causes issues on Spark 3.2 because the location of the child, children, left, right, etc. methods has changed, and code compiled against the Spark 3.0 code is no longer binary compatible with these classes in Spark 3.2+.

build

@tgravescs I think I've addressed your review comments.  I refactored GpuBatchScanExec to be a shim due to the use of LeafExecNode which required updating another shim interface.
@abellina during testing I saw canonicalization failures which I tracked down to mishandling canonicalization in aggregation expressions.  I've fixed those tests, but I'm not 100% positive it's the proper fix.  Please take a look at the change in AggregateFunctions.scala.

build

build

The premerge CI was failing for Spark 3.2 due to a crash.  I'm not quite sure what is causing the crash, but as far as I can tell it is directly associated with CostBasedOptimizerSuite.  If I remove that from test on Spark 3.2 it always passes for me, but fails almost every time if it is present.  If this stubbing is OK, I'll file a followup issue to investigate the instability on Spark 3.2 so that we can re-enable that test across all Spark versions.
I also added a few commits to fix more cases of unary classes in the class hierarchy.

build

I was trying to test this but another PR went into Spark that is breaking 3.2 (https://github.com/apache/spark/pull/32030). It seems related, but not sure if you want to address here.
Why the hash agg change is needed for filter is odd to me too. I'll look more at that today. I did notice the filter expression is not included here: https://github.com/NVIDIA/spark-rapids/blob/branch-0.5/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala#L150... it is in the CPU case. So was going to go down that route.
If we want to merge this and follow up with a 3.2.0 follow up that would be fine too.

So a bit of bad news.  SPARK-34989 recently updated the 3.2 code so that every node needs to override withNewChildrenInternal.  I've updated the code so it at least compiles, but it won't run as-is since the hierarchy isn't binary compatible, e.g.:
HashAggregatesSuite:
*** RUN ABORTED ***
  java.lang.AbstractMethodError:
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$withNewChildren$2(TreeNode.scala:279)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
  at org.apache.spark.sql.catalyst.trees.TreeNode.withNewChildren(TreeNode.scala:278)
  at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.insertRowToColumnar(Columnar.scala:514)
  at org.apache.spark.sql.execution.ApplyColumnarRulesAndInsertTransitions.$anonfun$insertRowToColumnar$1(Columnar.scala:514)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.TraversableLike.map(TraversableLike.scala:237)
  at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
  at scala.collection.immutable.List.map(List.scala:298)
  ...

Adapting to the changes in SPARK-34989 probably needs a different approach than just throwing what doesn't work in the shim manually.


Remove prefixes from DiscoveryService implementations

All of these structs are public. Previously when they were used, they'd
look like the following:

consul.ConsulDiscoveryService
etcd.EtcdDiscoveryService
file.FileDiscoveryService
nodes.NodesDiscoveryService
token.TokenDiscoveryService

As a result, they stuttered when read. This commit removes the
duplication of the package names on the struct so that they read better.
See also: https://golang.org/ref/spec#Import_declarations
Signed-off-by: Kushal Pisavadia<EMAIL_ADDRESS>

Coverage increased (+3.0%) to 42.6% when pulling bca710c4902cd0b854bef0ebb2649bf72a8472d5 on KushalP:master into<PHONE_NUMBER>ac1d60fd144f1cf51cc3361de72bca on docker:master.

Coverage increased (+3.0%) to 42.6% when pulling bca710c4902cd0b854bef0ebb2649bf72a8472d5 on KushalP:master into<PHONE_NUMBER>ac1d60fd144f1cf51cc3361de72bca on docker:master.

Coverage remained the same at 39.6% when pulling 78dba5d623da82167b10c5be4cfbb0aadecb3cfa on KushalP:master into<PHONE_NUMBER>ac1d60fd144f1cf51cc3361de72bca on docker:master.

LGTM

LGTM


support for roman numerical list style

This is in the latest version 2.1.0. Please test it out and confirm that it's working correctly!


Replace "pip" with "pip3" on linux systems

I recommend replacing only the string when attempting to install some packages on different operating systems.
For example, on windows, installation of a pip package would probably look like this:
pip install package_name
while on Linux, it would be like this:
pip3 install package_name
I think it would save some begginers a lot of time searching for the issue solution, by just finding the operating system of the user, and producing different strings according to that.

Closing this as a duplicate of https://github.com/pypa/warehouse/issues/1563, but thanks for the report!


Allow people to configure their own Rubygems

There's an increasing need for users of ScoutApp to be able to add rubygems they're interested in using to their ScoutApp project. Currently, all gems are bundled with ScoutApp and cannot be changed through the UI.
In the past, I have been adding more gems to ScoutApp, and pushing new releases. This is not a good solution. The other alternative is for users to customize their local installation of ScoutApp, which is also not a good solution.
Within the current UI, I was thinking of adding a new "Gemfile" tab next to "Configure" and "Log" which would be the place where users could add their own Rubygems. Since most gems give clear instructions about what to add to a Gemfile it seems straightforward enough to have this be a big text area that would be the contents of an actual Gemfile. If users messed up the formatting the next time they would try to run Scout they should see the output from bundler about their Gemfile being malformed. At least, this could be the first step.
This may require a few things about Scout to change.

Each project would need to track the contents of its Gemfile.

ScoutApp has its own Gemfile that it uses when running for a user in src/ and this would likely no longer be able to use this as-is since each project's Gemfile could differ. This could become a template Gemfile that each project starts with

ScoutApp would either need to maintain actual Gemfiles (on disk) for each project (so it could tell bundler which Gemfile to use when starting compass) OR it could start placing them in the user's project.

Every time the Gemfile changed ScoutApp would need to re-bundle. ScoutApp uses its own internal vendor/gems directory and it should likely continue to use this for all of a user's projects. So while the Gemfile will be different for each project, all gems should be installed to the same vendor/gems

Those are my thoughts so far.

The latest version of Scout-App is Node-based and no longer relies on Ruby. Since Compass and other Gems require Ruby to run, Scout-App 2 cannot use them.
Closing this issue as "will not fix".

Scout-App 2 (2016)


feat(svc): add servers create endpoint

Will close RVTEE-87, RVTEE-88, RVTEE-89

#740  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @AngelOnFira and the rest of your teammates on  Graphite

@NathanFlurry shouldnt it be renamed to api-server and pkg/server to be consistent with everything else?


Add a small graghics improvement on main screen

I added a line that delimits the status bar from the rest of the screen.
The functionality can be turned on or off from Makefile.
Simple and cute :)

Many thanks I like it, that should please @M7OCM too 😉
I saw that you'd changed the battery threshold from orange to green. Do you know which value you changed from in voltage or percentage?


Animating removal of carousel items.

Is this a bug report, a feature request, or a question?
Question
Question
I have a bunch of images in a carousel, and the user can press a button outside the carousel that removes them. This works fine, but it's a bit abrupt visually as the carousel doesn't animate the removal of the item, it just chunks into place.
I'd like to animate this removal smoothly. Is there a recommended way to do this or an example I could have a look at that does this?

@thekevinbrown I never had to do it, but I just remembered this old article: https://hackernoon.com/how-to-animate-the-items-of-a-react-native-flatlist-32c8cbf7ea3d
The exact same principles can be applied. However, all the website's code sections have been replaced with a picture of the author — don't know what's going on there...

I'm not sure how exactly this'd work with the carousel. Reason being, how do I replicate the animation the carousel does when it moves a page into the centre from the edges?
I might have a look at adding this to core as it seems like it'd be easier to do from within the component than trying to replicate all the logic outside of it.

@thekevinbrown Definitely feel free to push a PR for that; you'd make a lot of users happy 👍

bump :D did this ever get a PR?
wanting to animate the removal of cards in our carousel as well!
Thanks!!

Nope, couldn't figure it out.

Sorry, please allow me to advertise for my open source library! ~
I think this library react-native-reanimated-carousel will solve your problem. It is a high performance and very simple component, complete with React-Native reanimated 2

`carouselRef.current.snapToPrev();
		setTimeout(() => {
			setData(dummyData.filter((v, index) => index !== activeSlide));
			carouselRef.current.snapToItem(activeSlide - 1, false);
		}, 300);`


hot restart: provide a mechanism for obtaining a base-id dynamically

Commit Message:
Provides a --use-dynamic-base-id flag to select an unused base-id.
Primarily useful for testing, but generally available. Adds a
--base-id-path flag where Envoy writes the base id to a file.
Converts tests to use the dynamic base id selection rather than
trying to keep all base ids unique.
Additional Description:
This change preserves the oddball multiplication of base-id by 10, but moves it into the hot restart code. This has the side-effect of making the server_info command line options correct.
When the hot restart code gets EADDRINUSE trying to open the domain socket, MainCommonBase retries with a different random number. It gives up after 100 attempts (but immediately on other values of errno). If I've got my math right: the randomly chosen base ID has 2^28 possible values, so a collision with a single other Envoy has a likelihood of 1 in 2^28 (~268 million). The likelihood of failing N times in a row is 1 / 2^(28N). If M other Envoy processes are already running, the likelihood of N collision in a row is M^N / 2^(28N). So given N = 100 and M = 2^10 (~1000), the likelihood of a startup failure is 1 in 2^1800 and for M = 2^20 (~1M) it's 1 in 2^800.
Risk Level: low
Testing: updates existing tests, tests dynamic base id flags/retry
Docs Changes: updated
Release Notes: updated
Signed-off-by: Stephan Zuercher<EMAIL_ADDRESS>

Follow-on to #11016.


Responsive card layout

Cards should probably scale and position w/r/t to the layout...

An important note: we should make sure that the cards preserve the aspect ratio of tarot cards, cardHeight = (88.9 / 57.15) * cardWidth;

We probably need to mess with the minimum size of cards; avoiding overlap would be nice (my screen is pretty big, for instance, and I'm still getting overlaps for the celtic cross

I think we can take advantage of the magnify button to make these cards (and the text inside them) smaller. It would be really nice if we could fit the celtic cross in a regular sized laptop monitor without scrolling or big card overlaps.


Add Storybook Addon-Docs Dependency

Add the @storybook/addon-docs npm dependency that is required
to build the Storybook after updating other dependencies.

Going forward, we may want to make sure to bump the @storybook/* dependency versions together as a group


split physioevents and events json

See #90
Closes #90

some tests are failing because the timestamps of the output physio.tsv.gz are not continuous because eyetracker was probably turned on and off
will need to be fixed as we resolve issue #69


Explore Using SQLite

Explore using SQLite instead of MariaDB to see if it reduces image size as well as reducing dependency on another container.

Leaving as is to keep data and app separate. Allows users to have their own backup policies.


backend cannot be SEARCHAPI

Describe the bug
when I set backend as SEARCHAPI, it cannot deploy on lepton.ai.
To Reproduce

Expected behavior
I want to use 3rd party api for google search on lepton.ai
Screenshots

Environment (please complete the following information):
https://dashboard.lepton.ai/

Hi
The prebuilt photon only support lepton bing and google as backend, if you want to use customized search api, deploy the modified photon by yourself.
ref: https://www.lepton.ai/docs/build_you_own_model/anatomy_of_a_photon


Window resizing causes event processing to block

This can be seen on the demos as well.

I'm having this problem, too. I tried to asked in the forum ("Resize callback stall") first but I then found this issue. You mentioned here, and in #185, that one can try to "use the window size or refresh callbacks to update the contents of a window during resize". Could you please show how exactly do this, in an example perhaps?
BTW, I tried to subclass NSOpenGLView and it redraws even during resizing. This is how the it looks like:
- reshape(void) {
    NSSize size = [self.window.contentView frame].size;
    glViewport(0, 0, size.width, size.height);
}

Apple also provides "GLEssentials" example which works in this regard, too.

Also a duplicate of #408.


On top lists, ties are shown ordered

For example:

#
Name
# of decks

1
Tatyova, Benthic Druid
113

2
Zada, Hedron Grinder
113

3
Araumi of the Dead Tide
113

4
Fynn, the Fangbearer
107

Would be more correct if it read:

#
Name
# of decks

1
Tatyova, Benthic Druid
113

1
Zada, Hedron Grinder
113

1
Araumi of the Dead Tide
113

4
Fynn, the Fangbearer
107

(or the second and third 1s could be replaced by blanks).

Fixed by using Window/Rank in next deployment.


Allow passing predefined_acl via gcs blob_properties

Title
Allow passing predefined_acl via gcs blob_properties
Motivation
Similar to #726, it should be possible to set an ACL for gcs uploads, as it is for s3.
It turns out you can't set an ACL to a gcs blob object directly ref https://github.com/stevearc/pypicloud/issues/317.

Blocked by https://github.com/googleapis/python-storage/pull/878
Blocked by https://github.com/googleapis/python-storage/pull/838

Tests
If you're fixing a bug, consider test-driven development:

Create a unit test that demonstrates the bug. The test should fail.
Implement your bug fix.
The test you created should now pass.

If you're implementing a new feature, include unit tests for it.
Make sure all existing unit tests pass.
You can run them locally using:
pytest smart_open

If there are any failures, please fix them before creating the PR (or mark it as WIP, see below).
Work in progress
If you're still working on your PR, include "WIP" in the title.
We'll skip reviewing it for the time being.
Once you're ready to review, remove the "WIP" from the title, and ping one of the maintainers (e.g. mpenkov).
Checklist
Before you create the PR, please make sure you have:

[ ] Picked a concise, informative and complete title
[ ] Clearly explained the motivation behind the PR
[ ] Linked to any existing issues that your PR will be solving
[ ] Included tests for any new functionality
[ ] Checked that all unit tests pass

Workflow
Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.
Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.

Is it possible to set the ACL after the upload is complete? If we can do this bypassing smart_open, that would be ideal.

do you mean as a fallback for gcs <= 2.6?
or as primary/only approach?
it was my first thought too. but since i dont use gcs, i had to do a deep dive into the code and have not found a way to set ACL after upload:
there is blob.update(), but there is no attribute setter for ACL.
so using predefined_acl kwarg in one of the uplod methods is the only way I could find to achieve blob upload with ACL. all uploads methods have this kwarg for a long time, ONLY the one smart_open uses didn't have it

I meant as a primary (only) approach.
This looks relevant: https://cloud.google.com/storage/docs/xml-api/put-object-acls

looks like this PR can be superceded by https://github.com/RaRe-Technologies/smart_open/pull/729#issuecomment-1279949567

OK, thank you. Closing in favor of #729


Twitter video fails with 403 Forbidden when using cookies from browser

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2021.10.22. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
France
Description
I have --cookies-from-browser in my config file, and when I try to download a video from twitter, it fails. If I add --no-cookies-from-browser to the command line, it works again.
Verbose log
[debug] Command-line config: ['-Uv', 'https://twitter.com/SimonZawa/status/1455107386583232513']
[debug] User config file: /home/codl/.config/yt-dlp/config
[debug] User config: ['--no-playlist', '--embed-subs', '--sub-langs', 'all', '--add-metadata', '--embed-thumbnail', '--xattrs', '--paths', 'temp:/tmp/', '--cookies-from-browser', 'firefox']
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "/home/codl/.mozilla/firefox/p99w1gby.sept2021/cookies.sqlite"
[Cookies] Extracted 827 cookies from firefox
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8[debug] yt-dlp version 2021.10.22
[debug] Lazy loading extractors is disabled
[debug] Python version 3.9.7 (CPython 64bit) - Linux-5.14.16-arch1-1-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1, ffprobe 4.4.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
yt-dlp is up to date (2021.10.22)
[debug] [twitter] Extracting URL: https://twitter.com/SimonZawa/status/1455107386583232513
[twitter]<PHONE_NUMBER>583232513: Downloading guest token
ERROR: [twitter]<PHONE_NUMBER>583232513: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
  File "/usr/lib/python3.9/site-packages/yt_dlp/extractor/common.py", line 702, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/lib/python3.9/site-packages/yt_dlp/YoutubeDL.py", line 3252, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Login is not implemented for twitter. It is normal for some websites to reject the request when the logged-in cookies are present without actual login being performed


(*azblob.ContainerClient).ListBlobsFlat returns no error when using wrong cred

Bug Report

What happened?
I was trying to listBlobs with azidentity.NewClientSecretCredential as credential. However, I didn't get any errors with wrong tenantID, clientID or clientSecret.
What did you expect or want to happen?
pager.Err() should say something like "wrong auth" instead of nil when cred is wrong
How can we reproduce it?
Here is my code

package main

import (
	"context"
	"fmt"

	"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
)

func main() {
	tentID := "asdasdasda"
	clientID := "asdfasdfasdf"
	clientSecret := "1231323"

	cred, err := azidentity.NewClientSecretCredential(tentID, clientID, clientSecret, nil)
	if err != nil {
		panic(err)
	}
	azBClient, err := azblob.NewServiceClient("https://<storageAccount>.blob.core.windows.net/", cred, nil)
	if err != nil {
		panic(err)
	}
	conClient, err := azBClient.NewContainerClient("<containerName>")
	if err != nil {
		panic(err)
	}
	pager := conClient.ListBlobsFlat(
		&azblob.ContainerListBlobsFlatOptions{})
	fmt.Println(pager.Err())

	for pager.NextPage(context.TODO()) {
		resp := pager.PageResponse()
		fmt.Println("page len: ", len(resp.Segment.BlobItems))
		for _, v := range resp.Segment.BlobItems {
			fmt.Println(*(v.Name))
		}
	}
}

And the output is like:
$ go run main.go
<nil>

As you can see we are getting nil error with some random cred. I am wondering is it designed like this or there should be some error? If this is designed like this, how do we know there is authentication error or the container is just empty?
Looking forward to your replies

import path of package in question, e.g. .../services/compute/mgmt/2018-06-01/compute
As in the code
SDK version e.g. master, latest, 18.1.0

Specify the exact commit if possible; one way to get this is the REVISION
column output by dep status "github.com/Azure/azure-sdk-for-go.

github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.0.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1

output of go version
go version go1.18 darwin/arm64

Anything we should know about your environment.

Hey @sfc-gh-llin ,
Thanks for reaching out!
Let me shine some light on the issue you raised. The function conClient.ListBlobsFlat(&azblob.ContainerListBlobsFlatOptions{}) just creates a pager. But the actual request is sent when we try to fetch the first pager i.e. pager.NextPage(context.TODO()). Once we try to get the first pager, only after that can we expect err and resp to have non-null values.
Please check out this example for more information.
// Pager Constructor
pager := serviceClient.ListContainers(&listContainersOptions)

// Send the request to fetch the pager. Next() function is where we first try to fetch a page.
for pager.NextPage(context.TODO()) {
    // Now both resp and err are available to us.   err := pager.Err() will contain the error now.
    resp := pager.PageResponse()

    for _, container := range resp.ContainerItems {
	fmt.Println(*container.Name)
    }
}

if pager.Err() != nil {
     log.Fatal(pager.Err())
}

I hope my explaination helps. If you have any questions, don't hesitate to reach out to me.

Please reopen this if your query is not resolved.


[realsense2] build failure

Host Environment

OS: Windows 10
Compiler: MSVS2017

To Reproduce
Steps to reproduce the behavior:
./vcpkg install realsense2
Failure logs
Error: Building package realsense2:x86-windows failed with: BUILD_FAILED Please ensure you're using the latest portfiles with .\vcpkg update, then submit an issue at https://github.com/Microsoft/vcpkg/issues including:   Package: realsense2:x86-windows   Vcpkg version: 2020.02.04-nohash
install-x86-windows-dbg-out.log
Additional context
I have installed the english version of MSVS2017, Windows 10 SDK, I am kinda stuck now and don't know what to do. I attached all the logs, I couldn't find a clue about this issue within the log.
Thanks for your time

Please forgive me for taking the time to disturb you.
I found the error from the log:
c:\vcpkg\buildtrees\realsense2\src\dbc1bebf69-5a1aac81b3\src\winusb\device-winusb.h(14): fatal error C1083: Cannot open include file: 'atlstr.h': No such file or directory

Maybe you had missed ATL SDK or something.
When running the Visual Studio Installer, select the Individual components tab, and under SDKs, libraries, and frameworks make sure Visual C++ ATL Support is selected.

Installed "Visual C++ ATL support" and MFC and ATL support. The error still occurred in our x64 project.
You could try to fix some paths with the following two commands:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC>mklink /d atlmfc "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\atlmfc"
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\atlmfc\lib>mklink /d amd64 "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\atlmfc\lib\x64

More
Here are some links that helped me investigate your issue:

http://stackoverflow.com/a/16404536/796832
https://groups.google.com/forum/#!topic/microsoft.public.vc.language/pUU-NM4RRhA

Please forgive me for taking the time to disturb you.
I don't understand what you meant by saying this, however thanks for helping me and pointing out the issue, which is now solved. Thanks a lot !!


Add support for XACML v3.0 Related and Nested Entities Profile Version 1.0

Implemented everything specified by the XACML v3.0 Related and Nested Entities Profile Version 1.0
Fixed missing support of Content attribute in JSON requests
Delegate to StdFactory in CustomFactory instead of duplicating code

Looking good - just letting you know that I'm still reviewing it all.

I'm working on filing an internal authorization request to contribute to an open source project. I'm being asked if the project has "Contributor License Agreement (CLA) or Developer Certificate of Origin (DCO)". Please confirm.

This is great! Just some minor nits. I am asking legal also about the new files, I think there should be license headers there. I think its covered by the root license but I prefer to add them into the header. The question is, is it ok for it to be "Salesforce". I think so but not sure.

I'll can add Salesforce's standard MIT copyright header to the new files:
/*

Copyright (c) 2021, salesforce.com, inc.
All rights reserved.
SPDX-License-Identifier: MIT
For full license text, see the LICENSE file in the repo root or https://opensource.org/licenses/MIT
*/

@pdragosh ping


WISHLIST - Tasker

Allow Tasker via plugin to change the state of switches - e.g. toggle things on/off unattended.

Allow Tasker via plugin to change the state of switches - e.g. toggle things on/off unattended.


feat: support vite 3.0

@JohnCampionJr Can you merge this pretty please? 😆
Vite 3 is out now https://vitejs.dev/blog/announcing-vite3.html

Please! Merge! 😅

Are there any alternatives? Or temporary processing? I desperately need this to work

So very sorry, all.  I will do this today!
On Sat, Jul 23, 2022 at 2:17 AM HunterPHP @.***> wrote:

Are there any alternatives? Or temporary processing? I desperately need
this to work
—
Reply to this email directly, view it on GitHub
https://github.com/JohnCampionJr/vite-plugin-vue-layouts/pull/67#issuecomment-1193069586,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAILJJH7Q6MJKTNNUZS2IY3VVOE5XANCNFSM52H3UDMA
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
John Campion
Bright Shore Technologies, Inc.
Makers of Maryland PPT Pro


When yum.conf specifies a proxy, use that proxy to fetch the erratas

fixes: #16
I did some basic tests with a few proxies, but it should probably be tested more thoroughly before being merged.
It was somewhat painful to avoid any new dependency (such as requests), so there is a few lines of urllib2 stuff to open the Errata URL through a proxy.
Finally, it should be noted that Python supports opening HTTPS ressources through a proxy only since 2.6.2.

I'll need to see why the test is failing first.

Something related to ipython (???) install on Redhat, where it lacks gcc to compile the native modules. I don't really see how that could be related to the PR...

Yeah it probably isn't. I'll try and address it in master and then merge in the PR.

You're right, same problem on master build - https://travis-ci.org/wied03/centos-package-cron/jobs/149351998. Something environment/version wise since the master code hasn't changed in a while.

@stephane-martin - Fixed the master build by removing the ipython dependency that I don't really need anyways. Can you rebase your branch against master and push so that I can confirm travis passes?


Events from previous day are shown "done" on next day; they are not events crossing midnight though

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
I have some events that occurred Tuesday night:

And looking at them on the atomic calendar, they show under Wednesday as completed:

This is wrong; why would these events show under Wednesday, when they occurred on Tuesday?
Expected behaviour
I would expect Tuesday events to not be rendered under Wednesday.
Steps To Reproduce
No response
Mode
Event (default)
Card Version
9.0.0
Home Assistant Version
2023.9.3
Configuration
- type: custom:atomic-calendar-revive
            entities:
              - entity: calendar.holidays_in_united_states
                name: Holidays in US
                color: goldenrod
              - entity: calendar.birthdays
                name: Birthdays
                color: hotpink
              - entity: calendar.my_personal
                name: My personal
                color: mediumslateblue
              - entity: calendar.wife
                name: Wife
                color: orchid
              - entity: calendar.other
                name: Other
                color: palevioletred
            defaultMode: Event
            showWeekDay: true
            showMonth: false
            showProgressBar: true
            showCurrentEventLine: false
            showDescription: false
            disableEventLink: false
            disableLocationLink: false
            showNoEventsForToday: true
            showEventIcon: true
            showFullDayProgress: false
            showWeekNumber: false
            showCalendarName: false
            showDatePerEvent: false
            showTimeRemaining: false
            hoursOnSameLine: false
            calShowDescription: false
            showLastCalendarWeek: false
            showHiddenText: true
            hideFinishedEvents: false
            showLocation: true
            hideDuplicates: true
            showMultiDayEventParts: false
            showMultiDay: false
            compactMode: false
            showAllDayEvents: true
            allDayBottom: false
            offsetHeaderDate: false
            showDate: false
            showRelativeTime: true
            sortBy: start

Relevant log output
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi @nicholashead,
Please upgrade to 9.0.1-beta.2 where this may resolve your issue.
The latest update resolves an issue with timezones.

@marksie1988 that did it! Thank you. Verifying this is fixed by https://github.com/totaldebug/atomic-calendar-revive/issues/1191 - specifically https://github.com/totaldebug/atomic-calendar-revive/commit/d92b73dd146d3fa248231808134f701f4b1eca16


Renamed npm commands

:tada: This PR is included in version 7.6.5 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


vddl-list handleDrop callback not returning the callback result

transferredObject is alway true or false but can't be, as the documentation says it here
http://hejx.space/vddl/#/component/vddl-list the result of the callback.
In the doc :
"All other return values will be treated as the object to insert into the array. In most cases you simply want to return the item parameter, but there are no restrictions on what you can return."
The reason is invokeCallback() only returns true or false, but not the result of the invoked callback.
Current version :
    invokeCallback(expression, event, index, item) {
      var fn = this[expression];
      if (fn) {
       fn({
          event: event,
          index: index,
          item: item || undefined,
          list: this.list,
          external: !this.vddlDragTypeWorkaround.isDragging,
          type: this.vddlDragTypeWorkaround.isDragging ? this.vddlDragTypeWorkaround.dragType : undefined
        });
      }
      return fn ? true : false;
    },

Suggestion :
    invokeCallback(expression, event, index, item) {
      var fn = this[expression],
          ret = true;
      if (fn) {
        ret = fn({
          event: event,
          index: index,
          item: item || undefined,
          list: this.list,
          external: !this.vddlDragTypeWorkaround.isDragging,
          type: this.vddlDragTypeWorkaround.isDragging ? this.vddlDragTypeWorkaround.dragType : undefined
        });
      }
      return fn ? ret : false;
    },

@partikule I am sorry for late reply.
If there is a callback method, when the invokeCallback is executed, the callback method is executed directly and the parameters are returned. Returns true or false because other places need to determine if there is a callback.

Hi Heji !
Thanks for your answer !
I finally managed that without modify the method.

This still does not make sense to me. It does not work as described in the documentation as far as I can see.

If there is a callback method, when the invokeCallback is executed, the callback method is executed directly and the parameters are returned

What does this mean? Where is the result returned to?


Subscriptions - Unable to disable auto renew

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.81-1
Reproducible in staging?: y
Reproducible in production?: y
Issue found when executing PR: https://github.com/Expensify/App/pull/42990
Issue reported by: Applause - Internal Team
Action Performed:

Log in with a new Gmail account
Create a workspace
Navigate to https://staging.new.expensify.com/settings/subscription
Click on the Auto-renew toggle
Select any reason
Click on the "Submit" button

Expected Result:
I should be able to disable auto renew.
Actual Result:
Unable to disable auto renew. Nothing happens when I click on the "Submit" button
Workaround:
n/a
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/93399543/88d46723-58d6-44a6-ae0b-9c43e0c3e8e0
View all open jobs on GitHub

@strepanier03 FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors.

the same issue reported here https://github.com/Expensify/App/issues/43466, closing


Fix import statements for case-sensitive OSs

This is re: #30

Thank you! Should we also fix the paths in the demo folder? I can merge this and do them myself if you don't have time.

@mderrick This pr fixes imports in both demo/ and src/ :)

My bad, thank you :)


New Exercise: Promises

I thought it would be useful to add an exercise which focuses on Promises since they're such a core part of JS development. My idea was to ask learners to implement a function that behaves like NodeJS's util.promisify, as well as the Promise static helper methods in native JS:

Promise.all
Promise.allSettled
Promise.race
Promise.any

I've included some tests for each function as a starting point and an example implementation of all the functions. I'd love some feedback and additional contributions. I think this would make a great language-specific exercise!

@slaymance before reviewing, can you make the CI's pass?

@slaymance before reviewing, can you make the CI's pass?

Thanks <3

Thanks <3

@slaymance I performed the tasks, added introduction, added .meta/config files and added you as the Author.
Thanks for contributing :)


top get computed to negative value with large tooltips when not setting max-height to dom immediately in apply fn

If you dont update the dom immediately in your apply function in the size middleware the top value for the first render of the tooltip get calculated incorrectly  and the tooltip can even be positioned off screen.
Here's a codesandbox showing the bug. When you toggle the tooltip (only the first time) its not positioned at the top of the screen.  I emulated frameworks no updating immediately with a setTimeout. Try pressing the rightmost button to disable it and toggle the tooltip. It should now be calculated correctly.
https://codesandbox.io/s/floating-ui-dom-template-forked-yqiiws?file=/src/index.js
Frameworks often do not update the dom immediately. I see that you can use flushSync with react to do it (or just update the dom directly, so bypassing react).
I think this should be noted clearly in the docs of size and apply.
I am using ember and I have found that just changing this from:
    apply?.({
      ...middlewareArguments,
      ...dimensions,
    });

to
    await apply?.({
      ...middlewareArguments,
      ...dimensions,
    });

allows me to not have to bypass ember and update the dom directly by returning a promise from my apply function that resolves after ember has rendered, like this:
apply({
  availableHeight,
  availableWidth,
  x,
  y,
}) {
  this.maxHeight = `${availableHeight}px`;
  this.maxWidth = `${availableWidth}px`;

  this.left = `${x}px`;
  this.top = `${y}px`;

  return new Promise((resolve) => {
    schedule('afterRender', this, resolve)
  })

I guess other frameworks also have ways to wait for a render to have been finished.
If you want I can make a pr adding the await :)

Seems fine but I think this reduced the updating perf a bit, mutating it directly was faster (in Chrome at least)


Can't find sales_order_status_label table with prefix

If you are using a prefix on table for Magento2 DB, this tool gives you an error on "Sales Products" on 7th step because can't find sales_order_status_label table in database. To solve this you can create the table without prefix and go next, when the step is complete you can rename the table (deleting previous existing table with prefix) or migrate these datas manually.

@davideperrotta
You can open the file at path: protected/models/db/mage2/Mage2SalesOrderStatusLabelPeer.php
and find to the code line: return 'sales_order_status_label';
and replace it by code line: return '{{sales_order_status_label}}';
We are missing this in this model class.
Regards,
Mall.


Allow to track loop iterations using a user defined code call

The idea is that this delegate that we inject allow us to verify that the scripts isn't doing BAD THINGS.
See fuller discussion of this and the implications of what it enables here:
https://ayende.com/blog/179457/optimizing-javascript-and-solving-the-halting-problem-part-i?key=cf628fb3c9ae4f4d9dec980d04886248
https://ayende.com/blog/179458/optimizing-javascript-and-solving-the-halting-problem-part-ii?key=c9ab77877d474eb694fc4b901965177f

Very interesting, thanks. Being robust to badly behaving and actively malicious scripts is definitely of interest to quite a few people, but I've resisted doing anything about it because the only truly robust method is to run the malicious code in a separate OS process. Unfortunately, this solution is difficult and in a lot of cases (e.g. UWP) it's not even possible.
I can't think of a way that this + recursion limit + no CLR calls gives you a way to damage the parent process.
Note that we assume that actually handling the policy such as timeout / # of allocations is done by the supplied delegate.
Can you think about a way this will still open us some issues?

You're hooking into the built-in loop statements, but there are many other ways to execute lots of code.
Some I can think of:

Add tons of code to a string variable and eval() it.
Use Array.prototype.forEach.
Call methods that take a long time to complete (like Array.prototype.sort for example).

Hi,
note that I did similar changes in my branch implementScriptTimeout2 - see #85: At the beginning of every loop (and at the beginning of user-defined functions) I check a flag, and if it is true, a ScriptCanceledException will be thrown in order to cancel the script (in this case, finally blocks aren't executed any more as they now will only be executed if the exception could have been caught by JS code (if it is a JavaScriptException).
(I originally created code of the ScriptTimeoutHelper in the wiki page as that approach seemed to work for me, until I found that it does not work for JS code in finally clauses where a ThreadAbortException will not be thrown.)
I think that you should also check the delegate when user-defined functions are invoked, in order to handle the Array.prototype.forEach case that @paulbartrum mentioned, and because you can construct functions without a loop that run lots of operations (without exceeding a reasonable recursion depth limit), e.g.
function func(n) {
    if (n < 15) {
        func(n + 1);
        func(n + 1);
        func(n + 1);
        func(n + 1);
        func(n + 1);
        func(n + 1);
    }
}
func(1);


Bug: MPQs > 2GB fail to load

This appears to be two issues:
Incorrect llseek behavior when seeking >= 2 ** 31
These lines in the emscripten FS layer do not play well with offsets beyond 2 ** 31 bytes.
var stream = SYSCALLS.getStreamFromFD(), offset_high = SYSCALLS.get(), offset_low = SYSCALLS.get(), result = SYSCALLS.get(), whence = SYSCALLS.get();
// NOTE: offset_high is unused - Emscripten's off_t is 32-bit
var offset = offset_low;
FS.llseek(stream, offset, whence);

offset_low is read as a signed integer out of HEAP32 by SYSCALLS.get(). As a result, the value wraps after exceeding 2,147,483,647 bytes, even though it's not surprising to seek past that limit when working with large files.
Runtime assert intersecting with incompatible types
This line in SBaseFileTable.cpp fails, despite the values matching:
assert(pHeader->BlockTableSize64 <= (pHeader->dwBlockTableSize * sizeof(TMPQBlock)));

Presumably, some combination of the types of pHeader->BlockTableSize64, pHeader->dwBlockTableSize, and sizeof(TMPQBlock) prevent the assert's equality comparison from returning true.

@fallenoak Does this fix help at all with other files and devices larger than 2 ** 31? I am running into a similar problem with a device representing any file above that size.

@Auron52 The workaround I implemented should permit seeking past 2 ** 31 bytes for anything that uses the llseek syscall (ie. ___syscall140 in Emscripten). I wasn't clear if negative offsets are permitted, so it only works for llseek in SEEK_SET mode; it won't work for anything that uses llseek in SEEK_CUR or SEEK_END mode.
https://github.com/wowserhq/stormjs/blob/9e539882d77574125e5b86933a3ee386b8f6f64b/src/binding/post.js
I believe a proper workaround would be updating Emscripten to treat off_t as 64-bits, but that probably requires a bunch of work involving BigInt on the JS side.

Thanks @fallenoak. Were you able to call llseek from C/C++ code? (or did you call it directly from JavaScript?) If so, how did you do it? Up to this point I have been using fseek, but I assume I need to change this.


Unexpected token <

When running wh deploy on a custom server, I get:
Uploading, this might take a minute.undefined:1
<html>
^
SyntaxError: Unexpected token <
    at Object.parse (native)
    at Request._callback (C:\Users\Max\AppData\Roaming\npm\node_modules\wh\lib\push.js:199:21)
    at Request.self.callback (C:\Users\Max\AppData\Roaming\npm\node_modules\wh\node_modules\request\request.js:123:22)
    at Request.emit (events.js:110:17)
    at Request.<anonymous> (C:\Users\Max\AppData\Roaming\npm\node_modules\wh\node_modules\request\request.js:893:14)
    at Request.emit (events.js:129:20)
    at IncomingMessage.<anonymous> (C:\Users\Max\AppData\Roaming\npm\node_modules\wh\node_modules\request\request.js:844:12)
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

wh serve has no issues

I get this intermittently, and it's odd because it will generally go away on subsequent builds, but nothing seems to help consistently, including resetting my running processes on the server. I've done troubleshooting on the server and local files to no avail.

I just looked into the push.js that is causing the problem. It comes from JSON.parseing a server response that isn't actually JSON. In my case, the response is a 413 error (request entity too large). Adding client_max_body_size 10M to the server config in Nginx solved the problem for me.


paths override in .cargo/config stop working

After update nighty rust 13 November 2016 from first 1.14.0 nighty I get error on cargo build when building https://github.com/gtk-rs's gtk-related crates
error: failed to find lock source of glib-sys v0.3.1 (file:///home/eap/testdir/sys/glib-sys)

Maybe it related to #3191 but error is different (may be due using supcrates).
Versions:
rustc --version
rustc 1.14.0-nightly (cae6ab1c4 2016-11-05)
cargo --version
cargo 0.13.0-nightly (9c47815 2016-11-04)
uname -a
Linux EAPCOMP3 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce:
create folder and go to it (/home/eap/testdir in my case)
mkdir .cargo
nano .cargo/config
paths = [
 "/home/eap/testdir/sys/glib-sys",
 "/home/eap/testdir/sys/gobject-sys",
]
save and exit nano
git clone https://github.com/gtk-rs/glib.git
git clone https://github.com/gtk-rs/sys.git
cd cairo
cargo build

Also I get warning:
warning: path override for crate `gobject-sys` has altered the original list of
dependencies; the dependency on `glib-sys` was either added or
modified to not match the previously resolved version

This is currently allowed but is known to produce buggy behavior with spurious
recompiles and changes to the crate graph. Path overrides unfortunately were
never intended to support this feature, so for now this message is just a
warning. In the future, however, this message will become a hard error.

To change the dependency graph via an override it's recommended to use the
`[replace]` feature of Cargo instead of the path override feature. This is
documented online at the url below for more information.

http://doc.crates.io/specifying-dependencies.html#overriding-dependencies

But currently paths prefered for me as gtk-rs contains many coupled gtk-related crates.

Same error on Windows Msys64 with GNU ABI
eap@EAPCOMP3 MSYS ~
$ uname -a
MSYS_NT-10.0 EAPCOMP3 2.6.0(0.304/5/3) 2016-09-07 20:45 x86_64 Msys
eap@EAPCOMP3 MINGW64 ~
$ uname -a
MINGW64_NT-10.0 EAPCOMP3 2.6.0(0.304/5/3) 2016-09-07 20:45 x86_64 Msys

Last working version in archive https://static.rust-lang.org/dist/2016-10-04/index.html
rustc --version
rustc 1.14.0-nightly (9c31d76e9 2016-10-03)
cargo --version
cargo 0.13.0-nightly (ebcf0a8 2016-10-03)

Thanks for the report! Can you gist the full error here? (including the output of -v)
Note that the warning printed out is legitimate in the sense that it will likely cause spurious recompiles, spurious updates, etc, which is why [replace] is recommended.

https://gist.github.com/EPashkin/3ec94e7e5578d14446d872952173ff37

Updated after removing cargo.lock

Oh looks like this is fixed by https://github.com/rust-lang/cargo/pull/3251, which hasn't been merged yet

Thanks, I wait for merge


Added another array for tracking processed vs unprocessed requests.

Setup a second array for tracking purposes of what IPs have been pinged vs not.
-- With this in place, we will now only return the processed pings.
General function logging updates.
Moved PingQoSWorker functions into a cpp file.

Thanks bro!


Latest builds failing (lib-dynload file already exists)

Details

Project URL:  http://springboard-core.readthedocs.io/en/latest/
Build URL (if applicable): https://readthedocs.org/projects/springboard-core/builds/6167062/
Read the Docs username (if applicable): gajop

Expected Result
Builds should work as normally.
Actual Result
Builds seem to fail for bizarre reasons.
OSError: [Errno 17] File exists: '/home/docs/checkouts/readthedocs.org/user_builds/springboard-core/envs/latest/lib/python2.7/lib-dynload'

Same exact problem here, for about the last 5 hours.
python3.5 -mvirtualenv --no-site-packages --no-download /home/docs/checkouts/readthedocs.org/user_builds/pymiescatt/envs/latest

Seeing this too:
https://readthedocs.org/projects/conx/builds/6170070/

Looks like it has been fixed?

Hasn't been fixed for me.
`python3.5 -mvirtualenv --no-site-packages --no-download /home/docs/checkouts/readthedocs.org/user_builds/pymiescatt/envs/latest
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 351, in copyfile
os.symlink(srcpath, dest)
FileExistsError: [Errno 17] File exists: '/usr/lib/python3.5/lib-dynload' -> '/home/docs/checkouts/readthedocs.org/user_builds/pymiescatt/envs/latest/lib/python3.5/lib-dynload'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 2328, in
main()
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 713, in main
symlink=options.symlink)
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 925, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 1130, in install_python
copyfile(join(stdlib_dir, fn), join(lib_dir, fn), symlink)
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 354, in copyfile
copyfileordir(src, dest, symlink)
File "/usr/local/lib/python3.5/dist-packages/virtualenv.py", line 329, in copyfileordir
shutil.copytree(src, dest, symlink)
File "/usr/lib/python3.5/shutil.py", line 309, in copytree
os.makedirs(dst)
File "/usr/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/docs/checkouts/readthedocs.org/user_builds/pymiescatt/envs/latest/lib/python3.5/lib-dynload'
Using base prefix '/usr'`

Same problem here. Some examples:

https://readthedocs.org/projects/bezier/builds/6173086/
https://readthedocs.org/projects/bezier/builds/6171629/
https://readthedocs.org/projects/bezier/builds/6167853/
https://readthedocs.org/projects/bezier/builds/6167745/

It seems to have been fixed for me (https://readthedocs.org/projects/springboard-core/builds/6173917/ passed). I haven't done anything, except reissuing the build from the online interface.
I have no idea if the fix is going to persist though.

Same problem here since this afternoon :
https://readthedocs.org/projects/passhport-fr/builds/

We saw this last week too:
https://readthedocs.org/projects/treeherder/builds/6178579/
https://readthedocs.org/projects/treeherder/builds/6176994/
https://readthedocs.org/projects/treeherder/builds/6176988/

Looks like all the docs here now build with no problem, anyone still with this issue?
I think this is a virtualenv error, when running parallel processes (probably similar to https://github.com/conda/conda/issues/2837)

anyone still with this issue?

Nope

Let's reopen if anyone still has it. Thanks all.


Removing comment that is not needed

The DA is now  working after the ZCLR7.5 update then using PARMS instead of ARGS so no need for comments on the code.

Codecov Report

Merging #1513 (b6f4615) into dev (16f298e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##              dev    #1513   +/-   ##
=======================================
  Coverage   80.36%   80.36%
=======================================
  Files         227      227
  Lines        6704     6704
=======================================
  Hits         5388     5388
  Misses       1316     1316

Impacted Files
Coverage Δ

zhaquirks/tuya/ts004f.py
80.64% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16f298e...b6f4615. Read the comment docs.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 80.37%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
5388

Relevant Lines:
6704

💛  - Coveralls


Saml update metadata not working

We are having an intermittent saml issue, we see the following error "org.opensaml.common.SAMLException: Response doesn't have any valid assertion which would pass subject validation" , underlying exception is "SAMLProcessingFilter: Caused by: org.opensaml.xml.validation.ValidationException: Signature is not trusted or invalid". The interesting part is that the metadata for the idp is outdated, the certs were changed on the idp side recently and there was never a metadata update in UAA. However some requests are still passing and it fails intermittently instead of always. In this case we have multiple instances of UAA and multiple identity zones configured to use the same external idp.
Steps to reproduce in isolation:
We have a sample app, and one instance of UAA deployed in Cloud foundry. We configured the app to use an external identity provider. Our sample app was working fine and then we updated the idp metadata with incorrect certs. And still the app continued to work fine. We restarted UAA and then our sample app started failing. We then updated the idp metadata with the correct certs and still our app was failing until we restarted the UAA again.
When we get the idp config between these steps, the idp metadata is up to date. There seems to be some sort of cache based on what we observed.

A bit clarification - when there is intermittent failures on the sample app @bondugula  is referring to,  there are other identity zones in the uaa which do have the correct metadata.
@fhanik  this is the same issue we had discussed over slack - except we incorrectly concluded that a restart fixes the issue. This was because I didnt realize that the correct metadata/key was being picked from a global cache benefiting from correct metadata from a different identity zone.
The cache you had identified is in org.opensaml.security.MetadataCredentialResolver.

To further clarify if needed, the sample app, when pointed to the main UAA, should always be failing because we corrupted its metadata. The intermittent failures are more specifically intermittent successes caused by the cache presumably using correct credentials from other zones.

This issue has been addressed .


{EventHub}Fixed typo for --cluster-arm-id

Fixed typo for --cluster-arm-id fixes #20416
Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

fixes #20416

EventHub

@jiasli Could you please merge this PR once you get a chance ?


KAFKA-14299: Handle double rebalances better

The original implementation of the state updater could not
handle double rebalances within one poll phase correctly,
because it could create tasks more than once if they hadn't
finished initialization yet.
In a55071a99fabc9a706afa0e9acddf898c7cd05c4, we
moved initialization to the state updater to fix this. However,
with more testing, I found out that this implementation has
it's problems as well: There are problems with locking the
state directory (state updater acquired the lock to the state
directory, so the main thread wouldn't be able to clear the
state directory when closing the task), and benchmarks also
show that this can lead to useless work (tasks are being
initialized, although they will be taken from the thread soon
after in a follow-up rebalance).
In this PR, I propose to revert the original change, and fix
the original problem in a much simpler way: When we
receive an assignment, we simply clear out the
list of tasks pending initialization. This way, no double
tasks instantiations can happen.
The change was tested in benchmarks, system tests,
and the existing unit & integration tests. We also add
the state updater to the smoke integration test, which
triggered the double task instantiations before.
Committer Checklist (excluded from commit message)

[x] Verify design and implementation
[x] Verify test coverage and CI build status
[x] Verify documentation (including upgrade notes)

@cadonna could you please have a look?


Revert archived common-alpine feature

Currently the feature stored at src/common-utils is apaired down version of the older common-alpine feature.
Previously having a common-alpine as an available feature was a great help for those seeking a linux dev-container sensible settings.
With this being said, there are aspects of the dev-container feature that were very useful, and have since been removed/missing. Support for additional supplementary groups and specifying additional packages are vital to development.

Hi 👋
When we migrated over the common-utils Feature, we ensured to keep the installations and Feature options intact.

Support for additional supplementary groups

From https://github.com/devcontainers/features/tree/main/src/common-utils#options, the Feature has options to setup user groups. Kindly see username, userUid and userGid for reference.

specifying additional packages are vital to development.

Unfortunately, I don't see how we were adding additional packages to common-alpine.sh. Can you help me understand this point? thanks!
Let me know if you have any other questions, thanks!


Dynamic Component Transition Not Working >3.2.33

Vue version

3.2.34

Link to minimal reproduction
https://stackblitz.com/edit/vitejs-vite-dd7b22
Steps to reproduce
-> npm run dev
-> click the button to switch pages
-> the transition does not work
What is expected?
The page is expected to transition, as it does prior to version 3.2.34
What is actually happening?
The page is not transitioning. It only works if you remove mode="out-in" from the Transition tag.
System Info
No response
Any additional comments?
No response

You should change to this.
<template>
++

Page1

    <button @click="page = 'Page2'">click to switch to Page2</button>
++

</template>

The console has given a warning:


Use more adequate OpenSSL key

Use something like let's encrypt to provide SSL other than self-signed.

Lets make it simpler


[Feature request] Nostr Wallet Connect support

It'd be really cool to have NWC support in the app - That way you could easily send zaps from other devices.

Hi, thank you for your suggestion.
However, as of now I am unable to envisage NWC-like integration that would be practical to use. Minibits keeps the ecash directly on device and does all necessary cryptographic operation to mint / melt / swap on device.
That means that for NWC to work, app would need to be running AND remain in the foreground (otherwise Android will freeze or limit network connections and cpu heavy operations).
That's why zaps are implemented as a deeplink that passes the invoice from nostr client yet makes sure to wake up the wallet to pay it. If somebody comes with a viable concept except managing ecash on always-on server, let's do it.

Well, things change over time. Initial proof of concept integration of Nostr Wallet Connect has just landed in recent over the air Minibits update!
In this experimental implementation, NWC requests are routed to devices over push notification messages from Minibits server. That way, Minibits wallet can be backgrounded or even killed, then wake up on receiving NWC request event, decrypt the invoice and pay it.
Please note, that push notifications must be working and enabled on your device. As well, it will take time to make the implementation stable and such background NWC and payment processing might be suppressed by operating system depending on device type, battery level etc.
Will be glad if you test the implementation, will keep the issue open.

Neat! Thanks for taking a look at my suggestion! 🎉
I tried connecting Alby in my browser to Minibits with NWC and sending some zaps. However it looks like almost all of the Lightning payments fail. I get an error like "Pay invoice error: Lightning payment failed". I'm connected just to the Minibits mint and zapping various people.

Where do you put connection string in Alby? I know only about Alby providing one as a wallet, similar to Minibits, just fully on server. Thanks, I'll test then. For now, Amethyst and Primal should be ok for experimenting.

It's in Connect a Wallet > Bring Your own wallet > Nostr Wallet Connect


Application Dependency not working (spec.deps.packages is ignored) and (sparkConf:spark.jars.packages not propagated to executor)

Application dependencies added using the deps.packages seem to be ignored. It should work according to the docs but other users have mentioned in this issue #352 that they are facing the same problem.
Excerpt from yaml file:
spec:
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0

No mention of loading these packages in the Driver logs. It then fails with this error:
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=<IP_ADDRESS> --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class com.ketan.KafkaAvro local:///tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar kafka.kd-confluent.svc.cluster.local:9071 /tmp/data
21/01/15 04:22:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/01/15 04:22:29 INFO SparkContext: Running Spark version 3.0.0
...
21/01/15 04:22:44 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/opt/spark/work-dir/spark-warehouse').
21/01/15 04:22:44 INFO SharedState: Warehouse path is 'file:/opt/spark/work-dir/spark-warehouse'.
Exception in thread "main" org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide".;
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:666)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:194)
at com.ketan.StreamsProcessor.process(KafkaAvro.scala:57)
at com.ketan.KafkaAvro$.main(KafkaAvro.scala:28)
at com.ketan.KafkaAvro.main(KafkaAvro.scala)

I tried a workaround as mentioned in the same issue 352, which has worked for some people recently.
spec:
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0"

But that failed with this error (from the Driver logs)
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=<IP_ADDRESS> --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class com.ketan.KafkaAvro local:///tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar kafka.kd-confluent.svc.cluster.local:9071 /tmp/data
Ivy Default Cache set to: /opt/spark/.ivy2/cache
The jars for the packages stored in: /opt/spark/.ivy2/jars
:: loading settings :: url = jar:file:/opt/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-sql-kafka-0-10_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-7c17c84c-a8ab-48c8-ade4-26e4b9bb930b;1.0
confs: [default]
Exception in thread "main" java.io.FileNotFoundException: /opt/spark/.ivy2/cache/resolved-org.apache.spark-spark-submit-parent-7c17c84c-a8ab-48c8-ade4-26e4b9bb930b-1.0.xml (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:70)
at org.apache.ivy.plugins.parser.xml.XmlModuleDescriptorWriter.write(XmlModuleDescriptorWriter.java:62)
at org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.toIvyFile(DefaultModuleDescriptor.java:563)
at org.apache.ivy.core.cache.DefaultResolutionCacheManager.saveResolvedModuleDescriptor(DefaultResolutionCacheManager.java:176)
at org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:245)
at org.apache.ivy.Ivy.resolve(Ivy.java:523)
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1387)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)

I made the change to include spark.jars.ivy, as explained here. A volume for the ivy directory had to be added and mounted in both the driver and executor specs.
spec:
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"
    spark.jars.ivy: "/tmp/ivy"

Now the driver is able to find the dependencies and go further. However these options are not being passed to the executor and  it does not see any of the dependencies. It fails with java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition as below:
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_EXECUTOR_JAVA_OPTS[@]}" -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH" org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP)
+ exec /usr/bin/tini -s -- /usr/local/openjdk-8/bin/java -Dspark.driver.blockManager.port=7079 -Dspark.driver.port=7078 -Xms512m -Xmx512m -cp ':/opt/spark/jars/*:' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc:7078 --executor-id 1 --cores 1 --app-id spark-0a6417c1b3284e97a27f617c0f2dd034 --hostname <IP_ADDRESS>
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/01/18 14:10:13 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 14@kafka-avro-9aba567715d4cb68-exec-1
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for TERM
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for HUP
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for INT
21/01/18 14:10:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/01/18 14:10:14 INFO SecurityManager: Changing view acls to: 185,root
21/01/18 14:10:14 INFO SecurityManager: Changing modify acls to: 185,root
21/01/18 14:10:14 INFO SecurityManager: Changing view acls groups to:
21/01/18 14:10:14 INFO SecurityManager: Changing modify acls groups to:
21/01/18 14:10:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(185, root); groups with view permissions: Set(); users  with modify permissions: Set(185, root); groups with modify permissions: Set()
21/01/18 14:10:15 INFO TransportClientFactory: Successfully created connection to kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc/<IP_ADDRESS>:7078 after 129 ms (0 ms spent in bootstraps)
21/01/18 14:10:15 INFO SecurityManager: Changing view acls to: 185,root
21/01/18 14:10:15 INFO SecurityManager: Changing modify acls to: 185,root
21/01/18 14:10:15 INFO SecurityManager: Changing view acls groups to:
21/01/18 14:10:15 INFO SecurityManager: Changing modify acls groups to:
21/01/18 14:10:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(185, root); groups with view permissions: Set(); users  with modify permissions: Set(185, root); groups with modify permissions: Set()
21/01/18 14:10:15 INFO TransportClientFactory: Successfully created connection to kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc/<IP_ADDRESS>:7078 after 5 ms (0 ms spent in bootstraps)
21/01/18 14:10:15 INFO DiskBlockManager: Created local directory at /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/blockmgr-02756d41-fd31-4d4d-a650-ceb2fe83a533
21/01/18 14:10:15 INFO MemoryStore: MemoryStore started with capacity 117.0 MiB
21/01/18 14:10:15 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc:7078
21/01/18 14:10:15 INFO ResourceUtils: ==============================================================
21/01/18 14:10:15 INFO ResourceUtils: Resources for spark.executor:
21/01/18 14:10:15 INFO ResourceUtils: ==============================================================
21/01/18 14:10:16 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
21/01/18 14:10:16 INFO Executor: Starting executor ID 1 on host <IP_ADDRESS>
21/01/18 14:10:16 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44843.
21/01/18 14:10:16 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:44843
21/01/18 14:10:16 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/01/18 14:10:16 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:16 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:16 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:30 INFO CoarseGrainedExecutorBackend: Got assigned task 0
21/01/18 14:10:31 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/01/18 14:10:31 INFO Executor: Fetching file:/tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar with timestamp<PHONE_NUMBER>865
21/01/18 14:10:31 INFO Utils: Copying /tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar to /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/spark-119a54b8-42c9-42e5-9f26-c85c64f0605a/-21118480811610979002865_cache
21/01/18 14:10:31 INFO Utils: Copying /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/spark-119a54b8-42c9-42e5-9f26-c85c64f0605a/-21118480811610979002865_cache to /opt/spark/work-dir/./kafka-avro_2.12-1.0.jar
21/01/18 14:10:31 INFO Executor: Adding file:/opt/spark/work-dir/./kafka-avro_2.12-1.0.jar to class loader
21/01/18 14:10:31 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)

I made the change to include spark.jars.ivy, as explained here. A volume for the ivy directory had to be added and mounted in both the driver and executor specs.
spec:
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"
    spark.jars.ivy: "/tmp/ivy"

Now the driver is able to find the dependencies and go further. However these options are not being passed to the executor and  it does not see any of the dependencies. It fails with java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition as below:
+ CMD=(${JAVA_HOME}/bin/java "${SPARK_EXECUTOR_JAVA_OPTS[@]}" -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp "$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH" org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url $SPARK_DRIVER_URL --executor-id $SPARK_EXECUTOR_ID --cores $SPARK_EXECUTOR_CORES --app-id $SPARK_APPLICATION_ID --hostname $SPARK_EXECUTOR_POD_IP)
+ exec /usr/bin/tini -s -- /usr/local/openjdk-8/bin/java -Dspark.driver.blockManager.port=7079 -Dspark.driver.port=7078 -Xms512m -Xmx512m -cp ':/opt/spark/jars/*:' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc:7078 --executor-id 1 --cores 1 --app-id spark-0a6417c1b3284e97a27f617c0f2dd034 --hostname <IP_ADDRESS>
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/01/18 14:10:13 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 14@kafka-avro-9aba567715d4cb68-exec-1
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for TERM
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for HUP
21/01/18 14:10:13 INFO SignalUtils: Registered signal handler for INT
21/01/18 14:10:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/01/18 14:10:14 INFO SecurityManager: Changing view acls to: 185,root
21/01/18 14:10:14 INFO SecurityManager: Changing modify acls to: 185,root
21/01/18 14:10:14 INFO SecurityManager: Changing view acls groups to:
21/01/18 14:10:14 INFO SecurityManager: Changing modify acls groups to:
21/01/18 14:10:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(185, root); groups with view permissions: Set(); users  with modify permissions: Set(185, root); groups with modify permissions: Set()
21/01/18 14:10:15 INFO TransportClientFactory: Successfully created connection to kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc/<IP_ADDRESS>:7078 after 129 ms (0 ms spent in bootstraps)
21/01/18 14:10:15 INFO SecurityManager: Changing view acls to: 185,root
21/01/18 14:10:15 INFO SecurityManager: Changing modify acls to: 185,root
21/01/18 14:10:15 INFO SecurityManager: Changing view acls groups to:
21/01/18 14:10:15 INFO SecurityManager: Changing modify acls groups to:
21/01/18 14:10:15 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(185, root); groups with view permissions: Set(); users  with modify permissions: Set(185, root); groups with modify permissions: Set()
21/01/18 14:10:15 INFO TransportClientFactory: Successfully created connection to kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc/<IP_ADDRESS>:7078 after 5 ms (0 ms spent in bootstraps)
21/01/18 14:10:15 INFO DiskBlockManager: Created local directory at /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/blockmgr-02756d41-fd31-4d4d-a650-ceb2fe83a533
21/01/18 14:10:15 INFO MemoryStore: MemoryStore started with capacity 117.0 MiB
21/01/18 14:10:15 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@kd-kafkaavro-3a99e07715d477b2-driver-svc.spark-app.svc:7078
21/01/18 14:10:15 INFO ResourceUtils: ==============================================================
21/01/18 14:10:15 INFO ResourceUtils: Resources for spark.executor:
21/01/18 14:10:15 INFO ResourceUtils: ==============================================================
21/01/18 14:10:16 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
21/01/18 14:10:16 INFO Executor: Starting executor ID 1 on host <IP_ADDRESS>
21/01/18 14:10:16 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44843.
21/01/18 14:10:16 INFO NettyBlockTransferService: Server created on <IP_ADDRESS>:44843
21/01/18 14:10:16 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/01/18 14:10:16 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:16 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:16 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, <IP_ADDRESS>, 44843, None)
21/01/18 14:10:30 INFO CoarseGrainedExecutorBackend: Got assigned task 0
21/01/18 14:10:31 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
21/01/18 14:10:31 INFO Executor: Fetching file:/tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar with timestamp<PHONE_NUMBER>865
21/01/18 14:10:31 INFO Utils: Copying /tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar to /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/spark-119a54b8-42c9-42e5-9f26-c85c64f0605a/-21118480811610979002865_cache
21/01/18 14:10:31 INFO Utils: Copying /var/data/spark-07f269fd-4b5a-428e-90b1-90e9a250c0d0/spark-119a54b8-42c9-42e5-9f26-c85c64f0605a/-21118480811610979002865_cache to /opt/spark/work-dir/./kafka-avro_2.12-1.0.jar
21/01/18 14:10:31 INFO Executor: Adding file:/opt/spark/work-dir/./kafka-avro_2.12-1.0.jar to class loader
21/01/18 14:10:31 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)

@liyinan926 any suggestions? Appreciate any help you can provide.
The following is getting ignored altogether.
spec:
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0

And this works fine on the driver but is not getting passed to the executor at all.
spec:
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"

To debug this a bit, I tried adding stuff like the following. In that case, the executor logs do show these jars getting copied over and it is able to find those classes. But too many more jars are required so this isn't a practical workaround.
  sparkConf:
    spark.jars: "local:///tmp/ivy/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.0.0.jar,local:///tmp/ivy/jars/org.apache.kafka_kafka-clients-2.4.1.jar"

Here's the full yaml for completeness.
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: kd-kafkaavro
  namespace: spark-app
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v3.0.0"
  imagePullPolicy: Always
  mainClass: com.ketan.KafkaAvro
  mainApplicationFile: "local:///tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar"
  arguments:
    - "kafka.kd-confluent.svc.cluster.local:9071"
    - "/tmp/data"
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0
  sparkVersion: "3.0.0"
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"
#    spark.jars: "local:///tmp/ivy/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.0.0.jar,local:///tmp/ivy/jars/org.apache.kafka_kafka-clients-2.4.1.jar,local:///tmp/ivy/jars/org.apache.commons_commons-pool2-2.6.2.jar,local:///tmp/ivy/jars/org.xerial.snappy_snappy-java-<IP_ADDRESS>.jar,local:///tmp/ivy/jars/com.github.luben_zstd-jni-1.4.4-3.jar,local:///tmp/ivy/jars/org.lz4_lz4-java-1.7.1.jar,local:///tmp/ivy/jars/org.slf4j_slf4j-api-1.7.30.jar"
    spark.jars.ivy: "/tmp/ivy"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/volumes/spark/tmp"
        type: Directory
    - name: "data"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/spark/data"
        type: Directory
    - name: "dev"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/spark/devp"
        type: Directory
    - name: "ivy"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/volumes/spark/ivy"
        type: Directory
  driver:
    coreRequest: 100m
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.0.0
    serviceAccount: sparkop-spark
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
      - name: "data"
        mountPath: "/tmp/data"
      - name: "dev"
        mountPath: "/tmp/dev"
      - name: "ivy"
        mountPath: "/tmp/ivy"
  executor:
    coreRequest: 100m
    instances: 1
    memory: "512m"
    labels:
      version: 3.0.0
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
      - name: "data"
        mountPath: "/tmp/data"
      - name: "dev"
        mountPath: "/tmp/dev"
      - name: "ivy"
        mountPath: "/tmp/ivy"

@liyinan926 any suggestions? Appreciate any help you can provide.
The following is getting ignored altogether.
spec:
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0

And this works fine on the driver but is not getting passed to the executor at all.
spec:
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"

To debug this a bit, I tried adding stuff like the following. In that case, the executor logs do show these jars getting copied over and it is able to find those classes. But too many more jars are required so this isn't a practical workaround.
  sparkConf:
    spark.jars: "local:///tmp/ivy/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.0.0.jar,local:///tmp/ivy/jars/org.apache.kafka_kafka-clients-2.4.1.jar"

Here's the full yaml for completeness.
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: kd-kafkaavro
  namespace: spark-app
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v3.0.0"
  imagePullPolicy: Always
  mainClass: com.ketan.KafkaAvro
  mainApplicationFile: "local:///tmp/dev/kafkaavro/target/scala-2.12/kafka-avro_2.12-1.0.jar"
  arguments:
    - "kafka.kd-confluent.svc.cluster.local:9071"
    - "/tmp/data"
  deps:
    packages:
      - org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0
      - org.apache.spark:spark-avro_2.12:3.0.0
  sparkVersion: "3.0.0"
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,org.apache.spark:spark-avro_2.12:3.0.0"
#    spark.jars: "local:///tmp/ivy/jars/org.apache.spark_spark-sql-kafka-0-10_2.12-3.0.0.jar,local:///tmp/ivy/jars/org.apache.kafka_kafka-clients-2.4.1.jar,local:///tmp/ivy/jars/org.apache.commons_commons-pool2-2.6.2.jar,local:///tmp/ivy/jars/org.xerial.snappy_snappy-java-<IP_ADDRESS>.jar,local:///tmp/ivy/jars/com.github.luben_zstd-jni-1.4.4-3.jar,local:///tmp/ivy/jars/org.lz4_lz4-java-1.7.1.jar,local:///tmp/ivy/jars/org.slf4j_slf4j-api-1.7.30.jar"
    spark.jars.ivy: "/tmp/ivy"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/volumes/spark/tmp"
        type: Directory
    - name: "data"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/spark/data"
        type: Directory
    - name: "dev"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/spark/devp"
        type: Directory
    - name: "ivy"
      hostPath:
        path: "/run/desktop/mnt/host/c/kd/dev/bd/volumes/spark/ivy"
        type: Directory
  driver:
    coreRequest: 100m
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.0.0
    serviceAccount: sparkop-spark
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
      - name: "data"
        mountPath: "/tmp/data"
      - name: "dev"
        mountPath: "/tmp/dev"
      - name: "ivy"
        mountPath: "/tmp/ivy"
  executor:
    coreRequest: 100m
    instances: 1
    memory: "512m"
    labels:
      version: 3.0.0
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
      - name: "data"
        mountPath: "/tmp/data"
      - name: "dev"
        mountPath: "/tmp/dev"
      - name: "ivy"
        mountPath: "/tmp/ivy"

@ketanhdoshi were you able to solve this issue?

@vvavepacket
I was able to get this to work using a workaround. I used the spec.sparkConf settings for packages, jars, and repositories rather than spec.deps"
  sparkConf:
    spark.jars.packages: "org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0,..."
    spark.jars.repositories: "https://packages.confluent.io/maven"
    spark.jars: "local:///tmp/ivy/jars/com.101tec_zkclient-0.10.jar,\
      .....
      local:///tmp/ivy/jars/com.fasterxml.jackson.core_jackson-annotations-2.9.10.jar"
    spark.jars.ivy: "/tmp/ivy"

The documented solutions did not work. I believe the problem is in Spark not in the Operator. I tried running the same programs directly using spark-submit on Kubernetes, without Operator, and encountered the same problem when using "--packages", "--jars" and "--repositories"
  deps:
    repositories:
    packages:

@ketanhdoshi do you know if there is a ticket on spark for this? If so, can you let me know which it is? I want to keep track of the updates.

@ketanhdoshi do you know if there is a ticket on spark for this? If so, can you let me know which it is? I want to keep track of the updates.

@dwsmith1983, I hadn't opened one, but I think this one is for the same issue.

same issue im facing, any updates?

is there a solution for this seems like the above PR was never merged @ocworld

@GaruGaru @rahulbhatia2702 In spark, the pr (https://github.com/apache/spark/pull/38828) is merged now.

Don't know if it is new issue but for me it is still not working - dependencies are not added to SparkContext of the driver and so they are not available on executor afterwards. I'm using official apache/spark-py images to deploy drive and executors. I was trying to deploy a spark job manually with almost the same settings as the spark operator uses and it was also not working - the same issue. Then I find out that if I remove the spark property spark.kubernetes.submitInDriver=true it works fine. Does anybody know why ist this property set?

Don't know if it is new issue but for me it is still not working - dependencies are not added to SparkContext of the driver and so they are not available on executor afterwards. I'm using official apache/spark-py images to deploy drive and executors. I was trying to deploy a spark job manually with almost the same settings as the spark operator uses and it was also not working - the same issue. Then I find out that if I remove the spark property spark.kubernetes.submitInDriver=true it works fine. Does anybody know why ist this property set?

Ok, it was my fault. With the PySpark version 3.4.0 it's actually working.
It worked for me on older Spark version in client mode with the option spark.kubernetes.submitInDriver set to false (default). With the older version of Spark In cluster mode - as the spark-operator is working only in cluster mode - it didn't work anyway.
The only question is why is spark trying to resolve dependencies in cluster mode on the submit side, as this unnecessary slows the submit process down.

@javolek I understand your concern with slow down because of resolving dependencies on a spark-operator pod.
I'm not an author about it, so the reasons are just my guessing.

To early find dependency failures. Creating pod is expensive operator I think. It is need to reserve resources and sometimes downloading docker image. Event if in public cloud like aws, gcp, and azure, it make costs. Early resolving is helping to prevent them.

To managing dependencies in a property file as configmap to simplify.

In my experience, if you reduce waiting time, you can use a persistent volume. Sharing downloaded jars path with spark-operator, driver and executors pods is to prevent download again and again.

I had the same problem when starting spark driver pod with spark-operator. My workaround is to get all the jars inside the dockerfile with wget and put them into /opt/spark/jars directory. I had hoped spark.jars.packages should have worked as expected.


feat(comp.backend):  Implement filter values for component instance

Filtering for:

issueMatchId
service
ccrn -> will be renamed to resourceName
supportGroup


fix hostdb.host_file.interval to be set at reconfig time

This fixes an issue where shortening the check interval would require waiting for the current interval to expire before the value was updated.

Cherry-picked to v10.0.x


Added create-react-library

CLI for creating reusable, modern React libraries using Rollup and create-react-app.
This enables developers to share their react projects with others

@anmolrishi not appropriate


Robocopy Right-click Menu

Description of the new feature / enhancement
Love to have Robocopy at my finger tips.
Scenario when this would be used?
Copying files fast.  Syncing (/mir) folders.
Supporting information
No response

I second that.
Also a tool that can edit, add, and customize the right click menu from Microsoft itself would be great #33

/dup #26


App crash or stutter in background on custom rom without GAPPS included , if you flash GAPPS after , then also it sttuters. ONly works with roms that have come GAPPS

I've this working without issues on a huawei tablet that has no google stuff at all. Perhaps it is an issue with your custom firmware - specifically the webview component that is provided. There's nothing that can be done in the app to work around issues in the firmware. I suggest raising a bug with the firmware providers to say to them their webview component does not work correctly.


Region with 10 or 11 rows (Corporate HQ has been removed from the lab?)

Module: 02
Lab/Demo: Load Data in Power BI Desktop
Task: 6 Configure the Region query
Step: 5 In the status bar, verify that the query has four columns and 10 rows.
Description of issue
The Region query has 11 rows instead of 10. Please validate or update instructions and screenshot.

Or, alternatively, add the Filter again (to remove Corporate HQ). This is a screenshot of the solution file for this lab:

Repro steps:

@weslbo I was going to post this today as well! Thanks Wesley!

@weslbo  good catch. I see the mistake now was due to a previous issue opened that said the primary key was no longer including a 0, and upon further investigation based on your issue, I see that the issue is that it's the ALTERNATE key with 0 value. I'll revert to old instructions in next update.

@AngieRudduck Can you please advise how soon the fix will be published?

We will apply bug fixes by end of next week.

This is a duplicate of issue #109 so I'm closing this, but fix will still go.