Delete utilities table

Table class was deprecated earlier in favor of libfort. This PR removes the obsolete dependency of utilities/printing/table.hpp as mentioned here.

Codecov Report
Base: 71.17% // Head: 66.58% // Decreases project coverage by -4.58% :warning:

Coverage data is based on head (56ad4b4) compared to base (954aba9).
Patch coverage: 64.47% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #275      +/-   ##
==========================================
- Coverage   71.17%   66.58%   -4.59%
==========================================
  Files          29       76      +47
  Lines        1020     1739     +719
  Branches      377      655     +278
==========================================
+ Hits          726     1158     +432
- Misses         19      114      +95
- Partials      275      467     +192

Impacted Files
Coverage Δ

include/pluginplay/bounds_checking.hpp
74.07% <ø> (ø)

include/pluginplay/fields/module_input.hpp
100.00% <ø> (ø)

include/pluginplay/fields/module_result.hpp
100.00% <ø> (ø)

include/pluginplay/module.hpp
68.00% <ø> (ø)

include/pluginplay/submodule_request.hpp
71.42% <ø> (ø)

include/pluginplay/utility.hpp
100.00% <ø> (ø)

src/pluginplay/cache/database/rocksdb/rocksdb.hpp
0.00% <0.00%> (ø)

src/pluginplay/detail_/submodule_request_pimpl.hpp
88.88% <0.00%> (ø)

...c/pluginplay/fields/detail_/module_input_pimpl.hpp
100.00% <ø> (ø)

.../pluginplay/fields/detail_/module_result_pimpl.hpp
77.41% <ø> (ø)

... and 70 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


create-svelte: include globals.d.ts in tsconfig

Apparently globals.d.ts was being ignored entirely. Whoops.

Perhaps we should put it in src for better organization and include src/**/*.d.ts?

+1 on putting it inside src. +1 on adding src/**/*.d.ts. Name - I don't care 😄

If I recall next js puts the d.ts in src, seems pretty reasonable to think of it as a source file to me

Moved to /src/global.d.ts as recommended above.


Nodejs updates

Mainly to fix the security updates from https://nodejs.org/en/blog/vulnerability/december-2015-security-releases/

@zimbatm could we also do some renaming as well to "fit" nodejs new versioning model ... i've added commit on top of your feel free to cherry pick it -> https://github.com/garbas/nixpkgs/commit/3413a5e45b30cce1ac2fc9c5ba8ba84b37abfdf7

@garbas good idea, added ^

@zimbatm merged, i've also fix another small thing which i forgot in the first place -> https://github.com/NixOS/nixpkgs/commit/7ac0052ad59c4a826b3db8ad49518a79b0a9c03e

Thanks Rok !
On Sun, 6 Dec 2015 22:14 Rok Garbas<EMAIL_ADDRESS>wrote:

@zimbatm https://github.com/zimbatm merged, i've also fix another small
thing which i forgot in the first place -> 7ac0052
https://github.com/NixOS/nixpkgs/commit/7ac0052ad59c4a826b3db8ad49518a79b0a9c03e
—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/11497#issuecomment-162353623.


feat: cnn mut encoder

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

72 of 76 (94.74%)  changed or added relevant lines in 3 files are covered.
7 unchanged lines in 5 files lost coverage.
Overall coverage increased (+0.2%) to 94.607%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

dooc/nets/init.py
31
35
88.57%

Files with Coverage Reduction
New Missed Lines
%

dooc/nets/drugcell.py
1
95.0%

dooc/nets/init.py
1
86.7%

dooc/models.py
1
97.62%

dooc/datasets.py
2
98.11%

dooc/pipelines.py
2
96.91%

Totals

Change from base Build<PHONE_NUMBER>4:
0.2%

Covered Lines:
614

Relevant Lines:
649

💛  - Coveralls


"!d if" and "!d for" commands are grabbing from django docs.

The doc commands !d if and !d for (and possibly others) are grabbing from django docs. The expected behavior is the python syntax for if/elif/else.
!d label.if kind of pulls the right doc, but it's the grammatical usage and not very helpful for trying to understand the usage.
Maybe there could be some dedicated tags for them?
I've written a suggestion explanation for a dedicated !if-else tag here. This is just how I might explain it. Maybe it should be simplified, but I'll leave it to the pros.
if condition:
    # condition can be a variable or expression.
    # Ex. if condition > 0:
    print('condition is True'.)
elif condition2:
    # elif is optional and can be used to test the same condition
    # or another condition.
    print('condition is False, but condition2 is True.')
else:
    # The default action if the previous conditions are False
    print('Both conditions were False.')

If you are interested in writing a tag, go for it and can also open a PR for it.
The problem of the doc command fetching the Django template syntax for if-else will need some digging and investigation.

The django/python docs conflict has been solved in #1606. I've updated your issue to reflect the tag suggestion.


Fix issue with option disappearing when selected for new contacts

Details

Fixed Issues

$ https://github.com/Expensify/App/issues/29917
PROPOSAL:
Tests

Same as QA
Offline tests

Same as online
QA Steps

Create a new split / group chat
Select multiple participants
Search for a contact you have not chatted with and is not in your contacts list
Add them to the group and verify the option does not disappear

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that the left part of a conditional rendering a React component is a boolean and NOT a string, e.g. myBool && <MyComponent />.
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product is localized by adding it to src/languages/* files and using the translation method

[x] If any non-english text was added/modified, I verified the translation was requested/reviewed in #expensify-open-source and it was approved by an internal Expensify engineer. Link to Slack message:

[x] I verified all numbers, amounts, dates and phone numbers shown in the product are using the localization methods
[x] I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] If we are not using the full Onyx data that we loaded, I've added the proper selector in order to ensure the component only re-renders when the data it is using changes
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If any new file was added I verified that:

[x] The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))

[x] If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
[x] If a new page is added, I verified it's using the ScrollView component to make it scrollable when more elements are added to the page.
[x] If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: Native

Android: mWeb Chrome

iOS: Native

iOS: mWeb Safari

https://github.com/Expensify/App/assets/30609178/76c7886d-3d81-4226-a1f7-a315aea899af

MacOS: Chrome / Safari

https://github.com/Expensify/App/assets/30609178/8b57f395-521d-4141-b394-75e2bade4991

MacOS: Desktop
https://github.com/Expensify/App/assets/30609178/ee01f410-c980-4eb8-8345-d5f2f12880b8

hmm that's actually a pretty good suggestion, let me implement it rq

https://github.com/Expensify/App/assets/30609178/1adca8a4-cd3d-4df7-8988-51962ebdce69
I like the idea, though after testing it I'm realizing the only downside(?) is that a selected participant moves from one of the lists to the selected list and it creates this jump when selecting / unselecting

The header message is a separate thing, but I'm more mentioning the bouncing around with search. It's more prominent when you're searching but this is the example
With your changes:
https://github.com/Expensify/App/assets/30609178/6a0a3ad6-e798-4136-a8fd-4f2f90f20860
Currently:
https://github.com/Expensify/App/assets/30609178/4d5b1b21-61d7-4961-815f-196c0fc35f49

For right now, I think the second feels a bit smoother so let's just continue with that now but once we get more feedback on the UX we can explore different explanations, but thanks for the suggestion!

@allroundexperts Just updated hahah ⚡

I checked out this PR, and am seeing major lag (3-4s) when selecting participants in a high traffic account (large personal details array)

I think we can improve the lag in a follow-up PR, I will merge the PR since it's impacting the selection of participants

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Cherry-picked to staging by https://github.com/francoisl in version: 1.3.87-4 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@Expensify/applauseleads please QA this PR and check it off on the deploy checklist if it passes.

🚀 Deployed to production by https://github.com/francoisl in version: 1.3.87-12 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
failure ❌

🕸 web 🕸
success ✅


Add claims update for case insensitive queries when updating properties as a batch

Resolves : wso2/product-is#11578
Approach : Validation added to identify case insensitive queries.

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/762218700

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/762218700
Status: failure


Flip X axis

For 1H NMR the higher value is on the left and the smallest on the right.
You should invert the X axis

i filp the domain so the higher value will be at the left side and the smallest at the right,,
close 12


Fixes site name for MkDocs

@xerkus found this bug and reported it on Slack:
https://github.com/zendframework/zend-expressive-authentication/blob/486cb20dc81df864ee4706e8d191fc87ad0af91b/mkdocs.yml#L9
This results in a wrong header and installation command in the documentation:

Authentication middleware
$ composer require zendframework/Authentication middleware

https://docs.zendframework.com/zend-expressive-authentication/

@froschdesign thanks!


S.if() returned wrong result in IE 11

In the current sanctuary version this code returns wrong result in IE 11:
   S.is(Array, []) // It would be false
The problem is in how sanctuary determines the type. More precisely how it compares user's constructor (Array) and the value from R.type() in 386th line:
   R.type(x) === R.nth(1, R.match(/^function (\w*)/, String(type)))
The actual problem is that this code returns different results in different browsers:
   String(Array)
   // In chrome — "function Array() { [native code] }"
   // In IE 11  — "\nfunction Array() {\n    [native code]\n}\n"

ECMA 262 5.1 says that the result of String(Array) is not determined and the count of whitespaces depends on implementation:

<IP_ADDRESS> Function.prototype.toString ( )
An implementation-dependent representation of the function is returned. This representation has the syntax of a FunctionDeclaration. Note in particular that the use and placement of white space, line terminators, and semicolons within the representation String is implementation-dependent.

That is why it is very unsafe to obtain constructor's name in this way. In future we can use a new (ES2015) 'Array.name' property: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Function/name But for now I don't think there is a way to make this regular expression safe. We can accept the string "array" instead of an Array to elude this problem.

Thank you for submitting this patch, @kahless, and for writing a terrific description!
I know that @michaelficarra and others are working towards standardizing this behaviour, but it'll be some time before we can rely on that in our is implementation. ;)

:deciduous_tree: Thanks again!


I cant add kits

I cant add kits. In config.yml. When im start Server it show me "1 kit loaded" and this kit is no working :/
Sorry for my bad English

Pastern your config.yml and post the link here.

I dont have IT beacues i delete this plugin.


Roadmap for SimulAI

Short term development (fundamental steps) :

[ ] Coverage for more than 95 % of the source code.
[ ] Complete documentation for all the classes and methods.
[ ] Automating the execution of sanity tests and coverage.
[ ] Examples in notebooks for all the cases seen in the directory examples.

Medium term Development

[ ] Complete scalability tests for Data Distributed Parallelism (DDP) and Full-Sharded Data Parallelism (FSDP), using SimulAI models.
[ ] Pipelines execution optimized via JIT (TorchScript, maybe PyPy)

I will start on saturday 🦅🦅🦅


Several reconnecting with ManagedMqttClient

Describe your question
Hello. I've developed an open-source project for monitoring and interacting with Windows Desktop in general. This is a side project which I started to help a friend but it got heavily used in Home Assistant community as there is a lack of a proper Windows IOT tool.
I would like to request your help with my MQTT issues. I'm using your library to connect to MQTT server and publish my monitors values. However, several users (specially my friend which beta test with 10 computers on his company) have reported me disconnections (or reconnections) for timing out, while other MQTT clients in their networks are fine.
Here is how I'm using MQTTnet:

https://gitlab.com/iotlink/iotlink/-/blob/feature/mqtt-connection-rewrite/IOTLink/Service/MQTT/MQTTClient.cs
https://gitlab.com/iotlink/iotlink/-/blob/feature/mqtt-connection-rewrite/IOTLink/Service/MQTT/MQTTClientManager.cs

The MQTTClient.cs class should handle the MQTT connection itself using your client and send/subscribe/everything. The MQTTClientManager.cs is just a manager which should verify and recreate the entire MQTTClient if something goes south.
Here is an example of a log file:
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:20:37 -03:00][INFO][IOTLink.IOTLinkService]: Windows Service is started.
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:20:37 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::Connect() - Trying to connect to broker: tcp://<IP_ADDRESS>:1883.
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:20:37 -03:00][SYSTEM][IOTLinkService.Service.MQTT.MQTTClient]: ALL YOUR MQTT TOPICS WILL START WITH iotlink/workgroup/desktop-sala/
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:20:42 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:25:49 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 22:32:37 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:06:15 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:13:13 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:14:32 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:14:55 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:15:18 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected
[WORKGROUP\DESKTOP-SALA][2020-04-15 23:15:42 -03:00][INFO][IOTLinkService.Service.MQTT.MQTTClient]: MQTTClient::OnConnectedHandler() - MQTT Connected

And here it is possible to see the MQTT Server logs:

If you can take a quick look on these files or guide me how can I try to find how to fix this issue I would be grateful as the IOTLink users as well. I could not find any contact details of yours, so that is why I'm opening an issue. If you want to contact me directly, my Discord is alexslx#0984.
Just to be clear as a developer, I'm not asking for you fix the code for me or something else that might sound like this, just a little help using MQTTnet.
I'm sorry in advance if this issue has bothered you or your project anyway.
Which project is your question related to?

Client
ManagedClient

hi alex what is the purpose of your MQTTClientManager.cs? the managed client reconnects on its own.
I would remove it.
from your logs i cannot see whats wrong maybe plugin mqttnets trace
https://github.com/chkr1011/MQTTnet/wiki/Trace
CleanEvents can be:
OnMQTTConnected = null
and finally you dont await any of your calls. thats propably a big source of issue. you should await all methods that return a task. and that method should then also return a task. async void is a big nono

I assume this issue is closed.


Re-land "[cxx-interop] Check the presence of copy constructor correctly"

This reverts commit 3066bd6919b4b44524ca48a2241511353f44adb3.
This re-lands a change after it got reverted because of a regression in the build of SwiftCompilerSources.
rdar://136838485

@swift-ci please test

@swift-ci please build toolchain Ubuntu 22.04

@swift-ci please build toolchain Amazon Linux 2


ISPRK-3 Infinispan Query DSL from my Spark application

https://issues.jboss.org/browse/ISPRK-3
Added a couple of new operators to the InfinispanRDD:
val rdd = new InfinispanRDD[Integer, User](...)

//FilterByQuery
val query = Search.getQueryFactory(cache).from(classOf[User])
                           .having("name").equal("jerry").toBuilder[RemoteQuery].build

val filtered = rdd.filterByQuery(query, classOf[User]).sortByKey().first

// filterByCustom
val filtered = rdd.filterByCustom("my-deployed-filter-factory").filter(_.age > 10).collect()

Updated with support for the Java API and more tests covering deployed filters with parameters.

The rest looks good

@galderz A cache manager is created per node where the InfinispanRDD is executed, and closed in the listeners as you've seen it.
Wrt caching, it's cached when possible: there's a lazy val that stores the RemoteCache instance that preserves the underlying cache manager across calls in the same node. But since Spark actually serializes and sends the RDD to the remote workers, a new CM needs to be created to iterate the segments.
Btw, there was a typo in one of the listeners: it was calling remoteCache.stop instead of remoteCacheManager.stop, fixed and re-pushed

Ok, makes sense. Integrating...

Integrated


Resolve syncing conflicts from repo_sync_working_branch to main

Pull request opened by Docs to resolve syncing conflicts from repo_sync_working_branch to main.

Resolved the merge conflict in https://github.com/MicrosoftDocs/cloud-adoption-framework/pull/685; merging.


⚠ Bofrost.at has degraded performance

In 74a7918, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2005 ms

Resolved: Bofrost.at performance has improved in cafebca after 6 minutes.


Control flows hw

Homework merge

Changes included from Control Flows Youtube video homework

Changes included from Control Flows Youtube video homework


[SPARK-5654] Integrate SparkR

This pull requests integrates SparkR, an R frontend for Spark. The SparkR package contains both RDD and DataFrame APIs in R and is integrated with Spark's submission scripts to work on different cluster managers.
Some integration points that would be great to get feedback on:

Build procedure: SparkR requires R to be installed on the machine to be built. Right now we have a new Maven profile -PsparkR that can be used to enable SparkR builds

YARN cluster mode: The R package that is built needs to be present on the driver and all the worker nodes during execution. The R package location is currently set using SPARK_HOME, but this might not work on YARN cluster mode.

The SparkR package represents the work of many contributors and attached below is a list of people along with areas they worked on
edwardt (@edwart) - Documentation improvements
Felix Cheung (@felixcheung) - Documentation improvements
Hossein Falaki (@falaki)  - Documentation improvements
Chris Freeman (@cafreeman) - DataFrame API, Programming Guide
Todd Gao (@7c00) - R worker Internals
Ryan Hafen (@hafen) - SparkR Internals
Qian Huang (@hqzizania) - RDD API
Hao Lin (@hlin09) - RDD API, Closure cleaner
Evert Lammerts (@evertlammerts) - DataFrame API
Davies Liu (@davies) - DataFrame API, R worker internals, Merging with Spark
Yi Lu (@lythesia) - RDD API, Worker internals
Matt Massie (@massie) - Jenkins build
Harihar Nahak (@hnahak87) - SparkR examples
Oscar Olmedo (@oscaroboto) - Spark configuration
Antonio Piccolboni (@piccolbo) - SparkR examples, Namespace bug fixes
Dan Putler (@dputler) - Dataframe API, SparkR Install Guide
Ashutosh Raina (@ashutoshraina) - Build improvements
Josh Rosen (@joshrosen) - Travis CI build
Sun Rui (@sun-rui)- RDD API, JVM Backend, Shuffle improvements
Shivaram Venkataraman (@shivaram) - RDD API, JVM Backend, Worker Internals
Zongheng Yang (@concretevitamin) - RDD API, Pipelined RDDs, Examples and EC2 guide

@pwendell @rxin We might push some more fixes as they come in, but I think this should be ready for review

Test build #28899 has started for   PR 5096 at commit 3eacfc0.

This patch merges cleanly.

Test build #28899 has finished for   PR 5096 at commit 3eacfc0.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28899/
Test PASSed.

Regarding number 2. the YARN cluster mode. sparkRLibDir can be set to explicitly tell the workers the location of the R library path. This is how I have been telling the workers the location of the R library, which should contain the SparkR folder.

Test build #28910 has started for   PR 5096 at commit 463e28c.

This patch merges cleanly.

Additionally, one could think of setting sparkRLibDir as a way to remotely set .libPaths() on all of the workers. This is most useful when .libPaths() on the driver is different then on the workers. Note that it is implicitly assumed that all workers are configured identically, else setting sparkRLibDir will not work. If the function that is passed to parLapplyLB(cl,X,fun) calls other R packages the workers will look for these libraries sparkRLibDir, which by defalut is SPARK_HOME. There in lies the problem, SPARK_HOME is the spark home directory and not the path to the R packages on the workers.

Test build #28910 has finished for   PR 5096 at commit 463e28c.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28910/
Test PASSed.

@oscaroboto Good point about the rLibDir -- That does handle custom install locations on the executors. But in YARN cluster mode right now one needs it even on the driver.  BTW I don't think it should interfere with other packages because we only append to libPaths and don't overwrite it.

I wonder how does it deal with vcores management. One executor with 1 allocated core can execute one task, if this task is sparkR then even if it allocates 1 vcore, R process can consume as many CPUs as it pleases

@redbaron I have always wondered the same question myself

@redbaron @oscaroboto This quetion also applies to all the languages of Spark. One executor can have multiple allocated cores, each task also can consume one or more CPUs. A task consumes one cpu by default, it can be changed by spark.task.cpus.

hypothetical situation: Lets say we allocated 1 core per executor and the function within the SparkR lapply called the parallel package mclapply, but within mclapply we set mc.cores to 4. Would the distributed R code on the executors be forced to use just one core or would R have the ability of accessing 4 cores on the executors despite having allocated just 1 core? (assuming the chip has 4 cores)

@oscaroboto, @davies I am working on integrating sparkR into our workflow + integrating it with packrat.
To make things smooth, I had to drop custom .Rprofile which sets necessary options to limit CPU and RAM (drop Rscript override into /usr/local/bin which call real Rscript but ignoring any --vanilla option passed sparkR, otherwise your custom .Rprofile won't be loaded).
Unfortunately there is no standard way for R to limit CPUs as any library can use multithreading as it pleases. Currently I set mc.cores for mcapply  and OMP_NUM_THREADS for OpenBLAS to match executor cores.

@redbaron okay I see makes sense, thanks!

@redbaron, @oscaroboto The same applies to memory consumption I'm afraid. There isn't a way to constraint how much R (same for Python?) allocates (on Unix, 64-bit).
However, it looks like in Mesos and YARN this can be controlled via cgroups.

@felixcheung @redbaron @oscaroboto Does ulimit help? It can set memory usage limits for all procs started from a given shell.

@piccolbo ulimit is per process and it can't control RSS memory (ammount of actually used RAM) only vmem (ammount of allocated RAM) which can differ significantly.

Test build #29179 has started for   PR 5096 at commit e88b649.

This patch merges cleanly.

Test build #29182 has started for   PR 5096 at commit a1777eb.

This patch merges cleanly.

Test build #29183 has started for   PR 5096 at commit b045701.

This patch merges cleanly.

Test build #29183 has finished for   PR 5096 at commit b045701.

This patch fails RAT tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class UnregisterApplication(appId: String)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29183/
Test FAILed.

Test build #29186 has started for   PR 5096 at commit c300e08.

This patch merges cleanly.

Test build #29179 has finished for   PR 5096 at commit e88b649.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class UnregisterApplication(appId: String)

class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29179/
Test PASSed.

Test build #29182 has finished for   PR 5096 at commit a1777eb.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class UnregisterApplication(appId: String)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29182/
Test PASSed.

Test build #29186 has finished for   PR 5096 at commit c300e08.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29186/
Test PASSed.

@andrewor14 Could you help to review the spark-submit part?

What do you think about adding a basic test suite for SerDe?  This could be something pretty simple, such as just testing that serializing then deserializing objects give back equal objects.

We will do some refactor in SerDe, plan to add tests later. This PR is already huge, not convenient to review, I'd like to avoid large changes in this one.

What are the major areas where this needs review?  Is is pretty much just the spark-submit changes?  This patch doesn't significantly modify any existing Spark code, so it seems like it would be safe to merge soon in order to unblock collaboration on new features / tests.
It looks like R is already set up on AMPLab Jenkins, so it looks like we don't need do to anything else for testing infra.

Test build #29425 has started for   PR 5096 at commit 1d1802e.

Test build #29425 has finished for   PR 5096 at commit 1d1802e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29425/
Test PASSed.

Test build #29444 has started for   PR 5096 at commit 0e788c0.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29444/
Test FAILed.

I took a pass on this. I still haven't wrapped my head totally around the packaging, but here are a few comments:

I’m not quite sure what all the scripts are for. Can we document or remove some of these? It would be helpful to explain the general packaging mechanisms somewhere. For instance the install script seems important, but I don’t get what “install” means in this context (vs “package”).
We should file as a blocker for 1.4 integrating the docs into our doc build and our published API docs.
All of the examples use the Spark core API, but I imagine one of the biggest use cases would be using Data frames. What about an example that uses the data frame API?
Changes to Spark submit look good to me.

Spark submit changes look fine for the most part. The comments I left are mostly minor. One thing I noticed is that there are a lot of places where the return type is not specified. Even if a method has no return type, we should try to enforce adding Unit to the end so developers new to the project will do the same.
I intend to take another pass after the comments are addressed. I have not looked in detail at changes outside of Spark submit.

Test build #29746 has started for   PR 5096 at commit 5133f3a.

Test build #29746 has finished for   PR 5096 at commit 5133f3a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.1.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.5.0.jar
tachyon-client-0.5.0.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29746/
Test FAILed.

Test build #29758 has started for   PR 5096 at commit eb5da53.

Test build #29758 has finished for   PR 5096 at commit eb5da53.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.1.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.5.0.jar
tachyon-client-0.5.0.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29758/
Test FAILed.

Test build #29768 has started for   PR 5096 at commit 64eda24.

Thanks @pwendell @andrewor14 @JoshRosen for your comments. I am still investigating the return code problem for machines without R, but other than that most of the comments have been fixed.
Will be great if you could another pass !

Test build #29768 has finished for   PR 5096 at commit 64eda24.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29768/
Test FAILed.

@andrewor14 I think the test failure is because of a SparkSubmit unit test that is broken right now. If you see the test at https://github.com/apache/spark/blob/e40ea8742a8771ecd46b182f45b5fcd8bd6dd725/core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala#L97
we don't pass in a primaryResource or a mainClass and yet expect the argument parsing to succeed. The test will exit at https://github.com/apache/spark/blob/e40ea8742a8771ecd46b182f45b5fcd8bd6dd725/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L214 but we replace the exitFn and so it doesn't exit
Any good ideas on how to change the test case ? Should we just add a dummy primary resource, main class to it ?

@pwendell I tried to reproduce the build with R not installed on the machine with a clean Ubuntu VM, but I actually got a failure with a relevant error message
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec (sparkr-pkg) on project spark-core_2.10: Command execution failed. Process exited with an error: 127 (Exit value: 127) -> [Help 1]

And before the Maven summary I also see
../R/install-dev.sh: line 36: R: command not found

On a related note I also changed run-tests to not run SparkR tests if R is not installed.

Test build #29780 has started for   PR 5096 at commit f731b48.

Test build #29780 has finished for   PR 5096 at commit f731b48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29780/
Test FAILed.

Test build #29814 has started for   PR 5096 at commit 59266d1.

@andrewor14 We should had addressed all you comments, could you take another look?
Waiting for jenkins.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29814/
Test FAILed.

Test build #640 has started for   PR 5096 at commit 59266d1.

Okay LGTM from a packaging perspective. Once @andrewor14 sign's off on the spark-submit stuff I think this is ready to go.

Jenkins, retest this please

Test build #29828 has started for   PR 5096 at commit 59266d1.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29828/
Test FAILed.

@brennonyork @pwendell The new dependency checks seem to add around 20 minutes to a Jenkins run and this PR which has SQL and pom.xml changes has timed out thrice now (it didn't even get a chance to run the SparkR tests, so I don't think that is the problem). Is there anyway we can increase the time out or speed up the new dependency checks ?

We can certainly set the timeout to be something larger. Let me take a look at the previous builds and see if I can find a good timeout number and if there might be anything else we can do.
@pwendell any other ideas?

@brennonyork Thanks - another thing that might be helpful is to log how long it took to run the test. I am trying to figure out where the 120 minutes we have are being spent and its tricky to get a breakdown right now

Jenkins, retest this please (is the fourth time lucky ?)

Test build #29870 has started for   PR 5096 at commit 59266d1.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29870/
Test FAILed.

@shivaram a few things after looking at the build code some more...

The timeout value comes from the line here in dev/run-tests-jenkins. Its currently set at 120 minutes and doesn't include the time it takes for PR's to be tested against the master branch (i.e. for dependencies). We could certainly up that value, but I'd ask that since, I'm assuming, the dev/run-tests script on this PR runs all the new SparkR tests (plus any additional for core Spark you've added), that you run dev/run-tests locally and, for whatever additional time is needed, update the timeout in dev/run-tests-jenkins for this PR. The impetus for running locally first is that I'd much rather get a baseline for what it takes for all the new tests to run and then add 15ish minutes for fluff rather than throw a number into the wind.
Completely agree we should get some timing metrics for the various PR tests (thanks for the idea!). I'll generate a JIRA for that and take a look soon. That said, just to reiterate, those tests are not holding up the actual Spark test suite from finishing unless Jenkins has some deeper timing hooks than I know about. I assume though that it's merely a factor of the large corpus tests that were likely added in this PR.

SparkSubmit parts LGTM. We should merge this soon so people can start testing this well in advance of the release window.

@brennonyork The overall Jenkins build runner has a timeout of 130 minutes right now (cc @shaneknapp). So all the RAT tests, Mima checks, style checks, new dependencies plus all the unit tests have to run within 130 minutes and this PR seems to be failing that.
@shaneknapp can we increase the 130 min timeout to say 140 minutes ?

i'll up it to 180, just so we have some headroom.
On Wed, Apr 8, 2015 at 2:21 PM, Shivaram Venkataraman <
<EMAIL_ADDRESS>wrote:

@brennonyork https://github.com/brennonyork The overall Jenkins build
runner has a timeout of 130 minutes right now (cc @shaneknapp
https://github.com/shaneknapp). So all the RAT tests, Mima checks,
style checks, new dependencies plus all the unit tests have to run within
130 minutes and this PR seems to be failing that.
@shaneknapp https://github.com/shaneknapp can we increase the 130 min
timeout to say 140 minutes ?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/5096#issuecomment-91040259.

Thanks @shaneknapp ! Could you re-trigger this build once its upped ?

jenkins, test this please

also, i can't believe how long this build is...  sad panda etc.

Test build #29894 has started for   PR 5096 at commit 59266d1.

also, i can't believe how long this build is... sad panda etc.

Test parallelization is going to be a lot of work, but I think we could see huge speedups for the pull request builders if we didn't run all tests for every PR.  Most PRs touch the higher-level libraries and not core, so it should be safe to skip most of the tests if core hasn't been modified.

Test build #29894 has finished for   PR 5096 at commit 59266d1.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29894/
Test FAILed.

Test build #29908 has started for   PR 5096 at commit bac3a6b.

@shivaram - hey one thing I forgot to ask, how much time do the SparkR tests add to the overall Spark tests?

@pwendell Its around 2 minutes on my laptop. Here is the output on my machine
time ./run-tests.sh
....
....
./run-tests.sh  1:56.96 total

Test build #29908 has finished for   PR 5096 at commit bac3a6b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29908/
Test FAILed.

Test build #29919 has started for   PR 5096 at commit da64742.

Test build #29919 has finished for   PR 5096 at commit da64742.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29919/
Test PASSed.

Thanks @andrewor14 @pwendell for the reviews. Now that Jenkins is happy I am going merge this in and I'll file follow up issues for things like YARN cluster mode which we didn't get to in this PR.

:+1:


Provide support for buttons and D-pads mapped to half axes.

#38151 (and the 3.2 non-compat breaking version #38724) provided support for half axes and inverted axes, but they did not implement support for buttons or D-pads mapped to half axes. Worse, D-pad inputs are expected to be mapped to buttons; so break when they're not. This PR provides support for mapping buttons and D-pads to half axes.
Should fix #41065, but I don't have a controller that has this kind of mapping; so it needs testing.

Updated to include a fix for axes mapped to buttons and D-pads.

Updated to include fix for bug causing D-Pad buttons to stick.

Thanks!


[AutoDiff] Fix AST serialization of differentiation structs/enums.

Fix AST serialization of structs/enums synthesized during differentiation SIL transform.
Resolves TF-623. Asked about robust fix on Swift forums.

@swift-ci Please test tensorflow


Production build issue

Hello,
I'm attempting to build the /backend portion, and having difficulty removing the unused auth strategies.  I was able to get rid of the first couple errors I came across, but this one is stumping me, figured it doesn't hurt to ask :)
getting this error specifically when running npm run start:prod:
AssertionError [ERR_ASSERTION]: Unknown authentication strategy facebook in /auth/facebook
at new AssertionError (internal/assert.js:269:11)

fixed by commenting out unused auth endpoints for now


Rewrite build process

What the title says, gulp modules are maintained poorly and it is overly complicated to do it through the gulp flow while we only do some fairly basic stuff anyway.
Seems to me that a simple npm script is easier to maintain.

Fixed in c1334d34018048eb78eda30c3e82d88360aa58c2


[SPARK-47885][PYTHON][CONNECT] Make pyspark.resource compatible with pyspark-connect

What changes were proposed in this pull request?
This PR proposes to make pyspark.resource compatible with pyspark-connect.
Why are the changes needed?
In order for pyspark-connect to work without classic PySpark packages and dependencies.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Tested in https://github.com/apache/spark/pull/46090
Was this patch authored or co-authored using generative AI tooling?
No.

Merged to master.


Values show up in the left pane but not inside the tabs

Describe the bug
values exist and update in the left pane but when put into tabs no value is shown next to the name or in the graph.
To Reproduce
Steps to reproduce the behavior:
I am not sure how to reproduce it but closing and reopening the application seemed to solve it.
Expected behavior
I expected the values to show up as graphs correctly.
Screenshots

The value AbsoluteEncoderAngle is present and is updating on the left (under Hood) but isn't doing anything in the graphs.
Version (required):
Version: 4.0.0-beta-1
Distributor: FRC 6328
Platform: win32-x64
Build Date: 9/26/2024, 24:41:47 UTC
Electron: 32.0.2
Chromium: 128.0.6613.84
Node: 20.16.0

It looks like the line graph isn't rendering at all. If this occurs again, it would be helpful if you can check for errors in the developer console (go to "View" > "Toggle Developer Tools").
Without being able to recreate this, there's very little I can do to fix it.

My best guess is that this is a duplicate of #238, given that the screenshots show an image of the statistics tab. If the issue is not fixed on the next beta release of AdvantageScope, feel free to reopen.


Add proxy pass through test

Summary
Test is bad proxy info raises proxyerror in the API.  This means we correctly pass the proxies dict to the API.  Note that this test takes a while because the API does not check if a connection can be established before trying the max number of times.
Details and comments

The issue is really not the url that is being connected to, but rather the requests module does not care if it cannot establish a connection, it keeps trying up until the max number of connection attempts is made.

The issue is really not the url that is being connected to, but rather the requests module does not care if it cannot establish a connection, it keeps trying up until the max number of connection attempts is made. See issue #992

Hmm sounds odd - running the tests locally with <IP_ADDRESS> results in it passing in less than a second, which is actually sort of consistent with your explanation: it attempts to open a connection to the proxy a number of times (which instantly fail), and gives up after max_retries attempts. Do you have a proxy yourself on that port?
Anyway, seems part of another issue indeed - the PR itself is looking ready!


Better handling of invalid camera JSON

When a camera config file contains invalid JSON (for example due to a typo), camera2 will render a "positionable" camera panel at origin, except it's flat dark grey and can't actually be moved. This panel shows in every scene and is irremovable, even if you remove the broken config file and press ctrl+shift+f1. You need to restart the game to get rid of it.
This was a bit of a chore to diagnose, especially since the typo was very hard to see (; instead of :). It would be nicer if invalid cameras were ignored, possibly with an alert message. They should also reload properly.

If a cam fails to load for whatever reason it will already log the error to the Log file, however I have not thought of removing the in-broken-state camera. I'll change this behaviour.


Multiple download

HellO! How can I make multiple downloads at the same time? Thanks for your reply

It's in the Usage examples although this program doesn't work atm since it uses an outdated version of YoutubeExplode.

Sorry, wrong category


Rjf/expose graph ops

this PR adds a few simple commands to inspect graph attributes from CompassApp and CompassAppWrapper / python CompassApp. that includes:
    def graph_edge_origin(self, edge_id: int) -> int:
    def graph_edge_destination(self, edge_id: int) -> int:
    def graph_edge_distance(self, edge_id: int, distance_unit: Optional[str]) -> float:
    def graph_get_out_edge_ids(self, vertex_id: int) -> List[int]:
    def graph_get_in_edge_ids(self, vertex_id: int) -> List[int]:

each time one of these methods is called, we clone the Arc<DriverReadOnlyLock<Graph>>, which is a small performance hit which may be acceptable for some one-off operations but may slow things down for large batch queries. if that becomes a bottleneck we can add some batch operations that follow the pattern of these methods to amortize the clone runtime hit.
example
using the golden, co test case in the notebook example, i have this for my edges-compass.csv.gz file:
      edge_id  src_vertex_id  dst_vertex_id  distance
0           0              0            429   474.918
1           1              2              3    71.722
2           2              2            173    33.589
3           3              2            178   111.341
4           4              3              2    71.722
...       ...            ...            ...       ...
1755     1755            743            474    59.098
1756     1756            743            744    25.754     <---- this row in example
1757     1757            744            732    31.905
1758     1758            745            574    21.303
1759     1759            745            663   217.939

i run these queries and get this result:
(
    app.graph_edge_origin(1756),
    app.graph_edge_destination(1756),
    app.graph_edge_distance(1756, None),
    app.graph_edge_distance(1756, 'meters'),
    app.graph_edge_distance(1756, 'kilometers'),
    app.graph_edge_distance(1756, 'miles'),
    app.graph_get_out_edge_ids(743),
    app.graph_get_in_edge_ids(744)
)
(743,
 744,
 25.754,
 25.754,                  # meters is default
 0.025754000000000003,    # 25.75 / 1000
 0.0160062150410092,      # 25.75 / 1609
 [1755, 1756],            # includes test row id
 [1733, 1756])            # includes test row id

Closes #15.

i'll add an example of error handling for distance units here:
app.graph_edge_distance(1756, 'boog')

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
[/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/open_street_maps_example.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/open_street_maps_example.ipynb) Cell 10 line 1
----> [1](vscode-notebook-cell:/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/open_street_maps_example.ipynb#X40sZmlsZQ%3D%3D?line=0) app.graph_edge_distance(1756, 'boog')

File [~/dev/nrel/routee/routee-compass/python/nrel/routee/compass/compass_app.py:120](https://file+.vscode-resource.vscode-cdn.net/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/~/dev/nrel/routee/routee-compass/python/nrel/routee/compass/compass_app.py:120), in CompassApp.graph_edge_distance(self, edge_id, distance_unit)
    [119](https://file+.vscode-resource.vscode-cdn.net/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/~/dev/nrel/routee/routee-compass/python/nrel/routee/compass/compass_app.py:119) def graph_edge_distance(self, edge_id: int, distance_unit: Optional[str]) -> float:
--> [120](https://file+.vscode-resource.vscode-cdn.net/Users/rfitzger/dev/nrel/routee/routee-compass/docs/notebooks/~/dev/nrel/routee/routee-compass/python/nrel/routee/compass/compass_app.py:120)     return self._app.graph_edge_distance(edge_id, distance_unit)

Exception: could not deserialize distance unit 'boog'


Update location of breakpoint settings

Thanks!

As you can probably tell, I'm currently getting to grips with your library in prep of integrating it into a project I'm working on. Looks really good so far! :+1:

Yes, I guessed so! Great that it is useful to you. We are using it in several production projects ourself, but please feel free to let me know of any issues or feature requests.


Use dirty schedules

apply https://github.com/mururu/zstd-erlang/pull/9/files

actually let's coalese on {zstd, {git, "git@algithub.pd.alertlogic.net:alertlogic/zstd-erlang.git", {branch, master}}},

Sounds like a better plan. This fork is not even under my AL Public Github user :(

moved to https://github.com/alertlogic/zstd-erlang


Quick-start page link to deploy section (PR #69)

Small win can be made with internally linking the deploy text on the Quick-start page to the deploy page.
I will open a PR related to this issue.

Opened #69 for this

Closed with #69


feat: add framework name to the test classes BSP request

I've added [skip ci] because for now I only want to see what do you folks think about this PR.
This PR will stay as a draft until changes in bsp4j are merged, for now I've used locally published bsp4j version.
It is an implementation of the BSP proposal which is described here. This proposal adds an optional framework field to the returned class.
There's already implementation for bloop.

Resolution is failing because ch.epfl.scala:bsp4j:2.0.0+70-f6e47d42-SNAPSHOT is being resolved on the sonatype releases, but it's a snapshot. I'll wait until the official release is published.

@lefou I updated PR with the newly released bsp4j version.

Surprisingly, this is only for Scala projects and not for e.g. Java projects.

Do you mean that method names of method and classes contain ScalaTestClasses?

I mean, the framework name is only relevant for Scala projects, as it is implemented in ScalaBuildServer. So pure Java projects, for which we also sbt.testing, don't profit from this test framework selection.

@lefou It's true that it's more general than Scala and there was a small discussion about it here.

Thanks for the pointer. I think, this would be the right move, too.


Is selectAsync-like Functionality Available in ReArch for Asynchronous Property Updates?

In Riverpod, the selectAsync method allows for efficient listening to specific properties of asynchronous providers, enabling widgets to rebuild only when the selected property changes.
Is there an equivalent approach in ReArch to achieve similar functionality within a capsule?
Specifically, is it possible to perform a selection on an AsyncValue capsule to update only a particular property asynchronously?
Any guidance on implementing this pattern in ReArch would be greatly appreciated.

Hey! 👋
depending on your needs, you could use a combination of:

use(someCapsule.map((data) async => throw “todo”))
use.future(futureFromAbove);
You can also .map() the AsyncValue capsule itself to get the data out of it and avoid needing anymore async

normally, it is best to just make another capsule instead of using .map(), but it’s acceptable in a widget when you just want to get a filter accomplished quickly and/or it relies upon some ephemeral state (like a button toggle).

Specifically, is it possible to perform a selection on an AsyncValue capsule to update only a particular property asynchronously?

could you provide the specific capsules you have in mind and what you want out of them in your widget? That’d help me provide a clear example of what I’d recommend in your scenario

Hey again! 👋
Thanks for the clarification earlier. My question is a bit simpler, and it's about sync capsules instead of async ones. How can I make the widget rebuild only when a specific property changes in a sync capsule?
Here's a simplified example of my code:
class MyHomePage extends RearchConsumer {
  const MyHomePage({super.key});

  @override
  Widget build(BuildContext context, WidgetHandle use) {
    final gems = use(ExampleSync.gemsCapsuleState);
    // final (profile, setProfile) = use(ExampleSync.getProfileCapsuleState);
    print("MyHomePage rearch build");
    return Scaffold(
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.spaceEvenly,
          children: [
            Text("gems: $gems"),
            ElevatedButton(
              onPressed: () {
                print("Add Gems");
                final ccp = CapsuleContainerProvider.containerOf(context);
                final (profile, setProfile) = ccp.read(
                  ExampleSync.getProfileCapsuleState,
                );
                setProfile(
                  UserProfile(coins: profile.coins, gems: profile.gems + 1),
                );
              },
              child: const Text("Add gems"),
            ),
            ElevatedButton(
              onPressed: () {
                print("Add coins");
                final ccp = CapsuleContainerProvider.containerOf(context);
                final (profile, setProfile) = ccp.read(
                  ExampleSync.getProfileCapsuleState,
                );
                setProfile(
                  UserProfile(coins: profile.coins + 1, gems: profile.gems),
                );
              },
              child: const Text("Add Coins"),
            ),
          ],
        ),
      ),
    );
  }
}

class UserProfile {
  const UserProfile({this.gems = 0, this.coins = 0});
  final int coins;
  final int gems;
}

class ExampleSync {
  static (UserProfile, void Function(UserProfile)) getProfileCapsuleState(
    CapsuleHandle use,
  ) {
    return use.state(UserProfile());
  }

  static (UserProfile Function(), void Function(UserProfile))
  getProfileCapsuleData(CapsuleHandle use) {
    return use.data(UserProfile());
  }

  static int gemsCapsuleState(CapsuleHandle use) {
    final gems = use(getProfileCapsuleState.map((profile) => profile.$1.gems));
    return gems;
  }

  static int gemsCapsuleData(CapsuleHandle use) {
    final gems = use(getProfileCapsuleData.map((profile) => profile.$1().gems));
    return gems;
  }
}

class ExampleAsync {
  static (AsyncValue<UserProfile>, void Function())
  getProfileCapsuleRefreshable(CapsuleHandle use) {
    return use.refreshableFuture(() async {
      await Future.delayed(const Duration(seconds: 1));
      return UserProfile();
    });
  }

  static Future<UserProfile> getProfileCapsuleFuture(CapsuleHandle use) async {
    return UserProfile();
  }

  static AsyncValue<UserProfile> getProfileCapsuleAsyncValueFromFuture(
    CapsuleHandle use,
  ) {
    final profile = use(getProfileCapsuleFuture);
    return use.future(profile);
  }

  static AsyncValue<int> gemsCapsule(CapsuleHandle use) {
    final gems = use(
      getProfileCapsuleFuture.map((profile) async => (await profile).gems),
    );
    return use.future(gems);
  }
}

In the example above, I want the widget to rebuild only when gems changes. Currently, it seems like the widget rebuilds whenever UserProfile changes, regardless of which property is updated.
How can I filter the rebuild so that it only happens when the gems property changes? Any guidance would be much appreciated! 😊

static int gemsCapsuleState(CapsuleHandle use) {
  final gems = use(getProfileCapsuleState.map((profile) => profile.$1.gems));
  return gems;
}
static int gemsCapsuleData(CapsuleHandle use) {
  final gems = use(getProfileCapsuleData.map((profile) => profile.$1().gems));
  return gems;
}

This shouldn't impact rebuild frequency, but you should just be doing:
static int gemsCapsuleData(CapsuleHandle use) {
  return use(getProfileCapsuleData).value.gems;
}

ReArch has builtin build optimizations that stop downstream rebuilds when a capsule's data doesn't change (e.g. capsuleOldData == capsuleNewData). .map() should only be used from within a Widget, when you can't rely upon this optimization as easily/if at all.
Also, I'm guessing you created both the data/state capsules just to test what was going on, but I'd recommend just using the newer use.data side effect unless you explicitly want the API provided by use.state.

Currently, it seems like the widget rebuilds whenever UserProfile changes, regardless of which property is updated.

You often can minimize rebuilds by splitting up your state into smaller capsules (so perhaps 2 individual capsules for the gems and another for the coins, and perhaps another two for setters/incrementors). But, in your case, this was really an XY problem: you need to limit the scope at which you're introducing the capsules (often easiest through a RearchBuilder). When you use(someCapsule), the RearchConsumer that provides you with that use will rebuild whenever the capsules it's use-ing rebuild and produce new data. Thus, you need to introduce those capsules further down the Widget tree.
// This is written with the new, recently-stabilized capsule() syntax.
// Feel free to continue using the original syntax if you prefer it;
// it is 100% interoperable/no difference in functionality.

final Capsule<ValueWrapper<UserProfile>> userProfileManager =
  capsule((use) => use.lazyData(UserProfile.new));

final Capsule<int> userGemsCapsule = capsule((use) => use(userProfileManager).value.gems);

// https://rearch.gsconrad.com/paradigms/actions
final Capsule<void Function()> incrementUserGemsAction = capsule((use) {
  final userProfile = use(userProfileManager);
  return () {
    userProfile.value = UserProfile(
      coins: userProfile.value.coins,
      gems: userProfile.value.gems + 1,
    );
  };
});

class MyHomePage extends RearchConsumer {
  const MyHomePage({super.key});

  @override
  Widget build(BuildContext context, WidgetHandle use) {
    final gems = use(userGemsCapsule);
    // The following could also be acceptable, but I'd prefer the explicit capsule above:
    // final gems = use(userProfileManager.map((profile) => profile.value.gems));

    print("MyHomePage building...");
    return Scaffold(
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.spaceEvenly,
          children: [
            Text("gems: $gems"),
            RearchBuilder(
              builder: (context, use) {
                final incrementGems = use(incrementUserGemsAction);
                return ElevatedButton(
                  onPressed: incrementGems,
                  child: const Text("Add gems"),
                );
              },
            ),
          ],
        ),
      ),
    );
  }
}


Create Grafana-board that warns when deploy is failing in BFF

Description
As we saw when upgrading the docker job in this PR: https://github.com/Altinn/altinn-access-management-frontend/pull/541
that broke the deployment but we have no Github Actions that warns about this. This task is about researching if there are anything we can do about it (Create Grafana board, read from azure job or something else) and then implementing it.
After a talk with Jon Kjetil we should maybe spend some time grooming this task to find out what the scope of it should be.

Closing this


Use two spaces for indentation

Most of the code blocks in the readme already use 2 spaces for indentation, so this just updates a couple that use 4 spaces instead to be consistent.

Thanks, @dwightwatson!


Animixplay not working

many anime episodes are not opening in the animixplay . Please help me with this.

Please open extension issues in extension repository 🙏

To add/fix extensions, make a github issue here if it does not exist already
https://github.com/aniyomiorg/aniyomi-extensions


fix CanvasOverlay mocking

Merges the CanvasOverlay class with the mocked version of the class and resolves internal errors throwing in dependent projects jsdom testing environments.
Fixes #110

@robertleeplummerjr @trafficonese this is ready to get reviewed and merged


Enhance the platform tag definition for wheel files to cover platform variants

The current definition of the "platform" tag in PEP 425 is inadequate:

it doesn't account for Linux distribution variations
it doesn't clearly indicate the Windows MSVC runtime (this is technically implicit in the CPython runtime version, but could be clearer)
it doesn't account for variations in SSE support (None, SSE2, SSE3), which is critical for NumPy

Since distutils.util.get_platform() isn't providing sufficient platform discrimination, we need to come up with a new definition that provides the details we need, while also allowing pip to automatically fall back to the less specific wheels if the exact variant isn't available.

migrated from https://bitbucket.org/pypa/pypi-metadata-formats/issues/15/enhance-the-platform-tag-definition-for

What's the state of this?

I'm also interested in the state of this issue, especially the first of the three points (accounting for Linux distribution variations). I believe it's crucial to add basic Linux support, leaving the more specific discrimination (SSE support and whatnot) for the next version.
One reason to add basic Linux support is this: when building a docker container, pip install gevent (or similar) will take a lot of time, everytime, because every pip install actually behaves like pip install --no-cache-dir. This can easily become painful: if at the end of a docker build there's an error and you have to modify the Dockerfile, the next build will again need to recompile everything (as opposed to just re-download everything).

The "manylinux1" concept @njsmith recently posted about on distutils-sig seems like the most promising prospect for near term progress: https://mail.python.org/pipermail/distutils-sig/2016-January/027980.html

I know @ncoghlan was working on this with regard to identification of Linux distributions by means of lsb_realease and such. Nick, could you please link here relevant discussions that were taking place?

"working on" is a generous description - "kicking ideas around" was as far I got, although Nate Coraor took things further for use in https://galaxyproject.org/
The original incarnation of the idea was this proposal to use /etc/os-release: https://mail.python.org/pipermail/distutils-sig/2014-November/025297.html
Nate then set about turning that idea into reality: https://mail.python.org/pipermail/distutils-sig/2015-July/026616.html

FWIW there is also the manylinux proposal: https://code.activestate.com/lists/python-distutils-sig/27556/ and https://www.python.org/dev/peps/pep-0513/

Yes, we see the two proposals as complementary - manylinux1 for "one size fits most" wheels, and the more specific ones for when you want to build wheel files that are optimised for a specific distribution, but don't want them accidentally confusing matters for anyone using a different distribution.

SSE/AVX availability is important for other packages as well in case they wrap C code that uses it.
I wondered what could be done about it.
We could just parse /etc/cpuinfo where it's available.


Set isolatedDeclarations to true and skipLibCheck to false.

Requires Typescript 5.6, once it has been released.

one blocking issue preventing us from setting skipLibCheck to false is due to an incorrect typing entry in antlr4's package.json. the good news is that this issue has been fixed in antlr4 v4.13.2.
it looks promising to be able to disable skipLibCheck again with this issue out of the way.


test: test_recurring_job_kubernetes_status failed

_____________________ test_recurring_job_kubernetes_status _____________________

client = <longhorn.Client object at 0x7fefb60994d0>
core_api = <kubernetes.client.apis.core_v1_api.CoreV1Api object at 0x7fefb636bd50>
volume_name = 'longhorn-testvol-os1a5e'

    @pytest.mark.csi
    @pytest.mark.recurring_job
    def test_recurring_job_kubernetes_status(client, core_api, volume_name):  # NOQA
        """
        Test that a RecurringJob properly backs up the KubernetesStatus of a
        Volume.
        """
        host_id = get_self_host_id()
        client.create_volume(name=volume_name, size=SIZE,
                             numberOfReplicas=2)
        volume = common.wait_for_volume_detached(client, volume_name)

        pv_name = "pv-" + volume_name
        create_pv_for_volume(client, core_api, volume, pv_name)
        ks = {
            'pvName': pv_name,
            'pvStatus': 'Available',
            'namespace': '',
            'pvcName': '',
            'lastPVCRefAt': '',
            'lastPodRefAt': '',
        }
        wait_volume_kubernetes_status(client, volume_name, ks)

        # Simple Backup Job that runs every 2 minutes, retains 1.
        jobs = [
            {
                "name": RECURRING_JOB_NAME,
                "cron": "*/2 * * * *",
                "task": "backup",
                "retain": 1
            }
        ]
        volume.recurringUpdate(jobs=jobs)
        volume.attach(hostId=host_id)
        volume = common.wait_for_volume_healthy(client, volume_name)

        # 5 minutes
        time.sleep(300)
        snapshots = volume.snapshotList()
        count = 0
        for snapshot in snapshots:
            if snapshot["removed"] is False:
                count += 1
        # 1 from Backup, 1 from Volume Head.
        assert count == 2

        # Verify the Labels on the actual Backup.
        bv = client.by_id_backupVolume(volume_name)
        backups = bv.backupList()
>       assert len(backups) == 1
E       assert 2 == 1
E        +  where 2 = len({'resourceType': u'backup', 'data': [{'snapshotCreated': u'2019-08-27T02:48:02Z', 'name': u'backup-0f4633e6e94041e6', ...Size': u'16777216', 'volumeName': u'longhorn-testvol-os1a5e', 'size': u'0', 'volumeCreated': u'2019-08-27T02:48:02Z'}]})

test_recurring_job.py:247: AssertionError

The actual backup list is in bv.backupList().data rather than  bv.backupList().
Maybe Python syntactic sugar prevents this issue from being exposed in other similar use cases.

One more issue that fails this test:
Since DeleteBackup is aync and we will directly talk with backupstore in the recurring job pod, the pod immediately becoming complete after DeleteBackup call will fail the aync deletion process.


Make "name" property of host object optional

I find that the command itself is all I need to identify it, so all my "hosts" entries have the same "name" and "cmd". I think it would be a good idea to make the "name" default to the "cmd" if "name" is not set.
P.S. I am finding this useful for conveniently running other frequently used commands, besides ssh.

This is something that I'll consider.
Thanks! I'm happy you are finding it useful. :smile:


Please include compiled assets for dev-master

Please include compiled JS assets (dist dir) for current version of dev-master or improve documentation for manual compilation of assets.
Thanks.


Handle GLobalKnownHostsFile like other entries with multiple values

Coverage remained the same at 98.282% when pulling 1f0537957b5be04fde90d39f0bcaf4fff3604d6a on TheKPLD:master into df2b6627213491429276d7a739597fec0447b760 on hercules-team:master.

LGTM. Could you add a test for this entry please?

Coverage remained the same at 98.282% when pulling 78f38710d28a25f9dca252ab2df50287331c1d6e on TheKPLD:master into df2b6627213491429276d7a739597fec0447b760 on hercules-team:master.

Coverage remained the same at 98.282% when pulling 78f38710d28a25f9dca252ab2df50287331c1d6e on TheKPLD:master into df2b6627213491429276d7a739597fec0447b760 on hercules-team:master.


Wagtail 2.16 Invalid HTTP_HOST header: 'localhost:8000:8000'

Automatic redirect creation is a new feature in Wagtail 2.16.
But when the program runs with WAGTAILREDIRECTS_AUTO_CREATE = True(by default), an error "Invalid HTTP_HOST header: 'localhost:8000:8000'. The domain name provided is not valid according to RFC 1034/1035." will raise for changing page slug as well as moving page to another location.
I found that this problem can be handled by setting WAGTAILREDIRECTS_AUTO_CREATE = False.
Any other solution for this case?

Hi @aliceni81 - can you give the full stack trace of the error, please?

Managed to replicate this locally.

Traceback
Environment:

Request Method: POST
Request URL: http://wagtail.test:8000/admin/pages/83/move/76/confirm/

Django Version: 4.0.2
Python Version: 3.8.12
Installed Applications:
['bakerydemo.base',
 'bakerydemo.blog',
 'bakerydemo.breads',
 'bakerydemo.locations',
 'bakerydemo.search',
 'wagtail.contrib.search_promotions',
 'wagtail.contrib.forms',
 'wagtail.contrib.redirects',
 'wagtail.contrib.settings',
 'wagtail.embeds',
 'wagtail.sites',
 'wagtail.users',
 'wagtail.snippets',
 'wagtail.documents',
 'wagtail.images',
 'wagtail.search',
 'wagtail.admin',
 'wagtail.api.v2',
 'wagtail.contrib.modeladmin',
 'wagtail.contrib.routable_page',
 'wagtail.core',
 'rest_framework',
 'modelcluster',
 'taggit',
 'wagtailfontawesome',
 'corsheaders',
 'bakerydemo.custommedia',
 'wagtailmedia',
 'wagtailmarkdown',
 'django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.sitemaps',
 'wagtail.contrib.styleguide']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'wagtail.contrib.redirects.middleware.RedirectMiddleware']

Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/vagrant/wagtail/wagtail/admin/urls/__init__.py", line 125, in wrapper
    return view_func(request, *args, **kwargs)
  File "/vagrant/wagtail/wagtail/admin/auth.py", line 174, in decorated_view
    response = view_func(request, *args, **kwargs)
  File "/vagrant/wagtail/wagtail/admin/views/pages/move.py", line 72, in move_confirm
    action.execute()
  File "/vagrant/wagtail/wagtail/core/actions/move_page.py", line 109, in execute
    return self._move_page(self.page, self.target, pos=self.pos)
  File "/vagrant/wagtail/wagtail/core/actions/move_page.py", line 79, in _move_page
    post_page_move.send(
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 170, in send
    return [
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/dispatch/dispatcher.py", line 171, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/vagrant/wagtail/wagtail/contrib/redirects/signal_handlers.py", line 85, in autocreate_redirects_on_page_move
    create_redirects(page=instance, page_old=page_old, sites=sites)
  File "/vagrant/wagtail/wagtail/contrib/redirects/signal_handlers.py", line 127, in create_redirects
    old_urls = _page_urls_for_sites(page_old, sites, cache_target=page)
  File "/vagrant/wagtail/wagtail/contrib/redirects/signal_handlers.py", line 98, in _page_urls_for_sites
    site_id, root_url, page_path = page.get_url_parts(request)
  File "/vagrant/wagtail/wagtail/core/models/__init__.py", line 1125, in get_url_parts
    site = Site.find_for_request(request)
  File "/vagrant/wagtail/wagtail/core/models/sites.py", line 97, in find_for_request
    site = Site._find_for_request(request)
  File "/vagrant/wagtail/wagtail/core/models/sites.py", line 103, in _find_for_request
    hostname = split_domain_port(request.get_host())[0]
  File "/home/vagrant/.virtualenvs/bakerydemo/lib/python3.8/site-packages/django/http/request.py", line 135, in get_host
    raise DisallowedHost(msg)

Exception Type: DisallowedHost at /admin/pages/83/move/76/confirm/
Exception Value: Invalid HTTP_HOST header: 'wagtail.test:8000:8000'. The domain name provided is not valid according to RFC 1034/1035.


inst_uncertain.* should be named to match existing convention

inst_uncertain.* should be named to match existing convention (e.g., uncertain-upe.*). After #132 has been merged. @harrysarson, are you able to take care of this?

I should be able to take a look at this and fix it soon 🙂
For background, during my work on uncertainty I started by implementing a library and then moved to a hardware level design. The naming style of the functions reflect the fact that they simulate uncertainty propagation at the instruction level rather than at software level.
At some point there were lib_uncertain. functions which gave the naming convention more sense.


Programmatically Select Row

Is there a way to programmatically scroll to a row or index, after the picker has loaded?
I need to scroll to specific items, after the view has loaded, based on what the user selects in that View Controller
I've tried objectPicker.currentSelectedRow = 0
Any help is appreciated.

Hi @DhruveShah
In order to do that you should call the selectRow(_ row : Int, animated: Bool) method.

@filipealva Thanks for the help.
That works, however, it is changing my infinite PickerView to the default behaviour, and trying to change the style after calling selectRow does not seem to make a difference.

@DhruveShah it should not behave like that. I've used this method in a lot of projects and any of them had this issue (I just tried to reproduce it on some of them right now).
Can you provide a sample project that reproduces this issue?
tks

One thing I just noticed, if you scroll down it appears as infinite list and is fine, however if you scroll up, it doesn't appear to be an infinite list and stops at the first item.
So for example, calling selectRow(0, true), scrolls to the top, with no items appearing before that, but scrolling down however, makes it appear as an infinite list.
I'm not sure if this is due to the list size being rather small (15 items)
I will be sure to upload a sample project or code first thing in the morning.

This is essentially what my code is doing:
@IBOutlet weak var locationSelector: PickerView!
The following is inside my class:
override func viewDidLoad() {
        super.viewDidLoad()
        configurePicker()
}

override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(true)
        locationAuthStatus()
}

fileprivate func configurePicker() {
        locationSelector.dataSource = self
        locationSelector.delegate = self

        let scrollingStyle: PickerView.ScrollingStyle
        let selectionStyle: PickerView.SelectionStyle

        scrollingStyle = PickerView.ScrollingStyle.infinite
        selectionStyle = PickerView.SelectionStyle.defaultIndicator

        locationSelector.scrollingStyle = scrollingStyle
        locationSelector.selectionStyle = selectionStyle
}

func locationAuthStatus() {
        if CLLocationManager.authorizationStatus() == .authorizedWhenInUse {
            currentLocation = locationManager.location

            let exampleLocation = CLLocation(latitude: 48.770344, longitude: 9.171880)

            geocoder.reverseGeocodeLocation(exampleLocation) { (placemarks, error) in
                // Process Response
                self.processResponse(withPlacemarks: placemarks, error: error)
            }
        } else {
            locationManager.requestWhenInUseAuthorization()
            locationAuthStatus()
        }
}

private func processResponse(withPlacemarks placemarks: [CLPlacemark]?, error: Error?) {
        if let error = error {
            print("Unable to Reverse Geocode Location (\(error))")
        } else {
            if let placemarks = placemarks, let placemark = placemarks.first {
                stateSelected = placemark.administrativeArea!
                userCountry = placemark.country
                print(placemark.administrativeArea!)
                locationSelector.selectRow(0, animated: true)
            } else {
                print("No Matching Addresses Found")
            }
        }
}

And then I have the delegate and data source set up after:
extension SetLocationVC: PickerViewDataSource {
    func pickerViewNumberOfRows(_ pickerView: PickerView) -> Int {
        return DataView.states.count
    }

    func pickerView(_ pickerView: PickerView, titleForRow row: Int, index: Int) -> String {
        return DataView.states[index].stateName
    }
}

extension SetLocationVC: PickerViewDelegate {
    func pickerViewHeightForRows(_ pickerView: PickerView) -> CGFloat {
        return 50.0
    }

    func pickerView(_ pickerView: PickerView, didSelectRow row: Int, index: Int) {
        print(index)
        print(DataView.states[index].stateName)
    }

    func pickerView(_ pickerView: PickerView, styleForLabel label: UILabel, highlighted: Bool) {
        DataView.pickerStyle(label: label, highlighted: highlighted)
    }
}

What I'm doing is requesting the users location, and then wanting to select that index

@DhruveShah hey, sorry about the delay.
I could found the issue it was a bug on PickerView.
Just fixed it. I know it has been a long time, but if you could test it would be awesome. Just points to this branch https://github.com/filipealva/PickerView/tree/fix/select-row-issues in your Podfile.


Fix /x/net and /x/sys

Pull Request Motivation
Two commits, each self contained:

Adds go.mod and go.sum as sources so that we'll rebuild binaries locally if they change
Fixes CVEs in /x/net and /x/sys which we probably weren't actually vulnerable to but it's worth keeping on top of these things

Kind
/kind bug
Release Note
NONE

@wallrj I'd missed updating the LICENSES file, which I've now done. Could I get another LGTM?

/lgtm


Task ID

In our instance we load tasks from an external application (ticket/release/bug). Those have also a number assigned and that is what we refer to. Is there a way to either:

Replace the Task ID shown on the kanban card with the ID (number) that was assigned in the source application? The Task ID as currently is, isn't very useful to use and is taking up unnecessary white space on the card.
Add a number to the kanban card that is clickable and would open the ticket/release/bug in the source application when clicked on it?
This way we would be more easily able to link the kanboard cards with the actual ticket/release/bug in the source application.

There is a column "reference" in the table tasks, you can use that to store external ticket id.
It's actually used by the Github/Gitlab webhooks to synchronize issues in Kanboard. The url to the ticket is appended to the task description. The external id is shown on the card if not empty next to the Kanboard task id.
How do you load tasks into Kanboard from your external application?

Thanks for the quick and prompt response.
I use python and jsonrpc api. I love your application and we have big plans
to start using it in our IT teams.
OK, so I should be able to use the reference? Will I be able to set/provide
the reference (URL) to external application? If that works, I am all set.
On Sun, Jun 7, 2015 at 11:42 AM, Frédéric Guillot<EMAIL_ADDRESS>wrote:

There is a column "reference" in the table tasks, you can use that to
store external ticket id.
It's actually used by the Github/Gitlab webhooks to synchronize issues in
Kanboard. The url to the ticket is appended to the task description. The
external id is shown on the card if not empty next to the Kanboard task id.
How do you load tasks into Kanboard from your external application?
—
Reply to this email directly or view it on GitHub
https://github.com/fguillot/kanboard/issues/912#issuecomment-109787749.

I am having trouble with creating a reference between a task on the
Kanboard and a unique record (ticket/release/bug) in an external
application.
Creating tasks in Kanboard using createTask works awesome. But how do I
update if I don't know the relationship between task_id and the record in
the external application.
I guess, I could use title but that isn't 100% accurate. Titles can change,
are not unique, etc.
Now if I could generate my own unique task_id, based on an id in an
external application, I would be all set.
However the task_id appears like I have no control over.
On Sun, Jun 7, 2015 at 11:50 AM, Martien Truijen<EMAIL_ADDRESS>wrote:

Thanks for the quick and prompt response.
I use python and jsonrpc api. I love your application and we have big
plans to start using it in our IT teams.
OK, so I should be able to use the reference? Will I be able to
set/provide the reference (URL) to external application? If that works, I
am all set.
On Sun, Jun 7, 2015 at 11:42 AM, Frédéric Guillot <
<EMAIL_ADDRESS>wrote:

There is a column "reference" in the table tasks, you can use that to
store external ticket id.
It's actually used by the Github/Gitlab webhooks to synchronize issues in
Kanboard. The url to the ticket is appended to the task description. The
external id is shown on the card if not empty next to the Kanboard task id.
How do you load tasks into Kanboard from your external application?
—
Reply to this email directly or view it on GitHub
https://github.com/fguillot/kanboard/issues/912#issuecomment-109787749.

I modified the API to facilitate that (only in master branch). Use the field "reference" to store your external id (ticket number or whatever, 50 chars max, must be unique for each project).

Create a task with an external reference, here an example of payload:

{
    "jsonrpc": "2.0",
    "method": "createTask",
    "id":<PHONE_NUMBER>,
    "params": {
        "title": "Task with external ticket number",
        "reference": "TICKET-1234",
        "project_id": 1,
        "description": "[Link to my ticket](http:\/\/my-ticketing-system\/1234)"
    }
}

The url to your external ticket is defined in the description (in markdown format). You can do the same with the method updateTask.

If you would like to fetch this task later with the api, use the method getTaskByReference. See https://github.com/fguillot/kanboard/blob/master/docs/api-json-rpc.markdown#gettaskbyreference

With Kanboard the task_id is an autoincrement field, you cannot modify this field.


Allow for IP address configuration on prometheus instrumentation

The turbinia configuration should have an option to configure the IP address the prometheus instrumentation endpoint listens on. By default it will listen on <IP_ADDRESS>.

Setting the addr parameter in the start_http_server() function should fix this.
https://github.com/prometheus/client_python/blob/master/prometheus_client/exposition.py#L76

Fixed in #625