MGMT-17949: Proxy params for installation iso

MGMT-17949: Proxy params for installation iso
Setting proxy setting as part of installation service in installation iso

Background / Context
As part of installation flow on installation iso we need to pull seed image and precache seed images.  We need to add proxy settings to installation service in order for it to work

Issue / Requirement / Reason for change

Solution / Feature Overview
Set proxy params as part of installation service in installation iso

/approve


Ill-typed upgrades cause data-loss

Doing an ill-typed upgrade can cause stable data loss.
The problem is the recent change to candid to support defaulting of optional fields.
Motoko encodes stable variables in a record of options.
Doing an upgrade to an incompatible type will cause those optional fields to default to null, discarding any data those fields might have held (and just running the initializer instead).
A conformance check prior to upgrade could warn about this potential data loss, but absent tooling for that, we are in the situation where users can easily lose their data on upgrade by doing a simple, and natural, mistake like adding a field to stable variable of record type (or, more commonly, map thereof).
#2691 contains a repro of a sequence of ugrades that should fail but don't.
https://forum.dfinity.org/t/questions-about-data-structures-and-migrations/822/15 is a real world example of a user being tripped up by this, trying to "upgrade" a stable array of tuples containing records by adding a field.
Instead of failing the upgrade, the stable variable is regarded as uninitialized and initialized from this initializer expression (losing the data).
Solution: I briefly considered just changing the record of options encoding to a record of variants encoding for stable variables, which would be fine, but isn't  backwards compatible with previously compiled code, so a non-starter.
I'll investigate whether we can just change the deserialization behaviour for "T.Memory" records to avoid the option defaulting instead.
@rossberg, @nomeata whaddya think?
Of course, we could say we really need the static check, but I think a defensive implementation is called for anyway.

Hmm, for consistency, I wonder if we should just live with this but make the future static checker warn on lossy re-typing of stable vars.

Ouch, yeah. It's really overdue that we implement this upgrade check!

Yeah, but it's not entirely clear to me what to check.

Do we output the candidish type and check and extended candidish subtyping relation? Then we need to implement an extended candid just for that.

Or do we use a slightly extended relation on the Motoko (stable) types? If so, we need to serialize the Motoko types (that might involve mu types) to a format we can check. Probably useful for separate compilation later, but definitely not supported right now.

Another alternative, avoiding serialization of types, is just to type check both programs (before and after) in memory and check the relationship between the in memory types (but that seems gross and won't really accommodate language changes).

I think 1) is probably the way to go, since we have most of the machinery already and just need to extend candid a bit.

Another question is where to store this info. In a separate file, in a custom section or in the wasm binary as an additonal query method  as Joachim hacked up for the (external) Candid interface. The latter actually has some appeal.

Hi Claudio, has there been any progress on this issue? I recently lost some state due to this, would be good to prevent this from happening to others!

Still no progress ? What do you suggest to prevent data loss while we wait for a clean solution ?
I'm coding a bash script to dump the data (only for the canister owner) and restore the data. The thing is that i'm getting candid data and it's a bit hard to parse it and modify if there is a new field in a type.

PR #2887 (not yet merged) adds some support for manually checking compatibility of stable signatures using moc.
Roughly,
moc --stable-types foo.mo will write the stable signature of an actor(class) to file foo.most.
moc --stable-compatible old.most new.most will check that the stable interface can evolve form old.most to new.most in a   type safe way without unintentional data loss.
You can use tool didc to check that the dids are compatible
didc --check new.did old.did
The separate didc tool can be already be used to manually check compatibility of the candid interfaces.
What remains is integrating this with the replica and dfx to prevent an unsafe ugrade, but that's still underway and may take a while.
I'm also working on some documentation to explain what is allowed and what isn't. The basic rules are that you can evolve * a stable variable to a super-type with the same mutability (so that the upgrade can consume the old value)

add a new stable variable with different name and any type.
The Candid interfaces can evolve to a Candid subtype (note the inversion of the relation), so that old clients can use the new interface without breaking.

In practice, with the current implementation, one can safely change the mutability of a stable variable in an upgrade and drop an existing stable variable, but we have decided to error on those cases for now (but might only warn in future).

https://github.com/dfinity/motoko/pull/2887 is merged. Shall we close this?

The replica change is still not in production, but we can close this, as it's a motoko issue.


⚡ Implement Prometheus Operator Linter

User Story
As a developer I want to implement a Prometheus Operator Linter in this repo : https://github.com/ministryofjustice/analytics-platform-helm-charts
Hypothesis
This Linter should be ran on pull request on new PrometheusRule yaml if their is a error the binary will return with a error detailing the incorrect syntax.
Definition of Done
Example - [ ] Documentation has been written / updated

[ ] README has been updated
[ ] User docs have been updated
[ ] Another team member has reviewed
[ ] Tests are green

This ticket is to be closed due to the barrier in implementation due to the helm templating. Follow on stories will be raised in the backlog for a future sprint around alternative approached such as prom unit testing https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/ or potentially recreating the code in our own environments to suit our needs


Feat: Connected adding breakpoints with server and toast feature to inform user.

Connected adding breakpoints with server and toast feature to inform user

Description

Connected Breakpoint with server.
Added Toast functionality for informing the user.

Closes

#25
#24

@Ammoniya please review this pr.


Error: Failed to fetch plugin

I run this command in ionic cli
sudo ionic cordova plugin add cordova-plugin-telerik-imagepicker
and get this output
> cordova plugin add cordova-plugin-telerik-imagepicker --save
✖ Running command - failed!

[ERROR] Cordova encountered an error.
        You may get more insight by running the Cordova command above directly.

[ERROR] An error occurred while running cordova plugin add
        cordova-plugin-telerik-imagepicker --save (exit code 1):

        Error: Failed to fetch plugin
        git+https://github.com/Telerik-Verified-Plugins/ImagePicker.git via
        registry.
        Probably this is either a connection problem, or plugin spec is
        incorrect.
        Check your connection and plugin name/version/URL.
        Failed to get absolute path to installed module

my config is :
ionic -v3.5.0
cordova -v7.0.1
npm -v4.2.0
node -v7.10.1
android -v 6.2.1

please help me @EddyVerbruggen

me too, what's the problem?

Dunno, I just did this:
cordova create imagepicker
cd imagepicker
cordova platform add ios android
cordova plugin add cordova-plugin-telerik-imagepicker

Output of that last command:
Installing "com.synconset.imagepicker" for android
Subproject Path: CordovaLib
Installing "com.synconset.imagepicker" for ios
Adding com.synconset.imagepicker to package.json
Saved plugin info for "com.synconset.imagepicker" to config.xml

Downgrading to Cordova 6.x will make the install work.

Having the same issue here, it installed fine yesterday, I uninstalled it, and now when I try and reinstall today it's not working.

@alexstanbury Same here. Installed it earlier today with no problem. Now this error pops...

@matthieu80 my problem was the plugin did not uninstall properly previously. Manually removing the plugin folder from node_modules did the trick for me.
Incidentally, I have found a much better plugin that allows selecting videos as well, it is a lot more stable, has a nicer UI and more features - I would definitely recommend switching to it. I have only tested it on Android as I am using the html5 file input for iOS.

Did you resolve the trouble. I get the same error too when trying to install this plugin:
(node:39362) UnhandledPromiseRejectionWarning: CordovaError: Failed to fetch plugin https://github.com/Telerik-Verified-Plugins/ImagePicker.git via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Failed to get absolute path to installed module
    at /usr/local/lib/node_modules/cordova/node_modules/cordova-lib/src/plugman/fetch.js:173:37
    at _rejected (/usr/local/lib/node_modules/cordova/node_modules/q/q.js:864:24)
    at /usr/local/lib/node_modules/cordova/node_modules/q/q.js:890:30
    at Promise.when (/usr/local/lib/node_modules/cordova/node_modules/q/q.js:1142:31)
    at Promise.promise.promiseDispatch (/usr/local/lib/node_modules/cordova/node_modules/q/q.js:808:41)
    at /usr/local/lib/node_modules/cordova/node_modules/q/q.js:624:44
    at runSingle (/usr/local/lib/node_modules/cordova/node_modules/q/q.js:137:13)
    at flush (/usr/local/lib/node_modules/cordova/node_modules/q/q.js:125:13)
    at process._tickCallback (internal/process/next_tick.js:150:11)
(node:39362) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:39362) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

My ionic info:
cli packages: (/usr/local/lib/node_modules)

    @ionic/cli-utils  : 1.19.2
    ionic (Ionic CLI) : 3.20.0

global packages:

    cordova (Cordova CLI) : 8.0.0

local packages:

    Cordova Platforms : android 6.3.0 ios 4.5.4
    Ionic Framework   : ionic1 1.3.3

System:

    ios-deploy : 1.9.2
    ios-sim    : 5.0.13
    Node       : v9.3.0
    npm        : 5.7.1
    OS         : macOS High Sierra
    Xcode      : Xcode 9.2 Build version 9C40b

Environment Variables:

    ANDROID_HOME : not set

Misc:

    backend : pro

How can I resolve this? How can I install this plugin?


Which datasets are used for main paper (98 datasets) and "small data" (57 datasets)

Hi.
I'm trying to compare to some of the results in your work, but it's not clear to me which datasets were use for Table 1 and Table 2.
The Datasets A file contains 108 datasets, and the Datasets B file contains 69 datasets, so I'm not sure which the 98 ones are.
Really I care more about the 57 small datasets, but cutting off at those with 1250 or less instances doesn't yield 57 for either A or B or the combination.

The "easy_import" list seems to contain 175 classification tasks, 69 of which have less than 1250 instances.

Hi Andreas, below are the 98 datasets from Table 1 and the 57 datasets from Table 2. Please let us know if you have more questions.
datasets_table_1 = ['openml__visualizing_environmental__3602', 'openml__labor__4', 'openml__monks-problems-2__146065', 'openml__tic-tac-toe__49', 'openml__dermatology__35', 'openml__cardiotocography__9979', 'openml__lung-cancer__146024', 'openml__sonar__39', 'openml__anneal__2867', 'openml__analcatdata_chlamydia__3739', 'openml__iris__59', 'openml__irish__3543', 'openml__heart-c__48', 'openml__ionosphere__145984', 'openml__hayes-roth__146063', 'openml__fri_c3_100_5__3779', 'openml__fri_c0_100_5__3620', 'openml__analcatdata_authorship__3549', 'openml__rabe_266__3647', 'openml__balance-scale__11', 'openml__acute-inflammations__10089', 'openml__MiceProtein__146800', 'openml__banknote-authentication__10093', 'openml__mushroom__24', 'openml__kr-vs-kp__3', 'openml__analcatdata_boxing1__3540', 'openml__musk__3950', 'openml__transplant__3748', 'openml__cjs__14967', 'openml__synthetic_control__3512', 'openml__car-evaluation__146192', 'openml__fertility__9984', 'openml__postoperative-patient-data__146210', 'openml__breast-w__15', 'openml__wdbc__9946', 'openml__car__146821', 'openml__visualizing_livestock__3731', 'openml__mfeat-factors__12', 'openml__Satellite__167211', 'openml__colic__25', 'openml__lymph__10', 'openml__wall-robot-navigation__9960', 'openml__wilt__146820', 'openml__scene__3485', 'openml__mfeat-karhunen__16', 'openml__sick__3021', 'openml__dna__167140', 'openml__socmob__3797', 'openml__page-blocks__30', 'openml__PhishingWebsites__14952', 'openml__spambase__43', 'openml__splice__45', 'openml__churn__167141', 'openml__colic__27', 'openml__ecoli__145977', 'openml__semeion__9964', 'openml__ozone-level-8hr__9978', 'openml__heart-h__50', 'openml__pc1__3918', 'openml__qsar-biodeg__9957', 'openml__autos__9', 'openml__pc4__3902', 'openml__hill-valley__145847', 'openml__satimage__2074', 'openml__pc3__3903', 'openml__mfeat-fourier__14', 'openml__Australian__146818', 'openml__credit-approval__29', 'openml__cylinder-bands__14954', 'openml__mfeat-zernike__22', 'openml__kc2__3913', 'openml__bank-marketing__14965', 'openml__phoneme__9952', 'openml__elevators__3711', 'openml__breast-cancer__145799', 'openml__SpeedDating__146607', 'openml__kc1__3917', 'openml__adult-census__3953', 'openml__ilpd__9971', 'openml__vehicle__53', 'openml__ada_agnostic__3896', 'openml__tae__47', 'openml__blood-transfusion-service-center__10101', 'openml__jasmine__168911', 'openml__LED-display-domain-7digit__125921', 'openml__diabetes__37', 'openml__Click_prediction_small__190408', 'openml__profb__3561', 'openml__steel-plates-fault__146817', 'openml__jm1__3904', 'openml__glass__40', 'openml__dresses-sales__125920', 'openml__mfeat-morphological__18', 'openml__eucalyptus__2079', 'openml__libras__360948', 'openml__yeast__145793', 'openml__cmc__23', 'openml__analcatdata_dmft__3560']

datasets_table_2 = ["openml__Australian__146818", "openml__LED-display-domain-7digit__125921", "openml__MiceProtein__146800", "openml__acute-inflammations__10089", "openml__analcatdata_authorship__3549", "openml__analcatdata_boxing1__3540", "openml__analcatdata_chlamydia__3739", "openml__analcatdata_dmft__3560", "openml__anneal__2867", "openml__autos__9", "openml__balance-scale__11", "openml__blood-transfusion-service-center__10101", "openml__blood-transfusion-service-center__145836", "openml__breast-cancer__145799", "openml__breast-w__15", "openml__colic__25", "openml__colic__27", "openml__credit-approval__29", "openml__cylinder-bands__14954", "openml__dermatology__35", "openml__diabetes__37", "openml__dresses-sales__125920", "openml__ecoli__145977", "openml__eucalyptus__2079", "openml__fertility__9984", "openml__fri_c0_100_5__3620", "openml__fri_c3_100_5__3779", "openml__glass__40", "openml__hayes-roth__146063", "openml__heart-c__48", "openml__heart-h__50", "openml__hill-valley__145847", "openml__ilpd__9971", "openml__ionosphere__145984", "openml__iris__59", "openml__irish__3543", "openml__kc2__3913", "openml__labor__4", "openml__lung-cancer__146024", "openml__lymph__10", "openml__monks-problems-2__146065", "openml__pc1__3918", "openml__postoperative-patient-data__146210", "openml__profb__3561", "openml__qsar-biodeg__9957", "openml__rabe_266__3647", "openml__socmob__3797", "openml__sonar__39", "openml__synthetic_control__3512", "openml__tae__47", "openml__tic-tac-toe__49", "openml__transplant__3748", "openml__vehicle__53", "openml__visualizing_environmental__3602", "openml__visualizing_livestock__3731", "openml__wdbc__9946", "openml__yeast__145793"]

I just saw this issue. Are you aware that the datasets for Table 2 have a duplicate?
"openml__blood-transfusion-service-center__10101", "openml__blood-transfusion-service-center__145836"?

@LennartPurucker thanks for pointing this out - cc @crwhite14 . so we could remove the duplicate dataset from results that include it.
it looks like we accidentally pulled two different openML tasks (https://openml.org/search?type=task&id=145836 and https://openml.org/search?type=task&id=10101) which appear to be identical, because they are based on the same dataset (https://openml.org/search?type=data&id=1464)


Discussion on Pruning the 0 Layers

Hello! I may have found a bug in pruning.
Take vgg16 as an example. I chose a pruned_ratio of 0.6, and obtained the following cfg_mask:
[55, 64, 'M', 128, 128, 'M', 256, 253, 239, 'M', 142, 28, 8, 'M', 0, 1, 387]
One conv layer is pruned to zero. Based on my understanding, you skip the zero layer, build a ‘vgg15’ network based on the rest 12 conv layers. Then, you conduct the pruning by filling in the new network’s weight, based on the origin network’s weight.
In code implementation, you use start_mask and end_mask to mark the selected input and output channels. End_mask is actually an element of cfg_mask, which marks the origin network’s selected channels. When you encounter a layer with 0 channel, you have to skip it. So, I guess your code:

if torch.sum(end_mask) == 0:
continue

is designed to do the job. This line of code is placed right after the judgement of conv/BN layer. However, end_mask is updated only in the end of BN layer pruning. This result in the phenomenon: once the end_mask is all 0, it never changes, and the pruning algorithm directly skips all layers and ends.
Actually, I don’t think you have conducted pruning on the layers after the 0 layer. This can be seen by breakpoint debugging on the last BN layer, which weights are all 0.5(initial value). In addition, you print each conv layer right after conducting pruning on it. However, layers after the 0 layer have no printed information. I think those layers remain the initial weight.
Hope to receive your reply. Maybe I have problem understanding the code, if so please point it out. Thank you very much for your attention.
Best regards

Hi, thanks for your careful checking, we have addressed this issue and updated results during rebuttal: this “plunge” issue can be fixed by applying a simple “protective pruning” heuristic on top of our current algorithm. Specifically, we stop pruning a layer, when its remaining channels after pruning become less than 10% of the original, for avoiding overly slimming this layer; meanwhile we (uniformly) prune other layers more to meet the overall pruning ratio. Experiments show that such a simple protective strategy can effectively eliminate the plunges as shown in the updated Figures 1-2, without incurring overhead.
bn = []
protective_thre = {}
bn_index = 0
for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        bn_index += 1
        size = m.weight.data.shape[0]
        w = m.weight.data.abs().clone()
        # protective layer-wise threshold
        y, i = torch.sort(w)
        thre_index = int(size * 0.9) # protect top 10%
        protective_thre[bn_index] = y[thre_index]
        # remaining channels are needed to prune
        for item in y[:thre_index]:
            bn.append(item)


Added transpose to var<mat> (Issue #2101)

Summary
This is transpose for var<mat>.
Tests
It's all a lot of copy-paste of all the Eigen view things that were there.
Side Effects
No
Release notes
Added transpose for var<mat>
Checklist

[x] Math issue #2101

[x] Copyright holder: Columbia University
The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[x] the basic tests are passing

unit tests pass (to run, use: ./runTests.py test/unit)
header checks pass, (make test-headers)
dependencies checks pass, (make test-math-dependencies)
docs build, (make doxygen)
code passes the built in C++ standards checks (make cpplint)

[x] the code is written in idiomatic C++ and changes are documented in the doxygen

[x] the new changes are tested

@SteveBronder I remember you told me to make sure transpose doesn't malloc anything secretly. Not sure how to do that so I'll just nominate you for reviewer and lemme know if this isn't right :D
I just implemented it like a view like all the other access operators.

@t4c1 one thing I'm confused about here, should this pop off a new var<mat> and in the reverse pass do a transpose + accumulate? It's not like a block where we take a slice so a bit confused

I think this looks fine. It is just a view, like block. The only difference this view uses row-major indexing, so I think using it in an expression together with a col-major matrix won't use vectorization. But unless it is used many times this should be preferable to making a copy.


InputEventMouseButton with modifiers does not work in InputMap

Godot version
3.3.3 stable
System information
windows 10, gles3
Issue description
Adding InputEventMouseButton with alt modifier to action in InputMap does not work.
Steps to reproduce
extends Node2D

func _ready():
  InputMap.add_action("abc")
  var evt = InputEventMouseButton.new()
  evt.button_index = BUTTON_LEFT
  evt.alt = true     # I need alt + left_click to trigger the action
  InputMap.action_add_event("abc", evt)

func _input(_event):
  if Input.is_action_just_pressed("abc"):
    print(OS.get_ticks_usec())  # No Alt key is needed to print this

Minimal reproduction project
InputIssue.zip

Confirmed in 3.3.3-stable, 3.x (25bd0c3f7) and master (104a6191f).
It works fine with InputEventKey on the other hand.

It's a duplicate of #37835, but since this new issue has a reproduction project, I'll close the other one. There was a PR attempting to fix it but it wasn't working apparently: #48789.

CC @godotengine/input. Is there a specific limitation with supporting modifiers on mouse button actions, or is it just a plain oversight?

If it's not a limitation, can the fix be merged into 3.4 ?

So this fixes the issue:
diff --git a/core/input/input_event.cpp b/core/input/input_event.cpp
index 50b2099236..c77ed0bf2f 100644
--- a/core/input/input_event.cpp
+++ b/core/input/input_event.cpp
@@ -586,7 +586,7 @@ bool InputEventMouseButton::action_match(const Ref<InputEvent> &p_event, bool *p
                return false;
        }

-       bool match = mb->button_index == button_index;
+       bool match = is_match(mb, true);
        if (match) {
                bool pressed = mb->is_pressed();
                if (p_pressed != nullptr) {

It uses the new "exact match" feature added by @EricEzaM and backported to 3.4.
Here's a test project for master: InputIssue.zip
Now the question is whether there are any drawbacks to enforcing an exact match for action_match? I feared it could impact editor functionality but from a quick test it seems fine with this patch (e.g. stepped transformations with Ctrl+Left click, or faster freelook movement in 3D with Shift+Right click).

I think it is an oversight, yes. I don't think this is intended.
Your change did not affect the editor since for those shortcuts the editor queries the Input singleton directly (e.g. Input::get_singleton()->is_key_pressed(...)), not via actions,  so changes to action code wouldn't affect those particular scenarios.

Fixed by #54173 (I suppose). It's no longer reproducible in 3.5.


Validator behaviour when encountering unexpected enum values

Is your feature request related to a problem? Please describe.
It is presently unclear how the validator should behave when it encounters an unexpected enum value in an expected field.
Describe the solution you'd like
TODO: the result of the discussion will be resumed here
Additional context
Stem from https://github.com/MobilityData/gtfs-validator/pull/44#discussion_r387732469

Closing, addressed in the new v2 architecture

Closing, addressed in the new v2 architecture


[2] Support deployment of K8s on VMware vSphere using kubespray

Is your feature request related to a problem? Please describe.
Currently kubespray does not provide terraform scripts for VMware vSphere.
Describe the solution you'd like
Kubespray should support deployment on VMware vSphere by providing appropriate terraform scripts.
Describe alternatives you've considered
N/A
Additional context
Implementation of this feature will allow us to easily deploy CK8s on top of VMware vSphere.
Definition of done:
Kubespray successfully installs K8s cluster on VMware vSphere.

Excellent! Will you make the corresponding issue and subsequent PR over at kubespray, too? :)

From an internal Slack conversation, it seems like this may be considered "Done".
@jakubkrzywda, please confirm.

Link to the PR at kubespray (which is now merged) https://github.com/kubernetes-sigs/kubespray/pull/7306


Add kwargs to import_row, import_object and import_field

Added in resources.py kwargs on def import_row line 506, def import_object line 359 and def import_field line 348. Because I need the user id of the reqisition that is in kwargs, on import_field def. I need this user to filter objects by my permission system on a manager of the class.
Signed-off-by: GabrielPetz<EMAIL_ADDRESS>Problem
I have a customized permission system to different users, so I need them id before import fields, in my case I need subscribe this method import_field, adding the user of request.
Solution
How did you solve the problem?
The solution for my problem has been adding in resources.py argument kwargs on functions import_row line 506, import_object line 359 and import_field line 348
Acceptance Criteria
There is no need to write a different test, the test_resource.py file already calls the import_data > import_data_inner > import_row > import_obj > import_field , then the class of tests already coverage the changes. My changes are in bold.
There is no need to screenshots, because my changes are just in resources.py, there is no other changes.
I just added kwargs argument on functions, so there is no need to docs in function body to describe it.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

Coverage remained the same at 95.668% when pulling efd41a625ba7e9ee2ad9a659b5dd81848fed47a3 on simepar:master into 3c082f98afe7996e79f936418fced3094f141c26 on django-import-export:master.

I've made some changes into fields.py, on clean and save functions, and the same changes into save function call on import_field def on resources.py, so that way was easier to solve my problem without needed to change more your code. now I just need to take the user into def queryset of widget class.

@andrewgy8 Your Django version have conflicts with the 3.8 version of python, you guys need to do some checks on your coverage tests, on the script have a command: export DJANGO="Django>=2.0,<2.1", and this is making the conflict with python 3.8

I'm closing this pull request for now. But I'll open another with the same solutions that I'v made here.


Add constructor option to allow reusing an axios instance

This is for issue #77
This adds the axiosInstance property to the constructor options. This PR also includes some refactoring and new tests.
I added this because some users want the option to reuse another axios instance. The change will make this library compatible with more libraries.
I added tests to ensure that none of the properties of the default request config are changed. This partially addresses npdev453's 2nd point in the issue thread. The baseUrl property is the only one that might be modified, but only when it's undefined.
The documentation needs to be updated to inform the user about potential problems with using the axiosInstance property. It looks like the documentation is separate from this repo, so I can't edit it, but I added the new property to the function documentation.

when will this be applied? we need instances of axios to be used in our project and it seems to be the answer.
specifically we want to convert snake_case and camelCase using axios-case-converter package.


bruteforce: Fix comparions of unsigned shorts with 0.0f

Comparing test (of type cl_half) with 0.0f is incorrect because cl_half is a typedef of uint16_t (or unsigned short in c)

Merging as discussed in the October 1st teleconference.


Support --verbose/-v flag in resin ssh

Closes https://github.com/resin-io/resin-cli/issues/353

I'll release a new version with this change in a moment.

Published v4.1.0.

Thank you!


Objects not null when passed as parameter.

Hi there,
When I create the following PL/SQL type:
CREATE OR REPLACE TYPE PERSON
IS
  OBJECT
  (
    name     VARCHAR2(255),
    surname  VARCHAR2(255),
    age      NUMBER(10, 0)
  );

and the following procedure for testing purposes:
CREATE OR REPLACE PROCEDURE test_person(arg PERSON) AS
BEGIN
  IF arg IS NULL THEN
    RAISE_APPLICATION_ERROR(-20001, 'Person cannot be null.');
  END IF;
END;

and create the following test:
require "spec_helper"

describe "People Suite" do
  it "should break when called with a NULL parameter" do
    lambda do
      plsql.test_person(NULL)
    end.should raise_error(/ORA-20001/)
  end
end

I get the following error when I call the test_person procedure:
Failures:

People Suite should break when called with a NULL parameter
Failure/Error: lambda do
expected Exception with message matching /ORA-20001/ but nothing was raised
./spec/Test_spec.rb:5

I would have suspected the test_person to throw an exception when calling with a NULL parameter but it did not. Any help would be  very much appreciated.
Regards,
Wiehann

is it resolved?

Just tested it - it's resolved.
Can be closed.


DataRequest response is observed always on DispatchQueue.main ignoring any subscribeOn

The DataRequest response is always observed on DispatchQueue.main.
For example:
let queue = DispatchQueue(label: "RxAlamofireTest", qos: .background)
let schedule = SerialDispatchQueueScheduler(queue: queue, internalSerialQueueName: "RxAlamofireTest")
let urlString = "https://jsonplaceholder.typicode.com/todos/1"
let bag = DisposeBag()

request(.get, urlString)
    .subscribeOn(schedule)
    .do(onNext: { (_) in
        print("[1] isMainThread: \(Thread.isMainThread)")
    })
    .data()
    .subscribe(onNext: { (_) in
        print("[2] isMainThread: \(Thread.isMainThread)")
    })
    .disposed(by: bag)

This seems to be because there isn't any queue being passed along when calling DataRequest.response(queue: responseSerializer: completionHandler), which makes it to default to the DispatchQueue.main.
Since there is a .subscribeOn(schedule) shouldn't the subscribe(onNext:) be done on the schedule thread instead of being received on the DispatchQueue.main?

@rrodrigues , this is expected behavior. The wording is confusing. .subscribeOn doesn't specify on which queue/scheduler to deliver events for the subscription. Instead, it means where to execute code associated with initiating the subscription.
You will get the behavior you're looking for if you simply replace your .subscribeOn with .observeOn.

@epatey yes .subscribeOn should specify on which thread the whole chain starts processing, .observeOn allows to switch between threads.
I'm sorry if I wasn't able to explain me correctly.
What I think is not the expected behaviour is that when you create a sequence of operations like:
request(.get, urlString)
    .subscribeOn(schedule) // <= defines which thread the operation should be done
    .data()
    .subscribe(onNext: { (_) in
        // the current thread here should be one defined above by .subscribeOn(schedule)
        // and is not, this is run on the MainThread.
    })
    .disposed(by: bag)

if you don't call .observeOn anywhere the .subscribe(onNext: should be done on the initial thread/queue defined by .subscribeOn.
What is happening here is that on the .data() call internally Alamofire is calling DispatchQueue.main.async and this changes the execution thread/queue.
In my opinion should not be done, the queue where the operation started should be the done used when dispatching the DataRequest response.

if you don't call .observeOn anywhere the .subscribe(onNext: should be done on the initial thread/queue defined by .subscribeOn.

I don't think this makes sense because it doesn't happen like that anywhere in Rx resource processing. You are responsible for deciding where to process and where to receive events.
It really shouldn't (and probably can't) be a library's responsibility to force you to use some scheduler you used earlier, plus library doesn't know which scheduler it's running on usually, since it shouldn't know (it's an implementation detail).

ok.
So my understanding is wrong then.
Thanks for the clarification.


blog: scheduler plugin reusing

A blog post about scheduler plugin reusing as a followup of scheduling-sig discussions. In its own folder, because of the svg-images that are used.

/assign @sftim
/assign @onlydole

Discussion in sig-scheduling resulted in a proposal to move this to https://www.kubernetes.dev/blog/ which to me makes sense.
@sftim or @onlydole could you help directing me with how to transfer this there?

Discussion in sig-scheduling resulted in a proposal to move this to https://www.kubernetes.dev/blog/

Fork and clone https://github.com/kubernetes/contributor-site
Copy the text of the article into that site's blog directory. Make sure to adjust the content to match existing articles as the conventions are slightly different.
Finally, open a PR


Application of Mish on ReXNet

ReXNet: Diminishing Representational Bottleneck on Convolutional Neural Network
This paper addresses representational bottleneck in a network and proposes new design principles.
The authors apply this method on MobileNetV1 and V2 and gain performance that scales much better than EfficientNets
One of the tools they use is Swish. In this instance, they actually provide theoretical and empirical justification for this. (This involves the rank of inputs and outputs, and is explained within the paper)
The authors do not discuss or use Mish however, which is shown to be better in extensive use cases.
Since the ReXNet is quite simple, it would seem to be relatively straight forward to conduct studies investigating the replacement of swish with mish in the relevant settings.
I hope this is of interest to you.

Yes, I came across this paper a couple of days ago. I'm inclined towards testing Mish on this architecture however I don't have the compute resources for ImageNet experiments allocated for Mish as of now.

Since, I don't have access to any compute now and it won't be possible for me to run this model, I'm closing the issue as of now but will keep this on my checklist to run whenever I have access to a cluster. Thanks!


World Map Metathread

Top Down Map

Ninja rides on Capybara-Donkey on the world map

World Map Structure Key
Black = Main Level
Purple = Optional Level
Brown = Town
Orange = Aesthetic Structure/Feature


round-trip the fieldPath value in FromFieldPath patches through integer-preserving JSON unmarhsal

Fixes #101

this fixes the remaining issue reported in #101 w.r.t appendSlice patch policy in function-p-and-t having different behavior than the native P&T.
As explained in https://github.com/crossplane-contrib/function-patch-and-transform/issues/101#issuecomment-2022468812,  the from source field value is being round-trip'ed through Kubernetes JSON decoder, so that the json integers are unmarshalled as int64 consistent with to/dest value handling here.

@phisco @negz  PTAL.


New package: PETLION v0.1.0

Registering package: PETLION
Repository: https://github.com/MarcBerliner/PETLION.jl
Created by: @MarcBerliner
Version: v0.1.0
Commit: d4e4e1c64069c035142109dce4c39777ca37b60c
Git reference: HEAD

PETLION is an acronym for Porous Electrode Theory for Lithium-Ion batteries that follows a similar naming convention to another battery toolbox, LIONSIMBA, in MATLAB. Details of the PETLION package will be published in a journal.

PETLION is an acronym for Porous Electrode Theory for Lithium-Ion batteries that follows a similar naming convention to another battery toolbox, LIONSIMBA, in MATLAB. Details of the PETLION package will be published in a journal.

PETLION is an acronym for Porous Electrode Theory for Lithium-Ion batteries that follows a similar naming convention to another battery toolbox, LIONSIMBA, in MATLAB. Details of the PETLION package were accepted for publication.
I've also fixed the compat problems.
[noblock]


Adding latest and new innovations in AI for farming

@manikumarreddyu
I would like to add the new innovations in AI which will help farmers in agriculture
could you please assign me this task

not relevant..to this website


Remove redundant  call in ops.numpy.matmul

keras.backend.jax.numpy.matmul already calls convert_to_tensor() on both operands so we can safely remove this redundant call in keras.ops.numpy.matmul.
see: https://github.com/keras-team/keras/blob/master/keras/backend/jax/numpy.py#L48-L50

Codecov Report
All modified lines are covered by tests :white_check_mark:

Comparison is base (c9dc296) 77.58% compared to head (13da676) 57.28%.

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #18530       +/-   ##
===========================================
- Coverage   77.58%   57.28%   -20.30%
===========================================
  Files         334      334
  Lines       32211    32209        -2
  Branches     6286     6286
===========================================
- Hits        24990    18451     -6539
- Misses       5636    12334     +6698
+ Partials     1585     1424      -161

Flag
Coverage Δ

keras
57.28% <ø> (-20.21%)
:arrow_down:

keras-jax
?

keras-numpy
57.28% <ø> (-0.01%)
:arrow_down:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files
Coverage Δ

keras/ops/numpy.py
94.81% <ø> (-0.06%)
:arrow_down:

... and 179 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[CupertinoActionSheet] Match colors to native

This PR matches the various colors of CupertinoActionSheet more closely with the native one.
The following colors are changed.

Sheet background color
Pressed button color
Cancel button color
Pressed cancel button color
Divider color
Content text color

The resulting colors match with native one with deviation of at most 1 (in terms of 0~255 RGB).
The following are comparison (left to right: Native, Flutter after PR, Flutter current)

Note: The divider thickness is adjusted to 1/dpr instead of 0.3 in both Flutter version to make them look more native, as will be proposed in https://github.com/flutter/flutter/pull/149636.
Derivation
All the colors are derived through color picker and calculation. The algorithm is as followed:

Assume all colors are translucent grey colors, i.e. having the same value x for R, G, and B, with an alpha a.
Given the barrier color is x_B1=0 when the background is black, and x_B2=204 when the background is white.
Pick the target color x_t1 when the background is black, and x_t2 when the background is white
Solve the following equations for x and a

a * x + (1-a) * x_B1 = x_t1
a * x + (1-a) * x_B2 = x_t2

a = 1 - (x_t1 - x_t2) / (x_B1 - x_B2)
x = (x_t1 - (1-a) * x_B1) / a

These equations use a linear model for color composition, which might not be exact, but is close enough for an accuracy of (1/255).
The full table is as follows:

The first two columns are colors picked from XCode.
The 3~4 columns are the colors picked from the current Flutter. Notice the deviation, which is sometimes drastic.
The 5~6 columns are the colors picked from Flutter after this PR. The deviation is at most 1.
The last few columns are calculation.

There are two rows whose calculation is based on adjusted numbers, since the original results are not accurate enough, possibly due to the linear composition.

During the calculation, I assumed these colors vary between light and dark modes, but it turns out that both modes use the same set of colors.
Screenshots
Pre-launch Checklist

[ ] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] I read the Tree Hygiene wiki page, which explains my responsibilities.
[ ] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[ ] I signed the CLA.
[ ] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] I added new tests to check the change I am making, or this PR is test-exempt.
[ ] I followed the breaking change policy and added Data Driven Fixes where supported.
[ ] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Reason for revert: Appears to be failing tests in tree
01:48 +2991 ~18: /Volumes/Work/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/action_sheet_test.dart: Taps on a button can be slided to other buttons
 ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
 The following SkiaException was thrown while running async test code:
 Skia Gold received an unapproved image in post-submit
 testing. Golden file images in flutter/flutter are triaged
 in pre-submit during code review for the given PR.
 Visit https://flutter-gold.skia.org/ to view and approve
 the image(s), or revert the associated change. For more
 information, visit the wiki:
 https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
 Debug information for Gold --------------------------------
 stdout: Given image with hash 601f421e3bb643f437cc12395de96152 for test
 cupertino.cupertinoActionSheet.press-drag
 Expectation for test: ce8ef79c146857d162663b1161fd6d5c (positive)
 Expectation for test: 3bb57a8782f67836c6ad3ece7f00729a (positive)
 Expectation for test: 3f8c592774caf3c760fbbd318a7dc5af (positive)
 Expectation for test: 61863f38217aa3349c239eeb49b84930 (positive)
 Expectation for test: 98af16e9b7eb3fb810b44c74bb18ffb9 (positive)
 Untriaged or negative image:
 https://flutter-gold.skia.org//detail?grouping=name%3Dcupertino.cupertinoActionSheet.press-drag%26source_type%3Dflutter&digest=601f421e3bb643f437cc12395de96152
 stderr: Test: cupertino.cupertinoActionSheet.press-drag FAIL
 result-state.json: No result file found.

PR to manual revert: https://github.com/flutter/flutter/pull/149998

@dkwingsmt I have problems with the colors in dark mode, I suspect this PR has broken something.

@dkwingsmt I have problems with the colors in dark mode, I suspect this PR has broken something.
I see some colors are not resolved by CupertinoDynamicColor.resolve, maybe thats the issue.

Can you provide me with screenshots and code to reproduce? I might have used the wrong method to test.

Hi @dkwingsmt,
I´m using this code to reproduce:
void main() {
 runApp(TestScaffoldApp(
   theme: const CupertinoThemeData(brightness: Brightness.dark),
   actionSheet: CupertinoActionSheet(
     actions: [
       CupertinoActionSheetAction(
         onPressed: () {},
         child: Text('Button'),
       ),
       CupertinoActionSheetAction(
         onPressed: () {},
         child: Text('Button'),
       ),
     ],
     cancelButton: CupertinoActionSheetAction(
       isDefaultAction: true,
       child: Text("Cancel"),
       onPressed: () {},
     ),
   ),
 ));
}

// Shows an app that has a button with text "Go", and clicking this button
// displays the `actionSheet` and hides the button.
//
// The `theme` will be applied to the app and determines the background.
class TestScaffoldApp extends StatefulWidget {
 const TestScaffoldApp(
     {super.key, required this.theme, required this.actionSheet});
 final CupertinoThemeData theme;
 final Widget actionSheet;

 @override
 TestScaffoldAppState createState() => TestScaffoldAppState();
}

class TestScaffoldAppState extends State<TestScaffoldApp> {
 bool _pressedButton = false;

 @override
 Widget build(BuildContext context) {
   return CupertinoApp(
     theme: widget.theme,
     home: Builder(
       builder: (BuildContext context) => CupertinoPageScaffold(
         child: Center(
           child: _pressedButton
               ? Container()
               : CupertinoButton(
                   onPressed: () {
                     setState(() {
                       _pressedButton = true;
                     });
                     showCupertinoModalPopup<void>(
                       context: context,
                       builder: (BuildContext context) {
                         return widget.actionSheet;
                       },
                     );
                   },
                   child: const Text('Go'),
                 ),
         ),
       ),
     ),
   );
 }
}

This is where I use your PR using Dark Mode:

This is running the latest master (still not correct, so maybe some other PR has broken this behavior):

The first screenshot is exactly what I observed on an Xcode native app when the dark mode toggle on the status bar is on. Did I do it wrong? Can you show me what color it should be?

Maybe this is a bug in Xcode itself, I can't reproduce it myself in Xcode right now, but this is what a native action sheet looks like in dark mode in the Contacts app.

If I use CupertinoDynamicColors.resolve for all colors in _ActionSheetButtonBackground (currently _kPressed, _kActionSheetCancelPressedColor and secondarySystemGroupedBackground Color are not resolved),
then I get these colors (matching native)

Thanks you very much! Yeah I couldn't believe it either when I saw that the colors were the same in dark mode. I'll do more research (probably by running the native app on my phone).


feat(CompoundVault): Compound Voting Vaults

Compound Voting Vault that assigns voting power to cTokens weighted by the Time-weighted Average Borrow Rate (TWAR).
Compound is external to the governance system, and thus represents a leakage of governance tokens. Someone could lend out their Element token to receive cElement, and then borrow the Element token from Compound to essentially "double spend" a governance vote. To prevent this, we leverage the borrow utilization rate of cElement to weight the voting power of a cToken appropriately. To mitigate governance attacks that involve manipulating this borrow rate, we utilize the Time-weighted Average Borrow Rate (TWAR). TWAR operates similarly to Uniswap's TWAP. Thus, when the borrow rate is high (meaning more Element tokens have been borrowed), cTokens have less voting power (and vice-versa).
Design Choices
As the TWAR changes over time, the voting weight assigned to cTokens subsequently changes. There were some subtleties around how to calculate voting power around this, and there were two ways considered to handle this.

The first was to store the historical deposits in terms of cTokens. In a call to queryVotePower, the historical balance of cTokens at the desired block will be found, and that balance will be multiplied by the current TWAR multiplier (which is a function of the borrow rate).
The second way is to store voting power instead of cToken balances. During a deposit, the cTokens would be multiplied by the TWAR multiplier at the given block, and this value would be added to the current voting power. During withdrawals, we can’t simply use the current TWAR since deposits may have occurred at multiple time increments in which the TWAR was at different values. We handle this by taking the ratio of cTokens being withdrawn and deducting the same ratio from the voting power.

Both designs have trade-offs, but (1) seemed a cleaner UX. Two users who control the same amount of cTokens (same amount of underlying) would have the same voting power regardless of what the TWAR was when they deposited. Note that if the TWAR update interval is set to less than the voting period, the voting weight could fluctuate slightly and the same number of cTokens will be worth slightly different total votes over the course of the proposal voting period. Since the TWAR smoothens out volatility and borrow rates should remain relatively constant over a short time period, we don’t anticipate this being an issue.
Additionally, to maintain consistency with other Voting Vaults, we emit a VoteChange event on deposits & withdrawals. Because we deal with cToken balances, to emit these events we have to calculate the voting power for the newly minted/withdrawn amount of cTokens. This requires logic that accesses storage & increases gas costs for users simply for emitting an event. If a VoteChange event is not required, I propose swapping the event for a cTokenBalanceChange event instead, which doesn't require accessing extra storage.
Notes
This is un-audited code, and should be rebased to a development branch instead of being merged into main.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

96 of 96 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 94.465%

Totals

Change from base Build<PHONE_NUMBER>:
0.3%

Covered Lines:
602

Relevant Lines:
619

💛  - Coveralls

approved

This is stale so I'm closing this PR.  If there is interest for this later we can open another PR.


[SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

Why are the changes needed?
Recomputting shuffle blocks can be expensive, we should take advantage of our decommissioning time to migrate these blocks.
Does this PR introduce any user-facing change?
This PR introduces two new configs parameters, spark.storage.decommission.shuffle_blocks & spark.storage.decommission.rdd_blocks that control which blocks should be migrated during storage decommissioning.
How was this patch tested?
New unit test & expansion of the Spark on K8s decom test to assert that decommisioning with shuffle block migration means that the results are not recomputed even when the original executor is terminated.
This PR is a cleaned-up version of the previous WIP PR I made https://github.com/apache/spark/pull/28331 (thanks to @attilapiros for his very helpful reviewing on it :)).

Test build #123448 has started for PR 28708 at commit 4730a52.

cc reviewers from the WIP PR: @attilapiros  , @dongjoon-hyun , @prakharjain09  , @viirya

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28066/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28066/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28066/
Test FAILed.

Test build #123448 has finished for PR 28708 at commit 4730a52.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait MigratableResolver
class RDDBlockSavedOnDecommissionedBlockManagerException(blockId: RDDBlockId)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123448/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28071/
Test FAILed.

Test build #123452 has started for PR 28708 at commit 7fb4356.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28072/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28072/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28072/
Test FAILed.

cc @vanzin, @jiangxb1987, @Ngone51 FYI

cc @vanzin, @jiangxb1987, @Ngone51 too FYI

I think the k8s failure is unrelated (e.g.   java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: org.apache.spark.deploy.master.ui.MasterWebUISuite) seems like a problem with the UI suite.

Test build #123452 has finished for PR 28708 at commit 7fb4356.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait MigratableResolver
class RDDBlockSavedOnDecommissionedBlockManagerException(blockId: RDDBlockId)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123452/
Test PASSed.

I think we should test the offloading of the shuffle blocks at the unit test level so I suggest to add new test to the BlockManagerSuite:
  test("test migration of shuffle blocks during decommissioning") {
    val store1 = makeBlockManager(3500, "exec1")
    val store2 = makeBlockManager(3500, "exec2")

    val blockSize = 5
    val shuffleDataBlockContent = Array[Byte](0, 1, 2, 3, 4)
    val shuffleData = ShuffleDataBlockId(0, 0, 0)
    Files.write(store1.diskBlockManager.getFile(shuffleData).toPath(), shuffleDataBlockContent)
    val shuffleIndexBlockContent = Array[Byte](5, 6, 7, 8, 9)
    val shuffleIndex = ShuffleIndexBlockId(0, 0, 0)
    Files.write(store1.diskBlockManager.getFile(shuffleIndex).toPath(), shuffleIndexBlockContent)

    mapOutputTracker.registerShuffle(0, 1)
    mapOutputTracker.registerMapOutput(0, 0, MapStatus(store1.blockManagerId, Array(blockSize), 0))
    assert(mapOutputTracker.shuffleStatuses(0).mapStatuses(0).location === store1.blockManagerId)

    val env = mock(classOf[SparkEnv])
    when(env.blockManager).thenReturn(store1)
    when(env.conf).thenReturn(conf)

    SparkEnv.set(env)
    store1.offloadShuffleBlocks()

    eventually(timeout(1.second), interval(10.milliseconds)) {
      assert(mapOutputTracker.shuffleStatuses(0).mapStatuses(0).location === store2.blockManagerId)
    }
    assert(Files.readAllBytes(store2.diskBlockManager.getFile(shuffleData).toPath())
      === shuffleDataBlockContent)
    assert(Files.readAllBytes(store2.diskBlockManager.getFile(shuffleIndex).toPath())
      === shuffleIndexBlockContent)
  }

I think this could be a good starting point for any other tests at this level. This way we could check the streaming upload here and remove it from the BlockManagerReplicationSuite which is a higher level test (gravitate towards the integration testing). What do you think?

Test build #123499 has started for PR 28708 at commit d988e7f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28122/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28122/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28122/
Test FAILed.

So I don't want to stop the executor directly once the block migration is done. Instead, I have a follow-up JIRA which I've started working on that shutdowns the executor once the block migration has completed and there are no running tasks. I think it's ok to (temporarily) leak threads since decommissioning (as triggered currently) is only in the situation where the executor will be exiting soon anyways.

Test build #123510 has started for PR 28708 at commit b2da4c5.

I like the idea of adding a more specific unit test for the streaming upload so we can save the (slower) more integration style test for the other components, thanks for writing the test @attilapiros :)

Test build #123511 has started for PR 28708 at commit 7dde80e.

Test build #123499 has finished for PR 28708 at commit d988e7f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait MigratableResolver
class RDDBlockSavedOnDecommissionedBlockManagerException(blockId: RDDBlockId)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123499/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28133/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28134/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28134/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28134/
Test FAILed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28133/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28133/
Test FAILed.

Test build #123514 has started for PR 28708 at commit 9c8836a.

Hey @attilapiros can you explain to my why you think we need to test the different kinds of block fetches? When we migrate we're always migrating to disk so I'm not seeing how it should matter.

Test build #123515 has started for PR 28708 at commit ccb8827.

Test build #123510 has finished for PR 28708 at commit b2da4c5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123510/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28137/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28137/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28137/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28138/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28138/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28138/
Test PASSed.

Test build #123511 has finished for PR 28708 at commit 7dde80e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123511/
Test PASSed.

Test build #123514 has finished for PR 28708 at commit 9c8836a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123514/
Test PASSed.

Test build #123515 has finished for PR 28708 at commit ccb8827.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123515/
Test PASSed.

So I don't want to stop the executor directly once the block migration is done. Instead, I have a follow-up JIRA which I've started working on that shutdowns the executor once the block migration has completed and there are no running tasks. I think it's ok to (temporarily) leak threads since decommissioning (as triggered currently) is only in the situation where the executor will be exiting soon anyways.

I understand your point but as we are testing the decommissioning these threads are leaking during testing and even aggregating by each test using this feature.
We are logging this leaks in this method https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/ThreadAudit.scala#L106 at the end of the suite.
I believe it is better to keep such log completely clean if the required code change is easy otherwise the next person felt to be encouraged to add just one more violation.

Hey @attilapiros can you explain to my why you think we need to test the different kinds of block fetches? When we migrate we're always migrating to disk so I'm not seeing how it should matter.

I meant to test the different kinds of block uploads (not fetches): like streaming uploads and non-streaming ones. Those are currently tested by the BlockManagerDecommissionSuite controlled by the flag spark.network.maxRemoteBlockSizeFetchToMem.
But you are right shuffle uploads always should manifest in the file system. To achieve this we should change this condition:
https://github.com/apache/spark/blob/ccb8827c61e3178c83ae505dbddd30b1a6a2d361/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala#L171
to
    val asStream = blockId.isInternalShuffle ||
      blockData.size() > conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)

So @attilapiros looking at the Jenkins console logs we aren't leaking any threads during testing (nor would I expect us to). But I'll add something to more aggressively stop the shuffle migration threads.

So @attilapiros looking at the Jenkins console logs we aren't leaking any threads during testing (nor would I expect us to). But I'll add something to more aggressively stop the shuffle migration threads.

It will come when the BlockManager will be tested in BlockManagerSuite:
 ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.storage.BlockManagerSuite, thread names: rpc-boss-3-1, migrate-shuffle-to-BlockManagerId(exec2, localhost, 50804, None), shuffle-boss-9-1  , shuffle-boss-6-1 =====

So @attilapiros looking at the Jenkins console logs we aren't leaking any threads during testing (nor would I expect us to). But I'll add something to more aggressively stop the shuffle migration threads.

It will come when the BlockManager will be tested in BlockManagerSuite:
 ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.storage.BlockManagerSuite, thread names: rpc-boss-3-1, migrate-shuffle-to-BlockManagerId(exec2, localhost, 50804, None), shuffle-boss-9-1  , shuffle-boss-6-1 =====

Gotcha was looking for the explicit decom test. I'll eagerly shutdown the migrate-shuffle-to threads then.

Test build #123539 has started for PR 28708 at commit 81e29a8.

Test build #123539 has finished for PR 28708 at commit 81e29a8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123539/
Test FAILed.

Hey @attilapiros can you explain to my why you think we need to test the different kinds of block fetches? When we migrate we're always migrating to disk so I'm not seeing how it should matter.

I meant to test the different kinds of block uploads (not fetches): like streaming uploads and non-streaming ones. Those are currently tested by the BlockManagerDecommissionSuite controlled by the flag spark.network.maxRemoteBlockSizeFetchToMem.
But you are right shuffle uploads always should manifest in the file system. To achieve this we should change this condition:
https://github.com/apache/spark/blob/ccb8827c61e3178c83ae505dbddd30b1a6a2d361/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala#L171
to
    val asStream = blockId.isInternalShuffle ||
      blockData.size() > conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)

Yeah I know that we test it explicitly right now, but I'm not seeing why so I think maybe just taking out the test entirely is better.

Test build #123541 has started for PR 28708 at commit bce1613.

Test build #123542 has started for PR 28708 at commit a904030.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28162/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28164/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28165/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28162/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28162/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28164/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28164/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28166/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28166/

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28166/
Test FAILed.

Test build #123541 has finished for PR 28708 at commit bce1613.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123541/
Test FAILed.

Test build #123542 has finished for PR 28708 at commit a904030.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123542/
Test PASSed.

Test build #123548 has started for PR 28708 at commit 60bec89.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28172/

So @attilapiros looking at the Jenkins console logs we aren't leaking any threads during testing (nor would I expect us to). But I'll add something to more aggressively stop the shuffle migration threads.

It will come when the BlockManager will be tested in BlockManagerSuite:
 ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.storage.BlockManagerSuite, thread names: rpc-boss-3-1, migrate-shuffle-to-BlockManagerId(exec2, localhost, 50804, None), shuffle-boss-9-1  , shuffle-boss-6-1 =====

Gotcha was looking for the explicit decom test. I'll eagerly shutdown the migrate-shuffle-to threads then.

I think the latest changes have fixed this (e.g. grep "THREAD LEAK" consoleFull  |grep BlockManager returns nothing). Worth noting  we do leak threads in ~283 tests so I'm not sure how important this is.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28172/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28172/
Test FAILed.

Test build #123548 has finished for PR 28708 at commit 60bec89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123548/
Test PASSed.

The K8s test failure appears unrelated (- Run in client mode. *** FAILED ***) we don't do anything with the tokens. I'll investigate more tomorrow.

I have created one unit test and opened a PR with it: https://github.com/holdenk/spark/pull/5
With sbt it is passing as expected:
$ ./build/sbt -Pyarn -Phive -Pkubernetes
Using /Users/attilazsoltpiros/.sdkman/candidates/java/current as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
[info] Loading project definition from /Users/attilazsoltpiros/git/attilapiros/reviewing/project
[info] Resolving key references (21867 settings) ...
[info] Set current project to spark-parent (in build file:/Users/attilazsoltpiros/git/attilapiros/reviewing/)
> project core
[info] Set current project to spark-core (in build file:/Users/attilazsoltpiros/git/attilapiros/reviewing/)
> testOnly *.BlockManagerSuite -- -z "test migration of shuffle blocks during decommissioning"
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Packaging /Users/attilazsoltpiros/git/attilapiros/reviewing/core/target/scala-2.12/spark-core_2.12-3.1.0-SNAPSHOT.jar ...
[info] Done packaging.
[info] BlockManagerSuite:
[info] - test migration of shuffle blocks during decommissioning (278 milliseconds)
[info] ScalaTest
[info] Run completed in 3 seconds, 689 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 10 s, completed Jun 5, 2020 5:02:38 PM

Thanks for the PR :)

Oh also I think I understand some of our disagreement over the threads. I thought you were asking me to stop the Spark executor because I’ve started doing some separate work on that, and you were talking about the migration threads. Is my understanding correct now?

Test build #123572 has started for PR 28708 at commit 2ca6c08.

Oh also I think I understand some of our disagreement over the threads. I thought you were asking me to stop the Spark executor because I’ve started doing some separate work on that, and you were talking about the migration threads. Is my understanding correct now?

Yes :)
You can use the code from here if you agree with it:
https://github.com/apache/spark/pull/28708#discussion_r434578445

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28195/

Oh also I think I understand some of our disagreement over the threads. I thought you were asking me to stop the Spark executor because I’ve started doing some separate work on that, and you were talking about the migration threads. Is my understanding correct now?

Yes :)
You can use the code from here if you agree with it:
#28708 (comment)

I don't think we need that change, I believe setting running to false prevents the thread leak.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28195/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28195/
Test PASSed.

Oh also I think I understand some of our disagreement over the threads. I thought you were asking me to stop the Spark executor because I’ve started doing some separate work on that, and you were talking about the migration threads. Is my understanding correct now?

Yes :)
You can use the code from here if you agree with it:
#28708 (comment)

I don't think we need that change, I believe setting running to false prevents the thread leak.

If you run the test in the BlockManagerSuite then grep like this you will found the problem I have seen:
$ grep "THREAD LEAK" core/target/unit-tests.log  |grep BlockManager
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.storage.BlockManagerSuite, thread names: rpc-boss-3-1, shuffle-boss-9-1, migrate-shuffle-to-BlockManagerId(exec2, localhost, 49385, None), shuffle-boss-6-1 =====

Test build #123572 has finished for PR 28708 at commit 2ca6c08.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123572/
Test PASSed.

grep "THREAD LEAK"

So I'm not sure why it's not showing up in Jenkins:

holden@hkdesktop:~/repos/spark-website$ wget https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123572/consoleFull
--2020-06-05 13:32:19--  https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123572/consoleFull
Resolving amplab.cs.berkeley.edu (amplab.cs.berkeley.edu)... <IP_ADDRESS>
Connecting to amplab.cs.berkeley.edu (amplab.cs.berkeley.edu)|<IP_ADDRESS>|:443... connected.
 I'm not sure why it's showing up for you in sbt but not in Jenkins.

It is in the unit-tests.log on the jenkins too (this is the appender target see log4j.properties):
$ wget https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123572/artifact/core/target/unit-tests.log
$ grep "POSSIBLE THREAD LEAK.* migrate-shuffle-to-BlockManagerId" unit-tests.log                                                                                130 ↵
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.storage.BlockManagerSuite, thread names: shuffle-boss-7296-1, rpc-boss-7278-1, migrate-shuffle-to-BlockManagerId(exec2, localhost, 42830, None), shuffle-boss-7281-1, shuffle-boss-7242-1, shuffle-boss-7284-1, shuffle-boss-7287-1, rpc-boss-7260-1, shuffle-boss-7302-1, rpc-boss-7263-1, shuffle-boss-7269-1, rpc-boss-7290-1, rpc-boss-7240-1, shuffle-boss-7305-1, shuffle-boss-7253-1, shuffle-boss-7249-1, shuffle-boss-7275-1, rpc-boss-7257-1, shuffle-boss-7245-1, block-migration-thread, rpc-boss-7299-1, shuffle-boss-7272-1, rpc-boss-7266-1, shuffle-boss-7293-1 =====

Cool, I'll make the DISK_ONLY change, thanks for suggesting the cleanup :)

Test build #123644 has started for PR 28708 at commit 6a83c9d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28267/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28267/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28267/
Test PASSed.

Test build #123645 has started for PR 28708 at commit df030f6.

Test build #123644 has finished for PR 28708 at commit 6a83c9d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123644/
Test FAILed.

I suspect the date test is unrelated.

Test build #123647 has started for PR 28708 at commit dc2684d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28268/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28268/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28268/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28270/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28270/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28270/
Test PASSed.

Test build #123645 has finished for PR 28708 at commit df030f6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123645/
Test FAILed.

Test build #123647 has finished for PR 28708 at commit dc2684d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123647/
Test PASSed.

Hi @holdenk!
I have opened a new PR for your branch: https://github.com/holdenk/spark/pull/6
I know this seams to be a lot of change but it is really just moving code out from the BlockManager.
One advantage is testability: now the BlockManagerDecommissionManager can be tested quite easily without a BlockManager.
The other is now the BlockManagerDecommissionManager has very clean interface for its dependencies:

MigratableResolver to handle Shuffle Blocks
MigratableRDDResolver to handle cached RDD
BlockTransferService for handling block uploads
BlockManagerPeerProvider to find the peers where to migrate those blocks

I suggest to quickly check the commits and you will see it is not that risky and as refactors on production code is quite discouraged probably this is the best time to make these steps.

Test build #123704 has started for PR 28708 at commit 8f7d4eb.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28328/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28328/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28328/
Test PASSed.

Test build #123708 has started for PR 28708 at commit b087804.

Test build #123704 has finished for PR 28708 at commit 8f7d4eb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123704/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28333/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28333/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28333/
Test PASSed.

Test build #123708 has finished for PR 28708 at commit b087804.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BlockManagerDecommissionUnitSuite extends SparkFunSuite with Matchers

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123708/
Test FAILed.

Test build #123941 has started for PR 28708 at commit 7f00072.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28563/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28563/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28563/
Test PASSed.

Test build #123941 has finished for PR 28708 at commit 7f00072.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123941/
Test PASSed.

@holdenk, what are your thoughts on reducing the number of fetch failed exceptions (and thus potential job failures) due to decommissioning:

This PR tries to offload the shuffle blocks to another peer but that is somewhat best effort.
When the executor is eventually lost (detected much later by a heartbeat timeout), its shuffle files will be cleared. Until that happens the unmigrated blocks (for whatever reason) will result in fetch failures.

Block migration may not be possible in all cases since we may not have enough time to do the migration or the peer's disk might be full or it might soon after be decommissioned itself. This sort of "bulk decommissioning" can happen for example during a cluster scale down where many nodes are brought down in quick succession.
While block migration is great in that it avoids wasting work, we cannot guarantee it and I think we need a second line of defense by proactively calling MapOutputTracker.removeOutputsOnExecutor (and clearing DAGScheduler.cacheLocs) perhaps after some timeout. (I think FetchFailures may still happen if a reducer has already started.)
It would be nice to prevent/reduce job failures caused by fetch failures: Particularly because we already know that the executor would be yanked soon, so we should be able to do better.
Thanks.

Yeah I think supporting multiple ways of reducing the number of fetch failures makes sense here. I think migration is certainly a "best-case" scenario and we can't count on in migrating everything in overcommit environments.

Test build #123958 has started for PR 28708 at commit 0ea927d.

Although I'm a little fuzzy on what you mean by "eager" (if you mean as soon as the migrations are completed then yes)

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28580/

Test build #123960 has started for PR 28708 at commit fe34308.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28580/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28580/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28582/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28582/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28582/
Test PASSed.

Although I'm a little fuzzy on what you mean by "eager" (if you mean as soon as the migrations are completed then yes)

Thank you for confirming ! By eager, I specifically mean somehow triggering a code path that can ASAP trigger DAGScheduler#handleExecutorLost(_, workerLost = true) codepath, such that it can clear out the shuffle map files. This is more about not having fetch failures from decom as opposed to recouping resources.
One way that this is triggered today is by CoarseGrainedSchedulerBackend.DriverEndpoint#onDisconnected, but I don't really know if there is a timeout at play here. This workerLost = true bit is set only in a few cases unfortunately, so we might have to add some code (or do some testing) to achieve this.
I think https://issues.apache.org/jira/browse/SPARK-31197 is meant for this ?

Test build #123960 has finished for PR 28708 at commit fe34308.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ShuffleBlockInfo(shuffleId: Int, mapId: Long)

Test build #123964 has started for PR 28708 at commit da1db47.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123960/
Test FAILed.

Yeah so the plan is to trigger an exit as soon as migrations are completed. I think a good follow up to  https://issues.apache.org/jira/browse/SPARK-31197 would be adding a timeout so we also just exit if were likely to be killed anyways by the underlying cluster manager (k8s/yarn/etc.).
Also to make it a bit clearer I've put up two WIP PRs that build on top of this PR: https://github.com/apache/spark/pull/28818 & https://github.com/apache/spark/pull/28817 , there obviously a bit earlier on in terms of implementation/test coverage but I think they help illustrate some of the next steps.

Test build #123958 has finished for PR 28708 at commit 0ea927d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123958/
Test PASSed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28586/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28586/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28586/
Test PASSed.

Test build #123964 has finished for PR 28708 at commit da1db47.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123964/
Test FAILed.

jenkins retest this please

Test build #123981 has started for PR 28708 at commit da1db47.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28604/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28604/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28604/
Test FAILed.

Test build #123981 has finished for PR 28708 at commit da1db47.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123981/
Test PASSed.

The K8s integration test failure is irrelevant to this PR.
- Run SparkPi with no resources *** FAILED ***

Sure I'll share the design doc here as well, https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE

Hey @gatorsmile I noticed you marked this as "request changes", can you clarify what changes you are requesting?

Test build #124075 has started for PR 28708 at commit d9123f9.

Test build #124076 has started for PR 28708 at commit 6a77220.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28694/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28695/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28694/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28694/
Test FAILed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28695/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28695/
Test FAILed.

@holdenk The PR description is not clear. The current PR description does not have the first section in the template "What changes were proposed in this pull request?" . Could you please update it and explain how you did it?

Test build #124075 has finished for PR 28708 at commit d9123f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124075/
Test PASSed.

Test build #124076 has finished for PR 28708 at commit 6a77220.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124076/
Test PASSed.

I've updated the description @gatorsmile, let me know if there is any particular points you would like clarified.

Test build #124136 has started for PR 28708 at commit 56c5538.

Test build #124136 has finished for PR 28708 at commit 56c5538.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124136/
Test FAILed.

Build dependency failure is likely unrelated given the lack of any pom changes.

Test build #124137 has started for PR 28708 at commit 7da5130.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28753/

Test build #124137 has finished for PR 28708 at commit 7da5130.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124137/
Test FAILed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28753/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28753/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28754/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/28754/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28754/
Test FAILed.


Execution Timeout Milliseconds

Hello,
We have been using the timeout branch to test some of our guest function execution timeout code. All is well but we would  like the ability to limit by milliseconds as well. Would something like this be possible? Seconds at the moment is a bit limiting

@AshantiM We have no branch named timeout or similar names. Could you please share with us the link to it or your code if you don't mind? Thanks a lot!

Oh sorry thought I replied to this. This was for run_func_with_timeout on the VM

@AshantiM Hi, actually we've already done.
https://github.com/WasmEdge/wasmedge-rust-sdk/pull/76

@AshantiM Please help close the issue if it has been fixed. Thanks a lot!


v1.13 doesn't refresh view after submitting annotations in Safari - hangs view with loading symbol

Describe the bug
Who digitising in Safari, on clicking submit/update after adding labels in a segmentation project type the loading symbol is shown and the page doesn't update. Refreshing the page manually and navigating back to the same image reveals the labels were saved, but this is a large hassle.
To Reproduce
Steps to reproduce the behavior:
I've upgraded from label studio 1.12 to 1.13 on a Windows 10 with Python 3.12 installation using pip install -U label-studio.
I then added new images to an existing library.
I'm accessing label-studio in a web browser (Safari) from a Mac when labelling.
When I finish labelling and click submit/update the view in the web browser shows the spinning loading symbol and does not refresh the page.
This was not the experience when using 1.12.
Expected behavior
On clicking submit/update after adding a label the page should update either back to the list or a confirmation.
Environment (please complete the following information):
Label studio hosted on Windows 10 Pro on main release channel.
Digitising on macOS via Safari - up to date on main release channel.
Edit: Word in Chrome from the same machine (though Safari goes through far less battery than Chrome, so would prefer to use Safari)

Hi @sxwebster - unfortunately we don't officially support Safari: https://labelstud.io/guide/install.html#Web-browser-support - we'd welcome a PR (or additional information) if you're able to pinpoint the cause, though!


traj_1

Description: A simple trajectory plot. Each trajectory is a different color for clarity, and every fourth time step is marked with a circle. The start of each trajectory is marked with a green circle.
Script URL: http://ncl.ucar.edu/Applications/Scripts/traj_1.ncl
Plot URL: http://ncl.ucar.edu/Applications/Images/traj_1_lg.png

should this go in the XY plot folder or should a new trajectory plot folder be made? @kmpaul

I would make a new "Trajectories" folder

The directory structure should mirror the NCL structure seen here: https://www.ncl.ucar.edu/Applications/list_ptypes.shtml

The linked PR has been merged and closed, but this issue has remained open erroneously.


Add ability to exclude files from "quick open" (cmd-p)

Similar to the way you can exclude files from search (search.exclude) and workspace (files.exclude), it would be really nice to be able to exclude files from quick open (cmd-p).
My use case is that I have files like I like to be able to access rarely from the explorer pane (configuration and generated code), but I don't ever want them to show up in the quick open list.

search.exclude should exclude items from quick open results. If you are still seeing results from an excluded pattern, they could be coming from the editor history. #6502 tracks applying search.exclude to those patterns as well.
Try quick opening a file in an excluded folder that you have never opened before. Does it still show up?

I think this  might be a duplicate #6502...
It seems like once I open the item it starts showing up in my "Quick Open", even if I restart VSCODE.
This is not ideal (and actually quite an issue, because some of the file names are the same as ones I actually want in my workspace).

Thanks @ChuckJonas


Policy add deny rule test and benchmark

[ ] #33313 (@jrajahalme)

This is a partial backport of #33313, backporting only the new test and benchmark. This is being backported for comparison purposes between 1.16 and the forthcoming 1.17.
Add test and benchmark with a mix of deny and allow CIDR rules.
Add identity for each CIDR so that 'toMapState' has some work to do.
Run the new benchmark like this:
$ go test ./pkg/policy/... -bench BenchmarkRegenerateCIDRDenyPolicyRules -run ^$

Note: To run just the matching benchmark and run no tests, -run ^$ is used as ^$ is a regular expression that does not match any tests.

/test

/test

/ci-integration

/test

/ci-verifier

/ci-runtime

/ci-multi-pool

/ci-ingress

/ci-ipsec-e2e

/ci-ginkgo

/ci-external-workloads

/ci-eks

/ci-clustermesh

/ci-awscni

/ci-l4lb

/ci-ipsec-upgrade

/ci-e2e-upgrade