kicad: 5.1.2 -> 5.1.4

Motivation for this change
Update kicad and provide a partial fix for the ngspice issue.
With this the ngspice gui in eeschema Tools -> Simulation will open.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @berce

@GrahamcOfBorg build kicad

Fails on aarch64.
collect2: error: ld returned 1 exit status
[7904/7904] Linking CXX executable test/OCAFExport_test/OCAFExport_test
FAILED: test/OCAFExport_test/OCAFExport_test
: && /nix/store/bqaxflzgb6i1br71ilinf730lqc7wf2h-gcc-wrapper-8.3.0/bin/g++  -O3 -DNDEBUG  -rdynamic test/OCAFExport_test/CMakeFiles/OCAFExport_test.dir/OCAFExport_test.cpp.o  -o test/OCAFExport_test/OCAFExport_test  Unix/aarch64-Release-64/libFWOSPlugin.so.11.0.0 Unix/aarch64-Release-64/libTKBin.so.11.0.0 Unix/aarch64-Release-64/libgtest.a Unix/aarch64-Release-64/libTKCAF.so.11.0.0 Unix/aarch64-Release-64/libTKBO.so.11.0.0 Unix/aarch64-Release-64/libTKPrim.so.11.0.0 Unix/aarch64-Release-64/libTKV3d.so.11.0.0 Unix/aarch64-Release-64/libTKService.so.11.0.0 -lSM -lICE -lX11 -lfreeimage -lfontconfig Unix/aarch64-Release-64/libTKShHealing.so.11.0.0 -lGL -lGLU -lfreetype Unix/aarch64-Release-64/libTKMesh.so.11.0.0 Unix/aarch64-Release-64/libTKHLR.so.11.0.0 Unix/aarch64-Release-64/libTKTopAlgo.so.11.0.0 Unix/aarch64-Release-64/libTKGeomAlgo.so.11.0.0 Unix/aarch64-Release-64/libTKBRep.so.11.0.0 Unix/aarch64-Release-64/libTKGeomBase.so.11.0.0 Unix/aarch64-Release-64/libTKG3d.so.11.0.0 Unix/aarch64-Release-64/libTKG2d.so.11.0.0 Unix/aarch64-Release-64/libTKMath.so.11.0.0 Unix/aarch64-Release-64/libTKBinL.so.11.0.0 Unix/aarch64-Release-64/libTKLCAF.so.11.0.0 Unix/aarch64-Release-64/libTKCDF.so.11.0.0 Unix/aarch64-Release-64/libTKernel.so.11.0.0 -ldl -lm -lpthread && :
/nix/store/fm9jqjh7qgamlb2pllm6vaw7kgannx3s-binutils-2.31.1/bin/ld: /nix/store/19x7dfkrszzp540hi9ighhrx652h16x3-freeimage-3.18.0/lib/libfreeimage.so: undefined reference to `png_init_filter_functions_neon'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
builder for '/nix/store/9kifcs5r29rkr8gxckn8a5ks3g2wxwd7-opencascade-oce-0.18.3.drv' failed with exit code 1
cannot build derivation '/nix/store/2dvpbqnhpwzxjva65z3fqwgdlflylf18-kicad-5.1.4.drv': 1 dependencies couldn't be built
error: build of '/nix/store/2dvpbqnhpwzxjva65z3fqwgdlflylf18-kicad-5.1.4.drv' failed```

let's see if that fixes the aarch64 build.
@GrahamcOfBorg build kicad

another try....
@GrahamcOfBorg build kicad

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/kicad-version-update/4222/1

With the last commit I've successfully run a very basic simulation, so the ngspice simulation is now working.
Thanks to SnVIZQ for the patch.
The only remaining issue is the aarch64 build. I'll make a quick test if the old version had a successful build.
@GrahamcOfBorg build kicad

The aarch64 build has been broken for around 3 months now: https://hydra.nixos.org/job/nixpkgs/trunk/kicad.aarch64-linux/all?page=1
Perhaps it would be okay to just mark the package with broken = stdenv.isAarch64. If you want to fix it on aarch64 that would be great too. Perhaps you could try applying for https://github.com/nix-community/aarch64-build-box .

@veprbl I have no idea what the issue with this neon error is, so I've marked it as broken for aarch64.
@GrahamcOfBorg build kicad

Assuming that ofborg successfully builds kicad on x86_64, I believe this PR is now ready for merging.

I've picked these commits now, they allow me to run a very basic simulation, update kicad and disable the aarch64 build.
pick 06516fe13ad kicad: 5.1.2 -> 5.1.4
pick 0f39c86bdc7 kicad: mark is broken on aarch64 due to some neon issue.
pick 95a7a83df70 kicad: partial fix for the ngspice segfault

@GrahamcOfBorg build kicad

Hopefully this PR is now really done.

I didn't know about kicad-with-packages3d before, so that hopefully fixes everything now.

@tokudan Thank you!


[FR] Clone a simulator

This would be an easy thing to add in the contextual menu
simctl clone [device id] [new name]

Hi, thanks for the great development tool application.
This idea would be quite so helpful.
And Let me put my additional idea.
I'd like to get a function that can copy whole data on the simulator to the other device.
This would be helpful to check the UI behavior on the different screen sizes.
whole data means data directory.

Currently working on this and #34 at the same time

That looks perfect @davedelong 😄


Adding support for variance of the number operator

Context:
A recent addition in Strawberry Fields (https://github.com/XanaduAI/strawberryfields/pull/407) allowed support for extracting variances of the number operator for state objects. It also allowed for specifying more than 2 modes when the number expectation is returned using the BaseGaussianState.
With this change, the state.number_expectation method returns a tuple of (expval, var). For compatibility reasons, an adjustment was needed in #39, which also enabled calculating variances of the number operator. Adding complete support for this feature, however, was out of scope for #39.
Description of the Change:
Adds test cases for features allowed by the changes to the SF BaseGaussianState class which include cases for:

returning variances of the number operator and
specifying more than 2 modes when returning the expectation value of the number operator using the Gaussian backend.

Benefits:
The new features from SF are tested in the plugin.
Possible Drawbacks:
N/A
Related GitHub Issues:
N/A

@co9olguy @josh146 thank you for the comments!

the tests are now included in a test_shared.py
broke up the displaced squeezed test case into smaller ones
adjusted the SF reference function for better readability


refactor: remove isSupported util

it always returns true

Codecov Report

Merging #18 (e0bf7f6) into develop (73e1c44) will increase coverage by 0.27%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop      #18      +/-   ##
===========================================
+ Coverage    73.79%   74.06%   +0.27%
===========================================
  Files           42       42
  Lines         1240     1230      -10
  Branches       328      323       -5
===========================================
- Hits           915      911       -4
+ Misses         237      235       -2
+ Partials        88       84       -4

Impacted Files
Coverage Δ

src/charts/BaseChart.ts
45.61% <ø> (-2.72%)
:arrow_down:

src/svg/Svg.ts
67.77% <ø> (-0.71%)
:arrow_down:

src/axes/Axis.ts
94.73% <100.00%> (+9.37%)
:arrow_up:

src/charts/BarChart/BarChart.ts
64.86% <100.00%> (ø)

src/charts/LineChart/LineChart.ts
73.03% <100.00%> (ø)

src/core/creation.ts
93.93% <100.00%> (+2.76%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 73e1c44...e0bf7f6. Read the comment docs.


[Freature Request] Add ability to import and export Prompt Library (.csv format)

I would like to suggest adding the ability to import and export the Prompt Library, starting with the CSV format. This would allow users to import, save, and share lists such as the one available at https://github.com/f/awesome-chatgpt-prompts/blob/main/prompts.csv. (EDIT: looking at the Awesome prompt.csv, its full of non-CSV related code. Perhaps not the best example. Perhaps, It could be cleaned up and made available through FreeChatGPT project as a GitHub download).
By adding the CSV feature, users would be able to create a spreadsheet of their favorite prompts, save as CSV, and then import them into FreeChatgpt. This feature would also make it easier for users to edit their Prompt Library by allowing them to import the CSV file into a spreadsheet editor.
An advanced feature could be added to allow users to select all or specific prompts they want to import or export.  Perhaps this could be implemented through a separate dialogue box, where the user can check what they want to export or import.
PS: If it hasn't been done yet, the Prompts Library should be added to the JSON export/backup. Thanks!

What do you mean by "Awesome prompt.csv is full of non-CSV related code"?

My mistake. Somehow the way/where I downloaded it added a bunch of HTML code.  Looking at the raw version (https://raw.githubusercontent.com/f/awesome-chatgpt-prompts/main/prompts.csv), it looks like good ol' CSV to me.


Use cp-kafka.fullname for Kafka broker nodeport-service name

What changes were proposed in this pull request?
Use the cp-kafka.fullname helper to construct the name of the nodeport-service of individual Brokers.
This way, the names will more closely match the other services created by the cp-kafka chart:

How was this patch tested?
Forked chart, running on a project running in AWS EKS v1.11.8.

@confluentinc It looks like @mandrean just signed our Contributor License Agreement. :+1:
Always at your service,
clabot

@qshao-pivotal Thanks! So... Can we merge? 😄


running uv run pytest on laptop breaks backend python enviroment in developer mode

Describe the bug
In developer mode, the lumigator folder is mounted on the backend container. The fastapi server automatically reloads whenever any change in the backend folder is detected, so developers can update code in their IDE and see changes reflected immediately on the backend server.
Since the introduction of uv, the backend python environment is stored in /mzai/lumigator/python/mzai/backend/.venv and is generated automatically when the backend is deployed with docker compose via the make local-up command. Things work properly until the environment is updated locally (e.g. on laptop) via e.g. uv sync or uv run pytest: when the command is run, the previous .venv (which points to a python interpreter which does not exist on laptop) is rebuilt and thus updated on the backend too. At that point, fastapi reloads the server but then -probably because it finds a .venv which is now unsuitable for the backend- shuts down the server.
System Information (please complete the following information):

OS: macOS Sonoma
Lumigator version: 543a3a1

To Reproduce
> cd /tmp
> git clone https://github.com/mozilla-ai/lumigator.git
> cd lumigator
> ll lumigator/python/mzai/backend/.venv

ls: lumigator/python/mzai/backend/.venv: No such file or directory

> make local-up
[...]
 => [backend stage-0 5/5] RUN uv sync --frozen
 => => # Using CPython 3.11.10 interpreter at: /usr/local/bin/python
 => => # Creating virtual environment at: .venv
[... everything's up and running]

> test_dataset_upload

Connecting to http://localhost:8000...
{
  "id": "b80c3a52-8e64-4eec-b11e-f858697f061a",
  "filename": "thunderbird_gt_bart.csv",
  "format": "experiment",
  "size": 180528,
  "ground_truth": true,
  "created_at": "2024-10-24T10:32:51"
}
[the API is there, things work properly]

> cd lumigator/python/mzai/backend/
> ls -la
...
drwxr-xr-x   9 mala  wheel     288 24 Oct 11:31 .venv
...

[note that below here I am running tests without specifying the SQLALCHEMY_DATABASE_URL. It does not matter whether tests actually run to completion (one could as well run uv sync here), it's just the uv venv setup that breaks stuff]
> uv run pytest

warning: Ignoring existing virtual environment linked to non-existent Python interpreter: .venv/bin/python3 -> python
Using CPython 3.11.5 interpreter at: /Users/mala/miniconda3/bin/python
Removed virtual environment at: .venv
Creating virtual environment at: .venv
   Built schemas @ file:///private/tmp/lumigator/lumigator/python/mzai/schemas
Installed 103 packages in 282ms

[in the meantime on the backend:]

2024-10-24 11:35:28 WARNING:  WatchFiles detected changes in '.venv/lib/python3.11/site-packages...
[... every single file in .venv]
Reloading...
2024-10-24 11:35:28 INFO:     Shutting down
2024-10-24 11:35:28 INFO:     Waiting for application shutdown.
2024-10-24 11:35:28 INFO:     Application shutdown complete.
2024-10-24 11:35:28 INFO:     Finished server process [32]

[back to my laptop]

> ls -la
...
drwxr-xr-x@  8 mala  wheel     256 24 Oct 11:35 .venv
...
[venv has been updated]

> test_dataset_upload

Connecting to http://localhost:8000...
[hangs because the server is down]

Expected behavior
One should be able to run tests without breaking the backend or, more precisely, i.e. without updating the .venv that is used in the backend. This has another advantage, i.e. that by preserving the backend venv in place we wouldn't have to re-download dependencies every time we do make local-up.
This could be attained by running pytest in an isolated environment:
SQLALCHEMY_DATABASE_URL=sqlite:///local.db uv run --isolated pytest
I think updating documentation making the dependency between local and backend environments for developers more explicit and adding the above command for tests would already be a reasonable fix.

Hi there @aittalam , do you think we can close this one after https://github.com/mozilla-ai/lumigator/pull/323?

Yes @agpituk , thanks! There is a follow-up to this I have tracked in https://github.com/mozilla-ai/lumigator/issues/375


Fix build error for Android projects that use apply plugin: "com.android.library"

This PR allows Android projects that use apply plugin: "com.android.library" to build successfully. A recent regression caused by https://github.com/facebook/react-native/pull/20526 means that building one of these projects always fails with this error:
FAILURE: Build failed with an exception.

* Where:
Script '/Users/james/src/SampleLibrary/node_modules/react-native/react.gradle' line: 15

* What went wrong:
A problem occurred configuring project ':app'.
> Could not get unknown property 'applicationVariants' for object of type com.android.build.gradle.LibraryExtension.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s

This change updates react.gradle to cater to both application and library projects by selectively using android.applicationVariants or android.libraryVariants.
Fixes https://github.com/facebook/react-native/issues/22310.
Test Plan:
Build any Android project with apply plugin: "com.android.library" using ./gradlew build (I have a sample project here: https://github.com/jtreanor/sample-library-react-native). It will now succeed with these changes.
Changelog:
[Android] [Fixed] - Fixed build error with Android library projects

Thanks for the PR. This looks good to me.


Recommend way to share timeouts and intervals between the Supervisor and the Site

Spec 3.1.5. states the following on Watchdog section:
The interval duration for sending watchdog messages should be configurable at both the site and the supervision system. The default setting should be (1) once a minute.
That implies to me that there should be a way to convey Watchdog interval from Supervisor to the site. I couldn't find anything regarding how to perform this in the spec.
Ideally both sides should use the same interval otherwise one end may end up with Watchdog timeout for example.
This raises a general question:
What is the recommended way to share these kind of timeouts and intervals between the Supervisor and  the Site? A shared file (YAML or JSON) seems to be the straightforward solution but it is static and requires reboots when changed.
Any other suggestions?

Hi,
There is no build-in RSMP method for sharing this kind of information between sites and supervisors.
There are other things that must have matching configurations on sites and supervisor, like component ids. Making this smarter is on the roadmap, see e.g. #13 for one example.
Sharing config data manaually between sites and supervisor can really be done any way you find convenient. Usually there is no need to share files, all you do is manually edit config files on both sites to match e.g. watchdog interval and timeout.

Thanks for the feedback Erin.


Provide error message when user tries to upload duplicate files

I am trying to upload duplicate files to Wikimedia Commons, and I am indeed not succeeding in doing so, which is great. I am trying to upload files that already exist on Commons, with the exact same file name, from this project: (only the two first, flagged rows, using the filename2 column)
Jan-Geerling-photographs-Rotterdam-1503.openrefine (2).tar.gz
The upload process is stuck at 0% but the user doesn't get any feedback or error messages, and I think it would be good to have these; otherwise the user is left in the dark about what they may be doing wrong.

For upload-time error messages, we still need #4236 / #5032. We have sadly not got the infrastructure for this yet.
My hope is that we can do those checks earlier, as part of the Issues tab, so that this is reported even before the upload. Arguably that is even a better UX than ending up with a partially failed batch of uploads that you are not too sure how to fix afterwards (asking for deletion and uploading again is quite cumbersome).


When the cluster name is error,  It should be clear indicate what the error is

Describe the Bug
The cluster name contains spaces It should be clear indicate what the error is

Versions Used
Kubernetes:
Kubeclipper:
Environment
How To Reproduce
Steps to reproduce the behavior:
Expected behavior
It should be clear indicate what the error is

/kind optimize

/remove-kind bug

/milestone v1.2.0


about libraries

Which version of tensorflow is used in Ada-boundary?

Tensorflow 2.X can be used to run this code. Thanks.


*TVAL should capture the effective address on an exception

This is a clarification that needs to be added based on the following email thread:
https://lists.riscv.org/g/tech-privileged/topic/94419723
(I keep forgetting to update the spec, so capturing this change for posterity)

I'm not sure that the clarification proposed here is the right thing to do. The question being asked is:

When a breakpoint, access-fault, or page-fault exception occur on a CMO instruction (cbo.clean, cbo.flush, cbo.inval, cbo.zero), is the full virtual address provided in the register identified by rs1 saved into xtval or can a block-size aligned version of the VA be used?

The response (in the email thread) is:

The original intent (which apparently was not captured in the spec) was that the effective address would be captured in *tval, just like a normal load or store.
I will add a clarification to the extension specification.

Normal loads and stores do not necessarily capture the given effective address in the tval register. For example, if misaligned addresses are supported and a misaligned access is made that straddles page boundaries where the first page allows access but the second does not then the address saved in tval will be the base address of the second page, not the input effective address. Specifically, the Privileged Specification says this:

If mtval is written with a nonzero value when a misaligned load or store causes an access-fault or page-fault exception, then mtval will contain the virtual address of the portion of the access that caused the fault.

To me it feels that it would be more consistent to report the lowest failing address in tval, which would be the block-size aligned VA.
I'm also wondering about the interaction of this feature with PMP, which allows fine-grained protections to be specified. For example, suppose that PMP regions are configured so that an entire cache line has access allowed by a low-priority region except for four bytes in the middle that have access denied by a high-priority region. What happens if cbo.zero is used on that line? The PMP specification says:

The matching PMP entry must match all bytes of an access, or the access fails, irrespective of the L, R, W, and X bits.

Does this mean that the cbo.zero must fail without updating state if the line which it writes is not contained in a single PMP entry?
Thanks.

Note that the misaligned case that can occur for loads and stores, cannot
occur for any of the CMO instructions.  What David described applies for
aligned ld/st's and for the first part of misaligned ld/st's, and hence is
appropriate also for CMOs.
Regarding the PMP question, Yes.  Just like for a ld/st that encountered
the same situation.  Further, there are no defined architecture semantics
that allow for only partially executing RV instructions (until one talks
about vector ld/st's).
Greg
On Thu, Apr 13, 2023, 4:28 PM JamesKenneyImperas @.***>
wrote:

I'm not sure that the clarification proposed here is the right thing to
do. The question being asked is:
When a breakpoint, access-fault, or page-fault exception occur on a CMO
instruction (cbo.clean, cbo.flush, cbo.inval, cbo.zero), is the full
virtual address provided in the register identified by rs1 saved into
xtval or can a block-size aligned version of the VA be used?
The response (in the email thread) is:
The original intent (which apparently was not captured in the spec) was
that the effective address would be captured in *tval, just like a normal
load or store.
I will add a clarification to the extension specification.
Normal loads and stores do not necessarily capture the given effective
address in the tval register. For example, if misaligned addresses are
supported and a misaligned access is made that straddles page boundaries
where the first page allows access but the second does not then the address
saved in tval will be the base address of the second page, not the input
effective address. Specifically, the Privileged Specification says this:
If mtval is written with a nonzero value when a misaligned load or store
causes an access-fault or page-fault exception, then mtval will contain
the virtual address of the portion of the access that caused the fault.
To me it feels that it would be more consistent to report the lowest
failing address in tval, which would be the block-size aligned VA.
I'm also wondering about the interaction of this feature with PMP, which
allows fine-grained protections to be specified. For example, suppose that
PMP regions are configured so that an entire cache line has access allowed
by a low-priority region except for four bytes in the middle that have
access denied by a high-priority region. What happens if cbo.zero is used
on that line? The PMP specification says:
The matching PMP entry must match all bytes of an access, or the access
fails, irrespective of the L, R, W, and X bits.
Does this mean that the cbo.zero must fail without updating state if the
line which it writes is not contained in a single PMP entry?
Thanks.
—
Reply to this email directly, view it on GitHub
https://github.com/riscv/riscv-CMOs/issues/55#issuecomment-1506561126,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALLX6GVL45CTH2BFXUTSPBTXA62MPANCNFSM6AAAAAATQ7VVTA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Ok. If Hypervisor mode is implemented and the CMO ends up causing a stage 2 fault, presumably the address reported there has the same block offset as well.
Thanks.

Hello,
I want to understand the desired tval when a cache-block management instruction raises an exception.
Should the tval be the original address or the lowest effecitve virtual address on the cache block boundary?
https://github.com/riscv-software-src/riscv-isa-sim/pull/1376
Best regards,
YenHaoChen

IMO, there's nothing terribly problematic with either reporting the base address or reporting the effective address.  Since this was omitted from the ratified spec, is it not reasonable to allow either behavior (or really any address on the faulting cache line)?
@gfavor is it possible that "What David described applies for aligned ld/st's" is standard practice rather than something mandated?  Just wondering if we need a clarification there.  I suppose the spec says tval contains "the" faulting address, not "a" faulting address, which is justification enough.

As mentioned above (https://github.com/riscv/riscv-CMOs/issues/55#issuecomment-1506561126):

The original intent (which apparently was not captured in the spec) was that the effective address would be captured in *tval, just like a normal load or store.
I will add a clarification to the extension specification.

So the intent (not just a suggestion of common practice) is that the effective address is captured - the same as for other exceptions (modulo misalignment cases which don't apply to cbo's).
One can also note that the standard meaning of "the faulting address" in the RV Priv arch is a specific address (i.e. the effective address except when misalignment cases come into the picture), and not just any address covered by the access (or part of the overall access) being faulted.

Yeah, that works for me.

And finally tieing back to the beginning of this thread (to net out a bunch of this thread), it would be helpful to readers to add a non-normative comment about the recorded *tval address simply and always being the effective address - the same as what is captured in *tval for loads and stores (barring misalignment cases - which don't exist for cbo's).

Yup.  That would clear things up in an uncontroversial fashion.


Refactor Customer to Receiver

The codebase and API utilize the ACH term Originator for the company or individual that are generating the credit/debit transaction against a "Customer". Using ACH terminology the "Customer" should be called a "Receiver" throughout the code base and API.
The background is that "Originator" was previously labelled "Account" which had little to no meaning besides an Account had Customers. An Originator has a Receiver.

I've got this setup after the file transfer branches: https://github.com/adamdecaf/paygate/compare/file-transfer-controller...adamdecaf:rename-customer-to-receiver?expand=1

We will need to update the OpenAPI specification as well
https://api.moov.io/#tag/Customers

Done! https://api.moov.io/#tag/Receivers


Add introduction to Elixir video in the Newsroom page

Purpose
Add introduction to Elixir video in the Newsroom page
The purpose of this PR is to fix #606
Goals
Add video
Approach
Add video
Screenshots

@Tharindu37 can you add the following line to https://github.com/LSFLK/MedicinesforLK/blob/development/donor-portal/src/react-app-env.d.ts to this file and try rebuilding the project?
declare module "*.mp4";

Let's centre the video and add some padding.


[SPARK-46512][CORE] Optimize shuffle reading when both sort and combine are used.

What changes were proposed in this pull request?
After the shuffle reader obtains the block, it will first perform a combine operation, and then perform a sort operation. It is known that both combine and sort may generate temporary files, so the performance may be poor when both sort and combine are used. In fact, combine operations can be performed during the sort process, and we can avoid the combine spill file.
See  https://issues.apache.org/jira/browse/SPARK-46512 for details
Why are the changes needed?
Reduce the number of spills to disk when both sort and combine are used
Does this PR introduce any user-facing change?
No
How was this patch tested?
spark examples run in local and yarn mode.
Was this patch authored or co-authored using generative AI tooling?
No

@Ngone51
Thanks for you review! The combine is the key problem.
I still combine, but combine in ExternalSorter will never trigger extra spill. Combine happen in insertAllAndUpdateMetrics and merge:
(1) combine when insert, just read the exist key and combine the value, do it in memory.
(2) combine when merge, we sort the combined iterator, combine util next key occur, also do it in memory.
The two processes are not additional operations and are originally performed in the ExternalSorter.
Before the change, We use ExternalAppendOnlyMap to combine. If the memory is over threshold, will spill to disk. Then we use ExternalSorter to sort. If the memory is over threshold, will spill to disk. It means we may spill twice(ExternalAppendOnlyMap and ExternalSorter), if shuffle data is huge.
We know that we can combine during the sorting process.
After the change, we will use ExternalSorter, combine occurs during insert and merge processes. So there is no need to call ExternalAppendOnlyMap to combine. It means only spill once (ExternalSorter).
Spill is time consuming. For sort and combine shuffle, when shuffle date is huge, this PR will save time。

but combine in ExternalSorter will never trigger extra spill.

It could be triggered by ExternalSorter#maybeSpillCollection?

It looks like ExternalAppendOnlyMap uses the key for combine:
dep.aggregator.get.combineCombinersByKey(combinedKeyValuesIterator, context)

but ExternalSorter uses partition id:
map.changeValue((actualPartitioner.getPartition(kv._1), kv._1), update)

Do you think this would lead to the different result?

but combine in ExternalSorter will never trigger extra spill.

It could be triggered by ExternalSorter#maybeSpillCollection?

@Ngone51 Thanks for your reply!
Yes, It could be triggered by ExternalSorter#maybeSpillCollection. I mean that the Extra spill is trigger by ExternalAppendOnlyMap#spill
Before the change, bothExternalAppendOnlyMap#spill and ExternalSorter#maybeSpillCollection will be spill. After the change, only ExternalSorter#maybeSpillCollection will be spill.
I created a huge amount of data, and I found the following logs:
23/12/26 16:55:52 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 522.3 MiB to disk (1 time so far)
23/12/26 16:56:09 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 629.8 MiB to disk (2 times so far)
23/12/26 16:56:40 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 491.5 MiB to disk (3 times so far)
23/12/26 16:56:58 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 553.0 MiB to disk (4 times so far)
23/12/26 16:57:16 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 442.9 MiB to disk (5 times so far)
23/12/26 16:57:47 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 576.1 MiB to disk (6 times so far)
23/12/26 16:58:04 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 540.2 MiB to disk (7 times so far)
23/12/26 16:58:22 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 686.3 MiB to disk (8 times so far)
23/12/26 16:58:52 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 540.2 MiB to disk (9 times so far)
23/12/26 17:00:06 INFO ExternalAppendOnlyMap: Task 5 force spilling in-memory map to disk and it will release 942.2 MiB memory
23/12/26 17:00:27 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (1 time so far)
23/12/26 17:01:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (2 times so far)
23/12/26 17:02:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (3 times so far)
23/12/26 17:03:23 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (4 times so far)
23/12/26 17:04:30 INFO ExternalSorter: Thread 48 spilling in-memory map of 948.8 MiB to disk (5 times so far)
23/12/26 17:05:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 943.7 MiB to disk (6 times so far)
23/12/26 17:06:25 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (7 times so far)
23/12/26 17:07:08 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (8 times so far)
23/12/26 17:07:58 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (9 times so far)

After the change, we will only found ExternalSorter spill log.

It looks like ExternalAppendOnlyMap uses the key for combine:
dep.aggregator.get.combineCombinersByKey(combinedKeyValuesIterator, context)

but ExternalSorter uses partition id:
map.changeValue((actualPartitioner.getPartition(kv._1), kv._1), update)

Do you think this would lead to the different result?

@Ngone51
It seems that the key for combine is Tuple2(partitionid, key).

I see @zhengchenyu

Thanks for looking into this @Ngone51 !
Will take a look over this PR later this week/next - sorry for the delay, was away on a vacation.
+CC @otterc as well.

Can you update to latest @zhengchenyu ? That should hopefully fix the build.
Will give a couple of days more for folks to review the patch in meantime.

Merged to master.
Thanks for adding this @zhengchenyu !

but combine in ExternalSorter will never trigger extra spill.

It could be triggered by ExternalSorter#maybeSpillCollection?

@Ngone51 Thanks for your reply!
Yes, It could be triggered by ExternalSorter#maybeSpillCollection. I mean that the Extra spill is trigger by ExternalAppendOnlyMap#spill
Before the change, bothExternalAppendOnlyMap#spill and ExternalSorter#maybeSpillCollection will be spill. After the change, only ExternalSorter#maybeSpillCollection will be spill.
I created a huge amount of data, and I found the following logs:
23/12/26 16:55:52 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 522.3 MiB to disk (1 time so far)
23/12/26 16:56:09 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 629.8 MiB to disk (2 times so far)
23/12/26 16:56:40 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 491.5 MiB to disk (3 times so far)
23/12/26 16:56:58 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 553.0 MiB to disk (4 times so far)
23/12/26 16:57:16 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 442.9 MiB to disk (5 times so far)
23/12/26 16:57:47 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 576.1 MiB to disk (6 times so far)
23/12/26 16:58:04 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 540.2 MiB to disk (7 times so far)
23/12/26 16:58:22 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 686.3 MiB to disk (8 times so far)
23/12/26 16:58:52 INFO ExternalAppendOnlyMap: Thread 48 spilling in-memory map of 540.2 MiB to disk (9 times so far)
23/12/26 17:00:06 INFO ExternalAppendOnlyMap: Task 5 force spilling in-memory map to disk and it will release 942.2 MiB memory
23/12/26 17:00:27 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (1 time so far)
23/12/26 17:01:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (2 times so far)
23/12/26 17:02:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (3 times so far)
23/12/26 17:03:23 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (4 times so far)
23/12/26 17:04:30 INFO ExternalSorter: Thread 48 spilling in-memory map of 948.8 MiB to disk (5 times so far)
23/12/26 17:05:24 INFO ExternalSorter: Thread 48 spilling in-memory map of 943.7 MiB to disk (6 times so far)
23/12/26 17:06:25 INFO ExternalSorter: Thread 48 spilling in-memory map of 954.0 MiB to disk (7 times so far)
23/12/26 17:07:08 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (8 times so far)
23/12/26 17:07:58 INFO ExternalSorter: Thread 48 spilling in-memory map of 959.1 MiB to disk (9 times so far)

After the change, we will only found ExternalSorter spill log.

Hi @zhengchenyu , curious is the total number of spilling differ before and after this PR? Thanks!

@waitinfuture
(1) performance differs
Experimental data were recorded in https://issues.apache.org/jira/browse/SPARK-46512. How much performance improvement depends on the experimental environment. In my experiments, the correlation time was reduced from 75 seconds to 29 seconds.
(2) total number of spilling differs
The reduction in the number of spills is obvious.
Before this PR, we combine the unsorted records, then sort. When we combine the unsorted records, we us ExternalAppendOnlyMap. They may spill for large data. Then when we sort, we still spill for large data.
After this PR, when we sort, we can easily organize the same keys together, and then we no longer have to use ExternalAppendOnlyMap to combine.
After this PR, we will save the spilled process which we must do in combine.

@waitinfuture (1) performance differs Experimental data were recorded in https://issues.apache.org/jira/browse/SPARK-46512. How much performance improvement depends on the experimental environment. In my experiments, the correlation time was reduced from 75 seconds to 29 seconds.
(2) total number of spilling differs The reduction in the number of spills is obvious. Before this PR, we combine the unsorted records, then sort. When we combine the unsorted records, we us ExternalAppendOnlyMap. They may spill for large data. Then when we sort, we still spill for large data. After this PR, when we sort, we can easily organize the same keys together, and then we no longer have to use ExternalAppendOnlyMap to combine. After this PR, we will save the spilled process which we must do in combine.

Got it, thanks!


Adapt evm-bully for aurora-engine CI

Couple of things are done here:

Changing cwd by os.Chdir is not necessary, I altered git/make to do the same thing by setting cmd.Dir instead
Refactored replayer/neard quite a lot, to support more flexibility for different usage scenarios
Neard binary and aurora-cli paths can now be specified by argument
Thousands of empty blocks do not spam logs anymore (combined in ranges)

Looks good, thanks!


pipe-viewer doesn't know it's up to date

Any time I attempt to watch a video, I get the following:
WWW::PipeViewer version v0.1.4 required--this is only version v0.1.3 at /usr/bin/pipe-viewer line 49.
BEGIN failed--compilation aborted at /usr/bin/pipe-viewer line 49.

I am running the latest version from the AUR -git package.
pacman -Qi pipe-viewer-git
Name            : pipe-viewer-git
Version         : 0.1.4.r0.gc999250-1

Maybe you installed pipe-viewer manually before?
Check to see if the following file exists:
/usr/share/perl5/site_perl/WWW/PipeViewer.pm

If it does exist, then remove it, along with the following directory:
/usr/share/perl5/site_perl/WWW/PipeViewer

The version from the AUR is installed under /usr/share/perl5/vendor_perl/WWW/PipeViewer.

It was there....should I rebuild my AUR package, or should I just "grip it & rip it"?


[file_selector_android] Expose DISPLAY_NAME of the files selected using openFile

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Use case
I would like to display the name of the file that was selected by the user through the openFile function provided by the file_selector package. However, on Android, the name returned by (await openFile(...))?.name has no obvious relationship to the name the user saw in the selection dialog.
I briefly searched through the code of the Android implementation --- the name that should be returned is retrieved, but then it is dropped by XFile. The last path segment is returned instead.
(I encountered this issue when trying to migrate from file_picker to file_selector.)
Proposal
I believe that XFile.name should be decoupled from XFile.path on Android (similarly to web).
Any other way to retrieve the DISPLAY_NAME of the selected files would be appreciated too.

Hi @jjanku. Thanks for filing this. This seems to be similar to https://github.com/flutter/flutter/issues/113441. Can you confirm it is?

Hi, this is quite a different request. My issue concerns opening an existing file and retrieving its name, https://github.com/flutter/flutter/issues/113441 seems to deal with obtaining a path where a new file can be saved. My issue should be imho much easier to fix since the necessary pieces are already there, see the links.

This is likely blocked on https://github.com/flutter/flutter/issues/91869, since the ideal way to handle this would be an Android-specific XFile implementation, which would require making it an interface.

(fyi-android just for awareness; the fix here would be a trivial Dart-only change once XFile supports alternate implementations.)

+1

As a workaround, I override cross_file package with this fix https://github.com/flutter/packages/commit/4f472f5cf5a1a515f35f165a5eb860d185c505d9

any update on this issue. Am also interested in getting the name of the file selected by user.

Any update on this ticket?

@iamjay Could you create a PR for this? I really need this solution as well, thanks.


Add Clay Pot blocks rotation function

Directional blocks like stairs can be rotated (e.g. when part of the structure) which is useful feature to have for Clay Pots that have drawings on them

Can't be implemented since rotate method does not provide enough context to modify block entity

WARNING: Mirror function doesn't actually mirror shards, instead just rotates them 180 degrees, so shards that are placed perpendicularly to the rotation axis will switch places. I don't know how to fix this without overcomplications, so I hope this is not that big of a deal.


Multi label class encoding

Feature request
I have a multi label dataset and I'd like to be able to class encode the column and store the mapping directly in the features just as I can with a single label column.  class_encode_column currently does not support multi labels.
Here's an example of what I'd like to encode:
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.class_encode_column('labels')

I did some digging into the code base to evaluate the feasibility of this (note I'm very new to this code base) and from what I noticed the ClassLabel feature is still stored as an underlying raw data type of int so I thought a MultiLabel feature could similarly be stored as a Sequence of ints, thus not requiring significant serialization / conversion work to / from arrow.
I did a POC of this here and included a simple test case (please excuse all the commented out tests, going for speed of POC here and didn't want to fight IDE to debug a single test).   In the test I just assert that num_classes is the same to show that things are properly serializing, but if you break after loading from disk you'll see the dataset correct and the dataset feature is as expected.
After digging more I did notice a few issues

After loading from disk I noticed type of the labels class is Sequence not MultiLabel (though the added feature attribute came through).  This doesn't happen for ClassLabel but I couldn't find the encode / decode code paths that handle this.
I subclass Sequence in MultiLabel to leverage existing serialization, but this does miss the custom encode logic that ClassLabel has.  I'm not sure of the best way to approach this as I haven't fully understood the encode / decode flow for datasets.  I suspect my simple implementation will need some improvement as it'll require a significant amount of repeated logic to mimic ClassLabel behavior.

Motivation
See above - would like to support multi label class encodings.
Your contribution
This would be a big help for us and we're open to contributing but I'll likely need some guidance on how to implement to fit the encode / decode flow.  Some suggestions on tests / would be great too, I'm guessing in addition to the class encode tests (that I'll need to expand) we'll need encode / decode tests.

You can use a Sequence(ClassLabel(...)) feature type to represent a list of labels, and cast_column/cast to perform the "string to label" conversion  (class_encode_column does support nested fields), e.g., in your case:
from datasets import Dataset, Sequence, ClassLabel
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.cast_column('labels', Sequence(ClassLabel(names=["a", "b", "c", "d"])))

Great!  Can you elaborate on "class_encode_column does support nested fields"?  Do you mean that there is a way to class_encode_column on a Sequence?

Yes, exactly! This would be a nice contribution, though.

Sorry, I'm still not following.   Are you saying that there currently exists a way to call class_encode_column on a Sequence(ClassLabel) type?  Or that the underlying data structures support it and a contribution of a method to do that would be welcome?

class_encode_column  currently does not support Sequence(ClassLabel). Implementing support for this would be a nice contribution.
In the meantime, this limitation can be circumvented by fetching (unique) labels and calling .cast_column(col, Sequence(ClassLabel(names=labels))).

Ok makes sense, can you take a look at the POC implementation I did here?  Happy to take another pass / submit as a PR but would be helpful if I got a thumbs up that this was directionally correct with respect to implementation / architecture.

There is no need to introduce a new type (MultiLabel) for this feature. Also, I think we can keep the logic inside a single method instead of separating the two cases.
Maybe https://github.com/huggingface/datasets/pull/4277 can help with the implementation. We extended align_labels_with_mapping to support Sequence(ClassLabel(...)) in that PR (initially, it only worked with ClassLabel(...))


How to switch on/off hyperboard?

 Is there any easy way to do so except commenting the codes?
Thank you!

Unfortunately I didn't provide this feature. If you don't want your old records to be overwritten by your debugging records, you can specifically add a "hyperparameter" named debug and set it to False while training / True while debugging.

Thank you!


[SMALLFIX] Use static imports for standard test utilities for TransportProviderTest

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/17749/
Test PASSed.


PhantomJS crashes when setting file field value.

I have created an issue before at #158. There you fixed one of the problems, but it seems that it is not fixed completely. As also mentioned on the previous issue i had problems on Travis-CI too.
Now i took the moment to run specs on my vagrant VM and reproduce them.
The following specs are crashing PhantomJS intermittently (not always and not when running only failing spec):
https://github.com/jarmo/watirspec/blob/master/filefield_spec.rb#L127-L131
https://github.com/jarmo/watirspec/blob/master/filefield_spec.rb#L138-L146
https://github.com/jarmo/watirspec/blob/master/filefield_spec.rb#L151-L159
https://github.com/jarmo/watirspec/blob/master/filefield_spec.rb#L161-L165
Steps to reproduce:

Install Vagrant
Install and start Vagrant box:

vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
vagrant init lucid32
vagrant up

SSH into vagrant box:

vagrant ssh

Run the following commands inside of the vagrant VM:

sudo apt-get update
sudo apt-get install -y build-essential chrpath git-core libssl-dev libfontconfig1-dev curl libxslt1-dev libxml2-dev

curl -L -O "https://dl.dropbox.com/u/2731643/phantomjs/phantomjs-latest.tar.bz2"
mkdir phantomjs
tar -xvjf phantomjs-latest.tar.bz2 -C phantomjs
cat phantomjs/phantomjs.version
chmod +x phantomjs/phantomjs
export PATH=~/phantomjs:$PATH
cat phantomjs/phantomjs.version
phantomjs --version

git clone git://github.com/jarmo/watir-webdriver.git
cd watir-webdriver
git submodule update --init

\curl -L https://get.rvm.io | bash -s stable --ruby
source ~/.rvm/scripts/rvm
ruby --version
bundle install

export DISPLAY=ignored_really
export WATIR_WEBDRIVER_BROWSER=phantomjs
echo "\$DEBUG=true" >> spec/spec_helper.rb

Run file field specs inside of VM:

bundle exec rspec spec/watirspec/filefield_spec.rb

You may also run specific spec by appending a line number like so:
bundle exec rspec spec/watirspec/filefield_spec.rb:127

As mentioned above, running only crashing test would not crash PhantomJS - there is some state retained in PhantomJS between the specs.
Running these specs will crash PhantomJS almost always and you will see a message similar to this:
PhantomJS has crashed. Please read the crash reporting guide at https://github.com/ariya/phantomjs/wiki/Crash-Reporting and file a bug report at https://code.google.com/p/phantomjs/issues/entry with the crash dump file attached: /tmp/3e1184fb-b0de-1a80-23ae5b12-49abfbd6.dmp

Please note that $DEBUG=true inside of spec/spec_helper.rb will output a lot of debug information.

This issue still exists. Any updates on this?


go/types: gotype can't find vendored packages

go1.6 gotype seems to be unaware of vendoring

What version of Go are you using (go version)?
1.6

What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/john/src/gocode"
GORACE=""
GOROOT="/home/john/opt/go"
GOTOOLDIR="/home/john/opt/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?
john@ebola:~/src/gocode/src/github.com/the42/bevaddress$ gotype bevaddress.go

The tree is
john@ebola:~/src/gocode/src/github.com/the42/bevaddress$ tree -d
.
├── Godeps
└── vendor
└── github.com
├── gorilla
│   └── websocket
└── lib
└── pq
└── oid
8 directories

What did you expect to see?
no error (go build works)

What did you see instead?
bevaddress.go:9:2: could not import github.com/gorilla/websocket (can't find import: github.com/the42/bevaddress/vendor/github.com/gorilla/websocket)
bevaddress.go:10:4: could not import github.com/lib/pq (can't find import: github.com/the42/bevaddress/vendor/github.com/lib/pq)
bevaddress.go:18:16: undeclared name: websocket

@the42 It was recommended to me to run go install or go build -i after vending code or running for the first time.  This builds out the vendor code into $GOPATH/pkg/{packagePath}/vendor/... and then gotype stops complaining. I guess its similar to an npm i step.

@bcomnes what remains is whether your recommendation is the way it should be or if it's a bug. I think it's a bug.

Related to #11415. gotype is simply an app that invokes go/types. go/types doesn't build dependencies, it simply type-checks on package and relies on compiled versions of imported packages. If those don't exist (are not installed), go/types cannot find them.
Eventually, go/importer needs to support importing from source.

@the42 ¯\_(ツ)_/¯
@griesemer Thanks for the background.  :+1:

I'm going to close this as duplicate of #11415, and in turn mark that one for Go1.8.


Disabled classicFullStack

@mmayr-at please also change the default here:
https://github.com/Dynatrace/dynatrace-operator/blob/master/config/samples/cr.yaml#L160

Please also add it to the openshift section in the Readme.md

@mmayr-at @DTMad what is plan/roadmap here?
Can we expect that full stack monitoring will be implemented on some other way (hopefully on way which does need current oneagent at all) as this setting name is classicFullStack and now it is even disabled by default?

There will be a replacement where we change the way of injection in behind and improve usability.
However, it will still appear with the same visibility and insights you are used to.
I can't give an exact plan yet but it will be released with v0.3.0

That is nice to hear. I can also participate for piloting when any kind of beta/etc version is available.
I would be very happy to see that it would be implemented on way that it can support https://github.com/talos-systems/talos
ActiveGate and Kubernetes connection already works on it  but OneAgent cannot as it have long list requirements for binaries and libraries which does not exist on Talos.
PS. Just to clarifying: Company where I'm working on is Dynatrace customer and currently running Docker Swarm on private cloud. Eventually we will migrate to Kubernetes and it would be nice to be able to use Talos capabilities to minimize effort needed to maintain it without losing Dynatrace support.

We don't know if we will have some kind of beta - if so I'll let you know.
The data the agent and the AG delivers will not change.
It will be a change of how we do the injection within Kubernetes to solve pain points we used to have in the past and to make our injection cloud native.


 I don't think this is related to osgi-run, but to the osgi plugin. Please check with them how to fix this as I can't change this, osgi-run does not create your manifest... osgi does, using bnd behind the scenes.

By the way, don't explicitly declare Private-Package.... you should only declare Exported-Package, the osgi plugin will generate Private-Package for you.

Closing as this is not a bug in osgi-run.

yes you were right.
the correct way to override name or symbolic name is:
jar {
    manifest {
        name project.description
        symbolicName 'adm'
}

regards!


BED-5078: Fix bug in Kerberoastable users with most privs

Description
Updated the logic of cypher to properly filter all user objects considered for appropriate kerberoastability.
Motivation and Context
This PR addresses: BED-5078
The previous query only applied the filters to a subset of the matched results, rather than all users.
How Has This Been Tested?
Local env.
Screenshots (optional):
Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

[X] I have met the contributing prerequisites

Assigned myself to this PR
Added the appropriate labels
Associated an issue: https://github.com/SpecterOps/BloodHound/issues/672
Read the Contributing guide: https://github.com/SpecterOps/BloodHound/wiki/Contributing

[X] I have ensured that related documentation is up-to-date

Open API docs
Code comments (GoDocs / JSDocs)

[X] I have followed proper test practices

Added/updated tests to cover my changes
All new and existing tests passed

Approving since the issue isn't functional and we're having trouble diagnosing why gofmt isn't working as expected. Pull when ready


fix db upgrade #33

@ryansanford Corrected db upgrade script that broke integration server.
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

Once again, thanks for the help @ryansanford! Fixing was a breeze with the ability to test against a more complete db state. I didn't find any further issues with the db-upgrader since. Here's the relevant log entry:
./docker/run-cmd.sh db-upgrade
...
2017-07-05 14:55:24             root              database.py  1191:INFO Upgrade to version 33 complete.


Add a default value for the category to fix test issues

The policy report tests have been behaving poorly.  In environments where there is an insights NonCompliance, I think a metric gets created that causes the tests to pass. In some environments that NonCompliance doesn't exist causing this problem -- mainly because the policy resource provided doesn't have a NIST 800-53 category defined so no metric is created by the test. Providing a reasonable default for the category makes sense to me since the standard/category/controls for organizing compliance is optional -- but we don't want to optionally create the metric.
Refs:

https://issues.redhat.com/browse/ACM-8851

Related Issue:  stolostron/backlog#<ISSUE_NUMBER>
Description of changes

/cc @dislbenn @dhaiducek

Need an approver:
/cc @jlpadilla @SherinV

/cherry-pick release-2.9

/cherry-pick release-2.8

/cherry-pick release-2.7


KYLIN-3994: StorageCleanupJob may delete cube id data of new built segment because of cube cache in CubeManager

See: https://issues.apache.org/jira/browse/KYLIN-3994

Can one of the admins verify this patch?

Pull Request Test Coverage Report for Build 4466

3 of 5 (60.0%)  changed or added relevant lines in 4 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.003%) to 27.866%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core-cube/src/main/java/org/apache/kylin/cube/CubeManager.java
0
2
0.0%

Files with Coverage Reduction
New Missed Lines
%

core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java
2
80.23%

Totals

Change from base Build 4453:
-0.003%

Covered Lines:
22800

Relevant Lines:
81819

💛  - Coveralls

it's fine to me

Codecov Report

Merging #633 into master will decrease coverage by <.01%.
The diff coverage is 50%.

@@             Coverage Diff              @@
##             master     #633      +/-   ##
============================================
- Coverage     25.78%   25.78%   -0.01%
- Complexity     6008     6009       +1
============================================
  Files          1382     1382
  Lines         82202    82205       +3
  Branches      11519    11519
============================================
- Hits          21198    21197       -1
- Misses        58958    58962       +4
  Partials       2046     2046

Impacted Files
Coverage Δ
Complexity Δ

...c/main/java/org/apache/kylin/cube/CubeManager.java
37.83% <0%> (-0.2%)
44 <0> (ø)

...pache/kylin/rest/job/StorageCleanJobHbaseUtil.java
49.38% <100%> (ø)
7 <0> (ø)
:arrow_down:

.../org/apache/kylin/rest/job/MetadataCleanupJob.java
66.42% <100%> (ø)
26 <0> (ø)
:arrow_down:

...a/org/apache/kylin/rest/job/StorageCleanupJob.java
65.48% <100%> (ø)
31 <0> (ø)
:arrow_down:

...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java
69.3% <0%> (-0.92%)
7% <0%> (ø)

...ream/core/storage/columnar/ColumnarStoreCache.java
51.16% <0%> (+2.32%)
8% <0%> (+1%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 670c2a9...7ac7c45. Read the comment docs.


Getting Syntax error - though i copied from here and made hosts, datastage source file entries in main.yml and inventory file

Hi Folks
Need your help on this...!!!! Thanks in advance for your help.
When i run the DataStage ansible playbook getting as below error.
"ERROR! A playbook must be a list of plays, got a <class 'ansible.parsing.yaml.objects.AnsibleMapping'> instead
The error appears to be in '/etc/ansible/roles/ansible-role-infosvr/defaults/main1.yml': line 19, column 1, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
LOCATIONS
ibm_infosvr_install_location:
^ here
"

Hi @wangadae -- thanks for trying the role!
From your issue, one possible problem I can see is that you have cloned this repository as-is into your /etc/ansible/roles location with the repository name (ansible-role-infosvr).  The playbooks all expect the name of the role to be as it would be when installed via Ansible Galaxy, that is: IBM.infosvr.  You might be better off installing the role this way, as you'll be sure to get "released" functionality that way rather than what may be an interim state in the latest code in the repository.
You can install it this way using: ansible-galaxy install IBM.infosvr
(This will create the role with the appropriate name for you on your system.)
You'll still want to create an inventory defining your environments' hosts, and will probably want to override some of the variables in the main.yml.  Both of these activities can be done without modifying anything in the base role itself.  I've written a very brief tutorial on these aspects here which might help: https://cmgrote.github.io/ansible-tutorial/
(ie. to override variables I would suggest making use of a group_vars/all.yml file relative to where your playbook is defined.)
Finally, don't forget that the role is only a role -- not a playbook.  You need to define your own playbook that uses the role, like the example provided in the README: https://github.com/IBM/ansible-role-infosvr#example-playbook

Hi cmgrote
Thanks a lot for your response......
Is it possible to write a DataStage Playbook with response file of DataStage to do silent installation in a every New VM's? because right now i am using sample response file for DS silent installation in a New VM facing many issues....

Hi All,
I am new to IIS and Ansible  and would like to Automate ( DevOps ) using this. Can some one please guide me how we can run this playbook and what all configuration/directory, path etc i have to create and where to create and place inside ansible installed directory.
So far i have done below

Created/updated  main.yml and all the path ( media and installation ) has been provided correctly.
Created Play book for ansible
Modified hosts for expected inventory which was required

but when i m running the playbook i m getting error and not sure what all path/folder and files where i have to keep.
group_vars/all.yml where we can find this and is this inside ansible-role-infosvr-master\tasks\install
A complete setup guide will be highly appreciated.
Thanks

Hi @udaybit45,
a bit to late but you need to mention the error that you are getting.
With regards
Andrew


Adding progress bar on the top of the page.

As we scroll down the page the progress bar will increase. It will give a good look to the page.

please assign me this issue

Thanks for creating the issue in WildGuard! The issue has been assigned to you.
Before you start working on your PR, please make sure to:

⭐ Star the repository if you haven't already.
Pull the latest changes to avoid any merge conflicts.
Attach before & after screenshots in your PR for clarity.
Include the issue number in your PR description for better tracking.

Don't forget to follow @anuragverma108 – Project Admin & @PriyaGhosal -Project Collaborator  – for more updates!
We're excited to see your contribution as part of Hacktoberfest 2024! 🎉
Happy open-source contributing!


Workflow creation and rejection rights

Dear Developers,
it looks like currently people who can inject new workflows not always have rights to reject them. This leads to a situation that we need to clean up bad requests ourselves. It's better if the person who makes a mistake can fix it himself. It can be a potential issue if we want to remove a manual handshake between PPD and Computing.
Dima

Provided the workflow has not been assigned, after which point control of the workflow has to change hands to ops.

Adding to this discussion:
ReqMgr doesn't allow to create ACDC's when an user is registered to DATAOPS but not to PPD group.
In that case we get:
400 Bad Request
Create request failed, (400, 'Error in Request check-in: RequestCheckInError
Message: Error creating new request:
User mcremone is not a member of group ppd
User is associated to groups: ['DATAOPS']
\tModuleName : WMCore.RequestManager.RequestMaker.CheckIn
\tMethodName : checkIn


Removing activities causes strange behavior

If activity xyz is removed, the activities for a worker, with only the removed activity, will look like ["xyz", ""].
If a new activity is then added to the worker, it will remove the old activity but still be displayed as ["new activity", ""].

This is will be solved with the next pull request

solved


User Authentication Failed with Cluster, Client not Redirected to Cluster Member

Prerequisites

Can you reproduce? Sometimes
Are you running the latest version of SoftEtherVPN? Build 9674

SoftEther version: Version 5.01 Build 9674 Developer Edition
Component: [Server, Client, Bridge, etc.] Client
Operating system: [Windows, Linux, BSD, macOS, etc.] Linux
Architecture: [64 bit, 32 bit] 64 bit
[In case it's a computer with known specs, such as the Raspberry Pi, you can specify it omitting the details.]
Processor: [Specify brand and model. Example: AMD Ryzen 7 1800x] Intel(R) Xeon(R) CPU E5-2620 v3
Description
VPN client is not redirected to cluster member.
VPN client can successfully authenticate with cluster controller. It seems that then the controller tells the client to reconnect to one of the cluster member, and issues a ticket authentication to the member. However, the client reconnects back to the cluster controller and hence the authentication failed indefinitely.
Here are the controller server logs:
2023-03-16 21:13:37.330 On the TCP Listener (Port 1118), a Client (IP address x.x.x.x, Host name "redacted", Port number 54994) has connected.
2023-03-16 21:13:37.330 For the client (IP address: x.x.x.x, host name: "redacted", port number: 54994), connection "CID-463" has been created.
2023-03-16 21:13:37.542 SSL communication for connection "CID-463" has been started. The encryption algorithm name is "TLS_AES_256_GCM_SHA384".
2023-03-16 21:13:37.935 [HUB "Infra"] The connection "CID-463" (IP address: x.x.x.x, Host name: redacted, Port number: 54994, Client name: "SoftEther VPN Client Developer Edition", Version: 5.01, Build: 9674) is attempting to connect to the Virtual Hub. The auth type provided is "Certificate authentication" and the user name is "devx".
2023-03-16 21:13:37.935 [HUB "Infra"] The Virtual Hub's Security Account Manager has received the following certificate from the VPN Client and accepted its contents as the certificate for when user "devx" logs in: CN=devx, O=Local, S=Local, C=ID, SERIAL="2D5943FB9ECB3687" (Digest: MD5="C908B0E5F8BB6B31D7E4A02B7EAE79F4", SHA1="5757728854749DF46F253EA742BAF0BBECFFB5B9")
2023-03-16 21:13:37.935 [HUB "Infra"] Connection "CID-463": Successfully authenticated as user "devx".
2023-03-16 21:13:37.935 [HUB "Infra"] Connection "CID-463": The cluster controller is deciding the destination cluster member of this client.
2023-03-16 21:13:37.935 Connection "CID-463": The server "artemis" has been decided as a destination cluster member server. Instructing clients to redirect connection to this server.
2023-03-16 21:13:37.935 A new client authentication ticket has been issued for cluster member "artemis". Virtual Hub "Infra", User name "devx" ("devx"), Session name "SID-DEVX-64", Ticket "247DD9DA362BB64CBB11F03E09C06EB85207BB1C"
2023-03-16 21:13:38.136 Connection "CID-463" terminated by the cause "No error." (code 0).
2023-03-16 21:13:38.136 Connection "CID-463" has been terminated.
2023-03-16 21:13:38.136 The connection with the client (IP address x.x.x.x, Port number 54994) has been disconnected.
2023-03-16 21:13:38.257 On the TCP Listener (Port 1118), a Client (IP address x.x.x.x, Host name "redacted", Port number 55008) has connected.
2023-03-16 21:13:38.257 For the client (IP address: x.x.x.x, host name: "redacted", port number: 55008), connection "CID-464" has been created.
2023-03-16 21:13:38.478 SSL communication for connection "CID-464" has been started. The encryption algorithm name is "TLS_AES_256_GCM_SHA384".
2023-03-16 21:13:38.841 [HUB "Infra"] The connection "CID-464" (IP address: x.x.x.x, Host name: redacted, Port number: 55008, Client name: "SoftEther VPN Client Developer Edition", Version: 5.01, Build: 9674) is attempting to connect to the Virtual Hub. The auth type provided is "Ticket authentication" and the user name is "devx".
2023-03-16 21:13:38.841 [HUB "Infra"] Connection "CID-464": User authentication failed. The user name that has been provided was "devx", from x.x.x.x.
2023-03-16 21:13:39.022 Connection "CID-464" terminated by the cause "User authentication failed." (code 9).
2023-03-16 21:13:39.022 Connection "CID-464" has been terminated.

As you can see, it is said that user devx in Hub Infra has successfully authenticated with the controller and is decided to be redirected to member artemis. But, the client seems to reconnect back to the cluster controller with Ticket authentication.
Expected behavior:
The member server log outputs that the client is authenticating with the member. Some clients do this, but some other don't.
Actual behavior:
The client reconnects back to the controller and authenticates with the controller with ticket authentication, and it fails.
Steps to reproduce
Because this is flaky, I find this difficult to debug. But create a cluster of 2 nodes and with static hub, and try to connect as usual.

9674 was released years ago. Please build from the latest source and try again.

#1482 fixed this already

Thanks, is that patch for server or client functions?


Types exported from a Vue file in a monorepo package are not found in the importing package with the TypeScript Vue Plugin (Non-takeover mode)

Steps to Reproduce

Have VS Code with the "Vue Langauge Features (Volar)" & "TypeScript Vue Plugin (Volar)" extensions installed
Clone https://github.com/segevfiner/vue-typescript-plugin-issue
pnpm i
pnpm -C packages/a build
code .
Open packages/b/composables/foo.ts.

There will be no real type for Foo in non-takeover mode using the TypeScript Vue Plugin, and the incorrect assignment will not be flagged in VS Code. If you run pnpm type-check in packages/b it will flag the assignment. This doesn't happen in take over mode. It seems like something isn't working in this case in the TypeScript server plugin. Weird, cause I remember this used to work.
Using VS Code 1.80.1
Volar 1.8.5
Happens with either VS Code's builtin TypeScript 5.1.6 or with the workspace TypeScript 5.0.4

Thanks for the report!

This error seems to be like mine https://github.com/vuejs/language-tools/issues/3415
But it's not solved for me :(
I've created a reproduction repo here https://github.com/vuejs/language-tools/issues/3415


Tasks file -f support

Close https://github.com/AzureCR/azure-cli/issues/230 when this done as well


chore: Add template tagging for shorter cypress runs.

Description
This pull request adds template tagging functionality to improve the efficiency of cypress runs.
PR fixes following issue(s)
Fixes # (issue number)

if no issue exists, please create an issue and ask the maintainers about this first

Media

A video or a GIF is preferred. when using Loom, don’t embed because it looks like it’s a GIF. instead, just link to the video

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Chore (housekeeping or task changes that don't impact user perception)
This change requires a documentation update

Testing

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Also list any relevant details for your test configuration.
Delete anything that is not relevant

[ ] Manual
[ ] JUnit
[ ] Jest
[ ] Cypress

Test Plan

Add Testsmith test cases links that relate to this PR

Issues raised during DP testing

Link issues raised during DP testing for better visiblity and tracking (copy link from comments dropped on this PR)

Checklist:
Dev activity

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Speedbreak features have been covered
[ ] Test plan covers all impacted features and areas of interest
[ ] Test plan has been peer reviewed by project stakeholders and other QA members
[ ] Manually tested functionality on DP
[ ] We had an implementation alignment call with stakeholders post QA Round 2
[ ] Cypress test cases have been added and approved by SDET/manual QA
[ ] Added Test Plan Approved label after Cypress tests were reviewed
[ ] Added Test Plan Approved label after JUnit tests were reviewed

Summary by CodeRabbit

Tests

Enhanced end-to-end test coverage for template filtering, forking, and modal interactions.
Added new test cases to ensure template features behave correctly when adding pages and forking to apps.
Improved test reliability with additional assertions and waiting conditions.
Implemented mock server responses for more robust permission testing.
Verified tooltip functionality in the "Reconnect Datasources" feature within the template workflow.
Organized test suites with tags for better categorization.

/ok-to-test<EMAIL_ADDRESS>

Duplicate: #29790


Consider custom exceptions in OpenAPI documentation

Custom exception handlers should also appear as possible responses in the auto-generated OpenAPI specification.

Rought idea, but what do you think of this @MoritzWeber0?

basic error pydantic models
a function to create error responses
generating responses just above each route so they can then be used in the responses list (this is what makes them appear in OpenAPI)

The things I'm still trying to figure out:

This lets every error have its own example in the documentation, though then we're documenting in the route instead of in the pydantic model
Do we have any routes that have multiple ways to reach the same error status code? This might not allow us to document multiple reasons for a specific error code without resorting to something silly like a long string with a separator, but I haven't tested it/checked
We could pass both technical and user descriptions this way, but not sure if that's necessary or helpful

I just tested this out on the user routes for users/id:
users/user_id:

class ErrorDetail(pydantic.BaseModel):
    err_code: str
    reason: str

class ErrorResponse(pydantic.BaseModel):
    status_code: int
    detail: ErrorDetail

def create_error_response(
    status_code: int, err_code: str, reason: str
) -> dict:
    return {
        "model": ErrorResponse,
        "description": "Maybe a technical description as opposed to the user one?",
        "content": {
            "application/json": {
                "example": ErrorResponse(
                    status_code=status_code,
                    detail=ErrorDetail(
                        err_code=err_code,
                        reason=reason,
                    ),
                ).model_dump()
            }
        },
    }

NO_PROJECTS_IN_COMMON_RESPONSE = create_error_response(
    status_code=status.HTTP_403_FORBIDDEN,
    err_code="NO_PROJECTS_IN_COMMON",
    reason="You need at least one project in common to access the user profile of another user.",
)

@router.get(
    "/{user_id}",
    response_model=models.User,
    responses={status.HTTP_403_FORBIDDEN: NO_PROJECTS_IN_COMMON_RESPONSE},
)
def get_user(
    own_user: models.DatabaseUser = fastapi.Depends(injectables.get_own_user),
    user: models.DatabaseUser = fastapi.Depends(injectables.get_existing_user),
    db: orm.Session = fastapi.Depends(database.get_db),
) -> models.DatabaseUser:
    if (
        user.id == own_user.id
        or len(projects_crud.get_common_projects_for_users(db, own_user, user))
        > 0
        or auth_injectables.RoleVerification(
            required_role=models.Role.ADMIN, verify=False
        )(own_user.name, db)
    ):
        return user
    else:
        raise fastapi.HTTPException(status_code=status.HTTP_403_FORBIDDEN)

Ugh, just realized raising the exception that way would break the existing error interceptor. So it seems like it would be a even more duplication, because you'd need to repeat everything for both the responses and for raising the exception itself? Seems like there must be a way to avoid that

Rough idea, but what do you think of this @MoritzWeber0?

basic error pydantic models

We don't use pydantic models as of now and I want to avoid that we have to define exceptions twice. Have look at the exception files, e.g. capellacollab/users/exceptions.py. We already define the exceptions here, it would be great if we could use them.

generating responses just above each route so they can then be used in the responses list (this is what makes them appear in OpenAPI)
It would be nice if we could easily register all exceptions a API route can throw, e.g. via an annotation.
Better would be if it's completely auto-determined by scanning the AST of the function to detect which exceptions a function can raise: https://stackoverflow.com/questions/32560116/how-to-list-all-exceptions-a-function-could-raise-in-python-3

The things I'm still trying to figure out:

This lets every error have its own example in the documentation, though then we're documenting in the route instead of in the pydantic model
Do we have any routes that have multiple ways to reach the same error status code? This might not allow us to document multiple reasons for a specific error code without resorting to something silly like a long string with a separator, but I haven't tested it/checked
Don't have an example right now, but this case should exist somewhere. That's why we often use the exceptions files to define exceptions which can then be used in different places.
We could pass both technical and user descriptions this way, but not sure if that's necessary or helpful
Would leave it out for now.
I just tested this out on the user routes for users/id:

Looks good.

class ErrorDetail(pydantic.BaseModel):
    err_code: str
    reason: str

class ErrorResponse(pydantic.BaseModel):
    status_code: int
    detail: ErrorDetail

def create_error_response_and_exception(
    status_code: int, err_code: str, reason: str
) -> dict:
    return {
        "response": {
            "model": ErrorResponse,
            "description": "Maybe a technical description as opposed to the user one?",
            "content": {
                "application/json": {
                    "example": ErrorResponse(
                        status_code=status_code,
                        detail=ErrorDetail(
                            err_code=err_code,
                            reason=reason,
                        ),
                    ).model_dump()
                }
            },
        },
        "exception": {
            "status_code": status_code,
            "detail": {
                "err_code": err_code,
                "reason": reason,
            },
        },
    }

NO_PROJECTS_IN_COMMON = create_error_response_and_exception(
    status_code=status.HTTP_403_FORBIDDEN,
    err_code="NO_PROJECTS_IN_COMMON",
    reason="You need at least one project in common to access the user profile of another user.",
)

@router.get(
    "/{user_id}",
    response_model=models.User,
    responses={status.HTTP_403_FORBIDDEN: NO_PROJECTS_IN_COMMON["response"]},
)
def get_user(
    own_user: models.DatabaseUser = fastapi.Depends(injectables.get_own_user),
    user: models.DatabaseUser = fastapi.Depends(injectables.get_existing_user),
    db: orm.Session = fastapi.Depends(database.get_db),
) -> models.DatabaseUser:
    if (
        user.id == own_user.id
        or len(projects_crud.get_common_projects_for_users(db, own_user, user))
        > 0
        or auth_injectables.RoleVerification(
            required_role=models.Role.ADMIN, verify=False
        )(own_user.name, db)
    ):
        return user
    else:
        raise fastapi.HTTPException(**NO_PROJECTS_IN_COMMON["exception"])

In any case I'd keep the native Python way to use exceptions, to raise them.
But what we could do is to restructure the exceptions a bit.
Let's look at the current structure (example capellacollab/users/exceptions.py:
@dataclasses.dataclass
class UserNotFoundError(Exception):
    username: str | None = None
    user_id: int | None = None

async def user_not_found_exception_handler(
    request: fastapi.Request, exc: UserNotFoundError
) -> fastapi.Response:
    return await exception_handlers.http_exception_handler(
        request,
        fastapi.HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail={
                "title": "User not found",
                "reason": f"The user '{exc.username or exc.user_id}' doesn't exist.",
            },
        ),
    )

def register_exceptions(app: fastapi.FastAPI):
    app.add_exception_handler(
        UserNotFoundError, user_not_found_exception_handler  # type: ignore[arg-type]
    )

As you already can see, this is a lot of boilerplate. We could also generate the error message in the Exception directly:
@dataclasses.dataclass
class UserNotFoundError(Exception):
   def __init__(self, username: str | None = None, user_id: int | None = None):
      self.status_code=status.HTTP_404_NOT_FOUND
      self.title = "User not found"
      self.reason = f"The user '{username or user_id}' doesn't exist."

Then, we could remove the user_not_found_exception_handler function.
In addition, we can have a central registration of exceptions, for example a dictionary:
registered_exceptions = {UserNotFoundError}

The main function could pick it up and register all registered exceptions properly.
In addition, we can add create_error_responses([UserNotFoundError]) to the route and generate the docs from the Exception directly. What do you think about it?


[Object Oriented Programming using Java/CharFrequency.java] Add logic to count frequency of characters in string entered by user

File to add code

Hi, could you assign it to me?


DRAFT: Experiment with error handling

We have modified GetApp and SetCurrentDroplet in the app handler as an example
The main changes are as follows:
Errors:

We have a new single location for errors: the apierr package
Errors within apierr are objects that replace the writeXXXError methods from apis/shared.go.
These errors are raised throughout the system - mainly in repositories and handlers
These errors have all the information, i.e. http status and error details, to write out API error messages

Repositories:

Repo code calling k8s methods should map k8s errors to apierr errors using wrapK8sError which also takes the resourceType

Handlers:

Handler methods now have a new signature: handleXXX(authInfo authorization.Info, r *http.Request) (status int, responseObj interface{}, err error)

They do not get the response writer any more as this is now responsibility of the endpoint function wrapper (see below)
Return status and responseObj for successful requests
Return an error for error responses

An apierr error will render a custom API error
Any other error will be an 'UnknownError' 500

Handlers should modify ForbiddenError to NotFoundErrors where appropriate using the apierr.AsNotFoundIfForbidden(err) helper
Handlers should modify NotFoundError or ForbiddenError to UnprocessableEntityErrors where appropriate using the apierr.AsUnprocessableIfNotFoundOrForbidden(err) helper

We assume mapping NotFound and Forbidden to Unprocessable is the only general case. If not, we might think about callbacks we could pass in to match certain errors and be more flexible

A new wrapper in the routes wiring handles presenting the successful or error responses

One possible alternative would be to return a SuccessResponse struct that includes status and payload. This could also include the response encoding (e.g json, text), which would address Akira's concern.

I agree that maybe using a response object instead of two "sucess" values is maybe a good thing, it also enable future extensibility. However, if there are no examples of endpoints returning non-json object, I would recommend doing the simplest thing, i.e. always encode to json. If non-json cases appear, we could easily extend the response object to also contain encoding instructions.

Implemented by #798


Added documentation for #9733

Added documentation for PR https://github.com/mautic/mautic/pull/9733 .

@alfredoct96 please can you pull in the docs from the master branch? As this is on a private repo we are not able to do that as maintainers!

@alfredoct96 please can you pull in the docs from the master branch? As this is on a private repo we are not able to do that as maintainers!

@RCheesley i did the rebase

Awesome thank you!

Great - thanks for that @alfredoct96 - ready to go once the PR itself is merged :)


frontend: add sign in confirmation modal + reword show review limit alert

Notify users to sign out if their account is not stamford student (This is a very safe temporary solution - the review page is very critical and changes should be carefully tested in beta server before deployment which we can't do it yet....)
Force refresh when sign out to clear all the sessions and prevent from being able to do things you shouldn't (e.g. see their "My reviews" after signing out)
This modal will appear when clicking on the sign in button in both header and navbar

@chinathaip You want to change UI rhor?


Properties with reserved keywords cause inserts to fail

If i have a model called 'todo' and it has a property named 'order', todo.save() will fail when I try to call it. Renaming the 'order' field listed in the 'Field' enum to something like 'rank' lets you create the table and fetch from the table, but save will still fail.
Why does save fail?
The name 'order' is reserved by Cassandra for sorting. When it finds the word in the query, it gets confused and returns an error. I tried to workaround this by changing the Field name to 'rank'.
Save uses reflection to get the properties and values from the 'todo' object. The other functions use the 'Field' enum to get the field names. This means that every other function will use 'rank' except save, which uses the real property name.
How do we fix this then?
Allow for custom field names
We could allow people to map the property name to a field name and use it when saving. This definitely allows for more flexibility.
Escaping column names
This works:
CREATE TABLE IF NOT EXISTS fakeitem(documentID uuid primary key, userID text, title text, "order" int, completed boolean);
This doesn't:
CREATE TABLE IF NOT EXISTS fakeitem(documentID uuid primary key, userID text, title text, order int, completed boolean);
Suggested Reading
Cassandra Keywords

Small side effect of escaping column names. They become case sensitive. If they aren't escaped, they are transformed to lowercase.


Helping the user bring the right things based on the weather

If it's raining, bring an umbrella
If it's sunny, bring suncream
If it's cold, bring a coat

@akshitadixit Can you assign this issue to me and @niyoj , please?


snapcast component does not display correct name

Home Assistant release (hass --version):
0.32.3
Python release (python3 --version):
3.4.3
Component/platform:
SnapCast
Description of problem:
SnapCast component no longer shows the name of the snapcast client, but shows "Unnamed device"
I'm running SnapCast v0.10.0 (both on server and client)
Expected:
BY default, the hostname of the client is used as the name of the snapcast client.

I also have this problem and have done some digging.  Apologies if I'm off the mark with this as python isn't a language I'm particularly good with.
From what I can see the issue lies in the snapclient library.  Specifically in control.py lines 57, 58 (possibly others) the identifier gets set to config.name when it looks like it should be set to host.name.
I've modified the file manually in my test and the correct hostnames do then show up in the relevant components, but I'm not sure it's the most correct fix.  Anyone with more python knowledge than myself care to take a poke and see if there's a more correct way to do this?

Yes, the python-snapcast module has to be modified slightly and the dependency in HA updated. When I upgrade to 0.10.0 myself, I'll do it unless someone gets to it first.

Will that fix allow volume control to work?

Seems to work fine for me in my testing

I am a little lost here. I have snapcast server v0.10.0 and just upgraded to home assistant 0.37.1. My Android clients work well and show the names of my snapcast clients. However, I have the above mentioned "Unnamed device"-problem.
I also see a control.py in the snapcast github repository, though this seems to be nowhere installed on my home automation server running home assistant.
I have though also no snapcast specific errors in my home assistant log.
Where did you change the control.py file?

The name should either be the "name" of the client (which you can set in the app, or the snapserver config file), or the IP of the client if there is not a name set.  It looks like the IP field has moved in the datastructure, which is why you're getting "Unnamed device".
I'll update python-snapcast to try and use the config->name field with a fallback to the host->name field with a fallback to the host->ip field.
In the interim, if you set a name for your clients, you won't see the "Unamed device" any longer.

Is the python-snapcast update rolled into the latest file? where can I set the name for diff clients?

Not yet. Maybe next release!

Sweet. Is there a workaround meanwhile? I know you mentioned some stuff in your post on Feb 9 but I am not sure where I can set the client names (or how).

You can set them via the Android app, or the snapserver config file (man snapserver to get the path)

Fixed in #7079


Sampler SamplerV2 consistency

URL to the relevant documentation
https://docs.quantum.ibm.com/guides/run-jobs-batch#open-a-batch
https://docs.quantum.ibm.com/guides/run-jobs-session#set-up-to-use-sessions
Select all that apply

[ ] typo
[ ] code bug
[ ] out-of-date content
[ ] broken link
[x] other

Describe the fix.
Hi,
For historical reasons, here and there we meet Sampler and SamplerV2 as Sampler imports (see examples above)
Is Consistency possible ?
Thanks

thanks for flagging this! they should all say samplerv2 as sampler so the other way is a typo. we'll get that fixed asap


prepare_header_path breaks unicode

When adding things with (too) long names to an archive, the function prepare_header_path ('src/builder.rs', line 452) truncates the name to 100 bytes (line 471: let path = bytes2path(Cow::Borrowed(&data[..max]))?;). If the 100:th byte is part of of a multi-byte unicode character then this creates invalid unicode. On unix platforms this is blindly ignored, but on windows and wasm3 this causes an error ("only Unicode paths are supported on Windows" / "Invalid utf-8") even if the original name is valid unicode! This happens in the bytes2path function ('src/header.rs', lines 1574 and 1606).
Simple reproducible example:
    #[test]
    fn demo() {
        let mut archive = tar::Builder::new(Vec::new());
        let content = "test".as_bytes();
        let name = "aåååååååååååååååååååååååååååååååååååååååååååååååååååååååååååå";
        let mut header = tar::Header::new_gnu();
        header.set_size(content.len() as u64);
        archive
            .append_data(&mut header, &name, content)
            .expect_err("One of the 'å':s in the name was split");
    }

Possible solution:
Truncate at some earlier (safe) point if this issue is detected.

Oops, this has already been fixed (I just assumed I had the latest version). Sorry, for the unnecessary notifications.


Add brief Firebase intro (does not replace full documentation)

@halatmit LGTM. Do we want to merge this now or wait until we merge the Firebase component and merge it with it?

Either way is fine.  You saw the disclaimer I added at the beginning.  But
it might be safer to Wait.
On Dec 20, 2015 10:57 PM, "Jeffrey Schiller"<EMAIL_ADDRESS>wrote:

@halatmit https://github.com/halatmit LGTM. Do we want to merge this
now or wait until we merge the Firebase component and merge it with it?
—
Reply to this email directly or view it on GitHub
https://github.com/mit-cml/appinventor-sources/pull/638#issuecomment-166195852
.

@afmckinney @halatmit I "stole" this review from you. I'm checking it now, but my intention would be to merge this after the Firebase change is merged (alternatively, I can incorporate this change in the Firebase commit). Hal, what do you think?

@halatmit LGTM

@jisqyv  I'd merge this after the Firebase change is merged.

@halatmit MERGED


Isabelle Week 4 Review Ticket

CB Create Task completed and documented on GH Pages - Prep to submit to CB on or before May, 2nd. (2 Points - Individual)
CB Create Task Coding requirements documented
Runtime of Code(with comments) on Replit/Intellij

CB Create Task completed and documented on GH Pages - Prep to submit to CB on or before May, 2nd. (Individual)
Total Overall Grade: 2/2 Points
- [x] CB Create Task Coding requirements documented
- [x] Runtime of Code(with comments) on Replit/Intellij

Isabelle's total overall grade for this assignment is 2/2 points. Her Create Task Coding requirements were documented in detail, and she had all of the Runtime for her Create Task Coding. Her Rat Simulator is a very fascinating educational program.


"Pretty-printing" Stored configuration into JSON (overwritePersistedConfig = true)

Could it be possible to add a 3rd argument to JSON.stringify and pretty-print the configuration, for better readability and editing ?
Eg.:
...
function persistObject(fm, object, path) {
    let raw = JSON.stringify(object, null, 2);
    fm.writeString(path, raw);
}
...

Refer to official doc

I Can look into it

I Can look into it

Top :ok_hand:

Top :ok_hand:


vorpal.parse() not working

version: 1.11.4
node version: 6.3.1
I am following the docs but vorpal.parse() seems to be broken. Can you please tell me what is wrong here? Thanks.
index.js
const vorpal = require('vorpal');

const vcl = vorpal();
const chalk = vorpal().chalk;

vcl
  .delimiter(chalk.cyan('choreo$'))
  .show()
  .parse(process.argv);

vcl
  .command('duck', 'Outputs "rabbit"')
  .action(function(args, callback) {
    console.log('quack');
  });

Terminal output:
➜  choreograph-cli git:(master) ✗ node src/index.js duck
choreo$

  Invalid Command. Showing Help:

  Commands:

    help [command...]  Provides help for a given command.
    exit               Exits application.

choreo$

@tocttou I think you may have to process the args before passing into parse first. Ie. to get rid of [node, src/index.js]
I've been doing the following:
const parsedArgs = vorpal.parse(process.argv, {use: 'minimist'});
vorpal.parse(parsedArgs._)

Before I was running as you were, just passing raw argv and saw the same error. However now I am currently seeing my program execute without actually running any commands although I see no error. :(

Note I've documented my issue more thoroughly over in https://github.com/dthree/vorpal/issues/171


Weaveworks logo in downloaded graph SVG is yuge.

The logo is excessively large, it should be small enough that it is a discrete addition to the image.

Further info & correction. This feature works as expected in Chrome, Firefox.
In Safari, when the button is clicked the browser opens a URL similar to: "blob:http://<IP_ADDRESS>:4040/a57d0b50-82a5-409d-aa79-01f0a7d5d727"
(note the "blob:" prefix)
An example of the result:

The logo is weavescope. Should we mention weaveworks too, @pidster ?

Scope is fine for now, for the record.


[Question] Support for HuggingFace SwinV2

Hello,
Recently support for SwinV2 models released under SwinTransformer v2.0.0  has been added to FT. Would it be possible to add support for Swin models trained with the Transformers library such as swinv2-tiny-patch4-window8-256?
As far as I can tell, these models have slight differences compared to the originally released models such as layer naming:
For the original model:

For the HuggingFace model (classification):

Would it be feasible to apply FT to the latter model and if so, how?
Thanks,
/M

If the model architecture is fully same, you only need to pass the weights of HF in correct order.

Hi,
It seems the HF architecture is slightly different...
However when trying with the pretrained model from here I still get an error:
[ERROR][SwinTransformerWeights::__init__] missing weight layers.0.blocks.0.attn.relative_position_index or layers.0.blocks.0.attn.relative_coords_table or layers.0.blocks.0.attn.cpb_mlp.0.weight or layers.0.blocks.0.attn.cpb_mlp.0.bias or layers.0.blocks.0.attn.cpb_mlp.2.weight.
model is a SwinV2 model:
model = timm.create_model('swinv2_tiny_window8_256', pretrained=True, num_classes=2)

*train*

model.eval()
torch.save({"model":model.state_dict()}, "model.pt")

and I get the above mentioned error when running python builder.py --batch-size 8 --cfg ../../pytorch/swin/Swin-Transformer-Quantization/SwinTransformer/configs/swinv2/swinv2_tiny_patch4_window8_256.yaml --resume ../../../model.pt --th-path ../../../build/lib/libth_transformer.so --version 2 --fp16 --output model.engine
What am I doing wrong?
Thanks!

Do you encounter this error by using the checkpoint https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_tiny_patch4_window8_256.pth? Like running
python builder.py \
    --batch-size 32 \
    --cfg ../../pytorch/swin/Swin-Transformer-Quantization/SwinTransformer/configs/swinv2/swinv2_tiny_patch4_window8_256.yaml \
    --resume ../../pytorch/swin/Swin-Transformer-Quantization/swinv2_tiny_patch4_window8_256.pth \
    --th-path ../../../build/lib/libth_transformer.so \
    --version 2 \
    --fp16 \
    --output swin_transformer_fp16_v2.engine

When using the checkpoint you suggest everything is working .
However the model = timm.create_model('swinv2_tiny_window8_256', pretrained=True, num_classes=2) also uses that exact checkpoint:

For some reason the above mentioned layers get lost during training or is not saved correctly?
Thanks in advance,

When you use create_model, it may change some names of weights, removing some useless weights or add some additional weights.

Would you suggest that I just re initialize and add these back to the state_dict before saving?


Add a template for fish

Since setting the colorscheme for fzf is done by modifying an environment variable, it might be handy to support shells other than Bash (or by extension Zsh). This adds a template for fish.

thank you