Build Fails on wlroots 0.17.2-1

I'm almost sure the crash is due to differing wlroots versions due to things like implicit declaration of function ‘wlr_output_state_set_mode’; did you mean ‘wlr_output_set_mode’?, I cleaned up all old versions of wlroots I could find on my system, I'm not very familiar with meson is there a way to debug which header is actually included? Could also be that wlroots makes breaking changes in minor versions.
$ meson build
The Meson build system
Version: 1.4.0
Source dir: /mnt/2t/dev/cage
Build dir: /mnt/2t/dev/cage/build
Build type: native build
Project name: cage
Project version: 0.1.5
C compiler for the host machine: ccache cc (gcc 13.2.1 "cc (GCC) 13.2.1 20230801")
C linker for the host machine: cc ld.bfd 2.42.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: YES (/usr/bin/pkg-config) 2.1.1
Run-time dependency wlroots found: YES 0.17.2
Run-time dependency wayland-protocols found: YES 1.34
Run-time dependency wayland-server found: YES 1.22.0
Run-time dependency xkbcommon found: YES 1.7.0
Library m found: YES
Program wayland-scanner found: YES (/usr/bin/wayland-scanner)
Program git found: YES (/usr/bin/git)
Build-time dependency scdoc found: YES 1.11.3
Program /usr/bin/scdoc found: YES (/usr/bin/scdoc)
Program sh found: YES (/usr/bin/sh)
Configuring config.h using configuration
Message:
Cage 0.1.5-e7d8780 (branch 'master')

    xwayland: true

Build targets in project: 2

Found ninja-1.11.1 at /usr/bin/ninja

$ ninja -C bulid
ninja.log (apparently github comments have a 65536 character limit)

Hi @bbb651, which version/commit of Cage are you trying to build?

Latest master, commit e7d8780

Seems like there are conflicts between a wlroots install at /usr/ and another one at /usr/local/.

I can't find any more old installations, I searched in /usr:
/usr% fd wlroots
lib/libwlroots.so.12
lib/libwlroots.so
lib/pkgconfig/wlroots.pc
share/licenses/wlroots/

and libwlroots.so.12 is the version managed by pacman, libwlroots.so is a symlink to it and wlroots.pc seems correct, I also searched in $HOME/.local and couldn't find anything.

The logs reveal that both /usr/include/wlr/util/box.h and /usr/local/include/wlr/types/wlr_box.h exist and conflict.

I missed it, sorry for making this issue, thank you so much!


Cost functions in ceres_scan_matcher

We were surprised to see that the scan matching is tied to different poses for rotation and translation each.
In the translational case the previous_pose is used
  problem.AddResidualBlock(
      new ceres::AutoDiffCostFunction<TranslationDeltaCostFunctor, 2, 3>(
          new TranslationDeltaCostFunctor(options_.translation_weight(),
                                          previous_pose)),
      nullptr, ceres_pose_estimate);

While for rotation the initial_pose_estimate (hiding behind ceres_pose_estimate) is used.
  problem.AddResidualBlock(
      new ceres::AutoDiffCostFunction<RotationDeltaCostFunctor, 1, 3>(
          new RotationDeltaCostFunctor(options_.rotation_weight(),
                                       ceres_pose_estimate[2])),
      nullptr, ceres_pose_estimate);

Following the data-flow from local_trajectory_builder:

In case the real_time_correlative_scan_matcher is not used those two poses hold the same values.
In case the real_time_correlative_scan_matcher is used, only the rotation uses the refined pose.

Could you explain what this decision is/was based on?

previous_pose is due to past usage, it is the estimate from extrapolation now.
The interesting case in 2D SLAM without IMU where the rotation estimate can be pretty bad and the real_time_correlative_scan_matcher is used to find a good starting solution. In our experience trusting the orientation of this match is better than the extrapolated orientation estimate since changes in angular velocity can be quite fast. The opposite is true for translations: If the real_time_correlative_scan_matcher finds a better match, e.g. from a repetitive pattern, it is a bad idea to not penalize the change in velocity. If we would use the result from the real_time_correlative_scan_matcher here, huge changes in velocity could be accepted which break SLAM.

Great, thanks! That makes a lot of sense to me now.


devcontainers-cli: init at 0.54.1

Description of changes
Adds devcontainers-cli which allows parsing devcontainers.json into docker containers.
Useful for starting up dev containers within Vim via plugins or other non-vscode pathways.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Priorities
Add a :+1: reaction to pull requests you find important.

Useful for starting up dev containers within Vim via plugins

I am curious, do you have a plugin in mind  ?

Please squash all the review commits into the initial one. The review process for this simple package is not that interesting and just bloats the history.

Done in #301313


CdTe Treharne.yml added

Hi Mikhail,
I've added tabulated data for a CdTe film that our group use on a regular basis.
Kind regards
Rob

Hi Rob,
your data are in the database (I just modified the location of optiwhite glass and AZO and made a couple of minor edits to the yaml files).
http://refractiveindex.info/?shelf=main&book=CdS&page=Treharne
http://refractiveindex.info/?shelf=main&book=CdTe&page=Treharne
http://refractiveindex.info/?shelf=other&book=Al:ZnO&page=Treharne
http://refractiveindex.info/?shelf=glass&book=NSG-multipurpose&page=Pilkington-Optiwhite
Thank you!
Mikhail

Hi Mikhail.
Thanks very much for this.
Rob
Dr Robert Treharne
Centre for Doctoral Training
New and Sustainable Photovoltaics
Web: www.cdt-pv.org
Office: 0151 795 8125
Mobile: 0752 581 8229
Follow us on Twitter! @cdtpv

On 8 Dec 2015, at 03:37, Mikhail Polyanskiy<EMAIL_ADDRESS>wrote:
Hi Rob,
your data are in the database (I just modified the location of optiwhite glass and AZO and made a couple of minor edits to the yaml files).
http://refractiveindex.info/?shelf=main&book=CdS&page=Treharne
http://refractiveindex.info/?shelf=main&book=CdTe&page=Treharne
http://refractiveindex.info/?shelf=other&book=Al:ZnO&page=Treharne
http://refractiveindex.info/?shelf=glass&book=NSG-multipurpose&page=Pilkington-Optiwhite
Thank you!
Mikhail
—
Reply to this email directly or view it on GitHub.


require &Font instead of Font

Several functions requiring ownership of Font only really need a &Font reference.

Oops, I didn't see it was Copy, closing.


Fix missing Screen append on screen name

Seems that we were missing the appending of Screen in the generated screen file.
While it shouldn't cause any big issues, we do append Screen to the entered screen name in the CLI  here and we do it in the 2016 boilerplate...just being consistent ;) 👍

Even more fixed in #37 .


Checking for "currentUser" doesn't work in the localStorage after page refresh

So I have been following the tutorials (http://sahatyalkabov.com/create-a-tv-show-tracker-using-angularjs-nodejs-and-mongodb/, https://hackhands.com/building-instagram-clone-angularjs-satellizer-nodejs-mongodb/#/login-page) and in the header I also check for

Hello {{ currentUser.email }}

Hello {{ currentUser.name }}

When I first start the app, both of these work when the user is logged in. However if I refresh the page, both of them stop working. I double checked and currentUser is still in the localStorage. Any idea what I could be doing wrong?
Thanks!

My first guess would be that the state is lost when you refresh the page. Your Angular controller no longer knows about the currentUser object. It is set during the initial login, but probably does not get re-set again on page refresh by pulling the value from local storage.
Which is why, in the TV Show Tracker example I always fetch the currentUser when factory is loaded, be it a page refresh or an initial load:
$rootScope.currentUser = $cookieStore.get('user');
$cookieStore.remove('user');

So you will need to handle fetching currentUser from local storage yourself in your NavBar controller or one of the factories.

Thanks @sahat, that made sense


SampleWdpClient.UniversalWindows doesn't build

Here's the error output:
1>  Primary reference "WindowsDevicePortalWrapper.UniversalWindows".
1>      Could not find dependent files. Expected file "C:\Users\gershp\Desktop\WindowsDevicePortalWrapper-master\WindowsDevicePortalWrapper\WindowsDevicePortalWrapper.UniversalWindows\bin\Debug\WindowsDevicePortalWrapper.UniversalWindows.dll" does not exist.
1>      Could not find dependent files. Expected file "C:\Users\gershp\Desktop\WindowsDevicePortalWrapper-master\WindowsDevicePortalWrapper\WindowsDevicePortalWrapper.UniversalWindows\bin\Debug\WindowsDevicePortalWrapper.UniversalWindows.dll" does not exist.
1>      Resolved file path is "C:\Users\gershp\Desktop\WindowsDevicePortalWrapper-master\WindowsDevicePortalWrapper\WindowsDevicePortalWrapper.UniversalWindows\bin\Debug\WindowsDevicePortalWrapper.UniversalWindows.dll".
1>      Reference found at search path location "".
1>      The ImageRuntimeVersion for this reference is "".
1>C:\Program Files (x86)\MSBuild\Microsoft\WindowsXaml\v14.0\8.2\Microsoft.Windows.UI.Xaml.Common.targets(264,5): Xaml Internal Error error WMC9999: The system cannot find the path specified. (Exception from HRESULT: 0x80070003)
1>
1>Build FAILED.

My bad, it does build but only if you open the .sln and build that first.


[CI] Expand Build Step

TODO

Closing for now since we'd need to do more purposeful tool updating (set higher node version) + orchestration (way to auto-opt-in on size changes). Potential cooldown project.


[BUG] 1.18 mobs dont get removed and custom items are not droped when they are

mobs not droping items or being removed 1.18
Description
custom mobs when dieing in 1.18 dont drop any items and sometimes remain on the ground and i get a console error
Expected Behavior
Mob Should die and drop custom drop
Actual Behavior
Console Error and mob is not removed and remains dead on the ground
https://pastebin.com/ZmgjQrQd
Possible Fix
no clue
Steps to Reproduce

update mythic mobs on my server
kill a mob
mob remains
console error

Context
It has stoped my server players from getting the essential item that allows them to get custom items and such
Your Environment

rsion used (/version MythicDrops):1.13.1

Environment name and version (/version): 1.18 minecraft

I have several questions regarding this... Also I don't think a bug label is the right choice, since MythicDrops has not been released for 1.18 yet. Anwyays:

Where... does that version command come from? /version MythicDrops ? MythicDrops for 1.17 (latest) is on 8.0.1
I don't see MythicDrops in your plugin list. Were there errors on startup? If it's not installed, how did version return anything?
Step 1 in the reproduction steps seems odd, because this is not MytrhicMobs, but MythicDrops, a different plugin. You sure you're in the right place?

This is a MythicMobs issue, not a MythicDrops issue.

wrong forum idk how i got here sorry


page.client_storage doesn't work in web browser mode

Description

Code example to reproduce the issue:
(paste your program here)

Describe the results you received:

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):
Flet version (pip show flet):
(paste your output here)

Operating system:

Additional environment details:

Please give required details. I can't reproduce the issue

Description
I'm using "page.client_storage" to keep the password saved, once the user type it correctly once.
It works fine in app mode ft.app(target=main). But when I change to web mode, "ft.app(target=main, view=ft.AppView.WEB_BROWSER)" the function stop working.
Code example to reproduce the issue:
senha="dalci9"
valor = pg.page.client_storage.get("key")
myappbar=ft.AppBar(title=ft.Text('Login'),bgcolor='blue')
pg.set_appbar(myappbar)
def clicado(e):

    if campo_senha.value != senha:
        campo_senha.error_text='Senha Inválida'
        campo_senha.value=""
        pg.page.client_storage.remove("key")
    else:
        print(valor)
        campo_senha.error_text=""
        pg.page.client_storage.set("key",senha)
        pg.page.clean()
        pg.navigator.navigate('principal',pg.page,(senha))
    pg.page.update()

Describe the results you received:
On web browser, I have no results. I app mode, the program gives me the saved password
Describe the results you expected:
I  need the program gives me the saved password even on web browser
Additional information you deem important (e.g. issue happens only occasionally):
I think the issue may be that the task of saving the password is with the browser, in web browser mode
Flet version (pip show flet):
Version: 0.18.0
(paste your output here)
Operating system:
windows 10
Additional environment details:

Shouldn't it be page.client_storage.get rather than pg.page.client_storage.get?

@jvalle63, please share repro code we can run to test the issue.

"page.client_storage.get rather than pg.page.client_storage.get?"
Shure. The pg is because I'm using the module flet_navigator
Sorry

But  I testet on a server and seems to be OK..

It should be working. What error you get? Or what you want? Tell everything, so that we can "do" the issue accordingly

It doesn't work only when you call a webpage via Visual Studio Code...


add placholder parameter to work with codemirror placeholder addon

codemirror/codemirror.d.ts
to author(@mihailik). could you review this PR?
:+1: or :-1:?
check list

[ ] pass the Travic-CI test?

Bump

sorry for late. it merged!


[ko] Translate the glossary term 'Ephemeral Container' into Korean

This is a Feature Request

What would you like to be added

Translate the glossary term 'Ephemeral Container' into Korean

link to en entry: https://kubernetes.io/docs/reference/glossary/?fundamental=true#term-ephemeral-container
link to ko entry: https://kubernetes.io/ko/docs/reference/glossary/?fundamental=true#term-ephemeral-container

Why is this needed
Not translated yet
Comments

/language ko
/assign

/triage accepted

Resolved by

#35279 (→ dev-1.24-ko.2)
#35767 (dev-1.24-ko.2 → main)


Remove requirement for /opt mount

THIS DOESN'T WORK - PR FOR DISCUSSION
Currently tests requires /opt to be exported from the headnode and mounted on compute nodes. This is b/c installing s/w on the compute nodes was considered undesirable. However this is pretty odd and doesn't work very nicely with the refactor of the demo repo.
An attempt was made to install into the (already-required cluster shared filesystem) instead using dnf --installroot {{ openhpc_tests_rootdir }}/installs/ rather than yum install (we already require centos8 for tests as we want openhpc v2 for UCX so this is ok).
For the pingpong and pingmatrix tests which use openhpc's openmpi package this also requires:

rewriting the openhpc modulefiles to reference this new path
adding export MODULEPATH="{{ openhpc_tests_rootdir }}/installs/opt/ohpc/pub/modulefiles"

Having done this PATH and LD_LIBRARY_PATH looks right but both srun and mpirun are failing with something like:
-------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    opal_init:startup:internal-failure
But I couldn't open the help file:
    /opt/ohpc/pub/mpi/openmpi4-gnu9/4.0.4/share/openmpi/help-opal-runtime.txt: No such file or directory.  Sorry!

obvs. the help path is a distraction but implies mpirun doesn't know where it's installed. Tried using an absolute path for mpirun as per docs but that's no different.
For the intel-mpi based HPL-* tests $MKLROOT is hardcoded in mklvars.sh script to be:
CPRO_PATH=/opt/intel/compilers_and_libraries_2020.4.304/linux
export MKLROOT=${CPRO_PATH}/mkl

probably the same thing applies to other vars from e.g. mpivars.sh. So all those would need rewriting too. Haven't tried that yet.

Did #16 instead.

Did #16 instead.


Clean up code before git push

PHP CS Fixer need to run first, before code get pushed/committed.

Pre commit hook: https://gist.github.com/rjgamer/3b48de1626e5f39d16779b68a4ab9c83


fix(nginx): increase upstreams shared memory size

Resolves #123.
Increases the shared memory zone size from 64k to 256k.
This should fix problems reported by users on umbrelOS 1.x.

Thanks a ton for this @theborakompanioni! Would you and the team consider releasing an official Docker image with this change so that Raspberry Pi 5 users can run Jam?

Thanks a ton for this @theborakompanioni! Would you and the team consider releasing an official Docker image with this change so that Raspberry Pi 5 users can run Jam?

Yes, of course! I will tag it with v0.2.0-clientserver-v0.9.11-YYYYMMDD-patch (e.g. ``v0.2.0-clientserver-v0.9.11-20240514-patch`) after some more testing (there are some other updates included as well) and will let you know when it is finished.

Thanks, super appreciated!

Thanks, super appreciated!

Currently the automated docker builds are failing (due to network timeouts). Don't know if it is still related to the latest GitHub Actions incident. Will ping you as soon as it works again.

Thanks a ton for this @theborakompanioni! Would you and the team consider releasing an official Docker image with this change so that Raspberry Pi 5 users can run Jam?

Hey @nmfretz! Finally went through. Guess we need some adaptions to the build, as this was an unpleasant experience.
See the docker image here: https://github.com/joinmarket-webui/jam-docker/pkgs/container/jam-dev-standalone/219324914?tag=v0.2.0-clientserver-v0.9.11-patch-20240521
Prefixed with "jam-dev-standalone":
ghcr.io/joinmarket-webui/jam-dev-standalone:v0.2.0-clientserver-v0.9.11-patch-20240521
sha256:6716161d0a5d4514d6744bf8351c7490f3446a7ba52eb736e10ec62ffb0dc491

Do you need any more information? Are you updating the app in umbrel yourself, or should I create a PR? :pray:

amazing, thanks @theborakompanioni!

Do you need any more information? Are you updating the app in umbrel yourself, or should I create a PR? 🙏

Nothing else needed. I'll update Jam now. Thanks again for looking into this.


Create group returning error 500 Internal Server Error, but the group has been created with success

Description
Create group returning error 500 Internal Server Error, but the group has been created with success.
Environment

**WPPConnect version(s): 1.23.0
OS: Ubuntu
Node version: 6.14.17

Steps to Reproduce

send a http request post to /create-group sending the body:
{"name":".......","participants":"55119.....,........,.........",

Log Output
{
"status": "error",
"message": "Error creating group(s)",
"session": "........."
}

tests performed on the latest version and problem if found resolved


[TASK] Add Support for External Dependency Installation Script in Reusable Source Build Pipeline

Description of task
We need to extend the Reusable Source Build workflow to allow users to pass a URL pointing to an external raw Bash script. This script will handle the installation of dependencies not covered by rosdep. This addition will make the workflow more flexible by accommodating custom or organization-specific dependencies without requiring changes to rosdep or the main workflow.
Suggested Workflow

Insert a new step in the build job to check for dependency_script_url (this should be done after the setup ROS 2 environment job)
If dependency_script_url is not empty, the pipeline should do the following:

Download the script from the provided URL.
Mark it as executable.
Execute it, stopping the pipeline if any errors occur.

Specifications

[ ] Input Parameters: dependency_script_url:
Description: URL to a raw Bash script for installing dependencies not found by rosdep.
Required: No
Default: Empty string (indicating no script provided)

Contacts
@kluge7

Basically the purpose of this is if, lets say, a repository has a package, and this package requires a dependency, but rosdep does not have support for installing this. The solution we have come up with is that the developer of that package, will add a bash script to the repo, which will then install this dependency. This is a solution we will use for all of our repositories if needed!


Number of Conv blocks

According to the description of Appendix B.1 of the paper SimVLM, I guess there should be three blocks for the Conv Stage, but I see only 2 BottleneckBlock() in your code. Correct me if I have any misunderstanding. Thank you.

paper description:

"For the Conv stage, we use the first three blocks (excluding the Conv stem) of ResNet-101 and ResNet-152 (He et al., 2016) for our Base and Large models respectively, and a larger variant of ResNet-152 with more channels for the Huge model (matching its hidden dimension size).".

code in SimVLM.py:

self.ResNet=nn.Sequential(*[nn.Conv2d(in_channels=3, out_channels=64, kernel_size=patch_size, stride=patch_size, bias=True),
                                    BottleneckBlock(in_channels=64,out_channels=256,bottleneck_channels=64,),
                                    BottleneckBlock(in_channels=256,out_channels=d_model,bottleneck_channels=128)])

Yes,you are right.


[bluez] Remove filtering by service UUID

Problem
Bluez device filtering by service UUID is done by the following Linux kernel function: https://github.com/torvalds/linux/blob/bdb575f872175ed0ecf2638369da1cb7a6e86a14/net/bluetooth/mgmt.c#L9258. The function requires that devices advertise its services' UUIDs in UUID16/32/128 fields while the Matter specification requires only FLAGS (0x01) and SERVICE_DATA_16 (0x16) fields in the advertisement packets. Hence not all spec-compliant devices can be discovered using the current implementation.
Change overview
Revert filtering by UUID introduced by https://github.com/project-chip/connectedhomeip/pull/9583
Testing
Tested manually with nRF Connect examples which do not advertise UUID16/32/128 fields.

@cktakahasi FYI. To accept filtering by UUID the Matter specification would need to be updated first to require UUID16/32/128 fields.

@Damian-Nordic Thanks! I understood the problem.
Here is advertising data exposed by BlueZ when chip-all-clusters-app runs:
@ MGMT Command: Add Ext Adv Data (0x0055) plen 42                                                                                {0x0001} [hci0] 19.317468
Instance: 1
Advertising data length: 18
16-bit Service UUIDs (complete): 1 entry
Unknown (0xfff6)
Service Data (UUID 0xfff6): 00000f5a23fffe
Flags: 0x05
LE Limited Discoverable Mode
BR/EDR Not Supported
Scan response length: 13
Name (complete): MATTER-3840
I understand that for legacy devices adding to much fields/data to advertising data requires additional logic in the stack to "rotate" and broadcast all EIR data. However it sounds strange add a service data of service not listed at UUID16/32/128 fields.
Is the Matter Working group planning to change the spec to recommend adding the UUID to the list? Considering BT qualification & specs, is it correct to include a Service Data of a not listed service?

@Damian-Nordic Thanks! I understood the problem.
Here is advertising data exposed by BlueZ when chip-all-clusters-app runs:
@ MGMT Command: Add Ext Adv Data (0x0055) plen 42 {0x0001} [hci0] 19.317468
Instance: 1
Advertising data length: 18
16-bit Service UUIDs (complete): 1 entry
Unknown (0xfff6)
Service Data (UUID 0xfff6): 00000f5a23fffe
Flags: 0x05
LE Limited Discoverable Mode
BR/EDR Not Supported
Scan response length: 13
Name (complete): MATTER-3840
I understand that for legacy devices adding to much fields/data to advertising data requires additional logic in the stack to "rotate" and broadcast all EIR data. However it sounds strange add a service data of service not listed at UUID16/32/128 fields.
Is the Matter Working group planning to change the spec to recommend adding the UUID to the list? Considering BT qualification & specs, is it correct to include a Service Data of a not listed service?

@cktakahasi I'm not aware of any ongoing effort to put this requirement in the Matter spec. If the feature is crucial for you, feel free to file a spec issue and describe your use case.
The problem isn't really with legacy devices. It's just up to the implementation to automatically include this data or not. Zephyr Bluetooth Host implementation does not append UUID fields implicitly and that's how I hit on the issue :).
Also, I glanced at the Bluetooth spec supplement and didn't find any requirements regarding the UUID presence in an advertisement packet. Quite the contrary, I found the following sentence:
"An omitted Service UUID data type shall be interpreted as an empty
incomplete-list".


SharedWorkflow connection leak

Feature Request
Desired behaviour
get SharedWorkflows endpoint is using a temporary solution to avoid the connection leak.  Should investigate why the more optimized .filter(workflow -> !workflow.getUsers().contains(user)) causes a leak
[~nperez9] use performance testing framework
┆Issue is synchronized with this Jira Story
┆Issue Type: Story
┆Fix Versions: Dockstore 1.7
┆Sprint: Seabright Sprint 16 Pram
┆Issue Number: DOCK-824

https://github.com/dockstore/dockstore/pull/2629#discussion_r304652725 is worth a try
it does look suspicious to me

@coverbeck for review, maybe this can be hooked up to the performance tester.
Feel free to delegate

Ran curl for sharedworkflow endpoint 1000 times, staging seems to be fine
API - 1.7.0-rc.0
UI - 2.4.0-rc.0


ス○ブラと巨乳を愛する会 (あき)

https://fantia.jp/posts/205729
https://fantia.jp/posts/210318
https://fantia.jp/posts/135591

https://fantia.jp/posts/210318
https://fantia.jp/posts/210318
https://fantia.jp/posts/135591

What's the issue?

@mnao305 Can you download the three-picture code

Hmm, I don't know what the problem is.
If you add the browser extension, you'll see a button, just click on it.
Note that this only works if you have multiple images posted.

@mnao305 yes


Improve components that accept forward iterators and compute the size

Several components in the library still accept forward or bidirectional iterators/iterables and start by computing std::distance(first, last). This is sub-optimal when the container can already provide its size in O(1) such as std::list, which is why some sorters use utility::size to take advantage of it.
For version 1.8.0 we should inspect the library for sorters, adapters and other algorithms that could be improved with direct size information but are not.
Components that could obviously be improved:

[ ] equal_range: seldom used, unclear whether it's worth it.
[x] lower_bound
[x] probe::exc
[x] probe::ham
[x] probe::inv
[x] probe::max
[ ] schwartz_adapter
[x] stable_adapter
[x] upper_bound
[x] vergesort

Components where improvement is more involved if possible at all:

[ ] inplace_merge: I don't know whether the functions that can be improved are ever called.
[ ] introselect: seldom used, unclear whether it's worth it.
[x] probe::dis
[ ] probe::rem
[ ] quicksort

I hope that counted sentinels (#45) will make the situation even better and allow to take advantage if size information even when the algorithm only takes a part of a collection.

probe::dis got some love too because of this issue and the improvement is just ridiculously huge (to the point that the bars for the new speed are almost invisible):

Long story short:  probe::dis was accidentally cubic instead of quadratic for forward and bidirectional iterators. The latest commit made it O(n²) as it was expected to be. I've still got a few ideas to make probe::dis a bit faster again, but none would be that big: issue #85 tracks the measures of presortedness for which the time complexity could be improved according to literature, and Dis(X) is not among them.


[SPARK-17095] [Documentation] [Latex and Scala doc do not play nicely]

What changes were proposed in this pull request?
In Latex, it is common to find "}}}" when closing several expressions at once. SPARK-16822 added Mathjax to render Latex equations in scaladoc. However, when scala doc sees "}}}" or "{{{" it treats it as a special character for code block. This results in some very strange output.

Test build #63920 has started for PR 14688 at commit d50990c.

Test build #63920 has finished for PR 14688 at commit d50990c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63920/
Test FAILed.

The original problem in the JIRA was LaTeX using }}} and that failing. This is a related but opposite problem, were LaTeX is rendered as simple code. @sethah what instance(s) did you see? But I also think it's worth fixing problems like this too. I can see a few more just by searching for common LaTeX strings like \sum. Feel free to fix anything else of this form.

Test build #63922 has started for PR 14688 at commit 5cb09f7.

Test build #63922 has finished for PR 14688 at commit 5cb09f7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63922/
Test FAILed.

Test build #63924 has started for PR 14688 at commit 7c8fc6f.

Test build #63924 has finished for PR 14688 at commit 7c8fc6f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63924/
Test PASSed.

Merged build finished. Test PASSed.

Ya, this is not the problem I had. Here is an example:
\frac{e^{\vec{x}_i^T \vec{\beta}_{K-1}}}
The above will cause the scala doc to barf some weird output at the end because of the terminating }}}. The fix I made temporarily would be:
\frac{e^{\vec{x}_i^T \vec{\beta}_{K-1}}\,}
I do not have other instances of this happening in Spark code (that's not to say it doesn't), but it happened on a PR I am currently working on.

@jagadeesanas2 can you follow up on this to address other instances, including the one you opened the other PR for?

@srowen I do not find any other instances. Perhaps I send some fixes for Latex string.

OK I was going to say we would also have to fix the instance @sethah notes above, but looks like that was already fixed or was fixed before committing it. OK this looks fine; I'll verify the rendering.

Jenkins retest this please

Test build #64192 has started for PR 14688 at commit d358569.

Test build #64194 has started for PR 14688 at commit d358569.

Test build #64194 has finished for PR 14688 at commit d358569.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #64192 has finished for PR 14688 at commit d358569.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64194/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64192/
Test PASSed.

Merged to master


Bump chronic ~> 0.10.0 and update Gemfile.lock

BTW, eventmachine ~> 1.0.3 doesn't work with ruby >= 2.2.0, so this PR can be rather helpful for this issue too.


Minor refactor JDBCMetadataStore

Why are the changes needed?

This is a pure code refactor extracted from https://github.com/apache/kyuubi/pull/4790 to reduce the diff.
How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Please ignore the first commit, this PR should be rebased after https://github.com/apache/kyuubi/pull/5078 get merged

Codecov Report

Merging #5081 (42a7f4c) into master (14818cf) will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff           @@
##           master   #5081   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         564     564
  Lines       31308   31313    +5
  Branches     4098    4097    -1
======================================
- Misses      31308   31313    +5

Impacted Files
Coverage Δ

...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala
0.00% <0.00%> (ø)

...ubi/server/metadata/jdbc/JdbcDatabaseDialect.scala
0.00% <0.00%> (ø)

... and 3 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Thanks, merged to master


Please Explain Encoder Math, How to drive XL Motor about 120° with 50 % speed

Hello,
can you give me an example for the calculaton on how to fill in the command
Example Data send: 0F-00-81-01-11-0C-00-98-06-80-CE-32-64-00-00
When i want to drive XL Motor for +120 / and in secondly example -120°, with speed of 50%
I don't understand calculation examples .
Thanks:
by the way:
Data revieved: 05-00-82-01-05
Data revieved: 05-00-82-01-0C
Code "05", and 0C are not listed in doc , or i haven'f find it ;-)
Marc

Never mind:
Meanwhile i have noticed that your apps using positioning command 0x0d instead:
"Position" simply is desired angle.
It would be nice when you will regroup your documentation my means that you will have chapters / sections describing which commands and Options are available for a specific HUB , drive, …..
The actual try and error and reverse Engineering is very time consuming.


partially wrong code in english and french pages for

Request type

[] Please close this issue, I accidentally submitted it without adding any details
[ ] New documentation
[x] Correction or update

Details
MDN page : input type="date"
In english, please see following part : Handling browser support > Examples > Javascript, code , line 44:
const isLeap = new Date(year, 1, 29).getMonth() === 1;
Français : Gérer la prise en charge des navigateurs > Exemples > 2° bloc de code  ligne 52, même déclaration de constante.
I think this can't work for 2 reasons:

we want to look for februar 29th and not january (wrong month number);
as surprising as it can be, new Date(2021, 2, 29).getMonth() will return 2 even if the date doesn't exist.

I would suggest to replace this line with:
const isLeap = parseInt(selectedYear, 10) % 4 === 0;

I'm pretty sure this isn't wrong. Let's walk through the code in question:
const isLeap = new Date(year, 1, 29).getMonth() === 1
For a start, JS counts from 0, so January is month 0, February is month 1, etc.
Here we saying, if I create a new Date that is February 29th of the current year, if I run getMonth()  on it does it return 1 (February) still? If there are only 28 days in that year's February it will roll over to March the 1st, and the value returned by getMonth() will be 2.
You can test this like so
console.log(new Date(2019, 1, 29).getMonth())
console.log(new Date(2020, 1, 29).getMonth())
2020 is a leap year, so the value returned from the second one will be 2
So we do a test to see if the result is 1, and store it in a constant.
If it is 1, we set the numbers of days to 29 for that month, 28 otherwise:
dayNum = isLeap ? 29 : 28;
Does that help?
I think originally I had something closer to your suggestion, but then I read somewhere that this isn't going to be accurate 100% of the time, so I looked for something more robust.

For a start, JS counts from 0, so January is month 0, February is month 1, etc.

OK, so my mistake comes from there ... as often, wrong explanation of an issue. The code did'nt work but it had to be for another reason then.

I read somewhere that this isn't going to be accurate 100% of the  time

That's right, I did'nt know it : "Every year that is exactly divisible by four is a leap year, except for years that are exactly divisible by 100, but these centurial years are leap years if they are exactly divisible by 400. For example, the years 1700, 1800, and 1900 are not leap years, but the years 1600 and 2000 are." https://en.wikipedia.org/wiki/Leap_year

Does that help?

Certainly ! Thanks and sorry for the wrong report.

The code did'nt work but it had to be for another reason then.

Do you mean my code, or some code your are writing?

Certainly ! Thanks and sorry for the wrong report.

No need to apologise; it's always worth investigating things if they don't seem to work, or don't make sense!

Do you mean my code, or some code your are writing?

I mean my own implementation (and interpretation) of your code in a function which turns a single field "date" into 3 fields "text" (day-month-year), so it is very likely that the issue came from me.

I mean my own implementation...

Ah, OK! Glad things seem to be working OK for your case, but feel free to ping me if you need any help in the future.

Thanks!


Why do I not see a reply message logging for these "Person" actors?

import logging
from thespian.actors import Actor, ActorSystem, ActorTypeDispatcher
from fastcore import utils

class JoinGroup(object):
  def __init__(self, name):
    utils.store_attr()

class SetName(object):
  def __init__(self, name):
    utils.store_attr()

class Greet(object):
  def __init__(self, name, addr):
    utils.store_attr()

class Person(ActorTypeDispatcher):
  def __init__(self, *args, **kw):
      super(Person, self).__init__(*args, **kw)
      self.friends = {}

  def receiveMsg_SetName(self, message, sender):
    self.name = message.name
    logging.debug("My Name is [%s]"%self.name)
    self.send(sender, Greet(self.name, self.myAddress))

  def receiveMessage_Greet(self, message, sender):
    self.friends[message.name] = message.addr
    logging.debug("Hi, %s"%message.name)

class GroupManager(ActorTypeDispatcher):
  def __init__(self, *args, **kw):
      super(GroupManager, self).__init__(*args, **kw)
      self.people = {}

  def receiveMsg_JoinGroup(self, message, sender):
    logging.debug("got msg to join")
    if message.name not in self.people:
      person = self.createActor(Person)
      self.people[message.name] = person
      init = SetName(message.name)
      self.send(person, init)

  def receiveMessage_Greet(self, message, sender):
    for person in self.people.values():
      if sender != person:
        self.send(person, message)

class actorLogFilter(logging.Filter):
    def filter(self, logrecord):
        return 'actorAddress' in logrecord.__dict__

class notActorLogFilter(logging.Filter):
    def filter(self, logrecord):
        return 'actorAddress' not in logrecord.__dict__

logcfg = { 'version': 1,
           'formatters': {
               'normal': {'format': '%(levelname)-8s %(message)s'},
               'actor': {'format': '%(levelname)-8s %(actorAddress)s => %(message)s'}},
           'filters': { 'isActorLog': { '()': actorLogFilter},
                        'notActorLog': { '()': notActorLogFilter}},
           'handlers': { 'h1': {'class': 'logging.StreamHandler',
                                'stream': 'ext://sys.stdout',
                                'formatter': 'normal',
                                'filters': ['notActorLog'],
                                'level': logging.DEBUG},
                         'h2': {'class': 'logging.StreamHandler',
                                'stream': 'ext://sys.stdout',
                                'formatter': 'actor',
                                'filters': ['isActorLog'],
                                'level': logging.DEBUG},},
           'loggers' : { '': {'handlers': ['h1', 'h2'], 'level': logging.DEBUG}}
         }

def run():
  asys = ActorSystem(systemBase='multiprocQueueBase', logDefs=logcfg)
  group = asys.createActor(GroupManager)
  asys.tell(group, JoinGroup('Buffalo Bill'))
  asys.tell(group, JoinGroup('Norman Bates'))
  asys.tell(group, JoinGroup('Patrick Bateman'))

if __name__ == "__main__":
  run()

Execution output:
❯ python3 tmp.py
DEBUG    ActorAddr-Q.ThespianQ.b => got msg to join
DEBUG    ActorAddr-Q.ThespianQ.b => got msg to join
DEBUG    ActorAddr-Q.ThespianQ.b => got msg to join
DEBUG    ActorAddr-Q.ThespianQ.b.a => My Name is [Buffalo Bill]
DEBUG    ActorAddr-Q.ThespianQ.b.b => My Name is [Norman Bates]
DEBUG    ActorAddr-Q.ThespianQ.b.c => My Name is [Patrick Bateman]

I expected there to be a "Hi, {name}" logged by each "person", but it is like the msg is never sent/received.

Clarify, "each person"; I mean every actor already in the group. Say 1 person joins, there would be no "hi" from a group with a single actor. N-1 "hi" for N actors.

Clarify, "each person"; I mean every actor already in the group. Say 1 person joins, there would be no "hi" from a group with a single actor. N-1 "hi" for N actors.

🤦🏻  receiveMsg_Greet vs receiveMessage_Greet
I overlooked it everytime.

🤦🏻  receiveMsg_Greet vs receiveMessage_Greet
I overlooked it everytime.


use networkx to fix routing

@sebastian-goeldi

Hmm, but this only works in  the case of single (and somewhat isolated) port location, not for the general case of having a component with many ports in different directions, does it?

yes, it works for some cases,
marking it as draft


Unifying recurse semantics across backends

Our backends don't agree on recurse semantics when multiple paths to the same output are inolved:

Our Match backend would represent each result once, even though it was reached multiple times by different paths.
Our SQL backend would duplicate the output row once for each path
Our Neo4j backend would find all different paths that use each edge at most once, and duplicate the result for each one.

My proposal is to deduplicate the result in all backends, and (eventually) add a compiler flag called "skip_recurse_deduplication" to be used when the user does not want to pay the performance price of this deduplication. Below I talk about this proposal from multiple perspectives.
Backend support:
If we want unified semantics across backends, all backends should be able to implement this. This is the easiest kind of recurse to implement across backends, because it's hard to make OrientDB or Neo4j reduplicate their deduplicated results without significant performance drawbacks and engineering effort.
Performance:
Deduplication is an additional operation to be performed, which might be costly. That's why I propose the "skip_recurse_deduplication" flag. It is standard compiler practice to have a default behavior that's easy to use, and give the user an option to get more performance by paying with well documented side effects that break compliance. See for example gcc's option --ffast-math, which increases float operation performance, sacrificing associativity.
Future proofing:
We might want to add fields similar to _x_count, called _x_path, or _x_depth, which would return for each result of the recursion, the exact path taken, or the length of that path. If we deduplicate all different paths, these fields would have to return lists, effectively "folding" the result. However, folding is only unwanted because it removes any information about column relationships, when a tag-filter pair is used. This is not a problem because you can't tag or filter the intermediate vertices along the path anyway.


Nunjucks macro paths within nunjucks code snippets

The import path for the Nunjucks macros are incorrect, it expects the macro template to be in the folder /components/component-name/ eg. {% from 'components/action-link/macro.njk' import actionLink %} however this requires users to have configured their application to read files from the /node_modules/nhsuk-frontend/packages/ folder when referencing /components/component-name/.
The configuration for this needs documenting or there needs to be some guidance telling users how to import the macros from a different directory.
The paths are also incorrect for the nhsuk-prototype-kit they are imported with {% from 'action-link/macro.njk' import actionLink %} and do not require the /components/ prefix.

This was fixed in release 1.0.2


msgbacklog is not accurate between shutdowns

Describe the bug
the msgBacklog is not accurate after restarts and request msgBacklog for subscription even specify --get-precise-backlog
To Reproduce
Steps to reproduce the behavior:

get msgBacklog by 'pulsar-admin topics partitioned-stats --get-precise-backlog --get-subscription-backlog-size "
alias ts="pulsar-admin topics stats --get-precise-backlog --get-subscription-backlog-size'
restart broker
request for backlog again, or consume msg in a particular subscirption
we can consume more message than the 'msgBacklog'

Expected behavior
the msgBacklog can indicate the precise backlog in a subscription in any case.

What is your Pulsar version? I didn't reproduce the behavior in v2.9.2

@Radiancebobo I am using 2.10.0

@lujiwen pulsar count number of entries. Did you enabled batching in producer side?

@yapxue yes I did


Bypass interactive approval for column RENAME detection

I'm getting stuck on the interactive approval for column RENAME detection for an atlas schema apply . . . --auto-approve invocation in a automated deployment context
I see this feature was added in v 0.22 - https://atlasgo.io/blog/2024/05/01/atlas-v-0-22#rename-detection
When atlas detects a column rename, it interactively opens the following prompt:
? Did you rename "<table>" column from "<old>" to "<new>":
  ▸ Yes
    No

I figured that the --auto-approve flag would bypass this, but apparently not
Is there any way to bypass this and automatically apply detected column renames?

Hi @acaruso7,
Thanks for reaching out. We just pushed a fix that should resolve this for situations where the user isn't connected to a TTY.
Can you try again with the most recent version of Atlas?
curl -sSf https://atlasgo.sh | sh

Hi @rotemtam I just tried again on version v0.25.1-965bcf5-canary (installed from above command), but it's still interactively asking for approval
My atlas command:
atlas schema apply --url 'postgresql://postgres:admin@db/app?search_path=app&sslmode=disable' --to 'file://schema.sql' --dev-url 'postgresql://postgres:admin@db/atlas?search_path=app&sslmode=disable' --auto-approve

The fix was for non interactive environments, such as CI pipelines. Do you get that there as well?

Yes still getting stuck on the same in a CI environment (GitHub Actions invoking kubernetes migration job runners)

Hey! The issue was fixed in the latest build. Please, upgrade Atlas and report if you still face issues with this.


PPO Atari performance

For atari training,
i run ppo as mpirun -np 8 python -m baslines.ppo1.run_atari or simply run run_atari.py and
I see the result by average 100 episode reward mean
whether i changed clippling reward or vf clipping as old version, or not,
the score of atari game do not increase like paper, so why they are different?
How do i get same atari results in ppo?

The printouts are reporting the clipped reward, which might not be the same as the actual reward. look at the monitor.csv files instead.


Implement 1 Camera SLAM for Underwater Testing

Competition this year has a lot of 3D Mapping tasks. Using an already built tool will prevent me from losing more hair.
Team is undecided between attempting ROS2 Camera-only SLAM or using a harder tool such as LIDAR. Throwing a camera underwater and seeing how it behaves is the easiest way of determining if distortion due to waterproofing and strange lighting effects due to surface perturbations actually matters for 3D mapping.

To speed this along, I've gotten permission from a company I used to work at to use the VSLAM code I helped to implement while I was there. This is, of course, proprietary code, so we shouldn't go flinging it around to aggressively. That code has been mirrored to here: https://github.com/Colorado-School-of-Mines-Robotics-Club/ExternalVslam
I'm now working on tearing into that code and making a number of modifications to it, including making it run on our Ubuntu18/ROS2 Foxy platform and having it use Jetson acceleration. It currently runs purely on the CPU, which that Nano is almost certainly not equipped to handle.
The ORB feature extraction step is particularly low-hanging fruit for GPU acceleration.

Actually, I might hold off on porting/implementing everything until we've done a proof-of-concept with the unported VSLAM running on a laptop and the camera moving around in a pool.
Need to see if the thing will  have enough tracking data and be tolerant to the funky underwater lighting.


"no buildable Go source files"

Following the guide and getting and error when I run:
go get github.com/hyperledger/fabric/core/chaincode/shim
The error is:
src/github.com/hyperledger/fabric/core/crypto/attributes/attributes.go:28:2: no buildable Go source files in /Users/jp/go/src/github.com/hyperledger/fabric/core/crypto/attributes/proto
I have version 1.6:
$ go version

go version go1.6 darwin/amd64

Go works when I test out the hello world app:
$ $GOPATH/bin/hello

hello, world

Really unsure of what is needed. I am running on Mac OS X.

I met the same error and y env is Win7 and go v1.6.3.
When I try on Centos( Linux), this error disappears... ;)
But shows another issue "Can't build code due to Chaincode interface change". ;(

@johnnyensign  - you might have run into the issue where we were had removed the generated protobuf code in the master branch.  This was resolved so you can likely try again

Closing due to inactivity.  I have verified that the code builds properly if you clone the v0.5 release linked below into the $GOPATH/src/hyperledger/ folder on your machine.
https://github.com/hyperledger-archives/fabric/tree/v0.5-developer-preview

Its not working now in ubuntu. Can somebody help?
go version go1.6.2 linux/amd64

Is not working, followed tutorial to the T with go1.6.2 osx
Can anyone advise? https://console.ng.bluemix.net/docs/services/blockchain/ibmblockchain_tutorials.html

@deefalcon - what error are you seeing?

I have same error on both v0.5 and v0.6


Add class name and current time to tensorboard log dir

Proposed changes:
As I explained here #10183, I add class name and current time to tensorboard log dir.

...

Status (please check what you already did):

[ ] added some tests for the functionality
[ ] updated the documentation
[ ] updated the changelog (please check changelog for instructions)
[ ] reformat files using black (please check Readme for instructions)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@cep-ter thanks for proposing this! Sorry for the delay, I've now looked properly at what you're proposing and also why (and since when) we don't log TB information separately for each model. I'll have a think about the way you're solving the issue and will push this PR a step further 🙂

Right, so it looks like we removed the model- and run-specific grouping of TensorBoard logs in this PR and dump all logs into one directory. If I'm honest, I don't see why. I tried out the current approach and @cep-ter is right that it makes it hard to compare models or even just know which logs come from where. I think the reason for these problems is that, for a long time after the big e2e refactor, TensorBoard didn't really work with rasa, so we couldn't test things and make sure that the logging was done in the most sensible way (@dakshvar22 correct me if I remember something incorrectly).
In any case, I agree that we should separate TB logs by model name and timestamp like we did previously and like @cep-ter is proposing. However, I think this should be implemented in only one place -- rasa.utils.train_utils.create_common_callbacks -- and not in individual models like DIET or TED. @cep-ter could you make this change?
I've also dropped this PR into our Research inbox so that the effort is visible, someone takes this on fully and together with @cep-ter drives this to completion 🙂

I think the reason for these problems is that, for a long time after the big e2e refactor, TensorBoard didn't really work with rasa, so we couldn't test things and make sure that the logging was done in the most sensible way

The graph logging didn't work and might not be working even now but the metric logging should have worked always. Anyways, I agree that segregating the logs with timestamp is much better.

Hi there! Thanks for your contribution. As the PR has been stale for a while, I'm closing it. Please feel free to reopen an issue / PR if the problem still occurs on latest rasa versions.


Support for DateTimeFormat for RequestParam of type DateTime

I am trying to move from Spring FeignClient to ReactiveFeignClient. The service I'm trying to consume expects dates as request parameters in format "yyyy-MM-dd".
With FeignClient I was able to achieve by passing joda time DateTime object and using DateTimeFormat("yyyy-MM-dd") to format the date before sending the request.
But in ReactiveFeignClient this annotation is somehow ignored and the DateTime object is passed in default format.
I also tried using a formatter to achieve this transformation and providing it as part of configuration. But I could not see a way of using Formatters with ReactiveFeignClient and the one with Feign does not seem to work.
Is there anything I've missed? I cannot seem to find any examples related to this.
Dependencies used
org.springframework.cloud:spring-cloud-dependencies:2020.0.1
com.playtika.reactivefeign:feign-reactor-spring-cloud-starter:2.0.26
com.playtika.reactivefeign:feign-reactor-spring-configuration
com.playtika.reactivefeign:feign-reactor-webclient
org.springframework.boot:spring-boot-starter-webflux

Original Code with Feignclient
@DateTimeFormat(pattern = "yyyy-MM-dd")
        @RequestParam("startDate") startDate: DateTime,

Trial with FeignFormatter
Configuration
@Bean
    fun registerVehicleServiceLocaleConverter(): FeignFormatterRegistrar {
        return FeignFormatterRegistrar {
            it.addConverter(DateTimeConverter())
        }
    }

Converter:
class DateTimeConverter : Converter<DateTime, String> {
    override fun convert(source: DateTime): String {
        return DateTimeFormat.forPattern("yyyy-MM-dd").print(source)
    }
}

Didn't manage to find if OpenFeign client supports DateTimeFormat annotation.
Just found the link for workaround
https://stackoverflow.com/questions/60380148/feignclient-is-changing-the-format-of-localdate-passed-to-it

Will appreciate if somebody point me how it's implemented in regular Feign client

In a non-reactive Feign, below works

@Configuration
class FeignConfig : FeignFormatterRegistrar {

    override fun registerFormatters(registry: FormatterRegistry) {
        registry.addFormatter(DateFormatter())
    }
}

class DateFormatter : Formatter<LocalDate> {

    internal var formatter = SimpleDateFormat("yyyy-MM-dd")

    @Throws(ParseException::class)
    override fun parse(text: String, locale: Locale): LocalDate {
        val d = formatter.parse(text)
        val zdt = d.toInstant().atZone(ZoneId.systemDefault())
        return zdt.toLocalDate()
    }

    override fun print(localDate: LocalDate, locale: Locale): String {
        val date = Date.from(localDate.atStartOfDay(ZoneId.systemDefault()).toInstant())
        return formatter.format(date)
    }
}


[SPARK-12875] [ML] Add Weight of Evidence and Information value to Spark.ml as a feature transformer

jira: https://issues.apache.org/jira/browse/SPARK-12875
As a feature transformer, WOE and IV enable one to:

transform categorical values into continuous values for further training, especially when the feature has many different values and oneHotEncoder becomes impractical, e.g. district, types of goods, age etc.
Consider each variable’s independent contribution to the outcome.
Detect linear and non-linear relationships.
Rank variables in terms of "univariate" predictive strength.
Visualize the correlations between the predictive variables and the binary outcome.

http://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/ gives a good introduction to WoE and IV.
The Weight of Evidence or WoE value provides a measure of how well a grouping of feature is able to distinguish between a binary response (e.g. "good" versus "bad"), which is widely used in grouping continuous feature or mapping categorical features to continuous values. It is computed from the basic odds ratio:
(Distribution of positive Outcomes) / (Distribution of negative Outcomes)
where Distr refers to the proportion of positive or negative in the respective group, relative to the column totals.
The WoE recoding of features is particularly well suited for subsequent modeling using Logistic Regression or MLP.
In addition, the information value or IV can be computed based on WoE, which is a popular technique to select variables in a predictive model.
Next: Currently we support only calculation for categorical features. Add an estimator to estimate the proper grouping for continuous feature.

Test build #49588 has started for PR 10803 at commit 8bb38ab.

Test build #49588 has finished for PR 10803 at commit 8bb38ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49588/
Test PASSed.

Test build #49858 has started for PR 10803 at commit 762e091.

Test build #49858 has finished for PR 10803 at commit 762e091.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49858/
Test PASSed.

Test build #3027 has started for PR 10803 at commit 762e091.

Test build #3027 has finished for PR 10803 at commit 762e091.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Closing stale PR.

No it's not merged. Feel free to use the code as you wish.

Hi yuo,
i have used this class in my code but it not given the output as i expected. the value which it produces is not continuous
val startTimeMillis = System.currentTimeMillis()
val s1 = System.nanoTime()
import org.apache.spark.sql.types.{ StringType, DoubleType, IntegerType}
import org.apache.spark.mllib.classification.{LogisticRegressionModel, LogisticRegressionWithLBFGS}
import org.apache.spark.mllib.evaluation.MulticlassMetrics
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.ml.feature.{ Tokenizer, HashingTF, IDF }
import org.apache.spark.sql
import org.apache.spark.rdd.RDD
import org.apache.spark.annotation.Since
import org.apache.spark.ml.Transformer
import org.apache.spark.ml.feature.{ VectorAssembler, StandardScaler }
import org.apache.spark.ml.param.{Param, ParamMap, ParamPair, Params}
import org.apache.spark.ml.util.Identifiable
import org.apache.spark.sql.{DataFrame, Dataset, Column}
import org.apache.spark.sql.functions._
import org.apache.spark.ml.classification.RandomForestClassifier
import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
import org.apache.spark.ml.tuning.CrossValidator
import org.apache.spark.ml.Pipeline
import org.apache.spark.mllib.linalg.{ Vector, Vectors, SparseVector }
import scala.util.parsing.combinator._
import org.apache.spark.ml.feature._
import sqlContext.implicits._
import org.apache.spark.ml.feature.Bucketizer
var dataDf = spark.sql("select (.75 * longacc_0to1point5 + 2.25 * longacc_1point5to3 + 3.75 * longacc_3to4point5 + 5.25 * longacc_4point5to6 + 6.75 * longacc_6to7point5 + 8.25 * longacc_7point5to9 + 9.75 * longacc_9to10point5 + 11.25 * longacc_10point5to12 + 12.75 * longacc_12to13point5 + 14.5 * longacc_13point5)/( longacc_0to1point5 + longacc_1point5to3 + longacc_3to4point5 + longacc_4point5to6 + longacc_6to7point5 + longacc_7point5to9 + longacc_9to10point5 + longacc_10point5to12 + longacc_12to13point5 + longacc_13point5) as Long_Acc, ((5 * Speed_FLM_0to10 + 15 * Speed_FLM_10to20 + 25 * Speed_FLM_20to30 + 40 * Speed_FLM_30to50 + 65 * Speed_FLM_50to80 + 100 * Speed_FLM_80to120+135 * Speed_FLM_120to150 + 175 * Speed_FLM_150to200 + 225 * Speed_FLM_200toInf)/( Speed_FLM_0to10 + Speed_FLM_10to20 + Speed_FLM_20to30+ Speed_FLM_30to50 + Speed_FLM_50to80 + Speed_FLM_80to120 + Speed_FLM_120to150 + Speed_FLM_150to200 + Speed_FLM_200toInf)) as Speed_FLM, ((1 * dist_0to2km + 3.5 * dist_2to5km + 7.5 * dist_5to10km + 15 * dist_10to20km + 35 * dist_20to50km + 100 * dist_50to150km + 175 * dist_150to500km + 600 * dist_500km)/( dist_0to2km + dist_2to5km + dist_5to10km + dist_10to20km + dist_20to50km + dist_50to150km + dist_150to500km + dist_500km)) as Dist, ((5 * Pedal_0to10perc + 15 * Pedal_10to20perc + 25 * Pedal_20to30perc + 35 * Pedal_30to40perc + 45 *  Pedal_40to50perc + 55 * Pedal_50to60perc + 65* Pedal_60to70perc + 75 * Pedal_70to80perc + 85 *  Pedal_80to90perc + 95 *  Pedal_90perc)/ ( Pedal_0to10perc +  Pedal_10to20perc +  Pedal_20to30perc +  Pedal_30to40perc +  Pedal_40to50perc +  Pedal_50to60perc +  Pedal_60to70perc +  Pedal_70to80perc +  Pedal_80to90perc +  Pedal_90perc)) as Pedal, ((1 * time_1min + 3 *  time_1to5min + 7.5 * time_5to10min + 20 * time_10to30min + 45 *  time_30to60min + 90 * time_60to120min + 210 *  time_120to300min+400 *  time_300min)/( time_1min +  time_1to5min +  time_5to10min + time_10to30min + time_30to60min +  time_60to120min +  time_120to300min +  time_300min)) as Time, ((1 * Pauses_less_1min + 5Pauses_1_10min+20Pauses_10_30min + 45 * Pauses_30_60min + 90Pauses_60_120min + 210 * Pauses_120_300min + 3750 * Pauses_300_24hr + 5040 * Pauses_24hr_1week + 21600 * Pauses_1week_1month+ 43200Pauses_greater_30days)/(Pauses_less_1min+ Pauses_1_10min+Pauses_10_30min + Pauses_30_60min + Pauses_60_120min + Pauses_120_300min + Pauses_300_24hr + Pauses_24hr_1week + Pauses_1week_1month + Pauses_greater_30days)) as Pauses, ((750 * Speed_RPM_1000 + 1250 * Speed_RPM_1000to1500+1750 * Speed_RPM_1500to2000 + 2250 * Speed_RPM_2000to2500+2750 * Speed_RPM_2500to3000+3250 * Speed_RPM_3000to3500 + 4750 * Speed_RPM_3500to4000 + 4250 * Speed_RPM_4000to4500 + 4750 * Speed_RPM_4500to5000+5250 * Speed_RPM_5000to5500 + 5750 * Speed_RPM_5500to6000 + 6250 * Speed_RPM_6000to6500 + 6750 * Speed_RPM_6500to7000 + 7250 * Speed_RPM_7000)/( Speed_RPM_1000 + Speed_RPM_1000to1500 + Speed_RPM_1500to2000 + Speed_RPM_2000to2500 + Speed_RPM_2500to3000 + Speed_RPM_3000to3500 + Speed_RPM_3500to4000 + Speed_RPM_4000to4500 + Speed_RPM_4500to5000 + Speed_RPM_5000to5500 + Speed_RPM_5500to6000 + Speed_RPM_6000to6500 + Speed_RPM_6500to7000 + Speed_RPM_7000)) as Speed_RPM, Speed_ReverseDirection, Speed_FLM_Standstill, Distance_Per_day, Starts_Per_Day, Driving_Time_Per_Day, Target from fisher_vin_data")
dataDf = dataDf.na.fill(0)
println("Needed Output 1 ----> " + dataDf.count())
def getMaximumValue(x: Column) = dataDf.agg(max(x)).collect().map(r => r.toSeq(0).asInstanceOf[Double]).toList(0)
def getMinimumValue(x: Column) = dataDf.agg(min(x)).collect().map(r => r.toSeq(0).asInstanceOf[Double]).toList(0)
def Normalize(x: Column): Column = {
val max_x = getMaximumValue(x.cast("double"))
val min_x = getMinimumValue(x.cast("double"))
return (x.cast("double") - min_x)/ (max_x - min_x)
}
dataDf = dataDf.withColumn("Long_Acc", Normalize(dataDf.col("Long_Acc"))).withColumn("Speed_FLM", Normalize(dataDf.col("Speed_FLM"))).withColumn("Dist", Normalize(dataDf.col("Dist"))).withColumn("Pedal", Normalize(dataDf.col("Pedal"))).withColumn("Time", Normalize(dataDf.col("Time"))).withColumn("Pauses", Normalize(dataDf.col("Pauses"))).withColumn("Speed_RPM", Normalize(dataDf.col("Speed_RPM"))).withColumn("Speed_ReverseDirection", Normalize(dataDf.col("Speed_ReverseDirection"))).withColumn("Speed_FLM_Standstill", Normalize(dataDf.col("Speed_FLM_Standstill"))).withColumn("Distance_Per_day", Normalize(dataDf.col("Distance_Per_day"))).withColumn("Starts_Per_Day", Normalize(dataDf.col("Starts_Per_Day"))).withColumn("Driving_Time_Per_Day", Normalize(dataDf.col("Driving_Time_Per_Day"))).withColumn("Target", Normalize(dataDf.col("Target")))
//------------------------------------------------------
trait HasInputCol extends Params {
final val inputCol: Param[String] = new Param[String](this, "inputCol", "input column name")
final def getInputCol: String = $(inputCol)
}
trait HasLabelCol extends Params {
final val labelCol: Param[String] = new Param[String](this, "labelCol", "label column name")
setDefault(labelCol, "label")
final def getLabelCol: String = $(labelCol)
}
trait HasOutputCol extends Params {
final val outputCol: Param[String] = new Param[String](this, "outputCol", "output column name")
setDefault(outputCol, uid + "__output")
final def getOutputCol: String = $(outputCol)
}
class WeightOfEvidence(override val uid: String) extends HasInputCol with HasLabelCol with HasOutputCol {
def this() = this(Identifiable.randomUID("woe"))
def setInputCol(value: String): this.type = set(inputCol, value)
def setLabelCol(value: String): this.type = set(labelCol, value)
def setOutputCol(value: String): this.type = set(outputCol, value)
def transform(dataset: DataFrame): DataFrame = {
//validateParams()                  --------------***** Important *******-----------
val sorted_dataset = dataset.sort($(inputCol))
val woeTable = WeightOfEvidence.getWoeTable(sorted_dataset, $(inputCol), $(labelCol))
val woeMap = woeTable.map(r => {
val category = r.getAsString
val woe = r.getAsDouble
(category, woe)
}).rdd.collectAsMap
val trans = udf { (factor: String) =>
  woeMap.get(factor)
}
dataset.withColumn($(outputCol), trans(col($(inputCol))))

}
override def copy(extra: ParamMap): VectorAssembler = defaultCopy(extra)
}
object WeightOfEvidence {
def getWoeTable(dataset: DataFrame, categoryCol: String, labelCol: String): DataFrame = {
val data = dataset.select(categoryCol, labelCol)
val tmpTableName = "woe_temp"
data.createOrReplaceTempView(tmpTableName)
val err = 0.01
val query =
  s"""
     |SELECT
     |$categoryCol,
     |SUM (IF(CAST ($labelCol AS DOUBLE)=1, 1, 0)) AS 1count,
     |SUM (IF(CAST ($labelCol AS DOUBLE)=0, 1, 0)) AS 0count
     |FROM $tmpTableName
     |GROUP BY $categoryCol
    """.stripMargin
val groupResult = data.sqlContext.sql(query).cache()

val total0 = groupResult.selectExpr("SUM(0count)").first().getAs[Long](0).toDouble
val total1 = groupResult.selectExpr("SUM(1count)").first().getAs[Long](0).toDouble
groupResult.selectExpr(
  categoryCol,
  s"1count/$total1 AS p1",
  s"0count/$total0 AS p0",
  s"LOG(($err + 1count) / $total1 * $total0 / (0count + $err)) AS woe")

}
}
import org.apache.spark.sql.Row
//  val test = dataDf.select("Speed_FLM").rdd.map {
//        case Row(string_val: String) => (string_val, functionApplying(string_val, dataDf))
//    }.toDF("Speed_FLM", "Speed_FLM_1")
// test.show()
var dataDf1 = dataDf.withColumn("Speed_FLM" , dataDf("Speed_FLM").cast(StringType) ).withColumn("Long_Acc" , dataDf("Long_Acc").cast(StringType) ).withColumn("Dist" , dataDf("Dist").cast(StringType) ).withColumn("Pedal" , dataDf("Pedal").cast(StringType) ).withColumn("Time" , dataDf("Time").cast(StringType) ).withColumn("Speed_RPM" , dataDf("Speed_RPM").cast(StringType) ).withColumn("Speed_ReverseDirection" , dataDf("Speed_ReverseDirection").cast(StringType) ).withColumn("Speed_FLM_Standstill" , dataDf("Speed_FLM_Standstill").cast(StringType) ).withColumn("Distance_Per_day" , dataDf("Distance_Per_day").cast(StringType) ).withColumn("Starts_Per_Day" , dataDf("Starts_Per_Day").cast(StringType) ).withColumn("Driving_Time_Per_Day" , dataDf("Driving_Time_Per_Day").cast(StringType) )
dataDf1.show()
val splits = Array(Double.NegativeInfinity, -12.5, 0.0, 12.5, Double.PositiveInfinity)
var jk = new WeightOfEvidence().setInputCol("Speed_FLM").setOutputCol("Weight_Of_Evidence_Speed_FLM").setLabelCol("Target").transform(dataDf1)
var bucketizer: Bucketizer = new Bucketizer().setInputCol("Weight_Of_Evidence_Speed_FLM").setOutputCol("Speed_FLM" + "_Target").setSplits(splits)
dataDf = bucketizer.transform(jk)
here is what i do im trying to replicate woe binning in R

i have attached my output file.
export (8).zip

Why is this not merged?

Why is this not merged?


Change style of Bodies

I am trying to create a stack of Circles, that are filled blue. However I can't get rid of the stroke or set a new fill.
This is my code:
var width = window.innerWidth,
     height = window.innerHeight;

var render = Render.create({
    element: document.getElementById('wrapper'),
    engine: engine,
    width: width,
    height: height,
    wireframes: false
});

render.canvas.width = width;
render.canvas.height = height;
render.options.wireframeBackground = 'transparent';
render.options.background = 'transparent';

Render.run(render);

// x, y, amt x, amt y, gap x, gap y
var stack = Composites.stack(0, 0, 20, 10, 10, 10, function(x, y) {
    return Bodies.circle(x, y, 6, {
    	friction: 0.1 ,
    	fillStyle: "0045FF",
    	render: {
             lineWidth: 0
    	}
    });
});

Try this:
  render: {
    fillStyle: 'red'
  }

Yes, works, thanks! Also had to add:
render.options.wireframes = false;
It is kinda weird. It seems like when I create the render the options are not being accepted and I have to override them after creation.

Ah yes that's because you need to put that into options when creating:
var render = Render.create({
    element: document.getElementById('wrapper'),
    engine: engine,
    width: width,
    height: height,
    options: {
      wireframes: false
    }
});

hi


Add semantic block validation during the ZIP-212 grace period after Canopy activation

Motivation
We want to fully validate Canopy blocks and abide by the ZIP-212 grace period where note plaintext lead bytes are allowed to be either 0x01 or 0x02 so that Regtest is as close to the Mainnet consensus rules as possible. Prior to Canopy activation those lead bytes must be 0x01, and after the grace period, they must be 0x02.
This rule only applies to nodes and not wallets, and librustzcash may only allow for lead bytes of 0x02 after Canopy activation, so Zebra is currently mandating checkpoint validation until the end of the ZIP-212 grace period.
Possible Design
The grace period after Canopy should only apply until the next network upgrade (NU5) is activated, so it should be the lesser of Canopy Activation Height + ZIP_212_DURATION or NU5 Activation Height
Testing

Ensure that Zebra can validate transactions with note plaintexts that have lead bytes of 0x01 as well as 0x02 during the ZIP-212 grace period
Ensure that Zebra will only validate transactions with note plaintexts that have lead bytes of 0x02 after the ZIP-212 grace period

Both the latest version of zcash_primitives and the version Zebra currently uses (0.13.0) allow for plaintext lead bytes of 0x01 or 0x02 during the grace period, see:

latest version
0.13.0

Let's wait until Zebra is using zcash_primitives v0.15.0 and then pass Zip212Enforcement::On

depends-on: #8540


Fix inconsistencies in lang.xml:strip()

Purpose

$title

Added tests for #29954
Fixes #24200
Check List

[ ] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

As the function comment suggests, I think we need to check for the white spaces [1] and skip them.
[1] - w3.org/TR/xml/#sec-white-space

Oh my bad, this is on the old function doc, in the current spec it doesn't say this.

Plese take a look at https://github.com/ballerina-platform/ballerina-spec/issues/834

As per the clarification, I have removed the extra code that was put to look inside xml elements

Changes looks good to me


🐛 BUG: few string keywords show TS errors when imported

What version of astro are you using?
1.0.0
Are you using an SSR adapter? If so, which one?
None
What package manager are you using?
npm
What operating system are you using?
Mac
Describe the Bug
Hey, not completely sure if this is the place to have this fixed, but I am having this weird issue at VSCode where some components show TS errors when imported (see below) if they have some specific keywords within the HTML. The error is from TS: File '<path_goes_here>' is not a module.ts(2306)
https://user-images.githubusercontent.com/32722432/181135354-06e5c0c2-0741-43fe-97be-b490ee24abcd.mov
I've tested few JS specific keywords and the only one that seems to trigger such issue is the foras you can see at the video above. Not sure if this happens with some other framework-specific keywords as well, not using any at this project.
I believe this is some VSCode default config for TS or something like that (I am using the Astro extension by the way), but I thought it would be worth it to open an issue since this can possibly be a bug. If it's not, please feel free to close this and/or point me to the right place/resource so I can fix this - it's getting really annoying.
How to reproduce
You can try to reproduce that by creating a new project using create-astro as the docs says and select the minimal template option, then create a component and import it for another .astro file
Link to Minimal Reproducible Example
none
Participation

[ ] I am willing to submit a pull request for this issue.

Hello!
This issue is caused by a bug in our parsing logic inside the extension, we are currently working on using a new, better parser that should fix this issue. I'm sorry for the inconvenience!
For another issue on the same subject, see this one https://github.com/withastro/language-tools/issues/350

Actually, taking a deeper look now, while the result is the same as the other issues, I don't think in this particular case it's a parsing issue because the generated code is right 🤔
I currently have no idea why it happens with this particular keyword, will have to take a deeper deeper look 😓 Could be some sort of limitation of TypeScript that I'm unaware of

@Princesseuh spent some time scraping the net to check if someone had any issue with reserved keywords or something like that but no success. Not sure what can be causing it since it's only happening with the for and not any other TS/JS keywords, but I am able to run the dev server at least 😅 - but it's definitely something that has been bugging me when I open that file 🤣

Hello, this seems to have been fixed in one of the latest version of TypeScript. I can't reproduce in 4.8.3

Fixed by https://github.com/withastro/language-tools/pull/335 and the recent upgrade of TypeScript


Animation components cut-off (using expo)

Description
Some portions of the animation are not appearing when rendering using LottieView.
Expected behavior:
https://media.giphy.com/media/BF8e3IZV9cC3kXPXYg/giphy.gif
Actual behavior: [What actually happened]
https://media.giphy.com/media/C51KYEgJXAC6ke4Ado/giphy.gif
As you can see, some of the components of the animation are cut-off.
Versions
lottie-react-native: 4.0.3
lottie-ios: 3.2.3
expo: 43.0.3

Have same issue +1

Not all After Effects features are supported by Lottie. If you notice there are some layers or animations missing check this list to ensure they are supported.


Add provide entry.

Upstream imgui doesn't seem to like the idea of installing imgui, linking to it as an external project, or attempting to pkg-config it -- whether as a header-only library or a static or shared library. Obviously it's much more convenient to in fact do so anyway, but the thing is that dependency('imgui') can't ever mean anything other than a wrap.
Which kind of makes me wonder if it might be a bit confusing to let it be used in a manner that implies pkg-config rather than treating it like a normal object variable. I'm not sure whether it's a good idea or a bad idea. It just made me think.

dependency('foo') does not really imply Pkg-config or any other method. It just says "get me dependency foo however you see fit". Pkg-config is the most common way to get deps, true, but it is not implied merely by using dependency.


Storage Docs: Simple Bug

Azure.Storage.Cryptography.ClientSideEncryptor
Where found CryptoStreamMode.Read
should be Write not Read

Thank you for your feedback.  Tagging and routing to the team best able to assist.

@ak98 Can you please share the documentation url as well, where you are seeing this issue. Thanks.

Thanks for bringing this to our attention. We will now close this issue. If there are further questions regarding this matter, please tag me in a comment. I will reopen it and we will gladly continue the discussion.


arbitrary remote code execution through zipscript command injection

https://gist.github.com/kSapewpew/51d5975ccb0f49a9ce8b3901d4cb5056 as a copy
The following describes a remote code execution issue (or rather a systemic issue around command injection problems). The root cause for this is insecure handling of file names when executing zip actions in zipscript.

Root cause
----------

The following snippet comes from zipscript-c.c

636                         d_log("zipscript-c: Testing file integrity with %s\n", unzip_bin);
637                         if (!fileexists(unzip_bin)) {                                           <<<<<<<<<<<<<<<< does unzip binary exist?
638                                 d_log("zipscript-c: ERROR! Not able to check zip-files - %s does not exist!\n", unzip_bin);
639                                 sprintf(g.v.misc.error_msg, BAD_ZIP);
640                                 mark_as_bad(g.v.file.name);
641                                 write_log = g.v.misc.write_log;
642                                 g.v.misc.write_log = 1;
643                                 error_msg = convert(&g.v, g.ui, g.gi, bad_file_msg);
644                                 if (exit_value < 2)
645                                         writelog(&g, error_msg, bad_file_zip_type);
646                                 exit_value = 2;
647                                 break;
648                         } else {
649 #if (test_for_password || extract_nfo)
650                                 if ((!findfileextcount(dir, ".nfo") ||
651                                   findfileextcount(dir, ".zip")) && !mkdir(".unzipped", 0777)) <<<<<<<<<<<<<<<<<<< if no nfo, look for .zip file and create .unzipped in the same dir
652                                         sprintf(target, "%s -qqjo \"%s\" -d .unzipped", unzip_bin, g.v.file.name); <<<<<<< construct command line for unzip
653                                 else
654                                         sprintf(target, "%s -qqt \"%s\"", unzip_bin, g.v.file.name);
655 #else
656                                 sprintf(target, "%s -qqt \"%s\"", unzip_bin, g.v.file.name);
657 #endif
658                                 if (execute(target) != 0 || (allow_error2_in_unzip == TRUE && errno > 2 )) { <<<<<<<<<<<<<<< execute through system()
659                                         d_log("zipscript-c: Integrity check failed (#%d): %s\n", errno, strerror(errno));
660                                         sprintf(g.v.misc.error_msg, BAD_ZIP);
661                                         mark_as_bad(g.v.file.name);
662                                         write_log = g.v.misc.write_log;
663                                         g.v.misc.write_log = 1;
664                                         error_msg = convert(&g.v, g.ui, g.gi, bad_file_msg);
665                                         if (exit_value < 2)
666                                                 writelog(&g, error_msg, bad_file_zip_type);
667                                         exit_value = 2;
668                                         removedir(".unzipped"); <<<<<<<<<<<<<<<<<<<<<<<<< remove .unzipped again
669                                         break;
670                                 }

In the above code I have marked the important bits to understand with <<<. These essentially run when zipscript is executed (e.g. also through site rescan).
The issue with this code is that it treats the name of the .zip file basically as untrusted and injects it into a shell command line.

The command executed through this looks as follows is strace:
execve("/bin/sh", ["sh", "-c", "/bin/unzip -qqjo \"ZIPFILENAME.zip\" -d .unzipped 2>.delme"]

Now imagine ZIPFILENAME would be an arbitrary file name that is attacker controlled.
If this contains a " character, you can essentially escape the predetermined command line and inject arbitrary additional commands, thanks to executing this through system/sh.
During a race or a rescan, the shell will happily executed a crafted unzip command line, giving an attacker essentially remote arbitrary code execution from an uploaded file.

Exploitation
------------
The only limitation an attacker has is that the file name itself cannot contain / as that won't be a valid UNIX file name. It's left as an exercise to the CTF-affine reader to figure out which
command lines can be constructed that don't require this however. What's important to understand is that bash is on board of a typical gl installation. In combination with the ability to upload files, the possibilities are essentially endless.

Now when you first see this issue, you may wonder how you exploit this issue, because your typical gl installation does not allow to upload a file that contains " in the name. Most gl installations prevent that through a default setting such as:
path-filter     *       /ftp-data/misc/pathmsg          ^[-A-Za-z0-9_.()[:space:]]*$  ^\.  ^-

This is no limitation to an attacked though because of the logic around how the .zip file is unpacked. It's unpacked to a directory called .unzipped to which the attacker has access to.
It's in the same directory as the originally uploaded file.  At this point it's beneficial to have a large first .zip file so that the race condition between the first unzip and the .unzipped deletion at the end can realistically be won.

Following is an example file and a path to exploit this issue. If you want to test this, replace INSERTATTACKERCONTROLLEDINPUTHERE with sleep 1000 or whatever to have a harmless example.:
Archive:  test.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  08-28-2021 23:10   foo";INSERTATTACKERCONTROLLEDINPUTHERE; #.zip
100000000  08-28-2021 23:36   a.nfo
100000000  08-28-2021 23:36   c.nfo
1000000000  08-28-2021 23:46   d.mp3
---------                     -------
1200000000                     4 files

First, this test.zip is uploaded. The first file that is unpacked is the one with the .zip extension inside. After that the nfo files and the 1GB mp3 file (all junk essentially) are extracted from the zip.
This keeps the .unzipped directory around for a reasonable amount of time to win the race condition to enter .unzipped and before deletion of the directory.
As said, the files are first extracted into .unzipped. From there, you can enter this directory, perform a rescan (contains the same flaw) on the crafted .zip file, this bypassesbypass path-filter logic in gl, and run arbitrary shell commands.

[!] zipscripts handling of file names is messy and duplicated through different parts of zipscript. Like I mentioned rescan contains the same vulnerability as the main race code. There exist a number of execute() calls in the same code that are equally vulnerable to such attacks. This particular one is maybe the most convenient, because it can bypass path-filter entirely. You may find other creative ways for the others though.

Mitigation
----------
Patch your sites! Seriously, just do it[tm].  This is a command injection issue coming straight from first grade infosec books.

For just this issue, the easiest fix is just disabling test_for_password and extract_nfo in the zipscript settings or removing that code. I believe this place is the most convenient one to exploit this particular issue. However, as mentioned, there's more issues like this in zipscript as you'll discover looking for execute and system calls. I did my best below to identify these and patch them, but do your own due diligence.

My recommendation is adding *"* to /ftp-data/misc/banned_filelist.txt, making sure you have check_for_banned_files enabled in the configuration, and applying the below patch to zipscript. Be aware, the code of zipscript is bad enough to not have a warm feeling wrt this overall problem space. Monitor your environments and configure firewalls accordingly for outbound network connections!

In any case, the file handling in zipscript is a mess. Maybe this motivates anyone to rewrite this tooling at some point (haha).

---- kSa

---------------------------------
diff --git a/zipscript/include/zsfunctions.h b/zipscript/include/zsfunctions.h
index 0ff98f84..df6b0a90 100644
--- a/zipscript/include/zsfunctions.h
+++ b/zipscript/include/zsfunctions.h
@@ -159,3 +159,4 @@ extern unsigned int match_lenient(DIR *, char *);
 extern unsigned int insampledir(char *);
 #endif

+extern int _err_file_banned(const char *fn, struct VARS *v);
diff --git a/zipscript/src/postdel.c b/zipscript/src/postdel.c
index 99cb2d55..2def2491 100644
--- a/zipscript/src/postdel.c
+++ b/zipscript/src/postdel.c
@@ -337,6 +337,7 @@ main(int argc, char **argv)
 		if (!fileexists("file_id.diz")) {
 			temp_p = findfileext(dir, ".zip");
 			if (temp_p != NULL) {
+				_err_file_banned(temp_p, &g.v);
 				d_log("postdel: file_id.diz does not exist, trying to extract it from %s\n", temp_p);
 				sprintf(target, "%s -qqjnCLL \"%s\" file_id.diz", unzip_bin, temp_p);
 				execute(target);
diff --git a/zipscript/src/postunnuke.c b/zipscript/src/postunnuke.c
index 56edc0c1..81c86974 100644
--- a/zipscript/src/postunnuke.c
+++ b/zipscript/src/postunnuke.c
@@ -276,6 +276,7 @@ main(int argc, char *argv[])
 				g.v.file.size = fileinfo.st_size;
 				g.v.total.start_time = 0;

+				_err_file_banned(g.v.file.name, &g.v);
 				if (!fileexists("file_id.diz")) {
 					sprintf(exec, "%s -qqjnCLL \"%s\" file_id.diz 2>.delme", unzip_bin, g.v.file.name);
 					if (execute(exec) != 0) {
diff --git a/zipscript/src/race-file.c b/zipscript/src/race-file.c
index 9ded8eb3..7b7a07c5 100644
--- a/zipscript/src/race-file.c
+++ b/zipscript/src/race-file.c
@@ -365,6 +365,7 @@ testfiles(struct LOCATIONS *locations, struct VARS *raceI, int rstatus)
 				d_log("Failed to undupe '%s' - '%s' does not exist.\n", rd.fname, unduper_script);
 			} else {
 				sprintf(target, unduper_script " \"%s\"", rd.fname);
+				_err_file_banned(rd.fname, NULL);
 				if (execute(target) == 0)
 					d_log("testfiles: undupe of %s successful (%s).\n", rd.fname, target);
 				else
@@ -1534,40 +1535,6 @@ void create_dirlist(const char *dirnames, char *affillist, const int limit)
 	return;
 }

-int
-filebanned_match(const char *filename)
-{
-	int             fd;
-	char            buf[500];
-	FILE            *fname_fd;
-	char            fbuf[strlen(filename)+1];
-
-	bzero(fbuf, sizeof(fbuf));
-	strncpy(fbuf, filename, sizeof(fbuf) - 1);
-
-	if ((fd = open(banned_filelist, O_RDONLY)) == -1) {
-		d_log("filebanned_match: failed to open banned_filelist - open(%s): %s\n", banned_filelist, strerror(errno));
-		return 0;
-	}
-	if ((fname_fd = fdopen(fd, "r")) == NULL) {
-		d_log("filebanned_match: failed to open banned_filelist - fdopen(%s): %s\n", banned_filelist, strerror(errno));
-		return 0;
-	}
-	strtolower(fbuf);
-	while ((fgets(buf, sizeof(buf), fname_fd))) {
-		buf[strlen(buf) - 1] = '\0';
-		if ( *buf == '\0' || *buf == ' ' || *buf == '\t' || *buf == '#' )
-			continue;
-		strtolower(buf);
-		if (!fnmatch(buf, fbuf, 0)) {
-			close(fd);
-			d_log("filebanned_match: found match: %s\n", fbuf);
-			return 1;
-		}
-	}
-	return 0;
-}
-
 /*
  * lenient_compare - compare two filenames - ignore case and mix certain chars depending on config.
  * psxc r2173 (v1)
diff --git a/zipscript/src/rescan.c b/zipscript/src/rescan.c
index 860de8f0..da2eee1f 100644
--- a/zipscript/src/rescan.c
+++ b/zipscript/src/rescan.c
@@ -393,6 +393,7 @@ main(int argc, char *argv[])
 					g.v.file.speed = 2005 * 1024;
 					g.v.file.size = fileinfo.st_size;
 					g.v.total.start_time = 0;
+					_err_file_banned(g.v.file.name, &g.v);
 #if (test_for_password || extract_nfo)
 					tempstream = telldir(dir);
 					if ((!findfileextcount(dir, ".nfo") || findfileextcount(dir, ".zip")) && !mkdir(".unzipped", 0777))
@@ -805,6 +806,7 @@ main(int argc, char *argv[])
 				d_log("rescan: Warning - rescan_script (%s) - file does not exist!\n", rescan_script);
 			} else {
 				snprintf(target, sizeof(target), rescan_script " \"%s\"", g.v.file.name);
+				_err_file_banned(g.v.file.name, &g.v);
 				if (execute(target) != 0)
 					d_log("rescan: Failed to execute rescan_script: %s\n", strerror(errno));
 			}
diff --git a/zipscript/src/zsfunctions.c b/zipscript/src/zsfunctions.c
index fdc4db25..efb10ffa 100644
--- a/zipscript/src/zsfunctions.c
+++ b/zipscript/src/zsfunctions.c
@@ -1911,3 +1911,56 @@ chmod_each(const char *list, mode_t mode)

     return fail;
 }
+
+int
+filebanned_match(const char *filename)
+{
+        int             fd;
+        char            buf[500];
+        FILE            *fname_fd;
+        char            fbuf[strlen(filename)+1];
+
+        bzero(fbuf, sizeof(fbuf));
+        strncpy(fbuf, filename, sizeof(fbuf) - 1);
+
+        if ((fd = open(banned_filelist, O_RDONLY)) == -1) {
+                d_log("filebanned_match: failed to open banned_filelist - open(%s): %s\n", banned_filelist, strerror(errno));
+                return 0;
+        }
+        if ((fname_fd = fdopen(fd, "r")) == NULL) {
+                d_log("filebanned_match: failed to open banned_filelist - fdopen(%s): %s\n", banned_filelist, strerror(errno));
+                return 0;
+        }
+        strtolower(fbuf);
+        while ((fgets(buf, sizeof(buf), fname_fd))) {
+                buf[strlen(buf) - 1] = '\0';
+                if ( *buf == '\0' || *buf == ' ' || *buf == '\t' || *buf == '#' )
+                        continue;
+                strtolower(buf);
+                if (!fnmatch(buf, fbuf, 0)) {
+                        close(fd);
+                        d_log("filebanned_match: found match: %s\n", fbuf);
+                        return 1;
+                }
+        }
+        return 0;
+}
+
+
+/* kSa - yes this is somewhat inefficient as we'll open the banned file every time and it's not cached
+   i'm not going to rewrite the code here and rely on fs caching. also, someone should rewrite zipscript in a
+   memor-safe language - sigh
+*/
+int
+_err_file_banned(const char *fn, struct VARS *v) {
+	if (0 == filebanned_match(fn)) {
+		d_log("%s: seems like someone is trying to mess with you (%s):\n", __FILE__, fn);
+		printf("you're a bitch\n");
+		unlink(fn);
+		if (NULL != v) {
+			remove_lock(v);
+		}
+		exit(EXIT_FAILURE);
+	}
+	return 0;
+}

Is Skeddie still alive? He didn't get a chance to speak at the last issues either. nothing has been updated on the zipscript itself for a long time. so that this is best firmly intrigued into his git

yeah i wish i knew... actually didn't want to put this on github, but didn't reach anyone for weeks. so hope this helps some folks. don't forgot to edit banned_filelist.txt also and ideally harden your servers. i'm assuming there's more issues.

Thanks for sharing this @kSapewpew

Yes, probably would have been better to use execve(unzip_bin, args, NULL) instead of system()
Also better to use santized env.. etc.. it's playing whack-a-mole.
Or maybe.. don't use a dead project with code not touched for 20 years, made by cool dudes long gone and for a different time but still used by it's change resistant brain-dead target audience.

As if it is wrong to report this kind of stuff and keep on going with a project that is open source, keep attitude at the right level and stop cry, clown.

Yes, probably would have been better to use execve(unzip_bin, args, NULL) instead of system()
Also better to use santized env.. etc.. it's playing whack-a-mole.
Or maybe.. don't use a dead project with code not touched for 20 years, made by cool dudes long gone and for a different time but still used by it's change resistant brain-dead target audience.

i'm glad you sprinkled just enough technical knowledge on top of your opinion to make it obvious enough that you're definitely a pro. i love the irony seeing your github forks and your commits. why the hate bro?


UnboundLocalError when using pytest-splinter in Docker container

I am attempting to shift my automation at work from Selenium to Splinter. As part of this, I have chosen to utilize the pytest-selenium plugin for better QOL. Unfortunately, it appears that Splinter is raising an UnboundLocalError when run from a Docker container, not when running from command line. This error is raised at line 68 in browser.py within the Splinter library.

        When working with ``firefox``, it's possible to provide a profile name
        and a list of extensions.

        If you don't provide any driver_name, then ``firefox`` will be used.

        Common selenium errors are caught and a retry attempt occurs.
        This can mitigate issues running on Remote WebDriver.

        """
        for _ in range(retry_count):
            try:
                return driver(*args, **kwargs)
            except (IOError, HTTPException, WebDriverException, MaxRetryError) as e:
                pass

>       raise e
E       UnboundLocalError: local variable 'e' referenced before assignment

/usr/local/lib/python3.6/site-packages/splinter/browser.py:68: UnboundLocalError

Dockerfile I am using:
FROM python:3.6-slim-buster

# Set working directory
WORKDIR /python/tests/automation/

# Install third-party dependencies
RUN pip install pipenv

COPY Pipfile .
COPY Pipfile.lock .
RUN pipenv install --system --deploy --ignore-pipfile

# Copy automation data and configuration
COPY . .

# Clear .pyc files for conftest.py import
RUN find . -name *.pyc -delete

# Install Google Chrome
RUN apt-get update && \
    apt-get install -y wget dpkg && \
    wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && \
    dpkg -i google-chrome-stable_current_amd64.deb; apt-get -fy install

RUN webdriverdownloader chrome

RUN pytest -m splinter

Hello,
I'm having troubles trying to run in a Docker container too. I'm using an Alpine 3.11 based container, with chromium and chromedriver installed.
I launch the browser like this :
    opts = Options()
    opts.add_argument("--no-sandbox")
    opts.add_argument("--disable-setuid-sandbox")
    opts.add_argument("--disable-gpu")
    opts.add_argument("--disable-software-rasterizer")
    opts.add_argument('--start-maximized')

    with Browser('chrome', headless=True) as browser:
        browser.driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=opts)
        browser.visit('https://google.com')

and it always fail with the following error :
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/lib/chromium/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

I'm using splinter Github master version, installed with :
python -m pip install --upgrade "git+https://github.com/cobrateam/splinter.git"

What am I doing wrong ?

Ok just find my mistake.
Selenium Chromedriver options was no correctly set, with the following code it works fine :
import logging

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from splinter import Browser

logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
                    level=logging.DEBUG)

logger = logging.getLogger(__name__)

opts = Options()
opts.add_argument("--disable-gpu")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-setuid-sandbox")
opts.add_argument("--disable-software-rasterizer")
opts.add_argument('--start-maximized')

logging.info('Trying to start Chromium via Chromedriver ...')

selenium_opts = {'options': opts}

with Browser('chrome', headless=True, **selenium_opts) as browser:
    browser.visit('https://google.com')

logging.info('Hello Chromium !')

Sorry for the noise :disappointed:

Going to close this out as my team and I decided to not move forward with Splinter.


2015-08-12

Reverts the unmerged patch and updates to master with the merged one.
@angelini @solackerman @kmtaylor-github

:100: great job!

Built successfully: http://pack.chi.shopify.com/packages/Shopify/spark/1a98747efae7df352162d997c638d78ae8a6c502.json

:boat:


Feat creation of indicator component

Margin left unecessary, you can apply a flex in all div with values

Remove position relative, not did do effects.


Added support for service annotations.

With this, one can configure the service with custom annotations, for
example:
service:
  annotations:
    cloud.google.com/load-balancer-type: "Internal"

Merged by hand (with a couple more changes) into flynn/rel/0.35.1 -- thanks! and sorry for the delay getting to this.