Allow unfolding all foldables by default

Hi @go101! 👋
I proposed a feature to allow unfolding all foldables by default with this PR.
I found out clicking the foldables over-and-over. By having all foldables being unfolded, I can see the whole documentation and fold whatever I need.
I'm sorry I didn't write the test for the changes as I can't run the test runner on my machine.
Thank you very much. Looking forward to hearing from you about this one.

Seems also mentioned here.

As the new option is defaulted as false, I don't reject it.
How about to change the option name to unfold-all-initially?
BTW, when JS is on, we can press the a key to switch folding status for all.

How about to change the option name to unfold-all-initially?
Sounds good. Will change later in leisure time.

BTW, when JS is on, we can press the a key to switch folding status for all.
Yes, I'm aware of this, but unfortunately, I dunno, but it doesn't unfold everything as I still have to click some of inner folding for a complete unfolded documentation. I dunno if that's intended or not, or if there's something wrong here.

I've checked that this changes unfolds all of the folding, except the private implementation folding, but once it's clicked, all of the inner folding has been unfolded, which is pretty okay for me.

Hi @go101!
I've got time today for this one and I've pushed the changes. Please kindly take a look.
Thank you.


Check for ActiveSupport presence in railtie definition

This PR is related to the accidental introduction of an ActiveSupport dependency into the gem. api_auth should be able to be used without Rails (and therefore without ActiveSupport). Because the railtie is always included when the gem is loaded, the railtie is loaded even when Rails is not.
This was not historically an issue because before the merging of my pull request to fix a related deprecation warning, we checked for the definition of ActiveController::Base, which meant that the railtie file could be required even if Rails was not.
Because the definition check was the source of the deprecation warning that I fixed in my original PR, my change had the effect of breaking the railtie inclusion if ActiveSupport is not bundled.
This PR solves the issue by checking for the definition of ActiveSupport before using the on_load hooks. An alternative solution (which might make more sense) would be to conditionally include the railtie in its entirety here:
# ...
require 'api_auth/railtie' if defined?(::Rails)

Let me know what you think about how this problem should be solved.

👍

Thanks for the follow up PR @taylorthurlow. 👍


How to turn off automatic module rename refactoring?

One of my macOS systems is acting as if I chose Don't ask again in the Extension 'Python' wants to make refactoring changes with this file move dialog. I doubt I did, but anything is possible, I suppose.
At any rate, I need the auto-rename behavior to stop, but can't find any way to turn it off.
I've even peeked in the SQLite DB at ~/Library/Application Support/Code/User/globalStorage/state.vscdb, but failed to find anything that looked applicable.
Is there anything I can do, short of reinstalling VS Code?

Are you trying to turn it back on? Or are you trying to turn it off?

That depends on your definition of "it," I suppose :)
I'm trying to turn automatic (that is, proceeds without confirmation) module rename refactoring off.
To accomplish that, I'm operating under the assumption I need to bring back (or "turn on") the confirmation dialog with the Don't ask again option.
That said, if there's another way (short of no longer using the extension) to get the Python extension to stop auto-refactoring when I rename modules, I'm all ears.

Hoping I'm not rudely "bumping" this issue, but does anyone have any insight on how to turn off automatic refactoring triggered by the renaming of a module? Hopefully short of reinstalling VS Code, but I'm nearing the point of trying that.

Related -- https://github.com/Microsoft/vscode/issues/24815

Thanks very much for that.
In the meanwhile, @judej, is there any way I can dig into the Pylance extension settings and get the pre-refactoring confirmation dialog to reappear? I'm willing to dive into sqlite database files, and don't need my hand held, but a hint at the naming of the setting/property/field I should be searching for would be greatly appreciated.

I combed the VS Code source a bit, latched onto what was going on in mainThreadFileSystemEventService.ts, and worked out that the "file.particpants.additionalEdits" (yes, spelled "particpants") record in ~/Library/Application Support/Code/User/globalStorage/state.vscdb database needed to be removed.
After closing VS Code and making that change, the Extension 'Python' wants to make refactoring changes with this file move dialog returned, and I'm now able to control refactoring decisions again.
(I should note that I also removed "file.particpants.additionalEdits" from within the value of the neighboring __$__targetStorageMarker record, but I suspect that was temporary data storage and not really necessary.)
In the end, @debonte was exactly right in the sharing of the link above--this issue is on the VS Code side of the fence, rather than the Pylance side.
I'd surely appreciate it if the Pylance team would pressure the VS Code team to make this user experience better!
(Editorial: On one hand, who can complain about free software as great as VS Code? On the other hand, how can a piece of software as fantastic and polished as VS Code have "one-shot" settings that can't be undone without tracing source code and editing sqlite databases?)

unfathomable that this is still an issue today. I cant rename my files or folders, without vscode freezing up... and i have to wait for the dialog box to pop open and then if I am lucky, I can cancel the action in time. Searching online, I see other frustrating comments, but no attention. This is a serious bug, as it can be destructive.

unfathomable that this is still an issue today. I cant rename my files or folders, without vscode freezing up... and i have to wait for the dialog box to pop open and then if I am lucky, I can cancel the action in time. Searching online, I see other frustrating comments, but no attention. This is a serious bug, as it can be destructive.

unfathomable that this is still an issue today.

@debonte Why was this closed? Looks like myself, @Philipflyg, @StefanKarlsson987 are all having this same problem.


Taskbar icon not visible

Describe the bug
The icon is not visible in the taskbar
To Reproduce
Open Maestral
Expected behaviour
There should be an icon in the taskbar
System:

Maestral version: 1.4.8
Python version: 3.9.6
OS: [e.g. Ubuntu] Fedora 34 Workstation
Desktop environment: [e.g. Gnome 3.32] GNOME 40.4.0
PyQt version (for Linux GUI): 5.15.4

Additional context
Log when I try to start maestral_qt:
$ maestral_qt
QSocketNotifier: Can only be used with threads started with QThread
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
QImage::pixel: coordinate (0,0) out of range
^CTraceback (most recent call last):
  File "/home/raffaele/.local/lib/python3.9/site-packages/maestral_qt/main.py", line 163, in show_when_systray_available
    def show_when_systray_available(self):
KeyboardInterrupt
^CTraceback (most recent call last):
  File "/home/raffaele/.local/lib/python3.9/site-packages/maestral_qt/main.py", line 152, in update_ui
    def update_ui(self):
KeyboardInterrupt
^C^Z
[1]+  Stopped                 maestral_qt

I'm back :D

It's showing now also on Wayland.
Probably you need to reboot or something like that


Direct-to-BIO SSL mode

At the moment, all OpenSSL versions will first read/write record to the internal buffer, and only later call BIO_write with this pointer.
This is absolutely fine for BIO_s_connect()/BIO_s_accept(). However for memory BIOs (including custom BIOs that we use in Node.js) set through SSL_set_bio, this performs slower because of the extra memcpy call that needs to be done.
In case of Node.js, our custom BIO is powered by a circular buffer, and it is pretty easy to pre-allocate enough data to fit the record.
Hereby, I propose to add APIs for OpenSSL to introduce Direct-to-BIO operation mode in which special ctrl methods will be called on BIO, and if it recognizes them - the record data will be read/written directly to the BIO's internal buffer.
Does this sound reasonable?
cc @richsalz

It’s a good goal, not sure about the best way to implement it.  Ok? :)

Yep, thank you for response!
As far as I got it there has been some recent changes with the way write buffers are managed. There seem to be multiple pipelines for them now.
cc @mattcaswell since you implemented this.

The APIs may look like this:

BIO_CTRL_GET_FRAGMENT_BUFFER - to get a pointer to the bio's internal buffer
BIO_CTRL_COMMIT_FRAGMENT - to commit fragment to BIO

Yes, this could be possible. Probably not a top priority though. I'll put this against the post 1.1.0 milestone.

Thank you!

Hi,
I would just like to say I am writing an openssl channel on the new .net channels. This would be useful as the entire stack is close to no copy except for this one tiny part. I have a custom bio pointing to my own buffers, and this would work perfectly.
https://github.com/davidfowl/Channels/pull/128
Thanks

Marking as inactive, to be closed at the end of 3.4 dev barring further input


New workflow to deploy the lambda function

Build the lambda function, package it as expected by AWS lambda, then update it on AWS.
For now this is only triggered manually, so that the update can be performed only when it makes sense to the developers. In the future we can make a release being tagged trigger this workflow.

we should probably explain what secrets need to be added too

explain where? I can add it to the doc on deploying the lambda function I guess, after it has been validated


[DUOS-2527] Create an Edit Study page

Addresses
https://broadworkbench.atlassian.net/jira/software/c/projects/DUOS/boards/123?modal=detail&selectedIssue=DUOS-2527
Summary
Two things to note here:

Some of the data is corrupted in the previous studies. It is best to use a new study with all the proper info when testing
The submission is not working properly

Have you read Terra's Contributing Guide lately? If not, do that first.

Label PR with a Jira ticket number and include a link to the ticket
Label PR with a security risk modifier [no, low, medium, high]
PR describes scope of changes
Get a minimum of one thumbs worth of review, preferably two if enough team members are available
Get PO sign-off for all non-trivial UI or workflow changes
Verify all tests go green
Test this change deployed correctly and works on dev environment after deployment

We need to improve the efficiency here. I see 6 calls to get all dacs, and similarly 6 calls to get dac id 4

Clicking on the first option in NIH and AnVIL Use:

Should trigger the expanded view with additional questions for that section as well as NIH Administrative Information and NIH Data Management & Sharing Policy Details sections.


Lang guide

Initial version of Sysl language guide.
@anz-bank/sysl-developers

@anzdaddy asked me to look at this again
thanks @sahejsingh for making some changes in response to my feedback earlier . it doesnt seem like much  has changed since i reviewed it earlier. is it worth talking through my other feedback (e.g. lack of explanation about what top-level elements in the sysl files are) that hasn't been addressed by the most recent commit?

dumb questions: is there anything special about the word "MobileApp"? how does sysl know "MobileApp" is an application?
maybe could say something like "the name MobileApp is user-defined and has no special meaning to sysl" or perhaps, "in sysl, any top-level word followed by a colon such as "MobileApp:" or "Foo:" or "Banana:" defines a an application" or perhaps "a named application".
that probably isnt clearest way to explain it, hopefully you roughly get what i'm getting at

@anz-rfc , the explanation clearly states the following:

To explain these concepts, we will design an application called MobileApp which interacts with another application called Server.

and later...

MobileApp is a user-defined Application that does not have any endpoints yet. We will design this app as we move along.

I think it is clear that user is defining an app called 'MobileApp'.
I have also shows that a name like 'MobileApp' followed by ':' and an indent makes it an application.


ramlog: Add CONFIG_RAMLOG_KEEPLOG option

Summary
Adds a new config option to prevent reading /dev/kmsg from clearing the log buffer. This way the latest log messages can be read multiple times, until the log gets overwritten when buffer is filled up.
This matches the behavior of /dev/kmsg on Linux.
Impact
Old behavior is retained by default. When CONFIG_RAMLOG_KEEPLOG is enabled, reading /dev/kmsg no longer clears the buffer.
Testing
Tested on custom STM32F4 board by using cat to read & write /dev/kmsg.

@PetteriAimonen we also made a similar change recently, but a little bit flexible which support multiple readers. Could you review our change whether suitable your needs?

@xiaoxiang781216 I'd be glad to review. But somehow I can't see that change in pull request queue nor in the master branch?

@xiaoxiang781216 I'd be glad to review. But somehow I can't see that change in pull request queue nor in the master branch?

We are preparing the patch set, will be available soon.

@xiaoxiang781216 Ok, sounds good :) No hurry on my part, I'll use my implementation for now.

@xiaoxiang781216 Ok, sounds good :) No hurry on my part, I'll use my implementation for now.

@PetteriAimonen here is the patch:
https://github.com/apache/nuttx/pull/11393


Bugfix: QPU qubit overflow

Description

Added n_qubits attribute to DeviceQiskit that extracts the maximum number of qubits compatible for the particular IBMQ Backend selected

Added qubit number check between CircuitParams Object and DeviceQiskit. An Exception is raised if the number of qubits in the backend is less than the number of qubits required to create the circuit.

Updated unittests in test_qiskit_qpu.py

Added n_qubits attribute to DeviceAWS that extracts the maximum number of qubits compatible for the particular Braket Backend selected. (Since not all backends may have the total number of qubits available, a function that catches the possibility that the number of qubits is not available is added for this Device object.)

Added qubit number check between CircuitParams Object and DeviceAWS. Exception is raised as above.

Updated unittests in test_braket_qpu.py

Added n_qubits attribute to DevicePyQuil that extracts the maximum number of qubits compatible for the particular PyQuil QPU Backend selected.

Added qubit number check between CircuitParams Object and DevicePyquil. Exception is raised as above.

Checklist

[ ] I have performed a self-review of my code.
[ ] I have commented my code and used numpy-style docstrings
[ ] I have made corresponding updates to the documentation.
[ ] My changes generate no new warnings
[ ] I have added/updated tests to make sure bugfix/feature works.
[ ] New and existing unit tests pass locally with my changes.

Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Name the new unit-tests that you have added along with this change.

Comments on PyQuil Backend:
The backend does not fail silently like the others when a larger circuit is specified than is supported by the actual QPU.
The main reason for this is that within the init function of the PyQuilQPUBackend Class, the paramteric circuit is compiled for the particular QPU selected. This compilation step throws the error. Since the error is not thrown within the get_counts method, the code does not silently fail.
The new addition to the code just makes sure that the failure due to "qubit overflow" has a consistent message throughout all the backends.
Additional Comments:
With the slightly different design of the DevicePyQuil, n_qubits attribute assigned at the initialisation of the class since this information is readily available. This is not True for DeviceQiskit and DeviceAWS where the check_connection() must first be executed before n_qubits becomes available.
There is no direct effect in the code when n_qubits is assigned. Though we may want to consider if it might be appropriate to have similar behaviours for all Device Objects.


Network packet location changed

The network packet location has been changed to come immediately after the header word, instead as the fifth word. This move was motivated by a desire to not have the network packet ID be dropped in the middle of the precision scalars.
This will affect data, PPG, and scalar packets. Changes will need to be made to GRSISort to accommodate this, but I expect all changes to be backward-compatible. I am updating the data processing in TDataParser.cxx. Are there any other files this change will effect?

The data parsing is encapsulated in the TDataParser Class so nothing else should need to be changed. We can make this change backwards compatible, but the code will have to check both before and after the 00 words instead of assuming their existence. It would actually already work if it was the fourth word after the 0 and 00 words. Is it now immediately after the header word and before the filter and ppg patterns? Or did after the header word mean immediately before the channel trigger Id word.

On Sep 9, 2015, at 7:22 PM, SmithJK<EMAIL_ADDRESS>wrote:
The network packet location has been changed to come immediately after the header word, instead as the fifth word. This move was motivated by a desire to not have the network packet ID be dropped in the middle of the precision scalars.
This will affect data, PPG, and scalar packets. Changes will need to be made to GRSISort to accommodate this, but I expect all changes to be backward-compatible. I am updating the data processing in TDataParser.cxx. Are there any other files this change will effect?
—
Reply to this email directly or view it on GitHub https://github.com/GRIFFINCollaboration/GRSISort/issues/482.

What you described is basically what I've implemented. I'm confirming a PPG time difference change and then I'll push the changes. It is now immediately after the header word and before the filter and PPG patterns (if they exist) or before the channel trigger id (if the filter and PPG words do not exist).

Pushed in #483 .


Random uniform is inconsistent given same seed values

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
I was trying to use one of example usages from tensorflow docs and wanted to make results reproducible. After investigation it looks to me that gen_random_ops.random_uniform is inconsistent given seeds. That is why I have written a test ( custom code)
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Darwin-18.0.0-x86_64-i386-64bit
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from : pip install tensorflow
TensorFlow version : v2.1.0-rc2-17-ge5bf8de410 2.1.0
Python version: python version: 3.7.6
Bazel version (if compiling from source): brew installation
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: -
GPU model and memory: -

Describe the current behavior
gen_random_ops.random_uniform return different values even with same seeds
Describe the expected behavior
consistent behaviour is expected give same seeds
Standalone code to reproduce the issue
shape = (8,12)
dtype = 'float32'
seed = 5
seed2 = 1234
tf.random.set_seed(seed)
rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed, seed2=seed2)
rnd2 = gen_random_ops.random_uniform(shape, dtype, seed=seed, seed2=seed2)
# rnd and rnd2 will be different

Other info / logs
code does not throw exceptions. Just expected behaviour is different from results.
From random_seed.py I can see that probably it is expected behaviour but I don't understand why is that. This counter that keeps changing seed  by default is a way to avoid generation of same values? But usually for tests we need reproducibility. Does it mean that It should be done through re-setting of the tf.random.set_seed(1234)?

@deil87
Code provided seems incomplete please provide complete stand alone code for us to replicate the issue faced.

@Saduf2019 I have realised that this is probably not a bug but a feature. It is just a bit unexpected behaviour -  usually you don't need to re-run script or set seed one more time to get consistency. But random_seed.py explains that this is how it is done in tensorflow. I'm going to close this one.

Yes this is an undesirable property of the legacy stateful random uniform ops. Use either the generator API or the stateless random ops for sane behavior.


SAK-30154: Improve awkward animations for assignments allow resubmission and send feedback panels

Looks good to me. Would appreciate another set of eyes on this though.


Player stops except for playing next audio track in a playlist when the app is in background.

Version of react-native-audio-recorder-player : 2.6.0-rc3
Platforms you faced the error (IOS or Android or both?) : Android
Expected behavior : It should play next audio track in a playlist when the app is in background.
Actual behavior : It's not playing next audio track when the app is in background.
Steps to reprodue the behavior : 1. Create a playlist of some audios, 2. Implement playing logic sequentially, 3. Put the app in background while playing any audio. 4. once the currently playing audio track finishes, next one won't launch(only when the app is in background)
Note: It's android specific.

.

.


Error appeared when try to edit

Describe the bug
messages: text content blocks must be non-empty
the above message appeard to me and canot do any edit or deploy

Link to the Bolt URL that caused the error
https://bolt.new/~/sb1-fccgfp
Steps to reproduce
Submit
Expected behavior
messages: text content blocks must be non-empty
the above message appeard to me and canot do any edit or deploy
Screen Recording / Screenshot

Platform

OS: Windows
Browser: Edge
Version: Version 130.0.2849.80 (Official build) (64-bit)

Additional context
ed

I have the same issue, did you find a fix?

same problem

Hello,
Remove from. BOLT  DB the project and open it again
Get Outlook for iOShttps://aka.ms/o0ukef

From: eventstubsol @.>
Sent: Thursday, November 21, 2024 3:26:01 PM
To: stackblitz/bolt.new @.>
Cc: Muhammad Musbah @.>; Author @.>
Subject: Re: [stackblitz/bolt.new] Error appeared when try to edit (Issue #2373)
same problem
—
Reply to this email directly, view it on GitHubhttps://github.com/stackblitz/bolt.new/issues/2373#issuecomment-2491003013, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BJ4XJFH7DIF3ARQGX447SF32BXGNTAVCNFSM6AAAAABRZXNRO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJRGAYDGMBRGM.
You are receiving this because you authored the thread.Message ID: @.***>


Update code to handle API change in Senzing SDK API 1.15

Describe the bug
The Senzing python SDK API no longer returns an object.

This has been delivered to customer and seems to have fixed the issue.  We'll close.   If any new problems are discovered, we'll open a new issue.


Initial code owner groups

PR Template
Purpose

Define Code owners for some subdirectories

Does this introduce a breaking change?

[ ] Yes
[X] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[X] Other... Please describe: Github stuff

Great contribution ! :)


Bug: npx zenstack@latest init is giving error becouse is installing prima 6

Description and expected behavior
npx zenstack@latest init is giving error becouse is installing prima 6 Installing package<EMAIL_ADDRESS>with npm
Environment (please complete the following information):
New project
Additional context
npm error code ERESOLVE
npm error ERESOLVE unable to resolve dependency tree
npm error
npm error While resolving<EMAIL_ADDRESS>npm error Found<EMAIL_ADDRESS>npm error node_modules/prisma
npm error   dev<EMAIL_ADDRESS>from the root project

fixed in 2.10.0


Corner plot shows different errors than the best fitted values

Hi!
The corner plot shows the same mean the fitter gave but not the same errors. I tried to figure out what the issue in the code was, but I couldn't find it.

Hi!
This is expected, as the corner plot was always considered a neat visualization add-on and not a final report of parameters. For the actual errorbars we use the Highest Density Regions of the final posterior distribution, which in some cases (that are not uncommon!) can be multimodal and thus the usual way one would obtain the credible intervals might not be representative of the distribution. For the corner plot, I realize the calculation is done the 'usual way', which is basically getting the upper and lower percentiles corresponding to 1 sigma. I recon this should be an easy thing to change, but changing it would make the run time during the plots slower (although I'm sure it wouldn't be an unmanageable overhead).
Let me know if this is absolutely crucial to you and I can change it.
Cheers

Hey! It is not absolutely necessary, but it would be great if the values matched in case I wanted to use the image of the corner plot in the paper.  Is it possible just to add manually the values from the best fit to the corner plot? or to read them from the file? Not calculate them again?

Hi!
which one i can use in paper to give errors,  best_fit_average.dat or best_fit_sample.dat ?
corner gives different errors. but mean is same as mentioned before.


it takes a lot time when writing and  clear back a code from editor

We have written the needed data into your clipboard because it was too large to send. Please paste.

their is the time delay problems when writing on the vs code and using back space on the vs code ...after I updated my dell laptop


OSX support

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

More precisely, are   'libRDKitChemReactions' and 'libRDKitChemTransforms' found and linked on linux ?

In the CMakeCache.txt of the rdchiral lib on linux I can indeed find libRDKitChemReactions but not libRDKitChemTransforms.

I also checked and they are both available in the osx and linux conda packages.

Conpilation and linking works but test fails now when importing rdchiral. It seems like the package is corrupted.


Docker build fails

See https://github.com/Mudlet/Mudlet/runs/6125396129?check_suite_focus=true - the docker build has started to fail for some reason.

Oops, we're using ubuntu:latest which is now 22.04 and has different clang-tidy packages: https://packages.ubuntu.com/search?keywords=clang-tidy&searchon=names&suite=jammy&section=all
Trivial fix, although it would change the versions of clang-tidy

This still fails, even when specifying ubuntu-20.04
https://github.com/Ultimaker/CuraEngine/pull/1652

clang-tidy-review uses its own docker container, which is unrelated to the parent Action's container: https://github.com/Ultimaker/CuraEngine/runs/6175255484?check_suite_focus=true#step:2:5
I'll pin to ubuntu-20.04 for now.

thank you for the quick response and fix


DATE_PART doesn't work for ISO_DATE

I have a collection with following "schema"
{ "_id" : ObjectId("57f53249b746cc464b9fac1f"), "dt" : ISODate("1926-08-11T21:00:00Z"), "ts" : -1369278000, "value" : 0.4842846439255769 }
{ "_id" : ObjectId("57f53249b746cc464b9fac20"), "dt" : ISODate("1993-03-21T21:00:00Z"), "ts" : 732747600, "value" : 0.47271090326168097 }
{ "_id" : ObjectId("57f53249b746cc464b9fac21"), "dt" : ISODate("1968-02-05T21:00:00Z"), "ts" : -60058800, "value" : 0.020977698066937056 }
{ "_id" : ObjectId("57f53249b746cc464b9fac22"), "dt" : ISODate("1978-07-23T21:00:00Z"), "ts" : 270075600, "value" : 0.5733601616341653 }
{ "_id" : ObjectId("57f53249b746cc464b9fac23"), "dt" : ISODate("1955-03-16T21:00:00Z"), "ts" : -466916400, "value" : 0.7161196150201754 }
{ "_id" : ObjectId("57f53249b746cc464b9fac24"), "dt" : ISODate("1924-07-04T21:00:00Z"), "ts" : -1435633200, "value" : 0.11387358349052235 }
{ "_id" : ObjectId("57f53249b746cc464b9fac25"), "dt" : ISODate("1937-06-20T21:00:00Z"), "ts" : -1026615600, "value" : 0.9649450316634763 }
{ "_id" : ObjectId("57f53249b746cc464b9fac26"), "dt" : ISODate("1917-03-05T21:00:00Z"), "ts" : -1667012400, "value" : 0.16969064247916554 }
{ "_id" : ObjectId("57f53249b746cc464b9fac27"), "dt" : ISODate("1911-04-26T21:00:00Z"), "ts" : -1851908400, "value" : 0.19975430639659764 }
{ "_id" : ObjectId("57f53249b746cc464b9fac28"), "dt" : ISODate("1940-05-31T21:00:00Z"), "ts" : -933649200, "value" : 0.6063859894813727 }

This query
select distinct date_part("year", dt), date_part("month", dt), dt, value
from `/mongo/testDb/dates`

Returns nulls in first two columns for every date before 1970.

Does select * show reasonable values for the “dt” field?
If you’re inserting using ISODate on any version of Mongo we support (2.6+), it should have no problem storing values from before 1970. But calling date_part("year", dt) should be translated to simply {"$year": "$dt"} or dt.getUTCFullYear (depending on which of Mongo’s languages end up being generated).
It seems we don’t have any StdLib tests that cover pre-epoch dates. That’s definitely an oversight that we should remedy with this issue.

Found this comment: // FIXME: should be some negative value (this will miss any date before 1970) while looking for the culprit 😄

This was an easy fix, and only affected old Mongo (although it could affect new Mongo at some point). But the bug was in our aggregation pipeline type checking.

I also added tests in the StdLibSpec to deal with pre-1970 dates, which identified a similar bug (also fixed) in MarkLogic Interval handling.


DVD with mastering errors fails to dump

Myself and at least one other user have run into an issue with a PS2 DVD with mastering errors, corresponding to a dummy file on the disc. To verify that this was an issue with the DVD, multiple programs were used to generate ISOs, all of which hit the same error in the same sectors. For programs that supported it, filling those sectors with \0 ended up producing a matching hash for the known information on the disc.
This seems to only affect one particular pressing of this disc as well, since other pressings of the disc have the same dummy file (filled with \0) and do not hit the same issue. This is definitely a corner case for handling protected discs.
Attached here is the output logs from the DiscImageCreator run. Of note, it fails on the first of the bad sectors and then entirely stops after 5 attempts:
KATAMARI.zip
The eventual ask from this is either:

Add a flag to be able to ignore these read errors for DVD specifically
Or, to log the error but fill with \0 bytes in the affected sectors

I ordered it by Amazon.com. It'll arrive on October 9.

DiscImageCreator_test.zip
added

   /ps     The sector is padded when reading error occurs
                  val     0: Padded by 0x00
                  val     1: Padded by 0xAA


Pandora.com is darkified & Twitch.tv theatre mode now inverses properly

Thanks for your help, I only have two comments:

...> * > * > * > * > * > * > * > * > * > * > * > * > * >... selectors should be refactored.
Sites fixes should be sorted in alphabetical order.

Both of your points have been fixed in the second commit I just added.

Thanks.


Can we delete kube-state-metrics from this repo

Are we still using kube-state-metrics in this repo? it looks like we are pulling directly from upstream, so cant we just delete the folder from this repo?
https://github.com/neticdk/k8s-oaas-observability/tree/main/charts/kube-state-metrics

The kube-state-metrics chart has been marked as deprecated a while ago:
https://github.com/neticdk/k8s-oaas-observability/blob/main/charts/kube-state-metrics/Chart.yaml#L17
We are not using it anymore, we could possibly delete it as the old revisions will still be available from the releases - I guess.


Update machine v.0.15.0-rancher-9-1

Problem:
We do not support configuring azure VMs to use managed disks. This also block the user from provisioning with certain options we provide that require managed disks.
Solution:
Add managed disks to azure in docker machine
Issue:
https://github.com/rancher/rancher/issues/15788

LGTM


Foundations

Trying to build up some intuitions about traditional machine learning and deep learning techniques from scratch - with a focus around expressing it neatly in terms of code + visualizations.
Preferably not using any notebooks, just runnable make commands.

One specific path I want to go is a depth-first exploration from stable diffusion down to the foundational concepts that it is built on.

very inaccurate, but somewhat useful to know in practice - closest I can get to max BW in raw CUDA is ~24GiB, which is not nearly theoretical maximum of PCIe 4.0 x16
python3 naive.py
completed in 0.06398421s | throughput=0.14022472499855704GiB/s
completed in 0.00047826s | throughput=18.759905774907672GiB/s
completed in 0.00046302s | throughput=19.377580837624777GiB/s
completed in 0.00045833s | throughput=19.575824939497323GiB/s
completed in 0.00046212s | throughput=19.41527600456674GiB/s

python3 preallocate_on_gpu.py
completed in 0.00051289s | throughput=17.4933911495675GiB/s
completed in 0.00046213s | throughput=19.414772242060117GiB/s
completed in 0.00045613s | throughput=19.6701131964522GiB/s
completed in 0.00045330s | throughput=19.79322017749102GiB/s
completed in 0.00045695s | throughput=19.635071544645644GiB/s

python3 make_example_tensor.py
python3 load_from_disk.py
completed in 0.01094502s | throughput=1.6394985232611694GiB/s
completed in 0.00328608s | throughput=5.460715172017624GiB/s
completed in 0.00299614s | throughput=5.989145333375944GiB/s
completed in 0.00287382s | throughput=6.244077469324594GiB/s
completed in 0.00115082s | throughput=15.592652184365365GiB/s

Benchmarking imagenet + regular torch dataloader (unprocessed) + image folder vs. HF datataset
batch_size=32, workers=16

Testing out HDF5 as a data format for training data, but it was pretty annoying trying to figure out how to quickly write all of imagenet into the single file.
First attempt at fixing the transform I didn't realize there was an RGBA image just chilling inside of imagenet-1k, so raised an exception bc I was expecting 3 channels instead of 4. I couldn't find any good suggestions for this, so I just tried to slice the tensor.
tv_transforms = T.Compose([
    T.Resize(opt.imageSize),
    T.CenterCrop(opt.imageSize),
    T.ToImage(),
    T.ToDtype(torch.float32, scale=True),
    T.Lambda(lambda x: x[:3]),
    T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ])

Then it was raising errors because there were grayscale images with just 1 channel it basically getting an index error (implicit data not resizable error). I figured there had to be a better option so I looked a little harder and found that probably the cleanest solution is to do the conversion while it's still a PIL image.
tv_transforms = T.Compose([
    T.Lambda(lambda x: x.convert("RGB")),
    T.Resize(opt.imageSize),
    T.CenterCrop(opt.imageSize),
    T.ToImage(),
    T.ToDtype(torch.float32, scale=True),
    T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
    ])

In terms of making it go fast, the config I ended up running was workers=16, batch_size=1024. Large batches didn't work well presumably because of CPU memory bottlenecking - the actual file write itself is relatively trivial (~500ms write vs. ~2s processing) so more frequent writes didn't end up being much of an issue.

Very naive benchmark, but HDF5 is pretty solid. I bet it's basically the performance of loading from a single file mmap'd numpy array, but with a more sane interface. The performance of just data access is pretty much the same w/ and w/o moving data onto GPU. I was surprised that by BS=8192 it actually finished loading all of imagenet-1k before flushing the buffer for the first time (~100s total).

danbooru2021 dataset incoming

Clearly loading from disk is a serious bottleneck I always knew this, but I never really felt the gap. I literally have one of the fastest read speeds you're going to get over PCIE 4.0 x4 (WD Black SN850X @ 7300MB/s read)
You're looking at best 6K img/s reading off disk vs. upwards of 300K img/s off RAM

After all this effort, I've found that np.memmap through torch dataloader just works pretty seamlessly and is 3x faster than hdf5 (mainly due to enabling parallelism) w/ data loader. It is 4x faster than HDF5 without having to pre-load everything into RAM.

many such cases

The dataloader has to be literally the biggest mystery ever... just out of curiousity I wanted to check if the torch dataloader would get in the way of trying to get 300K img/s when I pre-load the dataset into RAM.

I just tried pre-loading the memmap'd array into RAM w/ 8 workers (it was terrible)
Then I suspected it's due to whatever multi-process initialization causing it to be slow, so I changed it to 1 worker (assuming this gets around the problem, but still terrible)
Then I changed it to 0 workers because I assume this might have a behaviour more similar to loading data in main process, and it yielded 200K img/s, but still not quite the 300K img/s that I know is possible with a regular loop in the main process. Additionally, smaller batch sizes seemed to let me get higher throughput, but practically speaking when writing an actual training loop - this will probably be a terrible idea.

closing this out for now, will be actually doing the training loops and stuff like that - not stalling anymore


Doesnt work with shared projects.

It asks for my class name an then says that it couldn't find the project.json nor the csproj file. Shared projects don't have either of them, they have .shproj files.

Version 1.3
OS: Ubuntu 14.0.4


Error with old-style print function

Error with old-style print function
print`space`

TypeError: Cannot read property 'length' of undefined
TypeError: Cannot read property 'length' of undefined
at extractRange (/Users/-/.atom/packages/linter-flake8/lib/main.coffee:99:67)
at /Users/-/.atom/packages/linter-flake8/lib/main.coffee:177:22
Python lib:
flake8==2.4.1
pep8==1.6.2
linter-flake8 version 1.9.1

I'm also getting this error message in Atom 1.1.0 - on both OS X 10.11 and Ubuntu 14.04.   In trying to reproduce, it seems to happen consistently if there is folded code above the edit point, and not until a space is typed.
TypeError: Cannot read property 'length' of undefined
    at extractRange (/Users/chris/.atom/packages/linter-flake8/lib/main.coffee:99:67)
    at /Users/chris/.atom/packages/linter-flake8/lib/main.coffee:177:22

Plugins:
advanced-open-file: 0.12.0
atom-runner: 2.4.6
autocomplete-python: 0.10.3
linter: 1.10.0
linter-flake8: 1.9.1
pigments: 0.18.1

Dependencies:
flake8: 2.5.0
pep8: 1.5.7

Tried disabling all other plugins (all but linter and linter-flake8) with same result.

Looks like more issues with the workaround for flake8's terrible highlighting. The issue is that lineTextForScreenRow seems to be returning undefined for the line we are asking it for, I'll see if I have time to debug this further tonight.

Fix published in v1.9.2, enjoy and please report any further issues.

Hi all! maybe I'm too late to answer but I use Atom v1.12.x and I had this error. The way i resolved it was inserting the error code of this issue (E901) in ignoring error code in Atom/settings/packages search for linter-flake8. this workaround hide the issue when I insert a "print somevariable"

@anargu The currently released versions are impossible to reproduce this in, as the entire code path that caused the issue has been removed. It's rather more likely you are using an out of date version of flake8. If you are still seeing this after updating to flake8>=3.2.1 please file a new issue with code that shows this, thanks!


How to pass data and functions between Svelte and Elixir functions?

I am new to Phoenix/Elixir. This project seems to combine my two favorite  frameworks into one. Now I wonder how do you pass data back to Elixir from Svelte component? Via dispatch? How about passing to Svelte component a Elixir function? A simple increment example with a on click handler button will suffice to understand the flow. Thank you

So the way this lib works is that it essentially extends esbuild to package svelte components as single files, which are then attached to a div (and passed initial data via props). What this means is that the back-and-forth you get between Phoenix and Svelte is not as close as you would with an ecosystem using only JS for example (because then it becomes easier to pass a component to a function, for example).
What you can do is the following:

use fetch to call a POST route that you define in Phoenix, which will allow you to send and receive information from the Svelte component.
or, set up a websocket connection with Phoenix from the svelte component which will allow you to do the same thing, but is much more complex (i haven’t tried this personally, might be worth building in the support for this in the future).

Let me know if this clears things up a bit - I’ll be happy to provide an example when I’m not on mobile.

I guess a good way of thinking about it is to consider svelte components as microfrontends. It makes sense in certain cases, however it brings in lots of boiler plate and verbosity for trivial tasks. I wish Svonix could integrate more tightly Svelte into Phoenix. A good article to read on it is here https://nathancahill.com/phoenix svelte maybe could be an inspiration for future


[Tests] Exposing promise lib on global breaks async tests

The addition of
global.Promise = require('promise');

in
https://github.com/facebook/react-native/commit/3ff39870ce776c653823e1733363be0401896294
leads to all tests with Promises to not work because the Promises never resolve. When the line is removed everything works fine again.
The above mentioned commit does not mention the reason for the line and only talks about the other lines added (mocking the batched bridge).

cc @cpojer

Please see
https://github.com/facebook/react-native/issues/6104
Which has a simple test case to recreate the issue.

Collapsing this issue into #5410


Async context manager for BitcoinRPC

As noted in https://github.com/bibajz/bitcoin-python-async-rpc/issues/6#issuecomment-819139200, remembering to call BitcoinRPC.aclose to close the underlying httpx.AsyncClient feels repetitive and one may forget it.
Implement __aenter__ and __aexit__ coroutines which will handle the closing automatically.

Resolved in https://github.com/bibajz/bitcoin-python-async-rpc/commit/510ebe42616bb72a0db64e081be94002bc164fdf


AppSync upload of Complex Objects to S3 fails with CORS error on local mock server

Describe the bug
When utilizing Complex Objects (as described here) to automatically upload files to S3, the upload will fail when utilizing amplify mock due to a CORS error. Based on the access errors I am getting, it looks like the client is trying to upload to the S3 bucket in AWS instead of utilizing the mock version on localhost. If I switch the implementation to manually uploading the file using Storage, the correct behaviour happens and the file is uploaded to the mock bucket hosted locally.
When not running amplify mock, the complex objects upload functionality works correctly with the cloud resources.
To Reproduce
Implement a public file upload using complex objects as described in the docs. Run amplify mock. Attempt to upload a file.
Expected behavior
Files automatically uploaded utilizing the complex objects API should use the local S3 mock server instead of the cloud environment bucket when running amplify mock.

@smithad15
What frontend project do you have? (Javascript, Android or iOS? )
After you run amplify mock, did you rebuild your frontend project to configure the frontend client with the updated frontend configuration file? e.g. <project-root>/src/aws-export.js for javascript project.

I have a Vue frontend project. The aws-export.js file was correctly recompiled to point to local endpoints and I had restarted my frontend build server to ensure that it wasn't a build chain hiccup.

I have a Vue frontend project. The aws-export.js file was correctly recompiled to point to local endpoints and I had restarted my frontend build server to ensure that it wasn't a build chain hiccup.

Talked to @elorzafe offline and the issue is that AppSync SDK needs a way to support the mocked storage endpoint similar to how the Amplify storage category handles it.
Tranferring to Amplify JS repo.

Talked to @elorzafe offline and the issue is that AppSync SDK needs a way to support the mocked storage endpoint similar to how the Amplify storage category handles it.
Tranferring to Amplify JS repo.


Indexing with negative step sizes

Currently arrays cannot be indexed with negative step sizes.

Do you want to support this? It doesn't seem so easy actually, so maybe raising an error is the best.

The C++ side can almost handle negative step sizes, the only obvious roadblock I see is there being no way to specify seq.begin at the end of the array. Using -1 triggers a special code path which I don't quite understand the purpose.

I think the main issue is that I don't quite understand how seq() is supposed to work for negative steps.

Can this be closed after the PR ?

I'm afraid not, the issue is deeper. There's no easy way to do array[::-1] because there's no way (that I know) to specify that seq.begin corresponds to the end of the array in the C/C++ API. The fixes I did only handle cases where the begin and end are specified.

af::seq(end,0, -1) does not work ?

Nope it gives:
af::exception: ArrayFire Exception(203): Invalid input size
In /Users/filipe/src/arrayfire/src/api/cpp/array.cpp:725

Code:
#include <arrayfire.h>

int main(){
  af::array a = af::randu(3);
  af_print(a);
  af_print(a(af::seq(af::end,0,-1)));
}

OK, I think I can still handle this in python itself. If you are not working on this and #26 I will give a shot at fixing them.

If you look at calcDims when end is 0 you end up on:
DIM_ASSERT(1, seq.begin >= -DBL_MIN && seq.begin < parentDim);
which will fail as af::end is smaller than -DBL_MIN

I'm not working on it, I have workarounds for my code.

Ok, I think I am going to handle this upstream. The workarounds are ugly.

This is fixed in 3.1.1 release. I think we can safely close this.


Remove e2e test

The end-to-end test introduced in PR #63 uses ilab to run our code. This is wrong because the training library doesn't rely on the ilab CLI.

See my comment on #76 that explains the purpose of this workflow.


Schema/Object types

@daavoo do you or the DVC team have any strong feelings about using pydantic over voluptuous? I think it has better support for clearer type-hinting / IntelliSense

@daavoo do you or the DVC team have any strong feelings about using pydantic over voluptuous? I think it has better support for clearer type-hinting / IntelliSense

I prefer pydantic. The reason I used voluptuous is that it is already a dependency in DVC, to avoid introducing a new dependency

gto does and is being added to DVC, after that is complete I'd like to PR this and DVC to use pydantic over voluptuous.
Do you foresee any issues with that? cc @iterative/dvc


Training code

Hi cedriclmenard,
Could you please provide the training code for this repo that would be very helpful.
Thanks

Hi @shreyasashwath,
You can take a look at the class definition in irislandmarks.py, especially at the predict and the forward methods. Training can be done as with any PyTorch models if you feed the right data. For a simpler implementation of training, I suggest looking into PyTorch-Lightning.
The goal of this repo is to reproduce the same results using the same weights given by the original authors (Google's Mediapipe), this is why no training is implemented here.

Hi @shreyasashwath,
You can take a look at the class definition in irislandmarks.py, especially at the predict and the forward methods. Training can be done as with any PyTorch models if you feed the right data. For a simpler implementation of training, I suggest looking into PyTorch-Lightning.
The goal of this repo is to reproduce the same results using the same weights given by the original authors (Google's Mediapipe), this is why no training is implemented here.

@cedriclmenard Yes agree training can be done using any PyTorch model with right data.
Can we train this model with few key-points like if i have to extend this to face landmarks (98 keypoints)? and which places we should use Batch normalised layer ?

For face landmarks, I suggest looking at the FaceMesh repo. You can definitely use this model, but you'll have to modify it according to the output size you need.
Batch norm layers, if there were any in the original training implementation by Google Mediapipe, have been folded in the convolution weights, so you can figure where batch norm layers could go given that.


show how many timeouts a person has

Is your feature request related to a problem? Please describe.

show how many timeouts a person has
Describe the solution you'd like

show how many timeouts a person has if mod
Describe alternatives you've considered

not using chaterino
Additional context

There is no api for what you're suggesting from twitch.

There probably is some GQL or other undocumented endpoint we can use, but that would mean that every time the API changes the client breaks.

Chatterino does not use GQL endpoints since they're only meant for twitch's 1st part applications

You can do this as a mod by doing /user <user> - should show exactly how many timeouts they have.

The solution alazymeme posted is the supported one


⚠ SceneTime has degraded performance

In 0d6b7a0, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 1718 ms

Resolved: SceneTime performance has improved in 2bddd94 after 10 minutes.


Back propagation on ChebyShev polynomial

Hi @mdeff,

l'm wondering if Chebyshev polynomial  is concerned by the backpropagation or just the result of Chebyshev polynomial.

in
https://github.com/mdeff/cnn_graph/blob/master/lib/models.py#L495

    def _inference(self, x, dropout):
        with tf.name_scope('gconv1'):
            N, M = x.get_shape()  # N: number of samples, M: number of features
            M = int(M)
            # Transform to Chebyshev basis
            xc = tf.transpose(x)  # M x N
            def chebyshev(x):
                return graph.chebyshev(self.L, x, self.K)
            xc = tf.py_func(chebyshev, [xc], [tf.float32])[0]
            xc = tf.transpose(xc)  # N x M x K
            xc = tf.reshape(xc, [-1, self.K])  # NM x K
            # Filter
            W = self._weight_variable([self.K, self.F])
            y = tf.matmul(xc, W)  # NM x F
            y = tf.reshape(y, [-1, M, self.F])  # N x M x F
            # Bias and non-linearity
#            b = self._bias_variable([1, 1, self.F])
            b = self._bias_variable([1, M, self.F])
            y += b  # N x M x F
            y = tf.nn.relu(y)
        with tf.name_scope('fc1'):
            W = self._weight_variable([self.F*M, NCLASSES])
            b = self._bias_variable([NCLASSES])
            y = tf.reshape(y, [-1, self.F*M])
            y = tf.matmul(y, W) + b
return y

l would like to know if only :
 y = tf.nn.relu(y) https://github.com/mdeff/cnn_graph/blob/master/lib/models.py#L514
which concerned by backpropagation or also :
            xc = tf.py_func(chebyshev, [xc], [tf.float32])[0]
            xc = tf.transpose(xc)  # N x M x K
            xc = tf.reshape(xc, [-1, self.K])  # NM x K
            # Filter
            W = self._weight_variable([self.K, self.F])
            y = tf.matmul(xc, W)  # NM x F
y = tf.reshape(y, [-1, M, self.F]) # N x M x F

https://github.com/mdeff/cnn_graph/blob/master/lib/models.py#L503
?
Thank you a lot for your answer.

I don't understand the issue. Please rephrase and reopen if that's still relevant.


Integration test per Sketch file version

Make it easy to track the ability of our sketch parser to parse different versions of SketchApp by adding different version of demo files.

We will switch from protactor to cypress for all e2e and integration test.

Fixed in #175


Fix infinity symbols looking terrible in ui

Turns out this isn't ready to merge yet, I need to get some changes that were in #247 for icon support.


Rules like "Weekly on Tuesdays from 10:30 to 21:00"?

In my application I need to have rules like "Weekly on Tuesdays from 10:30 to 21:00".
I think it's impossible in this version of ice_cube? I'd love to see this feature in future versions, though.

If you can put together a pull request with a reasonable way to word that in the documentation I'd gladly accept

Sure... doing it :)

Closing this issue now


streamids limited to 100 chars

As far as I can tell, the datalink protocol places no limit on the length of the streamids, other then they have to fit within the 255 header limit. But dlclient.c, about line 611 assumes max length of 100. Probably 255 would be safer.
char replystr[100];
char streamid[100];
char flags[100];

True, the stream ID length limit is determined by the header limit combined with the length of the other fields (packed ID, flags, packed size, ...), which also depends on whether it's a WRITE or PACKET, er, packet.
ringserver actually limits the stream IDs to 60 bytes:
/* Define a maximum stream ID string length */
#define MAXSTREAMID 60

which, honestly is quite large.  This:
IU_ANMO_00_BH1/MSEED

is 20 bytes, and that can be tripled, just to visualize that:
IU_ANMO_00_BH1/MSEEDIU_ANMO_00_BH1/MSEEDIU_ANMO_00_BH1/MSEED

Even if we went full crazy with forthcoming FDSN URI-style identifiers, it's hard to imagine that 60 bytes doesn't buy a lot of head room.
So in this case ringserver is imposing a limit less than the protocol itself.  I'll review making the messages bigger and think about the stream IDs too.
Increasing the length of the stream IDs in ringserver does have some implications though: each packet has a buffer for the maximum length ID (memory is not dynamically allocated for each stream ID for performance and simplicity), this directly impacts the amount of RAM and disk that is used to store the ring buffer.  So for a ringserver like IRIS that is maxed out, having 16,777,215 packets in the buffer, increasing the stream ID by another 60 bytes adds 959 MiB of needed space to store the buffer.

Whatever restrictions on length are fine, but would be nice also to have that in the datalink protocol documentation.
Also, there are some weirdos out there that are trying to use the ringserver to send non-fdsn, non-miniseed data, and in that case the net_sta_loc_chan/MSEED streamid pattern may not apply. Its just a generic buffer, right? :)

Whatever restrictions on length are fine, but would be nice also to have that in the datalink protocol documentation.

That would be the wrong place for it.

Also, there are some weirdos out there that are trying to use the ringserver to send non-fdsn, non miniseed data, and in that case the net_sta_loc_chan/MSEED streamid pattern may not apply. Its just a generic buffer, right? :)

Of course, which is why it should be documented in ringserver, which is imposing the limit.

In addition to documentation, there should be a clear error message if a stream ID longer than supported is submitted.  Need to verify that.


Fix improper streaming in Azure Client

Ensure streaming works in "real-time"
fix: Ensure streaming chunks are emitted individually to the client
Problem:
The previous implementation combined multiple chunks into a single response, causing all data to be sent to the client at once instead of streaming each chunk individually. This behavior was due to the use of reduce and concatMapIterable, which aggregated the data before emitting it.
Solution:

Removed the use of reduce and concatMapIterable, which were combining chunks into a single response.
Updated the code to use flatMap on the window to ensure each item within the window is processed and emitted individually.
Streamed ChatCompletions directly to maintain the streaming behavior, ensuring that each chunk is processed and passed downstream as soon as it is received.

Changes:

Removed the reduce call to avoid combining chunks.
Replaced concatMapIterable with flatMap to process each chunk individually.
Modified the windowUntil logic to correctly handle function calls and stream each chunk separately.

This change ensures that each chunk of data is streamed to the client individually, providing a smoother and more immediate streaming experience.
Tested:

Verified that the client receives each chunk individually as it is streamed.
Confirmed that the function call handling logic works correctly with the updated streaming approach.

@joshlong @markpollack I think the issue lies with the actual Azure SDK client having a piece of logic that is making the streaming of the chunks a blocking call which essentially makes using streaming with Azure OpenAI and Spring AI impossible atm. Do you think this could get higher priority?

@bruno-oliveira I can not confirm your observation. In fact there are integration tests that confirm that the streamed response is indeed chunked for both function and non-function calls:

https://github.com/spring-projects/spring-ai/blob/16c531c36c143babdda59eff32b24ac05b291bc6/models/spring-ai-azure-openai/src/test/java/org/springframework/ai/azure/openai/function/AzureOpenAiChatModelFunctionCallIT.java#L89
https://github.com/spring-projects/spring-ai/blob/16c531c36c143babdda59eff32b24ac05b291bc6/spring-ai-spring-boot-autoconfigure/src/test/java/org/springframework/ai/autoconfigure/azure/AzureOpenAiAutoConfigurationIT.java#L87

Please review the above tests and let me know if there is an issue with them.
If you still think there is an issue please write a test that illustrates it.

@tzolov Thanks for reaching out! It's a pleasure to be able to engage with the community who builds such amazing tools that tons of devs use everyday!
I have created a ticket, not here, but in the Azure SDK repo: https://github.com/Azure/azure-sdk-for-java/issues/40629#issuecomment-2177944316
Note several people confirmed this was an issue and observe this exact behavior. If I make a "streaming call" in Postman, the response "blocks" until all the chunks are "internally streamed" which makes it exactly the same as the normal, non-streaming call.
Initially I started by investigating Spring AI library but that lead me to think that the bug needs to be in Azure SDK after all.
Maybe this one can be closed as its not directly related to Spring AI. However, for all effects and purposes, streaming "doesn't work" at the moment, with the Azure OpenAI client

@bruno-oliveira can you check https://github.com/spring-projects/spring-ai/pull/1054 if that solves the issue for you?


fix method compute amound line all

Problem with returning an item from the pos which was impossible causing an error message .
Fix that replaces Pull request #1477 :

https://github.com/itpp-labs/pos-addons/pull/1477

ping @gaelTorrecillas @yelizariev


[SPARK-48742][SS] Virtual Column Family for RocksDB

What changes were proposed in this pull request?

Introducing virtual column family to RocksDB. We attach an 2-byte-Id prefix as column family identifier for each of the key row that is put into RocksDB. The encoding and decoding of the virtual column family prefix happens at the RocksDBKeyEncoder layer as we can pre-allocate extra 2 bytes and avoid additional memcpy.

Remove Physical Column Family related codes as this becomes potentially dead code till some caller starts using this.
Remove useColumnFamilies from StateStoreChangelogV2 API.

Why are the changes needed?

Currently within  the scope of the arbitrary stateful API v2 (transformWithState)  project, each state variable is stored inside one physical column family within the RocksDB state store instance. Column families are also used to implement secondary indexes for various features. Each physical column family has its own memtables, creates its own SST files, and handles  compaction independently on those independent SST files.
When the number of operations to RocksDB is relatively small and the number of column families is relatively large, the overhead of handling small SST files becomes high, especially since all of these have to be uploaded in the snapshot dir and referenced in the metadata file for the uploaded RocksDB snapshot. Using prefix to manage different key spaces / virtual column family could reduce such overheads.
Does this PR introduce any user-facing change?

No. If useColumnFamilies are set to true in the StateStore.init(), virtual column family will be used.
How was this patch tested?

Unit tests in RocksDBStateStoreSuite, and integration tests in TransformWithStateSuite.
Moved test suites in RocksDBSuite into RocksDBStateStoreSuite because some previous verification functions are now moved into RocksDBStateProvider
Was this patch authored or co-authored using generative AI tooling?

No.

Also @jingz-db - test failure seems related ?
[error] Failed tests:
[error] 	org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreIntegrationSuite
[error] (sql / Test / test) sbt.TestsFailedException: Tests unsuccessful

Thanks! Merging to master.


Fix a couple of trivial typos in Postfix docstring

Please submit changes to http://propellor.branchable.com/todo/ instead of using github pull requests, which are not part of my workflow. -- A robot acting on behalf of Joey Hess


App crashed when deploy to BTP using MTA

Dear colleagues,
I was trying to deploy the mock server to BTP, and I got the following error logs:

Time
Event
Actor
Description

27 Jun 2023, 18:30:21 (GMT+02:00)
app.crash
mock-srv
instance: 8f7a18b2-eaac-4ecb-5c0e-6ac5, index: 0, cell_id: c839c7b1-088e-41af-b9b7-66256aacd457, exit_description: Instance never healthy after 1m0.246s: Timed out after 1m0s (30 attempts) waiting for readiness check to succeed: failed to make TCP connection to <IP_ADDRESS>:8080: dial tcp <IP_ADDRESS>:8080: connect: connection refused; process did not exit, reason: CRASHED

27 Jun 2023, 18:30:21 (GMT+02:00)
audit.app.process.crash
web
instance: 8f7a18b2-eaac-4ecb-5c0e-6ac5, index: 0, cell_id: c839c7b1-088e-41af-b9b7-66256aacd457, reason: CRASHED, exit_description: Instance never healthy after 1m0.246s: Timed out after 1m0s (30 attempts) waiting for readiness check to succeed: failed to make TCP connection to <IP_ADDRESS>:8080: dial tcp <IP_ADDRESS>:8080: connect: connection refused; process did not exit, crash_count: 1, crash_timestamp:<PHONE_NUMBER>598257000, version: d61a09ed-ce83-4647-8b87-d8242d0571b2

I am using a BTP trial account in Zone US10. Could you please help me with this?
Regards,
Zhongpin Wang

Hi @ZhongpinWang Please specify your node version.

Hi @navinkrishnan,
Thanks for the quick reply. These are the versions:
node: v16.20.0
npm: 8.19.4
cf cli: cf version 8.7.1+9c81242.2023-06-15
mbt: Cloud MTA Build Tool version 1.2.23
Regards,
Zhongpin

Now, it seems to work only after I change the instance memory from 256MB to 512MB.
Since the problem is "solved", I will close this issue now. But it would be nice if the memory requirement could be documented somewhere. Also in this mission Install Mock Server.
Thanks a lot!


AbstractHibernateEntityManagerFactoryIntegrator  Index 3 out of bounds for length 3

org.springframework.boot
spring-boot-starter-parent
2.5.8

SpringBoot version : 2.5.8
There are some errors in my app:
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for length 3
at com.blazebit.persistence.integration.hibernate.base.function.AbstractHibernateEntityManagerFactoryIntegrator.(AbstractHibernateEntityManagerFactoryIntegrator.java:93) ~[blaze-persistence-integration-hibernate-base-1.6.6.jar:1.6.6]
... 42 common frames omitted
I think it might appear in the version on divide operation

Are you using an uber-jar deployment? In that case you must provide the Hibernate version through a version provider SPI

Are you using an uber-jar deployment? In that case you must provide the Hibernate version through a version provider SPI

ok, seems I saw some error messages about uber-jar.  Could pls give some examples or docs about config provider SPI?

Here is the SPI you have to implement https://www.javadoc.io/doc/com.blazebit/blaze-persistence-integration-hibernate-base/latest/com/blazebit/persistence/integration/hibernate/base/spi/HibernateVersionProvider.html
You will need a compile dependency on baze-persistence-integration-hibernate-base to see this class though.


OptionFactory refactor

Unless I'm mistaken, it is currently very unintuitive how to set a field option via JS after it has been initialized:
// Either:
$input.parsley().OptionsFactory.staticOptions.trimValue = true
// or must be destroyed first:
$input.parsley().destroy()
$input.parsley({trimValue: true})

One can't simply change the options because these are reset by the OptionFactory by combining all different layers (e.g. global, form, field) of options.
An alternative approach to the OptionFactory approach would be to use a prototype chain instead. The global options would be the prototype of form data options, which in turn would be the prototype of the form static options, itself prototype of field data options, with the field static level being the last level.
The main advantage would be that it would allow setting options directly. For example, the following would work naturally:
$input.parsley().options.trimValue = true

This would work as actualizeOptions would only change (in place) the values for the prototypes of options, and keep the own properties of options intact.
Another advantage would be that it fixes the fact that the static level (which combine default options and options passed as a parameter) is currently overridden by data options. Especially for the field level, it makes no sense to me that specific options passed via JS would be overridden by data options of the form...
I think the solution would also be quite elegant (and consume a bit less memory, but that's really negligible in the big scheme of things).
Before I work on the PR, does this sound good to you?

Hi,
OptionsFactory is by far the worst bit of Parsley 2.x. Like you clearly stated above, this is very unintuitive, a bit overcomplicated a rework would clearly help.
I agree too that javascript options should prevail on data binded values.
I might see what you want to do here, that seems great, it may be clearer with your PR. Hope actual tests would help you do this refactoring peacefully :)
Best

Ok, so it wasn't too bad to write after all.
One drawback I didn't think of is that this all works beautifully, but I haven't coded for nested options. So someone doing $field.parsley().options.validators.foo = function() would actually modify the ParsleyConfig.validators object. I think validators is the only option that isn't flat, right?


Simple fix to use https wiki images.

We receive big count of redirects and mixed content warning because of http images.

Thanks for the patch! I'll merge and update the site asap.

Done! It even seems that images are loading much faster. Maybe the browser only uses one connection with HTTPS, I think I read something about that some time ago…


Configure online editor

This issue is part of the migration to v3. You can read full details about the various changes here.
In Exercism v3, students can now choose to work on exercises directly from their browser, instead of having to download exercises to their local machine. The track-specific settings for the in-browser editor are defined in the top-level "online_editor" field in the track's config.json file. This field is defined as an object with two fields:

"indent_style": the indent style, either "space" or "tab".
"indent_size": the indent size, which is an integer (e.g. 4).

You can find a full description of these fields in the spec.
Goal
The "online_editor" field should be updated to correspond to the track's best practices regarding indentation.
Example
"online_editor": {
  "indent_style": "space",
  "indent_size": 4
}

Tracking
https://github.com/exercism/v3-launch/issues/2

@ErikSchierboom this too should be closed... only CoffeeLint by default uses two spaces. Might be change the current value (of 4)?

What do most people use when writing coffeescript?

Two spaces. See this and this, two popular CoffeeScript repos here on GitHub.

Then it should be changed

@ErikSchierboom #230.


android.permission.READ_MEDIA_IMAGES requires approval

According to https://support.google.com/googleplay/android-developer/answer/14115180 the permission android.permission.READ_MEDIA_IMAGES (added in https://github.com/Cap-go/capacitor-inappbrowser/pull/132) is only allowed for "apps with a need for broad access to photos" and requires declaration and approval. Apps with one-time or infrequent use of photos are requested to use Android photo picker instead and remove READ_MEDIA_IMAGES.
Since our last dependency update we are prompted to submit a declaration or remove the permission.

If your app doesn't use the permission, add the following to your android/app/src/main/AndroidManifest.xml.

Add xmlns:tools="http://schemas.android.com/tools" to manifest tag attribute
Add  <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" tools:node="remove" /> element

Example:
<manifest xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools">
  <uses-permission android:name="android.permission.READ_MEDIA_IMAGES" tools:node="remove" />
</manifest>

tools:node="remove" will remove the permissions element from the merged manifest. See https://developer.android.com/build/manage-manifests#node_markers


test

Hi,
when we want to run test.py we need model
in line 49 of test.py
load_model = "model-80.out" # you should choose the proper model to load
what is model-80.out ?
where did you save it?
actually there is some problem with line 153
where:
model.load_state_dict(torch.load(load_model))
Thanks

Hi,
Yes, sorry the command to save the model is missing in the current train.py
I have updated the code to save the model ([here])(https://github.com/nadgeri14/KGPool/blob/master/Context-Aggregator/train.py#L252)
You can find the saved model at save_folder.

Thanks


ERROR: thread number (--thread) should be 1  16, suggest 1  8

Dear @sfchen,
Firstly I want to congratulate your work, fastp is a great tool.
Secondly, my issue: I am running fastp as part of a Nextflow pipeline named QUARS, where I can use as many threads as the user specify, however, when threads' number is greater than 16 I get the following fastp error:
ERROR: thread number (--thread) should be 1 ~ 16, suggest 1 ~ 8
After reading the issue #13 I understand why the limit. However, such error keeps halting the whole pipeline execution. Is there any way to disable such limit?
I look forward to hearing from you.
Sincerely
@TainVelasco-Luquez

I can change the error to warning in new version, but why you don't decrease the thread number?
8 threads are much more than enough since fastp is so fast

I just encountered this same issue. Ideally, fastp should simply adjust the number of threads accordingly, if necessary, and emit a warning documenting what occurred.

Done, now you can try it with latest code, or download the latest prebuilt binary from http://opengene.org/fastp/fastp
Next release will include this change.

Excellent.
Thank you for your answer and your package.
Tain.


Dark mode Version History table unreadable

What You Are Seeing?

What is Expected?
Being able to read the text inside the table better
How Did You Get This To Happen? (Steps to Reproduce)

Go to https://community.chocolatey.org/packages/codeql
Make sure Dark mode is enabled
Expand the Version History table

System Details

OS Build (In PowerShell run [System.Environment]::OSVersion.version.tostring()): N/A
Windows PowerShell version (Run: $PSVersionTable): N/A
Chocolatey version (Run choco --version): N/A

Output Log
N/A

After force refreshing the page:


java.io.InvalidClassException with Spark 3.1.2

This was reported on the mailing list.

----

Using spark downloaded from below link,

https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz

I get below error when submitting a pipeline.
Full error is on https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693.

------------------------------------------------------------------------------------------------------------------
21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection from /<IP_ADDRESS>:35601
java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID =<PHONE_NUMBER>829468865, local class serialVersionUID =<PHONE_NUMBER>549731694
at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
...
------------------------------------------------------------------------------------------------------------------

SDK Harness and Job service are deployed as below.

SDK Harness

sudo docker run --net=host apache/beam_spark3_job_server:2.31.0 --spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true
Job service

sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool

apache/beam_spark_job_server:2.31.0 for spark 2.4.8

SDK client code

https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2
 Spark 2.4.8 succeeded without any errors using above components.

https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz

Imported from Jira BEAM-12762. Original Jira may contain additional context.
Reported by: ibzib.
This issue has child subcomponents which were not migrated over. See the original Jira for more information.

I can confirm that I'm facing the same issue using a job services based on apache/beam_spark3_job_server:2.41.0 trying to run a beam pipeline on a bitnami/spark:3.1 based spark cluster.
stderror log from the task:
java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID =<PHONE_NUMBER>829468865, local class serialVersionUID =<PHONE_NUMBER>549731694
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:109)
	at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$deserialize$2(NettyRpcEnv.scala:299)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:352)
	at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$deserialize$1(NettyRpcEnv.scala:298)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at org.apache.spark.rpc.netty.NettyRpcEnv.deserialize(NettyRpcEnv.scala:298)
	at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$askAbortable$7(NettyRpcEnv.scala:246)
	at org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$askAbortable$7$adapted(NettyRpcEnv.scala:246)
	at org.apache.spark.rpc.netty.RpcOutboxMessage.onSuccess(Outbox.scala:90)
	at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:195)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:142)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

Based on my understanding, this issue is usually linked to having different versions of spark between the version deployed on the cluster and the client submitting the job into the cluster. In this case, I'm guessing the client is the job service.
I can also confirm, after running a quick test, downloading the same spark sdk (as the cluster version) locally and submitting a job from a local client to the cluster via something like:
./bin/spark-submit \
     --class org.apache.spark.examples.SparkPi
     --master spark://localhost:7077  \
     -- ./examples/jars/spark-examples_2.12-3.1.3.jar 10000

Works fine.
I don't know what goes on inside the beam spark job server but this leads me to thinking, that this is probably an incompatibility between what the spark client version is on the beam job server and what the target cluster is running.
Any idea if there is an easy work around, around this? Is there any way to know the exact version of Spark that the apache/beam_spark_job_server:2.31.0 would support? I could adapt the Spark cluster version to that if needed.
Would greatly appreciate any feedback on this. Thanks!

@aymanfarhat Just FYI, this is a known Scala issue. You can find more details on the problem here including hints on how to resolve it.
The original Beam Jira also contains a bit more context, see https://issues.apache.org/jira/browse/BEAM-12762.

This may be a known issue with Spark 3.1.2. I think the issue goes away with spark 3.1.3 onwards. Can beam be migrated to v3.1.3? I am trying to use Beam's Python SDK with Spark but this issue is preventing me to use it.

@aymanfarhat Just FYI, this is a known Scala issue. You can find more details on the problem here including hints on how to resolve it. The original Beam Jira also contains a bit more context, see https://issues.apache.org/jira/browse/BEAM-12762.

From the link you provided, it appears that Beam is using scala v2.12 and when we build beam it is being built using scala2.12.14 which is incompatible with Spark v3.1.2, How do we set the scala version for beam's job server?

@aromanenko-dev I'm not really familiar with the portable runner / the job server, would you have additional insights here?
As far as I can see, both beam-spark and beam-spark-job-server are using Scala 2.12.15.
Spark 3.1.2 / 3.1.3 depend on Scala 2.12.10 with the mentioned binary incompatibility, so I suspect the problem is in fact the cluster running with a broken Scala version.
An option might be the bump Beam to use Spark 3.2.2 or later (with Scala 2.12.15).

@mosche AFAIK, there is no specific Scala dependencies for Spark portable runner and job server. They are sitting in the same package branch and have the same Spark/Scala dependencies as a native runner. I'm sure that @ibzib should know much better than me.
Also, I'm wondering why Scala version was specifically set to 2.12.15 here since by default it is that way for Spark 3:
  spark_version = '3.1.2'
  spark_scala_version = '2.12'

Indeed, the right solution would be align the spark/scala versions. The question here is do we need to support the different versions or not?

@mosche AFAIK, there is no specific Scala dependencies for Spark portable runner and job server. They are sitting in the same package branch and have the same Spark/Scala dependencies as a native runner. I'm sure that @ibzib should know much better than me.
Also, I'm wondering why Scala version was specifically set to 2.12.15 here since by default it is that way for Spark 3:
  spark_version = '3.1.2'
  spark_scala_version = '2.12'

Indeed, the right solution would be align the spark/scala versions. The question here is do we need to support the different versions or not?

I tried setting scala to 2.12.10 in spark_runner.gradle; it did not work. Do I have to set scala version somewhere else?

Also what is the working and testing spark job server version and it's compatible Spark version. If I cannot use beam with Spark 3.1.2; the I will have to downgrade the Spark version.

@nitinlkoin1984 I finally found some time to look deeper into this. Sorry for the hassle, finding the job-server in this state is a bit disappointing.

Also what is the working and tested spark job server version and it's compatible Spark version.

Unfortunately this is a weakness of the existing test infrastructure, it uses Spark in local mode. In that setup such a classpath issue won't be discovered.
Anyways, I've done some testing:

You can fairly easily build yourself a custom version of the beam_spark3_job_server image on the latest Beam master for Spark 3.2.2 (or later). This versions of Spark are using Scala 2.12.15 and don't suffer from the Scala bug causing this. Here's the detailed steps how to do it:

Pull the source code of Beam from https://github.com/apache/beam and checkout tag v.2.14.0
Update the Spark version to 3.2.2 in these two places:
https://github.com/apache/beam/blob/f37795e326a75310828518464189440b14863834/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L496
https://github.com/apache/beam/blob/f37795e326a75310828518464189440b14863834/runners/spark/3/build.gradle#L23
In the project directory, run the gradle command to build the docker container:./gradlew :runners:spark:3:job-server:container:docker

This will build apache/beam_spark3_job_server:latest, it will be available for local use.

Downgrading Scala in the job-server image to 2.12.10 is also possible, but not as obvious. The Scala version is bumped by a transitive dependency.

Append the following lines to  runners/spark/3/job-server/build.gradleconfigurations.runtimeClasspath {
  resolutionStrategy {
    force "org.scala-lang:scala-library:2.12.10"
  }
}

In the project directory, run the gradle command to build the docker container:./gradlew gradle :runners:spark:3:job-server:clean
./gradlew :runners:spark:3:job-server:container:docker

This will build apache/beam_spark3_job_server:latest, it will be available for local use.

Alternatively you could build yourself a custom Spark 3.1.2 image that contains Scala 2.12.15 (instead of 2.12.10) on the classpath. But I don't think that's generally a feasible option.
Let me know if any of these options help!
@aromanenko-dev The 2nd option would fix the job-server without having to bump Spark. One the other hand bumping to Spark 3.2.2 seems to be the more robust and longterm solution. But the biggest concern there is the Avro dependency upgrade (1.10). What do you think? Anyone else who could chime in?


Introducing a delay on calculation of values associated to Slider

Is it possible to make sliders somehow pass on the value once the slider stops moving? I have somewhat heavy computations downstream of the slider, and would like to be able to only make them once the slider stops moving.
thanks!

Can you check if the PlutoUI.confirm function is what you need?
Take a look here: https://juliapluto.github.io/sample-notebook-previews/PlutoUI.jl.html

It really helps! I see only two problems with that particular implementation:

It's in english (the target audience are spanish speaking chemistry students)
I use multiple sliders, so it seems a bit cumbersome to have to confirm every change you decide to make (perhaps something along the lines of a single PlutoUI.Button would be better)

Thanks for your answer!

Hey all, happy new year!

I think that's a great point, @egonik-unlp, and I think that would be great additional functionality to add
@pankgeorg and the rest of the Pluto wizards have been hard at work to make this a reality! Here's a quick example from the Live docs for confirm (I just took a few liberties with the mixing and matching of markdown and html + libraries used):

using PlutoUI
using MarkdownLiteral: @mdx

@bind speeds confirm(
    PlutoUI.combine() do Child
        @mdx("""
        ### Wind speeds
        <ul>
        $([
            @mdx("<li>$(name): $(Child(name, Slider(1:100)))")
            for name in ["North", "East", "South", "West"]
        ])
        </ul>
        """)
    end
)

speeds

Let's discuss 1. in a new issue! https://github.com/JuliaPluto/PlutoUI.jl/issues/183

I think the best answer to the original question is a debounce, see #69 , #105 , https://lodash.com/docs/4.17.15#debounce and https://css-tricks.com/debouncing-throttling-explained-examples/ for more info

Thank you all very much!


summaryManagerCreateSummarizerMaxThrottleDelay errors in production

I see 286 instances of eventName = "fluid:telemetry:Container:ContainerWarning" && errro == "summaryManagerCreateSummarizerMaxThrottleDelay" over the last week (for latest version of runtime in prod).
I'm not 100% sure why we need throttler for starting summarizer client (this case actually might help us understand it better).
I'd expect that this error is either points us to something that needs to be fixed / adjusted (for example, that delay should be actually maxed out at this.startThrottler.maxDelayMs), OR (if we determine it's normal and expected flow) we remove this error logging from telemetry.
Relevant code:
    private async delayBeforeCreatingSummarizer(): Promise<boolean> {
        // throttle creation of new summarizer containers to prevent spamming the server with websocket connections
        let delayMs = this.startThrottler.getDelay();
        if (delayMs > 0 && delayMs >= this.startThrottler.maxDelayMs) {
            this.emit(
                "summarizerWarning",
                createSummarizingWarning("summaryManagerCreateSummarizerMaxThrottleDelay", false),
            );
        }

Note - it's widespread, i.e. affects a lot of documents  - 70 over last week to be exact:

union Office_Fluid_FluidRuntime_Error
| where Event_Time > ago(7d)
| where Data_error == "summaryManagerCreateSummarizerMaxThrottleDelay"
| summarize count() by Data_docId

The logic in getDelay() allows us to use a delay == maxDelayMs, which seems reasonable, but delayBeforeCreatingSummarizer expects delay < maxDelayMs.


CALL FOR CONTRIBUTORS

CALL FOR CONTRIBUTORS
I'm reaching out because I want to fix Slick. I mean, generally speaking, the happy path of slick is stable, and has been. However, there are indeed some bugs when getting fancy with the options, and that shouldn't be the case.
So I show up to github issues in the morning, and say "Hey man, what can I fix today?". I look around and I have a real hard time determining:

Priority
Actual bugs
What I should fix

I have super limited time to work on this, and I want to make the time I do have count. I'm reaching out because I need some help with issue management. Are there any fine folks who would like to go through the current issues list and do the following:

Close anything that that doesn't have a jsfiddle showing the issue being reproduced
Close anything that even remotely looks like a support request
Label issues as bugs with severity and by environment
Close PRs that are out of date or aren't valid anymore
Flag valid PRs as valid and test them

If anyone is up for the task, it would be amazing. People often ask if they can donate, and no I don't want money. Instead, help me make slick continue to be the best carousel out there.
I have a working 2.0 branch here https://github.com/kenwheeler/slick/tree/2.0 if anyone hasn't seen it, where I'm introducing some really cool stuff that will make slick more reliable. I'm setting the stage for CI integration to make pull requests easily verifiable, and adding modern tooling that brings this project up to current standards.
Also, as an aside, when I wrote this plugin I was writing carousels from scratch all the time. I haven't even implemented a carousel in over a year. I've been in React-land and more recently writing native applications in React Native full time. This puts me a bit out of touch with modern carousel needs. Please comment and let me know if I'm overlooking some cool new thing that I should be supporting but I'm not.
As always, thank you rascals so much for using this plugin, and I deeply apologize if I haven't responded to a PR or issue in a timely fashion. There are a ton of them and I just don't have the bandwidth and I feel terrible. Lets fix that.
Big shout out to @simeydotme and @ahmadalfy for helping me so far, you should all buy them a beer for being amazing.

I am open to culling the issues, etc...
I am currently using slick in a project, and I have encountered some of the complexity issues, as well as needs that I have hacked a solution for.  So it behooves me to pay attention to the project.

@Mcashley email me at<EMAIL_ADDRESS>

I'm down to help. Profoundly appreciate your efforts along with @simeydotme and @ahmadalfy over the years (also emailed).

@mcashley @leggomuhgreggo :+1:

I'll be available to help clean up any issues with this project. Whether it's cleaning up the issue list, or knocking out some of the issues themselves. I can be contacted at<EMAIL_ADDRESS>

firstly, thanks for the work you guys've done so far. use this on a personal project & have since introduced it to a project at work. so definitely up for helping you out with issues. will drop an email.

At work I use your awesome slider on now 9 sides in many different situations. Maybe you have noticed that I try to give you a little bit back.
You've got an email :smile:

@kenwheeler dropped you email but didn't have response - do you still need contributors?

How's this closed? o.0... I'm seeing 332 issues opened and 115 PRs... that's A LOT... =/ is this project discontinued? If it is, please, someone must warn it somewhere...

This issue isn't closed.

@leggomuhgreggo I may have seen it closed, seriously. Is it possible that it got opened after I've commented? =P Anyway, it's open now, thanks.

We are in year 2018 now. Is this project still maintained by anybody?

Probably not much.
I was also interest in to be a more active contributor in the past, I've also got in touch with @kenwheeler but later on things have happened and I couldn't assume this. I'm still unavailable right now, if something changes, I'll got in touch with you guys again.
Please keep the project, slick is still one of the best carousels out there.
I'm unsubscribing this issue.


Upgrade to React 0.14.

This required adding react-dom and fbjs as dependencies.

react-dom includes the ReactDOM.render and ReactDOM.findDOMNode
fbjs includes CSSCore

From there it was a mostly mechanical conversion.
Note this has not been extensively tested. I basically just tried the
demo.
Fixes #63

Thanks! I manually merged this commit into the react14 branch. I'll keep it in sync with the latest changes in master.


[SPARK-32039][YARN][WEBUI][2.4] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

What changes were proposed in this pull request?
When a Spark Job launched in Cluster mode with Yarn, Application Master sets spark.ui.port port to 0 which means Driver's web UI gets any random port even if we want to explicitly set the Port range for Driver's Web UI
Why are the changes needed?
We access Spark Web UI via Knox Proxy, and there are firewall restrictions due to which we can not access Spark Web UI since Web UI port range gets random port even if we set explicitly.
This Change will check if there is a specified port range explicitly mentioned so that it does not assign a random port.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Local Tested.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @Ngone51  @dongjoon-hyun Can you guys review?

Hi, @rajatahujaatinmobi . If this is required for 3.1/3.0, please make a PR to master branch.

Hi, @rajatahujaatinmobi . If this is required for 3.1/3.0, please make a PR to master branch.

Hi @dongjoon-hyun We are using Spark 2.4 as of now. So it is required in Spark 2.4 but yes I will make a separate Master Pull request too.

Thank you for confirming, @rajatahujaatinmobi . Apache Spark community have a backporting policy. To prevent any regression, we fix master/branch-3.0 first always. Otherwise, Apache Spark 2.4.7 may have this fix while 3.0.1 has a regression due to lack of this patch. So, please close this first and make a PR to master branch.

I left more comment on your your new PR. This issue was discussed before and technically SPARK-29465 has been tracking this as an Improvement. So, we cannot backport this. I'll close this first and let's gather our discussion on new PR against master branch.


ModuleNotFoundError: No module named 'weave.monitoring'

ModuleNotFoundError: No module named 'weave.monitoring'

Sorry about that, I just pushed a fix to the main branch.  It should work now.l

thanks,well done