LogicalKeyboardKey.arrow* constants don't map to keys encountered in events (macOS)

The LogicalKeyboardKeys I'm receiving from arrow key presses on macOS do not match up with the LogicalKeyboardKey.arrow* constants.
 They are meant to be used in different ways. If you change your layout, the physical key should always be the same, as opposed to the logical ones. They don't necessarily need to match. We get the the physical keys from the "keycode" value that macos provides.
Hope this helps.

I did a quick test with a slightly modified version of testbed, and I'm seeing the same thing Scott is. Logging event.data.logicalKey, event.data.physicalKey, LogicalKeyboardKey.arrowLeft and pressing the left arrow I see:
LogicalKeyboardKey#0f702(keyId: "0x0000f702", keyLabel: "", debugName: "Key ")
PhysicalKeyboardKey#70050(usbHidUsage: "0x00070050", debugName: "Arrow Left")
LogicalKeyboardKey#70050(keyId: "0x100070050", keyLabel: null, debugName: "Arrow Left"

Note that the logical key constant's ID matches the reported physical key, not the logical key. Seems like there's a bug in the code that's handling conversions between codes/ID for the different types.

Ah sorry, I see which constants you were referring to. IIRC, the challenge here is that macOS doesn't provide a logical keyCode, that is dependent on the layout, so we rely on the charactersIgnoringModifiers to generate the logical key, which doesn't match the constant.
@gspencergoog any ideas on how to approach this?

Maybe before generating the LogicalKey based on the charactersIgnoringModifiers, we should check for the modifier flags (in this specific case, NSEventModifierFlagFunction for arrow keys), and in that case, search for the constant based on the physicalKey.usbHidUsage as we do when keyLabel is null. I'll do some tests with that

Thanks for looking at this, Francisco.

Yes, the problem is probably that the platform is producing a Unicode character for those keys, whereas our pre-defined logical keys assume that those are not keys which produce a character.

Landed! (sorry for the delay)

Huzzah! Thanks :)


Cannot redeclare Mockery_3_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType::mockery_init()

Larastan Version: 5.7.3
--level used: default

Description

Installed phpstan 0.13.0
Installed larastan 0.2.10
ran php artisan code:analyse

Got this:
$ php artisan code:analyse
  56/525 [▓▓░░░░░░░░░░░░░░░░░░░░░░░░░░]  10%PHP Fatal error:  Cannot redeclare Mockery_3_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType::mockery_init() in /vagrant/api.project.io/vendor/mockery/mockery/library/Mockery/Loader/EvalLoader.php(34) : eval()'d code on line 905

   Symfony\Component\Debug\Exception\FatalErrorException  : Cannot redeclare Mockery_3_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType::mockery_init()

  at /vagrant/api.project.io/vendor/mockery/mockery/library/Mockery/Loader/EvalLoader.php: 34
  30:         if (class_exists($definition->getClassName(), false)) {
  31:             return;
  32:         }
  33:
  34:         eval("?>" . $definition->getCode());
  35:     }
  36: }
  37:

   Whoops\Exception\ErrorException  : Cannot redeclare Mockery_3_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType_Mockery_1_PHPStan_Type_ObjectType_PHPStan_Type_ObjectType::mockery_init()

  at /vagrant/api.project.io/vendor/mockery/mockery/library/Mockery/Loader/EvalLoader.php: 34
  30:         if (class_exists($definition->getClassName(), false)) {
  31:             return;
  32:         }
  33:
  34:         eval("?>" . $definition->getCode());
  35:     }
  36: }
  37:

Laravel code where the issue was found:
I don't know where this comes from, the larastan docs say by default it's run for app only where I don't use mockery.
Note: I've an existing phpstan only setup which does not exhibit this crash (tuned for level 6):
    "phpstan": "vendor/bin/phpstan analyse --level=6 --configuration=phpstan.neon app"

and phpstan.neon contains
parameters:
  bootstrap: vendor/autoload.php
  ignoreErrors:
    # Lots of errors I'm ignoring

But it does not work with 0.10.3 yet (in the sense I haven't fixed all errors, hence I'm trying laratan and see if I can reduce this).
I can't even say if this is really a larastan issue or just a accumulation of stuff.
Any pointers highly appreciated!

@mfn This issue got addressed on the version v0.2.11. Can you test it?

@nunomaduro awesome, works!


AMD vs Intel for WSL2

i was meant to ask. How compatible is WSL2 on AMD processors?  I've been using virtualization with Intel and I was wondering about using WSL2 with AMD.

This is a good question for reddit, twitter. Ref #4193 which has some discussion.

that's talking about nested VM and KVM. not what I wanted.  All I want to know if u enable WSL and Virtual Machine Platform. Does it work out of the box for AMD


fix(Component): props name conflict

Deploy preview ready!
Built with commit f8e16baef8edcc1cec8deb7fddf0696ad680e5e6
https://deploy-preview-3--react-progressive-bg-image.netlify.com


memory usage optimization

Reduced memory usage for better performance according to our heavy load testing

Correction pushed. it fixes the regression

I stronglyy advise against subclassing ThreadLocal, it causes nasty memory leaks with used inside a servlet container. If you insist on using ThreadLocals, please initialize it lazily instead (with a null-check).

Changes commited
2015-03-19 17:22 GMT+01:00 jochenberger<EMAIL_ADDRESS>
I stronglyy advise against subclassing ThreadLocal, it causes nasty
memory leaks with used inside a servlet container. If you insist on using
ThreadLocals, please initialize it lazily instead (with a null-check).
--
Reply to this email directly or view it on GitHub
https://github.com/jayway/JsonPath/pull/71#issuecomment-83650080.

@srouthiau I'm about to release version 2.0 but I will not have time to look into @jochenberger's concerns about ThreadLocal. I will not include this in pull request in the upcoming release. Do you see any alternatives to ThreadLocal and can you provide some stats about the memory using?

I'd also be interested in some benchmarks. Does it really make a difference? The JVM has become pretty good at creating and throwing away large numbers of objects.

I would recommend against memory optimizations that use ThreadLocal which can lead to memory leaks.
I would suggest you add a custom hook that users can pass in. For example, you could create an interface that clients can implement:
interface BufferFactory { StringBuilder createBuilder(); }
But even then...

I don't think this is an issue. ThreadLocals are likely to produce more issues that they solve.


Update README.md

Formatting Issues Fixed: I fixed formatting issues where there were extra asterisks at the beginning and end of the "Contribution Guidelines" section. The guidelines now appear as a markdown table properly.
Consistent Alignment: I ensured that headings and text are consistently aligned, which improves the overall readability of the README.
Improved Section Headers: I replaced  tags with markdown headers (#) for consistency, and I made section headers more prominent.
Added Links: I ensured that all links are properly formatted and working. This includes links to external websites, contribution guidelines, and more.
Tech Stack Section: I formatted the "Tech Stack Used" section to display technology badges more clearly and consistently.
Contributors Section: I added a note to acknowledge and thank contributors for their contributions.
Clarified Mentors/Maintainers: I provided more clarity on the role of project mentors/maintainers.
Overall Readability: The improved formatting and alignment make the README more visually appealing and easier to read.
These changes enhance the overall presentation and readability of the README, making it more informative and user-friendly for visitors to your GitHub repository.
Fixes Issue
Closes #2069

Changes proposed

Screenshots

Checklist

[✅ ] You've linked this PR to the correct issue.

[ ✅] You have checked that the code is working correctly.

[✅ ] You ⭐ the repository!

Note to reviewers

The proposed changes in the README include fixing formatting issues, improving alignment and visual consistency, verifying and enhancing links, and providing clear acknowledgments to contributors. These changes make the README more organized, informative, and user-friendly. We welcome your contribution to our project, and these improvements will help provide a better experience for contributors like you.


 No, limiting the date widget width was done on purpose, and it works fine. See the example from the demo:

Can you show us the visual difference between before and after your PR?

I had an example, if I reproduce it again I post it here, want to close this until then?
I've been thinking that row, label and field classes should be configurable, I've been thinking on how to implement this, when I make a demo I open an issue.

Found it!

I can give you guys access to our server so you can look at it by your selfs if anyone is willing to look at it. Just look for me at gitter.

Did you customize the CSS?

@fzaninotto nope, the date field is set to col-sm-4 by default which is too short for displaying this dates.

Superseded by #673


pretty-quick --check is fixing files

Hi,
I wanted to use pretty-quick --check during CI to prevent that not formatted code is landing in master. But the --check property seems not to work correctly
pretty-quick version: 2.0.1
yarn pretty-quick
yarn run v1.22.4
$ \node_modules\.bin\pretty-quick
�🔍  Finding changed files since git revision 159bf55e4.
�🎯  Found 1 changed file.
✍  Fixing up add-user.actions.ts.
✅  Everything is awesome!
Done in 3.02s.
PS yarn pretty-quick --check
yarn run v1.22.4
$ node_modules\.bin\pretty-quick --check
�🎯  Finding changed files since git revision 159bf55e4.
�🔍  Found 1 changed file.
✍  Fixing up add-user.actions.ts.
✅  Everything is awesome!
Done in 2.47s.

--check and without check are doing the fixing but I just want to fail the job with --check flag

Hey @elvirdolic. Are you still experiencing this? This isn't the expected behaviour at all. If you are able to gather some extra diagnostic info that would go a long way to solving your issue.

A minimal but runnable online reproduction is required.


Update copyright notice to 2016 [ci skip]

Merged this, but the README ought to be reformatted in general

You mean its reads Copyright 2014 and needs to be changed to 2016?


Make CI build docs

We need to run yarn build:docs on Zeit's build so that we can preview what happens when docs are generated.

done


[BUG] {camera.launch.py not working}

Describe the bug
I can't get topics related to depth image. Here is the bug report when launching ros2 launch depthai_ros_driver camera.launch.py
It is the similar error with ros2 launch turtlebot4_bringup oakd.launch.py
Before the node is down, I can see oak topics with rgb and stereo in topic lists(I attached the picture below). I wanna use all the topics for extracting the data, so I think launching camera.launch.py is the solution but it is down showing an error message.
So, what do I do execute the code successfully?
Please help.
and I have some questions.
Q1.
In an existing camera.yaml, there is no content related to rgb and stereo.
Then, what is the reason of showing topics related to them?
Q2.
When I read the official document, if I use RGBD as the value of i_pipeline_type, it says that I have to include left & right camera content.
But, an existing camera.yaml doesn't include left & right camera content.
Q3.
Can you tell me what the required parameters are if I modify the yaml file to customize topics?
System information
I am using ROS2 humble on Turtlebot4 (ubuntu 22.04).
Attach system log
[INFO] [launch]: All log files can be found below /home/ubuntu/.ros/log/2024-04-09-06-21-36-602339-ubuntu-1579
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container-1]: process started with pid [1593]
[component_container-1] [INFO] [1712643700.753357558] [oak_container]: Load Library: /opt/ros/humble/lib/librobot_state_publisher_node.so
[component_container-1] [INFO] [1712643700.762664553] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1712643700.762804922] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<robot_state_publisher::RobotStatePublisher>
[component_container-1] [INFO] [1712643700.841839772] [oak_state_publisher]: got segment oak
[component_container-1] [INFO] [1712643700.842024252] [oak_state_publisher]: got segment oak-d-base-frame
[component_container-1] [INFO] [1712643700.842071844] [oak_state_publisher]: got segment oak_imu_frame
[component_container-1] [INFO] [1712643700.842107548] [oak_state_publisher]: got segment oak_left_camera_frame
[component_container-1] [INFO] [1712643700.842139307] [oak_state_publisher]: got segment oak_left_camera_optical_frame
[component_container-1] [INFO] [1712643700.842169547] [oak_state_publisher]: got segment oak_model_origin
[component_container-1] [INFO] [1712643700.842197843] [oak_state_publisher]: got segment oak_rgb_camera_frame
[component_container-1] [INFO] [1712643700.842225880] [oak_state_publisher]: got segment oak_rgb_camera_optical_frame
[component_container-1] [INFO] [1712643700.842255732] [oak_state_publisher]: got segment oak_right_camera_frame
[component_container-1] [INFO] [1712643700.842283861] [oak_state_publisher]: got segment oak_right_camera_optical_frame
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/oak_state_publisher' in container 'oak_container'
[component_container-1] [INFO] [1712643700.865168112] [oak_container]: Load Library: /opt/ros/humble/lib/libdepthai_ros_driver.so
[component_container-1] [INFO] [1712643701.389025588] [oak_container]: Found class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1712643701.389210476] [oak_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<depthai_ros_driver::Camera>
[component_container-1] [INFO] [1712643701.513268889] [oak]: No ip/mxid specified, connecting to the next available device.
[component_container-1] [INFO] [1712643704.428852394] [oak]: Camera with MXID: 18443010F1D6990F00 and Name: 1.1.2 connected!
[component_container-1] [INFO] [1712643704.430665175] [oak]: USB SPEED: SUPER
[component_container-1] [INFO] [1712643704.478820471] [oak]: Device type: OAK-D-PRO
[component_container-1] [INFO] [1712643704.482657735] [oak]: Pipeline type: RGBD
[component_container-1] [INFO] [1712643704.840804483] [oak]: NN Family: mobilenet
[component_container-1] [INFO] [1712643705.228526574] [oak]: NN input size: 300 x 300. Resizing input image in case of different dimensions.
[component_container-1] [INFO] [1712643705.619172529] [oak]: Finished setting up pipeline.
[component_container-1] [18443010F1D6990F00] [1.1.2] [1712643709.256] [host] [warning] Device crashed, but no crash dump could be extracted.
[component_container-1] terminate called without an active exception
[ERROR] [component_container-1]: process has died [pid 1593, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container --ros-args --log-level info --ros-args -r __node:=oak_container -r __ns:=/'].

List of topics checked before node is turned off

Thanks for response, and I provide more information.
Describe the bug
Running ros2 launch depthai_ros_driver camera.launch.py, the node is down.
Expected behavior
Running ros2 launch depthai_ros_driver camera.launch.py or ros2 launch turtlebot4_bringup oakd.launch.py, I expect topics below to extract the data.
/oak/rgb/camera_info
/oak/rgb/image_raw
/oak/rgb/image_raw/compressed
/oak/rgb/image_raw/compressedDepth
/oak/rgb/image_raw/theora
/oak/stereo/camera_info
/oak/stereo/image_raw
/oak/stereo/image_raw/compressed
/oak/stereo/image_raw/compressedDepth
/oak/stereo/image_raw/theora
Attach system log

Provide output of log_system_information.py
Which OS/OS version are you using? Ubuntu 22.04.4
Which ROS version are you using? Ros2 Humble
Which ROS distribution are you using ?
Is depthai-ros built from source or installed from apt? built from source (git clone --branch humble https://github.com/luxonis/depthai-ros.git)
Is depthai/depthai-core library installed from rosdep or manually? For rosdep install, check if ros-<rosdistro>-depthai package is installed, manual install can be checked with ldconfig -p | grep depthai
Please include versions of following packages

Package: ros-humble-depthai Version: 2.24.0-1jammy.20240308.165854
Package: ros-humble-depthai-ros Version: 2.9.0-1jammy.20240327.182037
Package: ros-humble-depthai-bridge Version: 2.9.0-1jammy.20240327.165719
Package: ros-humble-depthai-ros-msgs Version: 2.9.0-1jammy.20240219.090343
Package: ros-humble-depthai-ros-driver Version: 2.9.0-1jammy.20240327.180744

To get additional logs, set DEPTHAI_DEBUG=1 and paste the logs, either from command line or from latest log in ~/.ros/log

Additional context

Q3.
Can you tell me what the required parameters are if I modify the yaml file to customize topics?
A3.
Not sure what you exactly mean here.

I have referred to various issues to solve the error.
I noticed one solution of them( here ).
It said that in order to enable the stereo topic, the oakd_pro.yaml file needed to be modified by referring to the parameters. ( It is for launching ros2 launch turtlebot4_bringup oakd.launch.py )
As you know, there are many parameters.
So, if I have to modify the camera yaml file, I wonder what parameters should be included or removed, and what values should be set to.

Thanks for response.
I provide debug logs below.

To do that, run export DEPTHAI_DEBUG=1 and then ros2 launch depthai_ros_driver camera.launch.py

The process stops with a message indicating that it is dead.
debug_log1.txt

As I can see, turtlebot4_bringup oadk.launch.py uses rgb_stereo_node.launch.py from depthai_examples package which uses a different driver and won't react to the same parameters.

There is no rgb_stereo_node.launch.py in oakd.launch.py and I found rgb_stereo_node.launch.py in oakd_example.launch.py.
So, I launch ros2 launch turtlebot4_bringup oadk_example.launch.py
The process dies the same as above.
debug_log2.txt

Could you also try running the driver on your host PC just to make sure everything is OK with the camera?

If I run the driver on the host PC, the driver can't recognize the camera.
As far as I know, shouldn't we run it on the TurtleBot 4 Raspberry Pi image?
You can see [oak]: No ip/mxid specified, connecting to the next available device [oak]: No available devices in the debug log.
Could you tell me the right way to use it, if I'm mistaken?
debug_log3.txt

Regarding parameters, to get depth you can set camera.i_pipeline_type: RGBD or Depth.

On your advice, I set the parameters' values launching the codes.
As you check the debug logs, it isn't adjusted by set values.
ros2 launch depthai_ros_driver camera.launch.py camera.i_pipeline_type:=RGBD left.i_publish_topic:=true right.i_publish_topic:=true
debug_log4.txt
ros2 launch depthai_ros_driver camera.launch.py camera.i_pipeline_type:=DEPTH left.i_publish_topic:=1 right.i_publish_topic:=1
debug_log5.txt
Lastly, I have a question about executing the OAK-D sensor.
As I know, before launching the files related to OAK-D sensor ( ex. turtlebot4_bringup or depthai_ros_driver pkgs ), I have to press 'Restart' or 'Install' in turtlebot4-setup, is it right?
I always press 'Restart' before executing launch files.
It's been working well for a while, but an error message suddenly pops up, and the sensor is not recognized. ( [oak]: No ip/mxid specified, connecting to the next available device [oak]: No available devices )
Thank you for taking the time.

Hi, I tried connecting the sensor with Host PC directly, and it was a success.
I attached the result of image and the log.
As you see, the node loades completely, and there are topics I expected.
In the result, it is sure that the launch code and yaml files are not the point of the problem.

Hi, I tried connecting the sensor with Host PC directly, and it was a success.
I attached the result of log.
As you see, the node loades completely, and there were topics I expected.
success_log.txt
In the result, it is sure that the launch code and yaml files are not the point of the problem.
I think the problem is for networking between OAK-D sensor connected with Raspberry Pi image and Host PC.
In addition, I don't think it is a Wi-Fi issue at least because I kept checking when I first executed the Turtlebot4.
So, I don't know what the problem is.
If you grasp the problem, I would like to ask your opinion concerning this issue.
I look forward to hearing from you.
Thank you.

Hi, sorry for the delay, just to make sure, could you also test if you could run some python depthai examples? In general, if camera works on PC without issues then the issue might be somewhere on the turtlebot side, I would contact their team and verify if the setup is up to date.

Hi, thanks for response!
I executed 'hello_world.py' in depthai_tutorials and here the result is.
I think there is no error.
After you contact the turtlebot team, I would appreciate it if you could share the answers.


feat(learn): Add two static pages for use with the current curriculum and project-based curriculum

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the master branch of freeCodeCamp.
[ ] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

This PR fixes a bug that has been present for quite a while relating to invalid urls being used as links in the Basic HTML section.
The first such link is introduced in the Link to External Pages with Anchor Elements challenge.  The user is told to create an anchor element that links to http://freecatphotoapp.com which is a non-existent website.  This results in a 404 error if the user happens to click on the link in the Preview.  To remedy this problem, I created a new html file located at client/static/cat-photos/index.html.  It is a fairly plain page with an h1 element and two images (of cats).
The second url is introduced in the Create a Form Element challenge.  The user is told to add action="/submit-cat-photo" to the form element.  The issue arises when the user clicks on the Submit button in the next challenge (Add a Submit Button to a Form).  Currently, nothing actually happens which is a problem in itself. I have created a new html file located at client/static/submit-cat-photo/index.html that contains a generic thank you message to simulate what a user might see if they were actually submitting the form.
I believe both of these changes make the app being created more realistic.  Also, we plan on using these same URLs for the project-based curriculum's Basic HTML Cat Photo App.  In fact, the https://www.freecodecamp.org/cat-photos/index.html is used in several steps.
Note to reviewers:

Husky complained about several of the non-English files not having an extra line between the the last set of triple backticks and last closing </secton> tag, so you will see several files where I had to add that extra line.
I went ahead an changed the url for all the languages, instead of creating separate PRs for the different languages.  I changed no other content in the non-English files.

We do control the domain, its config seemed to have been changed during a move from one registrar to another.
We have pointed it to freecodecamp.org for now, but I assume we should be able to point it to independent static pages like we had in the past.

@raisedadead In that case, where can I see the source code for that site.  Would rather use that domain for the links or would it be fine to proceed with just putting them in the static folder as I have already done (they appear to work locally)?

where can I see the source code for that site.

The old code was lost in the move, but it was only a image and embedded in a html file. We can create a new repository and put static files in there. We could use netlify to host the site.
If you would like that let me know, I can setup an empty repo for you to work with and hook up the deployment on netlify once you are ready.

I prefer the cdn approach if possible, just so we can keep /learn focused.

Let's use Netlify for these two static pages.  We may have a use for the same site for other pages as we work through designing the content for the next curriculum. Should we also move the cats.json file to the Netlify site?

This repo is ready: https://github.com/freeCodeCamp/freeCatPhotoApp. I will deploy it once you have something in there.

Should we also move the cats.json file to the Netlify site?

Ideally, yes.  I'll just check if any tests depend on the location, but it should be an easy migration.

Okay the site is live (with a maintenance page), feel free to make a PR and remove it.


Export Options for Mesh Colours and Double-Sided

Thanks for creating a really useful tool!
A couple of queries:

I really like the option to export meshes with the display colour if no material assigned to the object. This is really handy. However I also have some meshes where I'd like to export the vertex colours IF they are assigned. Would it be possible to have the behaviour so that if both "use display colour for objects with no material set" AND "export vertex colours" are ticked then meshes with vertex colours are exported with them while those with no vertex colours are given the display colour? If this is too niche a use case then please ignore me.
Would it be possible to have an option to export open meshes so they are double-sided? For a simple floor plane object it's handy to limit face count in the model by having it double-sided?

Thanks again for a great tool!

In order to support the display color when no material is assigned a gltf material with the display color is created.
https://github.com/Stykka/glTF-Bin/blob/28e8828dc9cb61fd12477c9613ab62edbde35105/glTF-BinExporter/RhinoDocGltfConverter.cs#L195-L200
Vertex colors are always written if they exist and the option is enabled
https://github.com/Stykka/glTF-Bin/blob/28e8828dc9cb61fd12477c9613ab62edbde35105/glTF-BinExporter/RhinoMeshGltfConverter.cs#L180-L183
I'm guessing whatever renderer you're using is defaulting to the material instead of vertex colors. I can add an extra check to only create a material from the display color if no vertex colors are present.
It looks like doubleSided triangles are supported.
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#double-sided
I'll see how best to wire this up.


Download microservice: Check task status, plus some refactoring

Main Point (bug fix)
Should Fix: https://github.com/pulsar-edit/package-frontend/issues/71
First commit in this PR fixes an issue with failing to find built binaries.
Details of the issue (click to expand):
The issue happened because, after we narrow in on the most-recent overall "COMPLETED" (passing) status build, we were implicitly assuming all tasks (jobs) were also passing. So we only checked that these tasks were the OS+arch we were looking for. Whereas it's possible for a job to fail and be re-run, for instance if Cirrus aborts and auto-reschedules the task. (Or if the first run simply fails and a subsequent run passes.)

Solution:
Indeed, overall passing builds can contain failing tasks/jobs (which would have been later re-run until they passed, but the failing tasks/jobs are still stored separately in there). We must filter for only the passing tasks/jobs.
Miscellaneous Changes (refactoring/renaming/pruning some stuff)
The other commits I have included in this job are refactoring some code style, renaming a few variable names, and pruning out any query parameters on the Cirrus queries where we're not currently using the resulting data.
For details of the refactoring/renaming/pruning stuff, please see the commit messages and diff for details.
Thoughts / Priorities
It's not a super huge PR, hopefully reviewer(s) will not mind the refactoring bits. In my view, the bug fix (first commit) is my main priority, the rest is more debatable or subjective and I would be willing to back those out (or move them to a separate PR) if it helps this PR be more easily reviewable and mergeable.
Thanks for taking a look.

Here's some debugging data recorded for posterity, and to aid with testing:
Note: Highly recommended to see https://cirrus-ci.com/explorer to help explore these queries.
First Query (GraphQL) -- repository
query getRepositoryBuildStatuses {
  repository(id:<PHONE_NUMBER>158528) {
    builds(branch: "master", last: 10) {
      edges {
        node {
          id
          status
        }
      }
    }
  }
}

First Query Output (JSON) -- repository
{
  "data": {
    "repository": {
      "builds": {
        "edges": [
          {
            "node": {
              "id": "5781658918453248",
              "status": "COMPLETED"
            }
          },
          {
            "node": {
              "id": "5477904067330048",
              "status": "COMPLETED"
            }
          },
          {
            "node": {
              "id": "6441209531465728",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "5784867191914496",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "6071996828614656",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "5774358245801984",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "5881383696990208",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "6372356373872640",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "5447264106184704",
              "status": "FAILED"
            }
          },
          {
            "node": {
              "id": "6289185137491968",
              "status": "FAILED"
            }
          }
        ]
      }
    }
  }
}

Second Query (GraphQL) -- build
query GetTasksFromBuild {
  build(id:<PHONE_NUMBER>453248) {
    tasks {
      name
      id
    }
  }
}

Second Query Output (JSON) -- build
{
  "data": {
    "build": {
      "tasks": [
        {
          "name": "linux",
          "id": "4698661909692416"
        },
        {
          "name": "arm_linux",
          "id": "5824561816535040"
        },
        {
          "name": "silicon_mac",
          "id": "5261611863113728"
        },
        {
          "name": "intel_mac",
          "id": "6387511769956352"
        },
        {
          "name": "windows",
          "id": "4980136886403072"
        }
      ]
    }
  }
}

Third Query (GraphQL) -- task
query GetTaskDetails {
  task(id:<PHONE_NUMBER>692416) {
    name
    artifacts {
      name
      files {
        path
        size
      }
    }
  }
}

Third Query Output (JSON) -- task
{
  "data": {
    "task": {
      "name": "linux",
      "artifacts": [
        {
          "name": "binary",
          "files": [
            {
              "path": "binaries/Pulsar-1.101.2023021106.AppImage",
              "size": 227462595
            },
            {
              "path": "binaries/pulsar-1.101.2023021106.tar.gz",
              "size": 227794903
            },
            {
              "path": "binaries/pulsar-1.101.2023021106.x86_64.rpm",
              "size": 144555980
            },
            {
              "path": "binaries/pulsar_1.101.2023021106_amd64.deb",
              "size": 142098590
            }
          ]
        },
        {
          "name": "junit",
          "files": [
            {
              "path": "report.xml",
              "size": 5546
            }
          ]
        },
        {
          "name": "videos",
          "files": [
            {
              "path": "tests/videos/opening-first-time/5f1ccac952efa7bd7c0b8dacd6a90014.webm",
              "size": 68922
            },
            {
              "path": "tests/videos/opening-first-time/d9029511d71f7529495adaa7f483ad09.webm",
              "size": 4481010
            }
          ]
        }
      ]
    }
  }
}

This is all stuff which would have given our download microservice no problems, even before/without this PR's fix.
I can post the queries/responses that broke our microservice in the next comment.
The first query+response are our Cirrus repo build statuses as of a few hours ago, and the second query+response are for the most-recent passing/"COMPLETED" build as of then, and lastly the third query+response are for the x64 Linux task from that build.

In this comment, I'll be posting data corresponding to the build that temporarily broke Linux downloads from our download microservice. This is again for debugging purposes, and recording the data for posterity.
Let's pretend the first query from the comment above still returned this "problematic" (for us) build as the most-recent passing CI run for our repo in Cirrus.
Second Query (GraphQL) -- build ("problematic" build, no status)
query GetTasksFromBuild {
  build(id:<PHONE_NUMBER>330048) {
    tasks {
      name
      id
    }
  }
}

Second Query Output (JSON) -- build ("problematic" build, no status)
{
  "data": {
    "build": {
      "tasks": [
        {
          "name": "linux",
          "id": "6043622328500224"
        },
        {
          "name": "linux",
          "id": "6249477560860672"
        },
        {
          "name": "linux",
          "id": "5403887687958528"
        },
        {
          "name": "arm_linux",
          "id": "5480672375078912"
        },
        {
          "name": "arm_linux",
          "id": "6529787594801152"
        },
        {
          "name": "arm_linux",
          "id": "6421753262505984"
        },
        {
          "name": "silicon_mac",
          "id": "6606572281921536"
        },
        {
          "name": "silicon_mac",
          "id": "4700200246181888"
        },
        {
          "name": "intel_mac",
          "id": "4776984933302272"
        },
        {
          "name": "intel_mac",
          "id": "5826100153024512"
        },
        {
          "name": "windows",
          "id": "5902884840144896"
        },
        {
          "name": "windows",
          "id": "5263150199603200"
        }
      ]
    }
  }
}

Note there are multiple tasks for each OS+arch, unlike the second query results from the comment above. This means there were re-runs.
I'll post below the modified query/results that can reveal which tasks are properly passing...
Second Query (GraphQL) -- build (modified to show status)
query GetTasksFromBuild {
  build(id:<PHONE_NUMBER>330048) {
    tasks {
      name
      id
      status
    }
  }
}

Second Query Output (JSON) -- build (modified to show status)
{
  "data": {
    "build": {
      "tasks": [
        {
          "name": "linux",
          "id": "6043622328500224",
          "status": "ABORTED"
        },
        {
          "name": "linux",
          "id": "6249477560860672",
          "status": "COMPLETED"
        },
        {
          "name": "linux",
          "id": "5403887687958528",
          "status": "COMPLETED"
        },
        {
          "name": "arm_linux",
          "id": "5480672375078912",
          "status": "COMPLETED"
        },
        {
          "name": "arm_linux",
          "id": "6529787594801152",
          "status": "ABORTED"
        },
        {
          "name": "arm_linux",
          "id": "6421753262505984",
          "status": "COMPLETED"
        },
        {
          "name": "silicon_mac",
          "id": "6606572281921536",
          "status": "COMPLETED"
        },
        {
          "name": "silicon_mac",
          "id": "4700200246181888",
          "status": "COMPLETED"
        },
        {
          "name": "intel_mac",
          "id": "4776984933302272",
          "status": "COMPLETED"
        },
        {
          "name": "intel_mac",
          "id": "5826100153024512",
          "status": "COMPLETED"
        },
        {
          "name": "windows",
          "id": "5902884840144896",
          "status": "COMPLETED"
        },
        {
          "name": "windows",
          "id": "5263150199603200",
          "status": "COMPLETED"
        }
      ]
    }
  }
}

So, without filtering for "COMPLETED" tasks, we would look at the first task matching each OS+arch. Here's one of the "ABORTED" runs -- this one is for x64 Linux -- which we would have selected before:
Third Query (GraphQL) -- task ("ABORTED" task)
query GetTaskDetails {
  task(id:<PHONE_NUMBER>500224) {
    name
    artifacts {
      name
      files {
        path
        size
      }
    }
  }
}

Third Query Output (JSON) -- task ("ABORTED" task)
{
  "data": {
    "task": {
      "name": "linux",
      "artifacts": [
        {
          "name": "junit",
          "files": [
            {
              "path": "report.xml",
              "size": 5549
            }
          ]
        },
        {
          "name": "videos",
          "files": [
            {
              "path": "tests/videos/opening-first-time/07dbf3440cab2bac8824af7b79c74ba5.webm",
              "size": 4164235
            },
            {
              "path": "tests/videos/opening-first-time/8ca9d366a34fd24370ab0aac175a862a.webm",
              "size": 66627
            }
          ]
        }
      ]
    }
  }
}

Note there's no "binaries" artifacts, only the JUnit report.xml, and the video output assets (from the visual tests).
Here are the query/results for the passing Linux x64 task:
Third Query (GraphQL) -- task ("COMPLETED" task)
query GetTaskDetails {
  task(id:<PHONE_NUMBER>860672) {
    name
    artifacts {
      name
      files {
        path
        size
      }
    }
  }
}

Third Query Output (JSON) -- task ("COMPLETED" task)
{
  "data": {
    "task": {
      "name": "linux",
      "artifacts": [
        {
          "name": "binary",
          "files": [
            {
              "path": "binaries/Pulsar-1.101.2023021001.AppImage",
              "size": 227462559
            },
            {
              "path": "binaries/pulsar-1.101.2023021001.tar.gz",
              "size": 227793134
            },
            {
              "path": "binaries/pulsar-1.101.2023021001.x86_64.rpm",
              "size": 144545556
            },
            {
              "path": "binaries/pulsar_1.101.2023021001_amd64.deb",
              "size": 142100498
            }
          ]
        },
        {
          "name": "junit",
          "files": [
            {
              "path": "report.xml",
              "size": 5548
            }
          ]
        },
        {
          "name": "videos",
          "files": [
            {
              "path": "tests/videos/opening-first-time/08f3763f4835446877cf8a846c37e007.webm",
              "size": 54269
            },
            {
              "path": "tests/videos/opening-first-time/f04d8dc70a46c22126ca51fb3bb69490.webm",
              "size": 3840364
            }
          ]
        }
      ]
    }
  }
}

Note: See the .Appimage, .tar.gz, .rpm and .deb binary assets.
We should obviously select the "COMPLETED"/passing run, which does actually include the binaries we are attempting to redirect/link to for folks to download.
This PR (particularly the first commit in this PR) sets us up to do that.

Testing Notes
I tested this PR locally, with the following patch to always query the "problematic" build<PHONE_NUMBER>330048 that this PR aims to handle properly:
patch (click to expand)
diff --git a/microservices/download/utils.js b/microservices/download/utils.js
index 61a9483..6b5f9e9 100644
--- a/microservices/download/utils.js
+++ b/microservices/download/utils.js
@@ -139,7 +139,7 @@ async function findLink(os, type) {

     let buildQuery = `
       query GetTasksFromBuild {
-        build(id: "${buildID}") {
+        build(id: "5477904067330048") {
           tasks {
             name
             id

And the following JS script to quickly exercise the findLink() function that this PR updates...
Script get-url.js (click to expand)
const utils = require('./utils');

(async () => {
  console.log(
    await utils.findLink('linux', 'linux_deb')
  );
})();

Sample output with the above patch and script, with/without this PR's changes, showing the fix:
Sample output (no fix, error) (click to expand)
{
  ok: false,
  code: 503,
  msg: 'Invalid TYPE Download Parameters Provided'
}

Sample output (with fix, working) (click to expand)
{
  ok: true,
  content: 'https://api.cirrus-ci.com/v1/artifact/task/6249477560860672/binary/binaries/pulsar_1.101.2023021001_amd64.deb'
}

By the way, I didn't have to read plain pages of docs, just copy paste the queries we use, like repositoryQuery or buildQuery, (the part within the  `` template strings) into the Cirrus API Explorer: https://cirrus-ci.com/explorer
I noticed the part of our files here that resembled the example query, and I copy-pasted one into there. Such as:
Example Query to Copy-Paste Into Cirrus API Explorer
query getRepositoryBuildStatuses {
  repository(id:<PHONE_NUMBER>158528) {
    builds(branch: "master", last: 10) {
      edges {
        node {
          id
          status
        }
      }
    }
  }
}

Then I was like "gee, it'd be nice if these tasks we're getting returned to us had statuses, hey I bet they do, I'm going to plug in the word status in this area where we seem to be specifying the bits of data we get about tasks" and hey it worked.
And you can hover and/or click (double click?) some keywords in the query half of the page, and it'll bring up a sidebar and jump to the definition of some of the API data. Like, you can get the definition of the repository and what types of data you can query for under/in it.
It's a lil' confusing, since there's a page for the data type itself as like an entity, but there's a separate page for what stuff it has in it. But I had a very specific thing I wanted out of the API and I got the hang of it enough to squeeze that one thing out of it.

@DeeDeeG thanks for the further info here, I'll have to keep that in mind when doing another PR here to improve things, but it's appreciated!


update-packages error message because of dart pub get usage

Reproduction
flutter update-packages --force-upgrade
Seen results
The following error happens, but upgrading seems to have worked.
$ flutter update-packages --force-upgrade
Upgrading packages...
WARNING: the following packages use hard-coded version constraints:
  - flutter_gallery_assets: 1.0.2
  - flutter_template_images: 4.0.0
  - shelf: 1.1.4
  - video_player: 2.2.11
Running "flutter pub upgrade" in flutter_update_packages.5VmtFB...      2,612ms
Because release_smoke_test depends on integration_test from sdk which doesn't exist (the Flutter SDK is not available), version solving failed.

Flutter users should run `flutter pub get` instead of `dart pub get`.
pub get failed (69; Flutter users should run `flutter pub get` instead of `dart pub get`.)

Expected results
No error message.

hmmm...

Hmm, this command works for me locally.
This is suspicious, and makes me think your local setup is wrong?
Because release_smoke_test depends on integration_test from sdk which doesn't exist (the Flutter SDK is not available)

@christopherfujino I do have local changes in device_lab tests.  I'm running the update-packages in the flutter directory itself.  I don't have any local changes for or related to release_smoke_test.  Does it matter where you run that command from?

I was able to get update-packages to work by going directly to master.  I was running it from a branch of master.  I'm not sure why my changes would have broken it but that was the workaround.


please help me i dont know what to do! please!

hi, i need to start with im really new to python (i had not done python before this series u made) but i understand what im doing and  the basics. but i get so much problem with tensorflow now and i have tried for long and i cant se fix anywhere on internet. error:
WARNING:tensorflow:From C:\Users\ththo\AppData\Local\Programs\Python\Python36\lib\site-packages\tflearn\initializations.py:119: UniformUnitScaling.init (from tensorflow.python.ops.init_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior.
WARNING:tensorflow:From C:\Users\ththo\AppData\Local\Programs\Python\Python36\lib\site-packages\tflearn\objectives.py:66: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
Traceback (most recent call last):
File "C:\Users\ththo\Desktop\Gta 5 AI Test\train_model.py", line 25, in
model.fit({'input': X}, {'targets': Y}, n_epochs=EPOCHS,
NameError: name 'X' is not defined

i cant fix it an i really need to fix it to continue. thank you for any help!!

Well it is saying that X was not defined?...
Send your code for train_model.py


Clean up a bit travis.yml

Remove installation of npm poltergeist package. It seems a typo.
--
TravisCI's recommended way to run Xvfb https://docs.travis-ci.com/user/gui-and-headless-browsers/#Using-xvfb-to-Run-Tests-That-Require-a-GUI

Note: Don’t run xvfb directly, as it does not handle multiple concurrent instances that way.

--
The npm config set spin false speeds things a bit up.

This change is

Changes Unknown when pulling 3d2e7d90ad4f4d924321fe24df62ec1cad0787a6 on AlexKVal:travis-config into ** on shakacode:master**.

@alexkval Thanks!
@dylangrafmyre You'd be better to review this me. If this looks good to you, let's merge this.

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

@dylangrafmyre Travis passed, FWIW.

@justin808 @alexkval I have not run the ci rake task locally for this repo for a while.
I do not know what is required for it to run successfully.
However, if it can pass by using PhantomJs and Poltergeist than maybe some of this .travis.yml before_install items can be deleted?
Sometime all of these things get patched together to make it work because of web driver prior failures with react,etc. Maybe now it might be worth stripping down the the basics in the .travis.yml and testing if the suite works with only phantomjs and poltergeist?
But if this is what it takes now to get CI passing that is fine too. Just sometimes you need to revisit the CI config because of changes to the node_modules and gems that have adapted and provide better support for our tool chain.

Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.

Comments from Reviewable

Maybe now it might be worth stripping down the the basics in the .travis.yml and testing if the suite works with only phantomjs and poltergeist?

I've tried to run tests with PhantomJS instead of Chome on TravisCI.
https://travis-ci.org/AlexKVal/react_on_rails/builds/151347102
PhantomJS crashes.

I did not do any additional research about it.
Though locally<EMAIL_ADDRESS>all runs OK.

Coverage remained the same at 82.36% when pulling 4b644cae46c6aaffdd240da033280827cb29a568 on AlexKVal:travis-config into a6373f0870a2006db5356a0a6c43db3af025dcf3 on shakacode:master.

I've made "install" section a bit simpler and added caching.

Coverage remained the same at 82.36% when pulling 4ff4656b47fe1a3332726ad476f77d47670a2378 on AlexKVal:travis-config into a6373f0870a2006db5356a0a6c43db3af025dcf3 on shakacode:master.

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

.travis.yml, line 34 [r2] (raw file):

install:
  - nvm install 5

@alexkval @dylangrafmyre Why not version 6? or just latest node or LTS:
https://github.com/creationix/nvm#usage

Comments from Reviewable

@alexkval Here is an example for getting phantomjs 2.1 + installed within .travis.yml
https://github.com/testdouble/present/blob/57deb323731c23607d4e3cafd709ec062d9b6666/.travis.yml

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

.travis.yml, line 34 [r2] (raw file):
Previously, justin808 (Justin Gordon) wrote…
> @alexkval @dylangrafmyre Why not version 6? or just latest node or LTS:
>
> https://github.com/creationix/nvm#usage

It should be updated to version 6.

Comments from Reviewable

Thank you for the help. I'll check it out and report on it.

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

.travis.yml, line 34 [r2] (raw file):
Previously, dylangrafmyre (Dylan Grafmyre) wrote…
> It should be updated to version 6.

I wasn't sure if it is right to propose the node's version update.
Then should I rewrite it as `nvm install 6`
or `nvm install node` to always get the latest stable ?

Comments from Reviewable

Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks failed.

.travis.yml, line 34 [r2] (raw file):
Previously, AlexKVal (Alexander Shemetovsky) wrote…
> I wasn't sure if it is right to propose the node's version update.
> Then should I rewrite it as `nvm install 6`
> or `nvm install node` to always get the latest stable ?
>

I think the latter. I want to know if one day we become incompatible with the latest.

Comments from Reviewable

I've just changed npm install node and added node -v.
--
TravisCI now has<EMAIL_ADDRESS>by default.
It crashes.
I've tried the provided link with<EMAIL_ADDRESS>It also crashes.
https://travis-ci.org/AlexKVal/react_on_rails/builds/152966747

Coverage remained the same at 82.36% when pulling e14e8dc28bad825335e15c49367413923cfea92d on AlexKVal:travis-config into ea9575985ce9518b63eb3b72dcfb55189b037273 on shakacode:master.

Coverage remained the same at 82.36% when pulling e14e8dc28bad825335e15c49367413923cfea92d on AlexKVal:travis-config into ea9575985ce9518b63eb3b72dcfb55189b037273 on shakacode:master.

Reviewed 1 of 1 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

Comments from Reviewable

Thanks @AlexKVal!


List available categories before creating resource

Using terraform I would like to list available categories for a type of resource before resource is created. Possible to support listing categories just by setting --resource-type, not requiring --resource?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 55b103cb-dc59-8949-663f-f968a48d204e
Version Independent ID: 387ce519-f57b-e817-6046-08552966744a
Content: az monitor diagnostic-settings categories
Content Source: src/azure-cli/azure/cli/command_modules/monitor/_help.py
Service: monitoring-and-diagnostics
GitHub Login: @rloutlaw
Microsoft Alias: routlaw

@MyronFanQiu please take a look.

@tesharp Hello. I need to check the server whether it's possible or not. Do you mean that terraform can  support such feature?

Terraform supports the data source azurerm_monitor_diagnostic_categories today, but that requires a resource_id as input. So if I want to create a resource + diagnostic settings for it in same module I would like to do something like:
resource "azurerm_something" "main" {
  ...
}

data "azurerm_monitor_diagnostic_categories" "main" {
  resource_id = azurerm_something.main.rd
}

resource "azurerm_monitor_diagnostic" "main" {
  dynamic "log" {
    for_each = azurerm_monitor_diagnostic_categories.main.logs
    ....
  }
}

However, what happens now is that since the resource azurerm_something does not exist the data source returns a list with 1 entry it seems when running plan. So last monitor resource says it want to create 1 log entry in dynamic block. It does calculate the correct order, so when applying resources the data source can return many more entries since resource now exists and it apply fails since there is an inconsistency between plan and apply.
As long as the underlying azure command requires resource_id I don't see a way to solve it in terraform unless underlying command can support asking on resource type. Then terraform could instead of referencing resource_id use resource_type and plan would be correct as it knows beforehand what log entries to make.

@tesharp Hello. I checked the dependency, looks like we cannot do it currently. I will invite the corresponding people to take a look at this issue.

@gucalder Hello. I saw your commit in swagger spec repo. Could you help me about this issue or help me route this issue to the right person?

@gucalder  and update ?


Repeat, Center, Rotation Texture parameters are not supported

is pathtracer not support uv mapping ?

testPathtracerblender.zip
houseglb.zip

UV mapping is supported but some texture parameters like repeat, center, and rotation are not fully supported since the textures must be written to a texture array. The repeat factor of your texture is set to (0.005, 0.005) which means only a small portion of the texture is written to the texture array.
I recommend keeping your texture repeat parameters in the range [0.0, 1.0] for now and updating your floor UVs to accommodate. Perhaps the extra parameters can be supported once #75 is addressed.

I can put a $200 USD bounty on this issue for an accepted PR. Sponsored by Threekit.

Great! @alvinalexander The following things need to happen at a high level:

The RenderTarget2DArray class needs to strip the texture transforms before rendering.
The texture transforms (center, rotation, repeat) need to be stored in the MaterialTextures definition.
The UV transforms need to be applied in the shader before sampling the textures.

As far as I understand three.js only supports one set of UV transforms per material and takes one of the map UVs but I'm not sure what priority they are used with. I'd have to do some digging in three.js to find out.
Adding a UV Texture to a plane and rendering it with different UV offsets would be a good way to test. Let me know if you have any questions! Happy to provide pointers along the way if you want to start a draft PR.

Thanks for the outline @gkjohnson !
Based on the above discussion I have narrowed it down to these parameters from the Three.js texture specification:

repeat: Vector2d, where x and y  > 0
offset: Vector 2d where x and y in [0,1]
wrapS: Clam, RepeatWrapping, MirroredRepeatWrapping
wrapT: Clam, RepeatWrapping, MirroredRepeatWrapping
center: Vector2d where x and y in [0, 1]
rotation: in [0, 2*PI]

Is the goal for this PR to support all these parameters for all textures, or can we limit it to subset, say center, rotation, repeat and wrap[s|t] in [RepeatWrapping, Clamp] ?
Also, when you say:

The RenderTarget2DArray class needs to strip the texture transforms before rendering.

do you mean that these values must be reset to Three.js defaults before the textures are rendered to the array?

The repeat (vec2), offset (vec2), center (vec2), and rotation (float) properties from the Texture docs are what I'm looking at - that will be an additional 7 floats per texture for at least 6+ textures.
The "wrapS" and "wrapT" values are a bit complicated because we can only set one mode per render target array and all textures are packed into that one target. We could emulate the "clamp" behavior in code but maybe that can be a next PR.

The RenderTarget2DArray class needs to strip the texture transforms before rendering.

do you mean that these values must be reset to Three.js defaults before the textures are rendered to the array?

Right. At the moment the textures are being transformed before being written into the render target array which breaks when UVs outside of [ 0, 1 ].
So with @bhouston's suggestion to specify these coordinates per texture here's what this looks like now:

The RenderTarget2DArray class needs to strip the texture transforms before rendering.
Add 12 new pixels (2 per texture) to the end of each MaterialsTexture definition with the above values.
Read each bit of texture info into the shader in this function and extend the MaterialStruct definition to have new texture transform fields. I think it would be cleanest to create a new TextureTransform struct to encapsulate all the definitions in if that makes sense.
The UV transforms need to be applied in the shader before sampling the textures.

Support all texture transforms is definitely more robust but will require more texture sampling. We should be able to cut down on that per texture if the texture isn't being used, though. But lets get it working first!

BTW I believe that this:

The repeat (vec2), offset (vec2), center (vec2), and rotation (float)

Can simply be represented by a 3x3 matrix, but you only need the top 6 elements, because the last row/column is (0,0,1) always - so I would add that texture transform to the parameters, rather than these individual elements.
The ThreeJS Texture class always has the ability to return a matrix I believe that combines these all together:
https://threejs.org/docs/#api/en/textures/Texture.matrix

Thanks @bhouston ! I was going to construct the UV transformation matrix inside the shader using those values but it's great to know that ThreeJS has a utility function for this.
This is the approach I am considering :

Disable the UV transform before rendering inside https://github.com/gkjohnson/three-gpu-pathtracer/blob/33936d3eaad18064ba849b6bf4f85e6bc5256682/src/uniforms/RenderTarget2DArray.js#L38
For each Texture, get  Texture.matrix an write the first two rows (6 floats) into the floatArray https://github.com/gkjohnson/three-gpu-pathtracer/blob/33936d3eaad18064ba849b6bf4f85e6bc5256682/src/uniforms/MaterialsTexture.js#L126
Read the UV matrix items inside readMaterialInfo and construct a mat3 uvTransformMat
Apply transform before sampling textures.


add integration support, found bugs?

It would be of great help if you would add integration support into other projects, as in adding the ability to import functions from you module.
I have tried this workaround:
in __init__.py adding from .cli import main
and in my code from pdf2pptx import main as create_pptx
and using it like this create_pptx(["INPUT","-o OUTPUT"])
which allowed me to use debug features on your code. only to find out that certian attributes and methods names' are defined slightly diffrently (maybe becuase of an update) e.g:
pageCount ->page_count
loadPage ->load_page
getPixmap -> get_pixmap
but I got stuck on image_data = pixmap.getPNGData() at line 63
as I didnt seem to find that function or something similar
would appreciate a fix, thanks

I've updated the code for the new version of the PyMuPDF API. I'll look into making it easier to integrate later

I updated the code to make integration easier. There is now a function in __init__.py called convert_pdf2pptx that can be called directly by other programs. You can also use the quiet=True option to suppress console output if needed.


Slippage tolerance should be set automatically

Is your request related to a problem? Please describe.
A clear and concise description of what the problem is.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@Antusias9 Thank you for your suggestions, we've taken your request and noted it within our potential enhancement document list!


oil: 0.8.10 -> 0.8.11

simple package update
@lheckemann leaving this open for a couple of days in case you are interested.
Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.


Fix mypy type check

Codecov Report

Merging #165 (7cf228b) into master (80256ad) will decrease coverage by 0.14%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master     #165      +/-   ##
==========================================
- Coverage   18.84%   18.70%   -0.15%
==========================================
  Files         123      123
  Lines       15537    15501      -36
==========================================
- Hits         2928     2899      -29
+ Misses      12609    12602       -7

Impacted Files
Coverage Δ

...s_contrib_drivers/drivers/SignalCore/SignalCore.py
0.00% <0.00%> (ø)

...b_drivers/tests/QDevil/test_real_qdac2_ieee_std.py
31.81% <0.00%> (-10.49%)
:arrow_down:

...ontrib_drivers/tests/QDevil/real_qdac2_fixtures.py
30.90% <0.00%> (-3.58%)
:arrow_down:

...contrib_drivers/tests/QDevil/sim_qdac2_fixtures.py
78.78% <0.00%> (-1.22%)
:arrow_down:

...trib_drivers/drivers/ZurichInstruments/ZIHDAWG8.py
55.33% <0.00%> (-0.88%)
:arrow_down:

...odes_contrib_drivers/tests/test_Keysight_M3201A.py
15.22% <0.00%> (-0.76%)
:arrow_down:

...ivers/drivers/Keysight/SD_common/memory_manager.py
79.64% <0.00%> (-0.36%)
:arrow_down:

...contrib_drivers/drivers/ERAInstruments/erasynth.py
0.00% <0.00%> (ø)

...contrib_drivers/tests/test_rohdeschwarz_HMC804x.py
100.00% <0.00%> (ø)

...ontrib_drivers/drivers/Keysight/Keysight_E8267D.py
0.00% <0.00%> (ø)

... and 7 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[HB-5791] Fix Notes & Warnings in GitHub Documentation

Description
We want to implement the following in our documentation:

Note
This is a note

Warning
This is a warning

This PR fixes that!

This is cool. I assume this will only work for GitHub?

Correct


High TTFB with PageSpeed

Hi,
TTFB is much higher when PageSpeed is enabled and I don't understand why
264ms TTFB with PageSpeed=off
https://speedvitals.com/report/www.keinegotterkeinemeister.de/eEuqEG/
1100ms TTFB with PageSpeed ON
https://speedvitals.com/report/www.keinegotterkeinemeister.de/tg1zwa/
Pagespeed config:
https://pastebin.com/fLqVcP1M

Thanks for the data including the config; it's very helpful.
It does look like the performance is pretty good even from the US, despite the TTFB. I assume the server is in Germany with no US CDN.
Of course TTFB will never get faster with PageSpeed, only slower. The question is whether the optimizations it makes during that time provide a better user experience.
I visited view-source:https://www.keinegotterkeinemeister.de/tshirt-sabotage-black-cat-D01004174288P0210/?PageSpeedFilters=+debug to see what timing info PageSpeed injected at the end, and found this:
#NumFlushes            0
#EndDocument after     5906us
#Total Parse duration  5685us
#Total Render duration 411563us
#Total Idle duration   221us
No critical images detected.
The following filters were disabled for this request:
CriticalSelectorFilter: No critical selector info in cache
DelayImages

So it looks like your server spent 5.7 seconds in PageSpeed parsing your HTML -- that's mostly CPU bound. Then it spent 452ms rendering it -- that may be a combination of image optimization, network operations, waiting for asynchronous optimizations and URL fetches (depending on the cache setup) That leaves 1100-270= 378ms unaccounted for.
I'm guessing that most of this comes from the file cache. It's possible your disk is I/O bound. My main suggestion is to try deploying memcached or redis as an asynchronous cache. That should be faster than the file-cache.

Hi, thanks for the reply
The server is a dedicated server in OVH Datacenter in Beauharnois, Quebec, Canada
Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz (16 core(s)) - 2499 MHz
64 GB Ram
NVME SSD
1GBPS network
CPU load is always under 25% https://prnt.sc/Xte_RHbEGSgR
We also integrated a benchmark into the website to calculate the page load speed of the different PHP modules and calculate SQL queries execution time. You can see it on the bottom of the page when adding ?benchmark=1 to the URL
https://www.keinegotterkeinemeister.de/tshirt-sabotage-black-cat-D01004174288P0210/?benchmark=1
Currently, it takes 0.2110 seconds to generate the page with PHP on the first page load. But out of this 0.21 seconds, 0.1865 is due to calling an external geolocation API. This is just because Cloudflare is currently disabled for our testing phase. When Cloudflare is enabled, we get the geolocation from $_SERVER["HTTP_CF_IPCOUNTRY"] so we avoid calling the external API and page load time goes down to around 0.01s
TTFB is not the only problem, we also have high LCP and it throws warnings in Google Search Console: https://prnt.sc/2PjB74ekUgdR
Our search ranking dropped a lot and we lost 50% of our search traffic since June 2021 and we suspect it could be partly due to that
How can I confirm whether the disk is I/O bound or not? I followed Plesk guide and here are the results:
https://support.plesk.com/hc/en-us/articles/360006437794-Plesk-for-Linux-server-is-running-slow-too-much-IO-interactivity
[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=dir                                                                              ect 5000+0 records in 5000+0 records out 2560000 bytes (2.6 MB, 2.4 MiB) copied, 13.634 s, 188 kB/s [root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=direct 5000+0 records in 5000+0 records out 2560000 bytes (2.6 MB, 2.4 MiB) copied, 14.1703 s, 181 kB/s [root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=direct 5000+0 records in 5000+0 records out 2560000 bytes (2.6 MB, 2.4 MiB) copied, 14.0026 s, 183 kB/s
This seems pretty slow. Could it be because we run pagespeed from too many domains?
Latency seems fine:
[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=1024M count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.05548 s, 522 MB/s

however iotop looks perfectly fine. Monitored it for around 10 minutes and IO never went above 1%
https://prnt.sc/F65mP1MkTeBC
also no high %WA with top:
https://prnt.sc/Oe1hhgcqrNDC

I was suspicious there may be spinning-disk latency but as you have an SSD I'm not sure. Nevertheless it might be worth trying redis/memcached, as we never really tuned the file-cache for SSD.

There could be an IO issue because according to the Plesk guide, the first test I ran should get a speed of 142 MB/s but I got only 183 kB/s
Memcached is now installed but I'm not sure how to check if it's correctly configured
https://www.keinegotterkeinemeister.de/phpinfo.php
I enabled statistics and I can see some data related to Memcached so it seems to be working:
https://www.keinegotterkeinemeister.de/mod_pagespeed_statistics?memcached
ModPagespeedMemcachedServers localhost:11211
ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 51200

Is 51200 enough? We have plenty of memory on the server and the websites are big with thousands of images
It seems to have lowered TTFB, great! Now it is almost the same as with Pagespeed disabled. Thanks!
Anything suggestions to improve LCP?

The shared-memory metadata cache does not get populated with images. It
just gets populated with metadata mapping the original image name and
context into which it is to be rendered into an optimized resource name.
The images are stored in memcached in your configuration. You can see the
hierarchy pictured here:
https://www.keinegotterkeinemeister.de/pagespeed_admin/cache#cache_struct
It look like you've got Prioritize Critical CSS enabled. I haven't
thought about this space for quite some time but I don't think we have much
else in PageSpeed I'd recommend. I"d try disabling defer_javascript -- it's
possible it's doing more harm than good to that metric.
Good luck!
On Mon, Apr 11, 2022 at 11:52 PM Ungov77 @.***> wrote:

There could be an IO issue because according to the Plesk guide, the first
test I ran should get a speed of 142 MB/s but I got only 183 kB/s
Memcached is now installed but I'm not sure how to check if it's correctly
configured
https://www.keinegotterkeinemeister.de/phpinfo.php
I enabled statistics and I can see some data related to Memcached so it
seems to be working:
https://www.keinegotterkeinemeister.de/mod_pagespeed_statistics?memcached
ModPagespeedMemcachedServers localhost:11211
ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 51200
Is 51200 enough? We have plenty of memory on the server and the websites
are big with thousands of images
It seems to have lowered TTFB, great! Now it is almost the same as with
Pagespeed disabled. Thanks!
Anything suggestions to improve LCP?
—
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095951580,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAO2IPIHW2BALPOWN7TII53VETXQJANCNFSM5TEGVEXQ
.
You are receiving this because you commented.Message ID:
@.***>

Thanks, we disabled defer_javascript and some other filters
However today we noticed that images are not getting converted to webp anymore after switching to Memcached
When commenting out Memcached in the config, the images are converted again to webp

My first suspicion is that they just haven't been (re)converted into webp yet.  By switching to memcached you effectively are flushing your cache. As you may have noticed, PageSpeed optimizes resources in the background when it learns they need to be optimized, but may serve the first few requests unoptimized. Is it possible that if you refresh your page a few times the images will start coming in as webp?

I tried CTRL+F5 many many times but I didn't see any webp images in the code. As soon as I turn off Memcached, I see webp images after 2 refreshes. We let Memcached be enabled for 18 hours and didn't see any webp images on the homepage.
Memcached disabled, many webp images in the code:
view-source:https://www.keinegotterkeinemeister.de/
Memcached enabled for 18 hours with zero webp in the code:
view-source:https://www.ni-dieu-ni-maitre.com/

It's possible the reason that TTFB was good for a while is that the memcached code is good at failing-fast if it looks like memcached is not working.
I can think of three reasons generally that PageSpeed would make TTFB noticeably worse:

HTML parsing overhead: Running with ?PageSpeedFilters=+debug shows us that parsing overhead is less than 6ms so I don't think it's that.
Rewrite Deadlines: in the config you pasted above it looks like that's set to 20ms which should be fine. But it might be worth looking at the config again to see if anything deadline-related is overridden
General overhead and queuing on your server.  PageSpeed keeps a limited number of threads for each process, with work queues for each thread, and at some scale the queue-depths might start growing. There are histograms for that on /pagespeed_admin.

Hi,
Makes sense. I didn't understand how it was faster with Memcached failing lol
I don't think the rewrite deadline is being overwritten. I have not modified any conf file, the entire config is in Plesk GUI except for  ModPagespeedCreateSharedMemoryMetadataCache
I have looked at the histograms but I'm not sure how to interpret the data

You can check the admin section, I put it public temporarily again
https://www.keinegotterkeinemeister.de/pagespeed_admin/

Hi @Ungov77 @jmarantz
Take a look on this:
https://www.keinegotterkeinemeister.de/pagespeed_admin/cache#physical_cache
See evictions and current objects stored:
curr_items:            53457
evictions:             25654735
I think is a big amount of evistions.
You are using only a 64Mb memcache database
bytes:                 63936177
In the config file for memcached (normaly /etc7memcached.conf or /etc/system/memcached.conf) you can increase the size of RAM memcached uses:
change m 64 whit the size you will use.

Hi,
The config file aas in /etc/sysconfig/memcached
What size should we increase it to? We have 64gb of ram on this server
I increased it from 64 to 6400 for now

hi @Ungov77
Take account that your filesystem cache  was near of 40GB, if I remember +30 (Yes, I´m Longinos in the goolge group email list)


details key feature page

Hi @PorePranav  Can you assign this issue to me with Hactoberfest tag, or you can assign any other relevant issue? on click of this i can opne this page in detail

Hi @sunny7899,
Could you elaborate on what you wish to add? Like it's just a card to show feature of the application, I don't understand why it would be necessary to create an entire page just to explain a feature.


deps(docker): bump docker image dependencies

Change Overview
Bump docker image dependencies to fix some vulnerabilities.
Test Plan

[ ] :muscle: Manual
[ ] :zap: Unit test
[x] :green_heart: E2E

Unlike #2523 does not update any major versions, so should not have high impact.


sorted movies, removed terminator 2 split of two lines; removed dupli…

…cate (Point Break was in there twice)

I appreciate the thought and will clean up the duplicate. But I like the groupings instead of the production year sort.


Write and run test for testing consensus reactor for 134MB (512x512 ODS) blocks across 100 validators

The purpose of this test is to (1) verify that the consensus reactor of CometBFT can deal with 134MB blocks across 100 validators, and (2) establish what is the minimum value for timeout_commit that can be used for such block sizes. We do not need to run any celestia-node nodes for the scope of this test. We should also disable mempool tx gossiping in this test and generate transactions locally, as the scope of this test is to test the consensus reactor only.
Setup

Create 100 validators using this branch that has a maximum ODS size of 512x512: https://github.com/musalbas/celestia-app/tree/musalbas/ods1024
Validators should have a realistic network latency setup
Set max_bytes in genesis.json to<PHONE_NUMBER> (1GB)
Set broadcast=false in config.toml
Set RecvRate and SendRate to 10000000 (10MB) in config.toml (we can try to adjust this later if that causes issues)
Set timeout_commit in config.toml to 3 - we should play with this to see what's the lowest we can get away with
Each validator should have 16 vCPUs, to ensure that constructing the erasure code isn't a bottleneck

Test

On each validator, run the txsim utility locally on a separate process, with the following options: --blob 500 --blob-sizes 1000000-1000000 --blob-amounts 1-1 --feegrant true. This will create 500 routines on each validator, that are sending PFBs with 1MB blobs. Validators will need an account with sufficient funds to do this.
Record (1) the block size of each block to verify that blocks are getting filled up; (2) the number of signatures of each block to verify that all validators are able to sign and commit each block; (3) network bandwidth statistics

To achieve this we would need to keep 2 chapters in mind:
Infrastructure for Setup
In order to fulfil 16 vcpus per validator node, the node instance for the k8s cluster should be at least c5.4xlarge(16 vcpus) or c5.9xlarge where we have 32 vcpus per instance. I'd rather start with c5.4xlarge(we have them by default rn) and decrease to 14-15 vcpus at the start for the validator containers that each pod will serve. This will make sure that we scale 100 aws's node instances to a 1/1 ratio
According to recent test runs with full validators' set and QGB, the current infrastructure implementation should not be the bottleneck
Testing Environment
Code base

As per @celestiaorg/celestia-core team members historical usage, it's much better to either branch out test-infra to remove celestia-node part of tests' code base or just fork out and make a canonical test-infra-consensus repo
Most of the setup of validators is complete
config.toml configuration should be straight forward to accommodate changes we need. Same can be said to genesis.json if we need to modify whatever

Network setup

Validators should have a realistic network latency setup

We already have the necessary requisites to start playing with 0/100/200/300+ ms latencies and do it per validator to make the network more 'realistic'. Unfortunately the latency is not dynamically changeable during test execution
Still, I would recommend to kickstart with no bandwidth and latency limitations and go to the monitoring of validators to see unrestricted figures on network per pod/validator in grafana dashboards
Txsim
Currently, we already have a docker image of txsim, that we can pull into the testable DockerFile that is built and run from testground's pov.
This means that we can just add another CLI call in a go test code and point the celestia address of each validator as the master account for the txsim to produce big blobs submissions
--blob 500 --blob-sizes 1000000-1000000 --blob-amounts 1-1 --feegrant true

ref https://github.com/celestiaorg/celestia-core/issues/945 and https://github.com/celestiaorg/celestia-app/issues/2033


When use Curl::post i get HTTP/1.1 400 Content Length Required

Hi @rkmax. Can you provide an example to reproduce? Thank you

The server you are POSTing to may be running an older version of Apache Traffic Server. The response is unique in that a 400 Content Length Required isn't actually a valid HTTP response. It probably meant 411 Length Required even though Content-Length isn't required for POST requests. This issue has been fixed in more recent versions of Apache Traffic Server (https://github.com/apache/trafficserver/commit/db3de18 / Allow POST requests without a Content-Length header).
A couple options:

Update the server
Specify a Content-Length header

Referenced
https://issues.apache.org/jira/browse/TS-2902


Non-fluent builder API

Currently the builder only provides a fluent API. While this has some advantages it also have some disadvantages:

Bad IDE support:

Auto-formatting would remove proper indentation
Easy to make indentation mistakes, no help from the IDE
IDE type hints can make code very verbose (and possibly also decrease IDE reaction speed)

More difficult to break creation appart into separate steps

Therefore it might be useful to also offer a non-fluent builder API, for example similar to this:
byte[] serialData = serializableObject(
    classData(
        SerializableClass.class,
        List.of(
            intField("i", 6),
            objectField(
                "array",
                int[].class,
                array(new int[] {1, 2, 3})
            ),
            objectField(
                "s",
                String.class,
                string("nested-test")
            )
        )
    )
).createSerialData();

This could be realized in a rather user friendly way by exposing these static factory methods all on one class for which the user can then use a wildcard import (e.g. import static marcono1234...FactoryMethods.*;).
As part of this, it might be reasonable to rename the existing builders (and their packages) to:

SerialBuilder: FluentSerialBuilder / LowLevelFluentSerialBuilder (possibly a bit too verbose)
SimpleSerialBuilder: SimpleFluentSerialBuilder (too verbose?) / FluentSerialBuilder

The new builder API could be for now mostly be implemented on top of the existing fluent API. But eventually, especially for handle support, it might be easier to implement the fluent API on top of that new API. That would however also require a low level non-fluent API.

The new API could for example be named CompositeSerialBuilder (?).


Fixed: missing step in install

Fixed: missing instruction about installing go project dependencies before building the project. This step wouldn't be obvious for people who are not familiar with Go and just want to use this customer builder with Packer.

Closing as stale, please reopen if this is still an issue.


OutOfDirectMemoryError for large uploads using HttpPostMultipartRequestDecoder

Expected behavior
With a HttpDataFactory that has useDisk=true, I thought files of any size could potentially be uploaded.
Actual behavior
Example error from below unit test:
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 62914560, max: 64487424)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:775)
	at io.netty.util.internal.PlatformDependent.reallocateDirectNoCleaner(PlatformDependent.java:748)
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.reallocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:34)
	at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.reallocateDirect(UnpooledByteBufAllocator.java:194)
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.capacity(UnpooledUnsafeNoCleanerDirectByteBuf.java:52)
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:307)
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:282)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1105)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1098)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1089)
	at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.offer(HttpPostMultipartRequestDecoder.java:351)
	at NettyUploadTest.itCanProcessLargeFiles(NettyUploadTest.java:46)
// snip

Steps to reproduce
Set -Xmx64m and run below unit test.
Minimal yet complete reproducer code (or URL to code)
import io.netty.buffer.ByteBuf;
import io.netty.buffer.Unpooled;
import io.netty.handler.codec.http.*;
import io.netty.handler.codec.http.multipart.DefaultHttpDataFactory;
import io.netty.handler.codec.http.multipart.FileUpload;
import io.netty.handler.codec.http.multipart.HttpDataFactory;
import io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder;
import org.junit.Test;

import java.nio.charset.StandardCharsets;
import java.util.Arrays;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;

public class NettyUploadTest {

    @Test
    public void itCanProcessLargeFiles() throws Exception {

        int fileSize = 100_000_000; // set Xmx to a number lower than this and it crashes
        int bytesPerChunk = 1_000_000;

        String prefix = "--861fbeab-cd20-470c-9609-d40a0f704466\n" +
            "Content-Disposition: form-data; name=\"image\"; filename=\"guangzhou.jpeg\"\n" +
            "Content-Type: image/jpeg\n" +
            "Content-Length: " + fileSize + "\n" +
            "\n";

        String suffix = "\n" +
            "--861fbeab-cd20-470c-9609-d40a0f704466--\n";

        HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/upload");
        request.headers().set("content-type", "multipart/form-data; boundary=861fbeab-cd20-470c-9609-d40a0f704466");
        request.headers().set("content-length", prefix.length() + fileSize + suffix.length());

        HttpDataFactory factory = new DefaultHttpDataFactory(true);
        HttpPostMultipartRequestDecoder decoder = new HttpPostMultipartRequestDecoder(factory, request);
        decoder.offer(new DefaultHttpContent(Unpooled.wrappedBuffer(prefix.getBytes(StandardCharsets.UTF_8))));

        byte[] body = new byte[bytesPerChunk];
        Arrays.fill(body, (byte)1);
        for (int i = 0; i < fileSize / bytesPerChunk; i++) {
            ByteBuf content = Unpooled.wrappedBuffer(body, 0, bytesPerChunk);
            decoder.offer(new DefaultHttpContent(content)); // **OutOfMemory here**
            content.release();
        }

        decoder.offer(new DefaultHttpContent(Unpooled.wrappedBuffer(suffix.getBytes(StandardCharsets.UTF_8))));
        decoder.offer(new DefaultLastHttpContent());
        FileUpload data = (FileUpload) decoder.getBodyHttpDatas().get(0);
        assertThat((int)data.length(), is(fileSize));
        assertThat(data.get().length, is(fileSize));

        factory.cleanAllHttpData();

    }

}

Netty version
Tested on 4.1.56 and 4.1.58.
JVM version (e.g. java -version)
jdk1.8.0_162 and 12
OS version (e.g. uname -a)
Windows 10

@fredericBregier  It could be related to https://github.com/netty/netty/pull/10623 given that right now file isn't written until a delimiter is found?

@fredericBregier  It could be related to https://github.com/netty/netty/pull/10623 given that right now file isn't written until a delimiter is found?

I think so too... I think we should never the commit in question for now as while it fixes some "perf issues" when running in with paranoid leak detection I don't think it has any other benefits in real world use-cases. @fredericBregier WDYT ?

I think so too... I think we should never the commit in question for now as while it fixes some "perf issues" when running in with paranoid leak detection I don't think it has any other benefits in real world use-cases. @fredericBregier WDYT ?

Hi, the issue is partially related but yet an issue.

the file is totally in the buffer while reading: previously it could be in memory but by chunk
once the file is over, if the "disk" based HttpData is used, the buffer is written to a temprary file, therefore leaving the memory free
Note that another bug was fixed in the same time since before buffers were free (discardReadBytes) wrongly

I agree that this should be changed to adapt the solution with the new way to find the delimiter. Perhaps this?
Currently:

The content is added to the HttpData along slices (see https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L1191 ).
But only once all slices are found (https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L1171)

If it is written (in Disk mode or Mixed mode and if size is greater than limit), the buffer might be cleared (free), so one could try to change this such as:

each time a slice is found (even if the delimiter is not found), one could add the content to the HttpData, but carefully take into consideration that perhaps not all bufer shall be taken (the end of the buffer could be in the middle of the delimiter, to not take into account)
therefore, try to release the memory (careful, since if Memory based, must not be released at all or bad content will occur) as in https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L348 https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L349) (we cannot simply reset the writerIndex and readerIndex to the beginning for this HttpData since the HttpData could be in Memory only)

Not sure it will work or easy to implement, but I think the idea is there...

Hi, the issue is partially related but yet an issue.

the file is totally in the buffer while reading: previously it could be in memory but by chunk
once the file is over, if the "disk" based HttpData is used, the buffer is written to a temprary file, therefore leaving the memory free
Note that another bug was fixed in the same time since before buffers were free (discardReadBytes) wrongly

I agree that this should be changed to adapt the solution with the new way to find the delimiter. Perhaps this?
Currently:

The content is added to the HttpData along slices (see https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L1191 ).
But only once all slices are found (https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L1171)

If it is written (in Disk mode or Mixed mode and if size is greater than limit), the buffer might be cleared (free), so one could try to change this such as:

each time a slice is found (even if the delimiter is not found), one could add the content to the HttpData, but carefully take into consideration that perhaps not all bufer shall be taken (the end of the buffer could be in the middle of the delimiter, to not take into account)
therefore, try to release the memory (careful, since if Memory based, must not be released at all or bad content will occur) as in https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L348 https://github.com/netty/netty/blob/5c522916f796aef7a9bf7a7b234bfb81232b52d0/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L349) (we cannot simply reset the writerIndex and readerIndex to the beginning for this HttpData since the HttpData could be in Memory only)

Not sure it will work or easy to implement, but I think the idea is there...

Hi all, I propose a fix for this (adding a test inspired from the one given, testing both in Memory and on Disk behaviors)

Hi all, I propose a fix for this (adding a test inspired from the one given, testing both in Memory and on Disk behaviors)

Thanks to both of you. The new version is working well now.
The performance degradation with paranoid detection is very noticeable (e.g. a test with a lot of data moving was taking 1 second in 4.1.58 takes 30 seconds in 4.1.59). For a while I thought there was a performance issue in the new version before I remembered what Frederic said about leak detection performance.
So if anyone else is experiencing this and wondering what is going on, try disabling PARANOID for the slow tests.

@danielflower or maybe wait for #11001 which fixes also this PARANOID issue, and improves without this level also performances (about 4 times)... ;-)

The #11001 PR is merged now.


TextField FloatingLabel

How can make floatinLabel for textField active by default?

This will help https://github.com/callemall/material-ui/pull/1497.

Can i add property for this function?


PeerSharingServer refactorisation

computePeerSharing should be defined in  Ouroboros.Network.PeerSharing.  We don't need to export it, it should be used directly by peerSharingServer.  This will simplify the interface we expose to consensus, e.g. no callback in the daApplicationInitiatorResponderMode, but instead requires access to PublicPeerSelectionState which is fine, we do that for other mini-protocols as well.  Ideally daApplicationIntiatorResponderMode wouldn't it as an argument, other protocols don't need to pass similar information if I recall correctly.

IIRC it is only how it is now because polymorphism shenanigans and because we need access to the PeerSelectionState. Since peerSharingServer is used in consensus code when setting up the miniprotocols and since we need access to PeerSelectionState, we have to partially apply computePeerSharingServer which is defined in ouroboros-network Diffusion.P2P (module that has access to PeerSelectionState) to daApplicationInitiatorResponderMode.
If we want to move computePeerSharing to Ouroboros.Network.PeerSharing and call it directly in peerSharingServer we still need a way to pass an STM action to read the PeerSelectionState to consensus. So maybe the cleanest way to do this is to define a PeerSharing consensus API dictionary and use that

We can pass TVar PublicPeerSelectionState to Diffusion through DiffusionArguments (even better just pass atomically . writeTVar publicPeerSelectionStateVar :: PublickPeerSelectionState -> m () - this requires some changes, but it's possible).  Then we will have access to the TVar outside of Diffusion and thus allow us to create peerSharingServer outside of the diffusion context, it will be responsibility of consensus to create the TVar and pass it to the right places.  This is already true for other mini-protocols, that they create some context outside of diffusion, e.g. the bracketSyncWithFetchClient requires FetchClientRegistry which is part of NodeKernel, maybe that's the right place for the TVar too.


Custom time and implementing sql.Scanner

I ran into a slightly confusing behaviour when implementing sql.Scanner for a custom time struct.
I opened a DB using the database/sql driver with a table that has a column of type TEXT storing RFC3339 timestamps. I add _timefmt=rfc3339 to ensure a consistent format is written and read when inserting from or scanning to a time.Time.
However, with a lot of APIs I end up needing some custom time type, so I fairly often have:
type myTime struct { time.Time }

You can have to implement Scan(value any) error. Since it's stored in a TEXT field I made the assumption I'd get something string-y back and so the first thing I did was driver.String.ConvertValue(value). This results in a string of the time, but it's not the string as stored in the DB because the driver already caught on to this and converted it to a time.Time object because of maybeTime (I think?).
So when looking into it, I hit the return fmt.Sprintf(...)case ofdriver.String.ConvertValue`:
func (stringType) ConvertValue(v any) (Value, error) {
	switch v.(type) {
	case string, []byte:
		return v, nil
	}
	return fmt.Sprintf("%v", v), nil
}

Then trying to parse that value with time.RFC3339 as the layout blew up, because the %v formatting of a time.Time is done with a different layout.
Having experimented a bit, it seems that in this case I can get either a time.Time if the driver recognised the time format and converted it, or a string if it didn't (for example some other process wrote into that column without the right format). This means that when implementing sql.Scanner on a custom time struct you need to do a type switch first thing. I'm thinking it may be worthwhile to document that, maybe with an example, for the driver package?

Yes, maybeTime is likely the culprit.
I need to consider the code paths, and yes, likely documentation (and an example) is the best way to deal with this.
For context, maybeTime is my best attempt at respecting SQLite's "dynamic typing" ethos: time.Time is stored as RFC3339 TEXT, but is magically recovered as either a string or time.Time (or []byte, or...).
The way that's done is that I parse and use a time.Time if (and only if) it round-trips back to the exact same string (so that nothing is lost).
I didn't consider sql.Scanner here, unfortunately.

About _timefmt, unless you're typing your columns as DATE, TIME, DATETIME, or TIMESTAMP (so STRICT tables are out), the default is the same as _timefmt=rfc3339.

About _timefmt, unless you're typing your columns as DATE, TIME, DATETIME, or TIMESTAMP (so STRICT tables are out), the default is the same as _timefmt=rfc3339.

Ah, yeah. I always have my tables set to STRICT. It would be nice if stuff like DATETIME worked as aliases in STRICT mode too because their self-documenting nature is kinda nice. But alas.


Adding conditional "outline:none" for :focus state

Hello,
ive added a boolean $hamburger-block-focus to your hamburger.scss.
related to this variable, ive added a conditional :focus{outline:none;} in the _base.scss.
Should work in every browser, because youre using the gulp-autoprefix.
Thank you for this nice package :)

The existence of an outline depends what kind of element that the developer chooses to use (some elements receive focus by default and may have an outline, others don't without tabindex), so that's a separate concern which I don't think should be mixed into this library.
For example, @jonsuh's documentation encourages improving accessibility by using <button> tags, but you could feasibly use a non-focusable <div> or <span> which would never render a focus outline when clicked/tapped.
Also, as a developer, if I wanted to get rid of the focus outline, my instinct would immediately be to just apply my own style via the following (see below). Also, the presence, color and appearance of an outline can be inherited, so I wouldn't want to encourage anyone to use variables for something that should probably be done directly via simple styles.
.hamburger {
  outline: none;
}

Which is pretty simple and intuitive and, IMHO,  doesn't necessitate a modification in the core library

Thanks for taking your time.
I did this PR because i really like the Sass-way of configuring a custom behavior in one file (like a e.g. variables-overwrite.scss ) because you have everything in one place and if variable-names are well-formed its easy to read what the value is supposed to do.
But i get your point, maybe my suggestion is a little too much for setting it before compilation.


Graph height is too big, reaching into gameplay area

Describe the bug
The timeline graph can reach outside the timeline section, and straight into the gameplay area. Not sure if intended or not, but this looks off.
To Reproduce (optional)
Steps to reproduce the behavior:

Open some map (with clear seperated calm and intense section if you want to see clearly)
Look at the timeline graph

Expected behavior
Graph height should stay inside timeline area, or have some fixed height.
Screenshots

Desktop (please complete the following information):
This shouldn't really matter but...

OS: Ubuntu (GNOME)
Version: 21.04, GNOME 3.38

Additional context
Please also provide the following if they are mentioned and related:

Beatmap: Playing with Ruby [Extra]
Replay
Skin

It was intended, but I can reduce it, if it's actually annoying (for many people)

look after the cursor a bit, the graph is touching into the gameplay area. With your design language with this app, reducing it a bit to make the timeline seperate from the gameplay area looks better imo

This is how it looks like in the upcoming versions, hope it looks better now.


Remove unfound profile: ossrh-gpg-lbruu

Implicitly fixed: The project now uses a completely different workflow from the time when this PR was created.