Monitoring Causes High Memory Uses/Crashes

Steps to reproduce

Enable Monitoring for any set of favorited tags.
Set any interval
Run the program

Context
When using the monitoring function, it appears that once the initial monitoring delay in the options is met, all configured monitors begin running on their normal interval, at the same time. For anyone with a moderate to large number of favorited tags, this leads to a very quick loss of performance within the program, to an eventual crash without a crash log being written.
Not sure if this should be more of a suggestion to stagger the monitoring so that only one tag is being searched at a time, or a crash report due to the inevitable crash that occurs. Typically around 1.7 GB of Ram on my machine.
System information

OS: Windows 10
Grabber version: 7.1.1
16GB Ram

Additional context
Wish I had some logs to upload, but there are no crash logs being generated from this particular crash. I do have the windows event viewer log however:
Faulting application name: Grabber.exe, version: <IP_ADDRESS>, time stamp: 0x5ce14e37
Faulting module name: Qt5Widgets.dll, version: <IP_ADDRESS>, time stamp: 0x5a7c81e9
Exception code: 0xc000041d
Fault offset: 0x0000000000041568
Faulting process id: 0x18a0
Faulting application start time: 0x01d5183309af5a74
Faulting application path: D:%appdata%logs\Programs\Grabber\Grabber.exe
Faulting module path: D:%appdata%logs\Programs\Grabber\Qt5Widgets.dll
Report Id: 5a016f26-ab45-4b18-8769-3837163b9816
Faulting package full name:
Faulting package-relative application ID:

Okay, so there seem to be two separate issues at play here:
Performance issue
While there is always only one monitor running at the same time, if you have multiple monitors set with the same interval, they will be loaded one after the other without any pause, and it seem to happen in the main thread, causing freezes/performance issues in the application.
A few possible solutions:

Add an interval between each check (so if two are supposed to be run at a given point, add a 1 second "pause" between both)
Run them all in parallel (might cause issues if multiple monitors use the same source, and might cause CPU "spikes" in the program, which I'd rather avoid)
Run the monitor checker in a separate thread (I don't know why it's not already the case, as it should be)

I guess I'll go with simply moving the monitoring to another thread, as it should solve all issues.
Memory usage
The high memory usage, followed by crashes, is very likely caused by a memory leak in the monitoring system, so I'll investigate and get rid of that.

Thanks for taking a look at it!

I am running into this pretty badly now. I've tried the nightly but it doesn't seem to help. Was the solution of moving to a separate thread done?
Anyone else running into it still?

The current nightly seems to have resolved this, fwiw.

I am running into this again, likely due to the number of things i am monitoring now.
The issue has a few side effects:
favorites.json's "lastcheck" entries will not be updated between the time of the program launching and crashing.  This is good and bad, but mostly bad. If it has been multiple intervals since the last time the file was saved, then at next launch, many files will attempt to download but will be met with "file already exists. And of course the program will crash again without manual editing of favorites.json, due to the same memory issue.
The only good part of that is that since the monitor's checks per tag goes faster than the downloads do, if favorites.json was saved upon the crash, it would save new timestamps for tags that had not yet had an opportunity to download. Upon the next time the program is launched, it will use these new timestamps and skip running for those other tags, forever skipping any files posted in that timeframe between the last successful download of the tag and the newest timestamp. This can be confirmed by having many tags monitored and then closing the program manually while it has a gap between the current monitor check and the current monitor download.
"While there is always only one monitor running at the same time, if you have multiple monitors set with the same interval,"
Is the same interval the issue, or is it that all of them are trying to start at the same time. My issue here is that when a crash does occur, i normally don't notice for most of a day, so when i relaunch grabber, it is well past my standard interval time of 18000s. Would varying that interval time actually help in the case where every tag is well beyond their respective interval time?
If the solution is for me to simply change all of my intervals to like 18001, 18002, etc, then that's not too big of a problem...

Has there been a solution to this problem yet?

No, I am still experiencing it. I use modifications to the favorites.json file as a workaround to get everything downloaded. It also seemed like setting simultaneous downloads to 10 helped a bit, as did increasing the interval time for the tags I am monitoring.
Hopefully a fix for the memory leak is on the road map.

@Bionus
I'm excited to see that you are working on the monitoring features! I don't suppose you have had a chance to look into this issue?

I'll have a look.
How many things are you monitoring at the same time, and how often do new images get added? (i.e. how many images must be downloaded each time you open Grabber)

Hundreds of items are being monitored. I could share my favorites.json file if that would help. I monitor a few types of things, artists, characters, copyright, and then certain more generic tags like "3D Bikini". Most of the tags are artists, but those are the least likely to bring in images each time. Whereas characters, copyrights, & generic tags are much fewer in number, but they generally result in more images per search. I'm fairly sure I was hitting these crashes back when i was only searching for artists.
If i shared the file, it would be very easy to reproduce. You could do a find/replace on the directory structure and then another find/replace to move the date back 24-48 hours(or i can just send an older copy) and then after opening grabber it would crash before monitoring half of them. On next open, it would monitor all the same tags but probably get a little further because it seems to process the "file already exists" event faster than the actual file downloads

I'll gladly take you up on your offer for your favorites.json file.
Because while I have a few ideas as to what can cause the crash, there might be multiple causes for said crash so having "real world" data would be more useful than me creating 200 "rating:safe" monitors.
If you're not comfortable sharing the file here, you can always send it to me at<EMAIL_ADDRESS>I'll take a look tonight after I'm off work. As you said, I'd like to have monitors a main feature of the next version, so if it keeps crashing that won't do 😅

Sent the files, thanks again!

There's indeed lots of issues with Grabber when using the files you sent, thanks 👌
I've been working on it yesterday and I'll keep working on it to:

Make the program more responsive
Stop eating that much RAM

Awesome.
Well, let me reinforce something I mentioned above. The saving of monitor status in the case of a crash might be an issue if it mirrors the saving that occurs when the program is normally closed during monitoring. This is particularly noticeable if you change the "simultaneous downloads" setting back down to "1". In my settings.ini, I have it set to 10 due to it helping with getting more downloaded before the crash.
Okay, so what I'm getting at is that the monitor has 2 main steps from what I can tell. Searching and downloading. When "simultaneous downloads" is set to a lower number, it's much more apparent that downloads can lag many entries behind the searches. I've seen instances of downloads still being on a word starting with like "C" when it's searching a "J" letter. In this scenario, manually closing the program results in the favorites.json being updated so that every tag up to and including the latest "J" tag having its lastCheck being set to a current timestamp. This is a problem because the downloading was still back at that "C" tag. All images for the tags between "C" and "J" will never be downloaded, even on future monitor runs.
If saving monitor status during a crash follows that mechanism, the issue could silently happen as well. Just a thought, Maybe it's possible to only update "lastCheck " based on when any download is finished or when it's flagged to not need a download, rather than when the search executes as part of the monitor?
Does that make sense? I hope I explained it well. Thanks again.

Monitors indeed have multiple steps:

Perform the search and see if there are new images
Add the images to some "need to download" list
In the background, the images to download will be taken from the "need to download" list then downloaded (using your "simultaneous downloads" setting)

Therefore, if you have only 1 simultaneous download but hundreds/thousands of monitors, they will all be "checked" and if new images are found, they will all be added to the "need to download" list. On the other hand, downloading images take time so images will be added faster than they are removed from the list. This causes the list to keep increasing in size, until it takes too much memory and crashes the program.
The main fix I worked on is to simply wait for the downloads before starting new monitors. So the case where "J" is being checked wile "C" isn't even finished downloading shouldn't ever happen anymore. It should also solve the saving issue.
The only side effect that I am aware of, is that if you have "simultaneous downloads" set to 10 for example, but each monitor return a single image every time, you will only actually download one image at a time (since new images won't be searched until the current monitor is finished downloading), possibly slightly lowering the overall download speed.
So there are indeed places where there could be improvements, but for now I think stability is the most critical issue.
You can try it out in the nightly version if you want:
https://github.com/Bionus/imgbrd-grabber/releases/nightly

Amazing improvement! After installing the nightly and running Grabber once, I backdated my LastChecks to 2 days ago for every single tag, started up Grabber again, and it went through my full list without crashing. That never would have worked on 7.2.1!
What's the best way to support development, Patreon or something else? This alone is such a wonderful update.
Now if only monitored tags wouldn't redownload files already grabbed per that tag. It seems like server side changes to tags can prompt redownloads with the way I have my filename configured. And unfortunately, I don't want to change the "if a file already exists globally" setting away from "Save" due to wanting a picture to exist in an artists folder and a characters folder separately.
Anyway, you're awesome, thanks for tackling this!

By the way, thank you for also addressing the issue of the search/downloads for the monitors. I like that solution, it's ideal imo. The stated downside is minor compared to the unintended behavior that could be encountered before.

Amazing improvement! After installing the nightly and running Grabber once, I backdated my LastChecks to 2 days ago for every single tag, started up Grabber again, and it went through my full list without crashing. That never would have worked on 7.2.1!

Glad to hear that! I was worried there'd be additional issues I didn't notice in my tests 😅

What's the best way to support development, Patreon or something else? This alone is such a wonderful update.

There's my Patreon for monthly/recurring donations, or PayPal for one-time ones!

Now if only monitored tags wouldn't redownload files already grabbed per that tag. It seems like server side changes to tags can prompt redownloads with the way I have my filename configured.

I'm not sure I understand properly. You mean your filename contains the image's tags for example, so if someone changes the tags, it will trigger a re-download since the filename changed? This is strange since already checked images should be directly skipped based on their date (compared to the monitor's last check).
Does this happen on every source? Or could it be that it only happens on sources where the "date" information fails to be fetched correctly? In that case it might cause Grabber to think all images are "new" and re-download them (if the filename changed). If that is indeed the reason, a fix could be to update the comparison to use the last checked image's ID and directly stop there next time.

And unfortunately, I don't want to change the "if a file already exists globally" setting away from "Save" due to wanting a picture to exist in an artists folder and a characters folder separately.

I know it doesn't change anything for the problem you mentioned, but know that there's also a "shortcut" setting to avoid downloading the same image X times in different folders, taking time and hard drive space. Duplicates will be created as shortcuts to the original. The downside being is that if you delete the original you'll have "orphan" shortcuts.

Anyway, you're awesome, thanks for tackling this!

No problems! I'm glad that my work is being helpful 👌

By the way, thank you for also addressing the issue of the search/downloads for the monitors. I like that solution, it's ideal imo. The stated downside is minor compared to the unintended behavior that could be encountered before.

That's also my opinion. Especially given speed is not the critical factor for monitors, while stability is. I'd rather have Grabber slowly run in the background (FYI you can reduce it to the taskbar and leave it there if you mostly use monitors), downloading everything at a steady pace rather than go crazy and crash. 💀

I'm not sure I understand properly. You mean your filename contains the image's tags for example, so if someone changes the tags, it will trigger a re-download since the filename changed?

Yes, exactly.

This is strange since already checked images should be directly skipped based on their date (compared to the monitor's last check).

Well, keep in mind that I had just backdated my LastCheck parameters for all of my tags, so it rechecked all files associated with those tags that had been posted over the last 2 days. While I was of course only doing so as a proof of concept at the time, I actually do this on purpose every few weeks. The reason is that many images get added to these sites without tagging and the tagging happens over the following hours and days. Think about it like this. An image by a great artist gets posted at 12:05, but it doesn't have the artist in the tag. My monitor has a tick for the actual artist of that image at 12:10 and doesn't download the image. At 12:15, someone on the website tags that image to the appropriate artist that i am monitoring. The next time the monitor runs, the image does not download because it's checking for posts that happened since 12:10, not 12:05. Separately, because I include %character% and %copyright% in my filenames, if those are changed between my initial download and when i backdate my tags, the images will be redownloaded with new names.
Obviously, you can only do so much to mitigate a problem like this, and they may not be worth your time. One mitigation that i may go ahead and embrace is significantly increasing the intervals for all of my monitors, but that is only able to cut down on the issue. I can also use separate tools to scan for duplicates. On your end, I guess checking for the MD5 in the folder and renaming the file would be a solid option, but that may be a ton of work.

There's my Patreon for monthly/recurring donations, or PayPal for one-time ones!

Is paypal better? I would assume that patreon takes more of a cut.
The shortcut solution doesn't work for me, but thank you for mentioning it.

I see two possible solutions for the problem you mention (and both can be used at the same time).
First option would be to have a setting to "delay" monitors, that is they won't check images that came out in the last X minutes/hours/days, giving some leeway for the tags to be updated. One downside being that if tags are updated much later the issue will remain, but it should at least occur much less often. And another being that the images won't be directly downloaded, since there will be a delay before treating the images.
The other is actually very similar to what you suggested:

On your end, I guess checking for the MD5 in the folder and renaming the file would be a solid option, but that may be a ton of work.

Rather than checking everytime a file is downloaded for all the MD5 in the directory, another option would be to split the "if a file already exists globally" into two: one for the current folder and one for other folders. So in your case, you would use "Copy" for the "other folders" and "Rename" for the same folder. But that wouldn't work if the folder name itself contains the changed tag.
I feel like the first solution makes a lot of sense, while the second might work only in some cases (where the tags are not in the folder name).

Is paypal better? I would assume that patreon takes more of a cut.

On PayPal there are indeed no fees, while on Patreon there'd be a cut of around 8% (5% Patreon fee + payment fee). But in the end whichever to use mainly depends on whether you want to make it recurring (Patreon) or one-time (PayPal), or also if you simply already have an account in either of them :sweat_smile:

The shortcut solution doesn't work for me, but thank you for mentioning it.

Yes, I guess using Windows hardlinks would make more sense for most users. It's a kind of shortcut where the data is only stored in one place, and will only be deleted when all "links" to it are deleted. And for the user/other programs, it's totally indistinguishable from a normal file copy.

Rather than checking everytime a file is downloaded for all the MD5 in the directory, another option would be to split the "if a file already exists globally" into two: one for the current folder and one for other folders.

I like that, but maybe I am misunderstanding. I only use %search% or %search_1% in the folder name, so if I'm understanding you right, this second option would be a great solution for my setup.
Option #1 is pretty great too. Having it enabled by default would likely confuse some users, but being able to enable it and tweak the timeframe would work. I guess the only thing is that you would have to subtract(add?) the delay to the next monitor tick.

I like that, but maybe I am misunderstanding. I only use %search% or %search_1% in the folder name, so if I'm understanding you right, this second option would be a great solution for my setup.

In your specific case, it would indeed be a perfect solution. And I think that would be the case for most people as well, which is why I plan on implementing it.

I guess the only thing is that you would have to subtract(add?) the delay to the next monitor tick.

It's ignoring the images which are less than X hours old, then setting the "last check" to "now - X hours". It shouldn't be very hard to do and would make a lot of sense for the specific issue you pointed out (images which are not properly tagged "yet").


Release v0.8.0

about this pr

release 0.8.0

Codecov Report

Merging #46 into master will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #46   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines          158       183   +25
=========================================
+ Hits           158       183   +25

Impacted Files
Coverage Δ

tldr/__init__.py
100.00% <100.00%> (ø)

tldr/cli.py
100.00% <100.00%> (ø)

tldr/config.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1235ad1...616a526. Read the comment docs.


Present proof problem report handler throws exception for OOB proof requests

When an abandoned state triggers on a 2.0 proof (in our BC Gov VCAuth-N ecosystem implementation, when a user declines the proof in their wallet) it invokes the V20PresProblemReportHandler. But if this is an OOB proof request with no connection an unhandled exception happens due to context.connection_record.connection_id
This was addressed for the 1.0 protocol  here https://github.com/hyperledger/aries-cloudagent-python/pull/2723 but I think should be added for the equivalent problem report handler for 2.0
Error:
2024-06-25 15:08:19 2024-06-25 22:08:19,935 aries_cloudagent.core.conductor ERROR Exception in message handler:
2024-06-25 15:08:19 Traceback (most recent call last):
2024-06-25 15:08:19   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
2024-06-25 15:08:19     result = coro.send(None)
2024-06-25 15:08:19   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 210, in handle_message
2024-06-25 15:08:19     await handler(context, responder)
2024-06-25 15:08:19   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/protocols/present_proof/v2_0/handlers/pres_problem_report_handler.py", line 32, in handle
2024-06-25 15:08:19     context.connection_record.connection_id,
2024-06-25 15:08:19 AttributeError: 'NoneType' object has no attribute 'connection_id'
2024-06-25 15:08:19 2024-06-25 22:08:19,936 aries_cloudagent.core.conductor ERROR DON'T shutdown on AttributeError 'NoneType' object has no attribute 'connection_id'
2024-06-25 15:08:19 2024-06-25 22:08:19,936 aries_cloudagent.core.dispatcher ERROR Handler error: Dispatcher.handle_message
2024-06-25 15:08:19 Traceback (most recent call last):
2024-06-25 15:08:19   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
2024-06-25 15:08:19     result = coro.send(None)
2024-06-25 15:08:19   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 210, in handle_message
2024-06-25 15:08:19     await handler(context, responder)
2024-06-25 15:08:19   File "/home/aries/.local/lib/python3.9/site-packages/aries_cloudagent/protocols/present_proof/v2_0/handlers/pres_problem_report_handler.py", line 32, in handle
2024-06-25 15:08:19     context.connection_record.connection_id,
2024-06-25 15:08:19 AttributeError: 'NoneType' object has no attribute 'connection_id'

Addressed here https://github.com/hyperledger/aries-cloudagent-python/pull/3068

@Gavinok @esune as discussed the issue with VCAuth-N erroring on decline with 2.0 proofs.


unable to download events-push

Even after giving the below details in the BuildConfig.groovy , The project is failing to resolve the dependencies . Any one have some resolution
Dependency:
compile ":events-push:1.0.M7"
Custom repositories:
mavenRepo "https://oss.sonatype.org/content/repositories/snapshots/"

try this:
dependencies {
    ...
    compile 'org.grails.plugins:events:1.0.0.M1'
    compile 'org.atmosphere:atmosphere-runtime:2.1.4' //Maybe not necessary
}

and
plugins {
     ...
     runtime ":events-push:1.0.0.BUILD-SNAPSHOT"
}

Thank you for your reply.
But still I was not able to download and it was showing unresolved dependency.
I have resolved the problem by placing the downloaded zip file with in the project lib folder
Thanks


refactor(applications): check CollectiveId instead of CreatedByUser

resolves https://github.com/opencollective/opencollective/issues/2029

We have two applications in the database with CollectiveId = NULL (sounds like something that we should not allow in the first place). We should do something about them before merging, otherwise we're going to break them.

Would it possible to update those rows?

), tho I think should probably also hide the whole list in the applications resolver if you're not allowed to - I don't see any benefit in allowing other users to list your applications (id, type, name, description).

I'm not sure I completely understand when you say hide the whole list in the applications resolver

Would it possible to update those rows?

Yes, the best to do that is to create a migration with npm run db:migration:create -- --name enforce-applications-collective-id.
You can use the following query to update the applications:
UPDATE ONLY "Applications" a
SET "CollectiveId" = u."CollectiveId"
FROM "Users" u
WHERE u.id = a."CreatedByUserId"
AND a."CollectiveId" IS NULL

Then, in the same migration. we should also add a NOT NULL constraint to the Applications.CollectiveId column to make sure it doesn't happens in the future. Let me know if you need more info on that.

I'm not sure I completely understand when you say hide the whole list in the applications resolver

In my opinion, there's no reason to allow users to list someone else's applications. So in the application resolver (https://github.com/opencollective/opencollective-api/blob/0f710e52086298cd7b1255a8eb34a8935978b5d2/server/graphql/v1/CollectiveInterface.js#L1798) I'd rather add:
applications: {
  type: new GraphQLList(ApplicationType),
  resolve(userCollective, _, req) {
    if (req.remoteUser && req.remoteUser.isAdmin(userCollective.id)) {
      return models.Application.findAll({
        where: { CollectiveId: userCollective.id },
      });
    }
  },
},

@Betree the two applications mentioned are different from the others, we should double check their use case internally.

@znarf You mean by contacting the users?
In any case I don't think that it should prevent us from fixing them, adding their CollectiveId should not break anything.

@Betree I mean: let's discuss these specific cases elsewhere.

@znarf Yes no problem with that, but I still think that we should create this migration no? It doesn't cost much and adding a constraint to the DB will make sure that this doesn't happen again in the future. It's also a requirement for this PR to solve it before merging because otherwise we will break these two applications.

Thank you @bolariin. Merging!


Add repeat KV to model builder

Description
This PR adds repeat_kv to the model builder for models where num_attention_heads != num_key_value_heads.
Motivation and Context
By supporting repeat_kv, models where num_attention_heads != num_key_value_heads can now run on both CPU and GPU.

Could you please run a benchmark on the Gemm 2b model on CPU?


Typo should be fixed in Part 6

Under the subheading "Saving and loading networks" there is a sentence:
The simplest thing to do is simply save the state **** with torch.save. For example, we can save it to a file 'checkpoint.pth'.
I'm going to out on a limb and say you wanted to type "dict" with a T not a K.

Sigh, yes. Thanks for spotting that

Fixed in https://github.com/udacity/DL_PyTorch/commit/dc8b7c2143f1911fc75b9893c5d253d1a1061130

It made my day at least Mat. So there is that. Thanks.


Various ValueError

Keeping track of various ValueError cases.

[ ] int("13a")
[ ] int("13x", base=16)
[ ] int("18", base=5)
[ ] float("42b")
[ ] pow(2, 4, 0) ValueError: pow() 3rd argument cannot be 0  from issue #182

Issue migrated to new project.


[Feature Request] Mark items I have already voted on in "Offline" board

As a user, I would like the ability to view which items I have already voted on in an "Offline" board.
It is a hassle to have to remember which item I left off on, especially when the session has 50+ items.  Opening each one becomes time consuming.  It would be great to just have a marking of some sort that shows which ones I have already voted on so that I can continue where I left off, or backtrack to one that I maybe did not have enough info on.

Hi!, good suggestion i will add it to the backlog


Species info in Guidebook and at chargen

About the PR
Wouldn't it be great if we had Species info in the Guidebook?
Wouldn't it be even better if this was available in the character editor? Now there will be an info button next to the species selection dropdown, complete with hover tooltip. Pressing the button opens the relevant Guidebook page and also loads the other species pages, so the player can click check them all without having to change species in the character editor.
This is currently in draft because the integration/functionality is done but the actual Guidebook pages are mostly... rough
... also it turns out the Guidebook looks awful when opened over the profile editor, due to being the same color and not having any visible borders. Is that something I also need to fix? Or can I declare that Somebody Else's Problem?
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog
:cl: Errant

add: Species info is now available in the Guidebook and in the character editor.

This would be nice, but I think the bullet point for the stats is not a great idea

Info icon was made smaller, and is no longer displayed for species that don't have a guidebook page

Gonna give a few nitpicks about stuff that should be in the guidebook page, but past that good addition (and honestly overdue)


How can i migrate into this framework

How can i migrate our current integration tests library into patrol, does it require any changes in the integration tests framework. Can you please help/suggest?

Hi @akramcshive!
We've got some resources to help you with this task. Please see the documentation website. We've also got an example app which demonstrates most of Patrol's features - I highly recommend you also take a look at it, maybe even clone the repo and run the example app on your machine so you can see how Patrol works :)
If you're having problems with migrating some integration tests to use Patrol, you can also post the .dart file here and I'll try to help you out :)


colcon.pkg

This PR adds the colcon.pkg file.

Build status:

Linux
Windows

Please:

merge https://github.com/micro-ROS/micro-ros-build/pull/96 before this PR
update dashing to develop in this repo after this PR

Done!

Can you move the tag @julibert ?


Overflow in size calculations can allow access to invalid memory

This program produces a segmentation fault:
/*
[dependencies]
grid = "=0.8.0"

cargo run --release
*/

use grid::Grid;

fn main() {
    let grid: Grid<i32> = Grid::new(2, usize::MAX / 2 + 1);
    println!("{}", grid.get(0, 0).unwrap());
}

The documentation on Grid::new() states that it will panic if rows * cols > usize::MAX. However, it currently only panics in debug mode; in release mode, rows * cols will silently overflow without panicking. In this example, rows * cols == 0 due to overflow, so Grid::new() creates a Vec of 0 elements, and we attempt to read from element 0. Since this is an invalid pointer (an empty Vec doesn't point to a real allocation), this results in a segmentation fault.
This issue can be fixed by replacing rows * cols with rows.checked_mul(cols).expect("...") in Grid::new() and Grid::init().

Thx for reporting. I will fix this as soon as I possible

Also, Grid::init() fails to set both dimensions to zero if either is zero, causing a segfault:
/*
[dependencies]
grid = "=0.8.0"
*/

use grid::Grid;

fn main() {
    let mut grid = Grid::init(0, 1, 0);
    grid.push_row(vec![]);
    println!("{}", grid.get(0, 0).unwrap()); // segfault
}

I will change the behavior of puh row or col to panic or nop if length of vec is zero.

If you're referring to the Vec storing the additional row or column, it doesn't have to be empty for the issue to occur:
/*
[dependencies]
grid = "=0.8.0"
*/

use grid::Grid;

fn main() {
    let mut grid = Grid::init(0, 2, &0);
    grid.push_row(vec![&0]);
    println!("{}", grid.get(0, 1).unwrap()); // segfault
}

The length check simply doesn't account for a zero-by-nonzero grid, which you probably don't intend to allow.

The init function is wrong here. It should be the same as for the new function. If either rows or cols is zero it should be initialized fully empty

Fixed with 0.8.1


volumeMounts.subPath error not detected in UI

Rancher versions:
rancher/rancher: 2.0.2
Steps to Reproduce:
1: Add a persistent volume of plugin NFS share on the cluster
2: Add that volume on the project/namespace
3: Deploy new workload and add existing persistent volume (claim), use a faulty path for "Sub Path in Volume", such as /subpath
Results:
The pod isn't starting at all, giving the following error:
I0619 07:48:43.266827       1 event.go:218] Event(v1.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"podname", UID:"c6639a8a-7391-11e8-83e3-ee12cc7a836a", APIVersion:"apps", ResourceVersion:"95818", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' create Pod podname-0 in StatefulSet confluence failed error: Pod "podname-0" is invalid: spec.containers[0].volumeMounts.subPath: Invalid value: "/subpath": must be a relative path

We need to fix it in backend.
Backend should validate the Sub Path when creating StatefulSet
Now backend only validate it for Deployment and DaemonSet.

Also noticed it's possible to save without selecting an actual volume.

rancher/rancher:master at july 26th
i was able to make sure that when using invalid subpath, rancher throws an error and it appears in the UI without creating the pod:

also made sure that you can't save without selecting an actual volume.


EK-155: Removed tabindex from main element to prevent focusing.

I'm going to make a PR so we can update the template query that will most likely cause this to not fail if we merge that branch into this one

Perfect. I have merged master into this branch, and now the deploy preview succeeds.


PUTs to Site Endpoint Requires Value for time_zone

Hi,
https://github.com/digitalocean/netbox/issues/2143 is closed, but it was only done for netbox 2.3.4 - 6.
I still get this issue with version 2.4.3.
Check master commits https://github.com/digitalocean/netbox/commits/master?after=8d4329197a916ed5704ce774a77ec1f59c8a6328+279
Browsing files:
Still OK:
Post-release version bump @jeremystretch committed on Jul 2
NOK:
Closes #2211: Removed Python 2 instructions from the installation docs @jeremystretch committed on Jul 2

@jeremystretch
Can you hint me? (This history for the file at that 2 revisions deviates from March 1st)
Best Regards,
Norman

This issue is being closed as it does not conform to one of the provided templates.


[Fix] Fix the line.separator problem for alert-plugin' test case #10277

Purpose of the pull request
Fix the line.separator problem for alert-plugin' test case #10277
Brief change log
Modified the line separator
Verify this pull request

This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Codecov Report

Merging #10278 (190b55e) into dev (62b85b9) will decrease coverage by 0.12%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##                dev   #10278      +/-   ##
============================================
- Coverage     41.27%   41.15%   -0.13%
+ Complexity     4761     4760       -1
============================================
  Files           854      855       +1
  Lines         34579    34619      +40
  Branches       3821     3826       +5
============================================
- Hits          14272    14246      -26
- Misses        18935    18998      +63
- Partials       1372     1375       +3

Impacted Files
Coverage Δ

...inscheduler/plugin/alert/email/EmailConstants.java
33.33% <100.00%> (+33.33%)
:arrow_up:

...scheduler/api/service/impl/ProjectServiceImpl.java
62.96% <0.00%> (-14.17%)
:arrow_down:

...r/plugin/registry/zookeeper/ZookeeperRegistry.java
48.21% <0.00%> (-7.15%)
:arrow_down:

...hinscheduler/api/service/impl/BaseServiceImpl.java
90.69% <0.00%> (-4.18%)
:arrow_down:

...dolphinscheduler/remote/future/ResponseFuture.java
81.96% <0.00%> (-1.64%)
:arrow_down:

...lphinscheduler/common/enums/AuthorizationType.java
0.00% <0.00%> (ø)

...permission/ResourcePermissionCheckServiceImpl.java
24.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62b85b9...190b55e. Read the comment docs.


Updates to Installation Instructions

what

Add terminal configuration instructions
Add installation instructions for fonts

why

Not everyone is familiar with what makes atmos TUI look good =)

@coderabbitai full review


Fix divide by zero in pck_packer

_get_pad attempt to divide by the alignment, so passing 0 here causes a crash. I've change the default to a valid value of 32.
In addition seemed weird to have the Encryption Key as the 2nd optional arg, also with an invalid default. So you always had to pass in both the alignment and key. Therefore gave it a valid default value.

Thanks!


ion-list: upcoming 1.0.0 syntax

List refactor:

Declarative syntax
More flexibility
Performance increase for scrolling - will only make the DOM for the item more complex as elements are added (eg delete button section only if delete button is added)

Simple case:
<ion-list>
  <ion-item ng-repeat="i in items">Hello!</ion-item>
</ion-list>

Everything case:
<ion-list show-reorder="shouldReorder"
  show-delete="shouldDelete"
  can-swipe="swipeIsAllowed">
  <ion-item ng-repeat="i in items" on-reorder="onReorder()">
    {{i}}
    <ion-option-button class="button-positive">Hello!</ion-option-button>
    <ion-option-button class="button-assertive">Goodbye!</ion-option-button>
    <ion-delete-button ng-click="items.splice($index, 1)" class="ion-minus-circled"></ion-delete-button>
    <ion-reorder-button class="ion-navicon"></ion-reorder-button>
  </ion-item>
</ion-list>

Thoughts, questions, concerns, complaints, insults, death threats?
/cc @adamdbradley @mlynch @mhartington @coenwarmer @rvanbaalen @D3CK3R @calendee and anyone else who has a comment

I would like to the ability to programmatically open and close list item drawers. For example, Android should be able to listen for a long tap, and invoke the opening of the option buttons for that applicable list item. Thanks for all the rad work!


CommonAuthID cookie is cleared on openID logout even when RememberMe option is selected

Steps to reproduce:

Use wso2is-5.8.0 version
Log in to an OIDC sp with "remember me" ticked
Try OIDC logout
Then try Log in to an OIDC sp, the username and password are not pre populated

Expected result: when user chooses the "remember me"  option while logging in . The system shall have saved the username and password
identity.xml config:
<JDBCPersistenceManager>
        <DataSource>
            <!-- Include a data source name (jndiConfigName) from the set of data
                sources defined in master-datasources.xml -->
            <Name>jdbc/WSO2CarbonDB</Name>
        </DataSource>
        <!-- If the identity database is created from another place and if it is
            required to skip schema initialization during the server start up, set the
            following property to "true". -->
        <!-- <SkipDBSchemaCreation>false</SkipDBSchemaCreation> -->
        <SessionDataPersist>
            <Enable>true</Enable>
            <Temporary>true</Temporary>
            <PoolSize>0</PoolSize>
            <SessionDataCleanUp>
                <Enable>true</Enable>
                <CleanUpTimeout>20160</CleanUpTimeout>
                <CleanUpPeriod>1440</CleanUpPeriod>
                <!--Instead of deleting all the records at once, we are deleting the records in chunks to prevent the -->
                <!--possible deadlock and lock scenarios. The following property defines the chunk size.-->
                <DeleteChunkSize>50000</DeleteChunkSize>
            </SessionDataCleanUp>
            <OperationDataCleanUp>
                <Enable>true</Enable>
            </OperationDataCleanUp>
            <TempDataCleanup>
                <!-- Enabling separated cleanup for temporary authentication context data -->
                <Enable>true</Enable>
                <!-- When PoolZize > 0, temporary data which have no usage after the authentication flow will be deleted immediately
                 When PoolZise = 0, data will be deleted only by the scheduled cleanup task-->
                <PoolSize>20</PoolSize>
                <!-- All temporary authentication context data older than CleanUpTimeout value are considered as expired
                and would be deleted during cleanup task -->
                <CleanUpTimeout>40</CleanUpTimeout>
            </TempDataCleanup>
            <UserSessionMapping>
                <Enable>true</Enable>
            </UserSessionMapping>
        </SessionDataPersist>
    </JDBCPersistenceManager>

<CacheConfig>
        <!-- Identity cache configuration.
             Timeouts are in seconds.
             Capacity is the maximum cache size.
             Unless specifically mentioned, you do not need to set the isDistributed flag.
         -->
        <CacheManager name="IdentityApplicationManagementCacheManager">
            <Cache name="AppAuthFrameworkSessionContextCache"
                                                     enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="AuthenticationContextCache" enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="AuthenticationRequestCache" enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="AuthenticationResultCache"  enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="AppInfoCache"               enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="AuthorizationGrantCache"    enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="JWKSCache"                  enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="OAuthCache"                 enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="OAuthScopeCache"            enable="true"  timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="OAuthSessionDataCache"      enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="SAMLSSOParticipantCache"    enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="SAMLSSOSessionIndexCache"   enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="SAMLSSOSessionDataCache"    enable="true" timeout="300" capacity="5000" isDistributed="false"/>
            <Cache name="ServiceProviderCache"       enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="ServiceProviderCache.ID"       enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="ServiceProvideCache.InboundAuth"       enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="ProvisioningConnectorCache" enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="ProvisioningEntityCache"    enable="true" timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="ServiceProviderProvisioningConnectorCache" enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="IdPCacheByAuthProperty"     enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="IdPCacheByHRI"              enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
            <Cache name="IdPCacheByName"             enable="true"  timeout="900" capacity="5000" isDistributed="false"/>
        </CacheManager>
    </CacheConfig>

Observations:

When the user does not choose "remember me" option, commonAuthId cookie expiry is set to Session

When the user choose "remember me" option, commonAuthId cookie expiry is set to 2 weeks

In both cases on logout commonAuthId cookie is getting cleared

Logout supposed to clear all sessions. It should clear all remember-me sessions.
This is for security reasons, so that user can clear the session on the browser and leave.
Looks like the what mentioned on this issue is correct and expected behavior.

Thanks for the quick update @ruwanta  Understood it better now.
For anyone who will have similar question. Below information will be helpful:
How does RememberMe option work?
RememberMe option is not selected
By default, if the RememberMe option is not chosen, the commonAuthId cookie will have an expiry set to Session level. Below are the possible cases and behaviour:
=== Web App token expires, but user has not quit the browser yet and try to login again
In this case, since the commonAuthId expiry is at Session level. And the session is still valid. The user will not be prompted with login page again. He will be auto logged in.
=== Web App token expires, user has restarted the browser and try to login again
In this case, since the browser is restarted, all the Session level cookies will be deleted. Hence, when the user try to login, he/she will be prompted with the login page
== RememberMe option is selected
By default, if the RememberMe option is chosen, the commonAuthId cookie will have an expiry set to 2weeks. Below are the possible cases and behaviour:
=== Web App token expires, but user has not quit the browser yet and try to login again
In this case, since the commonAuthId expiry is set to 2weeks. The cookie will be  still valid. The user will not be prompted with login page again. He will be auto logged in.
=== Web App token expires, user has restarted the browser and try to login again
In this case, since the browser is restarted, all the Session level cookies will be deleted. But, since the commonAuthId expiry is set to 2weeks. The commonAuthId wouldn't have been delete. the Hence, when the user try to login, he/she will be auto logged In without the login page prompt.
The idle session timeout and the RememberMe timeout can be configured as follows:
FileName: /usr/lib/wso2/5.8.0/repository/conf/identity/identity.xml
<TimeConfig>
        <SessionIdleTimeout>15</SessionIdleTimeout>
        <RememberMeTimeout>20160</RememberMeTimeout>
</TimeConfig>

[NOTE]
This configuration changes will come into effect for only the new tenants that are created after the server is restarted with these changes. For all the older created tenants the timeout values will be the same as that were present when these tenants are actually created

@sheikirfanbasha , I think Better to update your explanation in the official documentation. Thanks.
I created new documentation issue to track the task.
https://github.com/wso2/docs-is/issues/1015


add ODCR support

Issue #, if available:
Description of changes:

Added ODCR Group and ODCR Policy resources in CloudFormation template
Added an ODCR post install script (50)
Modified the Cloud9 Bootstrap script to customize the ODCR post install script
Modified ParallelCluster templates to attach the ODCR policy and the post install script for the HeadNode

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Capacity reservations need to be added to the group manually post installation


⭐ 37 - Home or Business Energy Planner (Hacktoberfest Demo)

🤩 Thank you for participating in the CopilotKit Hacktoberfest event.
Follow This Link to Get Started!🎉
NOTE: CopilotKit will allow multi-assign issues
💡 Please show your support and give a ⭐ to the CopilotKit repository.
Documentation to:

CopilotKit
CopilotKit CoAgents (Advanced)

Demo:
Home or Business Energy Planner
Description of app functionality:
Develop a tool that helps users plan energy-efficient solutions for their homes or businesses using AI agents for guidance.
Technology Requirements
Using CopilotKit

Required: Your project must utilize CopilotKit as a core component.
Installation: Include CopilotKit by following the installation guide at CopilotKit Quickstart.
Documentation: Familiarize yourself with CopilotKit's features through the Official Documentation.

Styling with Shadcn-UI

Required: All UI components in your project should be styled using Shadcn-UI.
Installation: Install Shadcn-UI documentation

Hey @NathanTarbert I would like to work on this issue:)
@quest-bot embark

Hello @NathanTarbert , I would like to contribute to this issue. Please assign this issue to me.
Thank you.

Hello @NathanTarbert :)
I want to contribute to this issue. Please assign this issue to me.
Thankyou!


Remove unnecessary dependencies

Opencast did not use removed dependencies, according to the Karaf console.
Is this change still allowed in a release after a feature freeze? It could break Opencast because you can't test all possible configurations. It works for me ;)

[ ] have a concise title
[ ] close an accompanying issue if one exists
[ ] be against the correct branch
[ ] include migration scripts and documentation, if appropriate
[ ] pass automated tests
[ ] have a clean commit history
[ ] have proper commit messages (title and body) for all commits

I would probably put this into develop since we're close to release for 15, but I'm also not picky.  Thoughts from @Arnei and @berthob98?

Agreed, I'd rather have this point to develop. The immediate benefit is rather low while the risk to break some configuration we didn't think of seems comparatively high. I don't want to belittle your contribution or anything (thanks for all your contributions!), I just don't want risk a 15.x release on this ^^'.


feat: Target NET. Framework to 4.8 master

Change list
Target NET. Framework to 4.8 instead of 4.5 for both projects
Types of changes
What types of changes are you proposing/introducing to .NET client?
Put an x in the boxes that apply

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] New feature (non-breaking change which adds value to the project)
[x] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Documentation

[ ] Have you proposed a file change/ PR with appium to update documentation?

This can be done by navigating to the documentation section on http://appium.io selecting the appropriate command/endpoint and clicking the 'Edit this doc' link to update the C# example
Integration tests

[x] Have you provided integration tests to pass against the beta version of appium? (for Bugfix or New feature)

Details
Since .NET 4.8 is the recommended Framework by Microsoft we should progress from 4.5.
In addition, seems like Azure Pipeline-hosted VMS doesn't support .net4.5 any longer.

@Dor-bl I think we should keep the net45 target, just incase anyone is restricted by corporate policies, and add net48. CI can point to the desired target

Done


request help: connect() to unix:/usr/local/apisix/conf/apisix-1.sock failed (2: No such file or directory), client:

Issue description
apisix deploy  by  docker
start :
docker run -d --name apisix --cpus 8 --memory 16G --network host -v /home/work/apisix/gateway-plugin:/usr/local/apisix/apisix-plugin-runner -v /home/work/apisix/config.yaml:/usr/local/apisix/conf/config.yaml -v /home/work/apisix/logs:/usr/local/apisix/logs -v /etc/localtime:/etc/localtime apache/apisix:latest
Execute make build to generate a new go runner executable program，than  replace the  old  go-runner  ,restart  apisix
finally  err  !

Environment

APISIX Go Plugin Runner's version:
APISIX version:
Go version:
OS (cmd: uname -a):

You should make sure that your configuration is correct, whether there is ext-plugin related configuration in config, if you are in debug mode：
You can refer config:
ext-plugin:
path_for_test: /tmp/runner.sock


[Accessibility] There is not enough contrast on the light blue link text on the light gray background.

Our product which uses clarity has VPAT assessment report which contains issue that the used color light blue (#007CBB) on gray background (#FAFAFA) is not enough contrast.
[Issue]
There is not enough contrast on the light blue link text on the light gray background.
Computed Size: 12px
Foreground:#007CBB
Background:#FAFAFA
The contrast ratio is: 4.4:1
Text failed at Level AA
Text failed at Level AAA
Large text passed at Level AA
Large text failed at Level AAA
[Impact]
Ensure there is sufficient contrast between foreground and background colors. This requirement applies to text over solid and image backgrounds as well as images that contain text. Proper foreground and background contrast is necessary for users with low vision and some users with color blindness to accurately read content.
[Recommendation] Developers must ensure that the color contrast for all text (and images of text) meets the WCAG 2.0 Level AA requirements, unless specifically excluded. Designers or developers should modify the foreground color, background color, and/or font size so that sufficient contrast is attained. Standard text less than 18 point (or less than 14 point if bold) must have a luminosity contrast ratio of 4.5:1 or more. Text 18 point or larger (14 point or larger if bold) must have a luminosity contrast ratio of 3:1 or more. Use the Level Access contrast checking tool (https://www.levelaccess.com/color-contrast-checker/) to determine if contrast is sufficient.
Use case
We use the light blue (#007CBB) for buttons, links app level alerts.
Could you suggest with which color to be change the light blue (#007CBB) or the background color in order to meet the required contrast ratio of 4.5:1.
It’s good the clarity color palate  and examples  to be updated with the new color recommendations.
The purpose solution from @mathisscott  is update that specific blue to #0079B8 to get proper contrast
Thanks

This is one we will need to backport to 0.12 and 0.11


Delegate methods won't fire if I change presentation mode?

Both delegate methods fire just fine when I present it as modal like the example:
if QRCodeReader.isAvailable() {
            let reader: QRCodeReader = QRCodeReader(metadataObjectTypes: [AVMetadataObjectTypeQRCode])
            let vc: QRCodeReaderViewController = QRCodeReaderViewController(cancelButtonTitle: "Cancelar", codeReader: reader, startScanningAtLoad: true, showSwitchCameraButton: false, showTorchButton: false)

            vc.modalPresentationStyle = UIModalPresentationStyle.FormSheet
            vc.delegate = self
            self.presentViewController(vc, animated: true, completion: nil)
}

But if I insert a container view to my VC, and add the QRCodeReaderVC to it (so I can have a nice header menu, with some menu options and etc instead of modal), the delegate methods won't fire.
if QRCodeReader.isAvailable() {
            let reader: QRCodeReader = QRCodeReader(metadataObjectTypes: [AVMetadataObjectTypeQRCode])
            let vc: QRCodeReaderViewController = QRCodeReaderViewController(cancelButtonTitle: "Cancelar", codeReader: reader, startScanningAtLoad: true, showSwitchCameraButton: false, showTorchButton: false)

            vc.modalPresentationStyle = UIModalPresentationStyle.FormSheet
            vc.delegate = self

            self.addChildViewController(vc)
            vc.view.frame = CGRectMake(0, 0, self.containerView!.frame.size.width, self.containerView!.frame.size.height)
            self.containerView!.addSubview(vc.view)
            vc.didMoveToParentViewController(self)
        }

It behaves like this by design or am I doing something wrong?

Here it works fine. However the reader don't stop automatically. We have to stop it manually calling the stopScanning method.


vaultwarden: Insecure permissions of files in state directory

Describe the bug
The service vaultwarden creates a state directory /var/lib/bitwarden_rs. After creation, the state directory and all files in it (including the sqlite database) are world readable. Since the point of vaultwarden is to store sensitive information, this seems like a bad choice.
Steps To Reproduce
Steps to reproduce the behavior:

Enable the vaultwarden service (services.vaultwarden)
Check permissions of  /var/lib/bitwarden_rs

Expected behavior
All files in the directory /var/lib/bitwarden_rs (except for possibly rsa_key.pub.pem should have permissions 0600.
Notify maintainers

@K900 @ncfavier @pennae @dywedir @alarsyo
Metadata
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.43, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

That is actually not a major issue as Bitwarden is end-to-end encrypted, so the database dump is useless without the master password. That said, it should be an easy and harmless fix to set it to 0600.

I agree.

I suppose this is a security issue. We probably want to prevent someone from getting a database dump as much as possible.

Sorry, completely forgot to actually submit the change. Here it is.


Inventory for API8 (part 3) Queries

SpongeAPI | SpongeCommon
based on changes in: #1998
Merged QueryOperations and InventoryTransformations

Query, QueryType.NoParam|OneParam|TwoParam
transform replaced by query(Query)
new query type to query for a grid in another grid.

@Faithcaio can you rebase api8/redesign/inventory onto api-8 so that changes are better reflected?


jira PDC-960, Order resources in client

Print resources in order when running  'pdc_client -s qa ' shell command.
jira: PDC_960

Generally looks good to me.

Good to me

Looks good to me. I think the commit message would deserve a bit more love.


Improve README

What happens if you don't go the Drive section? It does not allow to open the JuputerLab?

What happens if you don't go the Drive section? It does not allow to open the JuputerLab?

If you don't go to the Drive section, you can still open JupyterLab, but it fails after selecting the Docker image with the error:
Error: HTTP 500: Internal Server Error ...

See screenshot:


Propose Move Forward replace Next Steps as section title

Summary
Details and comments

Thank you for your interest in contributing to Qiskit. There wording currently used in this heading is fine, so I'm going to close this PR. Please look through the various repositories in the Qoskit organisation for issues tagged "help wanted" if you would like to find something to work on.


Read Package Version Better

In composer, streaming, and MegaBlocks, we read the package version in the setup.py file from the _version.py file. However, in llm-foundry we are missing a _version.py file and instead read the version from the llmfoundry/__init__.py file. This PR changes this.
To ensure consistency with our other repos, I put the package version in the _version.py file. Having a _version.py file better follows good design principles as it is better to separate the version out into its own file.

Weird, I initially imported __version__ into the __init__.py file. Looking at the GA logs here, turns out the pycln hook was removing __version__, probably because it was an unused import.
To fix this, I added __version__ to the __all__.


Can you include this api call?

Can you augment the api to include calls to 'types'?
GET https://api.petfinder.com/v2/types'

You should be able to get animal types by using the following:
var pf = new petfinder.Client({apiKey: 'my-api-key', secret: 'my-api-secret'});

pf.animalData.types()
    .then(function (response) {
        // Do something with `response.data.types`
    })

what about
GET https://api.petfinder.com/v2/types/{type}?

Thank you!


how to download EcomXL model?

Hey, I've been looking at the comfyui workflow, and it seems to use the EcomXL model. I checked HuggingFace, but can not find it. Do any guys have any tips on downloading the EcomXL model?

I think it is supposed to be downloaded here, and then you can rename it to anything you like, like ecomxl.safetensors maybe, which will be detected by ComfyUI. But in my case, I got an error running this model, which is described in this issue. So, I cannot confirm if that is the correct EcomID model.

thanks for your reply, em...file size of model seems that it is not an SDXL model.

I overlooked that, yeah I think so

Sorry, our EcomXL model is not open-source yet, but it supports other models based on SDXL.

Sorry, our EcomXL model is not open-source yet, but it supports other models based on SDXL.

ok thanks, and is it right to set ip_weight to 0.3 (used in workflow) if using other SDXL models?


Custom device role icons and power topology

Merging it into master as to alleviate any confusion which branch should be installed.

This was implemented in the upstream repo so I'll close it here


feat(database): support optional endAt/equalTo key

Closes #837

Checklist

Issue number for this PR: #837
Docs included?: yes
Test units included?: yes
e2e tests included?: no
In a clean directory, npm install, npm run build, and npm test run successfully? yes

Description
This PR adds support for the optional key parameter for equalTo and endAt, as the SDK documentation now states the the optional parameter is supported for those methods, too. See linked issue for documentation links and references, etc.

😍

Is this feature included in the latest af version?
I'm getting Query: First argument passed to startAt(), endAt(), or equalTo() cannot be an object. when I try to add equalTo: { value: uid, key: 'user_id' } to my af query.
The doc at https://github.com/angular/angularfire2/blob/master/docs/4-querying-lists.md suggests it's possible. Thanks!

@vkniazeu If you have a problem, please raise an issue and follow the template, as it's more efficient than a back-and-forth conversation to establish versions, etc.


Add terradoc

Recreated README with terradoc, checked documentation for errors or inconsistencies.

Mistakenly put into R4R.


Upgrade pulumi-terraform-bridge to v3.85.0

This PR was generated via $ upgrade-provider pulumiverse/pulumi-acme --kind=bridge --target-bridge-version=latest.

Upgrading pulumi-terraform-bridge from v3.84.0 to v3.85.0.

Superseded by #53


[POC]: in memory projections

This is a container for a POC where we store frequently accessed data in memory instead of querying the database every time.
Starting point is the instance by host query which is executed on every request.
This issue is for state tracking and also to share results publicly.

I have a question of which I am uncertain: are we talking here how we can "access" a projection faster or how we can create projections faster, or both 😅

Question
I have a question of which I am uncertain: are we talking here how we can "access" a projection faster or how we can create projections faster, or both 😅

The problem is that we spend a lot of time waiting for a database connection to get free so it's both, the faster the queries get (or the less we need for a call) solve the underlying problem of slow response times.

To at least help my brain i tried to work out some thoughts below in an specific example... I.e. the instance projection.
Maybe it also helps us quickly defining what the "goal" of this improvement is by defining how we want to measure success... is it reduction of latency, better software architecture, less load on the db (or others)?

IMO it is clearly reducing latency. To answer questions like: https://github.com/zitadel/zitadel/issues/8352

Option 1 (persisten projection)
Adding to what we discussed so far I think we could use PGs notify ability to listen for events that would update a certain projection and then store it in a shared storage (PG or Redis) which supports locking to prevent race conditions. Also we might need to consider if we need to query a projection or if key value is sufficient.
-> To my understanding this is really close to what we already do today, altough with a different approach
Option 2 (in-memory projection)
One could use pg notify to fanout changes to each zitadel process to have a way of keeping an in memory storage up to date with the recent changes to instances.
However I see some challenges here:

This might introduce some more pressure on PG when many zitadel processes are running
One would need to create (or load) the projection when starting the zitadel process which could be expensive with many instances
I think it is hard to handle "consistency" here since it could happen that a process is overwhelmed and looses some of the changes

Option 3 (persisten projection + cache)
I think there is a third option where we could use pg notify to subscribe to changes (events) which create and store the projection and then a second pg notify to stream that something changed in the projection and notify all processes to purge/update their cache.

To me the process would then look somehow like this:
sequenceDiagram
    Query->>+Projection: get object
    Projection->>-Query:
    Command->>+Eventstore: push events
    Eventstore->>+Database: insert events
    Eventstore->>Database: upsert unique constraints
    Eventstore->>Database: notify "events added"
    Query->>+Projection: get object
    Projection->>-Query:
    Database->>Projection: notify with position
    Database-->>-Eventstore: commited
    Query->>+Projection: get object
    Projection->>-Query:
    Projection->>+Eventstore: FilterEvents between last position and notified one
    Eventstore->>+Database: select events
    Database->>-Eventstore:
    Eventstore->>-Projection:
    Projection->>Projection: Mark relevant objects dirty
    Query->>+Projection: get object
    Projection->>+Database: select relevant objects for update skip locked (to process in parallel)
    Database->>Projection: relevant objects
    Projection->>Projection: Update objects
    Projection->>-Query: (either return here or await the "projection updated" notification)
    Projection->>+Database: update relevant objects
    Projection->>Database: notify "projection updated"
    Database->>-Projection: commited
    Database->>Projection: notification "projection updated"

this would work with a shared or local cache with the main question being "how expensive" the fanout with notify is 😁

yes we need are currently lacking some data on that end.

Some additional things to consider

To my knowledge pg notify does not exist in CRDB so it essentially breaks that support (maybe there is a workaround with changefeed or we ditch it)

You are right we would need changefeeds for crdb

Introducing local and/or remote cache could be beneficial in some cases

My 2 cents
I would be most interested in option 3 with a shared cache (for local dev in-memory).
Main reason being I think we should avoid the whole "cluster coordination" problem that comes around when we need to update the cache of multiple zitadels, but I am open for a discussion on that 😁

would start with option 2 because it's easier to achieve

Fair, what concers me a little is that to be able to assign instances and their domains you need to have the projection ready, right? Which means either loading it at startup (which I hate) or to load it if you missed a "key".
Or am I missunderstanding that?

would start with option 2 because it's easier to achieve

Fair, what concers me a little is that to be able to assign instances and their domains you need to have the projection ready, right? Which means either loading it at startup (which I hate) or to load it if you missed a "key".
Or am I missunderstanding that?

No you are right. The biggest problem with instance by host that it is a search query.
This friday afternoon trial uses the current instance projection as a lookup table: https://github.com/zitadel/zitadel/pull/8564
What i think would work quite well for that case is to extend the lookup table by the instance domains and query it if we are missing data (or even do a join to the events table to get all events on a single query for that instance)

Hm yeah the lookup table could work for that.
But in that case the main concern is then cache lifecycle.
My take there is that it will be less expensive to run a shared cache that gets updated from a notification system. Rather than multiple processes holding their own cache that will be voided each time they restart (thundering herd when scaling would make this even worse), which IMO introduces more pressure to the primary storage.


Great Project! Will there be support for graphql?

First congrats on this great project!
I wanted to ask do see also integrating with the apollo server side of strapi?

That's a great idea haven't used the graphql side of strapi so this hasn't come to my mind yet.
But just so that I understand correctly, you would like graphql metrics or do you want the endpoints to be queryable with graphql?

Correct graphql metrics. I'm having issues knowing what queries are slow, but they all sue the same endpoint /graphql
As strapi uses apollo server I was wondering if something like this would be possible (https://www.apollographql.com/docs/technotes/TN0003-opentelemetry-traces-to-prometheus/#prometheus-setup)

Had a look today and found some plugins for apollo server but they did not show up in the metrics yet. But I'll update this thread if I manage to find a solution!

@XanderD99 do get which graphql query was called. As there might be different type of graphql queries being called. I want to narrow down the metrics on different ones


Add a second importer cronjob to run in delete mode

This runs the code introduced in #2030 that takes many hours to complete and isn't practical to run inline with regular imports.
Addresses #1467

Hmm, I think this is going to require additional tweaking based on an attempt at manually running the job: it looks like it will make the existing importer job unschedulable for the duration due to CPU limitations in the cluster?

Does the importer deleter need the checked out git repos on disk? The main reason the importer is in its own pool is to avoid having to perform a full pull of the git repositories every time it runs.

Does the importer deleter need the checked out git repos on disk? The main reason the importer is in its own pool is to avoid having to perform a full pull of the git repositories every time it runs

It doesn't, yet. Eventually deletion detection for Git repo sources will get reinstated as well, so it might be significant then?
Trying to think of the best places to put some signposting to avoid that becoming a problem later...

LGTM, though some minor nits

Thanks, deloused.


Build fails on SDK 2.1.300-rc1-008669 at BuildServer step

Environmental info
Windows Win 7 x86 ENU
VS:D15.7/27625.00 (15.7 Preview 5)
Node.js: V8.11.1 LTS
Yarn --version:1.6.0
CLI/SDK: 2.1.300-rc1-008669 (runtime - 2.1.0-rc1-26425-07)
Repro steps
Master branch of SAFE Bookstore
Install the needed deps for SAFE-Bookstore and run build.cmd. Error produced:

/cc @cartermp

@yunweiaa did you try with a RTM version?

This is for .NET Core 2.1 preview validation. Myself and @kevinransom can take a look, since the bits in the report are on an internal file share.

That would be great.

@forki @YunWeiaa the issue seems to be that the bookstore is downloading a 64-bit version of the CLI, but the VM that this was tested on is an x86 Windows 7 machine. So there isn't any action we'll take on our end. The FAKE module could be updated to handle this, or the requirements for the bookstore could be changed to x64.

Thanks for investigating this. I'll try to patch it in fake 4 tomorrow. /cc
8
Phillip Carter<EMAIL_ADDRESS>schrieb am Mo., 30. Apr. 2018,
23:10:

@forki https://github.com/forki @YunWeiaa https://github.com/YunWeiaa
the issue seems to be that the bookstore is downloading a 64-bit version of
the CLI, but the VM that this was tested on is an x86 Windows 7 machine. So
there isn't any action we'll take on our end. The FAKE module could be
updated to handle this, or the requirements for the bookstore could be
changed to x64.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SAFE-Stack/SAFE-BookStore/issues/328#issuecomment-385529822,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNEclOIlpO0rbb5KU5_CAn6LZkRmbks5tt322gaJpZM4Tpur4
.

Verified this issue on VS build: d15.7/27702.01 (15.7 Preview 7) + CLI/SDK 2.1.300-rtm-008701 (x86) on Win 7 x86 ENU, but it also failed.

Please let me know if you have any other concern.

@cartermp ideas?

Yoof, none come to mind. Sounds like some generic failure. The previous error message was typical of x86/x64 errors but this one doesn’t give a hint unfortunately. Do we know if webpack works on x86 windows?

That is a good question. Tbh I didn't even know that you could still use
x86. If we don't find something then I'm personally fine with requiring
x64.
Phillip Carter<EMAIL_ADDRESS>schrieb am Sa., 5. Mai 2018,
17:26:

Yoof, none come to mind. Sounds like some generic failure. The previous
error message was typical of x86/x64 errors but this one doesn’t give a
hint unfortunately. Do we know if webpack works on x86 windows?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/SAFE-Stack/SAFE-BookStore/issues/328#issuecomment-386813283,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADgNBm1jIf_7WPfcWe_DQnYhMegfIoFks5tvcSxgaJpZM4Tpur4
.


fix(cli): option --quiet suppresses output expect for error

Background
The --quiet option is supposed to "Suppress log output except on error" per the documentation:
https://github.com/sass/node-sass/blame/master/README.md#L547
however, it currently is only suppressing warn and info level and lets log level through.
If you ran node-sass --quiet /some/file.scss you would expect this to only emit on error, however, this currently will output the entire resulting .css to stdout.  An example of when you would not want the resulting css but would only want an error if one occurred is if you were using the command as a CI step to verify that the scss was valid.
Solution
This PR makes the --quiet option properly suppress log output except on error.
Closes #2599

If you ran node-sass --quiet /some/file.scss you would expect this to only emit on error, however, this currently will output the entire resulting .css to stdout

This assumption is incorrect. The default behaviour if not output flag is supplied is to output the compiled css to stdout.


[SPARK-49114] Sub-categorize cannot load state store errors

What changes were proposed in this pull request?

Reclassify a OOM error for both state store providers in loading state store and throw an informative error message
Within the loading state store call path, reclassify any non-error-class errors with error-class errors under the CANNOT_LOAD_STATE_STORE category

Why are the changes needed?

We need better error classification for load state store call path for easier diagnosis and resolution of issues.
Does this PR introduce any user-facing change?

No
How was this patch tested?

Tests passed.
Was this patch authored or co-authored using generative AI tooling?

No.

@HeartSaVioR requesting final review

Thanks! Merging to master.


Add pbstream_map_publisher_main.cc

Implements RFC 06

Finishes the work for this RFC started by @jihoonl with https://github.com/googlecartographer/cartographer_ros/pull/633 and https://github.com/googlecartographer/cartographer/pull/782

@wally-the-cartographer are you alright?

$ git show 7603183c86021e0fb660b33d76cb7550340dda20

commit 7603183c86021e0fb660b33d76cb7550340dda20
Merge: 92b1ce1 5cd81be
Author: Alexander Belyaev<EMAIL_ADDRESS>Date:   Wed Feb 7 18:11:53 2018 +0100

    Merge branch 'master' into serve-ros-map-from-pbstream

I guess he's complaining because of this mail. Can you edit the commiter e-mail in this one?

Nice, sorry for this.

No problem :D

@MichaelGrupp thank you for finishing this. I couldn't find the time to implement it feature recently.
One comment. You may want to refactor [occupancy_grid_node_main.cc:171-204] to use CreateOccupancyGridMsg as well.

Thanks for the hint, @jihoonl, will do that once this is merged.

Trigger Wally?


Create a new contact. Incorporate the REST API URI versioning.

Creating a new contact using the Spring JDBC metadata-driven insert client.
Adding the intermediate part of an SQL query, used to order contact records by ID.
Adding the Location response header with the URI of a newly created resource (customer contact).
Adding constants: regex patterns for contact phones and emails.
Adding a constant: the customer_id column name of the contact_phones and contact_emails database tables.
Changing the REST URI for the PUT /customers/contacts endpoint.
Updating Javadoc comments: what endpoints usually return and may return in case of error.
Incorporating the REST API URI versioning: currently /v1.
Adding a snippet to list contacts of the type phone for a given customer.
Updating the doc table with endpoint names/REST URIs regarding the REST API URI versioning.
Describing which values can take placeholders {customer_name}, {customer_id}, {customer_contact}, and {contact_type}.
Updating command-line snippets of consuming endpoints with regard to the REST API URI versioning.
Bumping version number to 0.2.0.

Ship it ! :+1: — Approved. :blue_heart:


Hopper Support?

Hey, I am running on latest main: d258f8f, using
"jax @ git+https://github.com/google/jax@d872812a359a3bafcfdeba1fcdb874ec77c209db",
"triton @ git+https://github.com/openai/triton@3452615d795bc0c69a189e41f1e775904e5659be#subdirectory=python"
When running on a hopper node I get the following error
    test_out = test_fn(**fn_kwargs, **extra_test_kwargs)    # My call to triton
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/traceback_util.py", line 166, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/custom_derivatives.py", line 620, in __call__
    out_flat = custom_vjp_call_p.bind(flat_fun, flat_fwd, flat_bwd,
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/custom_derivatives.py", line 770, in bind
    outs = top_trace.process_custom_vjp_call(self, fun, fwd, bwd_, tracers,
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/core.py", line 840, in process_custom_vjp_call
    return fun.call_wrapped(*tracers)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/linear_util.py", line 188, in call_wrapped
    ans = self.f(*args, **dict(self.params, **kwargs))
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/traceback_util.py", line 166, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/pjit.py", line 252, in cache_miss
    outs, out_flat, out_tree, args_flat, jaxpr = _python_pjit_helper(
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/pjit.py", line 165, in _python_pjit_helper
    out_flat = pjit_p.bind(*args_flat, **params)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/core.py", line 2596, in bind
    return self.bind_with_trace(top_trace, args, params)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/core.py", line 389, in bind_with_trace
    out = trace.process_primitive(self, map(trace.full_raise, args), params)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/core.py", line 821, in process_primitive
    return primitive.impl(*tracers, **params)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/pjit.py", line 1203, in _pjit_call_impl
    return xc._xla.pjit(name, f, call_impl_cache_miss, [], [], donated_argnums,
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/pjit.py", line 1187, in call_impl_cache_miss
    out_flat, compiled = _pjit_call_impl_python(
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/pjit.py", line 1143, in _pjit_call_impl_python
    return compiled.unsafe_call(*args), compiled
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/profiler.py", line 314, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/interpreters/pxla.py", line 1228, in __call__
    results = self.xla_executable.execute_sharded(input_bufs)
jax._src.traceback_util.UnfilteredStackTrace: jaxlib.xla_extension.XlaRuntimeError: INTERNAL: Failed to execute XLA Runtime executable: run time error: custom call 'xla.gpu.custom_call' failed: ptxas exited with non-zero error code 65280, output: ptxas fatal   : PTX with .target 'sm_90a' cannot be compiled for architecture 'sm_90'
; current tracing scope: custom-call.7; current profiling annotation: XlaModule:#hlo_module=jit_attention,program_id=22#.

Is hopper supported?

Yep, I have been able to run torch-triton kernels on Hopper for a few months now.

@chr1sj0nes do we expect this to work?

From our initial tests, It seems like this PR fixes the issue.

Awesome, thanks for the fix.


Fail terraform plan when reducing partition count for aiven_kafka_topic

What is currently missing?
Validation at plan stage on reducing partition count for aiven_kafka_topic
How could this be improved?
Using aiven/aiven v3.11.0
Change partitions to a lower number for an existing topic and run terraform plan will produce a good plan to update the resource in place. For example
module.topics.aiven_kafka_topic.x_topic["testing_topic"] will be updated in-place
~ resource "aiven_kafka_topic" "x_topic" {
...
      ~ partitions             = 24 -> 3

Running tf apply using the same plan errors with
Error: 400: {"errors":[{"message":"Reducing partition count is not supported","status":400}],"message":"Reducing partition count is not supported"} -
Preferable to catch the reduction in partition count at the plan stage.
Is this a feature you would work on yourself?
Sorry, I don't think I'll have time for that.

[ ] I plan to open a pull request for this feature.

hey, @apjm! 👋
thanks for reporting this! I'll take a look now and will get back to you!

I have introduced a PR which addresses your issue—you're welcome to compile the provider on your own if this is something urgent, in any other case—expect to see the fix in the next release of our provider, thank you!

Hello @Serpentiel . Thanks for looking at this so quickly. We will watch for progress


Support for Market-1501 dataset format

Motivation and context

Add support for Market-1501 dataset format
How has this been tested?

Unit tests
Checklist

[ ] I submit my changes into the develop branch
[ ] I have added description of my changes into CHANGELOG file
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] I have linked related issues (read github docs)
[ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[ ] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[ ] I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@yasakova-anastasia , there are some problems with tests:
    task.export_task(task_id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 749, in export_task
    task.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 631, in export
    exporter(dst_file, task_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 37, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/market1501.py", line 23, in _export
    for item in dataset._data._source:
AttributeError: 'Dataset' object has no attribute '_data'

@nmanovic , this PR depends on openvinotoolkit/datumaro#102, openvinotoolkit/datumaro#108 and openvinotoolkit/datumaro#114, so the tests fail.

Coverage increased (+0.1%) to 72.172% when pulling 829a18f7bd7e0c59c04b2653e10415c2eaace98c on ay/add-market1501 into 4191ea2df668b888c709babfe184fde389f9cc19 on develop.


Enable Remote Commands to ZabbixProxy Plugin

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
[X] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
[x] When the request is meant for an existing plugin, I've added its name to the title.

Is your feature request related to a problem? Please describe.
There is not EnableRemoteCommands for Zabbix Proxy.
Describe the solution you'd like
Add EnableRemoteCommands to Zabbix Proxy plugin directly in GUI.
Describe alternatives you've considered
Editing configuration file but at every restart it change to default vauls.
Additional context
I have also created a Pull Request:
https://github.com/opnsense/plugins/pull/2946
But it seams that someone closed it, but whitout explain why the changed are not right for pull.

@clanto007 I think you closed it yourself, better reopen if still a valid PR

Thank you, I found the problem, I'm sorry it's the first time that I use github to do PR.
Created a new PR:
https://github.com/opnsense/plugins/pull/2948

no problem, just noticed we didn't close it :)


Features to improve maintainability of plugin in managed service

Description
Adds several standard OpenSearch features to improve maintainability of plugin in managed service
Issues Resolved
https://github.com/opensearch-project/opensearch-learning-to-rank-base/issues/62
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This is a huge PR. Can you split it into smaller PRs?
Suggest add more details in description of all of the changes in the PR

@ylwu-amzn The PR is quite comprehensive, but the commits should be quite well separated:

Forst commit is for circuit breaker
Second commit is for settings
Third commit is for stats

We could split this into three separate PRs, if you consider this to be more suitable, but in general this PR is about adding features in org.opensearch.ltr and applying these features in the actual LTR rest, query, etc. stuff. How about a walkthrough together?

@noCharger  would you be able to help here with the review ? Within AWS Managed Service, if there are some learnings that can help @JohannesDaniel  make changes accordingly, it would benefit both targeted managed services, with @JohannesDaniel  getting a more robust version of LTR  ready for his managed service, and AWS inheriting a more user-friendly plugin from opensource.

Can we split into smaller prs for review please.
Thanks. Possible add a design overview the explain the changes that will help review faster

Closed in favor of smaller PRs.


fix: throw when trying to heal or discover neighbors for the controller

fixes: #5969

@zwave-js-bot automerge


PG::UndefinedTable error when running migration

I'm following the instructions, and when running rake db:migrate I get:
rake aborted!
PG::UndefinedTable: ERROR:  relation "que_jobs" does not exist
LINE 9:       FROM que_jobs

Am I missing something?
Thanks

What does rake db:migrate --trace show?

rake aborted!
PG::UndefinedTable: ERROR:  relation "que_jobs" does not exist
LINE 9:       FROM que_jobs
                   ^
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/base.rb:74:in `prepare'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/base.rb:74:in `block in execute_prepared'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/active_record.rb:5:in `block in checkout'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activerecord-4.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/active_record.rb:48:in `checkout_activerecord_adapter'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/active_record.rb:5:in `checkout'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/base.rb:65:in `execute_prepared'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que/adapters/base.rb:48:in `execute'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que.rb:73:in `execute'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/que-0.11.2/lib/que.rb:81:in `job_stats'
/Users/guillewu/Development/web/config/initializers/check_for_active_workers_job.rb:3:in `<top (required)>'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:268:in `load'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:268:in `block in load'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:240:in `load_dependency'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:268:in `load'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/engine.rb:652:in `block in load_config_initializer'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/notifications.rb:166:in `instrument'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/engine.rb:651:in `load_config_initializer'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/engine.rb:615:in `each'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/engine.rb:615:in `block in <class:Engine>'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:30:in `instance_exec'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:30:in `run'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:55:in `block in run_initializers'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:420:in `block (2 levels) in each_strongly_connected_component_from'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:419:in `block in each_strongly_connected_component_from'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:44:in `each'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:44:in `tsort_each_child'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:413:in `call'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:413:in `each_strongly_connected_component_from'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `call'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/initializable.rb:54:in `run_initializers'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/application.rb:352:in `initialize!'
/Users/guillewu/Development/web/config/environment.rb:5:in `<top (required)>'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `require'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `block in require'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:240:in `load_dependency'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:274:in `require'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/application.rb:328:in `require_environment!'
/Users/guillewu/.rvm/gems/ruby-2.2.3/gems/railties-4.2.4/lib/rails/application.rb:457:in `block in run_tasks_blocks'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:240:in `call'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:240:in `block in execute'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:235:in `each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:235:in `execute'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:179:in `block in invoke_with_call_chain'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:201:in `block in invoke_prerequisites'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:199:in `each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:199:in `invoke_prerequisites'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:178:in `block in invoke_with_call_chain'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/monitor.rb:211:in `mon_synchronize'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:172:in `invoke_with_call_chain'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/task.rb:165:in `invoke'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:150:in `invoke_task'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:106:in `block (2 levels) in top_level'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:106:in `each'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:106:in `block in top_level'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:115:in `run_with_threads'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:100:in `top_level'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:78:in `block in run'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:176:in `standard_exception_handling'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/lib/ruby/2.2.0/rake/application.rb:75:in `run'
/Users/guillewu/.rvm/rubies/ruby-2.2.3/bin/rake:37:in `<main>'
Tasks: TOP => db:migrate => environment

What's on /Users/guillewu/Development/web/config/initializers/check_for_active_workers_job.rb on line 3?

I imagine it's something that expects that que_jobs table is already setup, possibly asking for job stats. You could try rescuing the exception on that line so that the rest of the migration process can happen.

@joevandyk Thanks for pointing towards the right direction :+1: ! Closing issue.


ray tracing meets bug in GTX1050

When I run the code named kuafu.py provided by the Sapien toturial, I met the same bug mentioned in #51 on my laptop but everything worked fine on my Linux server. The major difference is that my laptop's GPU is GTX1050 while my linux server has several RTX2080Ti.
So I think there may be some bugs for Sapien's ray tracing on non-RTX GPU.

The return message is "vkCore: Missing physical device extension: VK_KHR_pipeline_library", which is the same as #51

I believe this issue indeed exists by according to reports. However, there is not much we can do since it looks like this required extension is not provided by the Nvidia driver for non RTX GPU.

I believe this issue indeed exists according to many reports. However, there is not much we can do since it looks like this required extension is not provided by the Nvidia driver for non RTX GPU.

Thanks for your reply


Unable to get streaming response from BedrockChat in a custom agent

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain.js documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain.js rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
  const llm = new BedrockChat({
    model: "anthropic.claude-3-sonnet-20240229-v1:0",
    modelKwargs: {
      temperature: 0,
    },
    streaming: true,
    callbacks: [
      {
        async handleLLMNewToken(token) {
          console.log({ token });
        },
      },
    ],
  });

  const memory = new BufferMemory({
    chatHistory: new DynamoDBChatMessageHistory({
      tableName: memoryTable,
      partitionKey: "SessionId",
      sessionId,
    }),
    returnMessages: true,
    memoryKey: "chat_history",
    outputKey: "output",
  });

  const agent = createToolCallingAgent({
    llm,
    tools,
    prompt,
  });

  const agentExecutor = new AgentExecutor({
    agent,
    tools,
    memory,
  });

  const result = await agentExecutor.invoke({
    input: message,
  });

Error Message and Stack Trace (if applicable)
No response
Description
Hi,
I'm following this example: https://github.com/langchain-ai/langchainjs/blob/main/examples/src/agents/streaming.ts
to enable streaming in an agent response.
I'm using BedrockChat as the model with the following options:
  const llm = new BedrockChat({
    model: "anthropic.claude-3-sonnet-20240229-v1:0",
    modelKwargs: {
      temperature: 0,
    },
    streaming: true,
    callbacks: [
      {
        async handleLLMNewToken(token) {
          console.log({ token });
        },
      },
    ],
  });

I'm expecting to see a response from the agent streaming in the console, token by token.
But i always get a single token with the full response:
{
  token: "the full response..."
}

Am I missing something?
System Info
{
  name: 'langchain',
  description: 'Typescript bindings for langchain',
  'dist-tags': {
    next: '0.2.3-rc.0',
    latest: '0.2.7'
  },
  version: '0.2.7',
  ...
}

yarn -v
1.22.22
node -v
v20.11.0

Sorry @dosu but this is not helping.
I'm using BedrockChat in the context of a custom agent, so I don't run the model directly.
This is done by the agent after executor.invoke

By default streamRunnable is set to true
I tried to explicitly set it to true without success

By default streamRunnable is set to true I tried to explicitly set it to true without success

Can you try the following code?
const result = await agentExecutor.stream({
  input: message,
});

@jeasonnow thanks but it was not working.
I finally made it work by implementing this solution: https://github.com/aws-samples/langchain-agents/tree/main/bedrock/langchain-js-stream-agent