Blacklist pruning

This PR is a significant pruning of the blacklisted websites list, removing old items, items that haven't had a hit in a while or have had lots of FPs since their last good hit.
The list of items to remove was generated by a script, which is available for review here; in essence, it calculates a value based on how long it's been since a good hit, how many false positives there have been since a hit, and how accurate the item has been, and if the value hits above a certain value it nominates the item for removal.
PRing this for visibility; I don't want to just drop this in on my own.

The blacklists and watchlists definitely need this type of attention, so it's great to be doing this.
This is just my very rough, initial thoughts, and is considerably biased by my pre-existing thoughts on pruning the blacklists. I need to go through the code to see what it's actually doing wrt. how decisions are being made prior to being able to comment reasonably.
I certainly agree that entries which are getting too many FP (or which don't have enough TP) should be removed, usually with placing them on the watchlist, unless the only reason they are getting the FP is a result of us not having bookending on the website blacklist.1
For the blacklists, accuracy is king
To me, the overriding criteria for being on, or off, the blacklist is accuracy: is the entry maintaining a high enough %TP rate to justify being on the blacklist? If it's not accurate enough, then it should be off the blacklist. I'm having a hard time coming up with exceptions to needing to maintain a high enough %TP, there probably are some, but they are quite rare.
There are a significant number of  these entries that I'm at a loss as to why they were ever added to the blacklist, because they don't appear to come remotely close to qualifying, even at the time they were added.
Removal based on time to last detection
Unless we have a need to reduce the amount of CPU cycles which SD uses per post for scanning, I'm not sure that time since last detection should be a significant criteria. It's something to warrant additional investigation, as the domain may have become unregistered, but we've definitely seen spam domains that have been dormant for years return to spamming.
What's the goal of trimming entries which haven't gotten recent hits?
Moving to the watchlist
It's unclear to me if this is proposing just straight removal of all identified entries, or if the intent is to move them to the watchlist after. It currently doesn't show putting the removed entries on the watchlist, but it's possible that the intent is to add a commit doing so after the list is finalized.

In which case, they should be adjusted to have bookending.
I'd really like to see a comparison of results between the no bookending we're currently using and using bookending that prevents entries in the website blacklist from matching domains that happen to encompass the entry. While that issue won't come close to accounting for all the FP, it's definitely been a noticeable number of those I've investigated.

There are a significant number of these entries that I'm at a loss as to why they were ever added to the blacklist, because they don't appear to come remotely close to qualifying, even at the time they were added.

There's a few, particularly near the start of the list, that don't appear to have ever had a hit. I suspect the cause of that, and the cause of some of the early very-low-accuracy items, is a lack of metasmoke at the point they were added.

What's the goal of trimming entries which haven't gotten recent hits?

Seemed like a good idea at the time? :)
Part of the reason that I wrote the code with a formula for calculating a removal value instead of just writing conditions based on each value was just this. I could've said "remove everything that hasn't had a hit in 2 years", but that didn't seem right; the calculation means that if an item's getting removed for old age, it's either really old and hasn't had hits in 4-5 years, or it's newer, but low accuracy.

This PR need to be either merged or closed quickly. As people keep blacklisting things, there will be an increasing amount of conflicts every day, and we'd better resolve them when we still can.

Script for generating watchlist lines from lines in remove.txt:
cat remove.txt | awk '{print systime() "\tArtOfCode\t" $1}' >> watched_keywords.txt

Given 22 days has past since this PR is opened, I would recommend closing it and if necessary, open a new one, due to severe merging conflicts.


Adding GitHub Marketplace topics

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md

I am:

[x] Suggesting edits to an existing topic or collection
[x] Curating a new topic or collection
[ ] Unaffiliated with the project (not self-promoting as e.g. a maintainer, creator, contractor or employee)

EDITING AN EXISTING TOPIC OR COLLECTION
I'm suggesting these edits to an existing topic or collection:

[ ] Image (and my file is *.png, square, dimensions 288x288)
[x] Content (and my changes are in index.md)

Please explain why these changes are necessary:
Helping out the internal topics team.
CURATING A NEW TOPIC

[x] I've formatted my changes as a new folder directory, named for the topic as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME])
[ ] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Please explain why you think this topic should be curated:
We need more things to be globally indexed.
Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.
cc: @brandonrosage @andymckay

@brandonrosage @andymckay Just pushed a ton of updates upstream. Hopefully this meets our requirements and passes CI

@leereilly @brandonrosage Can I get a re-review here now that things have passed CI 😃?
Please and thank you!

Friendly bump for @leereilly as this is slated to ship internally on Tuesday.

@leereilly Can we get one last look over as the feature that is dependent upon this PR goes out today?
cc: @andymckay


lang/latex: Support forward search with Skim

In LaTeX forward search doesn't work for me with the Skim PDF-reader. This fix appears to be necessary.
The format string for opening Skim is taken from Skim's documentation.

Thanks for the help!

Thanks for merging and thanks for the great work you're doing. I've had a blast with doom-emacs such far.


Rewrite

Hello! I like this implementation of Plutus.
Currently, I'm doing a full rewrite of the tool, will add a new way of simple database convertation, refactor and optimise the code.
Will do a PR soon.

Hello.
PR are welcome, thank you.

pr sent

thanks for your contribution


Find bitrot

Fixed a couple more IPTT tests.
97 passing (396.00s)
49 skipped
7 failing

After 3720fd4:
101 passing (471.20s)
49 skipped
7 failing

After 9aa4f94, which disables some tests and fixes others:
100 passing (489.80s)
54 skipped
3 failing


Cropping Leading & Trailing whitespace

As discussed in our meeting where @merapi  reviewed current text issues
details in this doc - https://docs.google.com/document/d/1wchYL4ZClv6b7kSadOR0mQuOZmFAHu9F3klKiFRkGYs/edit#

@merapi, is this not fixed by solution to #1952?

@barklund this one is about the line-height

@merapi, ah this is what we need the font metrics for, right? I'll set it as blocker.

Yep, this is one of the things that will utilize font metrics.

Looks good, couldn't break it!


Fixing grid artifacts when predicting with by_patch_of_size and noise-cancel weights

Thanks for the wonderful package!
I'm trying to upsample some large DSLR photos, I need to use by by_patch_of_size to control memory use
from ISR.models import RDN

rdn = RDN(weights='noise-cancel')
sr_img = rdn.predict(lr_img, by_patch_of_size=300)

Input:

Output:

There is a slight grid artifact along the edge of the patches, marked by decreased brightness. The interior of the patches appears brighter than the original image though, so it is possible the edges are the original brightness.
Any ideas for how to remove the artifact? I was thinking I could shift the image down and to the right by 10 pixels and run the prediction again, then shift back and average the two images.
Thanks!

Hi, there is a parameter for this that is defaulted to 2: padding_size=2.
Use a larger one and it should solve it.

Ahh I didn't see that options, thanks so much!


Enablee Public API analyzer

The public API analyzer was not running for all intended projects due to a global disable. Removing that, fixing the immediate violations. Using this PR to track anymore that may have been missed

@davidwengier, @DustinCampbell  this is my initial attempt to get us in sync with how dotnet/roslyn handles this analyzer. It definitely found a number of real violations. Are there other projects where you expected warnings to get triggered when this was turned back on that I should have a look at?

This is a surprisingly small list, but if it builds, its probably good to merge

Just realised I merged a or that will cause this to fail the main build if merged.
I'll push an update to this branch today.

/azp run


Small changes to certain commands + Command addition

Changed update command message to not say that streaming doesnt work
added TWLoader command
made netinfo command a embed
added 2 memes

dank may mays :ok_hand:

yes yes yes very good PR

fixed

This is no longer a total Meme PR

this will be manually merged due to conflicts, if that's fine with you.

I made some changes and resolved the Conflicts.
You can view the changes in my latest Commit 9cb381c
I removed my additions to the blacklisted words and names of piracy software

Im actually going to write a addition now to the togglechannel command to allow easy additions

Togglechannel has been rewriten and the result is subjective.
The bot has been written for only one server and it being easier to use may or may not be useful

Rewrite of togglechannel is shit and the rest of the cog isnt needed.
Closing


Y extent depends now on displayed series and indicators

Resolves #177

I like this.

Good suggestions :)

I think this looks good.

If this can be squashed into one commit, then I'd be happy to merge :+1:


Need to tag/release for Carthage to have something to pull against

When I do a carthage update right now against your repo it tells me:
No tagged versions found for github "yankodimitrov/SwiftPasscodeLock"
I believe you need to tag/create a release

Ahhh yes my bad. Done!


Include option to decrypt values on pull

Hi there!
First off, thanks for making this awesome library! I'm currently using it in my project and it's been incredibly helpful. While working with it, I noticed an opportunity to enhance its functionality a bit.
I’d like to contribute by adding an option to decrypt values when accessing them using SSM. This addition would be super useful for scenarios where encrypted data needs to be seamlessly integrated.

I made the requested changes. Could you tell me when the new version will be available for download using composer?

@picheidt Released, let me know if it all works or doesn't.

@picheidt Released, let me know if it all works or doesn't.

I tested some cases here and everything looks good!


Extremely high memory usage

I had used PicoscenesToolbox to parse 400K packets of raw CSI data. It cost 64 GiB memory usage.
Is it necessary for parsing numerous packets? Or maybe there is memory leakage.

I think something i write wrong,i will check it whether there is a memory leakage
------------------ 原始邮件 ------------------
发件人: @.>;
发送时间: 2023年12月2日(星期六) 中午12:15
收件人: @.>;
抄送: @.***>;
主题: [Herrtian/PicoscenesToolbox] Extremely high memory usage (Issue #4)
I had used PicoscenesToolbox to parse 40K packets of raw CSI data. It cost 64 GiB memory usage.
Is it necessary for parsing numerous packets? Or maybe there is memory leakage.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

I was checking the code recently. I believe the problem cause by the Cython (rather rxs_parsing_core) code logic. Return a generator instead of a list contain all frames may solve the problem. The generator yield the data frame by frame can reduce the memory usage.


feat(behavior_path_planner): add geometric_parallel_parking

Description

Add parallel parkin planner used in pull_over (plan to use pull_out also).
Generate arc-based parking path using simple geometry of the vehicle kinematic model.

(implementation of some part of Geometric Path Planning for Automatic Parallel Parking in Tiny Spots)
Related links

This is the part of https://github.com/autowarefoundation/autoware.universe/pull/873
depends on https://github.com/autowarefoundation/autoware.universe/pull/1257
Tests performed

check pull_over module https://github.com/autowarefoundation/autoware.universe/pull/873 can work.
forward arc parking.
https://user-images.githubusercontent.com/39142679/177542524-e769692a-02e9-4a90-bb30-96095e6dbaa2.mp4
backward arc parking
https://user-images.githubusercontent.com/39142679/177542658-66c9cfde-2f88-484b-bc6e-af35f85a7e3e.mp4
Notes for reviewers

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.
[ ] The PR has been properly tested.
[ ] The PR has been reviewed by the code owners.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.
[ ] The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

@yukkysaito
moved it under scene_module/utils as geometric_parallel_parking https://github.com/autowarefoundation/autoware.universe/pull/1260/commits/7c85943b06f0945c6ad24b7f8fb0ff4c2c0bc29e


HDDS-9217. Refine certificate renewer service to avoid it scheduled ahead of time

https://issues.apache.org/jira/browse/HDDS-9217

The failed unit tests in TestOzoneFileSystem is not relevant. #5217 is addressing it.


Bug: TypeError in SAM2ImagePredictor.predict() method

@cpuhrsch  Hi! I need your help with a bug in SAM2ImagePredictor.predict() method
# Bug: TypeError in SAM2ImagePredictor.predict() method

## Description
When using `SAM2ImagePredictor.predict()`, two errors occur:
1. When `return_logits=False`: RuntimeError: "Boolean value of Tensor with more than one value is ambiguous"
2. When `return_logits=True`: AssertionError at `postprocess_masks_1_channel()` due to incorrect tensor dimension (expecting channel dimension to be 1)

## Reproduction Steps
```python
from torchao._models.sam2.build_sam import build_sam2
from torchao._models.sam2.sam2_image_predictor import SAM2ImagePredictor

# Initialize model
sam2_checkpoint = "sam2.1_hiera_large.pt"
model_cfg = "sam2.1_hiera_l.yaml"
sam2_model = build_sam2(model_cfg, sam2_checkpoint, device="cuda")
predictor = SAM2ImagePredictor(sam2_model)

# Set image and input points
predictor.set_image(image)
input_point = np.array([[500, 375]])
input_label = np.array([1])

# This call raises the error
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
    return_logits=False  # or True
)

Error Messages
With return_logits=False:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

With return_logits=True:
AssertionError at transforms.py:128: assert masks.size(1) == 1

Additional Context
The input tensor has shape torch.Size([1, 256, 64, 64]). The error seems to occur in the parameter passing between _predict() and _predict_masks_postprocess() methods, specifically around the handling of return_logits parameter.
Environment

Python version: 3.11
CUDA: enabled

Could you please help me understand what's going wrong here? Thank you in advance!

Hey @dongxiaolong - the copy of SAM2 in torchao isn't intended to be general purpose just yet, but specialized towards the example in https://github.com/pytorch/ao/tree/main/examples/sam2_amg_server . The assert comes up because some assumption that was made along the development is being invalidated.

Thank you for your reply.

I'll keep this issue open so I can revisit it later on when this example works.


Feature: Obstacle Detection

This feature employs a ultrasonic sensor that would be positioned in such a way that obstacles at a distance of one metre and below will be detected. The system then uses the text to speech function to alert the user about the ditance to the obstacle.

Issue closed with commit 5bc0ef5


PSR-845, PSR-850, PSR-851 Added identity subject based pages

Replaced with https://github.com/hmrc/pension-scheme-return-frontend/pull/579


fix Fix the problem of concurrent execution in VExpr.

What problem does this PR solve?
In the past, a _can_fast_execute flag was maintained in VExpr.
However, since exec executes concurrently, errors would occur when using the _can_fast_execute judgment.
The correct approach is to put _can_fast_execute into VExprContext.
In addition, to prevent similar problems from occurring in the future, const has been added to all relevant execution functions.
Release note
None
Check List (For Author)

Test

[ ] Regression test
[ ] Unit Test
[ ] Manual test (add detailed scripts or steps below)
[x] No need to test or manual test. Explain why:

[x] This is a refactor/code format and no logic has been changed.
[ ] Previous test can cover this change.
[ ] No code files have been changed.
[ ] Other reason

Behavior changed:

[x] No.
[ ] Yes.

Does this need documentation?

[x] No.
[ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

run buildall

run buildall

run buildall

run p0

run cloud_p0

After some discussion, this PR will be split into two PRs.
The first PR will remove the _can_fast_execute variable.
The second PR will add const to exec.


suggest areas that can be improved in the project

we want to rearchitect how the goal ratings are stored. currently the experts edit in airtable and then we pull that info into a mongo db and convert the mongo collections into json objects and then access it directly. our thought process for going forward is to do the following

stop using airtable because there will not be lots of changes in the future
migrate the existing data in airtable into a postgres db
build an api around serving the ratings and information sheets
store the cover crop images in blob storage and maybe build an api around that
update the front end to use these apis
remove all the json objects that store the ratings/info sheets from the repo
remove all the images from the repo

once you have a feel for how things are architected we would like to get your input on the above plan and get your suggestions for impprovement

Made multiple suggestions to the confluence and added some notes to the README.md though it still needs some love by a seasoned veteran or two.


ZA2-Tersia Koetzee-Full-Stack-Project-Assessment

Hi @tersiakoetzee . There are 1494 files in this PR. Which ones were you hoping mentors review? 😉
Would it be a good idea to open a PR only with the pertinent files? (Never git add . in a codebase)

Hi Sally thank you for that I will get to fixing it.


remove IDCARD dependency on jumpsuit

About the PR

In the YAML files for all creatures that can hold a PDA, I removed IDCARD slots' dependency on the jumpsuit slot.
Why / Balance

as mentioned in issue #28604, one potential noob-trap mechanic is losing your PDA when you switch your jumpsuit. this causes the PDA to be unequipped, and might be forgotten & left behind. this aims to resolve the issue by removing the need for a jumpsuit to carry your PDA
Technical details

YAML file changes
Media

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes

None
Changelog

:cl:

tweak: Players no longer drop their PDA upon unequipping their jumpsuit.

Baby tiders rejoice! No more forgetting your PDA in the middle of the hallway 😆

its not the best solution...

This doesnt really fix the problem. The issue is that swapping clothing still raises the events for unequipping, causing your items to drop. This technically resolves that for IDs but the behavior still happens with pockets and any other slot dependencies that may exist in the future.

The other thing is that the item dropping gives a downside to using disguises. I'm not completely sure that's needed, but that's the effect that items dropping on uniform change has.

de fak,
Noobs must learn and overcome their mistakes, its kinda lazy solution
If you REALLY into "solving" this problem just make +1 slot for pda special, or...
just make a guide for beginers?
Anyway its not a problem and it doesnt require solution because its just
SKILL ISSUE

de fak, Noobs must learn and overcome their mistakes, its kinda lazy solution If you REALLY into "solving" this problem just make +1 slot for pda special, or... just make a guide for beginers? Anyway its not a problem and it doesnt require solution because its just SKILL ISSUE

Knowing that your PDA drops when changing clothes is a skill? That sure has a high skill ceiling...
I agree that for experienced players the real hindrance is that you can't quick-swap jumpsuit while running, which is good to prevent instant disguises. The issue for new players is knowing it drops to begin with.

as discussed on discord, ideally there'll be a do-after when swapping between clothing with content in its specific storage slots

Sticking the ID card pin directly into my flesh now that I can wear it without a jumpsuit.

Sticking the ID card pin directly into my flesh now that I can wear it without a jumpsuit.

you can do this with belt slot it just doesnt provide access

bro have you ever worn a belt before

you dont stick the belt into your flesh

you can wear pda completely naked as a belt on belt slot

spaceman technique

This is really not a problem and does not require fixing.  If a player doesn’t keep track of his things, it’s his fault, such is life.  What if next time he doesn’t keep track of the singularity?  Will you send her too so she doesn’t run away?  Sorry for the surprise, but this is roughly how it looks from the outside.

Side note, why not to spawn pda in belt slot so if people change their clothes nothing would drop down?
Or seperate PDA anad ID card so if a person left an id card they can go to hop and get a new one.

Based change.  SS14’s advantage over SS13 is being new player friendly, and this is a noob friendly change with pretty much zero genuine reasons to be against it other than old players wanting to spite normies which is cringe.  This ain’t skill issue, your characters should be smart enough to switch the contents of their clothing.
It it were up to me id make it so all pockets, jackets, and other such clothing dependent item slots autoswitch to new clothing

Discussed internally and thoughts remain the same.

Based change. SS14’s advantage over SS13 is being new player friendly, and this is a noob friendly change with pretty much zero genuine reasons to be against it other than old players wanting to spite normies which is cringe. This ain’t skill issue, your characters should be smart enough to switch the contents of their clothing.
It it were up to me id make it so all pockets, jackets, and other such clothing dependent item slots autoswitch to new clothing

There are some potential flaws with the change that broke general flow of gameplay - a lot of times the simplest option is likely the wrong one. There were some efforts to create a swapping system, which I had planned to take, but due to a lack of personal time, I had to drop it. It is likely there will be something underway in the near future which will make this more beginner friendly, but I agree with the sentiment that this particular change is not the right one.
I did, however, create PR #28722 which does somewhat address the issue, however it only does it by creating a notification. This has fallen under the radar I believe, or it is still under review, which one it is, I am uncertain, as I really have not had time to hop in discord and check either. If you want to pursue that, I'd recommend mentioning it in discord.


[1.20] Strainers not outputting all items

Strainers can output their items to the inventory below via something like hoppers. This, however, does not seem to work for the last item. Out of 8 items, only 7 get outputted.

Fixed in v1.0.0


Documented workspace setup does not work ?

I was unable to make the documented (in README.md) setup with http_archive work.
However this:
"""
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(name="bazel_pandoc",
remote="https://github.com/ProdriveTechnologies/bazel-pandoc.git",
branch="master")
load("@bazel_pandoc//:repositories.bzl", "pandoc_repositories")
pandoc_repositories()
"""
worked fine.

(figured it out)


[Question] Accessing Actor or Link state in _initialize_episode

In _initialize_episode, I'm able to access the state of an Actor or Link object. Previously, I set the robot's state and accessed the position of its link like this:
self.agent.robot.set_state(agent_state, env_idx=env_idx)
palm_xyz = self.palm_link.pose.p
# use palm_xyz for other initialization
...

I incorrectly assumed the link's position is updated immediately after setting the state. Based on the implementation, it seems this assumption is wrong.
https://github.com/haosulab/ManiSkill/blob/532bb97357faef00a790b010b764f61f4f59333e/mani_skill/envs/sapien_env.py#L808
Is reading states in _initialize_episode generally undefined?
Or should I explicitly ensure the state updates by calling:
if physx.is_gpu_enabled():
    # ensure all updates to object poses and configurations are applied on GPU after task initialization
    self.scene._gpu_apply_all()
    self.scene.px.gpu_update_articulation_kinematics()
    self.scene._gpu_fetch_all()

Looks like our documentation could be updated on this topic. The state update code of apply all, update kinematics, then fetch all is required whenever you want to read up to date link poses.
In general, when setting object pose or velocity (which is what occurs when setting any kind of sim state, state only ever includes pose, velocities, qpos, qvels), this will live update that data on the python side so if you read actor poses, or articulation qpos values, they will be whatever you set them to. Link poses/velocities are the only thing that i am aware of that do not get live updated. Perfectly fine to read actor poses/velocities, just not links at the moment. This is done to ensure we call apply/fetch as little as possible for efficiency.
gpu_apply_all writes all of that data internally to GPU buffers that you can't read. gpu_update_articulation_kinematics will generate link pose/velocity data based articulation qpos values. gpu_fetch_all then reads those GPU buffers. If you want the live link pose info, just call apply all, update kinematics, and fetch all.


Fix our nightly AWS tests

Description
This fixes our nightly AWS tests by allowing them to correctly read the secrets keys.
Related Issue
Fixes #N/A
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[X] Other (security config, docs update, etc)

Checklist before merging

[X] Test, docs, adr added or updated as needed
[X] Contributor Guide Steps followed

@Racer159 Interesting.. It looks like the test failed on the teardown stage because of mismatching package architecture
https://github.com/defenseunicorns/zarf/actions/runs/5059136034/jobs/9080234996#step:12:19


ockam_command: add optional --identity argument to CloudOpts

Depends on: https://github.com/build-trust/ockam/issues/3903
Current
Desired
The scope of this issue is to just add a new argument to all the commands using the CloudOpts.
This argument will be named identity, of type Option<String>, and represents the alias/name
of an identity. It's basically the same identity argument we use in the node create command.
In future PR's, we will modify the requests' payloads to include this new field and use it on the
commands.

We love helping new contributors!
If you have questions or need help as you work on your first Ockam contribution, please leave a comment on this discussion.
If you're looking for other issues to contribute to, checkout this discussion and labels -  https://github.com/build-trust/ockam/labels/good first issue or https://github.com/build-trust/ockam/labels/help wanted

Considering I picked up the #3903 issue, I figured I might take this as well to follow up if there isn't anyone onto it.

@nidnogg awesome. all yours.

I've added a PR with proposed changes, although I'm not sure this is the actual scope of the issue as it seems very tiny to me. Fingers crossed for the checks 🤞

This issue has a small scope because it's the foundation for a series of follow-up issues like this one. The benefit of keeping it small is that those follow-up issues could be done in parallel without conflicts.
I'll review the PR shortly, thanks!


Add options for disabling zopfli and brotli to the fusionrc file

Are there reasons for this? Is this only during local build or something, or would you want to do this for a prod build too?

We'd want this for local and production. The issue is to mitigate performance degradations in previous fusion-cli versions and give users the flexibility of enabling them if they're willing to take the performance hit.

Closed by https://github.com/fusionjs/fusionjs/pull/606


WinAFL crash with incorrect target_offset but ddrun run fine.

I had an issue when I tried to fuzz a correct target_offset in a Blackbox binary, seems like WinAFL is unable to find the offset.
drrun show the following correct output using the target_offset:
Command:
drrun.exe -c winafl.dll -debug -target_module binary.exe -target_offset 0xa06f0 -fuzz_iterations 5 -nargs 3 - C: \ binary.exe -e C: \ sample C: \ output
In pre_fuzz_handler
In OpenFileW, reading C: \ sample
In post_fuzz_handler
In pre_fuzz_handler
In OpenFileW, reading C: \ sample
In post_fuzz_handler
In pre_fuzz_handler
In OpenFileW, reading C: \ sample
In post_fuzz_handler
In pre_fuzz_handler
In OpenFileW, reading C: \ sample
In post_fuzz_handler
In pre_fuzz_handler
In OpenFileW, reading C: \ sample
In post_fuzz_handler
Everything appears to be running normally.
Coverage map follows: ...

But when I used the same offset in WinAFL, the just application crash:

The WinAFL log said:
Exception caught: e06d7363
Exception caught: e06d7363
WARNING: Target function was never called. Incorrect target_offset?
Coverage map follows: ...

What is going on? There is a workaround to this issue?

Note that you should remove the -debug flag from your command line when running afl-fuzz.exe. It is only meant to be used with drrun.exe directly and leaving it will prevent afl-fuzz from connecting to your target.
Also note that if you recompiled your target between running the debug mode and afl-fuzz, the target method offset could have changed.

Hi @ifratric, the application also crash without de -debug function applied, as can be seen in the following image:
Command:
afl-fuzz.exe -i C:\minset -o C:\crashes -D C:\DynamoRIO\bin32 -t 1000+ -- -coverage_module binary.exe -fuzz_iterations 01 -target_module binary.exe -target_offset 0xa06f0 -nargs 3 -- C:\binary.exe @@ C:\trash

The binary was not recompiled and it was not patched, its the original Blackbox binary.
Is there is a way to debug this behaviour?

At what point does the error occur? During the dry run or when fuzzing already starts? I'm thinking that maybe, on invalid (fuzzed) samples, your target throws some custom exception that WinAFL doesn't know how to handle currently.

Hi @ifratric the error happened during Dry Run!

Hi @ifratric, I was able to create a harness to avoid the above issue, but after a good time of fuzzing (around 1 hs) I got the following error message: OS message : Not enough space

Can you give me some insights about the issue?

PD: It is not a OS space problem due that I have 11.5 GB Free at my HD

Can't tell you much more than what you see from the error message (i.e. where it occured). Possibly your target hasn't closed the file at the end of the iteration, which is why WinAFL is unable to write to it. However, in that case, WinAFL should kill the target process (which will impact performance, but shoudln't cause a crash).

Hi @ifratric I will check the harness again to check possible solutions to this issue! Thanks for your fast reply!


Failed to run FastAPI Framework without flask installed

Blessed to see this excellent and fast updated framework.
code:
from fastapi import FastAPI
from fastadmin import fastapi_app as admin_app

app = FastAPI()

app.mount("/admin", admin_app)

console:
└─$ uvicorn main:app --reload
INFO:     Will watch for changes in these directories: ['/home/s1nh/project/fastadmin']
INFO:     Uvicorn running on http://<IP_ADDRESS>:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [902379] using WatchFiles
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
    target(sockets=sockets)
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/server.py", line 59, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/server.py", line 66, in serve
    config.load()
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/config.py", line 471, in load
    self.loaded_app = import_from_string(self.app)
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
  File "/home/s1nh/conda/miniconda3/envs/envp310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/s1nh/project/znss-admin/fastadmin/main.py", line 2, in <module>
    from fastadmin import fastapi_app as admin_app
  File "/home/s1nh/project/znss-admin/fastadmin/fastadmin/__init__.py", line 3, in <module>
    from fastadmin.api.frameworks.flask.app import app as flask_app
  File "/home/s1nh/project/znss-admin/fastadmin/fastadmin/api/frameworks/flask/app.py", line 4, in <module>
    from flask import Blueprint
ModuleNotFoundError: No module named 'flask'
^CINFO:     Stopping reloader process [902379]

@duchengyao yes I'm on it
Now I'm using extras
[tool.poetry.extras] fastapi = ["fastapi"] flask = ["flask"] django = ["django"]
Going to test and update documentation


Documentation website not working

What did you do?
I go to https://digia.online/react-foundation-docs/.
What did you expect to happen?
I expected a working site.
What actually happened?
It showed "404 There isn't a GitHub Pages site here".

Thanks for the report, we'll take a look.
In the meantime, you can see the docs at https://digiaonline.github.io/react-foundation-docs/

OK should be sorted now!

The digia.online recently expired and was renewed, and it seems GitHub Pages had unlinked from it.
The fix was to go to the top-level repo (https://github.com/digiaonline/digiaonline.github.io/settings/pages), switch the branch from master to a temporary branch (from master) and then switch it back.

It then picked up the CNAME containing digia.online and re-linked the pages to their URLs.
https://github.com/digiaonline/digiaonline.github.io/blob/master/CNAME
Thanks again!


Data Updates Details

Hi there, I am looking to use this data moving forward but noticed some data isn't as up to date compared to the CIA website, even though it says it is updated recently. Can you explain what level of detail goes into these updates for the data? For example, some topics/data shows from 2020, when on the CIA it has the updates from 2024.

find weekly updates here -> https://github.com/factbook/factbook.json  plus Q&A and more.

Hi, thank you for this. I still have the data update question unanswered... For example in the U.S, Military Expenditures is updated until 2024 but in the data it does not include this updated. Curious as to if only certain topics are updated weekly or do you updated all data across all countries listed on CIA, in a weekly basis?

from the readme:
factbook.json - Philosophy

Don't be "smarter" than the CIA
Use the same data as the CIA (including "known" typos /errors)

thus, you have to ask the CIA about the update frequency of countries / sections etc.


[bugfix]Using original string to embedding when quering instead of deliver tokenized list to embedding model.

Description
It will deliver tokenized query string to embedding service, which is not accepted sometimes.
{
  "input": [
    67933,
    12774,
    228,
    37687,
    37687,
    37985,
    40862,
    98657,
    52084,
    50285,
    13870,
    118,
    45059,
    10110,
    49,
    1929,
    7705,
    34208,
    85315,
    115,
    40862,
    96511,
    46729,
    10110,
    10754,
    36,
    7705,
    9554,
    98897,
    42052,
    51611,
    11883,
    20119,
    114,
    78935,
    198
  ],
  "model": "text-embedding-ada-002",
  "encoding_format": "base64"
}

Related Issues
[Reference any related issues or tasks that this pull request addresses.]
Proposed Changes

decode the token to original string instead of token just like what index do.

Checklist

[x] I have tested these changes locally.
[x] I have reviewed the code changes.
[ ] I have updated the documentation (if necessary).
[ ] I have added appropriate unit tests (if applicable).

Additional Notes
[Add any additional notes or context that may be helpful for the reviewer(s).]

Cloning locally to test

This one was included in #909
Thanks a lot for your valuable contribution! Much appreciated


Netherite armour durabilities broken

They are all the same durability value as the helmet (407)

Still broken

maybe fixed


Adding tests for RefreshingOAuth2CredentialsInterceptor

Making more methods private and using other methods in RefreshingOAuth2CredentialsInterceptorTest.
Moving variables to be more consistent.

LGTM - the reduced scopes, avoiding using syncRefresh directly, and the improved test cases look good. The overall functionality with ratelimited background token refresh through ClientCall interceptor is a bit dense :) Specially, this call. Is it adding a chain of interceptors till the refresh succeeds or fails on a stale token?
delegate().start(new UnAuthResponseListener<>(RefreshingOAuth2CredentialsInterceptor.this,
responseListener, headerCache), headers);

The UnAuthResponseListener is a way for us to find an unusual case that pops up.  We use OAuth2 tokens to establish authentication.  Most of the time, an OAuth token has to be refreshed before it expires.  Sometimes, the server-side can revoke the token before expiration and send UNAUTHENTICATED statuses.  UnAuthResponseListener will force a refresh of the token on UNAUTHENTICATED, but will only do that once every 10 seconds.


possible bug: some creatures/monsters from the default bestiary render with 0 XP

Hey there,
I noticed, that some creatures/monsters from the default bestiary render with 0 XP (theRaven for example).
Problem occurs both in a simple statblock and in use with the obsidian-initiative-tracker plugin (encounter).
I looked in the master/src/data/srd-bestiary.ts file and realized, that the experience points should be rendered based on the given challenge rating. So my first guess, that in the bestiary may be there is missing information, cannot be correct - what can be the cause of this problem instead?
Greetings :)

I think this happens with any cr: 0 value. For example, manually entering in Larva from p63 of the DM's guide, will produce 0 XP when the text declares 10 XP.

Ah indeed, it is currently set that anything with a CR of 0 is giving 0 XP when it should be 10xp.
@valentine195 Is there any technical limitations for me submitting an update to make 0 cr's be a value of "0.1" and 10xp respectively?


Decrementing minutes skips over "00"

This can be seen on the demo page too: http://rajeshwarpatlolla.github.io/TimePickerForIonicFramework/demo/#/tab/timepicker
On decrementing the minutes using the arrow button, the value goes straight from, say 10 to 50 instead of 10 -> 00 -> 50.
The increment functionality works as expected: 50 -> 00 -> 10

+1

This has been fixed in the latest commit to the master branch. Please check it in the latest commit


Fix GetBlockHeader() to include the auxpow for sending merkle block correctly

Currently the MERKLEBLOCK message is broken and makes server node crash when spv client send MSG_FILTERED_BLOCK message.
The fix will including the auxpow and resolve the assert error for not existing auxpow.

This PR has been superseded in <IP_ADDRESS>


Broken link to Gateway API release notes

This is a Bug Report

Problem: https://kubernetes.io/blog/2023/10/31/gateway-api-ga/#everything-else has a link to the wrong release notes.
Proposed Solution:
Edit the blog article to link to 1.0.0 not 0.1.0
Page to Update:
https://kubernetes.io/blog/2023/10/31/gateway-api-ga/#everything-else

Additional Information:
/area blog
/sig network
/language en

I will attempt to help with this.
/assign

We should fix the broken link in the blog.
/triage accepted

I have submitted PR https://github.com/kubernetes/website/pull/43762 to resolve the issue


failed: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory

I got the following failure on this task:
TASK [common : Pass bridged IPv4 traffic to iptables' chains] ****************** fatal: [black]: FAILED! => {"changed": false, "failed": true, "msg": "setting net.bridge.bridge-nf-call-iptables failed: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
To fix, I had to ssh to each of my nodes and run the following commands:
sudo modprobe br_netfilter sudo sysctl -p
The ansible playbook would then proceed

I had this same issue, re-running the playbook a 2nd time succeeded ... its not a fix, but it is a workaround.

It did work for me too. But, I don't understand why it would work the second time.  :(

Hi @aaronpkelly - Could you please close this issue?


exit the process anyway after a certain timeout

As titled, in some bad situations the close sequence might be stuck

Do you know the cases?  It's a bug or just a bad constellation?

sockets to dbs (or similar) not closing correctly, persistent websockets, some strange error conditions (automatic reconnects), and other bits.
Doing a proper shutdown in 100% of the cases is very hard.

Reopen if you find any issue.


Proposal: adding Unit tests

Hi ChatDev! enjoying the tool so far, though I've found that it's struggled to make a working version of a slightly more complex problem, and one part of that is the code not even running without code changes!
I think if we could add unit testing to this, it could reduce this problem, and I've had great success with AI-generated tests. I have asked this via the prompts, but it's not actually happening, so perhaps this needs to be in the code instead?
I Propose it is the engineer's responsibility to create test files for each file it creates (i.e. main.test.py), and the responsibility of the QA? to run the test - though one might argue it should be run before the code is submitted...

Hello! Thank you very much for this substantial suggestion, and we will certainly consider it. Unfortunately, due to recent scheduling constraints with our development team, we currently don't have immediate plans for unit testing. However, for optimizing before software generation, you might consider using the "--config 'Human'" option to enable aHuman-Agent-Interaction mode. Please stay tuned for our upcoming features, and we believe they will be arriving in the near future.😊😊😊


beakerlib missing for upgrade step

When adding an upgrade step which uses beakerlib framework, it seems beakerlib is not being installed and the upgrade tasks fail with ./task.sh: line 2: /usr/share/beakerlib/beakerlib.sh: No such file or directory
In the upgrade tasks main.fmf:
test: ./task.sh
framework: beakerlib

Example report (login-logout-stress is not using beakerlib framework):

Hopefully small enough reproducer which runs fast as it doesn't upgrade :)
Run as tmt run -avvv plan -n parent
Currently /check if beakerlib installed fails on old and new as beakerlib isn't installed and the  task-beakerlib runs inside 'perform the system upgrade' fails as beakerlib isn't installed.
tmt init

cat > parent.fmf <<EOF
discover:
    how: shell
    tests:
    - name: check if beakerlib installed
      test: test -e /usr/share/beakerlib/beakerlib.sh

provision:
    how: container
execute:
    how: upgrade
    path: upgrade-path
EOF

cat > task-beakerlib.fmf <<EOF
framework: beakerlib
test: |
    . /usr/share/beakerlib/beakerlib.sh || exit 1
    rlJournalStart
       rlPhaseStartTest
            rlPass "yes"
      rlPhaseEnd
    rlJournalEnd
EOF

cat > upgrade-path.fmf <<EOF
discover:
    how: fmf
execute:
    how: tmt
EOF

So if I understand it correctly (as a total noob), the Upgrade being Execute step and having it's own tasks discovery, resolving the framework/requires/recommends of the tasks would need one of:

Modifying the Discovery step to check for upgrade step, do the custom discovery and then pass the data to the upgrade step somehow.
Having a custom prepare step being run before the Upgrade. I believe it would be better to have all dependencies resolved before the first test execution, in order to have the system is in a same state for 'old' and 'new' runs.
Declaring (and informing a user) that upgrade tasks are not 'tests' and do not support these keywords (and adding beakerlib/other frameworks as dependency of upgrade plugin).

Having a custom prepare step being run before the Upgrade. I believe it would be better to have all dependencies resolved before the first test execution, in order to have the system is in a same state for 'old' and 'new' runs.

I'm not sure about this one. At one hand it makes it easier to compare old/new as just "upgrade" happen.
On the other hand I think it could be more useful to have 'Upgrade' run also prepare/finish and as such it can add repos which make dependencies available (they are not available for the OLD and if finish step removes them then neither for NEW).

So I'd advocate to change 'ExecuteUpgrade' to use provisioned guests of the parent run and run discover & prepare (including install require/recommend) & execute & finish.
Finish might be a feat for the future as currently this steps does also guest cleanup which should be moved to its own, new, step

So I'd advocate to change 'ExecuteUpgrade' to use provisioned guests of the parent run and run discover & prepare (including install require/recommend) & execute & finish. Finish might be a feat for the future as currently this steps does also guest cleanup which should be moved to its own, new, step

Do you mean by calling self.step.plan.prepare.go(), or something more complicated? I did try your reproducer, it worked very well, and I was thinking whether it'd be possible to somehow reconnect upgrade to the right flow of tmt pipeline. It feels very exceptional right now... But, not enough time spend there yet, may change my mind & trash it completely :)

Currently upgrade does this: https://github.com/teemtee/tmt/blob/main/tmt/steps/execute/upgrade.py#L288
So instead of calling just _discover_upgrade we should call also at least prepare.
I'm not sure how much work would be to change current implementation to just call prepare.go() after feeding it with relevant data.

Regardless of how much work would it be, I'd like to humbly suggest that it might be a good idea to take a step back and decide what the "upgrade" is actually for and how it should be implemented, because what you're describing sounds like a full tmt run within a tmt run, just with extra custom steps :)

because what you're describing sounds like a full tmt run within a tmt run, just with extra custom steps :)

With few _less _steps :) but yes. IMHO this is what upgrade plan is and I see no issue why it shouldn't be it.
@psss Where would be best place to sync a bit about it? This was implemented by LEAPP, right?

see no issue why it shouldn't be it.

All I'm saying is that conceptually there should be only one prepare step.

It might be a good opportunity to improve ways in which a plugin could invoke other steps and plugins when needed. It's not trivial today, upgrade needs to shuffle discover and execute a bit to make it work, now we need to run prepare properly...

All I'm saying is that conceptually there should be only one prepare step.

What is 'step' in this case? Step as a implementation or Step as a time when it runs? OK for implementation, opt-out for time.

As in resolving all dependencies and preparations before the first test(s) are being run.

As in resolving all dependencies and preparations before the first test(s) are being run.

Then I beg to differ that "upgrade dependencies and preparations" should be run as part of the upgrade itself or at least have it configurable.

Well we can't all have the same opinions on everything :)
@psss This might need some discussion and consensus. Can you help please?

Not directly related but https://github.com/teemtee/tmt/issues/2784 might change how/when tmt gathers require/recommend.


upgrading 4.9.0 to 5.0.0+ causes a problem with the provider

What happened?
The newer provider doesn't like the token value, do we need to reset it after upgrading the provider?
I am using Method 2: from https://www.pulumi.com/registry/packages/gitlab/installation-configuration/#configuring-credentials
View in Browser (Ctrl+O): https://app.pulumi.com/xxxxxx/gitlab-management/prod/previews/d77e2470-b284-493c-8dc1-74647560f7c7

     Type                          Name                       Plan       Info
     pulumi:pulumi:Stack           gitlab-management-prod
 +   ├─ xxxx:gitlab:Repository  xxxxx-infrastructure  create
     └─ pulumi:providers:gitlab    default_5_0_1                         1 error

Diagnostics:
  pulumi:providers:gitlab (default_5_0_1):
    error: rpc error: code = Unknown desc = config value 'token' mismatch between servers 1 and a previous server

reverting to 4.9.0 works

Installing collected packages: pulumi-gitlab
  Attempting uninstall: pulumi-gitlab
    Found existing installation: pulumi-gitlab 5.0.1
    Uninstalling pulumi-gitlab-5.0.1:
      Successfully uninstalled pulumi-gitlab-5.0.1
Successfully installed pulumi-gitlab-4.9.0

[notice] A new release of pip available: 22.2.2 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(310) xxxx@xxxx-LT01:~/code/gitlab-management$ pulumi preview
Previewing update (xxxxx/prod)

View in Browser (Ctrl+O): https://app.pulumi.com/xxxxx/gitlab-management/prod/previews/43ce044b-c2ec-4363-bffb-d65295716462

     Type                                           Name                                                       Plan       Info
     pulumi:pulumi:Stack                            gitlab-management-prod
 +   ├─ xxxxxxx:gitlab:Repository                   xxxx-infrastructure                                  create
 +   │  └─ gitlab:index:Project                     xxx-infrastructure-repository                       create
 +   │     ├─ gitlab:index:BranchProtection         xxxx-infrastructure-main-branch-protection           create
 +   │     └─ gitlab:index:ProjectLevelMrApprovals  xxx-infrastructure-merge-request-approval-settings  create

Expected Behavior
The token value doesn't change and the existing token configured should work
Steps to reproduce
Have a stack configuration like:
config:
  gitlab:token:
    secure: xcvxcvcxcxvxvxcvxc

and use provider version 5.0.0+ after using 4.9.0 or below.
Output of pulumi about
CLI
Version      3.65.1
Go Version   go1.20.3
Go Compiler  gc

Plugins
NAME      VERSION
aws       5.31.0
awsx      1.0.2
docker    3.6.1
gitlab    4.9.0
python    unknown
signalfx  5.9.0

Host
OS       ubuntu
Version  22.04
Arch     x86_64

Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

I Confirm the issue, same error since upgrade to 5.0

Thanks for reporting this, I can reproduce, in fact the issue appears to not be scoped to upgrades but affects a simple program on 5.0.1 that you try to pulumi up. Investigating.


Fix stuck recovery

Swaps that have been claimed by csv can get stuck on the peers side on startup.
This PR reworks the tx watcher to check all blocks for the desired tx since the block the swap started. If the tx is above the safety limit we return an error and fail the swap.

Please have a detailed look at and double check the observationLoop and the IsTxInMempoolOrRange method during review.

Blockstream Store is now running 63378ba8fff94052383f25d3dddcea04157cd870. Unfortunately this doesn't seem to have helped to unstick that swap.
Before
  {
     "id": "975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd",
     "created_at":<PHONE_NUMBER>,
     "asset": "lbtc",
     "type": "swap-in",
     "role": "receiver",
     "state": "State_SwapInReceiver_AwaitTxConfirmation",
     "initiator_node_id": "02d0e03736cbfc73f3c005bc3770327df0e84bd69bc8e557c279887344deb8bce2",
     "peer_node_id": "02d0e03736cbfc73f3c005bc3770327df0e84bd69bc8e557c279887344deb8bce2",
     "amount": 8000000,
     "channel_id": "798627x1031x0",
     "opening_tx_id": "9af850fd41bf916146f6bf43c4da4551c69ed9d3473ab61ff52eb381fd0465ae",
     "lnd_chan_id":<PHONE_NUMBER>23431168
  },

Log during startup
Aug 06 23:25:16 lightning-charge-production-hwwp docker[92511]: 2023-08-06T23:25:16.004Z INFO    plugin-peerswap: [Swap:4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f]: Recovering from state State_SwapOutReceiver_SendTxBroadcastedMessage
Aug 06 23:25:16 lightning-charge-production-hwwp docker[92511]: 2023-08-06T23:25:16.008Z DEBUG   plugin-peerswap: [FSM] event:id: 4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f, Event_ActionFailed on State_SwapOutReceiver_SendTxBroadcastedMessage
After
  {
     "id": "975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd",
     "created_at":<PHONE_NUMBER>,
     "asset": "lbtc",
     "type": "swap-in",
     "role": "receiver",
     "state": "State_SwapInReceiver_AwaitTxConfirmation",
     "initiator_node_id": "02d0e03736cbfc73f3c005bc3770327df0e84bd69bc8e557c279887344deb8bce2",
     "peer_node_id": "02d0e03736cbfc73f3c005bc3770327df0e84bd69bc8e557c279887344deb8bce2",
     "amount": 8000000,
     "channel_id": "798627x1031x0",
     "opening_tx_id": "9af850fd41bf916146f6bf43c4da4551c69ed9d3473ab61ff52eb381fd0465ae",
     "lnd_chan_id":<PHONE_NUMBER>23431168
  },

Ok some thoughts:
for 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd:

Starting Blockheight is 2447093
opening_tx_id=cf6edaa5f0240d652155c9921298922e8b97c8b117ce941dc4d18a60e1b6c523
opening_tx was confirmed the first time in block 2447094
opening_tx was spent in block 2447154
difference is 60 blocks, so csv worked out in the end.

https://github.com/ElementsProject/peerswap/blob/63378ba8fff94052383f25d3dddcea04157cd870/swap/actions.go#L657-L671
These lines from the AwaitTxConfirmationAction alone should cancel the ongoing swap, which would be intended behavior. The following test passes:
https://github.com/ElementsProject/peerswap/blob/63378ba8fff94052383f25d3dddcea04157cd870/test/recovery_test.go#L218-L234
We basically create a situation in which the csv limit passes and where the maker refunds itself. Restarting the taker node calls the recovery and cancels the ongoing swap that now exceeded the limits.
On the other hand we have 4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f where we are stuck in the state State_SwapOutReceiver_SendTxBroadcastedMessage which is not covered yet by this PR.
I suspect that 4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f being created earlier and/or having an Id of a lesser lexicographic order is the first we try to recover, but fail, as we do not have proper recovery for this state yet.
It seems that the recovery mechanism of the state machine works sequentially and returns on the first encountered error, leaving the other swaps untouched.
Give me some time to fix this on the state machine, that should be able to recover from 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd as well as from 4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f, and that does not exit on error and continues to recover the next swap.

Tested using signet L-BTC swaps since the BTC timeouts would take a while (about 3 days):
Ordinary swap-ins work.
Turning off PeerSwap on the remote peer and then starting it back up immediately results in State_ClaimedPreimage.
When turning off PeerSwap on the remote peer after 50% of the CSV timeout has passed, the result is State_ClaimedCoop, exceeded safe swap range.

This is a huge improvement! Blockstream Store PeerSwap is restored.
4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f remains stuck like above but it no longer prevents other recoveries to happen. 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd successfully recovers with the expected Coop state.
New problem ...
 It will go forever.
New problem: Retry is far too fast, overwhelming the log storage.
For now slow down the retry?

We can slow it down, it tries once a second as liquid is really fast :rocket: but I guess it would not hurt to try once every 30 sec or 1min.

2023-08-11T06:02:56.570Z DEBUG   plugin-peerswap: Waiting for all open swaps to recover.
This might be better as INFO?

Not a regression but think about this for later
If CLN and peerswap start at the same time there is no guarantee that these peers will be connected before these recoveries begin. Aren't these recoveries more likely to have a better outcome if the peer is connected at that moment? Is Waiting for cln to be synced... waiting for 4.6 seconds long enough? My guess is no.
In the below log you see plugin-peerswap restarted during the already running CLN. You see incoming polling at random times interspersed during the recovery messages which makes me suspect we are not waiting long enough before attempting the recoveries.
2023-08-11T06:02:44.891Z INFO    plugin-peerswap: Using core-lightning version v23.05.2
2023-08-11T06:02:44.891Z INFO    plugin-peerswap: Waiting for cln to be synced...
2023-08-11T06:02:55.114Z INFO    plugin-peerswap: Cln synced, continue...
... snip ...
2023-08-11T06:02:56.505Z INFO    plugin-peerswap: suspicious_peers: []
2023-08-11T06:02:56.505Z INFO    plugin-peerswap:
2023-08-11T06:02:56.569Z INFO    plugin-peerswap: [Swap:4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f]: Recovering from state State_SwapOutReceiver_SendTxBroadcastedMessage
2023-08-11T06:02:56.570Z DEBUG   plugin-peerswap: Waiting for all open swaps to recover.
2023-08-11T06:02:56.571Z INFO    plugin-peerswap: [Swap:70b00d81453a1a441904de2e51fb51f52a34d928ed2327766210b43f1e5bcfd0]: Recovering from state State_SwapInReceiver_AwaitTxConfirmation
2023-08-11T06:02:56.571Z INFO    plugin-peerswap: [Swap:975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd]: Recovering from state State_SwapInReceiver_AwaitTxConfirmation
2023-08-11T06:02:56.571Z INFO    plugin-peerswap: [Swap:ca17bae36e72c7348887e1d4c834ee0a9475a65ec22fbabb423b6cf841e3182a]: Recovering from state State_SwapOutReceiver_SendTxBroadcastedMessage
2023-08-11T06:02:56.571Z DEBUG   plugin-peerswap: [RedundantSender] start sending messages of type 42077 to 0360a41eb8c3fe09782ef6c984acbb003b0e1ebc4fe10ae01bab0e80d76618c8f4
2023-08-11T06:02:56.578Z DEBUG   plugin-peerswap: [FSM] event:id: 4d1b7d6f1dd4de3e93132d6b8c489829f52a277e5aedf699830040a025eef14f, Event_ActionFailed on State_SwapOutReceiver_SendTxBroadcastedMessage
2023-08-11T06:02:56.586Z DEBUG   plugin-peerswap: [FSM] event:id: ca17bae36e72c7348887e1d4c834ee0a9475a65ec22fbabb423b6cf841e3182a, Event_ActionSucceeded on State_SwapOutReceiver_SendTxBroadcastedMessage
2023-08-11T06:02:56.627Z DEBUG   plugin-peerswap: Received poll from peer 029a0f8111f0bcb94003413838451dfc1d0faed030ef46ff84d96c0dc5be3d1415: {\"version\":3,\"assets\":[\"btc\",\"lbtc\"],\"peer_allowed\":true}
2023-08-11T06:02:56.681Z DEBUG   plugin-peerswap: Received poll from peer 03a465772d45616bf6c8450a69191db8f3cf8cca19ff92138735fd5f1d436fe4dc: {\"version\":3,\"assets\":[\"lbtc\",\"btc\"],\"peer_allowed\":true}
2023-08-11T06:02:56.711Z DEBUG   plugin-peerswap: [FSM] event:id: 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd, Event_ActionFailed on State_SwapInReceiver_AwaitTxConfirmation
2023-08-11T06:02:56.714Z DEBUG   plugin-peerswap: [FSM] event:id: 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd, Event_ActionSucceeded on State_SwapInReceiver_SendPrivkey
2023-08-11T06:02:56.715Z DEBUG   plugin-peerswap: Received poll from peer 035b1ff29e8db1ba8f2a4f4f95db239b54069cb949b8cde329418e2a83da4f1b30: {\"version\":3,\"assets\":[\"btc\",\"lbtc\"],\"peer_allowed\":true}
2023-08-11T06:02:56.718Z DEBUG   plugin-peerswap: [FSM] event:id: 975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd, Event_ActionSucceeded on State_SwapInReceiver_SendCoopClose
2023-08-11T06:02:56.718Z INFO    plugin-peerswap: [Swap:975ca468baf5dc8843afc61acb99e567b615f9fc123af250d5541eb591fe60dd] Swap claimed cooperatively
2023-08-11T06:02:56.727Z DEBUG   plugin-peerswap: Received poll from peer 039cdd937f8d83fb2f78c8d7ddc92ae28c9dbb5c4827181cfc80df60dee1b7bf19: {\"version\":3,\"assets\":[\"btc\",\"lbtc\"],\"peer_allowed\":true}
2023-08-11T06:02:56.750Z INFO    plugin-peerswap: [TxWatcher] checkTxAboveCsvHeight returned: empty gettxout response
2023-08-11T06:02:56.757Z DEBUG   plugin-peerswap: [FSM] event:id: 70b00d81453a1a441904de2e51fb51f52a34d928ed2327766210b43f1e5bcfd0, Event_ActionFailed on State_SwapInReceiver_AwaitTxConfirmation
2023-08-11T06:02:56.760Z DEBUG   plugin-peerswap: [FSM] event:id: 70b00d81453a1a441904de2e51fb51f52a34d928ed2327766210b43f1e5bcfd0, Event_ActionSucceeded on State_SwapInReceiver_SendPrivkey
2023-08-11T06:02:56.762Z DEBUG   plugin-peerswap: [FSM] event:id: ca17bae36e72c7348887e1d4c834ee0a9475a65ec22fbabb423b6cf841e3182a, Event_OnClaimInvoicePaid on State_SwapOutReceiver_AwaitClaimInvoicePayment
2023-08-11T06:02:56.762Z INFO    plugin-peerswap: [Swap:ca17bae36e72c7348887e1d4c834ee0a9475a65ec22fbabb423b6cf841e3182a] Swap claimed with preimage 0b3d604b5364ae6dadfb33e08a56fc819c873e342efaa38681bd0360e2136000
2023-08-11T06:02:56.762Z DEBUG   plugin-peerswap: [RedundantSender] stop sending messages of type 42077 to 0360a41eb8c3fe09782ef6c984acbb003b0e1ebc4fe10ae01bab0e80d76618c8f4
2023-08-11T06:02:56.764Z DEBUG   plugin-peerswap: [FSM] event:id: 70b00d81453a1a441904de2e51fb51f52a34d928ed2327766210b43f1e5bcfd0, Event_ActionSucceeded on State_SwapInReceiver_SendCoopClose
2023-08-11T06:02:56.764Z INFO    plugin-peerswap: [Swap:70b00d81453a1a441904de2e51fb51f52a34d928ed2327766210b43f1e5bcfd0] Swap claimed cooperatively
2023-08-11T06:02:57.033Z DEBUG   plugin-peerswap: Received poll from peer 03c53ba23ea401b185a9d6e79b3ba776d4e78ab1a90e8ca28de90da27494002a12: {\"version\":3,\"assets\":[\"lbtc\",\"btc\"],\"peer_allowed\":true}

Is Waiting for cln to be synced... waiting for 11 seconds long enough? My guess is no.

There is no guarantee that any length of time will be long enough for peers to reconnect. However judging from the timing of those incoming polls for already-connected peers I think we would be better off waiting 10 seconds after Cln synced, continue... before proceeding with recoveries. That gives a high chance of a live peer reconnecting before recoveries without being too long for the vast majority of times we aren't waiting for recoveries.
Alternative: How easily can we detect if we have any pending recoveries? If we have none then we have no reason to wait during peerswap startup. Waiting for all open swaps to recover. could then conditionally instead print No pending swaps to recover.

Please rename Waiting for all open swaps to recover. to Waiting for all pending swaps to recover.

ACK, merging now because this is a strict improvement and we need other pull requests on top of this.
Fix minor comments in separate ticket.


Adding DVGeometryMulti to MPhys

Purpose
Adding DVGeometryMulti to MPhys. I added pass-through functions and modified the compute and compute_jacvec_product calls as needed.
Expected time until merged
1 week.
Type of change

[ ] Bugfix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (non-backwards-compatible fix or feature)
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Documentation update
[ ] Maintenance update
[ ] Other (please describe)

Testing
I tried this on my optimization and it appears to work (local analysis and derivatives on HPC). I would appreciate others verifying this on their cases.
Checklist

[X] I have run flake8 and black to make sure the Python code adheres to PEP-8 and is consistently formatted
[ ] I have formatted the Fortran code with fprettify or C/C++ code with clang-format as applicable
[ ] I have run unit and regression tests which pass locally with my changes
[ ] I have added new tests that prove my fix is effective or that my feature works
[ ] I have added necessary documentation

I didn't add the addIntersection() method to the MPhys wrapper. Let me know if you think it should be here or if it should be left as a direct reference to the underlying class.

Additional note: when adding a global DV the input was being added twice to OpenMDAO. This bug is fixed here too.

It turns out PR #201 broke the naming convention for child FFDs by prepending the child name by default. This meant that MPhys would not connect the OM variable to the DVGeo variable because the names would not match. I fixed that in this PR by setting prependName=False for all FFD-based design variables.

There is some weird behavior in the derivatives: the first check totals is fine, but a second check totals after a design perturbation makes the second function derivative wrong. This only happens when there is an intersection. I can't exclude this being an issue with DAFoam (or even IDwarp), but I do not have the bandwidth to look more into this now. Converting to a draft.

This is good to go. The issues with derivatives are coming from other code components, not DVGeo, and they are not unique to this PR. We should merge this in so it does not go stale again.

Looks good overall. Is there a corresponding addIntersection call? If not, how are you handling it?

Instead of having a pass-through call in MPhys, I just call the DVGeo method directly. For example:
DVGeo = self.geometry.nom_getDVGeo()
self.geometry.nom_addComponent("hub", "FFD/hubFFD.xyz", triMesh="hubTri.cgns")
self.geometry.nom_addComponent("blade", "FFD/bladeFFD.xyz", triMesh="bladeTri.cgns")
DVGeo.addIntersection(
    "hub",
    "blade",
    dStarA=1.0,
    dStarB=0.05,
    featureCurves=featureCurves,
    project=True,
    includeCurves=True,
    curveEpsDict=curveEpsDict,
    anisotropy=[1.0, 1.0, 0.75],
    slidingCurves=slidingCurves,
    trackSurfaces=trackSurfaces,
)

Is this fine with you?


molecular function regulator

I have used "molecular function regulator" to describe the
activity of a GTPase with the input of the gene product that it
regulates.
However,  "molecular function regulator" is not allowed for direct annotation.
I can't think of any other term to use here. How have others annotated this?
thanks,
Val

@ValWood
Can you provide some examples?
I've never been very happy with annotating GTPases just to 'GTPase activity' but I'm not sure if we're talking about the same thing.
Thx.

Because the GTPase makes the GTPase inactive for signalling. The GTP-bound form is the active form. It doesn't make sense to attach the input to the GTPase activity?
This was the annotation I was looking at yesterday. It isn't in PomBAse because I forgot to reapprove the session, it will be back tomorrow.

PMID:21035342
Actually for the GTPase in this case, I could use 'protein kinase activator'?, it was bit1 that I was struggling with. Bit1 is part of the TORC2 signalling complex and acts on Tor1 (Tor2 is called Tor1 in pombe), in some way to allow the ryh1 GTPAse to regulate it.
So  the example I was  stuck with wasn't actually a GTPase.
(note that this was curated a while ago,I was just dealing with the logs which reports "molecular function regulator" as a "do not annotate" term).

@ValWood
Yes, this is a similar curation issue.
I've been working on a GO-CAM model of the EGFR/Ras/MAPK signaling pathway and the GTPase activity MF of let-60/Ras is unsatisfying.
http://noctua.geneontology.org/editor/graph/gomodel:56d1143000000176
My understanding is that for this pathway GTP-bound Ras binds and recruits Raf to the plasma membrane, so maybe that's what I should be trying to capture better with the MF.

Yes! This comes up a lot. Most recently in this thread yesterday !
https://github.com/geneontology/go-ontology/issues/22482
scroll down to
https://github.com/geneontology/go-ontology/issues/22482#issuecomment-987885825
and then continue.
I ma vying for a term which means
"modification-dependent recruitment activity" (could have any label).
Personally I avoid curating "localization" because although this is true,  it's a 'given' from the 'recruitment' activity and the localization isn't really the  process.
If I could I would do
"modification-dependent recruitment activity" has input XXX
part of "XXXX signalling"  occurs at "location"
the 'localization process is a bit of a red herring detracting from the actual pathway.
I have used "chromatin adaptor" in this way here:
https://www.pombase.org/term/GO:0140463
If you can say where the recruitment happens, and during which process, the  explicit localization annotation is somewhat unnecessary (everything process has a localization component at some level).

I think what I am trying to say is that the 'localization' isn't an 'active'  process like transport. Things move around until they are 'recruited'

@ValWood
So what I currently have in my GO-CAM model for this particular Ras MF is: protein-membrane adaptor activity.
GTP-bound LET-60 recruits LIN-45/Raf to the plasma membrane.  Part of LIN-45/Raf activation, based on mammalian studies, is binding to phosphatidylserine at the plasma membrane and this is facilitated by its interaction with LET-60/Ras.

That sounds right actually. People talk about the TOR GTPasses 'activating' the kinase, but this could well be through recruitment.

@ValWood What0s the action here?


glob on Windows

When running on Windows, the '/' will result in error, int() problem with string.
Workaround is to change '/' to double backslash '\\', specifically:

def get_class(img_path):
return int(img_path.split('\\')[-2])

root_dir = 'GTSRB_Final_Training_Images\\GTSRB\\Final_Training\\Images\\'

all_img_paths = glob.glob(os.path.join(root_dir, '* \\ *.ppm'))

That fixed the int() error.
Coding solution will be:

def get_class(img_path):
return int(img_path.split(os.sep)[-2])

as is, non issue.

all_img_paths = glob.glob(os.path.join(root_dir, os.path.join(' * ', ' *.ppm')))

Thanks.

it's differnce bewteen windows and linux in path


fix(obstacle_avoidance_planner): remove the option march=native

Description

Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@kosuke55 Could you approve the PR.
FYI: https://star4.slack.com/archives/G016VP1ERDH/p1677742229407709?thread_ts=1677738866.941809&cid=G016VP1ERDH


i636 connect solar collector and pvt with new radiation results

Changes:

New folder solar under the cea/technologies
New settings.py files to store solar technology-related inputs
Refactored solar_collector.py
Refactored photovoltaic_thermal.py
add solar collector specifications to supply_systems.xls database
add calc_worst_hour function to determine the worst hour according to the desired hours of solar window on winter solstice.

@shanshanhsieh how to test this?

@daren-thomas do you mean that we should add a test for the "dodo test"?

No, basically, what should I run and how to verify it now works?

@shanshanhsieh would you mind checking for conflicts?

@JIMENOFONSECA conflicts resolved!
However, all the parameters are migrated to the settings.py file now, meaning that the inputs we get from the arcgis interface are no longer arguments in the function calc_PV, 'calc_SC, calc_PVT`.
Please first take a look to see if the parameters in the settings file make sense, and then we could change the interface accordingly.

@shanshanhsieh overall the script runs but the results need to be checked. Please consider:

There should be a separate tab for PVT and the last should be filled in in databases/systems...
The efficiency of the PVT and SC seems to not match the nominal. for PVT, the % of the energy generated in relation to the radiation should be around 70%. but we get at the best around 20%. It would be good checking what is wrong with it.
The pumping consumption is rather low. it seems it only consumes 10W or 20W to move water around for a whole installation in an hour. It seems rather unusual. A hint: a Led bulb consumes 3W, a vacum cleaner consumes 1000W.

Hi @JIMENOFONSECA , thanks for catching this, I will take a look this week.

@shanshanhsieh could you please take a look at the interface? I just pushed my stab at it... The other two interfaces could be done in a similar matter.

@shanshanhsieh could you let me know when this is ready to test again?

@JIMENOFONSECA , i will let you know once the conflicts are resolved!

@shanshanhsieh updates?

@JIMENOFONSECA there are more conflicts now after merging the refactoring branch. I will go on fixing it now

@shanshanhsieh thnaks!, after that, qouedl you mind @martin-mosteiro to take a look at this PR?


Further Resolving of Message Calls Within Functions

Component
Heimdall (Core)
Describe the feature you would like
The proposed improvement aims to enhance the decompilation functionality by providing a more detailed analysis of message calls within functions, specifically regarding call, staticcall, and delegatecall operations.
For example, considering the contract at address 0x9f00c43700bc0000Ff91bE00841F8e04c0495000 from the "examples", the current decompilation result is as follows:
(bool success, bytes memory ret0) = address(arg1).staticcall(abi.encode(0xa567a8fa00000000000000000000000000000000000000000000000000000000));

In the verify function of this contract, there is a staticcall. However, the function signature 0xa567a8fa can be found in the database as curtaPlayer(). Therefore, an ideal decompilation result might look like this:
(bool success, bytes memory ret0) = address(arg1).curtaPlayer();

I'm uncertain whether this idea is complete, and I welcome any feedback or suggestions.
Your contribution to this impressive project is highly appreciated.

This is a great idea! I'll tackle if not taken soon :)

up next

@Jon-Becker Any updates on this?


SDX 55 or snapdragon 865 Files not support

it using different header vtnv and using file 0003010

SDX55 and all module 5G more encrypt in modem partition.


Intellisense shouldn't suggest Table.Columns if that's deprecated

We're deprecating Table.Columns  - where Column returns a single column from table.
PowerApps supports; but Intepreter does not.
Intellisense still suggests this; it shouldn't if this is deprecated.

Do we need a feature flag on this?

No on the flag need, intellisense behavior has no back-compat impact, and tbh, it'd be good to not suggest this in canvas either, it's very rare that users intend to use single column table syntax, and we've seen incidents in the past from this confusion.

IntellisenseHelper.AddSuggestionsForNamesInType looks like where this change would live.


Resolve spotify problems neatly

As was talked about, this resolves #20, I've worked on a solution to the Spotify issue, and have done so in a way that fits the design language of the theme fairly well.
This PR is a refined version of the concept in the issue, even if not much has changed, to actually replace the setup in the source CSS file to ensure there's no values overwriting or messing with one another.
If there were to be any improvements made to this PR it would be:
Try to cut down on !important uses, I was more just being lazy and left those as is 'just in case'
There's also a rudementary solution to the call menu preventing access to the spotify modal with this PR, but it's both a very rough solution, and could probably be done better (Raising the spotify modal to be above the call menu and be a horizontal side menu instead of a corner menu? Putting it at the bottom of the member list? Dunno)
All in all though, this at works fairly well, and should serve as a suitable fix to the issue.

Thanks!


Problem with file output plugin after upgrade to 4.4.0

As td-agent 4.4.0 is using fluentd 1.15.1, there is a change in file output plugin. The plugin now creates  /tmp/fluentd-lock-* directories. While td-agent is run without --daemon flag everything works great, the temp dir is created and the app works as expected. But when --daemon option is set then the temp directory is created but then it is removed and after that the name of that directory is passed to the child processes. That blocks them from functioning properly and makes them throw lots of No such file or directory errors.
Related strace output
06:31:37.550299 mkdir("/tmp/fluentd-lock-20220818-5506-14yftxz", 0700) = 0
06:31:37.579237 lstat("/tmp/fluentd-lock-20220818-5506-14yftxz", {st_dev=makedev(259, 3), st_ino=25167781, st_mode=S_IFDIR|0700, st_nlink=2, st_uid=995, st_gid=992, st_blksize=4096, st_blocks=0, st_size=6, st_atime=1660804297 /* 2022-08-18T06:31:37.549858004+0000 */, st_atime_nsec=549858004, st_mtime=1660804297 /* 2022-08-18T06:31:37.549858004+0000 */, st_mtime_nsec=549858004, st_ctime=1660804297 /* 2022-08-18T06:31:37.549858004+0000 */, st_ctime_nsec=549858004}) = 0
06:31:37.580017 openat(AT_FDCWD, "/tmp/fluentd-lock-20220818-5506-14yftxz", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 10
06:31:37.582363 rmdir("/tmp/fluentd-lock-20220818-5506-14yftxz") = 0
06:31:37.600303 execve("/opt/td-agent/bin/ruby", ["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"], ["FLUENT_PLUGIN=/etc/td-agent/plugin", "GEM_PATH=/opt/td-agent/lib/ruby/gems/2.7.0/", "GEM_HOME=/opt/td-agent/lib/ruby/gems/2.7.0/", "TD_AGENT_LOG_FILE=/var/log/td-agent/td-agent.log", "FLUENT_SOCKET=/var/run/td-agent/td-agent.sock", "LD_PRELOAD=/opt/td-agent/lib/libjemalloc.so", "FLUENT_CONF=/etc/td-agent/td-agent.conf", "XDG_SESSION_ID=c1593", "HOSTNAME=<redacted>", "SHELL=/bin/bash", "TERM=xterm-256color", "HISTSIZE=1000", "USER=td-agent", "LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:"..., "MAIL=/var/spool/mail/td-agent", "PATH=/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin", "PWD=/var/lib/td-agent", "LANG=en_US.UTF-8", "HISTCONTROL=ignoredups", "SHLVL=1", "HOME=/var/lib/td-agent", "LOGNAME=td-agent", "LESSOPEN=||/usr/bin/lesspipe.sh %s", "XDG_RUNTIME_DIR=/run/user/995", "_=/opt/td-agent/bin/fluentd", "FLUENTD_LOCK_DIR=/tmp/fluentd-lock-20220818-5506-14yftxz", "SERVERENGINE_SOCKETMANAGER_PATH=/tmp/SERVERENGINE_SOCKETMANAGER_2022-08-18T06:31:37Z_5695", "SERVERENGINE_WORKER_ID=0", "SERVERENGINE_SOCKETMANAGER_INTERNAL_TOKEN=7b50c9427382eebbf950bffb9d6b1809"]) = 0

6:33:06.155735 open("/tmp/fluentd-lock-20220818-5506-14yftxz/fluentd-_var_log_<redacted>_log.lock", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = -1 ENOENT (No such file or directory)
06:33:06.157334 write(5, "2022-08-18 06:33:06 +0000 [warn]: #3 failed to flush the buffer. retry_times=0 next_retry_time=2022-08-18 06:33:07 +0000 chunk=\"5e67e2765ec0c69c43f4717523cd22b1\" error_class=Errno::ENOENT error=\"No such file or directory @ rb_sysopen - /tmp/fluentd-lock-20220818-5506-14yftxz/fluentd-_var_log_<redacted>_log.lock\"\n", 383) = 383

The problem does not occur in ealier td-agent version as it using version of tdagent that does not create this directory.
What kind of information should I attach to make it easier for you to find the solution?

@fujimotos @daipom Could you take a look this?

Sure! I will.

This is my bug. I implemented the tempdir creation as follows:
Evidently se.run will exit early with --daemon.
https://github.com/fluent/fluentd/blob/master/lib/fluent/supervisor.rb#L874-L877
      Dir.mktmpdir("fluentd-lock-") do |lock_dir|
        ENV['FLUENTD_LOCK_DIR'] = lock_dir
        se.run
      end

So what we probably need is revert https://github.com/fluent/fluentd/pull/3808/commits/75ef92f787106de0a054174c2423a7f2fee00350,
which introduced the automatic cleanup based on the PR feedback.

And this is the fix: https://github.com/fluent/fluent-package-builder/issues/411
I confirmed it now works with --daemon with the following config:
<system>
  workers 3
</system>

<source>
  @type dummy
  tag test.log
</source>

<match test.**>
  @type file
  path test.log
  append true
  <buffer>
    @type memory
    flush_interval 3s
    flush_mode interval
  </buffer>
</match>

and by running Fluentd as follows:
$ fluentd --daemon test.pid --log test.log -c test.conf

@ashie  Can we go on to release td-agent v4.4.1? Since --daemon is included
in the default config, so I think we should make a point release for it.
https://github.com/fluent/fluent-package-builder/blob/b4d97b92bedcc92b03c25dfba5dd59c12d886e96/td-agent/templates/etc/systemd/td-agent.service.erb#L27

Yea, we should release it ASAP.
In addition, I want to include fluent-plugin-kafka's fix: https://github.com/fluent/fluent-plugin-kafka/pull/466

Fixed by fluent/fluentd#3864. Schedule to be released early next week.

Thanks, I did not expect this to happen so fast. You are awesome.


Insert JsDoc comment templates for additional nodes

In response to https://developercommunity.visualstudio.com/content/problem/138023/typescript-editor-jsdoc-comments-not-auto-generate.html.
Previously, we were not inserting jsDoc comment templates before interface declarations, method signatures, property signatures, and enums, whereas we were for class declarations and method declarations. This adds this functionality in.
Also, following a suggestion by @andy-ms, we no longer return undefined if it is inappropriate to insert a doc comment template. Instead, we return an empty template.

I feel like commenting

I got rid of the empty template approach. Turns out it makes VS behave weirdly, whereas undefined induces the intended behavior.

This function will now:

return a single-line JsDoc comment unless there are parameters to document, in which case it will fill them in as before.
autocomplete everywhere except for inside strings, comments, template literals, regex literals, and JSXText


breaking refactor: use much lighter parse+compile engine

This package now uses the very lightweight simple-markdown
library as a baseline markdown parser. On top of their core
functionality, a bunch of improvements have been added:

All HTML is now supported natively and dangerouslySetInnerHTML
is no longer used under any circumstances.

Package size is now just ~8.7kB gzipped, down from ~37kB.

Anything rendered via the Markdown component can be overridden now,
even the wrapper, footnote footer, etc.

I've been working on this release for a while and it took some time
to get all the regexes working to my satisfaction. There will probably
be bugs, so PRs are welcome to add regression tests as needed and
get things tightened up.
Thank you!

@sapegin would love for you to test this out if you have some spare time

Sounds great! Any API changes? I’ll try it in Styleguidist ;-)

@sapegin no API changes :) should be a drop-in replacement

@sapegin published under markdown-to-jsx@next for testing

I already see that you’ve inherited this weird behavior: https://github.com/Khan/simple-markdown/issues/40

@sapegin I actually fixed that: https://github.com/probablyup/markdown-to-jsx/pull/115/files#diff-168726dbe96b3ce427e7fedce31bb0bcR333
A <div> might still be added to the result though if there are multiple returns since you can't return an array in React <16.