[util/reggen] Replace  with  wrappers

The reggen-generated tables don't break on  because that doesn't
change rendering inside of a .  Wrap things that are supposed to be
distinct in s to force them onto new lines.
Signed-off-by: Garret Kelly<EMAIL_ADDRESS>

There was a little bit of yapf collateral damage in here too.

The reggen-generated tables don't break on
because that doesn't
change rendering inside of a . Wrap things that are supposed to be
distinct in
s to force them onto new lines.
Signed-off-by: Garret Kelly<EMAIL_ADDRESS>
Okay, so HTML in commit messages is maybe a bad idea, but this looks like:
Author: Garret Kelly<EMAIL_ADDRESS>Date:   Fri Nov 22 14:38:38 2019 -0500

    [util/reggen] Replace
 with
 wrappers

    The reggen-generated tables don't break on
 because that doesn't
    change rendering inside of a .  Wrap things that are supposed to be
    distinct in
s to force them onto new lines.

    Signed-off-by: Garret Kelly<EMAIL_ADDRESS>

Hi Garret,
Thanks for changing the Reg table design. It was my TODO  but postponed until I have parameterized reggen.
Anyway, I think, the header information (Register name and the description) is better to be outside of the table. So, 32bit table can be a separate table and the bit description also can be an another table. Wrapping a register width
 tag.
I think current out most
 is for layout not for printing the data structure. So it would be better to be div .
WDYT?

@eunchan: just to clarify, how do you feel about landing this as an incremental improvement? Or would you rather this PR be abandoned/reworked for the larger change you suggest?

@eunchan: just to clarify, how do you feel about landing this as an incremental improvement? Or would you rather this PR be abandoned/reworked for the larger change you suggest?

Thanks for the reminding comment. I am ok having it merged.

@eunchan: just to clarify, how do you feel about landing this as an incremental improvement? Or would you rather this PR be abandoned/reworked for the larger change you suggest?

Thanks for the reminding comment. I am ok having it merged.

If you approve the change I'll land it now.


METRON-1545: Upgrade Spring and Spring Boot

Contributor Comments
This PR upgrades Spring to version 5.0.5.RELEASE and Spring Boot to version 2.0.1.RELEASE.  A summary of the changes included in this PR (all related to metron-rest):

Incremented Spring maven dependency versions
Added documentation for creating authentication tables
Incremented Jackson version to match the version Spring depends on
Added an explicit dependency on JsonPath to avoid conflicts with the provided metron-parsers uber jar
Disabled auto-configuration for Gson and Kafka to avoid version conflict problems that result from metron-parsers and metron-elasticsearch including these classes on the classpath.
Disabled automatic logging configuration to avoid version conflict problems that result from metron-elasticsearch including log4j2 classes on the classpath
Updated the JpaConfiguration class to match newer constructor
Configured a NoOpPasswordEncoder to satisfy the password encoder requirements introduced in newer versions of Spring
Fixed a bug in the test profile related to the Spring banner
Added the default application.yml as a configuration file since it is no longer added by default when the --spring.config.location option is used
Updated depedencies_with_url.csv to include all of the recent versions introduced by the upgrade

I tested this in full dev by manually interacting with all the REST endpoints including:

creating and editing alert UI settings
creating, updating, and deleting the various configs (global, parser, enrichment, indexing)
listing, getting and validating grok statements
creating, reading, updating, and deleting HDFS files
creating, listing, deleting Kakfa topics as well as consuming and producing messages
performing searches, lookups, groupings, and updates with our DAO endpoints
creating and editing metaalerts
stopping, starting and getting status of Storm topologies

I also went through the exercise of configuring a MySQL database for authentication.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

I'm +1 by inspection.  Thanks for the contribution!


Reduce resharding impact by redirecting data to new shards

Signed-off-by: Harkishen-Singh<EMAIL_ADDRESS>
Fixes: #7230
This PR adds support to reduce resharding impact by spinning up new shards at the time of resharding if there are existing shards. The new shards are fed with data from existing shards while blocking the incoming samples. Once this is done, the incoming samples are consumed. This keeps the samples in order. After this, the old shards are replaced with these new shards. This reduces the wait time of the samples due to slow shards, as mentioned in the related issue.

cc @cstyan @csmarchbanks

Thank you for the PR!
We are aware of this PR and will review it when we have time, probably Monday or Tuesday. Please avoid re-pinging people after only a couple days, especially when those days are a weekend.

On Oct 18, 2020, at 8:00 AM, Harkishen Singh<EMAIL_ADDRESS>wrote:

cc @cstyan @csmarchbanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Ahh, sorry. I didn't aim to re-ping, I felt like mentioning in PR description is not useful, so removed from there and mentioned in the comment.

It looks like runShard() would currently continue to pick up samples from its queue and try to send them rather than put them into a queue buffer, or a new queue. Then, only the very last request would be moved to the new queues in the current implementation?

As soon as the queue channel is closed, the pending samples in the shard will be sent to the buffer. It doesn't matter if its the last request or not, the pending samples are sent to buffers than waiting for them to be flushed (which is responsible for a slow shard).
Do we want also want to stop the existing sendSample (which is in process of sending) that may be slow? (assuming the answer to be no)

The issue is that using the default queue config, the queue channel might have 2,500 samples in it. All of those would be processed, and 4-5 requests would be sent, before the close is processed, at which point up to 500 samples would be moved over. Using those numbers, resharding would speed up by about 20%, but I think we can do better :).
I have seen queue configs where 100,000 samples might be buffered in the queue channel., which you can imagine may take quite a long time to successfully send to the remote before seeing that the channel was closed.
Does that make sense?

@csmarchbanks as far as i understand, as soon as the a queue channel is closed, the following iteration in runShard will make ok as false and hence shift the pending samples to the buffer and then to the new shards. The only way it can be slow is if the queue channel is closed but at that time, sendSamples() inside if nPending >= max is running blockingly, meaning we have to wait for that send to finish, which can make it slow, and hence i asked if we want to stop the currently running sendSamples().
Maybe, if you could explain more on how All of those would be processed, and 4-5 requests would be sent is the case. That would be very helpful.

Ahh, I think I might see where the misunderstanding is. When a channel is closed, ok only will be false when all of the samples from that channel are also drained. Since that channel can have thousands to hundreds of thousands of samples in it, all of those would be processed by the normal batch and send logic before the new logic is seen when ok comes back as false.
Here is a go playground example that shows the behavior in a simpler manner: https://play.golang.org/p/lmZ2po7P3OH. As can be seen, if we have 2300 samples in the queue when it is closed, we will send 4 requests of 500 samples before moving the final 300 samples to the new queue.
Did this explanation help at all?

Did this explanation help at all?

Yes, very much. Thank you for the awesome explanation. I will update the code soon.

With the recent code pushed, I feel that stopOldShards() is not required any more. This is because stopOldShards() is called only during resharding and when a reshard signal is sent, it goes into the t.reshard case, and then accepts incoming samples and redirecets to new queues, ending the old shards/go_routines.
What do other think on this?
I will try to make the code more cleaner. Sorry for the inconveniences :sweat_smile:

If the above sounds good, I would recommend two pull requests, one to refactor shards to be idempotent and replaced during reshard, and a second to implement the transferTo logic.

mmm, they both are part of a single feature, then why for two PRs? Also, as far as I have understood, I think for the implementation, they both are required.

@csmarchbanks I have updated with the suggestions. The renaming suggestions were helpful and I think the code looks more understandable now. There are some changes like the enqueue() now does not need to send false for blocking, instead is being handled by queuemanager and related changes that I will do after listening your thoughts on the updated implementation.
Thank you.

What would happen in the current implementation if a reshard is attempted but the remote endpoint is down? You would have shards stuck in sendSamples forever without a context cancellation I think?

For context cancellation or hard shutdown, should we use a TTL or limit a particular number of times a non-decreasing pending samples happens?

What would happen in the current implementation if a reshard is attempted but the remote endpoint is down? You would have shards stuck in sendSamples forever without a context cancellation I think?

The current implementation (with recent push) is such that after each shard is formed, we check if send sendSample() fails to send any sample, which is beyond the flush deadline, then we do a hardshutdown like earlier cases.

I am wondering it it is worth complicating this code now, since there is a consensus to make remote_write transactional, so all this code would change anyway.
https://docs.google.com/document/d/1vhXKpCNY0k2cbm0g10uM2msXoMoH8CTwrg_dyqsFUKo/edit

Hi @Harkishen-Singh, I am still on vacation this week, I plan to take another look at this sometime next week!

Hi @Harkishen-Singh, I am still on vacation this week, I plan to take another look at this sometime next week!

Hey @csmarchbanks. I have updated the code with a more cleaner approach in lines with the suggestions. Thank you.

Hey @csmarchbanks. I have updated the code with a more cleaner approach in lines with the suggestions. Thank you.

@csmarchbanks could you take a look at this PR? Thanks!
(PS: @Harkishen-Singh the tests are failing)

PS: This isn't ready for review yet. I think the new Github features automatically asks for reviews from the maintainers.

Yes, unless your mark the PR as draft.

@csmarchbanks @cstyan can you take another look at this, please?

(closed by mistake)

Rebased and updated.

Looks like there's now a data race and also maybe a routine that never ends somewhere in the resharding procedure, take a look at the failing tests.

Ah sorry, I missed a lock. It should be fine now.

Is this likely to be resurrected?  I see the original issue #7230 is still open.

I personally don't have time to take this over. @Harkishen-Singh do you plan/have any interest in picking this up again or should we mark it as open for contributors?

We have looked at this pull request during our bug scrub.
Given the lack of response, I have marked the issue as help wanted, and we have decided to close this PR.
Thank you for your contribution.


Decoupling: Use camel casing for expected API response from story editor API callbacks

Context
As decided in https://github.com/google/web-stories-wp/pull/9409 we want to use camel casing for the expected API response from all API callbacks. This PR updates wp-story-editor and story-editor for camel casing.
Summary

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

[x] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

Reviews
Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

[ ] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[x] I have tested this code to the best of my abilities
[x] I have verified accessibility to the best of my abilities (docs)
[x] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[x] This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
[x] I have added documentation where necessary
[x] I have added a matching Type: XYZ label to the PR

Partially addresses #456

Note that I had to move snakeToCamelCase to design-system ( in https://github.com/google/web-stories-wp/pull/9683/commits/a5c16c1f13bc6a9c54f40993a67a6970a794a797 ) because it was required in story-editor inside taxonomies.js
here
https://github.com/google/web-stories-wp/blob/a5c16c1f13bc6a9c54f40993a67a6970a794a797/packages/story-editor/src/components/panels/document/taxonomies/taxonomies.js#L73-L74
and here
https://github.com/google/web-stories-wp/blob/a5c16c1f13bc6a9c54f40993a67a6970a794a797/packages/story-editor/src/components/panels/document/taxonomies/taxonomies.js#L54-L55

Waiting for percy e2e tests to pass.

codecov is failing with a random negative percentage. Added some more coverage in https://github.com/google/web-stories-wp/pull/9683/commits/30159d9f0392de717811eb8102499699f5a53208 and it started showing even more negative difference

codecov is failing with a random negative percentage. Added some more coverage in 30159d9 and it started showing even more negative difference

I'd take it with a grain of salt :-)

As discussed yesterday, writing down some of my thoughts.
First of all, this is looking really good so far!
There's now camelCase in most places already, including getResourceFromAttachment, which is good. That means all image sizes keys are properly camelCased for example.
Also, I took another look at getCustomPageTemplates as well and now I better understand why you added the camelCasing there.
A few general observations

Anything in story_data is "versioned". If our data structure changes, we add a migration in packages/migration for it.
When loading a story in the editor, we run that migration (in useLoadStory)
When editing an existing story, story_data will theoretically still contain snake_case image sizes.
a. However, in this PR, the snakeToCamelCaseObjectKeys call in transformGetStoryResponse actually recursively updates all properties.
b. That call is a bit greedy because of that recursiveness. I don't think we should do that. We'd only want to camelCase the top level object keys. It should not modify anything inside story_data.
To update story_data and camelCase image sizes for existing stories, we should write a migration for that.
When loading existing custom page templates, story_data will theoretically still contain snake_case image sizes.
a. That said, I just noticed that custom page templates are not currently run through migration ⚠  That could lead to unexpected behavior when loading a super old page template.
b. To fix this, I think we need to call migrate() in loadTemplates within PageTemplatesPane or somewhere near there.
That said... do we even need the image sizes names? It doesn't seem like we do. So we could just simplify this by making resource.sizes an array instead (and adding a migration for it)

I don‘t think we should make design-system a dependency of migrations.
We don‘t need that for just updating image sizes names.

I don‘t think we should make design-system a dependency of migrations.

Yes, I realized that, was just working to revert that change.

Also, curious to hear your thoughts on making the image sizes a simple array instead.

It's being used at a couple of places.
https://github.com/google/web-stories-wp/blob/25d43b402e39f46de36ec4636a1893f8749b932e/packages/story-editor/src/app/highlights/quickActions/useQuickActions.js#L163-L166
https://github.com/google/web-stories-wp/blob/25d43b402e39f46de36ec4636a1893f8749b932e/packages/story-editor/src/components/library/panes/media/local/mediaPane.js#L222-L225
Why do we want to change that to an array?

Since PRs are constantly merging in main, one new source_url got added in the codebase as snake-case, updated in https://github.com/google/web-stories-wp/pull/9683/commits/206427ce76b15a249234edda17c391943dafcb0a, but going to review the entire change again as there may be more new instances like this.

Also, curious to hear your thoughts on making the image sizes a simple array instead.

It's being used at a couple of places.
https://github.com/google/web-stories-wp/blob/25d43b402e39f46de36ec4636a1893f8749b932e/packages/story-editor/src/app/highlights/quickActions/useQuickActions.js#L163-L166
https://github.com/google/web-stories-wp/blob/25d43b402e39f46de36ec4636a1893f8749b932e/packages/story-editor/src/components/library/panes/media/local/mediaPane.js#L222-L225
Why do we want to change that to an array?

So, I don't necessarily want to do it, but now is a great opportunity to rethink and refactor how things work in this area.
Some thoughts:

Resource sizes like medium  or web-stories-thumbnail are quite WP-specific. It's odd that these are now referenced as webStoriesThumbnail.
The only reason why we reference image sizes by name is convenience, to do things like resource.sizes?.medium?.source_url || resource.src so we get a reasonably sized image for insertion
Instead of accessing image sizes like that, we could just as well have some getSomeReasonablySizedImageSizeForInsertion(resource) function or whatever to get this image size.\
If we have such a function we don't need image sizes names. They could just be an array and we don't need a migration for them.

WDYT?
Related:

camelCasing things like capabilities and taxonomy names takes some time getting used to. For example capabilities[snakeToCamelCase(assign-${taxonomy?.restBase})] is not very readable and also a bit error prone, as the taxonomy is web-story-tag, but the capability is assignWebStoryTag.
Not sure how that could be improved though

Instead of accessing image sizes like that, we could just as well have some getSomeReasonablySizedImageSizeForInsertion(resource) function or whatever to get this image size.\

Do you mean getSomeReasonablySizedImageSizeForInsertion utility function would be able to find the closest image size? I think that would unnecessarily increase complexity. Would it be easier that we document what sizes keys ( with generic names ) we are expecting? Do you see any problem with it? For example:
'sizes': {
    'small': {
      'file': '',
      'width': 100,
      'height': 150,
      'mimeType': 'image/jpeg',
      'sourceUrl': '',
    },
    'medium': {
      'file': '',
      'width': 300,
      'height': 150,
      'mimeType': 'image/jpeg',
      'sourceUrl': '',
    },
    'large': {
      'file': '',
      'width': 1024,
      'height': 576,
      'mimeType': 'image/jpeg',
      'sourceUrl': '',
    },

camelCasing things like capabilities and taxonomy names takes some time getting used to. For example capabilities[snakeToCamelCase(assign-${taxonomy?.restBase})] is not very readable and also a bit error prone, as the taxonomy is web-story-tag, but the capability is assignWebStoryTag.

Agree it's confusing, will work on it in some other PR.

Converting this to draft as I feel like things like image sizes and caps need a bit more thinking. Let's sync about this.

I think that would unnecessarily increase complexity.
Would it be easier that we document what sizes keys ( with generic names ) we are expecting?

Yeah true. The advantage would be that the image sizes names don't matter.
Right now the only image size we use by name seems to be medium. And that's just because we don't want to use resource.src (the full size) when inserting on the canvas for performance reasons (don't wanna display a 4K picture there).
getResourceFromMedia3p / getImageUrls mentions a bunch more image sizes names, but we don't reference the sizes by names, ever, as far as I can tell.
Aside: We register a web-stories-thumbnail image size in WP but all the JS code seems to reference web_stories_thumbnail 🤔

Agree it's confusing, will work on it in some other PR.

Curious, what did you have in mind here?
I hesitate a bit to camelCase taxonomy names. If I have 1 taxonomy "web-story-tag" and one "webStoryTag", then there would be conflicts.
The same goes for image sizes names too.

Aside: We register a web-stories-thumbnail image size in WP but all the JS code seems to reference
web_stories_thumbnail 🤔

I think both web-stories-thumbnail and web_stories_thumbnail are not being used. They were probably used before. Found the PR where web_stories_thumbnail was added https://github.com/google/web-stories-wp/pull/3089

Do you have something like getSmallestUrlForWidth getSmallestUrlForWidth.js#L22-L30 in mind for getting the closest image for getSomeReasonablySizedImageSizeForInsertion ?

Curious, what did you have in mind here?
I hesitate a bit to camelCase taxonomy names. If I have 1 taxonomy "web-story-tag" and one "webStoryTag", then there would be conflicts.

I think the easiest is to add an ignore param in the snakeToCamelCaseObjectKeys,  which will be exempted in camel casing like I just did for story_data in https://github.com/google/web-stories-wp/pull/9683/commits/ab3220b196a85a8cd3bc95c6ee98e54283667721
Similarily we can use capabilities.
snakeToCamelCaseObjectKeys(post, ['story_data', 'capabilities']);

The only problem is undoing the changes made for capabilities.

Aside: We changed these many snake cases to camel cases, and considering so many changes that merged in main, I am not feeling very confident with this PR, I think I will create a fresh PR after we are done with our discussion here.
- permalink_template
- story_data
- style_presets
- preview_link
- edit_link
- embed_post_link
- assign-author
- assign-web_story_tag
- assign-web_story_category
- create-web_story_tag
- create-web_story_category
- lock_user
- featured_media
- publisher_logo
- date_gmt
- modified_gmt
- source_url
- mime_type
- **Remove ping_status and comment_status**
- web_stories_is_muted
- featured_media_src
- alt_text
- media_details
- media_type
- length_formatted
- file_sizes
- file_name
- manage_terms
- edit_terms
- delete_terms
- assign_terms
- add_new_item
- items_list
- separate_items_with_commas
- most_used
- singular_name
- search_items
- all_items
- parent_item
- parent_item_colon
- edit_item
- view_item
- update_item
- add_new_item
- new_item_name
- separate_items_with_commas
- add_or_remove_items
- choose_from_most_used
- not_found
- no_terms
- filter_by_item
- items_list_navigation
- items_list
- most_used
- back_to_items
- item_link
- item_link_description
- menu_name
- name_admin_bar
- publicly_queryable
- show_admin_column
- show_in_nav_menus
- show_in_quick_edit
- show_ui
- source_url
- web-stories-poster-portrait
- web-stories-publisher-logo
- web-stories-thumbnail
- web_stories_media_optimization
- web_stories_onboarding
- web_stories_tracking_optin
- web_stories_muted_id
- web_stories_optimized_id
- web_stories_is_poster
- web_stories_poster_id
- web_stories_trim_data
- web_stories_is_muted

@swissspidy As discussed on Tuesday, could you please add some more thoughts on media sizes?

Are we planning to keep the image sizes?

I'm still tinkering with this but it feels more reasonable to leave dynamic object keys like image sizes and taxonomy names (which are both out of our control) untouched.

Closing in favour of https://github.com/google/web-stories-wp/pull/9906


Firestore - Listen to real-time updates

Is there any implementation for getting real-time updates from a Firestore collection? If not, any advice accomplishing this in a KMP iOS/Android project?

Read these:
https://firebase.google.com/docs/firestore/query-data/listen?authuser=0#listen_to_multiple_documents_in_a_collection
https://github.com/GitLiveApp/firebase-kotlin-sdk?tab=readme-ov-file#flows


templates 给暴露出来可好?, 系统提醒邮件不能修改内容了.

sorry, I not see English.

Please speak English, this is the language everybody of us can speak and write.
Please ask questions or config/deploy problems on our Gitter channel: https://gitter.im/go-gitea/gitea
Please take a moment to search that an issue doesn't already exist.
Please give all relevant information below for bug reports, incomplete details will be handled as an invalid report.

You MUST delete the content above including this line before posting, otherwise your issue will be invalid.

Gitea version (or commit ref):
Git version:
Operating system:
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[ ] Not relevant

Log gist:

Description
...

Hi, first of all, Please write English so that your issue could be read by others.
About your problem, you could put your templates on custom/ then it will replace the default templates.

@lunny Maybe we can open a issues-only repo for questions in chinese? and link chinese README to that :)

@lunny duplicate of #529 ?

I think yes. So I close this one. Please go to #529 to continue.


mcu/nordic: Update NRFX to version 3.8.0

https://github.com/NordicSemiconductor/nrfx/releases/tag/v3.8.0

#AutoPTS run mynewt GAP/CON

#AutoPTS run mynewt GAP/CON

Scheduled PR https://github.com/apache/mynewt-core/pull/3328#issuecomment-2444366407, board: nrf52, estimated start time: 15:05:50, test case count: 32, estimated duration: 0:12:53Test cases to be runGAP/CONN/NCON/BV-01-CGAP/CONN/NCON/BV-02-CGAP/CONN/NCON/BV-03-CGAP/CONN/DCON/BV-01-CGAP/CONN/UCON/BV-01-CGAP/CONN/UCON/BV-02-CGAP/CONN/UCON/BV-03-CGAP/CONN/UCON/BV-06-CGAP/CONN/ACEP/BV-01-CGAP/CONN/ACEP/BV-03-CGAP/CONN/ACEP/BV-04-CGAP/CONN/GCEP/BV-01-CGAP/CONN/GCEP/BV-02-CGAP/CONN/GCEP/BV-05-CGAP/CONN/GCEP/BV-06-CGAP/CONN/SCEP/BV-01-CGAP/CONN/SCEP/BV-03-CGAP/CONN/DCEP/BV-01-CGAP/CONN/DCEP/BV-03-CGAP/CONN/DCEP/BV-05-CGAP/CONN/DCEP/BV-06-CGAP/CONN/CPUP/BV-01-CGAP/CONN/CPUP/BV-02-CGAP/CONN/CPUP/BV-03-CGAP/CONN/CPUP/BV-04-CGAP/CONN/CPUP/BV-05-CGAP/CONN/CPUP/BV-06-CGAP/CONN/CPUP/BV-08-CGAP/CONN/CPUP/BV-10-CGAP/CONN/TERM/BV-01-CGAP/CONN/PRDA/BV-01-CGAP/CONN/PRDA/BV-02-C

AutoPTS Bot results:
No failed test found.
Successful tests (32)GAP     GAP/CONN/ACEP/BV-01-C           PASSGAP     GAP/CONN/ACEP/BV-03-C           PASSGAP     GAP/CONN/ACEP/BV-04-C           PASSGAP     GAP/CONN/CPUP/BV-01-C           PASSGAP     GAP/CONN/CPUP/BV-02-C           PASSGAP     GAP/CONN/CPUP/BV-03-C           PASSGAP     GAP/CONN/CPUP/BV-04-C           PASSGAP     GAP/CONN/CPUP/BV-05-C           PASSGAP     GAP/CONN/CPUP/BV-06-C           PASSGAP     GAP/CONN/CPUP/BV-08-C           PASSGAP     GAP/CONN/CPUP/BV-10-C           PASSGAP     GAP/CONN/DCEP/BV-01-C           PASSGAP     GAP/CONN/DCEP/BV-03-C           PASSGAP     GAP/CONN/DCEP/BV-05-C           PASSGAP     GAP/CONN/DCEP/BV-06-C           PASSGAP     GAP/CONN/DCON/BV-01-C           PASSGAP     GAP/CONN/GCEP/BV-01-C           PASSGAP     GAP/CONN/GCEP/BV-02-C           PASSGAP     GAP/CONN/GCEP/BV-05-C           PASS (2)GAP     GAP/CONN/GCEP/BV-06-C           PASSGAP     GAP/CONN/NCON/BV-01-C           PASSGAP     GAP/CONN/NCON/BV-02-C           PASSGAP     GAP/CONN/NCON/BV-03-C           PASSGAP     GAP/CONN/PRDA/BV-01-C           PASSGAP     GAP/CONN/PRDA/BV-02-C           PASSGAP     GAP/CONN/SCEP/BV-01-C           PASSGAP     GAP/CONN/SCEP/BV-03-C           PASSGAP     GAP/CONN/TERM/BV-01-C           PASSGAP     GAP/CONN/UCON/BV-01-C           PASSGAP     GAP/CONN/UCON/BV-02-C           PASSGAP     GAP/CONN/UCON/BV-03-C           PASSGAP     GAP/CONN/UCON/BV-06-C           PASS

#AutoPTS run mynewt nrf53 SM

Scheduled PR https://github.com/apache/mynewt-core/pull/3328#issuecomment-2444560719, board: nrf53, estimated start time: 16:16:07, test case count: 61, estimated duration: 0:27:30Test cases to be runSM/CEN/PROT/BV-01-CSM/CEN/JW/BV-05-CSM/CEN/JW/BI-04-CSM/CEN/JW/BI-01-CSM/CEN/PKE/BV-01-CSM/CEN/PKE/BV-04-CSM/CEN/PKE/BI-01-CSM/CEN/PKE/BI-02-CSM/CEN/OOB/BV-01-CSM/CEN/OOB/BV-03-CSM/CEN/OOB/BV-09-CSM/CEN/OOB/BI-01-CSM/CEN/EKS/BV-01-CSM/CEN/EKS/BI-01-CSM/CEN/KDU/BV-05-CSM/CEN/KDU/BV-06-CSM/CEN/KDU/BV-10-CSM/CEN/PIS/BV-02-CSM/CEN/PIS/BV-03-CSM/CEN/SCJW/BV-01-CSM/CEN/SCJW/BV-04-CSM/CEN/SCJW/BI-01-CSM/CEN/SCPK/BV-01-CSM/CEN/SCPK/BV-04-CSM/CEN/SCPK/BI-01-CSM/CEN/SCPK/BI-02-CSM/CEN/SCOB/BV-01-CSM/CEN/SCOB/BV-04-CSM/CEN/SCOB/BI-01-CSM/CEN/SCOB/BI-04-CSM/PER/PROT/BV-02-CSM/PER/JW/BV-02-CSM/PER/JW/BI-03-CSM/PER/JW/BI-02-CSM/PER/PKE/BV-02-CSM/PER/PKE/BV-05-CSM/PER/PKE/BI-03-CSM/PER/OOB/BV-02-CSM/PER/OOB/BV-04-CSM/PER/OOB/BV-10-CSM/PER/OOB/BI-02-CSM/PER/EKS/BV-02-CSM/PER/EKS/BI-02-CSM/PER/KDU/BV-01-CSM/PER/KDU/BV-02-CSM/PER/KDU/BV-07-CSM/PER/KDU/BV-08-CSM/PER/KDU/BI-04-CSM/PER/PIS/BV-01-CSM/PER/PIS/BV-02-CSM/PER/SCJW/BV-02-CSM/PER/SCJW/BV-03-CSM/PER/SCJW/BI-02-CSM/PER/SCPK/BV-02-CSM/PER/SCPK/BV-03-CSM/PER/SCPK/BI-03-CSM/PER/SCPK/BI-04-CSM/PER/SCOB/BV-02-CSM/PER/SCOB/BV-03-CSM/PER/SCOB/BI-02-CSM/PER/SCOB/BI-03-C

AutoPTS Bot results:
No failed test found.


Replace scipy.fft calls with FFTW ones

Here is a comment on how I installed firedrake with threading in order to be able to run fftw that is provided by mpi4py:
INSTALLATION OF MPI4PY-FFT ON FIREDRAKE:
i) Install firedrake using:
curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install
PETSC_CONFIGURE_OPTIONS="--download-fftw=1 --download-fftw-configure-arguments='--enable-threads --enable-shared'" ~/firedrake/bin/python3 firedrake-install --venv-name firedrake-fftw1
ii) Install mpi4py-fft:
start venv
git clone git@github.com:mpi4py/mpi4py-fft.git (does not matter where to download this)
cd mpi4py-fft
export FFTW_LIBRARY_DIR=~/Programmes/firedrake-fftw1/src/petsc/default/lib/
export FFTW_INCLUDE_DIR=~/Programmes/firedrake-fftw1/src/petsc/default/include/
python -m pip install -e .
iii) Not clear if needed: linking libraries with ldd

The FFT takes up a very small proportion of the runtime with scipy.fft. Updating to FFTW is not a priority at the moment. See runtime profile for scalar advection below.


Extension doesn't start when code is started with inspect-extensions flag

When VSCode is started with code --inspect-extensions=9993 or in the extension development host mode, the extension doesn't load and shows this error:

This is due to some local issue in my dev machine. Running with the debugOptions set to {} doesn't cause this issue.


APPS-941 docker and circleci implementation

Includes a ruby update from ruby 2.1.6 to 2.3.7 because its better that way.
Also updated the rails_config gem to config.
Implemented rubocop and changes.

@jaredk-beta I may need you to help me symlink/mount /public with /srv/static/apps.biola.edu/idphotos

Looks like you're targeting 2.3.7 but your Dockerfile and circleci config are using 2.2.2. I assume that's not intentional?


get('hostname') returns the identifier of the host, not the URL

Deployer version:  6.8.0
Deployment OS: GH Actions Ubuntu 20.04.2 LTS

In my config I have:
host('staging')
	->hostname($stgServerAddress)
	->user($stgServerUser)
	->port($stgPort)
	->stage('staging')
	->roles('app')
	->set('branch', 'staging')
	->set('deploy_path', "~/www/{$stgDeployPath}")
	->addSshOption('UserKnownHostsFile', '/dev/null')
	->addSshOption('StrictHostKeyChecking', 'no');

With all the variables defined above in the file. In my task where I execute some scripts on the deployment server, I use $host = get('hostname'); to retrieve the URL.
However, that returns the staging string instead of the URL (which is set as host identifier). Am I missing something?

Yes, it’s confusing and was fixed in v7.
In v6 user real_hostname.

Oh forgot about that! The GH action uses 6.8, since 7 is in beta. Will write this down for future update. Thanks!


Add support for AES management keys

Once https://github.com/iqlusioninc/yubikey.rs/issues/330 is resolved, we should start using PIN-protected AES management keys for YubiKeys that support them. We should also migrate YubiKeys that we previously configured to use a PIN-protected TDES management key, if AES is supported.

We want to ensure that a YubiKey set up by age-plugin-yubikey is usable with yubikey-agent, so we use the same management setup (PIN-protected management key, PUK set to PIN) as yubikey-agent. We therefore need to synchronise on AES management key usage, which means we also need to block on https://github.com/go-piv/piv-go/issues/109.

It appears https://github.com/go-piv/piv-go/issues/109 has now been resolved and should hopefully no longer be blocking.
What all needs to be done for AES support?

I now have a new YubiKey with firmware 5.7.1, and it appears to come with AES as the default management key algorithm. The error message I previously added does trigger correctly (telling the user to switch to TDES).
@PriceHiller yes this is blocked on support in the yubikey crate.


Use catchError to mark thrown error as handled

New Pull Request Checklist

[x] I have read the Documentation
[x] I have searched for a similar pull request in the project and found none
[x] I have updated this branch with the latest master to avoid conflicts (via merge from master or rebase)
[ ] I have added the required tests to prove the fix/feature I am adding
[ ] I have updated the documentation (if necessary)
[ ] I have run the tests and they pass

This merge request fixes / refers to the following issues:
Debugger will halt execution while debugging an app that uses Dio to generate requests. This is really an issue with Dart that is known and is somewhat being worked on and this is a quickfix for everyone using Dio with debugger.
Pull Request Description
Use catchError method to let the debugger know that the thrown DioError will be caught so it doesn't hold execution while debugging.
Fix for flutterchina#1306

Please try with 4.0.2.  changes as follows:
    future = future.then(_requestInterceptorWrapper((
      RequestOptions reqOpt,
      RequestInterceptorHandler handler,
    ) {
      requestOptions = reqOpt;
      _dispatchRequest(reqOpt)
          .then((value) => handler.resolve(value, true))
          .catchError((e) {
        handler.reject(e as DioError, true);
      });
    }));


feat: add oss if-none-match support

Fix https://github.com/apache/incubator-opendal/issues/1977

The only thing left here is to make our CI happy! Please try running cargo fmt to make code foramtted.


Overhaul tenant state and  remote storage handling.

This is a massive patch. I wish to find ways to split it up, for
easier review, but I'm not sure if that's feasible. I'm all ears for
suggestions.
The refactorings are in three main areas: Tenant state handling,
remote storage synchronization i.e. S3 upload/download, and WAL
receiver.

Tenant state

Note: There is a one-to-one mapping between a tenant, and a
LayeredRepository object. I use those terms somewhat interchangeably.
Remove the concepts of Loaded and Unloaded timelines. Merge all the
state and functions from tenant_mgr.rs into LayeredRepository. What
remains in tenant_mgr.rs is a simple hashtable from tenant_id to
LayeredRepository.
There is still the concept of "tenant state". A tenant can be in one
of the following states: Loading, Attaching, Active, Stopping, Broken.
But regardless of the state, there is one LayeredRepository object for
the tenant, and the "Loaded" and "Unloaded" wrappers are gone.  See
TenantState enum for details on the states and state transitions.
Before performing an operation like GetPage@LSN on a timeline, you
should check the tenant state. If it's not Active, you need to wait
for it to become active, or return an error. There's a mechanism to
wait for the state to change. This is somewhat similar to how
GetPage@LSN waits for the WAL to arrive.
Previously, a timeline would be marked as "broken", if we failed to
load it on system startup. That is replaced by Broken state in tenant,
instead. That's not an improvement: instead of making one timeline
unusable, the whole tenant is unusable. But I think this is better in
the long run. When we implement on-demand download and will keep track
of state for individual layer files, we can also have a Broken state
for an individual layer file. With that, the whole tenant would only
be marked as Broken if we fail to load a metadata file, and that
should be rare, and it seems like the safer choice.

Remote storage

The storage_sync.rs code has been heavily refactored. The separate
thread with "sync loop" is gone. Instead, there's a per-timeline
upload queue. An async task is lanched for each upload.
IMHO this refactoring simplifies the code significantly. I believe the
old code structure emerged the way it was because previously the code
dealt with packaging files into "archives".

WAL receiver

The WAL receiver for each timeline is now more tightly integrated with
the LayeredTimeline struct. There's a reference to the timeline's WAL
receiver's "handle" in LayeredTimeline struct itself.
TODOs:

On pageserver startup, if there are some files missing locally, they
are not downloaded from s3. It is assumed that the pageserver
local state is always ahead or at he same state as S3. How did this
work on 'main'?

There are some broken tests. Some of the failures are related to
previous point.

killing async tasks is not implemented properly. There are now more
per-tenant and per-timeline async tasks that run in the background.
We should implement https://github.com/neondatabase/neon/issues/2137
to fix that.

This is what I've been working on lately. It's massive, and
I'm going to spend more time fixing comments and the various FIXMEs, but given the size, we better start discussing and reviewing this already. Did I break something that's not already listed in FIXMEs or the comments already? Comments on the overall code structure and the direction this is going? Ideas on how to split this up?

@SomeoneToIgnore what do you think of the rewrite in storage_sync.rs ? Does it make sense? Did I lose some functionality that we still need?
A couple of things I was not sure about the current implementation in main:

At pageserver startup, if some files are missing locally, but are present in S3, were they downloaded? With this PR, they are not. They probably should be. Similarly, if the metadata file in S3 has a higher disk_consistent_lsn than the file on local disk, what happens?

On 'main', if an upload keeps failing, we eventually give up:

    if current_attempt >= max_sync_errors {
        error!(
            "Aborting task that failed {current_attempt} times, exceeding retries threshold of {max_sync_errors}",
        );
        return ControlFlow::Break(sync_data);
    }

That seems bad. Later, we might remove the file locally too, and have no copies of it anywhere. Shouldn't we keep retrying forever? And alert humans if it keeps failing.

Do I correctly understand that our final goal is to provide on-demand load from S3 of individual layer files?
Such lazy loading can speed-up tenant activation and migration to different page server.
If so, I slightly confused by refactoring done in this PR.
Right now we have notion of loaded/unloaded timelines. Implementation of it IMHO looks really ugly, but at least it allows lazy loading of timelines. Now it is replaced with active/inactive tenants. So instead of providing more fine-grain load, we are going to move in opposite direction. If we really want to used timelines for creating snapshots, then I am absolutely sure, that some customers will make a new snapshot once per day of even more frequently. So there can be thousands of timelines (certainly if we will not limit  number of created timelines). In this case it seems toe very useful and important to load timelines on demand.
So I wonder why we want to implement some intermediate solution which actually  leads us away from final goal (on-demand layer loading)? I have not inspected much this S3-interaction code, but I wonder will it be easy to implement concept of "unloaded" or "remote" layer file right now? Ar there some principle problems with it?
As far as I understand optimal size of data chunk for interaction with S3 is about few GB. Size of  layer file now is several times smaller But there were plans to increase it... What else preventing us from on-demand loading of layer files?
May be I am wrong, but once we have on demand layer loading, there is no need in "Loading", "Attaching", "Active" tenant's states.
Also do we want to support migration of layer files from one page server to another bypassing S3? I.e. if we want to relocate tenant at some other pageserver,  can we just create "stub" layer file which refers not to S3 file, but to file at another pageserver?

Do I correctly understand that our final goal is to provide on-demand load from S3 of individual layer files? Such lazy loading can speed-up tenant activation and migration to different page server. If so, I slightly confused by refactoring done in this PR.
Right now we have notion of loaded/unloaded timelines. Implementation of it IMHO looks really ugly, but at least it allows lazy loading of timelines. Now it is replaced with active/inactive tenants. So instead of providing more fine-grain load, we are going to move in opposite direction.

Great question! You're right that in this PR, we're actually going in the opposite direction. But the next step is to make the layer-file download happen on-demand. That makes it better again. Attaching a tenant becomes a much lighter operation, as it only needs to fetch the metadata of each timeline, not the actual data. So with that, there isn't much need to be more fine-grained with the timelines.
Doing the attach/detach happens at tenant-level simplifies things. We can assume that once a tenant is attached, we have information about all its timelines. That's needed to do GC correctly, for example.

If we really want to used timelines for creating snapshots, then I am absolutely sure, that some customers will make a new snapshot once per day of even more frequently. So there can be thousands of timelines (certainly if we will not limit number of created timelines). In this case it seems toe very useful and important to load timelines on demand.
So I wonder why we want to implement some intermediate solution which actually leads us away from final goal (on-demand layer loading)? I have not inspected much this S3-interaction code, but I wonder will it be easy to implement concept of "unloaded" or "remote" layer file right now? Ar there some principle problems with it?
As far as I understand optimal size of data chunk for interaction with S3 is about few GB. Size of layer file now is several times smaller But there were plans to increase it... What else preventing us from on-demand loading of layer files?

Currently, without this PR, we don't have a natural place to track which layers exist remotely, which ones locally. There's a "local index" of files that exist locally, built when a timeline is loaded, and there's a separate "remote index" that tracks remote files. And both of those indexes are kept at an arms-length from the "layer map" that's used the code in LayeredTimeline to actually look up pages. We want to integrate the state into one place, in the layer map, so that the layer map holds all layer files, whether they're present only remotely or locally.
I'm sure we could approach that goal from a different angle, and start with some other refactoring first. But this is what I came up with, and I think this PR simplifies things and makes that next step easier. With this PR, the "remote index" still kind of exists, as the latest_files HashMap in storage_sync.rs.

May be I am wrong, but once we have on demand layer loading, there is no need in "Loading", "Attaching", "Active" tenant's states.

Hmm. There would still be a state where you have received the Attach command, and started downloading the index files from remote storage, but the downloads have not finished yet. I think it would still make sense to have the tenant present in the hash map, in Attaching state, while you're doing that. The index file download should finish in a few seconds, but could take longer if there are network problems connecting to S3 for example. But normally, the Attaching state would be much shorter than today, as you wouldn't need to download the layer files.

Also do we want to support migration of layer files from one page server to another bypassing S3? I.e. if we want to relocate tenant at some other pageserver, can we just create "stub" layer file which refers not to S3 file, but to file at another pageserver?

No plans for that. We could do that, but it would complicate things, and I don't see much benefit.

Doing the attach/detach happens at tenant-level simplifies things. We can assume that once a tenant is attached, we have information about all its timelines.

While doing relocation stuff I came to a conclusion that we cannot always guarantee that because of racy branch creation. Current proposed solution to that is to check after successful attach whether the branch list changed and if so call attach for added timelines and delete for deleted ones. With this solution we still need to have both attach flavours. Also it is possible that this process wont converge if new branches are constantly created. With per branch attach we still need to attach the hierarchy of branches and it will need to do almost the same thing similar to current tenant level attach, like scan remote storage for ancestors, or receive this structure directly through api. So what I want to say is that simplification is not that obvious, both ways contain similar pitfalls

No plans for that. We could do that, but it would complicate things, and I don't see much benefit.

With that it would be possible to "steal" layers from neighbours to combine them and create larger files in s3. But it is pretty niche use case and probably isnt worth it
@knizhnik

In this case it seems toe very useful and important to load timelines on demand.

I'm fully agree with you and current Loaded/Unloaded stuff doesnt look ugly to me, but the main problem with it is that it is almost useless because we load everything during gc, so the loading is lazy until first gc comes. This can be solved but currently it is not. Criteria for unloading can be a bit complicated with branches. It may be easier to unload whole tenant from pageserver using detach when there is no activity for some long period of time. Or do that explicitly in a snapshot case you've mentioned, like attach -> snapshot -> detach. It will affect startup time, but may still be reasonable for free tier. Probably we wil end up implementing both :)
As the changes are quite large I think it is simpler for now to give up on that and focus on transforming pageserver state so it is simpler to manage. Having another +500 or more will complicate review even further

Doing the attach/detach happens at tenant-level simplifies things. We can assume that once a tenant is attached, we have information about all its timelines.

While doing relocation stuff I came to a conclusion that we cannot always guarantee that because of racy branch creation. Current proposed solution to that is to check after successful attach whether the branch list changed and if so call attach for added timelines and delete for deleted ones. With this solution we still need to have both attach flavours. Also it is possible that this process wont converge if new branches are constantly created. With per branch attach we still need to attach the hierarchy of branches and it will need to do almost the same thing similar to current tenant level attach, like scan remote storage for ancestors, or receive this structure directly through api. So what I want to say is that simplification is not that obvious, both ways contain similar pitfalls

Yeah, our tenant and timeline operations, like creating, deleting, attaching, detaching, are somewhat racy :-(. I just bumped into another weird case. When pageserver is created, in the tests we create the initial tenant and timeline as part of the pageserver initialization. But we don't immediately upload them to S3 (or do we? not sure). We do the upload on first pageserver startup. That seems weird.
I think we need to write down how the atomicity of each operation is achieved. Or if they're not atomic, what exactly the guarantees are. When is it OK to have tenants / timelines in pageserver that don't exist in S3, if ever? And vice versa. In general, I think we should treat S3 as the source of truth. If a timeline doesn't exist in S3, it should be removed locally too. If a timeline exists in S3 but not locally, it should be downloaded.

But we don't immediately upload them to S3 (or do we? not sure)

We do and dont at the same time :) The upload is scheduled, but after init pageserver immediately exits without waiting for this task to complete. We use init only in tests, but the same problem can happen when pageserver crashes immediately after responding with OK to create_tenant request. Caller will never know about the failure. If it is just a restart pageserver will pick up that and continue with the upload. I think this is the reason why it makes sense for client to always provide desired tenant/timeline ids when calling pageserver. Because this way we can trace such issues, and client can explicitly decide to regenerate the id and call create another time to start from scratch without leaving too much broken stuff behind. Also if pageserver crashes in the middle of the create_timeline operation it will leave garbage on disk, but it should be marked as broken with current check for zero disk_consistent_lsn
Somewhat similar cases can happen with lost detach/delete. If pageserver crashes or is restarted it will loose in flight operations. With attach it cannot happen because attach creates directory for timeline before replying back (though it is not the case with your patch, so it can happen). The problem can be solved with additional flags in metadata so pageserver can continue with interrupted action after restart. This solution incurs additional synchronization around metadata. We need to take the same locks as checkpoint/gc to avoid lost updates. For timeline delete (it doesnt exist but is similar to detach with the only difference of removing files from s3 as well as from local disk) we need to reorder operations, delete from remote first and then when operation succeeds remove local directory so in case of restart/crash we can pick up this operation and continue with it because if we delete local files first we wont know that remote data is still there.
Another idea is to use some sort of a log, but having wal for pageserver looks too complicated. So another way is to try hard to make these operations idempotent so they can be retried from every step (we can test this with failpoints)
Created an issue to track that https://github.com/neondatabase/neon/issues/2238

If a timeline exists in S3 but not locally

In case there is no directory with tenant/timeline id it is impossible to determine that such timeline had ever existed on this pageserver (without using some separate mapping like checking with the control plane)

Hmm. There would still be a state where you have received the Attach command, and started downloading the index files from remote storage, but the downloads have not finished yet. I think it would still make sense to have the tenant present in the hash map, in Attaching state, while you're doing that.

It seems to me that to download timeline metadata, we need to download just one file from S3. Or may be even one file per tenant  (metadata of all timelines) - not sure. Certainly downloading of any file from S3, even if it is not so large, may take some time. During which tenant ill be in obscure state (loading/attaching/...).
My point is that if we are going to have on-demand loading of layer files, then we in  any case have to somehow handle such loading: if the same layer file is requested by multiple threads, then only one should load it from S3 and other - wait until load is completed. It is standard procedure for loading chunks from slow devices, fir example postgres buffer manager is doing the same.
The same mechanism can be used for loading not only data, but also metadata - if some other thread wants to access this timeline/tenant - it should wait until loading is completed.
So my main concern for this PR is whether it makes sense to implement some intermediate solution instead of implementing one-demand layer loading right now. Otherwise we will have to invent some mechanism and notions which will be thrown away in future. From my point of view implementation of on demand layer loading is not more complex than on demand loading/attaching timelines or tenants. May be even simple: the more simple thing we have control the more simple algorithms can be used, and layer file is definitely simpler than timeline (hierarchy of layers) or tenant (hierarchy of timelines). For example developing eviction algorithms for layers may be simpler than for tenants.

Doing the attach/detach happens at tenant-level simplifies things. We can assume that once a tenant is attached, we have information about all its timelines. That's needed to do GC correctly, for example.

We definitely need to stop unused compute nodes. But I am not quite sure about attaching/detaching tenants on pageserver.
If we are able to evict tenant  layer files (and may be also metadata files), then the only thing we have to keep in memory is some information that such tenant is present, Event if we have thousands of tenant per page server, such information will not take much space.
According to the comments to this ticket and #2239 RFC the problem of synchronizing pageserver and S3 state and atomicity of timeline/tenant operation is considered to be very important and challenged. I wonder if per-file synchronization can simplify this problem.
It seems to be obvious that that can be some new layer files at pageserver which are not yet uploaded to S3.
An at the same time it is possible that in case of pageserver local storage corruption we have to download all data from S3.
But do we need to enforce some complex atomicity rules here? It seems to me that standard 2-phase approach: first upload data and only then, atomically, metadata. If metadata is store in single file (which seems to be the only acceptable solution) then atomicity is naturally enforces by S3.
I am not sure if it is needed to upload metadata to S3 immediately when new tenant is created. But even if we have to do it (wait confirmation from S3 that data is uploaded),  I do not think that such delay during tenant creation is not acceptable.
So, may be I missing something and it is not possible or not desirable to implement our final goal (on demand image loading) right now and it is better to start with some intermediate solution. But I have feeling that it may be even simpler than current PR.
If there are some problems with on-demand layer loading which can not be solved right now, please describe them.
If there is some complex refactoring, then definitely it may be better to split it into several steps. But I am not sure pageserver<>S3 interaction is such example. This PR is quite large and complex, so it can not e considered as simple step forward. And at the same time, it seems to be that many things can be implemented much easier at lower level (deadling with layers rahter than with timelines/tenants).

At pageserver startup, if some files are missing locally, but are present in S3, were they downloaded? With this PR, they are not. They probably should be.

Yes, such files are downloaded in the current version and I think they should, before we switch to the on-demand download.

Similarly, if the metadata file in S3 has a higher disk_consistent_lsn than the file on local disk, what happens?

We do check metadata's disk_consistent_lsn on every synchronization and replace it, if the source Lsn is bigger. Yet, we download timelines currently on attach/startup only, so such scenario now can happen only once per timeline, if everything works normally (no concurrent writes in S3, etc.)

On 'main', if an upload keeps failing, we eventually give up:
That seems bad. Later, we might remove the file locally too, and have no copies of it anywhere. Shouldn't we keep retrying forever? And alert humans if it keeps failing.

We can attempt forever, current implementation just does "something" to raise the awareness to the issue.
The question is, what to do in such situations on download on-demand cases: if we cannot download a certain layer, get_page@lsn request could fail always for certain Lsns, which is not the good thing either.
One other implication that such failed uploads bring is an inability to delete layers remotely: we don't track which layer was compacted into what and postpone any deletions before all uploads for the timeline are successful.

According to the comments to this ticket and #2239 RFC the problem of synchronizing pageserver and S3 state and atomicity of timeline/tenant operation is considered to be very important and challenged. I wonder if per-file synchronization can simplify this problem.

I have not observed the entire PR, but also notice that this approach was meant by me when I've composed https://github.com/neondatabase/neon/pull/2152
I still think, we should not keep too many tenant statuses, ecept for maybe some specific "disabled" one for future maintenance use cases and "broken" for generic errors that might happen during tenant creation (index update as per https://github.com/neondatabase/neon/pull/2239, for example).
Otherwise IMO, all we need to ensure for "atomicity" are the corresponding metadata files created everywhere (locally, on S3) — after that, we can consider both a timeline and a tenant working properly and concentrate on layer sync/on-demand downloading and keep statuses per layer.

@knizhnik

I do not see big problem in keeping layer map always in memory. Number of layers is not expected to be very large.
GC is not working with layers data, just with metadata.

Currently transition from unloaded to loaded state consists only of loading the layer map. This logic was written when we had previous version of storage that generated a lot of layers and you reported that for 1TB database this operation can take up to 15 minutes.
So the remaining part is unloading, unloding needs to drop inmemory layers which is performed with checkpoint, right? With https://github.com/neondatabase/neon/pull/2228 checkpoint will happen automatically after some timeout so maybe there is no need to do something apart from that. What do you think?

I briefly looked into the changes. Will continue to look into details. Quite like where it is going so far
As an idea, what do you think about turning pageserver into a struct and pass various services as generic parameters?
Something like that:
struct Pageserver<R: RemoteStorage, B: Broker, L: LocalStorage /* this is Repository now */ {
    tenants: RwLock<HashMap<ZTenantId, Arc<Tenant<R, B, L>>>,
    remote_storage: R,
    broker: B,
    wal_receiver: WalReceiver<B>, // or it can be part of a tenant
    page_cache: PageCache,
    // other global stuff we have if any
}

It may be too restrictive for local storage because currently we use Layered{Repository, Timeline} methods directly trough type alias. So probably worth excluding LocalStorage from this list. Task manager can be part of a tenant struct too. We can directly store cancellation handles inside tenant so shutdown becomes fully contained inside Tenant struct. May be a good idea to use Service suffix for parameters.
With this approach we can mock services to simplify testing. Another point is that main pageserver entrypoint becomes thinner. All initialization routines can be moved into Pageserver::start. This way it is easy to construct several pageservers for unit tests without using pageserver binary and spinning up quite a lot of stuff like etcd or s3 server to run python tests.
What do you think about this approach? May be better to keep this transformation for another time as it is relatively orthogonal to changes in this PR

On pageserver startup, if there are some files missing locally, they
are not downloaded from s3. It is assumed that the pageserver
local state is always ahead or at he same state as S3. How did this
work on 'main'?

On main if something is missing locally on startup download task is scheduled here https://github.com/neondatabase/neon/blob/4227cfc96eda8ddde3c4b4df51efac6f0f1fc8c0/pageserver/src/storage_sync.rs#L1580 what were your steps when this happened? I tried it with local version if remote storage by deleting files from pageserver dir and they were "downloaded" when pageserver started

Ok, I implemented a a first draft version of on-demand download here. I think it needs some explanation and discussion on how it works. I'm going to break out for dinner now, but will write some notes on how it works later today, so that we can discuss it.

timeline.get() can now return a special error type, PageReconstructError::NeedDownload(future). The future can be used to download the missing file: if you call future.await, the future will start downloading the file. When the download finishes, the future will return. That allows the GetPage@LSN codepath to remain synchronous, rather than async. Each get_page is synchronous, and will return with the error if a file is missing locally, and the caller can perform the download further up the stack.
This introduces a new Layer type: `RemoteLayer'. It's a placeholder that can be stored in the layer map, alongside delta- and image layers, to stand for a file that's present in remote storage, but not locally. The code for loading/attaching a tenant has been changed, so that it doesn't immediately schedule downloads for remote layers, but just creates RemoteLayer placeholders for them.

Another big update on this branch: I switched a lot of things to async, including page_service.rs. The philosophy now is that everything runs in tokio tasks. We don't spawn plain threads anywhere anymore.
All network I/O happens using async mode, including S3 upload/download. It's considered OK to do disk I/O in synchronous mode, all the low-level code to read/write layer files is synchronous. The core get_page_at_lsn() codepath is still synchronous; it doesn't depend on waiting on network. The core storage code uses std::sync for locking, not the tokio counterparts. I consider this a feature: the compiler will complain if you try to call await while holding a RWLockGuard or MutexGuard. This enforces that you never sleep waiting for network while holding a lock.
thread_mgr.rs has been renamed to task_mgr.rs, and it now deals with tokio Tasks instead of threads. The shutdown mechanism works the same as it did with threads: there's a global registry of tasks, and each task can be registered with a particular tenant or timeline. You can request for all tasks associated with a particular tenant or timeline to shut down, and wait for them to finish. (This implements https://github.com/neondatabase/neon/issues/2137)
WDYT? I have mixed feelings about this. The async code is harder to debug. But this makes things more consistent, as we were already dabbling into the async world quite a lot. I probably should've extracted this out of this PR to be reviewed separately, but that's a lot of work too...

Not quite sure what does the ... in the end mean, but pretty sure we're not merging this PR, especially now, when big parts of it are merged and https://github.com/neondatabase/neon/pull/2433 is coming.
I will continue extracting some pieces of this PR, but most probably, not that many: big part of the remaining code touches remote storage and task handling, that we need to rework more anyway.


Use the ceiling value when deriving a max trip count for ONNXLoopOp

This patch uses the ceiling value when computing a max trip count for ONNXLoopOp to make sure the number of iterations is NOT decreased compared to the actual trip count.
This patch also removes a redundant function.

Jenkins Linux amd64 Build #6000 [push] Use the ceiling value wh... started at 20:01

Jenkins Linux s390x Build #6016 [push] Use the ceiling value wh... started at 21:02

Jenkins Linux amd64 Build #6000 [push] Use the ceiling value wh... passed after 1 hr 9 min

Jenkins Linux ppc64le Build #5138 [push] Use the ceiling value wh... passed after 1 hr 9 min

Jenkins Linux s390x Build #6016 [push] Use the ceiling value wh... passed after 1 hr 10 min


[google-cloud-cpp] update to latest release (v2.18.0)

Updates google-cloud-cpp to the latest release (v2.18.0)
Tested locally (on x64-linux) with:
for feature in securesourcemanager; do ./vcpkg remove google-cloud-cpp; ./vcpkg install "google-cloud-cpp[core,${feature}]" || break; done

and
./vcpkg remove google-cloud-cpp && ./vcpkg install 'google-cloud-cpp[*]'

[x] Changes comply with the maintainer guide
[x] SHA512s are updated for each updated download
[x] The "supports" clause reflects platforms that may be fixed by this new version
[x] Any fixed CI baseline entries are removed from that file.
[x] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

@microsoft-github-policy-service agree company="Google"

The CI seems to be giving unrelated errors.

Can one of the maintainers take a look?

@jimwang118 Can you take a look when you get the chance?


SOLR-15480 Added putAll method to Tuple and refactored the copy constructor, merge, and clone methods to use it.

https://issues.apache.org/jira/browse/SOLR-15480

Description
I added a new putAll method to resolve an issue where Tuple.fields could rehash multiple times when calling the copy constructor. The original clone and merge methods also wouldn't mark EOF and EXCEPTION if those tokens existed, but I refactored those methods to use the new putAll method to handle setting them.
Solution

Added a new putAll method to Tuple that supports marking EOF and EXCEPTION true when tokens are found.
Refactored the copy constructor, merge, and clone methods to use the putAll method.

Tests
I added no new tests since I saw Tuple already had coverage throughout the solrj package. I ran those tests along with the full test suite for the entire project.
Checklist
Please review the following and check all that apply:

[x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
[ ] I have created a Jira issue and added the issue ID to my pull request title.
[x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
[x] I have developed this patch against the main branch.
[x] I have run ./gradlew check.
[ ] I have added tests for my changes.
[ ] I have added documentation for the Reference Guide

This LGTM, @cpoerschke did you have thoughts on additional documentation around this change?

fieldNames and fieldLabels are not copied via the copy constructor, nor via clone and merge

probably should handle that here

I'll look into handling those as well!

Tuple.fieldNames and Tuple.fieldLabels should be covered in clone, merge, and a new copy constructor.

Awesome, my pleasure, and thank you @cpoerschke for taking the time to work with me!


Please help for below argumnets

Plz help me to understand the meaning of below arguments-
--text_threshold: text confidence threshold
--low_text: text low-bound score
--link_threshold: link confidence threshold
@ClovaAIAdmin @YoungminBaek

From what I've been able to understaind myself:
text_threshold = Certainity required for a something to be classified as a letter. The higher this value the clearer characters need to look.
low_text =  Amount of boundary space around the letter. The higher this value the less space
link_threshold =  Amount of distance allowed between two characters for them to be seen as a single word.

low_text = Amount of boundary space around the letter/word when the coordinates are returned
What exactly is meant by boundary space??
 Can anyone please let me know??


Bring EqualityChecker in line with new testing setup

Note, this issue is blocked by Issue 204
What?
There are a few required changes for this, summarised below from (https://github.com/lvgig/tubular/issues/150)

This transformer compares two columns, should obey arg convention 'lower_column' and 'upper_column'
new_column_name argument should obey arg convention new_column_names=[list of names]
Double check "minimal attribute dict" fixture has the correct set of minimal arguments required
Set up the TestInit TestFit, TestTransform, and TestOtherBaseBehaviour classes to inherit the appropriate test classes and then copy the setup_class class method into each test class, changing transformer_name to the transformer you are testing (examples linked) .
Extra tests specific to this class should be added, e.g. if this transformer also needed to check that when len(mappings)>2 then mappings is an ordered dict, you would add this to TestInit. Where extra tests would apply to all child classes, they should instead be added to the parent test class.
Delete any unnecessary tests from original transformer test file (i.e. tests testing inherited behaviour that is now tested by the inherited test classes). An example would be. In general asserting calls is a red flag, as this tests the applied process rather than the desired outcome.
Debug the failing tests from the base test suite.
Bring the test file in line with best practices by insuring there are no implementation tests (in this case looks like these will have been removed as part of the base test changes)
Bonus: If you identify any gaps in the testing, please set up new issues :)

Why?
Revamp tubular testing to make package more maintainable/modifiable

Completed as part of PR - https://github.com/lvgig/tubular/pull/240


I finished build, but no executable works

Hi, After I build lethe following guide

I cannot execute anyone. Do you have any ideas what's wrong here?

Hi @xshen053 ,
This is not an application but a test. If you are trying to run an example with Lethe you should go to the applications folder within your build folder, where you can find all the executable. You can follow the instructions written here.

Hi @xshen053 ,
This is not an application but a test. If you are trying to run an example with Lethe you should go to the applications folder within your build folder, where you can find all the executables. You can follow the instructions written here.

Thank you, I ran it with example yesterday, but it showed segmentation fault again, I will double check today. Is it because it doesn't support arm? I am running it on arm-based server (not apple)

Hello @xshen053, Welcome to the Lethe communauty. To start which example are you trying to run with which application? Can you send us an example of the commend you are trying to run?

no problem, will send it today

Dear @xshen053 have you managed to fix your issue? Otherwise, can you post the full error message you get.
Thanks
Bruno

Hi, I plan to check it when I get a chance. Thanks!!

@xshen053  I think I will close this issue.


More shortcuts for page elements

We need some more keyboard shortcuts to manage the page elements more easily:

[ ] Press [Arrow-Left] to close a parent element (like clicking on the arrow)
[ ] Press [Arrow-Right] to open a parent element
[ ] Press [Escape] to jump to the parent element
[ ] Press [Tab] to jump to the next sibling element

What about [Arrow-Up] and [Arrow-Down] to navigate from one sibling to the next?

Not sure if we should use arrow up/down for page scrolling in the preview pane... We could also depend the shortcuts on the pane you are actively working in.


*args are ignored by memoize.

Is it intentional that the *args after the declared arguments are ignored by the memoize decorator, or is that a bug? Below is an example in an ipython shell with python==2.7.6, django==1.4.22 of how memoize isn't taking into account the *args.
In [16]: @memoize()
def func(*args, **kwargs):
    return args, kwargs
   ....:

In [17]: func(1)
Out[17]: ((1,), {})

In [18]: func(2)
Out[18]: ((1,), {})

In [19]: func(1, 2)
Out[19]: ((1,), {})

In [20]: func(1, one=1)
Out[20]: ((1,), {'one': 1})

I added a pull request which resolves this and includes updated tests.
Pull Request #9

Solved.


http://dnd5eapi.co/api/classes/[class]/levels is returning an empty list.

It should return the leveling changes for the class, but instead I just get an empty list

I think this would get fixed as a result of this PR. The issue right now is right now it is looking for a URL with the index, and it's currently stored as the class name.

Also with this PR.

This should be fixed with #26


allow to build wheels for selected packages only

Currently packages are built based on the requirements files only. This PR adds a new manual workflow, which allows to build specific packages with requirement specifiers manually for selected operating systems/architectures.

Hi Juraj(@JurajSadel)/Roland (@dobairoland),
this is the first version and hopefully I haven't missed anything crucial. It would be great if you could take a look.
Thank you!

LGTM. Thank you @fhrbata!

@JurajSadel  secrets: inherit added, Could you PTAL? Thank you!

@fhrbata Thank you for making that changes! One more question from my side, what is the motivation behind the upload argument?

@fhrbata Thank you for making that changes! One more question from my side, what is the motivation behind the upload argument?

@JurajSadel I used it just for testing to avoid running the upload/index steps, which would fail anyways. If it all works we can remove it. Also I guess it should be enabled by default in the ad hoc wheels workflow. I have no strong opinion on this, so just let me know what you prefer and I will adjust the PR.
Thank you

@fhrbata I agree with you, let's remove it :)

@fhrbata I agree with you, let's remove it :)
@JurajSadel  again thank you for looking into this. The upload var should be removed now.


PeerJ R2C8

Section 3
'dressed up as ...' that sounds a bit weak.

The original sentence is:

JOSS offers developers a venue to publish their research software, dressed up as software articles, thus enabling citation credit for their work.

My suggested change is to:

JOSS offers developers a venue to publish their research software with short high-level articles, thus enabling citation credit for their work.

Maybe 'brief' instead of 'short' ? (Otherwise I agree with your suggested change.)

Yeah, I was going back and forth on those two words, but "brief" works fine for me.

How about:
JOSS offers developers a venue to publish their complete research software wrapped in relatively short high-level articles, thus enabling citation credit for their work.

That's fine with me as well 👍

I adopted @danielskatz's suggested wording:

JOSS offers developers a venue to publish their complete research software wrapped in relatively short high-level articles, thus enabling citation credit for their work.


I am getting following install error

+1.

@Rohan-Jalil , update package.json, gulp-sass to, say, version 3.1.0 (the latest one), and it should work.

Just updated all the dependencies and simplified the build steps, should be OK now.


refactor: Use a more engineering-oriented error code wrapper

Use "github.com/pkg/errors" to wrap error. In a word, If a function doesn't need to return an error, print the error and  stack information with logrus, otherwise wrap the error and append some information.

so many changes... can you cut this big pr into small pr?


Bump version of go-github library to v40

This release contains the following breaking API changes:

#2127

and the following additional changes:

#2126
#2128
#2129
#2131
#2133
#2141
#2193
#2195
#2119
#2197
#2165

plus several test- or documentation-only changes.

@Parker77 - can you please review this when you have time?


🤗 [REQUEST] - Qwen2 models

Model introduction
The models are created by Tongyi Qianwen, part of Alibaba Cloud, and are a general-purpose model, but incorporate the techniques used to train CodeQwen. More information can be found in their blog post.
Model URL
https://huggingface.co/collections/Qwen/qwen2-6659360b33528ced941e557f
Additional information (Optional)
No response
Decontamination
The authors did not mention decontamination, but it might be mentioned in the upcoming technical report.
Author
No
Data
No
Security

[X] I confirm that the model is safe to run which is not designed to produce malicious code or content.

Integrity

[X] I confirm that the model comes from unique and original work and does not contain any plagiarism.

They reported HumanEval, MBPP, and what looks to be the arithmetic mean of the HumanEval, HumanEval+, MBPP, and MBPP+ scores in their blog post.

Wonder why they are not logged in the leaderboard tho, feels very dated


[BUG] ERROR: Could not build wheels for hidapi, which is required to install pyproject.toml-based projects

Software Versions:

Operating System: Raspbian GNU/Linux 12 (bookworm)
Python version: 3.11.2
BusyLight: Kuando Omega

General Type of Problem

[x] Installing
[ ] Command-line
[ ] Integrating into another project
[ ] Web API
[ ] other

Describe the Problem
When running python3 -m pip install busylight-for-humans, I get the error in the title
Expected Behavior
I expect it to install...
Error Output
Building wheels for collected packages: hidapi
  Building wheel for hidapi (setup.py) ... error
  error: subprocess-exited-with-error

  python setup.py bdist_wheel did not run successfully.
  exit code: 1

  [19 lines of output]
  /home/user/.local/lib/python3.11/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
    dist.fetch_build_eggs(dist.setup_requires)
  running bdist_wheel
  running build
  running build_ext
  Compiling hid.pyx because it changed.
  [1/1] Cythonizing hid.pyx
  /home/user/.local/lib/python3.11/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-85i8kzpk/hidapi_c909f4a410e14dce968b40b1b7755855/hid.pyx
    tree = Parsing.p_module(s, pxd, full_module_name)
  building 'hid' extension
  error: unknown file type '.pxd' (from 'chid.pxd')
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hidapi
  Running setup.py clean for hidapi
Failed to build hidapi
ERROR: Could not build wheels for hidapi, which is required to install pyproject.toml-based projects```

Thanks for the bug report! Unfortunately, the source of that error is the hidapi package.  The install process couldn't find a prebuilt hidapi package for your operating system and CPU, so it attempted to build it from source and failed. This could be due to missing or down level software used to build the hidapi package (Cython I think). I've been able to install Busylight successfully on a Raspberry Pi in the past but haven't tested it recently.

This Stackoverflow question has a solution for a similar situation, maybe it will help you.

This Stackoverflow question has a solution for a similar situation, maybe it will help you.

Hey, thanks for giving it a shot.
Unfortunately, it didn't work out. I've spent hours trying to fix it, but no luck. As you mentioned, it seems to be a hidapi problem, and I might reach out to them and see if they can help.

@Swahgner Which kind of Raspberry Pi are you installing on? I've always installed on a Raspberry Pi … but also not recently.
I forget which model I'm using currently, but it's probably something like a Raspberry Pi 3.
I was not able to install on a Raspberry Pi Zero W back in the day. I seem to recall this was some issue involving Rust, which may have been due to a dependency. My memory is fuzzy on this, however. There may have been an issue about it.

@Swahgner Which kind of Raspberry Pi are you installing on? I've always installed on a Raspberry Pi … but also not recently.
I forget which model I'm using currently, but it's probably something like a Raspberry Pi 3.
I was not able to install on a Raspberry Pi Zero W back in the day. I seem to recall this was some issue involving Rust, which may have been due to a dependency. My memory is fuzzy on this, however. There may have been an issue about it.
Update: Yeah, it was #2 and it was an issue with a FastAPI dependency.

I have tried on a Raspberry Pi 3 Model A+. I tried switching to an ubuntu server OS instead, but everything leads back to hidapi failing to build wheels. I have tried countless of "solutions" I've found, but everytime it's the same issue


Add a preview option for files

in windows file explorer (the one that we have by default), you can preview certain types of files. This would be a great feature to add to Files UWP.

@pixl-8 currently Files UWP offers file preview through the integration with QuickLook, but the possibility of having built-in preview is being investigated.
Same as #130

Ok 👍


Fix with_scale_factor_override improperly setting scale_factor_override

Objective
Fixes #12282.
Solution
Use set_scale_factor_override in with_scale_factor_override.

Probably a candidate for 0.13.1 as well.


Integrate webcam

@chasmastin and someone else (sorry, I forgot who) came up with the concept that we would snap a picture of whomever filled the coffee last.
Open questions:

Where does the picture go and what do we do with it?
Do we do any processing on the image?
How does the person know their picture is being taken?

Anyone interested in making this feature cool please throw ideas in the comments!

To go cheap we're going to try the Logitech C270 camera - http://www.logitech.com/en-us/product/hd-webcam-c270
We can find them for under $20 now and it should do all we need.


[BUG] PlatformException (PlatformException(10005, error *errors.withStack not implement CodeError: GetFriendList failed: disk I/O error: read-only file system, null, null))

Describe the bug
When there is too much conversation messages, calling conversationManager.getConversationListSplit will report an error.

How to reproduce
Steps to reproduce the behavior:
Use open-im-flutter-demo

Create 1,000 group conversation messages.
After synchronization.
Kill the app process and re-enter.
Go to conversation.
Scroll to the bottom.
Scroll again.
When offset is 320.
Error occurred.

Expected behavior
Conversation messages are loaded correctly
Screenshots (If contains)

Version information

Device: OPPO REMO4 5G
OS: Android 10
Package Version: 3.8.1+31
Flutter Version: 3.22.3
Flutter Openim Sdk Version: ^3.8.0

Additional context

Hi, set count to a smaller value. For example, 50, 100.

Now the count is 40. I changed the count to 20, but the problem still remains.
final pageSize = 20;

void onLoading() async {
  late List<ConversationInfo> list;
  try {
    list = await _request(this.list.length);
    this.list.addAll(list);
  } finally {
    if (list.isEmpty || list.length < pageSize) {
      refreshController.loadNoData();
    } else {
      refreshController.loadComplete();
    }
  }
}

_request(int offset) =>
    OpenIM.iMManager.conversationManager.getConversationListSplit(
      offset: offset,
      count: pageSize,
    );

The function reporting the error is "getFriendList".

Please try this code.

List list = [];
final pageSize = 50;

getConversationList(int offset) async {
  var list = await OpenIM.iMManager.conversationManager.getConversationListSplit(
    offset: offset,
    count: pageSize,
  );
  return list;
}

FloatingActionButton(
  onPressed: () async {
    var result = await getConversationList(400);
    setState(() {
      list = result;
    });
  },
)

The exception information shows that something went wrong with "GetFriendList".


2015 pilot state machine.

I've started scribbling down an FSM similair to last years, but I think this one might be simple enough to just do linearly.
ie:
line-follow until T-intersection. stop.
turn.
line-follow to block.
(solve activity)
turn around.
line-follow to intersection.
(repeat to finish)

http://www.ewh.ieee.org/reg/3/southeastcon2015/track/index.html
This complicates things a lot. As it turns out, they will have intersections, so the state machine will have to account for this.

Some functionality that I would like is:

"follow_to_intersection":

Ability to differentiate intersection from right angles. (We'll avoid worrying about 4-ways for now, because I'm positive that the demo boards having them was a mistake).

Ideally, this would be handled within the follow function so that the previous function can continue working uninterrupted.

"follow_to_box"

Functionality would be similar to other follow_to_intersection. I think it might even be possible to just use follow_to_intersection, since to the IR's might not even see them as different.
It's worth noting that this function doesn't even have to tell the difference between a box and an intersection, because neither one would ever be called in a situation where it might hit the other.

turn_on_intersection

This should be easy enough to do without any huge changes. I'm working on a function in mec_driver for rotating 90 degrees, but some line-detecting intelligence in it would be useful if that's not accurate enough.

Update: turn_on_intersection is unecesary, blindly turning works fine and it just needs to get close enough for IR to see it anyway.

All that matters is being able to follow the line (including around corners) and stop whenever you hit an intersection or box.
Return values aren't important, but if you return information strings about what exactly it saw to stop, that has the potential to be useful.

Pilot will need to be updated for general navigation.


Saving plotly-resampler object for faster viewing

I am plotting large time series array and it takes considerable time to plot. Currently, I am loading the data and passing it to resampler which resamples and plots. My understanding is that resampling code is taking time.
Is it possible to save the plotly resampler object (first time) so that I can load the object and view plot faster for future use?

Hey @vinay-hebb,
Thx for submitting this issue. I observe the following questions in this issue;
A. Can you save plotly-resampler figures?
B. Can you view the plot faster?

A.
You can save Plotly-Resampler figures by serializing it.
fig_to_save = ...  # some plotly-resampler figure

# Save the figure (pickle it)
pickle.dump(fig, open("saved_fig.pkl", "wb"))
del fig_to_save  # you can delete the plotly-resampler figure

# Load the figure
fig_loaded = pickle.load(open("saved_fig", "rb"))

B.
Saving & loading should not result in faster downsample (as plotly-resampler is stateless).
Can you provide some more details;

What is slow? The initial loading/rendering (i.e., the construction of your figure)?

How do you construct your plotly-resampler figure? (do you pass the x and y data as hf_x and hf_y to the add_trace method) - https://github.com/predict-idlab/plotly-resampler#usage

What is the size of your data? (Number of signals, number of samples per signal)?

Feel free to submit some minimal working code on how you currently create your plots :)

Hope this answers your questions.
Cheers, Jeroen

I will close this issue for now.


AVT violation when there are two TableToolbarSearch components

Environment

Operating system :
Mac

Browser:
Chrome

Automated testing tool and ruleset :
DAP with "IBM Accessibility February 2019" ruleset

Detailed description

What version of the Carbon Design System are you using?
6.101.0

What did you expect to happen?
there is no AVT violations when there are two TableToolbarSearch existed in one page

What happened instead?
the below AVT violation is scaned by DAP tools : Multiple WAI-ARIA search landmarks must have unique labels specified with aria-label or aria-labelledby

What WCAG 2.1 checkpoint does the issue violate?
https://aat.w3ibm.mybluemix.net/token/c3b7a795-a87f-4fe4-8071-059d47e0571f/b5cf2a6d-027b-42c7-aaa0-1e99adf7ac78/latest///doc/w3/help/en-US/idhi_accessibility_check_g1097.html

Steps to reproduce the issue

Add two TableToolbarSearch components in one page.

Additional information

Screenshots

not stale ⚡
👋 Hey there -- are you still having this issue? I'm not sure if stacking data table search bars is something we support officially. Could you elaborate on what you were intending to do here and maybe provide an example I can test against?


Unable to open 'PlantUML Preview': write EPIPE.

During running preview I got following error:
Unable to open 'PlantUML Preview': write EPIPE.
Nothing is generated.
Any hint?

I want to see your environment value setting.
for example, screen shot.

Could you be more specific. Which files (or from which view) you want to have screenshot.

If your OS is Windows then Environment value setting window screen shot.
I want to know your real setting value.
Maybe,  If your setting value is "C:\Program Files\Java\j2sdk1.8." that include double quote then you should remove double quote.
Otherwise, If JAVA_HOME environment value is C:\Program Files\Java\j2sdk1.8.*\bin then you remove bin.

Ok, it is on my work computer I will provide your info more info on Monday. Thanks.

Some of my settings:
GRAPHVIZ_DOT=c:\Program Files (x86)\Graphviz2.38\bin\dot.exe
HOMEDRIVE=H:
HOMEPATH=
JAVA_HOME="C:\Program Files (x86)\Java\jre1.8.0_111"
LogPath=C:\WINDOWS\Logs\Siemens
Path=C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\RSA SecurID Token Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\Syst
em32\WindowsPowerShell\v1.0;C:\Program Files (x86)\WebEx\Productivity Tools;C:\Program Files (x86)\Sennheiser\SoftphoneSDK;C:\Program Files (x86)\SafeCom\Safe
ComPrintClient;C:\Python27;C:\Python27\Scripts;C:\Program Files\TortoiseSVN\bin;;C:\Program Files (x86)\Microsoft VS Code\bin;C:\Python27;C:\Python27\Scripts
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PLANTUML_HOME=C:\bin
PLANTUML_JAR=c:\bin\plantuml.jar

Thank you.
You should remove to double quote in JAVA_HOME.

Thanks you are right. Now it works :)

I am getting the same error:
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_171
Error: write EPIPE
at afterWriteDispatched (internal/stream_base_commons.js:149:25)
at writeGeneric (internal/stream_base_commons.js:140:3)
at Socket._writeGeneric (net.js:776:11)
at Socket._write (net.js:788:8)
at doWrite (_stream_writable.js:435:12)
at writeOrBuffer (_stream_writable.js:419:5)
at Socket.Writable.write (_stream_writable.js:309:11)
at c:\Users\calix\.vscode\extensions\jebbs.plantuml-2.13.14\out\src\plantuml\renders\local.js:112:35
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:94:5)


Add Save As function

Add a Save As function to save the current source as a new file, possibly even in a different project.

Pushing this issue back and breaking it into smaller issues, as it has 5 main parts that I can see.

Save an unsaved ("untitled") file as a named file (addressed in #107)
Save a file as a different name
Save a file into a different project
Add a "Save As" button into the UI
Add a "Save As" key command

Agreed.

This issue is split apart into 4 issues. For more information please go the sub-issues. #152, #153, #154, #155


build compiler-rt 14.0.0.rc4

Let's see how it goes

I think this is also hitting https://github.com/llvm/llvm-project/issues/53281. The fix for this was merged after rc2; unless there's a strong need for testing compiler-rt earlier, I think I'll just wait for rc3. Otherwise, it'd be possible to do a work-around à la https://github.com/conda-forge/llvmdev-feedstock/pull/143/commits/0f60bdd84dc128fee5ecda122b7cda73e957acfc

Hi! This is the friendly automated conda-forge-linting service.
I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.
Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

It looks like the 'compiler-rt' output doesn't have any tests.

So the current way to build seems to have its days numbered... It really wants to have the LLVM sources:
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:287 (message):
  LLVM source tree not found at
  "/Users/runner/miniforge3/conda-bld/compiler-rt-packages_1647512983761/llvm".
  You are not using the monorepo layout.  This configuration is DEPRECATED.
Call Stack (most recent call first):
  CMakeLists.txt:70 (load_llvm_config)

CMake Warning at cmake/Modules/CompilerRTUtils.cmake:315 (message):
  Consulting llvm-config for the LLVM source path as a fallback.  This
  behavior will be removed in the future.
Call Stack (most recent call first):
  CMakeLists.txt:70 (load_llvm_config)

-- Using LLVM source path (/Users/runner/miniforge3/conda-bld/llvm-package_1647096439118/work) from llvm-config
CMake Warning at cmake/Modules/CompilerRTUtils.cmake:352 (message):
  llvm-config finding testingsupport failed with status 1
Call Stack (most recent call first):
  CMakeLists.txt:70 (load_llvm_config)

CMake Warning at cmake/Modules/CompilerRTUtils.cmake:405 (message):
  LLVM_MAIN_SRC_DIR
  (/Users/runner/miniforge3/conda-bld/llvm-package_1647096439118/work) does
  not exist.  You can override the inferred path by adding
  `-DLLVM_MAIN_SRC_DIR=<path_to_llvm_src>` to your CMake invocation where
  `<path_to_llvm_src>` is the path to the `llvm` directory in the
  `llvm-project` repo.  This will be treated as error in the future.

@conda-forge/compiler-rt
Given that the standalone build is deprecated and not functional anymore currently (at least on linux), should we just bite the bullet and move to a mono-repo build for this?

Ah, looks like the glibc error can be solved with a little patch. I asked upstream, and the effective lower bound for glibc seems to be 2.17 (by virtue of not having any CI with lower versions).
With that, there shouldn't be any controversial parts of this PR, so I'm planning to merge it in 48h or so, unless there are voices to the contrary. There'll still be opportunity for more in-depth review on the merge to main for 14.0.0 GA.


[V6] [BUG] PlatformBuilder getConfiguration bug prevents routes in tests

Information

Version: 6.x
Packages: @tsed/common

This line wrongly assigns componentsScan configuration to mount, preventing any routes / controllers to be registered:
https://github.com/TypedProject/tsed/blob/2ac21a149a60f7b65798e8f67f1b3006678b8047/packages/common/src/platform-builder/utils/getConfiguration.ts#L16

Hello @tobiasstrebitzer
Thanks for your issue :)
The PR is ready!

:tada: This issue has been resolved in version 6.3.2 :tada:
The release is available on:

v6.3.2
GitHub release

Your semantic-release bot :package::rocket:


OP-20548 : Added new controller for SSD service

https://devopsmx.atlassian.net/browse/OP-20548

This PR  triggered this build in this execution, please wait for it  and child pipelines to succeed before merging:  https://isd.prod.opsmx.net/deck/#/applications/dev-oesaona-4x/executions/details/01H2TGK4QN82A9YMSG1K34BH76


Generate Button is Not Aligned in Add User Modal

Describe the Issue:
Generate button is not aligned in addUser modal as in the following screenshot.

How To Reproduce:

Go to console
Open User Management
Click Add User
Select Set a password for the user option
You will see the password input textbox and the generate button.

Expected behavior:
The Generate button should be aligned with the password text box. Or should be resized accordingly.
Device Information:

Device (e.g. PC, iPhone6):
OS (OSX, Android, iOS):
Browser + Version:

Developer tests in beta-4-snapshots


first commit

Vorolay lib commited

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.