KAFKA-12468, KAFKA-12566: Fix MM2 causing negative downstream lag

The primary issue being addressed here is the incorrect translation of offsets, the title issue KAFKA-12468.
Additionally, this PR stabilizes the MirrorMaker2 integration tests which were too flakey to properly verify this fix.
The MM2 KIP does not discuss the offset translation mechanism in detail, so I'll summarize the mechanism as it currently exists on trunk:

Records are mirrored from source topic-partition to target topic-partition by the MirrorSourceTask
MirrorSourceTask will (occasionally) emit OffsetSync messages to an Offset Syncs topic. Offset syncs contain the upstream and downstream offset of an emitted data record.
The MirrorCheckpointTask will consume from the offset syncs topic, and maintain an in-memory copy of the latest offset sync seen for each topic-partition (in OffsetSyncStore)
Periodically the MirrorCheckpointTask will poll consumer group offsets for the source topic, and use it's in-memory copy of the latest offset sync to translate upstream offsets to downstream offsets.
This is done by measuring the 'distance' between the MM2 offset sync and the upstream consumer group, and then assuming that the same distance applies in the downstream topic.

Step (5) is correct when assuming that every offset from the source topic has already been reproduced in the downstream topic. However, this assumption is violated when offsets are not present, which can happen for a variety of reasons, including:

Transaction markers take an offset but will never be emitted as records from the consumer
Records are dropped by SMTs and will never be emitted to the target topic
The source topic has been compacted and some offsets will never be emitted by the consumer
MM2 replication is lagging behind an upstream consumer group and some records have not been replicated yet

In any of these conditions, an upstream offset may be translated to a downstream offset which is beyond the corresponding record in the downstream topic. Consider the following concrete example of situation (4) resulting in negative lag:

Source topic A has 1000 records, all with contiguous offsets
An upstream consumer group cg is at the end of the log, offset 1000.
MM2 begins replicating the topic, and writes 500 upstream records to the target topic target.A, and writes offset-syncs correlating (A, 500) with (target.A, 500).
MM2 checkpoint reads cg offset 1000, translates the offset to 500 + (1000-500) = 1000, and writes to target.cg
Someone checks the target.cg offset for target.A and observes that the group offset is 1000, the topic end offset is 500, and the lag is -500.

And the following concrete example of situation (1) resulting in undelivered data.

Source topic A has 1000 records, all emitted with a transactional producer.
The 1000 records are interleaved with 1000 commit markers at every other offset.
An upstream consumer group cg is in the middle of the topic, at offset 1000.
MM2 begins replicating the topic, and writes 1000 records to the target topic target.A, and writes offset-syncs correlating (A, 500) with (target.A, 250), in addition to other offset-syncs.
MM2 checkpoint reads the cg offset 1000, translates the offset to 250 + (1000 - 500) = 750, and writes to target.cg
A system fails-over from cg to target.cg and someone notices that the cg application read records 0-500, target.cg application read 750-1000, but no consumer ever received offsets 500-750.

This PR adds a test that replicates transactional data, as in situation (1). It asserts that whenever an offset is translated, it does not pass the end of the downstream topic, and cannot cause negative lag. In addition the tests are strengthened to require the offset syncs to be emitted up to the end of the topic, requiring a fix for the offset-syncs topic starvation issue. This also exposed a number of mistakes and flakiness in the existing tests, so this PR also stabilizes the tests to make them useful for validating the negative offsets fix.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

After #13181 is merged, I'll rebase and remove my fairness patch.

Oh, and the Jenkins build seems to be consistently failing on the IdentityReplicationIntegrationTest::testNoCheckpointsIfNoRecordsAreMirrored test case. Probably want to look into that before we merge 😄

I notice that we don't do a read-to-end of the offset syncs topic in MirrorCheckpointTask before we begin syncing consumer group offsets, and we begin reading that topic from the beginning. This may cause us to sync offsets based on stale checkpoints if there are later checkpoints available in the topic that we haven't consumed yet. Do you think it might make sense to add a read-to-end for the offset syncs topic before we begin syncing consumer group offsets in the checkpoint connector?

Ouch, yeah that is certainly an issue that gets worse with my change.
Before the checkpoints would be non-monotonic for transactional/compacted topics, and after it's non-monotonic for everyone.
I think addressing this in a follow-up is a smart idea, this change is already messy enough.

Oh, and the Jenkins build seems to be consistently failing on the IdentityReplicationIntegrationTest::testNoCheckpointsIfNoRecordsAreMirrored test case. Probably want to look into that before we merge 😄

It looks like this was failing due to a typo in my offset.flush.interval.ms overrides. This should be fixed now.

Ouch, yeah that is certainly an issue that gets worse with my change.
Before the checkpoints would be non-monotonic for transactional/compacted topics, and after it's non-monotonic for everyone.
I think addressing this in a follow-up is a smart idea, this change is already messy enough.

And funnily enough, someone's already filed a ticket for that exact issue.
Out of an abundance of caution, what do you think about targeting your mm2-negative-offsets branch with a new PR to address KAFKA-13659, which can be reviewed separately but then merged to trunk in tandem with this PR?
This has the potential to be a fairly large change in behavior, and I'd like to do everything we can to minimize the chances that it breaks users' setups. Ensuring that this PR is merged if and only if a fix for KAFKA-13659 would help on that front.

@mimaison This is a moderately large change in behavior and if possible, it'd be nice to get another set of eyes on it before merging. We don't need another reviewer for the PR changes (although comments are always welcome); instead, I'd just like confirmation that this change is safe to make as a bug fix.
TL;DR: If an upstream consumer group is ahead of the upstream offset for the latest-emitted checkpoint, we will only sync offsets for that consumer group to the downstream cluster based on the offset pair for that checkpoint, instead of adding the delta of (upstream offset for consumer group - upstream offset in checkpoint), since there is no guarantee that that delta will be accurate in cases where the upstream topic is compacted, has transaction markers, or has some records filtered out via SMT.

Out of an abundance of caution, what do you think about targeting your mm2-negative-offsets branch with a new PR to address KAFKA-13659, which can be reviewed separately but then merged to trunk in tandem with this PR?

I tried to do this, but it appears that the "stacked PRs" feature for github only applies for branches on the same repository, not across forks. If i were to open a stacked PR, it would be on my fork, and review comments may be lost if I accidentally delete the repository.
In the interest of an atomic merge, and preserving comment history, I've added the commits here. To make it easier on your review, I will continue to tag the relevant commits with KAFKA-13659, so you can filter during your review. I hope this is acceptable, but sorry in advance for the difficult review.

I think my commit to synchronize the OffsetSyncStore introduced a dramatic performance regression that is causing the current test failures. Where previously the test would comfortably pass with consumeEfficiency = 10, it now fails with any value over about 0.5, for a ~20x decrease in read performance. I'll need to investigate this to confirm, and then look into ways to improve the performance of the OffsetSyncStore.

Hmmm... there appear to be some integration test failures. I've reproduced some of them locally too, which makes flakiness an unlikely cause. Can you look into the integration test failures and see if we can get a green run before merging this?

Unfortunately those test failures only appear in the EOS test and appear to be caused by EOS mode.
This is because MM2 doesn't do the periodic background commits that the offset syncs starvation fix relies on, and these assertions rely on the offset syncs starvation fix to assert that all the syncs are emitted.
I added a tweak to the firePendingOffsetSyncs which drains the offset syncs map on each commit. Do you think we can use the same blocking-drain for EOS and normal mode, or should this behavior only be enabled for EOS mode?

Hmmm... we make an effort to periodically invoke SourceTask::commit in non-EOS mode, even if there have been no new records written and there are no offsets to commit. I think we should do the same in EOS mode, instead of adjusting MM2 to accommodate this unexpected change in behavior.

I've split the periodic commit fix into a separate PR (https://github.com/apache/kafka/pull/13262) as it is not related to the changes here, only the test changes. This PR depends on that change landing first, and I'll update this branch once the other PR is merged.

Hi @gharris1727, if I understand this PR correctly, this will (almost always) cause duplication of data. Our problem is this:

We have a service reading from a Kafka topic and committing offsets for its consumer group.
We use MirrorMaker to replicate the topic to a different cluster.
We pause the service and check the current offset for the 2 streams (the one in the old cluster and the one in the new cluster).

In step 3, these offsets will be different, specifically, the offset from the old cluster will be the last message the service managed to commit an offset for. And the new topic will have as an offset the value: last_checkpoint_before_upstream_offset + 1.
Thus, when we restart the service (and make it consume from the new topic), it will re-process all messages from last_checkpoint_before_upstream_offset + 1 until downstream_offset. Isn't this a problem considering that Mirror Maker is providing Exactly Once Semantics for committing messages ?
This behaviour was verified by looking at the output of the kafka-consumer-groups script.

Hi @gkousouris Thanks for asking!

Thus, when we restart the service (and make it consume from the new topic), it will re-process all messages from last_checkpoint_before_upstream_offset + 1 until downstream_offset. Isn't this a problem considering that Mirror Maker is providing Exactly Once Semantics for committing messages ?

Your understanding of the offset translation (post-KAFKA-12468) is correct, and I would expect re-processing of messages downstream after a fail-over. I also understand that this doesn't satisfy "exactly once semantics" for some definition, because it allows for re-delivery of the same message to the same "application", when that application uses multiple Kafka clusters.
MirrorMaker2 currently provides "exactly once semantics" for replicating data, but not for offsets. I believe this is captured by the "MirrorSourceConnector" declaring that it supports EOS, but the"MirrorCheckpointConnector" does not. This means that when you replicate a topic with EOS mode, and use read_committed on the downstream topic from the beginning, EOS would mean that you read each record present in the upstream topic exactly once. When you instead start reading at the committed downstream offset, you may have records delivered downstream that have already been committed upstream.
This is not just caused by the offset translation that this PR implements, it's a limitation of the asynchronous offset translation that MirrorMaker2 uses. Consider this sequence:

MirrorCheckpointTask syncs offsets
MirrorCheckpointTask sleeps
MirrorSourceTask replicates some records
The upstream consumer group consumes upstream records and commits offsets to the upstream group
The downstream consumer group starts reading the topic with the stale offsets in the downstream group

Thanks for doing your due diligence on the claims of "exactly once semantics", and I hope that you can still make MirrorMaker2 work for your use-case. I suspect that EOS semantics across multiple Kafka clusters is a much larger effort than just changing the offset translation logic :) If you have a Jira account, please consider opening a ticket about this shortcoming.
Thanks!

Thanks a lot for your reply! I will look into creating a Jira account and creating a ticket for this.
I should have mentioned that for we were planning on using Mirror Maker to migrate the topic a service reads from from cluster A to cluster B. So we would not be limited by the asynchronous offset translation that MirrorMaker uses, since we would be:

Mirroring data from old topic in cluster A to new topic in cluster B.
Stopping the service and waiting for the consumer offset of the last message on cluster A to be replicated on the new topic on cluster.
Restart the service to read from the new topic.

We would have hoped that the offset would be translated exactly at some point, and would let us to seamlessly start consuming from the same point it was last stopped.
MirrorMaker seems like a great use case for us, but this might be a bit of a blocker. Using the old offset translation version before this PR could perhaps work if we were to disable EOS (to get rid of the transactional messages).
Otherwise, the only solution I can think of is the hacky approach of reading the offsets and trying to decipher what message to read on the application-side, which seems brittle.
Would you perhaps recommend a different approach to not re-process a message twice ?

@gkousouris Thanks for sharing your use-case. I think you are right to look towards MM2 for this sort of translation, and I think it's unfortunate that it isn't straightforward. The current offset translation doesn't "converge" for consumer groups which are inactive due to memory limitations, but for a single-shot migration use-case, that's not good enough.
Are you able to stop the producers to the upstream topic, and let the consumers commit offsets at the end of the topic before performing the migration? If you set offset.lag.max very low, MM2 should be able to translate offsets at the end of the topic.

Otherwise, the only solution I can think of is the hacky approach of reading the offsets and trying to decipher what message to read on the application-side, which seems brittle.

Yeah, if you want to get a 100% precise translation away from the end of the topic and don't want to modify MM2, you're going to need to "synchronize" the two topics and figure out which messages line up. Between offset.lag.max, the syncs topic throttle semaphore, and the OffsetSyncStore, a lot of intermediate offset syncs get discarded and the precision of the translation decreases significantly. If you let MirrorMaker2 perform a rough translation that you later refine with a custom script, you probably only need to compare a few hundred record checksums for each topic-partition-consumer-group. This would also allow you to compensate for the skipped offsets that EOS mode produces.
I think you could make such a script reliable enough for a one-off migration, with some manual spot-checking to make sure it doesn't do anything too incorrect.
If you're willing to hack on the MirrorMaker connectors, you could disable the throttling semaphore, the offset.lag.max parameter, and implement the full-depth OffsetSyncStore to get perfect translation. I don't think we could add those to mainline MM2 without a configuration, but you are certainly welcome to temporarily fork MM2 to get the job done.


Support unittesting metaconfig services with element option(s) set

Related to https://github.com/quattor/configuration-modules-core/pull/669

@jrha new build tools before 16.2 will be welcome. new version is not needed for https://github.com/quattor/configuration-modules-core/pull/669 itself, but it is for unittesting any services that are part of metaconfig that rely on element options.


Digital Art buying and selling APP - Milestone 2,3,4,5

Link to the original project proposal.
https://github.com/thenewboston-developer/Projects/issues/177
Link to most recent milestone payout request.
https://github.com/gautamkmahato/Digital-art-marketplace/tree/master
Milestone Description
Milestone 1 will be the design of the website layout and the frontend. In Milestone 2 the Front-end to be done and starting of backend architecture and design. In Milestone 3 we will be adding some extra features to the website. In Milestone 4 Back-end to be done and in Milestone 5 Testing and deployment will be done.
Requested Payment Upon Completion
Objective 1: 30th April 2021
complete the website design
completion of the frontend part like the design of the website layout, login/signup, creating the profile, uploading, images, navigation, landing page, dashboard, profile, etc.
ESTIMATED_AMOUNT_OF_COINS - 15,000 coins
Objective 2: 1st  June 2021
Backend architecture and designs
complete the basic backend functionality of the website like login/signup, creating the profile, uploading, displaying images category-wise.
creating a database and connecting it with the backend etc.
ESTIMATED_AMOUNT_OF_COINS - 25,000 coins
Objective 3: 5th July 2021
Adding additional features like biding functionality, payment service.
integration of blockchain/ TNB coin, etc.
Completion of Frontend and Backend
ESTIMATED_AMOUNT_OF_COINS - 35,000 coins
Objective 4: 15th August 2021
adding some extra features to the websites
Completion of Frontend and Backend
ESTIMATED_AMOUNT_OF_COINS - 40,000 coins
Objective 5: 15th  September 2021
Testing
deploying the website.
ESTIMATED_AMOUNT_OF_COINS - 45,000 coins
Project Reviewer
NAME OF PROJECT REVIEWER HERE
Project Wallet
fc8e5f6209a1861e3456ccec70a83217e01c036df9dc8c005e636f9f1c0beb2a

Closing this for now because this milestone proposal is related to this project: https://github.com/thenewboston-developers/Projects/issues/177


Microsoft Edge on iPad is supported ?

I can't find description about device support. I tried to access to Azure ML studio from iPad. But I got message "Unsupported browser". Please document about what device is supported clearly.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f8188c6a-2184-b7cb-08a8-5487b90a9fa3
Version Independent ID: a8e56c10-1fa4-5a2b-92c1-39cd5ccb0b73
Content: What is Azure Machine Learning studio?
Content Source: articles/machine-learning/overview-what-is-machine-learning-studio.md
Service: machine-learning
Sub-service: core
GitHub Login: @PeterCLu
Microsoft Alias: peterlu

@konabuta Thanks for the feedback! We will review and update accordingly.

@konabuta  Thanks for the feedback.

I tried to access to Azure ML studio from Microsoft Edge browser on iPad device. While Chrome browser on iPad device works.

@PeterCLu Can you please confirm on the Azure ML studio from Microsoft Edge browser on iPad device.

Hi @konabuta, thanks for the report!
I've reported this issue to the product team. They are now aware of this platform limitation, however they do not plan to add Edge support on Edge.
I've added a documentation work item to document platform support in the future. Our documentation team will prioritize this request along with other requests. Thanks so much!
#please-close


Pop contains invalid chars while provisioning in HomeKit Example (IDFGH-5397)

Environment

Development Kit:     ESP32-DevKitC
Kit version (for WroverKit/PicoKit/DevKitC): DevKitv1
Module or chip used:  ESP32-WROOM-32
IDF version (run git describe --tags to find it):
v4.2
Build System:        idf.py
Compiler version (run xtensa-esp32-elf-gcc --version to find it):
8.4.0
Operating System:     Windows
(Windows only) environment type: PowerShell
Using an IDE?: No
Power Supply:         USB

Problem Description
Pop contains invalid chars while provisioning in HomeKit Example.
Ex.:
https://rainmaker.espressif.com/qrcode.html?data={"ver":"v1","name":"PROV_8BEA1C","pop":"��X57�g","transport":"softap"}
I (1720) app_wifi: Provisioning Started. Name : PROV_8BEA1C, POP : ��X57�g
Expected Behavior
POP should be alphanumeric charset.
Actual Behavior
POP is invalids chars.
Steps to reproduce
Flash Homekit example.
Code to reproduce this issue
No change in code.
Debug Logs
https://rainmaker.espressif.com/qrcode.html?data={"ver":"v1","name":"PROV_8BEA1C","pop":"��X57�g","transport":"softap"}
I (1720) app_wifi: Provisioning Started. Name : PROV_8BEA1C, POP : ��X57�g
Log: esp.txt
Debug log goes here, should contain the backtrace, as well as the reset source if it is a crash.
Please copy the plain text here for us to search the error log. Or attach the complete logs but leave the main part here if the log is *too* long.

Other items if possible

[ sdkconfig.txt ] sdkconfig file
[ homekit_switch.zip ] elf file in the build folder
[ ] coredump (This provides stacks of tasks.)

Created here by mistake. Wanted to create in RainMaker repo.


[Error: im_local_dmask: nonsense mask parameters] When using convolve()

I am trying to do a simple sobel edge filter with the below code. When I trigger the request I get the following error [Error: im_local_dmask: nonsense mask parameters] and nothing is returned in the buffer. What am I doing wrong?
app.get('/', function (req, res) {
    var requestOptions = { encoding: null, uri: "https://hd.unsplash.com/photo-1470214304380-aadaedcfff84" };
    var imgReq = request(requestOptions, function (error, response, body) {
     sharp(body)
         .resize(1000)
         .convolve({
             width: 3,
             height: 3,
             kernel: [-1, 0, 1, -2, 0, 2, -1, 0, 1]
        })
        .webp()
        .toBuffer(function(err, buffer) {
            console.log(err);
            res.end(buffer, 'binary');
        });
    });
});

Hello, I wonder if this is because the default "scale" for a mask is calculated by adding up all the kernel values, which in this case results in zero. Are you able to try adding scale: 1 to the convolve parameters to work around this?

Perfect! That worked. Thanks so much. Maybe you want to change the docs to include the scale: 1
http://sharp.dimens.io/en/stable/api/#convolvekernel
Thanks again!

Thanks for confirming there's a (temporary) workaround. Rather than make the API more complex than it needs to be, I'm inclined to automagically clamp the scale to a minimum value of 1; leave it with me.

Commit af9d09f clamps the kernel scale to a minimum value of 1, plus adds the documented horizontal Sobel operator example as a test. Thanks for reporting this!

This fix is in v0.16.1, now available via npm.


Added style prop

name: 'Pull request'
about: A new pull request
New Pull Request
Checklist

[x] I am not disclosing a vulnerability.
[x] My code is conform the code style
[x] I have made corresponding changes to the documentation
[x] I have updated Typescript definitions when needed

Issue Description

Related issue: #851
Solution Description

Added style prop to:

Card component,
Button component,
and Typography.

@dani69654 why do you want this on card? i'm slightly worried lol
i considered it for Button, Input and Typography to cut out all the styled divs for margin in Admin but i think i have a 'safer' solution


Need to update guidance for multiple workspaces

Multiple workspaces are blocked by default: customer can only create one workspace.
Workspaces are moving to support multi-forest deployment
Need to remove this guidance :
In Azure ATP, you have the ability to manage and monitor multiple workspaces. This is especially helpful if you want to create a demo workspace and a test workspace in which you can POC Azure ATP before rolling it out to your whole organization. This is also needed to support deployments with multiple forests. A single workspace can only monitor multiple domains from a single forest.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6b1aaccd-7940-5e1a-5a33-9fad4b8e6377
Version Independent ID: 4e32d409-fc1a-d16c-756b-91742a9b735a
Content: Install Azure Advanced Threat Protection - Step 1
Content Source: ATPDocs/install-atp-step1.md
Service: azure-advanced-threat-protection
GitHub Login: @rkarlin
Microsoft Alias: rkarlin

Thank you for your feedback. Multi-forest support and workspace related guidance have been updated.


File Downloader: textfile.txt misses some content

Please add some content to the file to check for.

Ok, looks like there is not textfile.txt after a reset.
So i vote for some additional default files

textfile
csv file
xls


Bi-Directional AAD Provider Integration (Cloud)

[User Story]
AAD Integration

Epic Title
Active Directory Integration ([https://github.com/hyperledger/sawtooth-next-directory/issues/35])

User Story Title
Graph API Provider

T-shirt size:
It's HUUUGEE!

Description
Build API integration for Azure AD (Cloud). Collaboration with Microsoft team

Business Need (the why):
ensure Azure AD functionality for NEXT

Definition of Done
Bi-directional data sync with conflict resolution. Integration will resolve any user conflicts.

migrated to new tmobile/sawtooth-next-directory repo


Client.on('response') support

In the getting started we use:
support for client.on('response', function(error, response, body) in the stream_js client?
This no longer works. It's used to show error messages when you make a broken API call.

this is working correctly


fix example of the option in the proposals (the delimiter is a space, not a comma)

We changed the delimiter from a comma to a space in #11, but forgot to update the proposals.

Excellent! Closes #13
Looks ready to merge

Thanks, the new version is released!


Fix memory leak

ZeroClipboard was not properly destroyed when the directive was destroyed. I detected the problem while profiling the memory.
Calling ZeroClipboard.destroy() on the $destroy event handler fixes the problem.

Thanks, looks great!


fix(client): Fix print media queries for certificate

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #46965

The issues with the certificate printing layout were originated from two points:

Some styles coming from bootstrap css that were interfering with the layout:

The media queries had the screen media type, making the layout lose responsiveness when printing

This PR fixes these issues by overriding the bootstrap css rules and applying the certificate media queries to both print and screen layouts.

Thanks for working on this pr, @amorimr. This is a huge improvement over what we had. Here is what I see on my macOS Monterey 12.2:
Chrome Version 103.0.5060.114 (Official Build) (arm64):

Firefox 102.0.1 (64-bit):

You are in the right direction. It seems like it would need a little more touch ups and it should be good to go.

Thank you for the feedback, @ahmadabdolsaheb. I've made some adjustments to keep the certificate borders and backgrounds visible at all times when printing:
Firefox:

Chrome:

Let me know if there are any additional adjustments to be made!

Thank you for the review and suggestions, @moT01. I've made some adjustments on the margins and on the certificate for printing on landscape mode. Let me know if there are any other issues!

On portrait it looks good:

On landscape we are almost there:

__
  Summary: :hammer_and_wrench: This code needs a fix

On portrait it looks good:

On landscape we are almost there:

__        Summary: 🛠 This code needs a fix

Thank you for the comment, @naomi-lgbt. I've made some adjustments, let me know if there is still any issue.

This looks great on Firefox. I'm still getting a slight overflow on Chrome, but it might be something local to my setup?

I checked the certificate on both browsers and they look great. Solid work @amorimr and congrats on landing your first freeCodeCamp pull-request.


Improved Window focusable method documentation

focusable reference element must be focusable for it to work as you explained in the forum.

You might want to make it more explicit what focusable means:
“focused” element should either has tabindex defined or to be explicitly configured at runtime as element.state.focusable = true; ( STATE_FOCUSABLE if in native code )


Learn.md

Learn repos should have a LEARN.md file to teach student how to build your project step by step. You can explain how to build your project with text, code snippets, images, or even short (5 minute) long video lessons. As the maintainer of a Learn repo, the LEARN.md file requires you to think critically about how to explain the building of your project and how to also make it engaging. We don't expect you to be an expert teacher, but we would like you to reflect on how difficult it was to get to your level of knowledge, and then provide friendly guidance to help other students to learn.
Reference -

Contributing: https://github.com/Girlscript-WOB/Guidelines/blob/main/Project Admins/Contributing.md
Learn: https://github.com/Girlscript-WOB/Guidelines/blob/main/Project Admins/Learn.md
PROPOSAL: https://github.com/Girlscript-WOB/Guidelines/blob/main/Project Admins/PROPOSAL.md

I wouldm like to work on this issue, please assign me this issue

@Harish-2003 Please assign this issue to me.


Support config flags in line.request()

Hi, I noticed that config flags are not supported in the line request.
https://github.com/hhk7734/python3-gpiod/blob/master/py_src/gpiod/libgpiodcxx/__init__.py#L564 at this line the flags should have been taken from the config parameter. It seems that this was possible in the past, but this behavior was removed by this commit: https://github.com/hhk7734/python3-gpiod/commit/23d566b38ed6cc739ef93f6c865179b2d6c62f3b.
Is it possible to have this feature back?
Thank you,
Adrian

Sorry, my mistake. The variable names confused me. All is good. I will close the issue.


Minor Changes

Changed from the InstanceManager to .Instance, to maintain consistency throughout the rest of JunctionRestrictionManager.
The yield speed felt like they were just running the red... I know most people at least hit on their brakes before turning on red, so I removed the multiplication by itself and now it feels much more natural (imo).

Going through Constants.ManagerFactory is the intendend way to access another manager, not via *Manager.Instance. We should fix all other instances but please let us do this in krzychu's repo such that both mods can be synchronized.

I added the PR here so that, if it was merged, it would update your PR within his repo...

Ah, I see. xD I should not have pushed the branch into this repo then.


fix(tabs): hide mat-tab-nav-bar ink bar when no link is active

Currently the mat-tab-nav-bar updates the ink bar location whenever a link becomes active, however this doesn't account for the case where none of the links are active. These changes switch to checking for active links whenever the active state of any of the tabs changes and hiding the ink bar if none of them are active.
Fixes #9671.

Fixes #4436


Sometimes the customizations go haywire

Why is this? Hard to reproduce, but definitely a thing

Probably fixed by https://github.com/benfrancis314/reading_project/pull/112?
I have been playing around and didn't reproduce this.


Add osxlaunchhelper.scpt to the CLRDBG MIEngine package

This checkin adds osxlaunchhelper.scpt to the MIEngine package. With this I
was able to see the 'externalConsole' option work correctly on OSX.

@chuckries please review.

:+1:

I'm pretty sure this will cause the file to be dropped on all platforms, which isn't a big deal. The /any/any/ refer to language (such as cs or vb) and target framework moniker (such as net35 or wp81), so I do not think there's a way to only drop this on mac. LGTM as is.


profile_load: unable to load profile when using thumbnail_image_with_opts

Hi @augustocdias I am unsure if this is related https://github.com/olxgroup-oss/libvips-rust-bindings/issues/11 but I am encountering an issue when trying to us thumbnail_image_with_opts. When I attempt to thumbnail the image I receive the following error
[src/image_utils.rs:158] &error = ThumbnailImageError
[src/image_utils.rs:159] vips_app.error_buffer().unwrap() = "vips__file_read: error reading from file \"\"\nprofile_load: unable to load profile \"\"\n"

Here is a gist of some code and a sample image I found that produces the error
https://gist.github.com/jgould22/8e3a09497e5821d9dcd0575d9c830be0
As far as I can tell I am using libvips 8.10.2 installed with
git clone --depth 1 --branch v8.10.2 https://github.com/libvips/libvips.git && \
        cd libvips && \
        ./autogen.sh && \
        make -j 15 && \
        make install

and the bindings are
libvips = "1.4.2"

Thanks for the bindings and any help with the above error is greatly appreciated!

There's an issue with the bindings that it generates with empty profiles.
the Default implementation for each of this structs uses defaults from the library only when the introspection library provides so, otherwise it uses Rust's default for the type.
This is the case for the profile field. Although it has an internal default value in the libvips, the introspection doesn't say which value is that and the bindings set it to an empty string.
You could solve this by setting the profile field in the opts to "sRGB" for example.

Thank you
Adding the following has fixed my issue, thank you for the help
import_profile:String::from("sRGB"),
export_profile:String::from("sRGB"),


Create pa_spc.py

Summary
Issue: #107
Replace "ISSUE_NUMBER" with the number of your issue so that GitHub will link this pull request with the issue and make review easier.
Checklist
All checks are run in GitHub Actions. You'll be able to see the results of the checks at the bottom of the pull request page after it's been opened, and you can click on any of the specific checks listed to see the output of each step and debug failures.

[ ] Tests are implemented
[ ] All tests are passing
[ ] Style checks run (see documentation for more details)
[ ] Style checks are passing
[ ] Code comments from template removed

Questions
Include any questions you have about what you're working on.

Looks like this is against the wrong repo, closing


Can dark mode be deactivated for all components?

Hello,
I'd like to know if it's possible to disable dark mode for all components on a site where you don't want to have dark mode, because I have a problem when someone has activated dark mode on their computer.

This setting is controlled by tailwind, which uses prefers-color-scheme media feature by default and which has already been considered in the PrimeVue presets.
The quickest way to deactivate dark mode is by setting the "darkMode" value in your tailwind config to make use of manual toggling. Since it's unlikely you would have a generic "dark" class somewhere in a parent container, this should solve it: https://tailwindcss.com/docs/dark-mode#toggling-dark-mode-manually

Hi, In primevue-tailwind's tailwind config, we use the `dark mode: 'class. This way, the dark mode is not linked to your preferred color scheme. You can also check @WatCodeDatCode's suggested link from tailwindcss.
I will be closing this issue, but you can always re-open it.


The Docs pointed to the wrong app.Use  method

this works with .net core 1.0

Thanks - good catch


Regeneration Qiskit API historical docs v0.44-v0.42

Part of #542.
Regeneration of the Qiskit API docs (historical versions) with the methods inlined and the GitHub source links (thank you @Eric-Arellano). Given the number of files modified, this PR only updates the versions 0.44, 0.43, and 0.42. The rest of the versions will be added in a follow-up.

The new plan is to have a dedicated PR for 0.44, then subsequent PRs are 2 versions per batch. We want to closely review things because it's important we get the quality right.


New Updated APKs installing previous versions

General
I uploaded version 0.1.0 for my BETA crypto currency app a week ago and last night uploaded version 0.1.1 for it, the change was that previously my app supported 50 crypto currencies but in the newer version it supported 100.
Problem
The debugger was showing all my updated content, but when I uploaded the release apk on Playstore, it had the same content as the previous version, I tried building the apk again and testing it on my phone, it had the content from 0.1.0 when it was supposed to bue content from 0.1.1.

Are you sure you re-built the release version of the APK using the --release flag? The documentation for the release process is here. It may be possible that you were testing the debug variant and had an old release build from a previous run. If you want to clean your source tree completely, you can try git clean -fdx (beware that this will get rid of all uncommitted changes so want to commit or stash progress before you try this) and try to build a release version again.

Tried this... Still not working...

I often need flutter clean to make the running app reflect changes.

I'm seeing the same issue, repro steps:
flutter create foobar
flutter build apk
edit lib/main.dart to show 2 in the titel bar
flutter run -d <iOS Simulator>

=> title bar shows 2
edit lib/main.dart to show 3 in the titel bar
flutter build apk
flutter install -d <Android device>
launch app on device

=> title bar should show 3, but still shows the original title from flutter create

Finally, I got rid of the problem with the following steps,

run 'flutter clean'
run 'flutter build apk --release'

Closing this as the original issue is resolved.

Also ran into this problem.

I had the same problem for a few days on beta branch. Flutter run updates fine, but --release always needs flutter clean.

@jcollins-g any chance you can help with this flutter tools issue?

Is this the same as https://github.com/flutter/flutter/issues/16604 ? @cbracken @tvolkert

If this only happens on release builds, it's likely the problem that @cbracken fixed a few days ago with #17175.  Is anyone able to reproduce this on an updated master channel?


Better support for dropping attributes

I need to drop multiple types with different attributes and I've noticed that due to RepackImporter.ShouldDrop<TMerge>(), ILMerge supports only one attribute. Why? I suggest supporting multiple attributes separated by a semicolon (this won't collide with any naming conventions and complement MSBuild item collections).
Also, I noticed that the code checks only for fields, methods, types, events and properties. Hence I suggest including module and assembly attributes as well.

To clarify, are you looking to drop the members marked with an attribute or the attribute itself? Ideally if you could provide a tiny C# console app that demonstrates what you want and what your desired behavior is.

I'm thinking of this:
[Attribute1] //drop these
class A { }
[Attribute2] //also drop these
class B { }

If I want to drop both of these attributes, I need to run ILRepack twice with each configuration set to drop one of these two attributes. The bottleneck is in ILRepack/RepackOptions.cs at line 118:
public string RepackDropAttribute { get; set; }

And in ILRepack/RepackImporter.cs at line 248:
// skip members marked with a custom attribute named as /repackdrop:RepackDropAttribute
var shouldDrop = member.HasCustomAttributes
    && member.CustomAttributes.FirstOrDefault(attr => attr.AttributeType.Name == _options.RepackDropAttribute) != null;

If there was, say, string[] instead of string, ILRepack could effectively drop multiple attributes in one go. Since the argument is passed into the program by /repackdrop, I suggested separating multiple attributes via a semicolon. So, in ILRepack/RepackOptions.cs at line 286, this:
RepackDropAttribute = cmd.Option("repackdrop");

would be changed to this:
RepackDropAttribute = cmd.Option("repackdrop").Split(';');

Since semicolons are prohibited in C# namings, there would be no name conflicts in the process. As far as I'm aware there is nothing else stopping this from being implemented.

Thanks, your answer doesn't clearly answer my question:
To clarify, are you looking to drop the members marked with an attribute or the attribute itself?
In other words, for
[Attribute1] //drop these
class A { }

do you expect just the attribute to be deleted, or class A as well?

I want to drop the members marked with the attribute, which is exactly what ILRepack does. But it only drops one attribute per run, so I need to create multiple configurations and them run them all in succession before building. I'm proposing a way to drop all members marked with at least one of the attributes provided. So, to drop both classes A and B, I would be able to write:
<ILRepack Internalize="false" InputAssemblies="$(OutputPath)DLL.dll" TargetKind="Dll" DebugInfo="true" OutputFile="$(OutputPath)DLL.dll" RepackDropAttribute="Attribute1;Attribute2"/>

instead of running ILRepack twice:
<ILRepack Internalize="false" InputAssemblies="$(OutputPath)DLL.dll" TargetKind="Dll" DebugInfo="true" OutputFile="$(OutputPath)DLL.dll" RepackDropAttribute="Attribute1"/>
<ILRepack Internalize="false" InputAssemblies="$(OutputPath)DLL.dll" TargetKind="Dll" DebugInfo="true" OutputFile="$(OutputPath)DLL.dll" RepackDropAttribute="Attribute2"/>

Here you go:
https://www.nuget.org/packages/ILRepack/2.0.32


Auto-reconnect to IRC

The IRC gateway I'm running died last night. It's running under supervisorctl and didn't crash, it just disconnected from IRC (which can happen for various reasons).
Once disconnected, it doesn't seem to attempt to reconnect. It also doesn't crash or give any way for a supervisor to restart it.

https://github.com/thoj/go-ircevent, the library that mattermost uses should be doing this.
I'll have a look at it, but it's not high priority for now.

This is not only an issue with being disconnected from IRC, the same problem is when getting disconnected from Mattermost. This can happen for various reasons every now and then but Matterbridge doesn't seem to even try any re-connect, does it?

It does, if it doesn't could you run with debug, try to reproduce it and open a new issue?

Found some more details about my sub-issue and reported that in #69

We are using version 0.9.2 to connect Mattermost to IRC.  If the network goes down for more than a few minutes, the connect is lost.  Only restarting Matterbridge re-establishes the connection.  Is this the same issue?

I'm making some changes to handle disconnects/reconnects better.
It's in the status branch, https://github.com/42wim/matterbridge/tree/status
Only works for IRC now. Testing is appreciated :-)

Merged in master now, feel free to reopen if issue isn't fixed.

Hi, I am still having this issue as of April 2019. See FOSSRIT/infrastructure#45 for more information. Should I open a new issue for this?

@jwflory I just fixed this in master
Binaries can be found here: https://bintray.com/42wim/nightly/Matterbridge/v1.14.2-10-g380c386#files

fixed in  380c386


fix(client): socket is missing for mariadb client

I got following error:
% # URL is mysql://foo:bar@localhost/siremo?socket=path/to/mysql.sock
% prisma migrate dev --name init
Prisma schema loaded from prisma/schema.prisma
Datasource "db": MySQL database "app" at "localhost:3306"

Error: P1001: Can't reach database server at `localhost`:`3306`

Please make sure your database server is running at `localhost`:`3306`.

I noticed that the socket was not passed for mariadb and fixed it.
Please consider to merge it.

Additionally, it should be noted that In Google Cloud SQL, the Cloud SQL Auth Proxy is commonly used, and a UNIX domain socket is required to use it.
refs. https://cloud.google.com/sql/docs/mysql/connect-run
So, some Google Cloud users may have trouble with this issue. I'm too.

The file you edited is only used in testing, and is not really part of Prisma Client or its Query Engine. I don't think this will have any effect on usage of Prisma Client at all unfortunately.
Socket usage should work with Prisma Client and MySQL.

I see. Sorry, I misunderstood.
I tested again in a different way and found that the problem does not occur when connecting directly to MySQL residing on localhost without using the Cloud SQL Auth Proxy.
I will close this issue for now. Thank you very much.

I think this problem is similar to #11171


OCR runtime error -  Attempted to read or write protected memory.

I'm trying OCR sample for C#, when I try with spanish language a runtime error occurs.
{"Attempted to read or write protected memory. This is often an indication that other memory is corrupt."}
Code:
public OCRForm()
      {
         InitializeComponent();
         InitOcr(@"", "spa", OcrEngineMode.TesseractCubeCombined);

      }
private void InitOcr(String path, String lang, OcrEngineMode mode)
      {
         try
         {
            if (_ocr != null)
            {
               _ocr.Dispose();
               _ocr = null;
            }
            _ocr = new Tesseract(path, lang, mode);
            languageNameLabel.Text = String.Format("{0} : {1}", lang, mode.ToString());
         }
         catch (Exception e)
         {
            _ocr = null;
            MessageBox.Show(e.Message, "Failed to initialize tesseract OCR engine", MessageBoxButtons.OK);
            languageNameLabel.Text = "Failed to initialize tesseract OCR engine";
         }
      }

Guide for reporting issues
Please use the forum.

@Ivanzo1987, if the same runtime error also occurs with the original Tesseract executable, then reporting that as an issue would be correct. In this case we'd need additional information (Tesseract version, platform, command line used, copy of image).

Emgu Version 3.1

De: Stefan Weil<EMAIL_ADDRESS>Enviado: miércoles, 30 de noviembre de 2016 16:47
Para: tesseract-ocr/tesseract
Cc: Ivanzo1987; Mention
Asunto: Re: [tesseract-ocr/tesseract] OCR runtime error - Attempted to read or write protected memory. (#513)
@Ivanzo1987https://github.com/Ivanzo1987, if the same runtime error also occurs with the original Tesseract executable, then reporting that as an issue would be correct. In this case we'd need additional information (Tesseract version, platform, command line used, copy of image).

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/tesseract-ocr/tesseract/issues/513#issuecomment-263907829, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqJ0c5yM6KPenV405L5pTOtrmY84Lkqks5rDZqYgaJpZM4LAHHn.

Emgu is third party software, not the original Tesseract, so you will have to discuss or report any issue there.

Ok, where I can download tesseract OCR for C# ??

De: Stefan Weil<EMAIL_ADDRESS>Enviado: miércoles, 30 de noviembre de 2016 17:11
Para: tesseract-ocr/tesseract
Cc: Ivanzo1987; Mention
Asunto: Re: [tesseract-ocr/tesseract] OCR runtime error - Attempted to read or write protected memory. (#513)
Emgu is third party software, not the original Tesseract, so you will have to discuss or report any issue there.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/tesseract-ocr/tesseract/issues/513#issuecomment-263915331, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqJ0ZXzY3oNHBChfh1Ls04F_ULqedahks5rDaA4gaJpZM4LAHHn.

Tesseract is written in C++ (not C#). The Tesseract wiki has hints where you can get a binary installation.

Ok, I will try it, thanks

De: Stefan Weil<EMAIL_ADDRESS>Enviado: miércoles, 30 de noviembre de 2016 17:24
Para: tesseract-ocr/tesseract
Cc: Ivanzo1987; Mention
Asunto: Re: [tesseract-ocr/tesseract] OCR runtime error - Attempted to read or write protected memory. (#513)
Tesseract is written in C++ (not C#). The Tesseract wikihttps://github.com/tesseract-ocr/tesseract/wiki/Downloads has hints where you can get a binary installation.

You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/tesseract-ocr/tesseract/issues/513#issuecomment-263919191, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAqJ0fUPF2y5udPmN3gQ2Qdtu9RK8yd6ks5rDaMzgaJpZM4LAHHn.


fix: Set resource version for CRs

Set resource version for all monitoring.go GVRs. Earlier, this was not done by the machinery responsible for it, which caused manifest applications that had no resource version present to encounter the following error:
metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

This patch addresses that regression, which was introduced originally in #1575.

EDIT: I believe this is not a regression since 4.17 did not include #1575.

The code change looks good to me, pending the resolution of https://github.com/openshift/library-go/pull/1823#discussion_r1793445225

/approve
/assign @p0lyn0mial
for lgtm

@rexagod In general, I have a couple of suggestions for you to consider:

It seems that ApplyUnstructuredResourceImproved method could be made unexported/private. Do you agree?

The calling functions of the applyUnstructuredResourceImproved pass nil for defaultingFunc and equalityChecker, so I would like to remove these arguments from applyUnstructuredResourceImproved.

I would like to remove the Fail-fast if the resource versions differ code, since we merge into the existing object, not the required one.

I would like to remove an extra copy from the ensureGenericsSpec method.

I would like to create a new method, EnsureObjectMetaForUnstructured, where EnsureObjectMeta is currently located.

I would like to move the code that merges metadata into this new function (code from line 105 to line 144).

What do you think?

It seems that ApplyUnstructuredResourceImproved method could be made unexported/private. Do you agree?

The motivation to move to an unstructured.Unstructured approach was to accommodate for resources that do not have their own Apply<Resource> methods exposed. The ones in the library right now are only there for the sake of backwards-compatibility.

The calling functions of the applyUnstructuredResourceImproved pass nil for defaultingFunc and equalityChecker, so I would like to remove these arguments from applyUnstructuredResourceImproved.

To refer to my same reasoning as above, those are nil for the exposed resource-dedicated methods as that was the original behavior. However, when using ApplyUnstructuredResourceImproved for either (a) unknown resources (ones that do not have a dedicated method here), or (b) more control over the caching mechanism, those are still needed to leverage what the machinery has to offer. For instance, the exposed methods do not use a cache as well, to preserve older behavior, but nonetheless, something the user may still need.

I would like to remove the Fail-fast if the resource versions differ code, since we merge into the existing object, not the required one.

Done.

I would like to remove an extra copy from the ensureGenericsSpec method.

Done.

I would like to create a new method, EnsureObjectMetaForUnstructured, where EnsureObjectMeta is currently located. I would like to move the code that merges metadata into this new function (code from line 105 to line 144).

Done.

Thank you for the thorough review, @p0lyn0mial. I'd like to point out that I've refactored monitoring.go code to make it more consistent and readable (for instance, the ensureGenericSpec signature changes), in addition to the much-needed refactors you pointed out owing to the older code patterns that existed pre-#1575.

ApplyUnstructuredResourceImproved
Is ApplyUnstructuredResourceImproved method used outside of the library-go? If not, I would make it private and remove the nil parameters. I would only add them when they are truly needed.
Also, could you please remove fixup! regression: from the commit messages?

@rexagod  Thank you very much for your work - much appreciated!
would you be willing to open a PR similar to this one so we can be sure that the changes introduced in this PR are working as expected?

Raised https://github.com/openshift/cluster-kube-apiserver-operator/pull/1753.

ok, the ci/prow/e2e-aws-ovn-upgrade on the PR was green. Please rebase this PR and let's merge it.

@rexagod I think you need to add GO_TEST_PACKAGES :=./test/e2e-monitoring/... to the test-e2e-monitoring target.

mhm, I think that ci/prow/e2e-monitoring isn't running TestResourceVersionApplication.

Is it when you run make test-e2e-monitoring on your local machine ?

@p0lyn0mial The target was a placeholder to get the release PR in. I've amended that to run the tests now. However, test-e2e-encryption needs a similar refactor as that's still a placeholder IIUC.

/test e2e-aws-encryption

test-e2e-encryption is not a placeholder, the rule in invoking https://github.com/openshift/build-machinery-go/blob/5725581bdf8f467ee8fbd394f081a10df670c1ec/make/targets/golang/test-unit.mk#L12 on https://github.com/openshift/library-go/tree/master/test/e2e-encryption.

Ah, good to know why we specify GO_TEST_PACKAGES, however, I'm not sure how frequently this runs? /test e2e-aws-encryption also returns a make: Nothing to be done for 'test-e2e-encryption' at the moment, which led me to believe it to be a placeholder.
I ask this because I believe we would've caught the issue on master if it did anytime soon (PTAL below).
┌[rexagod@nebuchadnezzar] [/dev/ttys003] [master]
└[~/repositories/work/library-go]> gd upstream/master
┌[rexagod@nebuchadnezzar] [/dev/ttys003] [master]
└[~/repositories/work/library-go]> go vet ./...
# github.com/openshift/library-go/test/e2e-encryption
# [github.com/openshift/library-go/test/e2e-encryption]
vet: test/e2e-encryption/encryption_test.go:85:119: not enough arguments in call to genericoperatorclient.NewClusterScopedOperatorClient
        have (*rest.Config, schema.GroupVersionResource)
        want (clock.PassiveClock, *rest.Config, schema.GroupVersionResource, schema.GroupVersionKind, genericoperatorclient.OperatorSpecExtractorFunc, genericoperatorclient.OperatorStatusExtractorFunc)

let's try with https://github.com/openshift/library-go/pull/1836 - i have run it on my local machine:
❯ make test-e2e-monitoring
go test -mod=vendor -race ./test/e2e-monitoring/...
ok  	github.com/openshift/library-go/test/e2e-monitoring	(cached)

i'm waiting for the ci job to finish, if it works then we need something similar for the e2e-aws-encryption.

/test e2e-aws-encryption (and local invocation) returns a make: Nothing to be done for 'test-e2e-encryption' at the moment

Oh I see, I missed that. If you add the following line, it should fix it:
diff --git a/Makefile b/Makefile
index 43276499c..b466e269a 100644
--- a/Makefile
+++ b/Makefile
@@ -24,6 +24,7 @@ verify-podnetworkconnectivitychecks:
 	$(MAKE) -C pkg/operator/connectivitycheckcontroller verify

 test-e2e-encryption: GO_TEST_PACKAGES :=./test/e2e-encryption/...
+test-e2e-encryption: test-unit
 .PHONY: test-e2e-encryption

 test-e2e-monitoring

It must have been missed in the past

Hmm actually it reminded me of something and it looks like I had a PR for it that never merged: https://github.com/openshift/library-go/pull/1470

@rexagod https://github.com/openshift/library-go/pull/1836 worked, we have two options we can merge it or you can pull the changes to your pr.

i'm waiting for the ci job to finish, if it works then we need something similar for the e2e-aws-encryption.

It seems that on a fresh cluster the test fails due to the absence of encryptiontests.operator.openshift.io CRD, which I'm not sure which operator provides?

i'm waiting for the ci job to finish, if it works then we need something similar for the e2e-aws-encryption.

Since my contextual test-e2e-encryption-knowledge is limited and the fact that we've identified at-least a couple of issues [1][2] around the test, I'd reckon it'd be better if one of the previous authors with some spare cycles can take a look.
Otherwise, I'll revisit this as soon as I can find some.

Since my contextual test-e2e-encryption-knowledge is limited and the fact that we've identified at-least a couple of issues [1][2] around the test, I'd reckon it'd be better if one of the previous authors with some spare cycles can take a look.

@dgrisonnet has a pr for it - https://github.com/openshift/library-go/pull/1470

/lgtm
@rexagod many thanks for the PR.


Reduce size of CSVs, GIFs, and Images

This PR addresses the issue of GIFs: https://github.com/plotly/dash-sample-apps/pull/648
The code used to significantly reduce GIFs (resizing, heavy optimization):
files=$(find apps/dashr* -type f -name '*.gif')

for file in $files; do
    gifsicle -O3 --colors 128 --resize-fit-height 800 -o $file $file
done

With a quick filtering find/grep, we get a total of 500MB in csv files:
find ./ -type f -name '*.csv' -size +5M -exec du -csh {} + | grep total$
# => 498M    total

Could use this python code to compress some of the csv files:
import os
import gzip
from glob import glob
import shutil

from tqdm.auto import tqdm

def convert_to_gz(file, min_size=5*1e6):
    if os.path.getsize(file) > min_size:
        with open(file, 'rb') as f_in:
            with gzip.open(file+'.gz', 'wb') as f_out:
                shutil.copyfileobj(f_in, f_out)

files = glob('**/*.csv', recursive=True)
len(files)

for file in tqdm(files):
    convert_to_gz(file)

Which is significantly smaller:
find ./ -type f -name '*.csv.gz' -exec du -csh {} + | grep total$
# => 101M    total

For images, we could convert pngs to jpg. tho we'd only save <100MB in space:
find ./ -type f -name '*.png' -exec du -csh {} + | grep total$
# => 55M     total

To remove intermediary frames in GIFs:
gifsicle -U $file `seq -f "#%g" 0 2 370` -O2 -o $file

@HammadTheOne Hey can you read csv.gz files just like you read csv files in R? E.g.
EURUSD <- fread(file.path("pairs", "EURUSD.csv.gz"))

@xhlulu  We definitely can read csv.gz with fread in R. Based on the docs the only change we would need to make is adding the R.utils dependency to our init.R file for the sample app.

Compressed files with extension ‘.gz’ and ‘.bz2’ are supported if the R.utils package is installed.

Closed in https://github.com/plotly/dash-sample-apps/pull/648 and https://github.com/plotly/dash-sample-apps/pull/659


请问一下，您是怎么实现多码率的，经过我测试，可以自适应码率，求解答，谢谢！

Any solution here? I wondering if the framework includes Adaptive Bitrate feature for RTMP protocols.

Yes, it's support adaptive bitrate. And very low latency.

@xxycode How can I test this, or how to fetch an ABR RTMP stream?


Redirect / to /ui/

If the root handler detects that the /ui/ handler is installed, and the request path is exactly "/" (and not even "/?") then redirect to /ui/ for a warmer welcome.
The UI links to /status/ nowadays anyway.

While we're at it, can we make the redirect target URL an (optional) parameter in the root handler's config? With "/ui/" as the default value.
I'd like my "/" to redirect to a publisher. (Better still, I'd like "/" to be handled by an app handler rather than the root handler, but that might need a bit more work.)

https://camlistore-review.googlesource.com/4967
@saljam SGTM, but I haven't addressed it in that CL.
Feel free to do it in another CL? :-)

89488a3cf4a326b56015885973467cae17982b7e
@saljam feel free to reopen if/when you want to give a go at your proposition.


[Changed] Publish NPM on tag push

:tophat: What?

change the publish NPM package trigger to tag push
update description comments
added tests step before publish npm package

:thinking: Why?
Improve the workflow

Are we sure we want to avoid making releases in Github?

Are we sure we want to avoid making releases in Github? @raulmoyareyes

We can add this until we get semantic versioning:
https://github.com/marketplace/actions/github-create-tag-release


Add Vivado logs to captured results for debugging purposes.

Also fixes script when generating install collection lists.
Cleans up third_party, .git and env directories prior to kokoro doing artifact collection.

LGTM - My only concern is that there might be too many log files?

I don't think that'll be an issue.  We'll see.  We should add a check that the working directory has less than 100 GB total.  This was likely causing the problems discussed in https://github.com/SymbiFlow/symbiflow-arch-defs/issues/1768#issuecomment-725648866 and https://github.com/SymbiFlow/symbiflow-arch-defs/pull/1747
It is annoying that kokoro doesn't generate a log when this error is occuring :(

We should add a check that the working directory has less than 100 GB total.

I added a check to make sure the working directory has less than 45 GB, which is hopefully a good limit.  I also printed the size of the working directory so we can trace how big it has gotten.


fix: update ro templates

https://trello.com/c/5My9r4Bx/1727-2-change-detail-pages-for-presentation

@kmaid @jhonny-m I'm afraid I have to disagree, tbh I think it's clearer breaking the logic into two different templates.
But since you both agree on this, I reverted it and just updated the isGrantDocument variable,


Add subscription merging to docs

Adds a subscription merging example to the docs, introduced in v1.17.11.

Thank you very much! 😀


[Question] How to effectively use the intmm.py and intmm_triton.py

Hello AO Team! Thanks for this amazing package. I am extremely interested in using the Integer MatMul Kernels on A100 GPUs.
I wrote a simple matmul operation to see the effectiveness of the same.
import os
import torch
from torchao.kernel.intmm import int_matmul
from tqdm import tqdm

# print(f"Is Auto Tuner enabled: {bool(os.getenv('TORCHAO_AUTOTUNER_ENABLE', 0))}")
# print(f"A100 path: {os.getenv('TORCHAO_AUTOTUNER_DATA_PATH', None)}")

device = "cuda:0"

a = torch.rand(2048, 2048).to(torch.int8).to(device)
b = torch.rand(2048, 4096).to(torch.int8).to(device)

print(f"a: {a.shape}, a.dtype: {a.dtype}, a.device: {a.device}")
print(f"b: {b.shape}, b.dtype: {b.dtype}, b.device: {b.device}")

for _ in tqdm(range(100000)):
    c = int_matmul(a, b)
print(f"c: {c.shape}, c.dtype: {c.dtype}, c.device: {c.device}")

print("Using Float32 to do it")
a = a.to(torch.float32)
b = b.to(torch.float32)

print(f"a: {a.shape}, a.dtype: {a.dtype}, a.device: {a.device}")
print(f"b: {b.shape}, b.dtype: {b.dtype}, b.device: {b.device}")

for _ in tqdm(range(100000)):
    c = torch.matmul(a, b).to(torch.int32)
print(f"c: {c.shape}, c.dtype: {c.dtype}, c.device: {c.device}")

The Int Matmul is almost 1.5x compared to torch.matmul which is really great!

My question is, Am I using it right? At least looking through the source code it looks like I am not going via intmm_triton.py as I have not enabled the TORCHAO_AUTOTUNER_ENABLE. But when I enable it, it seems to take a long time to process. I even tried setting the TORCHAO_AUTOTUNER_DATA_PATH manually to the downloaded data_a100.pkl as I have an A100 GPU. I am kinda confused here on how should I use this triton kernel. Any help is appreciated. Also I want to use the int_scaled_matmul and it looks like running it without TORCHAO_AUTOTUNER_ENABLE  completely eliminates the memory benefits I get from fusing the scales.

Hey @balaabhijit - the A100 data has settings for specific shapes. Your shape might not be in it yet. By using the autotuner, you'll step through all the shapes and eventually end up with an updated .pkl file that has the best config for your specific input shape in it. This takes a long time, but if you're working with static shapes can help performance.


Improve settings for all users (Admin panel update)

User story
As a Collective, user, or fiscal host, I want to have all of my settings and management features in one place so that is easier for me to understand all the things I can do inside Open Collective
Best solution for this problem
Evolving the settings page to an admin panel for all types of users. In the case of fiscal hosts, they will have inside their admin panel all of the functionalities originally conceived for the former fiscal host dashboard (see #2896)
MVP
Migrate current settings to this new structure, we might want to start with fiscal hosts since the dashboard and host features are currently developed

Metrics
Less support tickets, can't think of another metric right now
Documentation
Fiscal hosts
See #2665

Help us sort Collectives' settings!
We would like to simplify the options users have available, Collectives' panel sidebar has too many elements now and we want to nest some of them in categories See example below

Current state
Proposal

Feel free to copy the next 🗒board and take a stab at this if you have another proposal for organizing these elements.
The final sorting/nesting decision will impact how fiscal host settings would look as well.
@opencollective/community @alanna @znarf @kewitz @sbinlondon @Betree @piamancini

@Memo-Es Should the navbar be displayed on this settings page?

@Betree No, you go back to the Collective page clicking the 'Exit Admin Panel' button

That makes sense, thank you!

Closing this issue since it's addressed in #3880


typstyle: 0.11.28 -> 0.11.29

Description of changes
Changelog: https://github.com/Enter-tainer/typstyle/blob/refs/tags/v0.11.29/CHANGELOG.md
cc @drupol

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.


User-space backtrace support for programs built without frame pointers

Is your feature request related to a problem? Please describe.
Currently, bpftrace supports generating userspace stack traces only for programs built with frame pointers. The majority of userspace software is built without frame pointers, making userspace stack traces generated by bpftrace not particularly useful.
ema@ariel:~$ sudo bpftrace -e 'uprobe:/bin/ls:format_user_or_group { print(ustack()) }'
Attaching 1 probe...
|
        format_user_or_group+0

        0x56141fa17f60

        0x56141fa17f60
[...]

Describe the solution you'd like

SystemTap supports printing the full userspace stack trace of programs regardless of whether they have been built with frame pointers or not by using debug information to unwind the user-space stack:
ema@ariel:~$ sudo stap -e 'probe process("/bin/ls").function("format_user_or_group") { print_ubacktrace() }'
 0x5627b7abbf60 : format_user_or_group+0x0/0xc0 [/bin/ls]
 0x5627b7ac126a : print_long_format+0x58a/0x9f0 [/bin/ls]
 0x5627b7ac1840 : print_current_files+0x170/0x3e0 [/bin/ls]
[...]

It would be awesome to have the same feature in bpftrace as well.

Overlaps with #1006 I think. Useful but I think it needs kernel support.

I'm guessing that systemtap can do this b/c it creates a custom kernel module and loads it into the system. bpftrace on the other hand is at the mercy of what the kernel provides for stack traces. And kernel developers are not very fond of any dwarf based unwinding (so much that they invented ORC for framepointer-less kernel unwinding)

Another idea: we could do what perf does and copy the stack to userspace for analysis. That'd get around the kernel limitation.

Initial plans here: https://dxuuu.xyz/stack-symbolize.html

Hi, guys, is there any new progress on this topic? There is a lot of programs compiled without frame pointer because of performance considering around us, what could we do to trace thess userspace programs?


serving xrefs: skip over further edges/pages once EdgesReply is full [phab:D461]

Created by schroederc at 2015-08-24 20:51:50:
This change also adds some a few logging statements.

shahms accepted this revision at 2015-08-24 20:55:19

schroederc closed via commit 610d5e85d0d926f0089410290b12b57b2825a214 at 2015-08-24 20:56:11


Which courses did you do to learn this?

Hi,
I like your project and want to do something similar but with a different game. I was wondering what courses / online MOOCs you took
to get to the level to code this?
Thanks

I learned RL from https://www.youtube.com/c/NicholasRenotte

Thank you! Did you just watch the "reinforcement learning" video or did you watch any others? Thanks

Hello?