Unable to build for ladder using wsl2

Describe the bug
Build fails with errors saying "This file was generated by an older version of protoc which is incompatible with your Protocol Buffer headers. Please regenerate this file with a newer version of protoc."
To Reproduce
Steps to reproduce the behavior, for example:

Download fresh install of Ubuntu with wsl2
Run the following commands:

apt-get update
sudo apt-get install cmake
sudo apt install build-essential
sudo apt install mingw-w64-x86-64-dev  g++-mingw-w64-x86-64  gcc-mingw-w64-x86-64 win-iconv-mingw-w64-dev
sudo update-alternatives --config x86_64-w64-mingw32-gcc

Choose option 1

sudo update-alternatives --config x86_64-w64-mingw32-g++

Choose option 1

Clone blank_bot repo (https://github.com/cpp-sc2/blank-bot)
Navigate to said repo
Generate CMake configuration with "cmake -B build -DWSL2_CROSS_COMPILE=ON"
Compile project with "cmake --build build -j"
Observe numerous errors similar to:

/mnt/c/Users/hocke/OneDrive/Documents/blank_bot/build/_deps/cpp_sc2-build/generated/s2clientprotocol/debug.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
   17 | #error This file was generated by an older version of protoc which is
      |  ^~~~~
/mnt/c/Users/hocke/OneDrive/Documents/blank_bot/build/_deps/cpp_sc2-build/generated/s2clientprotocol/debug.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
   18 | #error incompatible with your Protocol Buffer headers. Please
      |  ^~~~~
/mnt/c/Users/hocke/OneDrive/Documents/blank_bot/build/_deps/cpp_sc2-build/generated/s2clientprotocol/debug.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
   19 | #error regenerate this file with a newer version of protoc.
      |  ^~~~~

Expected behavior
The build should finish without errors.
Desktop (multiple version means multiple attempts with different version):

OS / Distros: Ubuntu 20.04.6/22.04.3
cpp-sc2 Version: 1.4.1
StarCraft II version 5.0.12
GNU Make versions: 4.2.1/4.3
CMake version: 3.16.3/3.22.1
gcc/g++ version: 9.4.0/11.4.0
protoc versions: None/3.12.4/3.23.4/3.23.1

Additional context
Tried with different combinations of the flags -DBUILD_FOR_LADDER=ON -DSC2_VERSION=5.0.12 -DSC2_VERSION=4.10.0 but no difference.
Also tried with Ubuntu 24.04 but the package "mingw-w64-x86-64-dev" does not exist for that version of Ubuntu so I could not progress past step 2.4
Also tried building this repo instead of the blank_bot repo with the same steps but encountered the same issue.

https://github.com/cpp-sc2/cpp-sc2/blob/master/cmake/assets/protos.cmake
Perhaps it has something to do with this?  Included when cross-compile WSL.
https://github.com/cpp-sc2/cpp-sc2/blob/998ca3b1d40013ebb774120787daade8f5f227ea/CMakeLists.txt#L81

Unfortunately, this issue is not so simple. This is about updated of precompiled toolchain for WSL2. We provide it as a separate artifact.

https://github.com/Nickrader/cpp-sc2/tree/wsl_proto
Was able to copy over protos from working windows build and point to that.
Had some trouble with map paths after getting compilation to work.
As a quick fix, could probably zip what I put in /thirdparty/winProto/ to replace old zip

But we are speaking about WSL, not regular windows. Please, test build inside it.

I am away from my windows machine till end of month.  I'm pretty sure I was running it from WSL, after I copied over the protos from windows (as I had to add a Maps folder to my windows Starcraft installation, where I never had to before)
I can test more by August.
Or if @nanogeo is still around, he could give it a try.
By the time I got done with the clean install of WSL and fixing what I messed up trying to mimic a normal Linux installation...
I just figured I'd get it posted for others to see after I got it to work on my system.  It may be as simple as just taking the protos from a working windows build.
From my reading, it appeared that the two approaches were:
1.  Cross compile on Linux machine using .configure command.  Which I didn't attempt or understand yet.
2.  Compile the protos on Windows and copy it over.  The internet said this was the easier way.

I'll check when I get back.  when I diff what I copied and what builds on linux, they are the same contents.

After investigation:

We should be able to use the *.pb.cc, *.pb.h files from any build (using appropriate protoc) to populate the asset. (I can't test mac, but win and linux have same content)

To build *.pb.cc/h in WSL:

Use a script for a two-step solution.  "when cross compiling, as the executables are built for the target platform and cannot run on the build host"  In our case, protoc.exe gets built, but compiler wants /bin/protoc.

Or possibly could link directly to protoc binary

But if we update the asset, we should be good to go for WSL.

@Nickrader could you please provide an archive with prebuilt protobuf files + sha of cpp-sc2 commit? Ideally the files you tested wsl build.
I'll upload the archive to s3.

https://github.com/Nickrader/cpp-sc2/tree/protos_asset

I was able to link directly to the precompiled protoc binary.
Everything is a mess at the moment, but I should be able to clean it up in a day or two.
So if you prefer this over the asset.  Big thing to watch out for is that downloaded protobuf version matches linked protoc version.  Otherwise will most likely get a runtime error from google.
https://github.com/Nickrader/cpp-sc2/tree/link_protoc

Fixed.


Benchmark KPI's

Have every institute review the list of KPIs, when it is ready. Do the institutes have any KPIs that must be included?
Depends on one review issue (in github) for each institute..

EUMETSAT data services KPIs are and target values are available at: https://www.eumetsat.int/media/45745 (pp. 105-107)
I took a screenshots of the pages here for convenience:

Check Oscar Rolling Requirements (Stuart promised to add link)

Checked https://space.oscar.wmo.int/requirements didn't see anything on availability but data timeliness is well defined for most parameters

WIS2 technical specification discuss availability requirements in "Part VI WIS Metrics" but do not specify them

ThX! I have included these sources in the introduction of the KPI document.


Worked on the Readme file. Added placeholder for later iterations.

Added placeholder text in a few places for things such as the documentation site, an example image of the program in use, and a banner up top.
My plan is to make the banner link to the discord server invite the way the following readme does it. https://github.com/mrousavy/react-native-vision-camera/blob/main/README.md

Overall, I would say: just be sure that the instructions you are providing actually work. If you haven't done part 2 of the rampup (follow the video in the link above) I would strongly encourage you to do so, since this will allow you to understand a lot better what is going on and thus to be able to verify that your README instructions are correct.

closes issue #3


Add CSS exports to package

Certain bundlers such as vite require that es module packages include all exports in the package.json, including CSS imports.

I've pushed attached, slightly simpler patch instead.


Extracting k8s server version

If I have kustomize installed on my system then this command kubectl version --short=true returns following on my system:
Client Version: v1.25.3
Kustomize Version: v4.5.7
Server Version: v1.22.5

And now if I just go ahead with current implementation of: KUBE_VERSION=$(kubectl version --short=true | awk 'FNR == 2 {print}'| awk -F '.' '{print $2}')
then KUBE_VERSION becomes integer 5 and the check fails.
Proposing new way to only capture the Server related line from the output and then proceed ahead.
Signed-off-by: Rachit Chauhan<EMAIL_ADDRESS>
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Type of changes
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Feature/Issue validation/testing:
Please describe the tests that you ran to verify your changes and relevent result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

[ ] Test A

[ ] Test B

Logs

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Checklist:

[ ] Have you added unit/e2e tests that prove your fix is effective or that this feature works?
[ ] Has code been commented, particularly in hard-to-understand areas?
[ ] Have you made corresponding changes to the documentation?

Release note:

This is duplicate


KAFKA-7149 Reduce assignment data size

The PR contains the changes related to Assignment Info re-design where the TopicPartitions are replaced with TaskIDs and GZIP compression is being done on assignmentInfo to reduce the assignment size in version 4.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Just a question. Does the source have to be compilable by Java 7 or is Java 8 language and API permissible? If it is, then several of those loops could be replaced with Java Streams, making the code shorter and more readable... I can volunteer to make this transformation for parts that are already changing if desired...

Are we sure we want to use gzip given how slow it is?

@ijuma What else would you propose? The goal to get good compression is more important for this ticket.
@brary Could you provide any numbers how long it take to (de)compress the metadata?

LZ4 is generally a good option. A newer option is zstd, but it's only supported by newer brokers.

@ijuma Why would we care what brokers support? We only compress rebalance metadata that the broker only passes through. Broker does not even know if the data is compressed or not (and does not need to know).

I was just sharing context in case this was ever used in produce/consume. If it's not, then sure, zstd could be used too. And it's certainly better than gzip.

Regarding the compression type, I'm fine with LZ4 / zstd too -- heard good stuff about zstd especially, compared to gzip. On the other hand, if with the new format we can already effectively reduce the size by large I'm fine with not doing compression as well. I remember @brary once did some benchmarks on the additional effectiveness of compression on top of the new format, v.s. the effectiveness of the new format v.s. the old format. Maybe he can chime in as well?

Regarding the compression type, I'm fine with LZ4 / zstd too -- heard good stuff about zstd especially, compared to gzip. On the other hand, if with the new format we can already effectively reduce the size by large I'm fine with not doing compression as well. I remember @brary once did some benchmarks on the additional effectiveness of compression on top of the new format, v.s. the effectiveness of the new format v.s. the old format. Maybe he can chime in as well?

When I did benchmark, I had 4 source topics with 512 partitions each and 400 machines in the cluster. The assignment size with the current schema came out to be 196 Mb. After changing the assignment info as proposed above without any compression it was 32 MB and with further compression, it was around 8 MB.

When I did benchmark, I had 4 source topics with 512 partitions each and 400 machines in the cluster. The assignment size with the current schema came out to be 196 Mb. After changing the assignment info as proposed above without any compression it was 32 MB and with further compression, it was around 8 MB.

Thanks for reminding me! Did you observe significant CPU usage (or in other words, drops in throughput) using GZIP? Could you retry the same benchmark comparing the throughput v.s. metadata size with LZ4 and zstd as well?

When I did benchmark, I had 4 source topics with 512 partitions each and 400 machines in the cluster. The assignment size with the current schema came out to be 196 Mb. After changing the assignment info as proposed above without any compression it was 32 MB and with further compression, it was around 8 MB.

Thanks for reminding me! Did you observe significant CPU usage (or in other words, drops in throughput) using GZIP? Could you retry the same benchmark comparing the throughput v.s. metadata size with LZ4 and zstd as well?

Sure, will do. In the meanwhile can you help here, I am constantly failing this test case.
kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup failed, log available in /Users/nsb000c/Documents/NaviBrar/production/kafka/core/build/reports/testOutput/kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup.test.stdout
kafka.api.ConsumerBounceTest > testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup FAILED
java.lang.AssertionError: Received 0, expected at least 68
I doubt it can be related to my changes, because I didn't change anything on the kafka.api side.

The build timed out. Retest this please.
@brary Is this ready for another round of review?

The build timed out. Retest this please.
@brary Is this ready for another round of review?

@mjsax and @guozhangwang you can start another round of interview, I feel I have addressed all the code review comments, meanwhile I while try to resolve build timeout.

There is one test in EosIntegrationTest.java -> shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances() which looks wrong to me. The basic wrong assumption in the test is that both objects streams1 and streams2 have same hostInfo(dummy:2142). This leads to partitionsByHostState being overwritten among the two subscriptions. After assignment, I checked the assignment of both streams objects. It is as below.
**streams1** = {KafkaStreams@5791}  streamsMetadataState = {StreamsMetadataState@5824} "GlobalMetadata: [StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}]\nGlobalStores: []\nMy HostInfo: HostInfo{host='dummy', port=2142}\nCluster(id = null, nodes = [], partitions = [Partition(topic = multiPartitionInputTopic, partition = 0, leader = none, replicas = [], isr = [], offlineReplicas = []), Partition(topic = multiPartitionInputTopic, partition = 1, leader = none, replicas = [], isr = [], offlineReplicas = [])], controller = null)\n"   builder = {InternalTopologyBuilder@5836}    allMetadata = {ArrayList@5837}  size = 1    0 = {StreamsMetadata@7019} "StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}"     hostInfo = {HostInfo@7021} "HostInfo{host='dummy', port=2142}"     stateStoreNames = {HashSet@7022}  size = 0     topicPartitions = {HashSet@7023}  size = 2      0 = {TopicPartition@7026} "multiPartitionInputTopic-0"      1 = {TopicPartition@7027} "multiPartitionInputTopic-1"   globalStores = {Collections$UnmodifiableSet@5851}  size = 0   thisHost = {HostInfo@5855} "HostInfo{host='dummy', port=2142}"   clusterMetadata = {Cluster@6907} "Cluster(id = null, nodes = [], partitions = [Partition(topic = multiPartitionInputTopic, partition = 0, leader = none, replicas = [], isr = [], offlineReplicas = []), Partition(topic = multiPartitionInputTopic, partition = 1, leader = none, replicas = [], isr = [], offlineReplicas = [])], controller = null)"   myMetadata = {StreamsMetadata@6908} "StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}"    hostInfo = {HostInfo@6915} "HostInfo{host='dummy', port=2142}"    stateStoreNames = {HashSet@6916}  size = 0    topicPartitions = {HashSet@6917}  size = 2     0 = {TopicPartition@6920} "multiPartitionInputTopic-0"     1 = {TopicPartition@6921} "multiPartitionInputTopic-1"
**streams2** = {KafkaStreams@5792}  streamsMetadataState = {StreamsMetadataState@5894} "GlobalMetadata: [StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}]\nGlobalStores: []\nMy HostInfo: HostInfo{host='dummy', port=2142}\nCluster(id = z84VPglESomInK50zKMMRQ, nodes = [localhost:57768 (id: 0 rack: null), localhost:57774 (id: 2 rack: null), localhost:57771 (id: 1 rack: null)], partitions = [Partition(topic = multiPartitionInputTopic, partition = 0, leader = 2, replicas = [2], isr = [2], offlineReplicas = []), Partition(topic = multiPartitionInputTopic, partition = 1, leader = 1, replicas = [1], isr = [1], offlineReplicas = [])], controller = localhost:57768 (id: 0 rack: null))\n"   builder = {InternalTopologyBuilder@5906}    allMetadata = {ArrayList@5907}  size = 1    0 = {StreamsMetadata@7001} "StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}"     hostInfo = {HostInfo@7003} "HostInfo{host='dummy', port=2142}"     stateStoreNames = {HashSet@7004}  size = 0     topicPartitions = {HashSet@7005}  size = 2      0 = {TopicPartition@7008} "multiPartitionInputTopic-0"      1 = {TopicPartition@7009} "multiPartitionInputTopic-1"   globalStores = {Collections$UnmodifiableSet@5948}  size = 0   thisHost = {HostInfo@5959} "HostInfo{host='dummy', port=2142}"   clusterMetadata = {Cluster@6979} "Cluster(id = z84VPglESomInK50zKMMRQ, nodes = [localhost:57768 (id: 0 rack: null), localhost:57774 (id: 2 rack: null), localhost:57771 (id: 1 rack: null)], partitions = [Partition(topic = multiPartitionInputTopic, partition = 0, leader = 2, replicas = [2], isr = [2], offlineReplicas = []), Partition(topic = multiPartitionInputTopic, partition = 1, leader = 1, replicas = [1], isr = [1], offlineReplicas = [])], controller = localhost:57768 (id: 0 rack: null))"   myMetadata = {StreamsMetadata@6980} "StreamsMetadata{hostInfo=HostInfo{host='dummy', port=2142}, stateStoreNames=[], topicPartitions=[multiPartitionInputTopic-0, multiPartitionInputTopic-1]}"    hostInfo = {HostInfo@6987} "HostInfo{host='dummy', port=2142}"    stateStoreNames = {HashSet@6988}  size = 0    topicPartitions = {HashSet@6990}  size = 2     0 = {TopicPartition@6993} "multiPartitionInputTopic-0"     1 = {TopicPartition@6994} "multiPartitionInputTopic-1"
Which means basically both the streams objects are saying at the same that time that they have been assigned the same tasks, if you look at myMetadata of both objects.

Thanks for updating the PR. I have some question about the upgrade path and backward compatibility.
Did your run the streams_upgrade_test.py system test locally to verify that version probing upgrading works?

I tried with ./gradlew cleanTest test and ./gradlew checkstyleMain checkstyleTest. Is there any specific command to it, or I can just run python tests/kafkatest/tests/streams/streams_upgrade_test.py?

How to run system tests is describe in tests README: https://github.com/apache/kafka/tree/trunk/tests

Inside OnAssignment, we want to convert taskByHosts to partitionsByHost. Now we have all the source topics from taskManager.builder().topicGroups() but to create TopicPartition we all need partitions along with topics which we get from partitionsForTask. We are getting partitionsForTask as below.
final Cluster fullMetadata = Cluster.empty().withPartitions(topicToPartitionInfo);
final Map<TaskId, Set<TopicPartition>> partitionsForTask = partitionGrouper.partitionGroups(sourceTopicsByGroup, fullMetadata);

Now, when we don’t add applicationServer, hostinfo is null so tasksByHost is empty so topicToPartitionInfo is empty so fullMetadata is empty and when the above line is called sourceTopicsByGroup has source topics but fullMetadata comes out to be null. So, internally the code breaks in while calling maxPartitions in DefaultPartitionGrouper.
What we can do otherwise is, rather than getting partitionsForTask via partitionGrouper, we can create from whatever is there in tasksByHosts. So, I mean corresponding to each topicGroupId, find the source topics and append in partitionsByHost by reading as below.
for (final Map.Entry<HostInfo, Set<TaskId>> entry : tasksByHosts.entrySet()) {
    final Set<TopicPartition> allTopicPartitions = new HashSet<>();
    for (final TaskId taskId : entry.getValue()) {
        final Set<TopicPartition> topicPartitions = new HashSet<>();
        for(String topic : sourceTopicsByGroup.get(taskId.topicGroupId)) {
            topicPartitions.add(new TopicPartition(topic, taskId.partition));
        }
        allTopicPartitions.addAll(topicPartitions);
    }
    partitionsByHost.put(entry.getKey(), allTopicPartitions);

    }

That makes all sense. What I am trying to say is, that we only need to convert taskByHost to partitionByHost iff application.server is specified. If a users does not specify application.server config, it's expected that there is no metadata. @guozhangwang please correct me if my understanding is wrong.
We only need to populate the metadata iff users wants to use interactive queries and if a user does not specify application.server and thus disables interactive queries, we don't need to bother to provide the metadata. With the current change it seems that we force users to specify application.server what would be a backward incompatible change and it seems unnecessary.
Does this make sense? Maybe I am missing something.

Btw: could you run the system tests locally to verify that this PR does not break anything?

Btw: could you run the system tests locally to verify that this PR does not break anything?

I ran TC_PATHS="tests/kafkatest/tests/streams/streams_upgrade_test.py::StreamsUpgradeTest" bash tests/docker/run_tests.sh on the trunk branch and it failed for test_version_probing_upgrade function while all other functions passed. Can you please if it works on trunk branch so I can run it on my local branch and see that nothing breaks.
Also, for my test I guess I might have to make some changes in StreamsUpgradeTest.java as FutureAssignmentInfo will have to be compressed as we are doing assignmentInfo for version >= 5. Am I right?

config, it's expected that there is no metadata. @guozhangwang please correct me if my understanding is wrong.

That's right. @brary if application.server is not specified, it is expected to see empty tasksByHost (which is normal), which means we cannot always rely on it to construct the partitionsByHost.

@brary The test you mentioned in the test in question. It passed in our nightly system test run (last run): https://jenkins.confluent.io/job/system-test-kafka/job/trunk/471 -- it also passed in run 470 and 469 and earlier
Thus, it should pass locally for you, too, using trunk. If it fails on this branch, it indicate an issue with this PR.

Also, for my test I guess I might have to make some changes in StreamsUpgradeTest.java as FutureAssignmentInfo will have to be compressed as we are doing assignmentInfo for version >= 5. Am I right?

I don't think this would be required to make the system test pass. This PR increases AssignmentInfo#LATEST_SUPPORTED_VERSION to 5 and FutureAssignmentInfo encodes AssignmentInfo#LATEST_SUPPORTED_VERSION + 1 as version number and thus is bump automatically.
Let us know if you need any input. Updating the metadata is tricky to get right with regard to backward compatibility... The error message from the test might help to figure out what is wrong.

@brary The test you mentioned is the test in question. It passes in our nightly system test runs (last run): https://jenkins.confluent.io/job/system-test-kafka/job/trunk/471 -- it also passed in runs 470 and 469 and earlier
Thus, it should pass locally for you, too, using trunk. If it fails on this branch, it indicate an issue with this PR.

Also, for my test I guess I might have to make some changes in StreamsUpgradeTest.java as FutureAssignmentInfo will have to be compressed as we are doing assignmentInfo for version >= 5. Am I right?

I don't think this would be required to make the system test pass. This PR increases AssignmentInfo#LATEST_SUPPORTED_VERSION to 5 and FutureAssignmentInfo encodes AssignmentInfo#LATEST_SUPPORTED_VERSION + 1 as version number and thus is bump automatically.
Let us know if you need any input. Updating the metadata is tricky to get right with regard to backward compatibility... The error message from the test might help to figure out what is wrong.

Cool, so in nutshell I will change onAssignment function to see if application.server is not mentioned partitionsByHost will be empty and we don't need partitionGrouper flow to create partitionsByHost from tasksByHost in that case. Also, I will remove application.server config from the test cases where I have added it.

I did a new clone of trunk branch today and ran StreamsUpgradeTest.test_version_probing_upgrade but the test again failed twice. I am adding some snipets below.
 Reported exception: java.lang.NullPointerException 	at org.apache.kafka.common.message.MetadataRequestData.toString(MetadataRequestData.java:292) 	at org.apache.kafka.common.requests.MetadataRequest$Builder.toString(MetadataRequest.java:100) 	at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299) 	at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271) 	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233) 	at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173) 	at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.writeLog(LogContext.java:428) 	at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.debug(LogContext.java:222) 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1073) 	at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:982) 	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:533) 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262) 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233) 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224) 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:161) 	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.ensureFreshMetadata(ConsumerNetworkClient.java:172) 	at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:336) 	at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1229) 	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1194) 	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1179) 	at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:940) 	at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:845) 	at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:804) 	at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:773)
Here's the summary of the test report.
"summary": "Could not identify leader\nTraceback (most recent call last):\n  File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 132, in run\n    data = self.run_test()\n  File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 189, in run_test\n    return self.test_context.function(self.test)\n  File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 330, in test_version_probing_upgrade\n    current_generation = self.do_rolling_bounce(p, counter, current_generation)\n  File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 546, in do_rolling_bounce\n    self.update_leader()\n  File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 364, in update_leader\n    raise Exception("Could not identify leader")\nException: Could not identify leader\n",
"test_id": "kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_version_probing_upgrade",
"test_status": "FAIL"
And this is the flow where I feel it is failing.
[2019-03-22 05:33:59,856] DEBUG [Producer clientId=StreamsUpgradeTest-f5843d4f-7ae7-4856-bde9-0dacb9646590-StreamThread-1-producer] Sending metadata request MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to node ducker03:9092 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient) [2019-03-22 05:33:59,864] DEBUG [AdminClient clientId=StreamsUpgradeTest-f5843d4f-7ae7-4856-bde9-0dacb9646590-admin] Updating cluster metadata to Cluster(id = 4u9fG_ebTH2wtVfC_Rdsow, nodes = [ducker03:9092 (id: 1 rack: null)], partitions = [], controller = ducker03:9092 (id: 1 rack: null)) (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [2019-03-22 05:33:59,867] INFO [Producer clientId=StreamsUpgradeTest-f5843d4f-7ae7-4856-bde9-0dacb9646590-StreamThread-1-producer] Cluster ID: 4u9fG_ebTH2wtVfC_Rdsow (org.apache.kafka.clients.Metadata) [2019-03-22 05:33:59,867] DEBUG [Producer clientId=StreamsUpgradeTest-f5843d4f-7ae7-4856-bde9-0dacb9646590-StreamThread-1-producer] Updated cluster metadata updateVersion 2 to MetadataCache{cluster=Cluster(id = 4u9fG_ebTH2wtVfC_Rdsow, nodes = [ducker03:9092 (id: 1 rack: null)], partitions = [], controller = ducker03:9092 (id: 1 rack: null))} (org.apache.kafka.clients.Metadata) [2019-03-22 05:33:59,869] DEBUG [Consumer clientId=StreamsUpgradeTest-f5843d4f-7ae7-4856-bde9-0dacb9646590-StreamThread-1-consumer, groupId=StreamsUpgradeTest] Sending metadata request [FAILED toString()] to node ducker03:9092 (id: -1 rack: null) (org.apache.kafka.clients.NetworkClient)

I was just checking MetadataRequestData.java:292 and is seems there is a bug. topics could be null but there is no check, and thus topics.iterator() gives a NPE.
What I am not sure though is, why topics is empty in your local test run. Can you try to just fix the bug locally and run the test again (just to see if this let the test pass) -- if you want, you can also do a PR to fix the bug -- otherwise, I can also do a fix (just let me know).

I was just checking MetadataRequestData.java:292 and is seems there is a bug. topics could be null but there is no check, and thus topics.iterator() gives a NPE.
What I am not sure though is, why topics is empty in your local test run. Can you try to just fix the bug locally and run the test again (just to see if this let the test pass) -- if you want, you can also do a PR to fix the bug -- otherwise, I can also do a fix (just let me know).

@mjsax Can you fix the bug? Since it would be merged faster that way. I checked the MetadataRequestData.java looks like it is being autogenerated and I am not sure exactly from where.

@mjsax I tried fixing the issue in trunk and even after commenting out the log line NetworkClient.java the version probing test is failing. Can you fix this bug, I guess that would do it faster.

I was just checking MetadataRequestData.java:292 and is seems there is a bug. topics could be null but there is no check, and thus topics.iterator() gives a NPE.
What I am not sure though is, why topics is empty in your local test run. Can you try to just fix the bug locally and run the test again (just to see if this let the test pass) -- if you want, you can also do a PR to fix the bug -- otherwise, I can also do a fix (just let me know).

@mjsax I tried fixing the issue in trunk and even after commenting out the log line NetworkClient.java the version probing test is failing. Can you fix this bug, I guess that would do it faster.

I'll look into the bug.
After you removed the log line, does it still fail with the same error, or is the exception gone? It not, maybe it did not pick up the change. You could also change the logging config to disable DEBUG level logging in NetworkClient I guess.

I'll look into the bug.
After you removed the log line, does it still fail with the same error, or is the exception gone? It not, maybe it did not pick up the change. You could also change the logging config to disable DEBUG level logging in NetworkClient I guess.

It didn't throw any exception but the version probing test case still failed. I see below logs but other than these, there are no other exceptions.
[INFO  - 2019-03-22 01:56:07,531 - runner_client - log - lineno:234]: RunnerClient: kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_version_probing_upgrade: FAIL: Could not uniquely identify leader Traceback (most recent call last):   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 132, in run     data = self.run_test()   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 189, in run_test     return self.test_context.function(self.test)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 330, in test_version_probing_upgrade     current_generation = self.do_rolling_bounce(p, counter, current_generation)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 546, in do_rolling_bounce     self.update_leader()   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 353, in update_leader     raise Exception("Could not uniquely identify leader") Exception: Could not uniquely identify leader[INFO  - 2019-03-22 01:56:07,531 - runner_client - log - lineno:234]: RunnerClient: kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_version_probing_upgrade: FAIL: Could not uniquely identify leader Traceback (most recent call last):   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 132, in run     data = self.run_test()   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 189, in run_test     return self.test_context.function(self.test)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 330, in test_version_probing_upgrade     current_generation = self.do_rolling_bounce(p, counter, current_generation)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 546, in do_rolling_bounce     self.update_leader()   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 353, in update_leader     raise Exception("Could not uniquely identify leader") Exception: Could not uniquely identify leader

Fix for the NPE: https://github.com/apache/kafka/pull/6489/files

@brary The test you mentioned is the test in question. It passes in our nightly system test runs (last run): https://jenkins.confluent.io/job/system-test-kafka/job/trunk/471 -- it also passed in runs 470 and 469 and earlier
Thus, it should pass locally for you, too, using trunk. If it fails on this branch, it indicate an issue with this PR.

Also, for my test I guess I might have to make some changes in StreamsUpgradeTest.java as FutureAssignmentInfo will have to be compressed as we are doing assignmentInfo for version >= 5. Am I right?

I don't think this would be required to make the system test pass. This PR increases AssignmentInfo#LATEST_SUPPORTED_VERSION to 5 and FutureAssignmentInfo encodes AssignmentInfo#LATEST_SUPPORTED_VERSION + 1 as version number and thus is bump automatically.
Let us know if you need any input. Updating the metadata is tricky to get right with regard to backward compatibility... The error message from the test might help to figure out what is wrong.

Cool, so in nutshell I will change onAssignment function to see if application.server is not mentioned partitionsByHost will be empty and we don't need partitionGrouper flow to create partitionsByHost from tasksByHost in that case. Also, I will remove application.server config from the test cases where I have added it.

Thanks @mjsax for the fix. Somehow the version probing test is still failing in my local, any idea on that?
And does the above statements make sense then I will send a commit with the changes.

I tried to run it locally on trunk and the test also failed. Investigating. Might take some time. Keep you posted.

I tried to run it locally on trunk and the test also failed. Investigating. Might take some time. Keep you posted.

Hey @mjsax any update on this?

@brary The test should be fixed now. Can you retry? If the test passed on trunk locally for you now, rebase this PR and see if the upgrade test passed.

@brary The test should be fixed now. Can you retry? If the test passed on trunk locally for you now, rebase this PR and see if the upgrade test passed.

@mjsax the test- test_version_probing_upgrade() again failed, below are the snipets.
[INFO  - 2019-04-16 00:12:00,685 - runner_client - log - lineno:234]: RunnerClient: kafkatest.tests.streams.streams_upgrade_test.StreamsUpgradeTest.test_version_probing_upgrade: Summary: Could not uniquely identify leader Traceback (most recent call last):   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 132, in run     data = self.run_test()   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 189, in run_test     return self.test_context.function(self.test)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 330, in test_version_probing_upgrade     current_generation = self.do_rolling_bounce(p, counter, current_generation)   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 549, in do_rolling_bounce     self.update_leader()   File "/opt/kafka-dev/tests/kafkatest/tests/streams/streams_upgrade_test.py", line 353, in update_leader     raise Exception("Could not uniquely identify leader") Exception: Could not uniquely identify leader

@brary I've retried running current trunk on jenkins and it works now.
For the failures you've seen, I'd suspect it's actually related to this PR since the version_prob test logic is very related to the logical change you've made. I'd suggest you taking a look at the collected logs (maybe also enable DEBUG / TRACE for more log info) and see if it reveals some bugs in the PR.

@brary I've retried running current trunk on jenkins and it works now.
For the failures you've seen, I'd suspect it's actually related to this PR since the version_prob test logic is very related to the logical change you've made. I'd suggest you taking a look at the collected logs (maybe also enable DEBUG / TRACE for more log info) and see if it reveals some bugs in the PR.

@guozhangwang I tried the version probing test which failed on a newly cloned trunk branch, without any changes from this PR. I remember last time when we came to know the issue in test, Matthias had said that the test is passing in the nightly test run but then it failed for him locally. Can you please test it locally too.

Sure I can do that.
At the mean time I've also triggered a jenkins job at your branch: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2517
It runs the whole test suite.
If that test passed then I think it also validates that there's some difference running it on jenkins v.s. locally.

Sure I can do that.
At the mean time I've also triggered a jenkins job at your branch: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2517
It runs the whole test suite.
If that test passed then I think it also validates that there's some difference running it on jenkins v.s. locally.

Sure, let me know whatever the end result.

@brary I'm still working on running your branch on my local box.
And for the jenkins test, it indeed failed and you can find the summary and download all the log files from here: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2019-04-18--001.1555624657--brary--assignmentinforedesign--431353d/report.html
Seems like failed tests on compatibility / upgrade path are related to the changes (on trunk they succeeded).

@brary I'm still working on running your branch on my local box.
And for the jenkins test, it indeed failed and you can find the summary and download all the log files from here: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2019-04-18--001.1555624657--brary--assignmentinforedesign--431353d/report.html
Seems like failed tests on compatibility / upgrade path are related to the changes (on trunk they succeeded).

Cool, thanks. I will look into the report. Did the trunk branch succeeded on local for version probing?

@guozhangwang @mjsax I have done rebase of my branch, now this has the NPE fix which was the reason behind the test cases failure in the job which @guozhangwang ran. Can you guys please do another round of review so that we can merge this change.

@mjsax @guozhangwang can you please take a look at this PR again.

@brary sorry for the late replies. I'm taking a look at it now and will also try to resolve conflicts by pushing to your branch directly.
Will also run the upgrade system test as well.

@brary sorry for the late replies. I'm taking a look at it now and will also try to resolve conflicts by pushing to your branch directly.
Will also run the upgrade system test as well.

Sure @guozhangwang, thanks. Let me know if you need any clarification.

@brary I tried to force push to your brary:assignmentinforedesign branch directly but somehow it messed up the branch a bit with old commits, do you mind if I close this PR and continue on https://github.com/apache/kafka/pull/6935?

@brary I tried to force push to your brary:assignmentinforedesign branch directly but somehow it messed up the branch a bit with old commits, do you mind if I close this PR and continue on #6935?

Hey @guozhangwang sure. Please do that. Lets work together and close this as soon as possible because everytime we lag too many commits behind, rebase kind of poses some issues.

Thank you @brary !


dead list awesome-rubymotion

the list awesome-rubymotion by @motion-open-source is dead.
https://github.com/sindresorhus/awesome/blob/9170c40dc9ae84fb4a6bd9708085de6dabea3a32/readme.md?plain=1#L217
❯ curl -I https://github.com/motion-open-source/awesome-rubymotion
HTTP/2 404
server: GitHub.com
date: Thu, 17 Feb 2022 03:34:35 GMT
content-type: text/html; charset=utf-8

looks like the user https://github.com/motion-open-source is dead in the first place.
but found some mirror (ig):

https://project-awesome.org/motion-open-source/awesome-rubymotion
https://github.com/rvollebregt/awesome-rubymotion (0 star | last commit on "1 Dec 2015")
https://awesomerepos.io/awesome/motion-open-source/awesome-rubymotion (dead server)

can i make a pull request removing the list ??
or something else should be done ??

can i make a pull request removing the list ??

👍


SVG not displayed

Hi,
Thank you for this Grunt package that allows us to nail down responsiveness issues on the websites we develop!
The only thing that seems not to be working is that the embedded SVG files do not seem to be displayed:here is an example as drawn by Grunt:Pageres:

Here is the page it comes from: http://boysinburbs.com/cda-drawing/how-to-break-daddy-and-papas-tv-remote-control/
Some details:

the page we tested was hosted on a local server accessed by <IP_ADDRESS>:8080 [but this should not be an issue]
the website is a WordPress website
there is a Lazy Page script [WP plugin] allowing to display the pictures progressively [this might be the culprit].

Best,
Stephen

You might want to try the delay option as the SVG might have been loaded async. Pageres only waits on the page load event, but there still might be other stuff being lazy loaded afterwards. No way for Pageres to know about that.

—
Reply to this email directly or view it on GitHub
https://github.com/sindresorhus/pageres/issues/164#event-262506661.

Thanks!


Adding non-A dns records does not work

According to the manual, the following should work:

dns:
  # Validate a PTR record
  PTR:<IP_ADDRESS>:
    resolvable: true
    server: <IP_ADDRESS>
    addrs:
    - "google-public-dns-a.google.com."

This does not work for me:

$ goss -v
goss version v0.3.6
$ goss add dns PTR:<IP_ADDRESS>
Adding DNS to './goss.yaml':

PTR:<IP_ADDRESS>:
  resolvable: true
  addrs:
  - <IP_ADDRESS>
  timeout: 500

In fact any dns record type is ignored and always an A record is requested. Examples:

$ dig +short -t MX google.com
30 alt2.aspmx.l.google.com.
40 alt3.aspmx.l.google.com.
50 alt4.aspmx.l.google.com.
10 aspmx.l.google.com.
20 alt1.aspmx.l.google.com.
$ goss a dns MX:google.com
Adding DNS to './goss.yaml':

MX:google.com:
  resolvable: true
  addrs:
  - <IP_ADDRESS>
  - 2a00:1450:400e:802::200e
  timeout: 500

Note that the example from the manual explicitly sets the server: key while your examples don't.
When I copy the PTR example from the manual as-is, it works for me as expected with goss 0.3.6.
On the contrary, when I comment out the server: key such that the DNS server of my AVM FRITZ!Box is used, the goss validate fails with:
DNS: PTR:<IP_ADDRESS>: addrs:
Expected
    <[]string | len:1, cap:1>: ["<IP_ADDRESS>"]
to contain element matching
    <string>: dns.google.

It does also work when I directly specify my provider's DNS server  (which the FRITZ!Box resolves too, as well).
When I test with dig @<IP_ADDRESS> -x <IP_ADDRESS> etc. I don't see any differences in the answers between the different DNS servers.
Thus, it looks like the Goss DNS code is incompatible to some DNS server implementations.

You have to explicitly set the sever attribute to test a PTR record.
"With the server attribute set, it is possible to validate the following types of DNS record:..."
Try setting the server attribute to your DNS server and let me know if that doesn't work.

Oh, I over-read this part - yes, when I explicitly set the server: to the address of my FRITZ!Box, it works as well.
FWIW, a dig -x <IP_ADDRESS> works without specifying the DNS server explicitly. Thus, this limitation is a bit unexpected.
Btw, testing the PTR of an IPv6 address isn't supported yet, right?
https://github.com/aelsabbahy/goss/blob/master/system/dns.go#L39-L40
looks like colons in an IP address aren't expected.
Should I open a new issue for this?

Glad the DNS test is working for you.
Yes, for ipv6 if you still need it, please open a new issue.


Python stopped working

The following command fails:
wapm install python

And then
wapm run python

Fails with:
exec error: wasi filesystem creation error: `Could not get metadata for file<EMAIL_ADDRESS>fd not a directory`
Error: wapm bus error - there was an error while deserializing the request or response.

This before gracefully failed via a WASI error (but was booting Python successfully)

Does it normally look like this...
https://tokera.sh/?init=wapm%20install%20python%20%26%26%20python

I think the new wasm-bus broke something on 'wapm run' command - I normally do regression tests with 'wapm run quickjs' which obviously isn't enough coverage - I'll push a fix with the next build.

This is now fixed it will work with either
wapm install python
wapm run python

...or it will work with
wapm install python
python


Use deprovision operation to remove VHost

fixes #123
@mixonic, for you

@mixonic refactored to use a {{delete-vhost}} component in 1b8babcaf881a3bb3afcf3362ba6f124c68c93bf
When tests go green this is good to go

also travis being a miscreant.

yeah travis status was reporting some issue w/ public repos


Clicking buttons in web ui return 'Forbidden'

After upgrade sidekiq to 3.4.2, clicking 'EnqueueNow', 'Disable', 'Enable' and 'Delete' button returns 'Forbidden'.
It may be caused by the new sidekiq requiring authenticity token for the post forms.

This issue is solved already here in the github code but not published on rubygems.

Any expected timeframe when a published version will include this fix?   I just encountered the same issue.

I just released sidekiq-cron v 0.3.1


Set up Rubocop

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Power Rename Enumerate does not work as espected

Microsoft PowerToys version
0.76.0
Installation method
GitHub
Running as admin
Yes
Area(s) with issue?
PowerRename
Steps to reproduce
Hi everyone,
I am having issues trying to rename files with Power Rename,
as shown in the screenshot, the Enumerator does not work for me.
What am I doing wrong?
Thanks for the help.
✔ Expected Behavior
2023-12-08_Module_01
2023-12-08_Module_02
2023-12-08_Module_03
2023-12-08_Module_04

❌ Actual Behavior
$YYYY-$MM-$DD_Module_${increment=1;padding=2;start=01}
$YYYY-$MM-$DD_Module_${increment=1;padding=2;start=01} (2)
$YYYY-$MM-$DD_Module_${increment=1;padding=2;start=01} (3)
$YYYY-$MM-$DD_Module_${increment=1;padding=2;start=01} (4)
Other Software
No response

Hi, you need to enable Enumeration in order to use this syntax:

Does this help? /needinfo

Hi, you need to enable Enumeration in order to use this syntax:

Does this help? /needinfo

Hi Stefan, I already tried it and it didn't change the end result, I will give a try tomorrow as I am now not in office.

can confirm that I'm getting the same issue

Anything else I need to do prior to "Report Bug"? Just tried it couple twice, it's waiting about a minute, printing message that .zip acrhive was saved to the desktop but there is nothing there.
Couldn't you reproduce the issue? Here's the video example
https://github.com/microsoft/PowerToys/assets/9417531/53fba9d6-c12c-4c15-b49f-cd90bb0d85f5

Anything else I need to do prior to "Report Bug"? Just tried it twice, it's waiting about a minute, printing message that .zip acrhive was saved to the desktop but there is nothing there.

No, it should be there named something like PowerToysReport_2023-12-11-08-54-13.zip
Which version of PowerToys are you using? /needinfo

Oh, turned out it wasn't updating automatically for some reason. The version was 0.71, now it's 0.78 and issue is resolved.

Oh, turned out it wasn't updating automatically for some reason. The version was 0.71, now it's 0.78 and issue is resolved.

Yes, the custom enumeration feature was introduced after v0.71 :) Thanks for checking, glad to hear it works now! :)
@Phas3s  Which version are you using? Could you attach /bugreport? Thanks!


Optimize parsing of invisible operator operands

Some expressions take a very long time to parse, scaling exponentially with the length of the expression.  The following example takes over 20 minutes to parse without this patch and less than a second with the patch.
xxxxxxxxxxxxxxxx\left[yyyyyyyy\left[zzzzzzzzzz\left[\left[\left[x.x\right]\right]\right]\right]\right]

It is an invalid expression.  If that could be determined in general without attempting to parse it, perhaps the parsing efficiency wouldn't matter, but I don't know of a way to achieve that.
The cause of the blow-up in execution time is a combination of backtracking to the opening braces due to the invalid part of the expression and continued attempts to parse forward again due to the loop over invisible operator operands at every level in the recursive chain of parseExpression() calls.
This patch avoids many redundant attempts to parse the next invisible operator operand by only doing that in the parent parseExpression() and not in its immediate children.  This is implemented in a somewhat ad-hoc way rather than splitting the multiplication precedence level in two.  If you would like, I'd be happy to submit a patch that does the latter.  I think that will disturb more code, so that is why I did it this way.
Thanks for listening, and thanks for the wonderful software!

Thanks, that's a great catch. Surprised parsing could take 20 mins, but I guess it's the power of exponential recursion!
Your patch works and is OK. As a stylistic thing, I prefer to avoid having boolean arguments, because they're not self-documenting, and it's easy to forget if you should pass true or false. Instead, I would rather have a named property in a structure. Not super important in this case, since the argument is only used in a recursive call of the function.
Nonetheless, I think I would rather add a invisibleOperator to the until (i.e. Terminator) structure, unless you want to go ahead and split the multiplication precedence (I assume that means there would be a "regular" one and one for "invisible" multiplication?).
Let me know how you want to proceed. If you'd like to explore either of those options, great, otherwise I'll pickup your patch and tweak it. Either way, thanks again for submitting it!

As an experiment, I have redone the patch by using until.minPrec itself to implement the optimization.  A new INVISIBLE_OP_PRECEDENCE that is less than MULTIPLICATION_PRECEDENCE was added.
In hindsight, it would be even simpler to change the invisible operator parsing loop to use minPrec=MULTIPLICATION_PRECEDENCE+1 when parsing arguments.  Then there is no need to introduce INVISIBLE_OP_PRECEDENCE.  The two different approaches produce semantically equivalent results when invisible operator is interpreted as multiplication, but are there other cases that need to be considered?
In the current patch, and prior to this patch, with canonical=false, ab*cd becomes:
["InvisibleOperator","a",["Multiply","b","c"],"d"]
If the alternative approach I mentioned above were used, or if INVISIBLE_OP_PRECEDENCE is set equal to MULTIPLICATION_PRECEDENCE, the result would instead be:
["InvisibleOperator",["Multiply",["InvisibleOperator","a","b"],"c"],"d"]
Whichever choice is made, this general approach introduces enforcement of precedence for the invisible operator, which causes important semantic changes.  Previously, parseExpression() did not check until.minPrec before entering the invisible operator loop.  This means, prior to this patch, 1/2a got parsed like 1/(2a), whereas now it is like (1/2)a.  The new behavior is consistent with 1/2*a being interpreted like (1/2)*a irrespective of this patch.  The latter is consistent with the defined precedence of division being higher than multiplication, which is at first surprising, since it is not like a calculator or programming language, but probably more consistent with mathematical tradition.
The power operator is implemented with a custom parser, so I don't think it is impacted.
The negation operator is defined with higher order of precedence than multiplication, so it is impacted.  I'm not aware of any case where that would create a semantic difference, but it certainly does impact the parsed forms of expressions such as -ab, which is now like (-a)b rather than -(ab).  All of the tests that fail are due to this change.
The \prod operator is impacted.  Previously, it was parsing its argument with minPrec=MULTIPLICATION_PRECEDENCE+1.  Therefore, \prod_{n=1}^3 n*m was interpreted the same as (\prod_{n=1}^3 n)*m, whereas \prod_{n=1}^3 nm was interpreted the same as \prod_{n=1}^3 (nm).  I believe the latter is intended, so this patch makes the visible multiplication case work this way as well.
The \sum operator is impacted as a side-effect of a fix to parseBigOp() for \prod.  The prec argument to parseBigOp() was being treated as one less than the minimum precedence.  To fix \prod, this patch changes the argument to minPrec, which is treated as the minimum precedence.  I had to choose a new precision to pass into parseBigOp() for \sum.  It could have been changed to ADDITION_PRECEDENCE+1 to avoid any change in behavior, but I am guessing that INVISIBLE_OP_PRECEDENCE is what is actually desired.  The precedence levels lower than that do not appear to be things that would be expected to be slurped into the argument of \sum.  I mention it in case you disagree.

Since I was wrong that setting INVISIBLE_OP_PRECEDENCE less than MULTIPLICATION_PRECEDENCE would preserve the old order of operations between multiplication and invisible operator, I don't see any reason to stick with that.  It seems to me it should either be equal or perhaps greater.  None of these choices conflict with the intended side-effect of fixing the invisible operator performance problem, because in all cases, the value of minPrec used when parsing the invisible operator operands would be greater than INVISIBLE_OP_PRECEDENCE, which would prevent the recursive chain of parseExpression() calls from all hammering on a failed expression like they were.
An advantage of making the precedence equal is that INVISIBLE_OP_PRECEDENCE could just go away.  However, I am wondering if there are branches of math where one might expect the invisible operator to have higher precedence than multiplication type operators.

In general, and for example when using TI calculators, the invisible operator is considered higher precedence than multiplication or division, so that 1/2a is parsed as 1/(2*a).

That looks good to me. The differences are either not very important (negation) or actually an improvement.
Thanks for looking into this and being so thorough.

I redid the patch.  There are two things worth noting.  One involves mixed numbers:
2\frac{5}{7}\times\frac{7}{9}
The above expression used to parse (non-canonically) to this:
["InvisibleOperator",2,["Multiply",["Divide",5,7],["Divide",7,9]]]
Now it parses (non-canonically) to this:
["Multiply",["InvisibleOperator",2,["Divide",5,7]],["Divide",7,9]]
The canonical parser interprets the invisible operator in the second case as a mixed number 2+5/7, whereas in the previous case, it interprets the invisible operator as multiplication.  I think this semantic change is reasonable, but since this case was in the test suite, perhaps you had a reason to prefer the previous behavior.
The second thing of note is that my previous assumption about the relative precedence of negation and exponentiation was wrong.  It doesn't work like this:
-a^bc = ((-a)^b)c
Instead, it works like this:
-a^bc = (-(a^b))c
As far as I can tell, this is intended, so all is well.
Other than those two things, I didn't find any surprises.

If you would like a second commit with changes to the test suite, I'd be happy to do that.  I didn't want to combine that with the code change, so you could examine the tests that fail and agree that those changes in behavior are all fine.

That looks good, thank you. I'll have a look at the mixed fraction case. It seems reasonable to me, but I've gotten some feedback previously on it, which may be why there was this test case. Either way, I'm sure we can make it work. I'll need before I can merge it, but the ball is in my court now.
Thanks again for this great contribution.

2\\frac{5}{7}\\times\\frac{7}{9} was parsed incorrectly previously, this PR fixes it.
All good, updated the tests.


Add support of Qwen models (7b, 14b, 72b) to DeepSpeed-FastGen

This PR adds support for Qwen models 7b, 14b and 72b.
Test Code
for mii pipeline:
from mii import pipeline
pipe = pipeline("Qwen/Qwen-7B-Chat")
pipe.tokenizer.tokenizer.eos_token_id = 151643
output = pipe(["DeepSpeed is"], max_new_tokens=128, do_sample=False)
print(output)

for huggingface:
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
inputs = tokenizer('DeepSpeed is', return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs, max_new_tokens=128, do_sample=False, repetition_penalty=1.0)
test = tokenizer.decode(pred.cpu()[0], skip_special_tokens=False)
print(test)

Qwen 7B
Huggingface output with prompt "DeepSpeed is":
 a high-performance, low-latency database management system designed for real-time analytics and machine learning applications. It is built on top of Apache Arrow, a columnar in-memory data format, and is optimized for processing large volumes of data in parallel.\n\nDeepSpeed offers several key features that make it well-suited for real-time analytics and machine learning applications:\n\n1. High Performance: DeepSpeed is designed to deliver high performance by leveraging parallel processing and optimized data structures. It can process large volumes of data in real-time, making it ideal for applications that require real-time analytics.\n\n2. Low Latency: DeepSpeed is designed to minimize latency by

DeepSpeed-FastGen output with prompt "DeepSpeed is":
 a high-performance, low-latency database management system designed for real-time analytics and machine learning applications. It is built on top of Apache Arrow, a columnar in-memory data format, and is optimized for processing large volumes of data in parallel.\n\nDeepSpeed offers several key features that make it well-suited for real-time analytics and machine learning applications:\n\n1. High Performance: DeepSpeed is designed to deliver high performance by leveraging parallel processing and optimized data structures. It can process large volumes of data in real-time, making it ideal for applications that require real-time analytics.\n\n2. Low Latency: DeepSpeed is designed to minimize latency by

Qwen 72B
Huggingface output with prompt "DeepSpeed is":
是一个开源的深度学习优化库，它提供了多种优化技术，包括模型并行、数据并行、混合并行、ZeRO内存优化等。它可以帮助用户在大规模GPU集群上训练深度学习模型，提高训练速度，减少内存使用。\n在Deepspeed中，模型并行是一种将模型的不同部分分配到不同的GPU上的技术。这样可以处理模型太大，无法放在一个GPU上的问题。数据并行是将数据集分成多个部分，每个部分在不同的GPU上进行训练。混合并行则是结合了模型并行和数据并行，以更有效地利用GPU资源

DeepSpeed-FastGen output with prompt "DeepSpeed is" with 8-way sharding:
是一个开源的深度学习优化库，它提供了多种优化技术，包括模型并行、数据并行、混合并行、ZeRO内存优化等。它可以帮助用户在大规模GPU集群上训练深度学习模型，提高训练速度，减少内存使用。\n在Deepspeed中，模型并行是一种将模型的不同部分分配到不同的GPU上的技术。这样可以处理模型太大，无法放在一个GPU上的问题。数据并行是将数据集分成多个部分，每个部分在不同的GPU上进行训练。混合并行则是结合了模型并行和数据并行，以更有效地利用GPU资源

@microsoft-github-policy-service agree

@ZonePG thank you for this contribution! Could you please run the pre-commit and commit any modified files? pre-commit run --all-files

@mrwyattii pre-commit checks have been run and all modified files have been committed. Thanks!


[QUESTION] Does COMET support multiple references?

Thanks for the tool! I'm wondering if COMET supports multiple references, or if we can just score each sentence with all the references and take the maximum value? Sorry if this has already been mentioned somewhere.

Hi! This is actually an open question. We did some experiments where we tried to experiment with multiple references and from our experiments, a simple average is what works best. We found that using a reliable reference is actually more important than having multiple references (if they are of lower quality)


whenever scheduled at the incorrect moment

Hi
In our deployment, using capistrano v2 with capinfony, whenever is scheduled as pretty much the first thing, even before bundler is called which leads to a execution failure for every first time deployment.
In the file:
https://github.com/javan/whenever/blob/334cfa01f373006cc032e23907b1777a8ea3f3b0/lib/whenever/capistrano/v2/hooks.rb#L5
Schould this not be 'after "deploy:finalize_update"'?
cheers,
Raffaele

Check your Capfile. See if you're explicitly require capistrano/bundler. You probably should, and it should be required before whenever/capistrano

There is a proposed solution for this at https://github.com/javan/whenever/pull/535. Please follow along there.


Add prisma and sequelize usage trend link

Hi Prisma Team,
I am a fan of the Prisma package and have greatly benefited from its features and capabilities. Thank you for your continuous efforts in maintaining and improving Prisma.
I have added a link to a usage trend comparison between Prisma and Sequelize. This comparison highlights Prisma's growing popularity over recent years. Including this link in the README can help new users quickly understand the benefits of choosing Prisma and how it stands out against a traditional ORM like sequelize.
I believe this small addition can make new users perceive and choose Prisma. Thanks for reviewing!

Thanks for the PR; but we are not interested in highlighting this angle in our documentation. Npm downloads are just one of many data points people should use to make a decision.


chore: add changelog and generator

I propose to start tagging "releases" in the same cadence as we do for backend - every sprint. So I created 0.11 (roughly to the sprint start) and 0.12 for today (demo time). And copied the changelog generation script and generated changelog. I think a copy of the script will do, we should not change it too often.
If this is approved and merged, I will also push the tags.

Changelog is published in GH repo only for now, but already used in GH actions, happy to merge the makefile and changelog tho :)

That's fine I will close there's not much. Pushing tags so we have a good starting point for the changelog for this sprint.
$ g push upstream --tags --no-verify
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/RHEnVision/provisioning-frontend
 * [new tag]         0.11.0 -> 0.11.0
 * [new tag]         0.12.0 -> 0.12.0


Unable to add tasks with / in its name

Tries to read it as a command instead of interpreting it as part of input

We will mention this in our User guide that the user is not allowed to use / in outside of commands.


[BUG] Sequences of fixed vs varying size handled inconsistently

🐛🐛 Bug Report
⚗ Current Behavior
Input Code
import hub
from hub.schema import Text, Sequence

@hub.transform(schema={"txts": Sequence(dtype=Text(max_shape=(100,)))})
def transform_ds(txts):
    return {"txts": txts}

# This works
rows = [
    ["one", "two"],
    ["one", "two", "three"],
]
transform_ds(rows).store("./tmp_ds")

# This throws `ValueError: parameter 'value': expected array with shape (2,), got (2, 2)`
rows = [
    ["one", "two"],
    ["one", "two"],
]
transform_ds(rows).store("./tmp_ds")

Expected behavior
I would expect it to work consistently across "ragged" sequences (rows with varying length) as well as ones where each row has the same number of elements.
⚙ Environment

Python version: 3.6.9
Packages: [ hub==1.3.5, zarr==2.7.1]

🧰 Possible Solution  (optional)
zarr.indexing.BasicIndexer.shape == (2,) in both of the above cases, which seems wrong. Not sure if this is a bug with zarr or hub is calling zarr incorrectly.

Hey there! Thanks for bringing this issue up. We'll try reproducing this issue and get back to you in some time!

@mattiasarro, apologies for the delay in answering! We still haven't come up with a permanent fix for this, but here's a quick patch to your issue: specifying different schemas to handle sequences of varying length vs static length.
Something like this:
from hub.schema import Text, Sequence
# Transform for sequences of varying length
@hub.transform(schema={"txts": Sequence(dtype=Text(max_shape=(100,)))})
def transform_ds(txts):
    return {"txts": txts}
rows = [
    ["one", "two"],
    ["one", "two", "three"],
]
transform_ds(rows).store("./tmp_ds")

# Transform for sequences of fixed length
@hub.transform(schema={"txts": Sequence(shape=(2,),dtype=Text(max_shape=(100,)))})
def transform_ds(txts):
    return {"txts": txts}
rows = [
    ["one", "two"],
    ["one", "two"],
]
transform_ds(rows).store("./tmp_ds")

Again, this is a quick patch and not a permanent fix. We'll be sure to work on this and fix it soon!

Thanks @dhiganthrao for the input! So does @FayazRahman 's PR not fix it? Or you mean the workaround can be used until the PR gets released in the next pip package?
For me, the schema of the dataset needs to be schema={"txts": Sequence(dtype=Text(max_shape=(100,)))}, because the txts field is variable in general. However when I use Hub transform to process the dataset in parallel / in chunks, it sometimes happens that a chunk has only rows where txts has the same number of elements, and then the exception is raised. Currently this happens at the end of my dataset (in the last chunk, which has very few elements), so my workaround has been to skip the data points that would go into the last chunk. Which is of course a temporary workaround until there's a permanent fix.

@mattiasarro I didn't get around to reviewing the PR, as it was closed. However, this issue should mostly get solved in 2.0 - since there's going to be a complete rewrite of the codebase, with some focus on dynamic tensors as well.

Just realised the PR is closed but not merged. Any rough timelines for when 2.0 will be released?

@mattiasarro @dhiganthrao Sorry guys I did something really stupid and saw the bug go away and thought it fixed it.

@FayazRahman don't worry about it! @mattiasarro, the alpha version of Hub 2.0 is expected to release on June 10!

Closing this issue, as dynamic sequences are properly handled in 2.0.
@mattiasarro, @FayazRahman - I'm also excited to announce we’ve released the 2.0 community beta - you can try it out using pip (pip install hub). You can access the docs here, and access some small tutorials to get started with 2.0 here! You can also let me know if you run into any issues, I'll be happy to help! If you haven't, please consider joining our Community Slack to receive further updates.


kms issue when the job bucket uses sse

The FrameworkProcessor failes twice if kms_key is not added to the class.

with S3Uploader.upload_string_as_file_body ; which can be fixed just adding the param from run
_upload_payload ; which fails on the estimator upload. The estimator class will only use kms_key (from output_kms_key) if the code_bucket is the same as the output_bucket -> https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/estimator.py#L2291.
So the fix here is to give the estimator an output_bucket with s3_prefix and pass in the kms_key from run to output_kms_key.

Happy to submit a pull request if needed. Thanks for the package, good stuff.

Hi @rareal, thank you for spotting this bug.
You're more than welcome to create a PR, preferably to this branch: https://github.com/verdimrc/sagemaker-python-sdk/tree/pr-framework-processor. This branch itself is being tracked by PR #2251 from the upstream sagemaker-python-sdk, and the code is updated much more than what's in this repo.
Do let me know.

Looks like the code in the other branch is considerably different then.
I see one issue, which is the function get_run_args, it is defined in the init but it needs to run _pack_and_upload_code, which would need the kms_key, but in the init it cannot take the kms_key from run.
Would the soluton be to pass the kms_key when instantiating FrameworkProcessor and not on run?


Casts defined in Trait are ignored

Larastan Version: 2.9.8
Laravel Version: 11.19

Description
Casts defined in Trait are ignored
Laravel code where the issue was found
trait MyTrait {

    public function initializeServiceTrait()
    {
        $this->mergeCasts([
             'price' => 'double',
             'quantity' => 'integer',
        ])
    }
}

Hello @makroxyz 👋🏻
Is your trait in use?

Yes it is, used in a couple of models

I think Larastan does not support mergeCasts.
https://github.com/search?q=repo%3Alarastan%2Flarastan mergeCasts&type=code

Yes, this is not supported. While we do use the getCasts method from the model (but not for the new L11 casts), we new up the model bypassing the constructor---which means traits are not booted/initialized:
https://github.com/larastan/larastan/blob/c8ec6f04f800d9425be743998574f1034e2ffb3f/src/Properties/ModelCastHelper.php#L240-L249

Hi,
I'm not sure if this can be ever solved without doing more voodoo black magic type of things. So I'm gonna close it for now.


Could NOT find WASM (missing:  WASM_CLANG)

Ubuntu 16.04 use eosio_build.sh install eosio error
error list:

-- Could NOT find WASM (missing:  WASM_CLANG)
CMake Error at CMakeModules/wasm.cmake:8 (message):
No WASM compiler cound be found (make sure WASM_ROOT is set)
Call Stack (most recent call first):
CMakeLists.txt:194 (include)

Beginning build version: 1.2
2018年 04月 05日 星期四 15:59:10 UTC
	git head id: c721c31d6fb0da210dccf03e2d7edfa0910c5144
	Current branch: * master

	ARCHITECTURE: Linux

	OS name: Ubuntu
	OS Version: 16.04
	CPU speed: 4294Mhz
	CPU cores: 1
	Physical Memory: 4449 Mgb
	Disk install: /dev/sda1
	Disk space total: 44G
	Disk space available: 31G

	Checking for installed dependencies.

	Package clang-4.0 found.
	Package lldb-4.0 found.
	Package libclang-4.0-dev found.
	Package cmake found.
	Package make found.
	Package libbz2-dev found.
	Package libssl-dev found.
	Package libgmp3-dev found.
	Package autotools-dev found.
	Package build-essential found.
	Package libbz2-dev found.
	Package libicu-dev found.
	Package python3-dev found.
	Package autoconf found.
	Package libtool found.
	Package curl found.

	No required dpkg dependencies to install.

	Checking boost library installation.
	Boost 1.66 found at /home/liang/opt/boost_1_66_0

	Checking MongoDB installation.
	MongoDB configuration found at /home/liang/opt/mongodb/mongod.conf.

	Checking MongoDB C++ driver installation.
	Mongo C++ driver found at /usr/local/lib/libmongocxx-static.a.

	Checking for secp256k1-zkp
	secp256k1 found

	Checking for LLVM with WASM support.
	WASM found at /home/liang/opt/wasm/bin

>>>>>>>> ALL dependencies sucessfully found or installed . Installing EOS.IO

-- Using custom FindBoost.cmake
-- Boost version: 1.66.0
-- Found the following Boost libraries:
--   thread
--   date_time
--   filesystem
--   system
--   program_options
--   signals
--   serialization
--   chrono
--   unit_test_framework
--   context
--   locale
--   iostreams
-- Configuring Eos on Linux
-- binaryen building with -std=c++11
-- binaryen building for platform x86-64
-- binaryen building with -Wall
-- binaryen building with -Werror
-- binaryen building with -Wextra
-- binaryen building with -Wno-unused-parameter
-- binaryen building with -fno-omit-frame-pointer
-- binaryen building with -fPIC
-- binaryen building with -O2
-- binaryen building with -UNDEBUG
-- Could NOT find WASM (missing:  WASM_CLANG)
CMake Error at CMakeModules/wasm.cmake:8 (message):
  No WASM compiler cound be found (make sure WASM_ROOT is set)
Call Stack (most recent call first):
  CMakeLists.txt:194 (include)

-- Configuring incomplete, errors occurred!
See also "/home/liang/liangwl/eos/build/CMakeFiles/CMakeOutput.log".
See also "/home/liang/liangwl/eos/build/CMakeFiles/CMakeError.log".

>>>>>>>>>>>>>>>>>>>> CMAKE building EOSIO has exited with the above error.

I also have the same problem.  -DLLVM_TARGETS_TO_BUILD= -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly -DCMAKE_BUILD_TYPE=Release ../
make -j4 install

@cppfuns do you have solve the problem

I get the same error (OS X + Xcode) and I don't have WASM installed.
. Maybe if you have time you can elaborate what you mean with "EOS will download and install...". Do you mean CMake will do this? But on the other hand - it does not seem to fix the missing WASM for me? Meanwhile, I will try to find WASM for Xcode toolchain somewhere...

@2liang Do you still encounter the problem with the latest code?

I'am having the same issue.
Deleting local WASM and then running eos build does not solve the issue.

@fredhampton suggest you prepare a clean system and clone eos rebuild it.