Prepare Istio 1.17

Modified version of #41015 (from 1.16)

This issue aims to track all steps needed to prepare the release 1.17 branches/builds.
This issue was created early to help document the steps using some new automation as well as any ordering changes, etc. Please update/add additional steps taken here so we can document for future releases.
Note that the actual coding steps done by the automation steps are left in this issue for reference only. You only need to run the automated steps, and not the individual instructions. Also note that the reference steps may be INCOMPLETE as the automation is updated, not necessarily the reference steps.
For now, the code to run is in https://github.com/istio/release-builder master branch. From within that branch, one can do a make shell and then within the shell run REPO_ORG=myorg STEP=x ./release/branch.sh (you can update the STEP in the trigger-branch file and not set it manually). This generates the changes for review (trygit status or git diff).  Adding DRY_RUN=false will cause the PRs to be created in the REPO_ORG.
Note that the automated steps 3 and on require the new release's branch in the various repos which are created by automated step 2.

[x] Set up release manager team - Make sure the new team has write access to the repositories. Istio_1.17_RMs

[x] Create a slack channel for the 1.17 release. Edit the description/topic of the channel to include RM's for the release, expected release date, etc.

[x] Send an announcement on Discuss pre-announcing branch cut (adding, but are branch cut notifications needed and where?)

[x] Send an announcement on Discuss announcing the start of branch cut and to stop merging of PRs (adding, but are branch cut notifications needed and where?)

(Automation step=1) Update dependencies. This must be done before branching so that PRs can be easily backported later. PR

Run go get github.com/envoyproxy/go-control-plane@main to update (Added in 1.17 in https://github.com/istio/release-builder/pull/1332).
Run the following command in istio/istio UPDATE_BRANCH=master ./bin/update_deps.sh; make gen (example)
[x] Merge PR in istio/istio. Wait for it to complete.
[x] Need to verify git code to allow commits and pulls to work for upcoming 1.18 branch cut.

(Automation step=2)  (new step because we are waiting on prior change to master before branching) Create a release-1.17 branch in every Istio repo - run commands export org=xyz ; export repo=xyz ; (git clone git@github.com:${org}/${repo}.git && cd $repo && git checkout -b release-1.17 && git push --set-upstream origin release-1.17).

istio/istio
istio/pkg
istio/api
istio/envoy  (NOT done as part of release-builder) ([x] Should update code to include this again? It seems Step=4 needs it, so will add it back) PR to add it back
istio/proxy
istio/client-go
istio/tools
istio/common-files
istio/release-builder
istio/enhancements
Explicitly skipped: istio-releases/pipeline, istio/tests, istio/istio.io, istio/test-infra, istio/gogo-genproto (No longer used and not done as part of release-builder 1.15 PR)
No PRs to merge as the new branches were created by automation

[x] Set up branch protection. Modify prow/config.yaml in test-infra to add release managers as owners for a branch. 1.17 PR

(Automation/Not very automated step=3) (doesn't work locally, appears to need credential to add a tag to an existing build-tools image)(new step because this might need to be done separately) Set up prow on release branches go run generate.go branch 1.17 (Note: Needed to run postsubmit in next step) (Contact Googler or ping in #test-and-release slack channel for help). 1.17 PR

Manual steps if Googler can't run automation:

Have Googler or someone with creds tag new build-tools, build-tools-proxy and build-tools-centos images, for example (replace hash with latest):

gcloud container images add-tag gcr.io/istio-testing/build-tools:master-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2 gcr.io/istio-testing/build-tools:release-1.17-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2
gcloud container images add-tag gcr.io/istio-testing/build-tools-proxy:master-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2 gcr.io/istio-testing/build-tools-proxy:release-1.17-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2
gcloud container images add-tag gcr.io/istio-testing/build-tools-centos:master-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2 gcr.io/istio-testing/build-tools-centos:release-1.17-81af2db75fc3fa7d1a27b08f6c2065aef10c5bb2

Create new branch off of latest test-infra master branch
Run make shell
You'll need to run a couple go get's for the tooling: go get github.com/kballard/go-shellquote go get istio.io/test-infra/tools/prowgen/pkg -- don't commit the modified go.mod and go.sum from these commands
Generate the jobs: go run tools/prowgen/cmd/prowgen/main.go branch 1.17
Generate the private jobs: go run ./tools/generate-transform-jobs/main.go branch 1.17
In the job configs prow/**/*-1.17.yaml, replace any image build-tools(-centos|-proxy):master with build-tools(-centos|-proxy):release-1.17
Verify there is no duplicated env entries of the following in the configs:- name: BUILD_WITH_CONTAINER
  value: "0"

Manually update testgrid/config.yaml manually, see last PR for entries to add and substitute the release (there is tooling to do this, just not sure where yet)
Manually update BASE_VERSION in the istio-{release} yaml from master to {release}
Run make gen
[ ] Update test-infra code to handle duplicate entries as well as master image changes done manually above for upcoming 1.18 branch cut.

[x] Merge PR in istio/test-infra. Wait for it to complete.

(Automation step=4) (new step since we want the automation from prior step to actually create the image after PR merges) PRS: 1 - istio/pkg 2 - istio/client-go 3 - istio/tools 4 - istio/common-files 5 - istio/enhancements 6 - istio/release-builder 7 - istio/istio 8 - istio/api 9 - istio/proxy 10 - istio/envoy

Updates istio/tools to build new release-1.17 build image (update BRANCH in docker/build-tools/build-and-push.sh. PR postsubmit will create new container images whose name will be used in the next step)
Update common-files in new release. You first have to manually update the common/Makefile.common.mk update-common target to point to the new release  (but not in common-files). In Step=5, the prow automation will actually call make update-common in these repos to do the actual make update-common against the new common-files release branch

istio/common-files - skipped. Covered in next step
istio/istio
istio/pkg
istio/api
istio/client-go
istio/proxy
istio/release-builder
istio/tools

Update CODEOWNERS to contain only the release managers for this release. Command: export org=xyz ; export repo=xyz ; (git clone git@github.com:${org}/${repo}.git && cd $repo && git checkout release-1.17 && git checkout -b release-1.17-codeowners && echo '* @istio/release-managers-1.17' > CODEOWNERS && git add CODEOWNERS && git commit -m 'Set release managers as CODEOWNERS for release-1.17' && git push --set-upstream origin release-1.17-codeowners).

istio/common-files  - skipped. Covered in next step
istio/istio
istio/pkg
istio/api
istio/proxy
istio/client-go
istio/tools
istio/release-builder
istio/enhancements

Stop publishing latest tags
Update istio/release-builder branch changes from master to new release in build.sh, publish.sh and manifests.
[x] Merge PRs from STEP 4. Wait for the new build images to be created at https://gcr.io/istio-testing/build-tools.
WARNING -- DO NOT MERGE As part of the STEP 4 PR post-submits, a new PR will be created in the istio/common-files repo with a title like: Automator: update build-tools<EMAIL_ADDRESS>in<EMAIL_ADDRESS>The work in this PR is only a portion of Step 5, and merging it will cause issues with reverting the repos to the main branch common-files. CLOSE this PR.

(Automation step=5) (new step since we need image from prior step)

Update istio/common-files to set release-1.17 build image (Update the UPDATE_BRANCH in files/common/Makefile.common.mk to be the new release name
Update IMAGE_VERSION in files/common/scripts/setup_env.sh  to be the new build image from prior step (found at https://gcr.io/istio-testing/build-tools).
Also Update CODEOWNERS to contain only the release managers for this release. 1.17 PR
[x] Merge PR from STEP 5. Wait until step 5 PR merges and all the postsubmit generated PRs (to update common files in repos) merge.
[x] Fix code getting image name to not use the arm version. PR

Additional steps that still need automation (if possible)

[x] Add new label: cherrypick/release-1.17: PR

[x] Create a 1.17 milestone: PR

[x] Fix up proxy automated job to pull from correct envoyproxy release branch if possible (set UPDATE_BRANCH). example (** NOTE there may need to be a revert of a PR for updates in main branch post Envoy 1.23.**) 1.17 PR

[x] Also, a manual update can be done to fix up the proxy and istio repos to point to latest in that branch. example - Not needed as automation already picked this change up.

[x] Bump master to version 1.17 in the release-1.17 branches

BASE_VERSION should update when a new base image is created (vulnerabilities scan fails)
UPDATE_BRANCH in update_deps.sh
[x] istio PR-1
[x] istio PR-2 (change actually in update_deps.PR).
[x] test-infra PR

[x] Bump 1.17 to 1.18 in the master branch

Update Istio: 1.18.0 to 1.18.0, 1.18-dev to 1.18-dev, etc: 1.16 PR
[x] istio PR

[x] Send an announcement on Discuss announcing branch cut complete and PRs can be merged again (adding, but are branch cut notifications needed and where?)

[x] Ask istio.io team to run job to update to use the 1.17 branch (probably @ericvn) PR

[x] Publish a beta release (from https://docs.google.com/document/d/1ilwz-HGSeiRDCFwP5rm-5d_UY6OOYL_nflx7bNggek0/edit#heading=h.qex63c29z2to, builds on release branch are labeled beta. Updating below names to reflect that.

[x] Run bin/update_deps.sh in istio/istio PR
[x] Trigger a 1.17 beta.0 build PR
[x] Some verification to verify build is good

You can download the build tar from the appropriate directory here: https://gcsweb.istio.io/gcs/istio-prerelease/prerelease/
untar the file and install, specifying the image repository. (ex: ./bin/istioctl install --set profile=demo -y --set hub=gcr.io/istio-prerelease-testing).
Run bookinfo from within the untar'd directory

[x] Publish a 1.17 beta.0 release

[x] Send an announcement on Discuss announcing availability of beta. May wait and add something about starting the first round of community testing. (Done in Slack) Discuss

[x] Prepare for Community Testing

Update the community testing worksheet
Updating the worksheet includes verifying the priorities and checking the tests are automated or not

[x] Verify the min and max K8s version supported for the release.

RMs usually determine the range of k8s versions and it appears on the TOC agenda at times. The max version is usually is the current K8s version and the min version is 3 past versions.

[x] Run the long-running tests on the beta and rc builds as you can. Unfortunately, Sam Naser seems to be the only person that knows how and can do this. Plan to run the long running tests on the latest rc build before the actual release. It take almost 48hrs for the test to run, so factor the time.

[x] Prepare the release notes for the release. Even though a major part of this is automated, it still takes time to clean up, sort and group the release notes. It also takes time for reviewers to review the content and make sure it is accurate. So factor in the time. Sample release note PR for 1.17
- To generate release notes, run the command ./gen-release-notes --notes ../../../istio --oldBranch 1.16.0 --newBranch release-1.17 (against the istio/istio and istio/api repos). Follow the instructions from here.

[x] Each time you publish a beta/rc/official release build, make sure that the build/helm charts/images are published correctly to Istio release repo/Helm repo/gcr repo.

[x] After official release, send out an announcement on announcement and release slack channel for Istio. Currently Craig Box handles all the twitter announcements related to Istio and he usually sends release related announcements on twitter. Coordinate with him. discuss, twitter

[x] Update the description in the release slack channel to reflect the official release date.

This is shipped! thanks everyone


Deletion and Reverse in Circular Linked List

Deletion and Reverse in Circular Linked List
Link
Difficulty: Medium
Given a Circular Linked List. The task is to delete the given node, key in the circular linked list, and reverse the circular linked list.
Note:
You don't have to print anything, just return the head of the modified list in each function.
Nodes may consist of Duplicate values.
The key may or may not be present.
Examples:
Input: Linked List: 2->5->7->8->10, key = 8

Output: 10->7->5->2
Explanation: After deleting 8 from the given circular linked list, it has elements as 2, 5, 7, 10. Now, reversing this list will result in 10, 7, 5, 2 & the resultant list is also circular.
Input: Linked List: 1->7->8->10, key = 8

Output: 10->7->1
Explanation: After deleting 8 from the given circular linked list, it has elements as 1, 7,10. Now, reversing this list will result in 10, 7, 1 & the resultant list is also circular.
Input: Linked List: 3->6->4->10, key = 9
Output: 10->4->6->3
Explanation: As there no key present in the list, so simply reverse the list & the resultant list is also circular.

Expected Time Complexity: O(n)
Expected Auxillary Space: O(1)
Constraints:

2 <= number of nodes, key  <= 105
1 <= node -> data <= 105

Solved
class Solution:
    # Function to reverse a circular linked list
    def reverse(self, head):
        if head is None or head.next == head:
            return head
        prev = None
        current = head
        next_node = None

        while True:
            next_node = current.next
            current.next = prev
            prev = current
            current = next_node

            if current == head:
                break

        head.next = prev
        head = prev

        return head

    def deleteNode(self, head, key):
        if head is None:
            return None

        if head.data == key and head.next == head:
            return None

        current = head
        prev = None

        if head.data == key:
            while current.next != head:
                current = current.next

            current.next = head.next
            head = current.next
            return head

        current = head
        while current.next != head and current.next.data != key:
            current = current.next

        if current.next.data == key:
            current.next = current.next.next

        return head


Reset subset of levels in multi-index from groupby

Multi-indices aren't supported, but are generated by groupby, which does allow grouping by multiple columns.  Should one immediately try to go back to a single index?  If so, it seems one is forced to do a computation because reset_index does not have a level argument, and set_index triggers a computation in most cases.  Can the computation be avoided if one wants to keep the partitioning from the groupby?
Basically, is there a way to drop just level b in the following w/o a computation?
import dask
import dask.dataframe as dd
import numpy as np
import pandas as pd

def make_df():
    raise RuntimeError

ddf_delayed = dask.delayed(make_df)()
meta = dd.utils.make_meta([('a', np.int64), ('b', np.int64), ('c', np.int64)])
ddf = dd.from_delayed(ddf_delayed, meta=meta)
ddf.groupby(['a', 'b']).sum().reset_index()

ddf = ddf.set_index('a')  # Triggers computation

I would say that multiindexes are kind of supported. Is there a workflow that you can share that doesn't work?

The dask code definitely seems to be hostile to multi-indices:
import pandas as pd
import dask.dataframe as dd
dd.from_pandas(pd.DataFrame({'a': [1], 'b': [2], 'c': [3]}).set_index(['a', 'b']), npartitions=1)

NotImplementedError: Dask does not support MultiIndex Dataframes.

You can back your way into multi-indices in certain ways, like by doing a groupby.  I've found, however, that when writing multi-index dataframes out and reading them back in (I use pyarrow), the ordering of the multi-index levels can change.  I've mostly abandoned trying to use them, once I've discovered they aren't 100% supported -- but I still have the occasional groupby.
It would be great to be able to drop a subset of levels (without reordering, so not triggering computation), so I'm posting this feature request.  It likely has to wait on the work to fully implement multi-indices (i.e. https://github.com/dask/dask/issues/1493)

Thank you for the additional context. I think it might be possible to handle this by supporting the as_index kwarg in pandas.groupby:
import pandas as pd

df = pd.DataFrame({"a": ["cat", "dog", "dog", "dog"], "b": [1, 0, 0, 1], "c": [1, 2, 3, 4]})
df.groupby(["a", "b"], as_index=False).sum()
#      a  b  c
# 0  cat  1  1
# 1  dog  0  5
# 2  dog  1  4

Does it seem like this would work for your situation?

Fundamentally, calling set_index on an unknown column will always trigger computation. Dask needs to figure out the divisions between each partition in the index. If you don't want to trigger that computation, only do reset_index.
If the issue is that you don't want to include a RangeIndex in your parquet file, then there is probably a kwarg in to_parquet to ignore the index.

The solution to avoid set_index is exactly what I so now, but it leaves me without having any of my columns as my index.
In the example above, I do not want to call set_index, but I do want a as the index.  I just want to drop the b column in the example above (at least, as long as true multi-index support is missing).  But I do not see a way to do it without calling reset_index followed by set_index.    So that is the crux of this issue.
If reset_index supported the level keyword, that would be one solution.   But without that, I was curious if there was some other magic combination of calls/keywords that could achieve the same effect.
[Technically, pandas does allowed unsorted indices.  Does dask always require a sorted index in the way it handles partitions?]

Even if there was some mechanism to include level in reset_index dask would still need to do some computation to know which rows are in which partition.

Does dask always require a sorted index in the way it handles partitions?

Yeah, the way that dask knows which rows are in which partition is it sorts the index (or you can tell it the index is already sorted to avoid this step) and then sets the index values at the boundary between partitions as divisions.
Some workarounds are: try to see if your function really has to have a as the index. If you find that it does, you might consider setting a as the index within your chunk function and then call reset_index at the end of the chunk function. But you should be careful with that approach since it might lead to wrong answers if your chunk function assumes that it has access to all adjoining rows.
Hopefully this is making some sense.

It does make sense.  I hadn't really understood how much indexing and partitioning are interlinked.    I am not morally opposed to a single a value was spread over a few partitions (say, if reset_index(level='b') has a parameter to keep the original partitioning), but I gather that does not fit the model.

I was chatting with @martindurant and he reminded me that you can use map_partitions as a layer of your graph. So you can do something like:
import pandas as pd
import dask.dataframe as dd

df = pd.DataFrame({"a": ["cat", "dog", "dog", "dog"], "b": [1, 0, 0, 1], "c": [1, 2, 3, 4]})
ddf = dd.from_pandas(df, 2).groupby(["a", "b"]).sum().reset_index()

ddf.map_partitions(lambda df: df.set_index("a"))

So for your original example:
import dask
import dask.dataframe as dd
import numpy as np
import pandas as pd

def make_df():
    raise RuntimeError

ddf_delayed = dask.delayed(make_df)()
meta = dd.utils.make_meta([('a', np.int64), ('b', np.int64), ('c', np.int64)])
ddf = dd.from_delayed(ddf_delayed, meta=meta)
ddf = ddf.groupby(['a', 'b']).sum().reset_index()

ddf.map_partitions(lambda df: df.set_index("a"))

Aha!  Neat idea.  That's great that a change of index in map_partitions gets propagated.
I wanted to see learn how partitioning would handle a values that were split across multiple partitions after a group by of [a, b].  But in the small test below, I didn't get far into the investigation.  The sum collapsed the frame to a single partition -- even for a groupby on single column.
I think I am still early-on in understanding dataframes.
import dask
import dask.dataframe as dd
import numpy as np
import pandas as pd

def make_df():
    raise RuntimeError

ddf_delayed = dask.delayed(make_df)()
meta = dd.utils.make_meta([('a', np.int64), ('b', np.int64), ('c', np.int64)])

ddf0 = dd.from_delayed(ddf_delayed, meta=meta)
ddf1 = dd.from_delayed(ddf_delayed, meta=meta)
ddf2 = dd.from_delayed(ddf_delayed, meta=meta)

ddf = dd.concat([ddf0, ddf1, ddf2])
print(ddf.npartitions)
ddf = ddf.groupby('a').sum()
print(ddf.npartitions)

Yeah by default aggregation on a groupby will do that. If you want to have more partitions in your output, you can use the split_out kwarg: For instance: ddf = ddf.groupby('a').sum(split_out=3)

I think this is good to close since it ended up being more of a usage question. But if you can think of a way to make the docs clearer please open a PR!


No settings button (gear icon) displayed (Windows)

Installed via BRAT
Browsed to plugin
Settings button (gear icon) is not displayed next to plugin

Perhaps this does not yet have Windows support?
Edit: my Obsidian version (v1.5.2):

Thank you for the report. You're right: I've tested everything except the very first run.
Fixed in https://github.com/pfrankov/obsidian-local-gpt/releases/tag/1.4.2
Please check and reply.
By the way, you can now install plugin from Obsidian store https://obsidian.md/plugins?id=local-gpt

Confirmed, appreciate the quick fix!

By the way, you can now install plugin from Obsidian store

Fwiw simply updating the plugin in Obsidian also works.


Installing Calligra error

I'm having latest homebrew installed on Lion. With fixed path (/usr/local/bin before /usr/bin) and while I'm trying to install calligra I'm getting such error:
$ brew install calligra
Error: /usr/local/Library/Taps/adymo/homebrew-kde/calligra.rb:64: syntax error, unexpected tSTRING_BEG, expecting tAMPER
        system "cmake", *cmake_args, "-DBUNDLE_INSTALL_DIR=."
                                      ^
/usr/local/Library/Taps/adymo/homebrew-kde/calligra.rb:72: syntax error, unexpected tSTRING_BEG, expecting tAMPER
                      "-DBUNDLE_INSTALL_DIR=#{bin}",
                       ^
/usr/local/Library/Taps/adymo/homebrew-kde/calligra.rb:72: syntax error, unexpected ',', expecting kEND
Please report this bug:
    https://github.com/Homebrew/homebrew/wiki/troubleshooting
/usr/local/Library/Homebrew/formulary.rb:65:in `require'
/usr/local/Library/Homebrew/formulary.rb:65:in `klass'
/usr/local/Library/Homebrew/formulary.rb:50:in `get_formula'
/usr/local/Library/Homebrew/formulary.rb:198:in `factory'
/usr/local/Library/Homebrew/extend/ARGV.rb:12:in `formulae'
/usr/local/Library/Homebrew/extend/ARGV.rb:12:in `map'
/usr/local/Library/Homebrew/extend/ARGV.rb:12:in `formulae'
/usr/local/Library/Homebrew/cmd/install.rb:27:in `install'
/usr/local/Library/brew.rb:131:in `send'
/usr/local/Library/brew.rb:131

Homebrew-kde was installed as in UG.

Calligra is no longer part of this repo


Allow proxy setting as param for sdk download via HTTP proxy

If the container is running in an environment with a HTTP proxy configured, we would not be able to download via the Python request module. The module respects environment variables, so allowing this to be passed as a param and set as an env var allows the download in this case

Tested on osm4 behind the proxy. Merging


With v3.1.0, Often parsed error for PDF417.

In fact the value is G0353141140061934133LBNLPNPNMHLBBOHGGD000, but often parse it to 47075146、02274849 or 47075146, and so on.

I do not get scanned pdf417. Other formats work.
Can you tell how you're doing?

I found very simple solution.


Add integration test for cache sync manager

$subject

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/9361225539

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/9361225539
Status: failure


OSV scan ref initial commit STO-6876

New topic: Open Source Vulnerabilities (OSV) scanner reference for STO
Added OSV to SAST open-source scanners list: Scanners supported by STO
Added to Early Access page: Open Source Vulnerabilities (OSV) scanner integration

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65a56d935a5ead1522530d16--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65b84202dd61696e461d5cc5--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65b904cadd6169f61c1d5b45--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65b98864996c4a5afed5a522--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65dcb3d980df4911b7a7a6ce--harness-developer.netlify.app


add support for TFVC in on-premise TFS resp. Visual Studio Team Services

both TFS 2015 and VSO support REST API to access TFVC besides GIT and therefore it would be possible to add basic support for TFVC.

@Daniel-Steiner I'm relatively certain that the REST API support for TFVC is relatively read-only.  You'd still need to use the heavyweight SOAP API to do things like checkout and checkin changes.
You could, however, shell out to tf to do TFVC integration.

YES!! :+1:

The other thing missing is an 'scm provider' API, that currently does not exist.

Does it mean implementing in javascript something like this? https://github.com/apache/maven-scm/tree/master/maven-scm-providers/maven-scm-provider-tfs
and/or SOAP in JavaScript:
http://www.codeproject.com/Articles/12816/JavaScript-SOAP-Client
I have not gone into the details of either project so don't know if those are too simplistic or incomplete projects.

@Code-DJ this is the next step. What I mean by an 'SCM provider API' is an API that avoids that you end up with a git viewlet, a TFS viewlet, a mercurial viewlet, etc. The provider API abstracts a bit from the underlying SCM so that you can build some common UI on top of it.

yes, please!!!!

Plugable SCM support in included in 1.11 🎉
http://code.visualstudio.com/docs/extensionAPI/api-scm
@joaomoreno was the guy and with luck we will see additional SCM providers popping up soon.  We know of at lest two and I'm going to let those authors tell you about them when they are ready but I know they are very close.  One of the teams works in Microsoft :)
Thanks everyone for pushing for this.
Sean - VS Code team member

Good news, the Visual Studio Team Services team just updated their extension:
https://marketplace.visualstudio.com/items?itemName=ms-vsts.team
The extension now supports TFVC using the new SCM Provider model.
So I'm going to close this request and let you start to give the extension a try and provide feedback in that repo.
Thank you all,
Sean


Missing "Buy me a coffee"

Hi,
I want to thank you so much for your work. This is the first and only easy to use working integration for any ventilation system. I think you should do more advertising for this because so lot people need this (i red 100000 of forums the last few month). It is simple and just working, also it is easy to use. And it is also working with more devices (master/slave) within one zone. Great job!!!
The only thing is, i cant thank you with more then an issue entry on github....maybe you give a chance for that with "Buy me a coffee" button or something.
Thx and kind regards Andreas

Hi TheACF,
I'm glad you are happy with this package. I find positive feedback or ideas for improvement is more valuable for me than a coffee. But if you want to 'Buy me a coffee', feel free to use this Link.
I don't do advertising, since i would be happier with an integration, not a package or any MQTT approach that displays a proper device and has better possibilities than this yaml code. Btw. have you tried the official HomeAssistant-integration-for-Ambientika? This repo was made public after I released my stuff. Maybe this is also a possible solution?
Best Regards,
Florian

First: Thank you for your donation :pray:
I'm closing this now, since Ko-fi Link is now in the Repo.


Adds notion of whether parsed timestamp for ShipmentEvent has a timezone

Information around timezones can be lost when creating tracking events.  We've attempted to normalize the shipment event time in tracking events to a "zoneless" fashion, but from an API consumer perspective, it can be handy to know if the UTC time value on the event is really UTC, or UTC only by virtue of the carrier not having supplied a timezone with their timestamp.
Examples:
USPS timestamps look a bit like:
      <EventTime>9:01 am</EventTime>
      <EventDate>April 28, 2015</EventDate>

with no other info as to convey the timezone.  It's probably based on the city/province/country in question, but perhaps it could be USPS server time.  In any case, it's ambiguous and timezone_ambiguous? would&should be true.
FedEx timestamps (most? all?) look like this:
<Timestamp>2014-11-18T18:13:00+11:00</Timestamp>

since the TZ is included in the timestamp, timezone_ambiguous? would&should be false.
This PR adds a method to be more sure whether an instance of ShipmentEvent is really zoneless or not

@qq99 We can backport this to 1.x when it's merged.

@jonathankwok any thoughts on naming method?
uses_timezone and has_timezone bug me a bit, since technically it's a Time instance so it always has some notion of a timezone on it.  Maybe timezone_unambiguous? or has_known_timezone??

reping for reviews

Alright, all should be well now

Spoke with and handed this off to @jonathankwok since I need to prioritize some other work.
I think we all agree we should properly note timezone ambiguity for each carrier doing tracking events before pushing this.  IMO this should be considered low priority since any new occurrences of the underlying issue are related to DST changeover time (old tracking numbers probably long irrelevant).
We should get it fixed up a bit before the next major DST changeover date, which sounds like it might be around Mar 12, 2017 for most of continental US

I think we can close this, @nicolaslupien found an elegant solution to handle these times differently, for any future readers see https://github.com/rails/rails/blob/e0397b0c25e790eba0c914d9ff97360219f49e29/activesupport/lib/active_support/time_with_zone.rb#L472-L478


Support Firefox in unit tests

Looks like a new playwright version is out that supports running from file protocol with Firefox: https://github.com/microsoft/playwright/releases/tag/v0.12.1
I have a branch https://github.com/microsoft/vscode/tree/ben/playwright-0.12.1 to play with and I can run the tests, though there are test failures:
4 pending
  5 failing

  1) URI URI#joinPath - /FIREFOX/:
     Missing expected exception..
  AssertionError@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:184:21
  fail@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:231:9
  _throws@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:444:9
  assert.throws@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:461:11
  @file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/base/test/common/uri.test.js:451:20

  2) URI URI#joinPath (posix) - /FIREFOX/:

      DIFFERENT from URL
      + expected - actual

      -file:///bazz
      +file://ser/bazz

  AssertionError@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:184:21
  fail@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:231:9
  equal@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:260:31
  assertJoined@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/base/test/common/uri.test.js:428:24
  @file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/base/test/common/uri.test.js:460:25

  3) Debug - Utils config presentation - /FIREFOX/:

      "a" == "p"
      + expected - actual

      -a
      +p

  AssertionError@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:184:21
  fail@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:231:9
  equal@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:260:31
  @file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/workbench/contrib/debug/test/common/debugUtils.test.js:125:20

  4) Decoration Render Options css properties - /FIREFOX/:

      false == true
      + expected - actual

      -false
      +true

  AssertionError@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:184:21
  fail@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:231:9
  ok@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:251:19
  @file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/editor/test/browser/services/decorationRenderOptions.test.js:49:19

  5) Decoration Render Options css properties, gutterIconPaths - /FIREFOX/:

      false == true
      + expected - actual

      -false
      +true

  AssertionError@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:184:21
  fail@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:231:9
  ok@file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/test/unit/assert.js:251:19
  @file:///Users/bpasero/Development/Microsoft/vscode-electron-4.0.x/out/vs/editor/test/browser/services/decorationRenderOptions.test.js:117:19

@bpasero steps to run these tests?
My test is failing due to different sorting on firefox, weird...

@isidorn

git co ben/playwright-0.12.1
git clean -xfd && yarn && yarn watch
yarn test-browser --browser firefox

@bpasero thanks. I have pushed a fix for the debug test.

Done as well

@bpasero time to merge your branch to master

@jrieken I merged the branch already a while ago, what is missing is to add a Firefox unit test to our builds. I did not do that in my branch, feel free to add it.

done. the default is now all browsers


Fix CI error

See: https://github.com/node-casbin/mongo-changestream-watcher/actions/runs/2927758423

@Sefriol

MongoDB 6.0 was released just today. I think that's the reason. Dockerfile uses the latest mongodb image. I'll check what are the breaking changes.

@Sefriol error: https://github.com/node-casbin/mongo-changestream-watcher/runs/8030455397?check_suite_focus=true

Hmmph. Tests complete before. I guess the semantic-release does not have access to the cluster anymore..
Should the release script run the tests anyway at this point?

@Sefriol refer to other repos: https://github.com/node-casbin

Other repos do not seem to have husky configured, but it seems that it should be ignored when semantic-release does the release commit.
It seems that I should just disable husky in CI.


Configure winrm connection on Terraform

This is to use provisioners on Terraform

Official documentation from Terraform: https://www.terraform.io/language/resources/provisioners/syntax

Other resource: https://ripon-banik.medium.com/using-winrm-in-terraform-82168362c2ed

Issue on Stackoverflow: https://stackoverflow.com/q/53440481/19465467

Enabling winrm: https://www.pdq.com/blog/how-to-enable-winrm-window-remote-management/

I decided to keep this as the base infrastructure for deploying VM on Azure.


2020 General Election Results

Can you provide any insight into how (if at all) data sources will be updated as election results roll in on Nov 3rd?
For instance, in the presidential toplines will the fields ecwin_inc and ecwin_chal (or some other value) be updated to reflect probabilities that take into account exit polls and/or precinct data as it comes in?
And/or will those (or other) fields change to 1 or 0 as states are called? If so, what source (i.e. authority) will be relied on to determine if a state can be called?
Thanks a lot for providing this data!
PS: If no such updates to the data will be made, can you indicate when the final update to this data will be released?
Thanks again!

States that cheat will be canceled by the New Republican House, if the
model fails to include that it is canceled now.
On Sun, Oct 18, 2020 at 7:28 PM Mike Vastola<EMAIL_ADDRESS>wrote:

Can you provide any insight into how (if at all) data sources will be
updated as election results roll in on Nov 3rd?
For instance, in the presidential toplines
https://projects.fivethirtyeight.com/2020-general-data/presidential_national_toplines_2020.csv
will the fields
https://github.com/fivethirtyeight/data/tree/master/election-forecasts-2020#presidential-forecast
ecwin_inc and ecwin_chal (or some other value) be updated to reflect
probabilities that take into account exit polls and/or precinct data as it
comes in?
And/or will those (or other) fields change to 1 or 0 as states are
called? If so, what source (i.e. authority) will be relied on to determine
if a state can be called?
Thanks a lot for providing this data!
PS: If no such updates to the data will be made, can you indicate when the
final update to this data will be released?
Thanks again!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fivethirtyeight/data/issues/266, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADEXKG5B3YP72LCXKRRUACDSLN2ZXANCNFSM4SVNHOGQ
.

States that cheat will be canceled by the New Republican House, if the model fails to include that it is canceled now.
@e-traudt well that was decidedly unhelpful.

For the Democratic primaries, they froze their forecasts on the day of major primaries. See South Carolina, Super Tuesday and a week after, and more that I can't find. I imagine they'll do the same for the general, though I don't work there and can't speak for sure!


Add support to read and write GenBank files

Interoperability with BioPython may be the easiest way to enable this.

Hi James,
Finally getting around to working on this, with my exams completed.
I was reading the SBOL specification manual and found this paragraph on GenericTopLevel.

Custom data can also be embedded at the top level of an SBOL document. The GenericTopLevel class is used to represent top-level entities whose purpose is to contain a set of annotations that are independent of any other class of SBOL object. Entities that have independent existence and are not recognized by the SBOL standard are deserialized to GenericTopLevel objects. These GenericTopLevel objects can be safely used by tools to exchange non-SBOL data.

Does the data from GenBank & similarly FASTA files go into this container?

Ideally a GenBank file would be converted into objects in the SBOL2 data model, such as ComponentDefinition, Sequence, SequenceAnnotation, etc.


Fix build on MSVC

Download all snapshots and such from static.rust-lang.org (drive-by fix)
Update dependencies to work and build on MSVC

ssh2-sys uses nmake to build libssh2
libgit2-sys uses cmake to build libgit2
curl-sys uses nmake to build libcurl
libz-sys uses nmake to build zlib
miniz-sys uses gcc-rs to drive cl.exe manually
term updated to pick up windows-specific deps on MSVC

Updated .travis.install.deps.sh to install MSVC target libraries on Windows,
so the compiler can be used to target MSVC. A 64-bit target is now used by
default as well.
Updated the configure script and Makefile to set appropriate environment
variables to build Cargo from a MSYS shell with a MSVC target. This is similar
to how Rust bootstraps on MSVC as well.

With this commit Cargo can successfully bootstrap itself on MSVC (use the MSVC
product to build itself again). The tests currently do not pass because
unwinding is not implemented, but soon!
Closes #1725

r? @wycats
(rust_highfive has picked a reviewer for you, use r? to override)

:umbrella: The latest upstream changes (presumably #1735) made this pull request unmergeable. Please resolve the merge conflicts.

r? @brson

:umbrella: The latest upstream changes (presumably #1740) made this pull request unmergeable. Please resolve the merge conflicts.

:pushpin: Commit 0da6c94 has been approved by brson

@bors r+

:hourglass: Testing commit 0da6c94 with merge a2dd2ac...

:sunny: Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64

:umbrella: The latest upstream changes (presumably #1724) made this pull request unmergeable. Please resolve the merge conflicts.

^This pull request made this pull request unmergeable???


JCenter to mavenCentral

currently, the jcenter is down, and it's better if we can fix this on open_filex.
thank you

@javaherisaber can you look into it?

@BetterB0y
According to (this)[https://blog.gradle.org/jcenter-shutdown] document, JCenter will be read-only indefinitely

JFrog has decided to keep JCenter as a read-only repository indefinitely. New package and versions are no longer accepted on JCenter. All Bintray services have been shutdown.

Have you seen any issues while building the library?

Yes. Yesterday the whole repo was 403 error and unreachable.
Here is stack thread
https://stackoverflow.com/questions/74258160/is-jcenter-down-permanently-31-oct

Today is also 403. Check by yourself.
https://jcenter.bintray.com

@BetterB0y
I just checked again and it's not throwing 403 for me, (I also cleaned my cache and retried)
Anyway I checked, replacing JCenter with MavenCentral satisfies the dependencies
So I can merge your change and make it future proof
Thanks for noticing and the PR


Doom purge doesn't purge straight build directory on version switch

What did you expect to happen?
If I have
.local/straight/build-28.0.50 and .local/straight/build-28.0.60 folders
And I run
doom sync -p
Then, as I'm using 28.0.60 version of emacs, build-28.0.50 is deleted
What actually happened?
The build-28.0.50 directory is not deleted, and I suspect this messes up with load-path and load-order (since I had a lot of xxxx.el is newer than byte-compiled file warnings)
Describe your attempts to resolve the issue
Deleting the old build-28.0.50 file manually seems to solve the issue (i.e. I have new bugs to deal with, solely on me :( )
Steps to reproduce

Keep 2 versions of Emacs
EMACS=emacs-28 doom sync
EMACS=emacs-29 doom sync -up

System Information
http://0x0.st/-7ht.txt

I think the crux of the issue is that doom-cli-packages-purge only ever mentions (straight--build-dir) https://github.com/hlissner/doom-emacs/blob/c93790bb5cbd8d70a778670b018442fc71f74727/core/cli/packages.el#L620 which changes value on change of Emacs version, so the reference to the previous build folder gets lost


Nieuws page - Discord channel messages [5]

https://github.com/Tristandemuijnck/Wum/commit/df45ffff71d7634775661caec198c989c687829b
Done


GenericStatusEffect chemistry effect does not add components

Description

It seems totally busted. Adjusting existing effects might work, but it isn't adding new ones. A good example right now is barozine, which is supposed to add the PressureImmunity status effect and component.
Reproduction

Take barozine
Open VV
Observe, contrary to what the YAML would imply, you have no pressure immunity component or status effect.

Did a test on 4fbea3cd1397fe88024713674ff5b16aee59aaac. Keep in mind that the space medipen contains Barozine.
Tried it, worked fine on both a human and a slime person.
You mentioned "taking" barozine - how was this performed?

Tried it, worked fine on both a human and a slime person. You mentioned "taking" barozine - how was this performed?
Using VV, it seems like it works some of the time when I test again (on a branch synced with latest master). However, anything other than removing time from an existing status effect seems to be somewhat unreliable. If I can get something more concrete I will reopen


Format validator.rs

Attempt to fix the linting for the other PRs

I don't like deleting large sections of code in PRs - it feels wrong somehow so apologies for that. On the other hand the lints on other PRs are failing and warning noise is not helpful. Happy to roll back the deletions (or abandon the PR) if you'd prefer.

hey, sorry for that... I ended up fixing the errors before realizing now you had written a PR...

No problem. Glad to see it's fixed


about the feature extraction??

I noticed a feature named 'alpha' in your file - feature_extraction_xx.m, does thie feature has any Physiological significance？ could you give any link about this feature definition?
S=real(ifft(fft(ppg))), can you tell me the function of this code?
thank you very much! Waiting your reply.

@v3551G

Yes, it is called Womersley number. Please check this out : https://www.ncbi.nlm.nih.gov/pubmed/31051486
I think you left a couple of lines there. The exact code is as follows:
Yy = fft(O1P); Z=Yy(1); Yy(1)=0; S=real(ifft(Yy));
So, here we convert the PPG signal to frequency domain and set the DC component to zero so that we only have the AC components of the signal. After that we convert it back to time domain and find the peaks of the signal ( in the later parts of the code as seen in feature_extraction_xx.m ) which helps us to find meu, which helps us find alpha.


feat(compiler-core): treat v-for with constant exp as a stable fragment

These are stable fragment?

Good job 👍


Use environment variables for credentials

Hello!
It would be great if we can use environment variables for credentials aws-envvars.
Me and my team using aws-vault to provide credentials to all tools we use.

Version 0.1.3 (https://github.com/dutchcoders/cloudman/releases/tag/0.1.3) will have an extra argument --use-env which will use credentials from environments variables.


javascript part I don't understand

Hi !
Sorry to bother you but I found out your article online and wanted to try all the solutions you did to insert some csv file into DynamoDB but i dont' get this part of code :/
const batchOf = (size = 25, execute) => {
  const items = [];
  return async (item, last = false) => {
    if (item) {
      items.push(item);
    }
    if (last || items.length === size) {
      await execute(items);
      items.length = 0;
    }
  };
};

First I didn't try with thousands of lines in the csv but just 5.
I don't understand how the  params last can ever be true because when I tried with only 5 items it never goes to the execute part.
Second i think it's more javascript related but I don't get how can you have the item as parameter of the function ?
I mean if i try to split the function :
const batchOf = (size = 25, execute) => {
    const items = [];
    async function passItem (item, last = false) {
        if (item) {
            items.push(item);
        }
        if (last || items.length === size) {
            await execute(items);
            items.length = 0;
        }
    };
    return passItem(?, ? );  // i should pass some parameters here..
};

How come is it working for the parameters to be pass to the function when you do the return async ? because in my example I need the declaration of the item parameter.
If you could help me for my 2 questions it'd be greatly appreciated.
Thanks a lot :)

Hi @cybernadinou, glad you asked the question!
re: I don't understand how the params last can ever be true because when I tried with only 5 items it never goes to the execute part.
The last param has to be passed in. I admit that it's a bit complicated, because batchOf is actually a function that returns a function. Even more complicated is that it's a non-async function that returns an async function...
I've written a standalone example that hopefully makes it clearer how it works:
const batchOf = (size = 25, execute) => {
  let items = [];
  return async (item, last = false) => {
    if (item) {
      items.push(item);
    }
    if (last || items.length === size) {
      await execute(items);
      items.length = 0;
    }
  };
};

const main = async () => {
  let processedItemCount = 0;
  const batchRunner = batchOf(2, items => {
    processedItemCount += items.length;
    console.log(`Executing batch of ${items.length} items`);
  });

  const itemsToProcess = [1, 2, 3, 4, 5, 6, 7, 8, 9];

  for (let i = 0; i < itemsToProcess.length; i++) {
    const item = itemsToProcess[i];
    const isLast = i === itemsToProcess.length - 1;
    console.log(`Adding item '${item}' to batch processor...`);
    await batchRunner(item, isLast);
  }

  // This won't work because the forEach won't wait for the async function to complete execution.
  //itemsToProcess.forEach(async (item, index) => {
    //console.log(`Adding item '${item}' to batch processor...`);
    //const isLast = index === itemsToProcess.length;
    //await batchRunner(item, isLast);
  //});

  console.log(`Processed ${processedItemCount} items.`);
};

main().then(() => console.log("Done"));

You should be able to run that with node <filename>.js.
Does that help?

Hi !!
Yes that does help to understand how things works.
So just 2 questions now :
Because in our case we need to have a stream so we can't do the for loop , does it means I need to do :

const parser = fastCsv
    .parseFile(fileName, { headers: true, delimiter })
    .on("error", (error) => console.error(error))
    .on("data", async (row) => {
      parser.pause();
      try {
        await processor(row);
        rowIndex++;
      } catch (err) {
        console.log(`Error processing row ${rowIndex}: ${err}`);
      } finally {
        parser.resume();
      }
    })
    .on("end", async (rowCount) => {
      await batch(null, true); => processor(null, true) ?
      console.log(`Parsed ${rowCount} rows`);
    });

Because on the part on("end" , you call a function batch but it's not defined anywhere :/
Second question :
  // This won't work because the forEach won't wait for the async function to complete execution.
  itemsToProcess.forEach(async (item, index) => {
    console.log(`Adding item '${item}' to batch processor...`);
    const isLast = index === itemsToProcess.length;
    await batchRunner(item, isLast);
  });

Is it because it's not supported by javascript yet that's why we need to go through the for loop ?
Sorry for the dummy question :/
Thanks for your help.

Yes, looks like a bug. Should have been a call to processor. I guess I did a bad git commit, because there's no way that would run, sorry for the confusion! I was testing on a huge file, so it's likely I never let it get to the end of it, it would have taken a long time with the JavaScript solution.

Async functions return a Promise instead of accepting a callback function as an argument. If you call an async function without awaiting it await fn() or handling the promise fn().then(() => doThis()) then execution just moves on to the next statement.

The forEach function accepts function argument, but it doesn't expect to have to look at the return value fo that function and wait for any promises returned by it to resolve.
So, the forEach happens instantly, while in the background n number of things are happening in promises that aren't being tracked - some might succeed, others might fail, but no code is tracking what happens to the promises. If the program exits, then all execution of that background activity stops.
For example, if you don't wait for the slow counter to complete before your program exits, the count won't complete:
const main = async () => {
  let count = 0;
  // Slow adder takes 250 ms to add 1 to a number.
  const slowAdder = n =>
    new Promise((resolve, reject) => {
      setTimeout(function() {
        count += n;
        resolve();
      }, 250);
    });

  // Wait for each count to be added up.
  for (let i = 0; i < 5; i++) {
    await slowAdder(1);
  }
  console.log(`Expected count of 5, got ${count}`);

  // If we don't wait, we won't finish our counting.
  for (let i = 0; i < 95; i++) {
    slowAdder(1);
  }
  console.log(`Expected count of 100, got ${count}`);
};

main().then(() => console.log("Done"));

Plop !
So now I'm trying to fix the issue about the items left but as your code is way to advance for me, here is what I'm trying to do :
let itemsToProcess = [];
let sizeChunk = 25;
stream
    .pipe(fastCsv.parse({headers : true}))
    .on("data", async (row) => {
        stream.pause();
        try {
            if(itemsToProcess.length === sizeChunk){
                console.log('on data');
                console.log(itemsToProcess.length);
                await batchWrite(client, itemsToProcess);
                itemsToProcess = [];
            }
            itemsToProcess.push(row);
        }catch (e) {
            console.log(e);
        } finally {
            stream.resume();
        }
    })
    .on("error", (err) => { console.log(err)})
    .on("end", async () => {
        console.log('end');
        console.log(itemsToProcess.length);
    });

The result is so weird...
I got into my console :
on data
25
on data
25
on data
25
on data
25
on data
25
end
25

Expected result :
on data
25
on end
5

I'm seriously lost right now...
Thx again for you help.

You probably want something like this, taking a copy of the data to prevent mutation of it.
import "path";
import fastCsv from "fast-csv";
import * as fs from "fs";
import * as path from "path";

const batchWrite = async (items) =>
  items.forEach((item) => console.log(`processing: ${JSON.stringify(item)}`));

const main = () =>
  new Promise((resolve, reject) => {
    let itemsToProcess = [];
    let sizeChunk = 5;
    const stream = fs.createReadStream(path.resolve("parse.csv"));

    stream
      .pipe(fastCsv.parse({ headers: true }))
      .on("data", (row) => {
        itemsToProcess.push(row);
        if (itemsToProcess.length === sizeChunk) {
          console.log("processing chunk...");
          stream.pause();
          batchWrite(new Array(...itemsToProcess)).finally(() => {
            stream.resume();
          });
          itemsToProcess.length = 0;
        }
      })
      .on("error", (err) => {
        reject(err);
      })
      .on("end", () => {
        stream.pause();
        console.log("processing end...");
        batchWrite(itemsToProcess).then(() => resolve());
      });
  });

main().then(() => console.log("done"));

With a file of parse.csv in the same directory:
id,value
1,a
2,b
3,c
4,d
5,d
6,e
7,f

It outputs:
processing chunk...
processing: {"id":"1","value":"a"}
processing: {"id":"2","value":"b"}
processing: {"id":"3","value":"c"}
processing: {"id":"4","value":"d"}
processing: {"id":"5","value":"d"}
processing end...
processing: {"id":"6","value":"e"}
processing: {"id":"7","value":"f"}
done

Thanks a lot it's working fine now ! Basically it's because I didn't put the stream.resume() at the good place..
But I thought that :
try {
  await myFunctionReturningPromise();
} catch (e) {
  console.log(e);
}
finally {
   //final code here
}

was equal to this :
myFunctionReturningPromise().finally(() => {
   // final my code
});

Am I wrong ?
One more time, thank you very much for your help !

Thanks a lot it's working fine now ! Basically it's because I didn't put the stream.resume() at the good place..
But I thought that :
try {
  await myFunctionReturningPromise();
} catch (e) {
  console.log(e);
}
finally {
   //final code here
}

was equal to this :
myFunctionReturningPromise().finally(() => {
   // final my code
});

Am I wrong ?
One more time, thank you very much for your help !

The thing you were missing was really that you'd done this:
.on("data", async (row) => {

If you look at my code, it does this:
.on("data", (row) => {

The function passed into the on("data", ...) call cannot be asynchronous due to the design of that library. Maybe the library could wait for the promise to complete, but there's probably a good reason it doesn't.


Just to notify you guys :)

Hey there,
That tool is still running, http://streamplayer.swisstxt.ch/.
Just to let you know :)
I hope you're all well, have a nice day ;)
@onitake @joschi36 @gildegoma
Liebi Grüess,
Maël

It's open source. :wink:
How's it going? Not getting bored at your new job?

@onitake Yeah, just wanted to let you know that there's a random tool, written by an ex-apprentice which is running publicly under a subdomain ;D
It's going pretty well, yes. I switched job again, from DevOps Engineer to Linux System Engineer. Currently attending a Cisco Hackathon.
How are you guys doing? I guess at the moment it's hard to meet each other physically, maybe we could organise a virtual FüBi with a couple of other ex-TXTlers and current TXTlers :)

Oh wow. Does that mean you also switched companies?
FüBi sounds good. I'll try to put something together with Nik.

@onitake Sorry, I always forget to check out Github. We use GitLab, so I'm not on here a lot :D
Yeah, I know work at SPIE in Bern. It's quite funny, the team structure and spirit is quite similar to when I was at SwissTXT, so it feels kind of "back to the roots". But it's very refreshing to be working in the Linux space again.
Nice :) See ya sometime soon, be it virtually or phisically.

By the way, if you go to http://streamplayer.swisstxt.ch/ and put "Daniel" in the text box, you can experience an easter-egg

Fabulous!
By the way, your website directly embeds content from Facebook, LinkedIn and Xing without asking. Haven't gotten any GDPR complaints yet? :stuck_out_tongue_winking_eye:

That tool was developed in the good old times, when you didn't have to make your users aware that you're abusing their data.
But no, haven't heard anything about that ;D If I do, I'll forward the complaints to<EMAIL_ADDRESS>

I meant your employer's website, not the streamplayer. :wink:

Ah, you're right, never noticed that. I'll see what happens if I mention it :)
swisstxt.ch does not show me one, either :smirk:

Oh yeah, I might want to drop our legal team a note... CH is not in the EU, but we're not blocking outside access, so....


What's new in .NET 7 RC 1

What's new in .NET 7 RC 1
This issue is for teams to highlight work for the community that will release .NET 7 RC 1
To add content, use a new conversation entry. The entry should include the team name and feature title as the first line shown in the template below.
## Team Name: Feature title

[link to the tracking issue or epic item for the work]

Tell the story of the feature and anything the community should pay particular attention
to be successful using the feature.

Preview 1: https://github.com/dotnet/core/issues/7106
Preview 2: https://github.com/dotnet/core/issues/7107
Preview 3: https://github.com/dotnet/core/issues/7108
Preview 4: https://github.com/dotnet/core/issues/7378
Preview 5: https://github.com/dotnet/core/issues/7441
Preview 6: https://github.com/dotnet/core/issues/7454
Preview 7: https://github.com/dotnet/core/issues/7455
RC 1: https://github.com/dotnet/core/issues/7716
RC 2: https://github.com/dotnet/core/issues/7717

Note: This is actually a preview 7 feature, but I missed the deadline with the update to #7455, so posting it here. Might be better to update the Preview 7 post with this info?
Detecting HTTP/2 and HTTP/3 protocol errors
When using HttpClient with the default SocketsHttpHandler, it's possible now to detect HTTP/2 and HTTP/3 protocol error codes (not to be confused with HTTP status codes!). This feature is useful for advanced applications like GRPC.
Usage
When calling HttpClient methods
using var client = new HttpClient();

try
{
    var response = await client.GetStringAsync("https://myservice");
}
catch (HttpRequestException ex) when (ex.InnerException is HttpProtocolException protocolException)
{
    Console.WriteLine("HTTP2/3 protocol error code: " + protocolException.ErrorCode)
}

When calling response stream methods
using var client = new HttpClient();
using var response = await client.GetAsync("https://myservice", HttpCompletionOption.ResponseHeadersRead);
using var responseStream = await response.Content.ReadAsStreamAsync();
using var memoryStream = new MemoryStream();

try {
    await responseStream.CopyToAsync(memoryStream);
}
catch (HttpProtocolException protocolException) {
    Console.WriteLine("HTTP2/3 protocol error code: " + protocolException.ErrorCode)
}

References
See https://github.com/dotnet/runtime/issues/70684

FYI: There is a feature we didn't get time to document for P7 (due to vacation, resourcing, etc.) - sorry for that! -- Perhaps we can include it in RC1 blog post? Or maybe the P7 response will be turned into P7 blog post update? @leecow can you please advise? Thanks!
https://github.com/dotnet/core/issues/7455#issuecomment-1225655505

@karelz we can add it to the RC1 blog post, it's not the first time that happens. If we updated the P7 post nobody would see it. 😅

WebSockets over HTTP/2
The WebSocket Protocol over HTTP/2 implementation is based on the RFC 8441. It uses a different approach than web sockets over HTTP/1.1 due to the multiplexing nature of the HTTP/2 connection. It allows to have streams of both protocols in one connection simultaneously and use the network more efficiently.
To enable web sockets over HTTP/2 in a specific connection, a server should explicitly indicate support by sending the SETTINGS_ENABLE_CONNECT_PROTOCOL. In case when this setting or the entire HTTP/2 protocol is not supported, you could allow downgrade by setting up the respective policy in the ClientWebSocketOptions.
By default ClientWebSocket uses the previous configuration of the HTTP/1.1 version and RequestVersionOrLower policy. To manage the connection with several streams of one or both protocols, you might want to provide a custom HttpMessageInvoker. In that case, you should use the overlapping options from the invoker instead of ClientWebSocketOptions.
Usage
var handler = new SocketsHttpHandler();

ClientWebSocket ws = new();
ws.Options.HttpVersionPolicy = HttpVersionPolicy.RequestVersionOrHigher;
ws.Options.HttpVersion = HttpVersion.Version20;

ws.ConnectAsync(uri, new HttpMessageInvoker(handler), cancellationToken);

//------------------------------------------------------------------
// alternative direct usage of HttpClient instead of ClientWebSocket
HttpRequestMessage request = new(HttpMethod.Connect, server.Address);
request.Version = HttpVersion.Version20;
request.Headers.Protocol = "websocket";

References
See https://github.com/dotnet/runtime/issues/69669

@ericstj @jeffhandley anything from your teams?

Support using ICU library by default on Windows Server 2019
In .NET 5.0, .NET started to use the ICU library for globalization support. Windows Server 2019 was lacking ICU support. Services and apps running on Windows Server 2019 and want to use ICU needed to deploy ICU and enable some configuration to use it as described in the doc. In .NET 7.0, the ICU will be supported by default on Windows Server 2019.
References
See https://github.com/dotnet/runtime/issues/62329, https://github.com/dotnet/runtime/pull/72656, and https://github.com/dotnet/docs/issues/30319

Improve the calculation precision of Add methods in DateTime and DateTimeOffset
Improved the calculation precision of the methods AddDays, AddHours, AddMinutes, AddSeconds, AddMilliseconds, and AddMicroseconds to produce a better result.
References
https://github.com/dotnet/runtime/issues/66815 and https://github.com/dotnet/runtime/pull/73198

System.Diagnostics.TraceSource can now be initialized from the app.config file
Primarily to make it easier to migrate from .NET Framework, support was added for initializing a TraceSource and related types including Switch and TraceListener from the application's config file.
Note that an explicit call must be made to enable this functionality through System.Diagnostics.TraceConfiguration.Register().
Usage
A package reference to System.Configuration.ConfigurationManager is required.
An example C# console app named ConsoleApp1 would use a config file named ConsoleApp1.dll.config and placed in the application's bin folder.
Sample of a config file:
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.diagnostics>
    <sources>
      <source name ="MyTraceSource" switchName="MainToggle" switchType="System.Diagnostics.SourceSwitch" />
    </sources>
    <switches>
      <add name = "MainToggle" value="Error" /> <!-- Set to Off to disable logging -->
    </switches>
  </system.diagnostics>
</configuration>

and C#
using System.Diagnostics;

namespace ConsoleApp1
{
    internal class Program
    {
        static void Main(string[] args)
        {
            TraceConfiguration.Register(); // Required to enable reading from the config file.

            // Get the configured TraceSource
            TraceSource ts = new TraceSource("MyTraceSource", SourceLevels.Error);

            // Add a listener
            TextWriterTraceListener listener = new("TextWriterOutput.log", "myListener");
            ts.Listeners.Add(listener);

            // The "MainToggle" SourceSwitch in the config file is set to Error, so this will log:
            ts.TraceEvent(TraceEventType.Error, 101, "Error message.");
            listener.Flush();
        }
    }
}

References
See https://github.com/dotnet/runtime/issues/23937 and https://github.com/dotnet/runtime/pull/73087

Support XmlSchema Import/Export with DataContractSerializer
.NET 7.0-RC1 brings the return of XmlSchema importing and exporting from .NET Framework 4.8 in the DataContractSerializer space. The API is as similar to the 4.8 API as possible to enable easy porting of code from .NET Framework. The export functionality is a built-in feature along with DataContractSerializer in the 7.0 SDK. The import functionality is available in an add-on package named System.Runtime.Serialization.Schema. (This package is not part of the 7.0 SDK as it depends on System.CodeDom, which is also shipped as a separate package.)
References
See https://github.com/dotnet/runtime/issues/72243, and the 4.8 Export and Import API docs for the Framework feature that preceded this work.

(copying https://github.com/dotnet/core/issues/7455#issuecomment-1225655505 here to make sure you don't miss it)
Detecting HTTP/2 and HTTP/3 protocol errors
When using HttpClient with the default SocketsHttpHandler, it's possible now to detect HTTP/2 and HTTP/3 protocol error codes (not to be confused with HTTP status codes!). This feature is useful for advanced applications like gRPC.
Usage
When calling HttpClient methods
using var client = new HttpClient();

try
{
    var response = await client.GetStringAsync("https://myservice");
}
catch (HttpRequestException ex) when (ex.InnerException is HttpProtocolException protocolException)
{
    Console.WriteLine("HTTP2/3 protocol error code: " + protocolException.ErrorCode);
}

When calling response stream methods
using var client = new HttpClient();
using var response = awaitclient.GetAsync("https://myservice", HttpCompletionOption.ResponseHeadersRead);
using var responseStream = await response.Content.ReadAsStreamAsync();
using var memoryStream = new MemoryStream();

try
{
    await responseStream.CopyToAsync(memoryStream);
}
catch (HttpProtocolException protocolException)
{
    Console.WriteLine("HTTP2/3 protocol error code: " + protocolException.ErrorCode);
}

References
See dotnet/runtime#70684

I didn't see list of above features in RC1 blog post https://devblogs.microsoft.com/dotnet/announcing-dotnet-7-rc-1/ -- did I miss it, or was it intentional? @JeremyLikness @leecow

[RegexGenerator] attribute was renamed to [GeneratedRegex]. https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/ is edited to note that.

@JonDouglas the above info was missed from the RC1 post. Could we please include in RC2? Cc @leecow

@danmoseley We missed this issue after looking for it last week. Which is also why the blog PR wasn't linked to this issue.
Since it's only been a week, perhaps we can just backfill this content to RC1? (Still healthy viewership).
I tagged us for the RC2 issue so we have visibility.
Thanks for the heads up!

@JonDouglas if we add it into RC1 blog post 1 week late (which is IMO fine), perhaps we can then mention the fact (of late update) in RC2 blog post and catch 1st-week set of viewers?

@karelz Sure thing. We will also include it in the RTM blog more expanded.

.NET 7 GA is available. Closing these pre-release issues.


What does ... mean?

I'm sorry I don't know where to ask. What does ... mean, Is it a legit syntax or something?

@mcleanbyron The PR passed the acceptance checklist. Please evaluate further and approve/complete the PR if appropriate. Thanks!

Thanks for your contribution!


Top level statements tutorial

Fixes #20593
Create tutorial for using top-level statements.
This tutorial enables readers to explore top level statements to learn the syntax rules for using top-level statements, learn to create script-like programs, and refactor those script-like programs into maintainable utilities.

@CamSoper Thanks for the review. I've made all the requested updates. Can you approve?

@BillWagner This introduced a build warning.


Can not run on Android: TypeError: undefined is not a function

Hi, I try install and run this lib for my project, but it show error: TypeError: undefined is not a function

@nguyenvanquan7826  android lib not link (use 'react-native config' see it) -> use 'worklet' key in function error when metro run

Also, did you follow the SETUP guide?

nope :) you not setup to auto link android . use react-native.config.js with content :
 dependencies: {     'react-native-multithreading': {       platforms: {         android: {           sourceDir: '../node_modules/react-native-multithreading/android',           folder: '../node_modules/react-native-multithreading',           packageImportPath:             'import com.reactnativemultithreading.MultithreadingPackage;',           packageInstance: 'new MultithreadingPackage()',           buildTypes: [],         },       },     },

Hi @mrousavy  and @hungnq139 . I was setup in MainApplication.java and create react-native.config.js, but when run, it show error:
`Could not determine the dependencies of task ':app:compileDebugJavaWithJavac'.

Could not resolve all task dependencies for configuration ':app:debugCompileClasspath'.
Could not resolve project :reactnativemultithreading.
Required by:
project :app
No matching configuration of project :reactnativemultithreading was found. The consumer was configured to find an API of a component, as well as attribute 'com.android.build.api.attributes.BuildTypeAttr' with value 'debug', attribute 'react-native-camera' with value 'general' but:
- None of the consumable configurations have attributes.
`

// react-native.config.js
module.exports = {
    dependencies: {
        'react-native-multithreading': {
            platforms: {
                android: {
                    sourceDir: '../node_modules/react-native-multithreading/android',
                    folder: '../node_modules/react-native-multithreading',
                    packageImportPath: 'import com.reactnativemultithreading.MultithreadingPackage;',
                    packageInstance: 'new MultithreadingPackage()',
                },
            },
        },
    }
}

@nguyenvanquan7826 react-native.config.js use for auto link ->  you need remove other manual link (ex : 'implementation : reactnativemultithreading ....) or use manual link an without react-native.config.js

I wonder what that 'react-native-camera' attribute is doing there. Also, can you run the example?


FART

; -- DROP TABLE issues;

Sorry thats feature, its hooks a bug in emacs to runs my CPU 100% to keep me warm.


Jasper doesn't install on Yosemite

Hey all. Following the thread here and everyones suggestions: https://github.com/jasperproject/jasper-client/issues/35
This project seems awesome and I would really love to help out...but..I've been trying for hours to get this to work -- no dice. Could we maybe get an updated tutorial? It's really difficult to attempt to follow a ton of various advice on a comment thread, and i'm worried that it isn't working because the steps are outdated for Yosemite.
Right now i'm stuck on trying to compile the phonetisarus libary, and it's a complete nightmare.:
g++ -O2 -I3rdparty/sparsehash -I3rdparty/utfcpp    -c Phonetisaurus.cpp -o Phonetisaurus.o
In file included from Phonetisaurus.cpp:39:
In file included from ./Phonetisaurus.hpp:36:
In file included from ./MBRDecoder.hpp:33:
3rdparty/sparsehash/google/dense_hash_map:106:10: fatal error: 'tr1/functional' file not found
#include HASH_FUN_H                  // defined in config.h
^
3rdparty/sparsehash/google/sparsehash/sparseconfig.h:10:20: note: expanded from macro 'HASH_FUN_H'
#define HASH_FUN_H <tr1/functional>
^
1 error generated.
make: *** [Phonetisaurus.o] Error 1

Please report the issue over here and have a look at the support forum.


[HACKTOBERFEST 2024] - Security and Privacy Quiz #56236

Checklist:

[x] I have read and followed the contribution guidelines.
[x] I have read and followed the how to open a pull request guide.
[x] My pull request targets the main branch of freeCodeCamp.
[x] I have tested these changes either locally on my machine, or GitPod.

Closes #XXXXX

I have completed the Security and Privacy Quiz. How can i send the pr ?


Allow regular expressions in text_contains / any_text_contains

I would like to find all script tags within a page with contain certain keywords.
Here's how it's done via bs4:
script_tags = soup.find_all("script", string=re.compile(r"window\.(keyword|another_keyword)"))

I'm not too sure how to go about it with selectolax.
I could use the matches provided by .select(css).text_contains(pattern) like so:
first_keyword_scripts = tree.select("script").text_contains("window.keyword").matches

However, if I were to do the same for the second keyword, it is difficult to create the original context created with bs4 in which:

script_tags preserved the order of the script tags. In this case, I have to figure out which nodes come first, and need to merge them somehow
some scripts might contain both keywords and as such can result in duplication

Another use case for regular expressions is the ability to ignore keywords.
It would be nice to have a text_does_not_contain function too.
I think these issues can be solved with regular expressions but I'm probably wrong.
I'd appreciate any feedback on how these examples can be run via selectolax.
For now I have a workaround that uses the .text() operation, which isn't really great for large script texts,
Thanks for your time and for maintaining this project!

I think I can implement a callback interface so that you can provide a custom function that executes the regex internally.

@rushter , thank you so much for considering this issue!


Chain Reductions pattern matching table

I wrote a table to help you know how to perform reductions on the chains. In a monadic or dyadic chain, reductions always result in an monad or dyad respectively. (Nilads can just be promoted to constant n-ads in an n-adic chain???)

chain
dyadic

2 2 …
2 2 2 …Φ₁ …2 2 …ε{",'} …

2 1 …
B₁ …

1 2 …
1 2 2 …Φ.₂ …1 2 …Δ …

1 1 …
B.₃ …

chain
monadic

1 2 …
1 2 1 …Φ …1 2 …Σ …

1 1 …
B …

2 …
2 1 …S …2 …W …

ε" f g = x y -> f(x, g(x,y))
ε' f g = x y -> g(f(x,y), y)
B.₃ f g = x y -> g(f(x))

I found some issues with Jello's combinator trees:

[ ] Same issue https://github.com/codereport/jello/issues/14 for B₁ as we had for Φ₁. Maybe B.₃ and B.₆?
[ ] ε' isn't the same combinator depending on the whether its an initial combinator or not, but Jello doesn't distinguish.
[ ] no combinator explanation for ε'

ε' issue
The pairs here associate to the right  [1, [1, 10]]
> 1 10 :: pair pair
           ,    ,
   ,  1 10 ➡  [1, 10]
   ,, 1 10 ➡  [1, [1, 10]]
    This is a 2-2 dyadic chain (ε')
              └┬┘
              ε'

but the pairs here associate to the left [[2, 11], 10]
> 1 10 :: pair add1 pair
           ,    ‘    ,
   ,   1 10 ➡  [1, 10]
   ,‘  1 10 ➡  [2, 11]
   ,‘, 1 10 ➡  [[2, 11], 10]
    This is a 2-1-2 dyadic chain (B₁ε')
              └┬┘ ⋮
              B₁  ⋮
               └┬─┘
               ε'

although 2-1-2 reduces to 2-2 so the only difference is whether the first dyad is a reduction or not.

chain
dyadic

2 2 2 …
Φ₁ …

2 2 …
ε{",'} …*

2 1 …
B₁ …

1 2 2 …
Φ.₂ …

1 2 …
Δ …

1 1 …
B.₅ …

chain
monadic

1 2 1 …
Φ …

1 2 …
Σ …

1 1 …
B …

2 1 …
S …

2 …
W …

Perhaps the table could be written this way, provided it is clear that we are matching the most specialized match first.

Thanks for this! Will address all in the future.
I also want to change Jellyfish to work such that the first chain in a multichain has a fixed arity (I think that is what I want at least). Added these spellings recently ⬇ but when I modified Jellyfish do have better support for D2 I couldn't figure out how to register the first chain of a multichain as monadic when it is called in a dyadic context. If you have thoughts let me know : )

@codereport why can you just start with a monadic separator to specify a subchain
> 1 10 :: . pair : pair
          µ  ,   ð  ,
   µ,   1 10 ➡  [1, 1]
   µ,ð, 1 10 ➡  [[1, 1], 10]
    This is a s-2-s-2 dyadic chain (WB)
                │   │
                W   d
                └─┬─┘
                  B

I kind of think your mental model is a bit confused on how the reductions happen. The arity of the whole chain is based on the arguments/context. What you are actually thinking of as the stuff after "1st chain" is actually a subchain that reduces down into a monad.
A less confusing way to denote this is to explicitly use parentheses ( for a monadic sub chain, curly parentheses {` for a dyadic subchain.
< . sort rev sums . sum . sum add1
{{< (sort rev sums) sum} (sum add1)}

sum add1 is not the rest of the chain, it is a subchain that reduces down to a monad. That monadic chain can refer back to the original argument, which is helpful, but you can add a separator to end the subchain and continue the chain.
Breaking change
Also are you aware that your better support for D2 is a breaking change
2 1 1 use to be B₁B₁ now it is D₂.

https://github.com/DennisMitchell/jellylanguage/wiki/Tutorial

Multi-chain links let us do just that! To construct them, we use the chain separators øµðɓ

Surely one of these will let you end a subchain without starting another

My mental model is/was definitely wrong. I didn't realize the chain is formed and then the context still matters. I also feel like i don't totally understand explicitly what the delineation of link/subchain/chain is.
How did you learn all of this? Just by trial and error? or looking at the source code?

@codereport I'm trained as a mathematician, it's just the logical way to do these operations. Denis implemented it quite different, more like a stack based language. So my mental model is a bit different to his implementation

Interesting. And yea, I realized that I could have just used a monadic separator to get what I wanted with the d2 combinator. But I like the expressions having a fixed meaning i.e. it can only be called monadically or dyadically. Maybe after I play around with it maybe I wouldn't like it as much though. You might be right that the subchain ender is what I want.

@codereport I agree that you should have to specify the arity at function definition, and throw an error if the wrong number of arguments are passed. However, your d2 change is orthogonal.
I think you got tripped up by the fact that Denis' implementation is using the fact the the following are equivalent:
f₁ g₂ h₁ k₂ : ...
. f₁ : g₂ . h₁ : k₂ : ...

IMO, It's better to think of jelly working as the second, each atom is implicitly separated until the first separator.
Also my mental model: jelly is a language for writing extremely nested compositions of functions. At the end of the day, it should be implemented by construct a single function then calling it, rather than Denis' the stack based implementation.


chore: add scripts to update discovery artifacts

These PR add the scripts from #1187 that are needed to update discovery artifacts using a Github action. The scripts will be removed from #1187 once all of the review comments from #1187 have been resolved.
This PR adds the following files under the scripts/ folder

README.md to provide instructions on manually updating discovery artifacts and API reference documentation.
buildprbody.py creates a summary of the changes detected in discovery artifacts and writes them to allapis.summary.
changesummary.py creates verbose change information for each API with discovery artifact changes.
createcommits.sh creates git commits for each API with discovery artifact changes or reference document changes.
updatediscoveryartifacts.py is the python file that can be used to update discovery artifacts.

I also moved describe.py under the scripts folder and modified it to save the discovery artifacts that are fetched.
TODO:

[x] Add tests for scripts
[x] Address review comments in #1187

@busunkim96 @tswast This is ready for review. PTAL.

@tswast All comments have been addressed. PTAL


ValueError: GraphDef cannot be larger than 2GB.

Hi Shangyu
When I tried your code to calculate Hessian matrix I got the error ValueError: GraphDef cannot be larger than 2GB.
How can I solve the problem?
thanks
Setya W.P

Hi Setya @setyawp25
You can set the batch size of input data as a small number, such as 2.
Best regards,
Shangyu

Dear Shangyu @csyhhu
I have tried it but it is failed. I run it on CPU mode tensorflow with Server having 112 GB RAM. Does it cause this problem?
Best
Setya W.P

Hi Setya,
When I did the experiment, the memory cost is about 100GB, so 112 GB may not be very enough.
You may also set a bigger convolution strides in models/resnet.py for every layer, so that it will reduce the number of patches extracted and reduce the memory cost.
Best,
Shangyu

Hi Shangyu @csyhhu
thanks a lot for your kind response.
Best,
Setya WP

@setyawp25
You're welcome, pls let me know if it doesn't work.


Portability suite failing - filename too long on Windows

https://grpc-testing.appspot.com/job/gRPC_portability_pull_requests/332/language=c,scenario=windows_x86_vs2015_bo/console
objectivec/ProtocolBuffers_iOS.xcodeproj/xcshareddata/xcbaselines/8BBEA4A5147C727100C4ADB7.xcbaseline/FFE465CA-0E74-40E8-9F09-500B66B7DCB2.plist (Filename too long)

(windows has a 256 character limit on path length).
jcanizales - any chance we could not check in that file or make its name shorter?

That file's inside the Protobuf submodule. We don't use it for anything, but I don't know if there are git commands to restrict what comes in when cloning a submodule. Maybe a .gitignore under third_party/?
What if we use FAT32 so it becomes less than 256 chars? :P
third_~1/protobuf/object~1/Protoc~2.xcodeproj/xcshar~1/xcbase~1/8BBEA4~1.xcbaseline/FFE465~1.plist
@TeBoring @thomasvl can this file be removed from the Protobuf repo?

Yea, we can trim it.

Are we still seeing this?

The file's still there in the Protobuf repo, but I haven't seen the test fail for that reason in a while.


cmd/go: unable to run go commands from within another GOROOT

First off: This is a weird issue. I acknowledge that. But it messes with my personal workflow, so I'm curious whether it can be satisfactorily resolved.
Reproduce:

Clone toolchain to .../a; run .../a/make.bash
Ditto in .../b
cd .../b/src
Execute .../a/bin/go test bytes

Result:
go: reserved module path std not allow outside of GOROOT/src
There are three things going on here:

Grammatical error in error message.
Mysterious error message; it took me a while to realize the problem.
I don't see why this no longer works. Why must this be an error?

For context: When working on the compiler, I generally hang out in GOROOT/src. I use compilecmp to generate built toolchains with different versions. Then I refer to those versions by absolute path to run benchmarks.
This also sometimes happens under the hood. I already had to fix compilecmp because it could not be run from GOROOT/src because of the same issue.
It's an annoyance to have to cd around to run benchmarks, take care with what dir you're piping results to, etc.
cc @bcmills @jayconrod

Execute .../a/bin/go test bytes

Do you mean .../a/bin/go test ./bytes, with a ./? Because otherwise I'm not sure how that would work in the first place: the name bytes would be ambiguous.

Because otherwise I'm not sure how that would work in the first place: the name bytes would be ambiguous.

Without a ./. Why is it ambiguous? It always used to work; it refers to the bytes in the GOROOT of the go command being executed.

Ah, I see what you mean.
If we have the invariant that the module std always refers to the standard library (and that its packages therefore always lack a prefix), then it's ambiguous because we don't know whether bytes refers to the std module in GOROOT, or to the std library in the current working directory.
On the other hand, if we treat a std module outside of GOROOT as just another module, perhaps that isn't a problem. (But it does mean that your workflow will break again if there is ever a standard subtree beginning with std/.)


evaluation does not work in SF 12

Hi Ilya,
I checked this several times with different settings, "stockfish.get_evaluation()" does not work in stockfish 12.
It works well in stockfish 8. Maybe this is a Stockfish 12 specific problem ??
here is a debug:
{'Write Debug Log': 'false', 'Contempt': 0, 'Min Split Depth': 0, 'Threads': 2, 'Ponder': 'false', 'Hash': 16, 'MultiPV': 1, 'Skill Level': 20, 'Move Overhead': 30, 'Minimum Thinking Time': 20, 'Slow Mover': 80, 'UCI_Chess960': 'false'}
^CTraceback (most recent call last):
File "/home/nnn/Desktop/pypro/stockfishTest.py", line 35, in
print(stockfish.get_evaluation()['type'])
File "/home/nnn/tf2/lib/python3.6/site-packages/stockfish/models.py", line 246, in get_evaluation
text = self._read_line()
File "/home/nnn/tf2/lib/python3.6/site-packages/stockfish/models.py", line 73, in _read_line
return self.stockfish.stdout.readline().strip()
Kind regards
Marius

duplicate #44

duplicate #44

Hello
Thank you for your bug report!
I fixed it in version 3.12.1
Please, check it now :)

Hello
Thank you for your bug report!
I fixed it in version 3.12.1
Please, check it now :)


Update link colors in normal state and hover state

I have added the hover color as the secondary color and the normal state color as the primary state color. These updates will be available in the npm package upon installation once you publish a new version.

good job, keep up the good work!


Make e-mail sanity check more precise

For security reasons, all e-mail addresses starting with non-alphanumeric characters were rejected. This is too broad and rejects perfectly valid e-mail addresses. Only leading hyphens should be rejected -- in all other cases e-mail address specification should follow RFC 5322.

... overlooked the unit tests, sorry.
Having a look at the tests: If e-mail address strings are indeed forwarded to the shell literally, then the dollar ($) and backtick (`) characters should be rejected everywhere in the address, as they could be interpreted in the shell (cf<EMAIL_ADDRESS> `pwd`@example.com).
It would be far better to escape user provided strings properly, before handing them to the shell, though. I don't know enough about the exact problem here, though. Please advise.

I think we really just should just use mail.ParseAddress. See https://github.com/go-gitea/gitea/pull/17688 for more context.

I think we really just should just use mail.ParseAddress, everything else is madness. See #17688 for more context.

We can't do that. That's what we changed from and we change it because of some security report about that method.

We can't do that. That's what we changed from and we change it because of some security report about that method.

Have these "security issues" been raised to golang? Can you provide examples of what was the problem?

We can't do that. That's what we changed from and we change it because of some security report about that method.

Have these "security issues" been raised to golang? Can you provide examples of what was the problem?

See #19102 and #17688

We ought to just correctly pass -- to sendmail in all cases to fix that. We should not have to restrict emails just because we are using tools like sendmail incorrectly.

From an external viewpoint, I totally agree with @silverwind. Meanwhile, as a stopgap solution, the sendmail problem can be remediated by checking only for a leading hyphen character.
Regarding my previous remarks about the $ and ` characters: I realized this would only pose a problem if the command was called in a shell context (which is a bad idea, due to exactly this problem). Without knowing details about your code, sendmail likely is not called this way: I tried a basic smoketest on the live test instance and, e.g. a$HOME@domain was processed correctly.
Unfortunately, with the restrictions in place, Gitea is totally broken for me as a user. I cannot access my account and I have no sane way to remediate the problem. I probably could scan the database and replace the offending e-mail address manually... but I would rather not go down that road, fearing what else might break along the way.

Indeed, we should never run commands in shell context as it's pretty much impossible to escape arguments safely for a shell. We should always use exec.Command(cmd, ...args) format which is safe as long as cmd is not a shell.

So regarding these changes, they seem okay. But there are various characters that have special meanings in shell context. Can it be demonstrated that this is safe to use with sendmail? Long them, we should of course move away from sendmail to SMTP only.

Codecov Report

Merging #20991 (8aab3a3) into main (bb0ff77) will increase coverage by 0.48%.
The diff coverage is 62.98%.

@@            Coverage Diff             @@
##             main   #20991      +/-   ##
==========================================
+ Coverage   46.65%   47.14%   +0.48%
==========================================
  Files        1003     1007       +4
  Lines      136654   137308     +654
==========================================
+ Hits        63761    64728     +967
+ Misses      65066    64661     -405
- Partials     7827     7919      +92

Impacted Files
Coverage Δ

cmd/migrate_storage.go
5.88% <0.00%> (+0.11%)
:arrow_up:

models/repo.go
50.22% <0.00%> (+0.11%)
:arrow_up:

modules/avatar/identicon/block.go
42.11% <ø> (ø)

modules/git/foreachref/parser.go
94.36% <ø> (ø)

modules/git/signature_nogogit.go
61.53% <ø> (ø)

modules/log/file.go
68.78% <ø> (ø)

modules/log/smtp.go
82.35% <ø> (ø)

modules/markup/mdstripper/mdstripper.go
61.15% <ø> (ø)

modules/pprof/pprof.go
0.00% <ø> (ø)

modules/structs/issue.go
0.00% <0.00%> (ø)

... and 70 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

So regarding these changes, they seem okay. But there are various characters that have special meanings in shell context. Can it be demonstrated that this is safe to use with sendmail? [...]

In principle, that leads to the same problem @zeripath already outlined in #19102: To be sure, one would have to test every sendmail implementation out there. However, I would consider it unlikely that any sendmail argparser processes special characters other than hyphens.
Regarding characters interpreted by the shell (not sendmail), at first a similar problem would arise: There are different shells out there, which likely have different special characters. The obviously "dangerous" characters would be:

The dollar sign $ (as $HOME@domain might reveal the home dir of the current process);
The exclamation mark ! (as !13@domain might reveal command nr. 13 in the history list);
And the back tick ` (as `pwd`@domain might execute any command).

Note, that these characters do not necessarily depend on being in the first position: $HOME@domain would be as vulnerable as a$HOME@domain. All of these, however, are easily circumvented by not executing external commands in a shell context, in the first place (and I don't think you're doing that, anyway). The second best option would be to escape the characters properly.
In summary, my argument for the PR is: From a security perspective, it does not decrease the status quo, but it fixes the erroneous rejection of valid e-mail addresses for a large number of potential addresses.

I see we call sendmail with
exec.CommandContext(ctx, setting.MailService.SendmailPath, args...)

Which is save from shell interpretion at least. And anything beyond that are sendmail bugs, so LGTM.

ping LG-TM

This is currently a blocking issue in accessing my gitea install due to the incorrect email validation on my admin account. Can this be brought into a release version soon please?

If it's urgent, to bypass, you can update the admin's e-mail in the database.


2*c connection/goroutine leak fix (std client/ui)

[x] Phase 1 : repro with tests failing
[x] Phase 2: code fix

Also verified manually with lsof
Fixes #166

@kyessenov

Have you tried running it at 30-60kqps and observing system socket states?

my mac can only do 44k qps with 48 connections
$ lsof -p 98312|grep TCP|wc -l
     101

stable
All done 2658061 calls 1.098 ms avg, 43709.8 qps


Blueprints as yml

In the future the php extension might be used differently by Kirby. Blueprint as yml gives better colors in most code editors.

Why do you think .php files could be used differently in the future by Kirby? I think I just copied and modifies blueprints from the starterkit and I think the were .php files. I just checked, currently the starterkit uses .yml files. Without explaining the change, the commit fba71d12ab9d30347b9ca12704edeb4b3af970bc of the starterkit renamed and modifies .php to .yml files.
I will change to .yml, just as the starterkit did, but I am still unsure about why.
(Maybe it’s just because of the better syntax highlighting.)

It's not set in stone. It's just a "maybe" but I think that yml is more correct for the yaml language anyway.
https://github.com/getkirby/panel/issues/979
@lukasbestle

Maybe we should just kill backwards-compatibility here.
I agree that it's better to remove that old behavior to keep this clean.

The reason for this change would be to use php for blueprints with logic.
Until we know about the change you can see it like that you save two rows in the file and get better syntax highlighting.

This is interesting! Logic in blueprints is something I always wanted for Kirby Comments. As long as getkirby/panel/issues/979 is not resolved I’ll be switching to YAML.


review "muscle tissue disorder" branch

following https://github.com/monarch-initiative/mondo/issues/5658 and meeting with experts on March 2nd, 2023.
A lot of changes have been done directly on the existing PR during the call.
Please refer to the https://github.com/monarch-initiative/mondo/pull/5984 to see the changes made in the classification
In addition, some terms were put on the obsoletion list, and additional classification needs to happen. Decisions and discussions are here

In addition to these changes, (and maybe in the long term) we can review the genetic skeletal disorders following this paper: https://onlinelibrary.wiley.com/doi/10.1002/ajmg.a.63132

@nicolevasilevsky
MONDO:0019201 should be obsoleted, but it has a lot of x-ref to omimps, gard,...

Obsoleted this term was discussed on a call with Chris and Peter.


bpf: Get rid of CGO in bpf_linux.go

This PR replaces the usage ofC.CString() with &[]byte()[0] which is safe and allows us to get rid of the CGO dependency in bpf_linux.go.

This change is

test-me-please

Coverage decreased (-0.0003%) to 42.102% when pulling 9a6cfd07445a5ed34a9a5ebd4eb18bc6e4fbae56 on pr/brb/rm-cgo-pkg-bpf-maps into 79b4eba1b2cf0b471699c1c53712a8e7a982db4c on master.

Checkout from version control messed up: https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated/12950/execution/node/7/log/

test-me-please


.etl log file can not be deleted.

I executed the commmand .\Install.ps1 -OnboardingScript .\WindowsDefenderATPOnboardingScript.cmd successfully.  But the generated .etl log file is very large (over 66GB) and can not be deleted. The error message as below:
`del : Access is denied
At line:1 char:1

del install-HOSTNAME-10.0.14393.5501.etl

CategoryInfo : PermissionDenied: (C:\win2016_onbo…14393.5501.etl:String) [Remove-Item], Unauthorized

AccessException

FullyQualifiedErrorId : ItemExistsUnauthorizedAccessError,Microsoft.PowerShell.Commands.RemoveItemCommand`

How can I delete it ?

I executed the commmand .\Install.ps1 -OnboardingScript .\WindowsDefenderATPOnboardingScript.cmd successfully. But the generated .etl log file is very large (over 66GB) and can not be deleted. The error message as below: `del : Access is denied
At line:1 char:1

del install-HOSTNAME-10.0.14393.5501.etl

CategoryInfo : PermissionDenied: (C:\win2016_onbo…14393.5501.etl:String) [Remove-Item], Unauthorized

AccessException

FullyQualifiedErrorId : ItemExistsUnauthorizedAccessError,Microsoft.PowerShell.Commands.RemoveItemCommand`

How can I delete it ?

The .etl log file disappeared after I restart the system, and the disk space is freed. So this issue was resolved.

As administrator, run "logman -ets" to figure out the session name (it should start with "install-") that is still active (and logging inside the problematic etl) and after that "logman stop -n  -ets" to stop the ETW session. After that you should be able to delete the .etl file.