jf scan ./image.tar reporting no vulnerabilities in GitHub action

Describe the bug

If I scan image.tar directly after creation, it's recognized as Generic and doesn't show any vulnerability.
If I load it into docker and save it again, it'll work

To Reproduce
name: Scan

on:
  pull_request:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: setup buildx
        uses: docker/setup-buildx-action@v2

      - name: build image
        uses: docker/build-push-action@v3
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64
          build-args: |
            BUILDCACHE_BASEURL_ARG=${{ secrets.BUILDCACHE_BASEURL }}
            BUILDCACHE_PUSH_ENABLED_ARG=false
            BUILDCACHE_USER_ARG=${{ secrets.BUILDCACHE_USER }}
            BUILDCACHE_PASSWORD_ARG=${{ secrets.BUILDCACHE_PASSWORD }}
          pull: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          outputs: type=docker,dest=image.tar

      # works
      - name: Run Trivy vulnerability scanner
        uses<EMAIL_ADDRESS>        with:
          input: image.tar
          trivy-config: trivy.yaml

      - uses: jfrog/setup-jfrog-cli@v2
        with:
          version: 2.24.1
        env:
          JF_ENV_1: ${{ secrets.JF_ENV_1 }}

      # doesn't work
      - run: |
          jf scan ./image.tar
        env:
          JFROG_CLI_LOG_LEVEL: DEBUG

      - uses: docker/login-action@v2
        with:
          registry: ${{ secrets.DOCKER_REGISTRY }}
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: load and push image
        run: |
          docker image load --input image.tar
          docker push ${{ secrets.DOCKER_REGISTRY }}/private/image:test

      # works
      - run: |
          jf docker scan ${{ secrets.DOCKER_REGISTRY }}/private/image:test
        env:
          JFROG_CLI_LOG_LEVEL: DEBUG

      - run: |
          container-diff diff daemon://${{ secrets.DOCKER_REGISTRY }}/private/image:test ./image.tar --type=history --type=file --type=size --type=apt

      - name: save
        run: |
          docker save ${{ secrets.DOCKER_REGISTRY }/private/image:test -o ./image2.tar

      # works
      - run: |
          jf scan ./image2.tar
        env:
          JFROG_CLI_LOG_LEVEL: DEBUG

results in
Run jf scan ./image.tar
  jf scan ./image.tar
  shell: /usr/bin/bash -e {0}
  env:
    JFROG_CLI_ENV_EXCLUDE: *password*;*secret*;*key*;*token*;*auth*;JF_ARTIFACTORY_*;JF_ENV_*
    JFROG_CLI_OFFER_CONFIG: false
    JFROG_CLI_BUILD_NAME: Vulnerability Scan
    JFROG_CLI_BUILD_NUMBER: 595
    JFROG_CLI_BUILD_URL: https://github.com/company/image/actions/runs/2840639249
    JFROG_CLI_USER_AGENT: setup-jfrog-cli-github-action/2.3.0
    JFROG_CLI_LOG_LEVEL: DEBUG
14:47:58 [Debug] Sending HTTP GET request to: https://company.jfrog.io/xray/api/v1/system/version
14:47:58 [Debug] Usage Report: Sending info...
14:47:58 [Debug] Sending HTTP GET request to: https://company.jfrog.io/artifactory/api/system/version
14:47:58 [Debug] Artifactory response: 200 OK
14:47:58 [Debug] The Artifactory version is: 7.41.7
14:47:58 [Debug] Sending HTTP POST request to: https://company.jfrog.io/artifactory/api/system/usage
14:47:58 [Info] JFrog Xray version is: 3.52.4
14:47:58 [Debug] Creating lock in:  /home/runner/.jfrog/locks/xray-indexer
14:47:58 [Info] JFrog Xray Indexer 3.52.4 is not cached locally. Downloading it now...
14:47:58 [Debug] Sending HTTP GET request to: https://company.jfrog.io/xray/api/v1/indexer-resources/download/linux/amd64
14:47:58 [Debug] Usage Report: Artifactory response: 200 OK
14:47:58 [Debug] Usage Report: Usage info sent successfully.
14:48:03 [Info] The downloaded Xray Indexer version is 3.52.4
14:48:03 [Debug] Releasing lock:  /home/runner/.jfrog/locks/xray-indexer/jfrog-cli.conf.lck.3298.1660229278636011461
14:48:03 [Info] [Thread 2] Indexing file: ./image.tar
14:48:07 [Info] 2022-08-11T14:48:04.17901381Z [jfxia] [DEBUG] [] [wire_gen:45                   ] [main                ] Initializing filtering service
2022-08-11T14:48:05.001930349Z [jfxia] [DEBUG] [] [indexer-app:43                ] [main                ] Indexing standalone file ./image.tar using artifactory folder /tmp/jfrog.cli.temp.-1660229283-497546448
2022-08-11T14:48:05.002024149Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Local path: /tmp/jfrog.cli.temp.-1660229283-497546448/6b42c647-e44f-456e-5704-1e4bf803459f/166022928500202044/image.tar
2022-08-11T14:48:05.002052049Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Scanning file from Artifactory with mimetype 'application/x-gzip'
2022-08-11T14:48:07.603531673Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Found archive file. Performing deep scan for file /tmp/jfrog.cli.temp.-1660229283-497546448/6b42c647-e44f-456e-5704-1e4bf803459f/166022928500202044/image.tar
2022-08-11T14:48:07.693722778Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/1152061f1151c742af79b176c806bf8e72bfbfd110835efd6317fb8bb4d254e9
2022-08-11T14:48:07.697623578Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/16f15939bf55211a98fafa76f9e247e9d319e76c6e6d81c7a91b82becb0c00ba
2022-08-11T14:48:07.697675978Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/1fe172e4850f03bb45d41a20174112bc119fbfec42a650edbbd8491aee32e3c3
2022-08-11T14:48:07.706308678Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/44d3aa8d076675d49d85180b0ced9daef210fe4fdff4bdbb422b9cf384e591d0
2022-08-11T14:48:07.706817778Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/53d8f3c0b37abd925ba94b581a31d167ddaa2b3c5687aa8a7ceeca150e15496a
2022-08-11T14:48:07.706857378Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/6ce99fdf16e86bd02f6ad66a0e1334878528b5a4b5487850a76e0c08a7a27d56
2022-08-11T14:48:07.758768281Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/9a27270b63ac3a43a90311711576840fe278679291b26993abbe581e8c466f93
2022-08-11T14:48:07.758833181Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/c0ab546c23d0497649e47056be2521d9211721303e9487e8aacbe7aec6d7a747
2022-08-11T14:48:07.800972883Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable blobs/sha256/cf0532f0204bdb5f0d5a35e14592233e9db15d5f1ca9fb001a44095ba8c98b31
2022-08-11T14:48:07.801297183Z [jfxia] [DEBUG] [] [archive_mgr:1264              ] [main                ] Found archive file. Performing deep scan for file /tmp/jfrog.cli.temp.-1660229283-497546448/6b42c647-e44f-456e-5704-1e4bf803459f/166022928780111878/manifest.json
2022-08-11T14:48:07.801337683Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] No classification found for manifest.json, classified as generic
2022-08-11T14:48:07.801358383Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] manifest.json was classified as Generic
2022-08-11T14:48:07.801383483Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] total running time for indexing tree construction of  manifest.json: 4.61e-05 seconds
2022-08-11T14:48:07.801440383Z [jfxia] [DEBUG] [] [archive_mgr:1245              ] [main                ] checking if the file is supported executable oci-layout
2022-08-11T14:48:07.801489283Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] No classification found for image.tar, classified as generic
2022-08-11T14:48:07.801508783Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] image.tar was classified as Generic
2022-08-11T14:48:07.801526683Z [jfxia] [DEBUG] [] [archive_mgr:232               ] [main                ] total running time for indexing tree construction of  image.tar: 5.29e-05 seconds
2022-08-11T14:48:07.801545683Z [jfxia] [DEBUG] [] [archive_mgr:195               ] [main                ] total running time for indexing image.tar: 0.19794911 seconds

14:48:07 [Debug] Sending HTTP POST request to: https://company.jfrog.io/xray/api/v1/scan/graph?scan_type=binary
14:48:08 [Info] Waiting for scan to complete...
14:48:08 [Debug] Sending HTTP GET request to: https://company.jfrog.io/xray/api/v1/scan/graph/84e868a0-f872-4530-76fb-b84f7e0bcb79?include_vulnerabilities=true
The full scan results are available here: /tmp/jfrog.cli.temp.-1660229288-2224644868
Note: no context was provided, so no policy could be determined to scan against.
14:48:08 [Info] Scan completed successfully.
You can get a list of custom violations by providing one of the command options: --watches, --repo-path or --project.
Read more about configuring Xray policies here: https://www.jfrog.com/confluence/display/JFROG/Creating+Xray+Policies+and+Rules
Below are all vulnerabilities detected.
+-------------------------------------+
| ✨ No vulnerabilities were found ✨ |
+-------------------------------------+

Expected behavior
Show all vulnerabilities, as on workstation
Versions

JFrog CLI version:  2.24.1
JFrog CLI operating system: ubuntu-latest
Artifactory Version: jfrog.io

Additional context

aquasecurity/trivy-action works as expected on the same image.tar

@timdittler,
Thanks for reaching out.
Few questions:

Do you run on the same OS and with the same Jfrog Platform server?

Can you please provide the JFrog CLI version?  The version: 3.52.4 is the JFrog Xray version. CLI version can be found using: jf --version.

Can you run the git action with the latest CLI version? Can be achieved by:

- uses: jfrog/setup-jfrog-cli@v2
  with:
    version: latest

Can you add the env - JFROG_CLI_LOG_LEVEL: DEBUG before running the scan command to show more log information?

   - uses: jfrog/setup-jfrog-cli@v2
        env:
          JF_ENV_1: ${{ secrets.JF_ENV_1 }}
         JFROG_CLI_LOG_LEVEL: DEBUG

Also just to let you know, we recently introduced the jf docker scan, which creates a tar file and scans it with one command: jf docker scan centos:latest
And another nice and easy way to scan docker images is using our new JFrog Docker Desktop Extension, available on your local docker desktop app.

Thanks for your comment @sverdlov93 . I tried many different things. Right now, I believe something is off with my image creation process. I'll investigate and re-open this ticket if necessary.

I dug a bit deep and come up with the example above. It's actually not about GH Actions vs. Workstation. I really don't know what's the problem. But jf scan won't detect anything in the first try, but on all the ones after importing it to docker. So I guess it's a problem of jf. Sadly, I can't share my image.tar.

This is beginning of the log of the second run with jf scan:
2022-08-11T14:50:30.0059600Z ##[group]Run jf scan ./image2.tar
2022-08-11T14:50:30.0059915Z jf scan ./image2.tar
2022-08-11T14:50:30.0113850Z shell: /usr/bin/bash -e {0}
2022-08-11T14:50:30.0114118Z env:
2022-08-11T14:50:30.0114471Z   JFROG_CLI_ENV_EXCLUDE: *password*;*secret*;*key*;*token*;*auth*;JF_ARTIFACTORY_*;JF_ENV_*
2022-08-11T14:50:30.0114850Z   JFROG_CLI_OFFER_CONFIG: false
2022-08-11T14:50:30.0115159Z   JFROG_CLI_BUILD_NAME: Vulnerability Scan
2022-08-11T14:50:30.0115478Z   JFROG_CLI_BUILD_NUMBER: 595
2022-08-11T14:50:30.0115859Z   JFROG_CLI_BUILD_URL: https://github.com/company/image/actions/runs/2840639249
2022-08-11T14:50:30.0116298Z   JFROG_CLI_USER_AGENT: setup-jfrog-cli-github-action/2.3.0
2022-08-11T14:50:30.0116633Z   JFROG_CLI_LOG_LEVEL: DEBUG
2022-08-11T14:50:30.0116896Z ##[endgroup]
2022-08-11T14:50:30.0269934Z 14:50:30 [Debug] Sending HTTP GET request to: https://company.jfrog.io/xray/api/v1/system/version
2022-08-11T14:50:30.0283730Z 14:50:30 [Debug] Usage Report: Sending info...
2022-08-11T14:50:30.0429139Z 14:50:30 [Debug] Sending HTTP GET request to: https://company.jfrog.io/artifactory/api/system/version
2022-08-11T14:50:30.4992161Z 14:50:30 [Info] JFrog Xray version is: 3.52.4
2022-08-11T14:50:30.4999911Z 14:50:30 [Debug] Creating lock in:  /home/runner/.jfrog/locks/xray-indexer
2022-08-11T14:50:30.5000571Z 14:50:30 [Debug] Releasing lock:  /home/runner/.jfrog/locks/xray-indexer/jfrog-cli.conf.lck.3542.1660229430499120246
2022-08-11T14:50:30.5001021Z 14:50:30 [Info] [Thread 2] Indexing file: ./image2.tar
2022-08-11T14:50:30.5114204Z 14:50:30 [Debug] Artifactory response: 200 OK
2022-08-11T14:50:30.5114887Z 14:50:30 [Debug] The Artifactory version is: 7.41.7
2022-08-11T14:50:30.5116634Z 14:50:30 [Debug] Sending HTTP POST request to: https://company.jfrog.io/artifactory/api/system/usage
2022-08-11T14:50:30.8857105Z 14:50:30 [Debug] Usage Report: Artifactory response: 200 OK
2022-08-11T14:50:30.8857952Z 14:50:30 [Debug] Usage Report: Usage info sent successfully.
2022-08-11T14:50:56.8747634Z 14:50:56 [Info] 2022-08-11T14:50:30.697780082Z [jfxia] [DEBUG] [] [wire_gen:45                   ] [main                ] Initializing filtering service
2022-08-11T14:50:56.8748627Z 2022-08-11T14:50:31.525523231Z [jfxia] [DEBUG] [] [indexer-app:43                ] [main                ] Indexing standalone file ./image2.tar using artifactory folder /tmp/jfrog.cli.temp.-1660229430-862749059
2022-08-11T14:50:56.8749640Z 2022-08-11T14:50:31.525635931Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Local path: /tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943152563193/image2.tar
2022-08-11T14:50:56.8750534Z 2022-08-11T14:50:31.525666231Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Scanning file from Artifactory with mimetype 'application/x-gzip'
2022-08-11T14:50:56.8751504Z 2022-08-11T14:50:35.314667199Z [jfxia] [DEBUG] [] [indexer_app:109               ] [main                ] Found archive file. Performing deep scan for file /tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943152563193/image2.tar
2022-08-11T14:50:56.8753260Z 2022-08-11T14:50:38.824982961Z [jfxia] [DEBUG] [] [tar:82                        ] [main                ] Docker image manifest scanning File: [Id=7246466352339916359, name=/***/private/image/timdittlertest/manifest.json, path=/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943882472426/, mime=application/x-docker, sha256=9dbd75b60387daa107389f7d67ce1e491b876e1ae2c150ff6da946ae9f57df54, parent=9dbd75b60387daa107389f7d67ce1e491b876e1ae2c150ff6da946ae9f57df54, childrens=0]
2022-08-11T14:50:56.8763332Z 2022-08-11T14:50:38.825214061Z [jfxia] [DEBUG] [] [tar:82                        ] [main                ] docker layers on message {"messageId":"bcf7deea-4476-4386-6d2a-9472bae16341","eventType":"","downloadUrl":"onDemand","artifactoryId":"","repoKey":"","repoPkgType":"","path":"/***/private/image/timdittlertest/manifest.json","checksums":{"md5":"1fb52bb5821f4174cf9a2ce14488467c","sha1":"50ac4d340ee9071331ec56ce207503434bfb3fa8","sha256":"9dbd75b60387daa107389f7d67ce1e491b876e1ae2c150ff6da946ae9f57df54"},"archivePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943882472426/manifest.json","downloadedDockerArchive":{"onDemand":{"DockerArchivesArray":[{"ArchiveName":"sha256__82b15c83930edda93acf7653ecdbad3507061f40d704bb1e5ada03a2c7a29e80.tar","MediaType":"","Sha":"82b15c83930edda93acf7653ecdbad3507061f40d704bb1e5ada03a2c7a29e80","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943531482009/45cb19a6236cea8ff70ec30070835a33a6946fd3962706f792455aa35cae1b6e/sha256__82b15c83930edda93acf7653ecdbad3507061f40d704bb1e5ada03a2c7a29e80.tar"},{"ArchiveName":"sha256__e8df80afc57d1d3442ee9c057b6c90d7750d1cee47c9eb3e1c8b5c462fd7de35.tar","MediaType":"","Sha":"e8df80afc57d1d3442ee9c057b6c90d7750d1cee47c9eb3e1c8b5c462fd7de35","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943628902755/612270a8219d297da7713f799e39f8c7c12a3499893a3bc20c686cb72fc7652c/sha256__e8df80afc57d1d3442ee9c057b6c90d7750d1cee47c9eb3e1c8b5c462fd7de35.tar"},{"ArchiveName":"sha256__9c1b6dd6c1e6be9fdd2b1987783824670d3b0dd7ae8ad6f57dc3cea5739ac71e.tar","MediaType":"","Sha":"9c1b6dd6c1e6be9fdd2b1987783824670d3b0dd7ae8ad6f57dc3cea5739ac71e","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943639807257/6e3692f03eefee9a819cd5ef747d7331ec6a1e20b65d5eb648b923469e74377e/sha256__9c1b6dd6c1e6be9fdd2b1987783824670d3b0dd7ae8ad6f57dc3cea5739ac71e.tar"},{"ArchiveName":"sha256__1390fc6b2a9fa968c8d267d5cb10dcba10c44616f2d221fcefcf495499dfa570.tar","MediaType":"","Sha":"1390fc6b2a9fa968c8d267d5cb10dcba10c44616f2d221fcefcf495499dfa570","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943689760635/77a833cd5f5c2b28b07eb05fc731f6afdf19996ebba3cd2a12ea719adf4c6115/sha256__1390fc6b2a9fa968c8d267d5cb10dcba10c44616f2d221fcefcf495499dfa570.tar"},{"ArchiveName":"sha256__13a34b6fff7804cf7f6e8f52a4cf25ceb2e32fc35a6f39e8158074c64831ebf0.tar","MediaType":"","Sha":"13a34b6fff7804cf7f6e8f52a4cf25ceb2e32fc35a6f39e8158074c64831ebf0","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943689811575/884857678af1292ad90fdf1a9c84a468d220fb6b5d5133692d3810da5274dd56/sha256__13a34b6fff7804cf7f6e8f52a4cf25ceb2e32fc35a6f39e8158074c64831ebf0.tar"},{"ArchiveName":"sha256__6be690267e47ddcfd965449d2af70a9eca9879f9436948ee83d7f4ad473b8e64.tar","MediaType":"","Sha":"6be690267e47ddcfd965449d2af70a9eca9879f9436948ee83d7f4ad473b8e64","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-9472bae16341/166022943692851615/db9e8c3bbea76eab9355035aa6bbe9453249192305f7e97756bbd67e943cd698/sha256__6be690267e47ddcfd965449d2af70a9eca9879f9436948ee83d7f4ad473b8e64.tar"},{"ArchiveName":"sha256__fba45aa3e67564317c0e0d31e5c7cad5a2e2b01a672251cb73b1ea0bbeb62423.tar","MediaType":"","Sha":"fba45aa3e67564317c0e0d31e5c7cad5a2e2b01a672251cb73b1ea0bbeb62423","DownloadLink":"","SavedFilePath":"/tmp/jfrog.cli.temp.-1660229430-862749059/bcf7deea-4476-4386-6d2a-

Differences begin after it detects a different mime type. Could this be the cause?!

@timdittler
Are you sure that the output from your build command is image.tar tar file and not a directory with the name image.tar?

Yes, I’m sure. We also store them as artifacts.

Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.

Really is an archive. Output from runner:
ls -l ./image.tar
file ./image.tar
  shell: /usr/bin/bash -e {0}
-rw-r--r-- 1 runner docker 384894976 Aug 15 07:23 ./image.tar
./image.tar: POSIX tar archive

Hi @timdittler ,
Can you reproduce that also using docker build on docker cli on your local machine?
I am trying to understand the difference between the tar and the manifest.json created by 'docker save' that we support correctly,
and the tar created by the docker build with output flag.

Hi @timdittler,
You can always create a free tier account https://jfrog.com/start-free/#saas without any time limits.
You can also create one from your CLI using the following command:
curl -fL "https://getcli.jfrog.io?setup" | sh

@timdittler
Could you please try to add the --bypass-archive-limits flag?

jf scan --watches service service.tar --bypass-archive-limits

Sadly, no change in behavior

What is your JFrog CLI version, @timdittler?
We added the support for this flag in 2.28.2.

I'm now loading the image to work around the problem
      - name: Build image
        uses: docker/build-push-action@v4
        with:
          context: .
          file: ./Dockerfile
          platforms: linux/amd64
          pull: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          outputs: type=docker,dest=service.tar
          tags: service

      - name: Load image
        run: docker load -i service.tar

      - name: Setup JFrog CLI
        uses: jfrog/setup-jfrog-cli@v3
        env:
          JF_ENV_1: ${{ secrets.JF_ENV_1 }}

      - name: Run vulnerability scanner
        run: jf docker scan --watches service --format json service

Related: PR to support docker scan from tar directly:
https://github.com/jfrog/jfrog-cli-security/pull/30


Last build logs button doesn't show up anything

Description:
In the UI, when you select an app, there is a button ‘Last Build logs’ but when I click on it. A new tab pops up but there is no log inside. I was expecting to see same logs I got when the app was building.
Version used:
epinio: 0.8.1
ui-dashboard-index: epinio-v0.6.2

Screenshot:

With latest epinio/epinio and using /wapi/v1/namespaces/from-ui/staging/<app deployment.stage_id>/logs?follow=true&authtoken=... I still see no staging logs. The socket is open and 'connected' but no messages are sent.

This has something to do with how follow: true works: https://github.com/rancher/dashboard/blob/fdd90feecbd6cbe0876b1debba6d54aa73a73663/pkg/epinio/windowComponents/ApplicationLogs.vue#L138
When I set it to false I can see the logs of the last staging process. I'll investigate more.

@richard-cox PR is merged, this is now fixed. Please re-open if you find otherwise.


Add nick_conf to weechat setup, fix conf parsing

Concatenation of configuration strings was resulting in empty conf variables being set, on my system at least.
$nick_conf was just missing from setup :)
Made servers autoconnect, so people can get to chadding immediately.

ouch sorry I went fixing it tonight before looking at PR, but am very grateful you also looked into it. my first prototype was in zsh which I'm more used to that's why the errors...

ook rebased to use your PR, nice to have someone else onboard :D

Thanks! I figured it was working fine in a shell, wasn't sure which one 😅
Happy to help where I can :)


convert temp coeff to %/C

[x] Closes #350 .
[x] I am familiar with the contributing guidelines.
[ ] Tests added.
[ ] Updates entries to docs/source/api.rst for API changes.
[x] Adds descriptions to appropriate "what's new" file in docs/source/whatsnew for all changes. Includes link to the GitHub Issue with :issue:`num` or this Pull Request with :pull:`num`. Includes contributor name and/or GitHub username (link with :ghuser:`user`).
[x] New code is fully documented. Includes numpydoc compliant docstrings, examples, and comments where necessary.
[x] Maintainer: Appropriate GitHub Labels and Milestone are assigned to the Pull Request and linked Issue.

API tests will probably fail until SolarArbiter/solarforecastarbiter-api#238

oh, is there something that needs to be done about the reference data set?

Yeah, adjusted here and the API migration (https://github.com/SolarArbiter/solarforecastarbiter-api/pull/238) will adjust those in the database already. Once this is in dev, I tag to prod relatively quickly so that any new sites a user makes are properly specified as %/C

Once this is in dev, I want to tag to prod relatively quickly so that any new sites a user makes are properly specified as %/C

Does that mean a beta 6 as soon as possible?


Feature Request: Servercow.de

What provider would you like to see added to NPM?

Have you checked if a certbot plugin exists?
would be cool if this could be integrated:
https://github.com/jhartlep/servercow-dns-api

Unfortunately the DNS challenge within nginx proxy manager is only available for certbot dns plugins. Servercow does not have a plugin for this. Using the acme.sh script is a very significant deviation from certbot and would require a just as significant amount of work. It also would require support for two separate DNS challenge mechanisms instead of just one. So I don't think it is very likely to be implemented for now, unless a certbot plugin is published.
The v3 rewrite of NPM will use acme.sh instead of certbot, but it is still fairly far away.

Hey @chaptergy,
could you confirm to integrate the acme.sh support for the V3.0 ?
i have also the same issue with servercow. Addtional benefit of acmee.sh is, that by using other targets like synolgy the acme deploy-hook can be parallel used on side to the proxy-manager.

I had opened this feature request some time ago.
It's a pity that so far no possibility has been added to use servercow with the npm.
Maybe you can try again to include servercow.
There is an official acme.sh script for the provider Servercow:
https://github.com/acmesh-official/acme.sh/blob/master/dnsapi/dns_servercow.sh


The scraper does not show that on validator page for Arcadis it does not appear the daily graph for number of jobs

ENVIRONMENT:
https://admin.peviitor.ro/
Browser: Chrome
Device: Laptop
OS: Windows 10
STEPS TO REPRODUCE:
Enter url in browser
In go to "https://admin.peviitor.ro/" and click to jobs for "Arcadis"
EXPECTED RESULTS:
The page shows the daily graph with the evolution of the number of the jobs
ACTUAL RESULTS:
The page shows the phrase "Nu exista date pentru aceasta companie in baza de date"
See below the screenshot
Attachment:
Page from Validator for Arcadis

Fixed...


Icon signifier

Hello.
How can I switch icon signifier to be inside the text field area like the guidelines on material.io website show?
https://material.io/design/components/text-fields.html#anatomy

Currently you can't.
The guideline has been updated several times and is now way different then when I started this project. There's even no such thing called Textfieldboxes in it ;)


Why I can't get remote video, always "waiting for others to join".

Can't get remote video
After "flutter run", I can't get remote video, always "waiting for others to join", but can get remote stream-ID.

I have started every service, like sfu, biz and so on.
Peer also can't get my video, his camera is open.
My camera is open too, and I can see my own video.

What's wrong? I can't firgure it out. Please help!

Because the back-end (ion) configuration is Incorrect.


gnome-shell doesn't see new applications without restarting

Describe the bug
When installing e.g. Slack in my user account, it creates the slack.desktop file in .nix-profile/share/applications and updates mimeinfo.cache as well.
However, Gnome-shell doesn't know about these applications until you restart it (for normal users this means actually rebooting), a bit drastic.
To Reproduce
Steps to reproduce the behavior:

Run gnome3 as your desktop
get Slack in your profile (nix-env or home-manager)
notice gnome-shell doesn't know about Slack

Expected behavior
NixOS would kick gnome-shell into rescanning
Additional context
It could be that gnome-shell has listeners on the mimeinfo.cache files, but they don't work on NixOS since all store files are immutable and have the same mtime. Furthermore, the parent directories are symlinks.
It seems to me that we need to patch gnome-shell to e.g. poll the inode of the cache files or fix the listener to understand trees of symlinks.
Notify maintainers
@hedning
@jtojnar
@worldofpeace

Duplicate of #12757, #86647.

(I agree that patching gnome-shell seems like probably the best fix there, though I wonder if we couldn't hack something up by making our own shadow symlink tree and ensuring stuff gets touched?)


misspelled word

[Just a misspelled word in the page "Sutdio"]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: f5c733e4-5a8c-56a3-ff18-b5e589ac6d91
Version Independent ID: aba35600-3d59-5751-2072-33cc24ec5601
Content: Power Apps portals Studio - Power Apps
Content Source: powerapps-docs/maker/portals/portal-designer-anatomy.md
Service: powerapps
Sub-service: portals
GitHub Login: @neerajnandwana-msft
Microsoft Alias: nenandw

Thanks for your comment, @IonutGhetau. This issue has been assigned to the doc author for review.

Thanks for bringing this to our attention.  The typo is fixed. Closing the issue.


Locale is ignored

No matter the value set to the locale, it seems to always be ignored
I have tried importing moment with locales and event setting the locale through moment, but even then, still ignored.
Unsure of what else to try or how to fix it.

@a-aznar
try to import this to your index.html file  :
<script src='https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.8.1/moment-with-locales.min.js'></script>


ux-select: collapse throwing an error on some instances

I'm working on an app for fun and decided to use aurelia-ux to try it out.
Doing so, I stumbled on the following bug:
TypeError: _this.optionWrapperEl is null vendor.aurelia-ux~3acd81b0.6e7494126752752e4a04.chunk.js line 23 > eval:265:13
    collapse ux-select.js:255

After looking it down, this is about : https://github.com/aurelia/ux/blob/master/packages/select/src/ux-select.ts#L339 where this.optionWrapperEl is null in some instances.
I suspect it is after the element has been unattached or unbound and that for some reason it tries to collapse.
Just adding an if to check if it's null before trying to remove the class seems to be fixing the issue, but I'm not sure this is the right way of doing it. Maybe the better one would be to store the timeout id somewhere and cancel it when needed?
I'm willing to make a pull request on this if you tell me what approach would be the right one for this component.

@qraynaud nice, thanks for the report. Could you please help with some pseudo code of how you'd get this error?

Even with my codebase, it's happening one time out of hundreds. I cannot really reproduce it reliably. I noticed it only because I show every error in a frame on the UI. It's not creating any issue I can see…
If I find some time, I'll try to do something. But the basic idea is just to have a component that contains a select and can be removed / re-added easily. Something along:
let(opened.bind="false")
ux-button(click.delegate="opened = !opened")
ux-select(if.bind="opened")
  //- options

In the end, it was not more complicated than that.
Hope it helps.

@qraynaud thanks. That should be enough to get started fixing this.

Here is the patch I'm currently using to bypass the issue. I'm pretty sure this is not the optimal fix but it helps to understand the issue:
diff --git<EMAIL_ADDRESS><EMAIL_ADDRESS>index 4f448d4..22ec131 100644
---<EMAIL_ADDRESS>+++<EMAIL_ADDRESS>@@ -252,7 +252,8 @@ var UxSelect = /** @class */ (function () {
         this.isCollapsing = true;
         this.optionCtEl.classList.remove('ux-select__list-container--open');
         setTimeout(function () {
-            _this.optionWrapperEl.classList.remove('ux-select__list-wrapper--open');
+            if (_this.optionWrapperEl)
+                _this.optionWrapperEl.classList.remove('ux-select__list-wrapper--open');
             _this.isCollapsing = false;
             _this.expanded = false;
             _this.setFocusedOption(null);


voteCount

GET /v1/contributors/9611450a-0663-45a5-8a08-f1c71320475e yields, as expected, the same model for information about contributors as does GETs against the active session, but with one difference: we don't get voteCount.
Expecting these two to endpoints present consistent models for contributors, and thus the browser doesn't have two different ways of conceiving them and fails to display the vote count in the contributor dialog.

Looking at the ETL in progress, I used
{...
"downvotingContributorIds": [ "12lasdfjka34", "2394239897sdlkfdfjd23" ],
"upvotingContributorIds": [ "12lasdfjka34", "2394239897sdlkfdfjd23" ]
...}
with plural keys, parallel to comments and answers.  Any strong opinion of plural vs singluar here (age old question I know) -- wer're using plurals elsewhere though - so leave like this @laurelnaiad ?
seed-data 1.7 is preceeding well.

I like plural. Not sure why I wrote it singularly before. Perhaps a typo.

Made great progress on this today.  Got all of the reimplementation in Java done.
I should even be able to complete the search transform changes today too.

Charles esta en fuego!

Great....on the issue of voting, I could not find separate issues for other aspects of voting, wondering if this single issue also addresses the fact that if I vote and then revisit that page, the visual record of my vote is lost (icon is blue indicating no vote). It allows me to vote again but throws an error. Let me know if I should file separate issue or wait for this vote fix.

@popzip  I noticed this too.  That behavior worked for a while, but there's no expectation that front-end can fix until the middle tier has been updated for the new logic for hasVoted.

@popzip In other words, it's a separate issue, but there's probably a failing or ignored test in browserland waiting for this middle tier fix (which is almost ready).

Yes, two failing tests in the browser marked as @broken so that they're skipped in automated e2e tests. I will reenable them once we have the fix in place.

Some adjustment of browser to accomodate voteCount in current PR.

Bumping to major.

No indication seen if the vote is already made  or not . When you do a vote up /vote down the arrow turned grey , with no indication what vote i made .
Also on the hover over the already voted answer I  see "You must be logged in to vote" tooltip .

There is already an existing issue for the feedback. This issue is about voteCount.
Please confirm voteCount here and use #270 for issues related to "Vote buttons should show up/down info after vote is cast", which is not in test, yet so no testing on #270 needed at this time.

Vote Up or Vote down count works fine inside the QnA details page .
One related behavior i am seeing is on doing (vote up or vote down)  ;the post tally on the QnA list is not getting updated correctly . i.e +1 for Vote up and -1 for vote down  .  Let me know if you need  a separate bug for this one .

the post tally on the QnA list is not getting updated correctly . i.e +1 for Vote up and -1 for vote down

I don't understand. Do you mean the vote count in the search results or something else?

On QnA list page.

Does it update after a page refresh?  It could be a bug in Java if it's not updating at all.

Its not updating i tried refresh .

Should the values be coming from a single place? Are we duplicating the vote counts in the db?

@laurelnaiad yes, now im confused.  Let me look more closely at this issue and see what im missing.

@gghai I think we need better reproduction -- this worked for me multiple times:

search for something
click through to QnA
vote.  voting is rather slow... we know how we could improve though.
click back button
the voteCount for that doc is updated.

of course if you vote one thing up and another down, then the voteCount would be unchanged.

@grechaw  you are correct . Could be possible i might have ran into a situation where i vote up and vote down and that caused the number to be unchanged .
I did verify it again seems to be working. sorry for the churn .


Flickering lines when multiple leds are lit

Hey,
I'm back with another issue (that's probably my fault):
When I light up multiple leds lines start to randomly flicker (as seen in this video). I tried multiple things which had little to no effect so far:

changing the main loop delay()

  void loop() {
  button.check();
  lmd.display(); //flush display
  delay(10);
}

removing lmd.display(); from the main loop
only using 4 8x8 modules

Have you experienced such issues? Could I try to change the SPI speed or do you think it is a power issue?
Thanks a lot!

I'd test it like this:

slow down the SPI,
clear display and display it when it's empty,
turn on LEDs one by one and check how far you can get.

could you hint me in the right direction: I can not find anything on how to change the SPI speed on an ESP32...
Thanks!

https://github.com/bartoszbielawski/LEDMatrixDriver/blob/cd14cdd4f3a2cc7d7789a798f7b97ee8b202f2b9/src/LEDMatrixDriver.cpp#L16

Let me know if the issue is solved.
I was actually considering making an API that does not force SPI speed and just leaves it alone, maybe that could be a reason to do it.

Thanks for the tip, SPI speed has no influence as far as I can tell (although I only tested it shortly). I will try to find out how many leds I can drive without issue and report back!
Thanks for your help!

It's okay, I can wait, take your time.
Some people after solving their problems never come back to even leave a comment...
Cheers!

Alright so I tried it out with this library and interestingly the issue shows as well. I guess this is good news for you and your library but bad news for me :) However I found out that the flickering only occurs at low intensity levels (less than half of the max brightness) which leads me to believe it is a hardware issue of the MAX7219 chip (maybe because supposedly the cheap ones are fake). I will try to dim the panels using the r_set resistor value.
Thanks for your support!

Update: Dimming using Rset works without flickering! On my boards a 20k resistor is used and I am now running 200k atm without issues. I'd say brighness is comparable to software intensity = 0

Thanks for letting me know. Did you try to get a better power supply? Maybe 5V was dipping...

I have tried different power supplies without success. But since I found out that flickering is not occuring on full brightness I ruled out power delivery problems. Also I think you can see in the video, that the flickering lines are not dimming but rather lighting up (which is in line with the vanishing of the flickering on full brightness).
Seeing as this is a hardware problem (possibly MAX2719 clone problem) I am fine with closing the issue.
Thank you for your patience and support!

Thanks for letting me know, cheers!


feat: leverage multiprocessing for map ops

Description
Small patch to leverage multi processing for data map operations. PR subject to change with https://github.com/foundation-model-stack/fms-hf-tuning/pull/230. Existing test cases should suffice this patch.
caveat - if your node has many cpus it might spawn too many processes.

@dushyantbehl @ashokponkumar you can close this once superseded, thanks.

This should be closed @kmehant


Switch Dart grammars (Sublime → Atom).

This PR switches Dart grammars from guillermooo/dart-sublime-bundle to dart-atom/dartlang.
The Sublime plugin is no longer maintained and its highlighting leaves a lot to be desired.
(A particularly egregious example: it highlights keywords that occur inside strings.)
The Atom plugin, on the other hand, works well and continues to be maintained.
(The VSCode / IntelliJ plugins would also work, but their grammars are not supported by Linguist.)

@rkirsling Thanks for the pull request!
@lildude I get an issue when I try to run the Atom grammar on Lightshow. Will that be an issue once on github.com?

Oooops. Looks like the CSON parsing on Lightshow has gone 💥  on the update.  Looks like I missed updating a library when updating it this morning.  I'll fix that in a mo.
@pchaigno Nope, we convert all the grammars to JSON within Linguist when building the gem and it's these that we use.
I converted the CSON in this PR to JSON and bunged it into a gist and it works

@pchaigno Lightshow is fixed now and your test no longer reports a problem.

@lildude Thanks!
The new grammar does look better (than the Sublime text one), although it still highlights a few keywords that occur inside variable names (or class fieds, I'm not sure what those are), such as TokenType.$return.


MM-36740 - fix close button styling

Summary
fixes styles (color) for modal-header close button by increasing class specificity
Ticket Link
Fixes MM-36740
Screenshots

before
after

Release Note
NONE

@calebroseland Please help prioritize review for tomorrow's 5.37 code freeze.

Cherry pick is scheduled.

Error trying doing the automated Cherry picking. Please do this manually
+++ Updating remotes...
Fetching upstream
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
From github.com:mattermost/mattermost-webapp
   5a7907867..35804dc74  MM-36731-ux-changes-cloud-starter-flat-fee -> upstream/MM-36731-ux-changes-cloud-starter-flat-fee
 * [new branch]          MM-36735     -> upstream/MM-36735
   156a14dee..50df9c67b  cloud        -> upstream/cloud
   ef5c885f2..bc47db6e2  master       -> upstream/master
   b0b95f310..7f0200b27  release-5.37 -> upstream/release-5.37
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Fetching origin
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
+++ Updating remotes done...
+++ Creating local branch automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625751697
Switched to a new branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625751697'
Branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625751697' set up to track remote branch 'release-5.37' from 'upstream'.

+++ About to attempt cherry pick of PR #8366 with merge commit bc47db6e2e5ec950ef0558f0f86e4ed31398a229.

error: could not apply bc47db6e2... MM-36740 - fix close button styling (#8366)
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

+++ Conflicts detected:

UU sass/components/_channel-switcher.scss
Aborting.

+++ Aborting in-progress git cherry-pick.

+++ Returning you to the master branch and cleaning up.

@amyblais cherry pick did fail ... i am attempting to do a manual pick now

/cherry-pick cloud

Cherry pick is scheduled.

Error trying doing the automated Cherry picking. Please do this manually
+++ Updating remotes...
Fetching upstream
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
From github.com:mattermost/mattermost-webapp
   35804dc74..e2ba23ef2  MM-36731-ux-changes-cloud-starter-flat-fee -> upstream/MM-36731-ux-changes-cloud-starter-flat-fee
   50df9c67b..d178d3146  cloud      -> upstream/cloud
Fetching origin
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
+++ Updating remotes done...
+++ Creating local branch automated-cherry-pick-of-mattermost-webapp-#8366-upstream-cloud-1625754022
Switched to a new branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-cloud-1625754022'
Branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-cloud-1625754022' set up to track remote branch 'cloud' from 'upstream'.

+++ About to attempt cherry pick of PR #8366 with merge commit bc47db6e2e5ec950ef0558f0f86e4ed31398a229.

The previous cherry-pick is now empty, possibly due to conflict resolution.

You are currently cherry-picking commit bc47db6e2.

nothing to commit, working tree clean
!!! git cherry-pick failed

+++ Aborting in-progress git cherry-pick.

+++ Returning you to the master branch and cleaning up.

/cherry-pick release-5.37

Cherry pick is scheduled.

Error trying doing the automated Cherry picking. Please do this manually
+++ Updating remotes...
Fetching upstream
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
Fetching origin
Failed to add the RSA host key for IP address '<IP_ADDRESS>' to the list of known hosts (/app/.ssh/known_hosts).
+++ Updating remotes done...
+++ Creating local branch automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625754046
Switched to a new branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625754046'
Branch 'automated-cherry-pick-of-mattermost-webapp-#8366-upstream-release-5.37-1625754046' set up to track remote branch 'release-5.37' from 'upstream'.

+++ About to attempt cherry pick of PR #8366 with merge commit bc47db6e2e5ec950ef0558f0f86e4ed31398a229.

error: could not apply bc47db6e2... MM-36740 - fix close button styling (#8366)
hint: after resolving the conflicts, mark the corrected paths
hint: with 'git add <paths>' or 'git rm <paths>'
hint: and commit the result with 'git commit'

+++ Conflicts detected:

UU sass/components/_channel-switcher.scss
Aborting.

+++ Aborting in-progress git cherry-pick.

+++ Returning you to the master branch and cleaning up.

@amyblais I did already cherry pick to the cloud branch

@michelengelen It needs a cherry-pick to release-5.37 too.

@michelengelen ^

@amyblais done^^


Show account public keys in more places

Creates a new common component SmallAccountID that initially displays XXXXXXXX...XXXXXXXX and expands and becomes selectable on click.

This adds SmallAccountID to the wallet header and sender/recipient in transaction details. Also adds the accountID in wallet settings in a CopyText. r? @keybase/react-hackers

@keybase/react-hackers bump

i guess my reply didn't go through. i believe the UX is that its supposed to expand on hover, not click
cc: @cecileboucheron

This could work too I think. Especially because it'll be that exact behavior on mobile.
@malgorithms @adamjspooner Thoughts?

@malgorithms and I talked a bit about this last week - a pro of this approach is we don't have to handle the case where the user click-and-drags outside the hover area and causes the ID to collapse

yeah, I prefer click, rather than hover


Unit of measure 0 is redundant

When there is for example margin: 0 it's redundant to add unit, example margin: 0em

The source in consistent with zero values, and I think that it’s going to be a waste of time to change it just for the sake of changing it, considering that units are getting removed when minified to semantic.min.css—it just doesn’t matter.

considering that units are getting removed when minified to semantic.min.css—it just doesn’t matter.

Right, I agree.


VS Code starting very slow

Extension Name: python
Extension Version: 2018.9.0
OS Version: Windows_NT x64 10.0.17763
VSCode version: 1.28.0

We have written the needed data into your clipboard. Please paste:
{
	"activationTimes": {
		"startup": false,
		"codeLoadingTime": 11679,
		"activateCallTime": 73,
		"activateResolvedTime": 145,
		"activationEvent": "onLanguage:python"
	}
}

Duplicate of #2827


Staging is ignoring a good chunk of CSS

Suspect it's a CSS file not getting pulled in. I'm working in style.css -- maybe it's not getting compiled properly into common.css.

UGH. So after doing some digging we mention in our settings.py file this statement:
ASSETS_DEBUG = True  # Don't bundle/minify static assets

which sets it so that assets aren't bundled. Soooo we can mark this as closed.


Add storage_options keyword to read_csv

Previously we had no way to pass down keyword arguments from read_csv
to a backend store like s3 or hdfs.  This is because kwargs was used
to supply pd.read_csv with options.  This commit adds a new
storage_options keyword that gets sent to the call to read_bytes.
cc @martindurant

Are there other read_* functions that need this treatment?

read_text pipes kwargs directly to read_bytes.  Nothing else exists at the moment

Should we still have the line dask.bytes.s3::70, the call to get_delegated_token? I suppose yes...
I changed its functioning such that it only generates a temporary token if the key/secret were explicitly provided - but there should be a warning somewhere that we are implicitly assuming the eight environment variables/config files/IAM roles are set for all workers for accredited login without explicit key/secret.

Should we still have the line dask.bytes.s3::70, the call to get_delegated_token? I suppose yes...

I'll leave that to you.  Should I make that change in this PR?

I think leave the line there, and merge this.
I anticipate still seeing some configuration questions on s3-in-dask in the future, this PR should make them easier to cope with.


v1.16 Backports 2024-08-12

[ ] #34277 (@aanm)

Once this PR is merged, a GitHub action will update the labels of these PRs:
 34277

/test-backport-1.16


[Bug] atomic.ads fails compilation

Hi Atomic-Developers,
I tried to build septum.gpr with alire.
septum.gpr uses  atomic              0.4.1.
The build however fails with
...
Note: Building septum/septum.gpr...
Setup
   [mkdir]        object directory for project Atomic
   [mkdir]        library directory for project Atomic
   [mkdir]        object directory for project Dir_Iterators
   [mkdir]        library directory for project Dir_Iterators
   [mkdir]        object directory for project Progress_Indicators
   [mkdir]        library directory for project Progress_Indicators
   [mkdir]        object directory for project Trendy_Terminal
   [mkdir]        library directory for project Trendy_Terminal
   [mkdir]        object directory for project Septum
   [mkdir]        exec directory for project Septum
Compile
   [Ada]          septum.adb
   [Ada]          trendy_terminal-windows.ads
   [Ada]          trendy_terminal-platform.adb
   [Ada]          trendy_terminal.ads
   [Ada]          trendy_terminal-vt100.adb
   [Ada]          trendy_terminal-string_vectors.ads
   [Ada]          trendy_terminal-maps.adb
   [Ada]          trendy_terminal-lines.adb
   [Ada]          trendy_terminal-lines-line_vectors.ads
   [Ada]          trendy_terminal-io.adb
   [Ada]          trendy_terminal-io-line_editors.adb
   [Ada]          trendy_terminal-histories.adb
   [Ada]          trendy_terminal-environments.adb
   [Ada]          trendy_terminal-completions.adb
   [Ada]          progress_indicators.ads
   [Ada]          progress_indicators-work_trackers.adb
   [Ada]          progress_indicators-spinners.adb
   [Ada]          progress_indicators-bars.adb
   [Ada]          dir_iterators.ads
   [Ada]          dir_iterators-recursive.adb
   [Ada]          dir_iterators-ancestor.adb
   [Ada]          atomic.adb
atomic.ads:91:19: error: ghost entity cannot appear in this context

   compilation of atomic.adb failed

gprbuild: *** compilation phase failed
ERROR: Command ["gprbuild", "-s", "-j0", "-p", "-P", "C:\path\to\septum\septum.gpr"] exited with code 4
ERROR: Compilation failed.

Looking at
https://github.com/Fabien-Chouteau/atomic/blob/14bb7db33dff4b6f1219967a809d0f787cae33cb/src/atomic.ads#L91
I was unable to understand why.
Hope you can solve it soon!
Greetings,
Pierre

Hi @Fabien-Chouteau
I have only installed GNATPRO 23.0.
I am using (according to alr toolchain --select)
Note: gnat is currently not configured. (alr will use the version found in the environment.)
Note: Currently configured: gprbuild=22.0.1

For more info:
> alr toolchain
CRATE        VERSION STATUS    NOTES
gprbuild     22.0.1  Default
gnat_arm_elf 12.2.1  Available
gnat_native  12.1.2  Available

>  alr config
last_build_profile=DEVELOPMENT
toolchain.assistant=false
msys2.install_dir=C:\Users\laarpjljvd\.cache\alire\msys64
toolchain.use.gprbuild=gprbuild=22.0.1

Note with gnat_native=12.2.1 compilation succeeds (with some warnings):
Note: Selected tool version gprbuild=22.0.1
Note: Deploying gnat_native=12.2.1...
############################################################################################################################################################################################################ 100.0%
Note: gnat_native=12.2.1 installed successfully.

PS C:\path\to\septum> alr build
Note: Building ansiada/ansi.gpr...
Compile
   [Ada]          ansi-demo.adb
   [Ada]          ansi.ads
Bind
   [gprbind]      ansi-demo.bexch
   [Ada]          ansi-demo.ali
Link
   [link]         ansi-demo.adb
Note: Building septum/septum.gpr...
Compile
   [Ada]          septum.adb
   [Ada]          trendy_terminal-windows.ads
   [Ada]          trendy_terminal-platform.adb
   [Ada]          trendy_terminal.ads
   [Ada]          trendy_terminal-vt100.adb
   [Ada]          trendy_terminal-string_vectors.ads
   [Ada]          trendy_terminal-maps.adb
   [Ada]          trendy_terminal-lines.adb
   [Ada]          trendy_terminal-lines-line_vectors.ads
   [Ada]          trendy_terminal-io.adb
   [Ada]          trendy_terminal-io-line_editors.adb
   [Ada]          trendy_terminal-histories.adb
   [Ada]          trendy_terminal-environments.adb
   [Ada]          trendy_terminal-completions.adb
   [Ada]          progress_indicators.ads
   [Ada]          progress_indicators-work_trackers.adb
   [Ada]          progress_indicators-spinners.adb
   [Ada]          progress_indicators-bars.adb
   [Ada]          dir_iterators.ads
   [Ada]          dir_iterators-recursive.adb
   [Ada]          dir_iterators-ancestor.adb
   [Ada]          atomic.adb
   [Ada]          atomic-unsigned.adb
   [Ada]          atomic-signed.adb
   [Ada]          atomic-unsigned_8.ads
   [Ada]          atomic-unsigned_32.ads
   [Ada]          atomic-unsigned_16.ads
   [Ada]          atomic-signed_8.ads
   [Ada]          atomic-signed_32.ads
   [Ada]          atomic-signed_16.ads
   [Ada]          sp.ads
   [Ada]          sp-config.adb
   [Ada]          sp-interactive.adb
   [Ada]          ansi.ads
   [Ada]          sp-file_system.adb
   [Ada]          sp-platform.adb
   [Ada]          sp-strings.adb
sp-interactive.adb:17:09: warning: unit "Ada.IO_Exceptions" is not referenced [-gnatwu]
sp-interactive.adb:26:08: warning: unit "SP.Platform" is not referenced [-gnatwu]
   [Ada]          sp-commands.adb
   [Ada]          sp-filters.adb
   [Ada]          sp-searches.adb
   [Ada]          sp-terminal.adb
   [Ada]          sp-contexts.adb
sp-searches.ads:29:05: warning: use clause for type "Ada.Containers.Count_Type" has no effect [-gnatwu]
   [Ada]          sp-memory.adb
sp-commands.adb:18:20: warning: unit "Ada.Containers.Vectors" is not referenced [-gnatwu]
   [Ada]          sp-progress.adb
sp-terminal.ads:20:06: warning: unit "System" is not referenced [-gnatwu]
   [Ada]          sp-cache.adb
sp-cache.adb:20:09: warning: unit "Ada.Task_Identification" is not referenced [-gnatwu]
sp-cache.adb:34:14: warning: function "+" is not referenced [-gnatwu]
Build Libraries
   [gprlib]       Atomic.lexch
   [gprlib]       Dir_Iterators.lexch
   [gprlib]       Progress_Indicators.lexch
   [archive]      libAtomic.a
   [gprlib]       Trendy_Terminal.lexch
   [archive]      libDir_Iterators.a
   [archive]      libProgress_Indicators.a   [archive]      libTrendy_Terminal.a

   [index]        libProgress_Indicators.a   [index]        libTrendy_Terminal.a

   [index]        libDir_Iterators.a   [index]        libAtomic.a

Bind
   [gprbind]      septum.bexch
   [Ada]          septum.ali
Link
   [link]         septum.adb
Build finished successfully in 78.22 seconds.

So Atomic is not compatible with GNAT Pro 23 yet, you can use GNAT Pro 22 in the meantime.


Remove start and done logging

Start and done logging are valuable in a pinch when diagnosing timed out requests but practically they produce a ton of data that usually doesn't tell us any more than access logs. Something like 25% of our total log volume is these two categories.
Should we encounter new timeouts, re-introducing this logging may be required. At that time we should probably introduce a feature so we can control the output of these data.

Is there any logging happening after this change?

Yes -- the on-response audit logging (GET /url/etc/ 500) will continue. That's where this should be safe -- we log successful requests three times, the start/done signals only add value when a request starts but never completes (and thus would never log).


Adapt last templates to use the new required args during creation

This PR is meant to address 2 things:

new required parameters during creation (acqera/procstr/etc)
updating releases that have been recently deprecated, see:
https://hypernews.cern.ch/HyperNews/CMS/get/relAnnounce/1783.html

This description will get updated as I go through these templates and make new commits:
*) properly handle the required creation args
*) delete templates using very old releases (actually, pre releases)
*) creating and validating new templates (TaskChain_ProdMinBias)
*) RelVal multicore template updated to a valid release
*) All DMWM templates were now updated
*) first half of the Integration templates (DQMHarvest, MC, MCFromGEN and ReDigi)
*) second half of the integration templates done (ReReco and TaskChain)
If I'm not wrong, the only template missing is a TaskChain reading lhe files.

For the record, list of releases deprecated this week:
https://hypernews.cern.ch/HyperNews/CMS/get/relAnnounce/1783.html
I've just removed all templates using 700_pre releases. They will be replaced in the coming days.

That should be basically it (besides the taskchain lhe). Now I have to update a couple of unittests that are using these json templates (I guess I'll just decouple unittests from them) and that's it.

Cool! In the end I had to keep the evil Neutrino PU dataset (> 100k files), though I didn't import it to dbs testbed. I haven't seen any problem with the agents while parsing/querying for that PU dataset as well.
We likely have to update some minor settings/comments, but I think we should merge it and if needed I can create another PR addressing those few issues. @ticoann what do you think?


[WIP] Switch notebook container from nteract to jlab

With this PR, running yarn docker:run:devel:notebook (after rebuilding) opens up the Jupyter Desktop app with an ijskernel installed (instead of nteract):

Some issues:

currently must require with absolute path, have not figured out how to get node to search /opt/rapids/node/node_modules by default
When jlab starts, must select JS "console" — trying to start JS "notebook" opens an ipython notebook for me

rebased on latest main. Still not sure why explicitly setting NODE_PATH does not function, or for that matter why the exact same JS kernel package is behaving any differently given the otherwise minimal/unrelated changes in the PR

I had thought of that too, I've been looking but hadn't turned up anything yet. What's odd is the current reported paths:
require("module")._nodeModulePaths(process.cwd()).concat(require("module").globalPaths)

]
  '/opt/rapids/node_modules',
  '/opt/node_modules',
  '/node_modules',
  '/usr/local/lib/node'
]

That's almost right?
Not sure why just the important one /opt/rapids/node/node_modules is missing

/opt/rapids is the homedir of the container user, and node automatically adds $HOME/node_modules to the require paths. I wonder if docker isn't parsing after /usr/local/lib/node_modules because of the colon?

I wonder if docker isn't parsing after /usr/local/lib/node_modules because of the colon?

Well, thats the thing too, /usr/local/lib/node  is what is inthe global paths, not /usr/local/lib/node_modules  (i.e. the NODE_PATH value)

Well, if I just run bash instead of jlab and then
rapids@bvhp:~/node$ NODE_PATH=/opt/rapids/node/node_modules/ /opt/JupyterLab/jupyterlab-desktop

it still does not work, so it definitely seems jlab is hiding env vars or something.

OK setting the working dir in the kernel.json does work :tada:

OK @trxcllnt @exactlyallan  This is ready, including updated notebooks:


Rescale only, length discrepancy

The program should warn the user about not using the  --seq-length option, when
using the --rescale-only flag.
Thanks to Morten to pointing this out

closing for lack of activity


Explain magic

There is a lot of unexplained magic, which makes it unclear what is happening, e.g., when is an account update created?
When is it added as child? There is a lot documentation that could be written.

Was this completed?


Update analytics.py

self.app should be available in init_app as well, otherwise it will fail when calling source function.

Coverage increased (+0.006%) to 99.254% when pulling 2be6ae06399dbcc1a00af35b4e6a4f5185300e48 on Xumeiquer:master into 48bea12574a038d6af0424c83470ebe6d70527fa on citruspi:master.


Hello, can i get iframe crunchyroll

https://static.crunchyroll.com/vilos-v2/web/vilos/player.html
I use your package, can i get iframe player ?

I believe it's not possible 😫
But, getting the episode object, you will get links/streams with a "temporary" link with the m3u8 addresses of streams.
Something like:
#EXT-X-STREAM-INF:BANDWIDTH=8126877,RESOLUTION=1920x1080,FRAME-RATE=23.974,CODECS="avc1.640028,mp4a.40.2"
https://pl.crunchyroll.com/evs3/b8f882125bc2e5f23701f654d4c03a39/assets/e655b0dabea7cdf181cfa709db5e3636_4887524.mp4/index-v1-a1.m3u8?res=1920x1080&Expires=1675649232&....

Then using a supported player (such as video.js with the HLS extension) you will be able to play them. However, it is something very complicated to do, and requires knowledge about refresh_tokens and the like, given that crunchyroll comes day after day creating barriers to make it difficult to access these links due to piracy.

I believe it's not possible 😫 But, getting the episode object, you will get links/streams with a "temporary" link with the m3u8 addresses of streams. Something like:
#EXT-X-STREAM-INF:BANDWIDTH=8126877,RESOLUTION=1920x1080,FRAME-RATE=23.974,CODECS="avc1.640028,mp4a.40.2"
https://pl.crunchyroll.com/evs3/b8f882125bc2e5f23701f654d4c03a39/assets/e655b0dabea7cdf181cfa709db5e3636_4887524.mp4/index-v1-a1.m3u8?res=1920x1080&Expires=1675649232&....

Then using a supported player (such as video.js with the HLS extension) you will be able to play them. However, it is something very complicated to do, and requires knowledge about refresh_tokens and the like, given that crunchyroll comes day after day creating barriers to make it difficult to access these links due to piracy.

Hi @Mssjim u still want to work on this? if yes i have a suggestion can we talk on discord? my user on discord is @codsec


Translate method never returns on Windows

Hi there! I am trying to compile the project on Windows. I have used the strategy on #3 and it compiled. The issue that I am facing is that the Translate method never returns.
I have added several logs inside the method and all of them are displayed in my terminal but the method itself never returns and the process keep running indefinitely.
The compiler I am using is MSVC 19.40.33811.0.
The compilation and execution work great on macOS and Linux.
I made some tests in Translate method and noticed that this issue just happens when translator.translate_batch method is called. But even when it is called all the other methods and logs called after it are executed without problem, except return and throw that are never executed.
Any ideas or tips on how to debug this would be very helpful.

this issue just happens when translator.translate_batch

I'm guessing this is a CTranslate2 issue

I'm guessing this is a CTranslate2 issue

That is true! Using -DOPENMP_RUNTIME=COMP as compilation option to CTranslate2 solved the issue


sendAnonymousUsage not working docker image: traefik:v2.9.8

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?
The latest docker image traefik:v2.9.8 when setting the sendAnonymousUsage in both
## STATIC CONFIGURATION in traefik.yaml
global:
  checkNewVersion: false
  sendAnonymousUsage: false

and in docker-compose.yaml file:
- "--global.sendAnonymousUsage false"
 - "--global.sendAnonymousUsage=false"

does not work!
What did you see instead?
time="2023" level=info msg="Traefik version 2.9.8 built on 2023-02-15T15:23:25Z"
time="2023" level=info msg="Stats collection is enabled."
time="2023" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2023" level=info msg="Help us improve Traefik by leaving this feature on :)"

What version of Traefik are you using?
traefik:v2.9.8
What is your environment & configuration?
## STATIC CONFIGURATION
global:
  checkNewVersion: false
  sendAnonymousUsage: false

log:
  level: INFO
  filePath: "/logs/traefik.log"

accessLog:
  filePath: "/logs/access.log"
  bufferingSize: 100
api:
  #insecure: true
  dashboard: true

pilot:
  dashboard: false

providers:
  docker:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: "web"
  file:
    directory: "/traefix"
    watch: true

entryPoints:
  console:
    address: ":8080"
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: ":443"
    http:
      middlewares:
        #- secureHeaders@file
    forwardedHeaders:
      trustedIPs:
        - "<IP_ADDRESS>/20" #https://www.cloudflare.com/ips/
        - "<IP_ADDRESS>/22"
        - "<IP_ADDRESS>/22"
        - "<IP_ADDRESS>/22"
        - "<IP_ADDRESS>/18"
        - "<IP_ADDRESS>/18"
        - "<IP_ADDRESS>/20"
        - "<IP_ADDRESS>/20"
        - "<IP_ADDRESS>/22"
        - "<IP_ADDRESS>/17"
        - "<IP_ADDRESS>/15"
        - "<IP_ADDRESS>/13"
        - "<IP_ADDRESS>/14"
        - "<IP_ADDRESS>/13"
        - "<IP_ADDRESS>/22"
        - "2400:cb00::/32"
        - "2606:4700::/32"
        - "2803:f800::/32"
        - "2405:b500::/32"
        - "2405:8100::/32"
        - "2a06:98c0::/29"
        - "2c0f:f248::/32"

Add more configuration information here.
If applicable, please paste the log output in DEBUG level
time="2023" level=info msg="Traefik version 2.9.8 built on 2023"
time="2023" level=info msg="Stats collection is enabled."
time="2023" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2023" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2023" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2023" level=warning msg="Traefik Pilot has been removed."
time="2023" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023" level=info msg="Starting provider *file.Provider"
time="2023" level=info msg="Starting provider *traefik.Provider"
time="2023" level=info msg="Starting provider *docker.Provider"
time="2023" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023" level=info msg="I have to go..."

time="2023" level=info msg="Many thanks for contributing to Traefik's improvement by allowing us to receive anonymous information from your configuration."
time="2023" level=info msg="Help us improve Traefik by leaving this feature on :)"
time="2023" level=info msg="More details on: https://doc.traefik.io/traefik/contributing/data-collection/"
time="2023" level=warning msg="Traefik Pilot has been removed."
time="2023" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
time="2023" level=info msg="Starting provider *file.Provider"
time="2023" level=info msg="Starting provider *traefik.Provider"
time="2023" level=info msg="Starting provider *docker.Provider"
time="2023" level=info msg="Starting provider *acme.ChallengeTLSALPN"
time="2023" level=info msg="Anonymous stats sent to https://collect.traefik.io/9vxmmkcdmalbdi635d4jgc5p5rx0h7h8: {\"global\":{\"sendAnonymousUsage\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"console\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{},\"http2\":{\"maxConcurrentStreams\":250}},\"web\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250}},\"websecure\":{\"address\":\"xxxx\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{\"trustedIPs\":[\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\",\"xxxx\"]},\"http\":{\"middlewares\":[\"RealIP@file\",\"ScanBlock@file\"]},\"http2\":{\"maxConcurrentStreams\":250}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"xxxx\",\"defaultRule\":\"xxxx\",\"network\":\"web\",\"swarmModeRefreshSeconds\":\"15s\"},\"file\":{\"directory\":\"/traefix\",\"watch\":true}},\"api\":{\"dashboard\":true},\"log\":{\"level\":\"INFO\",\"filePath\":\"xxxx\",\"format\":\"common\"},\"accessLog\":{\"filePath\":\"xxxx\",\"format\":\"common\"}}},\"bufferingSize\":100},\"pilot\":{},}}}}"

Hello,
the option works well but when you enable pilot this setting is forced to true.
Pilot has been stopped, so you have to remove the option from your configuration.


SQL Support

This is umbrella work item for all future work on SQL support
I envision following operations for alpha iteration

[ ] SELECT 5

[ ] Numbers support
[ ] Implicit data-type conversion

[ ] FROM Table

[ ] Table scan
[ ] Clustered Index seek
[ ] Non-clustered Index seek
[ ] Index scan

[ ] JOIN tables

[ ] INNER
[ ] LEFT/RIGHT/FULL

[ ] WHERE

[ ] basic comparison (<>, =)
[ ] IS NULL, IS NOT NULL
[ ] LIKE
[ ] BETWEEN

[ ] UPDATE support on single table

[ ] without WHERE
[ ] with WHERE

[ ] DELETE support on single table

[ ] without WHERE
[ ] with WHERE

Not part of alpha iteration

[ ] Multiple resultset
[ ] GROUP BY
[ ] Aggregated functions
[ ] Window functions
[ ] Tabular functions
[ ] Stored procedures

For SQL parser was selected FParsec. I personally consider resurrect https://github.com/fsprojects/Rezoom.SQL/ or start maintaining that library.
For query compilation was chosen Expression Trees to make life of general contributor slightly easier. Because of that decision, C# API cannot have SQL support for now because it has only TryGet(int, out TEntity)  method
https://github.com/StereoDB/StereoDB/blob/2318486ceb4de6944c7f1b3645884db0dd889f36/src/StereoDB/CSharp/Table.fs#L13
As part of implementation of SELECT was discovered that maybe existing ReadTransaction method should be expanded. Currently would be returned T voption, which is fine for single row resultsets, but if resultset has multiple records then list should be returned.
https://github.com/StereoDB/StereoDB/blob/2318486ceb4de6944c7f1b3645884db0dd889f36/src/StereoDB.FSharp/StereoDb.fs#L17C1-L18C1
@AntyaDev propose add separate method abstract ReadTransaction: transaction:(ReadOnlyTsContext<'TSchema> -> 'T array) -> 'T array. I personally think that returning array is bad for perf and would like to have somthing like S.C.G.List<T> or maybe T list to be idiomatic F#. But F# class library is not very friendly for codegen IMO so for now I would be returning List<T> voption until we make progress on discussion.

For implementing clustered index seek we probably somehow should detect what is primary key on the entity. This should happens either via metadata, or via some convention (for example look at Id or EntityId property)


Fix typo in is_instance_schema docstring

Change Summary
Just a quick typo fix in the docstring of the is_instance_schema function.

Related issue number
N/A

Checklist

[x] Unit tests for the changes exist
[x] Documentation reflects the changes where applicable
[x] Pydantic tests pass with this pydantic-core (except for expected changes)
[x] My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

please review


Fix TiRegion race when invalidating regions in RegionManager

What problem does this PR solve?
This is a fix for release-2.1 only.
In the previous implementation of the RegionManager, when the regions are invalidated, it is possible that the region in RegionManager is either not correctly invalidated due to race, or not correctly re-fetched.
RegionManager has race when encountering the following processes:

invalidateRegion() may be inconsistent when the handler tries to invalidate the region in RegionCache, but the region was previously updated by another process (e.g., updateLeader()).
the region was invalidated in keyToRegionIdCache, but the corresponding range is not correct.
if the invalidated range was smaller than the original, it creates a dangerous key range, where all keys in this range do not have a corresponding region id.
when fetching the region from the cache using getRegionByKey(), we look first into the key range cache, where it points to the non-existing region.

What is changed and how it works?
Use CAS operations when trying to invalidate regions. Forbid the usage of the region id.
Future works:
For current usage, the cache invalidator has to find the region by its id because they are not on the same JVM. This operation should be carefully dealt with.
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to be included in the release note

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

/run-all-tests tidb=v3.0.16 tikv=v3.0.16 pd=v3.0.16

LGTM


Add Volatility module

Volatility images can be submitted along with a profile and module to run:
turbiniactl -C rawmemory -l /home/mwatkins/vol-images/data.lime -P MacSierra_10_12_6_16G29x64 -m mac_pslist

Awesome!  I'm travelling tomorrow, but I'll take a closer look when I'm back in the office on Tuesday.

I signed it!

Whoops, looks like some unwanted commits were included in the last force push. Let me speak with @ericzinnikas about cleaning this up!

Ok, np.  Let me know when it's ready for review.
On Wed, Mar 20, 2019 at 3:34 AM Matt Watkins<EMAIL_ADDRESS>wrote:

Whoops, looks like some unwanted commits were included in the last force
push. Let me speak with @ericzinnikas https://github.com/ericzinnikas
about cleaning this up!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/google/turbinia/pull/361#issuecomment-474774474, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACiyu1W81KpJU2_eynF22iy_NzB3osvLks5vYg6tgaJpZM4b2FZ0
.

Ok, fixed the commit issues. Good to go here. @mwatkins-fb just looks like there are a couple of lint issues.


Better catalog example entities

The catalog-backend has two example yaml files that can be imported. They are very minimal and mock-like.
Replace them with one single file that has a believable set of entities, preferably around ten of them. Their names could for example be similar to some system of things inside Spotify, or at least be nicely chosen.
The end result should be that after import, the UI in Backstage is nice to view for example during a demo or when evaluating the catalog or Backstage itself.

Thanks! This is very important. Maybe we even have an empty state with a button that registers these example components. Let’s treat that as a separate issue in any case.

Fixed in #1245


extended tailwind classes not added to final classList.

Describe the bug
I have extended my tailwind theme with new a new class for a smaller fontSize:
fontSize: {
        mini: ['0.65rem', '0.75rem'],
      },

But when trying to use it in a tailwind-variants function. It never gets added to the final classes:
e.g.
export const itemDescription = tv({
  base: 'm-0 overflow-hidden whitespace-nowrap text-mini text-neutral-400',
})

The only workaround I have found is to override the class like tv({ class: 'text-mini '}) (this makes it work).
Expected behavior
extended tailwind theme options should work in the tv() base classes.
Desktop (please complete the following information):

OS: macOS
Browser: Chromium
Version  "tailwind-variants": "0.1.8",

They are working for me.
What doesn't work is when I use the extended utility with attribute variant like this: focus:text-mini

I have a similar issue. I customized the tailwind config and added different font sizes and colors, Tailwind variants do not understand the difference between text-[customized-color] and text-[customized-size], and the final class name does not include one of these. I mean the size overwrites the color or vice-versa. I have used  text-[customized-size] in  variants and  text-[customized-color] in compoundedVariants.

Tailwind merge doesn't load information from your Tailwind config automatically. You'll need to configure it for your custom class names following these guides:

https://www.tailwind-variants.org/docs/api-reference#twmergeconfig
https://github.com/dcastil/tailwind-merge/blob/v1.8.1/docs/configuration.md

Can you please elaborate a little bit more?
I expected tv to just work with my tailwind.config.js.

Tailwind variants and tailwind-merge do not load the tailwind.config.js file since they operate at runtime and importing the tailwind config file would result in the tailwindcss package being bundled into the application as well, which would greatly increase bundle size. You can see in the tailwind-merge docs exactly what I'm saying.

The tailwind-merge config is different from the Tailwind config because it's expected to be shipped and run in the browser as opposed to the Tailwind config which is meant to run at build-time. Be careful in case you're using your Tailwind config directly to configure tailwind-merge in your client-side code because that could result in an unnecessarily large bundle size.

That's the reason why it's not loaded automatically. Tailwind runs at build time, tailwind-variants/tailwind-merge run at runtime.

So I have the same problem. I have some special tailwind.config.js classes such as a button font size from my designer.
So in my tailwind.config.js I have text-button
twMerge and thus it seems tailwind variants drops these.
I'm curious if someone has a quick example of extending the tailwind merge config to just add a couple extra classes? If I find out how to I'll post back.

@GimpMaster See my answer for documentation links: https://github.com/nextui-org/tailwind-variants/issues/69#issuecomment-1725410594

@mskelton - I get that I'm just new to tv and tw-merge.
So I was looking for a simple working example. I'll post one when I can if I get it working.
I'm guessing it will be something like
const buttonVariants = tv({
	base: 'px-32 py-16 inline-flex items-center justify-center rounded-md font-semibold whitespace-nowrap ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2',
	variants: {
		variant: {
			contained:
				'hover:border-neutral-grey-800 hover:bg-button-primary-hover active:bg-button-primary-active bg-primary-blue-500 text-neutral-white rounded-20',
			destructive:
				'bg-button-danger-enabled hover:bg-button-danger-hover active:bg-button-danger-active text-neutral-white rounded-20',
			outlined:
				'hover:border-button-primary-hover hover:text-button-primary-hover active:border-button-primary-active active:bg-button-primary-active-alt active:text-button-primary-active border-solid border-2 border-primary-blue-500 text-primary-blue-500 rounded-20',
			text: 'text-primary-blue-500 px-32 py-16  hover:text-button-primary-hover active:text-button-primary-active active:bg-button-primary-active-alt  rounded-20'
		}
	},
	defaultVariants: {
		variant: 'contained',
		size: 'default'
	}
}, {
	twMergeConfig: {
		BLAH: 'BLAH'
	}
});

where I need to figure out how to do BLAH BLAH for my custom font sizes

Ok for anyone coming across this in the future here is a working example. It would be awesome to add something like this to the docs.
import { defaultConfig } from 'tailwind-variants';

defaultConfig.twMergeConfig = {
	classGroups: {
		customFontSizes: [
			'text-display1',
			'text-h1',
			'text-h1-desktop',
			'text-h2',
			'text-h2-desktop',
			'text-h2',
			'text-h3',
			'text-h3-desktop',
			'text-h4',
			'text-h4-desktop',
			'text-h5',
			'text-h5-desktop',
			'text-h6',
			'text-h6-desktop',
			'text-subheader1',
			'text-subheader2',
			'text-body1',
			'text-body2',
			'text-caption1',
			'text-caption2',
			'text-overline',
			'text-button'
		]
	}
};

In this case I modified the default config for all tv() functions. You could just as easily pass it in on each tv() function but this was better to cover all cases.
Also if you are using shadcn-svelte I went a step further since they use clsx and twMerge in a wrapper function called cn()
import { defaultConfig } from 'tailwind-variants';

// Since we are using custom font sizes, we need to define the class groups for both tailwind-variants and tailwind-merge
const classGroups = {
	customFontSizes: [
		'text-display1',
		'text-h1',
		'text-h1-desktop',
		'text-h2',
		'text-h2-desktop',
		'text-h2',
		'text-h3',
		'text-h3-desktop',
		'text-h4',
		'text-h4-desktop',
		'text-h5',
		'text-h5-desktop',
		'text-h6',
		'text-h6-desktop',
		'text-subheader1',
		'text-subheader2',
		'text-body1',
		'text-body2',
		'text-caption1',
		'text-caption2',
		'text-overline',
		'text-button'
	]
};

// This updates the tailwind-variants default config to include the customFontSizes class group
defaultConfig.twMergeConfig = {
	classGroups
};

// This is a custom tailwind-merge function that we can use with cn
const customTwMerge = extendTailwindMerge<'customFontSizes'>({
	//extend: { classGroups: { customFontSizes: ['text-button'] } }
	extend: { classGroups }
});

// This function combines clsx and customTwMerge to create a custom class name function.
// This allows us to use clsx style formatting such as cn('text-button', { 'text-primary-blue-500' : isPrimary}) and have it properly tailwind merge
export function cn(...inputs: ClassValue[]) {
	return customTwMerge(clsx(inputs));
}

Hopefully this helps someone else.


[Hub Generated] Review request for Microsoft.CostManagement to add version preview/2020-03-01-preview

This is a PR generated at OpenAPI Hub. You can view your work branch via this link.
Contribution checklist:

[x] I commit to follow the Breaking Change Policy of “no breaking changes
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

[x] Ensure to check this box if one of the following scenarios meet updates in the PR, so that label “WaitForARMFeedback” will be added automatically to involve ARM API Review. Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs, all “removals” and “adding a new property” no more require ARM API review.

Adding new API(s)
Adding a new API version
Adding a new service

[ ] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If there are following updates in the PR, ensure to request an approval from API Review Board as defined in the Breaking Change Policy.

[ ] Removing API(s) in stable version
[ ] Removing properties in stable version
[ ] Removing API version(s) in stable version
[ ] Updating API in stable version with Breaking Change Validation errors
[ ] Updating API(s) in preview over 1 year

Please follow the link to find more details on PR review process.

/azp run automation - sdk

Can one of the admins verify this patch?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

I see you are removing multiple types Microsoft.CostManagement/views, Microsoft.CostManagement/budgets! You need to follow deprecation flow as defined in https://armwiki.azurewebsites.net/api_contracts/APIDeprecationPolicy.html?q=deprecate for this type of change!

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

@chiragg4u Is this the proper configuration to avoid any deprecations and just add a new type with a new version from api perspective? Including the costmanagement.json in preview folder generated avocado errors so it seemed like this was the correct option, but want to make sure.

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

Signing off as per below comments from Sachin
[9:55 AM] Sachin Srivastava
Is it possible to treat the comments in a separate PR at a later time. We had an earlier review and was approved at that time. Hence we made the API and UX with the existing contract.. and any change at this time would have a lot of ripple effect.
[9:56 AM] Sachin Srivastava
We are in public-preview state.. and have to set up another iteration before we go GA.

/azp run automation - sdk

@zachraMSFT Could you please fix the CI error: LintDiff and Semantic?

/azp run automation - sdk

/azp run automation - sdk

/azp run automation - sdk

@njuCZ I fixed the errors when I run validation independently, but looks like checks here are stuck


Allow nested data objects to handle there own types

Currently, when creating a nested Data object from an array, the nested data that maps to another Data object either also has to be an array or has to already be that Data object.
This PR allows for those nested Data properties to be handled by the object itself.
Basic Example
Say I have the following Data classes:
class MemberData extends Data
{
    public function __construct(
        public string $name,
    ) {}

    public static function fromUser(User $user): static
    {
        return new static($user->name);
    }
}

and
class TeamData extends Data
{
    public function __construct(
        public string     $name,
        public MemberData $leader,
    ) {}
}

I'm able to pass a User model to create the MemberData (MemberData::from($user);), however, I'm not able to do the same with the TeamData using an array i.e. I would have to :
TeamData::from([
    'name'   => 'The best team ever!',
    'leader'  => [
        'name' => $user->name
    ],
    // or 'leader'  => MemberData::from($user),
]);

Which means I would always have to explicitly create the nested Data objects or update the array structures throughout the code base whenever one of those Data objects changes.
This PR would allow the nested UserData class to make use of it's creation methods:
TeamData::from([
    'name'   => 'The best team ever!',
    'leader'  => $user,
]);

Great addition! Thanks!


Running action now fails

When running action, it fails since 15th of may.
Container builds in workflow, but according action does fail with the message
"ls: cannot access 'code': No such file or directory"

From what I can see, is that the command ls in entrypoint.sh is in general correct. Removing ls from local build makes this working as expected. (Using nektos/act for local tests.)
This is the error message:

@totosan Thanks for submitting this. We will fix this today or tomorrow.

I already made one with this PR #12
Also please check following repos (will not spam issue lists :-) ):

aml-run
aml-compute
aml-registermodel

@totosan Sorry for the late reply. Are you running a special setup?
We ran a couple of tests and could not reproduce the issue. I would like to include your setup as a test case.

No, no self-hosted. Nothing special. I was using suggested the yml from MAzure Machine Learning example Repo.

Maybe a silly question, but how can the secret be set in the PR - Integration test of your repo.  My Pr fails, because secret is not set.
btw. I am using a private repo with ubuntu:latest as image.

You have to set a secret in your repo - here's instructions https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets

Hi @aronchick,
thx, the PR from my forked repo is running against this repo here. I set the secret at my repo, already. Maybe, I misunderstood something.

Maybe a silly question, but how can the secret be set in the PR - Integration test of your repo. My Pr fails, because secret is not set.
btw. I am using a private repo with ubuntu:latest as image.

@totosan  You have to add the secrets to this repo: https://github.com/totosan/aml-workspace
Can you please check the referenced branch and check whether it works as expected?
We will merge this to fix the issue. The same changes will be made to the other repos.

@totosan Can you please confirm whether the bugs are fixed in all master branches?
It is shown here how you can test the master branch of the AML Actions: https://github.com/marvinbuss/mlops/blob/be7a6470d76eb784c40d2bce46178d2d6cadbdd7/.github/workflows/train_deploy.yml#L16
I could no reproduce the error in my repo, but it would be great if you would provide some feedback. Draft releases will be made public once you confirm.

@totosan did you have a chance to test the changes in master of aml-workspace?

Yeah, it's working like a charm. From my perspective, the bug is resolved. Thx a lot.

Great. I will release a new version tonight. You can switch back to v1 tomorrow. Thanks for submitting the bug.
Feel free to submit additional feature requests and bugs in the future.

I released an update to all AML Actions, which should fix the bug.
Please reopen the issue, if it persists.


Add color list for navbar

Issue description
Add list for navbar colors

components: navbar
reactstrap version #5.0.0-beta

What is happening?
User should go to getbootstrap to know possible navbar colors.
What should be happening?
User see possible colors in navbar topic
Steps to reproduce issue

Open navbar
Don't find possible color list

Hey @AlexanderInsulin ,
Each component uses the same color palette.
I think it is redundant in every component to display this information.
But, we could add the background colors and gradients to the Utilities section,
like bootstrap did. https://getbootstrap.com/docs/4.0/utilities/colors/#background-color
@TheSharpieOne @supergibbs

Agreed, available colors are set by bootstrap not reactstrap. The prop value is used to add the class bg-<value> so best to refer to bootstrap docs which are valid. You can also make your own.

Yes, the bootstrap color names are well-known and this library is not limited to those.
I often have to add custom colors in addition to bootstrap, sticking with generic names such as "accent" or "alternative" and those values can be provided to color.
We do not limit the user to only using the defaults bootstrap colors out of the box like other libraries do and thus there is not set list of values which can be provided.


Added role assignments to LogicApp's identity

This PR contains fix for this Issue - Necessary "Role Assignments" for the LogicApp are not added

Code Changes

RoleAssignment resources added to the ARM template file

Details

NetworkContributor permissions are added to the Logic App's System-Assigned Identity

Updates required in the README.md file
The readme file in the playbook section needs to be updated with information on the additional parameters added to the template and the dependency on the appgateway and frontdoor deployments.
These components should be deployed using the templates provided in the WAF Attack Testing lab before the updated sentinel playbook could be run.

@tobystic, one of the checks (analyze/javascript) has failed, but I see no error that indicates problems with the checked-in code. Can you please take a look. Thanks.


carpentries-incubator/hpc-workflows

While the rest of the HPC Carpentry lessons have complicated templating and snippet libraries that will require manual effort to fix, we believe hpc-workflows is reasonably clean, and should be auto-convertible to the Workbench. I tried doing the conversion locally, but I could not get it working properly.
Is there a timeline for converting Incubator lessons, or a queue to have lessons in the Carpentries namespace converted? If so, would you be willing to convert hpc-workflows on our behalf, so we can get into hands-on learning with the Workbench using our own lesson development as motivation? (We'll keep working in the meantime.)

Nevermind -- we need to make major revisions to adapt the lesson to our purposes anyway, so we're archiving the old lesson and starting with a blank Workbench repository instead.


Added MacOS support

Added basic MacOS support
Added option to manually specify local Vulkan SDK path in vulkan_x repo rules

Hi @Morfly, thanks for the contribution!