File has unexpected size  https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/

Issue Title
It appears the checksums in the InRelease file have not been updated after a release.
General
2020-06-24T11:29:07.1014141Z Step 37/72 : RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
2020-06-24T11:29:07.1428743Z  ---> Running in 8f1f1f72af1b
2020-06-24T11:29:07.4318822Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-24T11:29:07.4324906Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-06-24T11:29:07.4326186Z
2020-06-24T11:29:07.5018269Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: apt-key output should not be parsed (stdout is not a terminal)
2020-06-24T11:29:07.5511675Z
2020-06-24T11:29:07.5512807Z 100   983  100   983    0     0   8191      0 --:--:-- --:--:-- --:--:--  8191
2020-06-24T11:29:08.0488990Z OK
2020-06-24T11:29:09.0280731Z Removing intermediate container 8f1f1f72af1b
2020-06-24T11:29:09.0281722Z  ---> 4116ce353101
2020-06-24T11:29:09.0282130Z Step 38/72 : RUN curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list | tee /etc/apt/sources.list.d/msprod.list
2020-06-24T11:29:09.0788654Z  ---> Running in 7172bb67770f
2020-06-24T11:29:09.3827705Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-06-24T11:29:09.3828204Z                                  Dload  Upload   Total   Spent    Left  Speed
2020-06-24T11:29:09.3828460Z
2020-06-24T11:29:09.4058013Z   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2020-06-24T11:29:09.4059060Z 100    79  100    79    0     0   3434      0 --:--:-- --:--:-- --:--:--  3434
2020-06-24T11:29:09.4059838Z deb [arch=amd64] https://packages.microsoft.com/ubuntu/16.04/prod xenial main
2020-06-24T11:29:10.0336367Z Removing intermediate container 7172bb67770f
2020-06-24T11:29:10.0337490Z  ---> b41c61050097
2020-06-24T11:29:10.0337947Z Step 39/72 : RUN apt-get update
2020-06-24T11:29:10.0794144Z  ---> Running in 21de1eb50b1e
2020-06-24T11:29:10.4557979Z Hit:1 http://deb.debian.org/debian buster InRelease
2020-06-24T11:29:10.4562997Z Hit:2 http://deb.debian.org/debian buster-updates InRelease
2020-06-24T11:29:10.4563741Z Hit:3 http://security.debian.org/debian-security buster/updates InRelease
2020-06-24T11:29:10.4611596Z Get:4 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease [4003 B]
2020-06-24T11:29:10.9540746Z Get:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [142 kB]
2020-06-24T11:29:10.9541253Z Err:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
2020-06-24T11:29:10.9541718Z   File has unexpected size (142137 != 142079). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
2020-06-24T11:29:10.9545512Z   Hashes of expected file:
2020-06-24T11:29:10.9546659Z    - Filesize:142079 [weak]
2020-06-24T11:29:10.9568588Z    - SHA512:1e2d73b7082e18c33f3b4a6a584b7929344cdc87b5c96d25e2dcb3e84d72733014364d098028d9dcaddb7fd194ec11f3a74df489a9f05f67e84638cd033525e1
2020-06-24T11:29:10.9569494Z    - SHA256:32a93cdfa7390dbc737271d543f7ae198d267a09aba44ca3e27beca49cc8471c
2020-06-24T11:29:10.9570119Z    - SHA1:3243a484d2fa337b05e6e4be925d4ed96ba2ca38 [weak]
2020-06-24T11:29:10.9571059Z    - MD5Sum:70ea695335d795cdf852fcdac44cb18e [weak]
2020-06-24T11:29:10.9573326Z   Release file created at: Fri, 19 Jun 2020 08:51:18 +0000
2020-06-24T11:29:10.9619688Z Fetched 4003 B in 1s (7692 B/s)
2020-06-24T11:29:11.6849439Z Reading package lists...
2020-06-24T11:29:11.7104230Z E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/main/binary-amd64/Packages.bz2  File has unexpected size (142137 != 142079). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
2020-06-24T11:29:11.7104962Z    Hashes of expected file:
2020-06-24T11:29:11.7105626Z     - Filesize:142079 [weak]
2020-06-24T11:29:11.7106397Z     - SHA512:1e2d73b7082e18c33f3b4a6a584b7929344cdc87b5c96d25e2dcb3e84d72733014364d098028d9dcaddb7fd194ec11f3a74df489a9f05f67e84638cd033525e1
2020-06-24T11:29:11.7107166Z     - SHA256:32a93cdfa7390dbc737271d543f7ae198d267a09aba44ca3e27beca49cc8471c
2020-06-24T11:29:11.7107756Z     - SHA1:3243a484d2fa337b05e6e4be925d4ed96ba2ca38 [weak]
2020-06-24T11:29:11.7108289Z     - MD5Sum:70ea695335d795cdf852fcdac44cb18e [weak]
2020-06-24T11:29:11.7108667Z    Release file created at: Fri, 19 Jun 2020 08:51:18 +0000
2020-06-24T11:29:11.7109071Z E: Some index files failed to download. They have been ignored, or old ones used instead.
2020-06-24T11:29:11.9092606Z The command '/bin/sh -c apt-get update' returned a non-zero code: 100
2020-06-24T11:29:11.9218316Z ##[error]The command '/bin/sh -c apt-get update' returned a non-zero code: 100
2020-06-24T11:29:11.9237722Z ##[error]The process '/usr/bin/docker' failed with exit code 100

The same for: https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2

Same issue here with the same package. Please let us know if we can help giving more info.

Same applies also for:
https://packages.microsoft.com/debian/10/prod/dists/buster/main/binary-amd64/Packages.bz2
Get:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 Packages [43.7 kB]
 Err:5 https://packages.microsoft.com/debian/10/prod buster/main amd64 Packages
   File has unexpected size (43969 != 43653). Mirror sync in progress?
   Hashes of expected file:
    - Filesize:43653 [weak]
    - SHA512:770b9079119523e27b68efe2742e196564815069321388ad9b1ce270da25b9c7f432fb57b210a6af4788d4460c1f80877c2a34de3a89e3d1584d74ed16c5f7c4
    - SHA256:cd6baf1fba149709cb0a9904cb455666f295e1ceb1cef6b92dc3ba138b90a8f3
    - SHA1:3cb42e61744f3cdb436f96e65d9669aa90623c88 [weak]
    - MD5Sum:4864e88c059a44fe020d60d1e2a80872 [weak]
   Release file created at: Wed, 17 Jun 2020 00:12:12 +0000

I'm taking a look at this (filing a ticket with the team, etc.) and will report back here with more info shortly.

@dagood re: #4167, it has been significantly longer than that amount of "Expected time".  Also, based on that issue, it appears the maintainers are not addressing the issue given the timeline, (nor contact-able).   :(

I have the same:
[12:51:31]	[Step 3/5] Preparing to unpack .../apt-transport-https_<IP_ADDRESS>_all.deb ... [12:51:31]	[Step 3/5] Unpacking apt-transport-https (<IP_ADDRESS>) ... [12:51:31]	[Step 3/5] Setting up apt-transport-https (<IP_ADDRESS>) ... [12:51:31]	[Step 3/5] Hit:1 http://security.debian.org/debian-security buster/updates InRelease [12:51:31]	[Step 3/5] Hit:2 http://deb.debian.org/debian buster InRelease [12:51:31]	[Step 3/5] Hit:3 http://deb.debian.org/debian buster-updates InRelease [12:51:31]	[Step 3/5] Get:4 https://packages.microsoft.com/debian/9/prod stretch InRelease [4009 B] [12:51:31]	[Step 3/5] Get:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [93.2 kB] [12:51:31]	[Step 3/5] Err:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [12:51:31]	[Step 3/5]   File has unexpected size (93446 != 93230). Mirror sync in progress? [IP: <IP_ADDRESS> 443] [12:51:31]	[Step 3/5]   Hashes of expected file: [12:51:31]	[Step 3/5]    - Filesize:93230 [weak] [12:51:31]	[Step 3/5]    - SHA512:6d9952cd289566e58aad057b0c24d1d450c2dc8adfd3871b06f843c6a8ea84853f6cda10e390ffddc9621299d2dcfe516e488a6fea215928d5446a52754dfa9a [12:51:31]	[Step 3/5]    - SHA256:c5c7ad271c940d9533231e61043d923d93f5bde7ab9927d8da9be3964c440bd4 [12:51:31]	[Step 3/5]    - SHA1:46696f97ceccbacdcdd9dda6d68fdbc1f7b76cd0 [weak] [12:51:31]	[Step 3/5]    - MD5Sum:49e53481a1cbdac343dd7d54cc3c2c52 [weak] [12:51:31]	[Step 3/5]   Release file created at: Wed, 17 Jun 2020 00:12:39 +0000 [12:51:31]	[Step 3/5] Fetched 4009 B in 1s (6488 B/s) [12:51:32]	[Step 3/5] Reading package lists... [12:51:32]	[Step 3/5] E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2  File has unexpected size (93446 != 93230). Mirror sync in progress? [IP: <IP_ADDRESS> 443] [12:51:32]	[Step 3/5]    Hashes of expected file: [12:51:32]	[Step 3/5]     - Filesize:93230 [weak] [12:51:32]	[Step 3/5]     - SHA512:6d9952cd289566e58aad057b0c24d1d450c2dc8adfd3871b06f843c6a8ea84853f6cda10e390ffddc9621299d2dcfe516e488a6fea215928d5446a52754dfa9a [12:51:32]	[Step 3/5]     - SHA256:c5c7ad271c940d9533231e61043d923d93f5bde7ab9927d8da9be3964c440bd4 [12:51:32]	[Step 3/5]     - SHA1:46696f97ceccbacdcdd9dda6d68fdbc1f7b76cd0 [weak] [12:51:32]	[Step 3/5]     - MD5Sum:49e53481a1cbdac343dd7d54cc3c2c52 [weak] [12:51:32]	[Step 3/5]    Release file created at: Wed, 17 Jun 2020 00:12:39 +0000 [12:51:32]	[Step 3/5] E: Some index files failed to download. They have been ignored, or old ones used instead. [12:51:32]	[Step 3/5] The command '/bin/sh -c apt-get install -y apt-transport-https && apt-get update && apt-get install -y aspnetcore-runtime-3.1 dotnet-runtime-3.1 cron' returned a non-zero code: 100 [12:51:32]	[Step 3/5]  [12:51:32]	[Step 3/5] Process exited with code 100 [12:51:32]	[Step 3/5] Process exited with code 100 (Step: Docker build (Docker)) [12:51:32]	[Step 3/5] Step Docker build (Docker) failed

same problem!!!!!
root@9075ffe6ea60:/# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu eoan InRelease
Hit:2 http://archive.ubuntu.com/ubuntu eoan-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu eoan-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu eoan-security InRelease
Get:5 https://packages.microsoft.com/ubuntu/19.10/prod eoan InRelease [10.5 kB]
Get:6 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main armhf Packages [504 B]
Get:7 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main arm64 Packages [762 B]
Get:8 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main amd64 Packages [27.7 kB]
Err:8 https://packages.microsoft.com/ubuntu/19.10/prod eoan/main amd64 Packages
  File has unexpected size (27870 != 27700). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
  Hashes of expected file:
   - Filesize:27700 [weak]
   - SHA512:9b70e7c196b4aa1c30d66c9f3557217c32bff6ac6d330d9ed31cbb2993eb1146fd8820fe806668279d6f5cd6ee820b3a94494b67c122aa7d14c1291d18a85bbf
   - SHA256:7ff431b24135808bcb270b508ca9e81ef4b42b2c8fb24d9e7468aedbdd22449d
   - SHA1:4b94fc1de31196082b326cb61465a0c2c694a678 [weak]
   - MD5Sum:3f192fc18261e734bf26080c713ac5a0 [weak]
  Release file created at: Wed, 17 Jun 2020 00:13:22 +0000
Fetched 11.7 kB in 1s (22.2 kB/s)
Reading package lists... Done
E: Failed to fetch https://packages.microsoft.com/ubuntu/19.10/prod/dists/eoan/main/binary-amd64/Packages.gz  File has unexpected size (27870 != 27700). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
   Hashes of expected file:
    - Filesize:27700 [weak]
    - SHA512:9b70e7c196b4aa1c30d66c9f3557217c32bff6ac6d330d9ed31cbb2993eb1146fd8820fe806668279d6f5cd6ee820b3a94494b67c122aa7d14c1291d18a85bbf
    - SHA256:7ff431b24135808bcb270b508ca9e81ef4b42b2c8fb24d9e7468aedbdd22449d
    - SHA1:4b94fc1de31196082b326cb61465a0c2c694a678 [weak]
    - MD5Sum:3f192fc18261e734bf26080c713ac5a0 [weak]
   Release file created at: Wed, 17 Jun 2020 00:13:22 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

@dagood , I know this team is not responsible for the msodbcsql17 package, but it is also impacting that package.  Like others have said, the packages.microsoft.com ubuntu repo is hosed for 16.04 across the board.

re: #4167, it has been significantly longer than that amount of "Expected time".

Yes, that "expected time" is just an advisory that the service might get healthy again in some situations. This is clearly an exceptional case.
On my end I've filed https://icm.ad.msft.net/imp/v3/incidents/details/193848733/home with the team. (Internal link, posting here for our own easy access in future.)

Same issue here:
apt-get update
Hit:1 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://eu-west-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://ppa.launchpad.net/ansible/ansible/ubuntu xenial InRelease
Get:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease [4,003 B]
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [142 kB]
Ign:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [158 kB]
Ign:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
Get:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages [791 kB]
Err:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial/main amd64 Packages
  Writing more data than expected (792466 > 791391)
Fetched 4,003 B in 0s (7,632 B/s)
Reading package lists... Done
E: Failed to fetch https://packages.microsoft.com/ubuntu/16.04/prod/dists/xenial/main/binary-amd64/Packages  Writing more data than expected (792466 > 791391)
E: Some index files failed to download. They have been ignored, or old ones used instead.

Is this related to unable to install netcore 2.2 on ubuntu 16.04, when doing apt-get install dotnet-sdk-2.2:
E: Unable to locate package dotnet-sdk-2.2
E: Couldn't find any package by glob 'dotnet-sdk-2.2'
E: Couldn't find any package by regex 'dotnet-sdk-2.2'

Is this related to unable to install netcore 2.2 on ubuntu 16.04, when doing apt-get install dotnet-sdk-2.2:

Yes, I expect your apt update failed earlier, and this caused the package metadata about dotnet-sdk-2.2 to not be downloaded. Ubuntu 16.04 is one of the distros where this problem is occurring. (I feel obliged to point out that 2.2 is EOL: you should try to upgrade to 3.1, or use 2.1 if the downgrade doesn't break you is more feasible.)

So far these repos have been reported to be affected, and I've repro'd a few of them to confirm:

Ubuntu 16.04
Ubuntu 19.10
Debian 9
Debian 10

The Linux package repository team is working on applying a fix, no ETA yet.

(I feel obliged to point out that 2.2 is EOL: you should try to upgrade to 3.1, or use 2.1 if the downgrade doesn't break you and it's more feasible.)

Upgrade to 3.1 is in full progress, but until it's done, 2.2 needs to work :-)

Upgrade to 3.1 is in full progress, but until it's done, 2.2 needs to work :-)

Cool. 🙂 And yeah, it's definitely our intention to keep EOL versions available on download endpoints and fix up issues like this as soon as we can.

@dagood what about .net core 3.1? i can't install it. So it's not about 2.2

what about .net core 3.1? i can't install it. So it's not about 2.2

You're correct, this outage applies to the entire Microsoft Linux package repository (every package) for each affected distro. The fix that the Linux package repo team is applying should restore access to all packages on these feeds, once they do it.

Hi there. Same here:
*****@**********:/# apt-get update
Hit:1 http://security.debian.org stretch/updates InRelease
Ign:2 http://deb.debian.org/debian stretch InRelease
Hit:3 http://deb.debian.org/debian stretch-updates InRelease
Hit:4 http://deb.debian.org/debian stretch Release
Get:7 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [103 kB]
Err:7 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
  Writing more data than expected (103561 > 103377)
  Hashes of expected file:
   - Filesize:103377 [weak]
   - SHA512:fa3d4bc9b2634e2c25be0c0afa362c32933f1df199213972bf7b45676afd852da61f6efc8b1ca2dd8617ec9d672425d354e528d4baf599e7ae621649c1e0ce9e
   - SHA256:9fe010504f92e67c9fb22d37e6af9e02f72fff3de54edc28f5e726ece0d2a7d6
   - SHA1:40ebf0eb5d1fb7bd2916d2cc2b8b334309c6daf9 [weak]
   - MD5Sum:ee0f5824b41c62a11fb2dda476171a7e [weak]
  Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
Reading package lists... Done
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.gz  Writing more data than expected (103561 > 103377)
   Hashes of expected file:
    - Filesize:103377 [weak]
    - SHA512:fa3d4bc9b2634e2c25be0c0afa362c32933f1df199213972bf7b45676afd852da61f6efc8b1ca2dd8617ec9d672425d354e528d4baf599e7ae621649c1e0ce9e
    - SHA256:9fe010504f92e67c9fb22d37e6af9e02f72fff3de54edc28f5e726ece0d2a7d6
    - SHA1:40ebf0eb5d1fb7bd2916d2cc2b8b334309c6daf9 [weak]
    - MD5Sum:ee0f5824b41c62a11fb2dda476171a7e [weak]
   Release file created at: Wed, 17 Jun 2020 00:12:39 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Same issue here for debian 10.

Same here for both Debian 10 and 9

it works now for ubuntu 18.04

Looks like back up for 16 as well.  Closing.

Just tried debian 10 and still not working

NVM, working now

Fixed in debian 9 from now.

Fixed on Ubuntu 16.04

Thanks all for checking and posting status. The repo admins haven't gotten back to us since their last update, but @rbhanda applied a fix that worked for us in the past that seems to have cleared things up.

Fixed on Debian 10

The repo admins found some more repositories that needed fixups (Ubuntu 18.10, 17.10, 19.04), and they've resolved those and closed our ticket--all repositories should be working at this point.

@dagood  thank you for facilitating with the internal team!

@dagood thnx man!

Hi there, happened again on:
Linux d7bd89b85a20 5.7.7-100.fc31.x86_64 #1 SMP Wed Jul 1 20:37:05 UTC 2020 x86_64 GNU/Linux

Debian 10.4

Eror log:
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Get:4 https://packages.microsoft.com/debian/9/prod stretch InRelease [4009 B]
Get:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages [96.4 kB]
Err:5 https://packages.microsoft.com/debian/9/prod stretch/main amd64 Packages
  File has unexpected size (96509 != 96366). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
  Hashes of expected file:
   - Filesize:96366 [weak]
   - SHA512:383e408b929517161a2c48b7b0ac38eaccc58bddbdca88250941867f31ef9f12138196552cd0549a2b129b0add9dd4a61f2b474165a7ca5f4a7231652b03f7fc
   - SHA256:63ce57b0a494d005bcccd829efe2b28dadec3823eea3047deb2a191aa7ce6287
   - SHA1:16a3b6bf3754e25c7dbc323b2144e6258ace9a78 [weak]
   - MD5Sum:9d5568e53a8d61f77d8449dcb31a262b [weak]
  Release file created at: Tue, 14 Jul 2020 16:28:55 +0000
Fetched 4009 B in 0s (8191 B/s)
Reading package lists...
E: Failed to fetch https://packages.microsoft.com/debian/9/prod/dists/stretch/main/binary-amd64/Packages.bz2  File has unexpected size (96509 != 96366). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
   Hashes of expected file:
    - Filesize:96366 [weak]
    - SHA512:383e408b929517161a2c48b7b0ac38eaccc58bddbdca88250941867f31ef9f12138196552cd0549a2b129b0add9dd4a61f2b474165a7ca5f4a7231652b03f7fc
    - SHA256:63ce57b0a494d005bcccd829efe2b28dadec3823eea3047deb2a191aa7ce6287
    - SHA1:16a3b6bf3754e25c7dbc323b2144e6258ace9a78 [weak]
    - MD5Sum:9d5568e53a8d61f77d8449dcb31a262b [weak]
   Release file created at: Tue, 14 Jul 2020 16:28:55 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

@dagood  the  https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2  File has unexpected size happens again in ubuntu 18.04 when apt-get update:

E: Failed to fetch https://packages.microsoft.com/ubuntu/18.04/prod/dists/bionic/main/binary-amd64/Packages.bz2  File has unexpected size (141364 != 141252). Mirror sync in progress? [IP: <IP_ADDRESS> 443]
Hashes of expected file:
- Filesize:141252 [weak]
- SHA512:dffbaf15f537ae643df110b279248993b2828e99352dff51a5df9648cf39531ffb7022ec2f0d9e553287c9889addcb5e734d9df95f749c456db04fd328a67ef4
- SHA256:6757f7f09fb2a0491657aac899faa29bd67fe7b9dfd849844f7fafca9afcc33e
- SHA1:37be76673db70e808db91294af54fddf476aae7f [weak]
- MD5Sum:ecc0d5d8d9373eee7aeceda58d7be8f9 [weak]
Release file created at: Tue, 13 Oct 2020 22:35:00 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

Thanks for reporting, I will follow up on this and keep you posted. Is this an issue with both .NET Core 2.1 and 3.1?

It looks like the new outage was also reported at https://github.com/dotnet/core/issues/5373. It affects the entire 18.04 feed, not specific .NET packages.
I'm going to close this issue as resolved: https://github.com/dotnet/core/issues/5373 tracks the new 18.04 outage, so check there for the latest info.


Checkstyle checks for AvoidStaticImport, UnusedImports.

Many times I see comments like "unused import" and "avoid static imports" in PRs,
this will automate that kinds of review. I also removed the small number of those we
already had in production code.
This excludes tests from AvoidStaticImport, since those are used often there and
I didn't want to make this changeset too large.

👍

👍

From travis on jdk7,
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (validate) on project druid: Execution validate of goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check failed: An API incompatibility was encountered while executing org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check: java.lang.UnsupportedClassVersionError: com/puppycrawl/tools/checkstyle/api/CheckstyleException : Unsupported major.minor version 52.0

Maybe the newer checkstyle doesn't support java7, will look into that.

Downgraded to Checkstyle 6.19, which supports jdk7.

@gianm maybe activate different versions of checkstyle depending on the build jdk?

@leventov I don't see a really compelling reason to do that rather than just using the same version for both jdks.

Merging, since the only change since two 👍s was to fix the jdk7 issue.


WIP: build-script: teach the script to install SwiftSyntax build.

Quick comment: Indention in the Python script has been 4 spaces so far, looks like you have been using 2 in some places.


EXSWHTEC-363 - Implement tests for Surface Management APIs

hipCreateSurfaceObject
hipDestroySurfaceObject.

PR has been merged into develop branch via commit 7685c9c46a52cb267d8b1afff0d5f2049bb3d02e.


Redo query construction and worker mapping

I broke down the process of assigning workers to fragments
into 6 steps, with rules of precedence:

Obey user-overrides of fragment workers.
Fragments that scan tables must use the workers that contain the data.
Edge constraints between fragments. E.g., a {@link LocalMultiwayProducerEncoding} must use the same set of workers as its consumer.
Singleton constraints: Fragments with a {@link CollectConsumerEncoding} or a {@link SingletonEncoding} must run on a single worker. If none is still set, choose an arbitrary worker.
Repropagate edge constraints.
Unspecified: Any fragments that still have unspecified worker sets will use all workers in the cluster. (No need to repropagate edge constraints).

I then refactored the existing 2 functions that did all of this, incorrectly, into
six and heavily commented each one. There is now a lot more sanity-checking of plans
as well, and all the existing tests pass!

[x] Fix #693.
[x] add more tests.

Coverage increased (+0.19%) to 58.17% when pulling 601b59a1998958216b891aeff00f26976767aade on fix-local-multiway-producer into 34170b1638141eaa2fcd535e9e2934d453fa4bd9 on master.

Coverage increased (+0.17%) to 58.15% when pulling 5b5ce0e8241b264f28b9a4a6532e7bf97a7d2c69 on fix-local-multiway-producer into 34170b1638141eaa2fcd535e9e2934d453fa4bd9 on master.

Coverage decreased (-1.05%) to 57.99% when pulling c600440656fb84a1d522774bafc76a01406c62fb on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

Coverage increased (+0.18%) to 59.22% when pulling c600440656fb84a1d522774bafc76a01406c62fb on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

Coverage increased (+0.2%) to 59.24% when pulling 9e87fb0ba0a13a4f3bdea24e918a8449ea07de68 on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

Coverage increased (+0.2%) to 59.24% when pulling 9e87fb0ba0a13a4f3bdea24e918a8449ea07de68 on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

Coverage increased (+0.34%) to 59.37% when pulling 94f95902bb383c400b3ac5782a3a4d5cda349e52 on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

@jingjingwang I think you would be the best one to review this PR, if you have time. The main question marks for me have to do with my sanity checks of edges, e.g., that only MultiwayProducer and EOSController can have multiple affiliated consumers.

A future PR should remove some of the redundancy in the code... I recompute a lot of things in those 6 functions, but it was the best way IMO to make it very clear what was going on.

@jingjingwang bump ;)

Coverage increased (+0.04%) to 59.08% when pulling 59cb01c4ae0bbb40cb476901a66af6270670ad57 on fix-local-multiway-producer into 4a07c4cc4dcea0a13df84aaad400e9eed927ee61 on master.

I can imagine to have more tests in JsonBadPlanTests, but this one is good enough as a start  :)

Looks good, except for this LMP/LMC issue: https://github.com/uwescience/myria/pull/694#commitcomment-10034455.
Do you have any specific tests in mind you'd like to add now? Obviously I was unable to add the tests for the EOS stuff I didn't understand :)


Too Close Together?

Industrial Foregoing Version: industrialforegoing-1.12.2-1.12.13-237
TeslaCoreLib Version: 1.12.2-1.0.15
Crashlog (If Any): N/A
I have several plant harvesters and sowers near each other and I just can't get them to work. They are set with range addons and there is one block between the fields. the second one in line only harvests and plants the first row closest to the last field and the rest just don't work. I have potato fields next to each other as well that don't seem to work next to each other.

Make sure that the area isn't protected by Minecolonies or something similar


Fix minor formatting and go.sum

Minor formatting fix and go.sum update.

@mengqiy Fails due to starlark image unavailable


[Docs] Add to readme: links, getting help, load balancing

Purpose
This PR makes a handful of README updates:

Updated screenshot that says "AI search" vs "Cognitive search". @charris-msft the response changed since the last time we took it, more succinct, is okay?
More links to recent videos and blog posts
A getting help section that declares that Microsoft support doesn't support this repo.
As part of that, I also enabled discussions, so that we can move things into discussions that aren't real bugs. Let me know if you want to keep this repo as issues-only. The drawback is that there'll be one more place to search for answers, but the benefits are a more focused issue tracker and lower barrier for discussion-like questions.
Tips for load balancing OpenAI in the productionizing doc.

Does this introduce a breaking change?

[ ] Yes
[X] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[X] Documentation content changes
[ ] Other... Please describe:

Fixes #1012

@pamelafox - I'm impressed that we got the same citations, but the response is a bit succinct... and do you know if we did anything intentionally that would have eliminated the follow-up questions? Those are a really nice feature to show.

@charris-msft Ah oops, we default to them being off in this repo, I'll turn them on for the screenshot.


.Local Domain Support?

Many years ago, Microsoft encouraged the use of .local for Active Directory domains -- AD domains can't be renamed with Exchange in-place.
While I have a forward-zone in place for this domain, it doesn't resolve hostnames to an IP from a client using unbound for DNS.
The first example forwards correctly while the second does not resolve.
forward-zone:
	name: "example1.net."
	forward-addr: <IP_ADDRESS>
	forward-addr: <IP_ADDRESS>

forward-zone:
        name: "example2.local."
        forward-addr: <IP_ADDRESS>
        forward-addr: <IP_ADDRESS>

Using unbound 1.9.0.

There is no special handling for local. in unbound.
What do you see in the logs when trying to resolve a local subdomain? I assume you are getting SERVFAIL as the result?
I assume the server listening there is authoritative for that zone, in that case I would advise to use stub-zone: instead of forward-zone:. As also stated in the "Stub Zone Options" section of the man page (latest version here), if you are doing DNSSEC validation in unbound you would either need to:

specify domain-insecure: for that domain, or
add a trust anchor key for that domain in case you serve your own DNSSEC signed records there.

Perfect, stub-zone works great!

.local is mDNS over port 5353 and AFAIK is completely unrelated to any typical DNS service, you'd use something like Avahi, although systemd-resolved also has mDNS support IIRC.
Someone had mentioned on Reddit of forwarding .local DNS queries to mDNS via Unbound, which isn't something I thought could be done? Android lacks mDNS support(unless you're the developer of the app and use certain APIs), so a web browser cannot make such queries, hence the advice given.
I did come across this potentially related PR, but I am lacking clarity on how to utilize it. I see no mention of plugins when searching for documentation for Unbound. There is little discussion on the PR, but it sounds like it'd allow for Unbound to get local device IPs if receiving a .local domain request from a device that doesn't use mDNS directly?

The aforementioned PR added example code to be used with the python module (module that executes python code during operation).
You can find the file at pythonmod/examples/avahi-resolver.py in the source folder or online.
This requires unbound to be compiled with python module support and some extra python packages in your Python installation.
Documentation on how to configure unbound and run the example can be found at the top of the file.

As someone not familiar with Unbound or Python and how to use that with Unbound, I only browsed the public docs and wasn't particularly interested in building them locally to view anymore on the matter. There wasn't much information online when I did some searches on google. Unbound appears to be a solid choice for handling DNS, but for what I wanted to do I wasn't looking forward to attempting it as my first choice.
I came across CoreDNS and an external plugin for that. Possibly a similar situation, and while the docs weren't fantastic it was slightly more accessible for me. Cloned the repo, added the external plugin repo to a plugin.cfg file, ran three go commands to build a binary that the docs covered online, ran another command to verify that all went well and then tried out a basic config from some snippets.
Got it working, so I won't be trying out Unbound, I realize it's a niche feature but definitely an area that could be improved for less experienced users. No clue how long the online docs have stated the pythonmod docs have been in the "final stages", I assume it's been a while and may continue to be.


Raspberry 4 boot fails with rpi4 kernel, systemd-boot, UEFI

Describe the bug
After rebuilding with some hardware and boot upgrades (see below), my Raspberry 4 fails to boot using the linux_rpi4 kernel. Latest kernel (6.11 as of now) works.
Booting to a generation with these changes, it doesn't even get to stage 1 of the NixOS boot process, this is the only thing I see:
EFI stub: Booting Linux Kernel...
EFI stub: Loaded initrd from LINUX_EFI_INITRD_MEDIA_GUID device path
EFI stub: Generating empty DTB
EFI stub: Exiting boot services...

Previous configuration
I had an HDD with MBR partition table with an ext4 partition. I didn't change the bootloader, so it was using U-Boot. I'm also using the raspberry-pi-4 module from nixos-hardware. This worked well.
{
  boot = {
    initrd.availableKernelModules = [ "xhci_pci" "usbhid" "usb_storage" ];
    loader = {
      grub.enable = false;
      generic-extlinux-compatible.enable = true;
    };
  };

  # this is also set from nixos-hardware
  boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;

  system.stateVersion = "23.05";
}

New configuration
I've got a new SSD, put a GPT partition table on it and made a main btrfs partition and a vfat boot partition, and installed NixOS on it.
After cloning the previous configuration and enabling systemd-boot, I ran into this error: The option system.build.installBootLoader is defined multiple times. (related: https://github.com/NixOS/nixpkgs/issues/173948). To fix this I disabled boot.generic-extrlinux-compatible.
I also updated system.stateVersion, the hardware config.
After making these changes to the previous configuration, rebuilding, and rebooting, the latest boot entry fails with the aforementioned EFI stub messages.
Changing the kernel to the latest (non-rpi) fixed this issue.
{
  boot = {
    supportedFilesystems = [ "btrfs" ];
    loader = {
      efi.canTouchEfiVariables = true;
      systemd-boot.enable = true;
      # systemd-boot and this tries to install bootloader, disable this
      generic-extlinux-compatible.enable = lib.mkForce false;
    };
  };

  # rpi kernel set from nixos-hardware fails with EFI stub error on boot
  boot.kernelPackages = lib.mkForce pkgs.linuxPackages_latest;

  system.stateVersion = "24.05";
}

Full diff between previous and new config: https://github.com/ymstnt-com/raspi-dotfiles/compare/96aae2d3b44dcf32665f83fb0394c565cb5cc12e...f8fb07b50601dbe0ac74844743e617e2051a2b31
Expected behavior
This should work with rpi4 kernel, or at least throw some useful errors.
Notify maintainers
I couldn't find who is the official maintainer and searched for people who contributed to similar lower-level stuff, feel free to ignore if it's not related to you.
@ElvishJerricco @Majiir  @arianvp @K900 @spiage

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
NOTE: I ran this on the latest kernel where this is not an issue.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
install: cannot remove '/tmp/env-vars': Operation not permitted
 - system: `"aarch64-linux"`
 - host os: `Linux 6.11.2, NixOS, 24.11 (Vicuna), 24.11.20241009.5633bcf`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Lix, like Nix) 2.91.0
System type: aarch64-linux
Additional system types:
Features: gc, signed-caches
System configuration file: /etc/nix/nix.conf
User configuration files: /home/gep/.config/nix/nix.conf:/etc/xdg/nix/nix.conf:/home/gep/.nix-profile/etc/xdg/nix/nix.conf:/nix/profile/etc/xdg/nix/nix.conf:/home/gep/.local/state/nix/profile/etc/xdg/nix/nix.conf:/etc/profiles/per-user/gep/etc/xdg/nix/nix.conf:/nix/var/nix/profiles/default/etc/xdg/nix/nix.conf:/run/current-system/sw/etc/xdg/nix/nix.conf
Store directory: /nix/store
State directory: /nix/var/nix
Data directory: /nix/store/40i0syl2n33kakvlk24ymmg1vr6nhh0f-lix-2.91.0/share`
 - nixpkgs: `/nix/store/60sn02zhawl3kwn0r515zff3h6hg6ydz-source`

Add a :+1: reaction to issues you find important.

Please don't use vendor kernels.

How are you booting the RPi4 with UEFI? That's not a standard part of the pi and there's multiple ways to do it, so it's a relevant detail.
Though whenever you see no output following EFI stub: Exiting boot services..., it usually means it is actually booting fine but the kernel just isn't using the right console. So try to e.g. connect to it over ssh or something, and/or use a console= kernel param to get it to output to the right console.

We are using pftf/RPi which provides installable builds of official EDK2 Raspberry Pi 4 UEFI firmware.
When this "error" happens, the Raspberry does not boot, which means we can't connect over SSH, nor using a monitor and keyboard.
We haven't tried the console= param though, but considering SSH doesn't work and the Raspberry doesn't appear online, I'd say it really gets stuck at the boot process.

Sounds like a very early kernel panic then. I've had my fair share of experiences with that sort of problem with UEFI implementations on the RPi and I haven't learned much about how to deal with them. Suffices to say this is not a configuration that the RPi kernel devs support, so mainline might be worth a try. But, like I said, getting kernels to reliably boot on the available UEFI implementations for the RPi4 is rather painful.

Thanks, we will be using the mainline kernel, so far we have no issues with it.


Broker Load support Huawei OBS

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[Feature Request] smaller url width

The related Mod/Theme: Super URL Bar
Your Enhancement Suggestion/Feature Request option for smaller url width! it'd be nice if it conformed to the width of the website domain like in arc browser, tho i'm not sure if that's possible with zen. just a suggestion anyway 😅 thank you!

It's a duplicate of #45 yet this one is better described and titled.


AVRO-3219: Add nullable enum type field support.

Signed-off-by: Zike Yang<EMAIL_ADDRESS>Jira

[x] My PR addresses the following Avro Jira issues and references them in the PR title.

https://issues.apache.org/jira/browse/AVRO-3219

Tests

[x] My PR adds the following unit tests: TestNullableEnumResolution

Commits

[ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] This is just a bug fix. No need doc.

Re: @blachniet @zcsizmadia @KyleSchoonover
This is marked for release 1.11.1 -- would it be possible to take some time to review?

If you can address the comments, we can probably get this into 1.11.1!

Great. I will do it today.

Thanks for your review. PTAL again. @KalleOlaviNiemitalo @RyanSkraba

Thanks very much for the contribution @RobertIndie and the review @KalleOlaviNiemitalo


Fix CLI Validator and Catalog bugs

Closes #1
Closes #2
This package is awesome -- thanks so much for contributing! Once you push it to PyPI, please let me know -- I'd love to promote it

Great finds! Thanks for the fixes. Will certainly let you know once it's ready for primetime.


Move assignment of url throws exceptions

The docs for url's move assignment say that it throws nothing. (See here.)
Yet, the move assignment throws an exception in this function:
void fill(urls::url& url)
{
  urls::url new_url = urls::parse_uri("https://www.gmail.com/my/path").value(); // ok
  url = std::move(new_url); // throws!
}

The exception is of the type invalid_argument and is caused by another fishy inheritance in the library: static_url derives from url but does not adhere to url's contract. The calling function looks like this:
int main()
{
    urls::static_url<10> url;
    fill(url);
}

If the goal was to reuse some implementation from url in static_url then I suggest to either use composition or to introduce another class url_base that derives from url_view and is used as a base class both for url and static_url.

Unlike the other report, this is an actual violation :) Yes we need url_base and the refactoring that follows.

fishy inheritance

LOL...


[SPARK-20435][CORE] More thorough redaction of sensitive information

This change does a more thorough redaction of sensitive information from logs and UI
Add unit tests that ensure that no regressions happen that leak sensitive information to the logs.
The motivation for this change was appearance of password like so in SparkListenerEnvironmentUpdate in event logs under some JVM configurations:
"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password ..."
Previously redaction logic was only checking if the key matched the secret regex pattern, it'd redact it's value. That worked for most cases. However, in the above case, the key (sun.java.command) doesn't tell much, so the value needs to be searched. This PR expands the check to check for values as well.
How was this patch tested?
New unit tests added that ensure that no sensitive information is present in the event logs or the yarn logs. Old unit test in UtilsSuite was modified because the test was asserting that a non-sensitive property's value won't be redacted. However, the non-sensitive value had the literal "secret" in it which was causing it to redact. Simply updating the non-sensitive property's value to another arbitrary value (that didn't have "secret" in it) fixed it.

Test build #76051 has started for PR 17725 at commit 2f5148a.

Test build #76051 has finished for PR 17725 at commit 2f5148a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76051/
Test FAILed.

Test build #76054 has started for PR 17725 at commit 0603686.

Few decisions that I made here:

I considered if just sun.java.command property should be checked and redacted but that seemed to specific and likely a bandaid to the current problem, not a long-term solution, so decided against doing it.
Redaction for the SparkListenerEnvironmentUpdate event was solely being done on Spark Properties, while sun.java.command is a part of System Properties. I considered doing redaction for System Properties in addition to Spark Properties (that would have gone somewhere around here) but decided against it because that would have even more hardcoding and I didn't see why these 2 special kinds of properties are special enough to be redacted but the rest of them. So, decided to redact information from all kinds of properties.
One way to redact the property value would have been to redact the minimum possible set from the value while keeping the rest of the value intact. For example, if the following were the unredacted case:
"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password --conf spark.other.property=2"
One option for the redacted output could have been:
"sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=*********(redacted) --conf spark.other.property=2"
However, such a redaction is very hard to maintain. For example, we would had to take the current regex (which is (?i)secret|password by default and add matchers to it like so (?i)secret|password like "("+SECRET_REDACTION_DEFAULT+"[^ ]*=)[^ ]*". That would allow us to squeeze out and replaced just the matched portion. But this all seemed very fragile and even worse when the user supplies a non-default regex so I decided it was easiest to simply replace the entire value, even though only a small part of it contained secret or password in it.
One thing which I didn't explicitly check was the performance implications of this change. The reason I bring this up is because, previously we were comparing keys with a regex, now if the key
doesn't match, we match the value with the regex. So, in the worst case, we are twice as many regex matches as before. Also, before we were simply doing regex matching on Spark Properties, now we do them on all properties - Spark Properties, System Properties, JVM Properties and Classpath Properties. I don't think this should have a big performance impact so I didn't invest time in it, mentioning here in interest of full disclosure.

Thanks in advance for reviewing.

Test build #76055 has started for PR 17725 at commit 14b0d72.

Test build #76054 has finished for PR 17725 at commit 0603686.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76054/
Test PASSed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76055/
Test FAILed.

Jenkins, retest this please.

Test build #76114 has started for PR 17725 at commit 14b0d72.

Test build #76114 has finished for PR 17725 at commit 14b0d72.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76114/
Test PASSed.

Merged build finished. Test PASSed.

Test build #76143 has started for PR 17725 at commit 80e40ba.

Test build #76143 has finished for PR 17725 at commit 80e40ba.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76143/
Test FAILed.

Hmm, this is weird, it ran for 4h10 mins and got killed due to a timeout. Looking...

Jenkins, retest this please.

Kicking off another run here while I am running a run locally.

Test build #76156 has started for PR 17725 at commit 80e40ba.

Test build #76156 has finished for PR 17725 at commit 80e40ba.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76156/
Test FAILed.

Merged build finished. Test FAILed.

This time it finished on regular time (2h 29m) but failed a test. I ran that test locally (org.apache.spark.streaming.ReceiverSuite's 'receiver life cycle') and it passed for me, so for mostly laziness, I am going to issue another run here.

Jenkins, retest this please.

Test build #76194 has started for PR 17725 at commit 80e40ba.

Test build #76194 has finished for PR 17725 at commit 80e40ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76194/
Test PASSed.

Finally it's passing!

Merging to master / 2.2.

Thanks Marcelo!
On Apr 26, 2017 5:06 PM, "Marcelo Vanzin"<EMAIL_ADDRESS>wrote:

Merging to master / 2.2.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/spark/pull/17725#issuecomment-297575053, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABoVi9s2uyakNm-81qWP3aBpRkFHtm3Jks5rz9vugaJpZM4NE66l
.


multiply fractions with symbol in denominator

https://github.com/socraticorg/mathsteps/issues/139

I fixed this issue the cheating way, sorry! Refactoring how cancelling works will have to happen another time :)

Good call re: constant in numerator -- I removed that constraint! I don't totally remember why I had it in the first place 😆

just add that comment explaining what's up
then 🚢  :D


WMI Windows Agents plugin incorrectly reported as "not deprecated"

The WMI Windows Agents plugin is deprecated, but its plugin health score page lists that it is not deprecated.  That gives it a better score than it deserves.
Expected result
Plugin health score for WMI Windows Agents plugin is much lower than 70.
Actual result
Plugin health score for WMI Windows Agents plugin is 70.

This is interesting. I see that the plugin is marked as such in plugins.jenkins.io, but it is not in the update-center.
The update-center is not marking the plugin as deprecated (see https://github.com/jenkins-infra/update-center2/blob/master/resources/deprecations.properties). The plugin repository has the topic deprecated.
I can see that the Plugin site API has correctly the label deprecated for this plugin (https://plugins.jenkins.io/api/plugin/windows-slaves), and the same for the update-center.
The problem is that in https://github.com/jenkins-infra/plugin-health-scoring/blob/ace129cab7c0a23ff7ecb83519a8437e2736e7ef/src/main/java/io/jenkins/pluginhealth/scoring/probes/DeprecatedPluginProbe.java#L44-L50, the code is only looking at the deprecation section of the update-center. This is when the update-center itself mark a plugin as deprecated (first link), but not when a plugin marks itself as deprecated (topic on plugin repository).
This is a bug.
Thank you @MarkEWaite for finding this!


InvalidURL error with default values

Hello,
I tried the code getHorizon as is with the default values, and I think I am using a valid MapQuest key. However, the code exists with the following error:
http.client.InvalidURL: URL can't contain control characters. '/elevation/v1/profile?key=xtiAnYRiTRqAqfKIjzi4c1AZi6V6fZdv\n&shapeFormat=cmp6&latLngCollection=sulgxA_edwQ' (found at least '\n')
Any ideas what the problem is, and how this can be fixed?

It seems there was a newline in the API key file. Now it worked


collection parametrize

now, for save all models in collection i use array. But for improved performance possible add variant for use any data structure: RBTree, HashTable, or any other. For this, need have interface with base methods: first, takeFirst, collect, map, push, remove...
And use as

MyCollectionWithRedBlackTree = Sirius.Collection.parametrize_for(RedBlackInterface)

my_collection = new MyCollectionWithRedBlackTree(MyModel)

or just add index for collections

closed. see https://github.com/fntzr/sirius/issues/23


GoDaddy DNS support

Could this be a way to support GoDaddy DNS?
It gets a bit tricky due to GoDaddy's lack of a delete API, but I believe that can be accommodated.

I'm assuming you mean support for the DNS challenge when your DNS provider is GoDaddy.
I'm not using certbot, but it seems the underlying library I'm using does have GoDaddy support (GoDaddyDnsProvider): https://github.com/sjkp/letsencrypt-azure/issues/2 - I see you spotted that issue as well.
So it should be possible, but I don't have any concrete plans to implement it any time soon...

I don't have any concrete plans to implement it any time soon

I understand.
The solution provided in that issue seems to require a rebuild; it'd be nice to be able to effect a change at runtime, thereby permitting use of the official codebase.

I'm pretty sure it can be implemented without requiring a rebuild.
One of the reasons I don't have plans to add GoDaddy DNS is that I don't have GoDaddy DNS to test it.
But if someone were to submit a PR that could work :)
I'm reopening this because it's definitely a legitimate feature request...

I'm not at all handy with C#. Otherwise, I'd consider giving it a run.
Testing suggestion: You could register a test domain at GD and point its zone to your Azure stuff. You'd be out twenty bux, though, so that may not be desirable. (Maybe you've already got a spare that you're not using that you could transfer over.)

Follow-up: If you decide to look into this at some point, I might be able to help by retargeting one (or more) of my parked domains for your needs.
I suppose I could even add you as a DNS admin.

I admit this will be a low priority, but sure if you make me an admin it might speed things up when relevant...

Perfect. Just let me know, should ever the time come.

Also needed it for our need so created a PR for it.
#120


Custom login, lack of scrollbar when there are too many configuration fields on the right of the registration page

自定义登录，注册页右侧配置字段过多时，缺少滚动条，添加的字段显示不全

#955


Add rake task for updgrading JB versions

There is no obvious way to upgrade Jekyll-Bootstrap versions.

What's the non-obvious way?

You can add jekyll-bootstrap as a remote to your repository, fetch it, then compare it to your HEAD. This is better outlined here: https://github.com/plusjade/jekyll-bootstrap/pull/35
There are 4 main integration points between Jekyll-Bootstrap and Jekyll.

_config.yml
There are still some variables in use that are not namespaced into JB hash. But ideally we'll place every variable in JB and then find a clean way to merge the file without reorganizing the user data.
_includes/JB
These files are properly namespaced. You should always override the methods as outlined here: http://jekyllbootstrap.com/api/bootstrap-api.html#method_overrides  rather than edit the files directly so this should be easily upgradable.
Rakefile
The Rakefile can be upgraded cleanly, provided we add this feature in: https://github.com/plusjade/jekyll-bootstrap/pull/33 which allows users to extend the rakefile freely with their own tasks.
Themes
Themes should be namespaced and versioned independently from the JB core so with the exception of a major version release the theme API should not be affected at all when updating.

Are there any more areas I've missed? Ideally we need to fully automate this task. However it's better to not do anything than to risk overwriting user data. So I want to make sure to avoid any such possibility.
I would like to hear your thoughts on this, thanks!

The upgrade guide from #35 appears to have gone missing.


Manage Xcode project with XcodeGen:

Remove project from Git
Update gitignore
Add project.yaml for specifying the project
Add Make command

This change allows us to maintain custom build phases (swiftlint) until Swift PM supports it.

@maicki - Agreed, it'll be nice if there is a way to do this through Swift PM. I'll abandon in favor of not adding more dependencies to the project

@rahul-malik Thanks for looking into that though. I would love if Swift PM would be able to handle this kind of things in the near future.


Outdated dependencies reported during install

🐛 Bug Report
I attempted to install meta on an Ubuntu 22.04.4 LTS Linux machine using the command given.  The following output was generated.
npm WARN deprecated<EMAIL_ADDRESS>This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported
npm WARN deprecated<EMAIL_ADDRESS>Rimraf versions prior to v4 are no longer supported

added 134 packages, and audited 135 packages in 10s

6 packages are looking for funding
  run `npm fund` for details

1 critical severity vulnerability

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.

To Reproduce
Execute npm i meta -g
Expected behavior
No warnings about critical severity vulnerabilities reported.

Accepting pull requests. Vulnerabilities are low risk for a passage that is
being run locally.
On Tue, Jun 4, 2024 at 2:34 AM agrimstrup @.***> wrote:

🐛 Bug Report
I attempted to install meta on an Ubuntu 22.04.4 LTS Linux machine using
the command given. The following output was generated.
npm WARN deprecated @.: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated @.: Glob versions prior to v9 are no longer supported
npm WARN deprecated @.***: Rimraf versions prior to v4 are no longer supported
added 134 packages, and audited 135 packages in 10s
6 packages are looking for funding
run npm fund for details
1 critical severity vulnerability
Some issues need review, and may require choosing
a different dependency.
Run npm audit for details.
To Reproduce
Execute npm i meta -g
Expected behavior
No warnings about critical severity vulnerabilities reported.
—
Reply to this email directly, view it on GitHub
https://github.com/mateodelnorte/meta/issues/336, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAEHOXYXZSAYNGEN3UIL5I3ZFVUXJAVCNFSM6AAAAABIX5RR4GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZTENZXGYYTONY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


fix(date-picker): Fix cupertino date picker minuteInterval

The Cupertino date picker does not work correctly when using minuteInterval and scrolling past the endDate
List which issues are fixed by this PR. You must list at least one issue.
Issue 97010
If you had to change anything in the flutter/tests repo, include a link to the migration guide as per the breaking change policy.
Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x]  I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///).
test-exempt I added new tests to check the change I am making, or this PR is test-exempt.
test-exempt All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Hi, @ansolantz
You should write tests code or request to be approved.
Please follow link below: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests

Hi, @ansolantz You should write tests code or request to be approved. Please follow link below: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests

Hi @SuhwanCha,
I honestly do not know how to write a test case for this, and since the solution is pretty straightforward I prefer to wait for the ticket to be answered.

@ansolantz that fix is probably correct (i honestly don't know the code well enough to say for sure) but @SuhwanCha is right, before we can review it it really needs a test. You can find the other datepicker tests here: https://github.com/flutter/flutter/blob/master/packages/flutter/test/cupertino/date_picker_test.dart

Thanks for the contribution! I'm going to close this PR since it needs tests, as per the comments above. If anyone wants to take over please don't hesitate to take this code and submit a new PR!


Fixes issue matching constructor casing with field casing

Fixes issue matching constructor casing with field casing by querying all fields for any match regardless of casing.
Closes #4811

@michaelstaib Did you mean to assign the ticket to me rather than the PR?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: michaelstaib:x: da1rrenYou have signed the CLA already but the status is still pending? Let us recheck it.

@da1rren you still need to sign the CLA so we can finally merge this.

@michaelstaib Signed & conflicts resolved.

I've closed this as I'm not sure its the right approach.


Update schema as per last PR Review and add architecture.md

I have implemented the aspects discussed in the last PR to start. Answering question 9 within the README.md would also help shine some light on certain aspects of creating the schema.
Please ignore the first commit, and look at the latest commit (from today) for this PR. The future-schema.graphql is the eventual schema layout for the subgraph, whereas the schema.graphql within the root of the merge-v1 directory is the working schema as discussed.
I am not sure if the schema.graphql within the build subdirectory is autogenerated, but I did not change that file, I have just edited the one within the root of the merge-v1 directory.
You can ignore the architecture.md file I have created also as discussed. The context on it is below in italics for future reference.
Thanks!
--
Extra info that you do not to read, but is here for reference:
I created architecture.md file because it crossed my mind that this schema is a good exercise in thinking of a standard NFT schema and subgraph format for most NFT projects today. I figured it would be good to get practise making at least one more NFT subgraph too afterwards and see if the schema layout I had made would work well or not.

there is no question 9!

Currently getting this compile error (see screenshot at bottom of this comment) when running yarn codgen for merge subgraph. This is after changing the code within the massNFTs field code so that it resolved the previous compile error Field 'massNFTs': Value of the @derivedFrom 'field' argument must be a string
Resolving code:
"Details about NFTs being held by this User"
  massNFTs: [NFTs!]! @derivedFrom(field: "owner") #pak: allows one to query info about NFTs belonging to this user

Not sure why I am getting this as it is starting to confuse my understanding of using derivedFrom in reverselookups. It is discussed a little bit within this git issue but it is unresolved currently.
Error I am getting right now:

@davekaj if you could provide your insight on this when you have a moment that would be greatly appreciated. Going to merge this PR once I can resolve these compile errors.
Currently getting this compile error (see screenshot at bottom of this comment) when running yarn codgen for merge subgraph. This is after changing the code within the massNFTs field code so that it resolved the previous compile error Field 'massNFTs': Value of the @derivedFrom 'field' argument must be a string
Resolving code:
"Details about NFTs being held by this User"
  massNFTs: [NFTs!]! @derivedFrom(field: "owner") #pak: allows one to query info about NFTs belonging to this user

Not sure why I am getting this as it is starting to confuse my understanding of using derivedFrom in reverselookups. It is discussed a little bit within this git issue but it is unresolved currently.
Error I am getting right now:

This was resolved after talking with Dave more offline. This is Reverse Lookups#
I completely didn't see that token: Token! was pointing at the Token entity within the example there. So to resolve the issue here I had to do something similar with the respective fields.


Update docs to use K8s manifest generator

@abuehrle this is for your branch 👍

@errordeveloper pls target site/installing.md on the docs branch, not the readme.  The readme is about to be massively reduced in size.

@tomwilkie I've included @errordeveloper changes.  Let me push the updates.

Ok, this looks odd now. This mean to be merged to docs-reorg branch and appear in #1416.

@errordeveloper I added this section to "Install Scope on Your Cluster". It couldn't be merged exactly as you wrote it, so please make sure that I haven't missed anything (there's a preview copy on the dev server as well, which I can't post here, but choose the 'latest' tag).  I would just close this branch.

@abuehrle, I will take a look at what's on the dev site. I believe this PR should be closed now, or what do you want me to do, it has dozens of commits that do not relate to the title of the PR, and it was made to your branch...

I'll make PR agains master once all docs rework had been merged.


Build failed v0.2.4

I upgraded my flutter today and started getting this error with your package.
/home/david/.pub-cache/hosted/pub.dartlang.org/flutter_billing-
0.2.4/android/src/main/java/com/lykhonis/flutterbilling/BillingPlugin.java:163: error: ';' expected
    }
     ^
1 error

FAILURE: Build failed with an exception.

Also, if I download your project and load into android studio I get another error :
The current Dart SDK version is 2.1.0-dev.3.0.flutter-760a9690c2.

Because flutter_billing_example depends on synchronized >=1.0.0 <1.5.1 which requires SDK version >=1.9.1 <2.0.0, version solving failed.
pub get failed (1)

[✓] Flutter (Channel master, v0.7.6-pre.11, on Linux, locale en_CA.UTF-8)
• Flutter version 0.7.6-pre.11 at /work/installs/flutter
• Framework revision 251e82d211 (32 minutes ago), 2018-08-31 13:31:56 -0700
• Engine revision f612a761c2
• Dart version 2.1.0-dev.3.0.flutter-760a9690c2
[✓] Android toolchain - develop for Android devices (Android SDK 27.0.3)
• Android SDK at /home/david/Android/Sdk
• Android NDK location not configured (optional; useful for native profiling support)
• Platform android-27, build-tools 27.0.3
• ANDROID_HOME = /home/david/Android/Sdk
• Java binary at: /work/installs/android-studio-3.0/jre/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
• All Android licenses accepted.
[✓] Android Studio (version 3.1)
• Android Studio at /work/installs/android-studio-3.0
• Flutter plugin version 27.1.1
• Dart plugin version 173.4700
• Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
[!] IntelliJ IDEA Community Edition (version 2018.1)
• IntelliJ at /work/installs/idea-IC-181.4445.78
✗ Flutter plugin not installed; this adds Flutter specific functionality.
• Dart plugin version 181.4892.1
• For information about installing plugins, see
https://flutter.io/intellij-setup/#installing-the-plugins
[✓] Connected devices (1 available)
• Android SDK built for x86 • emulator-5554 • android-x86 • Android 5.1.1 (API 22) (emulator)
! Doctor found issues in 1 category.

Will be fixed in v0.2.5


Update publications from Giorgio Mariani

In this commits, I update and improve my publications, by adding a featured image (size: 640x480) and the GitHub repository link. I also made a minor change to my bio.

👷 Deploy Preview for gladia processing.

Name
Link

🔨 Latest commit
d2973b9aa4c386562c2096ede752700aaa29fb0c

🔍 Latest deploy log
https://app.netlify.com/sites/gladia/deploys/63f4b3e2e335da000890906c


Fix scan for stacktrace usage (bug never released)

Fix scan for stacktrace usage. Also recurse into chain.
Added missing unit test for this.
As discussed at
https://github.com/NLog/NLog/pull/1310#issuecomment-199523087
Thanks @Taloth!

Current coverage is 75.01%

Merging #1316 into master will increase coverage by +0.01% as of 4875292

@@            master   #1316   diff @@
======================================
  Files          267     267
  Stmts        15458   15467     +9
  Branches      1622    1626     +4
  Methods          0       0
======================================
+ Hit          11594   11602     +8
- Partial        382     383     +1
  Missed        3482    3482

Review entire Coverage Diff as of 4875292

Uncovered Suggestions

+0.09% via ...gingConfiguration.cs#282...295
+0.09% via ...gingConfiguration.cs#248...261
+0.08% via ...argets/FileTarget.cs#1713...1725
See 7 more...

Powered by Codecov. Updated on successful CI builds.


An error in an example about the Filtered and Sorted

In https://github.com/vuejs/vuejs.org/blob/master/src/guide/list.md#displaying-filteredsorted-results
the last example isn't working. Showed me this message: 'Uncaught TypeError: numbers.filter is not a function'. You can test here: https://jsfiddle.net/9qnugcfc/

You forgot to change computed to methods in your fiddle. The actual example works:
https://jsfiddle.net/chrisvfritz/nfym3byf/


How to get bands from custom area.

I'm trying to get S2-L1C bands for custom area with slightly changed code from examples directory.
input_task = S2L1CWCSInput('TRUE-COLOR-S2-L1C', resx='10m', resy='10m', maxcc=0.8, instance_id=INSTANCE_ID)
workflow = LinearWorkflow(input_task)
time_interval = ['2017-01-01','2017-12-31']
from sentinelhub import WebFeatureService, BBox, CRS, DataSource
bbox = BBox(bbox=[<PHONE_NUMBER>,-32.4219569,150.6583071,-32.4383657], crs=CRS.WGS84)

results = workflow.execute({input_task:{'bbox':bbox, 'time_interval':time_interval}})

but got an Error:
Server response: "Layer TRUE-COLOR-S2-L1C not found"
Is there any way to do?

Hi @GalacticF!
Is the TRUE-COLOR-S2-L1C layer defined in your configuration utility panel?
Cheers

You get this error because your Sentinel Hub instance does not have a layer with this specific ID.
In case you are working with default (exemplary) instance, you can try with layer ID named "TRUE_COLOR".
To work efficiently with sentinelhub-py, we recommend to create a new instance using "Python scripts template":

Go to Configuration utility (https://apps.sentinel-hub.com/configurator/)
Click "Add new configuration"
Choose "Python scripts template"
Use new instance's ID in the scirpt.

@gmilcinski It works, thank you!


Add gl-enum-value lisp type

(deftype gl-enum-value () '(signed-byte 32)) then go update everything that uses the old style

Easy win. Should do this. Glenum is typedefd from unsigned it, we can always use cffi to guarantee size too.

done


Drop Haskell Compatibility for .eta but retain it only for .hs

Since one goal is:

accessibility for beginners from imperative languages, especially Java

It might be more flexible to drop the compatibility for ETA language (.eta files) but retain it for .hs files and even detect and support many Haskell version. The implication of this is at the start ETA infrastructure can understand the ETA language itself, Java ByteCode and GHC 7.10.3's Haskell, but can later extended. (Currently ETA infrastructure can understand the Java ByteCode and GHC 7.10.3's Haskell as there is no ETA language.)
Dropping the compatibility for eta files means this can evolve independently to pursue the above goal better than also dragging the technical debt of compatibility. This can be addressed separately through https://github.com/typelead/eta/issues/117 in a more broader context which can include "compatibility with Haskell, OCaml, Java, C#, F#, C/C++, R, Python, Scala, etc."

I intended that .hs files will always be recognised and only support the features of GHC Haskell 7.10.3 and before. Any .eta files will have full support of the Eta language. Is this what you meant?

To start with. Depending on ETA it might be easy to map even 8.0.1 to ETA hence support different version. But having said that I feel it is better to design ETA independently and from scratch than base it on Haskell and have Haskell also translate into ETA starting with GHC Haskell 7.10.3 and later other versions.

Basing it on GHC 7.10.3 was to have a starting point so that it's easy to get something production-ready sooner. Another reason was to get access to Hackage so I can solve the libraries problem early on. It's a lot of baggage to carry and I'll probably feel the weight once the language evolves in drastic ways. Making it in an independent language altogether I can only see happening if I happen to acquire sufficient resources and there's a dire need for it. I'm not aiming for a perfect solution from day one. Just one that works and keeps getting better over time.

Initially keep GHC 7.10.3 and ETA experimental for a while and some point when ETA is somewhat mature transpile GHC 7.10.3 to ETA but from the onset designing ETA will be able to address some of the more pressing needs in typed programming not envisaged during the design or Haskell. Also haskell is hard for anybody from an imperative background.
Perhaps I can open another issue from what is best expected from a language designed today without any previous technical debt.
My feeling is this is the best way to go. You get your MVP out using Haskell and then to ETA with Haskell -> ETA so any investment in Haskell does not break.

...some of the more pressing needs in typed programming not envisaged during the design or Haskell.

Can you expand on this? I don't want to venture too far off since I'm trying to attract people who have already been sold on Haskell/Typed FP and want something that is geared towards industry since the alternatives are quite limited in their abilities to apply purely function techniques.

Perhaps I can open another issue from what is best expected from a language designed today without any previous technical debt.

That would be great!

I will put this together and open another issue.
There is Haskell, OCaml, F#, ML, Scala, etc. the best way to package ETA would be best and cream or FP than just geared to one language to attract users from all languages. There are things you can borrow from OCaml but in which case you cannot have change ETA.

I'm open to taking ideas from other languages, but for the most part I'm sticking with Haskell as it is fixing the warts as much as possible. I don't want to make Eta complex with too many features either. If the features from the other language can't be implemented in Eta, or is very useful and should have special compiler/runtime support I can think about it.

What I am going to propose is: making the language simple but like lego brings which can com together to build something bigger. Also make it accessible from people from C, Java, C# background. Also easier for the programmer. Also support Haskell by translating it into ETA and not by makes ETA Haskell++--. Wait until I start another issue with more information.

Closing as this is a major language change, this should follow the Eta Proposal Process:
https://github.com/typelead/eta-proposals


Dynamic Credentials (AWS) Not working on terrakube 2.22

Feedback
Hey guys, i am facing an issue while setting up my terrakube environment.
I was trying to set up Dynamic credentials, did everything told on Dynamic credentials Documentation.
I created the Public/private keys, on docker compose copied it to the api container, and then assigned its two new variables with the keys path inside the api container:
,
When the api containers were deployed, i logged into its shell, and checked the folders, and they were there.
Then i went to "terrakube-api.mydomain.com/.well-known/jwks  and the other path: terrakube-api.mydomain.com/.well-known/openid-configuration", and both gave the expected responses, as told on the docs.
I ran the Dynamic credentials terraform, creating the policy, role and oidc config on my aws account.
The only thing i changed was the name of the workspace i was working with.
Here is my trust relationship: (replaced real info for privacy reasons)
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::1234567890:oidc-provider/terrakube-api.mydomain.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "terrakube-api.mydomain.com:sub": "organization:company-develop:workspace:iac-terrakube-teste",
                    "terrakube-api.mydomain.com:aud": "aws.workload.identity"
                }
            }
        }
    ]
}

My original intention was to create the AWS envs on the global vars, but docs were doing it on workspace, so i did the same.
I ran a new plan to create a S3 bucket, and got this error. what am i doing wrong here?

Planning failed. Terraform encountered an error while generating this plan.
���
��� Error: No valid credential sources found
���
���   with provider["registry.terraform.io/hashicorp/aws"],
���   on main.tf line 1, in provider "aws":
���    1: provider "aws" {
���
��� Please see https://registry.terraform.io/providers/hashicorp/aws
��� for more information about providing credentials.
���
��� Error: failed to refresh cached credentials, failed to retrieve
��� credentials, operation error STS: AssumeRoleWithWebIdentity, https response
��� error StatusCode: 400, RequestID: (TheRequestIDgoesHere, I removed it), api
��� error ValidationError: 1 validation error detected: Value at
��� 'webIdentityToken' failed to satisfy constraint: Member must have length
��� greater than or equal to 4

I'm trying to solve this problem for days, and i still have no clue about how to solve it.
OBS:
My Terrakube is the docker compose version 2.22. ( I'm using terrakube on docker in an EC2 instance because my intention is to use terrakube to manage my EKS clusters.)
My EC2 instance with terrakube on it has a public IP, but its inbound access is filtere, allowing only bitbucket webhooks and a few other internal services.

Did you check if your PEM file is in "pkcs8" format???

Did you check if your PEM file is in "pkcs8" format???

Yes, i did, i took a first look when it was generated and it looked fine
After i got the error, i tried used the command on the docs to convert the key to the pkcs8 format
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in private_temp.pem -out private.pem,
but looks like it still wasnt the root cause of the error

You can enable debugging for dynamic credentials in the API container using the following env variable, so you can see the internal values inside the logs
JAVA_TOOL_OPTIONS="-Dlogging.level.org.terrakube.api.plugin.token.dynamic=DEBUG -Dlogging.level.org.terrakube.api.plugin.scheduler.job.tcl.executor=DEBUG"

allright, im going to put these debug args and try again

Using that you should be able to see the values from this part of the code
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L101
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L103
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L104
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L105
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ExecutorService.java#L194

Using that you should be able to see the values from this part of the code
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L101
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L103
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L104
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L105
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/scheduler/job/tcl/executor/ExecutorService.java#L194
That can help you to validate if everything is working correctly

I got the logs with the debug output. I noticed that the accessToken item is with **** value. Is it normal?
2024-10-17T16:56:01.626Z DEBUG 1 --- [ryBean_Worker-4] o.t.a.p.s.j.t.executor.ExecutorService   : Sending Job: /n ExecutorContext(commandList=null, type=terraformPlan, organizationId=281f4a85-5974-43da-bc1c-4890feeabbd0, workspaceId=fb5d00fb-e9c9-4cda-abdf-ad46ed754cb8, jobId=10, stepId=3646c137-ebaf-440b-87dc-b1f73b95a88e, terraformVersion=1.9.7, source=https://bitbucket.org/company/iac-terrakube-teste, branch=develop, folder=/, vcsType=BITBUCKET, refresh=true, refreshOnly=false, showHeader=true, accessToken=****, moduleSshKey=****, commitId=null, tofu=false, agentUrl=https://terrakube-executor.mydomain.com/api/v1/terraform-rs, environmentVariables={ENABLE_DYNAMIC_CREDENTIALS_AWS=true, AWS_REGION=us-east-1, organizationName=company-develop, WORKLOAD_IDENTITY_ROLE_AWS=arn:aws:iam::1234567890:role/terrakube-roleeft, TF_IN_AUTOMATION=1, AWS_WEB_IDENTITY_TOKEN_FILE=/home/cnb/.terraform-spring-boot/executor/281f4a85-5974-43da-bc1c-4890feeabbd0/fb5d00fb-e9c9-4cda-abdf-ad46ed754cb8/terrakube_config_dynamic_credentials_aws.txt, AWS_ROLE_ARN=arn:aws:iam::1234567890:role/terrakube-roleeft, workspaceName=iac-terrakube-teste, WORKLOAD_IDENTITY_AUDIENCE_AWS=aws.workload.identity, TERRAKUBE_AWS_CREDENTIALS_FILE=}, variables={})
AWS_WEB_IDENTITY_TOKEN_FILE returns a path, that i tried to follow when opened the api and  after the executor shell. Found the path but not the file inside it
TERRAKUBE_AWS_CREDENTIALS_FILE returns empty.

Check for this specific line log.debug("TERRAKUBE_AWS_CREDENTIALS_FILE: {}", awsWebIdentityToken);, it will show the token that will be used inside the logs you should see a base64 enconded json web token
public HashMap<String, String> generateDynamicCredentialsAws(Job job, HashMap<String, String> workspaceEnvVariables) {
        String awsWebIdentityToken = generateJwt(
                job.getOrganization().getName(),
                job.getWorkspace().getName(),
                workspaceEnvVariables.get("WORKLOAD_IDENTITY_AUDIENCE_AWS"),
                job.getOrganization().getId().toString(),
                job.getWorkspace().getId().toString(),
                job.getId()
        );

        log.debug("TERRAKUBE_AWS_CREDENTIALS_FILE: {}", awsWebIdentityToken);

        workspaceEnvVariables.put("TERRAKUBE_AWS_CREDENTIALS_FILE", awsWebIdentityToken);
        workspaceEnvVariables.put("AWS_ROLE_ARN", workspaceEnvVariables.get("WORKLOAD_IDENTITY_ROLE_AWS"));
        workspaceEnvVariables.put("AWS_WEB_IDENTITY_TOKEN_FILE", getDefaultExecutorPath(job) + "/terrakube_config_dynamic_credentials_aws.txt");

        return workspaceEnvVariables;
    }

If you dont see the token I guess it is not generated correctly here:
https://github.com/AzBuilder/terrakube/blob/b46286656a1fbd0fb4e7176e723c15605ebbb6bd/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java#L61

If you see from your logs I think the value is empty

If you see from your logs I think the value is empty

I guess in your logs you should see some error message

i found an error message on the private key thing, and the credential file variable empty

looks like theres still something wrong with the key, idk if its permissions (i pasted the .pem on the /etc/ folder), the key itself look normal

i found some errors, but nothing related to dynamic credentials, looks like something related to not find something the s3 state bucket (it is being authenticated with its own secret key. + id), and is already writing to bucket.

i found some other errors, but nothing related to dynamic credentials, looks like something related to not find something the s3 state bucket (it is being authenticated with its own secret key. + id), and is already writing to bucket.

You dont need to worry for those

i'm going to move the .pem and put it in another place, to see if i get a different output

Maybe it is a permission issue, this app is running using the following user.
uid=1002(cnb) gid=1000(cnb) groups=1000(cnb)

Yeah, may be it. im going to try applying a chmod + chown here

and yes, permissions were exactly the problem. idk why i thought putting the keys on /etc/ would be a good idea haha
they're now on /workspace folder with the permissions you told me above.
The debug output now gives me a value for TERRAKUBE_AWS_CREDENTIALS_FILE.
When i run a plan, it still gives me an error, but it may be something from my account blocking something for the amount of requests.  I'll try recreating the oidc config and roles to see if it solves the problem.
��� Error: No valid credential sources found ���  ���   with provider["registry.terraform.io/hashicorp/aws"], ���   on main.tf line 1, in provider "aws": ���    1: provider "aws" { ���  ��� Please see https://registry.terraform.io/providers/hashicorp/aws ��� for more information about providing credentials. ���  ��� Error: failed to refresh cached credentials, failed to retrieve ��� credentials, operation error STS: AssumeRoleWithWebIdentity, exceeded ��� maximum number of attempts, 3, https response error StatusCode: 400, ��� RequestID: (RequestIDHere-I-Removed-It), InvalidIdentityToken: ��� Couldn't retrieve verification key from your identity provider,  please ��� reference AssumeRoleWithWebIdentity documentation for requirements

You need to check this endpoint, AWS should be able to connect to those without any kind of restriction
https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/jwks
https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/openid-configuration

@isbig5 If you allow only those 2 endpoints it should work

@isbig5 If you allow only those 2 endpoints it should work

You can check that here

Thanks! My tk is finally working. I ended up using Cloudfront + and some  WAF rules to restrict access. limited the inbound to receive only traffic from cloudfront prefix list. Unfortunately looks like the IAM Identity provider doesnt have a specific range of IPs, and runs on EC2 range. And allowing the whole ec2 range would be too risky for me.
Another question: Its possible to create the terrakube trust policy for all workspaces on the organization, and not only one?
I would like to use this role on the global variables for all my organization.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::1234567890:oidc-provider/terrakube-api.mydomain.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "terrakube-api.mydomain.com:sub": "organization:company-develop:workspace:iac-terrakube-teste",
                    "terrakube-api.mydomain.com:aud": "aws.workload.identity"
                }
            }
        }
    ]
}

Can i remove this "workspace:iac-terrakube-teste" and allow all? If yes, how?

Hello @isbig5 I guess it should work if you remove the workspace:iac-terrakube-test you should be able to use that for any workspace inside the organization.

I am not an AWS expert just to clarify XD I just implemented the dynamic credentials following this documentation

In the end AWS will check the subject of the JWT that is generated by terrakube in this part of the code and validate the signature using the enpoint /.well-known/jwks
log.debug("TERRAKUBE_AWS_CREDENTIALS_FILE: {}", awsWebIdentityToken);

You could decode the token to check the content of the JWT token to see all the values

The documentation mentioned something related to the trust policy here


How capture slow-mo with PBJVision ?

How capture slow-mo with PBJVision ?

any info?

Need help for capturing slow-motion video using PBJVision can I have a sample code for that. because I didn't get anything. I'm set video framerate 120fps.
videoFrameRate = 120
_currentDevice.activeVideoMinFrameDuration = fps;
_currentDevice.activeVideoMaxFrameDuration = fps;
when I save the video and play it normally.
Can you help me @piemonte to send sample demo for slow-motion video.
FYI : my device is supported 120 and 240 fps.
Thanks in advance.


session lost after bluebird.delay

I'm trying to integrate cls into an existing project running both async/await and bluebird. Everything works terrific, until bluebird.delay is called, at which point the session gets lost. Example code:
var Promise = require('bluebird');
var cls = require('continuation-local-storage');
var session = cls.createNamespace('blah');

session.run(async () => {
    session.set('one', 1);
    await Promise.resolve().delay(1);
    console.log(`========== one: ${cls.getNamespace('blah').get('one')} ==========`);
})

Help would be greatly appreciated

replace
var cls = require('continuation-local-storage');
with
var cls = require('cls-hooked');

Thank you so much @carlisliu, I had the same problem with TypeORM 👍


Add a correct index offset in apply_copy and apply_skip

This change fixes the crash of "edit_cargo_toml" benchmark in the hashmap branch.
This fn was crashing because, as of the original Vec implementation, removing a range [0..nb_lines] from the vector works as expected (=the first nb_lines in the vector are removed) but in case of HashMap<u64, Line>, there is no ordered array of lines, and the hash's keys are "some_index, some_index+1, ..., etc" (going in unspecified order) rather than "0, 1, ...". Thus, one should not index into HashMap with something like [0..nb_lines].iter.map(|i| old_lines.remove(&i).
The correct action is removal by keys min_index, min_index+1, ..., min_index+nb_lines.

Nice, thanks!
Seems like it isn't quite right still though, if I open a document (e.g. the shortcut_win.glade of Tau) in Tau, the linecache crashes like this after a few ops:
[2019-10-17T15:46:56Z ERROR xrl::cache] 46 lines left to copy, but only 15 lines in the old cache
thread '<unnamed>' panicked at 'cache update failed', /home/rasmus/Projects/xrl/src/cache.rs:206:13
stack backtrace:

Anyway, good work and thank you very much. Do you want to take a look at that?

Thanks :)

OK, found it. Quite expected, I missed a couple more similar issues with index ranges.
Tried xrl with my today's update on xi-term and it views/edits files fine now.
Can you please try this xrl fix on tau? I'm having a hard time setting up the tau dependencies.
Note, I expect the same problems <similar to what I just fixed> on the xrl clients' side. Switching from Vec<Line> to HashMap<Line> changes the meaning/behavior of some API, eg iter() (which now iterates in an unspecified order), extend(), remove(), etc. So the xrl users should not simply update the signatures of LineCache methods in the client code, but also probably change the client behavior. See my xi-term branch for the hashmap version of xrl: c4eater/xi-term@7f1db97.
BTW hashbrown::HashMap became the part of the standard library as std::collections::HashMap as of Rust 1.36, so you can probably drop this dependency from xrl.

BTW hashbrown::HashMap became the part of the standard library as std::collections::HashMap as of Rust 1.36, so you can probably drop this dependency from xrl.

Yes, but I had some panics with std's HashMap which I didn't have with hashbrown, but maybe that's fixed now :)

So the xrl users should not simply update the signatures of LineCache methods in the client code, but also probably change the client behavior.

Ah, Tau doesn't rely on that (it just iterates on first_line to last_line (which is in the view) and calls get_line for those), so I hadn't thought of that.
Will try with Tau, thanks for the help!

Sadly it still seems to be a bit broken when scroll up&down and panics after a bit with: 96 lines left to copy, but only 47 lines in the old cache :/

Good, I managed to crash xrl with a synthetic test that simulates repeated up/down scrolls.
Another indexing issue in apply_copy. >_<
I will also try playing with a live tau install today. (Will drop a comment if I manage to crash it)

Great, thank you :)

Just built tau and tried opening some random files (syntect enabled). Didn't see any xrl crash during scrolling.
That doesn't really prove anything, because for some reason tau +xrl from yesterday didn't crash for me either.
(That's probably toolchain/hardware specific. I've seen this before. xrl::LineCache in xi-term already needs different conditions on my 2 different machines in order to crash.)
PS: The only tau crash I encountered was when I made a few rapid 2-finger scrolls up/down and then clicked in the middle of the file. Tau panicked with an int overflow at edit_view.rs:766. Most likely this is a different problem.

Hm, it doesn't crash for me anymore either, but seems like the lines get a bit messed up after scrolling anyway :/
Thanks for the info about the other panic, will look into that :)

You mean, there is a small line offset after scrolling?
Whoops, I think I know where made a mistake.
Just added another commit, did it fix this offset issue?

Hmm, it doesn't crash, but seems like lines are duplicated sometimes?

Can't reproduce this yet (xi-term and tau), but I'm wondering whether I am using a correct tau branch.
From what I see, you tau "master" branch is using its own linecache not xrl::LineCache. I was probably using the stock tau and linecache for the whole time. Do you have a tau branch which exploits xrl LineCache?

Oh, I don't have an extra branch for that, I just changed that locally. I'll make a branch for you, wait a sec

I've made a xrl branch which just pulls from your git. You might want to change xrl to a path dependency to test your local changes in src/editview/Cargo.toml and src/tau/Cargo.toml.

A couple days ago, I got back to working on this branch (attempting to rebase it onto master), for I thought that the recent fix of xi-editor/xi-editor#1301 eliminates the remaining problem <in HashMap based linecache> with the invalid line indexing.
Shortly after that, I found a fundamental problem with using HashMap in place of Vec for line cache representation. The problem is: when line wrapping is set to true, some lines in UpdateOps will have ln == None, because a logical line (an entity spanning from ^ to $), when wrapped, is represented in xi-core as an array of Lines (= visual lines) where only the 1st Line has "ln" set. (See the frontend protocol spec.) In a Vec-based cache, it is easy to re-assemble the array slice of visual Lines back into a logical line: a logical line is just a concatenation of Lines between Line(ln=given #) and Line(ln=given # plus 1) in the global array of lines. In a HashMap, which is un-ordered, once a "ln"-less Line gets into the hash map, its line number information is completely lost. Thus, it is logically impossible to execute UpdateOps like eg. "Copy 2 lines" if the 1st logical line is wrapped: the cache will have a lot of Lines with ln=None as candidates for the 2nd Line, with no way to identify the correct one. Calling something like old_lines.take(2) will only yield 2 random lines as HashMap provides no guarantees on the traversal order. Identifying a line number by hash map key is also not possible.
This probably can be fixed by using Vec<Line> (logical lines) as HashMap values or by switching to something ordered, like B-tree. But, given the obvious performance problems of HashMap<Line>  in addition to data representation problems, I'd rather stop working in this direction. Here is what "cargo bench" yields in xrl on my machine:
Vec based
load_cargo_toml: 14
edit_cargo_toml: 107
HashMap based (the line numbers are messed up in the current implementation but that is unimportant here)
load_cargo_toml: 29
edit_cargo_toml: 472
I am not surprised, given the fact that moving/deleting a slice is the most-used operation in xrl linecache, and this operation is very cheap in Vec but expensive in an unordered map.
Having that said, I will not make any future attempts to fix or rebase this branch, as I see HashMap as a dead end here. A line cache implementation that outperforms Vec< Line>  is possible, but to me, more prominent canditates are:

The classical Gap Buffer.

Xi-core's Rope (Node<Line> to be precise).

Directly reading xi-core's line cache and dropping the whole json-rpc communication layer (heard a talk in xi's zulip).


very extremely minor spelling mistake

On On is crazy

https://github.com/chrultrabook/docs/blob/4af1e7668864e39f9fb143838941e4ae1e192760/docs/supported-devices.md?plain=1#L7