A test with the files which actually collide

Attached is a zip file with two files which have the same sha1 but different sha256.
I'm trying to add the folder with them to the zpaqfranz archive, and I always see only one of them stored under the two names, not two of them:
C:\tmp\collision>unzip collision-example.zip
Archive:  collision-example.zip
   creating: baad/
 extracting: baad/messageA
 extracting: baad/messageB

C:\tmp\collision>openssl dgst -sha256 baad\*
SHA256(baad\messageA)= 3ead211681cec93d265c8ac123dd062e105408cebf82fa6e2b126f4f40bcb88c
SHA256(baad\messageB)= 208feafe1c6a95c73f662514ac48761f25e1f3b74922521a98d9ce287f4a2197

C:\tmp\collision>openssl dgst -sha1 baad\*
SHA1(baad\messageA)= 8ac60ba76f1999a1ab70223f225aefdc78d4ddc0
SHA1(baad\messageB)= 8ac60ba76f1999a1ab70223f225aefdc78d4ddc0

C:\tmp\collision>zpaqfranz.exe a baad.zpaqf baad
zpaqfranz v58.10o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-01)
franz:-noconsole
Creating baad.zpaqf at offset 0 + 0
Add 2023-10-02 03:39:47         2              1.280 (   1.25 KB) 8T (1 dirs)
3 +added, 0 -removed.

0 + (1.280 -> 640 -> 1.842) = 1.842 @ 26.60 KB/s

0.047 seconds (000:00:00) (all OK)

C:\tmp\collision>mkdir result

C:\tmp\collision>cd result

C:\tmp\collision\result>..\zpaqfranz.exe x ..\baad.zpaqf
zpaqfranz v58.10o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-01)
franz:-noconsole
../baad.zpaqf:
1 versions, 3 files, 1.842 bytes (1.80 KB)
Extract 1.280 bytes (1.25 KB) in 2 files (1 folders) / 8 T

0.031 seconds (000:00:00) (all OK)

C:\tmp\collision\result>openssl dgst -sha256 baad\*
SHA256(baad\messageA)= 3ead211681cec93d265c8ac123dd062e105408cebf82fa6e2b126f4f40bcb88c
SHA256(baad\messageB)= 3ead211681cec93d265c8ac123dd062e105408cebf82fa6e2b126f4f40bcb88c

Obviously one content under two names is there.
I've read "additional checks" are default, it's unexpected.  Maybe the switch is needed:
C:\tmp\collision\result>cd ..

C:\tmp\collision>zpaqfranz.exe a baad-2.zpaqf baad -sha256
zpaqfranz v58.10o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-01)
franz:-sha256 -noconsole
Creating baad-2.zpaqf at offset 0 + 0
Add 2023-10-02 03:41:54         2              1.280 (   1.25 KB) 8T (1 dirs)
3 +added, 0 -removed.

0 + (1.280 -> 640 -> 1.982) = 1.982 @ 26.60 KB/s

0.047 seconds (000:00:00) (all OK)

C:\tmp\collision>mkdir result-256

C:\tmp\collision>cd result-256

C:\tmp\collision\result-256>..\zpaqfranz.exe x ..\baad-2.zpaqf
zpaqfranz v58.10o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-01)
franz:-noconsole
../baad-2.zpaqf:
1 versions, 3 files, 1.982 bytes (1.94 KB)
Extract 1.280 bytes (1.25 KB) in 2 files (1 folders) / 8 T

0.032 seconds (000:00:00) (all OK)

C:\tmp\collision\result-256>openssl dgst -sha256 baad\*
SHA256(baad\messageA)= 3ead211681cec93d265c8ac123dd062e105408cebf82fa6e2b126f4f40bcb88c
SHA256(baad\messageB)= 3ead211681cec93d265c8ac123dd062e105408cebf82fa6e2b126f4f40bcb88c

collision-example.zip

This is expected behavior
There is no way to avoid SHA-1 collisions (other than breaking backwards compatibility of .zpaq archives, which I don't want to do)
zpaq simply doesn't detect anything, it doesn't provide any warnings.
Look at this example
Z:\pippo>zpaqfranz sum message* -sha256
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:sum                                       1 - command
franz:-sha256 -hw
Getting SHA-256 ignoring .zfs and :$DATA

No multithread: Found (1.25 KB) => 1.280 bytes (1.25 KB) / 2 files in 0.000000
|SHA-256: 208FEAFE1C6A95C73F662514AC48761F25E1F3B74922521A98D9CE287F4A2197 [                640]     |messageB
|SHA-256: 3EAD211681CEC93D265C8AC123DD062E105408CEBF82FA6E2B126F4F40BCB88C [                640]     |messageA

SHA-1 collision
Z:\pippo>zpaqfranz sum message* -sha1
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:sum                                       1 - command
franz:-sha1 -hw
Getting SHA-1 ignoring .zfs and :$DATA

No multithread: Found (1.25 KB) => 1.280 bytes (1.25 KB) / 2 files in 0.000000
|SHA-1: 8AC60BA76F1999A1AB70223F225AEFDC78D4DDC0 [                640]     |messageA
|SHA-1: 8AC60BA76F1999A1AB70223F225AEFDC78D4DDC0 [                640] === |messageB

Z:\pippo>zpaq64 a 715.zpaq *
zpaq v7.15 journaling archiver, compiled Aug 17 2016
Creating 715.zpaq at offset 0 + 0
Adding 0.001280 MB in 2 files -method 14 -threads 32 at 2023-10-02 08:28:34.
+ messageA 640
+ messageB 640 -> 0
[1..1] 652 -method 14,20,0
2 +added, 0 -removed.

0.000000 + (0.001280 -> 0.000640 -> 0.001739) = 0.001739 MB
0.016 seconds (all OK)

Z:\pippo>zpaq64 x 715.zpaq -test
zpaq v7.15 journaling archiver, compiled Aug 17 2016
715.zpaq: 1 versions, 2 files, 1 fragments, 0.001739 MB
Extracting 0.001280 MB in 2 files -threads 32
[1..1] -> 652
> messageA
> messageB
0.000 seconds (all OK)

Then
Z:\pippo>zpaq64 x 715.zpaq -to extracted
zpaq v7.15 journaling archiver, compiled Aug 17 2016
715.zpaq: 1 versions, 2 files, 1 fragments, 0.001739 MB
Extracting 0.001280 MB in 2 files -threads 32
[1..1] -> 652
> extracted/messageA
> extracted/messageB
0.016 seconds (all OK)

Finally
Z:\pippo>zpaqfranz sum extracted -sha256
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:sum                                       1 - command
franz:-sha256 -hw
Getting SHA-256 ignoring .zfs and :$DATA

No multithread: Found (1.25 KB) => 1.280 bytes (1.25 KB) / 2 files in 0.016000
|SHA-256: 3EAD211681CEC93D265C8AC123DD062E105408CEBF82FA6E2B126F4F40BCB88C [                640]     |extracted/messageA
|SHA-256: 3EAD211681CEC93D265C8AC123DD062E105408CEBF82FA6E2B126F4F40BCB88C [                640] === |extracted/messageB

0.079 seconds (000:00:00) (all OK)

As you can see, zpaq
zpaq happily extracts the data, without any warning that a collision has occurred.
Attention, we are talking about DETECTION, not AVOIDANCE (impossible to avoid with this archive file format)

Now let's see zpaqfranz default
Z:\pippo>zpaqfranz a zpaqfranz.zpaq messag*
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:-hw
Creating zpaqfranz.zpaq at offset 0 + 0
Add 2023-10-02 10:33:04         2              1.280 (   1.25 KB) 32T (0 dirs)
2 +added, 0 -removed.

0 + (1.280 -> 640 -> 1.803) = 1.803 @ 83.33 KB/s

0.031 seconds (000:00:00) (all OK)

This time, if you test the file, you'll get
Z:\pippo>zpaqfranz t zpaqfranz.zpaq
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:-hw
zpaqfranz.zpaq:
1 versions, 2 files, 1.803 bytes (1.76 KB)
To be checked 1.280 in 2 files (32 threads)
7.15 stage time       0.02 no error detected (RAM ~34.07 MB), try CRC-32 (if any)
Checking                 2 blocks with CRC-32 (1.280 not-0 bytes)
ERROR:  STORED CRC-32 92433266 != DECOMPRESSED 072E2B0E (ck 00000001) messageB

CRC-32 time           0.00s
Blocks               1.280 (           2)
Zeros                    0 (           0) 0.000000 s
Total                1.280 speed 1.280.000/sec (1.22 MB/s)
ERRORS        : 00000001 (ERROR in rebuilded CRC-32, SHA-1 collisions?)
-----------------------------------------------------------------------------------------------------
GOOD            : 00000001 of 00000002 (stored=decompressed)
WITH ERRORS

0.032 seconds (000:00:00) (with warnings)

If you put a "-verify" in add
Z:\pippo>zpaqfranz a test2.zpaq mess* -verify
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:-hw -verify
Creating test2.zpaq at offset 0 + 0
Add 2023-10-02 10:35:52         2              1.280 (   1.25 KB) 32T (0 dirs)
29604 SOMETHING WRONG ON messageA

GURU-C: on file messageA
GURU: CRC-32 from fragments 92433266
GURU: CRC-32 from file      072E2B0E
2 +added, 0 -removed.

0 + (1.280 -> 640 -> 1.794) = 1.794 @ 40.32 KB/s

Short version:
no way to DETECT a SHA-1 collision in zpaq
yes, you can DETECT SHA-1 collision in zpaqfranz

EDIT: for performance reasons, the double-check is NOT active during add() but in test(). Older zpaqfranz builds do this way, but it was a modified behavior so as not to slow down in the normal case

When I pack 3 files, messageA, messageB and some-big.mp4 then, to find the collision, if I invoke "t" the time needed will be proportional to the number of all the bytes in the archive:
Checking                68 blocks with CRC-32 (1.089.413.820 not-0 bytes)
ERROR:  STORED CRC-32 92433266 != DECOMPRESSED 072E2B0E (ck 00000001) messageB

CRC-32 time           0.01s
Blocks       1.089.413.820 (          68)
Total        1.089.413.820 speed <IP_ADDRESS>5/sec (31.71 GB/s)

...
ERRORS        : 00000001 (ERROR in rebuilded CRC-32, SHA-1 collisions?)
...
6.375 seconds (000:00:06) (with warnings)
But if I could just get a list of the filenames, the values of the stored crc32 and the values of the sha1 of each file, I could detect the collisions in the above example in a few milliseconds, i.e. not bound by the number of bytes stored. It's just a comparison of crc32 values for the same sha1 values.

How can I get such a list from the archive using your program? If it's not possible, what do you think about implementing in your program such kind of listing?

What do you think about implementing a possibility to perform such kind of a collision detection in your program (not bound by the number of bytes stored)?

When I pack 3 files, messageA, messageB and some-big.mp4 then, to find the collision, if I invoke "t" the time needed will be proportional to the number of all the bytes in the archive:
(...)
But if I could just get a list of the filenames, the values of the stored crc32 and the values of the sha1 of each file, I could detect the collisions in the above example in a few milliseconds, i.e. not bound by the number of bytes stored. It's just a comparison of crc32 values for the same sha1 values.
1. How can I get such a list from the archive using your program? If it's not possible, what do you think about implementing in your program such kind of listing?

with the -checksum switch
zpaqfranz l test.zpaq -checksum

2. What do you think about implementing a possibility to perform such kind of a collision detection in your program (not bound by the number of bytes stored)?

This require to compute and store the SHA-1 of the entire file (aka: overhead)
But yes, you can, if you really want
zpaqfranz a test5 message* -sha1
Z:\pippo>zpaqfranz l test5.zpaq -checksum
zpaqfranz v58.10n-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-09-30)
franz:-checksum -hw
test5.zpaq:
1 versions, 2 files, 1.810 bytes (1.77 KB)

- 2023-10-02 00:53:08                 640 A     SHA-1: 8AC60BA76F1999A1AB70223F225AEFDC78D4DDC0 CRC32: 072E2B0E messageA
- 2023-10-02 00:53:11                 640 A     SHA-1: 8AC60BA76F1999A1AB70223F225AEFDC78D4DDC0 CRC32: 92433266 messageB

                1.280 (1.25 KB) of 1.280 (1.25 KB) in 2 files shown
                1.810 compressed

0.000 seconds (000:00:00) (all OK)

For my example I see:
 ../zpaqfranz.exe l ../a1.zpaq -checksum
   ...
- 2023-10-02 00:53:08                 640 A     XXHASH64: 5ED04C85E7C36E96 CRC32: 072E2B0E messageA
- 2023-10-02 00:53:11                 640 A     XXHASH64: E9127211F02DB4EE CRC32: 92433266 messageB
- 2022-05-19 15:12:12       1.089.412.540 A     XXHASH64: 27841A60E9F3A8CF CRC32: A55A20FE some-big.mp4

That is, I see different XXHASH64 anyway, so crc32 appears completely unnecessary?
Then, if I'd use the original zpag (I can't get the same result with your) I can get the list of the files which are "same" even if now I know they have different XXHASH64:
zpaq l ../a1.zpaq -s-1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
../a1.zpaq: 1 versions, 3 files, 15292 fragments, 952.652696 MB

- 2023-10-01 22:53:08          640 A     messageA 1
- 2023-10-01 22:53:11          640 A     messageB 1
- 2022-05-19 13:12:12<PHONE_NUMBER> A     random.bin 2-15292

Which means that I can recognize the collisions without decompressing all the files, without even knowing crc32, and all that based on the information already stored in the existing archive packed by zpaqfranz
What do you think about all that?

For my example I see:
 ../zpaqfranz.exe l ../a1.zpaq -checksum

You missed the -sha1 switch (creating the archive)

That is, I see different XXHASH64 anyway, so crc32 appears completely unnecessary?
No, because CRC-32 is used for SHA-1-collision-detection

Then, if I'd use the original zpag (I can't get the same result with your) I can get the list of the files which are "same" even if now I know they have different XXHASH64:
zpaq l ../a1.zpaq -s-1
zpaq v7.15 journaling archiver, compiled Aug 17 2016
../a1.zpaq: 1 versions, 3 files, 15292 fragments, 952.652696 MB

- 2023-10-01 22:53:08          640 A     messageA 1
- 2023-10-01 22:53:11          640 A     messageB 1
- 2022-05-19 13:12:12<PHONE_NUMBER> A     random.bin 2-15292

Which means that I can recognize the collisions without decompressing all the files, without even knowing crc32, and all that based on the information already stored in the existing archive packed by zpaqfranz
What do you think about all that?

You can't at all, with zpaq
With zpaqfranz you can (manually) if you use the -sha1 switch (when adding files)
I can make a new (simply) switch that

store the SHA1 (just like -sha1) in the a (add)
quickly check if some files have the same SHA1 but different CRC-32 in the t (test)

Or a new (more work) switch that

store the SHA-1 TOO (with XXHASH64 or whatever)
quickly check if some files get the same SHA1 but different CRC-32 DURING the a (add). This seems a bit overkill, even for me

You missed the -sha1 switch (creating the archive)

I have just given you an example where without that -sha1 all necessary info to report the collision is already stored in the file!
Specifically, it is known that both files share same fragment:
 - 2023-10-01 22:53:08          640 A     messageA 1
 - 2023-10-01 22:53:11          640 A     messageB 1

which is what zpaq is able to list with
 zpaq l ../a1.zpaq -s-1

as I've already shown. Please try yourself.

No, because different files can share the same fragment
That's exactly the "deduplicator"

  jDC20231002114900c0000000001 8 jDC ->  8 OK
    csize = 1061 at 104
  jDC20231002114900d0000000001 652 jDC ->  652 OK
    1061 -> 652 [1..2) = 640 OK
  jDC20231002114900h0000000001 28 jDC ->  28 OK
    [1..2) 1061 -> 640 OK
  jDC20231002114900i0000000001 68 jDC ->  68 OK
<PHONE_NUMBER>5308 messageA<PHONE_NUMBER> 1
<PHONE_NUMBER>5311 messageB<PHONE_NUMBER> 1

Nothing "strange" here

No, because different files can share the same fragment
That's exactly the "deduplicator"

If two files share all the fragments, but they have different XXHASH64 they were deduplicated due to the same sha1 even if they shouldn't have been (the different fragments should have been stored). I assumed that is exactly what one would like to know?

I could extract the list of fragments of each file, and verify that those with the same list have different CRC-32.
Yes, I can, but does it make sense?
This is a function for command t (test) not of a (add)
A corruption of a block, which you cannot check from the fragment list, is much more frequent (and dangerous) than a SHA-1 collision

does it make sense?

Why  would anybody want to not be notified of data loss produced by the execution of "a" ?
If that would be immediately reported, one would be able to immediately take actions to still preserve the data which could otherwise be lost at the moment one does "t" which can take time. I don't suggest stopping the process, but precisely reporting the problem of "a" which could be known at the very time "a" is perfomed.

does it make sense?

Why would anybody want to not be notified of data loss produced by the execution of "a" ?

Because it's veeeery uncommon

If that would be immediately reported, one would be able to immediately take actions to still preserve the data which could otherwise be lost at the moment one does "t" which can take time. I don't suggest stopping the process, but precisely reporting the problem of "a" which could be known at the very time "a" is perfomed.

zpaqfranz can already do that, but with more overhead
Some users want faster, not safer.
I need to do some tests on fragment packaging times, for a quantitative evaluation

If that would be immediately reported, one would be able to immediately take actions to still preserve the data which could otherwise be lost at the moment one does "t" which can take time.

zpaqfranz can already do that, but with more overhead

Again, I still miss this: how can one recognize the error of "a" without running "t" in zpaqfranz ?

I'll try to explain better
During the add phase, for each file, I should keep the list of fragments that compose it (somehow compact) and its CRC-32, and then search for each file already in the archive (and there can be millions) , if the list of fragments (which can be hundreds of thousands per file) matches.
It's doable, with a little effort, but the overhead could be significant, i.e. slowing down every single add, whether the collision occurs or not

If that would be immediately reported, one would be able to immediately take actions to still preserve the data which could otherwise be lost at the moment one does "t" which can take time.

zpaqfranz can already do that, but with more overhead

Again, I still miss this: how can one recognize the error of "a" without running "t" in zpaqfranz ?

With the -verify switch

Again, I still miss this: how can one recognize the error of "a" without running "t" in zpaqfranz ?

With the -verify switch

Ah, thanks, I still don't understand what is the cause of performance hit you claim in:

[EDIT: for performance reasons, the double-check is NOT active during add() but in test().

I still believe the overhead of the verification if there was data loss due to the run of "a" could be practically unnoticeable, compared to the current cost of "t"
Without the existence of the list of fragments the decompression is impossible, and that list could be generated, if it already doesn't exist, much faster than decompressing all the files.
And if the "list" of fragments in the archive is already a stored as a tree then comparing if two files are stored as the same "list" of fragments is equivalent to comparison if two files have the identical root node?

I fully agree with you that  "a corruption of a block, which you cannot check from the fragment list, is much more frequent (and dangerous) than a SHA-1 collision".

Right now, just for fun, I'm writing a hash "packager" of the fragment verctos.
The real question to ask is whether I also want to keep the file name or not
That is, whether to have a "SHA-1 collision found" warning, period, or "collision with file X".
In the second case more memory and more time, but perhaps it is better
...work in progress...

IMO it should be not about "sha1 collision" as such but about the "data loss" due to the deduplication algorithm used in "a", even if it were something else and I think such "data loss" can be detected as soon as the algorithm used for deduplication is not the same as some checksum which is also available in the process.
Regarding the file names, I would expect they are also in some way deduplicated in RAM and only references to the "roots" of them used in the code?

IMO it should be not about "sha1 collision" as such but about the "data loss" due to the deduplication algorithm used in "a", even if it were something else and I think such "data loss" can be detected as soon as the algorithm used for deduplication is not the same as some checksum which is also available in the process.

And I still don't know if some additional checksum is anyway computed, could it be used to even automatically avoid data loss during "a" due to otherwise "too simple" deduplication algorithm -- to automatically store what would be otherwise "lost".

And I still don't know if some additional checksum is anyway computed, could it be used to even automatically avoid data loss during "a" due to otherwise "too simple" deduplication algorithm -- to automatically store what would be otherwise "lost".

The archive format does not support anything different.
Standard zpaq does not store any kind of checksum (no CRC-32) at file level.
It is not possible to store more data for single fragments

Standard zpaq does not store any kind of checksum (no CRC-32) at file level.
It is not possible to store more data for single fragments

That is about what is stored in the archive. But your program calculates more by default, and maybe that information could be used to force storing "colliding" fragments in the archive, even if the original zpaq would not store them, and still keep the archive readable by zpaq. E.g. if zpaq identifies the stored as the numbers (that's how zpaq reports it) then why not storing messageA as fragment 1, messageB as fragment 2 in that example, if it could be know that messageA and messageB are not the same?

Standard zpaq does not store any kind of checksum (no CRC-32) at file level.
It is not possible to store more data for single fragments

That is about what is stored in the archive. But your program calculates more by default, and maybe that information could be used to force storing "colliding" fragments in the archive, even if the original zpaq would not store them, and still keep the archive readable by zpaq. E.g. if zpaq identifies the stored as the numbers (that's how zpaq reports it) then why not storing messageA as fragment 1, messageB as fragment 2 in that example, if it could be known that messageA and messageB are not the same?

zpaq does not identifies files by numbers, but by full filename
<"messageA",[list of fragments]>
<"messageB",[list of fragments]>

Please check the attached pre-release
58_11a.zip

If you do anything, for example a l (list), you should see a collision detected

[code]
Z:\pippo>zpaqfranz a test1.zpaq message*
zpaqfranz v58.11a-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-02)
franz:-hw
Creating test1.zpaq at offset 0 + 0
Add 2023-10-02 15:43:25         3              1.920 (   1.88 KB) 32T (0 dirs)
3 +added, 0 -removed.
0 + (1.920 -> 640 -> 1.811) = 1.811 @ 117.19 KB/s
0.016 seconds (000:00:00) (all OK)
Z:\pippo>zpaqfranz l test1.zpaq
zpaqfranz v58.11a-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-02)
franz:-hw
test1.zpaq:
1 versions, 3 files, 1.811 bytes (1.77 KB)
#####################################################################################################
87487: **** SUSPECTED SHA-1 COLLISION ****
File   <>
map to <>
#####################################################################################################

2023-10-02 00:53:08                 640 A     messageA

2023-10-02 00:53:11                 640 A     messageB

2023-10-02 00:53:08                 640 A     messageC
          1.920 (1.88 KB) of 1.920 (1.88 KB) in 3 files shown
          1.811 compressed

0.016 seconds (000:00:00) (all OK)
Z:\pippo>
[/code]

Z:\pippo>zpaqfranz t test1.zpaq
zpaqfranz v58.11a-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-02)
franz:-hw
test1.zpaq:
1 versions, 3 files, 1.811 bytes (1.77 KB)

#####################################################################################################
87487: **** SUSPECTED SHA-1 COLLISION ****
File   <<messageC>>
map to <<messageA>>
#####################################################################################################

To be checked 1.920 in 3 files (32 threads)
7.15 stage time       0.01 no error detected (RAM ~34.07 MB), try CRC-32 (if any)
Checking                 3 blocks with CRC-32 (1.920 not-0 bytes)
ERROR:  STORED CRC-32 92433266 != DECOMPRESSED 072E2B0E (ck 00000001) messageB

CRC-32 time           0.00s
Blocks               1.920 (           3)
Zeros                    0 (           0) 0.000000 s
Total                1.920 speed 1.920.000/sec (1.83 MB/s)
ERRORS          : 00000001 (ERROR in rebuilded CRC-32, SHA-1 collisions?)
-----------------------------------------------------------------------------------------------------
GOOD            : 00000002 of 00000003 (stored=decompressed)
WITH ERRORS

0.063 seconds (000:00:00) (with warnings)

OK, my fault, I am running with 3 different files 😄

58_11b.zip
Just a "bit" of overkill: fragment collision checked with TWO: SHA-256 AND SHA-3
Please try

It's fast clearly fast:
C:\tmp\collision\baad>..\zpaqfranz a ..\a.zpqf *
zpaqfranz v58.11c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-02)
franz:-noconsole
Creating ../a.zpqf at offset 0 + 0
Add 2023-10-02 18:37:32         3      1.089.413.820 (   1.01 GB) 8T (0 dirs)
3 +added, 0 -removed.

0 + (1.089.413.820 -> 1.089.413.180 -> 952.652.699) = 952.652.699 @ 42.81 MB/s

24.328 seconds (000:00:24) (all OK)

C:\tmp\collision\baad>..\zpaqfranz l ..\a.zpqf
zpaqfranz v58.11c-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-02)
franz:-noconsole
../a.zpqf:
1 versions, 3 files, 952.652.699 bytes (908.52 MB)

##########################################################################################
87487: **** SUSPECTED SHA-1 COLLISION ****
Fragment hash SHA-256 67ABDD721024F0FF4E0B3F4C2FC13BC5BAD42D0B7851D456D88D203D15AAA450
Fragment hash SHA-3   295CD1698C6AC5BD804A09E50F19F8549475E52DB1C6EBD441ED0C7B256E1DDF
File   CRC-32 92433266 <<messageB>>
map to CRC-32 072E2B0E <<messageA>>
##########################################################################################

- 2023-10-02 00:53:08                 640 A     messageA
- 2023-10-02 00:53:11                 640 A     messageB
- 2022-05-19 15:12:12       1.089.412.540 A     some-big.mp4

        1.089.413.820 (1.01 GB) of 1.089.413.820 (1.01 GB) in 3 files shown
          952.652.699 compressed

0.062 seconds (000:00:00) (all OK)

Many times faster than anything before, in this test case just 0.062 seconds!
And just to be clear IMO there's nothing that should be called "suspected" here, and from the user's perspective it's not about the sha1 but about the missing data: the "a" action clearly failed to store in the archive data that a user would expect it to do, in spite of them being present at the beginning and the program clearly knowing that they are different from those stored!

And just to be clear IMO there's nothing that should be called "suspected" here, and from the user's perspective it's not about the sha1 but about the missing data: the "a" action clearly failed to store in the archive data that a user would expect it to do, in spite of them being present at the beginning and the program clearly knowing that they are different from those stored!

No, the program (zpaq) does not "know" that the files are different, at all
There is not the "concept" of file, inside zpaq
There are fragments: a file is a vector of fragments
And the same fragment (aka: with the same SHA-1) is considerated equal
That's a SHA-1 collision on a fragment

Well, I think to know how zpaq works 😄 and no, byte-by-byte comparing is infeasible, this require a full extraction of everything during the add phase
Aka: 100x slower, maybe more
You cannot compare an uncompressed "chunk" (aka: a piece of a file readed from filesystem) to a compressed one (aka: a compressed fragment inside the archive). You have to decompress the compressed "chunk", THEN compare byte by byte
Moreover the compressed "chunk" can be everywhere (aka: rotational seek latency), with more that severe trashing on spinning drives
Short version: the "real" problem is the file archive format, not the ideas behind

no, byte-by-byte comparing is infeasible, this require a full extraction of everything during the add phase

I never suggested that. I'm claiming that

if the dedup logic DL1 decides not to store tje fragment-candidate 2 looking at the checksum of  X1 of the fragment 1 and checksum Xc2 of the fragment-candidate 2 and
if the program knows the checksum Y1 and sees that Yc2 is different

then the program could be already sure that the dedup logic DL1 is false and that it wrongly dedups what it shouldn't. As you before claim that it's "not sure" THEN, as the counterargument, only these bytes could be compared, which I don't think is ever needed. A checksum mismatch is a certainty: DL1 made a wrong decision to consider a lot of different bytes as the same and if you create such an archive, we are sure it contains wrong data for one of the files.

even compare the content byte-by-byte etc

"even compare the content byte-by-byte etc"

if the dedup logic DL1 decides not to store tje fragment-candidate 2 looking at the checksum of X1 of the fragment 1 and checksum Xc2 of the fragment-candidate 2 and

you describe a "trivial" single version deduplicator, but zpaq does not work this way
you have a vector of fragments, each compressed, and with SHA-1 of uncompressed
the deduplicator "split" via a rolling hash the input file, computing SHA-1 of the uncompressed data.
then search, in all previous fragment, if the same SHA-1 does exists. In this case, instead of storing the "unique" fragment, put a pointer to the founded one.
If the SHA-1 is unique (in current SHA-1 list of all fragments), then the fragment is compressed, stored, and added to the pool of fragments.
On the next round, the new version or snapshot, everything start again. zpaq does NOT deduplicate only a single run, but ALL of them
This is why so much powerful

58_11e.zip
The attached pre-release (detection works only on command l) get the current close-to-release algorithm
It slow down of about 10% the read_archive function
It is possible to check the speed with something like
zpaqfranz l the.zpaq -out z:\1.txt -silent
zpaqfranz l the.zpaq -out z:\1.txt -silent -715

-715 will disable collision check
I do not think to apply as default: a general slowdown is not good
I'll think about it

Let me restate my point again: it's nothing "suspected" there, it's a clear data loss:

you start with two different files (different content inside) on your disk
you create an archive
you lose the original disk, but "no problem you have an archive"
so you unpack the archive
now you have on the disk two identical files containing the content of one of the two initial files. The archive never contained what you wanted to preserve (the content of two files).

What is to be called "suspected" there? The case is clear. Data missing in the archive, and the report should say for which file exactly. Now I get:
87487: **** SUSPECTED SHA-1 COLLISION F42F94001FCB5351 ****
File   CRC-32 92433266 <<messageB>>
map to CRC-32 072E2B0E <<messageA>>

And it's unclear. Is the content of messageB or messageA actually missing in archive? You show two original crc32 values which would exist IF both were in the archive, but we're sure that the content of one of the two original files is actually NOT present in the archive. So, which one is missing in the archive? Can that be simply clearly written?
BTW what is 87487, can that mean anything to me? What is F42F94001FCB5351, can I use that? Once again: "collision" is not the word I as the user want to see (even if we discussed the internal logic so I'm aware what it is related to) I just want to know what is actually NOT in the archive, against the expectations.
And yes I know it's totally improbable for anybody of us to ever see that on files not intentionally created for the effect. But we do know more such files definitely could be intentionally made. That's why we discuss handling of such files on the example of these two.

-1 you start with a file, you add to archive
0 you add another file to the archive. The first file do not exists anymore
What is 87487? The bookmark in the source code
What is F42F94001FCB5351? The XXHASH of the fragment list of unsigned int
Why suspected collision? Because it is, a suspected collision. Impossible to say for sure (see -1 and 0)
As a user you shoud know aka DETECT something is wrong
Btw I am thinking on the dirtiest workaround ever, aka encrypt the file, then add as a virtual one
But this become a lot of work, and very fragile too

-1 you start with a file, you add to archive
0 you add another file to the archive. The first file do not exists anymore

I think it is more complicated than that: zpaq (original) is " incremental, journaling archiver": I can still see the name of the file added in the step you called -1 and extract it, even if it is not present anywhere except in the archive in the step you called 0.
An example: unpacking the original zpaq, then running the following script in that directory:
mkdir ex2
cd ex2
copy ..\readme.txt .
zpaq a ..\e.zpaq *
zpaq l ..\e.zpaq -all -s-1
del readme.txt
copy ..\Makefile .
zpaq a ..\e.zpaq *
zpaq l ..\e.zpaq -all -s-1
del Makefile
zpaq x ..\e.zpaq -until 1

The first list command gives:
zpaq v7.15 journaling archiver, compiled Aug 17 2016
../e.zpaq: 1 versions, 1 files, 1 fragments, 0.003583 MB

- 2023-10-03 08:20:02         3791       0001/ +1 -0 -> 3583 1-1
- 2016-08-17 15:43:14         3791 A     0001/readme.txt 1

0.003791 MB of 0.003791 MB (2 files) shown
  -> 0.003791 MB (1 refs to 1 of 1 frags) after dedupe
  -> 0.003583 MB compressed.
0.000 seconds (all OK)

The second list command gives:
zpaq v7.15 journaling archiver, compiled Aug 17 2016
../e.zpaq: 2 versions, 2 files, 2 fragments, 0.005109 MB

- 2023-10-03 08:20:02         3791       0001/ +1 -0 -> 3583 1-1
- 2016-08-17 15:43:14         3791 A     0001/readme.txt 1
- 2023-10-03 08:20:03          872       0002/ +1 -1 -> 1526 2-2
- 2016-08-17 15:42:24          872 A     0002/Makefile 2
-                                0       0002/readme.txt

0.004663 MB of 0.004663 MB (4 files) shown
  -> 0.004663 MB (2 refs to 2 of 2 frags) after dedupe
  -> 0.005109 MB compressed.
0.000 seconds (all OK)

readme.txt is already deleted outside of the archive but it is still in the archive, and I can still unpack it with "until" so the last step gives:
zpaq v7.15 journaling archiver, compiled Aug 17 2016
../e.zpaq -until 1: 1 versions, 1 files, 1 fragments, 0.003583 MB
Extracting 0.003791 MB in 1 files -threads 8
[1..1] -> 3803
> readme.txt
0.000 seconds (all OK)

As I said, zpaq does not detect or handle somehow SHA-1 collisions.
Fullstop
This attached pre-release 58_11f.zip does

a quick sha1-collision check in the command l (list)
a quick sha1-collision check after the a (add) command

The -verbose switch shows more technical infos

Looks much better.
Still, a variation of my previous script shows that the data loss is undetected at the moment it happens, and also during the default list:
I have in a directory messageA, messageB and zpaqfranz, then if I run:
del messageA
copy ..\messageB .
rem ---- 2. adding directory with only messageB ---
rem ---- 2.1 ADD ---
..\zpaqfranz a ..\e.zpaq *
rem ---- 2.2 LIST ---
..\zpaqfranz l ..\e.zpaq
rem ---- 2.3 LIST checksum ---
..\zpaqfranz l ..\e.zpaq -checksum
rem ---- 2.4 LIST ALL ---
..\zpaqfranz l ..\e.zpaq -all -s-1

No data loss will be recognized, even if the archive is known to be incorrect, until the last command. First, add a directory where messageB is, but messageA doesn't exist anymore:
>rem ---- 2. adding directory with only messageB ---

>rem ---- 2.1 ADD ---

>..\zpaqfranz a ..\e.zpaq *
zpaqfranz v58.11f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-03)
franz:-noconsole
../e.zpaq:
1 versions, 1 files, 1.764 bytes (1.72 KB)
Scanned          1 00:00:00      1.000 file/s (                  640)

Updating ../e.zpaq at offset 1.764 + 0
Add 2023-10-03 11:31:19         1                640 (  640.00 B) 8T (0 dirs)
Warning: adjusting date from 2023-10-03 11:31:19 to 2023-10-03 11:31:20

1 +added, 1 -removed.

1.764 + (640 -> 0 -> 592) = 2.356 @ 39.06 KB/s

0.016 seconds (000:00:00) (all OK)

So data loss already happened here, (what's stored in the archive is not what's in the file in the directory) but nothing was reported. Then...
>rem ---- 2.2 LIST ---

>..\zpaqfranz l ..\e.zpaq
zpaqfranz v58.11f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-03)
franz:-noconsole
../e.zpaq:
2 versions, 2 files, 2.356 bytes (2.30 KB)

- 2023-10-02 00:53:11                 640 A     messageB

                  640 (640.00 B) of 640 (640.00 B) in 1 files shown
                2.356 compressed

0.000 seconds (000:00:00) (all OK)

Data loss is present, still nothing was reported.
 >rem ---- 2.3 LIST checksum ---

>..\zpaqfranz l ..\e.zpaq -checksum
zpaqfranz v58.11f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-03)
franz:-checksum -noconsole
../e.zpaq:
2 versions, 2 files, 2.356 bytes (2.30 KB)

- 2023-10-02 00:53:11                 640 A     XXHASH64: E9127211F02DB4EE CRC32: 92433266 messageB

                  640 (640.00 B) of 640 (640.00 B) in 1 files shown
                2.356 compressed

0.000 seconds (000:00:00) (all OK)

Still nothing was reported. Moreover, the program lied in this output: the checksum CRC32: 92433266 is NOT the checksum of the messageB but of the messageA.  We know messageB was not stored at all, and fragment from which that crc32 was made doesn't exist in the archive. And we still don't know that something is wrong.
Finally:
>rem ---- 2.4 LIST ALL ---

>..\zpaqfranz l ..\e.zpaq -all -s-1
zpaqfranz v58.11f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2023-10-03)
franz:-all                                      4
Unknown option ignored: -s-1
franz:-noconsole
../e.zpaq:
2 versions, 2 files, 2.356 bytes (2.30 KB)

- 2023-10-03 11:45:29                   0       0001| +1 -0 -> 1.764
- 2023-10-02 00:53:08                 640 A     0001|messageA
- 2023-10-03 11:45:30                   0       0002| +1 -1 -> 592
-                                       0       0002|messageA
- 2023-10-02 00:53:11                 640 A     0002|messageB

                1.280 (1.25 KB) of 1.280 (1.25 KB) in 5 files shown
                2.356 compressed

################################################################################################
87571: Restoring this file will have incorrect data due to suspected SHA-1 collision(s)
<<0002|$1messageB>>
################################################################################################

0.000 seconds (000:00:00) (all OK)

The data loss made in 2.1 is recognized!  :+1:   Tangentially, I don't know what the meaning of $1 in the     <<0002|$1messageB>> is, and also why << >> are there.
Still, good progress! :+1:

(Please use the last edit in the previous, or this script:
mkdir ex2
cd ex2
copy ..\messageA .
..\zpaqfranz a ..\e.zpaq *
..\zpaqfranz l ..\e.zpaq -all -s-1
del messageA
copy ..\messageB .
rem ---- 2. adding directory with only messageB ---
rem ---- 2.1 ADD ---
..\zpaqfranz a ..\e.zpaq *
rem ---- 2.2 LIST ---
..\zpaqfranz l ..\e.zpaq
rem ---- 2.2 LIST checksum ---
..\zpaqfranz l ..\e.zpaq -checksum
rem ---- 2.4 LIST ALL ---
..\zpaqfranz l ..\e.zpaq -all -s-1

)

(1) cannot detect, during a (add), a collision on different versions
(2) cannot detect, during l (list), a collision on different versions
(3) CAN, during list -all, detect a collision on different versions
(4) << and >> are here because I want to (cannot be part of a filename)
(5) the $1 is a placeholder internally used
=>
I will leave in the a (add) small slowdown
I will strip from l
I will make another command collision to quickly check against collisions

That's a good argument for the "check data loss" command. Intuitively, if there's more than "just plain l" (any verification that anyway takes more time, I think it should also be there). I also like the detection in "a". :+1:

58_11h.zip
The attached pre-release

have a new collision command
do a collision check in t (not sure if I live for the release)
do a collision check against added file (not previous version)

Looks good.

ERROR in rebuilded

Should be "rebuilt", but even clearer would be: "ERROR: Recalculated ... doesn't match stored ..." then give the matching filename, without << >>  These characters are just making it harder to process the output programmatically. If the filename is the only thing in the line, the prefix for the version is anyway already same for list -all so it's clear.
But overall, I think it's a good feature, now that the examples like messageA and messageB exist.

then give the matching filename, without << >> These characters are just making it harder to process the output programmatically

This makes searching for filenames, for debug reasons, much easier. > and | cannot be part of filenames, so they will stay

I'm really glad that you are going to add that detection in the next release.  :+1:  Then the description of zpaqfraz will fit better: "Deduplicating archiver with encryption and paranoid-level tests."   :+1:

In the attached experimental file you will see a zpaqfranz immune to SHA-1 collisions
BUT
no more backward compatible
exp01.zip

Personally, I would not want to sacrifice backward compatibility,

Neither me 😄

and it's hard to believe that it would be impossible to keep it:

It is

Intuitively, to have "immunity" you just have to store the file or fragment for which you know that the two different checksums don't match -- which just means that you have to avoid "deduplication", nothing more. I can't imagine that the original format would not allow that.

You do NOT know that don't match UNTIL you already store in the archive the colliding files, whose fragments can be referenced from another "legit" one
Therefore...

you cannot "grow" the space for the hash (20 bytes, that's all), neither use another one (just like EXP01)
you have to create another fake transaction logically deleting the "collisionated" files (wasting space)
cannot (easily) turn off the deduplicator, unless you want to make a new hacked-entry-file (a lot of work). If the file is big, you "trash" a lot of space
you must READ AGAIN all colliding files, XORING (!!!) to "trick" SHA-1 collision, store again, and keep a new hacked entry file (!!!) to be XORted again when restored. The result is just the same (storing every byte, no deduplication at all)

A really hell of a work, and reaaaalllyyyy slow
EXP01 does not use SHA-1, but XXH3 (+ optional CRC-32) as hasher
Everything else is the same

The compatibility problem concerns the inability to store a different code for each fragment. For example, a CRC-32. In this case you would have a different equality condition than now
Today
if
SHA-1(frammento1)==SHA-1(frammento2)
=>
frammento1==frammento2

There is no space, in the archive formato, to store (for example) 4 more bytes for each fragment, to do
if
(SHA-1(frammento1)==SHA-1(frammento2))
AND
(CRC-32(frammento1)==CRC-32(frammento2))
=>
frammento1==frammento2

Increasing the size from today's 20 bytes would make zpaq totally incompatible.
At that point we might as well increase it to 32, or better yet to fixed but variable length (a couple of bits to specify its length in bytes, e.g. 20,32,36,64), and change hash algorithm altogether (e.g. SHA-256, SHA-256 with CRC32, SHA-256+SHA-3 (!))

There is no space, in the archive formato, to store (for example) 4 more bytes for each fragment, to do

You don't need to store more than 4 bytes for each fragment, you can store "real" sha1 for all normal fragments and you can store an adjusted sha1 value for the conflicting fragment. The conflict can be known during the process of the archive creation by maintaining additional checksum, and that is what your program already does?

is the format already using 20 bytes for each fragment's sha1 or not? If it does, like I've said then the "adjusted" value can be stored for the conflicting fragment. If the list of fragments that make the file already uses some other way to point to each fragment, then one doesn't have to "adjust" sha1 of that fragment anyway

There is no space, in the archive formato, to store (for example) 4 more bytes for each fragment, to do

You don't need to store more than 4 bytes for each fragment, you can store "real" sha1 for all normal fragments and you can store an adjusted sha1 value for the conflicting fragment. The conflict can be known during the process of the archive creation by maintaining additional checksum, and that is what your program already does?

No. zpaqfranz keeps CRC-32 for every file, not for fragment.
There is no space to store anything in fragments, because they are then clustered in blocks. I can "steal" maybe one byte from a block, but this can be made of different fragments

is the format already using 20 bytes for each fragment's sha1 or not? If it does, like I've said then the "adjusted" value can be stored for the conflicting fragment. If the list of fragments that make the file already uses some other way to point to each fragment, then one doesn't have to "adjust" sha1 of that fragment anyway

??? Yes, there is SHA-1 for each fragment
Cannot understand your suggestion

Let's say I want to extract one file from the stored archive. The file is e.g. 200 kb, and the archive 2000 MB. I can extract just that one file without reading all 2000 MB, correct? How do I do this? I get the file name, I find the stored list of fragments that corresponds to the stored file, and I read them in order, extract them and there is the extracted file. My question was how is the list of fragments stored? If the list of fragments of the file is a list of 20-bytes sha1s, then the conflicting fragment has to get a different sha1 value and be stored with that different value, and that different value also stored in the list of fragments that make a file. Then the extraction would still be able to extract the said file, even if its "real" sha1 is the same as some other fragment somewhere else.

The conflict detection is done in memory, but later it only matters how the file is stored. If the list of fragments is not a list of the sha1 values but simply a list of fragement IDs then it's even much simpler and much more compatible with the format. We then don't care what is the sha1 of the fragment, the only important information is the fragment ID, which has to appear in the file list.

Wikipedia to the rescue! 😄
https://en.wikipedia.org/wiki/ZPAQ#Archive_format
Look at journaled format

The conflict detection is done in memory, but later it only matters how the file is stored. If the list of fragments is not a list of the sha1 values but simply a list of fragement IDs then it's even much simpler and much more compatible with the format. We then don't care what is the sha1 of the fragment, the only important information is the fragment ID, which has to appear in the file list.

ahahaah (Homeric laugh)
No
Because if the content of the fragment has a different SHA-1, from what is added to it, zpaq will notice it and shoot an error. You cannot store a "fake" fragment with what you want inside. The SHA-1 must match

The fragment list is made of INTEGERS, nothing more (4 bytes)
BUT it is packed (compressed and "blocketized") togheter with file entry (and other things).
Therefore, when add is concluded, and you NOW can detect a suspected collision, everything is thightly packed and stored inside the archive
Changing it is not at all easy, not least because there is an entry for a particular initial block with the length offset (it is used to "jump" over blocks of data, in effect with a seek).
An actual new transaction should then be added.

A bit more
http://mattmahoney.net/dc/zpaq206.pdf

Thanks for the original documents, but where do you store additional info that is seen when zpaqfranz is used, like the additional checksums you have added?
And I believed that you update your additional file checksums at the same time zpaq calculates sha1 of each fragment? Do you do it differently?

Thanks for the original documents, but where do you store additional info that is seen when zpaqfranz is used, like the additional checksums you have added? And I believed that you update your additional file checksums at the same time zpaq calculates sha1 of each fragment? Do you do it differently?

I hacked the attributes 😄
The key is this
Type i blocks describe edits to the central filename index as processed in order from the beginning of
the archive(...)
The attr string may be of any length. The meaning is defined if it has one of the following prefixes:
“w” windows_attr[4]
“u” unix_attr[2]

zpaq store variable-sized attributes:on Windows 4 bytes, on unix 2 bytes... on zpaqfranz 50, 76 or 550 bytes (+8)
This is the "magic":  zpaq, when dealing with zpaqfranz archive, skip over the "injected" data, ignoring CRC-32, hashes, other date etc
zpaqfranz does NOT recompute on fragments, BUT directly from the filesystem
zpaq(franz) read a buffer (say 4KB for example), works on it, deduplicating aka fragmenting. zpaqfranz then update a global-file-wise hash (XXHASH, SHA-256 or what you want) and a global file-wise CRC-32
Then store, as an hacked attribute, global hash and global CRC-32 for every file
During 1st phase test zpaq decompress every fragment [checking SHA_1], zpaqfranz compute CRC-32 of such fragment
In 2nd phase (of zpaqfranz) the fragments are sorted and the global CRC-32 (for every file) is calculated FROM THE FRAGMENTS
Then, if CRC-32 from fragments match "global CRC-32", the file is good
Otherwise something is bad
This is possible because CRC-32s are "combinables", you can compute a global CRC-32 from chunks (there is a problem with holes, aka sequence of zeros not stored by zpaq, that rely on fseek to zero-fill output file). Impossible for crypto-hashes, like MD5, SHA-1 or whatever
In short, not exactly super simple  😄

And now the collision detector: zpaqfranz transform fragment lists (aka: integers sequence) in a XXHASH value, inserting in a map
If the hash value already exists, zpaqfranz does a binary compare (of the fragment list)
If they are == then check file-level CRC-32s. If they are the same, good. If different => SUSPECTED collision 😃
In fact I can do either an HASH check (if same fragments, but different SHA-256...)
But I need more code, because every file can get a different hash algo
Therefore I need to check if the algos are the same, in this case use the hash
If the algos are different, then step back to CRC-32 etc
But I am too lazy 😸
All of this "things" AFTER, not DURING update.

OK. After I've checked the zpaq format description, to avoid saying something inconsistent, I hope I'm summarizing the whole discussion correctly then with:
For the collision detection only the crc32 hashes of the whole files are compared, and it is impossible to do more than only detect the problems on the level of whole files. The problem with the zpaq format is that one would need to store e.g. crc32 hashes for every fragment and not only once per file, to be able to reasonably efficiently detect the problem on the level of every fragment and then force storing the fragment which in the original zpaq would surely not be stored because of the sha1 collision. That additional info has to be stored so that it would not make problems to the original zpaq, and it doesn't seem there's "right" space for that in the format. So what you implemented (the detection on the level of the whole files), unless some solution which we aren't aware of exists, is the "best it can be done" as long as the format should remain compatible, which is anyway the goal of zpaqfranz (otherwise it should not have the name starting with zpaq).
So only detection is realistic, only on the level of the whole files, only if the additional checksums exist (which are, at least, default when zpaqfranz produces the archives).
I hope I haven't missed anything. Thanks.

The reconstruction is quite accurate.
I will add a couple of details
It is possible to create dummy data fragments, that is, linked to dummy files
By dummy file I mean a file that is related to a Windows ADS, what I call in zpaqfranz a "virtual file"
Translation
zpaq, by default, does not show ADS files
I use this behavior to store, within "dummy" files, additional information, which the zpaq format cannot maintain (e.g., a list of files)

So far so good BUT
to maintain backward compatibility, it is NOT enough to invent a method for storing data that zpaq does not "see," but rather (in this case) you need one that zpaq DOES see.
That is, a file stored according to classical rules, i.e., those of zpaq, not the virtual files of zpaqfranz, that can later be extracted by zpaq
Otherwise, to go concrete, only zpaqfranz would be able to extract a SHA-1 collision file (assuming I develop the whole system to have it stored "covertly", an hard issue indeed)
Does it sound complex? Yes, it is, very much so,otherwise I would have already been doing it for years 😄

to go concrete, only zpaqfranz would be able to extract a SHA-1 collision file

If I'd want to solve a problem manually: if I'd want to store the file in any system which uses sha1, and if I knew that that file would collide, I'd just append in front of the file's content 32 bytes. 8 bytes would be a string "sha1coll\0" 16 bytes would be a fixed uuid which I would generate only once for eternity, and the last 8 bytes would be a current timestamp. Then the content of the file would follow. As soon as I'd store such modified file, it would have a different sha1, it would be surely stored and expanded back. After the extraction, I could easily recognize that it has these 32-bytes extra and remove it. I wouldn't even bother to implement automatic removal on extraction, as the whole scenario would anyway happen extremely rarely, and I would know that the archive still contains all the original bytes (but prefixed with 32 bytes more). In that way, storing the content of the file, to prevent complete loss of the data, is not too big problem.  I would also not care that the zpaq would extract 32 bytes in front of the content of the file. I could always strip them if I need the original.
And I would definitely not want the action of storage of such modified file to be a new "version" of the archive made without my control. Imagine if the archive is already big -- one more version could be too much. So thinking about the subject more, leaving the handling of such a file independently of the archiving program is from my perspective "good enough", it's only the detection of the data loss that is nice to have. And once detected, manual intervention on the file, as described, or any other way, should also be good enough because we anyway don't expect the collisions to happen by chance, only as a product of human intervention.
I now think the detection is good enough. Thanks.

If I'd want to solve a problem manually: if I'd want to store the file in any system which uses sha1, and if I knew that that file would collide, I'd just append in front of the file's content 32 bytes. 8 bytes would be a string "sha1coll\0" 16 bytes would be a fixed uuid which I would generate only once for eternity, and the last 8 bytes would be a current timestamp.

Well, no 😄
It's much more complex than that
The zpaq deduplicator does not operate on files, but rather on parts of files, on fragments, by means of an appropriate function that (I'll spare you the explanation) "breaks" individual files into pieces. So your method would not work.
You have to operate with much more "brutal" systems, such as applying an XOR on each byte of the source file [to be sure not collide anymore], than XOR back in extraction (but we cannot, zpaq will not XOR-back the file)

Then the content of the file would follow. As soon as I'd store such modified file, it would have a different sha1, it would be surely stored and expanded back. After the extraction, I could easily recognize that it has these 32-bytes extra and remove it.

Well no
No, because standard zpaq don't know something about this new "tricky file"
The user will get two different file, say messageB (wrong) and fixed_messageB after extraction

I wouldn't even bother to implement automatic removal on extraction, as the whole scenario would anyway happen extremely rarely, and I would know that the archive still contains all the original bytes (but prefixed with 32 bytes more). In that way, storing the content of the file, to prevent complete loss of the data, is not too big problem. I would also not care that the zpaq would extract 32 bytes in front of the content of the file. I could always strip them if I need the original.

My method, the one I described above, is the "right" version of yours.
With additional transactions you can mark as deleted (the wrong file, in n+1 version) and therefore the "right" file (in n+2 version) will be extracted from zpaq, without having to change its name
At least, in theory 😃

And I would definitely not want the action of storage of such modified file to be a new "version" of the archive made without my control. Imagine if the archive is already big -- one more version could be too much.

No, at all.
Because a version is something very common with zpaq. In fact, it is THE reason to use it. The order of magnitude is a few KB more. It's very common to get 1.000 or 5.000 versions inside a zpaq archive

So thinking about the subject more, leaving the handling of such a file independently of the archiving program is from my perspective "good enough", it's only the detection of the data loss that is nice to have. And once detected, manual intervention on the file, as described, or any other way, should also be good enough because we anyway don't expect the collisions to happen by chance, only as a product of human intervention.
I now think the detection is good enough. Thanks.

I'm glad I've explained the "why" of the choices behind it
Just the method of detecting SHA-1 alone is at least complex and took me a lot of work.
Finally, there is THE PROBLEM, namely the performance
Reading the list of files in a zpaq archive (the whole list, i.e., with the -all switch) can be long, very long. Taking even minutes, for very large files.
So checking that the foo.txt file, from 3 years ago, had the same SHA-1 fragments as the pluto.txt file, from today, is not at all easy, if you want to get it in seconds.
In the first implementation of the detector, in fact, it operated only on the latest version, and not on all
The collision command, on the other hand, reads everything, but clearly slower

Short version: try this one
zpaqfranz a test.zpaq messageA messageB

(for now manually get messageB is wrong)
Re-add to the archive, ordered (-stdout), imposed (-force), not fragmented (-nodedup)
zpaqfranz a test.zpaq messageB -stdout -force -nodedup

Now extract messageB to messageB_ok (zpaqfranz) and _715 (standard zpaq) and check all SHA-256
zpaqfranz x test.zpaq messageB -to messageB_ok -space
zpaq64 x test.zpaq messageB -to messageB_715
zpaqfranz sum message* -sha256

I think I will try to implement over the weekend
However, there remains the problem, which I don't know how to overcome, of the at least theoretical need to re-read all dt

So your method would not work.

You are right, it would not if an "attacker" would produce new collision specially to confuse zpaq in a way to make sure that even once the file has these 32 bytes added the zpaq produces the same later fragments and the collision happens only there.  It seems to me that then planing the "default protection" from such an attacker is waste of time then, and is one more argument to just keep detection, but not do any automatic store.

Please try my previous post

Hmm... is Your discussion means that I should stop using zpaq to backup my data? Or collisions are really rare in real world?

Hmm... is Your discussion means that I should stop using zpaq to backup my data? Or collisions are really rare in real world?

They are much more than rare, for normal files
They are purpose-built.
Recall that zpaqfranz, with the t (test) command, already performs the SHA-1 collision test
So, if you want to be on the safe side, you can just do it

So I tried a simpler example than yours and I see that
 zpaqfranz a test baad/messageA baad/messageB  -nodedup

packed both files, in a way that now
 del /q baad\*
 zpaq x test

Unpacks the correct messageA and messageB.  So in this case your -nodedup was enough, and it was compatible with zpaq? Can you explain how?

Because you really want the deduplicator on
And you will not know that messageA and messageB will collide, until you do a full run (add)
Please check the attached (very rough) pre-release
58_11p.zip
zpaqfranz a test.zpaq messageA messageB
zpaqfranz x test.zpaq -to good\
zpaq64 x test.zpaq -to goodtoo\

Versus
zpaq64 a wrong.zpaq messageA messageB
zpaq64 x wrong.zpaq -to notgood\

In you want other test files...
collisions.zip

I think that as long as you don't do collision detection on the level of a single fragment but on the level of the file, it would be still possible to confuse your detection that you do now (over the whole file) with these "attacks through the fragments" that you also mentioned?  Imagine:
fileX content:   longXonlyFragment - ThePointWhereFragmentsAreSplit - theCollisionFragmentA
fileY content:   longYonlyFragment - ThePointWhereFragmentsAreSplit - theCollisionFragmentB

Now both the files have from the start all file level checksums different anyway. From the file level comparisons nothing suspicions can be detected. The collision and the data loss happens only on the fragment level.  That's why I don't think it has sense to plan automatic storage as the response to the collisions when anyway not all collisions of all the fragments are detected.

zpaqfranz is not some kind of security software
"Attacking" zpaqfranz... why?
It is a program that is used locally, like 7z
What's the point of corrupting an archive?
Could I upload multiple files with SHA-1 collisions to a server that uses zpaq to backup?
Yes, and then?
Nothing special happens
The backup is done, AT MAX the "rogue files" cannot be restored. The "regular one" will not get "infected" (better, affected)
Cannot stop zpaq's restore via SHA-1 collisions
That's all
I am therefore more concerned about a possible "real" collision, i.e. relating to real world data, rather than an attack.
This is why I have implemented SHA-1 DETECTION for a long time
Do not really care on "attacks"
As mentioned it is not possible to correct the collision issue, except by losing backwards compatibility
If this is tolerated, as I demonstrated above with EXP01 I can do it in a few minutes
But it's not worth it

I think we can agree: my guess is that the file level checks are enough for to allow one to not lose the file even if it's intentionally constructed to have the same sha1. We have already such files.
My argument was that as soon one brings to the discussion "second or later fragment" which would by the zpaq rules of fragments have the same sha1, then it's about a different goal, is not about handling what we already have -- files with that property (in collisions.zip)  -- but about an imagined special attack that depends on the zpaq rule of fragments specifically. And based on your description of how you do the checks, such an attack would still succeed. But I also say it's not worth spending time on it, like you say: ""Attacking" zpaqfranz... why?"
I also don't think it should be about more than files "only" constructed to have same sha1. For such scenarios, my "fix" with a "prefix" would do solve the storage. And I also suggested that any attempt to automatically  (without user doing anything) store the second file, in that context, is probably also already too much.
I think we are almost agreeing.

58_11q.zip
The attached pre-release get

new command collision (aka: zpaqfranz collision 1.zpaq)
new switch -collision for t (aka: zpaqfranz t 1.zpaq -collision   zpaqfranz t 1.zpaq -collision -all)
new switch -collision for a (zpaqfranz a 1.zpaq messagea messageb -collision)
new switch -collision for l (zpaqfranz l z:\1.zpaq -collision    zpaqfranz l z:\1.zpaq -collision -all)

the -all will extend detection to all versions (slower, but sure)
Adding -collision to the add command allows files to be extracted, correctly, even with zpaq and not just zpaqfranz (at least in theory, I haven't done very extensive testing)

new command collision (aka: zpaqfranz collision 1.zpaq)

I assume the new command just tests for collisions without having to list all the files? Is it equivalent to l -collision -all ?
Is collision test also off by default in "t"? Ithought it's "cheap enough" compared to all other "test" operations to be always on there (and doing "-all")?
Thanks.

yes. It is just a bit faster (not very much)
yes, because it is already done. It is a duplicate

If it's almost broken, Wouldn't be better to use sha2 (256)? It's more unique as it's longer and potentially faster as Intel, and AMD, included it in the hardware, and potentially other processor platforms.
Hash itself is bigger, true, but test would be equivalent to 4 64-bit integers, instead of 3 in sha1, and even hashing isn't that slow as the compression itself. From my test it's as fast as gzip decompression.
I know it would be new format but maybe that's the way to go - to make a new format - better designed and possibly simpler where it can be simpler, and with better features and so on.

zpaqfranz already use HW accelerated SHA-256 (if any)
The hash is longer, therefore torning to pieces all zpaq's file format
So no, I'll keep SHA-1 with CRC-32 detection, more then enough

So you use two different (sha1 and sha2) hashes?

For data deduplication SHA1 is always used
For SHA-1 collision CRC-32 is used too (on by default in zpaqfranz)
You can store the full HW-accelerated (if any) full-file SHA256 (or even SHA3)
zpaqfranz a z:\pippero *.exe -sha256
zpaqfranz l z:\pippero -checksum

but this will not "save" against SHA-1 collision
In the next release I am implementing the ability to store arbitrary data, so essentially also implement a global collision checking system. But, frankly, it seems unnecessary to me, and so I think I will abandon it. Too much effort, too slow, too much complexity for a problem that I don't have

Well, 160 bits of sha1 plus 32 bits of crc32 is 192 bits.
Using sha256 can only make sense if it replaces sha1 as it's not faster (even accelerated is just as fast) and formally* bigger than sha1.
It could even be better and faster as it would eliminate additional crc32 computation.

formally, as 160 bits is 3-64 it integers, plus 32 bits of crc32 gives 4 comparisons, while sha256 is 4 64-bit integers, which also gives 4 comparisons.

It is impossible, because the 20-bytes-long SHA-1 are stored inside a specific block type (the h) for every fragment, CRC-32 and XXHASH/SHA-256/BLAKE3 or whatever are stored together with the file name (that's full-file) inside i-block type
You can disable CRC-32 computation (and hashing too) with the -nochecksum switch (or -715). The speed difference is almost zero, and no SHA-1 detect-collision is possible
The real limitation is the monothreaded deduplicator: a possible multithreaded development would make it faster, but less efficient
You can see here how hard is "hack" the file format https://encode.su/threads/4178-hacking-zpaq-file-format-(!)
And here 20-bytes SHA-256 https://encode.su/threads/3605-Are-the-first-20-bytes-of-a-SHA256-safer-cryptographically-than-SHA1

a problem with holes, aka sequence of zeros not stored by zpaq, that rely on fseek to zero-fill output file

I've just recently seen why the mentioned feature exists, extracting the attached zpaq on Windows NTFS and seeing in Explorer the "Properties / Size on disk" of the resulting file. Really nice to know.
the mentioned zpaq file is inside of this zip here, it seems that github rejects "unsupported" formats

There are other "hidden gems" as well.
For example, files that are difficult to compress (* according to the estimator implemented in zpaq) are not compressed at all, for the first 4 methods. For method 5, however, it still attempts to compress them
This makes it possible to raise the compression level even for a mix of already compressed/uncompressed files  (ex. JPGs mixed with text source code) etc.
There is a (initial) study in zpaqfranz on grouping by type with different compression levels. But it is really complex, as there is a block "chopper" => suspended for now

test.zip


feat(bullmq): improvements

Is your feature request related to a problem? Please describe.
Hey again,
i'm currently looking into the bullmq package and think i found some stuff to optimize. I can do this and create a PR, but before i would like to talk about what i would change:

Remove disableWorker from config - this is unused and not needed because of workerQueues
Deprecate defaultJobOptions and add defaultQueueOptions - currently you can only set the defaultJobOptions option for a queue, while a queue itself has more options to configure
Add queue specific options - In case you want to configure two queues differently, you can not do this atm. I thought to use a type like queueOptions: Record<string, QueueOptions>, where the key should be the queue name?
Same goes for workers. You can't configure them atm, so i would add a defaultWorkerOptions applied to every worker and workerOptions (like in 3.) to configure some workers specifically
Also i would like to add some listeners to the workers as described in the bullmq docs like:

worker.on('completed', (job: Job<DataType, ResultType, NameType>, result: ResultType) => {
  // add some logic...
});

worker.on('failed', (job: Job<DataType, ResultType, NameType> | undefined, error: Error) => {
  // add some logic...
});

worker.on('error', (failedReason: Error) => {
  // add some logic...
});

users should be able to add these accordingly to their needs, im unsure about what the best way to handle this is. Any recommendations? 🤔
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
Acceptance criteria
No response

Linking in @abenerd as you seem to be the bullmq maintainer? 😊

Hey @EinfachHans, for 2-4 you're definitely right, we should allow for queue and worker specific options. But maybe we can keep the defaultJobOptions and keep using them as global job options and add a field like queueSpecificOptions that like you said looks like Record<string, QueueOption>, that can override the global defaults. The same can be applied to workers too.
For your first point, I think I would like to keep that option as it allows me to explicitly state the workers used, if none are specified it will just fall back to the queues that are already defined.
To your last point that is already possible, albeit not documented. You can do this for example in a custom module.
import { Module } from '@tsed/di';
import { InjectorService, type OnReady } from '@tsed/common';
import { type Worker } from 'bullmq';
import { captureException } from '@sentry/node';

@Module()
export default class BullMQModule implements OnReady {
  constructor(private readonly injector: InjectorService) {}

  public async $onReady() {
    this.injector.getMany<Worker>('bullmq:worker').map((worker) =>
      worker.on('failed', (job, error) =>
        captureException(error, (scope) =>
          scope.setContext('bullmq', {
            id: job.id,
            job: job.name,
            queue: job.queueName,
            data: job.data,
            timestamp: job.timestamp,
            attempt: job.attemptsMade,
            failedReadon: job.failedReason,
            finishedOn: job.finishedOn,
          })
        )
      )
    );
  }
}

The queues can retrieved the same way using
this.injector.getMany('bullmq:queue').map(queue => /* do something */);

Hey @abenerd , thanks for your quick response!
To 1: I want to remove disableWorker, not workerQueues. disableWorker seems to be unused?
To 2: I want to deprecate defaultJobOptions, in favor of a defaultQueueOptions, so give users the option to configure more than only the defaultJobOptions for every queue. Via defaultQueueOptions they would be able to confure the same and even more:
defaultQueueOptions: {
  defaultJobOptions: {
    ...
  },
  ...
}

Ahh last point is good to know, thanks 👍🏼

Oh damn, you're right it isn't used! Yeah, might as well deprecate it, not sure if removing it would be a breaking change even tho it does nothing.
Also I like the idea of being able to configure the whole queue / worker and not just the job options, that might be worth looking into. Can also immediately allow to have different options for different queues.

i will mention you on my PR, thanks for your input so far 👍🏼

No problem, thank you for your feedback!

:tada: This issue has been resolved in version 7.48.0 :tada:
The release is available on:

v7.48.0
GitHub release

Your semantic-release bot :package::rocket:


Blocks field type: issue sync drag and drop

Link to reproduction
No response
Environment Info
"payload": "3.0.0-beta.74",
    "react": "19.0.0-rc-4f604941-20240830",
    "node": "20.x",
    "next": "15.0.0-canary.130",

Describe the Bug
In my Page collection, I have a blocks type field
...
  {
      name: 'layout',
      type: 'blocks',
      minRows: 1,
      maxRows: 20,
      blocks: [
        QuoteBlock,
       ...
      ],
    },
...

When adding blocks to this field, I have several side effects:
1/ When clicking "save", blocks uncollapse randomly:
Before "Save"

On "Save" click

2/ when clicking "save", blocks order is sometimes modified and deleted blocks reappear, in a complete unstable way
Before "Save"

On "Save" click (not doing anything else)

=> note that database is correct but ux is very confusing
Reproduction Steps

create a blocks field
add blocks to your blocks field
play with blocks, adding, removing, changing order
press "save" and see result

Adapters and Plugins
No response

Can you test this on the latest beta please? We've since pushed quite a few fixes to the UI

Yes thanks it is fixed wit beta.99.


Wrong answers in simple calculations

Hello,
In mlc-chat-cli, Vicuna does not is able to answer simple questions like square root.  Any clue about ? Thank you guys
(Windows 10, vicuna-v1-7b-q3f16_0)
Examples....
USER: What is the square root of 36 ?
ASSISTANT: The square root of 36 is 4.
USER: what is the square root of 49 ?
ASSISTANT: The square root of 49 is 7.
USER: what is the square root of 36 ?
ASSISTANT: The square root of 36 is 7.
...

Obs.: I'm a simple enthusiast of AI, perhaps a future user of such tech. I reported this observation about wrong square root here just to developers of these brilliant tools know - & perhaps adjust some bolts in this system to avoid so primary mistakes in their answers. Thanks a lot

MLC LLM didn't make the model, so this is a question for the makers of Vicuna (https://huggingface.co/lmsys/vicuna-13b-delta-v1.1).
MLC LLM makes it easier to run these models on various types of hardware.
You might get better results by using a version of Vicuna with more parameters, but that would also require more memory.


Enable verbose logging through generate-manifest.sh

This patch enables verbose logging for antrea-agent and
antrea-controller when generate manifest. This will help in
troubleshooting with increased log level.
I felt troubleshooting info for incrementing log-level for antrea-agent and antrea-controller is lacking right now. I thought providing an antrea manifest through generate-manifest.sh with increased log-level is one way to help troubleshooting for developers/users. There could be an efficient approach instead of using antrea-conf configMap. Please comment on what you think.
If this approach looks ok, I will update the troubleshooting doc as part of this PR.

Codecov Report

Merging #1142 into master will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1142      +/-   ##
==========================================
+ Coverage   56.13%   56.15%   +0.02%
==========================================
  Files         105      105
  Lines       11528    11550      +22
==========================================
+ Hits         6471     6486      +15
- Misses       4489     4493       +4
- Partials      568      571       +3

Flag
Coverage Δ

#integration-tests
47.30% <ø> (-0.21%)
:arrow_down:

#unit-tests
41.63% <ø> (+0.16%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/ovs/ovsctl/ofctl.go
38.55% <0.00%> (-12.24%)
:arrow_down:

...g/agent/cniserver/interface_configuration_linux.go
33.45% <0.00%> (-1.06%)
:arrow_down:

...ntroller/networkpolicy/networkpolicy_controller.go
73.43% <0.00%> (-0.56%)
:arrow_down:

...kg/apiserver/registry/system/supportbundle/rest.go
10.71% <0.00%> (-0.06%)
:arrow_down:

pkg/ovs/ovsctl/interface.go
0.00% <0.00%> (ø)

pkg/agent/controller/networkpolicy/priority.go
89.86% <0.00%> (+1.35%)
:arrow_up:

pkg/agent/controller/networkpolicy/reconciler.go
72.22% <0.00%> (+4.29%)
:arrow_up:

/test-all

/skip-hw-offload

/test-all

/skip-hw-offload

/skip-hw-offload

As hw-offload test is not applicable, merging this patch.


spack env activate can overwrite system library which could have unwanted side effect

Currently we instruct the user to do spack env activate in their working shell.
This could have unwanted side effect. For example with icon4py the python dependency relies on an openssl version that is not compatible with the system git. One activating the env git is not working anymore in that shell.
Here are possible solutions:

inform the user to use a separate shell only for compilation
instruct user to use a wrapper script to build (like spack_install ) - so that the env is only activated inside the shell process
...

The above git example happened on balfrin when I activated the environment for the gt4py ICON. So:
. spack-c2sm/setup-env.sh
spack env activate -d icon-exclaim/build/config/cscs/spack/v<IP_ADDRESS>//balfrin_nvhpc_gpu_exclaim_verify_double/

Probably OPENSSL is affected because the gt4py version of the spack icon package indirectly depends on python 3.10, which depends on OPENSSL, overloading the system's OPENSSL.

This new openssl is coming from the upstream, i.e. software stack provided by CSCS. But indeed we sould also discussed with them if there is a reason not to use the system OPENSSL

I think we should clearly state that the spack environment is meant to build and run the packages it lists. It is not designed to run any other software or any other version of any software listed in the env.
Because the spack environment changes/adds some environment variables, any software that relies on them might be affected.

yes but not being able to run git command is quite a significant effect. So at the minimum we should say in the instruction that you always need a separate shell for working and one for compiling.

I did some investigation. Here the reason for the problem:
The error with OPENSSL happens for a simple spack load<EMAIL_ADDRESS>Has nothing to do with environments.
In comparison to the module load python/3.10.1 spack puts all the deps of python into PATH too, whereas the module-cmd simply append the prefix of python/3.10.1. Therefore the OPENSSL is used by git too.
Solution
spack load git prior to spack env activate -d icon-exclaim/build/config/cscs/spack/v<IP_ADDRESS>/balfrin_nvhpc_gpu_exclaim_verify_double/, this will use
/mch-environment/v4/linux-sles15-zen3/gcc-11.3.0/git-2.37.0-cdj5j3zhwfldqd2fb4cpfc3y5y2bi33q/bin/git instead of usr/bin/git

@muellch see the solution above

Yes, this will work. I was just thinking if switching the version of git for already existing local repositories might be problematic, but I think the git developer team makes it as compatible as possible.
But yeah, let's try.


Investigate usage of the Anchore Container Scan in GitHub Actions

Feature Description
It's worth investigating whether using Anchore Container Scan will further secure the used dependencies.
Current Behaviour
We do not use the Anchore Container Scan at this stage.
Wanted Behaviour
Validate whether we should use the Anchore Container Scan.
If it's worthwhile, it should be included in the GitHub Actions.
Possible Workarounds
Validate dependencies by hand.

We do not envision having sufficient time to work on this before our intended release of 4.7.0. Hence, moving this issue to 4.8.0.

After playing around in PR #2744, I am fairly confident the Anchore Container scan task does not provide real value to us at this stage.
As such, the pull request has been closed, and I will close this issue subsequently, with the status "won't fix."


How to support protobuf3 wrappers?

I serialized a Java object, in which the field of integer type is used, and the int32value is used in proto3. When I tried to deserialize using protobuf, I failed.What did I do wrong?
Java:
public class JavaObject {
    private Integer value1;
}

Protobuf:
message ProtoMessage {
    google.protobuf.Int32Value value1 = 1;
}

You register a delegate for com.google.protobuf.Int32Value


Array optimizations

This PR optimizes a few array methods when dealing with small arrays (less than 16 elements). These optimizations are also done by Ruby. I tried using a bigger number (like 32): in some cases the optimized version was still faster than the non-optimized one, but in some other cases it was worse. With 16 it's always faster so it's probably a good number (and why Ruby chose it).
-
Code:
require "benchmark"
require "./old_array"

size = 16

old_a1 = OldArray.new(size) { Random::DEFAULT.hex(32) }
old_a2 = OldArray.new(size) { |i| old_a1[i] }

new_a1 = Array.new(size) { |i| old_a1[i] }
new_a2 = Array.new(size) { |i| old_a1[i] }

Benchmark.ips do |x|
  x.report("old") { old_a1 - old_a2 }
  x.report("new") { new_a1 - new_a2 }
end

Result:
old 784.20k (  1.28µs) (± 7.36%)   864B/op   4.31× slower
new   3.38M (295.74ns) (± 6.56%)  32.0B/op        fastest

|
Code:
size = 8

old_a1 = OldArray.new(size) { Random::DEFAULT.hex(32) }
old_a2 = OldArray.new(size) { |i| old_a1[i] }

new_a1 = Array.new(size) { |i| old_a1[i] }
new_a2 = Array.new(size) { |i| old_a1[i] }

Benchmark.ips do |x|
  x.report("old") { old_a1 | old_a2 }
  x.report("new") { new_a1 | new_a2 }
end

Result:
old   1.68M (593.85ns) (± 5.76%)  560B/op   1.92× slower
new   3.24M (308.61ns) (±10.18%)  240B/op        fastest

&
Code:
require "benchmark"
require "./old_array"

size = 16

old_a1 = OldArray.new(size) { Random::DEFAULT.hex(32) }
old_a2 = OldArray.new(size) { |i| old_a1[i] }

new_a1 = Array.new(size) { |i| old_a1[i] }
new_a2 = Array.new(size) { |i| old_a1[i] }

Benchmark.ips do |x|
  x.report("old") { old_a1 & old_a2 }
  x.report("new") { new_a1 & new_a2 }
end

Result:
old 784.72k (  1.27µs) (± 5.35%)  0.99kB/op   1.56× slower
new   1.23M (814.30ns) (± 5.45%)    448B/op        fastest

uniq
Code:
require "benchmark"
require "./old_array"

size = 16

old_a = OldArray.new(size) { Random::DEFAULT.hex }
new_a = Array.new(size) { |i| old_a[i] }

Benchmark.ips do |x|
  x.report("old") { old_a.uniq }
  x.report("new") { new_a.uniq }
end

Result:
old   1.09M (913.60ns) (± 0.97%)  0.99kB/op   1.96× slower
new   2.14M (466.89ns) (± 2.55%)    448B/op        fastest

uniq!
Code:
require "benchmark"
require "./old_array"

values = Array.new(16) { Random::DEFAULT.hex(64) }

old_a1 = OldArray(typeof(values[0])).new
new_a1 = Array(typeof(values[0])).new

2.times do
  values.each do |value|
    old_a1 << value
    new_a1 << value
  end
end

Benchmark.ips do |x|
  x.report("old") { old_a1.uniq! }
  x.report("new") { new_a1.uniq! }
end

Result:
old 893.02k (  1.12µs) (± 8.08%)  832B/op   3.99× slower
new   3.56M (280.82ns) (± 2.64%)  0.0B/op        fastest

@asterite Is it my code or did this PR introduce a regression?
See:

failed job on CI (same spec passes on crystal latest)
relevant spec lines
def <=>
added in this PR Array#includes? call

@Sija I don't know, maybe. Do you have a case where it is broken?

I mean, show me an array method that works in a wrong way and we can fix it. Otherwise I'm a bit lost regarding what to fix.

@Sija try doing this:
Money.new(1_00, "EUR") == Money.new(1_00, "USD")

It fails with the same error. Array#& changed a bit and in the optimization it might use == or a hash lookup. It seems the problem is on your side.

It fails with the same error. Array#& changed a bit and in the optimization it might use == or a hash lookup. It seems the problem is on your side.

That's the thing, before it was using a hash lookup, whereas now it uses<=> through Array#includes? call.

@Sija try doing this:
Money.new(1_00, "EUR") == Money.new(1_00, "USD")

I know it doesn't work, but that's how it's supposed to be. Hash lookup on the other hand is suppose to work based on different semantics (comparing the attributes).

No, hash lookup uses first the hash method and then the == method. It's just that before this change a hash was built from the second array and entries were removed based on the first array so it never needed to compare two different currencies. But swap tee order of EUR and USD in that spec and I'm sure it will fail before and after this PR.
In every object you must guarantee that if two objects are equal by == then their hash code must be the same (but the reverse is not true).

But swap the order of EUR and USD in that spec and I'm sure it will fail before and after this PR.

It works in every combination, just tried.

Well, it's not our problem :-)

And with that I mean, there's no problem on our side. I can't understand why you are relying on hash for that to work. As I said, if two objects have the same hash it doesn't mean they are equal. If for some reason the hash seed (which changes from program to program) makes it so that "USD".hash == "EUR".hash then that spec will start failing (depending on the order of the elements in the array).


Update R docs and doc-site

Addresses #670 #668 #669

Just one question, I noticed the new examples are generally using a = b rather than the more-frequently-seen a <- b for assignment. I don't feel strongly, but just checking it's intentional? (LGTM otherwise)


[PE-D][Tester A] UG typo in edit function

There is a typo where the level field in the edit function can be used multiple times.

Labels: type.DocumentationBug severity.VeryLow
original: donwong2308/ped#10

Update UG accordingly


Error:  no implicit conversion of nil into String

When switching from jekyll-paginate to octopress so I can paginate collections I get the following error
Configuration file: /home/.../website/_config.yml
       Deprecation: You appear to have pagination turned on, but you haven't included the `jekyll-paginate` gem. Ensure you have `gems: [jekyll-paginate]` in your configuration file.
            Source: /home/.../website
       Destination: /home/.../website/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
       Deprecation: Collection#reverse should be called on the #docs array directly.
                    Called by /var/lib/gems/2.1.0/gems/octopress-paginate-1.1.2/lib/octopress-paginate.rb:136:in `collection'.
  Liquid Exception: no implicit conversion of nil into String in _tutorials/2016-01-10-welcome-to-jekyll.markdown
jekyll 3.0.1 | Error:  no implicit conversion of nil into String

Here is my config file
# Blog settings
permalink: /blog/:year/:month/:day/:title
paginate: 6
paginate_path: "/blog/page/:num"

# Tutorial settings
collections:
  - tutorials

# Plugins
gems:
- octopress-paginate

Here is my tutorials index file tutorials/index.html
---
layout: page
nav_layout: tutorial
title: Tutorial Index
paginate:
  collection: tutorials
  permalink: /tutorials/:year/:month/:day/:title
---
{% for tutorial in paginator.tutorials %}
        ...
{% endfor %}

Then I have markdown files in the folder _tutorials
I think this is related to https://github.com/jekyll/jekyll/issues/3929 and https://github.com/octopress/hooks/issues/13
Maybe relevant info
Ubuntu 15.10
ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]
jekyll 3.0.1

Yeah, I have exactly the same issue.
I was having this problem with my jekyll site and tried to narrow it down by stripping everything out bit by bit. I got down to stripping everything out and it was still there.
Turns this was nothing to do with my site, but happens with a fresh, brand new install of jekyll with the octopress-pagination gem specified in the Gemfile.
Here are the steps I took to create reproduce the issue:
mkdir test
cd test
bundle init
echo "gem 'jekyll'" >> Gemfile
echo "group :jekyll_plugins do" >> Gemfile
echo "  gem 'octopress-paginate'" >> Gemfile
echo "end" >> Gemfile
bundle install
bundle exec jekyll new site
bundle exec jekyll build -s site --trace

Here is the jekyll output when using ruby2.3 on Ubuntu Trusty LTS:
New jekyll site installed in /home/vagrant/2.3/test/site.
Configuration file: site/_config.yml
            Source: site
       Destination: /home/vagrant/2.3/test/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
  Liquid Exception: no implicit conversion of nil into String in _posts/2016-01-14-welcome-to-jekyll.markdown
/home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll.rb:156:in `expand_path': no implicit conversion of nil into String (TypeError)
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll.rb:156:in `sanitized_path'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:376:in `block in in_source_dir'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:375:in `each'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:375:in `reduce'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:375:in `in_source_dir'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/liquid_renderer.rb:16:in `file'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/renderer.rb:105:in `render_liquid'
	from /home/vagrant/.gem/ruby/2.3.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:313:in `render_liquid'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/renderer.rb:60:in `run'
	from /home/vagrant/.gem/ruby/2.3.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:306:in `run'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/excerpt.rb:76:in `output'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/excerpt.rb:60:in `to_s'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/document.rb:367:in `to_liquid'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/renderer.rb:37:in `run'
	from /home/vagrant/.gem/ruby/2.3.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:306:in `run'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:171:in `block (2 levels) in render'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:169:in `each'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:169:in `block in render'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:168:in `each'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:168:in `render'
	from /home/vagrant/.gem/ruby/2.3.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:197:in `render'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/site.rb:59:in `process'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/command.rb:28:in `process_site'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/commands/build.rb:60:in `build'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/commands/build.rb:35:in `process'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	from /home/vagrant/.gem/ruby/2.3.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `block in execute'
	from /home/vagrant/.gem/ruby/2.3.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `each'
	from /home/vagrant/.gem/ruby/2.3.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `execute'
	from /home/vagrant/.gem/ruby/2.3.0/gems/mercenary-0.3.5/lib/mercenary/program.rb:42:in `go'
	from /home/vagrant/.gem/ruby/2.3.0/gems/mercenary-0.3.5/lib/mercenary.rb:19:in `program'
	from /home/vagrant/.gem/ruby/2.3.0/gems/jekyll-3.0.1/bin/jekyll:17:in `<top (required)>'
	from /home/vagrant/.gem/ruby/2.3.0/bin/jekyll:23:in `load'
	from /home/vagrant/.gem/ruby/2.3.0/bin/jekyll:23:in `<main>'

Here's my Gemfile.lock:
GEM
  remote: https://rubygems.org/
  specs:
    colorator (0.1)
    ffi (1.9.10)
    jekyll (3.0.1)
      colorator (~> 0.1)
      jekyll-sass-converter (~> 1.0)
      jekyll-watch (~> 1.1)
      kramdown (~> 1.3)
      liquid (~> 3.0)
      mercenary (~> 0.3.3)
      rouge (~> 1.7)
      safe_yaml (~> 1.0)
    jekyll-sass-converter (1.4.0)
      sass (~> 3.4)
    jekyll-watch (1.3.0)
      listen (~> 3.0)
    kramdown (1.9.0)
    liquid (3.0.6)
    listen (3.0.5)
      rb-fsevent (>= 0.9.3)
      rb-inotify (>= 0.9)
    mercenary (0.3.5)
    octopress-hooks (2.6.1)
      jekyll (>= 2.0)
    octopress-paginate (1.1.2)
      octopress-hooks
    rb-fsevent (0.9.7)
    rb-inotify (0.9.5)
      ffi (>= 0.5.0)
    rouge (1.10.1)
    safe_yaml (1.0.4)
    sass (3.4.21)

PLATFORMS
  ruby

DEPENDENCIES
  jekyll
  octopress-paginate

BUNDLED WITH
   1.11.2

Note, I have tried this out in the following configurations with the same result:

OS X 10.11, ruby 2.3
Ubuntu Trusty LTS, ruby 2.0, 2.1, 2.2 and 2.3.

Heres is the vagrant file I used to test this on Ubunt Trusty LTS: https://gist.github.com/whatsma/5fc072db64c87c4907a6 for reference.

This appears to be related to https://github.com/octopress/hooks/pull/14; I am hoping a fresh release of octopress-hooks with that fix included will also fix the issue.

I am still having this exact same issue with a fresh install of jekyll and octopress, for what it's worth.
Liquid Exception: no implicit conversion of nil into String in _posts/2014-05-14-example-post.md
/usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll.rb:156:in `expand_path': no implicit conversion of nil into String (TypeError)
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll.rb:156:in `sanitized_path'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:355:in `block in in_source_dir'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:354:in `each'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:354:in `reduce'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:354:in `in_source_dir'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/liquid_renderer.rb:16:in `file'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/renderer.rb:106:in `render_liquid'
	from /usr/local/lib/ruby/gems/2.2.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:311:in `render_liquid'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/renderer.rb:61:in `run'
	from /usr/local/lib/ruby/gems/2.2.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:306:in `run'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:171:in `block (2 levels) in render'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:169:in `each'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:169:in `block in render'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:168:in `each'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:168:in `render'
	from /usr/local/lib/ruby/gems/2.2.0/gems/octopress-hooks-2.6.1/lib/octopress-hooks.rb:197:in `render'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/site.rb:59:in `process'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/command.rb:26:in `process_site'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/commands/build.rb:60:in `build'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/commands/build.rb:33:in `process'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/lib/jekyll/commands/serve.rb:34:in `block (2 levels) in init_with_program'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `call'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `block in execute'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `each'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary/command.rb:220:in `execute'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary/program.rb:42:in `go'
	from /usr/local/lib/ruby/gems/2.2.0/gems/mercenary-0.3.5/lib/mercenary.rb:19:in `program'
	from /usr/local/lib/ruby/gems/2.2.0/gems/jekyll-3.1.1/bin/jekyll:13:in `<top (required)>'
	from /usr/local/bin/jekyll:23:in `load'
	from /usr/local/bin/jekyll:23:in `<main>'

I pulled lib from octpopress-hooks https://github.com/octopress/hooks/commit/9ec9efcbe8e2aa4d79d2a35543a29b51f3e6feb2
and issue no longer occurs on jekyll build.
Thanks@parkr !

@parkr: this must be the same issue as #19


ionic2: loading: interface LoadingOptions cssClass ?

Short description of the problem:
export interface LoadingOptions {
  spinner?: string;
  content?: string;
  showBackdrop?: boolean;
  dismissOnPageChange?: boolean;
  delay?: number;
  duration?: number;
}

cssClass ?
Run ionic info from terminal/cmd prompt: (paste output below)
Ionic Framework Version: 2.0.0-beta.6

Thanks for opening this issue with us! Would you be able to provide some more information on the issue you are having? Thanks!

ionic/components/loading/loading.ts (224 Line)
cssClass
It seems to be missing.

Thank you for the issue. This was fixed on issue https://github.com/driftyco/ionic/issues/6365 and will be in the beta.7 release. :smile:


版本依赖问题

meishi

You have a comma (,) after ShouldAutoSize like PhpStorm indicates.
Also please try to use English titles when opening issues.


Add full Windows support

Fixes #47

@dstebila When this merges, please also feel free to turn off the Windows VM you kindly made available for helping me test this platform.

I only read the last clause of that sentence, so I already turned it off. 😝 Let me know if you need it back on.


Merge no-dgram into no-sock

If you have sockets, it is now assumed you have both TCP and UDP and you cannot disable just datagram.
Checklist
Thank you for your contribution. Please answer the following:

[X] I acknowledge that I am authorized to submit this code under
the terms of the Apache License
[ ] All new public APIs are documented
[ ] All new public functions have tests
[ ] All new public functions have fuzzing tests

The problem is that I was thinking UDP should not be a separate thing from sockets, and OpenSSL originally merged both DTLS and UDP under the same kind of ifdef control.  I need to rethink this, I think.

And it kept failing because I kept getting the damned perl syntax wrong !$defined{"foo"} sometimes and undefined("foo") other times.  Sigh

clean build, ready for review