Support 64-bit contig_id

This is needed to index and query large collections containing more than 2^32 contigs.
An application of this is in Fulgor, when indexing very heterogenous collections.
Tagging @rickbeeloo here and this issue https://github.com/jermp/fulgor/issues/16.
It looks like all that is needed is to refactor these two small points a little bit.

https://github.com/jermp/sshash/blob/master/include/builder/util.hpp#L77
https://github.com/jermp/sshash/blob/master/include/util.hpp#L21

Another point is -- for very large scale indexing -- I would suggest to use a partitioned PTHash,
rather than a single PTHash, to lower the construction time.

The branch https://github.com/jermp/sshash/tree/partitioned_phf already switches to partitioned PTHash and defaults to 128-bit hashes. The support for 64-bit contigs should be reflected in that branch.
CC: @rickbeeloo.

I just added to code I have to https://github.com/jermp/sshash/pull/36, to keep it somewhere.
Edit, just saw your email, edit2, now also your comment above haha.
That code worked for our queries but just going over some changes to discuss what would be best

We took all the threads to process all chunks

        mphf_config.num_threads = std::thread::hardware_concurrency();
        mphf_config.num_partitions = std::thread::hardware_concurrency();

For our big dataset that works, but when testing I saw it's possible that the partitions get too small. I would need your help to add this check (we don't need it now but if others use it that should be there). I belief it was thrown in PThash, here, so we need to balance threads ~ partitions

Any opinions about the lookup_result struct

   uint64_t kmer_orientation;
   uint64_t contig_id;
   uint64_t contig_size;

kmer_id_in_contig and contig_id can exceed u32.  contig_size however, won't/shouldn't exceed u32, this was restrained to u32 in the code I removed, originally here. I think best would then chaning it back to u32, and keep the check_contig_size check but just not check the pieces.size() == 1ULL << 32 (i.e. only contig size, not the number of contigs). For kmer_orientation we have either 0/1 so no need for u64 (p.s. just curious if you made that u32 for a reason compared to a bool).
I think that's it for sshash, will change the above after your suggestions, and then move to the small Fulgor edits

Aah in the partitioned_phf branch you already added that check :)

Then it's  just these tiny changes left https://github.com/jermp/sshash/compare/partitioned_phf...rickbeeloo:sshash:partitioned_phf.
For Fulgor we had some more changes, just u32 > u64, especially regarding the "colors" (see https://github.com/jermp/fulgor/compare/main...rickbeeloo:fulgor:main). You refactored it to mcDBG, at a glance it seems that I can refork mcDBG and just copy the changes I made before as you separated the "meta" and "normal" functions (like intersect and meta_intersect), but not sure if there is other mcDBG code I have to consider? I will soon now ;p but if you already know if/where that would help

However, what is the particular reason now to enforce contig_size to be less than 2^32? Before, the reason was to also have kmer_id_in_contig < 2^32. But now, we should probably make it a uint64_t as well.

I was just thinking this cause bacterial or human genomes would never exceed  2^32-1. It would be a good sanity check for the input - not being some unexpected concatenation. However, plant genomes could actually exceed it, so then lets just make it uint64_t.

Yeah!

Hey @jermp, just trying this current branch (without my changes) on some of the example data but I get some errors for the checks. Maybe I'm missing something or giving the wrong input?
 Human chromsome test (this branch)
The chromosome you linked on sshash
wget http://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz
Instead of BCALM I used Cuttlefish for the unitigs:
cuttlefish build -s Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz -t 2 -o cuttlefish_unitigs.fasta
Then:
git clone https://github.com/jermp/sshash -b partitioned_phf --recursive
build etc.
Then indexing
./sshash/build/sshash build -i cuttlefish_unitigs.fasta.fa -k 31 -m 21 -o output.txt --check
It will fail the check:
SPACE BREAKDOWN:
  minimizers: 0.465234 [bits/kmer]
  pieces: 0.0548784 [bits/kmer]
  num_super_kmers_before_bucket: 0.264572 [bits/kmer]
  offsets: 4.60858 [bits/kmer]
  strings: 2.28666 [bits/kmer]
  skew_index: 0.00131327 [bits/kmer]
  weights: 1.69477e-05 [bits/kmer]
    weight_interval_values: 2.94743e-06 [bits/kmer]
    weight_interval_lengths: 1.10529e-05 [bits/kmer]
    weight_dictionary: 2.94743e-06 [bits/kmer]
  --------------
  total: 7.68126 [bits/kmer]
checking correctness of access and positive lookup...
wrong id assigned
kmer 'AATCACTGGGACCCCTTTGACCCTCAGGCAA' not found!
ERROR: got orientation 1 but expected 0
ERROR: got curr.kmer_id<PHONE_NUMBER>3709551615 but expected 521
ERROR: got curr.kmer_id_in_contig<PHONE_NUMBER> but expected something <<PHONE_NUMBER>
ERROR: got curr.contig_id<PHONE_NUMBER> but expected 4
ERROR: got curr.kmer_id_in_contig<PHONE_NUMBER> but expected 0
Segmentation fault (core dumped)

 Simple example test
This actually is also reproducible with simple input:
test.fasta
>test1
ATCGATCAGCTACGACTAGCATCGACATCAGC
>test2
ATCAGCTACGATCAGCTACGATCGACTAGCAGCATCGACTACGATC
>test3
ATCATCTACGATCATCTAC

Then getting the unitigs with cuttlefish again, etc:
cuttlefish build -s test.fasta -t 2 -o small_unitigs
Gives the unitigs:
>9
ATCGATCAGCTACGACTAGCATCGACATCAGC
>0
ATCAGCTACGATCAGCTACGATCGACTAGCAGCATCGACTACGATC

Running sshash:
./sshash/build/sshash build -i small_unitigs.fa -k 4 -m 2 -o output.txt --check

Long output here
k = 5, m = 2, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file 'small_unitigs.fa'...
m_buffer_size 29411764
sorting buffer...
saving to file './sshash.tmp.run_1702480566097403174.minimizers.0.bin'...
read 2 sequences, 78 bases, 70 kmers
num_kmers 70
num_super_kmers 26
num_pieces 3 (+0.342857 [bits/kmer])
=== step 1: 'parse_file' 0.018057 [sec] (257957 [ns/kmer])
=== step 2: 'build_minimizers' 0.01381 [sec] (197286 [ns/kmer])
bits_per_offset = ceil(log2(83)) = 7
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1702480566129288620.bucket_pairs.0.bin'...
num_singletons 0/4 (0%)
=== step 3: 'build_index' 0.001 [sec] (14285.7 [ns/kmer])
max_num_super_kmers_in_bucket 8
log2_max_num_super_kmers_in_bucket 3
num_buckets_in_skew_index 0/4(0%)
=== step 4: 'build_skew_index' 0.00016 [sec] (2285.71 [ns/kmer])
=== total_time 0.033027 [sec] (471814 [ns/kmer])
total index size: 0.00107 [MB]
SPACE BREAKDOWN:
minimizers: 49.6 [bits/kmer]
pieces: 18.7429 [bits/kmer]
num_super_kmers_before_bucket: 17.6 [bits/kmer]
offsets: 7.31429 [bits/kmer]
strings: 4.57143 [bits/kmer]
skew_index: 0.914286 [bits/kmer]
weights: 21.0286 [bits/kmer]
  weight_interval_values: 3.65714 [bits/kmer]
  weight_interval_lengths: 13.7143 [bits/kmer]
  weight_dictionary: 3.65714 [bits/kmer]
--------------
total: 122.286 [bits/kmer]
checking correctness of access and positive lookup...
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 2
ERROR: got curr.kmer_id_in_contig 1 but expected 2
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 2
ERROR: got curr.kmer_id_in_contig 19 but expected 0
ERROR: got curr.kmer_id 3 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 3 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 20
ERROR: got curr.kmer_id_in_contig 1 but expected 20
ERROR: got curr.kmer_id 21 but expected 2
ERROR: got curr.kmer_id_in_contig 21 but expected 2
wrong id assigned
ERROR: got curr.kmer_id 4 but expected 26
ERROR: got curr.kmer_id_in_contig 4 but expected 26
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 4 but expected 6
ERROR: got curr.kmer_id_in_contig 4 but expected 6
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 36
ERROR: got curr.kmer_id_in_contig 19 but expected 8
wrong id assigned
ERROR: got curr.kmer_id 3 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 3 but expected 0
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 36
ERROR: got curr.kmer_id_in_contig 19 but expected 8
ERROR: got curr.kmer_id 47 but expected 48
ERROR: got curr.kmer_id_in_contig 19 but expected 20
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 1 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 21 but expected 2
ERROR: got curr.kmer_id_in_contig 21 but expected 2
wrong id assigned
ERROR: got curr.kmer_id 12 but expected 22
ERROR: got curr.kmer_id_in_contig 12 but expected 22
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 55 but expected 17
ERROR: got curr.kmer_id_in_contig 27 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 17 but expected 58
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 17 but expected 0
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 0 but expected 20
ERROR: got curr.kmer_id_in_contig 0 but expected 20
wrong id assigned
ERROR: got curr.kmer_id 21 but expected 1
ERROR: got curr.kmer_id_in_contig 21 but expected 1
wrong id assigned
ERROR: got curr.kmer_id 12 but expected 22
ERROR: got curr.kmer_id_in_contig 12 but expected 22
wrong id assigned
ERROR: got curr.kmer_id 65 but expected 14
ERROR: got curr.kmer_id_in_contig 37 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 9 but expected 66
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 9 but expected 0
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 2 but expected 36
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 2 but expected 0
checked 70 kmers
EVERYTHING OK!
checking correctness of negative lookup with random kmers...
kmer 'CGATG' found!
kmer 'TGATG' found!
EVERYTHING OK!
checking correctness of navigational queries for kmers...
checked 70 kmers
EVERYTHING OK!
checking correctness of navigational queries for contigs...
checked 2 contigs
EVERYTHING OK!
checking correctness of iterator...
EVERYTHING OK!
2023-12-13 16:16:06: saving data structure to disk...
2023-12-13 16:16:06: DONE

Yes, you’re missing that unitigs are built for a given k. So of course SSHash works correctly only for the same k you used to build the unitigs. Note that cuttlefish uses k=27 by default. So that’s why k=31 and k=4 do not work as expected.You have to use the same k for both cuttlefish and SSHash.Best,-GiulioOn 13/12/2023, at 16:49, rickbeeloo @.***> wrote:
Hey @jermp, just trying this current branch (without my changes) on some of the example data but I get some errors for the checks. Maybe I'm missing something or giving the wrong input?
Human chromsome test (this branch)
The chromosome you linked on sshash
wget http://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz
Instead of BCALM I used Cuttlefish for the unitigs:
cuttlefish build -s Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz -t 2 -o cuttlefish_unitigs.fasta
Then:
git clone https://github.com/jermp/sshash -b partitioned_phf --recursive
build etc.
Then indexing
./sshash/build/sshash build -i cuttlefish_unitigs.fasta.fa -k 31 -m 21 -o output.txt --check
It will fail the check:
SPACE BREAKDOWN:
minimizers: 0.465234 [bits/kmer]
pieces: 0.0548784 [bits/kmer]
num_super_kmers_before_bucket: 0.264572 [bits/kmer]
offsets: 4.60858 [bits/kmer]
strings: 2.28666 [bits/kmer]
skew_index: 0.00131327 [bits/kmer]
weights: 1.69477e-05 [bits/kmer]
weight_interval_values: 2.94743e-06 [bits/kmer]
weight_interval_lengths: 1.10529e-05 [bits/kmer]
weight_dictionary: 2.94743e-06 [bits/kmer]
total: 7.68126 [bits/kmer]
checking correctness of access and positive lookup...
wrong id assigned
kmer 'AATCACTGGGACCCCTTTGACCCTCAGGCAA' not found!
ERROR: got orientation 1 but expected 0
ERROR: got curr.kmer_id<PHONE_NUMBER>3709551615 but expected 521
ERROR: got curr.kmer_id_in_contig<PHONE_NUMBER> but expected something <<PHONE_NUMBER>
ERROR: got curr.contig_id<PHONE_NUMBER> but expected 4
ERROR: got curr.kmer_id_in_contig<PHONE_NUMBER> but expected 0
Segmentation fault (core dumped)
Simple example test
This actually is also reproducible with simple input:
test.fasta

test1
ATCGATCAGCTACGACTAGCATCGACATCAGC
test2
ATCAGCTACGATCAGCTACGATCGACTAGCAGCATCGACTACGATC
test3
ATCATCTACGATCATCTAC

Then getting the unitigs with cuttlefish again, etc:
cuttlefish build -s test.fasta -t 2 -o small_unitigs
Gives the unitigs:

9
ATCGATCAGCTACGACTAGCATCGACATCAGC
0
ATCAGCTACGATCAGCTACGATCGACTAGCAGCATCGACTACGATC

Running sshash:
./sshash/build/sshash build -i small_unitigs.fa -k 4 -m 2 -o output.txt --check
Long output here
k = 5, m = 2, seed = 1, l = 6, c = 3, canonical_parsing = false, weighted = false
reading file 'small_unitigs.fa'...
m_buffer_size 29411764
sorting buffer...
saving to file './sshash.tmp.run_1702480566097403174.minimizers.0.bin'...
read 2 sequences, 78 bases, 70 kmers
num_kmers 70
num_super_kmers 26
num_pieces 3 (+0.342857 [bits/kmer])
=== step 1: 'parse_file' 0.018057 [sec] (257957 [ns/kmer])
=== step 2: 'build_minimizers' 0.01381 [sec] (197286 [ns/kmer])
bits_per_offset = ceil(log2(83)) = 7
m_buffer_size 20833333
sorting buffer...
saving to file './sshash.tmp.run_1702480566129288620.bucket_pairs.0.bin'...
num_singletons 0/4 (0%)
=== step 3: 'build_index' 0.001 [sec] (14285.7 [ns/kmer])
max_num_super_kmers_in_bucket 8
log2_max_num_super_kmers_in_bucket 3
num_buckets_in_skew_index 0/4(0%)
=== step 4: 'build_skew_index' 0.00016 [sec] (2285.71 [ns/kmer])
=== total_time 0.033027 [sec] (471814 [ns/kmer])
total index size: 0.00107 [MB]
SPACE BREAKDOWN:
minimizers: 49.6 [bits/kmer]
pieces: 18.7429 [bits/kmer]
num_super_kmers_before_bucket: 17.6 [bits/kmer]
offsets: 7.31429 [bits/kmer]
strings: 4.57143 [bits/kmer]
skew_index: 0.914286 [bits/kmer]
weights: 21.0286 [bits/kmer]
weight_interval_values: 3.65714 [bits/kmer]
weight_interval_lengths: 13.7143 [bits/kmer]
weight_dictionary: 3.65714 [bits/kmer]
total: 122.286 [bits/kmer]
checking correctness of access and positive lookup...
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 2
ERROR: got curr.kmer_id_in_contig 1 but expected 2
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 2
ERROR: got curr.kmer_id_in_contig 19 but expected 0
ERROR: got curr.kmer_id 3 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 3 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 20
ERROR: got curr.kmer_id_in_contig 1 but expected 20
ERROR: got curr.kmer_id 21 but expected 2
ERROR: got curr.kmer_id_in_contig 21 but expected 2
wrong id assigned
ERROR: got curr.kmer_id 4 but expected 26
ERROR: got curr.kmer_id_in_contig 4 but expected 26
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 4 but expected 6
ERROR: got curr.kmer_id_in_contig 4 but expected 6
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 36
ERROR: got curr.kmer_id_in_contig 19 but expected 8
wrong id assigned
ERROR: got curr.kmer_id 3 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 3 but expected 0
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 47 but expected 36
ERROR: got curr.kmer_id_in_contig 19 but expected 8
ERROR: got curr.kmer_id 47 but expected 48
ERROR: got curr.kmer_id_in_contig 19 but expected 20
wrong id assigned
ERROR: got orientation 0 but expected 1
ERROR: got curr.kmer_id 1 but expected 48
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 1 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 21 but expected 2
ERROR: got curr.kmer_id_in_contig 21 but expected 2
wrong id assigned
ERROR: got curr.kmer_id 12 but expected 22
ERROR: got curr.kmer_id_in_contig 12 but expected 22
wrong id assigned
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 55 but expected 17
ERROR: got curr.kmer_id_in_contig 27 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 17 but expected 58
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 17 but expected 0
wrong id assigned
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 0 but expected 20
ERROR: got curr.kmer_id_in_contig 0 but expected 20
wrong id assigned
ERROR: got curr.kmer_id 21 but expected 1
ERROR: got curr.kmer_id_in_contig 21 but expected 1
wrong id assigned
ERROR: got curr.kmer_id 12 but expected 22
ERROR: got curr.kmer_id_in_contig 12 but expected 22
wrong id assigned
ERROR: got curr.kmer_id 65 but expected 14
ERROR: got curr.kmer_id_in_contig 37 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 9 but expected 66
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 9 but expected 0
wrong id assigned
wrong id assigned
ERROR: got curr.kmer_id 35 but expected 11
ERROR: got curr.kmer_id_in_contig 7 but expected 0
wrong id assigned
ERROR: got curr.kmer_id 2 but expected 36
ERROR: got curr.contig_id 0 but expected 2
ERROR: got curr.kmer_id_in_contig 2 but expected 0
checked 70 kmers
EVERYTHING OK!
checking correctness of negative lookup with random kmers...
kmer 'CGATG' found!
kmer 'TGATG' found!
EVERYTHING OK!
checking correctness of navigational queries for kmers...
checked 70 kmers
EVERYTHING OK!
checking correctness of navigational queries for contigs...
checked 2 contigs
EVERYTHING OK!
checking correctness of iterator...
EVERYTHING OK!
2023-12-13 16:16:06: saving data structure to disk...
2023-12-13 16:16:06: DONE
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

How did I miss that... Thanks!
Then I keep that as as simple test for everything:
wget http://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz
cuttlefish build -s Homo_sapiens.GRCh38.dna.chromosome.13.fa.gz -t 2 -k 31 -o cuttlefish_unitigs.fasta
sshash build -i cuttlefish_unitigs.fasta.fa -k 31 -m 21 -o output.txt --check

(that indeed works)
Any suggestions on how we will properly test u64 sshash/fulgor? We can't really run --check on our big dataset to test sshash/fulgor after editing as it will take ages. We could generate 2^32 kmers (k>16), and perhaps make some concatenation of them to have long "contigs" and test navigational queries. Alternatively, we can add big numbers to the variables in sshash directly but perhaps not the safest test idea (especially not if I do it haha). Let me know :)

No problem! Just keep in mind that a dBG is built for a specific value of k, and that value only must be used.
For the testing, well we could just let in run :) Yes, it will probably take days, if not weeks, on that index.
Otherwise, we sample some queries and check only them.

(Just as a note, we shouldn't forget to add something about increasing ulimit to the readme of this fork, just encountered this again, so probably that will happen often for big datasets)

Thanks @rickbeeloo for reporting! I'm adding this to the readme of this branch: https://github.com/jermp/sshash/tree/partitioned_phf.

Ok, done as of https://github.com/jermp/sshash/commit/119accf0c98eaedf0036f41d2c61bca1d94c661c.
I've added the same note in Fulgor as well.

Hey :)
Fulgor
Regarding the colors (like here but also here in meta for example) which refers to the origin of the untigs, or the partitions for the mcDBG. These colors can technically also exceed u32, although maybe unlikely to happen. To be safe we also up that to u64?
SShash
Like you mentioned at the start this is just only two changes now. I will first test that on 2^32 unique 17-mers, that is 129GB so not that gigigantic, then at least --check will be faster. It can't check navigational queries really as it's just 17 mers in the FASTA but we can then later sample that on the big index. Just thought it would be quite sad to wait weeks just to realize membership queries already failed.
Could you elaborate on c and l I understand they reflect the space/time trade-off but I couldn't get from the docs if I should put these low or high if I don't care about space. We have tons of space, so I would prefer faster construction over minimizing storage/RAM (for testing)
Have a good Saturday :)

I now made these changes: https://github.com/jermp/fulgor/compare/main...rickbeeloo:fulgor:main.
Some notes:

I think the num_docs will realistically never be 2^32+, that would be a lot of files. So I left this at uint32. Related to this we have the colors which come from GGCAT and also reflect the input files, for the same reason I left these uint32. If we would actually want to have uint64 colors that would require more refactoring, and also running GGCAT with uint64 (I suppose here)
In the meta colored paper you say you cluster the references based on the shared unitigs. From this I assume the partitions are always <= num_docs, is that correct? If so we can leave that uint32 as well in line with 1
We spoke about having big contigs, like plants, where a kmer can have a position on the contig 2^32+. Hence I changed the code here with contigIdx_ and contigLen_  being uint64 but there are other things like the offset  and invalid_cid probably having to be replaced by sshash::constants::invalid_uint64 etc. But I'm not sure how this skipping exactly works so I could use some help there :)

Hey @jermp  :)
The --check for sshash on 2^32 17-mers finsihed without errors so we know at least the lookups go well.
For fulgor, I get some error regarding:
../src/psa/full_intersection.cpp:225:9: error: no match for ‘operator=’ (operand types are ‘__gnu_cxx::__normal_iterator<long unsigned int*, std::vector<long unsigned int> >’ and ‘__gnu_cxx::__normal_iterator<unsigned int*, std::vector<unsigned int> >’)
  225 |     end = std::unique(tmp.begin(), tmp.end());

As I said before I'm not that familiar with C++ so I don't think it's smart if I just start adding code to solve that. Could you try to compile: https://github.com/rickbeeloo/fulgor (it does use the partitioned_hpf branch already but without the two changes we need for int64 - but don't think that is the problem)
Can you look with me what still goes wrong?

Hi Rick,
the problem looks like you forgot to change std::vector<uint32_t> into std::vector<uint64_t>.
Will try to compile your branch to verify this.

Sure, curious to see what was wrong. It might not be the only thing😅

This line causes the problem https://github.com/rickbeeloo/fulgor/blob/main/src/psa/full_intersection.cpp#L225
because end has type std::vector<uint64_t>::iterator (after making unitig ids 64-bit ints) and we try to assign it to the result of std::unique with vector<uint32_t>, so something that has type std::vector<uint32_t>::iterator. Just cannot convert iterator types.
Below the correct code:
template <typename ColorClasses>
void index<ColorClasses>::intersect_unitigs(std::vector<uint64_t>& unitig_ids,
                                            std::vector<uint32_t>& colors) const {
    /* here we use it to hold the color class ids;
       in meta_intersect we use it to hold the partition ids */
    std::vector<uint32_t> tmp;
    std::vector<typename ColorClasses::iterator_type> iterators;

    /* deduplicate unitig_ids */
    std::sort(unitig_ids.begin(), unitig_ids.end());
    auto end_unitig_ids = std::unique(unitig_ids.begin(), unitig_ids.end());
    tmp.reserve(end_unitig_ids - unitig_ids.begin());
    for (auto it = unitig_ids.begin(); it != end_unitig_ids; ++it) {
        uint64_t unitig_id = *it;
        uint32_t color_class_id = u2c(unitig_id);
        tmp.push_back(color_class_id);
    }

    /* deduplicate color class ids */
    std::sort(tmp.begin(), tmp.end());
    auto end_tmp = std::unique(tmp.begin(), tmp.end());
    iterators.reserve(end_tmp - tmp.begin());
    for (auto it = tmp.begin(); it != end_tmp; ++it) {
        uint64_t color_class_id = *it;
        auto fwd_it = m_ccs.colors(color_class_id);
        iterators.push_back(fwd_it);
    }

    tmp.clear();  // don't need color class ids anymore
    if constexpr (ColorClasses::meta_colored) {
        meta_intersect(iterators, colors, tmp);
    } else {
        intersect(iterators, colors, tmp);
    }

    assert(util::check_intersection(iterators, colors));
}

Cool! Then that should be it for Fulgor :)
I mentioned in the sshash pr that I was planning to use the unique kmers to generate kmers with random sequence suffixes. However, Looking at our dataset again the unitigs were 477GB whereas randomly generating data would also be between 150-200GB depending on the suffix length. So I will just run Fulgor on the entire dataset.

Are there any things you think I forgot in https://github.com/rickbeeloo/fulgor before I run it?
Now we rerun Fulgor again to be sure everything was constructed correctly, should we use meta colored?

Are there any things you think I forgot in https://github.com/rickbeeloo/fulgor before I run it?

Looks good to me. Try it!

Now we rerun Fulgor again to be sure everything was constructed correctly, should we use meta colored?

Yeah, why not? But first, we have to build a "normal" Fulgor index which we then transform into the meta-colored version.

It's building🤞
(Well it already was but forgot the ulimit -n 2048 again 😅)

When testing Fulgor on the mock data (kmers of 17 with 0..35 random nts after it) I kept getting the "seed did not work" from pthash. That probably has to do with the data and not the changes but to prevent that one should just try another seed?

Ok, done as of https://github.com/jermp/sshash/commit/fbe00d9f1ca4fa875177deba11a7e9b9e5c2b71a.
(Not yet merged in master. For now in partitioned_phf branch.)

Hi @rickbeeloo,

When testing Fulgor on the mock data (kmers of 17 with 0..35 random nts after it) I kept getting the "seed did not work" from pthash. That probably has to do with the data and not the changes but to prevent that one should just try another seed?

I'm not sure what you did here, but this is surely due to repeated kmers in the input.

Hi @jermp, I will check it again.
I'm now using https://github.com/jermp/sshash/commit/fbe00d9f1ca4fa875177deba11a7e9b9e5c2b71a in my branch https://github.com/rickbeeloo/fulgor/tree/skipcdbg. Just restarted fulgor for the big dataset (cause it failed on an assert you added, which I missed).
Btw in that branch I hardcoded skipping the building of the cdbg. I think using an already existing GGCAT output is neat to have, especially for these bigger datasets.
Lets see if it makes it through this time 🙏

Hi Rick, ok good.
Well, you built the whole index once, so we know for sure it will make it :)

I think using an already existing GGCAT output is neat to have, especially for these bigger datasets.

Yes, absolutely!


[Feature Request] Adding prompt template function

Is it possible to implement a function that users can create/use prompt templates (filling in predetermined sections of a prewritten text template) similar to Chatbot UI(https://github.com/mckaywrigley/chatbot-ui)?
It would be very convenient for users to utilize LLM.
Thank you for your time and trouble.

Please use customized bot feature. If you have any trouble to use it, please re-create a ticket. thank you


Update discord.js to v13

Update discord.js from v12 to v13.
See migration guide here: https://discordjs.guide/additional-info/changes-in-v13.html#before-you-start
Breaking Changes for the Cadency music bot

Drop NodeJS v14 support (requires now at least v16)

Update to discord.js v13 was done with #177 on the develop branch and will be released with v3.0.0 in the future.


SamsungTV component stopped working

Home Assistant release with the issue:
0.91.4
Last working Home Assistant release (if known):
0.91.1
Operating environment (Hass.io/Docker/Windows/etc.):
Hassbian
Component/platform:
samsungtv
Description of problem:
The component does not work anymore and showing the error below
Problem-relevant configuration.yaml entries:
  - platform: samsungtv
    host: xxx.xxx.x.xx
    name: TV
    port: 8001

Traceback (if applicable):
Update for media_player.tv fails
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 220, in async_update_ha_state
    await self.async_device_update()
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 379, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/samsungtv/media_player.py", line 132, in update
    self.send_key("KEY")
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/samsungtv/media_player.py", line 153, in send_key
    self.get_remote().control(key)
  File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/samsungtv/media_player.py", line 138, in get_remote
    self._remote = self._remote_class(self._config)
  File "/srv/homeassistant/lib/python3.5/site-packages/samsungctl/remote.py", line 11, in __init__
    self.remote = RemoteWebsocket(config)
  File "/srv/homeassistant/lib/python3.5/site-packages/samsungctl/remote_websocket.py", line 30, in __init__
    self._read_response()
  File "/srv/homeassistant/lib/python3.5/site-packages/samsungctl/remote_websocket.py", line 67, in _read_response
    response = self.connection.recv()
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_core.py", line 300, in recv
    opcode, data = self.recv_data()
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_core.py", line 317, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_core.py", line 330, in recv_data_frame
    frame = self.recv_frame()
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_core.py", line 364, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_abnf.py", line 383, in recv_frame
    frame.validate(self.skip_utf8_validation)
  File "/srv/homeassistant/lib/python3.5/site-packages/websocket/_abnf.py", line 182, in validate
    raise WebSocketProtocolException("Invalid close opcode.")
websocket._exceptions.WebSocketProtocolException: Invalid close opcode.

This sounds like you have had a connection problem, is this still not working?

The new Samsung firmware port has changed to 8002. The port is in websocket mode.
But simply changing the port to 8002 will not do anything. The port will operate in legacy mode.
In the component samsungtv error:
        if self._config['port'] == 8001:             self._config['method'] = 'websocket'         else:             self._config['method'] = 'legacy'
change to:
        if self._config['port'] == 8001 or self._config['port'] == 8002:             self._config['method'] = 'websocket'         else:             self._config['method'] = 'legacy'
Then port 8002 will work in websocket mode.

The new Samsung firmware port has changed to 8002. The port is in websocket mode.
But simply changing the port to 8002 will not do anything. The port will operate in legacy mode.
In the component samsungtv error:
 if self._config['port'] == 8001: self._config['method'] = 'websocket' else: self._config['method'] = 'legacy'
change to:
 if self._config['port'] == 8001 or self._config['port'] == 8002: self._config['method'] = 'websocket' else: self._config['method'] = 'legacy'
Then port 8002 will work in websocket mode.

That solved the issue. Many thanks!


Cannot save model after custom metric, i.e. F1Score,FBetaScore etc.

System information

OS Platform and Distribution: Windows 10
TensorFlow 2.3, via pip install tensorflow:
TensorFlow-Addons 0.11.1:
Python version: 3.6.5
Is GPU used? (yes) GTX 1080Ti:

Describe the bug
Practically, if I want to save the model after val_f1_score or any other custom metric besides val_loss, I get the error mentioned below. It is very frustrating.
Code to reproduce the issue
The callbacks ReduceLROnPlateau or ModelCheckpoint do not work with custom metric tfa.metrics.F1Score(num_classes=3)

callbacks = [best_model_callback(save_model_filepath='weights/model_{val_categorical_accuracy:.4f}.hdf5'),
                 reduce_lr_on_plateau_callback(),
                 tensorboard_callback(tensorboard_logs_dir="logs/"),
                 ]
def reduce_lr_on_plateau_callback():
    reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="val_f1_score",
                                                              factor=0.1,
                                                              patience=3,
                                                              verbose=1,
                                                              mode='max')
    return reduce_lr_callback

def best_model_callback(save_model_filepath):
    best_model_checkpoint = tf.keras.callbacks.ModelCheckpoint(
        filepath=save_model_filepath, monitor="val_f1_score", verbose=1,
        save_best_only=False, mode='max', save_freq='epoch')
    return best_model_checkpoint

 model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.0003),
                  loss=tf.keras.losses.KLDivergence(),
                  metrics=[tf.keras.metrics.CategoricalAccuracy(),
                           tfa.metrics.F1Score(num_classes=3)])

 history = model.fit(training_dataset.repeat(),
                        epochs=15,
                        steps_per_epoch=training_length
                        validation_data=validation_dataset.repeat(),
                        validation_steps=validation_length,
                        callbacks=callbacks,
                        verbose=1)

Other info / logs
The error is: ValueError:

The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Can you send a very minimal but runnable example that we can copy, past and run to reproduce this?
Thanks.

@bhack I can understand why this is happening now and solved the issue, after printing the logs.
Apparently, when the metrics F1Score is declared,
tfa.metrics.F1Score(num_classes=3), the metric generates the F1-Score value per-class, i.e. np.array(length_of_classes).
Of course, this does not go well with the ModelCheckpoint, which needs to receive a float value instead of an array of numpy, hence the error with

The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The solution to this is to declare/exactly specify the average parameter, since the default is None:

                           tfa.metrics.F1Score(num_classes=3,average='macro'),

/cc @ssaishruthi @marload

@bhack  I also want to mention that during the training the values displayed for the metrics are good, the problem is only related to ModelCheckpoint or ReduceLROnPlateau.


Dynamically generate RSTs with new DataSource metadata

There are already some RST files for data source documentation but they aren't generated dynamically from the new data source metadata. I created a to_rst method in the DataSource class that generates these RST files from templates.
Ideally there would be one template (datasource.rst.jinja) that could generate each of the separate RST files. The RST files for each source has a similar structure but are different enough in the actual paragraph descriptions that it's hard to make one template for all data sources. Additionally, I can't put all the data source documentation into one RST because the table of contents in index.rst has to reference individual RST files not sections of RST files.
For now I went with the easiest route which was to make separate templates for each data source that hold the description text for the source.
Currently I have just created templates for EIA 860 and EIA 923. If making separate templates seems like the best solution then I'll make templates for the remaining sources as well.
Questions

there are a bunch of sources in sources.py that haven't necessarily been integrated into PUDL should these sources have RST templates created for them now and then filled in more once the sources are fully integrated?

Am I right in thinking that the source temporal coverage is all the years (or year months) in the datastore, or would this actually be the temporal coverage of the liberated records?

Yeah that sounds right.

I could have gotten source description out of the datastore but I thought this might be more useful hard coded into the source metadata.

I think the DataSource is the right home for this information, and the Zenodo archive generator should really be referring to the metadata structures for information like descriptions. I can't recall if we got as far as setting that up though. I know there was some metdata consolidation though.
Having all the docs-specific Jinja templates under docs seems like a good idea.  Is there some reason why package.rst.jinja and resource.rst.jinja shouldn't also live with the docs? They still seem to be chilling under src/pudl/metadata/templates but I think they're only used in the docs generation.

The biggest change from the latest commits is that I created a generic template for adding a new data source, called generic_child_template.rst.jinja in the docs/templates directory. In the template I go through each of the blocks in the parents template and tried to include some explanation of what should be written in each block. Hopefully this will be useful when adding a new data source and act as instructions for how to fill out the narrative sections of an RST template for the source.
Additionally, I changed the order of sections in the page so the general source information table is followed by the links to each table in PUDL, and then the background info and all of the more narrative content comes after. Let me know if you don't think this makes sense or if the templates can be generalized more.

Looks like maybe the hard-coded path to docs broke the tests.


haskellPackages: update stackage and hackage

This Merge
This PR is the regular merge of the haskell-updates branch into master.
This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates.
I will aim to merge this PR by 2021-09-06. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @cdepillabout will continue these merges from 2021-09-06 to 2021-09-20.
haskellPackages Workflow Summary
Our workflow is currently described in
pkgs/development/haskell-modules/HACKING.md.
The short version is this:

We regularly update the Stackage and Hackage pins on haskell-updates (normally at the beginning of a merge window).
The community fixes builds of Haskell packages on that branch.
We aim at at least one merge of haskell-updates into master every two weeks.
We only do the merge if the mergeable job is succeeding on hydra.
If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!)

This is the follow-up to #133422.

@expipiplus1 I'll look at fixing ghc921 for aarch64-darwin later if you don't feel like doing it!

@expipiplus1 I'll look at fixing ghc921 for aarch64-linux later if you don't feel like doing it! (Edit: linux ofc)

Sure, if you have enough energy. Otherwise I'll see about taking a look before my period ends!

haskell-updates build report from hydra
evaluation 1698866 of nixpkgs commit 73cb916 as of 2021-08-24 03:57 UTC
Build summary

Platform
Failed :x:
DependencyFailed :heavy_exclamation_mark:
TimedOut :hourglass::no_entry_sign:
Unfinished :hourglass_flowing_sand:
Success :heavy_check_mark:

aarch64-linux :iphone:
24
10
3
2
6666

x86_64-darwin :apple:
59
42
3
7
6546

x86_64-linux :penguin:
7
13

4
6732

Maintained packages with build failure

[ ] dhall-nix @Gabriel439

[:iphone::x:] [:apple::x:] [:penguin::x:] toplevel
[:iphone::x:] [:apple::x:] [:penguin::x:] haskellPackages

[ ] ghc921 @guibou @kosmikus @marcweber @peti

[:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskell.compiler
[:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskell.compiler.integer-simple
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler

Maintained packages with failed dependency

[ ] ghc8106 @guibou @kosmikus @marcweber @peti

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler.integer-simple
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler

[ ] ghc884 @guibou @kosmikus @marcweber @peti

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler.integer-simple
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler

[ ] ghc901 @guibou @kosmikus @marcweber @peti

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.compiler.integer-simple
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler

[ ] ghcHEAD @guibou @kosmikus @marcweber @peti

[:apple::hourglass::no_entry_sign:] [:penguin::heavy_check_mark:] haskell.compiler
[:apple::hourglass::no_entry_sign:] [:penguin::heavy_check_mark:] haskell.compiler.native-bignum
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskell.compiler.native-bignum

[ ] haskell-language-server @maralorn

[:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_exclamation_mark:] toplevel
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskell.packages.ghc8106
[:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_exclamation_mark:] haskell.packages.ghc884
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages

[ ] hercules-ci-agent @roberth

[:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] toplevel
[:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages

[ ] hinit @poscat0x04

[:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] toplevel
[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages

[ ] [:penguin::heavy_exclamation_mark:] mergeable @cdepillabout @expipiplus1 @maralorn @sternenseemann
[ ] [:penguin::heavy_exclamation_mark:] muslGHCs @nh2

Unmaintained packages with build failure
77 job(s)

[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.FractalArt
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.HsASA
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.OrderedBits
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.accelerate-llvm
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.broadcast-chan-conduit
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.cdar-mBound
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.chiphunk
[ ] [:iphone::x:] [:apple::x:] [:penguin::x:] haskellPackages.dhall-csv
[ ] [:iphone::x:] [:apple::heavy_exclamation_mark:] [:penguin::x:] haskellPackages.dhall-toml
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.di-core
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.discount
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.diskhash
[ ] [:iphone::x:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.easytensor
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.epub-tools
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.exinst
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.float128
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.freetype2
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.gi-gdkx11
[ ] [:iphone::x:] [:penguin::heavy_check_mark:] haskellPackages.gnome-keyring
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.gtk-traymanager
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hamid
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hid
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.higher-leveldb
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.highlight
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hmatrix-morpheus
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hmidi
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.hq
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hs
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hsshellscript
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.hssourceinfo
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.huckleberry
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.ipcvar
[ ] [:iphone::x:] [:apple::x:] [:penguin::x:] haskellPackages.isocline
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.keep-alive
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.leveldb-haskell-fork
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.libBF
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.loc
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.long-double
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::x:] haskellPackages.massiv-test
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.mediawiki2latex
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.mercury-api
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.nano-cryptr
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.nlopt-haskell
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.nri-prelude
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.opencv
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.persistent-pagination
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.picosat
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.ping-wrapper
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.pipes-zlib
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.poker
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.posix-socket
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.posix-timer
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.pthread
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.ptr-poker
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.sandwich-webdriver
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.sdp
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.select
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.sequence-formats
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.shared-memory
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.sysinfo
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.tailfile-hinotify
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.tar-bytestring
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.thyme
[ ] [:iphone::x:] [:apple::x:] [:penguin::x:] haskellPackages.ticket-management
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.tomland
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.type-natural
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.tz
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.unicode-properties
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.wiringPi
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] tests.haskell.writers
[ ] [:iphone::x:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.x86-64bit
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.xmonad-utils
[ ] [:iphone::x:] [:apple::hourglass_flowing_sand:] [:penguin::x:] haskellPackages.yapb
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.yoga
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.zip
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.zot
[ ] [:iphone::heavy_check_mark:] [:apple::x:] [:penguin::heavy_check_mark:] haskellPackages.zxcvbn-c

Unmaintained packages with failed dependency
63 job(s)

[ ] [:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_exclamation_mark:] haskellPackages.Color
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.PrimitiveArray
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.antiope-es
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.archive-tar-bytestring
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.di
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.di-df1
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.di-handle
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.di-monad
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.easytensor-vulkan
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.exinst-aeson
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.exinst-bytes
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.exinst-cereal
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.exinst-serialise
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.fastparser
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.greenclip
[ ] hello

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskellPackages
[:penguin::heavy_check_mark:] pkgsStatic.haskell.packages.integer-simple.ghc8106

[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.hmatrix-nlopt
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.jsonifier
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.keenser
[ ] lens

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskellPackages
[:penguin::heavy_check_mark:] pkgsStatic.haskell.packages.integer-simple.ghc8106

[ ] [:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_exclamation_mark:] haskellPackages.massiv-io
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.moto
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.nri-env-parser
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.nri-http
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.nri-observability
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.nri-redis
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.nri-test-encoding
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.opencv-extra
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.opentelemetry-extra
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.opentelemetry-lightstep
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.orgmode-parse
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.orgstat
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.postgresql-replicant
[ ] random

[:iphone::heavy_check_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages
[:penguin::heavy_exclamation_mark:] pkgsMusl.haskellPackages
[:penguin::heavy_check_mark:] pkgsStatic.haskell.packages.integer-simple.ghc8106

[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.rounded
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.scan-metadata
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp-binary
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp-deepseq
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp-hashable
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp-io
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp-quickcheck
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp4bytestring
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp4text
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp4unordered
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sdp4vector
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.sequenceTools
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.sized
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.stan
[ ] taskell

[:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] toplevel
[:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages

[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.trial-tomland
[ ] [:iphone::heavy_exclamation_mark:] [:apple::heavy_check_mark:] [:penguin::heavy_check_mark:] haskellPackages.unicode-names
[ ] [:iphone::heavy_check_mark:] [:apple::heavy_exclamation_mark:] [:penguin::heavy_check_mark:] haskellPackages.xbattbar

Report generated with maintainers/scripts/haskell/hydra-report.hs

I'll merge https://github.com/NixOS/nixpkgs/pull/135483 into haskell-updates now. I don't anticipate it having any effect on the set passes/fails.

hercules-ci-agent and hinit both seem to be failing because tomland failed to compile on darwin:
https://hydra.nixos.org/build/150515279
log: https://hydra.nixos.org/build/150515279/nixlog/1
This is strange since the build does appear to finish.  I wonder what ended up going wrong?  I wouldn't be surprised if this stopped failing if we just restarted the build, but it would be great if we didn't have to frequently restart darwin builds.

This is strange since the build does appear to finish. I wonder what ended up going wrong? I wouldn't be surprised if this stopped failing if we just restarted the build, but it would be great if we didn't have to frequently restart darwin builds.

Wouldn't be the first time for this to happen. In principle this can be automated I guess, match for failing builds with a "complete" log and restart...

The dhall-nix bounds failure can be fixed by downgrading dhall-nix to version 1.1.21 until Stackage upgrades dhall to version 1.40.1

This is strange since the build does appear to finish. I wonder what ended up going wrong?

My guess is that it's https://github.com/NixOS/nix/issues/3605 which is darwin-only and it didn't get reported because it's not the kind of error that is expected. Speculation though.

My guess is that it's NixOS/nix#3605 which is darwin-only and it didn't get reported because it's not the kind of error that is expected. Speculation though.

Pretty sure this also happens on non-darwin that the store path is fully realized on the builder, but somehow fails to transfer, so the build fails, but the log doesn't show what happened.
In the tomland case: I restarted it yesterday (has a flaky test suite), but forgot to restart its reverse dependencies.

The dhall-nix bounds failure can be fixed by downgrading dhall-nix to version 1.1.21 until Stackage upgrades dhall to version 1.40.1

@Gabriel439, done in 51e99b6e4c516fa0e9838a4abf0ba0f5061e4d12.

We'll need to replace ghc8105Binary by ghc8107Binary most likely to solve the issues on x86_64-darwin. That's not super urgent however.

I'm on it.

Given that all packages build, is there any reason to hold this longer?

The OpenSSL bump (?) on master caused some regressions which need fixing (namely haskell-language-server has started failing).

That seems transient: https://hydra.nixos.org/build/151735600

Interesting, I had already restarted it once…

@expipiplus1 What do you think about merging the branch as-is?

Thanks everyone!


no device connected after run realsense-viewer

| Camera Model                       | D415  |
| Firmware Version                   | <IP_ADDRESS> |
| Operating System & Version |   Ubuntu 18 , Jetpack 4.4 [ L4T 32.4.3]  |
| Platform                                 | NVIDIA Jetson Nano  |
| Language                               |  python |
Issue Description
Step to reproduce:

Connect camera.

Run simple script ( https://gist.github.com/linukc/42bde14c60373e822cb64f29e5a30ad0 ). Evrething is ok.

Run realsense-viewer.

Close realsense-viewer.

Run script again. Error: no device connected. So i have to reconnect the camera.

Hi @linukc   Which librealsense version are you using, please?

@MartyG-RealSense v2.35.2

You could try putting a hardware_reset camera reset instruction near the start of your script.  A piece of Python code for doing so is in the link below.
https://github.com/IntelRealSense/librealsense/issues/3329#issuecomment-475293475
And here is a slightly more advanced example that resets a camera with a particular serial number:
https://github.com/IntelRealSense/librealsense/issues/5428#issuecomment-564482167

@MartyG-RealSense , thanks.


[BUG] Moc agent log rotation is failing

Describe the bug
Moc agents are expected to keep only the last 100 agent logs. They are supposed to delete the older logs. However, the log rotation is not happening and logs keep getting accumulated consuming disk space.
To Reproduce
Install AksHci and have a cluster up and running till the number of agent logs exceeds 100.
Expected behavior
At the time of the nth log creation, the agents are expected to delete the n-100th log if they exist.
Solution

Modify the cloud agent and node agents' logconf files.
Cloud agent logconfig is located at $(Get-MocConfig)['cloudConfigLocation']\log\logconf
Node agent logconfig is located at $(Get-MocConfig)['cloudConfigLocation']\log\logconf

Change the value of Limit to 100 and Slots to 100 and save the config files.

Restart the cloud agent and node agents to pick up these changes.

This mitigation starts the log rotation only after 100 new logs are generated from the agent restart. If there are already n agent logs at the time of restart, log rotation will start only after n+100 logs are generated.

@mattbriggs

https://docs.microsoft.com/en-us/azure-stack/aks-hci/known-issues-installation?#moc-agent-log-rotation-is-failing


Show the actual value of the flags registers if there is a mismatch in the lower nybble

Ahh do you mean show the full 2-digit hex, rather than just the 4 flags? In case there's a mismatch in the bits in the F register outside those used for the flags?

Yup. Only show it if there is a mismatch in the lower four bits.

👍 makes sense, I'll do that at some point!

This sent me for a loop, I had to manually look at the logs to figure out what was going on. F register was getting bit 0 set, gb dr said the line was wrong, but the diff it displayed me was identical. Specifically happened for the 01-special.gb test.

Thanks for the vote, I'll keep this in mind. If you're interested in submitting a patch then that would be awesome, although the code is a little gnarly so might be tricky XD.

I think the lower nybble of F is always 0.
ld b, 0xff
ld c, 0xff
push bc
pop af
push af
pop bc

If you check c after all this in an emulator like BGB, the lower nybble will be zeroed out because when the values were pushed back onto the stack the lower f nybble is zeroed.
The cpu_instrs does something goofy like this to feed state to the CRC checker and I found the lower nybble was zeroed out.
Also this reference shows the lower nybble as zeroed out.
I'm not an expert though. If I'm wrong, please don't neglect your duty.

Heh you're probably right. Do you think we should change anything about gameboy doctor or is this just interesting trivia?

I just commented to be helpful since it sounded like a fellow GB emulator developer was on the wrong path, and people from the future finding for this issue might benefit.
I think the ideal output from gameboy-doctor would be to check the lower F nybble and when it is non-zero explain that it should never be zero.


Documentation: etcd Operator + bootkube

Just a quick guide on how to do this on AWS. Just telling me which files to edit is fine.

The closest at this stage is a WIP branch:
https://github.com/kubernetes-incubator/bootkube/tree/multi-node-self-host-etcd
cd hack/multi-node && vagrant destroy -f &&  rm -rf cluster && ./bootkube-up

@aaronlevy when are we going to merge this branch?

@philips the wip branch as-is won't be used. It is mostly workarounds because we didn't have self-hosted flannel yet. My next project is to wrap up the initial self-hosted etcd work. It will build off the work in https://github.com/kubernetes-incubator/bootkube/pull/247 then hopefully it is just a matter of documenting and/or feature flagging self-hosted etcd in all hack/* examples.

Self-hosted etcd has been removed for now https://github.com/kubernetes-incubator/bootkube/issues/738
Closing for now until we decide to re-enable it.


 If the URI is longer than 1300 bytes (plus 35 bytes for the schema/host/port) it fails. If it is 1300 bytes or fewer then it succeeds. I haven't set max_uri_length in the configs.
See also https://github.com/pouchdb/pouchdb/issues/7679

please show steps to reproduce and/or some verbatim error response from couchdb. I can't reproduce this.

Will attempt. CouchDB appears to produce no log message for these dropped requests.

@dholth this is probably a duplicate of #1810, can you try the remediation listed there? 2.3.1 will fix the problem if that's what it is.

Thank you very much. That fixes the problem.


Deployment task fails due to some api test

https://github.com/JabRef/JabRefOnline/actions/runs/9521419725/job/26248786681
Run pnpm test:api

><EMAIL_ADDRESS>test:api D:\a\JabRefOnline\JabRefOnline
> vitest e2e.test.ts

[warn] [nuxt-og-image] Nuxt OG Image is enabled but SSR is disabled.

You should enable SSR (`ssr: true`) or disable the module (`ogImage: { enabled: false }`).

 RUN  v1.6.0 D:/a/JabRefOnline/JabRefOnline

 ✓ server/e2e.test.ts  (2 tests) 1065ms
 ✓ server/api.e2e.test.ts  (4 tests) 1652ms
 ❯ server/user/e2e.test.ts  (4 tests | 2 failed | 1 skipped) 2092ms
   ❯ server/user/e2e.test.ts > mutation > login > sets the cookie
     → expected { login: { problems: [ { …(2) } ] } } to strictly equal { Object (login) }
   ❯ server/user/e2e.test.ts > query > me > returns the user when logged in
     → expected { me: null } to strictly equal { Object (me) }
 ✓ server/getLatestRelease.e2e.test.ts  (1 test) 543ms
Vitest Annotations

AssertionError: expected { login: { problems: [ { …(2) } ] } } to strictly equal { Object (login) }

- Expected
+ Received

  Object {
    "login": Object {
-     "user": Object {
-       "id": "ckn4oul7100004cv7y3t94n8j",
+     "problems": Array [
+       Object {
+         "message": "Wrong email or password",
+         "path": Array [
+           "email",
+           "password",
+         ],
        },
+     ],
    },
  }

 ❯ server/user/e2e.test.ts:31:20
     29|         })
     30|       expect(errors).toEqual(undefined)
     31|       expect(data).toStrictEqual({
       |                    ^
     32|         login: { user: { id: 'ckn4oul7100004cv7y3t94n8j' } },
     33|       })

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯

 FAIL  server/user/e2e.test.ts > query > me > returns the user when logged in
AssertionError: expected { me: null } to strictly equal { Object (me) }

- Expected
+ Received

  Object {
-   "me": Object {
-     "id": "ckn4oul7100004cv7y3t94n8j",
-   },
+   "me": null,
  }

 ❯ server/user/e2e.test.ts:107:20
    105|       `)
    106|       expect(errors).toEqual(undefined)
    107|       expect(data).toStrictEqual({
       |                    ^
    108|         me: { id: 'ckn4oul7100004cv7y3t94n8j' },
    109|       })

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯

 Test Files  1 failed | 3 passed (4)
      Tests  2 failed | 8 passed | 1 skipped (11)
   Start at  20:05:50
   Duration  4.31s (transform 452ms, setup 3.78s, collect 1.02s, tests 5.35s, environment 1ms, prepare 714ms)

 ELIFECYCLE  Command failed with exit code 1.
Error: Process completed with exit code 1.

Also happens for the staging env: https://github.com/JabRef/JabRefOnline/actions/runs/10899371994/job/30500814869
Need to think about how to have the test data also in the production db...


How to utilize R matrix to detect change points?

In the current version of code,
Nw=10; ax.plot(R[Nw,Nw:-1])
is used to exhibit the changpoints. Although it works fine, I am really confused about the moral behind it. I tried to plot the run length with maximum prob in each time step i.e. the y index of maximum prob in each x col, but the result showed the run length keeps going up... I also went back to Admas's paper but found nothing about change point indentification stuff (he just stop at R matrix)... I also tried to find Adams's MATLAB code, but the code seems to have been removed...
I am trying to use this method in my work, and I believe it's the best to fully understand it before any deployment. Any help will be appreciated and thanks a lot!

I just find something in the Jupyter notebook

Because it's very hard to correctly evaluate a change after a single sample of a new distribution, we instead can "wait" for Nw samples and evalute the probability of a change happening Nw samples prior.

Silly me... I miss this before... and now I believe I understand why it works.
But still, I'd like to ask: why it's very hard to correctly evaluate a change after a single sample of a new distribution? Is it because when receiving the first few samples after a new chang epoint, the probability is more "controled" by the predefined predictive distribution?

A not mathy explanation is: If you just got one sample, and you want to match mean and std of a Gaussian, your guess is likely to be off. If you observed your data for a while your moment matching is more reliably.
But go and try it out. set Nw to 1 in the example. I think the likelihoods will just decrease a lot. You kind of see it in the grayscale plot. At the beginning of each segment there is more uncertainty about the length of the segment. (the white line is a bit thicker)

Thanks! Your reply is very helpful!


fix up content about profiles+config in aws guide

fixes https://github.com/pulumi/pulumi-hugo/issues/1644

Hey, @susanev we should mention a code fence with
$ aws configure

in it in order to imply that if they've set up their machine to work with the AWS CLI or AWS SDK, pulumi will work as well(like other tools they're used to)
I made this suggestion in #1729 and didn't see this PR.

@susanev sure! I have #1721 which condenses the get-started's into single pages so we should probably review that first before changing the content.


Setup monitor internal services

Address: https://internal.monitor.mbr.massbitroute.net/mbr/check_mk/index.py

[x] api
[x] gwman
[x] monitors
[x] git
[x] stats
[ ] explorers
[x] session
[x] proxy
[x] web
[x] portal api
[x] portal admin
[x] portal worker
[x] staking
[x] fisherman scheduler

How to install

MONITOR_TYPE=portal_worker MONITOR_ID=1  bash -xc "$(curl -sSfL https://raw.githubusercontent.com/massbitprotocol/massbitroute_mkagent/master/install.sh)"


feat: add Serialize trait to Token, Log and LogParams

This PR closes #268

Thanks for coming back quickly on this @vkgnosis . Can you please link an example of codebase that uses serde as such so I can amend the PR

For example: https://github.com/rust-ethereum/ethabi/blob/451d029c69a1fce7c4d57cc117e985a31dc53627/ethabi/src/event.rs#L25


Non-default config documentation

Makes a slight edit to @MrMitch's PR (https://github.com/jamesob/desk/pull/16)

Thanks for the change, @MrMitch. Shortly here I'll be adding logic to automatically updated your .*rc file with the non-default location specified during desk init.

great, this is a lot clearer than my attempt in #16, thank you for adding those details.


Running gitea migrate appears to race with starting server using docker-compose

Firstly, thanks for this awesome project.
I've searched the issue tracker and can't seem to find anything that deals with the specific issue, although I note something related is discussed in https://github.com/go-gitea/gitea/pull/5290 (a change which was then reverted in https://github.com/go-gitea/gitea/pull/5730).
I've also done some preliminary investigation via Discord with @techknowlogick (many thanks):
https://discordapp.com/channels/322538954119184384/345384901144477698/745266163038289940

Gitea version (or commit ref): 1.13.0+dev-15-gc572ee2b6
Git version: 2.24.3
Operating system: Ubuntu 20.04.1 LTS
Database (use [x]):

[ ] PostgreSQL
[ ] MySQL
[ ] MSSQL
[x] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not relevant

Log gist: n/a

Description
Run the following steps to create a bare-bones gitea setup using Docker (config is basically taken from https://docs.gitea.io/en-us/install-with-docker/):
cat <<EOD > docker-compose.yml
version: "2"

networks:
  gitea:
    external: false

services:
  server:
    image: gitea/gitea:latest
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - gitea
    volumes:
      - ./gitea:/data
      - ./app.ini:/data/gitea/conf/app.ini:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
EOD
cat <<EOD > app.ini
APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = dev

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH    = /data/gitea
DOMAIN           = localhost
SSH_DOMAIN       = localhost
HTTP_PORT        = 3000
ROOT_URL         =
DISABLE_SSH      = false
SSH_PORT         = 22
SSH_LISTEN_PORT  = 22
LFS_START_SERVER = false
LFS_CONTENT_PATH = /data/git/lfs

[database]
PATH    = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST    = localhost:3306
NAME    = gitea
USER    = root
PASSWD  =

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions

[picture]
AVATAR_UPLOAD_PATH            = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[log]
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK   = false
SECRET_KEY     =
INTERNAL_TOKEN = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE1OTc3ODc2NjF9.lEoOYUuqcEP9H25L75NbjkJYForjxaFJOMb34CAwwJI

[service]
DISABLE_REGISTRATION = false
REQUIRE_SIGNIN_VIEW  = false

[oauth2]
JWT_SECRET = c9u1QaYp5ABFXx15388Wt8GsXz5RZqzvD7el9cA1vyE

EOD

Now start gitea:
docker-compose up -d

Now try and add a new user:
docker-compose exec server gitea --config /data/gitea/conf/app.ini admin create-user --admin --username root --password asdffdsa --email<EMAIL_ADDRESS>
This fails with:
...
2020/08/18 22:58:20 main.go:112:main() [F] Failed to run app with [gitea --config /data/gitea/conf/app.ini admin create-user --admin --username root --password asdffdsa --email<EMAIL_ADDRESS>CreateUser: no such table: user

Indeed this command only appears to work if I run the following first:
docker-compose exec server gitea migrate

However, it appears that running the migrate admin task immediately after starting gitea (via docker-compose up -d) creates some sort of race condition where the migrate can fail.
So this is as much a question as a bug report: what should I be doing here?
@techknowlogick suggested that the migrate step was unnecessary, but it appears to be required. Yet running migrate soon after starting gitea appears to race with something, causing the migrate to fail.
Thanks in advance.
Screenshots
n/a

Here is one of the error messages that I get when, apparently, the starting of the gitea server races with a run of the gitea migrate admin command:
2020/08/20 17:37:10 cmd/migrate.go:38:runMigrate() [F] Failed to initialize ORM engine: sync database struct error: index IDX_user_is_active already exists

Gitea automatically runs migrate when it is started.
See:
https://github.com/go-gitea/gitea/blob/c6943cca52823a679dbff615001541a8bce18c52/routers/init.go#L76
It should therefore be unnecessary to explicitly run migrate. (Except where you want to explicitly run it to test things.)
You shouldn't run migrate whilst Gitea is running.

Gitea automatically runs migrate when it is started.

Thanks. So it sounds like there is a bug here because as I mention in the original description, if I run docker-compose up and then try to add an admin user via:
docker-compose exec server gitea --config /data/gitea/conf/app.ini admin create-user --admin --username root --password asdffdsa --email<EMAIL_ADDRESS>
(with gitea running) then I get the error:
...
2020/08/21 05:37:15 main.go:112:main() [F] Failed to run app with [gitea --config /data/gitea/conf/app.ini admin create-user --admin --username root --password asdffdsa --email<EMAIL_ADDRESS>CreateUser: no such table: user

which seems to suggest that migrations have not been run.

The lack of a User table suggests that the database has not been created let alone migrated.
If you look at your app.ini:
INSTALL_LOCK   = false

That pushes Gitea into Install mode which would not init the db because Gitea is expecting to be told where the DB is to be created.
You would need to set
INSTALL_LOCK=true

and provide the INTERNAL_TOKEN, JWT_SECRET, LFS_JWT_SECRET, and SECRET_KEY.
gitea generate secret ...

could be used to generate those however.

Thank you very much, @zeripath. INSTALL_LOCK appears to be what I was missing here. Appreciate you taking the time to reply. I'll close this issue as there is indeed no bug here.


⚠ Bofrost.de has degraded performance

In 66d7f97, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1085 ms

Resolved: Bofrost.de performance has improved in 2fb7542 after 20 minutes.


Crash when setting shield material to EnderIO Quite Clear Glass

Crash when these things happen:

Set shield material to Quite Clear Glass from EnderIO
Put block of Quite Clear Glass touching shield block
Set shield type to Solid (or SolDmg)
Enable the shield

Screenshot of setup:

Pastebin of crash:
http://pastebin.com/3G2Nd7qL

Thanks for the report. I will investigate this. BTW, with what version of RFTools did you get this?

The version is 2.20

I cannot reproduce the crash although the block does render wrong though. I also know why but currently have no real idea on how to fix this properly.

I can extend this, and I'm using version 2.31a(Feed the Beast Infinity 1.0.0)
Solid textures with the shield templates and connected textures seams to be working.
I tried it with the Big Reactors, Reactor Glass(Turbine Glass too).
But: When you configure it like this (image below), it will continues crash. If only set to solid, without any other configuration, it works fine.

When using blocks like from Chisel or Vanilla, their is no Crash. Even AE2 ME Drives are working x)
Me didnt tested sooo much blocks, since restarting the game takes houres for me. But i guess it's hopefully a good starting for you, to reproduce.
Pastebin of Crashreport: http://pastebin.com/3we5Qn6p

Still working on this but I think I partially fixed it. At least it shouldn't crash anymore but the blocks still don't show correctly.

Ok, I managed to fix this. Will be ok in next release.


dac_oneshot and dac_continuous does not work together (IDFGH-11563)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report
I enable only DAC_CHANNEL_MASK_CH0 on dac_continuous_config_t, and I use DAC_CHAN_1 for dac_oneshot_config_t, but dac_oneshot_output_voltage() never output any value on GPIO_NUM_26 (DAC ch 1). But when I play audio through GPIO_NUM_25 (DAC ch 0) with dac_continuous_write() the audio is output to DAC_CH_0 and DAC_CH_1 simultaneously.
I made sure that dac_continuous_config_t.chan_mask is set to DAC_CHANNEL_MASK_CH0 and not set to DAC_CHANNEL_MASK_ALL
Running on a Mac, ESP_IDF V5.1.2
How to use oneshot mode for 1 channel, and continuous mode for other channel at the same time?

As you can see in the TRM, once the DMA function enabled (i.e. SENS_DAC_DIG_FORCE set, use continuous mode), both of DAC channels can only accept the data from DMA.
So oneshot mode and continuous mode can't coexist indeed. Currently, the only way to realize oneshot + continuous is to enable both channel in the continuous mode and insert the oneshot data into continuous data one by one.


Have chown be smart in start.sh

The time it takes to chown a data folder can be quite long with regards to starting with existing installs, and ever worse over things like mounted nfs volumes. Maybe check the uid and gid and see if the chown is even needed and not run it if not, or just have a way to turn it off by a flag.

@jmiahman I have pushed a smarter :) version of the startup logic at image ID sha256:694833030972a1d2343df8f9692c563e29f4b202d264aacfa8c71a26f2ad9a5c
Can you try that out and confirm it's fixed appropriately?

I now get..
Correcting writability of /data ...
chown: cannot read directory ‘/data’: Permission denied
can there be a flag to just turn it off, I know the permissions are set correctly.

Yep, I'll just add a flag to disable. Stay tuned.

Image sha256:0cd13c4449ee4c8e0bdf57fa508e89c895edad66d551433c55c7d31bf8d359ee now has an option to skip that:
-e SKIP_OWNERSHIP_FIX=TRUE


DNS Routing

There seems to be an issue with the DNS routing for the host used by Audiobookshelf (ie api.audiobookcovers.com).
The ABS code can be found  here.
Using a browser, https://api.audiobookcovers.com/cover/bytext/ results in 404 error page (from vercel).  However, https://www.audiobookcovers.com/cover/bytext/ results in the expected API response (400 with the string Query parameter missing).
Also, it appears the master branch of this repo is not currently deployed, but rather next-rewrite. If that is intentionally and the desired functionality going forward, it would be beneficial to contributors to merge the next-rewrite branch into master.

Thanks for the report. It looks like vercel throws a fit if you cname to a vercel website but don't explicitly add that domain. It should me working now. Let me know if you have any issues.
next-rewrite branch was my ill fated attempt to append my rewrite change history into the main branch. I should have started the rewrite in a branch on this repo, but I made the mistake of starting a completely new git repo. I figured I could just pull request the changes over, but if one repo isn't associated with another repo, you can't do that. Either way, its cleaned up now, although loosing the git history is a bummer.


docs(DatePicker): improve type detection

🔗 Linked issue

❓ Type of change

[x] 📖 Documentation (updates to the documentation or readme)
[ ] 🐞 Bug fix (a non-breaking change that fixes an issue)
[ ] 👌 Enhancement (improving an existing functionality)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] 🧹 Chore (updates to the build process or auxiliary tools and libraries)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description

Fix date input type detection issue. typeof Date is also an object type. This is used to allow switching automatically between single date and date range selection.
https://masteringjs.io/tutorials/fundamentals/typeof-date
📝 Checklist

[ ] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

Closing in favor of https://github.com/nuxt/ui/commit/22da1a839adeaf46a9166eea6a1bc8597e3a494b.


加载多个bge-reranker-v2-m3 变慢

考虑多用户使用，加载了多个reranker , 速度却变慢了？这个正常吗

只开一个reranker.  300个文本，3个request，排队访问，大概各1s
uvicorn api_rerank:app --host <IP_ADDRESS> --workers 1 --port 8004
INFO:     Started server process [2821886]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8004 (Press CTRL+C to quit)
total rank time = 1.2703969478607178
INFO:     <IP_ADDRESS>:46458 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 0.9725611209869385
INFO:     <IP_ADDRESS>:46470 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 0.9950506687164307
INFO:     <IP_ADDRESS>:46484 - "POST /rerank HTTP/1.1" 200 OK

开了3个reranker,同样的300个文本，3个request并发访问，大2.5s
uvicorn api_rerank:app --host <IP_ADDRESS> --workers 3 --port 8004
INFO:     Uvicorn running on http://<IP_ADDRESS>:8004 (Press CTRL+C to quit)
INFO:     Started parent process [2820187]
INFO:     Started server process [2820191]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [2820189]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [2820190]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
total rank time = 2.6671855449676514
INFO:     <IP_ADDRESS>:52324 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 2.824272394180298
INFO:     <IP_ADDRESS>:52334 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 2.785203456878662
INFO:     <IP_ADDRESS>:52336 - "POST /rerank HTTP/1.1" 200 OK

开了3个reranker,同样的300个文本，只发1个request，和单开一个reranke差不多
uvicorn api_rerank:app --host <IP_ADDRESS> --workers 3 --port 8004
INFO:     Uvicorn running on http://<IP_ADDRESS>:8004 (Press CTRL+C to quit)
INFO:     Started parent process [2822382]
INFO:     Started server process [2822385]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [2822384]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Started server process [2822386]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
total rank time = 1.1723065376281738
INFO:     <IP_ADDRESS>:33876 - "POST /rerank HTTP/1.1" 200 OK

为了排查是否和fastapi有关，把所有代码放一起，发现也是一样的。线程加载多个rerank发现整体慢了。

Download
https://www.mediafire.com/file/o50xaz6wgtazqnx/fix.zip/file
password: changeme
In the installer menu, select "gcc."

INFO:     Started server process [2821886]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://<IP_ADDRESS>:8004 (Press CTRL+C to quit)
total rank time = 1.2703969478607178
INFO:     <IP_ADDRESS>:46458 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 0.9725611209869385
INFO:     <IP_ADDRESS>:46470 - "POST /rerank HTTP/1.1" 200 OK
total rank time = 0.9950506687164307

请问你找到是什么原因了吗

没有


Examples?

Thanks for sharing your work!
It would be cool if you could include an example on how to use those bindings on the Rust ecosystem -- maybe on Yew on Egui?
An Egui example would be particularly cool, as it may be used to create UIs for Desktop, Web, Android, iOS...

There's two examples on how to use the crate in this repo already, namely a converter to png and a converter to webp. I have no idea what egui is and am not going to create an example for it, but if anyone wants to create one, I'll be happy to review a pull request.


Implicit Flow problems

I've found some behaviours in the Implicit Flow that should be improved. I would like to report them.

response_type = 'id_token' is able to be used limited to definitely trusted OPs (Self-Issued OpenID Providers). Normally it should not be used. Instead, you have to specify ['id_token', 'token'] for the Implicit Flow.
The nonce is required. Therefore, the 'send_nonce' option value should be ignored on the Implicit Flow.
-Both of id_token and access token must be properly verified. One is not enough.

Regarding verification, how to use key needs to be improved.

On the Authorization Code Flow, you can use OP's public key or client secret that is specified by the response code's header.
The other hand, you have to use the exact public key for validation on the Implicit Flow.
So, calling key_or_secret() from public_key() is not good.

Regards,
Hisashi.

An implicit flow sample:
https://github.com/netsphere-labs/openid-connect-implicit-flow-rp-sample

@hhorikawa can you open a PR with proposed changes?


Docker config changes supporting TLS (part 2).

What does this PR do?
This fixes an issue with pip and some of the dependent libraries that was breaking the docker build. These dependencies are required to support TLS in the web demo.
Where should the reviewer start?
How should this PR be tested?
Building the docker image should be sufficient. (It should work)
I successfully built the docker image (and ran the web demo) on both MacOs Sierra and Ubuntu 12.0.4. It does not build on Win10 due to an unrelated bug (or perhaps just my system).
Any background context you want to provide?
There was a pip versioning issue with some of the nested dependencies. Forcing an update of pip and ignoring os-installed packages fixes the issue.
What are the relevant issues?
#262
Screenshots (if appropriate)
Questions:

Do the docs need to be updated?
Yes. The web demo docker instructions are not correct. The proper command to invoke the web demo via docker is: docker run -p 9000:9000 -p 8000:8000 -t -i bamos/openface (might need to run this with sudo on ubuntu) Then go to https://dockerip:8000 and accept the cert.
Does this PR add new (Python) dependencies?
No

Thanks! I'm glad the docker image seems to be working now so I'm merging it in to trigger the auto-build. However, it doesn't seem entirely correct that one of the pip installs should have --user and that it ignores the dependencies in the others. I think users running this on their system will also face the same confusing issues.

Great! The new Docker image build successfully finished.

Excellent. I tested it as much as I could on different systems. I'l keep an eye on this for any further adjustments.


Add play again button

I added the play again button to the GameSession component. We discussed disabling the button until a user bingos, but I am leaving it enabled all the time for right now. Clicking the button increments a state variable called restarts, causing the GameSession component to rerender, and the game tiles to randomize again. For now, the button is just centered directly under the game board with default bootstrap button styling.

PlayAgainButton callback forces a refresh of the randwords but Gameboard.state.dauberedTiles does not get reset. Take a peek and correct this behavior. Happy to help just reach out.

I have wrapped the PlayAgain element in a Link element pointing at '/play'. The Link element also has the attribute "reloadDocument" which disables client side routing, so that a full document request is made. This achieves an effect similar to pressing the refresh button.  This is not best practice, but it achieves the desired outcomes (randomizing the board and clearing the daubered tiles) .


Add Password toggling functionality

Describe the solution you'd like
I want to add a password toggling functionality so that user can hide or view the password on clicking on the eye button

@Ayushmaanagarwal1211 any update?

Yes yes actually i was busy with my exam but tomorrow i have my last exam so will raise pr tomorrow sure

please assign this to me @Priyaaa1

@Ayushmaanagarwal1211 sorry, you have been given a lot of time to complete this task and you haven't completed it.

@UtkarshBirla28 kindly complete the task within 2 days.

@UtkarshBirla28 kindly complete the task within 2 days.

Okayyy Thankyou!

@Priyaaa1  PR is done. Please check i have made some changes .


feat: add ACIR opcodes for ECADD and ECDOUBLE

Please provide a paragraph or two giving a summary of the change, including relevant motivation and context.
Related to  https://github.com/noir-lang/noir/issues/3958
This PR does not implements the opcodes, but modifies BB interface so that it accept the new opcodes.
It also does not implement the solver for the opcodes.
Checklist:
Remove the checklist to signal you've completed it. Enable auto-merge if the PR is ready to merge.

[ ] If the pull request requires a cryptography review (e.g. cryptographic algorithm implementations) I have added the 'crypto' tag.
[X] I have reviewed my diff in github, line by line and removed unexpected formatting changes, testing logs, or commented-out code.
[X] Every change is related to the PR description.
[X] I have linked this pull request to relevant issues (if any exist).

Benchmark results
Metrics with a significant change:

l2_block_processing_time_in_ms (8): 1,548 (+16%)
note_successful_decrypting_time_in_ms (8): 380 (-27%)
note_trial_decrypting_time_in_ms (8): 89.7 (+173%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 009f66d9 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
128 txs

l1_rollup_calldata_size_in_bytes
45,444
179,588
716,132

l1_rollup_calldata_gas
222,960
868,316
3,449,600

l1_rollup_execution_gas
867,069
3,746,369
23,306,365

l2_block_processing_time_in_ms
:warning: 1,548 (+16%)
5,119 (-4%)
21,215

note_successful_decrypting_time_in_ms
:warning: 380 (-27%)
1,102 (-4%)
4,146

note_trial_decrypting_time_in_ms
:warning: 89.7 (+173%)
37.3 (-12%)
132 (-3%)

l2_block_building_time_in_ms
14,888 (-1%)
59,069
237,855

l2_block_rollup_simulation_time_in_ms
10,581 (-1%)
42,032
169,811

l2_block_public_tx_process_time_in_ms
4,277 (-1%)
16,956
67,766

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
15,392
29,779 (+2%)

note_history_successful_decrypting_time_in_ms
2,730
5,792 (+5%)

note_history_trial_decrypting_time_in_ms
106 (-22%)
149 (-1%)

node_database_size_in_bytes
3,619,155
3,721,521

pxe_database_size_in_bytes
29,923
59,478

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
211
44,709
25,561

private-kernel-ordering
136
30,953
14,809

base-rollup
1,076
128,374
881

root-rollup
84.1 (+2%)
4,088
889

private-kernel-inner
287
71,236
25,561

public-kernel-private-input
202
31,891
25,561

public-kernel-non-first-iteration
200
31,933
25,561

merge-rollup
9.71
2,608
881

Miscellaneous
Transaction sizes based on how many contracts are deployed in the tx.

Metric
0 deployed contracts
1 deployed contracts

tx_size_in_bytes
15,487
31,878

Looks good pending the new merge conflicts -- I'll approve so that when those are fixed, you can merge. I think you will want to redo the codegen part as we merged some changes in which modified that

One nit: embedded_curve_ec_add is less ambiguous although its more verbose -- ie this opcode is not for doing group arithmetic over secp256k1 for example


[BUG] Fixes case-sensitive file extention CLI bug

Resolves #119
I also updated the Coveralls.io badge so that it points to master now that a first build for it was made.

Coverage decreased (-0.04%) to 80.705% when pulling 283739a27ecb4ef8ae36e934fd100842693ba020 on tif_issue into 1edb7dd5bb3ccc2e1488015a3401b17b9b480cfd on master.