[Benchmarks] Add a whole lot more benchmarks for Data

This adds a number of new benchmarks for Data that test specific common cases of empty, small, medium and large Datas. Additionally this adds new benchmarks for creation as well as conversion from String to Data using Sequence/Collection initializers.

@swift-ci please smoke test

@swift-ci please smoke benchmark

Build comment file:
Performance: -O

TEST
OLD
NEW
DELTA
RATIO

Improvement

ObjectiveCBridgeStubToNSDate2
1538
1364
-11.3%
1.13x (?)

Added

DataAccessBytesLarge
1409027
1488463
1460729
—

DataAccessBytesMedium
1140
1352
1211
—

DataAccessBytesSmall
1137
1139
1138
—

DataAppendBytesLarge
1810983
1813382
1812004
—

DataAppendBytesMedium
4890
5360
5201
—

DataAppendBytesSmall
4477
4744
4580
—

DataCopyBytesLarge
15268
16127
15639
—

DataCopyBytesMedium
453
472
459
—

DataCopyBytesSmall
291
293
292
—

DataCountLarge
1419208
1489729
1464522
—

DataCountMedium
37
40
38
—

DataCountSmall
34
35
34
—

DataCreateEmpty
23600
23819
23685
—

DataCreateEmptyArray
26934
27026
26980
—

DataCreateLarge
1433268
1494583
1473405
—

DataCreateMedium
17312
18078
17605
—

DataCreateMediumArray
3102
3157
3134
—

DataCreateSmall
98144
102382
100847
—

DataCreateSmallArray
27711
27881
27785
—

DataMutateBytesLarge
1817912
1891248
1866258
—

DataMutateBytesMedium
4279
4723
4526
—

DataMutateBytesSmall
4712
5021
4887
—

DataSetCountLarge
525
527
526
—

DataSetCountMedium
517
534
523
—

DataSetCountSmall
508
509
508
—

DataSubscriptLarge
1393758
1487858
1449201
—

DataSubscriptMedium
220
222
221
—

DataSubscriptSmall
220
224
222
—

DataToStringEmpty
2985
3416
3130
—

DataToStringMedium
11626
11816
11690
—

DataToStringSmall
5140
5363
5221
—

StringToDataEmpty
2590
2641
2607
—

StringToDataMedium
2680
2739
2713
—

StringToDataSmall
2599
2646
2615
—

Removed

DataAccessBytes
1137
1408
1227
—

DataAppendBytes
5139
5356
5224
—

DataCopyBytes
531
531
531
—

DataCount
34
35
34
—

DataMutateBytes
4332
4482
4383
—

DataSetCount
517
517
517
—

DataSubscript
220
222
221
—

Code size: -O

TEST
OLD
NEW
DELTA
RATIO

Regression

DataBenchmarks.o
26476
56388
+113.0%
0.47x

Performance: -Osize

TEST
OLD
NEW
DELTA
RATIO

Improvement

CharacterLiteralsLarge
111
100
-9.9%
1.11x

CharacterLiteralsSmall
345
322
-6.7%
1.07x

Added

DataAccessBytesLarge
1477658
1515304
1496091
—

DataAccessBytesMedium
1154
1288
1199
—

DataAccessBytesSmall
1152
1152
1152
—

DataAppendBytesLarge
1818139
1820459
1819406
—

DataAppendBytesMedium
4869
4950
4897
—

DataAppendBytesSmall
5087
5136
5104
—

DataCopyBytesLarge
15565
16309
15882
—

DataCopyBytesMedium
484
484
484
—

DataCopyBytesSmall
291
294
292
—

DataCountLarge
1402564
1460620
1441213
—

DataCountMedium
37
40
38
—

DataCountSmall
34
35
34
—

DataCreateEmpty
23530
23928
23727
—

DataCreateEmptyArray
27031
27143
27100
—

DataCreateLarge
1492492
1502613
1499147
—

DataCreateMedium
17147
17547
17306
—

DataCreateMediumArray
3100
3156
3119
—

DataCreateSmall
100609
103595
102597
—

DataCreateSmallArray
27718
27998
27844
—

DataMutateBytesLarge
1850114
1903383
1869186
—

DataMutateBytesMedium
4287
4411
4342
—

DataMutateBytesSmall
4363
4463
4425
—

DataSetCountLarge
529
532
530
—

DataSetCountMedium
517
517
517
—

DataSetCountSmall
508
509
508
—

DataSubscriptLarge
1498272
1509822
1504916
—

DataSubscriptMedium
220
223
221
—

DataSubscriptSmall
220
223
221
—

DataToStringEmpty
3192
3424
3269
—

DataToStringMedium
12146
12181
12169
—

DataToStringSmall
5763
5901
5813
—

StringToDataEmpty
2633
2805
2690
—

StringToDataMedium
2700
2755
2720
—

StringToDataSmall
2640
2747
2676
—

Removed

DataAccessBytes
1142
1316
1200
—

DataAppendBytes
5877
6070
5942
—

DataCopyBytes
665
665
665
—

DataCount
37
38
37
—

DataMutateBytes
4912
5367
5069
—

DataSetCount
517
517
517
—

DataSubscript
220
222
221
—

Code size: -Osize

TEST
OLD
NEW
DELTA
RATIO

Regression

DataBenchmarks.o
21781
51013
+134.2%
0.43x

Performance: -Onone

TEST
MIN
MAX
MEAN
MAX_RSS

Added

DataAccessBytesLarge
1407747
1478167
1434889
—

DataAccessBytesMedium
2351
2496
2400
—

DataAccessBytesSmall
2370
2473
2423
—

DataAppendBytesLarge
1823424
1855672
1844661
—

DataAppendBytesMedium
5077
5239
5144
—

DataAppendBytesSmall
4877
4948
4902
—

DataCopyBytesLarge
15465
16268
15957
—

DataCopyBytesMedium
503
509
505
—

DataCopyBytesSmall
363
365
364
—

DataCountLarge
1411637
1474797
1434953
—

DataCountMedium
223
227
224
—

DataCountSmall
225
228
226
—

DataCreateEmpty
24247
24514
24363
—

DataCreateEmptyArray
60473
60733
60580
—

DataCreateLarge
1448701
1554400
1508303
—

DataCreateMedium
18212
18294
18265
—

DataCreateMediumArray
9367
9467
9403
—

DataCreateSmall
97285
100790
98485
—

DataCreateSmallArray
91824
92012
91935
—

DataMutateBytesLarge
1813483
1900503
1844943
—

DataMutateBytesMedium
5775
5977
5855
—

DataMutateBytesSmall
5820
5954
5888
—

DataSetCountLarge
572
574
573
—

DataSetCountMedium
535
535
535
—

DataSetCountSmall
526
563
544
—

DataSubscriptLarge
1441155
1457993
1450813
—

DataSubscriptMedium
443
443
443
—

DataSubscriptSmall
443
448
446
—

DataToStringEmpty
3829
4169
3984
—

DataToStringMedium
12289
12418
12370
—

DataToStringSmall
5772
5845
5799
—

StringToDataEmpty
3665
3733
3689
—

StringToDataMedium
3986
4063
4028
—

StringToDataSmall
3678
3718
3694
—

Removed

DataAccessBytes
2283
2430
2332
—

DataAppendBytes
5601
5672
5639
—

DataCopyBytes
568
568
568
—

DataCount
222
226
223
—

DataMutateBytes
5940
6024
5969
—

DataSetCount
534
534
534
—

DataSubscript
442
450
445
—

How to read the data
The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.
If you see any unexpected regressions, you should consider fixing the regressions before you merge the PR.
Noise: Sometimes the performance results (not code size!) contain false alarms. Unexpected regressions which are marked with '(?)' are probably noise. If you see regressions which you cannot explain you can try to run the benchmarks again. If regressions still show up, please consult with the performance team (@eeckstein).

Hardware Overview
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

Among other things, this has also reintroduced setup overhead to some of the benchmarks I previously cleaned up. All the new Large variants are ~100% overhead of buffer creation. I see they were disabled in #20411, but I really don't think benchmarks that allocate over 1GB of memory are something that belongs to Swift Benchmarking Suite, at all.
Also all the DataCreate measures is the performance of arc4random_buf. I don't see why we need truly random data for these tests… pseudorandom would be just as good, but since we are not doing anything with it, why not just fill it with repeated sequence of bytes like we do in IterateData?
I'll clean this up, but in the future, please do run Benchmark_Driver check locally before committing modified benchmarks. We now have hooked this up to CI, but it only validates newly added benchmarks. Here's what it say on my 2008 MBP:

Benchmark Check Report

✅
Benchmark Check Report

⛔⏱
DataAccessBytesLarge has setup overhead of 15980062 μs (102.0%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⚠Ⓜ
DataAccessBytesLarge has very wide range of memory used between independent, repeated measurements.  DataAccessBytesLarge mem_pages [i1, i2]: min=[258239, 180882] 𝚫=77357 R=[3927, 81283]

⛔⏱
DataAccessBytesMedium has setup overhead of 16 μs (7.4%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataAppendArray execution took at least 16619 μs.  Decrease the workload of DataAppendArray by a factor of 32 (100), to be less than 1000 μs.

⛔⏱
DataAppendBytesLarge has setup overhead of 14701060 μs (92.3%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataAppendBytesLarge execution took at least 1231514 μs (excluding the setup overhead).  Decrease the workload of DataAppendBytesLarge by a factor of 2048 (10000), to be less than 1000 μs.

⛔⏱
DataAppendBytesMedium execution took at least 15679 μs.  Decrease the workload of DataAppendBytesMedium by a factor of 16 (100), to be less than 1000 μs.

⚠🔤
DataAppendDataLargeToLarge name is composed of 6 words.  Split DataAppendDataLargeToLarge name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataLargeToLarge has setup overhead of 27390 μs (13.1%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataAppendDataLargeToLarge execution took at least 181985 μs (excluding the setup overhead).  Decrease the workload of DataAppendDataLargeToLarge by a factor of 256 (1000), to be less than 1000 μs.

⚠Ⓜ
DataAppendDataLargeToLarge has very wide range of memory used between independent, repeated measurements.  DataAppendDataLargeToLarge mem_pages [i1, i2]: min=[119, 95] 𝚫=24 R=[18, 29]

⚠🔤
DataAppendDataLargeToMedium name is composed of 6 words.  Split DataAppendDataLargeToMedium name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataLargeToMedium execution took at least 80878 μs.  Decrease the workload of DataAppendDataLargeToMedium by a factor of 128 (100), to be less than 1000 μs.

⚠Ⓜ
DataAppendDataLargeToMedium has very wide range of memory used between independent, repeated measurements.  DataAppendDataLargeToMedium mem_pages [i1, i2]: min=[54, 51] 𝚫=3 R=[19, 22]

⚠🔤
DataAppendDataLargeToSmall name is composed of 6 words.  Split DataAppendDataLargeToSmall name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataLargeToSmall execution took at least 76184 μs.  Decrease the workload of DataAppendDataLargeToSmall by a factor of 128 (100), to be less than 1000 μs.

⚠🔤
DataAppendDataMediumToLarge name is composed of 6 words.  Split DataAppendDataMediumToLarge name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataMediumToLarge has setup overhead of 19424 μs (17.8%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataAppendDataMediumToLarge execution took at least 89827 μs (excluding the setup overhead).  Decrease the workload of DataAppendDataMediumToLarge by a factor of 128 (100), to be less than 1000 μs.

⚠Ⓜ
DataAppendDataMediumToLarge has very wide range of memory used between independent, repeated measurements.  DataAppendDataMediumToLarge mem_pages [i1, i2]: min=[52, 53] 𝚫=1 R=[24, 23]

⚠🔤
DataAppendDataMediumToMedium name is composed of 6 words.  Split DataAppendDataMediumToMedium name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataMediumToMedium execution took at least 14957 μs.  Decrease the workload of DataAppendDataMediumToMedium by a factor of 16 (100), to be less than 1000 μs.

⚠🔤
DataAppendDataMediumToSmall name is composed of 6 words.  Split DataAppendDataMediumToSmall name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataMediumToSmall execution took at least 14235 μs.  Decrease the workload of DataAppendDataMediumToSmall by a factor of 16 (100), to be less than 1000 μs.

⚠🔤
DataAppendDataSmallToLarge name is composed of 6 words.  Split DataAppendDataSmallToLarge name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataSmallToLarge execution took at least 120114 μs.  Decrease the workload of DataAppendDataSmallToLarge by a factor of 128 (1000), to be less than 1000 μs.

⚠Ⓜ
DataAppendDataSmallToLarge has very wide range of memory used between independent, repeated measurements.  DataAppendDataSmallToLarge mem_pages [i1, i2]: min=[74, 61] 𝚫=13 R=[12, 18]

⚠🔤
DataAppendDataSmallToMedium name is composed of 6 words.  Split DataAppendDataSmallToMedium name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataSmallToMedium execution took at least 13885 μs.  Decrease the workload of DataAppendDataSmallToMedium by a factor of 16 (100), to be less than 1000 μs.

⚠🔤
DataAppendDataSmallToSmall name is composed of 6 words.  Split DataAppendDataSmallToSmall name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming

⛔⏱
DataAppendDataSmallToSmall has setup overhead of 1210 μs (9.0%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataAppendDataSmallToSmall execution took at least 12171 μs (excluding the setup overhead).  Decrease the workload of DataAppendDataSmallToSmall by a factor of 16 (100), to be less than 1000 μs.

⛔⏱
DataAppendSequence execution took at least 65873 μs.  Decrease the workload of DataAppendSequence by a factor of 128 (100), to be less than 1000 μs.

⛔⏱
DataCopyBytesLarge execution took at least 81330 μs.  Decrease the workload of DataCopyBytesLarge by a factor of 128 (100), to be less than 1000 μs.

⚠Ⓜ
DataCopyBytesLarge has very wide range of memory used between independent, repeated measurements.  DataCopyBytesLarge mem_pages [i1, i2]: min=[34, 34] 𝚫=0 R=[13, 24]

⚠⏱
DataCopyBytesMedium execution took at least 1352 μs.  Decrease the workload of DataCopyBytesMedium by a factor of 2 (10), to be less than 1000 μs.

⛔⏱
DataCountLarge has setup overhead of 15325398 μs (99.0%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataCountLarge execution took at least 160287 μs (excluding the setup overhead).  Decrease the workload of DataCountLarge by a factor of 256 (1000), to be less than 1000 μs.

⛔⏱
DataCountMedium has setup overhead of 18 μs (22.8%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataCountSmall has setup overhead of 4 μs (7.5%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataCreateEmptyArray execution took at least 14273 μs.  Decrease the workload of DataCreateEmptyArray by a factor of 16 (100), to be less than 1000 μs.

⛔⏱
DataCreateLarge has setup overhead of 14540010 μs (95.3%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataCreateLarge execution took at least 714406 μs (excluding the setup overhead).  Decrease the workload of DataCreateLarge by a factor of 1024 (1000), to be less than 1000 μs.

⛔⏱
DataCreateMedium execution took at least 157904 μs.  Decrease the workload of DataCreateMedium by a factor of 256 (1000), to be less than 1000 μs.

⚠⏱
DataCreateMediumArray execution took at least 9136 μs.  Decrease the workload of DataCreateMediumArray by a factor of 16 (10), to be less than 1000 μs.

⛔⏱
DataCreateSmall execution took at least 223690 μs.  Decrease the workload of DataCreateSmall by a factor of 256 (1000), to be less than 1000 μs.

⛔⏱
DataCreateSmallArray execution took at least 16932 μs.  Decrease the workload of DataCreateSmallArray by a factor of 32 (100), to be less than 1000 μs.

⛔⏱
DataMutateBytesLarge has setup overhead of 18031434 μs (100.5%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataMutateBytesMedium execution took at least 12779 μs.  Decrease the workload of DataMutateBytesMedium by a factor of 16 (100), to be less than 1000 μs.

⚠⏱
DataMutateBytesSmall execution took at least 3367 μs.  Decrease the workload of DataMutateBytesSmall by a factor of 4 (10), to be less than 1000 μs.

⛔⏱
DataReplaceLarge execution took at least 84975 μs.  Decrease the workload of DataReplaceLarge by a factor of 128 (100), to be less than 1000 μs.

⛔⏱
DataReplaceLargeBuffer execution took at least 186047 μs.  Decrease the workload of DataReplaceLargeBuffer by a factor of 256 (1000), to be less than 1000 μs.

⚠Ⓜ
DataReplaceLargeBuffer has very wide range of memory used between independent, repeated measurements.  DataReplaceLargeBuffer mem_pages [i1, i2]: min=[68, 61] 𝚫=7 R=[13, 26]

⛔⏱
DataReplaceMedium execution took at least 16692 μs.  Decrease the workload of DataReplaceMedium by a factor of 32 (100), to be less than 1000 μs.

⛔⏱
DataReplaceMediumBuffer execution took at least 33516 μs.  Decrease the workload of DataReplaceMediumBuffer by a factor of 64 (100), to be less than 1000 μs.

⛔⏱
DataReplaceSmall execution took at least 11302 μs.  Decrease the workload of DataReplaceSmall by a factor of 16 (100), to be less than 1000 μs.

⛔⏱
DataReplaceSmallBuffer execution took at least 27040 μs.  Decrease the workload of DataReplaceSmallBuffer by a factor of 32 (100), to be less than 1000 μs.

⛔⏱
DataReset execution took at least 10305 μs.  Decrease the workload of DataReset by a factor of 16 (100), to be less than 1000 μs.

⛔⏱
DataSetCountLarge has setup overhead of 498 μs (17.3%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⚠⏱
DataSetCountLarge execution took at least 2377 μs (excluding the setup overhead).  Decrease the workload of DataSetCountLarge by a factor of 4 (10), to be less than 1000 μs.

⚠⏱
DataSetCountMedium execution took at least 2477 μs.  Decrease the workload of DataSetCountMedium by a factor of 4 (10), to be less than 1000 μs.

⛔⏱
DataSubscriptLarge has setup overhead of 14769664 μs (97.1%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataSubscriptLarge execution took at least 443749 μs (excluding the setup overhead).  Decrease the workload of DataSubscriptLarge by a factor of 512 (1000), to be less than 1000 μs.

⛔⏱
DataSubscriptMedium has setup overhead of 16 μs (10.0%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

⛔⏱
DataToStringMedium has setup overhead of 44 μs (5.8%).  Move initialization of benchmark data to the setUpFunction registered in BenchmarkInfo.

Large variants are useful in the regards that it takes a completely different code path and has different performance characteristics. I understand we should not do that for every test but it still should not regress imho.
The arc4random is being used to ensure the kernel does not just hand out a non faulting page of memory. Just being an allocation is not enough to properly identify used cases. It must be written to: and patterns like repeated are a completely different code path in the sequence initialization.

If we want to transition to a different random source that is fine in my book. It does not need to be arc4 but it should be something portable to Linux is all.

Can you clarify this part?

It must be written to: and patterns like repeated are a completely different code path in the sequence initialization.

To the best of my understanding, even the .veryLarge case just calls sampleData(size: 1024 * 1024 * 1024 + 128) which looks like this:
func sampleData(size: Int) -> Data {
    var data = Data(count: size)
    data.withUnsafeMutableBytes { getRandomBuf(baseAddress: $0, count: size) }
    return data
}

How is it different initialization pattern than this from IterateData?
  var data = Data(count: 16 * 1024)
  let n = data.count
  data.withUnsafeMutableBytes { (ptr: UnsafeMutablePointer<UInt8>) -> () in
    for i in 0..<n {
      ptr[i] = UInt8(i % 23)
    }
  }

I think that from the perspective of Data, which hands an UnsafeMutablePointer to the closure that fills it to capacity. As long as all the content is written to, it should look the same from kernel's perspective, too. Why do we need random data here?

Oh that differential is fine, I was just saying that Repeated (the sequence type) would behave differently.


CSS issue in Public Notification component

CSS issues are showing up when the PN content overflows vertically

This is looking good.


Add "WithError" versions of NewReader/NewGenericReader

It's currently difficult to use NewReader / NewGenericReader in a situation where an invalid file might be read, since they can panic when trying to read an invalid file.
This PR adds WithError versions of NewReader and NewGenericReader, which return an error if something goes wrong rather than panicing.
This PR also adds a testdata file which causes a New(Generic)Reader to panic, and adds tests ensuring that NewReaderWithError and NewGenericReaderWithError return an error when trying to read that file. It was that file which alerted me to this issue!
I figured it was easier to raise a PR for discussion than to create an issue for this - if this needs any work or if this isn't something you want to add to the library, please let me know!
Alternative Approach
Another approach would be to break the API and change the signatures to e.g. func NewReader(...) (*Reader, error) - given that this library isn't at v1 yet, that might be an option which some libraries would take.
I assumed that breaking existing code wouldn't be desirable here, though - if this library ever publishes a v2, that might be a thing to do then.
That said, if you'd be happy to break the API I'd gladly change this PR to do that instead!

Hi @SgtCoDFish thanks for your PR! This is certainly a change we can accept. I agree that it's better to not break the API just yet.
Something I'm not sure tho is the name of the function. Usually (in the stdlib for example) functions with WithError will expect the user to pass an error in parameter. Example: https://github.com/segmentio/parquet-go/pull/313.
Wondering if we should use something else like NewReaderOrError. What do you think ? Have you seen WithError used in other package already ?

Wondering if we should use something else like NewReaderOrError. What do you think ? Have you seen WithError used in other package already ?

Ah I much prefer OrError, yeah!
I thought about the name a bit and ended up just going with my first idea; I'm definitely happy to change it. I've made that change :grin:

Something I had documented (but maybe not highlighted enough) is the pattern which is currently supported to handle configuration errors; here is an example https://pkg.go.dev/github.com/segmentio/parquet-go#NewBuffer
I wonder if this would be an acceptable approach, or whether we feel like the new APIs are worth introducing still?

I wonder if this would be an acceptable approach, or whether we feel like the new APIs are worth introducing still?

I think if configuration errors were the only way that creating a (Generic)Reader could fail that would be an acceptable tradeoff - the issue I encountered though was that creating the reader will panic on an invalid parquet file, which I think is demonstrated by the test that I added (not_enough_columns.invalid.parquet). I don't think the configuration approach you've shown there would solve that issue?

Thanks for sharing these details.
You could open the file ahead of time to handle errors caused by invalid files, for example:
f, err := parquet.OpenFile(...)
if err != nil {
    ...
}
r := parquet.NewGenericReader[Row](f)

You could open the file ahead of time to handle errors caused by invalid files, for example:
Let me know if this is an acceptable solution for your use case.

This would probably work in our use case in the short term but it's quite verbose and fiddly. Using OpenFile means I need to manually calculate the size of my file, and it's just more boilerplate overall.
I like that NewGenericReader does all of this for me, so it would still be nice to be able to simply use NewGenericReaderOrError instead.
Another issue with panics is that there's no easy way for me to know whether a new failure mode which could cause a panic might be added to NewGenericReader in the future, which could cause my application to start to panic after I upgrade to a newer version of parquet-go. If it instead returns errors, my existing error handling code will handle those gracefully.

@achille-roussel gentle bump - does my reasoning above seem reasonable?


Corrupted pretrained model

I downloaded the pertained model and ran the code, but it seems that the pretrained model is corrected due to the following error:
Traceback (most recent call last):
  File "/home/code/SSAH-adversarial-attack/main.py", line 156, in <module>
    fid = return_fid(benign_img, adv_img)
  File "/home/code/SSAH-adversarial-attack/utils/fid_score.py", line 256, in return_fid
    dims=2048)
  File "/home/code/SSAH-adversarial-attack/utils/fid_score.py", line 239, in calculate_fid_given_paths
    model = InceptionV3([block_idx]).to(device)
  File "/home/code/SSAH-adversarial-attack/utils/inception.py", line 82, in __init__
    inception = fid_inception_v3()
  File "/home/code/SSAH-adversarial-attack/utils/inception.py", line 208, in fid_inception_v3
    state_dict = torch.load(os.path.join("checkpoints", "pt_inception-2015-12-05-6726825d.pth"))
  File "/home/anaconda3/envs/ssah/lib/python3.6/site-packages/torch/serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/anaconda3/envs/ssah/lib/python3.6/site-packages/torch/serialization.py", line 781, in _legacy_load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: unexpected EOF, expected 559999 more bytes. The file might be corrupted.

As given in the error message, you might want to download or create the file again, as it seems to be corrupted.  Or you can download again from the link https://github.com/mseitzer/pytorch-fid/releases/download/fid_weights/pt_inception-2015-12-05-6726825d.pth


Upgrade path from 0.8?

After we upgraded to 0.9 and rebooted the system, we get the following error:
/etc/init.d/api-umbrella: line 41: api-umbrella: command not found

As a result, the API Umbrella service will not load.

Can you try runninng the following commands and posting the output?
ls -l /usr/bin/api-umbrella*
ls -l /opt/api-umbrella/bin

Sorry for the delay in response, we got the 'broken' upgrade VM back online today. Here are the results of the commands you posted:
# ls -l /usr/bin/api-umbrella*
ls: cannot access /usr/bin/api-umbrella*: No such file or directory

$ ls -l /opt/api-umbrella/bin
total 0
lrwxrwxrwx 1 root root 46 marra 28 03:10 api-umbrella -> ../embedded/apps/core/current/bin/api-umbrella

Additionally, the following commands return nothing:
$ sudo ls -l /usr/bin | grep api
$ sudo ls -l /usr/bin | grep umbrella

If I understand where you are going with this, I ran the following commands:
$ sudo ln -s /opt/api-umbrella/bin/api-umbrella /usr/bin/api-umbrella
$ sudo service api-umbrella start
  * Starting API Umbrella api-umbrella

Still, the API Umbrella application does not load when requesting the server through a web browser.

We are trying the upgrade again with the following steps:
$ sudo lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

$ sudo apt-get update && sudo apt-get upgrade -y

Warnings during this process:
Unpacking api-umbrella (0.9.0-1~trusty) over (0.8.0-1) ...
dpkg: warning: unable to delete old directory '/opt/api-umbrella/embedded/apps/web/shared/log': Directory not empty
dpkg: warning: unable to delete old directory '/opt/api-umbrella/embedded/apps/web/shared': Directory not empty
dpkg: warning: unable to delete old directory '/opt/api-umbrella/embedded/apps/web': Directory not empty

Then, some configuration overwrite messages:
Configuration file '/etc/init.d/api-umbrella'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** api-umbrella (Y/I/N/O/D/Z) [default=N] ?

Configuration file '/etc/logrotate.d/api-umbrella'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer
...

I answered Y to both.

Thanks for reporting this. I've identified a couple issues with the upgrade process and some extra files being marked as config files in the .deb packages. I think we should have new packages released Monday or Tuesday to resolve these issues.

This has been fixed in https://github.com/NREL/api-umbrella/commit/629dc347f930b65fb963d7d372dae9368783a64b New v0.10 packages will be released tomorrow, which should make upgrading as easy as apt-get install api-umbrella.

v0.10 packages are now published. Give a shout if you're still having any trouble upgrading.

@GUI, I ran the install process using 0.10 packages on an 0.8 instance. The install seems to have succeeded, but the ApiUmbrellaWeb service does not respond to requests to / or /admin.
Additionally, I got an error when I tried to upgrade instead of install:
$ sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  api-umbrella

@brylie: After upgrading, can you try running sudo /etc/init.d/api-umbrella start? Sorry I forgot to mention that, but if you're upgrading from v0.8 on Ubuntu or Debian, you have to explicitly start API Umbrella again after upgrading. This shouldn't be an issue moving forward, since we're now explicitly testing the various upgrade paths for packages, but due to the post-removal package scripts in v0.8 and the ordering of callbacks for .debs, you'll need this manual start after upgrading this time (and we can't do much to solve it, since it's an issue with the v0.8 package).
Let me know if that doesn't do the trick, though.
Regarding apt-get upgrade not working, based on http://askubuntu.com/a/602, I think that's maybe expected since we had to add a dependency to the packages. It sounds like any dependency changes might trigger this, so apt-get install api-umbrella should probably be the preferred way to update the package (api-get upgrade also tries to update the whole system, so that may update more packages than you want).

@elnzv reported that he was able to successfully upgrade from 0.8 to 0.10, and will test on a clone of our main VM.

tailf /var/log/api-umbrella/nginx/current is looping error:
[emerg] 17617#0: could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32
@GUI any idea how I could fix this ? where I can find "server_names_hash_bucket_size" ?

@gui, may we please re-open this issue until we have a successful upgrade path?

@jykae the hash_bucket_size issue was discussed and marked as resolved here.


Add NumPy constants to ChainerX

Fixes #7368
See also: https://docs.scipy.org/doc/numpy/reference/constants.html

Jenkins, test this please.

Jenkins CI test (for commit 9ee3b376ee23fcf08481c812a83441889ebd163c, target branch master) failed with status FAILURE.

LGTM, failure is unrelated.


Clock Tower

The game works, but I can't put it in full screen

As far as I understand it's a windowed-only game, it doesn't have a fullscreen mode. DDrawCompat doesn't have any features to display windowed applications in fullscreen mode or vice versa, and it's unlikely to have them in the near future, if ever.
There are better tools for this, such as DxWnd. There are instructions on how to set it up at PCGamingWiki. Alternatively, you can ask for help on DxWnd forums, there seems to be a topic for this game already here: https://sourceforge.net/p/dxwnd/discussion/general/thread/d900d059/


tf.nn.ctc_loss behaviour changes depending on whether dense or sparse labels are provided

The tf.nn.ctc_loss changes its behaviour unexpectedly based on whether the labels provided are sparse or dense.
In particular, it accepts empty target label sequences in the dense version, and maximises (as one would expect) the log probability of the blank token at each time frame in the logits prediction matrix. It does not output any warning or error.
However, if you convert the label to be sparse by using tf.sparse_from_dense, given the same inputs (with empty target label sequence), it errors out with
tensorflow.python.framework.errors_impl.InvalidArgumentError: Labels length is zero in batch 0 [Op:CTCLoss]

This inconsistency in behaviour should be fixed.
Option 1: Ideally, we would implement support for empty labels in the case of sparse tensors.
Option 2: Add a warning to the CTC loss doc that empty labels are not supported in the case of sparse tensors.
Another inconsistency is that the dtype for logit_length can be int64 for the dense version, but needs to be int32 for the sparse version, and errors out if int64 is provided. Again, ideally this should be converted internally, but if it really cannot be fixed, the docs should reflect this, which they do not at the moment.
Function affected:
https://www.tensorflow.org/api_docs/python/tf/nn/ctc_loss
Usage example
This runs fine:
tf.nn.ctc_loss(
    labels=tf.ones((4, 50), dtype=tf.int32),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.constant(200, dtype=tf.int64, shape=4),
    logits_time_major=False,
    blank_index=0,
)

Converting to sparse labels:
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 50), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.constant(200, dtype=tf.int64, shape=4),
    logits_time_major=False,
    blank_index=0,
)

suddenly raises
tensorflow.python.framework.errors_impl.InvalidArgumentError: cannot compute CTCLoss as input #3(zero-based) was expected to be a int32 tensor but is a int64 tensor [Op:CTCLoss]

which is fixed by running
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 50), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

Now introducing empty label sequences:
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 0), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(0, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

raises
tensorflow.python.framework.errors_impl.InvalidArgumentError: Labels length is zero in batch 0 [Op:CTCLoss]

wheras the dense version is fine:
tf.nn.ctc_loss(
    labels=tf.ones((4, 0), dtype=tf.int32),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(0, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

Yes, can confirm, I get
InvalidArgumentError: cannot compute CTCLossV2 as input #3(zero-based) was expected to be a int32 tensor but is a int64 tensor [Op:CTCLossV2]

for box 4 (with sparse, non-empty labels)
Everything else runs fine. However, you didn't include the run with sparse and empty labels that I talked about above
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 50), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

@f90 Thank you for the quick update!
I have included  the run with sparse and empty labels ,please find the gist here.
Thanks!

No problem. I actually still see only the dense empty labels in the notebook. Do you want to link to the cell that you mean?

@f90,
I could able to reproduce the issue.Could you please confirm.
Sparse Tensors
import tensorflow as tf
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 50), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.constant(200, dtype=tf.int64, shape=4),
    logits_time_major=False,
    blank_index=0,
)

Output
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
[<ipython-input-2-70fe3e7f7321>](https://localhost:8080/#) in <module>()
      6     logit_length=tf.constant(200, dtype=tf.int64, shape=4),
      7     logits_time_major=False,
----> 8     blank_index=0,
      9 )

1 frames
[/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py](https://localhost:8080/#) in raise_from_not_ok_status(e, name)
   7184 def raise_from_not_ok_status(e, name):
   7185   e.message += (" name: " + name if name is not None else "")
-> 7186   raise core._status_to_exception(e) from None  # pylint: disable=protected-access
   7187
   7188

InvalidArgumentError: cannot compute CTCLossV2 as input #3(zero-based) was expected to be a int32 tensor but is a int64 tensor [Op:CTCLossV2]

Sparse tensor with cast int32
import tensorflow as tf
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 50), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(50, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

Output
<tf.Tensor: shape=(4,), dtype=float32, numpy=array([324.07513, 324.07513, 324.07513, 324.07513], dtype=float32)>
Sparse tensor with empty labels
import tensorflow as tf
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 0), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(0, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

Output
<tf.Tensor: shape=(4,), dtype=float32, numpy=array([458.2139, 458.2139, 458.2139, 458.2139], dtype=float32)>
Dense tensor with Empty labels
import tensorflow as tf
tf.nn.ctc_loss(
    labels=tf.ones((4, 0), dtype=tf.int32),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(0, dtype=tf.int64, shape=4),
    logit_length=tf.constant(200, dtype=tf.int64, shape=4),
    logits_time_major=False,
    blank_index=0,
)

Output
<tf.Tensor: shape=(4,), dtype=float32, numpy=array([459.82333, 459.82333, 459.82333, 459.82333], dtype=float32)>
Issue is with logit_length, It expects int32 Tensor.

@gadagashwini I can confirm these results with one exception: Running empty sparse labels does not work on my end:
import tensorflow as tf
tf.nn.ctc_loss(
    labels=tf.sparse.from_dense(tf.ones((4, 0), dtype=tf.int32)),
    logits=tf.zeros((4, 200, 10), dtype=tf.float32),
    label_length=tf.constant(0, dtype=tf.int64, shape=4),
    logit_length=tf.cast(tf.constant(200, dtype=tf.int64, shape=4), tf.int32),
    logits_time_major=False,
    blank_index=0,
)

gives me
2022-04-05 11:59:04.820963: W ./tensorflow/core/util/ctc/ctc_loss_calculator.h:499] No valid path found.
2022-04-05 11:59:04.822817: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at ctc_loss_op.cc:213 : INVALID_ARGUMENT: Labels length is zero in batch 0
Traceback (most recent call last):
  File "/Users/dstoller/PycharmProjects/lyric-align-baseline/lyric_align_baseline/ctc_example.py", line 32, in <module>
    blank_index=0,
  File "/Users/dstoller/.pyenv/versions/lyric-align-baseline/lib/python3.7/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/dstoller/.pyenv/versions/lyric-align-baseline/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 7186, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InvalidArgumentError: Labels length is zero in batch 0 [Op:CTCLoss]

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


Illegal reflective access

When instrumenting with the 2.1 snapshot i always get the following warning:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.javalite.instrumentation.ActiveJdbcInstrumentationPlugin (file:/Users/bla/.m2/repository/org/javalite/activejdbc-instrumentation/2.1-SNAPSHOT/activejdbc-instrumentation-2.1-SNAPSHOT.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of org.javalite.instrumentation.ActiveJdbcInstrumentationPlugin
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

Maybe you could have a look at it? Thank you!

what version of Java are you using?

9.0.4

JavaLite has not yet been certified for Java 9.

Will you do that sooner or later?

yes, it is coming. The pain of releasing Java 9 is that it is not compatible with previous versions of Java on the byte code level. This means that all JavaLite projects need to have parallel Jenkins builds, separately maintained branches, and appropriate documentation, all of which is a tom of extra work. Initial work has been already done: https://github.com/javalite/activejdbc/tree/java9 though. Additionally, the line
WARNING: Illegal reflective access by org.javalite.instrumentation.ActiveJdbcInstrumentationPlugin (file:/Users/bla/.m2/repository/org/javalite/activejdbc-instrumentation/2.1-SNAPSHOT/activejdbc-instrumentation-2.1-SNAPSHOT.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
is complaining about a hack in the Instrumentation plugin, which adds the project artifacts to the plugin classpath. I do not know of a good workaround yet.

Uuuhhh, i didn't know that.
So it's luck that it works quite well within my current Java 9 project...

yeah, unfortunately Java 9 is not backwards compatible with Java 8, which means I cannot compile JavaLite for Java9 and have it work on Java 8 or even 7. The project has to fork to multiple builds to get on Java 9. That warning has to be fixed though because in future versions (10, 11?) it will become an error.

additionally, there is already a fork for people stuck in Java 7:( == more work.

Any updates?

Will ActiveJDBC be JPMS-friendly once you move to 11? It's perfect for an app I'm writing, but my project uses JPMS and it looks like, as of 7/14/19, ActiveJDBC doesn't. So I'm seeing a lot of conflicts. And I cloned the Github project figuring that maybe I could make the mods myself,  but there are a half-dozen different Maven projects...

@stevesobol we will be focusing on releasing the next snapshot version on Java 11, so this should take care of JPMS, although I'm yet to learn  it :)

Module support is something you specifically have to add, even with Java 11. I can help, I’ve done it before...

On Jul 15, 2019, at 12:52, Igor Polevoy<EMAIL_ADDRESS>wrote:
@stevesobol we will be focusing on releasing the next snapshot version on Java 11, so this should take care of JPMS, although I'm yet to learn it :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@stevesobol that would be cool, feel free to fork the repo and give it a shot!

Could you do me a favor please, and shoot an email to<EMAIL_ADDRESS>when the snapshot hits your Github repo? Or let me know here, with a link to the commit. That way, I can work off a codebase that is working with Java 11. Thanks.

@stevesobol do you mean a snapshot with Java11 support?

@ipolevoy yes.


Backtesting and profit trailing support

Hi
Just discovered this piece of software. I was wondering if it is possible to do backtesting before deploying the bot and if it supports profit trailing.

This software is not supporting backtesting, if you want to backtest your strategy, you can try howtrader project , the following link is : https://github.com/51bitquant/howtrader.

Hi
Just discovered this piece of software. I was wondering if it is possible to do backtesting before deploying the bot and if it supports profit trailing.

This software is not supporting backtesting, if you want to backtest your strategy, you can try howtrader project , the following link is : https://github.com/51bitquant/howtrader.


Raise a more helpful error when join table can't be found

I was trying to create some joins and kept getting this cryptic error:
NoMethodError: undefined method `[]' for nil:NilClass
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/schema.rb:58:in `join'
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/relation/reading.rb:541:in `block in __join__'
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/relation/reading.rb:541:in `each'
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/relation/reading.rb:541:in `reduce'
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/relation/reading.rb:541:in `__join__'
  from /Users/george/.rvm/gems/ruby-2.3.3/bundler/gems/rom-sql-c46d3c34c071/lib/rom/sql/relation/reading.rb:436:in `left_join'

turns out that I was doing it wrong: inner_join(:task, user_id: :id), when
actually I should have passed :tasks as the first argument (i.e. the name of
the table.) An easy mistake to make since my user actually only has one task,
so I'm used to thinking about 'user.task' in the singular, not plural.
This PR raises a more helpful error when you get the name of the table wrong...
except it doesn't actually work, because ROM::SQL::Error has some extra magic
with original_exception that prevents raise ROM::SQL::Error, 'message' from
working as you'd expect.
Before I finalise this I figured I'd open the PR and a) make sure this is
actually a change you'd accept and b) get some feedback on whether I'm better
off raising ROM::SQL::Error (with some tweaks or hacks to make it actually
work) or using a different error class altogether.

Thanks, I actually thought about doing something else - it feels like it'd be better to actually accept relations as arguments in *_join methods, so inner_join(tasks) instead of inner_join(:tasks). This way the lookup in schema is just relations[relation.name]. This has nice benefit that aliases will automatically work, you people won't have to think about whether they need to use dataset name or relation name (in case it's the same). WDYT?

I ended up removing this whole thing, so I need to close this. Thanks anyway :)


Some Bug Views Are Too Wide

Yet another responsive sizing issue.

There are two issues. The first is a result of the table-like view. Particularly anything that has a common name will appear too wide in a narrow vertical view. The second is that if a view is opened in wide mode, and then rotated to a narrower view the scrollable widget may not be resized for width to fit within its container.

Invertebrate attributes stack vertically on small screens instead of layout left to right.


Update dependency bounds

All tests passed locally with DataFrames 0.20.0 and Unitful 0.18.0.

Thanks


translation.translatable_id shouldn't be nullable

Any consideration about this thread?

I don't think it's nullable right now. Is it?
https://github.com/KnpLabs/DoctrineBehaviors/blob/master/src/ORM/Translatable/TranslatableSubscriber.php#L218-L229

I'm going through the old issues/PRs and closing most of them so we can focus on now.
This one is missing any feedback or reasoning so I cannot deduce anything from it.
Thanks for you though, but closing for reasons above.
If the bug still remains, please send failing test case to verify it. Only with test we'll be albe to help.


The homepage navbar becomes unresponsive when the viewport width falls below 973 pixels.

Please assign this issue to me I want to work on it.


Purge pollers after device inactivity

To free up resources, we should automatically purge poller processes for devices that have become inactive.

Inactive devices have not synced for a certain period of time
The inactivity interval should be configurable (default: 30 days?)
If a device comes back after inactivity, polling should recover properly

The inactivity interval should be configurable (default: 30 days?)

My vote would be for choosing a sensible default rather than exposing a config option.
Proposed impl:

Hourly background job which looks for pollers whose devices haven't synced for $INACTIVITY_THRESHOLD.
Something like: select user_id, device_id from tokens group by (user_id, device_id) where max(last_seen_ts) < $NOW - $INACTIVITY_THRESHOLD. It's a scan of the tokens table, but no worse than we do on startup.
Needs a DB conn, so probably put it in handler2.Handler.
Ask the poller map to stop any of these devices' pollers.

For each such device: act as if the token has been expired (which means deleting it from the DB). In particular, emit V2ExpiredToken on pubsub.
Assuming that the poller startup has completed, there shouldn't be any devices we've found that are unknown to poller. (If there were, we ought to have expired/deleted them already).
To enforce that assumption, don't schedule hourly cleanup until the pollers have started up.
Log when we expire a token due to inactivity
Log how many inactive tokens we found and how many pollers we closed

For simplicity's sake, don't bother to do this kind of cleanup on startup.

My vote would be for choosing a sensible default rather than exposing a config option.

Alright, convention over configuration. We might change this later if needed. 30 days even seems a bit long but would cover e.g., longer vacation periods. Anything between 14 and 30 days works for me, I think.

Should be finished soon.


Class subscription with __class_getitem__ not working

Environment data

Language Server version: 2021.6.1-pre.1
OS and version: Windows 7
Python version (& distribution if applicable, e.g. Anaconda): 3.8.10 Anaconda

Sample
class C:
    def __class_getitem__(cls, key) -> str:
        return "C"

s = C[0]

Expected behavior
According to the signature of C.__class_getitem__, type of C[...] should be str.
Actual behavior

__class_getitem__ is meant to "Return an object representing the specialization of a generic class by type arguments found in key" according to the official Python documentation.
Can you explain what you're trying to do here?

I was writing something that supposed to be used like constants but with some arguments.
For example, there is a member in my module named RSV (Raw Stochastic Value) and it would be used with the form like RSV[10] ( a 10-day RSV factor ). The square brackets are here to make the expression differ from the form of function calls in visual. RSV[10] itself would be considered a constant, while RSV[5] would be considered another constant. I think it would be much more intuitive that RSV[10] is RSV[10] than RSV(10) is RSV(10).
I used to implement RSV as an instance of a private factory class, like
class _RSV:
    def __getitem__(self, key): ...

RSV = _RSV()

but then I found that there is the __class_getitem__ method and tried to eliminate one redundant class.
So, if __class_getitem__ is not supposed to be used like this, I will revert my recent changes.

After reading the document you gave, I see that I used __class_getitem__ wrong.


[CSS] SyntaxError: Unexpected character '?'

Hello,
When I tried to include font-awesome in vender.ts I got an error that "@" is not recognised.
SyntaxError: Unexpected character '�'
To fix it in webpack.config.js I replaced:
{test: /\.(png|jpe?g|gif|svg|woff|woff2|ttf|eot|ico)$/, loader: 'file?name=fonts/[name].[hash].[ext]?'},
with
{test: /\.(png|jpe?g|gif|svg|woff|woff2|ttf|eot|ico)(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: 'file?name=fonts/[name].[hash].[ext]?'},
Now I can just install font-awesome with:
npm install font-awesome
and include in vendor.ts
import 'font-awesome/css/font-awesome.css';

Thanks for the detailed issue. I will try what you did and your solution.

I think I did this long ago.


fix: remove useless type assertion

Currently makes tsc panic on "npm start"

Already fixed on master


TypeError: can't pickle NotImplementedType objects using KerasRegressor

When I use KerasRegressor for testing by scikit_learn cross-validation method, it gives this error "TypeError: can't pickle NotImplementedType objects". A script which can produce this error is given in "https://github.com/liyakun/time_series/blob/master/lib.py"

Having the same issue with Lambda function as output of a model, when trying to save the model.
state_in = ...
h = ...
scale_matrix = K.constant(c) # c is a numpy matrix
out = Lambda(lambda x: K.dot(x, scale_matrix), output_shape=(n_out,))(h)

model = Model(inputs=[state_in], outputs=[out])
model.save(filename)

trying the same as @hamzamerzic , a previous issue suggested using json to save the model and the save_weights method to save the learned weights.
this isn't working for me either.

I have the same TypeError: can't pickle NotImplementedType objects error that @hamzamerzic and @damnMeddlingKid have when using Lambda in my model when attempting to model.get_config()


include version numbers on release notes and on Data Metrics & Plots page

@andrewhercules commented on Wed Jul 11 2018
@deniseOme commented on Wed Jul 26 2017
Things such as EFO version 2.84, COSMIC version 81, EVA, ClinVar April release number collected from here:
https://github.com/opentargets/data-providers-docs/wiki/Data-(Evidence-Strings)-Submission-Process
Should go in the release notes, not blog as it's too much of dry piece of info.
Note we have not included those in the past, AFIK.
So if we decide to include them now, we should do it onwards too for consistency.
As per the first release in netlify, I suggest it on the release notes rather than blog.
The same for orthologues table, web bug fixes and protein/RNA expression data.
What do you think @emepyc @elipapa?

@deniseOme commented on Wed Jul 26 2017
Also I will point the release blog post to the release notes for more details.

@emepyc commented on Wed Jul 26 2017
We can include versions for the data sources in the release notes. Another place is the data sources doc.
As for the netlify, it is transparent to the user and I don't see it very useful from a user point of view.
The current release notes for 3.2 already includes the orthologues table, bug fixes, etc

@peatroot commented on Fri Feb 02 2018
@deniseOme, the URL above does not work anymore. Do you know where this information has moved to? The provider versioning does not appear to be in the release notes, but I agree it'd be useful for future versions.

@andrewhercules commented on Wed Jul 11 2018
Moving issue to the data_release repository as data provider version numbers could be integrated into the Data Metrics & Plots wiki page and the webapp could link directly to that wiki once the version numbers are available.

@deniseOme commented on Wed Jul 11 2018
This is the new URL @peatroot:
https://github.com/opentargets/data_release/wiki/Data-Submission

Closing as in hand with QC automation and release process teamwork.


Double-check docker_compose_path order

In models.py
I don't understand why we're mucking with the PATH here, this seems fragile and likely to break things.  @amfarrell - do we still need this code?

When I installed docker-compose as a python package and then ran it under sudo (as you need to do if you don't implement https://docs.docker.com/installation/ubuntulinux/#giving-non-root-access), It could not find docker-compose because miniconda the path is reset when you run commands under sudo. The answer is to either:

install docker-compose into some place already in the secure_path found in /etc/sudoers
run the docker client as a non-root user. (see https://docs.docker.com/installation/binaries/#giving-non-root-access)
continue to refer to the particular binary.

I suspect that #2 is the right choice.

Set the PATH on the sudo calls: sudo env "PATH=$PATH"

Alright, let's get this fixed.  I'm okay with 1, 2, or 4.

No longer relevant.


Incompatibilty with NumPy 2

I am in need of the magix in pydantic-numpy. The following combination of versions works :
uv pip install netCDF4==1.6.5 numpy==1.26.4 pydantic-numpy==5

Upgrading to
numpy              2.1.2
pydantic-numpy     6.0.0
netcdf4            1.7.1.post2

breaks my program, it returns
Segmentation fault (core dumped)

I have a quite complex program and I'd need some time to create a minimal non-working example. But, is the latest version of pydantic-numpy at all compatible with NumPy 2 ?

I haven't ran the tests for v2.1 yet, but they ran fine with v2.0. Will run them for v2.1; let's see if they fail.

All tests are passing unfortunately

Closing. I am open to revisiting the problem you are having just not under this issue, this is too vague! You may refer to this issue if you want.

I uploaded a new version of the package with bumped dependencies, might be worth a shot.

I have no idea what the problem is exactly with netCDF4. Ditching it in favor of https://github.com/h5netcdf/h5netcdf/tree/v1.4.0 along withNumPy 2.1.2, pydantic-numpy 6.0.1 just works -- set aside that all runs faster by quite some hundreds of milliseconds !


AccessTokenHandler - Pass the innerHandler into the TokenClient

When using the AccessTokenHandler ctor that created a TokenClient, need to pass the innerHandler into that also.
For this to work, TokenClient (similar to DiscoveryClient) ctors need to be able to take a null innerHandler and create a default instead of throwing a ArgumentNullException.

ok ;)


[output] Make the output on the first start more streamlined in the root.go

Is there an existing issue for this?

[X] I have searched the existing issues

Is your feature request related to a problem? Please describe
At the moment the output after the installation of a new version of DDEV is the following /cmd/ddev/cmd/root.go
Congratulations, you seem to have a new DDEV version.
Please note that the global ~/.mutagen folder is no longer used by DDEV so you can delete it if it's no longer used by other applications.
It looks like you have a new DDEV version. During an upgrade it's important to `ddev poweroff`. May I do `ddev poweroff` before continuing? This does no harm and loses no data.

Problem is for one that you have "congratulations you have a new version of DDEV" in the first sentence and in the third sentence you have more or less the same again. the second sentence could be removed imho.
the second sentence could be broken up into two. the first part should be a plain explanatory statement that ~/.mutagen isn't used by DDEV any longer. the second part is changed into a clear actionable item in case the folder isn't used by any other application.
Describe the solution you'd like
I would suggest the following draft:
Congratulations, you seem to have a new DDEV version.
Please note that the global ~/.mutagen folder is no longer used by DDEV. Delete the folder if it is no longer used by any other application.
During an upgrade DDEV has to be shut down. May I do `ddev poweroff` to be on the safe side before continuing? This causes no harm or data loss.

I'll try to open up a pull request later today to test the fork/branch process via the terminal.
Describe alternatives you've considered
No response
Additional context
No response

If you're still interested in this, a new PR is welcome. Otherwise, let's close it about 1 August 2023.

Closing, almost a year late :)


A few improvements in smoke-test script

Closing this PR as this code changes are already covered in #25


chore(deps): update common-dev-assets digest to 5d1db25

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
26c4df2 -> 5d1db25

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 5.10.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


MacOS sleep mode causes Keybase to crash

It's not a repeatable thing that happens. The only commonality I can detect is linked to putting my MacBook Pro to sleep. When I wake my laptop from sleep mode, I sometimes see an error dialog stating that keybase has crashed. This is the first time I saw the keybase log send terminal command.
my log id: c133a56e0a0d60a129dec81c

ATTN: Issue Support Team peeps
Apologies for posting this "new" issue. Should have checked before hitting submit. Seems #8377 is a duplicate of many other similar issues. Most notably, #3541 and #6540 (among others, I'm sure).
Feel free to close or link this issue as needed to remove issue "noise".
Thank you!


Work with libfabric community to decide a solution for FI_SELECTIVE_COMPLETION

The latest libfabric verbs implementation decide to turn on sq_sig_all on calling ibv_create_qp(). This will stop FI_SELECTIVE_COMPLETION from working: all RDMA write will receive a completion entry no matter if the FI_COMPLETION flag is set or not.
@dmitrygladkov insists this design but let libfabric keeps only demanded on in the completion queue. see this pull request.
I posted this solution because I believe the performance is going to be better.
I am discussing with @dmitrygladkov.

Verbs devices require the periodic generation of a completion, even if the application never wants one, or the send queue will block.  For example, if the send queue size is 512, then there MUST be a completion generated at least once for every 512 transfers. The completion indicates that all previously posted operations have completed, which frees the send queue slots for re-use.  Without a completion being generated, the verbs software has no way of knowing when it is safe to re-use a send queue slot for a new operation.
Because libfabric doesn't have this requirement, it needs to ensure that a completion is generated for apps that submit more "no completion requests" than allowed by the underlying device.  This results in a significant amount of overhead, which was removed from the upstream libfabric code.  The commit that removed this support is here:
https://github.com/ofiwg/libfabric/commit/00c1bb1a48a48a2d92d3f95517e4f0b53b2cbff1
To re-enable the ability to completely disable completion generation requires re-implementing something similar to what was removed in the above patch.  Basically, it's non-trivial, and most application developers aren't even aware of this limitation.

Yes, we have to periodically clear outstanding requests in the NIC. We turn the completion off when initializing the queue pair (qp_init_attr.sq_sig_all = 0;) and selectively post operations that enable completions every few milliseconds. A successful completion for an RDMA operation on a queue pair clears all the outstanding requests on that queue pair.

I think we can take libfabric fix #4475 for now. If we see significant overhead, we can re-consider a sophisticated solution later.

Upgrade to the latest libfabric with the fix. See commit f3db843807b6d2df4ab55933eedcd20e92291fb4.


⚠ Bofrost.be has degraded performance

In 5c2076f, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2958 ms

Resolved: Bofrost.be performance has improved in ac786e2.


save the GitHub commit SHA to the setup file

I think it would be a good idea to explore if it is possible to also write the GitHub commit SHA to the setup file for PredSims. In that way you always know which 'version' of the code you used.

to check; what happens when repo is downloaded as zip

to check; what happens when repo is downloaded as zip

then status is not 0, issueing a warning
https://github.com/KULeuvenNeuromechanics/PredSim/blob/c75abf3dee04e2b064b8ac6894cde96c0e0644ae/DefaultSettings/get_git_hash.m#L27-L29


Unable to run. Checked all factors but can not find the reason.

My platform is Nvidia Jetson Nano. Jetpack Version is 4.5.1. The CUDA version that comes with the system is 10.2. The Deepstream I installed is 5.1, It has been tested with official examples and installed successfully. The kafka I installed is 2.13-2.6.0 as the README in the repo. The quickstart event topic was successfully created.
I switched the libnvds_msgconv.so in the opt/nvidia/deepstream/deepstream-5.1/lib/ with the one in the repo cloned which is opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream-occupancy-analytics/bin/jetson/.
I also changed all the '5.0' to '5.1' in the config/test5_config_file_src_infer_tlt.txt
The Build was successful.
In the Run phase, it says:
(deepstream-test5-analytics:24279): GLib-CRITICAL **: 17:24:37.349: g_strchug: assertion 'string != NULL' failed
(deepstream-test5-analytics:24279): GLib-CRITICAL **: 17:24:37.349: g_strchomp: assertion 'string != NULL' failed
** WARN: <create_pipeline:1141>: Num of Tiles less than number of sources, readjusting to 2 rows, 1 columns
Warning: 'input-dims' parameter has been deprecated. Use 'infer-dims' instead.
** ERROR: main:1088: Failed to set pipeline to PAUSED
Quitting
ERROR from sink_sub_bin_sink2: Could not initialize supporting library.
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvmsgbroker/gstnvmsgbroker.c(359): legacy_gst_nvmsgbroker_start (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
unable to open shared library
ERROR from sink_sub_bin_sink2: GStreamer error: state change failed and some element failed to post a proper error message with the reason for the failure.
Debug info: gstbasesink.c(5265): gst_base_sink_change_state (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
Failed to start
App run failed
I have checked all the factors I know but still can not solve the problem. Can someone give me a hand?

Hi noobdeveloper,
The issue seems to be the kafka server. could you check if the kafka is running fine before starting the app?

Hi noobdeveloper,
The issue seems to be the kafka server. could you check if the kafka is running fine before starting the app?

Hi noobdeveloper,
The issue seems to be the kafka server. could you check if the kafka is running fine before starting the app?

Later I found the reason. The librdkafka was not installed correctly before the DS installation.


Optional sbml_warning argument to generate no SBML related warnings

We should tag SBML generated from bioscrape and biocrnpyler so that the warning about things not working doesn't show up every time we simulate from a file.

Fixed with #50.


Add tests for batch unlock - locked vs. unlocked varying amounts per participant

A participant P1's locked_amount1 (from P1's balance proof) actually contains:

amount_that_can_be_unlocked - if the secret was revealed => P2 must receive this amount
amount_that_cannot_be_unlocked - if the secret was not revealed => P1 must receive this amount back, because the transfer was never finalized.

We must add tests for the settlement calculations that will ensure we compute the locked amount properly, in a way that the batch unlock has enough tokens left in the contract for it to be resolved fairly.
Cases:

a very high amount_that_can_be_unlocked with a very low amount_that_cannot_be_unlocked
a very low amount_that_can_be_unlocked with a very high amount_that_cannot_be_unlocked

Test cases:
A. Both balance proofs are valid and are the last known balance proofs.
Participant P1
deposit1 = 10
withdrawn1 = 0
transferred1 = 0
locked1 = 25

Participant P2
deposit2 = 5
withdrawn2 = 0
transferred2 = 0
locked2 = 15

all secrets are revealed for the pending transfers P1 -> P2, locked1 must be transferred to P2
all secrets are revealed for the pending transfers P2 -> P1, locked2 must be transferred to P1

none of the secrets are revealed for the pending transfers P1 -> P2, locked1 must be transferred to P1
all secrets are revealed for the pending transfers P2 -> P1, locked2 must be transferred to P1

all secrets are revealed for the pending transfers P1 -> P2, locked1 must be transferred to P2
none of the secrets are revealed for the pending transfers P2 -> P1, locked2 must be transferred to P2

Note: both 2 & 3 are needed, due to the current settlement algorithm, which computes the participant amounts based on whether transferred1 + locked1 is >= or <= than transferred2 + locked2
B. One of the balance proofs is old. We take P1s balance proof as being outdated.
Participant P1
deposit1 = 10
withdrawn1 = 0
transferred1 = 0
locked1 = 25

Participant P2
deposit2 = 5
withdrawn2 = 0
transferred2 = 0
locked2 = 100

Same 1, 2, 3 cases as for A.

I have made a more detailed analysis about the settlement algorithm here: https://github.com/raiden-network/raiden-contracts/issues/188.
This issue is on hold until the issues raised in the above one are discussed.

@loredanacirstea I think this can be closed right?

This is actually solved in https://github.com/raiden-network/raiden-contracts/pull/189


Updated Build Instructions

The Build Instructions lacked information about things needed to build CZMQ.
Added That.

@nootanghimire This is the first ever pull request for unicorn and for Muktakosh! Lots of :heart: for getting this done. Really appreciated!
One small note: Can you please make another commit and add your information in the CREDITS file? We need that for the legalities :-)

Thanks @kaustavdm . I hope to contribute more in upcoming days! I've updated my info on CREDITS file. Thanks again! :)


Plugin 1.16.0 can not resolve IBM Secrets Manager secret that resolved in 1.14.0

Describe the bug
While preparing for the depreciation of the IBM Secrets Manager v1 API, tested the new 1.16.0 version with our current secret definitions that work under 1.14.0 we get an error about it not being able to resolve a placeholders
To Reproduce
Steps to reproduce the behavior:

Create secret in IBM secrets manager
Create a yaml secret for the secret
Run program

Expected behavior
The process should replace the value references in the template secret.yaml file with the correct values from IBM secrets manager.
Screenshots/Verbose output
secret.yaml
---
kind: Secret
apiVersion: v1
metadata:
  name: argocd-github
  namespace: argocd
  annotations:
    avp.kubernetes.io/path: "ibmcloud/kv/secrets/groups/034b2403-7908-692f-ed03-d98096ce9c98"
data:
  username: <argocd-github-creds | jsonPath {.username} | base64encode>
  password: <argocd-github-creds | jsonPath {.password} | base64encode>

Secret from secrets manager
% ibmcloud secrets-manager secret --id "66487b09-b1a1-5c33-2e28-33da6e505830" --service-url https://46900e8a-09a1-4c5a-beb2-b55a43d45b40.us-south.secrets-manager.appdomain.cloud --output json | jq -r
{
  "created_at": "2022-09-01T17:12:32.000Z",
  "created_by": "xxxxxxxx",
  "crn": "xxxxxxxx",
  "data": {
    "password": "xxxxxxxx",
    "username": "xxxxxxxx"
  },
  "downloaded": true,
  "id": "66487b09-b1a1-5c33-2e28-33da6e505830",
  "locks_total": 0,
  "name": "argocd-github-creds",
  "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
  "secret_type": "kv",
  "state": 1,
  "state_description": "active",
  "updated_at": "2022-09-01T17:12:53.000Z",
  "versions_total": 1
}

Output from command run
% kustomize build . | argocd-vault-plugin generate - --verbose-sensitive-output
2023/08/10 20:32:09 reading configuration from environment, overriding any previous settings
2023/08/10 20:32:09 AVP configured with the following settings:

2023/08/10 20:32:09 avp_kv_version: 2

2023/08/10 20:32:09 IBM Cloud Secrets Manager enabling 3 API call retries with 20 seconds between tries
2023/08/10 20:32:09 IBM Cloud Secrets Manager listing secrets of from group 034b2403-7908-692f-ed03-d98096ce9c98 starting at offset 0
2023/08/10 20:32:10 IBM Cloud Secrets Manager list secrets in group HTTP response: {
    "StatusCode": 200,
    "Headers": {
        "Cache-Control": [
            "no-store, no-cache, must-revalidate, proxy-revalidate,no-cache, max-age=0, no-store"
        ],
        "Cf-Cache-Status": [
            "DYNAMIC"
        ],
        "Cf-Ray": [
            "7f4c61418f9e1435-PIT"
        ],
        "Content-Security-Policy": [
            "default-src 'none'"
        ],
        "Content-Type": [
            "application/json; charset=utf-8"
        ],
        "Correlation-Id": [
            "20a22fef-d9d0-43e6-9108-4de80c9904f9"
        ],
        "Date": [
            "Fri, 11 Aug 2023 00:32:11 GMT"
        ],
        "Expires": [
            "0"
        ],
        "Pragma": [
            "no-cache"
        ],
        "Server": [
            "cloudflare"
        ],
        "Strict-Transport-Security": [
            "max-age=31536000; includeSubDomains; preload"
        ],
        "Surrogate-Control": [
            "no-store"
        ],
        "Vary": [
            "Accept-Encoding"
        ],
        "X-Content-Type-Options": [
            "nosniff"
        ],
        "X-Envoy-Upstream-Service-Time": [
            "510"
        ]
    },
    "Result": {
        "total_count": 4,
        "limit": 200,
        "offset": 0,
        "first": {
            "href": "xxxxxxxx"
        },
        "last": {
            "href": "xxxxxxx"
        },
        "secrets": [
            {
                "created_by": "xxxxxxxx",
                "created_at": "2022-09-19T21:11:22.000Z",
                "crn": "xxxxxxxx",
                "downloaded": true,
                "id": "5561dcf8-a1ae-6da8-6114-c567192d130d",
                "locks_total": 0,
                "name": "ds-nonprod-dal-vault-plugin",
                "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
                "secret_type": "iam_credentials",
                "state": 1,
                "state_description": "active",
                "updated_at": "2023-06-21T13:01:38.000Z",
                "versions_total": 5,
                "ttl": "7776000",
                "api_key_id": "xxxxxxx",
                "service_id": "xxxxxxxx",
                "service_id_is_static": true,
                "reuse_api_key": true,
                "rotation": {
                    "auto_rotate": false
                }
            },
            {
                "created_by": "xxxxxxxx",
                "created_at": "2022-11-15T15:09:31.000Z",
                "crn": "xxxxxxxx",
                "description": "Used by ArgoCD to get secrets for all nonprod clusters. API key owned by service id devops-nonprod. ServiceId-72778aae-373d-4a3f-a870-d45ddacaa9c9",
                "downloaded": true,
                "id": "ed311044-40c0-bed5-0a91-1468874d5fda",
                "locks_total": 0,
                "name": "nonprod-vault-plugin",
                "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
                "secret_type": "iam_credentials",
                "state": 1,
                "state_description": "active",
                "updated_at": "2023-06-19T14:33:30.000Z",
                "versions_total": 5,
                "ttl": "7776000",
                "api_key_id": "xxxxxxxx",
                "service_id": "xxxxxxxx",
                "service_id_is_static": true,
                "reuse_api_key": true,
                "rotation": {
                    "auto_rotate": false
                }
            },
            {
                "created_by": "xxxxxxxx",
                "created_at": "2022-09-01T17:12:32.000Z",
                "crn": "xxxxxxxx",
                "downloaded": true,
                "id": "66487b09-b1a1-5c33-2e28-33da6e505830",
                "locks_total": 0,
                "name": "argocd-github-creds",
                "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
                "secret_type": "kv",
                "state": 1,
                "state_description": "active",
                "updated_at": "2022-09-01T17:12:53.000Z",
                "versions_total": 1
            },
            {
                "created_by": "xxxxxxxx",
                "created_at": "2022-07-28T20:20:19.000Z",
                "crn": "xxxxxxxxx",
                "description": "Credentials for GitHub provider in order to manager GitHub teams",
                "downloaded": true,
                "id": "c00a6598-09ed-626d-1ee9-1e2da41fb42d",
                "locks_total": 0,
                "name": "github-provider-creds",
                "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
                "secret_type": "username_password",
                "state": 1,
                "state_description": "active",
                "updated_at": "2022-07-28T20:25:40.000Z",
                "versions_total": 1,
                "rotation": {
                    "auto_rotate": false
                }
            }
        ]
    },
    "RawResult": null
}

2023/08/10 20:32:10 IBM Cloud Secrets Manager get secret: getting secret argocd-github-creds of type kv from group 034b2403-7908-692f-ed03-d98096ce9c98
2023/08/10 20:32:11 IBM Cloud Secrets Manager get unversioned secret 66487b09-b1a1-5c33-2e28-33da6e505830 HTTP response: {
    "StatusCode": 200,
    "Headers": {
        "Cache-Control": [
            "no-store, no-cache, must-revalidate, proxy-revalidate,no-cache, max-age=0, no-store"
        ],
        "Cf-Cache-Status": [
            "DYNAMIC"
        ],
        "Cf-Ray": [
            "7f4c6145caaa1435-PIT"
        ],
        "Content-Security-Policy": [
            "default-src 'none'"
        ],
        "Content-Type": [
            "application/json; charset=utf-8"
        ],
        "Correlation-Id": [
            "249d6d2b-31ab-de50-d76f-8e8b291d589a"
        ],
        "Date": [
            "Fri, 11 Aug 2023 00:32:11 GMT"
        ],
        "Expires": [
            "0"
        ],
        "Pragma": [
            "no-cache"
        ],
        "Server": [
            "cloudflare"
        ],
        "Strict-Transport-Security": [
            "max-age=31536000; includeSubDomains; preload"
        ],
        "Surrogate-Control": [
            "no-store"
        ],
        "Vary": [
            "Accept-Encoding"
        ],
        "X-Content-Type-Options": [
            "nosniff"
        ],
        "X-Envoy-Upstream-Service-Time": [
            "285"
        ]
    },
    "Result": {
        "created_by": "xxxxxxxxx",
        "created_at": "2022-09-01T17:12:32.000Z",
        "crn": "xxxxxxxxxxx",
        "downloaded": true,
        "id": "66487b09-b1a1-5c33-2e28-33da6e505830",
        "locks_total": 0,
        "name": "argocd-github-creds",
        "secret_group_id": "034b2403-7908-692f-ed03-d98096ce9c98",
        "secret_type": "kv",
        "state": 1,
        "state_description": "active",
        "updated_at": "2022-09-01T17:12:53.000Z",
        "versions_total": 1,
        "data": {
            "password": "xxxxxxxx",
            "username": "xxxxxxx"
        }
    },
    "RawResult": null
}

2023/08/10 20:32:11 calling GetSecrets to get all secrets from backend because avp.kubernetes.io/path is set to ibmcloud/kv/secrets/groups/034b2403-7908-692f-ed03-d98096ce9c98
2023/08/10 20:32:11 found placeholder argocd-github-creds with modifiers [ jsonPath {.username}   base64encode]
2023/08/10 20:32:11 processing modifier jsonPath with args ["jsonPath" "{.username}"]
2023/08/10 20:32:11 found placeholder argocd-github-creds with modifiers [ jsonPath {.password}   base64encode]
2023/08/10 20:32:11 processing modifier jsonPath with args ["jsonPath" "{.password}"]
Error: Replace: could not replace all placeholders in Template:
jsonPath: username is not found for placeholder argocd-github-creds in string username: <argocd-github-creds | jsonPath {.username} | base64encode>
jsonPath: password is not found for placeholder argocd-github-creds in string password: <argocd-github-creds | jsonPath {.password} | base64encode>

Additional context

This has worked fine under previous version of argocd-vailt-plugin
IBM is removing support for api/v1 in October 2023

This is a bug - KV replacement works if you use the .data key in the jsonPath, but that's not how AVP should handle this. Proper fix is to copy the KV pairs so they can be directly referenced like you expect. Mentioned PR should fix it

Released here https://github.com/argoproj-labs/argocd-vault-plugin/releases/tag/v1.16.1


Make sure we use the default featurePrefix

This seems to be a change between 4.0.1 and 4.2.0, previously, when not specifying featurePrefix in a transaction, you would get feature but now you get undefined, this PR restores the old behaviour

@Jenselme any chance you could have a look at this one as a reviewer?

ah you're right @Jenselme I will wait for another review then, thanks for taking the time to look at it though

thanks @ahocevar for the review


Release 0.3.1 cannot be compiled

File wscript line 31, upstream:
VERSION = "0.3.1+"

release 0.3.1:
VERSION = "0.3.1"

That is causing following error during building package:
 File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/wscript", line 201, in configure
    VERSION_SUFFIX = VERSIONS[1]
IndexError: list index out of range

Full log:
Setting top to                           : /home/user/AUR/diorite-rolling/src/diorite-0.3.1
Setting out to                           : /home/user/AUR/diorite-rolling/src/diorite-0.3.1/build
Version                                  : 0.3.1
Upstream revision                        : unknown (unsupported build)
Target platform                          : LINUX
Install prefix                           : /usr
Checking for 'gcc' (c compiler)          : /usr/bin/gcc
Checking for program pkg-config          : /usr/bin/pkg-config
Checking for 'gobject-2.0'               : yes
Checking for 'gthread-2.0'               : yes
Checking for program valac-0.8           : not found
Checking for program valac               : /usr/bin/valac
Checking for valac version >= (0, 8, 0)  : (0, 34, 4)
Checking for program valac-0.22          : /usr/bin/valac
Checking for valac-0.22 version >= (0, 22, 1) : (0, 34, 4)
Compiler optimizations                        : ON
Checking for 'glib-2.0' >= 2.40               : yes
Checking for 'gthread-2.0' >= 2.40            : yes
Checking for 'gio-2.0' >= 2.40                : yes
Checking for 'gtk+-3.0' >= 3.10               : yes
Checking for 'gdk-3.0' >= 3.10                : yes
Checking for 'gdk-x11-3.0' >= 3.10            : yes
Checking for 'x11' >= 0                       : yes
Traceback (most recent call last):
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Scripting.py", line 97, in waf_entry_point
    run_commands()
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Scripting.py", line 153, in run_commands
    ctx=run_command(cmd_name)
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Scripting.py", line 146, in run_command
    ctx.execute()
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Configure.py", line 128, in execute
    super(ConfigurationContext,self).execute()
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Context.py", line 84, in execute
    self.recurse([os.path.dirname(g_module.root_path)])
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/.waf3-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Context.py", line 125, in recurse
    user_function(self)
  File "/home/user/AUR/diorite-rolling/src/diorite-0.3.1/wscript", line 201, in configure
    VERSION_SUFFIX = VERSIONS[1]
IndexError: list index out of range
==> ERROR: A failure occurred in build().
    Aborting...

same issue on Fedora 25 !

Hello @petRUShka & @martinkg. I apologize for that problem. I should have tested it more carefully. However, 0.3.1 is not a supported release but merely a development snapshot. Please use the latest git master snapshot instead.

Or you can apply https://github.com/tiliado/diorite/commit/26321c12ece8e67c2365883ff480fb75193161df#diff-dc6134e3d3c869a052520d86f46371dc


CANopen Support

Is your feature request related to a problem? Please describe.
CANopen is a CAN-based communication system. It comprises higher-layer protocols and profile specifications.

I am looking into a project in the near future, where we will need to integrate CANopen support in Zephyr. Still searching for an appropriate, small CANopen implementation on top of SocketCAN, that can be ported to Zephyr.

@henrikbrixandersen do you still intend to port a CANopen implementation to Zephyr?
The two libraries, that build upon SocketCAN, and I know of are CANopenNode and CANFestival. Both seem not be maintained very actively anymore.

https://github.com/CANopenNode/CANopenNode   license: GPL-2.0
https://canfestival.org/ license: LGPL(runtime) and GPL (some tools regarding the creation of the object dictionary)

I am pretty sure you already had a look at them, do you think any of these would be appropriate?

@str4t0m Yes, I am in the process of porting CANopenNode to Zephyr.

That sounds great.
I'm thinking about a project where I could use CANopen.
Currently I am just getting started with Zephyr and haven't worked with CANopenNode yet, but I like what I have seen so far.
The management of the object dictionary seems mature compared to the Objdictedit used for Canfestival. Do you use the Object dictionary editor from the CANopenNode, or the libedssharp
 editor to manage the Object dictionary?

@str4t0m I use the editor from libedssharp. It's quite good.
I have CANopenNode basic functionality up and running on Zephyr, still working on EEPROM and ROM save/load support.

* https://github.com/CANopenNode/CANopenNode   license: GPL-2.0

The GPL-2.0 license is not compatible with Apache-2 license that we have in Zephyr, so your port cannot be put upstream. Did you had plans to upstream your CANopenNode port?

* https://github.com/CANopenNode/CANopenNode   license: GPL-2.0

The GPL-2.0 license is not compatible with Apache-2 license that we have in Zephyr, so your port cannot be put upstream. Did you had plans to upstream your CANopenNode port?

CANopenNode is licensed under GPL-2.0 with linking exception and the CANopenNode repository is a separate module. I was hoping to upstream it, yes.

CANopenNode is licensed under GPL-2.0 with linking exception and the CANopenNode repository is a separate module. I was hoping to upstream it, yes.

ping @nashif any idea whether upstreaming this piece of code is possible?

ping @nashif any idea whether upstreaming this piece of code is possible?

The WIP integration can be seen here: https://github.com/vestas-wind-systems/zephyr/tree/canopennode

CANopenNode author comments on licensing of driver and object dictionary files can be found here: https://github.com/CANopenNode/CANopenNode/issues/105#issuecomment-533803323

I am not a lawyer, but the exception looks ok to me. Anyway, it would be better if you get approval from TSC for this.

I'm wondering if lely-co, instead of CANopenNode, could be a better choice in term of code quality and license (Apache 2.0).

I'm wondering if lely-co, instead of CANopenNode, could be a better choice in term of code quality and license (Apache 2.0).

Interesting. I am interested in learning more of how you view/compare the code quality of the two projects?


i don't get any response with fake tokens on send

@florianreinhart  hey,
İ have 1000 tokens, one is real and the rest is "xxxx..." i receive a notification to my phone but i dont get any response.
return apnConnection.send(note, token).then(result => {
  console.log(result);
  return result;
}).catch(err => {
  console.log(err);
  return err;
});

How is token initialized in your code?
Is your callback executed when just send the notification to one valid token?


MOFISTO-32 - Implement ledgerFY API

Purpose
https://issues.folio.org/browse/MODFISTO-32 - Implement LedgerFY API that combines ledger and FY
Approach

Implement new raml file for all miscellaneous APIs
Using RMB cross table query
Until https://issues.folio.org/browse/MODFISTO-37 is worked on , inserting data into ledgerFY table for unit tests

TODOS and Open Questions

[ ] Why is fundsTest used for CRUD tests instead of EntitiesCrudTest file
[ ] There are a few warning messages cql.serverChoice requested, but no serverChoiceIndexes defined. while running the tests with queries against parent table. Investigate the cause
[ ] Merge acq-models PR first

Pre-Merge Checklist
Before merging this PR, please go through the following list and take appropriate actions.

Does this PR meet or exceed the expected quality standards?

[ ] Code coverage on new code is 80% or greater
[ ] Duplications on new code is 3% or less
[ ] There are no major code smells or security issues

Does this introduce breaking changes?

[ ] Were any API paths or methods changed, added or removed?
[ ] Were there any schema changes?
[ ] Did any of the interface versions change?
[ ] Were permissions changed, added, or removed?
[ ] Are there new interface dependencies?
[ ] There are no breaking changes in this PR.

If there are breaking changes, please STOP and consider the following:

What other modules will these changes impact?
Do JIRAs exist to update the impacted modules?

[ ] If not, please create them
[ ] Do they contain the appropriate level of detail?  Which endpoints/schemas changed, etc.
[ ] Do they have all they appropriate links to blocked/related issues?

Are the JIRAs under active development?

[ ] If not, contact the project's PO and make sure they're aware of the urgency.

Do PRs exist for these changes?

[ ] If so, have they been approved?

Ideally all of the PRs involved in breaking changes would be merged in the same day to avoid breaking the folio-testing environment.  Communication is paramount if that is to be achieved, especially as the number of intermodule and inter-team dependencies increase.
While it's helpful for reviewers to help identify potential problems, ensuring that it's safe to merge is ultimately the responsibility of the PR assignee.

INFO: lint-raml-cop: Investigating and determining configuration: mod-finance-storage/ramls
WARNING: lint-raml-cop: Missing from configuration: finance-storage.raml
WARNING: lint-raml-cop: Configuration needs to be updated (FOLIO-903).
INFO: lint-raml-cop: Not assessing schema descriptions, as per option '--validate-only'.
INFO: lint-raml-cop: Assessing RAML files (https://dev.folio.org/guides/raml-cop/):
INFO: lint-raml-cop: Processing RAML v1.0 file: budget.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with budget.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: fiscal-year.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with fiscal-year.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: fund.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with fund.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: fund-distribution.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with fund-distribution.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: fund-type.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with fund-type.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: group.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with group.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: ledger.raml
ERROR: lint-raml-cop:   raml-cop detected errors with ledger.raml:
[ramls/raml-util/rtypes/collection.raml:64:1] ERROR Can not parse JSON example: Unexpected token '"acqUnitIds"'
INFO: lint-raml-cop: Processing RAML v1.0 file: transaction.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with transaction.raml
INFO: lint-raml-cop: Processing RAML v1.0 file: finance-storage.raml
INFO: lint-raml-cop:   raml-cop did not detect any errors with finance-storage.raml
ERROR: lint-raml-cop: There were processing errors. See list above.


Get window dimension in global scope will cause issues when az is not run in a terminal

$TERM is missing when the command is not run in a terminal. Therefore fetching dimension will fail no matter what. Running get_window_dim in global scope will cause an exception when simply azure-cli-interactive is just imported.
https://github.com/Azure/azure-cli/blob/4e25d149cf8b1ad2182e89a0c1c29e2638a5b6b8/src/command_modules/azure-cli-interactive/azclishell/gather_commands.py#L16

This was fixed in #3893


Performance Improvements

Add a description
This issue is to track any areas where performance improvements can be made in gittuf. There's likely a few places that we can do this in, such as code in the RSL package.
Raised in #384.
Relevant log output if the discussion pertains to existing gittuf functionality
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Currently, when we traverse the RSL, we do so linearly, even though what we actually want is an RSL entry for a certain ref. When verifying a ref, if we have an entry, we want the latest RSL entry before that entry for the same ref.
We could implement this by changing the structure of RSL entries also to include a LastID, which would point to the latest RSL entry that knows the current entry that is recording on the same ref; this means that if we are verifying any ref when most of the time we are searching for entry's of that ref (other than attestations and policy states) we could jump backward the "parent" RSL entry for that ref. We could also record the last policy commit ID and attestation commit ID. This would mean that anything needed for verification could be found in the RSL entry itself without the need for traversing the whole RSL linked list. This means we could eliminate the need to veer off the course of verification. We could jump to every single applicable entry, without having to encounter useless ones.
The cost of this comes in recording RSL entries. Recording would take more time, but in most scenarios, the policy state is just a short distance from the head of the RSL, and also, with the attestation state and the last RSL entry for that ref, it wouldn't mostly be too far. Even in the worst case, we iterate through the whole RSL to record, but in most scenarios, this will not be the case, and the time for verification will outweigh the cost of recording.

@adityasaky @patzielinski Thoughts?

Somthing like this

How do we guarantee that an entry doesn't point to an older policy? We walk back the RSL entry by entry to get the assurance we indeed have the most recent policy for some entry under test.
Also, our policy look ups are not so frequent I think? In sequential verification, we look up once for the start (minus the bootstrapping phase which is unavoidable) and then handle new policies as they come up. I was moreso wondering if instead of trying to remove repeated walks of the RSL, which I think we have some of in the policy package, we could just cache parent-child relationships for the ones we've already seen so we don't have to load and parse the same git object repeatedly.

To add, walking back entry by entry also guarantees the RSL is indeed linear.

It makes sense; any of these pointers could be spoofed by a malicious actor. Verifying the linearity of the RSL would not be possible, but having pointers like these would mean that if we had an example RSL that contained ten reference entries, but only two were for the ref you were trying to verify, you would only have to iterate through those two instead of 10.
Would there be a way to have a cache at every step of the RSL that tells us the latest state of the policy and attestations while also being secure?

Would there be a way to have a cache at every step of the RSL that tells us the latest state of the policy and attestations while also being secure?

We could do that, but I think we'd effectively solve that with #245 as well? Also, #476 made significant improvements to the RSL walks. I think the biggest source of overhead now is walking all the way back to the first entry, which we have to do everytime until we address #245. As such, I think we ought to prioritize that ticket next.

Rather than have each entry track pointers to the applicable policy / attestations, I wonder if we can add a field to each entry with an incrementing integer, then build skip lists as part of #245 for policy and attestations so we can identify the right policy / attestations entries for each entry by just comparing the integers. This still doesn't address the question of verifying RSL linearity though.

That said, we can persist the point until which we've verified RSL linearity as well, and verify since that point. For example, verify linearity every time policy changes (and that local skip list is updated), every 100 RSL additions (walk back the last 100) etc.