MODELIX-427 Caching in api-gen-gradle

:tada: This PR is included in version 2.5.4 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Support for explicit ID values

Request to support optional ability to explicitly set the ID values. This can be helpful for when working with matching objects (entitlements/policies/etc..) across multiple environments since it is the only immutable property

Note another approach we use for this in the k8s operator is to use the name as the identifier as far as automation is concerned.


[TASK] Capability Packager/catalog frontend development work

Description
create a frontend using HTML/CSS, and based on the prior mockup.

The basic frontend is in place. This is complete.


Updated API documentation to reflect changes to authentication scheme.

With #167, the API was changed to provide a more generic way of how Clair can retrieve the layers from a registry. This change reflects it in the documentation.

Thanks! I totally forgot to update that doc!


sibling to cask-repair: cask-delete

this is a wishlist item and not a bug-report.
it would be awesome if we had a script named 'cask-delete' that could speed up sending a pull request to delete a cask.
i often stumble upon casks that no longer exist and should be removed, but the overhead of submitting a pull request manually often prevents that.

What’s your current process?

trying to create a branch in my fork and sending the pull request manually but that needs at least 5 minutes

What if you add something like this to your shell startup files (assuming bash/zsh):
function cask-delete {
  local cask tap
  cask="${1}"
  [[ -z "${2}" ]] && tap='homebrew-cask' || tap="homebrew-cask-${2}"

  if [[ -z "${1}" ]]; then
    echo 'Give a cask name as the first argument, and optionally a tap name as the second'
    return 1
  fi

  open "https://github.com/Homebrew/${tap}/blob/master/Casks/${cask}.rb"
}

Then you do something like
cask-delete alfred # Opens https://github.com/Homebrew/homebrew-cask/blob/master/Casks/alfred.rb
cask-delete firefox-nightly versions # Opens https://github.com/Homebrew/homebrew-cask-versions/blob/master/Casks/firefox-nightly.rb

Then you need only press the delete button and create a PR from the web interface. Would that suffice=

awesome ;)
( i didn't know you could do it this easily on the website and just did everything manually on the command line )
i think it would still be nice if this could be added as a standalone script to your repository for easier discovery by other users

I’ll have a think about it, but my initial thought is I wouldn’t want to publicise it openly. Although I really like cask deletions, I’d want for them to be submitted when it’s right (i.e. I’d rather the user makes sure before suggesting the deletion).
But I’m open to accommodating people who use my scripts regularly and well, such as yourself. So if you’re happy with it as is, it’s fine by me too. If someone else finds this issue and decides to also use the code, it’s also fine. Because if they did find it, they probably wanted it enough to look for it.
I’m not going to hide the code, just also not suggest it as a possibility for people that haven’t faced the need for it.

thanks, sounds fine with me ;)


timeular: 6.7.6 -> 6.7.8

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/timeular/versions.
meta.description for timeular is: Timetracking by flipping 8-sided dice
meta.homepage for timeular is: https://timeular.com
Updates performed

Version update

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 6.7.8 with grep in /nix/store/ksk5svg6qdw42w4hk38pq47fqmws5i4i-timeular-6.7.8
found 6.7.8 in filename of file in /nix/store/ksk5svg6qdw42w4hk38pq47fqmws5i4i-timeular-6.7.8

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

timeular

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/ksk5svg6qdw42w4hk38pq47fqmws5i4i-timeular-6.7.8 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A timeular https://github.com/r-ryantm/nixpkgs/archive/00f5b673c2777df1b6618ed79ba40d2bdd1423a7.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/00f5b673c2777df1b6618ed79ba40d2bdd1423a7#timeular

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/ksk5svg6qdw42w4hk38pq47fqmws5i4i-timeular-6.7.8
ls -la /nix/store/ksk5svg6qdw42w4hk38pq47fqmws5i4i-timeular-6.7.8/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

timeular

Maintainer pings
cc @ktor for testing.

Result of nixpkgs-review pr 310489 run on x86_64-linux 1

1 package built:

timeular


Fix PSSA Issues in MSFT_xExchOwaVirtualDirectory (Fixes #83)

Fixing the PSSA Issues in MSFT_xExchOwaVirtualDirectory.
This fixes #83.
Running Invoke-ScriptAnalyzer with the following rules excluded no longer has any output:
PSUseShouldProcessForStateChangingFunctions
PSDSCDscTestsPresent
PSDSCDscExamplesPresent
These rules have been approved to be ignored.

This change is

Hi @kwirkykat, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Katie Keim). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

Looks good to me.


Add link to braintree-web to README

To better guide people looking for where to file issues on the current version of braintree.js.

:+1:


Update Composer dependencies (2023-02-12-00-09)

Visual regression test passed!
View the visual regression test report


FFMPEG vs PyAV

Is it possible or reasonable to build based on FFMPEG instead of PyAV? To my understanding, PyAV has to be built from source on Android (Termux) to my understanding. I believe that fails due to AV_CODEC_CAP_TRUNCATED, AV_CODEC_CAP_AUTO_THREADS, AV_CODEC_CAP_INTRA_ONLY, and AV_CODEC_CAP_LOSSLESS. It seems that these constants were removed in ffmpeg 6.0.0.
I wasn't able to figure out exactly what these constants are for to usefully modify the PyAV source, so I assume that I would need to build an older version of ffmpeg, which makes me concerned given the number of things dependent on it.
For faster-whisper, it seems like dependence on PyAV is limited to https://github.com/guillaumekln/faster-whisper/blob/f20bb258de81bdb9830c3ef1f2e468564ff4fbd2/faster_whisper/audio.py . I have limited experience, but I imagine that I could figure out how to modify this to use ffmpeg and change the requirements file.
Is that a reasonable plan? Or would that be likely to fail miserably?
EDIT: Apparently someone smarter than me already posted a commit for PyAV fixing this.
https://github.com/PyAV-Org/PyAV/issues/1106#ref-commit-9e7bf70
In case anyone else needs to build PyAV from source with ffmpeg 6.0.

The decode_audio function is basically running the following FFmpeg command:
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav

Then the waveform is normalized:
https://github.com/guillaumekln/faster-whisper/blob/d889345e071de21a83bdae60ba4b07110cfd0696/faster_whisper/audio.py#L58-L59
You can run these steps with another library if you want to.
We use PyAV because it does not add a system dependency (the FFmpeg libraries are bundled in the Python package).

Do you've any idea why results from PyAV are better?

They should not be. Either the audio decoding options are different or the normalization step mentionned above is missing.

They should not be. Either the audio decoding/resampling options are different or the normalization step mentionned above is missing.

That FFmpeg command is standard [I think, everyone using it like that].
Does Faster-Whisper skips normalization step if input is that "output.wav" from FFmpeg?

Does Faster-Whisper skips normalization step if input is that "output.wav" from FFmpeg?

No, all audio files go through the same decoding/resampling/normalization pipeline.

No, all audio files go through the same decoding/resampling/normalization pipeline.

Then mystery remains.
Here I made the test samples: https://we.tl/t-I9Lxeof475
Used settings: --language en --model "medium" --device "cpu" --beam_size 1
Output on original audio:

01 - [00:00.000 --> 00:02.000]  Feeling inspired yet?
02 - [00:07.000 --> 00:09.000]  Thank you.
03 - [00:09.000 --> 00:11.000]  I thought you said you were hungry.
04 - [00:13.000 --> 00:17.000]  There's a boat tour of the Trakla Island Formations this afternoon.
05 - [00:17.000 --> 00:23.000]  I was thinking we could go on that and make reservations in town for dinner.
06 - [00:23.000 --> 00:25.000]  We could try the Chinese place.
07 - [00:25.000 --> 00:28.000]  I don't think I'd survive another dinner in town.
08 - [00:28.000 --> 00:30.000]  Even the idea that...
09 - [00:30.000 --> 00:32.000]  Does anyone think it's a real town?
10 - [00:35.000 --> 00:37.000]  Why would they have a Chinese place?
11 - [00:42.000 --> 00:45.000]  Is it okay if I go? I'll meet you on the beach.
12 - [00:46.000 --> 00:48.000]  No, sure.
13 - [00:58.000 --> 01:00.000]  What's going on?
14 - [01:22.000 --> 01:24.000]  Someone's making a statement.
15 - [01:25.000 --> 01:27.000]  One of the locals, I guess.
16 - [01:29.000 --> 01:31.000]  What do you think he's trying to say?

Output on ffmpeg audio:

01 - [00:00.000 --> 00:03.000]  You feeling inspired yet?
02 - [00:08.000 --> 00:09.000]  Thank you.
03 - [00:09.000 --> 00:11.000]  I thought you said you were hungry.
04 - [00:14.000 --> 00:18.000]  There's a boat tour of the Trakla Island Formations this afternoon.
05 - [00:19.000 --> 00:23.000]  I was thinking we could go on that and make reservations in town for dinner.
06 - [00:24.000 --> 00:26.000]  We could try the Chinese place.
07 - [00:26.000 --> 00:29.000]  I don't think I'd survive another dinner in town.
08 - [00:30.000 --> 00:32.000]  Even the idea that...
09 - [00:32.000 --> 00:34.000]  Does anyone think it's a real town?
10 - [00:36.000 --> 00:38.000]  Why would they have a Chinese place?
11 - [00:43.000 --> 00:46.000]  Is it okay if I go? I'll meet you on the glitch.
12 - [00:47.000 --> 00:49.000]  No, sure.
13 - [01:00.000 --> 01:02.000]  What's going on?
14 - [01:02.000 --> 01:04.000]  Someone's making a statement.
15 - [01:05.000 --> 01:07.000]  One of the locals, I guess.
?? - [01:09.000 --> 01:11.000]  What is it?
?? - [01:11.000 --> 01:13.000]  I don't know.
?? - [01:13.000 --> 01:15.000]  I don't know.
?? - [01:15.000 --> 01:17.000]  I don't know.
?? - [01:17.000 --> 01:19.000]  I don't know.
?? - [01:19.000 --> 01:21.000]  I don't know.
?? - [01:21.000 --> 01:23.000]  I don't know.
?? - [01:23.000 --> 01:25.000]  I don't know.
?? - [01:25.000 --> 01:27.000]  I don't know.
16 - [01:28.000 --> 01:30.000]  What do you think he's trying to say?

Results from original audio:
Very accurate transcription.
Timestamp on 13 line is off by ~20s.
Results from FFmpeg audio:
Transcription is inaccurate at 1 & 11 lines , after 15 line it starts hallucinating.
Timestamps  on 13,  14 & 15 lines are off by ~20s.

When you say "ffmpeg audio", what’s the actual command that was run?

When you say "ffmpeg audio", what’s the actual command that was run?

ffmpeg -i test_original.aac -ar 16000 -ac 1 -c:a pcm_s16le test_ffmpeg6.wav

For completeness I ran original audio on OpenAI's Whisper [it's using FFmeg].
Same inaccurate result as in my previous post [FFmpeg audio], just the hallucination part is different:

[01:09.000 --> 01:11.000]  What is it?
[01:11.000 --> 01:13.000]  I don't know.
[01:13.000 --> 01:15.000]  I'm just wondering.
[01:15.000 --> 01:17.000]  What's going on?
[01:17.000 --> 01:19.000]  I don't know.
[01:19.000 --> 01:21.000]  I'm just wondering.
[01:21.000 --> 01:23.000]  What's going on?
[01:23.000 --> 01:25.000]  I don't know.
[01:25.000 --> 01:27.000]  I'm just wondering.
[01:27.000 --> 01:29.000]  What's going on?

I'm not reproducing your output on the original audio when running faster-whisper directly:
from faster_whisper import WhisperModel

model = WhisperModel("medium", device="cpu")

segments, info = model.transcribe("test_original.aac", beam_size=1, language="en")

for segment in segments:
    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))

Here's what I get, which seems to match the output from OpenAI's Whisper:
[0.00s -> 2.00s]  You feeling inspired yet?
[7.00s -> 9.00s]  Thank you.
[9.00s -> 11.00s]  I thought you said you were hungry.
[13.00s -> 17.00s]  There's a boat tour of the Trakla Island Formations this afternoon.
[18.00s -> 23.00s]  I was thinking we could go on that and make reservations in town for dinner.
[24.00s -> 26.00s]  We could try the Chinese place.
[26.00s -> 28.00s]  I don't think I'd survive another dinner in town.
[30.00s -> 32.00s]  Even the idea that...
[32.00s -> 34.00s]  Does anyone think it's a real town?
[36.00s -> 38.00s]  Why would they have a Chinese place?
[44.00s -> 46.00s]  Is it okay if I go? I'll meet you on the glitch.
[47.00s -> 49.00s]  No, sure.
[60.00s -> 62.00s]  What's going on?
[62.00s -> 64.00s]  Someone's making a statement.
[64.00s -> 66.00s]  One of the locals, I guess.
[69.00s -> 71.00s]  What is it?
[71.00s -> 73.00s]  I don't know.
[73.00s -> 75.00s]  I'm just wondering.
[75.00s -> 77.00s]  What's going on?
[77.00s -> 79.00s]  I don't know.
[79.00s -> 81.00s]  I'm just wondering.
[81.00s -> 83.00s]  What's going on?
[83.00s -> 85.00s]  I don't know.
[85.00s -> 87.00s]  I'm just wondering.
[87.00s -> 89.00s]  What's going on?

I can reproduce the repeated "I don't know" when I set compute_type="int8" but you didn't mention this option. Is it set by default by the program you are using?

I can reproduce the repeated "I don't know" when I set compute_type="int8" but you didn't mention this option. Is it set by default by the program you are using?

Yes, it's using int8 by default if on cpu and float16 by default on cuda.

The default value for compute_type here is "default" which infers the compute type from the converted model. More info here: https://opennmt.net/CTranslate2/quantization.html#quantize-on-model-loading

The default value for compute_type here is "default" which infers the compute type from the converted model. More info here: https://opennmt.net/CTranslate2/quantization.html#quantize-on-model-loading

If I understood correctly, int8 is the problem whether or not ffmpeg/PyAV is used?

In this example, yes. The output is different because the options are different and not because of FFmpeg/PyAV.

If I understood correctly, int8 is the problem whether or not ffmpeg/PyAV is used?

Not only int8, int16 too. No differences with float32, I can't test float16.

In this example, yes. The output is different because the options are different and not because of FFmpeg/PyAV.

Why you say that? Mention of OpenAI is irrelevant there, just ignore it.
Both comparisons were done with exact same settings:
--language=en --model=medium --device=cpu --beam_size=1 --compute_type=int8
Reuped test files: https://we.tl/t-S5gnRvMuQB

More info here: https://opennmt.net/CTranslate2/quantization.html#quantize-on-model-loading

Thanks for the link. After reading it still some things are not clear for me. What quantization is more precise or faster?
For me float32 is twice faster and looks less accurate than int8.

If I understood correctly, int8 is the problem whether or not ffmpeg/PyAV is used?

Not only int8, int16 too. No differences with float32, I can't test float16.

In this example, yes. The output is different because the options are different and not because of FFmpeg/PyAV.

Why you say that? Mention of OpenAI is irrelevant there, just ignore it. Both comparisons were done with exact same settings: --language=en --model=medium --device=cpu --beam_size=1 --compute_type=int8
Reuped test files: https://we.tl/t-S5gnRvMuQB

More info here: https://opennmt.net/CTranslate2/quantization.html#quantize-on-model-loading

Thanks for the link. After reading it still some things are not clear for me. What quantization is more precise or faster? For me float32 is twice faster and looks less accurate than int8.

If I understood correctly, int8 is the problem whether or not ffmpeg/PyAV is used?

Not only int8, int16 too. No differences with float32, I can't test float16.

In this example, yes. The output is different because the options are different and not because of FFmpeg/PyAV.

Why you say that? Mention of OpenAI is irrelevant there, just ignore it. Both comparisons were done with exact same settings: --language=en --model=medium --device=cpu --beam_size=1 --compute_type=int8
Reuped test files: https://we.tl/t-S5gnRvMuQB

More info here: https://opennmt.net/CTranslate2/quantization.html#quantize-on-model-loading

Thanks for the link. After reading it still some things are not clear for me. What quantization is more precise or faster? For me float32 is twice faster and looks less accurate than int8.

As far as I know, Pyav also uses ffpmeg.
If so, I can guess that the issue is due to a configuration or version difference between Pyav's ffmpeg and ffmpeg.
Just to add one thing, I tested the wav files extracted via ffmpeg on my mbp14 with faster-whisper int8 and float32. Results for int8 and float32 were different, and in general float32 produced timestamps more accurately, although it took about 1.3 times longer.

Tested on MacBook Pro 14

test_ffmpeg6.wav - [ int8 ]

00:00:00.000 --> 00:00:02.000
Feeling inspired yet?

2
00:00:07.000 --> 00:00:08.000
Thank you.

3
00:00:09.000 --> 00:00:11.000
I thought you said you were hungry.

4
00:00:14.500 --> 00:00:17.000
There's a boat tour of the Trakla Island formations this afternoon.

5
00:00:20.500 --> 00:00:23.000
I was thinking we could go on that and make reservations in town for dinner.

6
00:00:24.000 --> 00:00:25.000
We could try the Chinese place.

7
00:00:26.000 --> 00:00:28.000
I don't think I'd survive another dinner in town.

8
00:00:30.500 --> 00:00:33.000
Even the idea of it. Does anyone think it's a real town?

9
00:00:36.000 --> 00:00:38.000
Why would they have a Chinese place?

10
00:00:43.500 --> 00:00:46.000
Is it okay if I go? I'll meet you on the beach.

11
00:00:47.000 --> 00:00:48.000
Yeah, sure.

12
00:01:19.500 --> 00:01:21.000
What's going on?

13
00:01:23.500 --> 00:01:26.000
Someone's making a statement. One of the locals, I guess.

14
00:01:29.000 --> 00:01:31.000
What do you think he's trying to say?

15
00:01:32.500 --> 00:01:35.000
He's saying that he wants to put a long knife right through her.

16
00:01:37.500 --> 00:01:40.000
And after you die, he'll hang your body at the airport to scare off the other tourists.

17
00:01:42.000 --> 00:01:43.000
Seems a bit extreme.

18
00:01:46.000 --> 00:01:48.000
The Tokans are a melodramatic people.

19
00:01:55.000 --> 00:01:56.000
I loved your book.

20
00:01:59.000 --> 00:02:00.000
Sorry?

21
00:02:01.500 --> 00:02:04.000
You're James Foster. I loved your book.

22
00:02:07.000 --> 00:02:09.000
Sorry, is that good? I don't mean to put you on the spot.

23
00:02:10.000 --> 00:02:11.000
No, thank you.

24
00:02:12.500 --> 00:02:15.000
It's just not a lot of people read my book.

25
00:02:16.000 --> 00:02:17.000
I'm Gabby Bauer.

26
00:02:18.000 --> 00:02:19.000
I'm James Foster.

27
00:02:22.000 --> 00:02:23.000
Alvin!

28
00:02:26.000 --> 00:02:27.000
This is James Foster.

29
00:02:28.000 --> 00:02:29.000
Nice to meet you. Albon Bauer.

30
00:02:30.000 --> 00:02:32.000
He wrote a book that I love, The Variable Sheath.

31
00:02:33.000 --> 00:02:34.000
Oh yeah, I remember.

32
00:02:35.000 --> 00:02:36.000
I thought it was brilliant.

33
00:02:38.500 --> 00:02:41.000
James, do you think I could convince you to join us for dinner this evening?

34
00:02:43.500 --> 00:02:46.000
I've been seeing you around the resort for a few days now and I would love to get to know you.

35
00:02:47.000 --> 00:02:49.000
We have a reservation tonight at Yang's.

36
00:03:02.500 --> 00:03:05.000
Yeah, it was a good kind of a learning experience.

37
00:03:07.000 --> 00:03:09.000
Is there anything else I can get you?

38
00:03:10.000 --> 00:03:11.000
That's all I think.

39
00:03:12.000 --> 00:03:14.000
Alright everyone, please have a great meal.

40
00:03:16.500 --> 00:03:19.000
And let me know any time if I can make your experience even more enjoyable.

41
00:03:22.000 --> 00:03:23.000
Interesting guy.

42
00:03:24.000 --> 00:03:25.000
Yes.

43
00:03:26.500 --> 00:03:29.000
This resort is labelled in the resort guide as a multicultural dining experience.

44
00:03:30.000 --> 00:03:32.000
It certainly is an experience.

45
00:03:33.500 --> 00:03:36.000
So Albon, what is it you do for a living?

46
00:03:37.000 --> 00:03:39.000
Architecture, but I'm mostly retired.

47
00:03:40.000 --> 00:03:42.000
Now I run a journal out of Los Angeles called Glass Pane.

48
00:03:43.000 --> 00:03:44.000
You're French?

49
00:03:45.500 --> 00:03:48.000
Oh no, Swiss first, from Geneva, then Paris, then LA.

50
00:03:49.500 --> 00:03:52.000
I'm from London first, then Paris.

51
00:03:53.000 --> 00:03:54.000
That's how we met.

52
00:03:55.000 --> 00:03:57.000
I couldn't get work there, so I made Albon move with me.

53
00:03:58.000 --> 00:04:00.000
And what do you do?

54
00:04:01.000 --> 00:04:03.000
Well, I'm an actress of course.

55
00:04:04.000 --> 00:04:05.000
Oh really? She's great.

56
00:04:06.000 --> 00:04:07.000
For commercials.

57
00:04:08.500 --> 00:04:11.000
I have a contract with an LA company, they've been grooming me.

58
00:04:12.000 --> 00:04:14.000
I specialize in failing naturally.

59
00:04:15.500 --> 00:04:18.000
What does that mean, failing naturally?

60
00:04:19.500 --> 00:04:22.000
Finding a natural seeming way to fail at any given task.

61
00:04:23.500 --> 00:04:26.000
In each of the commercials that I'm in, I'm the one who simply can't go on without the product.

62
00:04:27.000 --> 00:04:29.000
It's ridiculous of me not to have the product.

63
00:04:30.000 --> 00:04:31.000
Okay.

64
00:04:32.000 --> 00:04:33.000
Show them.

65
00:04:34.000 --> 00:04:35.000
No, you should.

66
00:04:36.000 --> 00:04:37.000
Do you want to see?

67
00:04:38.000 --> 00:04:39.000
I want to see.

68
00:04:42.000 --> 00:04:43.000
She's amazing.

69
00:04:53.000 --> 00:04:54.000
I just...

70
00:04:55.000 --> 00:04:56.000
I...

test_ffmpeg6.wav - [ float32 ]

1
00:00:00.000 --> 00:00:02.000
Feeling inspired yet?

2
00:00:07.000 --> 00:00:08.000
Thank you.

3
00:00:09.000 --> 00:00:11.000
I thought you said you were hungry.

4
00:00:14.500 --> 00:00:17.000
There's a boat tour of the Trakla Island formations this afternoon.

5
00:00:20.500 --> 00:00:23.000
I was thinking we could go on that and make reservations in town for dinner.

6
00:00:24.000 --> 00:00:25.000
We could try the Chinese place.

7
00:00:26.000 --> 00:00:28.000
I don't think I'd survive another dinner in town.

8
00:00:30.500 --> 00:00:33.000
Even the idea of it. Does anyone think it's a real town?

9
00:00:36.000 --> 00:00:38.000
Why would they have a Chinese place?

10
00:00:43.500 --> 00:00:46.000
Is it okay if I go? I'll meet you on the beach.

11
00:00:47.000 --> 00:00:48.000
Yeah, sure.

12
00:01:19.500 --> 00:01:21.000
What's going on?

13
00:01:23.500 --> 00:01:26.000
Someone's making a statement. One of the locals, I guess.

14
00:01:29.000 --> 00:01:31.000
What do you think he's trying to say?

15
00:01:32.500 --> 00:01:35.000
He's saying that he wants to put a long knife right through her.

16
00:01:37.500 --> 00:01:40.000
And after you die, he'll hang your body at the airport to scare off the other tourists.

17
00:01:42.000 --> 00:01:43.000
Seems a bit extreme.

18
00:01:46.000 --> 00:01:48.000
The Tokans are a melodramatic people.

19
00:01:55.000 --> 00:01:56.000
I loved your book.

20
00:01:59.000 --> 00:02:00.000
Sorry?

21
00:02:01.500 --> 00:02:04.000
You're James Foster. I loved your book.

22
00:02:07.000 --> 00:02:09.000
Sorry, is that good? I don't mean to put you on the spot.

23
00:02:10.000 --> 00:02:11.000
No, thank you.

24
00:02:12.500 --> 00:02:15.000
It's just not a lot of people read my book.

25
00:02:16.000 --> 00:02:17.000
I'm Gabby Bauer.

26
00:02:18.000 --> 00:02:19.000
I'm James Foster.

27
00:02:22.000 --> 00:02:23.000
Alvin!

28
00:02:26.000 --> 00:02:27.000
This is James Foster.

29
00:02:28.000 --> 00:02:29.000
Nice to meet you. Albon Bauer.

30
00:02:30.000 --> 00:02:32.000
He wrote a book that I love, The Variable Sheath.

31
00:02:33.000 --> 00:02:34.000
Oh yeah, I remember.

32
00:02:35.000 --> 00:02:36.000
I thought it was brilliant.

33
00:02:38.500 --> 00:02:41.000
James, do you think I could convince you to join us for dinner this evening?

34
00:02:43.500 --> 00:02:46.000
I've been seeing you around the resort for a few days now and I would love to get to know you.

35
00:02:47.000 --> 00:02:49.000
We have a reservation tonight at Yang's.

36
00:03:02.500 --> 00:03:05.000
Yeah, it was a good kind of a learning experience.

37
00:03:07.000 --> 00:03:09.000
Is there anything else I can get you?

38
00:03:10.000 --> 00:03:11.000
That's all I think.

39
00:03:12.000 --> 00:03:14.000
Alright everyone, please have a great meal.

40
00:03:16.500 --> 00:03:19.000
And let me know any time if I can make your experience even more enjoyable.

41
00:03:22.000 --> 00:03:23.000
Interesting guy.

42
00:03:24.000 --> 00:03:25.000
Yes.

43
00:03:26.500 --> 00:03:29.000
This resort is labelled in the resort guide as a multicultural dining experience.

44
00:03:30.000 --> 00:03:32.000
It certainly is an experience.

45
00:03:33.500 --> 00:03:36.000
So Albon, what is it you do for a living?

46
00:03:37.000 --> 00:03:39.000
Architecture, but I'm mostly retired.

47
00:03:40.000 --> 00:03:42.000
Now I run a journal out of Los Angeles called Glass Pane.

48
00:03:43.000 --> 00:03:44.000
You're French?

49
00:03:45.500 --> 00:03:48.000
Oh no, Swiss first, from Geneva, then Paris, then LA.

50
00:03:49.500 --> 00:03:52.000
I'm from London first, then Paris.

51
00:03:53.000 --> 00:03:54.000
That's how we met.

52
00:03:55.000 --> 00:03:57.000
I couldn't get work there, so I made Albon move with me.

53
00:03:58.000 --> 00:04:00.000
And what do you do?

54
00:04:01.000 --> 00:04:03.000
Well, I'm an actress of course.

55
00:04:04.000 --> 00:04:05.000
Oh really? She's great.

56
00:04:06.000 --> 00:04:07.000
For commercials.

57
00:04:08.500 --> 00:04:11.000
I have a contract with an LA company, they've been grooming me.

58
00:04:12.000 --> 00:04:14.000
I specialize in failing naturally.

59
00:04:15.500 --> 00:04:18.000
What does that mean, failing naturally?

60
00:04:19.500 --> 00:04:22.000
Finding a natural seeming way to fail at any given task.

61
00:04:23.500 --> 00:04:26.000
In each of the commercials that I'm in, I'm the one who simply can't go on without the product.

62
00:04:27.000 --> 00:04:29.000
It's ridiculous of me not to have the product.

63
00:04:30.000 --> 00:04:31.000
Okay.

64
00:04:32.000 --> 00:04:33.000
Show them.

65
00:04:34.000 --> 00:04:35.000
No, you should.

66
00:04:36.000 --> 00:04:37.000
Do you want to see?

67
00:04:38.000 --> 00:04:39.000
I want to see.

68
00:04:42.000 --> 00:04:43.000
She's amazing.

69
00:04:53.000 --> 00:04:54.000
I just...

70
00:04:55.000 --> 00:04:56.000
I...

test_original.acc - [ int8 ]

1
00:00:00.000 --> 00:00:02.000
Feeling inspired yet?

2
00:00:07.000 --> 00:00:08.000
Thank you.

3
00:00:09.000 --> 00:00:11.000
I thought you said you were hungry.

4
00:00:14.500 --> 00:00:17.000
There's a boat tour of the Trakla Island formations this afternoon.

5
00:00:20.500 --> 00:00:23.000
I was thinking we could go on that and make reservations in town for dinner.

6
00:00:24.000 --> 00:00:26.000
We could try the Chinese place.

7
00:00:26.000 --> 00:00:28.000
I don't think I'd survive another dinner in town.

8
00:00:31.500 --> 00:00:34.000
Even the idea of it. Does anyone think it's a real town?

9
00:00:36.000 --> 00:00:38.000
Why would they have a Chinese place?

10
00:00:43.500 --> 00:00:46.000
Is it okay if I go? I'll meet you on the beach.

11
00:00:47.000 --> 00:00:49.000
Yeah, sure.

12
00:01:19.500 --> 00:01:21.000
What's going on?

13
00:01:24.500 --> 00:01:27.000
Someone's making a statement. One of the locals, I guess.

14
00:01:29.000 --> 00:01:31.000
What do you think he's trying to say?

15
00:01:32.500 --> 00:01:35.000
He's saying that he wants to put a long knife right through her.

16
00:01:37.500 --> 00:01:40.000
And after you die, he'll hang your body at the airport to scare off the other tourists.

17
00:01:42.000 --> 00:01:44.000
Seems a bit extreme.

18
00:01:46.000 --> 00:01:48.000
The Tokens are a melodramatic people.

19
00:01:55.000 --> 00:01:56.000
I loved your book.

20
00:01:59.000 --> 00:02:00.000
Sorry?

21
00:02:01.500 --> 00:02:04.000
You're James Foster. I loved your book.

22
00:02:07.000 --> 00:02:09.000
Sorry, is that good? I don't mean to put you on the spot.

23
00:02:10.000 --> 00:02:11.000
No, thank you.

24
00:02:12.500 --> 00:02:15.000
It's just not a lot of people read my book.

25
00:02:16.000 --> 00:02:17.000
I'm Gabby Bauer.

26
00:02:18.000 --> 00:02:19.000
I'm James Foster.

27
00:02:22.000 --> 00:02:23.000
Alvin!

28
00:02:26.000 --> 00:02:27.000
This is James Foster.

29
00:02:27.000 --> 00:02:28.000
Hi, nice to meet you. Albon Bauer.

30
00:02:29.000 --> 00:02:30.000
Pleasure.

31
00:02:31.000 --> 00:02:32.000
He wrote your book that I love, The Variable Sheath.

32
00:02:33.000 --> 00:02:34.000
Oh yeah, I remember.

33
00:02:35.000 --> 00:02:36.000
I thought it was brilliant.

34
00:02:38.500 --> 00:02:41.000
James, do you think I could convince you to join us for dinner this evening?

35
00:02:43.500 --> 00:02:46.000
I've been seeing you around the resort for a few days now and I would love to get to know you.

36
00:02:47.000 --> 00:02:49.000
We have a reservation tonight at Yang's.

37
00:02:58.000 --> 00:02:59.000
Is there anything else I can get you?

38
00:03:00.000 --> 00:03:01.000
That's all I think.

39
00:03:02.000 --> 00:03:03.000
Alright, everyone, please have a great meal.

40
00:03:04.000 --> 00:03:05.000
Thank you.

41
00:03:06.500 --> 00:03:09.000
And let me know any time if I can make your experience even more enjoyable.

42
00:03:11.000 --> 00:03:12.000
He's an interesting guy.

43
00:03:13.000 --> 00:03:14.000
He is.

44
00:03:15.000 --> 00:03:16.000
This is what is labelled in the resort.

45
00:03:17.000 --> 00:03:18.000
Oh, I see.

46
00:03:19.000 --> 00:03:20.000
So, what's your name?

47
00:03:21.000 --> 00:03:22.000
I'm Alvin.

48
00:03:23.000 --> 00:03:24.000
I'm Gabby.

49
00:03:25.000 --> 00:03:26.000
I'm Alvin.

50
00:03:27.000 --> 00:03:29.000
The resort guide is a multicultural dining experience.

51
00:03:30.000 --> 00:03:32.000
It certainly is an experience.

52
00:03:33.500 --> 00:03:36.000
So, Alvin, what is it you do for a living?

53
00:03:37.000 --> 00:03:38.000
Oh, architecture.

54
00:03:39.000 --> 00:03:40.000
But I'm mostly retired.

55
00:03:41.000 --> 00:03:42.000
Now I run a journal out of Los Angeles called Glass Pane.

56
00:03:43.000 --> 00:03:44.000
You're French?

57
00:03:45.000 --> 00:03:46.000
Oh no, Swiss first, from Geneva.

58
00:03:47.000 --> 00:03:48.000
Then Paris, then LA.

59
00:03:49.000 --> 00:03:50.000
I'm from London first.

60
00:03:51.000 --> 00:03:52.000
Then Paris.

61
00:03:53.000 --> 00:03:54.000
That's how we met.

62
00:03:54.500 --> 00:03:57.000
But I couldn't get work there, so I made Alvin move with me.

63
00:03:58.000 --> 00:03:59.000
And what do you do?

64
00:04:00.000 --> 00:04:02.000
Well, I'm an actress, of course.

65
00:04:03.000 --> 00:04:04.000
Oh, really?

66
00:04:05.000 --> 00:04:06.000
She's great.

67
00:04:07.000 --> 00:04:08.000
For commercials.

68
00:04:09.000 --> 00:04:10.000
I have a contract with an LA company.

69
00:04:11.000 --> 00:04:12.000
They've been grooming me.

70
00:04:13.000 --> 00:04:14.000
I specialize in failing naturally.

71
00:04:15.000 --> 00:04:16.000
What does that mean?

72
00:04:17.000 --> 00:04:18.000
Failing naturally?

73
00:04:19.500 --> 00:04:22.000
Finding a natural-seeming way to fail at any given task.

74
00:04:22.000 --> 00:04:23.000
In each of the commercials that I'm in,

75
00:04:24.000 --> 00:04:26.000
I'm the one who simply can't go on without the product.

76
00:04:27.000 --> 00:04:29.000
It's ridiculous of me not to have the product.

77
00:04:30.000 --> 00:04:31.000
Okay.

78
00:04:32.000 --> 00:04:33.000
Show them.

79
00:04:34.000 --> 00:04:35.000
No, you should.

80
00:04:36.000 --> 00:04:37.000
Do you want to see?

81
00:04:38.000 --> 00:04:39.000
I want to see.

82
00:04:42.000 --> 00:04:43.000
She's amazing.

83
00:04:53.000 --> 00:04:54.000
I just...

84
00:04:55.000 --> 00:04:56.000
I...

test_original.acc - [ float32 ]

1
00:00:00.000 --> 00:00:02.000
Feeling inspired yet?

2
00:00:07.000 --> 00:00:08.000
Thank you.

3
00:00:09.000 --> 00:00:11.000
I thought you said you were hungry.

4
00:00:14.500 --> 00:00:17.000
There's a boat tour of the Trakla Island formations this afternoon.

5
00:00:20.500 --> 00:00:23.000
I was thinking we could go on that and make reservations in town for dinner.

6
00:00:24.000 --> 00:00:26.000
We could try the Chinese place.

7
00:00:26.000 --> 00:00:28.000
I don't think I'd survive another dinner in town.

8
00:00:31.500 --> 00:00:34.000
Even the idea of it. Does anyone think it's a real town?

9
00:00:36.000 --> 00:00:38.000
Why would they have a Chinese place?

10
00:00:43.500 --> 00:00:46.000
Is it okay if I go? I'll meet you on the beach.

11
00:00:47.000 --> 00:00:49.000
Yeah, sure.

12
00:01:19.500 --> 00:01:21.000
What's going on?

13
00:01:24.500 --> 00:01:27.000
Someone's making a statement. One of the locals, I guess.

14
00:01:29.000 --> 00:01:31.000
What do you think he's trying to say?

15
00:01:32.500 --> 00:01:35.000
He's saying that he wants to put a long knife right through her.

16
00:01:37.500 --> 00:01:40.000
And after you die, he'll hang your body at the airport to scare off the other tourists.

17
00:01:42.000 --> 00:01:44.000
Seems a bit extreme.

18
00:01:46.000 --> 00:01:48.000
The Tokens are a melodramatic people.

19
00:01:55.000 --> 00:01:56.000
I loved your book.

20
00:01:59.000 --> 00:02:00.000
Sorry?

21
00:02:01.500 --> 00:02:04.000
You're James Foster. I loved your book.

22
00:02:07.000 --> 00:02:09.000
Sorry, is that good? I don't mean to put you on the spot.

23
00:02:10.000 --> 00:02:11.000
No, thank you.

24
00:02:12.500 --> 00:02:15.000
It's just not a lot of people read my book.

25
00:02:16.000 --> 00:02:17.000
I'm Gabby Bauer.

26
00:02:18.000 --> 00:02:19.000
I'm James Foster.

27
00:02:22.000 --> 00:02:23.000
Alvin!

28
00:02:26.000 --> 00:02:27.000
This is James Foster.

29
00:02:27.000 --> 00:02:28.000
Hi, nice to meet you. Albon Bauer.

30
00:02:29.000 --> 00:02:30.000
Pleasure.

31
00:02:31.000 --> 00:02:32.000
He wrote your book that I love, The Variable Sheath.

32
00:02:33.000 --> 00:02:34.000
Oh yeah, I remember.

33
00:02:35.000 --> 00:02:36.000
I thought it was brilliant.

34
00:02:38.500 --> 00:02:41.000
James, do you think I could convince you to join us for dinner this evening?

35
00:02:43.500 --> 00:02:46.000
I've been seeing you around the resort for a few days now and I would love to get to know you.

36
00:02:47.000 --> 00:02:49.000
We have a reservation tonight at Yang's.

37
00:02:58.000 --> 00:02:59.000
Is there anything else I can get you?

38
00:03:00.000 --> 00:03:01.000
That's all I think.

39
00:03:02.000 --> 00:03:03.000
Alright, everyone, please have a great meal.

40
00:03:04.000 --> 00:03:05.000
Thank you.

41
00:03:06.500 --> 00:03:09.000
And let me know any time if I can make your experience even more enjoyable.

42
00:03:11.000 --> 00:03:12.000
He's an interesting guy.

43
00:03:13.000 --> 00:03:14.000
He is.

44
00:03:15.000 --> 00:03:16.000
This is what is labelled in the resort.

45
00:03:17.000 --> 00:03:18.000
Oh, I see.

46
00:03:19.000 --> 00:03:20.000
So, what's your name?

47
00:03:21.000 --> 00:03:22.000
I'm Alvin.

48
00:03:23.000 --> 00:03:24.000
I'm Gabby.

49
00:03:25.000 --> 00:03:26.000
I'm Alvin.

50
00:03:27.000 --> 00:03:29.000
The resort guide is a multicultural dining experience.

51
00:03:30.000 --> 00:03:32.000
It certainly is an experience.

52
00:03:33.500 --> 00:03:36.000
So, Alvin, what is it you do for a living?

53
00:03:37.000 --> 00:03:38.000
Oh, architecture.

54
00:03:39.000 --> 00:03:40.000
But I'm mostly retired.

55
00:03:41.000 --> 00:03:42.000
Now I run a journal out of Los Angeles called Glass Pane.

56
00:03:43.000 --> 00:03:44.000
You're French?

57
00:03:45.000 --> 00:03:46.000
Oh no, Swiss first, from Geneva.

58
00:03:47.000 --> 00:03:48.000
Then Paris, then LA.

59
00:03:49.000 --> 00:03:50.000
I'm from London first.

60
00:03:51.000 --> 00:03:52.000
Then Paris.

61
00:03:53.000 --> 00:03:54.000
That's how we met.

62
00:03:54.500 --> 00:03:57.000
But I couldn't get work there, so I made Alvin move with me.

63
00:03:58.000 --> 00:03:59.000
And what do you do?

64
00:04:00.000 --> 00:04:02.000
Well, I'm an actress, of course.

65
00:04:03.000 --> 00:04:04.000
Oh, really?

66
00:04:05.000 --> 00:04:06.000
She's great.

67
00:04:07.000 --> 00:04:08.000
For commercials.

68
00:04:09.000 --> 00:04:10.000
I have a contract with an LA company.

69
00:04:11.000 --> 00:04:12.000
They've been grooming me.

70
00:04:13.000 --> 00:04:14.000
I specialize in failing naturally.

71
00:04:15.000 --> 00:04:16.000
What does that mean?

72
00:04:17.000 --> 00:04:18.000
Failing naturally?

73
00:04:19.500 --> 00:04:22.000
Finding a natural-seeming way to fail at any given task.

74
00:04:22.000 --> 00:04:23.000
In each of the commercials that I'm in,

75
00:04:24.000 --> 00:04:26.000
I'm the one who simply can't go on without the product.

76
00:04:27.000 --> 00:04:29.000
It's ridiculous of me not to have the product.

77
00:04:30.000 --> 00:04:31.000
Okay.

78
00:04:32.000 --> 00:04:33.000
Show them.

79
00:04:34.000 --> 00:04:35.000
No, you should.

80
00:04:36.000 --> 00:04:37.000
Do you want to see?

81
00:04:38.000 --> 00:04:39.000
I want to see.

82
00:04:42.000 --> 00:04:43.000
She's amazing.

83
00:04:53.000 --> 00:04:54.000
I just...

84
00:04:55.000 --> 00:04:56.000
I...

There's definitely a difference. Why does this difference occur?

@comienzo2093 Why you are spamming this? These tests were posted already.

@comienzo2093 Why you are spamming this? These tests were posted already.

Sorry, it's my mistake.
I repeatedly tested on the MBP14, large-v2 model, and the results are as follows:
test_original.aac
There was a significant difference between int8 and float32.
test_ffmpeg6.wav
There was no significant difference between int8 and float32.
Both int8 and float32 produce results similar to the int8 results of test_original.aac.

int8 vs float32 difference are expected. We talk about differences running exact same settings, like int8 vs int8.
Are float32 vs float32 different for you?

int8 vs float32 difference are expected. We talk about differences running exact same settings, like int8 vs int8.
Are float32 vs float32 different for you?

There was a significant difference between int8 and int8, but there was no significant difference between float32 and float32.

Strange, for me float32 vs float32 results were exact same. I'll redo tests to  be sure.
Btw, interesting that medium results are better than medium.en, maybe because of various English accents there.
Here are retail subs for test sample: https://we.tl/t-atMElx0fp0

Strange, for me float32 vs float32 results were exact same. I'll redo tests to be sure.
Btw, interesting that medium results are better than medium.en, maybe because of various English accents there. Here are retail subs for test sample: https://we.tl/t-atMElx0fp0

I ran the tests again.
In my case as well, the results between float32 were exactly the same. It could be expressed as follows. The difference between acc int8 and float32 was quite large, but the difference between wav int8 and float32 was very small.
acc int8: result B
acc float32: result A
wav int8: result Aa
wav float32: result A
I'm not sure why these differences occur.

Thanks for confirming. Maybe later a friend will do float16 tests for me.

Thanks for confirming. Maybe later a friend will do float16 tests for me.
What "A" and "B" means? "acc int8" with medium produces most accurate result, by accurate I mean transcription, not timestamps, timestamps are not accurate on with types, so personally I don't care about them.

A and B are just symbols to indicate similarity between results.
And I will comment more definitively about which transcription result is more accurate after a closer review.

@Purfview
--model "large-v2" --device "cpu" --beam_size 5
In my case, both transcription and timestamps were more accurate with the fp32 results (result A) than with int8(result B).

Sorry, there are indeed differences in the decoded audio. I did not expect the FFmpeg version to make a difference but it does actually. I downloaded FFmpeg binaries from https://www.gyan.dev/ffmpeg/builds/#release-builds.

FFmpeg 5.1.2 has the same result as PyAV which uses the same FFmpeg version
FFmpeg 6.0 has a different result

Strangely the decoded audio from FFmpeg 6.0 is also different than the one that was shared above test_ffmpeg6.wav.
import hashlib

from faster_whisper import decode_audio

def md5(path):
    audio = decode_audio(path)
    print(hashlib.md5(audio.data.tobytes()).hexdigest(), path)

md5("test_original.aac")
md5("test_ffmpeg6.wav")
md5("test_ffmpeg5_gyan.dev.wav")
md5("test_ffmpeg6_gyan.dev.wav")

1f9eff46a9e5a1316c4edf4786f554c4 test_original.aac
02dba39440e0fa9f35786abaaaea8203 test_ffmpeg6.wav
1f9eff46a9e5a1316c4edf4786f554c4 test_ffmpeg5_gyan.dev.wav
84bff41f130b7881ac4067a6891ead81 test_ffmpeg6_gyan.dev.wav

Could the problem be dithering being applied during resampling?

I checked the decoded audio for test_original.aac and test_ffmpeg6.wav. There are only a few samples that differ for ±1 in the s16 representation (41 out of 4894037). I'm not sure what happens in the audio at these times to have a different decoding with FFmpeg 6.

Time where a decoded sample is different, in seconds
[ 10.4261875  26.7394375  32.732375   63.516875   63.904625   65.14075
  65.467375   67.029125   67.044125   68.1555     69.457375   70.216875
  70.5359375  70.5680625  70.61475    72.3516875  75.186375   80.6776875
  85.0641875<PHONE_NUMBER> 120.663<PHONE_NUMBER><PHONE_NUMBER><PHONE_NUMBER>
 157.75425<PHONE_NUMBER> 179.50675   188.9505    202.828375  205.91625
<PHONE_NUMBER> 211.54825   229.986<PHONE_NUMBER> 236.940125  252.719625
 259.027     272.807375  277.77975<PHONE_NUMBER><PHONE_NUMBER>]

@Purfview When the transcription is different, is it consistently worse with FFmpeg 6 vs. FFmpeg 5 (PyAV)? Or is it sometimes worse and sometimes better?

I can confirm that ffmpeg v5.1.2 produces same results as PyAV. Btw, VAD is not affected by this issue.
I compared test_ffmpeg5.wav vs test_ffmpeg6.wav and I see only 30 one byte [±1] differences.
How one byte difference can propagate into such big differences in transcription and timestamps with int8?

When the transcription is different, is it consistently worse with FFmpeg 6 vs. FFmpeg 5 (PyAV)? Or is it sometimes worse and sometimes better?

I tested only this random file, and had reports from some users. My believe is that it shouldn't be consistently worse.
@guillaumekln What compute_type should be more precise or faster?

float32 is more precise and slower. int8 is less precise and faster.
int8 amplifies the difference in the input which could result in a different generation. Then this difference is further amplified by the Whisper transcription loop that feeds previous transcription as context.

float32 is more precise and slower. int8 is less precise and faster.

Hmm, on my Sandy Bridge CPU float32 runs ~1.7 times faster than int8 and in general int8 looks more precise.

In my project, I used ffmpeg to preprocess the audio ,and then sent the audio to faster-whisper.
I encounted the dead_lock when calling ffmpeg with subprocess in multi-thread. And I guessed the old centos may cause the problem, but the problem was still there even changing the OS to Ubuntu 18.
So, after rewriting this part with pyAV, this problem has gone.  I am still not sure if it's Python subprocess' bug or not.

Can you provide more info, and exact full, uncut, steps to reproduce your issues?

Can you provide more info, and exact full, uncut, steps to reproduce your issues?

What issue did you meant exactly? [Here are posted multiple ones]
"My" issue was that the outputs of ffmpeg v5 and ffmpeg v6 are different with this command:
ffmpeg -i audio.aac -ar 16000 -ac 1 -c:a pcm_s16le audio.wav
Probably it's reproducible on any audio file.

@Purfview
about hash part,
ffmpeg writes LIST chunk to output, you have to pass -bitexact to remove them. WAV is a lossy format, it may optimize for performance rather than reproducible. (yeah, that's odd but flac will get the same result)

WAV is a lossy format, it may optimize for performance rather than reproducible.

WAV is lossless format.

that's odd but flac will get the same result

What same result?

I'm closing this issue since there is nothing else to add in my opinion.
The original question was about replacing PyAV by FFmpeg which is totally possible. Then we found the results can be different between the 2. More specifically the results can be different between 2 FFmpeg versions. The differences in the decoded audio are small but can be amplified when running the model in low precision (e.g. int8).


Transaction rollback is inconsistent if exception raised during commit

OrientDB Version, operating system, or hardware.

v2.2.9 and earlier

Operating System

[* ] Linux
[ *] MacOSX
[ *] Windows
[ *] Other Unix
[ *] Other, all

Expected behavior and actual behavior
Assume Transaction that include changes in the separate blob cluster with ORecordBytes records. and changes in another ODocument record that has a field es linkedlist to these blob records. In that transaction two existing blobs were deleted and new one has been inserted. linked list field has been updated to point to the new blob and then transaction commit is called.
During commit an NullPointer exception has been raised due to (whatever reason, it doesn't matter actually). Expected behavior would be that new blob record is not inserted, two old blob records are not deleted and ODocument record still points to the old two records.
Actual behavior:
Old 2 Blob records are deleted.
New blob record is (half inserted). You can see it if you ask "clusters" in the console but you don't see it if you run statement "select from cluster:blob_cluster".
ODocument record still points to a non-existsing Blob records if you issue statement "select from #<document.rid>. In Java code attempt to get filed value returns a List with 2 null elements.
As a result we have a full loss of blob data instead of original values.
Steps to reproduce the problem
I have compiled a very simple test program that easily reproduces that problem.

Extract attached zip into a empty directory. (it contains 2 Eclipse Maven projects (NPESimulatorAgent and OOMTest)
Import bothe projects into empty eclipse workspace.
Run Maven "package" on the NPESimulatorAgent project. This will generate Java agent.jar that can throw NullPointerException from the com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect() method.
NPESimulatorAgent.jar should appear in the NPESimulatorAgent directory.
Now open class OutOufMemoryTest in the project OOMTest. This class has a main method and it can be launched using LaunchConfiguration OutOfMemoryTest.launch.

If you run this program it will create new Database, create one class "TestClass" with 2 fields: itemId and blob. Field blob is a linked list. Additionally it creates new cluster called "TestClass_blob" for storing blobs in it.
method writeDataAndCheck() will add one record with itemId = "1" and blob that refers to a chunk1 (512Kb) and Chunk2 (256Kb). Transaction is committed and reloaded to test that all data were saved correctly.
The fun starts in the simulateErrorAndCheck() method.
Here it will load that testclass record into ODocument and then start transaction that replaces chunk1 and chunk2 with new chunk3 (1MB).  In Line 264 you can control whether exception in the commit should be simulated or not. If exception is simulated it will printed to stderr and then database will be closed. and storage will be closed to release all in memory caches.
Then program tries to reopen database and read contents. You can see that if exception is happening during commit, contents of chunk1 and chunk 2 is lost. Chunk3 is actually stored in the cluster but it cannot be accessed anymore.
To me this is a showstopper bug. Blobs are lost without recovery.
OrientTest.zip

@nadestin I looked at the test which you provided,
and I think you misunderstand the meaning of ACID. ACID properties work if an implementation does not have errors. But what you do, you break implementation by bytecode manipulation and you claim that if the code in database implementation has errors this erroneous code should still support ACID properties does not matter how it changed. @lvca I think that test which breaks database code to simulate error can not be treated as a critical bug. Obviously, ACID properties are  implemented by code and we can not guarantee that if I break this code we will still have valid ACID properties.

@nadestin you throw NPE in .OByteBufferPool.acquireDirect()  do you have argumentation why this code should throw NPE if not , let suppose that in any RDBMS code which is responsible for rollback of data intentionally changed does it mean that rollback still should work ?

@nadestin I propose you to, for example, throw NPE in cluster or index manipulation code if you want to emulate such behavior but I am sure that infrastructure properties will work if and only if infrastructure code will work correctly.

@nadestin why a NullPointerException from the com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect() ? Is it what you had in the OrientDB logs?

I believe that there is a misunderstanding.
I have filled this bug report not out of curiosity.  The example program shows a Real problem and Real data loss. This sample program is a test case that I made to clearly and undoubtedly reproduce a real problem. I am trying to give you a good starting point for debugging Transaction behavior in the OrientDB. I am trying to help you, not attack you.
Here is extract from the real Stack Trace that we see (unfortunately) almost every day:
Caused by: java.lang.Exception: Failed to commit transaction
at com.riag.db.dbengine.DBConnection.commitTransaction(DrTax3_map:241) ~[DrTax3.jar:3.9.50 2016-09-09 17:55:21+0200]
at com.riag.db.dbengine.skin.orientdb.OrientDbSkin.saveItemContentToStorage(OrientDbSkin.java:4039) ~[na:na]
... 38 common frames omitted
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Error during transaction commit
DB name="DrTaxDB"
DB name="DrTaxDB"
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_101]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_101]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[na:1.8.0_101]
at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[na:1.8.0_101]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:428) ~[na:na]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:379) ~[na:na]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:261) ~[na:na]
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:153) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2133) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote$28.execute(OStorageRemote.java:1316) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote$28.execute(OStorageRemote.java:1289) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote$2.execute(OStorageRemote.java:198) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:230) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperation(OStorageRemote.java:195) ~[na:na]
at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:1289) ~[na:na]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:560) ~[na:na]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:106) ~[na:na]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2733) ~[na:na]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2703) ~[na:na]
at com.riag.db.dbengine.skin.orientdb.OrientDbSkin.commitTransaction(OrientDbSkin.java:924) ~[na:na]
at com.riag.db.dbengine.skin.orientdb.OrientDbSkin.commitTransaction(OrientDbSkin.java:113) ~[na:na]
at com.riag.db.dbengine.DBConnection.commitTransaction(DrTax3_map:185) ~[DrTax3.jar:3.9.50 2016-09-09 17:55:21+0200]
... 39 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect(OByteBufferPool.java:300) ~[na:na]
at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.cacheFileContent(OWOWCache.java:1281) ~[na:na]
at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.load(OWOWCache.java:636) ~[na:na]
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.updateCache(O2QCache.java:1111) ~[na:na]
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:353) ~[na:na]
at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.allocateNewPage(O2QCache.java:403) ~[na:na]
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.commitChanges(OAtomicOperation.java:419) ~[na:na]
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:418) ~[na:na]
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.endStorageTx(OAbstractPaginatedStorage.java:3078) ~[na:na]
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1486) ~[na:na]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:566) ~[na:na]
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:106) ~[na:na]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2733) ~[na:na]
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2703) ~[na:na]
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.commit(ONetworkProtocolBinary.java:1358) ~[na:na]
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:581) ~[na:na]
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:318) ~[na:na]
at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:194) ~[na:na]
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77) ~[na:na]
It is a bit long, but it clearly shows that source of exception was Line 300 in the class OByteBufferPool., in method acuireDirect().
That line looks like this:
final ByteBuffer db = bfh.buffer.duplicate();
Object bfh at that moment is definitely not null, otherwise exception would have been thrown much earlier. So it can only be member bfh.buffer that was not assigned (most likely due to problem allocating DirectMemory).
That is why I made a test case that simulates NPE exactly in this method. The only tiny difference that simulated exception is raised a bit earlier. Manipulating byte code was just a very simple and convenient way to simulate the problem. In the real application there are nor byte code manipulations at all.
The important point here is not why NPE happend. There is no way to guarantee that this or another exception will never be raised. Exception simple happens, that's life.
The point is: Transaction was not rolled back as it should in that case.
Here I need to disagree with @laa when he assumes that I don't understand ACID principle. Please re-read ACID definition at least here: https://en.wikipedia.org/wiki/ACID
"A" stands for "Atomicy" and it is defined:
(Start of citate)
Atomicity requires that each transaction be "all or nothing": if one part of the transaction fails, then the entire transaction fails, and the database state is left unchanged. An atomic system must guarantee atomicity in each and every situation, including power failures, errors, and crashes. To the outside world, a committed transaction appears (by its effects on the database) to be indivisible ("atomic"), and an aborted transaction does not happen.
(End of citate)
Sorry, but not atomic transaction cannot be called transaction at all.
So, please, consider this bug report as an attempt to help you to fix a problem and make OrientDB better. I am on your side, and trying to be constructive. I am sure that you know your code better and you can fix the problem faster.

Hi @nadestin sorry for the misunderstanding.
I suppose we may fix your issue without significant effort. Then we may continue to discuss atomicity concept.
I see that you run a server , do you run embedded server or standalone one?

Our application can run standalone and in the client/server mode.
In standalone mode OrientDB is embedded in the client.
In the Client/Server mode it is embedded in the server.

@nadestin so just to double check , this error happens in Client/Server mode when your application is embedded inside OrientDB server. Is it correct ? Could you send JVM settings which you use to run server /

No, This error happens in both modes. We saw it on standalone client and also in the client server mode.
And the problem is not NPE Exception.  Problem is that after this exception transaction is not properly rolled back what leads to the data loss, as shown in my test case. Rollback behavior should be fixed, NPE is not important here.

@nadestin As I wrote above this method is part of mechanics which is responsible for rollback to make rollback work we need to fix code which is responsible for rollback. Once again atomicity of database does not mean that it will immune to any error because if code responsible for correct error handling is broken it can not work correctly. Could you send me JVM settings which you use to run server ?

In that particular case Server was running on 32bit Java, with
-Xmx512m
-XX:MaxDirectMemorySize=512m
On a 32 Bit Java in Windows that is a Maximum what we can get.

@nadestin I see, what is disk cache size or you use default settings ?

Default settings

@nadestin I see now. I need time to think how to fix that problem without applying of special settings but for your case, we may resolve an issue quickly.  Could you use following additional settings -Dstorage.diskCache.bufferSize=448 -Dmemory.chunk.size=58720256.
If it works, I provide a detailed explanation of reason of issue.

Ok, since it is only System properties, it can be done during program start before OrientDB is loaded.
I will make these changes for all 32bit clients and servers. It will take some time before I get a feedback, but consequence of this error is data loss, and that still worries me.

@nadestin It worries me too that is why we are going to provide a solution which will not require manual configuration. But first of all, I need to now that I identified issue correctly.

So the problem seems to reside in the 32-bit JVM, right?

Null pointer Exception during commit was observed only in 32bit JVM (yet), but inconsistent transaction rollback can be reproduced on any JVM.

@nadestin as I mentioned above if code responsible for rollback is broken obviously rollback can not happen. So claim that npe from any part of code should be correctly handled is incorrect because of we need to make rollbacks without NPE to make them to work correctly.

@nadestin @lvca the problem as I can see it is following. The system has several GB of memory.
OrientDB detects that  and uses disk cache with few GB of memory. But it may address only hundred of MB of memory and as result if it try to acquire more for it needs system fails.
One of the approaches to resolving this situation and any other similar once is to allocate all memory required by disk cache during system start in such case if there will be any problems with system settings this will be known as head and data will not be broken.
Sure not an ideal solution but it will prevent surprises in the middle of execution of the system.
And of course we will try to detect 32 JVM inside our system and decrease disk cache size, but I am not sure now if that is possible.

AFAIK, there are no bullet proof way to figure out if JVM is 32 bit or 64, but following code is covering most real world cases. For some exotic JVMs another way should be found.
public static int getJavaBitWidth()
{
// Figure out whether bit width of running JVM
// Most of JREs support property "sun.arch.data.model" which is exactly what we need here
String dataModel = System.getProperty("sun.arch.data.model", "32"); // By default assume 32bit
int size = 32;
try
{
size = Integer.parseInt(dataModel);
}
catch(Throwable t)
{
// Ignore
}
return size;
}
By the way to @laa. The proposed parameters for 32bit JVM seem to work. I have performed tests with big databases that had crashed on 32bit before. With your settings for storage.diskCache.bufferSize and memory.chunk.size I was able to complete tests without errors. We are still waiting for response from the field, but that workaround looks promising.
Why don't you publish methods of calculating values in the Manual. When I install Application I usually know startup parameters, so I could properly initialize these properties without hacking around.
Could you share how did you calculate values 448 and 58720256?

Hi @nadestin . A calculation is simple. The maximum amount of direct memory which you may allocate is 512MB I put 64MB for service needs which are more than enough and also split 448 MBs into 8 chunks. So  I get 56 MB for each chunk or 58720256 bytes.

@nadestin once you confirm that issue is fixed I will provide fix.

Hello @laa. I confirm, with adjusted parameters we have no more cases of NPE during Direct Memory allocations on 32bit JVMs. (On 64bit JVMS it is working without parameter adjustments).
Your workaround had fixed the problem.
Thank you for your help.

Hi @nadestin , OK we will provide complete fix during next week. Thank you very much for your report and your time.

@nadestin could you build latest 2.2.x version and run it on the same case without any additional parameters ?

Hi @nadestin , I close this issue, please reopen issue if you see it again.

Sorry, I was on vacation last week, will try it and report status.


[Fix #118] Fix an incorrect autocorrect for Rails/Validation

Fixes #118.
This PR fixes an incorrect autocorrect for Rails/Validation when attributes are specified with symbol array literal.
The following auto-corrected causes NoMethodError.
validates %i[full_name birth_date], presence: true

# => validates_with': undefined method `to_sym' for
#    [:full_name, :birth_date]:Array (NoMethodError)

This PR will auto-correct it to the following code.
validates :full_name, :birth_date, presence: true

Before submitting the PR make sure the following are checked:

[x] Wrote good commit messages.
[x] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
[x] The PR relates to only one subject with a clear title
and description in grammatically correct, complete sentences.
[x] Run bundle exec rake default. It executes all tests and RuboCop for itself, and generates the documentation.

I imagine regular array literals also affected? 🤔
validates_presence_of [:foo, :bar]

What about splatted arrays?
validates_presence_of *[:foo, :bar]
validates_presence_of *%i[foo bar]

Thanks for your review. Array literals and frozen array literals are also expanded.
On the other hand, it is not applied to the following code because applying a splat to an array literal causes a syntax error.
*[:foo, :bar]
# => SyntaxError ((irb):1: syntax error, unexpected '\n', expecting '.' or &. or :: or '[')
*%i(:foo, :bar)
# => SyntaxError ((irb):2: syntax error, unexpected '\n', expecting '.' or &. or :: or '[')


ERROR: The cluster's authentication mode must be set to one of [API, API_AND_CONFIG_MAP] to perform this operation.

Description
We are trying to upgrade the EKS to 1.28 from 1.27 and the module (from 19.10.0 to 20.11.1) and also migrate the authentication mode from Auth_config to API-AND-CONFIG_MAP. I followed the upgrade v19-v20 document, upgraded the providers, and added the necessary entries. The execution is failing with the The cluster authentication must be set error.

[X ] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]:

Terraform version: v1.3.2

Provider version(s):

Terraform v1.3.2
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v5.50.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.4
+ provider registry.terraform.io/hashicorp/helm v2.9.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.30.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/time v0.11.1
+ provider registry.terraform.io/hashicorp/tls v4.0.5

Reproduction Code [Required]
module "eks" {
  # checkov:skip=CKV_TF_1
  source              = "terraform-aws-modules/eks/aws"
  version             = "~> 20.11.1"
  cluster_name        = local.cluster_name
  cluster_version     = var.kubernetes_version
  enable_irsa         = true
  vpc_id              = module.new_vpc.vpc_id
  subnet_ids          = module.new_vpc.private_subnets
  authentication_mode = "API_AND_CONFIG_MAP"

  cluster_endpoint_public_access           = true
  cluster_endpoint_private_access          = true
  enable_cluster_creator_admin_permissions = true

  cluster_endpoint_public_access_cidrs = concat(
    var.cidr_blocks,
    split(",", data.aws_ssm_parameter.nat_gw_eips.value)
  )

  # https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  cluster_encryption_config = {
    provider_key_arn = aws_kms_key.etcd_encryption_key.arn
    resources        = ["secrets"]
  }

  cluster_addons = {
    coredns = {
      addon_version = "v1.10.1-eksbuild.10"
      configuration_values = jsonencode({
        corefile = <<EOF
      .:53 {
          errors
          health
          hosts {
            ${local.coredns_host_config}
            fallthrough
          }
          kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
          }
          prometheus :9153
          forward . /etc/resolv.conf
          cache 29
          loop
          reload
          loadbalance
      }
      EOF
      })
    }
    kube-proxy = {
      addon_version = "v1.28.6-eksbuild.2"
    }
    vpc-cni = {
      addon_version            = "v1.18.1-eksbuild.1"
      service_account_role_arn = module.vpc_cni_irsa.iam_role_arn
      configuration_values = jsonencode({
        env = {
          # Reference docs https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html
          ENABLE_PREFIX_DELEGATION = "true"
          WARM_PREFIX_TARGET       = "1"
          WARM_IP_TARGET           = "5"
          MINIMUM_IP_TARGET        = "2"
        }
      })
    }
    # AWS Distro for OpenTelemetry
    adot = {
      addon_version = "v0.94.1-eksbuild.1"
      # https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2731
      most_recent = true
    }
    aws-efs-csi-driver = {
      addon_version            = "v2.0.1-eksbuild.1"
      service_account_role_arn = module.efs_csi_irsa_role.iam_role_arn
    }
  }

  iam_role_additional_policies = {
    additional = var.cloudwatch_server_policy_arn
  }

  node_security_group_additional_rules = {
    ingress_self_https = {
      description = "Enable https self traffic"
      protocol    = "TCP"
      from_port   = 443
      to_port     = 443
      type        = "ingress"
      self        = true
    }
    ingress_self_http = {
      description = "Enable http self traffic"
      protocol    = "TCP"
      from_port   = 80
      to_port     = 80
      type        = "ingress"
      self        = true
    }
  }

  # As we have enabled `attach_cluster_primary_security_group = true` we had to set the below tag as null
  node_security_group_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = null
  }

nodegroup configs
  eks_managed_node_group_defaults = {
    instance_types                        = var.managed_node_group_instance_type
    iam_role_attach_cni_policy            = true
    attach_cluster_primary_security_group = true # default is false
    iam_role_additional_policies = {
      eksKedaCloudWatchMetricsHPA = aws_iam_policy.keda_cloudwatch_metrics.arn
      additional                  = data.aws_iam_policy.cw_agent_policy.arn
      ssmPolicy                   = data.aws_iam_policy.ssm_agent_policy.arn # Added ssm policy to the nodes to login to the nodes
    }
    pre_bootstrap_user_data = <<-EOT
      export USE_MAX_PODS=false
      export KUBELET_EXTRA_ARGS="--max-pods=110"
    EOT
  }

  eks_managed_node_groups = {
    node-group-1 = {
      min_size     = local.eks_asg_min_node_count
      max_size     = local.eks_asg_max_node_count
      desired_size = local.eks_asg_desired_node_count

      instance_types = var.managed_node_group_instance_type
      capacity_type  = var.eks_capacity_type
      ami_id         = var.eks_worker_ami
      cpu_credits    = "unlimited"
      enable_bootstrap_user_data = true

      tags = local.cluster_autoscaler_tags
    }
  }

  fargate_profiles = {
    argo-events = {
      name = "argo-events"
      selectors = [
        {
          namespace = "argo-events"
          labels = {
            Application = "argo-fargate"
          }
        }
      ]
      tags = {
        Owner = "secondary"
      }
    }
  }

  tags = local.tags
}

module "eks_aws_auth" {
  # checkov:skip=CKV_TF_1
  source                    = "terraform-aws-modules/eks/aws//modules/aws-auth"
  version                   = "~> 20.11.1"
  manage_aws_auth_configmap = true
  aws_auth_roles            = local.eks_iam_rbac_mappings
}

Steps to reproduce the behavior:
 yes
 yes
 Upgrading the module from `19.10.0` to `20.11.1`
Expected behavior
EKS upgrade and the authentication_mode migrated to API_AND_CONFIG_MAP
Actual behavior
EKS is upgraded to 1.28 however the code failed to create the nodes and failed with the following error.
 Error: creating EKS Access Entry (aws_core_eks:arn:aws:iam::681496624581:role/admin): operation error EKS: CreateAccessEntry, https response error StatusCode: 400, RequestID: c77dd87a-bdc7-41fb-b1ba-b45027c03766, InvalidRequestException: The cluster's authentication mode must be set to one of [API, API_AND_CONFIG_MAP] to perform this operation.
│
│   with module.eks.aws_eks_access_entry.this["cluster_creator"],
│   on .terraform/modules/eks/main.tf line 191, in resource "aws_eks_access_entry" "this":
│  191: resource "aws_eks_access_entry" "this" {
│
╵

Additional context
Note:
Have performed the following state removal before applying the changes.
terraform state rm 'module.eks.kubernetes_config_map_v1_data.aws_auth[0]'

State module.eks.kubernetes_config_map.aws_auth[0] was not present.

Have downgraded the module version to 20.8.2 which solved the above issue and upgraded the EKS cluster.

Have downgraded the module version to 20.8.2 which solved the above issue and upgraded the EKS cluster.

After moving to authentication_mode=API, have upgraded to the latest tag.

We had the same issue and tried upgrading to k8s first 1.28 separately (with module version 19) and got the same errors even with pinning 20.8.2 in a subsequent change.

In the end we had to set enable_cluster_creator_admin_permissions = false and run terraform apply twice, since there appears to be a race condition (at least in v20.8.2 and v20.16.0) between changing the authentication_mode and module.karpenter.aws_eks_access_entry.node[0]. I found https://github.com/terraform-aws-modules/terraform-aws-eks/blob/v20.16.0/modules/karpenter/main.tf#L685 but clearly that's insufficient. Can we re-open this ticket? Or should I open a new one?
A (very) minimal example for the Karpenter issue is something like the following.
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0" # i.e. 20.8.2 or 20.16.0
  # ...
  # NOTE: was "CONFIG_MAP"
  authentication_mode = "API_AND_CONFIG_MAP"
  # ...
  # NOTE: was "1.27" but this part proved irrelevant in testing
  cluster_version = "1.28"
  # ...
}

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 20.0"
  # ...
}

since there appears to be a race condition (at least in v20.8.2 and v20.16.0) between changing the authentication_mode and module.karpenter.aws_eks_access_entry.node[0]

correct - changes to the access entry API mode are not immediate so if you try to change the mode and use auth based on that mode, there is a race condition. its recommended to change the access entry mode first, then make any downstream changes

N.B. We have not tried setting enable_cluster_creator_admin_permissions = true after successfully upgrading to 20.x and setting authentication_mode = "API_AND_CONFIG_MAP" but I assume that will result in a simliar error to the one we've seen:

This usually occurs with folks trying to create access entries for the managed node group IAM role and/or the cluster creator - you can see more details here
tl;dr - when converting from configmap only (default case for all clusters created prior to access entry), EKS will automatically add an access entry for any roles that are used by EKS managed node group, Fargate profiles, as well as the cluster creator identity. Our version via enable_cluster_creator_admin_permissions is not the same as the EKS API version - which is a one time decision at time of cluster creation. Our implementation here is using the access entry API to allow full control over those changes - now and in the future. (ref https://github.com/terraform-aws-modules/terraform-aws-eks/blob/c52bbb32a41e5f5cd2b1fd1d9b04ee1baa8e496e/main.tf#L38-L43)

Thanks very much for this explanation! Totally clears it up.

also - there are new docs coming that I hope will help with some of these questions - we have been woefully behind in terms of providing these details and nuances so thats on me to get those out https://github.com/terraform-aws-modules/terraform-aws-eks/compare/master...bryantbiggs:terraform-aws-eks:chore/new-docs


Support ANSI line manipulation (other than full clear)

I have a program that I was hoping to use asciinema to demonstrate, but it doesn't render correctly: https://asciinema.org/a/ufb6UgP828NL8noQxuii7c2rR
Escape sequences used:

CUU
CUF
CUB
EL (sometimes with option 0, others 1)

It also makes heavy use of printing newlines with \r to to go to beginning of the line. Below is what it actually looks like when running:
https://user-images.githubusercontent.com/8285973/107308333-af397c80-6a45-11eb-833d-da6ce6bff23c.mp4

This has been fixed in e363079197abd037fee439047a5db4a2ac046361


Automatic page_view events on router change

Describe the feature
Hey!
Are you open to the idea of calling useTrackEvent('page_view') (perhaps with route metadata) automatically on route change?
I'm thinking it's optional, disabled by default but you can opt-in via nuxt.config. This would have the module act a little more like a typical GA site tracker, where every page view is tracked automatically.
If this makes sense for the module, I can submit a PR. Thoughts?
P.S. Happy New Year
Additional information

[X] Would you be willing to help implement this feature?
[x] Can you think of other implementations of this feature?

Final checks

[X] Check existing issues.

Great idea! PR happily welcome. 🙂
I'm not sure if a Nuxt hook is the best choice of just using the useRouter().afterEach method. Maybe you can look into that. Thanks in advance.
Dito, happy New Year to you too.

Isn’t page tracking done by default already with the default setup (before this PR?)

@notflip Correct. See the PR discussion: https://github.com/johannschopplich/nuxt-gtag/pull/38#issuecomment-1874356691
Closing this for now, since enhanced management will already watch for history state changes, which should cover most use cases. Kudos to @marchantweb for the PR tho!


Revert "Create LayerDataProvider class  and CatalogLayer class"

Reverts heremaps/here-olp-sdk-typescript#99
This commit reverts PR #99 as the LayerDataProvider and CatalogLayer provide extensive functionality that eclipses both VersionedLayerClient and VolatileLayerClient, providing users the possibility to only use CatalogLayer. This will put pressure to extend CatalogLayer with new functionality all the time.
Relates-to: OLPEDGE-1241
Signed-off-by: Andrei Popescu<EMAIL_ADDRESS>

Codecov Report

Merging #122 into master will decrease coverage by 1.1%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master    #122     +/-   ##
========================================
- Coverage    83.5%   82.4%   -1.2%
========================================
  Files          43      39      -4
  Lines         852     787     -65
  Branches      130     120     -10
========================================
- Hits          712     649     -63
+ Misses        105     104      -1
+ Partials       35      34      -1

Impacted Files
Coverage Δ

@here/olp-sdk-dataservice-read/lib/index.ts
100% <ø> (ø)
:arrow_up:

@here/olp-sdk-dataservice-read/lib/cache/index.ts
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cce3227...b220018. Read the comment docs.


Change path

Hello i have a question i proved your screen recorder and it's amazing, but when i stop Record the video is saved in cache, and i would like to know if the path could be changed, and how.
Thanks

This feature will be added as soon as possible. Don't worry. Have a nice day.

Added Custom File Path - Directory feature. Can you please test it? @linnminhtet23 @DanteAlucard98

Thanks, i test it and wooow awesome :3


.\elan-init.ps1 fails with UnauthorizedAccess on a fresh Windows machine

The README installation instructions for Windows fails on a freshly installed Windows 10 machine with:
C:\Users\kim>powershell -f elan-init.ps1
File C:\Users\kim\elan-init.ps1 cannot be loaded because running scripts is disabled on this system. For more
information, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170.
    + CategoryInfo          : SecurityError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : UnauthorizedAccess

Does adding -ExecutiinPolicy Bypass to that line help?

Yes, that does work. Or rather, it gets us to the next error message:
info: downloading installer to C:\Users\kim\AppData\Local\Temp\
Elan failed with error code -1073741515
1


[ haddock ] first draft of comments on Normalize

This is my draft of comments on the Normalize module. It should be read over, and there were a few FIXMEs I left as I wasn't sure how to explain a few of them. It seems to me that only a few of these functions are even exported, but I did comments for all of the ones I could find!

This is nice, feel free to merge it yourself.

Ok can I merge this @CSchank ?

Okay, this should be good to merge! I don't have proper access to the repo to merge though, apparently. Or maybe the CircleCI needs to finish first. I'll try again when it does.


Separate loader asi and ScriptDomain dll

There's at least 3 reasons that separation will benefit us.
Known Benefits

Enables to use managed NuGet packages

I'd like to use ImGui via ImGui.NET so we won't have to deal with difficult output line counting for the console, but packing native code and managed code in one dll prevents from using managed NuGet packages 😭

Enables Rider's suggestions to work

Rider doesn't work at all for C# files in C++/CLI projects. Ffs moment.

Enables nullable in the ScriptDomain project

Unfortunately, you can't use nullable reference types in netmodule files (maybe only in C# files in C++/CLI projects) even if you manually define or import mandaroty attributes for nullable reference types with such as the Nullable package. I'd like to adopt nullable reference types and avoid silly mistakes on null references. Well, maybe not too beneficial as in .NET Core 3.x and .NET 5+ , since BCL for earlier runtimes don't have nullable attributes for notations.

Potential Benefits

Maybe we can unload asi if manually hosting .NET runtime AND the runtime doesn't pin handles of the dll that has initiated the runtime? If impossible to resolve, we'll close #547 with how to reproduce.

#1178 might be relevant


Enhancing User Profiles with Vibrant Colors

Related Issue
[Cite any related issue(s) this pull request addresses. If none, simply state “None”]
Fixes:  #3946
Description
The "Profile Coloring Remodified" initiative aims to enhance the user experience by implementing a vibrant and customizable color palette for user profiles. This update not only refreshes the visual appeal of profiles but also allows users to express their individuality through personalized color selections. The redesign focuses on improving user engagement and satisfaction, creating a more inviting and dynamic platform. Key features include seamless integration of color customization options, ensuring accessibility, and maintaining a cohesive aesthetic throughout the application. This project reflects our commitment to fostering a user-friendly environment that celebrates creativity and personalization.
#3946
Type of PR

[X] Bug fix
[ ] Feature enhancement
[X] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
[Attach any relevant screenshots or videos demonstrating the changes. Make sure to attach before & after screenshots in your PR.]
https://github.com/user-attachments/assets/fb68d933-40b5-4ef5-967a-15edf0f2bd2e
Checklist:

[X] I have made this change from my own.
[ ] I have taken help from some online resources.
[X] My code follows the style guidelines of this project.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly in hard-to-understand areas.
[X] I have made corresponding changes to the documentation.
[X ] My changes generate no new warnings.
[X] I have tested the changes thoroughly before submitting this pull request.
[X] I have provided relevant issue numbers and screenshots after making the changes.

this does not look vibrant at all . please enhance the UI of the section keeping in mind the color combo of the website .
u can add background similar to the one in the home page .
update the footer also ...  like that in on the home page .

work with updated branch @govindbairi


Fixing cases when ValueError from alphalens.tears.create_factor_tear_sheet

When we run create_factor_tear_sheet on either:

single value for factor per group (only one value over a date, or sector)
Field with discreet value (not sure if this is a correct use case for alphalens)

We are can run into exception seen below.
qcut over similar edges have been discussed but no solution is implemented yet "officially":

http://stackoverflow.com/questions/20158597/how-to-qcut-with-non-unique-bin-edges
https://github.com/pydata/pandas/issues/7751#issue-37814702

So this provided a simple work around for case when we run into this issue in a simple case, when we have only seen one value of the factor.
Exception seen
/usr/local/lib/python2.7/dist-packages/alphalens/performance.pyc in quantile_calc(x, quantiles)
    242
    243     def quantile_calc(x, quantiles):
--> 244         return pd.qcut(x, quantiles, labels=False) + 1
    245
    246     grouper = ['date', 'sector'] if by_sector else ['date']

/usr/local/lib/python2.7/dist-packages/pandas/tools/tile.pyc in qcut(x, q, labels, retbins, precision)
    167     bins = algos.quantile(x, quantiles)
    168     return _bins_to_cuts(x, bins, labels=labels, retbins=retbins,precision=precision,
--> 169                          include_lowest=True)
    170
    171

/usr/local/lib/python2.7/dist-packages/pandas/tools/tile.pyc in _bins_to_cuts(x, bins, right, labels, retbins, precision, name, include_lowest)
    187
    188     if len(algos.unique(bins)) < len(bins):
--> 189         raise ValueError('Bin edges must be unique: %s' % repr(bins))
    190
    191     if include_lowest:

ValueError: Bin edges must be unique: array([-0.0030303, -0.0030303, -0.0030303, -0.0030303, -0.0030303,
       -0.0030303])

So we have ran into this issue as well, the first and most obvious solution is to reduce the number of quantiles, but it looks like in from your trace that you are assigning groups of size N >> 1 a value and trying to look at the performance of each group? So are you classifying groups of stocks into varying degrees of under/over performing?

Closing due to inactivity.


[DO NOT MERGE] Avoid running tests in parallel

Test PR to see if running tests not in parallel makes a difference

/ok-to-test

/ok-to-test

/ok-to-test

This doesn't seem to be making a significant difference. Closing.


Add Traditional Chinese readme and corresponding texts in main.py

Type of changes

[ ] Bug fix
[ ] New feature
[x] Documentation / docstrings
[ ] Tests
[ ] Other

Checklist

[x] I've run the latest black with default args on new code.
[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
[ ] I've added tests for new code. (Not checked because it's a documentation change.)
[x] I accept that @willmcgugan may be pedantic in the code review.

Description
Add Traditional Chinese readme and texts in __main__.py's Asian language support section. The original Simplified Chinese readme file is renamed to README.zh-cn.md in order to reflect that both versions of readme in Chinese are available. Changes to the navigation links in other readme files are also applied to follow mentioned renaming.
Traditional Chinese readme is reviewed by @jw910731
New screenshot may be needed because __main__.py are changed.
Edit: Update information about the reviewer

This is great. Thank you so much.
My only concern is that links to the original Chinese readme will be broken with this change. Perhaps we should keep the original Chinese readme, and add the traditional under a new filename?

To verify the change won't break links to the original Chinese readme, I created a new branch readme_test in my organization to test. It turns out that the experiment is successful. Since the links in readme were hard coded to the master branch of Textualize's rich repo, I modify them to the readme_test branch in l10n-tw's rich repo. The modification was only applied to English version for testing purpose, which means the links to both Chinese readmes in other language are, of courese, still broken. Hope this alleviate your concern.
By the way, would you have a plan to change links from absolute URL to relative path like ./README.de.md?

It was actually links from outside of the project I was concerned about. If somewhere on the web links to the Chinese README, it would break with this change.

Understood. The latest commit reverts Simplified Chinese readme to its original file name. Links in other readme are also changed. It should work without problems.

謝謝!

Conflict on CHANGELOG.me is resolved.

Thanks!


Consider adding support for struct fields to have a default value of nothing

In issue #164, "gbc shouldn't allow struct fields to have default of nothing", the handling in gbc for the default value of structs was tightened up to match the documented/specified behavior:

An optional default value can be specified for fields of basic types. For integers the default can be specified as either a decimal number or a hexadecimal number prefixed with 0x. The only explicit default value allowed for containers is nothing. Enum fields must have an explicit default value which must be one of the enum named constants or nothing.

Nowhere is a default value for struct/bonded fields mentioned as allowed.
This behavior matches the behavior of an older, Microsoft-internal version of Bond. (The internal version of Bond is a different code base, which is how the discrepancy was introduced.)
However, there are cases when it can be useful to have a struct field with a default value = nothing. For example, a consumer can deserialize then check whether struct field has a value. If so, the producer knew about the struct and the values inside make sense. If not, the producer didn't know about the struct, so different behavior (e.g., substitute default values) should be taken.
We should consider whether this feature makes sense broadly and can be implemented in a way that preserves .bond file compatibility with the two Bond compilers (e.g., some sort of --compatibility-mode switch)

This would be a great feature to add to the Bond IDL grammar. The C++ library already seems to support it. I am able to do the following and it has been very useful:
struct Message {
0: T message = nothing;
}
struct MyStruct {
...
}
Message msg;


add spdlog library and improve logging

Summary
This PR integrates the spdlog library to enhance the logging functionality within the project. The spdlog library has been added as a git submodule under extern/helper/spdlog and incorporated into the build system via the CMakeLists.txt configuration.
Test Plan
To activate trace logging, the trace-enabled argument can be set in the system input file. The corresponding trace logs will be generated and stored in log/simulated-trace.txt.
Additional Notes
At this stage, the logging enhancements are limited to trace-related output. Future iterations may extend this logging improvement to cover all logging operations across the project.

Would fix #232

This feature has been resolved via https://github.com/astra-sim/astra-sim/commit/28b426ccb1a452bda3bcbe8275c22862d48d3043


Refactor tests for mp.jwt.verify.token.age into their own class and update suites

Signed-off-by: Teddy J. Torres<EMAIL_ADDRESS>Adds TokenAgeTest for testing mp.jwt.verify.token.age and update the base and full suites.

@sberyozkin Greetings. These are the TCK updates for token age. Thanks!

The new TokenAgeTest and DecryptKeyAlgorithmDefaultTest pass in the implementation.


Internal Server error, status code 500 returned when trying to create orgs

We have been seting up a cloud foundry ecosystem based on internal OpenStack hardware.
We have installed cloud foundry on the openstack system following the instructions at https://docs.cloudfoundry.org/deploying/openstack/index.html and everything seems to be working
We used the latest HEAD on master (as of about a week ago)
I can do a curl on the api info endpoint and it returns information about the system.
I then install the cf cli and logged onto the system (several problems here - but resolved)
[mtlgrr@host-192-168-0-86 ~]$ cf login
API endpoint: https://api.cf.memex.sas.com
Email> api.cf.memex.sas.com
Password> [mtlgrr@host-192-168-0-86 ~]$ cf login
API endpoint: https://api.cf.memex.sas.com
Email> admin
Password>
Authenticating...
OK
API endpoint:   https://api.cf.memex.sas.com (API version: 2.56.0)
User:           admin
No org or space targeted, use 'cf target -o ORG -s SPACE'
FAILED
Error finding available orgs
Server error, status code: 500, error code: 0, message:

THere are no orgs or spaces defined yet so the error is no great surprise. However when I try and create an org
[mtlgrr@host-192-168-0-86 ~]$ cf create-org svi
Creating org svi as admin...
FAILED
Server error, status code: 500, error code: 0, message:

I tried again with the -v flag but no real extra information
I tried tracing the request through the log files on the system. I can see it moving from the haproxy to the router and eventually to the nginx server on the api_z1 VM. The nginx server seems to pass it off to the cloud_controller but it is at this point that we are getting the 500 error on the server.
The cloud_controller doesn't seem to log anything useful!
Can anyone give any hints as to what might be going wrong?

#975. Hey, I am also facing a similar issue. Did you manage to fix it?
You could set CF_TRACE=true and see the HTTP requests that are going through

No solution yet. I had set the -v flag but there wasn't enything useful reported
for completness here is the command with CF_TRACE set
[mtlgrr@host-192-168-0-86 ~]$ cf logout
Logging out...
OK
[mtlgrr@host-192-168-0-86 ~]$ cf login -u admin --skip-ssl-validation
API endpoint: https://api.cf.memex.sas.com
Password>
Authenticating...
OK
API endpoint:   https://api.cf.memex.sas.com (API version: 2.56.0)
User:           admin
No org or space targeted, use 'cf target -o ORG -s SPACE'
FAILED
Error finding available orgs
Server error, status code: 500, error code: 0, message:
[mtlgrr@host-192-168-0-86 ~]$ export CF_TRACE=true
[mtlgrr@host-192-168-0-86 ~]$ cf create-org svi
Creating org svi as admin...
REQUEST: [2016-06-22T05:35:05-04:00]
POST /v2/organizations HTTP/1.1
Host: api.cf.memex.sas.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.19.0+b29b4e0 / linux
{"name":"svi"}
RESPONSE: [2016-06-22T05:35:05-04:00]
HTTP/1.1 500 Internal Server Error
Content-Length: 21
Content-Type: text/plain
Date: Wed, 22 Jun 2016 09:39:35 GMT
Server: nginx
X-Vcap-Request-Id: 129d304e-cb7f-4aae-4dec-7b90d5df09a3
Internal server error
FAILED
Server error, status code: 500, error code: 0, message:

As you can see there is nothing informative here.

Hey, I maanged to fix this issue (only to run into another:()
The fix for me was to use PEM encoded keys for jwt signing and verification.
ssh-keygen -t RSA -f testkeys
ssh-keygen -f testkeys.pub -e -m pem

Copy the private key to jwt.signing_key`` and the PEM public key to jwt.verification_key```
Hope that helps!

That fixed it we had a typo in  the PEM keys.
Could I suggest that we have some sort of reasonable error message that would help in this situation.

Did you folks both happen to have typos in your PEM keys, or did some documentation lead you astray?

The wrong key was generated (in our case).
It would be better if there was some sort of logged message rather than an ambiguous "Internal Server Error"
Putting a message in the internal log files saying something like "JWT keys do not match" or "JWT Authorization error" would have saved a lot of time as we would have known to look at the keys.
As it was we weren't sure if we had somehow corrupted the image or the platform we were running on or anything - there were no clues

@graememross that sounds pretty sensible.  The component that gets the key pair, and could validate that they match, is the UAA.  I would recommend opening an issue on the uaa-release repo with some details about what input you had, what error you saw, and what error message you'd like to see.
Cheers,
Amit


Add extra maintainers

There are a pile of unmerged pull requests, is it possible to add one or more maintainers to this project? I'm super excited about it and want to keep seeing the feature pile in.

anyone wanna apply?

I would be happy to start merging in PRs and getting releases cut.

@jakepearson added

@egoist I'd be interested, as well. I'd be interested to try to create a new major release sometime in the near future that integrates some of the PRs, etc. It's been a year since the last push to npm and I think there are some worthwhile things that could be brought online.