Whisper Python performance - Benchmarking

Running on OpenVoiceOS, RaspberryPi 4 - 2GB model. Using Python 3.10 and Tensorflow-lite 2.11
With the tiny model;
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper.tflite -t 4
Importing tensorflow, numpy and torch
Importing whisper
Loading tflite model models/whisper.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/test.wav
Samplerate: 16000, length: 30.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 Bili always listens to his mother. He always does what she says. If his mother says,

Inference took 4.74s for 30.0s audio file.

Loading audio file: samples/test_1.wav
Samplerate: 16000, length: 30.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 David lost his yellow pencil. He could not find it. Where is my yellow pencil? He asked his sister. His sister did not know. I don't know where your pencil is. She said David thought about it. He thought and thought. He used his yellow pencil for before lunch. He used it to write a note to his teacher. The notes said, dear teacher, thank you for helping me, David. He put the note in the envelope where was the envelope?

Inference took 8.57s for 30.0s audio file.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.28s for 11.0s audio file.

And with the two files from;
https://github.com/fquirin/speech-recognition-experiments/tree/main/test-files
Loading audio file: samples/en_sh_lights_70pct_4s.wav
Samplerate: 16000, length: 3.575875s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 Set the lights in the living room to 70%.

Inference took 3.5s for 3.58s audio file.

Loading audio file: samples/en_speech_jfk_11s.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.26s for 11.0s audio file.

https://github.com/ggerganov/whisper.cpp/issues/7#issuecomment-1397467197
Thinking an arg might be better as with big.Little often its better just to use the big as have found its sometimes faster than all cores.
Also being trying to work out how to get floating point times so we get fractions of a second
We have end.tv_sec-start.tv_sec
Whats the best way of adding tv_usec and is there just time float alternative?

Am I correct, it only decodes one pass of 30 seconds? I can't seem to get the full transcribe of >30 seconds wav files. It just continues to the next wav file with the test.py file.

orangepi@orangepi5:~/openai-whisper/minimal_build$ ./minimal ../models/whisper.tflite ../samples/test.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 2 seconds

[_SOT_][_NOT_] Bili always listens to his mother. He always does what she says. If his mother says, brush your teeth, Bili brushes his teeth.  Bili is a very good boy, a good boy listens to his mother. His mother does not have to ask him again. She asks him to do something one time and she does not ask again. Bili is a good boy. He does what his mother asks the first time. She does not have to ask again.

Seems something to do with the small model

Yeah I am talking about the above
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 12 seconds
terminate called after throwing an instance of 'std::out_of_range'
  what():  map::at
Aborted

Happens with the small model but fine with tiny but yeah we only get the 1st 30sec beamsearch
orangepi@orangepi5:~/openai-whisper/minimal_build$ ./minimal ../models/whisper.tflite ../samples/gb0.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 2 seconds

[_SOT_][_NOT_] Good morning. This Tuesday is Election Day. After months of spirited debate in vigorous campaigning, the time has come for Americans to make important decisions about our nation's future and encourage all Americans to go to the polls and vote. Election season brings out the spirit of competition between our political parties. And that competition is an essential part of a healthy democracy. But as the campaigns come to a close, Republicans, Democrats, and independents can find common ground on at least one point. Our system of

wget --quiet --show-progress -O samples/gb0.ogg https://upload.wikimedia.org/wikipedia/commons/2/22/George_W._Bush%27s_weekly_radio_address_%28November_1%2C_2008%29.oga
ffmpeg -loglevel -0 -y -i samples/gb0.ogg -ar 16000 -ac 1 -c:a pcm_s16le samples/gb0.wav

Yeah, posted that error over at the now closed issue at whisper.cpp
the small model is not yet correct.

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper.tflite samples/gb0.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 9 seconds

[_SOT_][_NOT_] Good morning. This Tuesday is Election Day. After months of spirited debate in vigorous campaigning, the time has come for Americans to make important decisions about our nation's future and encourage all Americans to go to the polls and vote. Election season brings out the spirit of competition between our political parties. And that competition is an essential part of a healthy democracy. But as the campaigns come to a close, Republicans, Democrats, and independents can find common ground on at least one point. Our system of

I ran  test.py and it worked fine with whisper-small.tflite, maybe on raspberry pi tflite version is bit older one
$python test.py
Importing tensorflow and numpy
2023-01-19 12:00:17.805994: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
Importing whisper
Loading tflite model models/whisper.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Loading audio file: ../test-files/en_sh_lights_70pct_4s.wav
Samplerate: 16000, length: 3.575875s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
!!!
Inference took 9.42s for 3.58s audio file.
Loading audio file: ../test-files/en_speech_jfk_11s.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
!! And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

@j1nx pls downloasd the latest whisper-small.tflite and I will also run using minimal c++ example

@nyadla-sys Will do in a bit.
That test.py you linked to uses full tensorflow, however we use the tensorflow-lite.
https://github.com/fquirin/speech-recognition-experiments/blob/main/whisper-tflite/test.py#L9
Could you flip the # at line 8 and 9 and try again?

PS the guys at tensorflow took pity on me :)
https://github.com/tensorflow/tensorflow/issues/59273#issuecomment-1384441333

@nyadla-sys Ran with the latest small model
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-small.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 3435 (WHILE) failed to invoke.
Error at ../minimal.cc:211

Same error

Could you flip the # at line 8 and 9 and try again?

If you do that don't forget to change line 24 to interpreter = tf.Interpreter(model_path, num_threads=int(args.threads)) as well

PS the guys at tensorflow took pity on me :)

Did you notice any difference? I need to check what they've actually changed since they rewrote a lot without comments

Yeah, indeed. Grabbed the snippet here which has that also flipped;
https://github.com/ggerganov/whisper.cpp/issues/7#issuecomment-1384419135
Anyhow, could you or @nyadla-sys check out both? As we do not train, all we need is the tflite runtime for inference.

I just ran whisper-small.tflite on my linux ubuntu and please refer the latest README.md
$ ./minimal ../models/whisper-small.tflite ../samples/jfk.wav
n_vocab:50257
mel.n_len3000
mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 26 seconds
[_extra_token_50258][_extra_token_50259]!! And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.

[_extra_token_50258][_extra_token_50259] are tokens output stays the same

are tokens output stays the same

tokens output gets changed depending on the english/multilingual model

To use a multilingual model in Python, you can simply change the line "wtokenizer = whisper.tokenizer.get_tokenizer(False, language="en")" to "wtokenizer = whisper.tokenizer.get_tokenizer(True, language="en")"

Continuation from here
Does this require the whisper-small.tflite model? Because I've tried that with whisper.tflite (en only?) but the output is completely scrambled and still English when I set for example "de".

To use a multilingual model in Python, you can simply change the line "wtokenizer = whisper.tokenizer.get_tokenizer(False, language="en")" to "wtokenizer = whisper.tokenizer.get_tokenizer(True, language="en")"

Continuation from here
Does this require the whisper-small.tflite model? Because I've tried that with whisper.tflite (en only?) but the output is completely scrambled and still English when I set for example "de".

Please use whisper-small.tflite  or whisper-medium.tflite

These two whisper-small.tflite or whisper-medium.tflite models are multilingual, "whisper.tflite" is identical to "whisper-tiny.en.tflite", I intended to change the name but I refrained from doing so because many people are still using it.

Strangs as I run those test with the right vocab multilang bin.
will double check in the morning.

I've updated test.py with ne parameters for "--lang", "--runtime" and the tweaks mentioned in the tensorflow issue:
$ python3 test.py -h
usage: test.py [-h] [-f FOLDER] [-m MODEL] [-t THREADS] [-l LANG] [-r RUNTIME]

Running Whisper TFlite test inference.

optional arguments:
  -h, --help            show this help message and exit
  -f FOLDER, --folder FOLDER
                        Folder with WAV input files
  -m MODEL, --model MODEL
                        Path to model
  -t THREADS, --threads THREADS
                        Threads used
  -l LANG, --lang LANG  Language used
  -r RUNTIME, --runtime RUNTIME
                        Tensorflow runtime, use '1' for tf.lite or '2' for tflite_runtime

On my Rpi400 tflite_runtime is still about 1.5s slower than tf.lite.
I could not test whisper-small.tflite because it keeps crashing (will post error in a minute).

Change the name @nyadla-sys as we can change things but think we are all used the the original naming convention

@fquirin dunno why but the tensorflow rewrote the test script for me, maybe its because the full tf has some special sauce optimisation with cmake for Arm that maybe we are missing with the TensorFlow Lite Python Wheel Package build?

Change the name @nyadla-sys as we can change things but think we are all used the the original naming convention

I updated with whisper-tiny.en.tflite model and need to change README,for bacword compatibility i will keep whisper.tflite for sometime

Error with whisper-small.tflite (I think we had this somewhere a few hours ago already?):
Traceback (most recent call last):
  File "/home/pi/whisper-tflite/openai-whisper/test.py", line 93, in <module>
    transcribe(args.folder + file)
  File "/home/pi/whisper-tflite/openai-whisper/test.py", line 68, in transcribe
    interpreter.invoke()
  File "/home/pi/whisper-tflite/venv/lib/python3.9/site-packages/tensorflow/lite/python/interpreter.py", line 917, in invoke
    self._interpreter.Invoke()
RuntimeError: gather index out of boundsNode number 35 (GATHER) failed to invoke.Node number 3435 (WHILE) failed to invoke.

dunno why but the tensorflow rewrote the test script for me, maybe its because the full tf has some special sauce optimisation with cmake for Arm that maybe we are missing with the TensorFlow Lite Python Wheel Package build?

It must be something like that, yes 🤔

Error with whisper-small.tflite (I think we had this somewhere a few hours ago already?):
Traceback (most recent call last):
  File "/home/pi/whisper-tflite/openai-whisper/test.py", line 93, in <module>
    transcribe(args.folder + file)
  File "/home/pi/whisper-tflite/openai-whisper/test.py", line 68, in transcribe
    interpreter.invoke()
  File "/home/pi/whisper-tflite/venv/lib/python3.9/site-packages/tensorflow/lite/python/interpreter.py", line 917, in invoke
    self._interpreter.Invoke()
RuntimeError: gather index out of boundsNode number 35 (GATHER) failed to invoke.Node number 3435 (WHILE) failed to invoke.

dunno why but the tensorflow rewrote the test script for me, maybe its because the full tf has some special sauce optimisation with cmake for Arm that maybe we are missing with the TensorFlow Lite Python Wheel Package build?

It must be something like that, yes thinking

Can you provide me with additional information such as the operating system, machine, and whether you are using a minimal C++ build or a Python script?

Can you provide me with additional information such as the operating system, machine, and whether you are using a minimal C++ build or a Python script?

Sure: Aarch64, Raspberry Pi 400, 4GB RAM, Debian Bullseye (11), Python script.
Maybe my Pi is actually out of memory when using the small model but according to OpenAI 2GB should be fine 🤔

Run 'htop' in another shell and you see what is going on.

Some recent benchmark results with my Rpi400:
Whisper TFlite - tiny-en - tensorflow.lite - 4 threads:
-------------------------------------------------------

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.083s for 11.000s audio file.

Whisper TFlite - tiny-en - tflite_runtime - 4 threads:
------------------------------------------------------

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 5.668s for 11.000s audio file.

Runs at 780MB (used) for a while than quickly maxes out at 1.8GB (~50%) then crashes.
What I did notice during the test: tensorflow.lite seems to use the 4 cores more efficiently than tflite_runtime 🤔

Can you provide me with additional information such as the operating system, machine, and whether you are using a minimal C++ build or a Python script?

I run C++ minimal and python build on and it works fine
Linux pop-os 5.19.0-76051900-generic #202207312230~1663791054~22.04~28340d4 SMP PREEMPT_DYNAMIC Wed S x86_64 x86_64 x86_64 GNU/Linux

I run C++ minimal and python build on below and both works fine

I've tested it on my x86 Debian 11 laptop and it worked as well (Python test.py). So it seems to be a ARM or Raspberry Pi issue 🤔 .
Language selection still doesn't work though. The small model with "de" setting adds "!!!" to the beginning of a line, removes some words or entire texts but never gives any results in German.

What does; "-funsafe-math-optimizations" exactly?
Because all Tensorflow Lite documentation shows, it should be used and so did I.
https://www.tensorflow.org/lite/guide/build_cmake_arm
However looking at the build script used within the repo, I can't find it (anymore);
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/pip_package/build_pip_package_with_cmake.sh
It is also not present using Bazel;
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
However the bazel based buildscript does explicitely set "-O3" which is not done with cmake. (I also do not set it explicitely and compile OpenVoiceOS-buildroot with "-O2".
Perhaps the ~25% comes from there?

Just double checked if I had used the right vocab multilingual filter bin used, but indeed I did. It has something to do with the model and the gather function being different between the tflite_runtime lib and the tensorflow lite lib.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-small.tflite -t 4 -l en -r 2    Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-small.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Traceback (most recent call last):
  File "/home/mycroft/whisper/test.py", line 93, in <module>
    transcribe(args.folder + file)
  File "/home/mycroft/whisper/test.py", line 68, in transcribe
    interpreter.invoke()
  File "/usr/lib/python3.10/site-packages/tflite_runtime/interpreter.py", line 917, in invoke
    self._interpreter.Invoke()
RuntimeError: gather index out of boundsNode number 35 (GATHER) failed to invoke.Node number 3435 (WHILE) failed to invoke.

And with the C++ version
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-small.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 3435 (WHILE) failed to invoke.
Error at ../minimal.cc:211

https://www.tensorflow.org/api_docs/python/tf/gather

funsafe-math-optimizations

A quirk of Neon in Armv7 devices is that it flushes all subnormal numbers to zero, and as a result the GCC compiler will not use it unless -funsafe-math-optimizations, which allows losing denormals, is turned on. "Enhanced" Neon defined since Armv8 does not have this quirk, but as of GCC 8.2 the same flag is still required to enable Neon instructions.[133] On the other hand, GCC does consider Neon safe on AArch64 for Armv8.

I managed to generate encoder and decoder tflite models and just pending to complete the decoder post processing to generate tokens with text
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_encoder_decoder_tflite.ipynb

https://shop.allnetchina.cn/products/rock5-model-a-pre-order-discount-redeem-code?variant=44314435813692
@fquirin

I rebuild the tflite_runtime with GPU support.
Running the python based inference now takes longer, while it still says it is using XNNPACK
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.en.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.en.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 6.56s for 11.00s audio file.

BTW @nyadla-sys You are converting the small model with SELECT_OPS wich is not available for the tflite_intepreter. Perhaps that is the reason why we can't run it while you can.

@StuartIanNaylor Am in process of crosscompiling ComputeLibrary and ArmNN for rpi4 (armv8a). Interested to see if ArmNN outperforms XNNPACK. They claim it does, so interested to see...

I successfully executed the TFLite encoder and decoder models.
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_encoder_decoder_tflite.ipynb

this will open up multilingual support for whisper tflite models

@StuartIanNaylor Am in process of crosscompiling ComputeLibrary and ArmNN for rpi4 (armv8a). Interested to see if ArmNN outperforms XNNPACK. They claim it does, so interested to see...

I have had some problems with ArmNN in that it seems very platform dependent and you might find if you use Ubuntu 22.04 for Pi it might work whilst you could have problems otherwise but see how you go.
Its whatever version they use in the Wav2Letter example.
@nyadla-sys This is interesting as those who can run on GPU/CPU/NPU what benchmarks can be provided, I haven't looked or tried yet but will.

@j1nx Have you managed to build it?
I am still playing with a delegate build and have in minimal.cc
/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==============================================================================*/
#include "tensorflow/lite/core/interpreter.h"
#include "tensorflow/lite/kernels/register.h"
#include "tensorflow/lite/model.h"
#include "tensorflow/lite/optional_debug_tools.h"
#include "whisper.h"
#include "input_features.h"

// This is an example that is minimal to read a model
// from disk and perform inference. There is no data being loaded
// that is up to you to add as a user.
//
// NOTE: Do not add any dependencies to this that cannot be built with
// the minimal makefile. This example must remain trivial to build with
// the minimal build tool.
//
// Usage: minimal <tflite model>

#define TFLITE_MINIMAL_CHECK(x)                              \
  if (!(x)) {                                                \
    fprintf(stderr, "Error at %s:%d\n", __FILE__, __LINE__); \
    exit(1);                                                 \
  }

int main(int argc, char* argv[]) {
  if ((argc != 2) && (argc != 3)) {
    fprintf(stderr, "'minimal <tflite model>' or 'minimal <tflite model> <pcm_file name>'\n");
    return 1;
  }
  const char* filename = argv[1];
  whisper_filters filters;
  whisper_mel mel;
  struct timeval start_time,end_time;
  std::string word;
  int32_t n_vocab = 0;
  std::string fname = "./filters_vocab_gen.bin";
  auto fin = std::ifstream(fname, std::ios::binary);
  {
    uint32_t magic=0;
    fin.read((char *) &magic, sizeof(magic));
    //@magic:USEN
    if (magic != 0x5553454e) {
        printf("%s: invalid vocab file '%s' (bad magic)\n", __func__, fname.c_str());
        return 0;
    }
  }

  // load mel filters
  {
      fin.read((char *) &filters.n_mel, sizeof(filters.n_mel));
      fin.read((char *) &filters.n_fft, sizeof(filters.n_fft));

      filters.data.resize(filters.n_mel * filters.n_fft);
      fin.read((char *) filters.data.data(), filters.data.size() * sizeof(float));
  }

  // load vocab
  {
    fin.read((char *) &n_vocab, sizeof(n_vocab));
    g_vocab.n_vocab = n_vocab;
    printf("\nn_vocab:%d\n",(int)n_vocab);

    for (int i = 0; i < n_vocab; i++) {
      uint32_t len;
      fin.read((char *) &len, sizeof(len));

      word.resize(len);
      fin.read((char *) word.data(), len);
      g_vocab.id_to_token[i] = word;
      //printf("len:%d",(int)len);
      //printf("'%s'\n", g_vocab.id_to_token[i].c_str());
    }

    g_vocab.n_vocab = 51864;//add additional vocab ids
    if (g_vocab.is_multilingual()) {
        g_vocab.token_eot++;
        g_vocab.token_sot++;
        g_vocab.token_prev++;
        g_vocab.token_solm++;
        g_vocab.token_not++;
        g_vocab.token_beg++;
    }
    for (int i = n_vocab; i < g_vocab.n_vocab; i++) {
        if (i > g_vocab.token_beg) {
            word = "[_TT_" + std::to_string(i - g_vocab.token_beg) + "]";
        } else if (i == g_vocab.token_eot) {
            word = "[_EOT_]";
        } else if (i == g_vocab.token_sot) {
            word = "[_SOT_]";
        } else if (i == g_vocab.token_prev) {
            word = "[_PREV_]";
        } else if (i == g_vocab.token_not) {
            word = "[_NOT_]";
        } else if (i == g_vocab.token_beg) {
            word = "[_BEG_]";
        } else {
            word = "[_extra_token_" + std::to_string(i) + "]";
        }
        g_vocab.id_to_token[i] = word;
        // printf("%s: g_vocab[%d] = '%s'\n", __func__, i, word.c_str());
    }
  }

  //Generate input_features for Audio file
  if (argc == 3) {
    const char* pcmfilename = argv[2];
    // WAV input
    std::vector<float> pcmf32;
    {
      drwav wav;
      if (!drwav_init_file(&wav, pcmfilename, NULL)) {
          fprintf(stderr, "%s: failed to open WAV file '%s' - check your input\n", argv[0],pcmfilename);
        //  whisper_print_usage(argc, argv, {});
          return 3;
      }

      if (wav.channels != 1 && wav.channels != 2) {
          fprintf(stderr, "%s: WAV file '%s' must be mono or stereo\n", argv[0], pcmfilename);
          return 4;
      }

      if (wav.sampleRate != WHISPER_SAMPLE_RATE) {
          fprintf(stderr, "%s: WAV file '%s' must be 16 kHz\n", argv[0], pcmfilename);
          return 5;
      }

      if (wav.bitsPerSample != 16) {
          fprintf(stderr, "%s: WAV file '%s' must be 16-bit\n", argv[0], pcmfilename);
          return 6;
      }

      int n = wav.totalPCMFrameCount;

      std::vector<int16_t> pcm16;
      pcm16.resize(n*wav.channels);
      drwav_read_pcm_frames_s16(&wav, n, pcm16.data());
      drwav_uninit(&wav);
      // convert to mono, float
      pcmf32.resize(n);
      if (wav.channels == 1) {
          for (int i = 0; i < n; i++) {
              pcmf32[i] = float(pcm16[i])/32768.0f;
          }
      } else {
          for (int i = 0; i < n; i++) {
              pcmf32[i] = float(pcm16[2*i] + pcm16[2*i + 1])/65536.0f;
          }
      }
    }

    //Hack if the audio file size is less than 30ms append with 0's
    pcmf32.resize((WHISPER_SAMPLE_RATE*WHISPER_CHUNK_SIZE),0);
    if (!log_mel_spectrogram(pcmf32.data(), pcmf32.size(), WHISPER_SAMPLE_RATE, WHISPER_N_FFT, WHISPER_HOP_LENGTH, WHISPER_N_MEL, 1,filters, mel)) {
      fprintf(stderr, "%s: failed to compute mel spectrogram\n", __func__);
      return -1;
    }

    printf("\nmel.n_len%d\n",mel.n_len);
    printf("\nmel.n_mel:%d\n",mel.n_mel);
  }//end of audio file processing

  // Load tflite model
  std::unique_ptr<tflite::FlatBufferModel> model =
      tflite::FlatBufferModel::BuildFromFile(filename);
  TFLITE_MINIMAL_CHECK(model != nullptr);

  // Build the interpreter with the InterpreterBuilder.
  // Note: all Interpreters should be built with the InterpreterBuilder,
  // which allocates memory for the Interpreter and does various set up
  // tasks so that the Interpreter can read the provided model.

  tflite::ops::builtin::BuiltinOpResolver resolver;
  tflite::InterpreterBuilder builder(*model, resolver);
  std::unique_ptr<tflite::Interpreter> interpreter;
  builder(&interpreter);
  TFLITE_MINIMAL_CHECK(interpreter != nullptr);

  // Allocate tensor buffers.
  TFLITE_MINIMAL_CHECK(interpreter->SetNumThreads(4) == kTfLiteOk);
  TFLITE_MINIMAL_CHECK(interpreter->AllocateTensors() == kTfLiteOk);
  //printf("=== Pre-invoke Interpreter State ===\n");
  // tflite::PrintInterpreterState(interpreter.get());
  // Get information about the memory area to use for the model's input.
  float* input = interpreter->typed_input_tensor<float>(0);
  if (argc == 2) {
    memcpy(input, _content_input_features_bin, WHISPER_N_MEL*WHISPER_MEL_LEN*sizeof(float)); //to load pre generated input_features
  }
  else if (argc == 3) {
    memcpy(input, mel.data.data(), mel.n_mel*mel.n_len*sizeof(float));
  }
  // Fill input buffers
  // TODO(user): Insert code to fill input tensors.
  // Note: The buffer of the input tensor with index `i` of type T can
  // be accessed with `T* input = interpreter->typed_input_tensor<T>(i);`
  gettimeofday(&start_time, NULL);
  // Run inference
  TFLITE_MINIMAL_CHECK(interpreter->Invoke() == kTfLiteOk);
  gettimeofday(&end_time, NULL);
  if (end_time.tv_usec-start_time.tv_usec <= 0) {
	  printf("Inference time %ld.%ld seconds \n",(end_time.tv_sec-start_time.tv_sec-1),(end_time.tv_usec-start_time.tv_usec+1000000));
  }
  else if (end_time.tv_usec-start_time.tv_usec >= 0) {
	  printf("Inference time %ld.%ld seconds \n",(end_time.tv_sec-start_time.tv_sec),(end_time.tv_usec-start_time.tv_usec));
  }
  int output = interpreter->outputs()[0];
  TfLiteTensor *output_tensor = interpreter->tensor(output);
  TfLiteIntArray *output_dims = output_tensor->dims;
  // assume output dims to be something like (1, 1, ... ,size)
  auto output_size = output_dims->data[output_dims->size - 1];
  //printf("output size:%d\n",output_size );
  int *output_int = interpreter->typed_output_tensor<int>(0);
  std::string text = "";
  std::string word_add;
  for (int i = 0; i < output_size; i++) {
    //printf("%d\t",output_int[i]);
    if(output_int[i] == g_vocab.token_eot){
      break;
    }
    text += whisper_token_to_str(output_int[i]);
  }
  printf("\n%s\n", text.c_str());
  printf("\n");

  //printf("\n\n=== Post-invoke Interpreter State ===\n");
  ////  tflite::PrintInterpreterState(interpreter.get());
  // Read output buffers
  // TODO(user): Insert getting data out code.
  // Note: The buffer of the output tensor with index `i` of type T can
  // be accessed with `T* output = interpreter->typed_output_tensor<T>(i);`
  return 0;
}

If I build with  cmake --build ../tensorflow_src/tensorflow/lite/examples/minimal -DTFLITE_ENABLE_XNNPACK=OFF
orangepi@orangepi5:~/openai-whisper/minimal_build$ ./minimal ../models/whisper.tflite ../samples/test_1.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
Inference time 2.469990 seconds

[_SOT_][_NOT_] David lost his yellow pencil. He could not find it. Where is my yellow pencil? Yes his sister. His sister did not know. I don't know where your pencil is. She said David thought about it. He thought and thought. He used his yellow pencil before lunch. He used it to write a note to his teacher. The notes said, dear teacher, thank you for helping me, David. He put the note in the envelope where was the envelope?

Then build with cmake ../tensorflow_src/tensorflow/lite/examples/minimal
orangepi@orangepi5:~/openai-whisper/minimal_build$ ./minimal ../models/whisper.tflite ../samples/test_1.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 2.490417 seconds

[_SOT_][_NOT_] David lost his yellow pencil. He could not find it. Where is my yellow pencil? Yes his sister. His sister did not know. I don't know where your pencil is. She said David thought about it. He thought and thought. He used his yellow pencil before lunch. He used it to write a note to his teacher. The notes said, dear teacher, thank you for helping me, David. He put the note in the envelope where was the envelope?

Is that the speedup XNNPACK gives !?

an apple iOS phone is only taking 0.7 seconds to run inference with whisper-tiny.en.tflite

Prob the Amx blocks are the same sort of secret sauce that the M1 has
The Apple A13 Bionic features an Apple-designed 64-bit six-core CPU implementing [ARMv8.4-A](https://en.wikipedia.org/wiki/ARMv8.4-A)[[1]](https://en.wikipedia.org/wiki/Apple_A13#cite_note-a13microarch-1) [ISA](https://en.wikipedia.org/wiki/Instruction_set_architecture), with two high-performance cores running at 2.65 GHz[[6]](https://en.wikipedia.org/wiki/Apple_A13#cite_note-:1-6) called Lightning and four energy-efficient cores called Thunder. The Lightning cores feature [machine learning accelerators](https://en.wikipedia.org/wiki/AI_accelerator) called AMX blocks

Prob the Amx blocks are the same sort of secret sauce that the M1 has

The Apple A13 Bionic features an Apple-designed 64-bit six-core CPU implementing ARMv8.4-A[1] ISA, with two high-performance cores running at 2.65 GHz[6] called Lightning and four energy-efficient cores called Thunder. The Lightning cores feature machine learning accelerators called AMX blocks

https://medium.com/swlh/apples-m1-secret-coprocessor-6599492fc1e1 Apple sauce.

Jup, now I know for sure; I am a software guy.🤣

To be honest I didn't think AMX was a 'coprocessor' and it was just some Apple specific instructions they added to the Arm IP and that is why they work just like CPU instructions, because they are.

Finished the whole cross compile infrastructure and everything builded fine in the end.
https://github.com/OpenVoiceOS/ovos-buildroot/commit/876ee82daa2c4846afec1d4d505dcf872e2ad93c
Can't test before tonight / 5 hours from now, but will keep you posted if it works and how it performs.

All fun and all, but purely CPUacc from ArmNN is not bringing a performance gain.
The ArmNN Delegate is also rather picky with models and operators as the whisper tiny wasn't working. Had to grab a typical used demo model to compare it againsty XNNPACK.
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite --num_threads=4 --report_peak_memory_footprint=true --warmup_runs=1
STARTING!
Log parameter values verbosely: [0]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite]
#threads used for CPU inference: [4]
Loaded model models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite
The input model file size (MB): 3.57776
Initialized session in 3.112ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=11 first=92585 curr=62982 min=28710 max=92585 avg=46423.7 std=17777

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=50817 curr=27858 min=25482 max=63801 avg=39285.8 std=10589

Inference timings in us: Init: 3112, First inference: 92585, Warmup (avg): 46423.7, Inference (avg): 39285.8
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=2.57812 overall=6.17969
Overall peak memory footprint (MB) via periodic monitoring: 12.1367
Memory status at the end of exeution:
- VmRSS              : 12 MB
+ RssAnnon           : 4 MB
+ RssFile + RssShmem : 8 MB

mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite --warmup_runs=1 --external_delegate_path="/usr/lib/libarmnnDelegate.so" --external_delegate_options="backends:CpuAcc;logging-severity:info"
STARTING!
Log parameter values verbosely: [0]
Min warmup runs: [1]
Graph: [models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite]
External delegate path: [/usr/lib/libarmnnDelegate.so]
External delegate options: [backends:CpuAcc;logging-severity:info]
Loaded model models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite
Info: ArmNN v31.0.0
Can't load libOpenCL.so: libOpenCL.so: cannot open shared object file: No such file or directory
Can't load libGLES_mali.so: libGLES_mali.so: cannot open shared object file: No such file or directory
Can't load libmali.so: libmali.so: cannot open shared object file: No such file or directory
Couldn't find any OpenCL library.
Info: Initialization time: 0.83 ms.
INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
EXTERNAL delegate created.

Info: Optimize ArmnnSubgraph time: 6.77 ms
Info: Load ArmnnSubgraph time: 146.02 ms
Info: Overall ArmnnSubgraph creation time: 156.61 ms

Explicitly applied EXTERNAL delegate, and the model graph will be completely executed by the delegate.
The input model file size (MB): 3.57776
Initialized session in 286.218ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=9 first=112925 curr=46424 min=35712 max=112925 avg=59661.4 std=27126

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=50733 curr=47343 min=33201 max=67901 avg=46443.1 std=7687

Inference timings in us: Init: 286218, First inference: 112925, Warmup (avg): 59661.4, Inference (avg): 46443.1
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=59.8477 overall=68.1016
Info: Shutdown time: 3.38 ms.

All fun and all, but purely CPUacc from ArmNN is not bringing a performance gain.
The ArmNN Delegate is also rather picky with models and operators as the whisper tiny wasn't working. Had to grab a typical used demo model to compare it againsty XNNPACK.
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite --num_threads=4 --report_peak_memory_footprint=true --warmup_runs=1
STARTING!
Log parameter values verbosely: [0]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite]
#threads used for CPU inference: [4]
Loaded model models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite
The input model file size (MB): 3.57776
Initialized session in 3.112ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=11 first=92585 curr=62982 min=28710 max=92585 avg=46423.7 std=17777

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=50817 curr=27858 min=25482 max=63801 avg=39285.8 std=10589

Inference timings in us: Init: 3112, First inference: 92585, Warmup (avg): 46423.7, Inference (avg): 39285.8
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=2.57812 overall=6.17969
Overall peak memory footprint (MB) via periodic monitoring: 12.1367
Memory status at the end of exeution:
- VmRSS              : 12 MB
+ RssAnnon           : 4 MB
+ RssFile + RssShmem : 8 MB

mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite --warmup_runs=1 --external_delegate_path="/usr/lib/libarmnnDelegate.so" --external_delegate_options="backends:CpuAcc;logging-severity:info"
STARTING!
Log parameter values verbosely: [0]
Min warmup runs: [1]
Graph: [models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite]
External delegate path: [/usr/lib/libarmnnDelegate.so]
External delegate options: [backends:CpuAcc;logging-severity:info]
Loaded model models/mobilenet_v2_1.0_224_quantized_1_default_1.tflite
Info: ArmNN v31.0.0
Can't load libOpenCL.so: libOpenCL.so: cannot open shared object file: No such file or directory
Can't load libGLES_mali.so: libGLES_mali.so: cannot open shared object file: No such file or directory
Can't load libmali.so: libmali.so: cannot open shared object file: No such file or directory
Couldn't find any OpenCL library.
Info: Initialization time: 0.83 ms.
INFO: TfLiteArmnnDelegate: Created TfLite ArmNN delegate.
EXTERNAL delegate created.

Info: Optimize ArmnnSubgraph time: 6.77 ms
Info: Load ArmnnSubgraph time: 146.02 ms
Info: Overall ArmnnSubgraph creation time: 156.61 ms

Explicitly applied EXTERNAL delegate, and the model graph will be completely executed by the delegate.
The input model file size (MB): 3.57776
Initialized session in 286.218ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=9 first=112925 curr=46424 min=35712 max=112925 avg=59661.4 std=27126

Running benchmark for at least 50 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=50 first=50733 curr=47343 min=33201 max=67901 avg=46443.1 std=7687

Inference timings in us: Init: 286218, First inference: 112925, Warmup (avg): 59661.4, Inference (avg): 46443.1
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=59.8477 overall=68.1016
Info: Shutdown time: 3.38 ms.

To run whisper-tiny model using gpu delegate, we need to generate full float model, for testing purpose I can generate full float whisper tiny model

ArmNN is not using GPU but their special CPU instructions.
Have not yet looked into GPU support. That one is next. But by all means, if you could convert and upload GPU enabled tiny model, please do. We kind of created a nice playground overhere to test out all different aspects of whisper inference and I am sure we will end up with the best way forward for certain boards and hardware.

@j1nx appreciated your effort

@j1nx if possible could you please provide memory usage for whisper-tiny.en.tfliyte model on rpi4

@j1nx if possible could you please provide memory usage for whisper-tiny.en.tflite model on rpi4

Sure no problem. Using which tool you want it?
your minimal c++ binary
Python
Or the bench_mark tool from tensorflow?

I am looking for minimal c++ binary.
If you can do for rest it may be useful for others

Looks like benchmark tool would also provide required information

I found the binary delegate very picky on the distro it ran and it wasn't easy but did get it working on a Mali based board.
The GPU doesn't have to be Float as my test in https://github.com/StuartIanNaylor/rock5b-wav2letter-bench
Which is just the https://developer.arm.com/documentation/102603/2211 tutorial as it seemed to have quite a lot of typo's.
You can switch between GpuAcc & CpuAcc and the 8bit quantised model they provide works, but if GpuAcc works on non Mali I don't know as I tested on a MaliG610.
With the Wav2Letter they omit the GPU setup which makes me think its Mali only if if its base is OpenCL.

@j1nx uploaded whisper-base.tflite model
https://github.com/usefulsensors/openai-whisper/blob/main/models/whisper-base.tflite
along with the colab notbook

Ok, here some memory data for the tiny model. This is running on a Raspberry Pi4 - 2GB model.
Some info first;
mycroft@OpenVoiceOS-e3830c:~/whisper $ uname -a
Linux OpenVoiceOS-e3830c 5.15.76-v8 #1 SMP Fri Dec 30 14:58:43 CET 2022 aarch64 GNU/Linux

mycroft@OpenVoiceOS-e3830c:~/whisper $ lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A72
    Model:               3
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r0p3
    CPU max MHz:         1800.0000
    CPU min MHz:         600.0000
    BogoMIPS:            108.00
    Flags:               fp asimd evtstrm crc32 cpuid
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   192 KiB (4 instances)
  L2:                    1 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

mycroft@OpenVoiceOS-e3830c:~/whisper $ sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 20.906206 seconds, 49.0MB/s
mycroft@OpenVoiceOS-e3830c:~/whisper $ sudo sh -c "sync && echo 3 > /proc/sys/vm/drop_caches"
mycroft@OpenVoiceOS-e3830c:~/whisper $ dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 4.949115 seconds, 206.9MB/s

mycroft@OpenVoiceOS-e3830c:~/whisper $ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G      182.8M      561.6M        7.1M        1.0G        1.5G
Swap:        359.5M           0      359.5M

mycroft@OpenVoiceOS-e3830c:~/whisper $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    1  29.9G  0 disk
├─sda1   8:1    1    64M  0 part /boot
├─sda2   8:2    1   1.1G  0 part /media/rfs/ro
└─sda3   8:3    1  28.7G  0 part /media/rfs/rw
zram0  254:0    0 359.5M  0 disk [SWAP]

Running the minimal C++ binary on the whisper-tiny.en.tflite while watching the memory consumption (don't know about an easy way to record) it shows 185 MB added use at max while running it;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-tiny.en.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 4 seconds

[_SOT_][_NOT_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Doing the same but runing it with the test.py showed 305 MB added use at max.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.en.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.en.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.45s for 11.00s audio file.

And finally, using the benchmark tool, which shows the memory consumption it self. (watching the mem usage showed 176 MB used)
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-tiny.en.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-tiny.en.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-tiny.en.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 40.9627
Initialized session in 15.03ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=3044595

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=2903102 curr=2876513 min=2870115 max=2903102 avg=2.8807e+06 std=12080

Inference timings in us: Init: 15030, First inference: 3044595, Warmup (avg): 3.0446e+06, Inference (avg): 2.8807e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=9 overall=251.41
Overall peak memory footprint (MB) via periodic monitoring: 257.414
Memory status at the end of exeution:
- VmRSS              : 222 MB
+ RssAnnon           : 176 MB
+ RssFile + RssShmem : 46 MB

Will see what we can do with the base model now.

The base model suffers from the same gather index out of bounds issue;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-base.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 1264 (WHILE) failed to invoke.
Error at ../minimal.cc:210

Ok, here some memory data for the tiny model. This is running on a Raspberry Pi4 - 2GB model.
Some info first;
mycroft@OpenVoiceOS-e3830c:~/whisper $ uname -a
Linux OpenVoiceOS-e3830c 5.15.76-v8 #1 SMP Fri Dec 30 14:58:43 CET 2022 aarch64 GNU/Linux

mycroft@OpenVoiceOS-e3830c:~/whisper $ lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A72
    Model:               3
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r0p3
    CPU max MHz:         1800.0000
    CPU min MHz:         600.0000
    BogoMIPS:            108.00
    Flags:               fp asimd evtstrm crc32 cpuid
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   192 KiB (4 instances)
  L2:                    1 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

mycroft@OpenVoiceOS-e3830c:~/whisper $ sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 20.906206 seconds, 49.0MB/s
mycroft@OpenVoiceOS-e3830c:~/whisper $ sudo sh -c "sync && echo 3 > /proc/sys/vm/drop_caches"
mycroft@OpenVoiceOS-e3830c:~/whisper $ dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 4.949115 seconds, 206.9MB/s

mycroft@OpenVoiceOS-e3830c:~/whisper $ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G      182.8M      561.6M        7.1M        1.0G        1.5G
Swap:        359.5M           0      359.5M

mycroft@OpenVoiceOS-e3830c:~/whisper $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    1  29.9G  0 disk
├─sda1   8:1    1    64M  0 part /boot
├─sda2   8:2    1   1.1G  0 part /media/rfs/ro
└─sda3   8:3    1  28.7G  0 part /media/rfs/rw
zram0  254:0    0 359.5M  0 disk [SWAP]

Running the minimal C++ binary on the whisper-tiny.en.tflite while watching the memory consumption (don't know about an easy way to record) it shows 185 MB added use at max while running it;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-tiny.en.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 4 seconds

[_SOT_][_NOT_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Doing the same but runing it with the test.py showed 305 MB added use at max.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.en.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.en.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.45s for 11.00s audio file.

And finally, using the benchmark tool, which shows the memory consumption it self. (watching the mem usage showed 176 MB used)
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-tiny.en.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-tiny.en.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-tiny.en.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 40.9627
Initialized session in 15.03ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=3044595

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=2903102 curr=2876513 min=2870115 max=2903102 avg=2.8807e+06 std=12080

Inference timings in us: Init: 15030, First inference: 3044595, Warmup (avg): 3.0446e+06, Inference (avg): 2.8807e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=9 overall=251.41
Overall peak memory footprint (MB) via periodic monitoring: 257.414
Memory status at the end of exeution:
- VmRSS              : 222 MB
+ RssAnnon           : 176 MB
+ RssFile + RssShmem : 46 MB

Will see what we can do with the base model now.

@j1nx Great and thanks for your time on this ..

The base model suffers from the same gather index out of bounds issue;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-base.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 1264 (WHILE) failed to invoke.
Error at ../minimal.cc:210

Can you try with the latest base model that i uploaded yesterday?

The base model suffers from the same gather index out of bounds issue;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-base.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 1264 (WHILE) failed to invoke.
Error at ../minimal.cc:210

Can you try with the latest base model that i uploaded yesterday?

Downloaded the base model this morning, so is using that latest.

Ok, here some memory data for the tiny model. This is running on a Raspberry Pi4 - 2GB model.
Some info first;
mycroft@OpenVoiceOS-e3830c:~/whisper $ uname -a
Linux OpenVoiceOS-e3830c 5.15.76-v8 #1 SMP Fri Dec 30 14:58:43 CET 2022 aarch64 GNU/Linux

mycroft@OpenVoiceOS-e3830c:~/whisper $ lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A72
    Model:               3
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -
    Cluster(s):          1
    Stepping:            r0p3
    CPU max MHz:         1800.0000
    CPU min MHz:         600.0000
    BogoMIPS:            108.00
    Flags:               fp asimd evtstrm crc32 cpuid
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   192 KiB (4 instances)
  L2:                    1 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Vulnerable
  Spectre v1:            Mitigation; __user pointer sanitization
  Spectre v2:            Vulnerable
  Srbds:                 Not affected
  Tsx async abort:       Not affected

mycroft@OpenVoiceOS-e3830c:~/whisper $ sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 20.906206 seconds, 49.0MB/s
mycroft@OpenVoiceOS-e3830c:~/whisper $ sudo sh -c "sync && echo 3 > /proc/sys/vm/drop_caches"
mycroft@OpenVoiceOS-e3830c:~/whisper $ dd if=tempfile of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 4.949115 seconds, 206.9MB/s

mycroft@OpenVoiceOS-e3830c:~/whisper $ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.8G      182.8M      561.6M        7.1M        1.0G        1.5G
Swap:        359.5M           0      359.5M

mycroft@OpenVoiceOS-e3830c:~/whisper $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    1  29.9G  0 disk
├─sda1   8:1    1    64M  0 part /boot
├─sda2   8:2    1   1.1G  0 part /media/rfs/ro
└─sda3   8:3    1  28.7G  0 part /media/rfs/rw
zram0  254:0    0 359.5M  0 disk [SWAP]

Running the minimal C++ binary on the whisper-tiny.en.tflite while watching the memory consumption (don't know about an easy way to record) it shows 185 MB added use at max while running it;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal  models/whisper-tiny.en.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 4 seconds

[_SOT_][_NOT_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Doing the same but runing it with the test.py showed 305 MB added use at max.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.en.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.en.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 4.45s for 11.00s audio file.

And finally, using the benchmark tool, which shows the memory consumption it self. (watching the mem usage showed 176 MB used)
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-tiny.en.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-tiny.en.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-tiny.en.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 40.9627
Initialized session in 15.03ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=3044595

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=2903102 curr=2876513 min=2870115 max=2903102 avg=2.8807e+06 std=12080

Inference timings in us: Init: 15030, First inference: 3044595, Warmup (avg): 3.0446e+06, Inference (avg): 2.8807e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=9 overall=251.41
Overall peak memory footprint (MB) via periodic monitoring: 257.414
Memory status at the end of exeution:
- VmRSS              : 222 MB
+ RssAnnon           : 176 MB
+ RssFile + RssShmem : 46 MB

Will see what we can do with the base model now.

@j1nx Great and thanks for your time on this ..

No problem. Hopefully we can get the base and small models to work as well. Tiny is fast enough but perhaps base as well.

VmRSS              : 222 MB

RssAnnon           : 176 MB
RssFile + RssShmem : 46 MB

Just for others who want to understand  about VmRSS,RssAnnon, RssFile..
These values represent the amount of memory used by a process, specifically the Resident Set Size (RSS) of the process.
VmRSS is the total amount of resident memory used by the process, measured in megabytes (MB).
RssAnnon is the amount of anonymous memory used by the process, measured in megabytes (MB). Anonymous memory is memory that is not associated with a file on disk.
RssFile and RssShmem are the amount of file-backed and shared memory used by the process, respectively, measured in megabytes (MB). File-backed memory is memory that is associated with a file on disk, while shared memory is memory that is shared between multiple processes.
In this case, the process is using a total of 222 MB of memory, with 176 MB of that being anonymous memory and 46 MB being file-backed and shared memory.

Very nice explaination. Thanks for that.

@fquirin create the runtime with bazel
grab the latest from https://github.com/bazelbuild/bazelisk likely wget to /usr/bin, chmod a+x, create symlink called bazel
export USE_BAZEL_VERSION=5.3.0
Find the version in the tensorflow_src .bazelversion file
/home/orangepi/tensorflow_src/tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh native
cmake tflite_runtime
test.wav
 Bili always listens to his mother. He always does what she says. If his mother says,

Inference took 1.54s
test_1.wav
 David lost his yellow pencil. He could not find it. Where is my yellow pencil? He asked his sister. His sister did not know. I don't know where your pencil is. She said David thought about it. He thought and thought. He used his yellow pencil before lunch. He used it to write a note to his teacher. The notes said, dear teacher, thank you for helping me, David. He put the note in the envelope where was the envelope?

Inference took 2.54s
jfk.wav
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 1.51s

bazel tflite_runtime
test.wav
 Bili always listens to his mother. He always does what she says. If his mother says,

Inference took 1.08s
test_1.wav
 David lost his yellow pencil. He could not find it. Where is my yellow pencil? He asked his sister. His sister did not know. I don't know where your pencil is. She said David thought about it. He thought and thought. He used his yellow pencil for before lunch. He used it to write a note to his teacher. The notes said, dear teacher, thank you for helping me, David. He put the note in the envelope where was the envelope?

Inference took 2.07s
jfk.wav
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 1.07s

I noticed you uploaded new tiny models as well. Hereby the model benchmark tool output on them.
The normal one;
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-tiny.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-tiny.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-tiny.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 69.3703
Initialized session in 162.167ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=4303281

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=3882868 curr=3828750 min=3828750 max=3882868 avg=3.85199e+06 std=17539

Inference timings in us: Init: 162167, First inference: 4303281, Warmup (avg): 4.30328e+06, Inference (avg): 3.85199e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=10.7656 overall=306.816
Overall peak memory footprint (MB) via periodic monitoring: 312.531
Memory status at the end of exeution:
- VmRSS              : 244 MB
+ RssAnnon           : 171 MB
+ RssFile + RssShmem : 73 MB

The EN one;
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-tiny.en.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-tiny.en.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-tiny.en.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 41.6556
Initialized session in 119.189ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=3247494

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=2934383 curr=2893835 min=2892774 max=2934383 avg=2.90437e+06 std=15345

Inference timings in us: Init: 119189, First inference: 3247494, Warmup (avg): 3.24749e+06, Inference (avg): 2.90437e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=10.7422 overall=264.406
Overall peak memory footprint (MB) via periodic monitoring: 268.031
Memory status at the end of exeution:
- VmRSS              : 227 MB
+ RssAnnon           : 181 MB
+ RssFile + RssShmem : 46 MB

The base model still errors out.
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-base.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-base.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-base.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 77.0875
Initialized session in 182.557ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
ERROR: gather index out of bounds
ERROR: Node number 35 (GATHER) failed to invoke.
ERROR: Node number 1264 (WHILE) failed to invoke.
count=1 curr=6372530

Benchmarking failed.

But the funny thing, the small model does work?!
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-small.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true
STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-small.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-small.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 387.698
Initialized session in 152.364ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=21885361

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=20743640 curr=20800234 min=20726743 max=20882339 avg=2.07767e+07 std=59027

Inference timings in us: Init: 152364, First inference: 21885361, Warmup (avg): 2.18854e+07, Inference (avg): 2.07767e+07
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=33.9414 overall=1207.91
Overall peak memory footprint (MB) via periodic monitoring: 1211.55
Memory status at the end of exeution:
- VmRSS              : 752 MB
+ RssAnnon           : 376 MB
+ RssFile + RssShmem : 376 MB

Look at that AMAZING low memory usage for the small model !!!!
@nyadla-sys Well done / Great work.

@j1nx I just uploaded new whisper-base.tflite model.
could you please try this model on rpi4 and let me know if you still see  crash issue

@j1nx I just uploaded new whisper-base.tflite model. could you please try this model on rpi4 and let me know if you still see crash issue

All good now;
mycroft@OpenVoiceOS-e3830c:~/whisper $ ./benchmark_model --graph=models/whisper-base.tflite --num_threads=4 --warmup_runs=1 --num_runs=5 --report_peak_memory_footprint=true                                                                 STARTING!
Log parameter values verbosely: [0]
Min num runs: [5]
Num threads: [4]
Min warmup runs: [1]
Report the peak memory footprint: [1]
Graph: [models/whisper-base.tflite]
#threads used for CPU inference: [4]
Loaded model models/whisper-base.tflite
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
The input model file size (MB): 125.575
Initialized session in 54.729ms.
Running benchmark for at least 1 iterations and at least 0.5 seconds but terminate if exceeding 150 seconds.
count=1 curr=10344233

Running benchmark for at least 5 iterations and at least 1 seconds but terminate if exceeding 150 seconds.
count=5 first=9610296 curr=9466280 min=9466280 max=9654249 avg=9.57554e+06 std=64674

Inference timings in us: Init: 54729, First inference: 10344233, Warmup (avg): 1.03442e+07, Inference (avg): 9.57554e+06
Note: as the benchmark tool itself affects memory footprint, the following is only APPROXIMATE to the actual memory footprint of the model at runtime. Take the information at your discretion.
Memory footprint delta from the start of the tool (MB): init=15.875 overall=497.102
Overall peak memory footprint (MB) via periodic monitoring: 502.785
Memory status at the end of exeution:
- VmRSS              : 333 MB
+ RssAnnon           : 207 MB
+ RssFile + RssShmem : 126 MB

However just as with the small model, the WER is terrible.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-base.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-base.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
th Wiv array Year, high Lend thisyear or D Mic, highendic or D M thisyear.

Inference took 11.85s for 11.00s audio file.

I suspect this has something to do with the language selection of some kind.
Is it possible that you convert and upload the small and base model ".en" models as well?

Just confirmed it;
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
th Wiv array Year high Lend thisyear or D Mic highendic or D M thisyear

Inference took 5.92s for 11.00s audio file.
mycroft@OpenVoiceOS-e3830c:~/whisper $ python3 test.py -f samples/ -m models/whisper-tiny.en.tflite -t 4 -l en -r 2
Importing tflite_runtime
Importing numpy
Importing whisper
Loading tflite model models/whisper-tiny.en.tflite ...
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Loading audio file: samples/jfk.wav
Samplerate: 16000, length: 11.0s
Calculating mel spectrogram...
Invoking interpreter ...
Preparing output data ...
Converting tokens ...
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 5.13s for 11.00s audio file.

This is interesting, plotted peak memoru usage against inference time in second

 python test.py --model=../openai-whisper/models/whisper-tiny.tflite --folder=../openai-whisper/samples/ --threads=4

jfk.wav
th Wiv array Year high Lend thisyear or D Mic, highendic or D M thisyear.

Inference took 1.25s

python test.py --model=../openai-whisper/models/whisper-tiny.en.tflite --folder=../openai-whisper/samples/ --threads=4

jfk.wav
 And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

Inference took 1.13s

Slower by 10% ?

Indeed, but the WER of the non EN fixed model is... Well, 5% so I guess something goed wrong there.
will have a look at the C++ binary in the morning to see how that one is performing on all models.

I presume the initial decoder tokens are hardcoded for a translation model / language or something or just missing, but the start tokens are control tokens
decoder_input_ids = torch.tensor([50258, 50266, 50358, 50363]) #<|startoftranscript|><|ja|><|translate|><|notimestamps|>

decoder_input_ids = torch.tensor([50258, 50259, 50359, 50363]) #<|startoftranscript|><|en|><|transcribe|><|notimestamps|>

Here the output of the C++ binary with the single vocab.bin;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 6 seconds

[_extra_token_50258][_extra_token_50259][_extra_token_50359][_BEG_]th Wiv array Year high Lend thisyear or D Mic, highendic or D M thisyear.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.en.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 5 seconds

[_SOT_][_NOT_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.

And the same multilingual model but this time with the multilingual vocab.bin;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 6 seconds

[_extra_token_50258][_extra_token_50259][_extra_token_50359][_BEG_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.[_SOT_]

And to complete the whole test package;
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 6 seconds

[_extra_token_50258][_extra_token_50259][_extra_token_50359][_BEG_] And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-base.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 12 seconds

[_extra_token_50258][_extra_token_50259][_extra_token_50359][_BEG_] And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-small.tflite samples/jfk.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 37 seconds

[_extra_token_50258][_extra_token_50259][_extra_token_50359][_BEG_] And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.[_SOT_]

Next up is some other different WAV files running the Python inference utilizing the tiny.en model to get some insight on the WER.

This also might be of interest to you @nyadla-sys The base model do translation to english where as the tiny and small models just returned the language detected.
mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 7 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50359][_BEG_] Für mich sind alle Menschen gleich unabhängig von Geschlecht, sexuelle Orientierung, Religion, Hautfarbe oder Geo-Kordinaten der Geburt.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-base.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 12 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50358][_BEG_] For me, all people are equally independent of gender, sex, orientation, religion, hate, or gender coordinates of birth.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-small.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 43 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50359][_BEG_] Für mich sind alle Menschen gleich, unabhängig von Geschlecht, sexueller Orientierung, Religion, Hautfarbe oder Geo-Koordinaten der Geburt.[_SOT_]

This also might be of interest to you @nyadla-sys The base model do translation to english where as the tiny and small models just returned the language detected.

thanks for your information and I will look into it

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-tiny.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 7 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50359][_BEG_] Für mich sind alle Menschen gleich unabhängig von Geschlecht, sexuelle Orientierung, Religion, Hautfarbe oder Geo-Kordinaten der Geburt.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-base.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 12 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50358][_BEG_] For me, all people are equally independent of gender, sex, orientation, religion, hate, or gender coordinates of birth.[_SOT_]

mycroft@OpenVoiceOS-e3830c:~/whisper $ minimal models/whisper-small.tflite de_speech_thorsten_sample03_8s.wav

n_vocab:50257

mel.n_len3000

mel.n_mel:80
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Inference time 43 seconds

[_extra_token_50258][_extra_token_50261][_extra_token_50359][_BEG_] Für mich sind alle Menschen gleich, unabhängig von Geschlecht, sexueller Orientierung, Religion, Hautfarbe oder Geo-Koordinaten der Geburt.[_SOT_]

@StuartIanNaylor it looks like the tflite_runtime issue was resolved? Did you understand what the problem was? Should we use this build script as default? 🙂

Yep the bazel native build on the rk3588 is almost 50% faster also on Pi if compiling on metal you should select native, I haven't tried but looks like the others are for cross compiling.
Tensorflow does this thing where it names new features experimental but many of them can be years old and the current defacto way. So I don't know if its just optimisation as been scratching my head with minimal_buiild as that could be the same, but boy the api is complex and not always clear what is the latest and greatest, but I think some of the so called 'experimental' methods also end up in the build with bazel.
I did send you a message to use bazel with a @fquirin that I have forgot where but should be in your 'mentions'

Hi @StuartIanNaylor , do you have a tflite_runtime wheel file built with the bazel script for download maybe? My Pi seems to be to weak to build it and I don't understand how the cross-compile is supposed to work because the examples are broken 😕  (the referenced Docker files don't exist).
I'd require the Python 3.9 aarch64 build 🙃

Good as my native compile on Armv8,2 prob would not work on a Pi.
I am not really sure what the bazel build does that is different but yeah it is notibeabilly faster than the cmake compile that maybe hasn't been updated and creates a legacy build.
I am thinking its the same for the minimal_build and maybe linking to bazel built lib than a static lib might also be faster due to whatever bazel does in the build as opposed to running cmake.
Shouldn't need docker though even though you got it working as you just specify https://www.tensorflow.org/lite/guide/build_cmake_pip#available_target_names
armhf or aarch64 and it will go off and cross-compile, whilst otherwise for the host even if one of those native

Good as my native compile on Armv8,2 prob would not work on a Pi

Actually could you try my build: download (if you have the same Python version). I'd like to know if they are compatible and perform the same :-).

Ran in a miniconda env as on Ubuntu with Python 3.10, but yeah runs OK.
conda create -n tlite-test python=3.9
Totally forgot what we where at, but yeah it runs.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
gb1.wav
 My fellow Americans, this day has brought terrible news and great sadness to our country. At 9 o clock this morning, mission control in Houston lost contact with our space shuttle Columbia. A short time later, debris was seen falling from the skies above Texas. The Columbia's lost. There are no survivors.

Inference took 2.1s
mm0.wav
 This is the micro machine man presenting the most miniature motorcade of micro machine. Each one has dramatic details for a picture of precision paint jobs. Plus incredible micro machine pocket place that's physical police station fire station restaurant service station and more. Perfect pocket portable to take any place. And there are many miniature places to play with. Each one comes with its own special edition micro machine vehicle and fantastic features that miraculously move. Raise the bolt. Lift at the airport. Marine a man. The gun turret at the army base. Clean your car at the car. Raise the toll bridge. And these place that's fitted together to form a micro machine world. Micro machine pocket place that's so tremendously tiny so perfectly precise. So doesn't we detailed you on a pocket them all micro machines and micro machine pocket place that sold separately from Glube. The smaller they are, the better they are.

Inference took 3.72s
hp0.wav
 Henry F. Phillips from Wikipedia, the free encyclopedia at en.wicopedia.org.

Inference took 1.5s
gb0.wav
 Good morning. This Tuesday is Election Day. After months of spirited debate in vigorous campaigning, the time has come for Americans to make important decisions about our nation's future and encourage all Americans to go to the polls and vote. Election season brings out the spirit of competition between our political parties. And that competition is an essential part of a healthy democracy. But as the campaigns come to a close, Republicans, Democrats, and independents can find common ground on at least one point. Our system of

Inference took 2.52s

Nice 😎 , thanks for testing 👍


UPDATED CODE FOR DARK THEME

sir as per your desired look of website is here

closes https://github.com/07sumit1002/CabRental/issues/147


Fix for application multi startup issue and subscription file path

Coverage decreased (-0.2%) to 92.093% when pulling 9f08f90e0bbb2e2857c96f21278690c1f05fa989 on manali-deshmukh:vcf-issues into 2efffe64bf3561cfdc273b16ecbfc03e7b846ac5 on HewlettPackard:master.


ASTMangler: Fix mangling of SILFunctionType with @pack conventions

Fixes rdar://problem/107151125.

@swift-ci Please smoke test

@swift-ci Please smoke test

@swift-ci Please smoke test macOS

@swift-ci Please smoke test macOS

@adrian-prantl The test already passed and merged in the 5.9 branch. The main branch PR was held up by an unrelated CI issue.


Looks like private messages aren't handled properly

Hooks don't get invoked for them.

This bug is still alive. Do you have any idea about its origin?


fix Fix the sql of AddPartitionRecord

Cherry-pick #37295
The range field is accidentally compared to DUMMY_ITEM.
It was introduced by #35461.

run buildall


able to define shared cache db across multiple threads

Hi,
I plan to use revm in a multi-threaded environment and want to share a cache database across multiple threads.
I have an implementation like this

and I define the db as :
let mut ethers_db = EthersDB::new(Arc::clone(&client), Some(block_id)).expect("error in create ether db");
let mut cache_db = CacheDB::new(ethers_db.clone());
If I clone the cache_db in each thread, I think caching mechanism will not work efficiently. also, using Arc and Mutex for whole evm.transact().unwrap(); will not work for me, because it will block all other threads and nearly all of my processing is about transacting.
Is there any clever way to handle this? I'm aware that internal hashMap of the cache db (hashBrown) is not designed for parallel lock-free access and I think changing it to something else would have a lot of work.
thanks

You would need to implement your own database that has locks. This is currently out of scope of revm.
You can look at foundry-fork-db that implements revm database. https://github.com/foundry-rs/foundry-fork-db


[WIP] Flutter Desktop (don't merge)

Adds support for Flutter Desktop

[x] macOS
[ ] Windows
[ ] Linux

Linked issue #37
macOS version test
export WORK_PATH=`pwd`

git clone https://github.com/cloudwebrtc/flutter-webrtc.git
git clone https://github.com/cloudwebrtc/flutter-webrtc-demo.git

cd $WORK_PATH/flutter-webrtc
git checkout desktop

cd $WORK_PATH/flutter-webrtc-demo
git checkout desktop

flutter run -d macos
#flutter build macos --release

Separate into
https://github.com/cloudwebrtc/flutter-webrtc/pull/138
https://github.com/cloudwebrtc/flutter-webrtc/pull/139
So close now.


Fix and cleanup retained map test

Thank you. It looks good to me.
I commented about some of minor coding style.


Add PKGBUILD for Archlinux from JKA Network

We created a PKGBUILD for your program to install it on Archlinux.

Thanks for the pull request, appreciate it.
I don't think this makes sense to keep in the main repository however. It seems a bit 'wrong' to keep a build script specific to a particular platform (Arch in this case) in the git repo itself. This already exists on the AUR and as far as I'm aware this would only serve to host it in another place effectively.
Let me know if you disagree or I've overlooked something, thanks!

No problem. The PKGBUILD is now in AUR.
If you change the repository tree or make a important change in the program please report it us to update the PKGBUILD in AUR and maintain the package updated.
Thanks


Config/types for windows

Reviewer:
Asana:
Description
Steps to test

Current dependencies on/for this PR:

main

PR #184   👈

This comment was auto-generated by Graphite.


GPIO connecting issue

OK, so I'm successfully deployed motion detector on GPIO PIN 3, I connected 1 line to PIN 3, but where I had to plug second?? GND? 5V? 3.3V? Pls help me!

Hi Alex,
Be carefull in the next release 0.3.2 input sensor are now pulled-up. So you will need to change the wiring to plug it to GND instead of 3.3V.
I made this modification to avoid errors as pull-up logic is more common.


ImproperlyConfigured: The Redis backend requires redis-py to be installed.

Describe the problem
The following exception is raised:
ImproperlyConfigured: The Redis backend requires redis-py to be installed.
Steps to reproduce

Run pip install "django-constance[database]"
Add constance to settings.INSTALLED_APPS:

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.staticfiles',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    ...
    'constance',
)

Navigate to /admin/constance/config/

System configuration

Django version: 3.0.3
Python version: 3.7.3

Fixed it adding:

constance.backends.database to INSTALLED_APPS
CONSTANCE_BACKEND = 'constance.backends.database.DatabaseBackend'

It was not very clear in the installation instructions, I think that installing the 'database' version this config should be implicit, update to docs please.


[Discuss] Raising visibility of golden changes as breaking changes

Description
According to the Changelog, it should include

changes to the layout or rendering which might affect golden file testing

but has not been reflected very reliably.
This small change adds mention in pull request templates to raise visibility of golden file changes being breaking changes that affect our developers. :)
Related Issues
Addresses #36667
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I signed the CLA.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require Flutter developers to manually update their apps to accommodate your change?

[x] No, this is not a breaking change.

Closing in favor of a bot that will label PRs with golden changes.


Tabs opened from pinned tabs positioned incorrectly

STR:

Pin a tab containing a link that opens in a new window
Pin another tab (so the first pinned tab has pinned tabs after it)
Open another tab (not pinned)
Click the link in the first tab

The new tab opens just to the right of the parent tab creating a situation where there are mixed pinned and non-pinned tabs
Bonus STR:

Double click the newly opened un-pinned tab (the one in the middle of the pinned tabs) which should pin it. It will pin itself, but it will jump into the un-pinned tabs as it pins itself.

What a great feature.


Fetching current song automatically

Another step towards fully autonomous cheating. This would be a very convenient feature to have.

Interestingly, I couldn't find the song in memory, except for calls to interface with Discord. Fetching it from the window title could be an option, but it's not clear how portable that is.

This has been implemented for Windows with commit 825784a10eddfec6d5498a421b21b93eac4f1766.


Abnormally high CPU usage compared to RAM

Describe the bug
Running a production server on AWS using a m6g.xlarge instance. CPU load average between 3-8 (depending on traffic) whilst memory usage is only hovers around 1GB (out of 16GB). Dataset about 1.5GB. Indexing works fine, this is during "regular" usage.
To Reproduce
This is obviously hard to reproduce, but I'm more looking for some feedback if this is normal. Some notes on how we use Meilisearch:

We only have a few thousand documents but each document contains a lot of data. We run a multi-language sites which means we index every translation. A document can for example have categories attached to them, this means we index each category with all its translations so that it's searchable.
We use facetDistributions on many places
We have 14 filterable attributes
We have 8 searchable attributes including a description (which can be up to 2000 characters).
The VPS is running ARM
Doing some testing locally it doesn't look like the searches themselves are slow. Some sample search times (in seconds):

Expected behavior
According to the docs and my own logic we should see a lot more RAM usage and less CPU usage during regular operations. Screenshots taken below is when the server is not indexing. We have a lot of requests mind you, but the resource usage looks abnormal to me.
Screenshots

(Some of this structure was built before nested attributes existed)
Meilisearch version:
0.27.1

Hello @antonhedling, a huge CPU usage usually happens because meilisearch is indexing (that’s a good sign, it means meilisearch is able to use 100% of your resource).
And for your RAM usage, the ram consumption is actually quite big.

As you can see in the help of htop, the yellow bar is related to the RAM cache usage.
But this is not unused RAM, especially for meilisearch, which is memory mapped.
A memory-mapped program relies on the cache a lot and lets the kernel decide when we can free or not the memory page.
So what you see is actually quite ordinary. The CPU usage is high, as well as the RAM usage.

Hello @antonhedling, a huge CPU usage usually happens because meilisearch is indexing (that’s a good sign, it means meilisearch is able to use 100% of your resource). And for your RAM usage, the ram consumption is actually quite big.  As you can see in the help of htop, the yellow bar is related to the RAM cache usage. But this is not unused RAM, especially for meilisearch, which is memory mapped. A memory-mapped program relies on the cache a lot and lets the kernel decide when we can free or not the memory page.
So what you see is actually quite ordinary. The CPU usage is high, as well as the RAM usage.

Thank you. I guess I missed that part in the documentation, and now I've also learned a bit more about memory usage. I'll simply upgrade the server. Closing this issue.