Update README.md

README.md

@@ -32,10 +32,18 @@ and [Multilingual LibriSpeech](https://huggingface.co/datasets/japanese-asr/en_a
 Kotoba-whisper-bilingual's loss objective consists of cross-entropy on both of ASR and translation tasks, while KL divergence loss only for ASR task.
 The student model consists the full encoder of the teacher large-v3 model and the decoder with two layers initialized from the first and last layer of the large-v3 model.
 
-Kotoba-Whisper is **6.3x faster than large-v3**, while retaining as low error rate as the large-v3.
+As kotoba-whisper uses the same architecture as [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3),
+it inherits the benefit of the improved latency compared to [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 
+(**6.3x faster than large-v3**, see the table below taken from [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)).
+
 
 ## Evaluation
 
+We compare our kotoba-whisper-bilingual with OpenAI whisper models, kotoba-whisper models, and cascaded models for translation.
+Worth noting that kotoba-whisper-bilingual is the only model that can do Japanese and English ASR and translation between Japanese and English, as 
+OpenAI whisper is not trained for English to Japanese translation, and other models are specific to the Task (eg. kotoba-whisper is Japanese ASR and
+distil whisper is English ASR only).
+
 ### Speech2Text Translation (Japanese->English)
 
 | model                                                                                                                                                                                                     |   [CoVoST2 (Ja->En)](https://huggingface.co/datasets/japanese-asr/ja2en.s2t_translation)|   [Fleurs (Ja->En)](https://huggingface.co/datasets/japanese-asr/ja2en.s2t_translation) |
@@ -105,20 +113,24 @@ Kotoba-Whisper is **6.3x faster than large-v3**, while retaining as low error ra
 | [japanese-asr/distil-whisper-bilingual-v1.0](https://huggingface.co/japanese-asr/distil-whisper-bilingual-v1.0) |                                                                        20.7 |                                                                               18.6 |                                                                                 2.4 |                                                                             6.4 |                                                                              10   |
 
 
+### Inference Speed
+Although the cascaded approach is better in translation task, due to the nature of cascaded approach, the pipeline has additional complexity compared to the single end2end models for the sake of high accuracy.
+Following table shows the mean inference time in second averaged over 10 trials on audio sample with different durations.
 
-
-
-
-
-- ***Latency***: As kotoba-whisper uses the same architecture as [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3),
-it inherits the benefit of the improved latency compared to [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 
-(**6.3x faster than large-v3**, see the table below taken from [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)).
-
-| Model                                                                                        | Params / M | Rel. Latency |
-|----------------------------------------------------------------------------------------------|------------|--------------|
-| **[kotoba-tech/kotoba-whisper-v2.0](https://huggingface.co/kotoba-tech/kotoba-whisper-v2.0)**| **756**    | **6.3**      |
-| **[kotoba-tech/kotoba-whisper-v1.0](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)**| **756**    | **6.3**      |
-| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                    | 1550       | 1.0          |
+| model                                                                                                                                                                                                     |    10 |    30 |    60 |   300 |
+|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------:|------:|------:|------:|
+| [kotoba-tech/kotoba-whisper-bilingual-v1.0](https://huggingface.co/kotoba-tech/kotoba-whisper-bilingual-v1.0)                                                                                             | 0.041 | 0.111 | 0.214 | 1.077 |
+| [japanese-asr/en-cascaded-s2t-translation](https://huggingface.co/japanese-asr/en-cascaded-s2t-translation) ([facebook/nllb-200-3.3B](https://huggingface.co/facebook/nllb-200-3.3B))                     | 0.173 | 0.247 | 0.352 | 1.772 |
+| [japanese-asr/en-cascaded-s2t-translation](https://huggingface.co/japanese-asr/en-cascaded-s2t-translation) ([facebook/nllb-200-1.3B](https://huggingface.co/facebook/nllb-200-1.3B))                     | 0.173 | 0.24  | 0.348 | 1.515 |
+| [japanese-asr/en-cascaded-s2t-translation](https://huggingface.co/japanese-asr/en-cascaded-s2t-translation) ([facebook/nllb-200-distilled-1.3B](https://huggingface.co/facebook/nllb-200-distilled-1.3B)) | 0.17  | 0.245 | 0.348 | 1.882 |
+| [japanese-asr/en-cascaded-s2t-translation](https://huggingface.co/japanese-asr/en-cascaded-s2t-translation) ([facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M)) | 0.108 | 0.179 | 0.283 | 1.33  |
+| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                                                                                                                                 | 0.061 | 0.184 | 0.372 | 1.804 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2)                                                                                                                                 | 0.062 | 0.199 | 0.415 | 1.854 |
+| [openai/whisper-large](https://huggingface.co/openai/whisper-large)                                                                                                                                       | 0.062 | 0.183 | 0.363 | 1.899 |
+| [openai/whisper-medium](https://huggingface.co/openai/whisper-medium)                                                                                                                                     | 0.045 | 0.132 | 0.266 | 1.368 |
+| [openai/whisper-small](https://huggingface.co/openai/whisper-small)                                                                                                                                       | 0.135 | 0.376 | 0.631 | 3.495 |
+| [openai/whisper-base](https://huggingface.co/openai/whisper-base)                                                                                                                                         | 0.054 | 0.108 | 0.231 | 1.019 |
+| [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)                                                                                                                                         | 0.045 | 0.124 | 0.208 | 0.838 |
 
 
 ## Transformers Usage
@@ -136,7 +148,8 @@ class to transcribe short-form audio files (< 30-seconds) as follows:
 
 Download sample audio.
 ```shell
-wget 
+wget https://huggingface.co/datasets/japanese-asr/en_asr.esb_eval/resolve/main/sample.wav -O sample_en.wav
+wget https://huggingface.co/datasets/japanese-asr/ja_asr.jsut_basic5000/resolve/main/sample.flac -O sample_ja.flac
 ```
 
 ```python
@@ -148,32 +161,35 @@ from datasets import load_dataset
 torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
 device = "cuda:0" if torch.cuda.is_available() else "cpu"
 model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
-
-# load model
 pipe = pipeline(
     "automatic-speech-recognition",
     model="kotoba-tech/kotoba-whisper-bilingual-v1.0",
     torch_dtype=torch_dtype,
     device=device,
-    model_kwargs=model_kwargs
+    model_kwargs=model_kwargs,
+    chunk_length_s=15,
+    batch_size=16
 )
 
-
+# Japanese ASR
 generate_kwargs = {"language": "ja", "task": "transcribe"}
+result = pipe("sample_ja.flac", generate_kwargs=generate_kwargs)
+print(result["text"])
 
-# load sample audio
-dataset = load_dataset("japanese-asr/ja_asr.reazonspeech_test", split="test")
-sample = dataset[0]["audio"]
+# English ASR
+generate_kwargs = {"language": "en", "task": "transcribe"}
+result = pipe("sample_en.wav", generate_kwargs=generate_kwargs)
+print(result["text"])
 
-# run inference
-result = pipe(sample, generate_kwargs=generate_kwargs)
+# Translate Japanese speech to English text
+generate_kwargs = {"language": "en", "task": "translate"}
+result = pipe("sample_ja.flac", generate_kwargs=generate_kwargs)
 print(result["text"])
-```
 
-- To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline (make sure the audio is sampled in 16kHz):
-```diff
-- result = pipe(sample, generate_kwargs=generate_kwargs)
-+ result = pipe("audio.mp3", generate_kwargs=generate_kwargs)
+# Translate English speech to Japanese text
+generate_kwargs = {"language": "ja", "task": "translate"}
+result = pipe("sample_en.wav", generate_kwargs=generate_kwargs)
+print(result["text"])
 ```
 
 - For segment-level timestamps, pass the argument `return_timestamps=True` and return the `"chunks"` output:
@@ -182,152 +198,12 @@ result = pipe(sample, return_timestamps=True, generate_kwargs=generate_kwargs)
 print(result["chunks"])
 ```
 
-***Sequential Long-Form:*** Kotoba-whisper is designed to be compatible with OpenAI's sequential long-form transcription algorithm. This algorithm uses a sliding window for buffered 
-inference of long audio files (> 30-seconds), and returns more accurate transcriptions compared to the [chunked long-form algorithm](#chunked-long-form).
-As default, if long audio files are passed to the model, it will transcribes with the sequential long-form transcription.
-The sequential long-form algorithm should be used in either of the following scenarios:
-
-1. Transcription accuracy is the most important factor, and latency is less of a consideration
-2. You are transcribing **batches** of long audio files, in which case the latency of sequential is comparable to chunked, while being up to 0.5% WER more accurate
-
-If you are transcribing single long audio files and latency is the most important factor, you should use the chunked algorithm
-described [below](#chunked-long-form). For a detailed explanation of the different algorithms, refer to Sections 5 of 
-the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf). The [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) 
-class can be used to transcribe long audio files with the sequential algorithm as follows: 
-
-
-### Chunked Long-Form
-This algorithm should be used when a single large audio file is being transcribed and the fastest possible inference is required. In such circumstances, 
-the chunked algorithm is up to 9x faster than OpenAI's sequential long-form implementation (see Table 7 of the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf)).
-To enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For distil-large-v3, a chunk length of 25-seconds
-is optimal. To activate batching over long audio files, pass the argument `batch_size`:
-
-```python
-import torch
-from transformers import pipeline
-from datasets import load_dataset
-
-# config
-model_id = "kotoba-tech/kotoba-whisper-v2.0"
-torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
-generate_kwargs = {"language": "ja", "task": "transcribe"}
-
-# load model
-pipe = pipeline(
-    "automatic-speech-recognition",
-    model=model_id,
-    torch_dtype=torch_dtype,
-    device=device,
-    model_kwargs=model_kwargs,
-    chunk_length_s=15,
-    batch_size=16
-)
-
-# load sample audio (concatenate instances to create a long audio)
-dataset = load_dataset("japanese-asr/ja_asr.reazonspeech_test", split="test")
-sample = {"array": np.concatenate([i["array"] for i in dataset[:20]["audio"]]), "sampling_rate": dataset[0]['audio']['sampling_rate']}
-
-# run inference
-result = pipe(sample, generate_kwargs=generate_kwargs)
-print(result["text"])
-```
-
-
-### Additional Speed & Memory Improvements
-You can apply additional speed and memory improvements to further reduce the inference speed and VRAM 
-requirements. These optimisations primarily target the attention kernel, swapping it from an eager implementation to a 
-more efficient flash attention version.
-
-#### Flash Attention 2
-
-We recommend using [Flash-Attention 2](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#flashattention-2) 
-if your GPU allows for it. To do so, you first need to install [Flash Attention](https://github.com/Dao-AILab/flash-attention):
-
-```
-pip install flash-attn --no-build-isolation
-```
-
-Then pass `attn_implementation="flash_attention_2"` to `from_pretrained`:
-
-```diff
-- model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
-+ model_kwargs = {"attn_implementation": "flash_attention_2"} if torch.cuda.is_available() else {}
-```
-
-
-## Model Details
-See [https://huggingface.co/distil-whisper/distil-large-v3#model-details](https://huggingface.co/distil-whisper/distil-large-v3#model-details).
-
 
 ## Training
 Please refer to [https://github.com/kotoba-tech/kotoba-whisper](https://github.com/kotoba-tech/kotoba-whisper) for the model training detail.
 Datasets used in distillation and the whole model variations can be found at [https://huggingface.co/japanese-asr](https://huggingface.co/japanese-asr).
 
 
-## Evaluation
-The following code-snippets demonstrates how to evaluate the kotoba-whisper model on the Japanese subset of the CommonVoice 8.0. 
-First, we need to install the required packages, including 🤗 Datasets to load the audio data, and 🤗 Evaluate to 
-perform the WER calculation:
-
-```bash
-pip install --upgrade pip
-pip install --upgrade transformers datasets[audio] evaluate jiwer
-```
-
-Evaluation can then be run end-to-end with the following example: 
-
-```python
-import torch
-from transformers import pipeline
-from datasets import load_dataset
-from evaluate import load
-from transformers.models.whisper.english_normalizer import BasicTextNormalizer
-
-# model config
-model_id = "kotoba-tech/kotoba-whisper-v2.0"
-torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
-generate_kwargs = {"language": "japanese", "task": "transcribe"}
-normalizer = BasicTextNormalizer()
-
-# data config
-dataset_name = "japanese-asr/ja_asr.reazonspeech_test"
-audio_column = 'audio'
-text_column = 'transcription'
-
-# load model
-pipe = pipeline(
-    "automatic-speech-recognition",
-    model=model_id,
-    torch_dtype=torch_dtype,
-    device=device,
-    model_kwargs=model_kwargs,
-    batch_size=16
-)
-
-# load the dataset and sample the audio with 16kHz
-dataset = load_dataset(dataset_name, split="test")
-transcriptions = pipe(dataset['audio'])
-transcriptions = [normalizer(i['text']).replace(" ", "") for i in transcriptions]
-references = [normalizer(i).replace(" ", "") for i in dataset['transcription']]
-
-# compute the CER metric
-cer_metric = load("cer")
-cer = 100 * cer_metric.compute(predictions=transcriptions, references=references)
-print(cer)
-```
-
-The huggingface links to the major Japanese ASR datasets for evaluation are summarized at [here](https://huggingface.co/collections/japanese-asr/japanese-asr-evaluation-dataset-66051a03d6ca494d40baaa26).
-For example, to evaluate the model on JSUT Basic5000, change the `dataset_name`:
-
-```diff
-- dataset_name = "japanese-asr/ja_asr.reazonspeech_test"
-+ dataset_name = "japanese-asr/ja_asr.jsut_basic5000"
-```
-
 ## Acknowledgements
 * [OpenAI](https://openai.com/) for the Whisper [model](https://huggingface.co/openai/whisper-large-v3).
 * Hugging Face 🤗 [Transformers](https://github.com/huggingface/transformers) for the model integration.