fix vocabulary size based on all unique word count
fix vocabulary size based on all unique word count

len(tokenizer.word_index) return all unfiltered vocab size. We filtered this vocabulary while building Tokenizer.

For example:

import tensorflow as tf
tk = tf.keras.preprocessing.text.Tokenizer(num_words=2)
texts = ["AAA BBB", "CCC DDD", "AAA DDD"]
tk.fit_on_texts(texts)

print(tk.word_index)  ## ---> prints {'aaa': 1, 'ddd': 2, 'bbb': 3, 'ccc': 4}
print(len(tk.word_index)) ## ---> prints 4
tensorflow/docs,tensorflow/docs,tensorflow/docs