Machine learning anomaly detection for continues data I need to find anomalies in logs created by network monitor tool Bro. Right now I am referring to the example given by Bro Analyze Tool. But the example only uses recent logs.

So I need to make it work with full log meaning to train on past logs and perform anomaly detection in real time on each new log. In the example, as far as I understand, unsupervised machine learning with IsolationForest and KMeans in scikit-learn python are used.

So I have the following question.

For unsupervised machine learning is it possible to store the model with trained data which is already created on previous run?

If possible can I add new data with existing model and then perform anomaly detection in real time?

You can save sklearn models with pickle. This blog post shows how.


How to use a NN architecture that is too big for GPU? Initially posted in Stack Overflow.
I would like to implement a model which is actually 2 neural networks stacked together. However the size of these 2 architecture is too big to fit in GPU at the same time.
My idea was the following :

Load the first model and run it for 1 batch
Unload the first model from GPU, load the second model
Run the second model from the output of the first model
Unload the second model from GPU
Repeat for every batch

I actually don't need to train the first model, since it's pre-trained. But I need to train second model.
Is it possible to do something like this ? Is my approach correct ? What are the pitfalls I should be aware of ? What about performance ?
Edit
I already tried the idea of computing the output of the first model for the whole dataset at first, and then use it as input for the second dataset. However the output of the first dataset is really big, and I don't have the available space for storing the whole pre-processed dataset. That is why I wanted to do it each batch.
Edit 2
After the very nice answer from @Gal Avineri, just one more precision :
I would like to implements my architecture using only one GPU.

I suggest a method similar to what @ignatius offered.
Since you don't need to train the first model and only the second one you could do the following:

Use the first model over the entire dataset and save the extracted results in your memory.
Train the second model using the results from the previous step.

This way you will only have to load one model at a given time.
In addition this will make the training of the second model to be faster.
Edit 1
(the op has specified he does not have enough memory for the inference results)
In this case I can suggest to parallelize between the inference of the first model and the training of the second model.
This could be achieved through prefatching.
Let's denote the original dataset as data1 and the corresponding inference results from the first model as data2.
Let's also denote the first model as M1 and the second model as M2.
Thus you can do the following:

load M1 to cpu, and M2 to gpu.
Draw a batch from data1 and use M1 to prepare a batch of data2 for M2.
Use the gpu to train M2 on the batch received from the previous step
while step 3 is being executed, prepare the next batch of data2 by executing step 2.

In this way, while M2 is being trained on a batch, the next batch is being prepared. This is called prefetching the next batch.
This will parallelize the training of M2 and the making of data2.
The method i described above has a very simple way to implement using the tensorflow "Data" api.
Here is a code example:
import numpy as np
from tensorflow.keras import backend as K
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.data import Dataset

with K.device('cpu0'):
    input = Input((10,))
    x = Dense(100)(input)
    m1 = Model(input_shape=input, outputs=x)

with K.device('gpu0'):
    input = Input((100,))
    x = Dense(5)(input)
    m2 = Model(input_shape=input, outputs=x)

features = np.random.rand(10, 2000)
labels = np.random.randint(2, size=2000)
dataset = Dataset.from_tensor_slices((features, labels))

def preprocess_sample(features, label):
    inference = m2.predict(features, batch_size=1)
    return inference, label

dataset = dataset.shuffle(2000).repeat().map(preprocess_sample).batch(32)
dataset = dataset.prefetch(1)

m2.fit(dataset, epochs=100, steps_per_epoch=2000//32)


Why is input preprocessing in VGG16 in Keras not 1/255.0 I am just trying to use pre-trained vgg16 to make prediction in Keras like this.

from scipy import ndimage
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input

im = scipy.misc.imread("cat_dog/validation/cats/cat.1362.jpg").astype(np.float32)
im = scipy.misc.imresize(im, (224, 224)).astype(np.float32)

#im /= 255.0
#im = im - np.mean(im, axis=2, keepdims=True)

im = np.expand_dims(im, axis=0)
im = preprocess_input(im)

out = vgg16_model.predict(im)
np.argmax(out)

It seemed that im /= 255.0 give very bad prediction. I commented it out and it started making good prediction. I also added preprocess_input(...) but that doesn't seem to affect prediction for the few random trials I did.

The question is that according to this great blog:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

under "Using the bottleneck features of a pre-trained network: 90% accuracy in a minute", pre-trained VGG16 is in a transfer learning context. And if you look at this gist, you see this line of code:

def save_bottlebeck_features():
    datagen = ImageDataGenerator(rescale=1. / 255)
    etc.

The preprocessing of input seemed to be 1/255.0 during caching of features from the last conv layer. This is sort of puzzling. I further also looked up how preprocess_input(...) is defined in the code I have, and found for 'tf':

x /= 127.5
x -= 1.

Which you can check here.

The pre-trained weights that are available on Keras are trained with the preprocessing steps defined in preprocess_input() function that is made available for each network architecture (VGG16, InceptionV3, etc).

For example

from keras.applications.vgg16 import preprocess_input

If you are using the weights that comes with keras for fine tuning, then you should use the corresponding preprocess_input() function for the network. It can be different from the original preprocessing steps mentioned in the paper.

from keras.applications.vgg16 import VGG16
model = VGG16(weights='imagenet', include_top=False)

If you want to stick to the original preprocessing steps, you can find pre-trained weights that is trained with the original preprocessing steps instead of using the weights that comes with Keras.

Maybe at the time of writing the blog post the weights were trained with different preprocessing steps.

Checkout this github issue to learn more.


What is GELU activation? I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as
$$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{
2/π}(x + 0.044715x^3)])$$

Could you simplify the equation and explain how it has been approximated.

GELU function

We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows:
$$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$

Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a stochastic analogy, however mathematically, this is just a definition.

Here is the plot of GELU:

Tanh approximation

For these type of numerical approximations, the key idea  is to find a similar function (primarily based on experience), parameterize it, and then fit it to a set of points from the original function.

Knowing that $\text{erf}(x)$ is very close to $\text{tanh}(x)$

and first derivative of $\text{erf}(\frac{x}{\sqrt{2}})$ coincides with that of $\text{tanh}(\sqrt{\frac{2}{\pi}}x)$ at $x=0$, which is $\sqrt{\frac{2}{\pi}}$, we proceed to fit
$$\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+ax^2+bx^3+cx^4+dx^5)\right)$$ (or with more terms) to  a set of points $\left(x_i, \text{erf}\left(\frac{x_i}{\sqrt{2}}\right)\right)$.

I have fitted this function to 20 samples between $(-1.5, 1.5)$ (using this site), and here are the coefficients:

By setting $a=c=d=0$, $b$ was estimated to be $0.04495641$. With more samples from a wider range (that site only allowed 20), coefficient $b$ will be closer to paper's $0.044715$. Finally we get

$\text{GELU}(x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)\simeq 0.5x\left(1+\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+0.044715x^3)\right)\right)$

with mean squared error $\sim 10^{-8}$ for $x \in [-10, 10]$.

Note that if we did not utilize the relationship between the first derivatives, term $\sqrt{\frac{2}{\pi}}$ would have been included in the parameters as follows
$$0.5x\left(1+\text{tanh}\left(0.797885x+0.035677x^3\right)\right)$$
which is less beautiful (less analytical, more numerical)!

Utilizing the parity

As suggested by @BookYourLuck, we can utilize the parity of functions to restrict the space of polynomials in which we search. That is, since $\text{erf}$ is an odd function, i.e. $f(-x)=-f(x)$, and $\text{tanh}$ is also an odd function, polynomial function $\text{pol}(x)$ inside $\text{tanh}$ should also be odd (should only have odd powers of $x$) to have
$$\text{erf}(-x)\simeq\text{tanh}(\text{pol}(-x))=\text{tanh}(-\text{pol}(x))=-\text{tanh}(\text{pol}(x))\simeq-\text{erf}(x)$$

Previously, we were fortunate to end up with (almost) zero coefficients for even powers $x^2$ and $x^4$, however in general, this might lead to low quality approximations that, for example, have a term like $0.23x^2$ that is being cancelled out by extra terms (even or odd) instead of simply opting for $0x^2$.

Sigmoid approximation

A similar relationship holds between $\text{erf}(x)$ and $2\left(\sigma(x)-\frac{1}{2}\right)$ (sigmoid), which is proposed in the paper as another approximation, with mean squared error $\sim 10^{-4}$ for $x \in [-10, 10]$.

Here is a Python code for generating data points, fitting the functions, and calculating the mean squared errors:

import math
import numpy as np
import scipy.optimize as optimize

def tahn(xs, a):
    return [math.tanh(math.sqrt(2 / math.pi) * (x + a * x**3)) for x in xs]

def sigmoid(xs, a):
    return [2 * (1 / (1 + math.exp(-a * x)) - 0.5) for x in xs]

print_points = 0
np.random.seed(123)
# xs = [-2, -1, -.9, -.7, 0.6, -.5, -.4, -.3, -0.2, -.1, 0,
#       .1, 0.2, .3, .4, .5, 0.6, .7, .9, 2]
# xs = np.concatenate((np.arange(-1, 1, 0.2), np.arange(-4, 4, 0.8)))
# xs = np.concatenate((np.arange(-2, 2, 0.5), np.arange(-8, 8, 1.6)))
xs = np.arange(-10, 10, 0.001)
erfs = np.array([math.erf(x/math.sqrt(2)) for x in xs])
ys = np.array([0.5 * x * (1 + math.erf(x/math.sqrt(2))) for x in xs])

# Fit tanh and sigmoid curves to erf points
tanh_popt, _ = optimize.curve_fit(tahn, xs, erfs)
print('Tanh fit: a=%5.5f' % tuple(tanh_popt))

sig_popt, _ = optimize.curve_fit(sigmoid, xs, erfs)
print('Sigmoid fit: a=%5.5f' % tuple(sig_popt))

# curves used in https://mycurvefit.com:
# 1. sinh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))/cosh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))
# 2. sinh(sqrt(2/3.141593)*(x+b*x^3))/cosh(sqrt(2/3.141593)*(x+b*x^3))
y_paper_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + 0.044715 * x**3))) for x in xs])
tanh_error_paper = (np.square(ys - y_paper_tanh)).mean()
y_alt_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + tanh_popt[0] * x**3))) for x in xs])
tanh_error_alt = (np.square(ys - y_alt_tanh)).mean()

# curve used in https://mycurvefit.com:
# 1. 2*(1/(1+2.718281828459^(-(a*x))) - 0.5)
y_paper_sigmoid = np.array([x * (1 / (1 + math.exp(-1.702 * x))) for x in xs])
sigmoid_error_paper = (np.square(ys - y_paper_sigmoid)).mean()
y_alt_sigmoid = np.array([x * (1 / (1 + math.exp(-sig_popt[0] * x))) for x in xs])
sigmoid_error_alt = (np.square(ys - y_alt_sigmoid)).mean()

print('Paper tanh error:', tanh_error_paper)
print('Alternative tanh error:', tanh_error_alt)
print('Paper sigmoid error:', sigmoid_error_paper)
print('Alternative sigmoid error:', sigmoid_error_alt)

if print_points == 1:
    print(len(xs))
    for x, erf in zip(xs, erfs):
        print(x, erf)

Output:

Tanh fit: a=0.04485
Sigmoid fit: a=1.70099
Paper tanh error: 2.4329173471294176e-08
Alternative tanh error: 2.698034519269613e-08
Paper sigmoid error: 5.6479106346814546e-05
Alternative sigmoid error: 5.704246564663601e-05


Split unprocessed dataset into train and test sets I have two files namely - log.csv and label.csv. My aim is to to split these files into training and testing datasets before preprocessing the log_train file further.

The log.csv file has a no. of columns including id. For one value of id, the file has 'z' no. of rows (this 'z' is different for different value of id) . Whereas, in the label.csv file there is just one row for each value of id.  I wish to split the files into - log_train.csv, log_test.csv, label_train.csv and label_test.csv obviously such that all rows corresponding to one value of id goes either to train or test file with corresponding values in label_train or label_test file.

I have already tried doing:
X_train,X_test,y_train,y_test=train_test_split(df,df_label,test_size = 0.2, random_state = 0)

but this gives me the error:
Traceback (most recent call last):
File "tmp.py", line 35, in <module>
    X_train, X_test, y_train, y_test = train_test_split(df,df_label,test_size = 0.2, random_state = 0)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/model_selection/_split.py", line 2184, in train_test_split
    arrays = indexable(*arrays)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 260, in indexable
    check_consistent_length(*result)
  File "/home/yamini/virtual/home/yamini/virtual/lib/python3.6/site-packages/sklearn/utils/validation.py", line 235, in check_consistent_length
    " samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [608, 3]

How can I achieve this in python?

Use scikit learn:

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)


How to disable GPU with TensorFlow? Using tensorflow-gpu 2.0.0rc0. I want to choose whether it uses the GPU or the CPU.

I've seen some suggestions elsewhere, but they are old and do not apply very well to newer TF versions. What worked for me was this:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

When that variable is defined and equal to -1, TF uses the CPU even when a CUDA GPU is available.


machine learning on athlete performances to predict the time in a future race Example Data

I have a dataset (in R as a data frame) of race results for athletes.

athlete racedistance time    location tracktype       date    coach
A          100       10.0       UK     typeA       2014-01-01  carlos
A          200       20.0       US     typeB       2014-02-01  carla
A          100        9.5      AUS     typeC       2014-03-01  chris
B          100       11.0       UK     typeA       2014-01-01  carla
B          200       21.0       US     typeB       2014-02-01  carlos
B          400       61.0      AUS     typeC       2014-03-01  carla
B          100       10.5      GER     typeA       2014-04-01  clive
C          100        9.5       UK     typeA       2014-01-01  clive
C          200       21.5       US     typeB       2014-02-01  chris

Question

Is there an appropriate machine learning algorithm or method that can use the previous results of each athlete as a feature, when trying to predict the time for an athlete in a future race?

For example, athlete A has three races, with one month rest between them. In the third race he performs slightly better than the first race over the same distance.

Can an algorithm learn that the second race had an effect on the athlete, which meant he performed better in the third race?

From what I've read on the subject and the training examples I've completed it would appear that each 'row' of data should be independent, is this the case for all ML algorithms? Is there another prediction technique I should be considering to solve this type of problem?

I think the key here is what you believe to be dependent variables. You mentioned, for instance, a three month rest. Encoding the rest between races is likely to be a better idea than simply encoding the date of the rest as a lot of the date is redundant to what is actually being said. As with any machine learning algorithm, representation of the data is key and in many ways, the actual algorithm applied is less important.


Problem importing dataset I am new to machine learning and I am trying to build a classifier. My problem is that I am not able to import the dataset I need. In particular, I put my dataset in the Desktop and what I did is:

#pakages
import numpy as np
import pandas as pd
import jsonlines                   #edit

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import *
from sklearn.naive_bayes import *
from sklearn.metrics import confusion_matrix, classification_report
from sklearn import svm

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

now, after these imports I want to use the function

training_set = pd.read_json('\Desktop\training_dataset.jsonl')  #edit
print(training_set.head())

to import my dataset. The problem is that what i get is:

ValueError                                Traceback (most recent call last)
<ipython-input-16-f789503c3c7c> in <module>
----> 1 training_set = pd.read_json('\Desktop\training_dataset.json')
      2 print(training_set.head())

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates,
keep_default_dates, numpy, precise_float, date_unit, encoding, lines,
chunksize, compression)
    590         return json_reader
    591
--> 592     result = json_reader.read()
    593     if should_close:
    594         try:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in read(self)
    715             obj =
self._get_object_parser(self._combine_lines(data.split("\n")))
    716         else:
--> 717             obj = self._get_object_parser(self.data)
    718         self.close()
    719         return obj

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
_get_object_parser(self, json)
    737         obj = None
    738         if typ == "frame":
 --> 739             obj = FrameParser(json, **kwargs).parse()
    740
    741         if typ == "series" or obj is None:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in parse(self)
    847
    848         else:
--> 849             self._parse_no_numpy()
    850
    851         if self.obj is None:

~\Anaconda3\lib\site-packages\pandas\io\json\_json.py in
_parse_no_numpy(self)
   1091         if orient == "columns":
   1092             self.obj = DataFrame(
-> 1093                 loads(json, precise_float=self.precise_float),
dtype=None
   1094             )
   1095         elif orient == "split":

ValueError: Expected object or value

and I can't understand why. Can somebody please help me? Thank's in advance.

[EDIT] the file is a .jsonl file, but yet I don't know how to import the dataset because I cannot use .read_json
      ,I have tried this:

openfile=open('Desktop\training_dataset.jsonl')
jsondata=json.load(openfile)
df=pd.DataFrame(jsondata)
openfile.close()
print(df)

but gives me the following error message:

OSError                                   Traceback (most recent call last)
<ipython-input-28-2422c1a9a77b> in <module>
----> 1 openfile=open('Desktop\training_dataset.jsonl')
      2 jsondata=json.load(openfile)
      3 df=pd.DataFrame(jsondata)
      4 openfile.close()
      5 print(df)

OSError: [Errno 22] Invalid argument: 'Desktop\training_dataset.jsonl'

[EDIT 2] by doing as suggested, so:

with open("\Desktop\training_dataset.jsonl") as datafile:
data = json.load(datafile)
dataframe = pd.DataFrame(data)

I again obtain another error message, which is:

OSError                                   Traceback (most recent call last)
<ipython-input-47-1365f26e6db5> in <module>
 ----> 1 with open("\Desktop\training_dataset.jsonl") as datafile:
      2     data = json.load(datafile)
      3 dataframe = pd.DataFrame(data)

OSError: [Errno 22] Invalid argument: '\\Desktop\training_dataset.jsonl'

but I don' understand, because my dataset is placed in my desktop.

You can follow the below steps to load a json file,

First check whether the file is json or not using the following; https://jsonlint.com/. Once you are confirmed the file is a json, use the below code to read it.

with open("training_dataset.json") as datafile:
    data = json.load(datafile)
dataframe = pd.DataFrame(data)

I hope the above will help you.


python - What is the format of the WAV file for a Text to Speech Neural Network? I am creating a Text to Speech system for a phonetic language called "Kannada" and I plan to train it with a Neural Network. The input is a word/phrase while the output is the corresponding audio.

While implementing the Network, I was thinking the input should be the segmented characters of the word/phrase as the output pronunciation only depends on the characters that make up the word, unlike English where we have slient words and Part of Speech to consider. However, I do not know how I should train the output.

Since my Dataset is a collection of words/phrases and the corrusponding MP3 files, I thought of converting these files to WAV using pydub for all audio files.

from pydub import AudioSegment
sound = AudioSegment.from_mp3("audio/file1.mp3")
sound.export("wav/file1.wav", format="wav")

Next, I open the wav file and convert it to a normalized byte array with values between 0 and 1.

import numpy as np
import wave

f = wave.open('wav/kn3.wav', 'rb')
frames = f.readframes(-1)

#Array of integers of range [0,255]
data = np.fromstring(frames, dtype='uint8')

#Normalized bytes of wav
arr  = np.array(data)/255

How Should I train this?

From here, I am not sure how to train this with the input text. From this, I would need a variable number of input and output neurons in the First and Last layers as the number of characters (1st layer) and the bytes of the corresponding wave (Last layer) change for every input.

Since RNNs deal with such variable data, I thought it would come in handy here.

Correct me if I am wrong, but the output of Neural Networks are actually probability values between 0 and 1. However, we are not dealing with a classification problem. The audio can be anything, right? In my case, the "output" should be a vector of bytes corresponding to the WAV file. So there will be around 40,000 of these with values between 0 and 255 (without the normalization step) for every word. How do I train this speech data? Any suggestions are appreciated.

EDIT 1 : In response to arduinolover's Answer

From what I understand, Phonemes are the basic sounds of the language.  So, why do I need a neural network to map phoneme labels with speech? Can't I just say, "whenever you see this alphabet, pronounce it like this". After all, this language, Kannada, is phonetic: There are no silent words. All words are pronounced the same way they are spelled. How would a Neural Network help here then?

On input of a new text, I just need to break it down to the corresponding alphabets (which are also the phonemes) and retrieve it's file (converted from WAV to raw byte data).  Now, merge the bytes together and convert it to a wav file.

Is this this too simplistic? Am I missing something here? What would be the point of a Neural Network for this particular language (Kannada) ?

Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.

You can use a phonetic dictionary for converting your text files into sequence of phonemes. e.g play -> P L EY

If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).

But there are many things that affect the pronunciation. Some of them are listed below:

Context: 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.

Pitch: Female speakers usually have higher pitch than male speakers.

Speaking rate: Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.

length_of_output_phoneme: The length of wav file to generate

So in the end input to your NN will look something like this
[left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme] and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation.

If you don't have time marked data. You can use Hidden Markov model HMM for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.

Some speech synthesis resources are listed below:

1) CMU festvox

2) wavenet

3) Deep Learning in Speech Synthesis

The biggest challenge will be to make it sound like human voice.

Edit 1: Some of the problems with "whenever you see this alphabet, pronounce it like this" are listed below

1) context: From the above to and go example which pronunciation of 'o' will be used

2) discontinuity: Vocal tract vibrations (lip, tongue motions) produce phoneme. Phonemes diffuse into neighboring phonemes since vocal tract doesn't stop vibrating immediately. If you copy-paste phonemes then there will be an abrupt change. You can copy-paste sounds for words since they are independent (kind of, 'can not' becomes 'can't' while speaking) and insert small pauses in between words. But then you have to store pronunciations for all the words.

3) stress phoneme: While speaking one or more phonemes are stressed (more focused, longer), Copy-pasting leaves out the stress information since everything is same

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds.
With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated. Go through the first and third resource for detailed explanations.

All these problem arise because we want to make it sound more human. Phoneme transition and diffusion are the major hurdles.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Even if you don't change any condition each time there will be a slight variation in pronunciation of same word. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.


How to detect time for the future events in time series data? I am dealing with IOT data from a mechanical machine. On the input I have ~100 features that are measured every minute. On the output, I have labels of zeros and ones, where zero indicates the absence of the event and 1 indicate the presence of an event. The event represent the failure for the machine in place. Therefore, the goal is to predict at every time step the remaining "minutes" for a failure to occur. I would like to know how to tackle this problem, and if possible for some material to read.
Is there a way to know which features in the past leads to a failure in the future if I'm using an LSTM?

Try boosting and random forests. You would need a long enough time series. A good reference (with examples in R) is
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An Introduction to Statistical Learning: with Applications in R. Springer.
Usefulness of particular features can be assessed via variable importance plots.


Consequences of using XGBoost regressor for small dataset(< 500 rows) I am using XGBoost regressor to train my model for 322 rows of data and the train and test split is as follows: ((257, 9), (257,), (65, 9), (65,))
I am using the following parameters for hyper-parameter tuning:
{'max_depth': 3,
 'min_child_weight': 6,
 'eta': 0.3,
 'subsample': 0.9,
 'colsample_bytree': 0.7,
 'objective': 'reg:linear',
 'eval_metric': 'rmse',
 'reg_lambda': 0,
 'reg_alpha': 0.5,
 'gamma': 0}

I am getting the following results:
Train results:

MAE =  43.95317769328908
RMSE =  69.32233101307436
R2 score =  0.7500463354991436

--------------------------------------------

Test results :

MAE =  51.21307032658503
RMSE =  79.65759750390318
R2 score =  0.6569142423871053

What are the drawbacks of training XGBoost model on such a small dataset? I know about overfitting, but I can control it to some extend with regularization.

The one other problem you have because of this is you are stuck with just this solution and this may not be the best one. Yes, you can use regularization but still, what if some other better solution exists than this. Since, in boosted trees, you aim is to  reduce your errors, each iteration of boosting tries to overfit on the examples which the last iteration predicted wrong.
But there is a way to confirm if this is the best solution. Do cross-validation. Or try LightGBM and CatBoost also, since, they are also good libraries for the same purpose. And finally, you combine all three of them to come with an ensemble that will you the best of the three worlds.


Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce I understand Hadoop MapReduce and its features but I am confused about R MapReduce.

One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with Hadoop.

My doubt is:

R can do all stats, math and data science related stuff, but why R MapReduce?
Is there any new task I can achieve by using R MapReduce instead of Hadoop MapReduce? If yes, please specify.
We can achieve the task by using R with Hadoop (directly) but what is the importance of MapReduce in R and how it is different from normal MapReduce?

rhadoop (the part you are interested in is now called rmr2) is simply a client API for MapReduce written in R. You invoke MapReduce using R package APIs, and send an R function to the workers, where it is executed by an R interpreter locally. But it is otherwise exactly the same MapReduce.

You can call anything you like in R this way, but no R functions are themselves parallelized to use MapReduce in this way. The point is simply that you can invoke M/R from R. I don't think it somehow lets you do anything more magical than that.


Best Way to combine multiple datasets into one model I want to make a multilabel image classification model that can detect many different labels. For each label, I can get at least 5000 positive examples and 5000 negative examples. However, my question is how can I use data in this format to train a multilabel image classifier. Part of the challenge is, for example, I can download 10,000 images of a hand and know that they are positive examples but then if I want to detect a shoe as well, I don’t know how many of those hand photos might have also had a shoe in them. I’m trying to make a model this way because I will have a fairly high amount of labels and need to be continuously adding new labels. What is the best way to go about doing this?

Why don't you give Multi-label classification a try.you can actually train your model to predict multiple labels for one image by training the model on a dataset that has images with multiple labels the only problem you might face here is to where to get the data from you can actually use this dataset for practice:
https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/data.

Another better option that you have is as you said in your question you want to train your model on multiple classes you can actually use Imagenet dataset it mas 1000 classes and more than a million images.
Note:Use tranfer learning to build your model and do watch lesson 1,2,3 from fast.ai part1 2019 course you will have all your answers. Jeremy Howard is a god!!!!


Difference between training and test data distributions A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate shift addresses this issue. Can someone clear the following doubts regarding this?

How does one check whether two distribution are statistically different?
Can kernel density estimate (KDE) be used to estimate the probability distribution to tell the difference?
Let's say I have 100 images of a specific category. The number of test images is 50, and I'm changing the number of training images from 5 to 50 in steps of 5. Can I say the probability distributions are different when using 5 training images and 50 testing images after estimating them by KDE?

A good way to measure the difference between two probabilistic distributions is Kullbak-Liebler. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)


Is it possible to train the neural networks for programming I want to train my neural networks for the task of programming in a different language. So, I want to know whether is it possible or not to train a neural net for programming?

It depends on what kind of functionality you would want to achieve. In essence, a trained neural network is a mapping function, which takes some input and produces some output, which then may be variously interpreted (for example using a certain threshold for classification).

Therefore, if the desired functionality of your program would be coming up with a computational function, you could treat your trained neural network as the program itself. Multiple machine learning libraries allow that easily, producing a class instance which contains the trained neural network inside and allows you to call it with given parameters just like any other function.

However, I assume that you are looking for a way to output complex computer code with control structures, etc. - that unfortunately is very difficult to achieve, since you'd need an effective way to interpret the neural network outputs as code - not just the commands, but their structure and ordering as well. Such interpretations would be so complex that they'd make it very difficult for any Neural Network training method to converge to anything sensible.

To mitigate this problem, you can try machine learning techniques which manipulate and optimize the code "directly". The most famous of such is Genetic Programming - in which a computer program in encoded as a set of "genes" which are then improved upon through genetic algorithm techniques (crossovers which recombine large parts of code and mutations which randomly change small parts of the code). This has been successfully applied to some programming problems such as the Towers of Hanoi (see http://www.oakgp.org/towers-of-hanoi ).


Alternative regression model algorithms for machine learning I am looking for not so known regression models and if possible a python library that implements it.
In my quiver I have:

Generalized Linear Models, Linear, Lasso, Ridge...
Decision Tree based model: CART, Random Forest, Gradient Boosting Decision trees(xgb,lightgbm,catboost)
Support Vector Machines
Knn Regressor
Neural Networks

Are there regression algorithm less known? I am looking for alternatives models to add to my quiver.

Multiple regression
multivariate regression
Mixed Models
elastic search
Partial Least Squares regression
Logistic regression
Bayesian regression and many more..

When you google you'll find first page suggestions like https://www.listendata.com/2018/03/regression-analysis.html , https://www.techfunnel.com/information-technology/regression-analysis-types/ and https://www.kaggle.com/eraaz1/a-comprehensive-guide-to-advanced-regression


Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens? I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further downstream task.
Bert's last layer looks like this :

Where we take the [CLS] token of each sentence :

Image source
I went through many discussion on this huggingface issue,  datascience forum question,  github issue Most of the data scientist gives this explanation :

BERT is bidirectional, the [CLS] is encoded including all
representative information of all tokens through the multi-layer
encoding procedure. The representation of [CLS] is individual in
different sentences.

My question is, Why the author ignored the other information ( each token's vector ) and taking the average, max_pool or other methods to make use of all information rather than using [CLS] token for classification?
How does this [CLS] token help compare to the average of all token vectors?

It's because you need to fine-tune BERT for your specific task anyway. You can train it to classify based on either cls token, or mean of token outputs, or whatever.
In essence, CLS token of the last layer has connections with all of the other tokens on the previous layer. So, does it make sense to average manually?


Is there any good alternative for Keras library? I am currently using keras as a deep learning library on top of tensorflow. I just want to know is there any other library which is more efficient/easier than keras.

I can't say which is more efficient or easier but MXNet exists, has interfaces to most languages (python, scala, R, C++, perl, Julia), and the code looks fairly similar to keras.

https://mxnet.incubator.apache.org/

Another one that I've come across is caffe, mostly because people seem to share a lot of pre-trained nets in their model zoo.

http://caffe.berkeleyvision.org/

Update: I didn't see caffe 2 but I think this is quite separate to the original caffe.

Finally I've not fully had time to check out H2O's offering in deep water which is an alternative to Keras but still sits on top of TF/MXNet/Caffe


Are there deduplication algorithms that do not work on a metric space? Recently I got interested in the process of data cleansing and specifically in record linkage.

Thus far I read about deterministic and probabilistic approaches to deduplicate data sets and to some lesser degree also about machine learning methods. It struck me that the key part of all algorithm basically introduce a metric space. Through the metric space every two data points can be assigned a distance. The distance is then basically a measure of how close these two data points are related to another.

However I do wonder, if there were not also different kinds of algorithms that do not work on this principle?

One option is fingerprinting. If two objects have the same fingerprint, they are probably the same object. Depending the technique used, the fingerprint can not tell about approximate duplicates.


Text similarity using RNN Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the similarity APIs provided by the package.

There is a smaller subset of this data which is already tagged as duplicated. Currently, I have not factored in this as part of model training. Also, using the text2vec package, the results are not great on this test set. So now I am considering RNNs which are known to perform well in text similarity.

Now, I need help in feature engineering and preparing my input layer. Sentences S1 and S2 that need to be compared differ in length (the word representations differ in their dimensions). How to normalize this difference? Should I consider bag-of-words or glove word vectors (word vectors being much superior representations)? Any inputs in this regard will help.

Doc2Vec, Mikolov's paper will solve your problem. Here is the paper. You can find a gensim implementationhere. While using RNN, using GLOVE or Googl Word2Vec will be always useful even if your sequences are of the same length. To solve the variable sequence problem you can pad the sentences and use bucketing or just pad to one uniform length, truncating longer sentences and padding shorter sentences.


How to count the number of missing values in each row in Pandas dataframe? How can I get the number of missing value in each row in Pandas dataframe.
I would like to split dataframe to different dataframes which have same number of missing values in each row.

Any suggestion?

When using pandas, try to avoid performing operations in a loop, including apply, map, applymap etc. That's slow!

A DataFrame object has two axes: “axis 0” and “axis 1”. “axis 0” represents rows and “axis 1” represents columns.

If you want to count the missing values in each column, try:

df.isnull().sum() as default or df.isnull().sum(axis=0)

On the other hand, you can count in each row (which is your question) by:

df.isnull().sum(axis=1)

It's roughly 10 times faster than Jan van der Vegt's solution(BTW he counts valid values, rather than missing values):

In [18]: %timeit -n 1000 df.apply(lambda x: x.count(), axis=1)
1000 loops, best of 3: 3.31 ms per loop

In [19]: %timeit -n 1000 df.isnull().sum(axis=1)
1000 loops, best of 3: 329 µs per loop


Find most important features which differentiate two sets Assume that we have two small sets of feature vectors (each feature vector representing an item). What is a good way of finding which features have the biggest difference (in distribution) between the two (small, 50 samples) sets? Given that the distribution of the data per feature is not necessarily normal.

A reformulation of your question would be:

  Which are the features that differentiate the two datasets more accurately, because they differ a lot across them?

A common way to answer this is by checking the information gain of the attributes, based on the reduction of the entropy of the dataset. The attributes with the highest information gain are the ones that separate the dataset more accurately.

Assuming you have m attributes, you need to:

Label your samples based on which dataset they belong (A or B)
Attempt to split the dataset based on the values of each attribute. This means you need to perform m splits.
The split that maximizes the information gain, thus minimizes the individual sub-entropy of the two splitted groups, shows the attribute that separates the data the best. This attribute is the one that differs the most in both datasets and is the answer to your question.

FYI this is exactly what is done when training decision trees, to cut out (prune) features that are not informative enough in order to:

Reduce the tree's variance
Reduce training / inferencing time


Voice recognition with fourier transformation with audio input in python First of all is using the fourier transformation even a good method for recognizing different speakers? I'm not sure if it could recognize a voice if the things that are said are different. I know google and amazon have features of voice/speaker recognition in their voice assistants but what would be a good way to make that too if the fourier transformation doesn't work out?

I want to recognize voices using a neural network, to do that I need to first get a good input for the neural network but by just giving the sound recording as input I don't think it would work because it is based on frequency and time. So I found the Fourier transformation and now I'm trying to transform my audio file with Fourier and plot it.

My questions are:

How can I plot a Fourier transformation with audio input in python?
And if that is working, how can I input the Fourier transformation in the neural network (I thought perhaps give every neuron a y value with the neurons as the corresponding x value)

I tried something like (a combination of things I found on the internet):

import matplotlib.pyplot as plt
from scipy.io import wavfile as wav
from scipy.fftpack import fft
import numpy as np
import wave
import sys

spf = wave.open('AAA.wav','r')

#Extract Raw Audio from Wav File
signal = spf.readframes(-1)
signal = np.fromstring(signal, 'Int16')
fs = spf.getframerate()
fft_out = fft(signal)

Time=np.linspace(0, len(signal)/fs, num=len(signal))

plt.figure(1)
plt.title('Signal Wave...')
plt.plot(Time,fft_out)
plt.show()

but considering my input in the mic was 'aaaaaa' it does not seem right.

The vanilla version of Fourier Transform (fft) is not the best feature extractor for audio or speech signals. This is primarily due to that FT is a global transformation, meaning that you lose all information along the time axis after the transformation.

You need to be familiar with the concept of short-time Fourier transform (STFT). Basically STFT tells you what frequency components exist in your signal at each timestamp. The result of STFT (its squared magnitude, to be precise) is called the spectrogram, which is what people usually visualize. An example of spectrogram from the link above:

You may refer to matplotlib.pyplot.specgram or scipy.signal.stft regarding how to plot a spectrogram in Python.

The most commonly used speech feature (as input for neural networks) is the Mel-Frequency Cepstral Coefficients, or MFCC, which carry the similar semantic meaning as the spectrogram. Other commonly used features include PLP, LPCC, etc which you can google for more details. But directly feeding the result of FT or STFT into a neural network is not the best practice.


Implementing simple linear regression using a neural network I have been trying to implement simple linear regression using neural networks in Keras in hope of understanding how to work in the Keras library. Unfortunately, I am ending up with a very bad model.

Here is the implementation:

from pylab import *
from keras.models import Sequential
from keras.layers import Dense

#Generate dummy data
data = data = linspace(1,2,100).reshape(-1,1)
y = data*5

#Define the model
def baseline_model():
   model = Sequential()
   model.add(Dense(1, activation = 'linear', input_dim = 1))
   model.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['accuracy'])
   return model

#Use the model
regr = baseline_model()
regr.fit(data,y,epochs =200,batch_size = 32)
plot(data, regr.predict(data), 'b', data,y, 'k.')

The generated plot is as follows:

Can somebody point out the flaw in the above definition of the model (which could ensure a better fit)?

Your code works perfectly. The only problem is that the learning of the parameters is not finished. If you try with 10000 epochs this will works but this is way too much for this problem. As a matter of fact you can see that the loss is diminishing very slowly.

Solution : Increase the learning rate.

I set the batch size to one because this penalize the convergence speed here. Increasing the batch size is useful when you need to avoid overfitting. Here you want to overfit your data. Moreover I choose to use a simpler update rule with the SGD optimizer. With those changes, you will see that only 4 epochs are necessary to fit perfectly your data.

from pylab import *
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers

#Generate dummy data
data = data = linspace(1,2,100).reshape(-1,1)
y = data*5

#Define the model
def baseline_model():
   model = Sequential()
   model.add(Dense(1, activation = 'linear', input_dim = 1))
   sgd = optimizers.SGD(lr=0.2)
   model.compile(optimizer = sgd, loss = 'mean_squared_error', metrics = ['accuracy'])
   return model

#Use the model
regr = baseline_model()
regr.fit(data,y,epochs = 4,batch_size = 1)
plot(data, regr.predict(data), 'b', data,y, 'k.')


How can i increase the memory of Jupyter? What I have:
I have a data set (35989 rows × 16109 columns) and is unfortunately confidential.
But i receive this error massage:
Unable to allocate 4.32 GiB for an array with shape (16109, 35994) and data type float64

How can i solve this problem?

Assuming you cannot add more memory to your computer (or free up some of the memory), you could try 2 general approaches:

Read only some of the data into memory e.g. a subset of the rows or columns.

reduce the precision of the data from float64 to float32.

From your error, it looks like you are loading data into a numpy array, so somewhere in your code, you would need to add this argument to the array creation step e.g.  np.array(your_data, dtype=np.float32).
EDIT:
I don't think this is a rate limiting problem, or a max_buffer_size issue from Tornado (the library behind Jupyter).
You can try to see if your machine is actually running out of memory by using a tool called htop - just execute htop in a terminal (or first sudo apt install htop if it isn't already installed).  That shows the total amount of memory (RAM) available on your machine, it looks something like this:

This example shows the machine has 16 cores and 62.5 Gb memory - 6.14Gb of that is occupied.
Watch that view while you run your code. If the memory bar becomes full before the crash, you know you ran out of RAM.


Spacy tokenizer that strips HTML/XML keeping the positions I'm new to Spacy and infact new to Data Science. I would like to process some XML files for NER and then mark that in the original XML. I would like to know how to tokenize the XML for the NER. I think I should be filtering out the XML tags/code and then feeding the remaining text while retaining positions so that I can get the positions of the NER results in relation to the original XML. This would help me identifying and tagging the exact locations in the XML.
Any ideas and guidance on this would be most appreciated. Thanks

Generally, XML is first parsed. Then, the contents can be analyzed with something like spaCy.
xml.etree.ElementTree is the most common way to parse XML in Python.


How can I avoid requiring global information for performing regression on meter variables? Note: With a meter variable a timestamped value is the sum of all previous differences plus a difference to the most recent value. Think of a electricity meter counting the use of energy.

The goal here is to perform some form of regression (e.g. a Random Forest method) on a data series of a meter variable and then use the resulting model to fix gaps in the data series and possibly do further analysis on the data, for example removing noise from a faulty sensor.

However the possible patterns in the data are likely periodic on timescales smaller than the entire series. Thus we're not interested in modeling the sum of the values. We transform the data by calculating $$\delta_i=\frac{v_i-v_{i-1}}{t_i - t_{i-1}}$$ and then perform the regression on $\delta$.

With the generated model $m(t)$ we can calculate a missing value $v_i$ as $$v_i = v_{i-1} + (t_i - t_{i-1})\cdot m(t).$$

Easy, right?

But now let's throw in some serious noise. Depending on the chosen regression method this doesn't matter much and only makes the predictions worse, but still useful. But what if we want to calculate a missing $v_i$ and $v_{i-1}$ happens to be some bogus value? The model doesn't mind, but for the calculation of $v_i$ we have to assume that $v_{i-1}$ is correct.

Is there a way around that? Is it possible to calculate missing values using only (time-)local information?

Your "missing value" formula mentions $v_{i-1}$, and you seem to be suggesting that you'd prefer not to use that. Ok, fair enough. So use an estimate, $\hat{v}_{i-1}$, which is based on weighted regression of $v_{i-1}$ and several preceding values. Give greater weights to more recent values.

You may wish to implement a classifier that labels each reading as "good" or "bogus". Part of that is straightforward, as you already have a "missing" label for some of your samples. The classifier would additionally discard readings that swing implausibly high, or that show physically impossible behavior like having negative derivative.

You don't mention the generative source of bogus readings. If they are due to noise events on the digital channel used for obtaining each reading, then they don't affect the data source and will soon average out. Here is another technique. Suppose you're willing to suffer higher readout latency in exchange for numeric accuracy. Delaying by one sample would let you compute $\hat{v}_{i}$ as median of three values: $v_{i-1}$, $v_{i}$, $v_{i+1}$. Similarly, you could delay by two and find median of five values, or by three with median of seven values.


Deep Learning: Does starting the training on a smaller subset of the data make sense? I trained a deep neural network with a small subset of my data, which allowed me to go through many epochs in a short amount of time and allowed the model to perform reasonably, then I gave it the entire data set (10X more data) and it improved further. When I give it just the entire data set the model seems to perform terribly, but also it was difficult to reach many epochs due to the extra time needed for training.

My question is: does it make sense to "warm up" the training a large deep neural network with a smaller subset of the data, or should one always just provide the full data?

To add a bit more detail: I had a heavy class imbalance. The model always predicts 0 when I give it the full data set, but when I overfit on a smaller training set, it learns at least to not always predict 0. I notice this guide in step 5, suggests something similar to what I did, but I am not sure if it is theoretically sound or just a nice way of checking the model "can work given enough epochs".

It makes the most sense to train a deep learning model on the entire dataset. If you train on a subset of training, you are more likely to end up a local minimum.
One option is to adjust the size of the batch for stochastic gradient descent (SGD). The batch size is the number of data points given to the model before performing a learning update.


Forecasting Foreign Exchange with Neural Network - Lag in Prediction I have a question regarding the use of neural network. I am currently working with R (neuralnet package) and I am facing the following issue.
My testing and validation set are always late with respect to the historical data. Is there a way of correcting the result?
Maybe something is wrong in my analysis

I use the daily log return

I normalise my data with the sigmoid function (sigma and mu computed on my whole set)
I train my neural networks with 10 dates and the output is the normalised value that follows these 10 dates.

I tried to add the trend but there is no improvement, I observed 1-2 days late. My process seems ok, what do you think about it?

A couple of ways to improve your design:

Consider a different normalization: The sigmoid function will attenuate large moves.  It is likely precisely these large non-linear moves that attracted you to using neural networks in the first place.  Why remove them?  A simple whitening of the data may be better
As pointed out by Nima, your model can only predict what is knowable from the data you give it.  If you are only fitting on data using historical prices, it will only give you what is predictable from them.  Things such as news events / earnings surprises / holidays / option market flows will not be fed into your model.  Consider adding these series as well.
Lots more historical data.  Neural networks often require very large sample sizes because they are trying to estimate a very large parameter space.  Although, more data doesn't always imply more information, this will still likely help.
Experiment with different network architectures.  The number of layers / size of the layers / different gradient decent algorithms / different activation functions / dropout etc..


How to train a model to predict a time window than an event will occur on a website I have a list of historical timestamps of when a specific event occurred on a website. Currently the timestamp represents a 30 minute window that the event happened within. I am looking to train a model that will predict a time window that the given event is most likely to happen within.

The different events I am trying to predict can have various frequencies such as events that occur every day, week, month, quarter, year. To start off I am interested in looking at events that happen on a daily basis since I can have the most immediate feedback.

For example, I would like to predict when a website will update a given report on their page, I know that this typically happens once per business day. I have a process that will go to the website every 30 minutes to check and see if the report has been posted. Therefore my dataset is a list of timestamps that represent the end time of a 30 minute window the report was published in.

I would like to speed up my process in for a time window the event is most likely to occur so I can receive the report as quickly as possible, without speeding up the process 24 hours per day.

What would be the best way to attack this problem? The prediction will need to adjust every day based upon a new time from the day before.

You can use TICK (Telegraf, Influx DB, Chronograf and Kapacitor) stack for the same. You need to install the above mentioned stack on your web server. Chronograph is a visualization tool which gives you information about your server.

Morgoth is a unsupervised learning algorithm which can help you achieve your purpose. You need to combine Morgoth framework with Kapacitor so that the event is triggered once you have an anomaly


Why is the code provided in this book mostly commented out? I am new to text mining and have been playing around with the code provided in the book Applied Text Analysis With Python. I came to a problem with this specific part:

https://github.com/foxbook/atap/blob/master/snippets/ch08/oz.py

In this python script, most of the code is commented out. Is there a specific reason why someone would do that? What's the correct way to uncomment it?

I am using jupyter notebook, I pasted the code in a cell and deleted one by one each '#', but I think I messed up the indentation.

Welcome to the site!
It looks to me that the code you reference is just showing some example usage for the functions defined above it. There are several reasons someone might do this--sometimes it's so auto-documentation generating code (e.g., doxygen) can pick up the example code for downstream formatting, other times it's so that the reader can identify the code they might need to modify for their own use case. I suspect the latter here. If you're interested in trying to just run the code as written, you could copy it into a text editor like TextMate or VSCode and do a vertical select (usually shift+option+click/drag) to highlight and delete the comment characters!


Extract relevant vocabulary from a document I am training a DSSM model for QnA. I have 200 queries and their correspondent answers - the answer is answering what kind of information should an article related to the query contained. E.g:

Title:
literacy rates Africa

Description:
What are literacy rates in African countries

I have trained my model with the whole vocabulary but validation has no lead to great results. By the whole vocabulary I mean a list containing all the words used as I thought that eliminating prepositions, conjuctions and so on may lead to a loss in semantic meaning.
Now I am trying to find a way to extract the more relevant vocabulary of my documents. I have done some research and I have thought of n-grams. In fact there is a similar example in CNKT where the vocabulary they use for the answers is formed both for single words and n-grams but I couldn't find the way to do it by myself yet.
So far I have found a way to do the n-grams but this is not what I want as for example in the sentence:

the cow jumps over the moon

I get the following code:
the_cow_jumps
cow_jumps_over
jumps_over_the
over_the_moon

When I would be interested in the 4-gram:
cow_jumps_over_moon

Keep in mind that even eliminating the articles (or stopwords) I would still be getting more than one n-gram which is not what I want as my main objective is get the final vocabulary to train my model.
As an example of what I want could be the words:
book
book_character
book_editions_published
book_subject
books_published

Instead of:
book
character
editions
published
subject

Not sure if this is what you are looking for. Couldn't understand your question properly do you want 4-grams then this is the way you could do it. Assuming you have cleaned your text file by eliminating articles.

from nltk import ngrams
file=open('abc.txt','r')
txt = file.read()
n = 4
fourgrams = ngrams(txt.split(), n)
for grams in fourgrams:
  print (grams)

In case your file abc.txt contains "This is random text to demonstrate the use of n-grams"

OUTPUT:
('This', 'is', 'random', 'text')
('is', 'random', 'text', 'to')
('random', 'text', 'to', 'demonstrate')
('text', 'to', 'demonstrate', 'the')
('to', 'demonstrate', 'the', 'use')
('demonstrate', 'the', 'use', 'of')
('the', 'use', 'of', 'n-grams')

If you do not want repetition then

words = 'This is random text we’re going to split apart'
x=[]
for word in words.split():
    x.append(word)
    if len(x) == 4:
        print(x)
        x=[]
print(x)

OUTPUT:
['This', 'is', 'random', 'text']
['we’re', 'going', 'to', 'split']
['apart']


Output of evaluation metric for XGBoost - is it cumulative? On the 10th boosting round for XGBoost, I get an MAP of 0.32 on the test data. Does that reflect the performance of just that 10th tree? Or the performance of all 10 trees combined that have been created so far?

That reflects the performance of the boosting of the 10 trees together.
I do reccomend you this tutorial https://www.youtube.com/watch?v=sRktKszFmSk&t=4s
Gradient boosting is well explained there.
Using as a predictor only the 10th tree will give you weird results, since it is fitted on the residual errors of the previous 9


On interpreting the statistical significance of R squared I have performed a linear regression analysis to two series of data, each of which has 50 values. I did the analysis in SPSS and as a result got a table which says that my adjusted R squared is 0.145 and its significance is 0.004.

Being 0.004 < 0.05, I assume my adjusted R squared is significant.

1) Does it mean my adjusted R squared is credible?

2) What does happen if you get a significance which is > 0.05? Does it imply the adjusted R squared can be trusted with credibility but also that the two datasets are not or poorly correlated?

The p-value is the strength of evidence against the null hypotheses. In this case the null is that the coefficient is equal to zero. So your p-value says that this is very weak evidence against the null so you model is likely to be describing the underlying system of the data.

R-squared describes the percent of variation that is explained by the model. Your value is very low; 14.5%. Of all the "activity" in the data your model is only explaining 14.5% of it.

So you have a situation were model is most likely explaining variation in data but not explaining very much of it.  I would suggest altering the model and refitting.


Convolutional Neural networks Hi all: I have a very fundamental question on how CNN works.

I understand fully the training process as to take a bunch of images, start with random filters, convolve, activate, calculate loss, back propagate and learn weights. Fully understood.

But once the training is done, the last convolution layer has the most complex and complete features like faces, ears, wheels and such filters that can get activated by full features.

If that is so, during testing, do we need to pass our images through all the layers again? Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated? And pass that on to the fully connected layer for classification?

I understand there might me inconsistencies in the layers and the inputs but except that anything more important?

Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated?

The answer is that all the layers are fully dependent on the exact features of the previous layer. The last layer is simply not capable of taking a raw image as input and outputting meaningful values. Most likely it is not even going to be able to fit to the shape of the output, it will require dozens of channels, where a colour image only has 3. If you somehow forced it to fit, then the colour image data is meaningless to it. It might still be made to output something, but it would essentially be gibberish, and unrelated to the desired outcome.

Each layer is a function of the previous layer. Ignoring the details of convolution, a neural network is essentially a composition of multiple functions (let's call them $f, g, h, i, j$ for example) so that:

$$y = j(i(h(g(f(x))))$$

You are essentially asking here, can you just do $y = j(x)$ instead of running all those functions in sequence. And the answer is no.


How to build a classification pipeline that will pass to another model? Not sure if the title explained it, but I am trying to build a pipeline where it's like a decision tree, but also not. Say for example, I had a picture. The model classified the picture, but now I want to send it to another CNN. How could I create something like this? Sorry if it's not detailed, still figuring it out!

You can use Microsoft Azure, to make two different pipelines. If I am not wrong, I understand that you want to compare the two methods, using a pipeline, therefore Microsoft Azure ML Studio is an easy and quick method to test your ideas.


Calculating the average of gradient decent I am currently studying the backpropagation process and gradient decent algorithm form the book Neural Networks and Deep Learning written by Michael Nielsen and 3Blue1Brown channel in YouTube.

My question is about calculating the gradient in gradient decent algorithm(the whole dataset as input).

I have drawn a picture that shows my understanding about how the algorithm works:

For example we have 1 million hand written digit images and through the first iteration we feed the network with this 1 million images. Then the gradient is calculated for each images, summed together and averaged before updating the weights.

If my understanding is not wrong, this is the same thing that I saw in 3Blue1Brown channel. In this process the average of gradient is calculated with respect to the cost of each image and not the average cost of the whole data set(1 million) in one iteration, so the formula for calculating the cost has no effect here, rather its derivative is used for calculating the gradients for each image and we do not take the average of costs here.

First I want to know if this is a correct picture about how one iteration of calculating gradient decent works, second, why don't we take the derivatives of average costs with respect to the weights and biases and then take the average sum of all gradients?

And the last question, how the number of iterations and epochs can be assigned here? can we say number of epochs is always equal to the number of iteration since the whole data is used for each iteration?

Starting from the last part, as the entire dataset is used, number of epochs(run over entire dataset) equals number of iterations. Instead, one can do the calculation in "mini batches" (of 32, for example), then the run over each 32 samples is called an iteration.

As for the rest of the question, you can chose a batch that is equal to the entire dataset - this is called "batch gradient descent"; or update after every single sample (a batch size of 1) which is "stochastic gradient descent". Any other choice is called "mini-batch gradient descent.

Deep Learning course on Coursera offers a relatively better explanation of these matters compared to Nielsen's book or 3B1B videos. You can watch the videos for free. In particular here is the video on  Gradient Descent.


Explaining what a neural network is learning In more traditional statistical learning methods, such as logistic regression, the coefficients on the pre-selected features can be interpreted as increasing or decreasing the log-odds of success. For example, say we want to use logistic regression with a single predictor variable studying_hours to predict whether a student will pass the exam. We can estimate the parameters in the model:

$log(Y) = \beta_0 + \beta_1$studying_hours

where $Y = 1$ if the student passes the exam and
$Y = 0$ otherwise, and

studying_hours is a variable with values 1, 2, ..., 100 hours.

We can say that a one-unit increase (a one hour increase) in the number of hours studying for the exam increases the log-odds of passing the exam by $\beta_1$.

It stands to reason that we should be able to make a similar argument when the features are learned instead of pre-selected. But then we would need to understand exactly what the learned feature is. In my example above, instead of pre-selecting studying_hours to be the feature we want to use to predict whether a student passes the course, we would learn some informative feature instead.

I've heard of Hinton diagrams, and of visualizing the feature map for CNNs, but I don't think either of those techniques really works here. A Hinton diagram would just tell me the magnitude and sign of the weights. A feature map would not really be informative. I don't expect to understand what all of the features are, because the whole point is that the algorithm can do a better job of learning features than I can do designing features. But even so, at least some of the features should be human-intepretable, like a 'straight line detector' in a CNN.

How can we interpret the features being learned by a multilayer perceptron for logistic regression?

Not really sure what to tag here, so please, if there's an edit you would recommend, go ahead!

EDIT

This is my attempt at starting an answer, but clarification/other opinions would be appreciated.

Let's take a case of binary classification, and start from the simplest possible case: one scalar input, one hidden unit with a relu activation, and one output unit. Assume the hidden unit has a bias of 0. We can express the function that describes this network as:

$log(Y) = \beta_0 + \beta_1 max\{wx, 0\}$,

where

$\beta_0$ is the bias on the output unit
$\beta_1$ is the weight on the output unit
$w$ is the weight from the input to the hidden unit

Then we can expand that function to:

$log(Y) = \beta_0 + \beta_1wx$, if $|wx| > 0$

$log(Y) = \beta_0$ otherwise.

Which means that if $|wx| > 0$, then the hidden unit is giving us a factor by which to multiply the slope of the line separating the two classes. We can say that a one-unit increase in $\beta_1 w$ implies a one-unit increase in the log-odds of success.

If $|wx| <= 0$, then the best separating line between the two classes has 0 slope, regardless of the magnitude of the weight (I think this is related to the dying relu problem).

If we expand this to include a bias on the output, we have:

$log(Y) = \beta_0 + \beta_1(wx + b)$, if $|wx + b| > 0$

$log(Y) = \beta_0$ otherwise.

Through arithmetic, that first part can be expressed as:

$log(Y) = \beta_0 + \beta_1b + \beta_1wx$, if $|wx + b| > 0$

which means to me that the weight $w$ from the input to the hidden unit tells us both how to modify the y-intercept (which is now $\beta_0 + \beta_1b$) and how to modify the slope.

Does that make sense?

That's a popular question, since many use cases require to understand what's going on in the model.

I suggest to do just some simulation with the resulting NN. Like in your example you could have a test-dataset and den modify the 'studying_hours' by +5%, +10%, +15%, ... and then plot the result over this modification factor. This way you can see by an simulation what the NN is thinking of the 'studying_hours' and how it influences the result.


Keras fit_generator() for long signals I want to make an LSTM network and I have quite a long signal that I want to use as my training data

My X_train is a CSV-file which contains 12 signals with a length of 54 837 488
My y_train is an array that contains a One Hot encoded signal (9 categories) with a length of 54 837 488

If I try to upload the CSV-file as a data frame or as an array in Python I exceed the limit of RAM, so my thought is to use fit_generator(). I have used this before when making image models but then I just used some prebuild generators, but I can't really find any prebuild generators for signals so I decided to try to make one myself.

def generate_data_to_model(y_train):
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield (x, y_train[line])

model = Sequential()
model.add(LSTM(32, input_shape=(1, 12)))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(9, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer="Adam", metrics=['acc'])

model.fit_generator(generate_data_to_model(y_train),
                    steps_per_epoch=1, epochs=2, verbose=1)

When I start training I get this error:

<ipython-input-29-33b2195fea44> in generate_data_to_model(y_train)
      8                 x= x.reshape(1,1,12)
 ---> 9                 yield (x, y_train[line])

  only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices

I found a solution to this problem with some good help from this article.
What I found really convenient is to make two generators. One for features, or X, and one for labels, or y.

def generate_X():
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield x

def generate_y():
    while True:
        for i in range(len(y_train)):
            y= y_train[i]
            yield y

And then I go ahead and use these two generators as input to my third generator which will make batches of a desired size to the fit_generator()

def batch_generator(batch_size, gen_x,gen_y):

    batch_features = np.zeros((batch_size,1, 12))
    batch_labels = np.zeros((batch_size,9))

    while True:
        for i in range(batch_size):
            batch_features[i] = next(gen_x)
            batch_labels[i] = next(gen_y)
        yield batch_features, batch_labels

And last, but not least I am now able to use the batch_generator when training my model

model.fit_generator(batch_generator(128, generate_X(), generate_y()),
                    steps_per_epoch=(len(y_train)/128), epochs=2, verbose=1)


Extract key phrases for binary outcome I have a set of phrases that lead to a binary outcome (accept/reject) and I was wondering what techniques are most helpful for extracting key phrases that are most likely to determine the outcome, given that I have a training set of data that has the English-language phrase and the observed outcome.
To illustrate the idea let me give a simple example:
Accepted

Sounds great
That would be great
That's fine

key words: great, fine
Rejected

I'm not sure
I don't think so
No way

key words: not, don't, no

There are a variety of techniques that you could use, depending on what you would like to do.
If your goal is to gain insight into the phrases that are being used in each group, then I'd recommend looking for the most frequent N-grams of different lengths that appear in each class. Here is a related stackoverflow question that shows how you can use nltk and sklearn to extract these.
If your goal is to predict the outcome (accept/reject) given a phrase, then I'd recommend setting this up as a binary classification problem. Since those phrases are quite short, you could start with a Bag of Words approach - the scikit-learn documentation for working with text data is a good example that guides you through the steps.


what happens to the depth channels when convolved by multiple filters in a cnn (keras, tensorflow) I have a $15$-channel time series that I want to convolve using a $1$d CNN ($1\times n$ time-steps kernel). Now, let's say I want to have, as my first layer, $16$ filters. This would imply to my mind that the output would have a depth of $16 \times 15 = 240$, because each filter would be applied to each channel independently. However when I implement this in keras, (using Sequential) the filter dimensions in the summary do not reflect this. Here is a code fragment:

TIME_RANGE = 31
NUMBER_OF_CHANNELS = 15

model = Sequential()
model.add(Conv1D(input_shape = (TIME_RANGE, NUMBER_OF_CHANNELS),
                    filters=16,
                    kernel_size=7,
                    padding='valid',
                    data_format='channels_last'))
model.add(ReLU())
model.add(BatchNormalization())

model.add(Conv1D(   filters=32,
                    kernel_size=5,
                    padding='valid'))

and here is the corresponding summary output:

Layer (type)                 Output Shape              Param #

=================================================================

conv1d_1 (Conv1D)            (None, 25, 16)            1696

re_lu_1 (ReLU)               (None, 25, 16)            0

batch_normalization_1 (Batch (None, 25, 16)            64

conv1d_2 (Conv1D)            (None, 21, 32)            2592

as you can see output's shape along the time-wise axis decreases as expected due to the no-padding argument, from $31$ to $25$ to $21$, but the depth just reflects the number of filters-- so where have all my channels gone? At this point in the architecture I was expecting a depth of $15\times 16\times 32 = 7680$. It seems an implicit $1\times 1$ convolution is occurring somewhere, which I don't think I actually want-- I'd like to do my $1\times 1$ convolutions later on in the network. So what am I missing here?

When you have a image dimension of HxWxC and you apply a convolutional layer with filter size F and kernel size KxK, you will be sliding F different KxKxC kernels over your image, resulting in the output dimension of the total operation being H'xW'xF which is what you are seeing.


Can you combine two xgboost models into one? If you have built two different xgbost models, with say 100 trees each, is it possible to combine into an xgboost model with 200 trees?

No: the trees' results are added together to produce the final score, so combining two models would produce outputs roughly twice as large as desired.  (Gradient boosted trees change the target labels being fitted by each tree, so the 101st tree has "reset" the targets when training.)


Difference between Factorization Machines and PCA? Factorization Machines (FMs) are a means to express the high dimensional data into lower dimensions, despite the original data being sparse.

How is it different from PCA which itself is a dimensionality reduction technique?

Are there pros-cons of either approach?

The main difference is that PCA is a dimensionality reduction technique, while Factorization Machines are classifiers. You can use PCA to simplify/compress a given dataset, while you can use FMs to classify your observations.

The other difference is that PCA is a linear technique, while FMs are non-linear. PCA extracts later factors that are linearly associated with your input variables, while FMs, on the other side, can "learn" non-linear patters in your dataset.


Can we use a neural network to perform arithmetic operation between 2 numbers? How to develop a neural network which can perform subtraction?

It is an easy task. Make some training data, two inputs and one output. this is a regression task. You can train a simple MLP or you can employ RNNs such as LSTM for this task. Take a look at here.


Linear vs Non linear regression (Basic Beginner) So my doubt is basically in Linear regression, We try to fit a straight line or a curve for a given training set. Now, I believe whenever the features (independent variable) increases, parameters also increases. Hence computing these parameters is computationally expensive. So, I guess that's the reason we move to Non linear!?  Is my understanding right?

And my next doubt is, in overfitting for linear regression, we say that the model memorizes. What I understand is that the parameters are adjusted in such a way that it can only predict for the given input and will give bad results for output
 And the reason is because we haven't fitted a generalized line or curve. But we are plotting a line or curve that passes through all output values. Is my understanding correct?

Whether you use a linear model or a non-linear model depends completely on whether or not the data is linearly separable, not the number of features. If you have a dataset that is linearly separable, i.e a linear curve can determine the dependent variable, you would use linear regression irrespective of the number of features. Since real-world data is rarely linearly separable and linear regression does not provide accurate results on such data, non-linear regression is used.

And to answer your second question, yes you are correct. Overfitting occurs when your model has a very high accuracy for training data but a considerably low accuracy for validation/test data. It happens because the parameters are adjusted completely based on your training set, so there is not a generalised curve. Hence on newer data, the model would perform badly.


What will be the policy if the state space is continuous in Reinforcement learning I have started recently with reinforcement learning. I have few doubts regarding the policy of an agent when it comes to continuous space. From  my understanding, policy tells the agent which action to perform given a particular state. This makes sense when it comes to the maze example, where the state space is descrete and limited. What if the state space is continuous, will the agent have information of every possible state in the state space?
Also will an RL agent be able to take decision if its in a new state that it has not encountered during training ?

You can still define state value functions $v(s)$, action value functions $q(s,a)$ and policy functions $\pi(s)$ or $\pi(a|s)$ when the state $s$ is from a very large or continuous space. Reinforcement Learning (RL) is still a well-defined problem in that space.

What becomes harder is iterating through the state space. That rules out two simple approaches:

Tabular methods - that store lists of all states with the correct action or value.
Any method that needs to iterate through all states, e.g. the dynamic programming methods Policy Iteration or Value Iteration.

These are important methods for RL. With tabulation and assuming you can iterate through all possibilities, then you can prove that you will find the optimal policy.

However, RL methods can still work with large state spaces. The main method to do so is to use some form of function approximation, which then generalises the space so that knowledge learned about a single state is used to assess similar states.

Function approximation can simply be discretising the space to make the numbers more manageable. Or you can use a parametrisable machine learning approach, such as neural networks. The combination of neural networks with reinforcement learning methods is behind the "deep" reinforcement learning approaches that have been subject of much recent research.

If you use any function approximation with RL, then you are not guaranteed to find the most optimal policy. Instead you will find an approximation of that policy. However, that is often good enough for purpose.

To answer the questions more directly:

  What will be the policy if the state space is continuous in Reinforcement learning

There is no change at the theoretical level. You can express the policy as $\pi(s)$ for a deterministic policy, or $\pi(a|s)$ for a stochastic policy, regardless of the space of $s$.

At the implementation level, you will need to implement a parametric function that takes $s$ as one of its inputs. The function parameters $\theta$ are what is learned. For instance if you use an action value based method such as Q-learning, then you will create an approximation to $Q(s,a)$ - in the literature you may see this directly represented as $\hat{q}(s,a,\theta) \approx Q(s,a)$

Using a neural network for $\hat{q}(s,a,\theta)$ is one common way to achieve this, where the neural network's weight and bias values are in $\theta$.

  What if the state space is continuous, will the agent have information of every possible state in the state space?

Depends what you mean by "have information". The agent cannot possibly store separate data about each state. However, it may have information about similar states, or store its knowledge about states in a more abstract fashion (such as in the parameters $\theta$)

  Also will an RL agent be able to take decision if its in a new state that it has not encountered during training ?

Yes. For this to work well with function approximation, it relies on successful generalisation between similar states. So it is important that the state space representation works towards this. For instance, if two states are close together in the state space representation you use, it should be expected that value function and policy functions are often similar - not always, the function can have arbitrary shape, but trying to learn effectively random mapping would be impossible.


Can't reproduce results from GridSearchCV? I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with GridSearchCVs best results.
clf = KNeighborsClassifier(n_neighbors=15, weights='uniform')

clf.fit(features_train, labels_train)

print('Score using optimized parameters: {}'.format(clf.score(features_test,       labels_test)))

params = {'n_neighbors':[1,10,15,20,25,30,35,40,45,50,60,70,80,90,100],    'weights':['uniform', 'distance']}
grid = GridSearchCV(clf, params, cv=10, )
grid.fit(features_train, labels_train)

print('Optimized Parameters:{}'.format(grid.best_params_))
print('Best Score from GridsearchCV parameters{}'.format(grid.best_score_))

Output:

Score using optimized parameters: 0.928
Optimized Parameters:{'n_neighbors': 15, 'weights': 'uniform'}
Best Score from GridsearchCV parameters: 0.962666666667

The score from your GridsearchCV is biased.  You can use cross-validation either for estimating accuracy, or for choosing hyperparameters; but not both.  If you use cross-validation to pick the best choice of hyperparameters, by measuring the accuracy of each possible option, the accuracy you got for the option you chose will tend to overestimate the accuracy you'll see on the test set.

To avoid this bias, select a separate hold-out validation set for estimating the accuracy of your selected parameters, or use nested cross-validation (or a scikit-learn Pipeline).

See https://datascience.stackexchange.com/a/17835/8560.


Interpreting decision tree results after target encoding I am not sure how to interpret the results of my decision tree after I had used target encoding, could someone clarify? The example below doesn't need target encoding just for explanation of my confusion here.
For instance I am trying to classify if a fruit is rotten or not given its age and fruit type. I use target encoding for the fruit column:

I then get the following decision tree with default sklearn decision tree classifier parameters:

I believe after encoding I have lost information about fruit type and I can only say that if fruit_target <= 0.841 then the fruit is rotten if smaller, else not rotten. But then how do i interpret 0.841; what does it mean?

I believe after encoding I have lost information about fruit type and I can only say that if fruit_target <= 0.841 then the fruit is rotten if smaller, else not rotten. But then how do i interpret 0.841; what does it mean?

Recall what the target encoding actually is in this example: it is the share of rotten fruits per fruit type, e.g. $75 \%$ of data points with fruit == pear are estimated to be rotten (I say "estimated" because it depends on the type of target encoding whether this an exact number or an estimate).
Accordingly, you can infer from the decision tree that a data point will be classified as rotten iff its fruit type has more than $0.841 = 84.1\%$ rotten data points in the training set.


Naive about which Naive Bayes in article Working with the Naive Bayes spam filtering article on Wikipedia
(https://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering)

Is it a binary multinomial equation or a Bernoulli form or some other?

In this 2006 paper, it discusses that there are many Naive Bayes algorithms: Spam Filtering with Naive Bayes – Which Naive Bayes?
http://www.aueb.gr/users/ion/docs/ceas2006_paper.pdf

The paper states that binary Multinomial NB performs best.

If it is not multinomial, what changes are needed to make it so?

First off, yes there are different Naive Bayes algorithms. But they are all based on the same principle, namely Bayes Theorem where the features are assumed to be independent.

Here is a short guide on when to use which for spam detection (or document classification in general for that matter):

Bernoulli Naive Bayes

  Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters as much as if it isn't. For instance, the fact that "viagra" is in the e-mail might mean it's "spam" and the fact it isn't might mean it's "not spam"

Multinomial Naive Bayes with boolean features

  Each e-mail is represented as a binary vector, and in this case, whether a word is present in the e-mail matters more than if it isn't. For instance, if "grandma" is in the e-mail, it might mean that it's "not spam", but if it isn't in the e-mail, it doesn't necessarily mean that it's "spam".

Multinomial Naive Bayes with term frequency

  Each e-mail is represented by how many times each word occurs.

Gaussian Naive Bayes

  This deals with continuous values, so this isn't applicable here.


Adding Validation PyTorch First of all, I'm new in this field and it's my first this kind of work. I'm trying to train EfficientNet (CNN), the code below is working fine, but I can't succeed to add also validation set to the code below. My validation data is val_X and val_y.  I really will appreciate your help.
   def train(net):
        BATCH_SIZE = 64
        EPOCHS = 10
        for epoch in range(EPOCHS):
            for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs = net(batch_X)
                matches  = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)
                loss.backward()
                optimizer.step()
            print(f"Epoch: {epoch}. Acc: {round(float(acc),2)}  Loss: {round(float(loss),4)}")

    train(net)

The loop for the validation data would look very similar to your training loop, but for your validation data you only have to calculate the loss and not backpropagate the error. It would look something like this:
def train(net):
        BATCH_SIZE = 64
        EPOCHS = 10
        for epoch in range(EPOCHS):
            # training loop
            model.train()
            for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs = net(batch_X)
                matches = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)
                loss.backward()
                optimizer.step()

            print(f"Epoch: {epoch}. Training acc: {round(float(acc),2)} Training loss: {round(float(loss),4)}")

            # validation loop
            model.eval()
            for i in tqdm(range(0, len(val_X), BATCH_SIZE)):
                batch_X = val_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = val_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                outputs = net(batch_X)
                matches = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)

            print(f"Epoch: {epoch}. Validation acc: {round(float(acc),2)} Validation loss: {round(float(loss),4)}")


Is it correct to use non-target values of test set to engineer new features for train set? Suppose, I have a dataset with a feature_1 value and a target value. Now, I want to engineer a new feature by creating relative value by subtracting mean from each value.

Question: Can I (1) use feature_1 value of test set to calculate mean or (2) should I use only the train set values?

If (1) is correct than I can use the same mean for test set and train set by calculating the mean of feature_1 for all dataset. I'm not sure it's legal, because here we use information from the test set in the train set. On the other hand, we don't use target value, so it might be ok.

If (2) is correct, then, I suppose, we can't use test to calculate the mean for train set, but we can use train set feature_1 values to calculate mean for test set. But then train and test sets' means might be different and influence the correctness of the model for test set. I could use the mean of train set for test set, but again I'm not sure it's correct.

It might be irrelevant for a large dataset, because influence of each value for mean is negligible, so I suppose (1) is ok her. But if I have very small dataset of, say, less than 30 samples or, if e.g. I want to generate new feature by calculating relative value of feature_1 for each category in some categorical feature_2 by calculating mean of all the samples belonging to the same category. Then, it might turn out to be just a few samples in some category of feature_2, so that each sample would influence mean greatly.

Definitely (2). You should not include the testing values when calculating features over the data (or normalizing or scaling the data, etc).

Let's take a step back for a second. Why are we holding out a test set at all? Well, we're training a model on a dataset so that in the future, when we collect more data from the same domain, we can use our model to predict the target for that new data. We don't just want to do as well as possible on the data that we have, we already have the labels for it. So we hold out a test set to measure the performance on, so that we can get a sense of how well the model's predictions generalize to new data.

If we give the model information about the test set (which putting the test samples into a mean calculation would do), we'd improve the model's performance against that particular test set -- but presumably not against any new data that was collected later, when the model was actually deployed and in use! All we'd be doing is making our performance metrics less effective measures of whether or not the model generalizes.

So even while the performance would appear better, it would both possibly not reflect an actual improvement (depending on the actual distributions of train, test, and post-deployment collected data), and would harm our ability to say for sure whether our choices in model design improved our generalization performance at all.

I actually talked about this a little bit with respect to feature scaling in this answer.


How to generate training data for OCR I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to generate the training data by convolve  the image of license plat letter with kernels (somethings like gaussian blur,box blur) using python, but it does not resemble real data, here is the font look like:

and I want to generate individual letter, that look like this:

Do you have any idea how to generate training data like above picture, ?
thank you

That's a very good question. I am facing a similar issue with different data, but my research led me to this which should be pretty much what you are looking for.