Strategies for automatically tuning the hyper-parameters of deep learning models I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and suggestions/critiques on my proposed strategy below. I realise that experimentation may be the best way forward but I don't want to unnecessarily duplicate somebody else's work.

I know that a popular approach is to assume that the function of model error wrt. HPs is a bayesian process, the parameters of which are updated as new points are evaluated (ie. an expensive full training of the underlying deep learning model) so that a more rapid convergence on the optimal HPs can be achieved by quickly predicting better candidates for the next HPs to be evaluated than a simple grid search would be able to. Ilija Ilievski et al report more efficient convergence by using an RBF surrogate model (with pySOT) instead of bayesian https://github.com/ilija139/HORD.

Given that changes in some HPs (let's call them cheap) eg. ADAM initial learning rate and learning rate decay won't affect training time wheras doubling some HPs (let's call them expensive) eg. SGD batch size or total number of steps will double training time, does it make sense to approach this problem by trying to quickly refine cheap HPs whilst the object function is cheap(er) to evaluate and then allowing the object function to become more expensive gradually as the expensive HPs inflate?

One way of implementing this startegy might be to hold expensive HPs at relatively low values whilst refining cheap HPs and then allowing expensive HPs to grow and then iterating this process until some precondition is met, ie. model-error no longer goes down or processor-time budget is exceeded.

An alternative and possibly more 'organic' approach that I imagine would effectively achieve something similar would be to give each evaluation of the object function a certain processor time budget that would gradually increase. So when the object function over-ran its budget, it would be (gracefully) stopped and whatever error result that model gave, would be taken alongside whatever HPs were used (with number of steps being the actual number of steps taken, rather than what was requested). The idea being that this would pressure the HP surrogate to rapidly refine the cheap HPs initially whilst allowing the expensive HPs to be refined over time.

Does this make sense?

I like to start with an overfitting set of hyperparameters, train to convergence - usually zero training error - then start increasing the regularization until the test error starts declining. Time is saved by the fact that I never re-initialize, but jack up the regularization parameters in an already-fit network.

My regularization parameters are dropout rate and weight decay. I usually leave one of them fixed and move along a grid of the other. When trying for a given set of hyperparameters, I usually start with large batches (faster) and then move to small batches (deeper).

I also rely on ensembling, combining the prediction of 4 - 64 models trained analogously, maybe with different architectures

Ymmv, perhaps.


How to solve this classification problem: multi-class or multi-label? In a supervised cancer classification task which is given the data containing metrics we want to classify whether the patient has cancer or is at high risk (label 1) or low risk (0). However, there is another attribute which is screening status which I want to include in the prediction. But I don't know how -- should it be a label or an input feature? The information on patients screened have the screening year as the value. Those who were not screened have the value for the year empty. Let the label for is screened be 3 and not screened be 4. So, the idea is:
-- a patient is at high risk and was screened : [1,3]
-- patient is at high risk but not screened: [1,4]
-- patient is at low risk and was screened: [0,3]
-- patient is at low risk but not screened: [0,4]

Basically, I want to assess the importance of early screening in predicting benign/malignant. The patients who have been screened early may not have cancer. On the other hand, it may happen that those who were screened developed cancer in later stages. This implies that the patients who did not pass the screening tests were diagnosed with cancer. So, what are the factors that influence the patient to pass or reject the screening test and whether the screening test itself can help in predicting/classification of malignant/benign. This is the research problem.
Confusion 1: How can I include this screening status? Will the information on screening be included as a label ?
Confusion 2: Will it be multi-label or multi-class classification?
Is my approach correct? Please let me know if I am in the correct direction an please correct me otherwise. Links to papers with similar problem will also be helpful.

So, in regards to your two questions:

In terms of including the screening status as a label: this really depends on what exactly you want your model to do and the applications of this model. If you want this model to help clinicians understand which patients should be screened and which shouldn’t, then of course include this feature as an additional predictor.

In terms of screening status prediction task being multi-label or multi-class: in this case the classes appear to be mutually exclusive. Therefore, this would be a multi-class classification problem. A multi labelling problem is for example predicting which combination of symptoms would appear if a patient experienced a certain disease.


Is it correct to join training and validation set before inferring on test-set? I would like to know if is a correct procedure to join training-set and validation-set together, in order to train the model on this new dataset, before making predictions on the test-set.

Yes, once you optimized your model and parameters with the validation set, it is advised to train your final model on the combination of the training and validation sets before applying it to the test set. Indeed, you can see the validation set as a subset of the training set. It is used to tweak your models and your parameters, but once it is done, it would just be a waste to not use the validation set for training during the testing phase.

Remember that once you tested your model on the test set, you should not tweak it anymore.


Clean up a pandas column using regex I have a Python Pandas dataframe with a column named "codes". This column is very messy and contains the following values:
codes                       output
1234A, C (1AAA1), D, E      1234A 1234C 1234D 1234E
34240 & 1 & 3               34240 34241 34243
PS1A;D;E                    PS1A PS1D PS1E
CE11A, C & 2452D            CE11A CD11C 2352D
EOPSDCE11A, C & 2452D       EOPSDCE11A EOPSDCD11C 2352D

Basically, what we need is to complete the single letter/number (i.e. C) with the full code excluding the last char (i.e. 1234 in the first line), remove the phrase in the parenthesis, keep the code if it's a full code (length between 2-10), and use space as delimiter.
I have tried to use str.replace but was not able to manage to write the regex correctly.
Thanks for the help in advance!

Would something like the following work?
import re
from typing import List

import pandas as pd

df = pd.DataFrame(
    {
        "codes": [
            "1234A, C (1AAA1), D, E",
            "34240 & 1 & 3",
            "PS1A;D;E",
            "CE11A, C & 2452D",
            "EOPSDCE11A, C & 2452D",
        ]
    }
)

def complete_codes(codes: List) -> List:
    line_code = codes[0][0:-1]
    res = []
    for code in codes:
        if len(code) < 2:
            res.append(line_code + code)
        else:
            res.append(code)
    return res

def clean_up(text: str) -> str:
    text = re.sub(r"\(.*\)", "", text)  # Remove parenthesis
    multi_split = re.split(r",|;|&", text)  # Split text
    multi_split = [item.strip() for item in multi_split]
    codes = complete_codes(multi_split)
    res = " ".join(codes)
    return res

df["output"] = df.codes.apply(clean_up)
print(df)

Which outputs:
                    codes                       output
0  1234A, C (1AAA1), D, E      1234A 1234C 1234D 1234E
1           34240 & 1 & 3            34240 34241 34243
2                PS1A;D;E               PS1A PS1D PS1E
3        CE11A, C & 2452D            CE11A CE11C 2452D
4   EOPSDCE11A, C & 2452D  EOPSDCE11A EOPSDCE11C 2452D


How would one separate digits for number recognition? Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px.

How would one split an image of unknown size and an unknown amount of digits into a series of 28x28px images to feed to this network? (The order of the digits must be obtainable.)

For example:

How can this be split up to:

Assuming that there are not always 5 digits and the initial image is not always the same size.

Firstly, my thoughts would be to create a secondary neural network. This neural network would output an (x, y) coordinate. This could be used to crop the image to 28x28px about this coordinate. However, this neural network would only be able to locate 1 digit at a time.

Secondly, another idea is that a series of random crops could be performed and then all given to the number recognising neural network. However, this would create a high error rate and the number recognising neural network has no way of telling that there was no valid number given (Unless that output was added). But more importantly the order of the digits would/could be lost.

I am struggling to find any resource explaining a possible solution. Google's house number recognition takes the entire image of digits and returns a value. This can be found in How Google Cracked House Number Identification in Street View and Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks.

In Image Processing, this task is known as localization. You basically want to localize each digit in the image and then use your digit recognizer over the digits. A cursory google search for digit localization in images gives me following papers which seem to be very helpful.

Reading Digits in Natural Images with Unsupervised Feature Learning
Character Recognition In Natural Images


Problem with chain rule in softmax layer when differentiated separately I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax is derivated with Cross-Entropy error. Here's an example from a network with a single data-point with 2 output neurons using Softmax as the activation function.
$$ Z^{O} := (1.75, 1.75) $$
$$ A^{O} := (0.5, 0.5) $$
Here $ Z^{O} $ is the layer output before applying the SoftMax activation function, while $ A^{O} $ is the activation - guess - of the network.

If I were to derive the SoftMax activation function with the CE error function would give me:
$$ \frac{\partial{E(A^{O},T)}}{\partial{Z^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(A^{O})}}}{\partial{Z^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(Softmax(Z^{O}))}}}{\partial{Z^{O}}} = A^{O} - T $$
With:
$$ A^{O} = \begin{pmatrix}0.5&0.5\end{pmatrix} $$
$$ T = \begin{pmatrix}0&1\end{pmatrix} $$
This yields:
$$ A^{O} - T = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$

This is fine. Now I need to get the same result when applying the chain rule and differentiating the error function and the activation function separately:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} $$

The first part is trivial:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(A_{i}^{O})}}}{\partial{A^{O}}} = -\frac{T_{i}}{A_{i}^{O}} = \begin{pmatrix}-\frac{0}{0.5}&-\frac{1}{0.5}\end{pmatrix} = \begin{pmatrix}0&-2\end{pmatrix} $$

The second part is of course a matrix of $ Softmax^{'}(Z^{O}) $:
$$ \frac{\partial{Softmax(Z^{O})}}{\partial{Z^{O}}} = \begin{pmatrix}Z^{O}_{0}(1-Z^{O}_{0})&-Z^{O}_{0}Z^{O}_{1}&...&-Z^{O}_{0}Z^{O}_{n}\\-Z^{O}_{1}Z^{O}_{0}&Z^{O}_{1}(1-Z^{O}_{1})&...&-Z^{O}_{1}Z^{O}_{n}\\...&...&&...\\-Z^{O}_{n}Z^{O}_{0}&-Z^{O}_{n}Z^{O}_{1}&...&Z^{O}_{n}(1-Z^{O}_{n})\end{pmatrix} $$

def df2(d):
    def softmax_prime(f):
        jacobian = np.diag(f)
        for i in range(len(jacobian)):
            for j in range(len(jacobian)):
                if i == j:
                    jacobian[i][j] = f[i] * (1 - f[i])
                else:
                    jacobian[i][j] = -f[i]*f[j]
        return jacobian
return np.apply_along_axis(softmax_prime, axis=1, arr=d)

Which for the above $ \begin{pmatrix}1.75&1.75\end{pmatrix} $ yields:
$$ \frac{\partial{A^{O}}}{\partial{Z^O}} = SoftMax'(\begin{matrix}1.75&1.75\end{matrix}) = \begin{pmatrix}-1.3125&-3.0625\\-3.0625&-1.3125\end{pmatrix} $$

Now, according to the chain rule, if I multiply these two together I would get $ \delta^{O} $ for the output layer and it should be the same as the above $ A^{O} - T $. However this gives:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} = \begin{pmatrix}0&-2\end{pmatrix} * \begin{pmatrix}-1.3125&-3.0625\\-3.0625&-1.3125\end{pmatrix} = (6.125, 2.625) $$

Which is different from the the above original calculation. I'm sure I made a mistake, but I can't see where. The deltas must match that I'm sure about, and fair enough if I use the chain rule like this the error will increase for each epoch.

Can anyone tell me where I went off?

I'm just gonna answer my own question here so that in the future I can find it easier, and hopefully it comes handy for others asking about the derivative of Softmax and gets confused by the fact that it's a matrix. So the shown matrix form for $\frac{\partial{Softmax(x)}}{\partial{x}} $ in the opening question is incorrect. The correct formula should be:
$$ \frac{\partial{Softmax(Z^{O})}}{\partial{Z^{O}}} := \begin{pmatrix}\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{0})}}{\partial{Z^{O}_{N}}}\\\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{1})}}{\partial{Z^{O}_{N}}}\\\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{2})}}{\partial{Z^{O}_{N}}}\\...&...&...&&...\\\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{0}}}&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{1}}}&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{2}}}&...&\frac{\partial{Softmax(Z^{O}_{N})}}{\partial{Z^{O}_{N}}}\end{pmatrix}$$
Which is:
$$ \begin{pmatrix}S(Z^{O}_{0})(1-S(Z^{O}_{0}))&-S(Z^{O}_{0})S(Z^{O}_{1})&-S(Z^{O}_{0})S(Z^{O}_{2})&...&-S(Z^{O}_{0})S(Z^{O}_{N})\\-S(Z^{O}_{1})S(Z^{O}_{0})&S(Z^{O}_{1})(1-S(Z^{O}_{1}))&-S(Z^{1}_{1})S(Z^{O}_{2})&...&-S(Z^{O}_{1})S(Z^{O}_{N})\\-S(Z^{O}_{2})S(Z^{O}_{0})&-S(Z^{O}_{2})S(Z^{O}_{1})&S(Z^{2}_{2})(1-S(Z^{2}_{2}))&...&-S(Z^{O}_{2})S(Z^{O}_{N})\\...&...&...&&...\\-S(Z^{O}_{N})S(Z^{O}_{0})&-S(Z^{O}_{N})S(Z^{O}_{1})&-S(Z^{O}_{N})S(Z^{O}_{2})&...&S(Z^{O}_{N})(1-S(Z^{O}_{N}))\end{pmatrix} $$
For which I incorrectly used the df2 function with z1 argument, as it should be:
$$ \begin{pmatrix}A^O_0(1-A^O_0)&-A^O_0A^O_1\\-A^O_0A^O_1&A^O_1(1-A^O_1)\end{pmatrix} $$
Using this, I have:
$$ \frac{\partial{A^{O}}}{\partial{Z^{O}}} = Softmax^{'}(A^{O}) = \begin{pmatrix}.25&-.25\\-.25&.25\end{pmatrix} $$
Finally doing the multiplication:
$$ \frac{\partial{E(A^{O}, T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} = \begin{pmatrix}0&-2\end{pmatrix}*\begin{pmatrix}0.25&-0.25\\-0.25&0.25\end{pmatrix} = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$
Which matches the original:
$$ \frac{\partial{E(A^{O}, T)}}{\partial{Z^{O}}} = A^{O} - T = \begin{pmatrix}0.5&0.5\end{pmatrix} - \begin{pmatrix}0&1\end{pmatrix} = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$
And this is where I messed up.


Can't train input variable with Keras+Tensorflow I have a Keras model and I want to do some cool visiualizations with it. It's an object recognition network.

So I thought, It would be cool to input a blank image into the network and treat the image as the variable and not the weights, and then train the network to always output an icecream for example.

So I wrote the following code:

#loading the model
model = load_model('model.h5')

#create the input image as a variable
w = tf.Variable(tf.zeros([1,224,224,3]))

#create the flowgraph with the variable input
pred = model.call(inputs=w)

#create the desired output distribution
desired = np.zeros((1000))
desired[928] = 1.0

err = tf.reduce_mean(tf.subtract(pred,desired))
lr = tf.placeholder(dtype=tf.float32, shape=None)

#create an optimizer that can only affect the inital input variable I created
optimizer = tf.train.AdamOptimizer(learning_rate=0.0001).minimize(err, var_list=[w])

#train the network
for i in range(0,100):
    _,cost = sess.run([optimizer,err])
    print(cost)

So I thought the code would work well, but the cost literally doesn't change. It stays in place as if it's entirely unaffected.

Try explicitly setting the variable as trainable by setting Trainable=True.
Also after the network is created, check if the variable is in the list of
trainables https://www.tensorflow.org/api_docs/python/tf/trainable_variables


Activation function vs Squashing function This may seem like a very simple and obvious question, but I haven't actually been able to find a direct answer.
Today, in a video explaining deep neural networks, I came across the term Squashing function. This is a term that I have never heard or used. Our professor always used the term Activation function instead. Given the definitions I've been able to find, the two seem to be interchangeable terms.
Are they really synonymous or is there a difference?

Activation functions like sigmoid function, hyperbolic tangent function, etc. are also called squashing function because they squash the input into a small range like in sigmoid function output is in range of [-1,1]. But you cannot call ReLU as a squashing function because for a positive input value it returns the output as same.


Is regression or ANOVA the right approach for determining RRP based on a set of RRPs associated with categorical variables Basically I have scraped some retail site data and I would like to know if this is a regression problem:
If I have data from the website about shoes like RRP, what brand they are, what colour they are and what size and I collate that data to build a model around such that I can give the model known variables about a new pair of shoes (brand, size colour) to determine it's likely RRP based on the set of shoes i already know all the variables and RRP for, should I use ANOVA or regression. My reasonings for regression is that it seems like a classic problem in that price is a continuous variable. But I'm unsure to go about this, a lot of the regression examples i see use contiuous variables for both the predictor and outcome variables maybe ANOVA is better?

ANOVA is a linear regression...a regression on categorical variables.
Regressions can have any mix of continuous and categorical variables. Some, like ANOVA and ANCOVA, have special names that you may want to use when you're communicating with people who have less training in statistics and don't know how to unify them with linear regression.


When the shallow network has a small number of neurons, how it will need more sum-product? I am reading a paper, where I am not able to understand the meaning of the following statement, can someone please help.

a shallow network requires exponentially many more sum-product hidden units
than a deep sum-product network in order to compute certain families of polynomials.

Particularity my question is, when a shallow network has a small number of neurons, how will it need more sum-product?

Shallowness is when the Model has very few Or just one hidden layer.
When it has a lot of hidden layers it will be a Deep Neural Network.
Sum-product is basically a Neuron of a Hidden Layer.

From the paper-
Artificial neural networks with several hidden layers, called deep neural networks, have become popular due to their unprecedented success in a variety of machine learning tasks (see, e.g., Krizhevsky et al. 2012, Ciresan et al. 2012, Goodfellow et al. 2013, Hinton et al. 2012). In view of this empirical evidence, deep neural networks are becoming increasingly favoured over shallow networks (i.e., with a single layer of hidden units), and are often implemented with more than five layers

A single sum-product hidden layer summarizes a layer of product units followed by a layer of sum units.

The exact definition of Shallow and Deep is a moving number depending on the advancement of AI research.
While it is intuitive that if a 3-layered model needs 5-Neurons in each layer then a single Layered model should have 15 Neurons.
But it will actually require more than that
 It's very difficult to get a conclusive answer to this but the two famous reasons are -

Our data has often an underlying Hierarchical nature i.e. Image data
Analogy with Circuit theory (Check Ref#2 and #4)

Practical reasons seem that it worked with GD/Backprop and research moved in that direction. As you may read the listed references and find that None of the experts has a conclusive view on this.
Reference -
1. Deep Learing Book - 6.4 Architecture Design
2. Neural Networks and Deep Learning,  Michael Nielsen
 3. Stanford CS class CS231n
4. Andrew Ng Deep Learning course - C1W4L04


(0.0053726552728412666, 0.9415686658133314, 1), that 0.005 represents the test statistic Yes,
0.94 represents the P-value - Yes.
and 1 the degrees of freedom? No.
And 1,2,3 and 4 are ordinary serial numbers and not, degree of freedom.
At 10% level of confidence, you need at least 2.71(value of Chi-square). You have computed the chi-square = 2.576 and therefore, the null-hypothesis is acceptable. Your assertion  - the null hypothesis can absolutely not be rejected- is valid in this particular case.


What is the output of multivariate LSTM model? I am currently trying to build an LSTM model by using multivariate inputs, but I don't understand what exact output I am predicting.

I am currently using 5 features in the data i.e. 'Time', 'Avg CPU Load', 'F1', 'F2', 'F3' as input data.

X_data = data[['Time', 'Avg CPU Load', 'F1', 'F2', 'F3']]

My main goal is to predict 'Avg CPU load' by using the other features as an input data, but I am not sure if I am doing that.

In these features I have made 'Time' as an index in the dataframe.
Then I have used 60 timesteps in the below code.

new_X_train=[]
new_Y_train=[]
for i in range(len(X_data)-timesteps-1):
    t=[]
    for j in range(0,timesteps):

        t.append(X_data[[(i+j)], :])
    new_X_train.append(t)
    new_Y_train.append(X_data[i+ timesteps,1])

The variables 'new_X_train' and 'new_Y_train' have the inputs data that have to be inserted in the model. Then I define my model which is..

sq = Sequential()
sq.add(LSTM(units = 100, return_sequences = True,
           input_shape = (None, 4)))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100,return_sequences = True))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100,return_sequences = True))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100))
sq.add(Dropout(0.2))

sq.add(Dense(units=1)) # only one output

sq.compile(optimizer = 'adam', loss = 'mean_squared_error')

sq.fit(new_X_train, new_Y_train, epochs = 5, batch_size = 20)

Once the model is trained I load the test data and implement 60 timesteps to i.e I do the same for what I have done for the training data.
And then I use the X_test data to predict the values.

predicted = sq.predict(X_test)

Now I am not sure what output the above code is predicting, like what feature is it trying to predict?
And how to make my model predict 'Avg CPU load' feature?

If you want to predict "Avg CPU Load", your y_train should be only the "Avg CPU Load" column and your x_train should be rest of the columns (you might skip the time column if all timestamps are equally spaced).

The network will then learn a function mapping rest of the columns to "Avg CPU Load" and it shall be predicting the same.

Hope it helps.

edit: in testing phase, also you have to follow the same structure for x_test (and y_test i.e. ground truth).

edit2: In this case, for RNN, you are right by generating each training sample as all columns over 60 timestamps. You can check if the prediction is indeed  what you want by plotting against  ground truth. Although, I think you should  predict all the columns (new_y_train is all columns instead of avg CPU load only) i.e. make the output dimension same as input dimension. Make the network predict the whole thing and pick your column of interest for your consideration. This is probably the more natural way to do it.


Filter unwanted terms I have the following keywords retrieved from a text document.

natural language processing
sequential labeling
programmable
spell checking
techniques
forensics
important issue
categorial grammar
girls
applications

Now I want to remove unwanted words such as programmable, techniques, important issue, girls, applications from this keyword list. Is there a way to automate this? Can I consider some pos-tag pattern to do this?

Please help me!

Not sure what you are exactly looking for. There are many ways to do this. One simple way is like this,

word_list = ['natural language processing', 'sequential labeling', 'programmable', 'spell checking', 'techniques', 'forensics', 'important issue', 'categorial grammar', 'girls', 'applications']

Since you said, a list of key words,

stopwords = ['programmable','techniques', 'important issue', 'girls', 'applications']

resultwords  = [word for word in word_list if word.lower() not in stopwords]
result = ' '.join(resultwords)

print (result)

This will yield,

>> natural language processing sequential labeling spell checking forensics categorial grammar

Otherwise, if you just have a string of text, use split() method to put each word into a list splitting by space.

querywords = word_list.split()

resultwords  = [word for word in querywords  if word.lower() not in stopwords]
result = ' '.join(resultwords)

You can always warp them into a function and automate this. As I said there are many ways to do it, without knowing what have you tried, this is a simple way to do without using nltk library.


Bidirectional Encoder Representations from Transformers in R Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R.

https://github.com/google-research/bert
https://github.com/facebookresearch/XNLI
https://github.com/tensorlayer/seq2seq-chatbot
https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/
https://github.com/huggingface/pytorch-pretrained-BERT и чат ботов
https://stanfordnlp.github.io/coqa/
https://github.com/nyu-dl/bert-gen

You might be interested in the open-source R package RBERT:
https://github.com/jonathanbratt/RBERT

It's a work in progress, but the goal is to be able to use BERT directly in R.


Accuracy doesn't match in Keras I am using Keras to classify images. I am following the Keras blog. The accuracy from  predict_generator is not matching with the accuracy obtained from the confusion matrix, which I am computing using the scikit-learn package. I have included the relevant snippet of the code below

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
import numpy as np
import theano
from sklearn.metrics import classification_report, confusion_matrix
y_actual = np.ones((nb_test_samples),dtype = int)
y_actual[0:2817] = 0

train_datagen = ImageDataGenerator(
             featurewise_std_normalization=False,
             samplewise_std_normalization=False,
     rescale = 1./255)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
                    train_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
                    class_mode = 'binary')

test_generator = test_datagen.flow_from_directory(
            test_data_dir,
                    target_size = (img_width,img_height),
                    batch_size = 32,
            class_mode = 'binary',
            shuffle = False         )

model.fit_generator(
                    train_generator,
                    samples_per_epoch = nb_train_samples,
                    nb_epoch = nb_epoch,
                    validation_data = test_generator,
                    nb_val_samples = nb_test_samples)

score =   model.evaluate_generator(
                      test_generator,
                      4938)
print "Test fraction correct (Accuracy) = {:.2f}".format(score[1])
prediction = model.predict_generator(test_generator,nb_test_samples)

for i in xrange(0,len(prediction)):
    if prediction[i]<0.5:
       prediction[i] = 0
    else:
       prediction[i] = 1

#y_predicted = test_generator.classes
print np.sum(prediction)
CM = confusion_matrix(y_actual,prediction)
print CM

If I use y_predicted, I get a perfectly diagonal confusion matrix, when the console output shows an accuracy of 70% which doesn't make any sense at all. What is that I am doing wrong?

The score method used in keras does not calculate accuracy like the sklearn's accuracy_score method. Check the internal implementation of evaluate method to understand more. If you want to calculate accuracy, I suggest you to use sklearn's accuracy_score by getting the predictions or manually calculate if it is easier for you.


Converting similarity matrix before inputting to t-sne I have a cosine similarity matrix where I want to adjust it to inputto t-sne. I fond the following explanation in a FAQ. As mentioned there I have made the diagonals to zero. what does it mean by symmetrize the pairwise similarity matrix, and normalize it to sum up to one?

  Can I use a pairwise similarity matrix as input into t-SNE?

  Yes you can! For instance, we successfully applied t-SNE on a dataset
  of word association data. Download the Matlab implementation, make
  sure the diagonal of the pairwise similarity matrix contains only
  zeros, symmetrize the pairwise similarity matrix, and normalize it to
  sum up to one. You can now use the result as input into the tsne_p.m
  function.

You need to scale the values by some constant factor so the sum of every entry in the matrix results in 1.0.

You can achieve this by using mat /= mat.sum(), where mat is your matrix.


Unable to understand Kernel Ridge regression I am trying to read kernel ridge regression from this link

But , I am unable to get the intution behind the derivation.

Can anyone please help me ?

Equation (4) simply gives us an identity (without proof):

$$(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} = PB^T (BPB^T + R)^{-1}.$$

Let's check that it does indeed hold. The left hand side equals

\begin{align*}
(P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} &= \left(\left((P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1}\right)^{-1}\right)^{-1}\\
&= \left(RB^{-T}(P^{-1} + B^TR^{-1}B)\right)^{-1}\\
&= \left(RB^{-T}P^{-1} + B\right)^{-1}.
\end{align*}
The right hand side equals
\begin{align*}
PB^T (BPB^T + R)^{-1} &= \left(\left(PB^T (BPB^T + R)^{-1}\right)^{-1}\right)^{-1}\\
&= \left((BPB^T + R)B^{-T}P^{-1} \right)^{-1}\\
&= \left(B + RB^{-T}P^{-1} \right)^{-1}.
\end{align*}
So indeed, the equality holds.

If you take the right hand side of equation (3)
$$\left(\lambda\mathbf{I} + \sum_i \mathbf{x}_i \mathbf{x}_i^T\right)^{-1}\left(\sum_j y_j \mathbf{x}_j\right),$$
replace $\mathbf{x}_i$ with $\phi_i$  and rewrite it as
$$\left(\lambda\mathbf{I} + \Phi\Phi^T\right)^{-1}\left(\Phi\mathbf{y}\right),$$
then it matches the left hand side of the identity in equation (4) (right-multiplied by $\mathbf{y}$), where
$$P^{-1} = \lambda^{-1}\mathbf{I}$$
$$B^T = \Phi$$
$$R = \mathbf{I}.$$

Therefore, we can rewrite the right hand side of equation (4) as

$$\lambda^{-1}\mathbf{I}\Phi(\Phi\lambda^{-1}\Phi + \mathbf{I})^{-1}\mathbf{y}.$$ As $\lambda$ is a scalar, we can rewrite this as
$$\Phi(\Phi\Phi + \lambda\mathbf{I})^{-1}\mathbf{y}.$$


Understand the equations of quantile regression forest (Meinshausen)? I am trying to implement a quantile regression forest (https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf).
But, I have some difficulties to understand how the quantiles are computed. I will try to summarize the part of interest in order to then explain exactly what I don't understand.
Let be $n$ independent observations $(X_i, Y_i)$. A tree $T$ parametrized with a realization $\theta$ of a random variable $\Theta$ is denoted by $T(\theta)$.

Grow $k$ trees $T(\theta_t)$, $t = 1, . . . , k$, as in random forests. However, for every leaf of every tree, take note of all observations in this leaf, not just their average.
For a given $X = x$, drop $x$ down all trees. Compute the weight $\omega_i(x, \theta_t)$ of observation $i \in \{1, . . . , n\}$ for every tree as in (4). Compute weight $\omega_i(x)$ for every observation $i \in \{1, . . . , n\}$ as an average over $\omega_i(x, \theta_t)$, $t = 1, . . . , k$, as in (5).
Compute the estimate of the distribution function as in (6) for all $y \in \mathbb{R}$.

Where the equations (4), (5), (6) are given below.
$$ \omega_i(x, \theta_t) = \frac{ 1 \{ X_i \in R(x, \theta_t) \} }{\text{#} \{ j : X_j \in R(x, \theta_t) \} } \ \ \ (4)$$
$$ \omega_i(x) = k^{-1} \sum_{t=1}^k \omega_i(x, \theta_t) \ \ \ \ (5)$$
$$ \hat{F}(y|X=x) = \sum_{i=1}^n \omega_i (x) 1\{Y_i \leq y\} \ \ \ (6) $$
Where $R(x, \theta_t)$ denotes the rectangular area corresponding to the unique leaf of the tree $T(\theta_t)$ that $x$ belongs to.
I can compute (4) and (5) but I don't understand how to compute (6) and then estimate quantiles. I would also add that I don't know where all observations in leaves (first step of the algorithm) are used.
Can someone give some elements to understand this algorithm ? Any help would be appreciated.

It's a good idea to remember what you're trying to predict and that is: $\mathbb{E}[Y | X=x]$. The simplest estimate is a single tree:
$$
\hat{\mu}(x) = \sum_{i \leq n} w_i(x, \theta) Y_i
$$
with:

$w_i(x, \theta)$ the single tree node weights (equation 4 in the paper)
$Y_i$ your observations

As we all know this is not a great estimate (high variance among others) so he defines a better estimator (random forest) as:
$$
\hat{\mu}(x) = \sum_{i \leq n} w_i(x) Y_i
$$
where the $w_i(x)$ are averages over the multiple trees (equation 5). So what does that mean? Your best estimator for $\mathbb{E}[Y | X=x]$ is $\hat{\mu}(x)$. What do you do if you want an estimation of a function $\phi$ of $Y$? You just transform your observations of $Y$ and use the same formula, i.e.:
$$
\hat{\mu_{\phi}}(x) = \sum_{i \leq n} w_i(x) \phi(Y_i)
$$
which would then be an estimator of $\mathbb{E}[\phi(Y)|X=x]$. If you define $\phi(Y) = \mathbb{1}_{Y \leq y}$ then you get that:
$$
\hat{\mu_{y}}(x) = \sum_{i \leq n} w_i(x) \mathbb{1}_{Y_i \leq y}
$$
is an estimator of $\mathbb{E}[\mathbb{1}_{Y \leq y}|X=x] = F(y|X=x)$.
I avoided a lot of low-level details here but to be clear this relates to the delta method i.e. convergence of a function of estimator. This kind of convergence is not trivial to prove (but I guess that's why they wrote a paper about it)


How to perform node classification using Graph Neural Networks I'm am trying to perform node classification using graph neural network methods. My initial plan was to convert my graphs to adjacency matrices and train my network on that, with the node features being my target.

However, my graphs all have a different number of nodes, so I believe adjacency matrices will not work. I then found information about node embeddings and applications in biology (see here). It infers here that by embedding your nodes it no longer matters about graph size.

What I am wondering, is if using embeddings is actually the solution to my problem? Then, how, practically, do I build a Graph Neural Network out of this method? The link (in the slides for part 2) still says I need to train my network to learn the embeddings - but now I have my original problem, how can I train the network on many different sized graphs?

Doing node classification on small, differently sized graphs seems to be somewhat commonly done. What I am slightly confused by is the step from theory to practise.

Could anyone please offer any advice? Or links to more material that could solve some of my queries?

I'm not sure if that's what you need but this library seems pretty similar to your problem: https://github.com/Accenture/AmpliGraph/


Loading collections of datasets - Python code examples Sometimes you might want to check your ideas on multiple datasets.
There are several places with datasets collections.
Question: Please share some Python scripts how to download multiple datasets from these (or other)
datasets collection ?
Ideally one  should be able to : 1) get datasets list 2) select some desired by conditions 3) download those selected. But if you have something different please share anyway.
For "openml" database - I have a script - see my own answer.
But I do have for other collections : Kaggle, uci ...

Here some examples of datasets collections:
https://www.openml.org/
https://archive.ics.uci.edu/ml/index.php
https://ieee-dataport.org/datasets
Каggle contains lots of datasets, there are also specific collections:
graph collections see list here https://mathoverflow.net/a/359449/10446  ,
many biological data is here : https://www.ncbi.nlm.nih.gov/gds

How to fetch Kaggle data from python code?

Install kaggle package
C:\Users\TalgatHafiz>  pip install kaggle

login to your Kaggle account
click on the icon in the upper right corner -> My Account
Scroll down to API section
Click "Create New API Token"
"kaggle.json" file is created and saved locally

Create ".kaggle" dir
C:\Users\TalgatHafiz>mkdir .kaggle
and move "kaggle.json" into that directory

see all active competitions by running the following command
C:\Users\TalgatHafiz>kaggle competitions list

Select one of the competitions that you signed up for, eg
https://www.kaggle.com/c/contradictory-my-dear-watson/data#
Scroll down. Right before "Data Explorer" section there should be API line:
"kaggle competitions download -c contradictory-my-dear-watson"
copy it

run these commands from the notebook
import kaggle
!kaggle competitions download -c contradictory-my-dear-watson

zipped data file is downloaded into your the same directory where your notebook is:
C:\Users\TalgatHafiz\conda\contradictory-my-dear-watson.zip
so now you can unzip and start using the data

If you still have questions please read
https://medium.com/@jeff.daniel77/accessing-the-kaggle-com-api-with-jupyter-notebook-on-windows-d6f330bc6953


I have a csv file with time as the datetime format and want to edit the same column into miliseconds, how do I change that? Input file is :
time    event   earthAcceleration.x earthAcceleration.y earthAcceleration.z
2017-11-22T09:20:13.944 motion  -0.006379993    -0.001029071 -0.010781396
2017-11-22T09:20:13.954 motion  0.014739769 0.008146121 -0.001068155
2017-11-22T09:20:13.964 motion  0.008520941 -0.008222823    0.022573506
2017-11-22T09:20:13.974 motion  0.000573836 0.016701656 -0.016170829
2017-11-22T09:20:13.984 motion  0.016282924 0.003181049 0.006969224

I need to remove the date and hours and just need the milliseconds from the start. Example: Time2 = time2-time1 i.e. I need relative time in milliseconds such that I can plot a graph by time.
Output:
(the time column should change into)
time
0.00
0.15
1.12
1.34

Familiarize yourself with the pandas library, which allows you to do this:

from pandas import read_csv
from io import StringIO
csv_file = StringIO("""time event earthAcceleration.x earthAcceleration.y earthAcceleration.z
2017-11-22T09:20:13.944 motion -0.006379993 -0.001029071 -0.010781396
2017-11-22T09:20:13.954 motion 0.014739769 0.008146121 -0.001068155
2017-11-22T09:20:13.964 motion 0.008520941 -0.008222823 0.022573506
2017-11-22T09:20:13.974 motion 0.000573836 0.016701656 -0.016170829
2017-11-22T09:20:13.984 motion 0.016282924 0.003181049 0.006969224""")
df = read_csv(csv_file, sep=' ', parse_dates=['time'])
df.assign(delta_t = df.time-df.time.ix[0])

I believe your output is wrong; the difference between each line is exactly 0.01s. Welcome to the site!


Which phrase should be returned in case of multiple matches when comparing text? I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is:
I am playing football
and there are three more sentences that I want to compare my comparing sentence with. They are:
1. and I am playing Cricket
2. Why do you play Cricket
3. I love playing Cricket when I am at school
Now, if I compare my comparing sentence to the above three sentences by counting words, the number 1 and number 2 sentences have the same number of words that the comparing sentence has. and that is 3 (I, am , playing).
Now the question is, Which sentence is more related to my comparing sentence in this case? there are no semantic meanings involved at all.
In some places I saw, they say, it is less convoluted to return the shortest sentence in this case. What are your thoughts?

This is usually done by carefully choosing two things:

The sentence representation. Word count is the most simple option but there can be many others: TFIDF weights, with/without removing stop words, with/without lemmatization, etc. In a DL approach the sentence would be represented as a sentence embedding.
The similarity measure between two sentences. Again there are many options, in BoW approaches the standard ones would include counting words in common (e.g. Jaccard) and cosine TFIDF.

So the answer is: it depends on the similarity score. A complex similarity score like cosine TFIDF rarely produces ties so the highest score can be selected. More simple methods give ties, and then the logical answer is to return all the tied sentences.


Coding a Content Addressable Memory on a GPU I´m trying to code a CAM or more simply a dictionary storing the pointer of the data accessible by a key. I try to do it with a GPU but all attempts have been inefficient compared on using System.Collections.Generic.Dictionary. Does anybody know how to implement this with CUDA to obtain a better performance than with a CPU?

I similarity matched a billion 8 character strings at about 1.5fps on a gtx980 - is that fast enough for you?

similarity matching -> do it very block list like, and charge through all the patterns in the whole corpus/memory, match in lots of 32 bits and get the similarity score by getting the pop sum is the fastest way to do it I think. (gpu or cpu)

exact matching -> you get a lot more if you exact match because it needent access the whole memory per access, its approximately a root more optimized, put it in a tree, reading is simple, just go node to node, but writing is a problem, youll get thread conflicts if it tries to append the tree twice at the same place.

Interlocked operations make it alot simpler to do,  but its easier if you read in parallel and just dont bother to parallel the write, unless its a necessity.


How to convert Keras h5 to PyTorch pth format? I have trained a feature extractor in Keras and saved the weights as a h5 file. Now I want to load the same weights into the same model created and initialized in PyTorch for performance comparisons. Is there any way I can convert the h5 file to pth file so I can load that into the PyTorch model?

You must be careful with the versions from TF and PyTorch (as some commands may be different).
Basically you must:
1 - Know your layer and activation structure on Keras:
You can get the layer information with:
model_keras.summary()

If you can't get info about activation functions, try:
for layer in model_keras.layers:
    print(layer.output)

2 - Build a model on PyTorch that has the same layer structure (and activation) as on Keras
3 - The recently created PyTorch model (let's say model_pyt) has different weights and biases from your model on Keras, so you must copy those weights and biases from the Keras model to PyTorch model:
BE CAREFUL HERE, PYTORCH WEIGHTS ARE TRANSPOSED IN RELATION TO KERAS WEIGHTS
Then, for example, copying a weight would be like:
model_pyt.layer1.weight.data = torch.tensor(model_keras.layers[0].get_weights()[0].T)

And for the bias:
model_pyt.layer1.bias.data = torch.tensor(model_keras.layers[0].get_weights()[1])

repeat that for all layers.
Then, your PyTorch model has the same architecture and weights as the Keras model and might behave in the same way.


Precision and Recall Confusion matrix I was wondering, is it a proper method to convey information via separate Recall and Precision Confusion matrix? I recently came across a paper which reported the following scores. I am puzzled and unable to interpret them. Is this a usual standard in the academics?

Edit 1: The paper is related to activity recognition.

No. A confusion matrix is by definition a tabulation of real classes and predicted classes per subject. I've seen relative counts but they are not standard.

According to Wikipedia:

  Each row of the matrix represents the instances in a predicted class while each column represents the instances in an actual class (or vice versa).

The matrices listed above don't do that, so they are not confusion matrices. I'm not familiar with this format, and I don't think it is an academic standard. I'm not really sure how to interpet them, in that sense they are confusing matrices.

PS: You can obtain precision and recall from a confusion matrix.


Keras generator Function I am trying to write a batch generator function so that I can train my Keras model using it but I don't know from where to start. Does anyone have some examples or tutorials?

This tutorial explains in a nice and simple way how to create a data generator which you can pass to your Keras model to train using fit_generator(). Two things to keep in mind about Keras generators, in order to be compatible with Tensorflow 2.x requirements:

Your generator should inherit from keras.utils.Sequence, which allows for internal parallelization of training
Populate the __getitem__() and __len__() methods. The first makes your generator iterable and the second returns its length using the builtin len command on the generator object. Both are required by Keras.


What are some recommended visualisations to represent how attributes in a data-set influence each other? Looking at this data-set,  I want to illustrate ideally to what degree and in what direction each of the properties (fixed acidity,volatile acidity,citric acid,etc.) influence each other.

Right now I can only think of using a chord diagram or weighted network diagram to illustrate the magnitude of influence properties have on one another.

Are there any reasonable ways to visualise both the magnitude of influence properties have on one another and the direction of influence (positive/negative influence)?

Or any better ways to simply visualise the magnitude of influence?

Using seaborn in python, I usually plot the pairwise distributions with sns.pairplot. seaborn provides a wide variety of data exploration tools in a high level API, it is quite powerful for data exploration.

Here's what sns.pairplot looks like on the popular iris dataset:

Sometimes plots like the following (a specific kind of cobweb plot) can be useful too.

Those represent the distribution of each parameter on each vertical line, with a web of links joining values for a single entry in the dataset.


Why we add a constant value column in our DataFrame sometimes? Currently I'm learning data science and I'm in the beginners stage. I have seen many times we add a "constant" column in our data frame with all row cells of that column having value 1.

I need to know why we do so. And also what will happen if we don't use it.

Thank you.

In linear regression you need that column to have lines which are not constrained to pass through origin. Think of linear model $y = b_1 x_1 + b_2 x_2 + ...$. Iff all $x_i$ are 0, y must be 0, you need an additional parameter to pass that constraint.


What's the principal difference between ANN,RNN,DNN and CNN? I'm newer to deep learning domain.

I would like to know what is the principal difference between RNN,ANN,DNN and CNN?

How to implement those neural networks using the TensorFlow library?

Welcome to DS StackExchange. I'll go through your list:

ANN (Artificial Neural Network): it's a very broad term that encompasses any form of Deep Learning model. All the others you listed are some forms of ANN. ANNs can be either shallow or deep. They are called shallow when they have only one hidden layer (i.e. one layer between input and output). They are called deep when hidden layers are more than one (what people implement most of the time). This is where the expression DNN (Deep Neural Network) comes.
CNN (Convolutional Neural Network): they are designed specifically for computer vision (they are sometimes applied elsewhere though). Their name come from convolutional layers: they are different from standard (dense) layers of canonical ANNs, and they have been invented to receive and process pixel data.
RNN (Recurrent Neural Network): they are the "time series version" of ANNs. They are meant to process sequences of data. They are at the basis of forecast models and language models. The most common kind of recurrent layers are called LSTM (Long Short Term Memory) and GRU (Gated Recurrent Units): their cells contain small, in-scale ANNs that choose how much past information they want to let flow through the model. That's how they modeled "memory".

If you want to learn how to implement all these forms of ANN in TensorFlow, I suggest you this wonderful book:

Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron

this is the best book on ML and DL, IMHO.


How can precision be less than one in Leave-One-Subject-Out binary classification if each subject contains only one class Say I'm trying to classify a medical condition.
Theres only two classes: Sick and Healthy.
I build a model and I can't split the data because I don't want data from the same patient being in training and test set. So I elect to use Leave-One-Subject-Out, training the model on all subject except one and testing on the left out subject.
So for each test set I have one subject and they are either healthy or sick. So the confusion matrix only contains one class where precision is technically one every time and recall equals accuracy.
I've been reading some papers that claim to use leave-subject-out training and test splits for tasks where patients either have a medical condition or do not. I've seen papers that report accuracy, recall, and precision but I don't understand how you could have precision be less than one if each subject only contains one class. I doubt these papers are lying because I've seen this more than once.
I just want to know whats going on here for them to get precision values that are less than one. Are they doing some kind of averaging or am I missing something and thinking about this in the wrong way? None of the papers explain this either.

Your reasoning is correct that the gold standard class is the same for all the instances in a single leave-one-out test set (under the assumption that the test patient cannot become sick at some point in time, thus having both healthy and sick status).
What you're missing is the aggregation across multiple test sets: a full leave-one-out experiment repeats this process for every single patient, i.e. if there are N patients then there are N unique pairs of (training set, test set). Here is a pseudo code to show this clearly:
correct,incorrect=0
for every patient p:
    training_set = all the patients except p
    test set = patient p
    model = train(training_set)
    // to simplify I count the patient as one instance, it's easy to count instances instead
    if model.predict(test_set) == gold_standard(p):
       correct += 1
    else:
       incorrect += 1
accuracy = correct / (correct + incorrect)

The calculation across all the patients can lead to some predictions being correct and some others being incorrect, this is why the accuracy, precision or recall can be any value between 0 and 1.


Doc2Vec or Word2vec for word embedding Is there any benefits from using Doc2vec for word embedding ( replacing word2vec ) ? in other hand if I train word2vec and doc2vec with the same dataset will I have the same word vectors ?

As the name implies, doc2vec generates vectors representing documents (sentences, paragraphs) but not single words. So training doc2vec won't give you word vectors but document vectors. This means you can't replace word2vec by doc2vec at all.

Here's how the authors of the underlying paper describe what doc2vec does:

  Our algorithm represents each document by a dense vector which is trained to predict words in the document.


How would I load a database with multiple inconsistent headers? I have a database with multiple headers like so:

site_no,datetime,00060_00003
11481500,2019-10-05,7.54
[...]
site_no,datetime,00010_00001,00010_00002,00010_00003,00060_00003
11523000,2019-10-05,15.0,14.1,14.6,1920

I need to merge these into something that looks like this:

site_no,datetime,00010_00001,00010_00002,00060_00003
11481500,2019-10-05,-1,-1,7.54,-1
11523000,2019-10-05,15.0,14.1,14.6,1920

The code that I currently have to do this is:

    df = pd.read_csv(outputFileName, false_values = ["***"], header=headerlist[0] )
    i = 1
    while i+1 < len(headerlist):
        newdf = pd.read_csv(outputFileName, false_values=["***"], skiprows=headerlist[i]-1, header = headerlist[i]).head(n=(headerlist[i+1]-headerlist[i]),)
        df.join(newdf, on = "datetime", how="outer")
        i = i + 1
    df.to_csv(outputFileName)

I get an error:

KeyError: 'datetime'

$ cat one.csv
    site_no,datetime,00060_00003
    11481500,2019-10-05,7.54
$ cat two.csv
    site_no,datetime,00010_00001,00010_00002,00010_00003,00060_00003
   11523000,2019-10-05,15.0,14.1,14.6,1920

code:

df2 = pd.read_csv('two.csv')
df = pd.read_csv('one.csv')

pd.concat([df,df2], join='outer', join_axes=[df2.columns])

output:

   00010_00001  00010_00002  00010_00003  00060_00003  00060_00003        datetime   site_no
0          NaN          NaN          NaN          NaN             7.54  2019-10-05  11481500
0         15.0         14.1         14.6       1920.0              NaN  2019-10-05  11523000

the pd.join or pd.concat works on entire data frame, you don't need to loop through the variables.

using merge: (from the documentation)

t = pd.DataFrame({'key':['foo','foo'], 'lval':[1,2]})
>>> right = pd.DataFrame({'key':['foo', 'foo'], 'rval':[4,5]})
>>> left
   key  lval
0  foo     1
1  foo     2
>>> right
   key  rval
0  foo     4
1  foo     5
>>> pd.merge(left, right, on='key')
   key  lval  rval
0  foo     1     4
1  foo     1     5
2  foo     2     4
3  foo     2     5


Difference between Sum of Squares and Maximum Likelihood Linear Regression I'm new in Machine Learning and one of the first arguments I'm studying is linear regression. I understood that , in few words , the idea to use the Linear Regression is to learn an hypothesis that can map a new input x in a good approximation of y.

In order to do this , if my hypothesis is :

h(x) = wx + w0

I have to update my parameters minimizing an error function like Least Squares and optimize the w vector with the help of an optimization algorithm like Gradient Descent.

I understood how this works , but sometimes I see this "Maximum Likelihood Estimation" and I did not understand if it is another way to estimate the w parameters or something else.

Suppose you construct a probabilistic model, where $y_i$ are believed to be related to your $x_i$ under the formula;

$$ y_i = w_1 x_i + w_0 + \epsilon_i $$

So $w_1$ and $w_0$ are your target parameters from before and $\epsilon_i$ is an error term which you expect follows some probability distibution, e.g. $\mathcal{N}(0,\sigma^2)$. It is important that its expectation is zero.

Given your data you want to maximise the probability of returning every $y_i$ given your data $x_i$ subject to your model. The probability of getting $y_i$ from $x_i$ is equal to $f_{\epsilon}(y_i-w_1x_i-w_0)$, where $f_{\epsilon}$ is the probability density function of $\epsilon$, i.e. a normal distribution.

And the likelihood of getting every $y_i$ is the product of them all,

$$ L = \prod_i f_{\epsilon}(y_i-w_1x_i-w_0) $$

You want to maximise this value by tweaking $w_1$ and $w_0$, hence the name maximimum likelihood estimation.

Note that this is equivalent to maximising the log likelihood, so;

$$ \log L = \sum_i \log f_{\epsilon}(y_i-w_1x_i-w_0)$$

And if you look at the normal distribution density function you will see that (after ignoring some constants) this reduces to the problem of maximising..

$$ - \sum_i (y_i-w_1x_i-w_0)^2 $$

or in other words minimising the sum of squares akin to OLS.

But like using a different distance function in OLS you can parametrise a different error distribution in MLE.


Number of parameters in Neural Network Language Model The Neural Network language Model (NNLM) by Bengio et.al is a structure extensively used in machine translation, text summarization based on deep learning. What's the computational complexity of this model?

Knowing the complexity in terms of number of parameters helps in choosing the size of the training set and in determining what compute infrastructure is required.

NNLM has following sets of parameters ).
Using $V$ to denote the number of words in the vocabulary:

The matrix which creates the embedding for each word in the context . This distributed embedding is in $\mathbb{R}^{m}$ space. This matrix is $C :{V \times m}$
The matrix which transforms the concatenated list of word embeddings (active in the current context of size $n-1$) to the hidden layer (of size $h$). This matrix is $H:{(n-1)m\times h}$
The matrix which maps the hidden layer to unmormalized probabilities for each word in the vocabulary. This matrix is $V :{V \times h}$
The matrix connecting the context word embeddings to the output layer. This connection is optional and has dimensions $W:{V \times (n-1)m}$
Note: while training using SGD, for  a single example only $n-1$ words in the context are active out of words $V$ in the vocabulary. I have also omitted the parameter vectors for the bias terms $b:V \times 1$   when computing unnormalized outputs and when computing the hidden layer ($d: h \times 1$).

Hence my current understanding is that  number of parameters in NNLM is:
$$dim(C)+ dim(H)+ dim(V) + dim(W)= (n-1)m \times V + h \times (n-1)m+ V \times h + V \times (n-1)m$$


Should I normalize data to be similar to Normal distribution? As I understood, when training a neural network, it is preferable to have data with expectation of 0 and std. of 1.

Now if I have a feature with a Ratio distribution i.e., where median and expectation can differ much. If I apply Gaussian normalization - to subtract expectation and divide by std, I will get a distribution with again median different to expectation.

Should I be bothered by this anyway? I can easily move them to match by applying Log to it but I wanna know should I waste time on such details?

Normalising input data to a neural network is known to improve convergence properties, i.e. the model should converge quicker because the normalised data will not be so likely to produce huge or tiny gradients. Huge gradients make training erratic and less likely to converge (without applying other tricks), while tiny loss gradients lead to neuron death whereby certain neurons may get stuck, unable to get to an optimal weight value.

As you pointed out, normalising can be trivially simply to perform, so it might be worthwhile for both the final results and as a learning experience to try it out and compare your results.

You might also consider using other techniques such as batch normalisation, which inherently give you similar properties to normalising the entire dataset at the beginning.


Extract information from sentence I'm creating a simple chatbot. I want to obtain the information from the user response. An example scenario:

Bot : Hi, what is your name?
User: My name is Edwin.

I wish to extract the name Edwin from the sentence. However, the user can response in different ways such as

User: Edwin is my name.
User: I am Edwin.
User: Edwin.

I'm tried to rely on the dependency relations between words but the result does not do well.

Any idea on what technique I could use to tackle this problem?

[UPDATED]

I tested with named entity recognition together with part of speech tagger and parser. I found out that most model is trained in a way that the first character of the entity for the person name or the proper noun must be upper case. This may be true for normal document, but it is irrelevant for a chatbot. E.g.

User: my name is edwin.

Most NER failed to recognize this.

You can possibly use a combination of Named Entity Recognition and Syntactical Analysis -
while the word Edwin is certainly propping up, imagine a situation where the name is Edward Philip Martel. NER detects each word as a separate entities (hence 3 different entities) - thus, you will anyways have to string them together based on some logic. Further, in the case of multiple names being present, it can get harder to disambiguate (e.g. John & Ramsey dined at Winterfell).

This is where the analysis of the sentence syntax would also help (assuming that the end user enters a relatively coherent and proper sentence - if slang and short forms of text are used, even the Stanford NLP  can help upto a certain extent only).

One way of leveraging on syntax analysis / parsing and NER is in the following examples -

 1. User: Edwin is my name.
 2. User: I am Edwin.
 3. User: My name is Edwin.

In each of the cases (as is generically the case as well), the Entity Name (Proper Noun / Noun) is associated in close proximity to a Verb. Hence, if you first parse the sentence to determine verbs and then apply NER to surrounding (+/- 1 or 2) words, you may have a relatively decent way to resolve the problem.
This solution would depend primarily on the syntax rules you create to identify NERs as well as the window around the verbs.


Training CNN: Understanding number data generated while training the model I am training CNN on kaggle and my training and test datasets shapes are as follows:
(12271, 100, 100, 1) train shape
(3068, 1, 100, 100) test shape

(12271,) train target shape
(3068,) test target shape

But when I am training my network with below code, I am getting fllowing output while training, what does it mean. Does it mean  validation score is only calculated on 95 samples instead of 3068? I am not sure what does this number 95 denotes at the start of each epoch?
Why is it happening?

for i in range(len(model)):
    model[i].fit_generator(datagen.flow(x_train,y_train, batch_size=batch_size),
                                        epochs = epochs, steps_per_epoch=x_train.shape[0] // batch_size,
                                        validation_data = (x_test,y_test),
                                        callbacks=[ReduceLROnPlateau(monitor='loss', patience=3, factor=0.1)],
                                        verbose=2)
    models.append(model[i])

Epoch 1/700
95/95 - 12s - loss: 3.4509 - accuracy: 0.3898 - val_loss: 2.8512 - val_accuracy: 0.4198
Epoch 2/700
95/95 - 11s - loss: 3.2385 - accuracy: 0.4420 - val_loss: 4.5017 - val_accuracy: 0.0975
Epoch 3/700
95/95 - 12s - loss: 2.7225 - accuracy: 0.4928 - val_loss: 4.3058 - val_accuracy: 0.4654
Epoch 4/700
95/95 - 11s - loss: 2.5231 - accuracy: 0.5203 - val_loss: 3.7107 - val_accuracy: 0.4501
Epoch 5/700
95/95 - 11s - loss: 2.2600 - accuracy: 0.5358 - val_loss: 1.9277 - val_accuracy: 0.5639
Epoch 6/700
95/95 - 12s - loss: 2.0255 - accuracy: 0.5529 - val_loss: 1.7618 - val_accuracy: 0.5896
Epoch 7/700
95/95 - 11s - loss: 1.8013 - accuracy: 0.5756 - val_loss: 1.4528 - val_accuracy: 0.6040
Epoch 8/700
95/95 - 11s - loss: 1.6182 - accuracy: 0.5942 - val_loss: 1.1788 - val_accuracy: 0.6858
```

95 is not number of samples, it equals to testset divided by batch_size: testset // batch_size in other words steps in per epoch


How do I use matrix math in irregular neural networks such as those generated from neuroevolution (NEAT)? I understand how to structure the matrix when every node in a layer is fully connected to every node in adjacent layers and I understand that in "irregular" neural networks I can just process each node individually. However, there are no explanations or examples online of how to structure a matrix for an "irregular" neural network. How would I handle recurrent connections? Would I just fill in the "gaps" in the matrix with zeroes? Take the irregular neural network in this diagram:

Could I somehow combine (or get the dot-product of):

[i0  i1  i2] and

[[w0 w1 0  w9 0 ]
 [0  w2 w3 0  0 ]
 [0  0  0  w4 0 ]
 [0  0  0  w5 w7]
 [0  w8 0  w6 0 ]]

to find [o0 o1 o2]? Would I need to give the input vector an additional two values of 0?

It looks like I created a new way to use matrices with irregular neural networks in the process of answering my own question. Everyone must now refer to this method as the Capobianco Irregular Neural Network method. The CINN method basically treats the entire irregular neural network as a two layer fully connected network, albeit you treat most of the weights as being 0 (because they don't exist). This results in two sparse matrices. To be clear, you treat the input layer as being connected to EVERY hidden neuron, even if they are several layers away. Similarly, you treat the output layer as being connected to EVERY hidden neuron and input neuron.

The following example referencing the above picture uses initial h0 and h1 values of 0, which can be thought of as the values of the hidden states at the beginning of t0. There's two tricks I discovered.

First, just concatenate the [i0 i1 i2] input vector with the hidden state vector which gives us:

[ i0 i1 i2 h0 h1 ]

Next concatenate the weight matrix for the entire neural network except for the output weights:

     ( i0 i1 i2 h0 h1 )
(i0) [[ 1  0  0 w0  0 ]
(i1)  [ 0  1  0 w1 w2 ]
(i2)  [ 0  0  1  0 w3 ]
(h0)  [ 0  0  0 w9  0 ]
(h1)  [ 0  0  0  0  0 ]]

Every row shows the outgoing weight the value in parentheses has to the corresponding column value, with the exception of the area where the input vector values would otherwise "interact". We need this identity matrix to retain the original input values for the last step.

Before that it might be helpful to multiply out what we have using real values. To be clear, you want to find the dot product of:

                    [[ 1  0  0 w0  0 ]
                     [ 0  1  0 w1 w2 ]
[ i0 i1 i2 h0 h1 ] . [ 0  0  1  0 w3 ]
                     [ 0  0  0 w9  0 ]
                     [ 0  0  0  0  0 ]]

http://matrixmultiplication.xyz/ isn't perfectly accurate, but I really like how it shows you visually how each pair of terms are combined. This will result in a new vector [i0 i1 i2 h0* h1*]. h0* and h1* represent the final values of the hidden states at the end of the original time-step t0 (note that these will be the new h0 and h1 values at the beginning of t1), while the i0, i1, and i2 remain unchanged because we utilized the identity matrix.

Finally all we have to do is multiply this new vector by the matrix containing all the weights connecting to the output layer:

      ( o0 o1 o2)
(i0 ) [[ 0  0  0]
(i1 )  [ 0  0 w8]
(i2 )  [ 0  0  0]
(h0*)  [w4 w5 w6]
(h1*)  [ 0 w7  0]]

We can also do this all at once:

                    [[ 1  0  0 w0  0 ]  [[ 0  0  0 ]
                     [ 0  1  0 w1 w2 ]   [ 0  0 w8 ]
[ i0 i1 i2 h0 h1 ] . [ 0  0  1  0 w3 ] . [ 0  0  0 ] = [ o0 o1 o2 ]
                     [ 0  0  0 w9  0 ]   [w4 w5 w6 ]
                     [ 0  0  0  0  0 ]]  [ 0 w7  0 ]]

Note that is just for output vector at the end of t0. If you're not dealing with time data or using an RNN you don't need to worry about this, but if you're trying to determine outputs at a later time-step, you'll have to do something like this:

                  [[ 1  0  0 w0  0 ]                       [[ 1  0  0 w0  0 ]  [[ 0  0  0 ]
(input @ t0)       [ 0  1  0 w1 w2 ]   (input @ t1)         [ 0  1  0 w1 w2 ]   [ 0  0 w8 ]   (output @ t1)
[i0 i1 i2 h0 h1] . [ 0  0  1  0 w3 ] . [i0 i1 i2 h0* h1*] . [ 0  0  1  0 w3 ] . [ 0  0  0 ] = [o0 o1 o2]
                   [ 0  0  0 w9  0 ]                        [ 0  0  0 w9  0 ]   [w4 w5 w6 ]
                   [ 0  0  0  0  0 ]]                       [ 0  0  0  0  0 ]]  [ 0 w7  0 ]]

Note that this example is for an INN created using a genetic algorithm, so back-propagation is unnecessary in this example. If you wanted to utilize back-propagation you'd begin that process after calculating your output vector.


Does random forest re-use features at each node when generating a decision tree? Learning about random forest for the first time and I'm not clear if at each level, if we've already used a feature to make a decision, whether we can use the same feature again at further levels.

If the feature is discrete/categorial/binary, it won't use the already used feature to split at further levels in any given branch of the tree. But key thing to note here is that the same feature can be used in a different branch of the tree at a different level.

In the above diagram, we can see that same 'size' feature has been used at two levels 'level 1' and 'level 2', but in different branches of the tree.

On the other hand, if the variable is a continuous value, it uses threshold splits at each level and in this case, same feature can be used multiple times in any given branch of the decision tree.

Since random forests is an ensemble of decision trees, the same theory applies to Random Forests as well.

Reference: http://www.cse.msu.edu/~cse802/DecisionTrees.pdf


Splitting large multi class dataset using leave one out scheme into train and test I am doing some supervised learning using neural networks, and i have a Targets array containing 1906 samples, which contain 664 unique values. min. count of each unique value==2, by design. Is there a smarter way to split this dataset into train and test, using a leaveoneout scheme to pick randomly 1 sample from each class and put it in the test set and use the rest for training, before i get down to explicitly iterating over all my values? I am using python, numpy, sklearn and pytorch btw!

Never mind! Found it!

just doing a,indices,counts=np.unique(y) will return the starting indices of each unique value in indices, and then you can just slice this off as x_test=X[indices], and remember to delete it using np.delete(X,indices)


operating on a dataset with 125,497,040 records I'm trying to run add_datepart() which converts a df column from a datetime64 to many columns in place

Year', 'Month', 'Week', 'Day', 'Dayofweek', 'Dayofyear','Ismonthend', 'Ismonthstart', 'Isquarterend', 'Isquarterstart', 'Isyearend', 'Isyearstart' . etc

I'm using is Grocery Sales dataset. Total dates i believe in that are 125497040. What should I do to run this operation?

Every time I run this piece of code. The kernel dies(out of memory which is 17.2 GB RAM)

So I tried breaking down this data frame in smaller parts and then running add_datepart but still the same result

I wrote this code

def add_datepart_large(temp_df, size):
    list_df = [temp_df[i:i+size] for i in range(0,temp_df.shape[0],size)]
    for i in range(len(list_df)): add_datepart(list_df[i], 'date')
    return pd.concat(list_df)

final_df = add_datepart_large(df_all, 100000)

If after running this code, the kernel dies. What's wrong?

For 127 million rows, it is better to perform data prep on DB. It will be a select + insert query and will not require whole data to be loaded in memory.

SELECT YEAR(date) AS 'year', MONTH(date) AS 'month'
 FROM data

Edit : Once you start training / validation, even then it would be better to load few batches from DB (at a time). Most frameworks support this.

Foe example : https://github.com/keras-team/keras/issues/107


neural networks error function: is global minimum desirable? In "Elements of statistical learning" page 395 the authors state that, relative to R(θ), the regression/classification error function in a neural network such as a multi layer perceptron:

  Typically we don’t want the global minimizer of R(θ), as this is
  likely to be an overfit solution. Instead some regularization is
  needed: this is achieved directly through a penalty term, or
  indirectly by early stopping. Details are given in the next section.

However in Backpropagation, when momentum regularization is described the figure 9 shows how the back propagation process should steer the error function towards its global minimum, avoiding any local minimum found along the way.

How come these 2 reasonings are compatible? The only thing I can think of is that we want to get the global minimum for each coefficient error (as computed during the backprop process) while not adjusting for the "global" error function. Am I interpreting this correctly?

The figure describing momentum is a bit misleading because it only considers a very simple case. Usually momentum will help you not to get stuck too early in a local minimum, so you end up in a better but still local minimum.

Momentum can help you converge faster, but there are no guarantees to end up in a global minimum. In popular use cases of neural networks like image classification etc. the error function will be extremely complicated, so it will always be infeasible to reach a global minimum using gradient based methods. But practice has shown that driving down the error to a local minimum still leads to good results.

Also, momentum is compatible with other forms of regularization. You can apply momentum to any regularized cost function. The statement in Elements of Statistical Learning is more of general nature, meaning that even if you could achieve the global minimum it wouldn't be desirable because it is most likely not the solution that will give you the best generalization (even though it has the best training(!) error).


Metric to combine average with number of accurance I was analysing a dataset which has got two main columns which I can name like:

category:String
amount: Int

So I wanted to know the average of amount for each of the category. So I did something like:

df.groupBy("category").agg(avg("amount") as average)

The problem is for categories I don't have many observations, sometimes just one or two. But for other categories I have around 5000.

So I would like to know if there is any metrics where I can reflect that the average may be more or less accurate depending on the amount of observations used for the average.

The metric you are looking for is the standard error. From Wikipedia:

  the standard error equals the standard deviation divided by the square root of the sample size.

So as the sample size increases the standard error decreases. You can also see this for an explanation with some code in R.


Training set of a NN that classify Cat or Not Cat Is it possible to build a Convolutional Neural Network (using Keras, Tensorflow) that can give output as 1 for an image of a Cat and 0 for everything else?
How would the training set look like?
I mean, let's assume if the number of images of the Cat is ~1 million, then the number of images of "not a Cat" should be potentially trillion or even more.
More importantly, what classes of images should I include in the images that are labeled as "not a Cat", e.g. Dogs, Tigers, Zebras, Trees, Computers, Galaxies, etc?
Doesn't it mean that we should have an infinite number of training samples that represent "not a Cat"?

It is possible to build that kind of a CNN. It is important to maintain uniform distribution for both the classes ('cat' and 'not cat'). That is you should have an almost equal number of samples for each of these classes to avoid biasing your model to the 'non-cat' class just because it has huge number of examples.
The number of non-cat examples can be potentially trillion because it's everything in the world apart from cat. You don't want to include those many examples in your training set just because trillions of examples are available. As I told, it's important to match the class distributions. You can choose any animal or object you want for the 'non-cat' class. You can even mix multiple classes for 'non-cat'.


Distribution vs Histogram I am not very good at understanding statistics jargon

I tried to read articles on statistics every article says distribution and show a picture of bell curve. I understand that x axis of bell curve will take the value of data but does y-axis always has the count/frequency od the data

Is Distribution and Histogram same ?

I did find an article on difference between distribution and histogram

https://asq.org/quality-resources/histogram

But it says histogram is most commonly used for frequency distribution

So does that means there are other types of distribution ?
Or
Does distribution of variable always means frequency distribution ?

You could think of histogram as one way of plotting a distribution of values. Another way of plotting such distribution is a KDE (Kernel Density Estimate), but after all, these inform about the same concept, which is how frequent is each value (or values interval) of your values series (x-axis).

I like this picture from seaborn distplot to have a one shot info about the (univariate) distribution of your values, both with continuous (kde) and discrete (histogram) representations. https://seaborn.pydata.org/tutorial/distributions.html


Keras Fit Generator Function I want to know if my thinking is correct?

Total Images = 120,000

So in the Keras Fit Generator function,
I made the batch_size = 24 and steps per epoch = 500 ,which is 12000 and only one-tenth of the total dataset.

Therefore I should make the "epochs = 100" if I actually want the real epoch to be 10.

i.e. It will actually take 10 epochs to go through the entire 120,000 images once.

Your assumption is in part correct, as your model will see 120000 images (i.e. the size of the whole dataset) in 10 epochs.

However, because keras' default generators shuffle the dataset at the end of each epoch, it is very unlikely that the model will see each image once after 10 epochs.

My suggestion would be to set steps_per_epoch=5000.


Using a model for a different dataset I have generated a model for predicting the future input trend with a sample data using linear regression in Knime. I want to validate the model using a different data set.

Suppose the data set I used for creating the model came from the sensor of device A. The prediction is accurate for the dataset from device A. Lets say I saved the model as a PMML file  called A. Can I use the same PMML file A to predict values of Device B (Values of B are not comparable). If not, should I create a different model for all data set that I have?

Question

How do I combine all the models generated so that it predicts any given data set? Is it possible?

I think you answered yourself by "values of B are not comparable". Learning for predictions is based on a fundamental assumption which is the data for prediction has the same joint distribution as the data for learning. This is the link between those processes.

Now, if you want to handle that in a meaningful way you have to know somehow the source type. In your example the device type. One way would be to introduce the device type as a different column in your data set, so that the model can have the chance to differentiate between source type. Obviously you have to have training data for all the device types. Supposing you have 2 device types, A and B. Your training data should have some columns for the signal and also a factor column with type A and B. Also you have to have enough instances of data with type A and type B.


How to classify whether text answer is relevant to an initial text question I have a text classification problem in which i need to classify an answer to a message as either relevant or not.

In the first phase of my calculations, I have already used a SVM to determine if the original message was relevant or not, deciding whether a message contains a hint or question if somebody's twitter account has been hacked.

example:
"Hey @foobar, have you been hacked?"   <-- relevant
"My bank account has just been hacked" <-- not relevant

However, when I want to classify whether the answer is relevant, I would want to have both the original message and the answer as input, right? An answer is relevant in my case if it, in any way, responds to the original message. Is this approach possible using a SVM or any other machine learning tool? I'm using python with the scikit-learn library.

example:
"Hey @foobar, have you been hacked?"
"@barfoo it seems so, thx for suggesting" <-- relevant

"Hey @foobar, have you been hacked?"
"Lose 20 pounds quickly! http://blabla.com" <-- not relevant

I'm not very experienced in this field, so any input would be very appreciated.

Both message and answer are your input, so your feature vector should contain information about both.

Here's a simple structure of a possible solution using scikit-learn:

import numpy as np
from sklearn.svm import SVC
from sklearn.feature_extraction import DictVectorizer

dataset = (("Hey @foobar, have you been hacked?",
            "@barfoo it seems so, thx for suggesting",
            True), # True for relevant, False for not relevant
           ("Hey @foobar, have you been hacked?",
            "Lose 20 pounds quickly! http://blabla.com",
            False))

def extractMessageFeatures(message):
    # here comes your real feature extraction algorithm
    return { 'message_predicted_spam': False,
             'message_contains_valid_username': True }

def extractAnswerFeatures(answer):
    # here comes your real feature extraction algorithm
    return { 'answer_predicted_spam': False,
             'answer_contains_valid_username': True }

def extractFeatures(data):
    features = []
    for instance in data:
        instanceFeatures = extractMessageFeatures(data[0])
        instanceFeatures.update(extractAnswerFeatures(data[1]))
        features.append(instanceFeatures)
    return features

def trainClassifier(data):
    features = extractFeatures(data)
    vec = DictVectorizer()
    featureVector = vec.fit_transform(features)

    print vec.get_feature_names()
    print featureVector.toarray()

    svc = SVC()
    svc.fit(featureVector, np.array([i[2] for i in data]))
    return svc

clf = trainClassifier(dataset)

# now, you can clf.predict(...)

Now, the hardest part is to decide which features to extract from both messages and answers. It's up to you.

One of the simplest solutions would be to use n-gram features.

Other approach would be to use some spam detection to decide whether answer is spam or not and treat this information as a feature.

You can also use Twitter-specific information (for example, whether users are mentioning each other in their tweets, using the same hashtags etc.).

You can combine these features in whatever fashion you like, of course.

Except of creating feature extraction functionality you need a labeled dataset of messages, answers and relevant/non-relevant labels.

But once you have both (feature extraction functionality and a proper dataset), you're good to go with a task which clearly matches standard machine learning approaches.


How does TF-IDF classify a document based on "Score" alloted to each word I understand how TF-IDF "score" is calculated for each word in a document, but I do not get how can it be used to classify a test document. For example, if the word "Mobile" occurs in two texts, in the training data, one about Business (like the selling of Mobiles) and the other about Tech, then how does the "score" for word "Mobile", in both training and test document over the given dataset, help the algorithm to classify whether the text (a new test document) belongs to "Business" category or "Tech" category? I'm new to NLP, thanks in advance!

It's not a single TFIDF score on its own which makes classification possible, the TFIDF scores are used inside a vector to represent a full document: for every single word $w_i$ in the vocabulary, the $i$th value in the vector contains the corresponding TFIDF score. By using this representation for every document in a collection (the same index always corresponds to the same word), one obtains a big set of vectors (instances), each containing $N$ TFIDF scores (features).
Assuming we have some training data (labelled documents), we can use any supervised method to learn a model, for instance Naive Bayes, Decision Trees, SVM, etc. These algorithms differ from each other but they are all able to take into account all the features for a document in order to predict a label. So in the example you give maybe the word "mobile" only helps the algorithm eliminate the categories "sports" and "literature", but maybe some other words (or absence of other words) is going to help the algorithm decide between categories "Business" and "Tech".


How can we use the cosine similarity formula on document feature vector without a direction? In mathematics, a vector has both magnitude and direction.

In data science, for identifying document similarity we convert the document into a feature vector. Then apply cosine angle formula between the source and target document's feature vector.

However the cosine formula is applicable only for vectors. And a vector should have both magnitude snd direction. For a document that is represented as a vector, where is the direction?

From this
"Cosine similarity measures the degree to which two vectors point in the same direction, regardless of magnitude.

When vectors point in the same direction, cosine similarity is 1; when vectors are perpendicular, cosine similarity is 0; and when vectors point in opposite directions, cosine similarity is -1. In positive space, cosine similarity is the complement to cosine distance: cosine_similarity = 1 - cosine_distance.

For example, the cosine similarity between [1, 2, 3] and [3, 2, 1] is 0.7143."

Also for angle and "Direction", google results says-

Here is a another nice explanation-

https://www.machinelearningplus.com/nlp/cosine-similarity/

by this article-

"When plotted on a multi-dimensional space, where each dimension corresponds to a word in the document, the cosine similarity captures the orientation (the angle) of the documents and not the magnitude. If you want the magnitude, compute the Euclidean distance instead.

The cosine similarity is advantageous because even if the two similar documents are far apart by the Euclidean distance because of the size (like, the word ‘cricket’ appeared 50 times in one document and 10 times in another) they could still have a smaller angle between them. Smaller the angle, higher the similarity."


How to make a dataframe with lists or vectors as its elements This is something I have been wondering for ages but I am never able to get an answer.

I am trying to understand how to make a dataframe in R, where each element of the dataframe is itself a vector or a matrix.

For example, lets say we have a regular vector V with elements being real numbers.

Then to acess any number we would have

V[3] which would give the third element of said vector.

Now I want to know how to do this with say a dataframe D,

where each element of the dataframe is itself a vector or matrix, so that say

D[3] is not a real number, but a vector.

How can this be done in R?

Thanks all


What is the most efficient data indexing technique As we all know, there are some data indexing techniques, using by well-known indexing apps, like Lucene (for java) or Lucene.NET (for .NET), MurMurHash, B+Tree etc. For a No-Sql / Object Oriented Database (which I try to write/play a little around with C#), which technique you suggest?

I read about MurMurhash-2 and specially v3 comments say Murmur is very fast. Also Lucene.Net has good comments on it. But what about their memory footprints in general? Is there any efficient solution which uses less footprint (and of course if faster is preferable) than Lucene or Murmur? Or should I write a special index structure to get the best results?

If I try to write my own, then is there any accepted scale for a good indexing, something like 1% of data-node, or 5% of data-node? Any useful hint will be appreciated.

I think you messed up some things in your question. Lucene (I know nothing about Lucene,NET, but I suppose is the same) is a library used to analyze, split in tokens, and store documents in order to be able to query and retrieve them later. Lucene has a pretty old but effective model, it uses inverted trees to find and retrieve documents. Without further details, all documents are split in tokens (terms), and for each term is maintained a data structure, which stores all the documents which contains the given term. As a data structure could be used a BTree, a hash table and in the latest major revisions you can even plug in your own data structures.

A BTree (see Wikipedia page for further details), is a kind of a tree data structure, which is appropriate for working with big chunks of data and is often used for storing tree-like ordered structures on disk. For in-memory other trees performs better.

Murmur hash (see Wikipedia page for further details), is a family of hash functions used in hash table. The implementation of the hash table is not important, it could be a standard chained implementation or more advanced open hash addressing scheme. The idea is that the hash tables allows one to get fast a key, from an unordered set of keys, and can answer to tasks like: is this key part of this set of keys? which is the value associated with this key?

Now back to your main problem. You have one library (Lucene) and to data structures, both data structures are used in Lucene. Now you see that it is not possible to answer your question in these terms since they are not comparable.

However, regarding you footprint and performance part of the question. First of all you have to know which kind of operations you need to implement.

Do you need only get value for key, or do you need to find all elements in a range? In other words do you need order or not? If you do, than a tree can help. If you do not, than a hash table, which is faster could be used instead.

Do you have a lot of data which does not fit the memory? If yes than a disk-based solution would help (like BTree). If your data fit the memory, than use the fastest in-memory solution and use disk only as a storage (with a different structure, much simpler).


What is the best alternative for Fisher's Exact test for contigency tables that are NOT 2x2? I am a newbie to data mining. I am trying to find associations between two categorical variables. Since more than 20% of my expected frequencies are less than 5, I wanted to use Fisher exact test but it turns out it is generally used for contingency tables 2x2 but my variables have more than two values. Would really appreciate recommendations on the best course of action for me now. Here are some options I found after some search:

Use Freeman-Halton extension to Fisher's Exact test for more than 2x2 table.
Merge multiple attributes values so that I end up with 2x2 contingency table and then use Fisher's Exact test.
Merge multiple attributes values so that I end up with expected counts > 5 and then use chi square test for independence.
Use the Crammer V test.

I would like to know what is the standard practice in this case when your categorical variables with >2 possible values end up with less than 5 expected counts?
Thanks,

Fisher's exact test has a ready generalization to tables of an arbitary dimension and it is applicable here (Metha and Pathel, 2012).
You can for example use the fisher.test built-in function to compute this in R.


What is a "variable index" in the Gaussian perspective? I was going through this article about Gaussian processes, in which the author explains about the "variable index" in the form of a plot while writing about 2D Gaussian.

The explanation and plot are as below:

I understood the y-axis in this plot, but I'm having problems understanding the x-axis (variable index).

Where did the values 1 and 2 come from in that axis and how is the y-value 2 for both of them?

The motivation behind this diagram is to explain how the 2 dimensions of a 2D Gaussian are related to each other depending on the covariance matrix. The author of the article refers to the dimension as 'variable index'. By reading further in the article, the author explains that as the off-diagonal element of the covariance matrix is 0.9, the 2 dimensions are highly correlated. So if we sample a point from this 2D Gaussian, the first variable-index or dimension of the sample will have almost a similar value to the second dimension (=2) in the example. On the other hand, if the off-diagonal element is 0.0 then the two variable-indices or dimensions will be very different.

This example builds up to the idea of defining a kernel function in the Gaussian Process.


Efficient recurrent network for sequences of varying length Suppose I have a bunch of sequences of varying lenghts. The absolute majority of them are short, just a few dozens items long. However, very few of them are significantly longer - more than a hundred items long. The question is, how to organize them efficiently as an input to recurrent layer?

Padding doesn't work, since many sequences need to be heavily padded. Limiting batch size is not an option - these sequences are obtained as parts of a larger structure.

Sequence bucketing

Depending on the input length of sequences you can dynamically change the padding and speed it up.

Take a look at here Sequence bucketing


Sns Heatmap save as png jpeg image with full labels visible via python I am trying to save my heat map in png format using the below code, but the labels are getting trimmed when I am saving the heatmap. Please help to keep the labels inside the figure.

    array = confusion_matrix(gts, preds, labels=class_name)
    #print(gts)
    #print(preds)
    df_cm = pd.DataFrame(array, index = class_name, columns = class_name)
    print("Confusion Matrix")
    print(df_cm)
    plt.figure(figsize = (10,7))

    sns_plot = sns.heatmap(df_cm, annot=True, cmap =sns.cm.rocket_r,linecolor='white', linewidths=1)

    results_path = 'results.png'
    #print(results_path)
    plt.savefig(results_path, dpi=400)

results_path = 'results.png'`
#print(results_path)`
plt.savefig(results_path)`


How many images are generated when ImageDataGenerator is used, and when data augmentation is included as a part of the model? Is there any way to know the number of images generated by the ImageDataGenerator class and loading data using flow_from_directory method? I searched everywhere for the same but couldn't find anything useful.
Also, if I use image_dataset_from_directory fuction, I have to include data augmentation layers as a part of the model. In that case, how many images are generated?
My ImageDataGenerator code:
train_datagen = ImageDataGenerator(rescale=1./255,
                                   horizontal_flip=True,
                                   zoom_range=0.2,
                                   shear_range=0.2,
                                   rotation_range=15,
                                   fill_mode='nearest')

My code with data augmentation as a part of the model:
data_aug = Sequential([
                         layers.experimental.preprocessing.Rescaling(1./255),
                         layers.experimental.preprocessing.RandomFlip('horizontal'),
                         layers.experimental.preprocessing.RandomZoom(0.2,0.2),
                         layers.experimental.preprocessing.RandomRotation(0.06)
])

# in the model:

input_layer = layers.Input(shape=(150, 150, 3))
x = data_aug(input_layer)

In one epoch - It's the number of images in your Directory or the DataFrame
 In case of a custom Generator. It will be batch_size * steps_per_epoch
You may check this with any of these approaches -

Check the shape of prediction on train model.predict(traindata).shape
Save the images into a dir by using save_to_dir='/content/train_data'
Write a callback for batch_end
Get the steps_per_epoch in the training print statement i.e.
125/125 [==============================] - 50s 392ms/step - loss: 0.6637 - accuracy: 0.6000