How much of a problem is each member of a batch having the same label? I have a batch size of 128 and a total data size of around 10 million, and I am classifying between 4 different label values.
How much of a problem is it if each batch only contains data with one label?
So for example - batch 0 all have the 3rd label.  Batch 1 all have the 1st.  Batch 2 the 2nd.  Etc.

If you are interested in convergence, then most probably it shouldn't converge. I used "should" because although we have an intuition of how the convergence work but no one is 200% sure of everything.
 Let's understand a few of the aspects of the whole learning process.
 - Every data point has a distinct Loss space
 - Overall Loss space will be the average of all individual Loss surface
 - We can safely assume, that the overall Loss space of the data points in one class will be closer to each other but very different when compared to the other class.
 - Lastly, the Gradient is calculated at the end of a Batch
When we have well-shuffled batch -
In this case, the Gradient after each batch will be directed towards the overall Gradient because the Gradient will be averaged. It means a smooth Loss reduction with each batch
When we have each batch of one Class -
 In this case, every alternate batch will move on two different Loss space(One of each Class).
The issue will be that we will calculate Loss for one space, but after the weight update at the end of a batch, the next batch will continue in its own space but using the weight from the previous batch. This will make learning very unpredictable.
 Also, many logic applied in optimizers will not help because Gradient will randomly change with each batch.
I tried this on MNIST digit (just using 0,1 digit) and got this Loss per Batch. I have not out enough effort on LR optimization, LR decay, etc. So, can't conclude that result will be always similar.
$\hspace{3cm}$


Is it better to optimize hyperparameters or run multiple epochs? Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though many epochs.

However, whenever my neural network performs poorly, rather than have it go through more epochs, I try to optimize the hyperparameters. This approach has generally been successful as my neural networks are pretty simple.

But is training a neural network in this manner a bad practice? Are there disadvantages to immediately going to optimize the hyperparameters rather than running the neural network for more epochs?

Count of epochs is also a hyper-parameter. However, if you meant to ask what to choose to work upon, whether increasing epochs or some other methods like feature engineering , then below is my answer.

Increasing number of epochs is often attributed as hammer stroke to train the model where you yourself often don't have to think much about the data and deepNet does it for you generally. However, it comes at a cost of computational complexity, risk of overfitting, etc.

Instead, it is always advisable to work as much on tasks like feature engineering, tuning different other hyper-parameters while training. In fact, a data scientist is supposed to work on these things preferably and not just blindingly utilising the black-box concept of deepNet.

However, even if you choose to increase the count of epochs, you may utilise the technique called early stopping. It basically says to stop the training at a certain epoch if the validation loss does not improve which you can use without much thinking about the impacts of increasing the count of epochs because if the validation loss does not improve, it will stop there.


So one way to do it is to use python slice operators to grab every other value in the line and zip them (along with the filename) into 3-tuples, e.g.:

data = []
malletOutput = open('doc-topics','r').readlines()

for line in malletOutput:
    line = line.split('\t')[1:-1] # slicing out useless leading index and trailing \n
    _id = line[0]

    tIndicies = map(int,line[1::2])
    tVals = map(float, line[2::2])
    topics = sorted(zip(tIndicies, tVals))
    topics = [t + tuple([_id]) for t in topics] # add _id
    for t in topics:
        data.append(t)

Now you have a list of 3-tuples which you can add to a dataframe. A simple pivot from there results in the desired shape:

df = pd.DataFrame(data)
df = df.pivot(index=2,columns=0,values=1)


Hyperopt vs Default Values When I use the hyperopt library to tune my Random Forest classifier, I get the following results:

Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0}

However, when I train the model using its default hyperparameters, all of the evaluation metrics (Precision, Recall, F1, iba, AUC) return higher values compared to the tuned model. Should I still follow the tuned parameters? Or ignore the results of the tuning process, as it is not helping to improve the results?

According to me your approach of oversampling(as mentioned in comment) is not correct.
When you apply SMOTE on each FOLD(assuming K-FOLD), it will over sampled the minority class. And you will always get the mixed response from K-FOLD, which is not the purpose of K-FOLD validation.

Let me try to explain you with an example:

Consider you have a dataset of 4 classes A, B, C and D.
Samples of each class, Class A has 10000, B has 2000, C has 1000, D has 6000.
During K Fold cross validation, sample picked up by system for training having majority of A and D classes.
Whereas for testing it has majorly B and C. Since you have used SMOTE it will over sampled the other two classes and you will get decent results in this case as well.

NOTE:It could be the one of the reason.

My suggestion:
 1. Shuffle the whole data set.
 2. Divide into training and test data.
 3. Apply SMOTE on training data.
 4. Follow other steps of Model training, without manipulating the data at this stage.
 5. Evaluate model on the actual test data.


Why I get AttributeError: 'float' object has no attribute '3f'? I am getting this error:

AttributeError: 'float' object has no attribute '3f'

I don't understand why I am getting it, I am following the example straight from the book "applied text analysis"

The chunk of code in python is:

total = sum(words.values())
for gender, count in words.items():
pcent = (count / total) * 100
nsents = sents[gender]
print(
"{0.3f}% {} ({} sentences)".format(pcent, gender, nsents)
)

I see that pcent clearly will return a float, why the author tries to apply .3f what  am I missing?

Try this instead,

  print(
"{:.3f}% {} ({} sentences)".format(pcent, gender, nsents)
)

Refer the latest docs for more examples and check the Py version!


How can you map the exceedance of a threshold into an activation function of a Neural Network? I am totally new to Artificial Neural Networks. Let’s say that the model you are trying to turn into an artificial neural network has an output that is triggered only by the exceedance of a threshold: $y\geq y_{1}. $Therefore, you need to find a way to use this inequality as an activation function. Is this feasible?

This is feasable. This is also called a Binary/Step activation function. You must only use this activation function on the output neurons.

The Step function will round down an answer that is lower than 0.5 to 0, and an answer that is higher than 0.5 to 1. However, please note that you do not need to use a binary activation function to output 1 - I advise you to just use TanH or sigmoid and backpropagate a whole bunch of iterations.

However, in other comments, you mentioned that you know what y1 is. That is not of importance, the network will act as a black box and will figure out a treshold itself. Don't set up your own activation function just to get the right output - that avoids the whole point of backpropagation.


Recurrent Neural Network on Panel Data There are 2 parts to this question. Suppose we are looking at sales $S$ of a product across $> 1000$ stores where a it sells. For each of these $1000$ stores we have 24 months recorded data.

We want to be able to predict $S_t \leftarrow f(S_{t-1})$. We could build a RNN for each of the store time series, calculate test RMSE and then take an average after taking care of normalizing values etc. But the problem is there are very few samples per time series. If we were to segment stores into groups (by say Dynamic Time Warping) then could we create a monologue of text sentiment mining where like in text two sentences are separated by a dot here we would have two time series separated by a special symbol (let's say). In that case, we would generate a RNN model on

$$Train_1 | Train_2 |\ldots|Train_t$$

data and predict on

$$Test_1 | Test_2 |\ldots|Test_t$$

After this, we would like to set it up as a panel data problem where $S_t\leftarrow f(x_{t1},x_{t2},\ldots,x_{tn})$. In that case should I build a separate neural network for each $t$ and then connect the hidden layers from $t \rightarrow t+1 \rightarrow t+2 \ldots$.

How should I implement these through packages like Keras/Theano/Mxnet etc.? Any help would be great!

This is a good question - one that is of interest (or should be of interest) to those working in applied social sciences and operations or basically in any situation where the objective function is on panel data. There is a paper by Bai, Zhong and Gao on this topic: https://arxiv.org/abs/1708.00185


Will reinforcement learning work if states wont get repeated again? I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I don't understand is how to define states if I want to implement this with reinforcement learning?If I define each query as a state, all states will be different .Can anyone shed some light on this please!!!

I am not convinced that framing a document retrieval success as reinforcement learning problem will be easy to get working.

However, your core question is answerable. This problem occurs in other problems that are too large to explore all possible states. For instance, in the game of Go, the AlphaZero system cannot possibly explore all game positions. Similarly, in Atari games, many of which have also had reinforcement learning applied successfully, it is very unlikely that the agent has seen all possible states of the system - typically it will have trained on a million video frames.

The answer to this issue is to use some form of function approximation that can generalise to new unseen data. That is precisely what supervised machine learning models do, so the RL agent will use one internally, typically a linear regression or neural network, so that values and or policy learned from states it has explored will be associated with similar unseen states (for some interpretation of similar).


Split columns by finding "£" and then converting to minvalue, maxvalue I have a pandas dataframe with a salary column which contains values like:

  £36,000 - £40,000 per year plus excellent bene...,
  £26,658 to £32,547 etc

I isolated this column and split it with the view to recombining into the data frame later via a column bind in pandas.

I now have an object with columns like the below. The columns I split the original data frame column I think are blank because I didn't specify them (I called df['salary']=df['salary'].astype(str).str.split()
)

So my new object contains this type of information:

[£26,658, to, £32,547],
[Competitive, with, Excellent, Benefits]

What I want to do is:

Create three columns called minvalue and maxvalue and realvalue
List items starting with £ (something to do with "^£"?
Take till the end of the items found ignoring the £ (get the number out) (something to do with (substr(x,2,nchar(x)))?
If there are two such items found, call the first number "minvalue" and call the second number "maxvalue" and put it below the right column. If there is only one value in the row, put it below the realvalue column.

I am very new to pandas and programming in general, but keen on learning, your help would be appreciated.

This is more of a general regex question, rather than pandas specific.
I would first create a function that extracts the numbers you need from strings, and then use the pandas.DataFrame.apply function to apply it on the pandas column containing the strings. Here is what I would do:
import re
def parseNumbers(salary_txt):
    return [int(item.replace(',','')) for item in re.findall('£([\d,]+)',salary_txt)]

#testing if this works
testcases = ['£23,000 to £100,000','£34,000','£10000']
for testcase in testcases:
    print testcase,parseNumbers(testcase)

Here, I just used re.findall, which finds all patterns that look like £([\d,]+). This is anything that starts with £ and is followed by an arbitrary sequence of digits and commas. The parenthesis tells python to extract only the bit after the £ sign. The last thing I do is I remove commas, and parse the remaining string into an integer. You could be more elegant about this I guess, but it works.
using this function in pandas
df['salary_list'] = df['salary'].apply(parseNumbers)
df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)
df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)

Checking if this all works:
import pandas
df = pandas.DataFrame(testcases,columns = ['salary'])
df['minsalary'] = df['salary'].apply(parseNumbers).apply(min)
df['maxsalary'] = df['salary'].apply(parseNumbers).apply(max)
df

    salary  minsalary   maxsalary
0   £23,000 to £100,000 23000   100000
1   £34,000 34000   34000
2   £10000  10000   10000

The advantages of moving the parsing logic to a separate function is that:

it may be reusable in other code
it is easier to read for others, even if they aren't pandas experts
it's easier to develop and test the parsing functionality in isolation


Error Metric for Prediction of sales quantity (by using Attributes of item) I trying to predict sales quantity of particular item by using attributes of item.Not by using Time series.I have currently 3 years of past data.I am performing this by using Neural Networks.
What is the Error metric need to follow
in this case ?

This is a fairly common task, especially in fields like economics, but the choice of using a Neural Network makes your approach different from most.

Having said that, I think you can still use common validation metrics:

Mean Error
Mean Absolute Error
Absolute Error ("delta")

You probably don't need anything more fancy than that, especially if those are the error metrics that your business clients will be using to measure your success (which I imagine would be the case).

Having said that, if you want a fancier ML metric to use whilst developing your algorithm you might try Root Mean Squared Error (RMSE). Here's an article on the topic.

If you wanted to, you could also construct a pseudo-R-squared metric, but those are less used with ANN's and I wouldn't choose it myself for this particular case.


Classify tweets by topic I am approaching machine learning for the first time because of my studies. I have been given a bunch of tweets and the goal is to classify them per topic. I really have no clue on how this should be done. Is there a particular way to follow?
Until now, I have only found topics and was thinking about making a DTM-like dataframe for the training data containing not only the number of times not-sparse words occur but also the number of times particular N-grams occur and a ground truth column with the topic.
Is this totally wrong? How else could I train a classifier without having features?

Since there are no predefined topics, the task is unsupervised: the goal is to group tweets which are semantically similar together (as opposed to classification, which requires training a model to predict among specific classes).
The standard unsupervised approach is topic modelling. In the traditional LDA approach, a topic model groups the documents into clusters and also provides the probability of a word given a topic, so a list of "top words" by topic can be extracted from the model. LDA requires the number of topics as input parameter but Hierarchical Dirichlet Processes can be used to avoid this issue (it's less common however).


Input for Sales Forecasting I want perform demand forecast for particular item based on attributes.Did I need to train the model with unsold items ? by maintaining sales Quantity as zero or  go with only items sold in training period.

If you want to estimate demand, no, you do not need to train the model with unsold items for which you have supply, since they do not affect it. Having an excess supply/capacity does not affect your customer behaviour.

If you have supply, but no demand, it needs to be there.

However, also take into account that if you have had a demand greater than the supply at any moment, you have to do proper demand unconstraining to have a good forecast of your demand.


How to train an image dataset in TensorFlow? As I am new to TensorFlow, I would like to do image recognition in TensorFlow using Python. For this Image Recognition I would like to train my own image dataset and test that dataset. Please answer me how to train a dataset and how to select the dataset..

If you are interested in training tensorflow model for image recognition in general, it is a good idea to go through tensorflow tutorials on MNIST:

https://www.tensorflow.org/get_started/mnist/beginners
https://www.tensorflow.org/get_started/mnist/pros

If you want to collect the data yourself and train model on it, here is something which could be interesting: https://digits-draw-recognize.herokuapp.com/

This is my site with handwritten digit recognition. I have collected dataset by myself - I drew ~1k digits and processed the data. After this I trained two models, one of them is CNN in Tensorflow. And these models train on the new images drawn by people. You can find some description of functionality on this site.


Classifier for large number of labels I have a merchants dataset with 800,000 samples and 18,000 labels. Each sample is associated with a single label and the labels are independent.

An example sample looks like

description, label
int'l 0028240525 amazon uk retail amazon.co.uk, Amazon

In addition to the existing samples there will be new retailers added to the dataset. In this case there may well only be a single sample for that new retailer.

To summarise, I need a classifier that

handles a large number of labels (~18,000, independent, single label per sample)
is able to classify undersampled labels (i.e. a single retailer)

Is there an approach that will handle both? Perhaps two separate classifiers makes more sense?

For multiclass classification problems there are multiple algorithms which are inherently built in a way to be able to solve them.
Some examples: kNN, naive bayes, decision trees...

For the performance to be accurate on all labels and for the classifier to show little bias, you can use other approaches: you can oversample minority classes or undersample majority classes, in a way that all the labels have the same number of points associated with them.

Here you can find some interesting answers about how to fight against class imbalances on decision tree classification: https://stats.stackexchange.com/questions/28029/training-a-decision-tree-against-unbalanced-data


why naive is needed in Naive Bayes ,what happens if naive is not included in Bayes theorem? Im trying to understand why naive is needed in Naive Bayes and everyone says Naive Bayes assumes the input features (predictors) are not correlated hence they are not dependent on each other .

i want to understand what will happen if features are dependent i.e Non Naive (the opposite part of being naive )

if we have a sentence "You won lottery for 1million" and we need to classify it as spam and not spam using naive bayes .

p(y|x)=p(x|y).p(y)

In the likelihood partwe will model the probability as p(x|y)

here x="You won lottery for 1million" and y=spam or not spam

p('You won lottery for 1million'|y=spam)

p('You won lottery for 1million'|y=notspam)

what is the correct way of writting this probability and finding its value without considering independence of event among X ?

should it be written as

**to find probaiblity of spam given feature are depenent**
p('You |won, lottery, for, 1million,spam) *
p('won| lottery, for, 1million,spam) *
p('lottery| for, 1million,spam)*
p(for| 1million,spam)*
p( 1million|spam)

**to find probaiblity of not spam given feature are depenent**
p('You |won, lottery, for, 1million,notspam) *
p('won| lottery, for, 1million,notspam) *
p('lottery| for, 1million,notspam)*
p(for| 1million,notspam)*
p( 1million|notspam)

is this correct way of finding the probability of the X considering its events are dependent on each other ? Should the spam/notspam also be included in the dependence part ?

What is the probelm in finding the above 2 probability and why is it so hard that naive has to pitch in and make the features as independent inorder to calculate probability .

I do not think your formulation is correct. What you have described are just conditional distributions for each word in the sentence but not the joint conditional distribution, given a specific class.

In your case, we have by Bayes rule:

$$ Pr(spam | X) \propto Pr(X | spam) \times Pr(spam) = Pr(you, won, lottery, for, 1million | spam) \times Pr(spam).$$

Now, $\hat Pr(spam) = \frac{\text{# of spam occurences in train}}{\text{total observations in train}}$, the MLE that uses none of the features.

In Naive Bayes, we make the assumption that:

$$Pr(you, won, lottery, for, 1million | spam) \times Pr(spam) = Pr(you | spam) * Pr(won | spam) * Pr(lottery | spam) * Pr(for|spam) * Pr(1million | spam) * Pr(spam).$$ That is, we assumed that each of the features are conditionally independent given a class. If we do not wish to make this assumption, we simply leave $Pr(you, won, lottery, for, 1million | spam)$ as is in the above expression and posit some sort of multivariate/joint conditional distribution.

The reason why we make this assumption is due to the difficulty in determining what the conditional joint distribution of the features is given a particular class. Is the joint distribution a multivariate Poisson? A complex mixture of many? Some other multivariate counting distribution? What is the dependence structure, and how should we represent it? What if we have a thousand features? How do we represent the dependence structure now in a reasonable way? Is the dependence structure so complicated that representing it reasonably is more annoying and/or time-consuming than just assuming independence?

These questions are not trivial and are very difficult, if not impossible, to answer. Thus, we make the independence assumption to simplify the problem. We may not know the joint conditional distribution but perhaps we can derive and/or visualize the marginal conditional distributions for each feature. This, of course, comes at the cost of assuming independence between features which is highly unlikely to be true (especially for word frequencies) but nevertheless occasionally effective for some problems (kind of).


Lowering learning rate makes my accuracy on the validation set go down I'm using XGBoost and my mean absolute error on the validation set goes up when I change it from 0.05 to 0.03, I thought a smaller learning rate only makes it run slower and will if anything increase the accuracy of the model because all it does is make the step smaller, so if anything it's less prone to over shooting.

So maybe I don't know the learning rate as well as I thought.

Why does this happen?

PS. Is there a good way to figure out the learning rate and other parameters like n_estimators (in this case for XGBoost) and so on other than trial and error?

Optimizer functions in ML algorithms update themselves with learning rate to converge local minimums. When the step size (here learning rate = eta) gets smaller the function may not converge since there are not enough steps with this small learning rate (step size).

You should increase your learning rate or number of steps while keeping the learning rate constant to deal with the problem.

You may want to look at gradient descent: https://en.wikipedia.org/wiki/Gradient_descent

Also following image explains very well how to tune learning rate. (source: https://www.jeremyjordan.me/nn-learning-rate/)


Mapping of categorical features into binary indicator features I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator features, the following question is proposed:

  Is it a good idea to map a categorical feature with n values to log2(n) binary features?

Why wouldn't that be the case, seeing as how much resources could be spared by working with fewer features? Does this approach depend on the model that is being used?

Compactifying the data like this saves space in memory, but it adds false relationships that one-hot encoding doesn't.

Let's consider a categorical feature with levels A, B, C, D, that you decide to encode as 00, 01, 10, 11 respectively.

In a linear model, you only get three parameters (a constant and one for each new feature); you can fit "the right" parameters to hit A, B, and C, but then fixing those parameters determines what happens on D, and may be a very poor fit there.  (You're now assuming essentially that (B-A)+(C-A)=D-A.)

In k-NN, the distance between any two levels in the one-hot encoding is the same, 1.  In this encoding, the distance between A and B is 1, but the distance between A and D is $\sqrt{2}$.

In SVM, the set {A,D} is not separable from {B,C}.  With one-hot encoding, everything is separable.

In a tree model, using the raw categorical allows the tree to split any subset of the levels against the rest; using this binary encoding forces the tree to split (AB|CD) or (AC|BD) [missing (AD|BC)]; using one-hot encoding forces the tree to split (A|BCD) or (B|ACD) or (C|ABD) or (D|ABC).  The tree can split differently subsequently and eventually recover an arbitrary split, but a greedily built tree might never accomplish that.

Notice in particular that in these last three models, we've made A and D "more different" from each other than we might have reason to believe.  And in this small example, the catches were fairly small/few, but as the dimension increases these tend to become more exaggerated.

Now, it may still be the case that it's worth doing, but these are some things to consider.


Tool/dataset for matching first names and nicknames I'm trying to identify the same individuals in a large dataset where sometimes the individuals may be listed by their full first name (e.g., "Michael Douglas") and sometimes by its nickname (e.g., "Mike Douglas"). Does anyone know of a dataset that has (at least) English first names and their corresponding common nicknames? Or a tool that will do such matching for you? I searched for a while and couldn't find anything like this.

Maybe this helps. I could not check the content, but it says that it provides a csv mapping names and nicknames.

https://github.com/carltonnorthern/nickname-and-diminutive-names-lookup


CNN with Multi channel input or CNN with Multi instance learning? I have 500 Dicom images of medical scan of patients. These are 3 dimension scans , shape = [300 x 300 x 3]. From these I have extracted Front and side views. So, for each patient I have 2 images of shape [300 x 300].

In order to build a classifier,
Should stack these 2 views and train a CNN {[300 x 300 x 2] x 500} ->  Multi channel input,

Or should i pass each view as a new data altogether {[300 x 300 x 1000]} -> Multi instance learning?

One of the other approach you can try is multi input & single output approach in which passing these 2 images to 2 different layer from there different layer then concatenation of the layers to perform classification task.

It can be done using keras Functional API

input1 = keras.layers.Input(shape=(1,))
input2 = keras.layers.Input(shape=(1,))

cnn1   = keras.layers.Conv2D(32, 3, activation=keras.activation.relu)(input1)
cnn2   = keras.layers.Conv2D(32, 3, activation=keras.activation.relu)(input2)

merged = keras.layers.Concatenate(axis=1)([input1, input2])

dense1 = keras.layers.Dense(2, activation=keras.activations.relu)(merged)
output = keras.layers.Dense(1, activation=keras.activations.sigmoid)(dense1)
model10 = keras.models.Model(inputs=[input1, input2], output=output)

When passing data to model create separate arrays of input those 2 images set.

model0.fit([array_1, array_2],output, batch_size=16, epochs=100)


What is the simplest neural network for the simplest nonlinear function $f(x,y) =xy$ How do I capture $y = x_1 x_2$ using a simple neural network with commonly used activation functions? I assume that I need at least one hidden layer. What mix of commonly activation functions should I use?

So far, I have used $max(0,x)$ and $tanh$ activation for the hidden layer, but the gradient descent diverges very quickly.

Some thoughts which may be useful:

$$ (x_1 y_1)^2 = \exp(\log(x_1^2) + \log(x_2^2)) $$

Probably you need to do one or more of:

Decrease learning rate. Diverging loss is often a symptom of learning rate too high.
Increase number of hidden neurons. The output function will be the combination of many "patches" each created by a neuron that has learnt a different bias. The shape and quality of each patch is determined by the activation functions, but almost any nonlinear activation function used in an NN library should work to make a universal function approximator.
Normalise inputs. If you are training with high values for $x_1$ or $x_2$, this could make it harder to train unless you normalise your training data.

For your purposes, it might be an idea to skip the need for normalisation by training with $x_1$ and $x_2$ in range -2.0 to 2.0. It doesn't change your goal much to do this, and removes one potential problem.

You should note that a network trained against an open-ended function like this (where there are no logical bounds on the input params) will not learn to extrapolate. It never "learns" the function itself, but an approximation of it close to the supplied examples. When you supply  a $(x_1, x_2)$ input far from the training examples, the output is likely to completely mismatch your original function.


You can use the pandas rolling method for that.

back = 3
cols = ['high','low']

# calculate rolling mean on the dataframe's columns of interest
df_back_average = df[cols].rolling(back).mean()

# rename columns before concatenating the result back
df_back_average.columns = str(back) + '_back_average_' + df_back_average.columns

pd.concat([df[cols],df_back_average], axis=1)

high        low         3_back_average_high     3_back_average_low
0.000442    0.000445    NaN                     NaN
0.004580    0.004650    NaN                     NaN
0.004820    0.004890    0.003281                0.003328
0.004890    0.004920    0.004763                0.004820
0.004750    0.004920    0.004820                0.004910
0.004610    0.004860    0.004750                0.004900
0.004620    0.004800    0.004660                0.004860


Should I use MongoDB instead of storing data in CSV in python? I am currently storing data crawled from multiple websites having same but still different structure so every crawler is saving data in separate csv. I am planning to store the data using MongoDB instead of storing it in csv.

Will this be beneficial in saving space ?
Overall will this be advantageous to do or will there be any drawbacks apart from me having to change the code ?

If you're a beginner it might be useful to experiment with a database. In realistic settings, data is usually found in databases. So knowing how to interact with a database is also important for a data scientist.

cv data is tabular, while mongodb is key-value so it is also an opportunity to explore new data representation schemes.


Does recall has different interpretation when comes to classification and information retrieval Recall Definition

In terms of classification

  The recall is defined as no of positive instances that are correctly detected by the classifier.

$$
TP = \frac{TP}{(TP+FN)}
$$

In terms of Information retrieval

  Recall is defined as No of relevant documents that are retrieved.

 Let p = No of relevant documents that are retrieved   and
     q = Total number of relevant documents

 Formula  = p/q

Do both the definition work on any common understanding?. I still can't get my head through this. If both are different, why are they using the same name recall?

According to wikipedia, Recall is defined as-

In information retrieval, recall is the fraction of the relevant documents that are successfully retrieved.

In your second formula, recall = p/q, where q is total number of relevant documents. If you define a confusion matrix for binary classification, you will see that q is actually sum of TP and FN. And p is already (by definition) is TP.

Now lets get the definition of TP and FN. TP (True Positive) is number of predicted positive docs out of number of actual positive docs. FN (False Negative) is number of predicted negative docs out of number of actual positive docs. So, basically when you add TP and FN, it becomes total number of positive docs.

So, let say out of 50 positive (actual) docs, you get 35 positive and 15 negative predictions. So, recall by your first formula is

$35/(35+15) = 35/50$

and by your second formula, $p = 35$ and $q = 50$, so recall is

$p/q = 35/50$

Hope this helps. Please comment if I am wrong about something or missed anything.


How do I build an image dataset for CNN? I don't understand how images are actually fed into a CNN. If I have a directory containing a few thousand images, what steps do I need to take in order to feed them to a neural network (for instance resizing, grey scale, labeling, etc)? I don't understand how the labeling of an image works. What would this dataset actually look like? Or can you not look at it at all (something like a table)?

This is a very packed question. Let's try to go through it and I will try to provide some example for image processing using a CNN.

Pre-processing the data

Pre-processing the data such as resizing, and grey scale is the first step of your machine learning pipeline. Most deep learning frameworks will require your training data to all have the same shape. So it is best to resize your images to some standard.

Whenever training any kind of machine learning model it is important to remember the bias variance trade-off. The more complex the model the harder it will be to train it. That means it is best to limit the number of model parameters in your model. You can lower the number of inputs to your model by downsampling the images. Greyscaling is often used for the same reason. If the colors in the images do not contain any distinguishing information then you can reduce the number of inputs by a third by greyscaling.

There are a number of other pre-processing methods which can be used depending on your data. It is also a good idea to do some data augmentation, this is altering your input data slightly without changing the resulting label to increase the number of instances you have to train your model.

How to structure the data?

The shape of the variable which you will use as the input for your CNN will depend on the package you choose. I prefer using tensorflow, which is developed by Google. If you are planning on using a pretty standard architecture, then there is a very useful wrapper library named Keras which will help make designing and training a CNN very easy.

When using tensorflow you will want to get your set of images into a numpy matrix. The first dimension is your instances, then your image dimensions and finally the last dimension is for channels.

So for example if you are using MNIST data as shown below, then you are working with greyscale images which each have dimensions 28 by 28. Then the numpy matrix shape that you would feed into your deep learning model would be (n, 28, 28, 1), where $n$ is the number of images you have in your dataset.

How to label images?

For most data the labeling would need to be done manually. This is often named data collection and is the hardest and most expensive part of any machine learning solution. It is often best to either use readily available data, or to use less complex models and more pre-processing if the data is just unavailable.

Here is an example of the use of a CNN for the MNIST dataset

First we load the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

  Training data shape:  (60000, 28, 28) Testing data shape :  (10000,
  28, 28)

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

Then we need to reshape our data to add the channel dimension at the end of our numpy matrix. Furthermore, we will one-hot encode the labels. So you will have 10 output neurons, where each represent a different class.

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Now we design our model

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Finally we can train the model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))


Document classification - optimal classifier & embedding Planning a classifier to recognize specific documents based on text. Categories are mutually exclusive and include: job application resumes, paychecks, quarterly financial reports etc, and data is text extracted from doc, pdf, xls etc.

At the feature engineering stage, with questions in mind:

One Multiclass classifier or a number of One vs. Rest (one per category)?
Word embedding (Word2Vec/GloVe) - one embedding for all categories or a separate embedding for each category?

My tendency is towards a number of one vs. rest classifiers (to allow for dynamic use of just some of the categories), and one embedding for all categories (to save prediction time).

Project architecture: initially basic LogReg, but will soon develop to CNN / CNN+RNN.

Project scale:

Training: a few hundreds per category
Prediction: millions of files

Please feel free to share any insights or references, thank you

I would suggest to have a look at fasttext, it will learn word embeddings for you from your corpus (advantage over word2vec embedding is that it learns representations based on character ngrams, so if a word is there, but its plural is not, it will still have a similar representation) and then in the supervised learning mode it uses something very simple like regression for classification, provided you have examples of labelled data.
I tried it with trivial 20newsgroups and it works well for distinct categories. Maybe not a production-friendly solution (it's in c++, but can be stretched to python, albeit not without pain), but can give an idea.
Given the nature of the algorithm, it's quick, compared to nets.


NN with 1 hidden layer: Number of half planes visualised Using the tensorflow playground I get the result below for the

circle shaped sample data
1 hidden layer with relu
3 hidden units

From the 3 hidden units and relu 3 splitting lines are generated. From these I expect ${3\choose 0} + {3\choose 1} + {3\choose 2} = 7$ segments, of which the finite segment should be a triangle. However, the plot (below) on the right shows a hexagon.

I then manually extracted the weights that the tensorflow app optimized and generate my own plots below in R. Note that the 3 half planes generated by the straight lines (of each hidden layer) look the same as in the screenshot of the web application. However the final output (top left of the plot below) is a triangle not a hexagon. See the code below.
What am I a doing wrong here?

library(tidyverse)
library(gridExtra )
#Weights as taken from the optimal solution in the tensorflow playground
W <- data.frame(
    W11 = c(1.2, 0.31,-1.4),
    W12 = c(1,-1.5,0.56),
    b = c(-0.51,-0.68,-0.51),
    W2 = c(-1.6,-1.7,-1.6 )
)

#Grid
G <- expand.grid(
    x = seq(-6,6,0.05),
    y = seq(-6,6,0.05)
) %>%
mutate(
    Node1 = NA,
    Node2 = NA,
    Node3 = NA,
    output = NA
)

#fill the Grid
for (i in 1:nrow(G))
{
    for (j in 1:nrow(W))
    {
        G[i,paste0("Node",j)] <- max(G$x[i] * W[j,1] +  G$y[i] * W[j,2] + W[j,3],0)
    }
    G$output[i] <- G$Node1[i] * W$W2[1] + G$Node2[i] * W$W2[2] + G$Node3[i] * W$W2[3]
}

p1 <- G %>% mutate( Node1 = ifelse( Node1<=0,-1,1)) %>% ggplot(aes(x = x, y = y, color =  Node1) ) + geom_point()
p2 <- G %>% mutate( Node2= ifelse( Node2<=0,-1,1)) %>%     ggplot(aes(x = x, y = y, color =  Node2) ) + geom_point()
p3 <- G %>% mutate( Node3= ifelse( Node3<=0,-1,1)) %>%  ggplot(aes(x = x, y = y, color =  Node3) ) + geom_point()
p4 <- G %>% mutate( output = ifelse( output<0,-1,1)) %>%     ggplot(aes(x = x, y = y, color =  output) ) + geom_point()
grid.arrange(p4, p1, p2,p3, nrow = 2)

A nice experiment!  I think you've just forgotten to include the final neuron's bias term.
As to the original approach, I think you're right that the hidden layer's output will be piecewise linear with 7 regions.  But after the activation at the final neuron you can cut each of those regions into two based on sign.  Plotting more of a heatmap for the actual outputs might be worthwhile.


Ensemble Techniques - Bagging | Subset size I do have a question on ensemble techniques Baggging/Boosting.
- What would be the subset size for Bagging?

like said in a previous answer, the exact subsample parameter value depends on your data.

But a usual starting parameter that gets you good results in general, and doesn't hurt the data distribution much is 0.9.

Taking out 10% of your data at each iteration or newly constructed tree, makes your model generalize a little better. You can try out different variations of the 0.9 and see the results.


Does an RL agent learn during exploitation? I have started with RL and have some doubts regarding it.

Does an RL agent learn during exploitation, or does it only learn during exploration?
Is it possible to train a model only using exploitation (i.e. where exploration is not allowed)?

It depends on the game the agent is playing. If there are rewards all over the environment, the agent learns only when the coefficient of exploration is greater than zero. That is, if you are only allowing it to exploit, the agent may aswell just exploit the first reward it meets which ends a game.

In this case, it will find the first reward that ends the game, and will not change its algorithm (will not learn any other way of playing). On the other hand, if you allow it to explore, it may eventually find another better strategy (learn).

It is always good to conciliate the ratio of exploration and exploitation. It should always be capable of exploring, even if the coefficient is low. That is the whole advantage of Reinforcement Learning.


Merging common Columns values in two DataFrame Pandas I have a dataset of patients from which I want to predict whether patient suffering from diabetes or not.
In that I have a DataFrame  res_total_Df with columns suppose [patient_id, urine output, Blood pressure] and

another DataFrame key_df with columns suppose [patient_id , haemoglobin, Blood pressure].

I want to merge into single dataFrame in which common columns values should be added as list(for which later I would take mean). Resultant dataFrame would be [patient_id, urine output, haemoglobin, Blood pressure].

How can I achieve it using pandas. the number of columns in second dataFrame can vary because I am extracting them from the text. I cannot paste the dataFrame snapshot because it is health data.

Here is my code

def find_common_col(res_total_df, key_df):
    # patient_id is always common
    common_list = ['patient_id']
    for i in res_total_df.columns:
        if i in key_df.columns:
            if i not in common_list:
                common_list.append(i)

common_col = find_common_col(res_total_Df, key_df)
res_total_Df = pd.merge(res_total_Df, key_df, how='outer', on=common_col)

The problem with above code it has duplicates rows. Suppose Blood pressure value for patient_id 1993 in res_total_Df is 180 and in key_df is 200. Then it adds two rows one with value 180 and other with value 200 for patient_id 1993.

It want Blood pressure for patient_id 1993 as [180, 200].

Dropping your duplicate rows and merging on 'Patient ID' should solve your issue.

data = pd.merge(res_total_Df, key_df,
                        left_on=['patient_id'], right_on = ['patient_id'], how = 'left')


How to aggregate features to a group level as a feature in machine learning model? I am building a model to predict some behavior at a household level. I could roll up income or number of cars etc so that I can take everyone into consideration. But how can I roll up something like age? I don't know if it makes sense to add up ages or education. Is there a cleaver way to still use those features at a aggregated level?

Ages
I guess adding up the ages won't makes sense, since the number 30 means that there is one person of 20 years and one of 10, or a bunch of 5 years old?
What you could do is have number_of_adults_in_household and number_of_children_in_household. This way you still aggregate the ages but keep more information.
Education
Education might be a more difficult feature to aggregate. My educated suggestion would be to have highest_education_degree_in_household.


Handling categorical variables in linear regression and random forest In the linear regression, when we have a categorical explanatory variable with $n$ levels, we usually remove one level and call it a baseline level and fit the model on the remaining levels. And the final intercept is the intercept plus the coefficient of baseline level. Now my questions are:

Does it matter which level I choose to remove? I am working on a dataset to predict house price. When I use linear model with intercept on the testset for prediction, I get negative values for some houses which is not correct, as house price cannot be negative. But when I fit a regression without an intercept all the prediction values are positive as expected. I guess it has to do something with the baseline level I chose.
Do I have to remove one level from all my categorical variables in the dataset before fit a random forest, KNN, Ridge, Lasso,... too?!

For linear regression, we have to do one hot encoding and it creates one less number of variables then levels of the categorical variable. In latest tools you don't have to do it manually it automatically does I have tried in R. One hot encoding has no impact on which level you choose, it is basically a binary way of representing the level True or False. This will create whole new variables which with just 1 and 0.
For tree-based algorithms, you basically don't need to apply the one hot encoding. These algorithms are capable enough to handle the categorical variables.

Reference: https://gerardnico.com/wiki/data_mining/dummy


Creating a pie chart with historical motion in Rstudio So, I created several pie charts showing voting percentages for republicans and democrats, from 2008 to 2016 with plotly package in Rstudio. What I would like to do, to make it sleek, is create a menu for a user to click on(2008, 2012, 2016), and the pie chart would motion into the correct vote totals. Would I need to create a shiny app to do this?

Shiny App, will be the place to display the graphics, however the animation you are looking for is not something that plotly would create, to the best of my knowledge.

d3 however is a different beast. It can create aesthetically pleasing graphics. For your specific case, here is the example

https://bl.ocks.org/mbostock/1346410

Or you can look at the entire gallery


How to detect cardboard boxes using Neural Network I'm trying to train a Neural Network how to detect cardboard boxes along with multiple classes of persons (people).

Although it's easy to detect persons and correctly classifies them, it's incredibly hard to detect cardboard boxes.

The boxes look like this:

My suspicion is that box is too simple of an object, and the neural network has a hard time detecting it because there are too little features to extract from the object.

The division of the dataset looks like this:

personA: 1160
personB: 1651
personC: 2136
person: 1959
box: 2798

Persons are wearing different safety items, based on the items are classified, while detected as whole person, not just the item.

I tried to use:

ssd300_incetpionv2
ssd512_inceptionv2
faster_rcnn_inceptionv2

All of these are detecting and classifying persons much better than boxes. I cannot provide exact mAP (don't have it).

Any ideas?

Thanks.

As you said:

  My suspicion is that box is too simple of an object, and the neural
  network has a hard time detecting it because there are too little
  features to extract from the object.

... and that is exactly the problem with this task. I suggest you to train your Network using a whole lot of image augmentation. If you are using Keras/TensorFlow 2.0, they have built-in functions that do that.

I also suggest you to train a first model, then study accurately the images that your Network thinks are boxes but aren't (the false positives). At that point, you select the "correct" cases, together with the false positive cases, and build a subset of the dataset that you have. That sub-dataset can be useful to train the Network to distinguish between actual boxes and objects that look like boxes.

Once again: use a massive load of image augmentation. That's my main point, and it's what I would do.


How to draw a simple LSTM network I'm new to deep learning, I am learning LSTM for my PhD work. This is a simple LSTM network for sequence classification.
This code is from MATLAB tutorial:

layers = [sequenceInputLayer(1)
    lstmLayer(5,'OutputMode','last')
    fullyConnectedLayer(3)
    softmaxLayer
    classificationLayer];

For simplicity, the dimension of the input sequence is 1, there are 3 classes.

I am trying to draw the diagram of this network. This is my attempt:

Is this right? Should the LSTM blue units be connected? The orange units are the softmax layer, is there any symbol the should be on each unit (like ∑)? The same question for each layer? Should be any extra layer to represent the "classificationLayer"? The fullyConnectedLayer is represented implicitly in the full connection in the last layer, Do I need to add any extra layer for this? Any other comments, please?

When thinking of rnn/lstm/gru layer keep few points in mind.

What is your inputs size? In this case we have 5 word sentence so 5 input circle going into lstm layer with that every word value will be multiple by same weight value.
How many lstm cell will be present inside lstm layer? To keep it simple as we have 5 words we are keeping 5 lstm cell for each word then carry forwarding it's memory into next lstm cell.
How many output categories we have? The last lstm cell will be connected to the all 3 dense layer neurons and at each of the neurons softmax operation will happend just like normal fully connected neural network dense layer works.


Minimal example: Keras functional API & multi-input/multi-output regression Problem:

I have a regression problem, where I want to predict two or more numerical outcomes $y_i$ based on a number of numerical features $X_i$. The model would look like: $$y_{1,i}, y_{2,i} = \beta X_i + u_i.$$

I understand that the Keras functional API could be used to solve a problem like this. In an NN setting I would use ordinary densely connected layers to solve the problem. However, I have no experiance with the functional API and I'm not sure if the problem can be solved (at all) using the functional API.

Question:

Can someone point me to a full-fledged minimal example on how to use the functional API for multi-input/multi-output regression as outlined above? Ideally, I'm looking for something like the code examples provided for more basic problems as described in "Deep learning with R" (code can be Python or R as I work with both).

Any additional comments on model/method choice are highly appreciated as well. However, please note that I don't want to estimate/train individual models for each outcome $y_i$, I really aim at predicting all outcomes in one model.

Here is a possible solution:

import numpy as np
import pandas as pd
from keras.datasets import boston_housing
(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()

# Standardise data
mean = train_data.mean(axis=0)
train_data -= mean
std = train_data.std(axis=0)
train_data /= std
test_data -= mean
test_data /= std

# Add an additional target (just add some random noise to the original one)
import random
train_targets2 = train_targets + random.uniform(0, 0.1)
test_targets2   = test_targets + random.uniform(0, 0.1)

# https://keras.io/models/model/
from keras import models
from keras import layers
from keras.layers import Input, Dense
from keras.models import Model
from keras import regularizers
from keras.layers.normalization import BatchNormalization

# Input and model architecture
Input_1=Input(shape=(13, ))
x = Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.05))(Input_1)
x = Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(128, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = Dense(8, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)

# Outputs
out1 = Dense(1)(x)
out2 = Dense(1)(x)

# Compile/fit the model
model = Model(inputs=Input_1, outputs=[out1,out2])
model.compile(optimizer = "rmsprop", loss = 'mse')
# Add actual data here in the fit statement
model.fit(train_data, [train_targets,train_targets2], epochs=500, batch_size=4, verbose=0, validation_split=0.8)

# Predict / check type and shape
preds = np.array(model.predict(test_data))
#print(type(preds), preds.shape)
# is a 3D numpy array

# get first part of prediction (column/row/3D layer)
preds0 = preds[0,:,0]
# second part
preds1 = preds[1,:,0]

# Check MAE
from sklearn.metrics import mean_absolute_error
print(mean_absolute_error(test_targets, preds0))
print(mean_absolute_error(test_targets2, preds1))


Tfidfvectorizer in R for character n-grams Is there a similar tool as the tfidf vectorizer which converts a collection of raw documents to a matrix of TF-IDF features in a R package?

See this for reference.

You should check out the text2vec R package. It has TF-IDF capabilities.

Here's the link to the package info:
text2vec


Can neural networks be considered a method for data analysis? I am in the process of writing my thesis.
In this thesis I am stating that neural networks can be considered a method for data analysis.
After a bit of though I am now unsure if this claim is justified.
Wikipedia states:

"Data analysis is a process of inspecting, cleansing, transforming,
and modeling data with the goal of discovering useful information,
informing conclusions, and supporting decision-making."

So, a feedforward network does model the data, but because of its black-box nature it is problably less useful to discover useful information. However, because of its predicive capabilities it certainly could support decission-making. However, the main goal when using neural networks is still making plain predictions, not necessarily support decission making.
What is your take on this?
Can feedforward neural networks be considered a method for data analysis?

There's a lot to say on this topic. First of all, outside of academia, there is an unofficial classification (more slanted toward business types) of machine learning systems (and thus neural networks), which splits them into three categories in order of increasing utility/autonomy: descriptive, predictive and prescriptive.
A descriptive system summarizes data, a predictive one predicts future trends (or anything else not explicitly present in the data) and a prescriptive one recommends courses of action. Neural networks certainly do more than summarize data, so most of them fall in the predictive category, but with additional techniques they can and do reach the prescriptive category.
To support decision making, as you mentioned, there are a number of desirable properties and capabilities a system should have, but the one I will focus on since you emphasized it is model interpretability, which will bring us further down the line to neural networks as methods of data analysis.
Neural networks, as is, are eminently uninterpretable, black-box models: often, this leads people to adopt simpler models, such as decision trees or random forests. However, this isn't a fact of life: a lot of useful information can be extracted from what a neural network has learned, information that humans can understand.
In the case of networks for simple tabular data (e.g. house prices), [https://eli5.readthedocs.io/en/latest/blackbox/index.html] offers permutation important to estimate the predictive power of features (e.g. surface area matters, color of walls not so much). It also offers a more complex method, LIME, which builds a surrogate linear model of the neural network (which is much more easily interpreted). LIME can also be used for CNNs. Other interpretability methods include partial plots and SHAP.
Since we're on the topic of CNNs, other methods tailored specifically for them exist, such as the (basically industry standard by now) Grad-CAM, which highlights the contribution of parts of an image to the output of a CNN.
If you consult literature review papers on explainable AI (XAI), you will notice that one of the main goals is knowledge extraction from trained models.
On top of all that, in my opinion, plain neural networks, without any fancy explanation method applied, can be considered data analysis methods as is. Image segmentation, semantic segmentation, autoencoding, for instance, sound like the definition you cited for data analysis, and they are all done with neural networks.


What conclusion can I get when the variable is influenced by other but there isn't any correlation? I am doing an analytic exploratory analysis.
If the target is a continuous variable and the attributes are all categorical (discrete values), in order to know if exist any influence on the target from the each attribute I am doing the ANOVA-test like this:
fvalue, pvalue = stats.f_oneway(df[y], df[x])
pvalue < 0.5

If that condition is true, there is a dependency between variables.
For all variables I get true dependency with ANOVA, but the values of the correlation are between -0.1 and 0.2.
What kind of conclusion can I do with this?
Can I say for example that if in the variables with negative correlation, I get new values in order to make that correlation a high positive value, then the target will get a higher value?
Any ideas?

First of all as your target value is continuous so it's a regression problem.
If the correlation doesn't give desired value it means there exist no linear relationship between variable and target variable you should look into mutual information.
Also, while interpreting a relationship, one should be careful to not confound correlation and causality, because although a correlation demonstrates that a relationship exists between two variables, it does not automatically imply that one causes the other.
So if you increase the values to get positive correlation of that variable with target it might not necessarily increase the same as there might exist a confounder.
Mutual information versus correlation


Can categorical features be linearly distributed? Still on my early days, so I cannot really say whether categorical values can be used when linear regression is a key element.

Judging from the below plot, based on my dataset, I'd say this possible (judging from this answer) and the graph below shows I can further use them, can't I?

My question is, can I use the linear regression (e.g. the one from sklearn) to calculate the RSME?

I am not sure what you mean with a linear distribution, but you can use a categorical features with a linear regression (that is, if your dependent variable is continuous). One thing to make though is to encode the categorical feature as such, e.g. use one hot encoding.


Accumulators in Spark (PySpark) without global variables? BACKGROUND

Consider the following textbook example which uses accumulators to add vectors.

from pyspark import AccumulatorParam

class VectorAccumulatorParam(AccumulatorParam):

    def zero(self, value):

        dict1 = {i: 0 for i in range(0, len(value))}

        return dict1

    def addInPlace(self, val1, val2):

        for i in val1.keys():
            val1[i] += val2[i]

        return val1

rdd1 = sc.parallelize([{0: 0.3, 1: 0.8, 2: 0.4},
                       {0: 0.2, 1: 0.4, 2: 0.2},
                       {0: -0.1, 1: 0.4, 2: 1.6}])

vector_acc = sc.accumulator({0: 0, 1: 0, 2: 0},
                            VectorAccumulatorParam())

def mapping_fn(x):

    global vector_acc

    vector_acc += x

rdd1.foreach(mapping_fn)
print(vector_acc.value)

This prints {0: 0.4, 1: 1.6, 2: 2.2}, i.e., the element-wise sum of the vectors.

QUESTION

The use of the global scope in mapping_fn() gnaws at me, since it's usually bad practice. Is there a simple way to illustrate how accumulators work without resorting to a global variable? This is all the more confusing since, as far as I understand, the beauty of Spark lies in a share nothing philosophy, and global is precisely the opposite of that.

The problem with sharing nothing is that for reduce functions like sum you need results from multiple elements. There are Spark ways to do this and there is also a sum already implemented, but the easiest code for reduction is using variables and looping. You need to get used to functional programming that Spark uses in order to write Spark way code. Functional programming also allows easier parallelization and faster distributed code.

It would probably be faster to use Spark dataframes like here. And in my opinion they are easier to understand than map-reduce logic.

A possible map-reduce approach (this is how the whole code looks like):

rdd1 = sc.parallelize([{0: 0.3, 1: 0.8, 2: 0.4},
                       {0: 0.2, 1: 0.4, 2: 0.2},
                       {0: -0.1, 1: 0.4, 2: 1.6}])

def acc_fn(acc, row):
    res = {}
    for key,value in row.items():
        res[key] = acc[key] + value
    return res

vector_acc = rdd1.reduce(acc_fn)

print(vector_acc)

vector_acc is now defined as a Python dictionary and not a class. You should access it as a dictionary.


Normalize / Standardize in a Random Forest? If I have a matrix of co-occurring words in conversations of different lengths, is it appropriate to standardize / normalize the data prior to training?

My matrix is set up as follows: one row per two-person conversation, and columns are the words that co-occur between speakers. I cannot help but think that, as a longer conversation will likely comprise more shared words than shorter ones, I should factor this in somehow.

Thanks for the clarification by commenting. Tree-based models do not care about the absolute value that a feature takes. They only care about the order of the values.
Hence, normalization is used mainly in linear models/knn/neural networks because they're affected by absolute values taken by features.

You don't need to normalize/standardize.

Check this post.


Having trouble understanding None in the summary of my Keras model The above code is a sample of a CNN model built using Keras. The first layer is a convolutional layer which will receive images of input_shape = (64, 64, 3), thus meaning that the format of the images is in RBG. The output of the first layer is (None, 62, 62, 32), but isn't in supposed to be (62, 62, 32)? Where does the None come from? Also since my images are (64, 64, 3) doesn't it mean that the filter should have 3 dimensions to cater for the color channels of the image?

The output of the first layer is (None, 62, 62, 32), but isn't in
  supposed to be (62, 62, 32)? Where does the None come from?

The None is just a placeholder saying that the network can input more than one sample at the time. If your input shape was (64, 64, 3) it would mean that your network could one hold one image at the same time. But you want to train with batches of varying length or predict several images at the same time, that's why it's (None, 64, 64, 3).

As an example, if you would send in a batch of 10 images your shape would be (10, 64, 64, 3) and if you changed it to 25 you would have (25, 64, 64, 3).

  Also since my images are (64, 64, 3) doesn't it mean that the filter
  should have 3 dimensions to cater for the color channels of the image?

Technically you could use a 3D filter with Conv3D on images. However, it is not something people do for RGB images. Probably because it gives worse results than using 2D. Instead you treat RGB as three separate 2D channels rather than one 3D channel. Using three dimensions is usually more relevant in video where you instead use time as your third dimension. Here is a Quora where it is discussed more at length:

Why do we use 2D convolution for RGB images which are in fact 3D data cubes? Shouldn't we use 3D convolution for RGB and 4D for video?


bad input shape (5634, 2) I tried everything and I am not sure how to resolve the following error:

  ValueError: bad input shape (5634, 2)

This is my first machine learning example so please bear with me. This is the python code:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
from pylab import rcParams
from IPython import get_ipython

ipy = get_ipython()
if ipy is not None:
    ipy.run_line_magic('matplotlib', 'inline')
# Loading the CSV with pandas
data = pd.read_csv('...WA_Fn-UseC_-Telco-Customer-Churn.csv')

# Data to plot
sizes = data['Churn'].value_counts(sort = True)
colors = ["grey","purple"]
rcParams['figure.figsize'] = 5,5
explode = (0.1, 0)  # explode 1st slice
labels = "Yes","No"
# Plot
plt.pie(sizes, explode=explode, labels=labels, colors=colors,
        autopct='%1.1f%%', shadow=True, startangle=270,)
plt.title('Percentage of Churn in Dataset')
plt.show()

data.drop(['customerID'], axis=1, inplace=True)
tc = (data['TotalCharges'].str.strip())
#print(tc)
data['TotalCharges'] = pd.to_numeric(tc)

data["Churn"] = data["Churn"].eq('Yes').astype(int)
Y = pd.get_dummies(data["Churn"].values).fillna(0)
X = pd.get_dummies(data.drop(labels = ["Churn"],axis = 1)).fillna(0)

print(X.shape) #(7043, 45)
print(Y.shape) #(7043, 2)

# Create Train & Test Data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=101)

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
result = model.fit(X_train, y_train)

Why does the error

  bad input shape (5634, 2)" when X.shape is (7043, 45) Y.shape is
  (7043, 2)?

Logistic Regression expects labelled y_train, you do not need to do OneHotEncoding.

  y : array-like, shape (n_samples,)

Above is from sklearn LogisticRegression.fit.

I think as long as it is label encoded(using LabelEncoder), basically this tool will automatically generate label mapping from your categorical values to unique integers between 0 to n-1, where n represent count of distinct value of the category. LogisticRegresion will work fine with label encoded target value.

Protip : Please use labelencoder or OneHotEncoder to make mapping of categorical values. The problem with get_dummies is that it relies on , suppose you are doing transformation on different separated dataset(e.g. test set contained in another dataframe), suppose that on trainset you have 3 unique categorical values but on test set there are only 2 unique values. Then pd.get_dummies will only give you 2 columns (other than 3) when you are perform it on the test set.


How to use scikit metrics for a statsmodel or vice versa? Am working on binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3558.

I did refer this post but not sure whether it is updated now or anyone has any way to compute this metrics

Currently I use logit model as shown below

model = smm.Logit(y_train, X_train_std)
result=model.fit()
y_pred = result.predict(X_test_std)
print("Accuracy is ", accuracy_score(X_test_std, y_pred))  #throws error from here and all the line below
print(classification_report(X_test_std, y_pred))
print("ACU score is ",roc_auc_score(X_test_std, y_pred))
print("Recall score is",recall_score(X_test_std,y_pred))
print("Precision score is",precision_score(X_test_std,y_pred))
print("F1 score is",f1_score(X_test_std,y_pred))

The reason why I am trying to do this is because statsmodel has p-values, coeff, intervals etc and I was hoping to get the usual metrics through scikit metrics as shown above but it isn't accepted.

On the other hand, Through scikit logistic regression I can get usual metrics and coeff, but what about p-values, conf intervals? Is there anyway to do the reverse?

Can someone help me with this?

Interpreting the code (since the error received is not mentioned), it looks like the code is passing in the X matrix and y-pred to the metrics. According to the documentation, the metrics want the y-true and y-pred. This would lead to an error mentioning incorrect dimensions.

I have used statsmodels and call scikit metrics. Many of the scikit examples, like the above documentation, show arrays being passed in, not from a specific model.

If that is not it, please post the error received.


Ensemble Techniques for multilabel data I observed that Adaboost or Bagging ensemble classifiers present in sklearn only work for single label training data. How do I use these for multilabel data?

It turns out the Adaboost or VotingClassifier doesn't work for multilabel classification data. We get a NotImplementedError. I believe they are working towards it and might be out in next version or so.


python tsne.transform does not exist? I am trying to transform two datasets: x_train and x_test using tsne. I assume the way to do this is to fit tsne to x_train, and then transform x_test and x_train. But, I am not able to transform any of the datasets.

tsne = TSNE(random_state = 420, n_components=2, verbose=1, perplexity=5, n_iter=350).fit(x_train)

I assume that tsne has been fitted to x_train.

But, when I do this:

x_train_tse = tsne.transform(x_subset)

I get:

AttributeError: 'TSNE' object has no attribute 'transform'

Any help will be appreciated. (I know I could do fit_transform, but wouldn't I get the same error on x_test?)

That's a particular peculiarity of TSNE in sklearn:  https://github.com/scikit-learn/scikit-learn/issues/5361


Which machine learning/AI method is this problem related to? I am new to machine learning and AI, and I am working on a problem in which I need to clean a table (database) of text/words. For example I should delete words like and, the, etc. , or replace words like COMP with company or substitute other acronyms with the suitable words. I wanted to find some resources on the techniques proposed for this purpose and to find what else I can do to better clean the table. Thanks for your help.

For example I should delete words like and, the, etc. ...

In natural language processing, this first task is called stop word removal. You can identify them by looking at the words' frequency over the documents; uninformative words appear very often.


What is the Space & Time Complexity of Mini-Batch K-Means clustering algorithm? For vanilla K-Means clustering algorithm I know that the time complexity is:

  Time complexity: O(tknm),

where n is the number of data points, k is the number of clusters, and
t is the number of iterations, m is the dimensionality of the vectors.

So, when I studied about Mini-batch K-Means to make the algorithm converge faster, I wanted to find out what is the Space & Time complexity of it?

Essentially so that I understand well, how much we are optimizing over vanilla K-Means.

Infinite.

Mini-batch k-means never converges, you need to use an iteration limit or similar heuristic, and you can never guarantee to have found a local optimum.

In essence, mini-batch k-means is:

draw a random sample
perform one iteration of k-means using this sample
repeat

Assuming that your sample size is N, 2 takes O(k N m t) time.


Dummy variable for Categorical values The question is in reference to solution of Titanic survival predictionat kaggle . As many have did the similar kind of feature extraction,

They have converted some of the numerical features (Age, Fares) to Categorical types with Nominal labels (1,2,3) using bands.
Does it help for models to fit efficiently if we have all features in categorical values form?

I have been knowing that if you have categorical values, you should convert them into numerical first, and than to DUMMY Variables. But in the solution it is not done, is dummying not needed?

Some libraries auto encode categorical features into numbers before running the model. Pre-processing the categorical features is done explicitly by the programmer or within framework. Can you post more details with your questions?


PCA formulation - Deep Learning book by Ian Goodfellow I am reading this deep learning book by Ian goodfellow. In the PCA formulation in the first chapter i.e Linear Algebra, he mentions the following:

  we need to choose the encoding matrix D. To do so, we revisit the idea of minimizing the L2 distance between inputs and reconstructions. Since we will use the same matrix D to decode all the points, we can no longer consider the points in isolation. Instead, we must minimize the Frobenius norm of the matrix of errors computed over all dimensions and all points.

I can't understand the explanation for why can we no longer see the points in isolation. Would really appreciate if someone could help me out here.

He means that $D$ must work well on all data points, not just a single one. I.e., $$ z_i = Dx_i \;\;\text{ so that }\;\; || (I - D^TD)x_i ||_2^2 \;\;\text{ is small } $$ for all $i$. If we could choose a different matrix, say $M_i$ for each $x_i$, then we could minimize the error $E_i = || x_i - M_i^TM_i x_i ||_2^2$ separately for each $x_i$ in isolation. But since we want a single matrix that works for all points, so we stack them all together and instead optimize
$$ D^* = \arg\min_D \mathcal{E}= \arg\min_D|| X - D^TDX||_F^2 $$
i.e., minimizing across the whole dataset simultaneously.

See e.g. here or here for more.


Reduce multiclass classification targets to binary classification targets in scikit-learn I would like to reduce multiclass classification targets to binary classification targets. Ideally, this mapping would happen within scikit-learn so the same transformation applies during both training and prediction.
I looked at transforming the prediction target (y) documentation but did not see something that would work. Ideally, it would be a classifier version of TransformedTargetRegressor.
Something like this mapping:
targets_multi  = {'A', 'B', 'C', 'D'}
targets_binary = {0: {'A', 'B'},
                  1: {'C', 'D'}}

Of the three stated purposes of pipelines, you'd get the "convenience and encapsulation" one, but not the others:

Joint parameter selection: you don't have any parameters for this transformation.
Safety (from data leak): your transformation is context-specific, so there is no data leakage in applying it to the entire dataset up front.

This feels like something that is the definition of the targets, and is best considered a part of the data retrieval.

sklearn expects transform methods to have input just X and not y.  For the most part, you can work around that by overriding fit_transform from TransformerMixin.  However, nothing downstream will expect to get two return values (transformed X and y), so this won't work.
You can make a little more headway with the imbalanced-learn package, which provides its own Pipeline with more flexible transformation syntax.  The purpose there is to implement resamplers, and that throws a major issue: resamplers do not apply at prediction time.


what is Tensorflow Quantum(TFQ)? Google announced a new open-source library called TensorFlow Quantum(TFQ) so I am curious to know about Tensorflow quantum.

what is TensorFlow Quantum?
How it is useful with an existing TensorFlow library?
what is hybrid quantum-classical machine learning?

I'm one of the SWEs who worked on TensorFlow Quantum. I'll do my best to answer your three questions.

TensorFlow Quantum is a piece of software to incorporate the quantum computing primitives in Cirq into TensorFlow in a native and scalable fashion. It is primarily targeted at researchers.
Quantum computing is statistical in nature, TensorFlow is very useful for (among other things) statistical things like analyzing data and building models at very large scale. Having this ability to leverage the tools of TensorFlow alongside quantum computing workflows will hopefully lead to new research at scale that might not have otherwise been possible.
Hybrid quantum-classical machine learning is machine learning that involves both classical and quantum data. TensorFlow Quantum was designed with the goal of developing a better understanding of quantum data.

If my answers were too short for your liking, these sorts questions are also answered in greater detail over here and here.


Regarding "modification" of feature columns in supervised learning I have a training set with columns as follows:
$x_1,x_2,x_3,x_4, x_3/x_4$.
I want to know if I should consider $x_3$,$x_4$ and $x_3/x_4$ as separate features (as $x_3/x_4$ is derived from other features)? Also I found that dropping $x_3$ as a feature increases my k-fold neg-loss score (marginally!).

The underlying question is how would I know if entertaining $1/x_1$ or $x_1/x_2$ or even $\sqrt{x_4}x_3^4/x_1$... (you get the idea) as another feature column will/won't increase my score?

Is there any method/algorithm to know which "modified" features will increase my score?

I think what you're talking about is called compound features, and it's extremely important because sometimes it captures interactions between certain features which are not readily apparent in considering each column vector (or column) independently.

The answer to your question is, it may or may not, there's really no way to know without looking at the data, for example if we have a dataset consisting of the following features $x_1, x_2, x_3, x_4$. You could create many datasets consisting of mathematical operations between these variables, like you have mentioned in the question.

Now for the reason as to why someone would want to do this. Consider a supervised learning scenario where you're trying to classify a nominal feature with respect to some other features. This is a purely hypothetical example, but let's say we're predicting skin color which has $3$ or $4$ distinct classes based on facial features (absolute nose position in a grid, position of left eye, etc). While you may attain low error, it may be worthwhile considering compound features like position of right eye - position of left eye which could give us more information than their absolute positions.

To answer the latter question, there isn't any real algorithm which does this because they are feature combinations which are based on the data and anything that does this would have to be a trial and error mechanism. Because underlying data similarities may not be apparently visible. One way you could solve this is in your elementary data processing phase, consider the actual possible relationships between data, or at least, what you perceive them to be.


Are there ML Libs in Python robust to missing data? So I was searching on how to handle missing data and came across this post from Machine Learning Mastery.
This article states that some algorithms can be made robust to missing data, such as Naive Bayes and KNN.

Not all algorithms fail when there is missing data.
There are algorithms that can be made robust to missing data, such as k-Nearest Neighbors that can ignore a column from a distance measure when a value is missing. Naive Bayes can also support missing values when making a prediction.

But then it says that sklearn implementations are not robust to missing data.

Sadly, the scikit-learn implementations of Naive Bayes, decision trees, and k-Nearest Neighbors are not robust to missing values.

Are there ML libs (preferably in Python, but could also be in other languages) that these algorithms are robust to missing data?

In general Weka can handle missing values, which are marked as ? in the .arff format. However this doesn't mean that every learning algorithm in Weka has a specific mechanism to deal with them, and unfortunately the documentation is a bit poor about this point. So as far as I'm aware, one has to study the source code and/or do some experiments in order to find out exactly what a particular class does with missing values.
Weka is in Java but there is a wrapper for Python.


Solution for TF-IDF Vectorization in Angular project? While making an Angular project to use my text-classification model on unseen data, i struggle in finding a way how to transform text to TFIDF features.

Anyone faced same issue? Maybe a solution on how to complete this in another way?

Angular has nothing to do with TF-idf vectorization. Its a web framework, so if you python/java etc code is modularised and has the right workflow it will work.

Just google how to structure it in angular, its a HUGE topic to give an answer.


What algorithm is used to extract keywords from unstructured texts? I'm using StanfordNLP (java pakage) for twitter sentiment analysis. I want to extract keywords from the tweets, classify the keywords as positive, negative, or neutral, and use those counts to classify the polarity of the tweet.

How can this be done in Java? Please share few examples. What is the most used algorithm for keyword extraction?

The OP asks two different questions: (1) how to extract key words and (2) how to assign keywords a sentiment class (pos/neg/neu). I will address the keyword identification piece in this answer as many others have discussed how to do sentiment analysis (e.g., this post).

The approach I would suggest is a key keyword approach advocated by Mike Scott (author of WordSmith Tools) and Chris Tribble (a computational linguist). As discussed here, the basic approach is to create two corpora, your target corpus which is composed of texts sampled from the texts you are interested in, and a reference corpus, which is typically a much larger corpus (which is often more general in content).

The procedure begins by computing word (or n-gram) frequencies for both corpora. During this process, if a word’s frequency in the target corpus is found to be statistically probable in comparison to the reference corpus (as computed by a chi-squared test and a user defined p-value), it is considered a keyword (Baker, 2004). According to Scott (2006), the process typically identifies three types of words as key:
proper nouns, words that characterize a text’s “aboutness,” and high frequency words that are indicators of style or genre. I discuss the approach in greater length in this 2007 article where I use the method to extract the salient features of academic discourse.

To illustrate with a concrete example, imagine your are interested in identifying the key topics surrounding a target brand (say, Pepsi Cola) as expressed in social media (e.g., twitter). Create two corpora: for your target corpora you create a search for "pepsi" and for the reference corpora you could search for competitor brands of soft drinks ("coke", "mountain dew", "dr. pepper", etc,). When the keyword process has terminated you will be left with all the keywords/topics that differentiate pepsi from other soft drink brands (and as a bonus you will also identify negative keywords... words that occur statistically less frequent in the target corpus).

As you might surmise, the results you get depend upon how the reference corpus is constructed. This, in my opinion, is a feature - as it gives the researcher much greater flexibility in hypothesis testing and data exploration.


Image classification: Strategies for minimal input count I want to classify product images into 8 discrete classes. For several reasons the number of input images need to be as small as possible. Related to this I have 2 questions:

What strategies can I employ to reduce the number or required images
What is the magnitude of images per classes needed (A) in a simple scenario where the image is always isolated, see below or (B) in a complex scenario where the image is a scenery (also below). I know this is hard to tell in general but a rough number would be fine.

Additional information

Architecture

Currently a CNN seems to be most fitting for the problem. I am also thinking about a pre-trained network (removing the deeper layers of a powerful network such as AlexNet).

Data

Imagine I am trying to classify dinner tables by the type of legs they have. In the simple scenario images from 3 different classes could look like this:

Class: regular

Class: outward

Class: Wire

In a complex scenario they might look like this:

Techniques to reduce input data needed

There are a couple of ideas I have to reduce the need for data (please add your own):

Use a pre-trained network (good idea or not?)
Pre-process images (how? fiddle with image orientation?)
Does it make sense to change the images more deeply (such as chopping off the top 50% since I only care for the legs?)

I am looking forward for your thoughts!

After running a few tests and collecting some training data I got a rough impression of how much data is needed. The question is still not answered entirely but I figured these insights might help others:

So how many images did I need?

Architecture:
- Pre-trained Inception V3 model with Imagenet data
- Removed last layer

Data / Images per category:

101 images
258 images
45 images
1064 images
259 images
607 images
161 images

Complexity:
Mixed complex sceneries with simple images.

Result:
The quality depends a lot on how difficult the task is. In my scenario the images to be classified where rather similar to the ones in the training data. In these situations all categories with 160 images and above are recognised rather reliably.

I will do some more testing and update my answer.


Why are the weights of my first layer and last layer in the CNN change while the middle layers don't? The weights of my first and last convolution layers do change in a noticeable way. However, the rest of my convolution layers, in the middle, do not.

I should add that all convolution layers' biases change noticeably, however the kernel weights of the middle layers (as opposed to the first and last layers) do not change significantly.

Has anyone else encountered this?

Can anyone suggest why this is might be happening?

Can I rule out vanishing gradients...?

Any ideas for resolution (if this is at all a problem)?

Thanks in advance.

I'm happy to add details based on comments.

Its ofcourse dataspecific, but this behavior is mostly likely not harmful. What could indicate dangerous behaviours is wild behaviour of cv-metric, learning rates  etc...

You can think of every cnn layer task to learn specific features/pecularities of a photo. In the first layers they will catch the most general patterns, edges etc... Middle layers are more specific and they look at some finer details but not all too specific. And final layers are there for the small details that can really discriminate the classes and make the right prediction.

So in your case weights are not updated significantly because information in middle layers is not that significant.


Derivation of Ward's formula for agglomerative clustering Figure describe formula of ward method for hierarchical or agglomarative clustering i.e. increment of total error after merging two cluster.

How did they get

$$\frac{n_A\cdot n_B}{n_A+n_B} ||\vec m_A-\vec m_B||^2 $$

from first line of formula as shown in figure?

I will be using these facts without proof (but the proofs either follow directly from definitions or are straightforward): The vector of the centroid of a set of points is arithmetic mean of all vectors of the points in the set. Also, the centroid of the union of two sets of point is on the straight line connecting centroids of these two sets, and it divides that straight line into two parts whose ratio is determined by the ratio of number of points of in these sets.

Let's consider following series of identities:

$\sum_{i\in A} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 =$

$$= \sum_{i\in A} \Big\Vert \vec{x}_i-\Big(\vec{m}_{A} - \frac{n_B}{n_A+n_B} (\vec{m}_{A} - \vec{m}_{B})\Big)\Big\Vert^2$$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big) + \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2 $$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + 2 \cdot \sum_{i\in A}\big(\vec{x}_i-\vec{m}_{A}\big) \cdot \frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big) + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$

$$= \sum_{i\in A} \Big\Vert\big(\vec{x}_i-\vec{m}_{A}\big)\Big\Vert^2 + n_A \cdot \Big\Vert\frac{n_B}{n_A+n_B} \big(\vec{m}_{A} - \vec{m}_{B}\big)\Big\Vert^2\big)$$

$$= \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +  \frac{n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Similarly:

$$\sum_{i\in B} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 +  \frac{n_A^2 n_B}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

By adding:

$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 = \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 +\sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 + \frac{n_A^2 n_B + n_A n_B^2}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Or:

$$\sum_{i\in {A\cup B}} \|\vec{x}_i-\vec{m}_{A\cup B}\|^2 - \sum_{i\in A} \Vert\vec{x}_i-\vec{m}_{A}\Vert^2 - \sum_{i\in B} \Vert\vec{x}_i-\vec{m}_{B}\Vert^2 = \frac{n_A n_B (n_A + n_B)}{(n_A+n_B)^2} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$

Or:

$$\Delta \big(A, B\big) = \frac{n_A n_B}{n_A+n_B} \Vert\vec{m}_{A} - \vec{m}_{B}\Vert^2$$


What algorithms are available to identify similarity scores between two records? I have data that looks like this:

priority    task         area
1           clean room   living room
2           clean room   living room
2           water plants bedroom

I want to identify records with a priority of 2 that should be a priority of 1 based on similarity between the priority 2 task and each priority 1 task.

The result data would look like this:

priority    task         area         propensity_for_priority_1
1           clean room   living room  1
2           clean room   bedroom      .5
2           water plants bedroom      0

String similarity does not need to be performed (i.e. living room and bedroom are mutually exclusive). Is there a matching algorithm that can be used to compare various fields and result in one propensity_for_priority_1 score for each record?

2 Approaches.

a) Siamese networks, let the net tell you whats similiar and what lies together.

b) just perform multiclassification if the data permits, I dont know all the details but it looks like it.


Spectral clustering with heat kernel weight matrix I am studying normalized graph cuts, and one of the way to define weight matrix is using heat kernel, which is $W_{ij} = e^{\frac{−∥x_i − x_j∥^2}{σ^2}}$.

I want to ask: what's the meaning of sigma? Does it affect on the partition of the data? How do we pick sigmas? And what happens if its too large or too small?

What happens to the Laplacian matrix L as $\sigma → 0$ and $\sigma → \infty$? What are the eigenvectors and eigenvalues of this matrix?

$\sigma$ represents a typical distance between points. If all points of one cluster of your graph are separated from all points of another by a distance that is significantly higher than $\sigma$, then the spectral clustering will probably use this as a cut.

If you already know the number of clusters (or cuts) that you want to make, $\sigma$ does not need to be tuned very finely. But it becomes really important if you have no idea how many cuts you want to make.

A good approach is to make a parametric study over $\sigma$, and make the decision through the eigen values. Here's an example from one of my study cases, giving eigen values (in ascending order) of the Laplacian matrix, for 4 different values of $\sigma$:

What you are looking for is a break in the increase of eigen values: theoritically, a wide gap between two consecutive values ($n$ and $n+1$) should tell you that it is a good idea to make $n$ clusters.

As you can see, small sigma values (upper left plot) lead to very low eigen values, sometimes even numerically negative. Very small values tend to make very small clusters containing outliers, and a very big cluster with all other points. High sigma values (lower right plot) will make your matrix look like the identity, with high eigen values, this usually won't bring anything interesting.

What I usually do is try a wide window for sigma values, and shorten the range progressively, until I find a satisfying result. This is all graphical, and depends on the prior knowledge about your problem (how many clusters do you approximately expect? 2, 10, 100?).

Eigen values represent the quality of the cut. Small values correspond to very distant clusters. For instance, in my upper right case, I could make 4 very well separated clusters (first 4 eigen values are almost 0). The next cut (to reach 5 clusters) would be less effective than the 3 previous ones. 13 clusters could also have been a good try. But I actually selected 9 clusters, because it was closer to what I actually expected.


Adding a bias makes Q-learning algorithm ineffective I've been working through the Q-Network learning example in this Arthur Juliani's blog. It's based on the pretty trivia Open Gym Frozen Lake example. It's base implementation get's about 47% success rate over 3000 iterations. I decided to add a bias to the implementation, and found that it severely harmed the results to no better than random.

That is, I added the bias term below:

inputs1 = tf.placeholder(shape=[1,16],dtype=tf.float32)
bias = tf.Variable(tf.zeros(shape=[1,4]))
W = tf.Variable(tf.random_uniform([16,4],0,0.01))
Qout = tf.matmul(inputs1,W) + bias
predict = tf.argmax(Qout,1)

The rest of the code is identical to the original solution. Any ideas why this would so negatively affect performance?

Update
It looks like someone else ran into this issue, and the answer given was that

  Having a bias term with the one-hot encoding prevents each state’s Q
  values from being independent

Any ideas why this is the case? The bias is added after the multiplication, so it's in the dimension of the actions, not the inputs. I don't see why this would make learning fail.

Having a bias term with the one-hot encoding prevents each state’s Q values from being independent

Any ideas why this is the case?

The usual point of using linear regression or other function approximation in Q-learning is to generalise, and thus prevent the Q values from being independent deliberately. So this is not a general statement about Q-learning and linear function approximation. In general yes you can add bias terms - also, in general when using linear regression or neural networks to estimate action values, then you should have bias terms.
Having said that, I would expect the bias should make little difference here, since the one hot coding of state makes it redundant - you can already express any mapping of state to Q value of each possible action using just the weights. Although you are using what looks like a linear regression model, essentially this is just the tabular form of Q learning.
Q learning with function approximation can be unstable. The combination of off-policy, bootstrap updates and function approximation is known as the "deadly triad", and you need to add tricks like experience replay to keep it stable. Although in theory the linear regression should learn a correct bias, probably adding bias puts the algorithm into that unstable zone.