Intuition Behind Restricted Boltzmann Machine (RBM) I went through Geoff Hinton's Neural Networks course on Coursera and also through introduction to restricted boltzmann machines, still I didn't understand the intuition behind RBMs.

Why do we need to compute energy in this machine? And what is the use of the probability in this machine? I also saw this video. In the video, he just wrote the probability and energy equations before the computation steps and didn't appear to use it anywhere.

Adding to the above, I am not sure what the likelihood function is for?

RBM's are an interesting beast. To answer your question, and to jog my memory on them, I'll derive RBMs and talk through the derivation. You mentioned that you're confused on the likelihood, so my derivation will be from the perspective of trying to maximize the likelihood. So let's begin.

RBMs contain two different sets of neurons, visible and hidden, I'll denote them $v$ and $h$ respectively. Given a specific configuration of $v$ and $h$, we map it the probability space.

$$p(v,h) = \frac{e^{-E(v,h)}}{Z}$$

There are a couple things more to define. The surrogate function we use to map from a specific configuration to the probability space is called the energy function $E(v,h)$. The $Z$ constant is a normalization factor to ensure that we actually map to the probability space. Now let's get to what we're really looking for; the probability of a set of visible neurons, in other words, the probability of our data.
$$Z = \sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}$$
$$p(v)=\sum_{h \in H}p(v,h)=\frac{\sum_{h \in H}e^{-E(v,h)}}{\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}}$$

Although there are a lot of terms in this equation, it simply comes down to writing the correct probability equations. Hopefully, so far, this has helped you realize why we need energy function to calculate the probability, or what is done more usually the unnormalized probability $p(v)*Z$. The unnormalized probability is used because the partition function $Z$ is very expensive to compute.

Now let's get to the actual learning phase of RBMs. To maximize likelihood, for every data point, we have to take a gradient step to make $p(v)=1$. To get the gradient expressions it takes some mathematical acrobatics. The first thing we do is take the log of $p(v)$. We will be operating in the log probability space from now on in order to make the math feasible.

$$\log(p(v))=\log[\sum_{h \in H}e^{-E(v,h)}]-\log[\sum_{v \in V}\sum_{h \in H}e^{-E(v,h)}]$$
Let's take the gradient with respect to the paremeters in $p(v)$

\begin{align}
\frac{\partial \log(p(v))}{\partial \theta}=&
-\frac{1}{\sum_{h' \in H}e^{-E(v,h')}}\sum_{h' \in H}e^{-E(v,h')}\frac{\partial E(v,h')}{\partial \theta}\\ & + \frac{1}{\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}}\sum_{v' \in V}\sum_{h' \in H}e^{-E(v',h')}\frac{\partial E(v,h)}{\partial \theta}
\end{align}

Now I did this on paper and wrote the semi-final equation down as to not waste a lot of space on this site. I recommend you derive these equations yourself. Now I'll write some equations down that will help out in continuing our derivation. Note that: $Zp(v,h)=e^{-E(v,h')}$, $p(v)=\sum_{h \in H}p(v,h)$ and that $p(h|v) = \frac{p(v,h)}{p(h)}$

\begin{align}
\frac{\partial log(p(v))}{\partial \theta}&=
-\frac{1}{p(v)}\sum_{h' \in H}p(v,h')\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}\\
\frac{\partial log(p(v))}{\partial \theta}&=
-\sum_{h' \in H}p(h'|v)\frac{\partial E(v,h')}{\partial \theta}+\sum_{v' \in V}\sum_{h' \in H}p(v',h')\frac{\partial E(v',h')}{\partial \theta}
\end{align}

And there we go, we derived maximum likelihood estimation for RBM's, if you want you can write the last two terms via expectation of their respective terms (conditional, and joint probability).

Notes on energy function and stochasticity of neurons.

As you can see above in my derivation, I left the definition of the energy function rather vague. And the reason for doing that is that many different versions of RBM implement various energy functions. The one that Hinton describes in the lecture linked above, and shown by @Laurens-Meeus is:
$$E(v,h)=−a^Tv−b^Th−v^TWh.$$

It might be easier to reason about the gradient terms above via the expectation form.

$$\frac{\partial \log(p(v))}{\partial \theta}=
-\mathop{\mathbb{E}}_{p(h'|v)}\frac{\partial E(v,h')}{\partial \theta}+\mathop{\mathbb{E}}_{p(v',h')}\frac{\partial E(v',h')}{\partial \theta}$$

The expectation of the first term is actually really easy to calculate, and that was the genius behind RBMs. By restricting the connection the conditional expectation simply becomes a forward propagation of the RBM with the visible units clamped. This is the so called wake phase in Boltzmann machines. Now calculating the second term is much harder and usually Monte Carlo methods are utilized to do so. Writing the gradient via average of Monte Carlo runs:

$$\frac{\partial \log(p(v))}{\partial \theta}\approx
-\langle \frac{\partial E(v,h')}{\partial \theta}\rangle_{p(h'|v)}+\langle\frac{\partial E(v',h')}{\partial \theta}\rangle_{p(v',h')}$$

Calculating the first term is not hard, as stated above, therefore Monte-Carlo is done over the second term. Monte Carlo methods use random successive sampling of the distribution, to calculate the expectation (sum or integral). Now this random sampling in classical RBM's is defined as setting a unit to be either 0 or 1 based on its probability stochasticly, in other words, get a random uniform number, if it is less than the neurons probability set it to 1, if it is greater than set it to 0.


Machine Learning on Curved Background Ok, maybe this is a really dumb question, but has anyone ever considered extending some of the statistical learning methods to live in some space other that $\mathbb R^n$? My guess is that you're just shuffling parameters around, but I wonder if there's any intuition to be gained.

This question came up as I was working through some course notes on neural nets, and realized that computer scientists may be even more cavalier with tensor indices than physicists :)

Essentially, each non-linear layer in a neural network is a map from $\mathbb{R}^n$ input to a $\mathbb{R}^m$ output. There is no requirement for these dimensions to be separate and uncorrelated, nor for the map to be reversible.

The best performance is usually found if each layer is roughly scaled to approximate a normal distribution with mean 0 and standard deviation 1. However, that is an independent issue to any kind of space-like mapping.

If you know a suitable mapping that relates to your problem and would simplify its expression, then you can apply a spacial transform between co-ordinate systems (typically by mapping the input features) and use it within the neural network. You could map co-ordinates to any curved space if you think it might help. Choosing between polar or cartesian co-ordinates for features is one choice that may crop up for example.

However, even without making this helpful mapping, you can analyse a neural network layer-by-layer, and see that it can effectively learn these kind of spacial mappings from the training data anyway. This is an intuition that has been called the "manifold hypothesis" - the link has some images showing how a neural network might learn to separate classes depending on how they are arranged in feature space, and this article from a DARPA researcher has a nice series of images showing successive transformations of a 2D spiral that allow the last layer to perform linear separation.


Relationship between two continuous variables in time series data I have a dataset that collects daily data based on transactions between two entities.
I wish to find the strength, direction, and kind of relationship between two continuous variables i.e. Number of transactions (No_of_transactions) and Error counts (Error_Counts) over the period of approximately two years.
Error counts are of degree 10²-10⁵ and the number of transactions in 10¹-10³.
I'm pretty sure the relationship is nonlinear because when I draw the scatterplot it is more towards inverse relationship (rightly so) and I believe a simple corr() function won't help much.

You could look at the cross-correlation between the two series for different lags. In short, you would look at the values from one series and see how they correlate with previous (or lagged) values from the other series. Auto-correlation is a special case of cross-correlation where you correlate a series with itself (or its lagged values).

To capture non-linear relationships you could see if you can use one of the series to predict the other. You could phrase it as a forecasting problem and see if adding the other variable improves forecasting accuracy.

For more details, check out this chapter.


Extracting values in Pandas I have a case like below:

  col1
  {hjjlhllllh
  :59:kjkjkjdd
  :50:/kjhjhjhsa:ddd
  :70:jnjjh

  kjkjytdvdjjh:fhghghghg
 :52:jkjkt
 :50K:kkcknkknn
 :72:nbnbds

  mvmmbmnnm{hghghgsl
 :65A:jjhjhjh
 :50B:iut6565vdjbj

:kjkjkj

I want it like below:

  col1                                                            col2

 {hjjlhllllh
  :59:kjkjkjdd
  :50:/kjhjhjhsa:ddd
  :70:jnjjh
                                                               /kjhjhjhsa

 kjkjytdvdjjh:fhghghghg
 :52:jkjkt
 :50K:kkcknkknn
 :72:nbnbds
                                                               kkcknkknn

 mvmmbmnnm{hghghgsl
:65A:jjhjhjh
:50B:iut6565vdjbj:kjkjkj                                     iut6565vdjbj

Now, I am able to get col 2 value for the 1st row by using a logic that whatever appears between '50' & ':', extract it. The Same logic does not stand correct for the other 2 rows where in I have to get that appears between '50K' or '50B' & ':'. I want to extract irrespective whatever be the prefix be for '50' without hardcoding. Can anyone help me in this?

This is a problem of identifying a good regular expression to parse the strings into their component parts. I made an assumption that this data is sitting in a DataFrame and you are trying to add a new column with the extracted string.

Code

import pandas as pd
import re

def extract_second_group(full_string):
    pattern = '^\S*:50.*:\W*(\w*):\S*$'

    match = re.search(pattern, full_string)
    if match is not None:
        return match.groups(0)[0]
    else:
        return None

df = pd.DataFrame(columns=['col1'])

df['col1'] = [
            '{kjkjkjdd:50:/kjhjhjhsa:ddd',
              'kjkjytdvdjjh:50K:kkcknkknn:nbnbds',
              'mvmmbmnnm{:50B:iut6565vdjbj:kjkjkj',
              'not a matching string'
]

df['col2'] = df['col1'].apply(lambda col1:extract_second_group(col1))
df

Explanation

The regex pattern is ^\S*:50.*:\W*(\w*):\S*$. Let's break it down to see what it is doing.

^   : Start of string
\S  : Matches any character which is not a whitespace character
:50 : Match this exact set of characters (in that sequence)
.  : matches any character except a newline
\W : Matches any character which is not a word character (matches the (/) character, if any)
(\w*): This is the most important part of the expression. The \w* matches Unicode word characters, while the parentheses tell us to create a group. We are interested in capturing this group
:   : Another colon separates the group from the rest
\S*  : Another non-whitespace sequence of characters
$   : End of string

The regex is enclosed in a function so that we can check when a regex search doesn't return a match. Call the apply() function to run each row through the function and return the matched group.

Edit Jul 24 2019

This edit is to reflect the change in the data from OP.
The problem remains a question about regular expressions. Here's the code for the updated dataset

Updated Code

import pandas as pd
import re

# https://docs.python.org/3.6/library/re.html
def extract_second_group(full_string):
    pattern = '\W*:50\w{0,1}:(/*[\w]*):\S*$'

    match = re.search(pattern, full_string)
    if match is not None:
        return match.groups(0)[0]
    else:
        return None

df = pd.DataFrame(columns=['col1'])

df['col1'] = [
            '{hjjlhllllh:59:kjkjkjdd:50:/kjhjhjhsa:ddd:70:jnjjh',
              'kjkjytdvdjjh:fhghghghg:52:jkjkt:50K:kkcknkknn:72:nbnbds',
              'mvmmbmnnm{hghghgsl:65A:jjhjhjh:50B:iut6565vdjbj:kjkjkj',
              'not a matching string'
]

df['col2'] = df['col1'].apply(lambda col1:extract_second_group(col1))
df

Output

$$\begin{array}{c|c|}
  & col1 & col2\\
\hline
0& \{hjjlhllllh:59:kjkjkjdd:50:/kjhjhjhsa:ddd:70:j... & /kjhjhjhsa\\
\hline
1& kjkjytdvdjjh:fhghghghg:52:jkjkt:50K:kkcknkknn:... & kkcknkknn\\
\hline
2& mvmmbmnnm\{hghghgsl:65A:jjhjhjh:50B:iut6565vdjb... & iut6565vdjbj\\
\hline
3& not a matching string & None\\
\end{array}$$

Updated Regex
The updated pattern is \W*:50\w{0,1}:(/*[\w]*):\S*$.
I updated it as follows

\W* Start with some non-word characters
:50\w{0,1} Followed by :50 and 0 or 1 Unicode word characters
:  Followed by a colon
(/[\w]) Followed by the group we want. I included one or more / and any number of Unicode word characters.
:  Followed by a colon
\S*$ Followed by any number of non-space characters to end of string

If there are other patterns you need to match, you can use the resources from https://docs.python.org/3.6/library/re.html to locate the different options available in regex patterns.


How to create a new column based on two other columns in Pandas? I am searching for a way to create a new column in my data. I have tried using iterows() but found it extremely time consuming in my dataset containing 40 lakh rows. So here is what I want.
Consider I have 2 columns:
Event ID, TeamID ,I want to find the no. of unique TeamID under each EventID as a new column. In other words, I want to find the number of teams participating in each event as a new column.

You can try something like this to get a new dataframe that has pairs of (EventID, TeamCount):

event_id_team_count = data.groupby('EventID').agg({'TeamID': lambda x: x.nunique()})
event_id_team_count.rename(columns={"TeamID": "TeamCount"}, inplace=True)

If you want to have this new column in the original dataframe, all you need to do is to join the original dataframe with the one you have just created:

data = data.join(other=event_id_team_count, on="EventID")


Determining first name and surname from a name pair? Suppose I'm given a list of name pairs:

[
    [ 'John', 'Smith'],
    [ 'Alex', 'Gordon'],
    ...
]

I wish to know (or be given reasonable certainty) which column is the surname, and which is the first name, assuming that all columns are consistent, and each name pair consists of exactly one first name and one surname.

An obvious challenge would be checking name pairs where both names can be used as first names, e.g. ['Graham', 'Tyler'].

Is this model something that is reasonable to achieve?

You can apply a simpler strategy here that will likely lead to better results than a neural network. Luckily we have census data for most countries. For example you can use US census data.

We can find the commonality of the two names in the array using their search function. Then we can say that the name with the higher rank is the first name. For example we can see that for the name ['John', 'Smith'], John has a rank of 27 for the most recent year whereas Smith is not listed. So we can assume that John is the first name and Smith is the last name.

This technique is a very old school means of prediction, we already have the distribution of the data through polls. So we can use it directly, we do not need to train a model to learn the underlining distribution (this would be very difficult with something like names which is has a lot of noise).

If you have additional data for each person like a date of birth or a gender then you will get better results through this prior information. This is similar to what you would achieve when applying Naive Bayes algorithm, however in this instance once again the distribution has already been determined.


Difference between using RMSE and nDCG to evaluate Recommender Systems What kind of error measures do RMSE and nDCG give while evaluating a recommender system, and how do I know when to use one over the other? If you could give an example of when to use each, that would be great as well!

nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1.

RMSE (Root Mean Squared Error) is typically used to evaluate regression problems where the output (a predicted scalar value) is compared with the true scalar value output for a given data point.

So, if you are simply recommending a score (such as recommending a movie rating), then use RMSE. Whereas, if you are recommending a list of items (such as a list of related movies), then use nDCG.


Sub-sampling so that sample statistics match population statistics I want to investigate the impact of various testing strategies on a product. Let's say chairs. I start with 500 random chairs that I've picked up from garage/yard sales. They come in all shapes and sizes, different manufacturers etc, but I've carefully measured each one and record: manufacturer, height, width, depth and fabric type. I calculate my population parameters. I find that some parameters are normal, and others are uniform, fabric types are mostly cotton but some leather.

I want to split my 500 chairs into groups of 100 such that each group has similar sample statistics. This way I can differentiate the impact of the various test on the chairs without worrying that I'm actually observing differences in my input distribution. I.e. I don't want all the leather chairs in one group.

I've tried randomly grouping my dataset, but I always end up with bad bias in one statistic. I thought that it might be possible to start with random grouping and then randomly selecting a pair of chairs to swap groups; recalculate group statistics; if they get closer to population parameter, keep the swap, else revert; repeat. That seems dreadfully slow.

I'm sure that there are many solutions available, but I'm not sure what they're called: what should I be searching for?
If you have a solution handy, I don't really mind what language it is presented in.
I'll add additional tags to help others find this sort of thing in the future.
Thanks!

The term you are looking for is stratified sampling : https://en.wikipedia.org/wiki/Stratified_sampling. It's a way to sample from population that can be partitioned into sub-populations. More specifically for your problem, look at what they do for clinical trials, when they parition patients in sub-groups.


how to save deep learning model and test it after training? I have a CNN model written using tensorflow for python, the model is for classifying lung CT images (cancer/no-cancer), after training the model with training and validation data and get a reasonable accuracy, after all, that I need to test the model with test data, but I don't know how to do that? how to save the model and using it for testing?

You can find the details in this tutorial: Save CNN model

To summarize:

Tensorflow variables are only alive inside a session. So, you have to save the model inside a session by calling save method on saver object.

import tensorflow as tf
saver = tf.train.Saver()
sess = tf.Session()
sess.run(tf.global_variables_initializer())
saver.save(sess, 'my_test_model')

For saving the model after 1000 iterations, call save by passing the step count:

saver.save(sess, 'my_test_model',global_step=1000)

To use  pre-trained model for fine-tuning:

with tf.Session() as sess:
    saver = tf.train.import_meta_graph('my-model-1000.meta')
    saver.restore(sess,tf.train.latest_checkpoint('./'))
    print(sess.run('w1:0'))
##Model has been restored. Above statement will print the saved value of w1.

To add more operations to the graph by adding more layers and then train it.

sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))

# Now, let's access and create placeholders variables and
# create feed-dict to feed new data

graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

#Add more to the current graph
add_on_op = tf.multiply(op_to_restore,2)

print sess.run(add_on_op,feed_dict)
#This will print 120.


TypeError: object of type 'int' has no len(), LogisticRegression() I'm trying to fit my Logistic Regression model, but I'm running into an error that I don't understand. Looked around and haven't found a straight answer.

Shape of independent features (X): (495,30)

Shape of dependent feature (y): (495,)

The dependent feature holds binary values (0,1) and the values are currently of type "int". This is probably where the problem is occurring, but I don't understand what I need to do to fix the issue.

This is the error I'm getting when I try fitting to sklearn LogisticRegression():

c_param = [1,10,100,1000]

rand_grid = {'kbest__k':list(range(5,10)),
            'log__C':c_param,
            'log__random_state':42}

log_rand = RandomizedSearchCV(pipe_opt('log',LogisticRegression()),rand_grid,n_iter=100,cv=3,
                              random_state=42,n_jobs=-1)
log_rand.fit(X_train,y_train)

RandomizedSearchCV expects you provide dictionary of parameters-values to try, like in your code above:

rand_grid = {'log__C':[1,10,100,1000],
             'log__random_state':42} # values should be in []

Since the idea is to search values which give the best performance, you are expected to provide several values, as a list of values. Though in your case you only provided only one value, it won't raise error as long as it was written as list (inside []).

However it's a bit weird if you only have one value to search. So better you fix those values when you initialize the estimator (i.e LogisticRegression())., e.g:

# you want to see which values perform best for params C and max_iter.
rand_grid = {'log__C':[1,10,100,1000]
             'log__max_iter': [100,200,300]}

# you have fix value for params random_state, penalty and fit_intercept
logReg = LogisticRegression(random_state=42, penalty='l1', fit_intercept=False)

log_rand = RandomizedSearchCV(pipe_opt('log',logReg),
                              rand_grid,n_iter=100,cv=3,
                              random_state=42,n_jobs=-1)


Word2vec outperforming BERT, possible? I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After numerous tries, it seems that Word2vec-CNN keeps outperforming BERT-CNN by a slight bit, here's a result from my last try:
Word2vec-CNN
precision (macro): 0.89
recall (macro): 0.87
f1-score (macro): 0.88
accuracy (test set): 0.81
hamming loss: 0.062

BERT-CNN
precision (macro): 0.86
recall (macro): 0.88
f1-score (macro): 0.87
accuracy (test set): 0.74
hamming loss: 0.073

Question is:

Could it be possible that Word2vec (or any static word embeddings) outperforms BERT (or any contextual word embeddings)? If so, what is the rationale? If there's any research paper on this it would be really helpful.
If not, what could possibly be the cause?

FWIW: Model is trained using TensorFlow-Keras (I kind of suspect this is SOMEHOW caused by how TF-Keras calculates its metrics but I still haven't figured out why and, if any, a solution), and both embeddings are pretrained (BERT model was trained on a bigger corpus, around 200:1).

Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss.
This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model needs to capture relationships between fewer tokens and can model those relationships better than a subword model which loses the relationships between fixed tokens in your data and instead tries to generalize relationships across >30K subword tokens (in the bert-based-uncased model), which could lead to noise.


Ideal difference in the training accuracy and testing accuracy In a data classification problem (with supervised learning), what should be the ideal difference in the training set accuracy and testing set accuracy? What should be the ideal range? Is a difference of 5% between the accuracy of training and testing set okay? Or does it signify overfitting?

Theoretically speaking, in a perfect scenario, training and test data both represent the distribution of your problem accurately. Therefore, in an ideal case, training and testing should not have any significant differences in accuracy. This becomes more and more true when you have lots of data.

A difference of 5% is perfectly fine. In practice, it is common that training accuracy is slightly better than the test accuracy. I will say that the difference may not be the best indicator. What you should look at is correlation. Meaning that, as long as training and testing accuracy improve together at a similar rate, you're in the clear, regardless of how far apart they are. You can investigate that by training and evaluating on increasingly bigger subsets of the data. Ideally, training and testing should both improve as you add data. If test data starts decreasing, you have overfitting.


How to limit a function's cosine value as it approaches 0 How do I get a cosine to peak up to a certain value as it hits 0? With the above code, the y-value peaks to infinity as it approaches 0.
import matplotlib.pyplot as plt
import numpy as np

amp = 1
time = np.linspace(-5, 5, 1000)

signal = np.cos(1 * np.pi * time) * abs(1/time)

plt.figure(figsize=(10, 10))
plt.ylim(-4, 4)
plt.plot(time, signal, color="black")
plt.savefig('output.png', bbox_inches='tight', pad_inches=0,
            transparent=True)

Put another way, based on my google-fu, how can I create a symmetrical exponentially decaying sinusoid with a maximum y-value as it approaches 0?

$\alpha e^{-\beta x^2}cos(x)$ where $\alpha$ is your peak and $\beta$ is proportional to the strength of the decay
This one's slightly longer and the peak is at $\alpha+1$ but the graph is easier to manipulate:

$cos(x)e^{-\beta x^2}+\alpha e^{-x^2}$


What is the difference between a hashing vectorizer and a tfidf vectorizer I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer

I understand that a HashingVectorizer does not take into consideration the IDF scores like a TfidfVectorizer does. The reason I'm still working with a HashingVectorizer is the flexibility it gives while dealing with huge datasets, as explained here and here. (My original dataset has 30 million documents)

Currently, I am working with a sample of 45339 documents, so, I have the ability to work with a TfidfVectorizer also. When I use these two vectorizers on the same 45339 documents, the matrices that I get are different.

hashing = HashingVectorizer()
with LSM('corpus.db')) as corpus:
    hashing_matrix = hashing.fit_transform(corpus)
print(hashing_matrix.shape)

  hashing matrix shape (45339, 1048576)

tfidf = TfidfVectorizer()
with LSM('corpus.db')) as corpus:
    tfidf_matrix = tfidf.fit_transform(corpus)
print(tfidf_matrix.shape)

  tfidf matrix shape (45339, 663307)

I want to understand better the differences between a HashingVectorizer and a TfidfVectorizer, and the reason why these matrices are in different sizes - particularly in the number of words/terms.

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here.

Hash functions are an efficient way of mapping terms to features; it doesn’t necessarily need to be applied only to term frequencies but that’s how HashingVectorizer is employed here. Along with the 45339 documents, I suspect the feature vector is of length 1048576 because it’s the default 2^20 n_features; you could reduce this and make it less expensive to process but with an increased risk of collision, where the function maps different terms to the same feature.

Depending on the use case for the word vectors, it may be possible to reduce the length of the hash feature vector (and thus complexity) significantly with acceptable loss to accuracy/effectiveness (due to increased collision). Scikit-learn has some hashing parameters that can assist, for example alternate_sign.

If the hashing matrix is wider than the dictionary, it will mean that many of the column entries in the hashing matrix will be empty, and not just because a given document doesn't contain a specific term but because they're empty across the whole matrix. If it is not, it might send multiple terms to the same feature hash - this is the 'collision' we've been talking about. HashingVectorizer has a setting that works to mitigate this called alternate_sign that's on by default, described here.

‘Term frequency - inverse document frequency’ takes term frequencies in each document and weights them by penalising words that appear more frequently across the whole corpus. The intuition is that terms found situationally are more likely to be representative of a specific document’s topic. This is different to a hashing function in that it is necessary to have a full dictionary of words in the corpus in order to calculate the inverse document frequency. I expect your tf.idf matrix dimensions are 45339 documents by 663307 words in the corpus; Manning et al provide more detail and examples of calculation.

‘Mining of Massive Datasets’ by Leskovec et al has a ton of detail on both feature hashing and tf.idf, the authors made the pdf available here.


What is the difference between Freidman mse and mse? The original question was posted on StackOverflow. However, taking into account the recommendations of Sergey Bushmanov, I'm providing an answer through this medium.

As Sergey has pointed out on the original question, going through the following link one finds the reference to the original article. So, we're going to proced by unraveling the components of each formula (explaining each of them) and then doing a proper interpretation of the splitting criterion that Jerome Friedmann proposed.

1. Probabilities $p_{k}$

First, let's consider
\begin{align}
    p_{k} = exp(F_{k}(x)) / \sum_{l = 1} ^{K} \exp(F_{l}(x)),
\end{align}

where $F(x)$ is a logistic transformation. The term $p_{k}$ determine an exponential way to consider probabilities to belong to a particular $k$ class, that gives a mayor probability to the events that are more likely to happen and a lesser probability to the ones that are less likely to happen. For example, consider that the values that $F(x)$ could take are 1, 7 and 49, thus

\begin{align}
     \frac{\exp(49)}{\exp(1) + \exp(7) + \exp(49)} \approx .9999 > \frac{\exp(7)}{\exp(1) + \exp(7) + \exp(49)} \approx 0.
\end{align}

In other words, greater values of $F(x)$ are more likely to happen.

2. Weights $w_{l}$

Once we know that $p_{k} \in [0,1)$, we arrive at the definition of the weights

\begin{align}
    w_{l}  = \sum_{i \in R_{l}} w_{l}(x_{i}) = \sum_{i \in R_{l}}p_{k}(x_{i})(1 - p_{k}(x_{i})),
\end{align}
for the $x$'s in the region $R_{l}$. In the other words, the weights are the sum of the probabilities, that in the given region $l$, every $x_{i}$ belongs to a certain $k$-class. This is fundamental to understand the criteron proposed by Friedman, as we would see next.

3. Least-Squares Improvement Criterion

The least-squares improvement criterion is

\begin{align}
      i^2(R_{l}, R_{r}) = \frac{w_{l}w_{r}}{w_{l} + w_{r}}(\overline{y_{l}}-\overline{y_{r}})^2  \:\:\:\:\:\:\:\:(1),
\end{align}

considering the fact that we would like to split the current region $R$ into two subregions $(R_{l}, R_{r})$. Remember that at the end we would like to do the split as clean as possible, that is to say, the outcomes are close to the average outcome of all $k$ cases at that node. Therefore, this splitting criterion allow us to take the decision not only on how close we're to the desired outcome, but also based on the probabilities of the desired $k$-class that we're going to find in the region $l$ or in the region $r$.

Disclaimer: This is what I understood by reading the original paper by Friedman, any errors should be pointed out. Thanks


In Neural network, if one node is deleted, where should other nodes be connected? If it's a fully connected neural network, should we just remove those lines that were originally connected to the deleted node, and hence no actual changes on the remaining nodes? Except there weights and bias will be updated?

Whether you are actually removing the nodes or simply deactivating them, more or less the same concept would apply. The idea is called Dropout and is employed to inhibit the tendency to overfit. Suppose you're removing a node from the $i^{th}$ layer which has $n_i$ nodes. Removing it will not only involve removing the connections from the previous layers (${n_{i-1}}$ connections) but also removing the connections from the removed node to the next layer ($n_{i+1}$ connections) which would  essentially remove $n_{i-1}+n_{i+1}$ connections.
It would be much better to simply set the weights and biases to $0$. Here's the math behind the idea and you can find one possible implementation here.


How to download a Jupyter Notebook from GitHub? This is a fairly basic question.

I am working on a data science project inside of a Pandas tutorial. I can access my Jupyter notebooks through my Anaconda installation. The only problem is that the tutorial notebooks (exercise files) are on GitHub.

My question: how do I download the exercise files from GitHub and then have them display in the Jupyter notebook section on my computer so that I can use them interactively?

I am very new to Jupyter Notebooks. So far, I have been using PyCharm to execute all of my code.

You can do :

Open the jupyter notebook you want to run.
Click on: Raw

Save Ctrl+S
Remove:  .txt

Now navigate to the directory where notebook is downloaded in jupyter notebook and open it.


Organization of layers in Keras for a NLP problem I have been trying out an NLP problem where I have to predict multi-label-sentiments for some text.

I have 8 labels and 170k training examples, and 140k for the test set.
My final dictionary size is around 190k.

I am using Keras for trying out an NN approach, although I am not sure if my architecture is right, Below is the model which gives me a 95 % on the test set, I am testing out accuracies on 0.7 - 0.3 split while training:

model = Sequential
model.add(Embedding(max_indexes + 1, 100, weights=[embeddings],
                  input_length=100))

model.add(Bidirectional(LSTM(256, return_sequences=True)))

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 5, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPool1D())

model.add(Convolution1D(256, 4, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 3, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())

model.add(Convolution1D(256, 3, padding='same'))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(MaxPool1D())

model.add(Flatten())

model.add(Dense(64, activation='relu',
                kernel_regularizer=keras.regularizers.l2(0.02)))

model.add(Dense(target_classes_len, activation='sigmoid',
                kernel_regularizer=keras.regularizers.l2(0.02)))
model.add(Dropout(0.1))

adam_opt = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

early_stopping = EarlyStopping(monitor='val_loss', patience=5, mode='min')

save_best = ModelCheckpoint('model_x.hdf', save_best_only=True,
                               monitor='val_loss', mode='min')
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2,
                              patience=5, min_lr=0.0001)

model.compile(adam_opt, 'binary_crossentropy', metrics=['accuracy'])

Over the internet, and while watching a video of CSE231N and other StackOverflow questions, I found out I should follow:

[(CONV-RELU)*n - POOL? ]*m - (FC-RELU)*K, Softmax
 N~5, M~very large, K>=0 and K<=2

Although it was for ImageNet classification problem and I have an NLP problem at hand.

Also, I found out that should place DropOut after final FullyConnected and BatchNormalization in a sequence with CONV-RELU.

I justed wanted to know what would a good organization of layers for an NLP problem, or any way of evaluating different architectures?

Thanks

In this kind of problems, one needs to be sure what model he/she wants to use. A convolution network would be great with final softmax layer giving 8 outputs. you don't need a bidirectional layer in your network architecture. Try a simple conv network and make sure your data pipeline is working then you can add more layers and make it complex.


Neural style transfer loss fucntion In the loss function between generated image and content image, we calculate the error taking the only activation of the corresponding channel but for calculation of loss function between style and generated image we calculate the gram matrix.
But we should do the same for the loss function between content and generated image to find the correlation between different channels.

So why we do not use gram matrix in loss function calculation between content and generated?
Computational efficiency is not an option :)

So why we do not use gram matrix in loss function calculation between content and generated?

"Same content" means same objects in same locations in the image. If the original content shows a face at the top, we want to see a stylised face at the top.
"Same style" means same combinations of textures and line style together in any location in the image. The gram matrix is a good proxy measure of this, because it is invariant to the locations at which the style correlations occur, and similar line shapes and textures will tend to trigger the same combinations of low-level feature map channels.

If you used gram matrix for the higher order content channels, you would probably find that larger objects would be treated like a style and would be moved, duplicated and distorted in the final output. Depending on the trained network, which layer(s) you picked as representing content etc, the effect might still be interesting or artistic. But it would no longer be style transfer, but perhaps some hybrid of style transfer and deep dream.


Generate pdf from jupyter notebook without code I have jupyter notebook which contains markdown, code and outputs (graphs). I want to generate pdf from this notebook.
I tried to hide code using html code which I get from here then I tried to download as pdf but again code shows up. But when I download as html it don't show any code but again when I tried to convert html to pdf it again shows code.

Try this:
jupyter nbconvert --to pdf --TemplateExporter.exclude_input=True  my_notebook.ipynb

This also works for html output.
You will find the documentation for this and other options here.
FYI, for complex notebooks, this may generate errors depending on your version of nbconvert, LaTeX and other components. In that case try to convert to html then print to pdf using a browser.


Quantifying the imbalanceness of a dataset after looking a lot in the literature there is really a lot of how to work with imbalanced dataset but so far I can not find a

definition of a imbalanced
metric that quantifies how much imbalanced is dataset A compared to dataset B.

I have even tried to define one such metric myself but I fail to capture with a single metric the following two cases:

One class has all the measurements
Measurements are more balanced but one or two classes have very low measurement or even zero.

Can you give me some more information from the literature that perhaps I was not able to find out or at least try to discuss the issue of definition of imbalanceness .

Thanks a lot
Alex

I think "balanced" can have different meanings in different contexts, but typically a "balanced" dataset is one in which the class labels are uniformly distributed.  Of course, an "imbalanced" dataset is one in which the distribution of class labels is not uniform.

I'm not aware of discussions about balance metrics in the literature.  One straightforward option is to measure the KL-divergence between the actual distribution of the class labels and a uniform distribution over the class labels.  A KL-divergence of zero would indicate a perfectly balanced dataset.  The KL-divergence will grow to infinity as the actual distribution differs from the perfectly-balanced distribution. This metric would allow you to compare the degree of imbalance between two datasets.

Unfortunately, KL-divergence is pretty fragile.  It's technically undefined when one of the classes has zero examples.  In practice you could treat such datasets as if they're infinitely imbalanced. This is what scipy.stats.entropy does.

However, this treatment has one undesirable side-effect.  Any dataset in which there are zero examples of a class gets the same "imbalance score" (infinity). This doesn't match our intuition about imbalance.  Suppose we have two datasets with examples from the classes "dog", "cat", and "horse".  Let's say dataset A has 19 dogs, 1 cat, and 0 horses.  Dataset B has 10 dogs, 10 cats, and 0 horses.
Ideally our metric would say that Dataset A is more imbalanced than Dataset B, but KL-divergence will give the same result for both datasets.

We can correct this problem by adding a tiny epsilon to each of the class distributions before calculating KL-divergence.  This both solves the problem of being undefined when a class has zero examples, and it also corrects for the above scenario where two different datasets get the same imbalance score.


ValueError trying to use a pickled scikit-learn model I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am getting the below error.

trainFile=os.path.join('D:\PYPrograms','Data','POS','collected.csv')
#load the data
train  = pd.read_csv(trainFile)
dataTemp=train
nullInTrain=train.shape[0] - train.dropna().shape[0]
print("Null values in Train data "+str(nullInTrain))
dataTemp.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
dataTemp.loc[:,"title"] = dataTemp.title.apply(lambda x : " ".join(re.findall('[\w]+',x)))
df1 = dataTemp.dropna()
cv1 = CountVectorizer()
df_x = df1["tickettype"]+" "+df1["title"]
df_y = df1["type"]
X_train, X_test, y_train, y_test = train_test_split(df_x, df_y, test_size=0.2, random_state=0)
x_traincv = cv1.fit_transform(X_train)
x_testcv = cv1.transform(X_test)
clf = RandomForestClassifier(n_estimators = 1000, max_depth = 6)
clf.fit(x_traincv,y_train)
pred=clf.predict(x_testcv)
pred
#make prediction and check model's accuracy
predictions_test = clf.predict(x_testcv)
acc =  accuracy_score(np.array(y_test),predictions_test)
print ('The accuracy of Random Forest is {}'.format(acc))
import pickle
modelFile=os.path.join('D:\PYPrograms','Data','model2')
with open(modelFile, 'wb') as picklefile:
    pickle.dump(clf,picklefile)

with open(modelFile, 'rb') as training_model:
    model = pickle.load(training_model)

cv2 = CountVectorizer()
File=os.path.join('D:\PYPrograms','Data','POS','Report_one_wk08.csv')
data = pd.read_csv(File)
data.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
test = cv2.fit_transform(data['title'])
model.predict(test)

Error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-205-c0ac8462bce6> in <module>
----> 1 model.predict(test)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
    543             The predicted classes.
    544         """
--> 545         proba = self.predict_proba(X)
    546
    547         if self.n_outputs_ == 1:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, X)
    586         check_is_fitted(self, 'estimators_')
    587         # Check data
--> 588         X = self._validate_X_predict(X)
    589
    590         # Assign chunk of trees to jobs

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in _validate_X_predict(self, X)
    357                                  "call `fit` before exploiting the model.")
    358
--> 359         return self.estimators_[0]._validate_X_predict(X, check_input=True)
    360
    361     @property

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, X, check_input)
    400                              "match the input. Model n_features is %s and "
    401                              "input n_features is %s "
--> 402                              % (self.n_features_, n_features))
    403
    404         return X

ValueError: Number of features of the model must match the input. Model n_features is 6639 and input n_features is 3

Data available at https://drive.google.com/open?id=1xaKKSXzpr7THezqU_8jycfvAueg0nnCQ

You have to use the same CountVectorizer instance on all data and have a method to handle out of training sample tokens.


Simple way to visualise word2vec vector space I want to visualize my word2vec vector space (with zoom in and zoom out). I found a really interesting GitHub project named word2vec explorer. However, it does not seem to work in windows and mac.

Hence, I am interested in knowing if there are any other off-the-shelf project I can use for this?

You can easily visualize word2vec vectors using TensorBoard, which is very easy to understand.

I have made a video about using TensorBoard.


is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN? is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN ?

No, the word artificial in this context is only used to separate the neural networks we build from the natural/biological neural networks that they are loosely based on. But in the realm of data science you can basically use NN and ANN interchangeably and fully connected layers are irrelevant to the terms.

Here is a hierarchy that might make things clearer:

Neural Network
 ├── Natural Neural Network
 |    ├── Brain                         (Not fully connected)
 |    └── ...
 └── Artificial Neural Network
      ├── Dense Neural Network          (Fully connected)
      ├── Convolutional Neural Network  (Not fully connected)
      ├── Recurrent Neural Network      (Not fully connected)
      └── ...


Righteous data analysis approach for this industrial process monitoring I am a R&D operator which is in charge to conduct an evaluation study for implementing a monitoring system for our process of production. I generalized what the software must do but I don't know which is the approach/algorithm to perform what I designed.
Here's the process description:
we produce brake hoses. The hoses are produced in a continuous process. It starts from (1) extruding a plastic tube which get wrapped in reels of about 1000mt. Then (2) these 1000mt of extruded tube get braided with a textile or steel wire. The last phase (3) is coating the braided tube from phase (2) with an extrusor with a "T" head. At the end we get batches of brake hoses in reels of about 1000mt.
The variables in this process are detected by sensors and in order are:
(1)-extruder speed - temperature - vacuum pressure
(1)-diameter of the extruded tube
(1) human operator
(2) speed of the braiding machines
(2) temperature and humidity of the braiding room
(2) human operator of the braiding machine
(3) coating extruder speed - temperature - vacuum pressure
(3) diameter of the coated hose (which the brake hose = the process OUTPUT)
(3) human operator
(3) tests performed by the laboratory onto the brake hose
This process is in the time-domain as the three phases are in chronological order and the process is a linear process (it starts from an extrusion of the inner tube and ends with its coating).
The operators of the laboratory will perform tests on the finished product (the process output) which is identified by a univoque batch code. The software must correlate these data and the diameter detected in the phase (3) to the sensors data of the process of that batch of hose.
Everytime we input data into the system the software must learn to recognize patterns and find correlations among the variables.
For example: last year we had a leakage problem on a hose which wasn't determined and disappeared after one month without significant actions made onto the process. If the software had already been implemented we expected a result from its analysis like this (simplifying in a spoken language):
-"I detected that the batch n°3 had a leakage in the test of "pressure holding" same as batch n°2 and batch n°1. In all these batches I detected the coexistence of these three factors:
1) the presence of the operator John Doe which stopped the braiding machine 2 times
2) the humidity of the braiding room was 2% over the average level of the past 3 months
3) there plastic grains for the extrusion of the inner tube (phase 1) has dried only for 20 minutes instead of 40.

Of course, the software next time the process shows the same conditions listed above warns us that the same defect might happen.

Which kind of algorithm (supervised-reinforced-unsupervised) is best suited for this task and which kind of approach (decision three-neural network?
As I'm making a study to be submitted to the top management I would like to be well prepared and also don't depend to what a software house says it is the best solution because they tend to be "smart" when somebody is ignorant on an argument.
Thank you
Alessio Cazzaniga

The problem you are looking at is a common researched problem in the field of reliability engineering. If I understand you example correctly your goal is to gain an understanding in the following:

What manufacturing conditions cause certain parts (in your situation hoses) to fail sooner than others?

For each variable you mention you will have to establish whether these are constant over the life time of a part or whether they change. Here I measure time as the time that elapses once the part is used by your client in production. In reliability engineering the variables that do not change over time e.g. human operator of the manufacturing process, production batch, etc. are called time independent covariates the ones that change over time, e.g. amount of pressure put on a hose in production (if you have a sensor that reads that kind of information at your client site), are called time dependent covariates.

You will have to transform your data such that it becomes censored survival data. This type of data means that every dependent variable is amount of time in operation from production. Since some of your hoses will likely never have a defect they will leave your observation period. Note that you do need to take these points into consideration too. When you combine all these data points you get something called right censored survival data.

With the right censored survival data and the covariates together you can start to build a accelerated failure time model. Note that it is recommended to first start with a Kaplan Meier estimate of your survival rates before diving into AFT, but I just want to give you some pointers.

A book that I can highly recommend on reliability engineering is:

Practical Reliability Engineering
The New Weibull Handbook

Both don't dive deep into AFT but they do explain the fundamentals of physical systems and how this links to statistics. From experience I know that not many people are working on this field and that you can have a lot of wrong pointers.

In terms of bringing a model like this in production will heavily depend on the quality of your data. Typically sensor data can be of poor quality, moreover, user input tends to be even of worse quality. An article that gives you an idea of what kind of organizational challenges you may face can be found here. Expect a project like this to take several months to years.


How to make k-means distributed? After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:

def kmeans(data, k, c=None):
    if c is not None:
        centroids = c
    else:
        centroids = []
        centroids = randomize_centroids(data, centroids, k)

    old_centroids = [[] for i in range(k)]

    iterations = 0
    while not (has_converged(centroids, old_centroids, iterations)):
        iterations += 1

        clusters = [[] for i in range(k)]

        # assign data points to clusters
        clusters = euclidean_dist(data, centroids, clusters)

        # recalculate centroids
        index = 0
        for cluster in clusters:
            old_centroids[index] = centroids[index]
            centroids[index] = np.mean(cluster, axis=0).tolist()
            index += 1

    print("The total number of data instances is: " + str(len(data)))

I have tested it for serial execution and it is OK. How to make it distributed in Hadoop? In other words, what should go to the reducer and what to the mapper?

Please note that if possible, I would like to follow the tutorial's style, since it's something I have understood.

Unless you are trying to do this as a learning exercise, just use Spark which has ML libraries made for distributed computing. See here


Relationship between VC dimension and degrees of freedom I'm studying machine learning and I feel there is a strong relationship between the concept of VC dimension and the more classical (statistical) concept of degrees of freedom.

Can anyone explain such a connection?

As stated by Prof Yaser Abu-Mostafa-

  Degrees of freedom are an abstraction of the effective number of parameters. The effective number is based on how many dichotomies one can get, rather than how many real-valued parameters are used. In the case of 2-dimensional perceptron, one can think of slope and intercept (plus a binary degree of freedom for which region goes to +1), or one can think of 3 parameters w_0,w_1,w_2 (though the weights can be simultaneously scaled up or down without affecting the resulting hypothesis). The degrees of freedom, however, are 3 because we have the flexibility to shatter 3 points, not because of one way or another of counting the number of parameters.


Divide a column by itself with mutate_at dplyr Hi I'd like to turn each non zero value of my selected columns to a 1 using mutate_at()

 BRAND  MEDIA_TYPE    INV1       INV2
   <chr>  <chr>      <dbl>     <dbl>
  b1     newspapers    2         27
  b1     magazines     3         0
  b2     newspapers    0         0
  b3     tv            1        145
  b4     newspapers    4         40
  b5     newspapers    5         0
  b1     newspapers    1         0
  b2     newspapers    0         28

The final result should be like follow:

 BRAND  MEDIA_TYPE    INV1      INV2
   <chr>  <chr>      <dbl>     <dbl>
  b1     newspapers    1         1
  b1     magazines     1         0
  b2     newspapers    0         0
  b3     tv            1         1
  b4     newspapers    1         1
  b5     newspapers    1         0
  b1     newspapers    1         0
  b2     newspapers    0         1

Do you have any suggestion on how to solve it?
Thank you!!

mutate_at(my_data, vars(starts_with("INV")), sign)

Although I see your INV1 column is a date? I'm not sure how much that'll affect things..


Is there a R implementation of isolation forest for anomaly detection? Is there a R implementation of isolation forest for anomaly detection?

Similar to the implementation from sklearn.

See the iforest package on Sourceforge or on R-Forge:

This package implements an anomaly detection method that detects data-anomalies using binary trees. Using the property that anomalies are more susceptible to isolation, anomalies can be detected as data points which have short expected path lengths.

  Isolation Forest detects data-anomalies using binary trees. Platform:
  R Reference: Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou, “Isolation
  Forest”, IEEE International Conference on Data Mining 2008 (ICDM 08)

or the isofor package on GitHub:

One efficient way of performing outlier detection in high-dimensional datasets is to use random forests. The isofor ‘isolates’ observations by randomly selecting a feature and then randomly selecting a split
value between the maximum and minimum values of the selected feature.


Adding hand-crafted features to a convolutional neural network (CNN) in TensorFlow Let's say I want to add a few hand-crafted features to a convolutional neural network CNN in TensorFlow.

The CNN can be a simple one as described here.

Naturally I'd like to add these features right after the second pooling and right before the first fully-connected layer (FC1 in the example).

Is that easy to express my method in code? I'd have to append my features to the h_pool2_flat vector/tensor:

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

I figured it out. If we denote the additional features as x_feat, I changed the lines from

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

to

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_pool2_flat = tf.concat( [h_pool2_flat, x_feat ], 1 )
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)


Multiple linear regression for multi-dimensional input and output? Assume that I have $N$ points $x_i,i=1,...,N$ in some $A>1$-dimensional  space $\mathbb{R}^A$ with pointwise evaluations of some function $f:\mathbb{R}^A \rightarrow \mathbb{R}^B$, i.e. $f(x_i),i=1,...,N$ where $f(x_i) \in \mathbb{R}^B$.
It is my goal to find a multiple linear regression between $x_i$ and $f(x_i)$. Now sklearn has a function (sklearn.linear_model.LinearRegression) for a multiple linear regression for functions of the type $f:\mathbb{R}^A \rightarrow \mathbb{R} $, but my output is $B$-dimensional. I assume that I could make independent multiple linear regressions for each output dimension and then combine the results, but there must be an easier way of achieving this.
Do you know of a more efficient way?

You are asking about multioutput regression. The class you talked about sklearn.linear_model.LinearRegression supports this out of the box.
import numpy as np
from sklearn.linear_model import LinearRegression

# features
A = 10
# number of values to predict
B = 15
# number of rows in dataset
m = 100

x = np.ones((m, A))
y = np.ones((m, B))

model = LinearRegression()
model.fit(x, y)

sklearn.linear_model.LinearRegression actually just creates B models. However it optimises calculations using vectorisation.
It actually is exactly the same as a fully connected layer in a neural network which has no activation function.
You can read more about it here:
https://machinelearningmastery.com/multi-output-regression-models-with-python/


Encoding for k-level qualitatative variable I have a qualitative variable, e.g. userId, which could take around 30,000 different coded values ($k$). I would like to represent this variable as a dummy variable. Coding this into a vector of size $k$ doesn't seem to be a good approach. Is there a more compact method for coding for this variable?

There is always clustering techniques you could do that would help you determine how to group userId into a dummy variable.


Can I add categories to a dataset after and retrain an H5 file? Without retraining the previous ones? This is the situation. I'm training a model to recognize letters of the Alphabet. There are 26 classes. When writing the code for 26 classes, and loading nearly 100,000 images to train, I'm having a lot of issues. However, I was able to successfully train the model to work on 10 letter increments. As in, A-J, K-T, and then U-Z. These three work perfectly fine (A-Z does not).

Question: Can I train my A-J model, save the H5. Then, train the K-T, and U-Z after and then MERGE the H5 files together? I understand it's possible to train all A-Z using fewer images and then retrain the model with a different image set, but the issues are coming when I'm doing a 26 class system - hence I'm asking if I can do it in increments of 10 and merge it after.

To get around this issue, I would suggest doing training in batches. This solves the issue of not being able to load 100,000 images and also removes the need to figure out how to merge the models.


Get specific object dimensional position from image I need to get specific object like logo dimensional position from an image. I checked open-cv, pillow package etc but it does not help to me. Can anyone advice which is the best process to do this? Do we need deep learning for it or we can do it any other way.

Assuming you already have a logo sample to detect. In that case you can use OpenCV template detection. You need to pass source logo and target image.
Link - Template Matching using OpenCV


Several fundamental questions about CNN I am trying to make a CNN for 3D image recognition but everything is predicted to only one class out of three. And the prediction even quickly converges during the first epoch.   I have been working on this for an week and totally lost.
I have my own several guess why it always converges to one class.

My 3D tensor is huge as 40x35x30 and there are a lot of white spaces because I had to put my objects in cubic box. Would this be problematic?
I have only ~5000 samples for training and ~500 for test sets. Do I have too little samples?
My labels have 3 classes and ~75% of the whole data belong to the class number 1. During the first epoch,  my model quickly converges to predict 99% of data as this class. Would this unbalanced data cause the problem?
My model is designed as
conv1(kernel_size=10,stride=1,filter=32)-batch_norm1-maxpool1-conv2(8,1,64)-batch_norm2-maxpool2-conv3(8,1,64)-maxpool3-fc4(1024)-batchnorm4-dropout4(20%)-fc5(384)-batchnorm5-dropout5(20%)-fc6(3)
I standardized the input between -1 and 1 I use leaky-relu activation for conv layers. I use Adam optimizer with decay rate of 0.99.
First, I am not sure if it is okay to perform batch normalization at every layer. Do I miss any important concept for designing CNN model here?
or maybe my data is just bad.. I kind of suspect that my data are not significantly different from each other and there is no pattern at all.  In this case, is there any statistical method/model to check if my data have meaningful differences? (The 3D images are some chemical/physical data in 3D space that I converted as numpy matrix)

But I think it is more like vanishing problem because when I initialize the variable with Xavier's way, the convergence to the class #1 is slower. Please someone help me :(

Here's my take on each of your points.

You have very sparse data. Are you storing these data optimally in a sparse object, for example a csr sparse matrix if you're using Keras? This is more likely to affect training time than accuracy, but is something to think about.
5000 training examples and 500 testing examples sounds alright to me, except you may have too few data to fit the type of model you have (you may be severely underfitting). Try a simpler model and see if you can improve results with that (try something stupidly simple like an MLP with one layer and see what happens).
This seems the most likely problem. My first guess is this has to do with the unbalanced classes. If you're using a metric like accuracy to evaluate training at each epoch, I'd recommend instead using something like average_precision or roc_auc_score or f1, all available through scikit-learn. If you're  using Keras, try using class_weights as well, which will weight underrepresented classes higher in the loss function, essentially biasing the algorithm to consider underrepresented classes on an equal playing field. If you're not using Keras, try implementing some similar class weighting scheme.
Batch normalization is less popular now than it used to be. It's worth trying the architecture with and without batch normalization to see if it provides a clear benefit.
Yes, if your data are purely random then you will not detect a signal at all. This goes back to point number 2 (try a simpler model and see if you can detect a pattern). You can also try visualizing some of your samples to see if you can visually see a difference between the classes.

With regard to Xavier initialization, I don't think that's the likely cause of the effect you're seeing. The type of initialization can affect results, of course, but from what you're describing, I strongly suspect this is due to too little signal from the data or the unbalanced classes problem.

Hope that helps!


Looking for other opinions on approach to classification problem I'm looking to implement an "opt-out" filter for my company. The input is short, text-message style messages. A few examples of opt-out messages are:

"remove me from your list"
"remove from list"
"please unsubscribe from list"
etc.

All other messages are "good", and should not be removed.

My thoughts on approaches:

I was thinking of using a Bayesian classifier here, but not really knowing the solution space (or having much of a background in ML), want to be sure I'm not wasting time on a sub-optimal solution.

I'm fine with not having the most cutting-edge solution, but want to be sure I'm not missing an approach that might be equally as straightforward but more effective.

You should use text classification techniques. The most basic one is multinomial naive Bayes classifier with tf-idf features. for this method, take a look at this:
https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html

If you don’t get enough accuracy (or maybe precision, recall or f-score), you could test more complex techniques e.g. using deep LSTM networks with word embedding. For this method, take a look at this:
https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/


Visualize interval type data graphically and regression Suppose I have data like this:

x
y

5
10

10
12-30

15
45

20
50

25
55-60

I want to visualize it graphically. How can I do it? I googled how to describe intervals graphically, but didn't find anything. And I also want to do regression analysis on the data.

You can visualize intervals using histograms. For regression, try out OLS. Or you can reframe it as a classification task.


Cannot install package: dlx I'm new to anaconda.
How can I install the dlx package (https://pypi.org/project/dlx/) in anaconda?
I found it here: (https://anaconda.org/auto/dlx).
conda install auto::dlx
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:
  - auto::dlx
  - secretstorage
Use "conda info <package>" to see the dependencies for each package.

I get no result from conda info auto::dlx.
Can anyone help me?
SOLUTION:
Thanks to vivekwisdom for leading me on the right path.
PyCharm created a virtual environment. I had to activate it first (source activate <name_of_venv>) and then install dlx using pip. It didn't work via conda install auto::dlx because it was missing cython, which I also installed via pip.

As per the link you provided, conda install -c auto dlx is the correct way to install the package in Anaconda.


ML algorithm for Music Features I am a newbie in machine learning topic and I need to create model from music data.

It contains features of the songs but it is not labeled. How can I create a model from that ?

Do I need to use unsupervised learning algorithms ? Which one is better or is it better if I use deep learning methods.

Data is looking like this:

      danceability  loudness  valence  energy  instrumentalness  acousticness
136         0.795    -8.334    0.578   0.409          0.000000      0.684000
442         0.502    -4.556    0.720   0.912          0.000173      0.000025
92          0.713   -14.590    0.560   0.258          0.006060      0.877000
67          0.505   -14.951    0.723   0.782          0.930000      0.921000
127         0.470    -6.740    0.490   0.809          0.006710      0.272000

If the problem you are trying to solve is "I need to guess if the new songs are fit to user's taste with the song feature data I got."

AND if you have some indication what user has liked / listened-to in past; you can use "Recommendation Engine" approach.

Example : https://tampub.uta.fi/bitstream/handle/10024/101198/GRADU-1495623946.pdf?sequence=1

It works on the basis of "likeness"; E.g.:

Assumption : All Hip-Hop songs have some common pattern among the
features like energy , loudness etc.
If a user likes Hip-Hop (can be judged from user's history based on features of songs), then other Hip-Hop songs can be suggested


Residual plots: why do we want to know the error? The residual plot represents the error between the actual value.

Y-axis: residuals
X-axis: the predictor variable or fitted values.

Why do we want to know the error, what benefits us in getting the residuals?
I've uploaded an image from the video. I don't understand what is being sampled in the illustration.

Where is the independent and dependent variable(s)?
How do you identify the error?

They take one calculation and apply to the right-side of the graph and so on.

I am taking this class and don't know anything about what the instructor is talking about.

Can someone break this all the way down? (This is a beginner's course!)

Here is the transcript from the video:

  Examining the predicted value and actual value we see a difference. We
  obtain that value by subtracting the predicted value, and the actual
  target value. We then plot that value on the vertical axis with the
  dependent variable as the horizontal axis. Similarly, for the second
  sample, we repeat the process. Subtracting the target value from the
  predicted value. Then plotting the value accordingly. Looking at the
  plot gives us some insight into our data. We expect to see the results
  to have zero mean, distributed evenly around the x axis with similar
  variance. There is no curvature. This type of residual plot suggests a
  linear plot is appropriate

Let me try to give an example: Let's say we would like to predict a housing price by looking at the number of bathrooms. The x-axis represents your feature/independent variable (#bathrooms) and your y-axis represents the response/dependent variable. The points represent training samples you collected and know both the independent and dependent variables of.

Now in Linear Regression our goal is to predict the housing price based on a new example of #bathrooms we didn't see yet. For that we have to create a line through the data points that represents the training data the best. So, what qualifies a line as being a good or bad line? The answer is a cost/loss function. In linear regression we often use the squared mean of the error/residuals (MSE):

$$
\frac{1}{n} \sum (y_i - \hat{y_i})^2
$$

where $y_i$ is the true housing price and $\hat{y_i}$ is the predicted value.
Our goal is to minimize this loss.

Plotting the residuals can give us many insights: For example on which training samples we make the highest errors. Also we can see how the residuals are distributed. For more complicated applications we need to check this distribution for the so called Markov Theorem which would go too much in detail.


How to determine x and y in 2 dimensional K-means clustering? Recently, we were taught K-means clustering. I understood the basic idea of the algorithm and successfully implemented it for data with a single dimensional. Now we are told to implement it for two dimensional data. As far as I understood x and y can be two attributes of a dataset but our professor said otherwise. She said that the we have to determine x and y of an attribute in the data set to cluster the data. She used a simple 2D matrix as an example. This has got me confused. How can one determine x and y of an atttribute? Row number and column number seems silly to me to use it for distance calculation.

So, my question is how does one determine x and y for 2-D k means clustering?

As per this question the two attributes (weight and height) are used as x and y. Is this correct?

I see no problem with the example of clustering on two numerical attributes like height and weight like in the example.

The only thing I can think of is somewhere lost in translation, your professor was trying to explain the concept of reducing many dimensions (attributes) to two and then clustering on those derived dimensions. That is a common technique for trying to visually spot clusters in a complex data set.


What methods can be used to detect anomalies in temporal texual data? I've been looking for methods that can help figure out anomalies in textual data stored in databases. Major goal is to use a unsupervised learning method to detect the anomalies. Further how can I find a context in the data set and figure out contextual anomalies?

This assumes you’re happy with neural networks. If you’re not, this answer probably isn’t of much use to you.

Firstly, a little bit about anomaly detection via autoencoders. Apologies if you’re already familiar with this.

An autoencoder is a neural network which learns to reproduce its own input when compressed through a “bottleneck” layer. For example, you may want to find a lower-dimensional feature representation of a set of 100 x 100 images. Your neural network architecture has an input layer of 10,000 elements, an output layer of 10,000 elements, and one of its hidden layers will be relatively narrow compared to the input space: say 100 nodes.

The objective is to train the network to produce an output as close to the input as possible, while throwing away all but 100 nodes’ worth of activation. You are trying to produce as lossless a compression of the input as possible, so those 100 nodes should be a very information-rich representation of the kind of data you trained it on.

“How does this have any bearing on anomaly detection?” I hear you ask. Well, if you train your autoencoder on non-anomalous data, it will learn a non-anomalous lower-dimensional feature representation. This will mean the reconstruction error from pushing something through the autoencoder will be lower for data similar to what it was trained on than it will for other arbitrary data. If it receives input that is substantively different from what it was trained on, the reconstruction error will be higher. So, given a set of novel inputs, those inputs with the highest reconstruction error are the most anomalous, as they are poorly reconstructed from the non-anomalous feature representation.

If your data has a temporal structure, and you have plenty of training samples, you might want to consider constructing an autoencoding LSTM. An LSTM is a neural network architecture for encoding and decoding sequentially dependent data, and a full description of how this works is beyond both the scope of this post and my own abilities. There are many magnificent resources available online for getting to grips with this.

Here is a relevant paper on using LSTMs for anomaly detection in time series in general. It may be that LSTMs are unnecessary for your purposes if the data isn’t strongly sequentially dependent.


How can I add custom numerical features for training to BERT fine tuning? I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy.

My original dataset consists of a pair of sentences (like MRPC dataset). I want to increase the accuracy of classification by adding some numerical features (which I will separately calculate). I wanted to know if I could train bert using this after I have already fine tuned it ?

I have read about some solutions people have proposed in the past like :
'Extracting word embeddings (extract_features.py on Bert GITHUB) and combining that with my custom data to feed a single layer CNN network.' I dont want to lose out on the accuracy the BERT network is providing me by only extracting features from the pretrained model.

So is there any way I can create a kind of hybrid model which first fine tunes BERT and then I add my features and it feeds into another model for an improved classification ?

P.S

As I am new to tensorflow and Deep learning, please let me know if there is something fundamentally wrong in my understanding. Thank you for your help

With BERT I am assuming you are using finally the embeddings for your task.

Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model for the task.

Solution 2: Here you will play with the network.

Now here left one is the normal BERT, in the right we have another MLP network to deal with other numeric features, on top we are combining embeddings from both and then passing through one more MLP to give the final output.
Here the advantage is, while backpropagation, the BERT weight updates will have some contribution because of the loss due to numeric features.

As you mentioned you are new to TF and Bert, refer https://towardsdatascience.com/bert-in-keras-with-tensorflow-hub-76bcbc9417b to see how BERT can be implemented, and with this only doing the tweak to create suggested network will be easy.
Feel free to reach out to me in case of any further questions.


Does pruning a decision tree always make it more general? If I prune a decision tree, does that make the resulting decision tree always more general than the original decision tree?

Are there examples where this is not the case?

If you filter something out by choosing one branch over another branch in the tree, the observations you did not choose are forever lost.

But to directly answer your question - no, it does not always make it more general. If you construct a tree where all the decision are exactly the same, then pruning does not make it more general.


Machine Learning and Natural Language Processing : Project Initiation I am in the research phase of a long project and am willing to get some useful feedback from your side about the most appropriate project path to take.

Current situation:

A large team of so called production editors (PEs) is semi-automatically processing MS word files in order to do the copy-editing and layout.
This is currently semi-automatized with word macros where the PEs will apply different macros on different text paragraphs in order to set the appropriate style, position, font etc.
Still many actions are manual and depending on the human eye checks, resulting at the end many mistakes and an average quality

Plan

Start using machine learning, later maybe neuronal networks, statistics probability and artificial intelligence
The usage of machine learning (to start with) could be in the way that the system will learn the manual tasks done by the user (meaning the tool was not able to do those correctly in the automatized way) and then try to apply those in next similar situation
The term of similar situations is important here as this will imply natural language processing (NLP). The staff is dealing with word documents that could contain anything (within a similar layout)

The dilemma of the day is :) :

Where to start from in order to do some use case analyses? What I currently have:

More than 100k word files that are correctly done (except the user mistakes that may still lay around)
A list of rules that apply to the word documents describing the exact styles that needs to be used (non exhaustive)
Basic Machine Learning knowledge (starting with AI studying)

Starting with a small chunk of documents (~50) as a training set with a set of rules implements would be a good starting point. Train using a few algorithms and see if your training accuracy is coming out as acceptable or better than the human accuracy.

Then slowly increase your training batch size and see the impact on the results.

Once you believe that the training set is a partially good representation of the dataset, go ahead do a test run. Then keep growing from there.


Criteria for saving best model during training neural network? I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function.
General approach to save best model during training is to monitor validation loss at each epoch and save the model if val loss decreases than previous minimum.
But, I am interested in the "model which gives best the average dice score of 4 classes".
During training in my case, using validation loss as criteria doesn't lead to best avg dice score. So what should I consider the best model the one with less validation loss or highest validation dice score?
Below is my validation loss and avg dice score after each epoch. Out of this which epoch gives the best model?
epoch  1/10     validation loss: 0.95     avg dice score: 0.17
epoch  2/10     validation loss: 0.86     avg dice score: 0.23
epoch  3/10     validation loss: 0.77     avg dice score: 0.34
epoch  4/10     validation loss: 0.74     avg dice score: 0.40
epoch  5/10     validation loss: 0.71     avg dice score: 0.45
epoch  6/10     validation loss: 0.69     avg dice score: 0.34
epoch  7/10     validation loss: 0.79     avg dice score: 0.45
epoch  8/10     validation loss: 0.75     avg dice score: 0.51
epoch  9/10     validation loss: 0.76     avg dice score: 0.36
epoch 10/10     validation loss: 0.75     avg dice score: 0.38

If I go by val loss as criteria epoch-6 gives the best model, if i choose avg dice score as critieria epoch-8 gives the best model? how to choose?

The loss is mostly a measure that helps the model learn and is not looked at too much when deciding which model to select. A more business oriented measure is often used for this, e.g. accuracy. Since in this case you are mostly interested in the dice score it would make most sense to select the model from epoch 8.


Predicting crowd density using real time streaming data I would like to make predictions about how crowded a location (postal code mapped to lat/lng coords) would be. I would like to provide predictions to questions like: what’s the crowd density going to be like for the area surrounding postal code 12345 for 50metre next Monday between 6pm to 8pm?

For a start I’m working with historical transaction data (as a proxy for human activity) which contain time stamps and postal codes. I would also like to incorporate new transaction data into my model as they occur.

Does this make sense?

Filter the historical transactions based on requested time (between 6pm to 8pm).
Cluster the transactions based on the desired proximity to get a daily transaction totals for the area.
Use the daily totals as training data for my regression model.
Use the model to predict

Would I need to rerun the entire process to account for new data which arrives in real time (e.g every 5mins)? Are there any incremental ML models which could be suitable?

Are there any incremental ML models which could be suitable?

You have to be able to access the model when it's in production mode and actually be able to update it with a new version. all ML models should be adjusted when new data comes because of the so-called data shifting. you could periodically evaluate how much the model is performing.

I suggest modeling the problem as a sequence model, use multiple output at each day to predict densities for each postal code.


Is consecutive multiple transformation of Y variable a valid approach statistically? I am going to run a regression analysis. But my response variable is highly right-skewed

Firstly I used a log(x+1) transformation and it increased the situation dramatically. But there is still right-skewness in my transformed data.

My question is; can I apply another transformation to my already transformed variable?

For example I have applied a sqrt transformation to my log(x+1) transformed variable and it is now very close to normality.

Is it valid in terms of data science and statistics? Can I apply such multiple data transformations to my data?

Thanks

Combining analytical transformations (say $f$ and $g$), as you suggest, is nothing else than applying a single $u = f\circ g$ transformation, so it is as valid as applying any simple transformation.

You just need to make sure that the transformation works (in this case, $y>0$ in the whole dataset and in the predictions, so that $\sqrt{log(y+1)}$ is real), and of course that it remains invertible, i.e. the initial variable can be retrieved from the transformed value, without ambiguity (this requires the transformation to be injective).

The transformation you suggest seems valid. You could also try a second log-transformation instead of the square root.


Document clustering to merge common labels I am building a recommendation system and I have to clean up some of the labels that I have. For example of the data
df['resolution_modified'].value_counts()

Gives
105829
It is recommended to replace scanner                                                                                1732
It is recommended to reboot station                                                                                1483
It is recommended to replace printer                                                                                881
It is recommended to replace keyboard                                                                                700
                                                                                ...
It is recommended to update both computers in erc to ensure y be compliant with acme                                                    1
It is recommended to configure and i have verify alignement printer be work now corrado                                                 1
It is recommended to create rma for break devices please see tt for more information resolve this in favor of rma ticket create         1
It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update                        1
It is recommended to switch out dpi head from break printers                                                                            1

Notice that It is recommended to replace keyboard and It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update are very similar. Ideally, I would like to just converge to the string that occurs more frequently thus the second string should convert to the first.
I am thinking of using document clustering to handle this approach. I have tried using fuzzywuzzy but since I have a lot of strings the process below is too slow
from fuzzywuzzy import fuzz

def replace_similars(input_list):
    # Replaces %90 and more similar strings
    for i in range(len(input_list)):
        for j in range(len(input_list)):
            if i < j and fuzz.ratio(input_list[i], input_list[j]) >= 90:
                input_list[j] = input_list[i]

def generate_mapping(input_list):
    new_list = input_list[:]  # copy list
    replace_similars(new_list)

    mapping = {}
    for i in range(len(input_list)):
        mapping[input_list[i]] = new_list[i]

    return mapping
res = h['resolution_modified'].unique()
res.sort()
mapping = generate_mapping(res)
for k, v in mapping.items():
    if k != v:
        h.loc[h['resolution_modified'] == k, 'resolution_modified'] = v

I wanted to know if there is some document clustering I could apply that weighs in the strings that occur more than once and thus I would just take the common strings related to them that occur less and converge them to the more frequent occuring string. Does anyone have any recommendation on which method to use?
What I have Tried Thus Far:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
v = TfidfVectorizer()
x = v.fit_transform(df['resolution_modified'])
kmeans = KMeans(n_clusters=2).fit(x)
test_strings = ['It is recommended to replace keyboard', 'It is recommended to replace keyboard manually clear hd space add to stale profile manager instal windows update']
kmeans.predict(v.transform(test_strings))

Which gives
array([1, 0], dtype=int32)

Obviously not working so far, will try to increase the number of clusters.

I might misunderstand something but it looks to me like you're trying to find a complex method for a simple problem: if there are many strings which occur multiple times in the list, you should deduplicate the list before comparing all the pairs. You could use a set, but since you will need to count how frequent each string is you should probably directly create a map (dictionary) which stores the frequency for every string (just iterate over the list of strings, then increment the frequency of this string (key) in the map).
Depending how many distinct strings you have, this simple step might be enough to allow you to compare all the pairs of strings efficiently.
Then you could for example decide on a threshold for frequency, for instance keep only the strings which appear at least 10 times. For any string which appear less than 10 times, replace with the frequent string (more than 10 times) which has the highest similarity with it.


1x1 Convolution. How does the math work? So I stumbled upon Andrew Ng's course on $1x1$ convolutions.
There, he explains that you can use a $1x1x192$ convolution to shrink it.

But when I do:

input_ = torch.randn([28, 28, 192])
filter = torch.zeros([1, 1, 192])

out = torch.mul(input_,filter)

I obviously get $28x28x192$ matrix. So how can I shrink it?

Just add the result of every $1x1x192 * 1x1x192$ kernel result? So I'd get a $28x28x1$ matrix?

Let's go back at normal convolution: let's say you have a 28x28x3 image (3 = R,G,B).

I don't use torch, but keras, but the principle applies I think.

When you apply a 2D Convolution, passing the size of the filter, for example 3x3, the framework adapt your filter from 3x3 to 3x3x3! Where the last 3 it's due to the dept of the image.

The same happens when, after a first layer of convolution with 100 filters, you obtain an image of size 28x28x100, at the second convolution layer you decide only the first two dimension of the filter, let's say 4x4. The framework instead, applies a filter of dimension 4x4x100!

So, to reply at your question, if you apply 1x1 convolution to 28x28x100, passing number of filters of k. You obtain an activation map (result) of dimension 28x28xk.

And that's the shrink suggested by Ng.

Again to fully reply to your question, the math is simple, just apply the theory of the convolution using 3D filters. Sum of multiplication of overlapping elements between filter and image.

Edit: Simple example

I am going to show you a simple example of the operations that occur during the deep convolution between M, a tensor of dimension (z= 2, x=2, y=2), where you can see z as your k that you want to shrink to 1 and W, a filter of dimension (2, 1, 1). You will have to implement your own function with a loop to operate the stride of the filter.

import numpy as np

M = np.array([[[1, 2],[3, 4]], [[5, 6],[7, 8]]])
W = np.array([[[2.]],[[3.]]])

C = np.ones(shape=(2, 2))

c[0,0] = np.sum(M[:, 0, 0]*w.T)
c[0,1] = np.sum(M[:, 0, 1]*w.T)
c[1,0] = np.sum(M[:, 1, 0]*w.T)
c[1,1] = np.sum(M[:, 1, 1]*w.T)


predict_proba compared I want to predict the probability that an individual credit_balance is more than value N is true

I need to compare 3 classification methods: Logistic regression, Least discriminant and quadratic discriminant.

For 1 sample prediction I get predict_proba for each of them with different values, how can I decide which model is the best for my prediction?

linear regression predict_proba result is [[0.93227393 0.06772607]]
LDA predict_proba is [[0.94144572 0.05855428]]
QDA precit_proba is [[9.99999999e-01 1.24419207e-09]]

What parameters should we look at to decide which classification is the best to predict the model?

You need to take a step back in order to decide which model would suit best for your use case. Before doing that predict_proba is definitely important to calculate the posterior probability of the class labels, but it isn't great for comparing against other model outputs (especially to decide which models suits best for your prediction).

Logistic Regression, QDA and LDA all have different approaches. Logistic regression is based on maximum likelihood estimation while LDA and QDA are based on Bayes theorem. To understand which classifier best suits your model we need to go over the assumptions (assuming you know the mathematical expressions) and then you can judge which best apply to you.

1. Logistic Regression

In Logistic regression, it is possible to directly get the probability of an observation for a class (Y=k) for a particular observation (X=x). There is nothing to assume to run logistic regression for classification. Its generally a safe go to method which is not exigent and is robust.

2. LDA & QDA

LDA and QDA algorithm is based on Bayes theorem and classification of an observation is done in following two steps.

Identify the distribution for input X for each of the class (or groups ex Y=k1, k2, k3 etc )
Flip the distribution using Bayes theorem to calculate the probability Pr(Y=k|X=x)

Following are the assumption required for LDA and QDA:

LDA Assumption:

Common covariance across all response classes σ2 ( for ex σk1 = σk2 =
σk3 for k1, k2 , k3 response classes )
Distribution of observation in each of the response classes is normal
with a class-specific mean (µk) and common covariance σ.

QDA Assumption:

Different covariance for each of the response classes. For ex – σk1,
σk2, σk3 for response class k1, k2, k3 etc.
Distribution of observation in each of the response class is normal
with a class-specific mean (µk) and class-specific covariance (σk2).

Notes:

LDA (Linear Discriminant Analysis) is used when a linear boundary is required between classifiers.
QDA (Quadratic Discriminant Analysis) is used to find a non-linear boundary between classifiers.
LDA/QDA, when all its requirements met classifies better than logistic regression (more efficient).
Logistic regression is not sensitive to outliers while LDA/QDA is.

To conclude:

LDA and QDA work well when class separation and normality assumption holds.
Logistic regression has an edge on LDA/QDA for dataset that is not normal.


Geometric interpretation of MLP output I am really interested in the geometric interpretation of perceptron outputs, mainly as a way to better understand what the network is really doing, but I can't seem to find much information on this topic.

I know a perceptron with no hidden layers can be seen as defining a hyperplane in between two classes, which works fine if they are linearly separable.

However, I find more difficult to understand what exactly does a multi layer perceptron do, and how it can classify an input that is initially not linearly separable.

I found this page http://www.borgelt.net/doc/mlpd/mlpd.html that says the hidden layers can be interpreted as making a coordinate transformation on the input. I would like to read more about this, but I didn't find much online, so I would really appreciate if you could elaborate on this or point me to some paper or book that covers this.

I am also puzzled by the visualisation that appears here http://tiny.cc/2l7o7y , a 3 neuron hidden layer manages to separate the input, which makes perfect sense since separating that input in a 3 dimensional space is trivial.
However, adding another 2 neuron layer after that creates in them shapes that are obviously non-linear. My guess is that it represents the separation created in the original 2-dimensional space. Is that right? If not, what are we seeing here, exactly?

You might want to check out this article on Neural Networks, Manifolds, and Topology for a "geometric interpretation of MLP output".
Here is an excerpt that you might be most interested in:

There are a variety of different kinds of layers used in neural
networks. We will talk about tanh layers for a concrete example. A
tanh layer tanh(Wx+b)  tanh(Wx+b)  consists of:
1.A linear transformation by the “weight” matrix W
2.A translation by the vector b
3.Point-wise application of tanh.

and here is an animated gif of the geometric description he gives:

At the beginning of this animation, you will see the lines are straight, and undergo a linear transformation (weights are multiplied by inputs, Wx) where the lines move, but remain parallel, with the origin also remaining at (0,0); this is what happens in all linear transformations.
Then, you will see the transformation translated by the bias vector b.  That is, you will see all the lines move together in the direction of the bias vector b.  In this case, you will see the entire picture move down and to the right.
Finally, you will see the "kernel trick" which transforms the inputs to the next layer in a nonlinear way, causing the nice parallel lines to warp.  This non-linear warping allows the neural network to solve non-linear problems.  That is, if the space is distorted, and the points you are trying to classify move apart under this non-linear transformation, it may then be possible to create a separating hyperplane between points which should have different classifications, thus allowing the neural network to differentiate between your classes.
Hopefully, this will give you a geometric interpretation of what is happening at each layer of a neural network.  From there, you can extrapolate as you add more and more layers to your network.
In the example shown above, he is using tanh for the "kernel trick", but a number of other nonlinear functions could also be used (e.g. Sigmoid, ReLU, etc.)
If you find this animated gif hard to follow, I would strongly recommend you check out 3Blue1Brown's Essence of linear algebra series where he has a number of videos with intuitive explanations of Linear Algebra and great geometric animations.