how to visualize InceptionV3 hidden layers I am following google codelab:

Tensorflow for poet  to train my custom model. This google codelab use the Inception-V3 model for training.

The inception-V3 model have 48 layer. My question is that how can i visualize image features at the hidden layers?

Examples

Here are two adapted functions from the first of the links below that should get some weights out for a given input sample for you and plot the activations for some filters of that layer:

def getActivations(layer,sample):
    units = sess.run(layer,feed_dict={x:np.reshape(sample, [1,784], order='F'), keep_prob:1.0})
    plotNNFilter(units)

def plotNNFilter(units):
    filters = units.shape[3]
    plt.figure(1, figsize=(20,20))
    n_columns = 6
    n_rows = math.ceil(filters / n_columns) + 1
    for i in range(filters):
        plt.subplot(n_rows, n_columns, i+1)
        plt.title('Filter ' + str(i))
        plt.imshow(units[0,:,:,i], interpolation="nearest", cmap="gray")

This is the result, showing the second layer of activations from their model used for MNIST classification - the sample used was clearly a 7:

Rather than paste the entire code here, I suggest you read their instructions/explanations in context. The first link contains the function above.

Inspect activations at given layer using Tensorflow - (blog version)
Visualising layer activations using Keras
Using the tf.summary.image operator on a Variable, which will log the given Variable, allowing you to view it in Tensorboard. Examples here.

Some background

You can do this a couple of ways:

extract the activations for a given sample and plot them - you will get plots of varying sizes as you move through the network, correspoding to the dimensions of the weight matrix.
select your target layer, freeze all layers before that layer, then perform backbrop all the way to the beginning. This essentially extrapolates the weights back to the input, allowing you to display them over an input sample - giving insight as to what a particular layer is focusing on.

Here is a lecture presented by Matt Zeiler, who co-authored the original paper that showed the second method above. Their approach gives some results like this:

They use a "deconvolutional model" - which is also named the transpose convolution, which describes it more accurately. A deconvolution is actually a mathematical operator, which has a different meaning, so many people choose not to use the term.

Another approach to this, which goes a bit beyond what you asked, was proposed a few months later, by Simonyan et al., which looks at the saliency maps of weights/images - this is "the spatial support of a given class in a given image". IT can help localise objects - read the paper if that sounds good. In Section 4 of their paper, they claim their method to be a generalisation of the approach from Zeiler and Fergus.


What is difference between Fully Connected layer and Bilinear layer in CNN? What is the difference between Fully Connected layers and Bilinear layers in deep learning?

I quote the answers from What is a bilinear tensor layer (in contrast to a standard linear neural network layer) or how can I imagine it?.
A bilinear function is a function of two inputs $x$ and $y$ that is linear in each input separately. Simple bilinear functions on vectors are the dot product or the element-wise product.
Let $M$ be a matrix. The function $f(x,y)=x^TMy=\sum_iM_{ij}x_iy_j$ is bilinear in $x$ and $y$. In fact, any scalar bilinear function on two vectors takes this form. Note that a bilinear function is a linear combination of $x_iy_j$ whereas a linear function such as $g(x,y)=Ax+By$ can only have $x_i$ or $y_i$. For neural nets, that means a bilinear function allows for richer interactions between inputs.
Now what if you want a bilinear function that outputs a vector? Well, you simply define a matrix $M_k$ for each coordinate of the output and you end up with a stack of matrices. That stack of matrices is called a tensor (3-mode tensor to be exact). You can imagine the bilinear tensor product with two vectors as $x^⊤M_ky$ computed on each “slice” of the tensor.
Bilinear Models consists of two feature extractors whose outputs
are multiplied using an outer product at each location of the image
and pooled to obtain an image descriptor. 1

Its advantage is that it can model pairwise feature interactions in a translationally invariant manner, which is particularly useful for fine-grained categorization. It also allows end-to-end training using image labels only, and achieves state-of-the-art performance on fine-grained classification.


What kind of images/objects are the most easy for neural networks to detect? I need to design a marker image that should be detected by the neural network. I am aware that it is not a complex task just to detect an image and this can be done with OpenCV alone. However the detector but this should also work also in pure visibility conditions (low light, out of focus, worn out/partially damaged, makes tiny part of the image only, etc). The background around the marker image is just the real world, difficult to predict what it could be there.

What kind of images are the most easy for neural networks to work with? Most important

Should it have sharp contrast like letters or road signs or color gradients may work better?
Would repeating pattern (like chevron frame around) as part of the object help or better to have every piece of the tag unique?

I know that neural networks are mostly used to the "real world" images like humans and horses, but I am not sure if they are better objects or just it is more impressive to get them recognized.

From the other side, objects that humans make on purpose just to recognize them (like road signs) mostly use sharp contrast. I do not remember any road or other sign with any kind of color gradient but repeating pattern like chevron frame around is sometimes used. From the other side, a road sigh with color gradients is more complex to manufacture, and maybe the technology simply did not exist when they were first invented.

Is there any consensus which kind of objects are the most easy to recognize by neural networks under "noisy input" conditions?

You are describing the field of robust object detection.
It depends greatly on the specifics of the data and model. Similar to person looking for their keys at night only under a street light, most robust object detection research is related to autonomous driving.
The simplest answer might be something out of sample from the training data. If the marker object is 100% synthetic, a model would quickly learn it since there would be no confusion with other inputs. Again, the definition of 100% synthetic would depend on the training set.


Automated tuning of Hyperparameter Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, random search etc. that could allow faster discovering of best parameters)? I heard of hyperopt but seems there are some problems, and not sure it can train traditional machine learning algorithms?

There are a number of methods to automate the optimisation of your hyper-parameters, such as GridSearch and RandomSearch which the article you linked discusses briefly.

The main reason to choose one over the other is if you want the best possible parameters, and don't care how long it takes to get them: go for GridSearch. On the other hand, if you don't want the optimisation to take a long time, but still want some good parameters then go for RandomSearch.

These two implementations in ScikitLearn aren't exactly "advanced packages" but they'll get the job done for any model in Scikit (Random Forest, MLPClassifiers, etc). Emre's comment also has some pretty cool advanced packages which are Scikit compatible or Tensorflow compatible.


is it possible to do feature selection for unsupervised machine learning problems? I started looking for ways to do feature selection in machine learning.

By having a quick look at this  post , I made the assumption that feature selection is only manageable for supervised learning problems:

Still, I have to ask: are there methods to do feature selection without having a known variable that will be used for a classification/regression problem?

Take a look at these links-

https://stats.stackexchange.com/questions/108743/methods-in-r-or-python-to-perform-feature-selection-in-unsupervised-learning

http://www.jmlr.org/papers/volume5/dy04a/dy04a.pdf

https://www.researchgate.net/post/What_is_the_best_unsupervised_method_for_feature_subset_selection


How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier? Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. Assume, for some reason (probably some training problem or declaring classes), C is confused and doesn't perform well. The way we assign a class using C to each test data $I$ is: $class(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$, where $p_j(I)$ is the probability estimate of $I$ corresponding to the $j$-th class, given by C.

Now, on top of this previous classifier C, I'd like to use a Hidden Markov Model (HMM) to "correct" the mistakes made by the previous context-free classifier C, by taking into account the contextual/sequential information not used by C.

Hence let in my HMM, the hidden state $Z_i$ denote the true class of the $i$-th sample $I_i$, and $X_i$ be the predicted class by C. My question is: how could we use the probabilistic information $cl(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$ to train this HMM? I understand that the confusion matrix of C can be used to define the emission prob. of the HMM, but how do we define the transition and start/prior prob.? I'm tempted to define the start/prior prob. vector as $\pi:=(p_1(x_1), ..., p_K(x_1))$.  But I may be wrong. This is my main question.

A follow up question: One can define an HMM in the above way (using confusion matrix and the prob. information from C); call the resulting parameter set $\Theta_0$. But after doing so, is it advisable to estimate the parameters to best fit the data $I$ used for C, while initializing a parameter set with the values mentioned in the previous paragraph?

As far i know u can't saything about hidden class, hidden class value at time t is 'some intermediate values of weighted values of all hidden classes'.

see the point-  2) hidden state sequence in blog-
https://machinelearningstories.blogspot.com/2017/02/hidden-markov-model-session-1.html

so ur statement-

"Hence let in my HMM, the hidden state Zi denote the true class of the i-th sample Ii, and Xi be the predicted class by C- is incorrect.

How can u compare hidden class value with actual, u could have compared emission value and actual.

You can try ensembling of HMM and Cs. I wonder how exactly you trying to use parameters from classification problem to time series/sequence (HMM).


AUC with sklearn vary each time script is started I'm using the following code to perform a tree classification. I set up an int value for random_state in train_test_split function but each time I got different values for auc or accuracy_score.
I don't see what I am missing...

X_train, X_test, y_train, y_test = train_test_split(X, Y, random_state=1,stratify=Y, test_size=0.33)

clf = clf.fit(X_train, y_train)

predicted_probas = clf.predict_proba(X_test)
y_predict = clf.predict(X_test)

print(accuracy_score(y_test, y_predict))

print(classification_report(y_test, y_predict))

classes = np.unique(y_test)
probas = predicted_probas

fpr = {}
tpr = {}
roc_auc = {}

for i in range(len(classes)):
        fpr[i], tpr[i], _ = roc_curve(y_test, probas[:,i],pos_label=classes[i])
        roc_auc[i] = auc(fpr[i], tpr[i])
        print(classes[i])
        print(fpr[i], tpr[i])
        print("roc_auc")
        print(roc_auc[i])

Ok  I miss the fact that you can set also random_state here to remove this variability... but I don't see how it works here exactly so if someone want to explain...your are welcome.

   clf = tree.DecisionTreeClassifier(random_state=5)


Are Denoising Autoencoders for anomaly detection on structured data? Can denoising autoencoders be used for anomaly detection on structured data?

I know I can use denoising autoencoders for anomaly detection on images, but I don't know if they can do it for structured data as well? If not, why?

When you run denoise auto-encoders or any process on image data, what you did first was to convert run image data on a structured dataset with RGB arrays arranged in a column-like order.

Which means you turned your image data into structured data and then you applied the denoise-autoencoders, this means that you used the autoencoders on structured data.

Actually, almost no process is done in unstructured data. They run on the structured version of the structured data.

So the answer is yes, the denoise autoencoders does anomaly detection on structured data.


Scikit alternative for categorical data modeling? So, sklearn doesn't support categorical data in its models. Is there a known alternative for categorical data modeling (such as random forests, etc.) for Python?

There are definitely ways to process your data to make categorical data compatible with sklearn (e.g one-hot encoding). An alternative you can look into is h2o, which supports categorical features natively (although it doesn't offer the breadth of models of sklearn).


Is it possible to create a predictive model for a dataset that consists of only positive occurrences of the dependent variable? Lets say I want to predict earthquakes.
My dataset would only contain data about earthquake occurrences and no data about non-earthquake occurrences as that would basically be any other period of time which is not kept in the dataset.
In that case, I assume a decision tree or logistic regression would not work as we don't have a dichotomous dependent variable (as the occurrence only gets into the dataset if an earthquake occurred).
Are there any models which would suit this situation, or is a different approach needed?

My dataset would only contain data about earthquake occurrences and no
data about non-earthquake occurrences as that would basically be any
other period of time which is not kept in the dataset.

It would be nice if you could specify the features used for the prediction.

In that case, I assume a decision tree or logistic regression would
not work as we don't have a dichotomous dependent variable (as the
occurrence only gets into the dataset if an earthquake occurred).

Here you are right!. So it's better to resort to anomaly detection algorithms for the same. Here are some examples

Unsupervised anomaly detection
Outlier detection
Anomaly detection in python


A/B Testing Suitability for Hypothesis I'm obscuring the data I'm working on, but I think this should get the point across. I'm trying to design a test for the following:

A client sells various popsicles. They think that on days when it is very hot, one particular kind of popsicle sells more than on days when it's not hot. How can I prove if he's correct or not?

My thought is that I could do an A/B test, splitting up between not hot & hot days. For each of those groups, I could find the proportion of sales for that popsicle from all popsicles sold that day. If that proportion rises on hot days, then I can do a test for statistical significance and come to a conclusion.

But I think there have to be other tests/models I can apply based on sales counts alone right? What tests can I run that only involve things like the raw counts of sales on days when it was hot and not? Note that I am planning to use the temperature as a categorical variable but I can also make it continuous if that makes more sense.

I think you need to start by thinking very carefully about what exactly your client wants to know, because there are two different things you could be testing:

Is each individual customer is more likely to buy a popsicle of the designated type on a hot day than on a cool day?
Is the client likely to sell more popsicles of the designated type on hot days than on cool days?

For the first one, you would analyze proportions; for the second, you would analyze absolute sales counts for the designated type of popsicle, and in fact you could probably ignore sales of all other types of popsicles.

If that latter question is the one you want to analyze, I do think running a regression analysis on sales of the particular type of popsicle vs temperature (both as continuous variables) is a reasonable place to start. In fact you could start by just plotting them and looking at the graph, to see if any trend (linear, logarithmic, logistic, etc.) jumps out at you. I wouldn't go for A/B testing right off the bat because you're not dealing with a categorical variable, and there's not a single natural way to convert it.

Of course it could make sense to do that conversion, but in doing so, I'd suggest looking at overall sales figures and other indicators of whether people perceive a day to be hot or cool to try to pin down the boundaries. Something like logistic regression (possibly multinomial) might come in handy, though I could easily see it giving worthless results if the data aren't clean enough. Perhaps you could exploit a clustering algorithm to identify groups of temperatures that have similar sales figures and use that to determine your categories. You could also try to correlate external data sets like community pool attendance, ice cream sales, electricity usage (for air conditioners), etc., if those are available, to further inform your analysis of what temperature class each day would fall in. (Maybe that's overkill, but it's an idea at least.)

Incidentally, A/B testing typically designates a process where you control the assignment of the classes A and B. Like in web design, you choose (or program a computer to choose, perhaps at random) whether each visitor gets version A or version B. You don't get to choose whether each day is hot or not, so it's not quite the same thing. That being said, you can probably use most of the same methods of analysis you would on a proper A/B test.


Stanford NER - Increase probability for a certain class I'm new to machine learning so I apologize if this question is silly. I'm using Stanford NER's english 4class classifier with good results. However, since my dataset is mostly focused on organizations, I think the results could be improved if I could boost the probability for an entity to be an organization to the detriment of other classes.(Ex: I would prefer "Carl Zeis" to be identified as an organization rather than a person). Is my supposition correct? If so, can it be achieved in an easier way than retraining the model?

Your data set will influence the labeling results. If it is focused on organizations, the NER should favor them simply by virtue of the data it's fed. So you might need to do anything. But if you do observe undesirable behavior in the resulting NER, you can adjust the weights.

It's the same with any machine learning algorithm. A humorous demonstration was provided by Google's Deep Dream: it kept seeing dogs every where. Why? Because the data set they used for training had an abundance of dogs.

(And Carl Zeis should be labeled as a person. The company is Carl Zeiss.)


Assigning a new document to a cluster based on keywords extracted and tf-idf I have about 40 clusters of documents defined by a combination of k-means clustering algorithm and hand curation. For example, some of the clusters given by k-means are too noisy so they have been further subdivided.

Now I want to assign new documents to these clusters.

I found that it is possible to extract keywords using tf-idf based methods as mentioned here.

My approach is to extract key terms from each of these clusters using tf-idf based method and I can extract the keywords from the new document using the same method.

My question is, how do I assign the new document to the cluster that has the most similarity?

Edit: I do not have enough reputation to comment on Marks answer: the input to kmeans are document vectors (from doc2vec) of all documents -- and I get the centroids of the initial clusters i.e. centroids = kmeans_model.cluster_centers_. But I have split many of these clusters manually into sub clusters. For example, original cluster 3 is now two clusters -- 3_1 and 3_2. How do I generate a representative vectors (like a centroid) for the documents in these sub clusters?

It seems that you are already half-way there.
If you have divided the documents to clusters, it implies that you already have some feature extraction method that was used to quantize each document to some feature vector.

Now, when you get a new document, you need to extract its features the same way and then use its feature vector to find the best fitting cluster for it.

The best fitting cluster can be determined by using 1NN (Nearest Neighbor) algorithm on either the cluster centers or one of their affiliated document vectors. If you use the vectors and not the cluster centers, you can also you try KNN algorithm (K-Nearest Neighbors).


When to choose linear regression or Decision Tree or Random Forest regression? I am working on a project and I am having difficulty in deciding which algorithm to choose for regression. I want to know under what conditions should one choose a linear regression or Decision Tree regression or Random Forest regression? Are there any specific characteristics of the data that would make the decision to go towards a specific algorithm amongst the tree mentioned above? What are those characteristics that I should look in my dataset to make the decision? And are there some reasons that would make one choose a decision tree or random forest algorithm even if the same correctness can be achieved by linear regression?

Let me explain it using some examples for clear intuition:

  When do you use linear regression vs Decision Trees?

Linear regression is a linear model, which means it works really nicely when the data has a linear shape. But, when the data has a non-linear shape, then a linear model cannot capture the non-linear features.

So in this case, you can use the decision trees, which do a better job at capturing the non-linearity in the data by dividing the space into smaller sub-spaces depending on the questions asked.

  When do you use Random Forest vs Decision Trees?

I guess the Quora answer here would do a better job than me, at explaining the difference between them and their applications. Let me quote that for you:

  Suppose you're very indecisive, so whenever you want to watch a movie,
  you ask your friend Willow if she thinks you'll like it. In order to
  answer, Willow first needs to figure out what movies you like, so you
  give her a bunch of movies and tell her whether you liked each one or
  not (i.e., you give her a labeled training set). Then, when you ask
  her if she thinks you'll like movie X or not, she plays a 20
  questions-like game with IMDB, asking questions like "Is X a romantic
  movie?", "Does Johnny Depp star in X?", and so on. She asks more
  informative questions first (i.e., she maximizes the information gain
  of each question), and gives you a yes/no answer at the end.

  Thus, Willow is a decision tree for your movie preferences.

  But Willow is only human, so she doesn't always generalize your
  preferences very well (i.e., she overfits). In order to get more
  accurate recommendations, you'd like to ask a bunch of your friends,
  and watch movie X if most of them say they think you'll like it. That
  is, instead of asking only Willow, you want to ask Woody, Apple, and
  Cartman as well, and they vote on whether you'll like a movie (i.e.,
  you build an ensemble classifier, aka a forest in this case).

  Now you don't want each of your friends to do the same thing and give
  you the same answer, so you first give each of them slightly different
  data. After all, you're not absolutely sure of your preferences
  yourself -- you told Willow you loved Titanic, but maybe you were just
  happy that day because it was your birthday, so maybe some of your
  friends shouldn't use the fact that you liked Titanic in making their
  recommendations. Or maybe you told her you loved Cinderella, but
  actually you really really loved it, so some of your friends should
  give Cinderella more weight. So instead of giving your friends the
  same data you gave Willow, you give them slightly perturbed versions.
  You don't change your love/hate decisions, you just say you love/hate
  some movies a little more or less (you give each of your friends a
  bootstrapped version of your original training data). For example,
  whereas you told Willow that you liked Black Swan and Harry Potter and
  disliked Avatar, you tell Woody that you liked Black Swan so much you
  watched it twice, you disliked Avatar, and don't mention Harry Potter
  at all.

  By using this ensemble, you hope that while each of your friends gives
  somewhat idiosyncratic recommendations (Willow thinks you like vampire
  movies more than you do, Woody thinks you like Pixar movies, and
  Cartman thinks you just hate everything), the errors get canceled out
  in the majority. Thus, your friends now form a bagged (bootstrap
  aggregated) forest of your movie preferences.

  There's still one problem with your data, however. While you loved
  both Titanic and Inception, it wasn't because you like movies that
  star Leonardio DiCaprio. Maybe you liked both movies for other
  reasons. Thus, you don't want your friends to all base their
  recommendations on whether Leo is in a movie or not. So when each
  friend asks IMDB a question, only a random subset of the possible
  questions is allowed (i.e., when you're building a decision tree, at
  each node you use some randomness in selecting the attribute to split
  on, say by randomly selecting an attribute or by selecting an
  attribute from a random subset). This means your friends aren't
  allowed to ask whether Leonardo DiCaprio is in the movie whenever they
  want. So whereas previously you injected randomness at the data level,
  by perturbing your movie preferences slightly, now you're injecting
  randomness at the model level, by making your friends ask different
  questions at different times.

  And so your friends now form a random forest.


How the original data can be written in the space defined by these M principal components? Suppose you apply PCA on the data $x_1,...,x_6$ and find that data can be fully described using M principal components $u_1,...,u_M$. How the original data can be written in the space defined by these M principal components?

Assuming your basis vectors are sorted by decreasing eigenvalues, a sample $x_i$ can be transformed into the new space using dot products (projections), i.e. $[u_1^Tx_i,...,u_M^Tx_i]$. Note that, you first de-mean your data.


Different performance for splitting into test/train data vs. using cross-validation I am training a linear model using the following scikit-learn setup:
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.metrics import f1_score
from sklearn.model_selection import cross_val_score

[...]

random_state=786543

max_iter=5, tol=None)
clf = LinearSVC(random_state=random_state, dual=True, C=1.5)

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)
clf.fit(X_train, y_train.values)
predicted_train = clf.predict(X_train)
predicted_test = clf.predict(X_test)
print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))

Train Accuracy: 0.985129495926343
Test Accuracy: 0.9601936525013448
Test F1 micro: 0.9601936525013448
Test F1 macro: 0.9000889214688401
Test F1 weighted: 0.9590331562500389

But now I run
scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')
print(scores)

array([0.65860981, 0.84306338, 0.82113645, 0.83414211, 0.64665942])

How can this discrepancy be explained? I tested this using different random states.
A couple of points to consider:

I am having multible classes (but only one label per sample)
The dataset is skewed (so there are classes with many samples and some with very few classes)
I have 45066 samples, 5222 features, 259 classes

The number of samples per class is:
sorted(list(np.unique(y, return_counts=True)[1]))

[1, 1, 1, 1, 1, 1, 1, 2, 3, 4, 4, 4, 4, 4, 7, 7, 8, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 18, 18, 18, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 21, 21, 21, 22, 22, 22, 22, 23, 23, 24, 24, 24, 25, 25, 25, 26, 26, 27, 27, 27, 27, 27, 29, 29, 29, 29, 30, 30, 30, 30, 32, 32, 32, 34, 34, 35, 35, 35, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 40, 40, 40, 41, 41, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 48, 49, 50, 50, 52, 55, 56, 59, 59, 60, 60, 61, 61, 61, 65, 65, 67, 67, 69, 72, 73, 74, 75, 77, 77, 79, 80, 84, 85, 87, 93, 96, 97, 97, 103, 110, 112, 117, 123, 130, 139, 139, 141, 143, 146, 146, 147, 147, 150, 159, 161, 169, 170, 177, 180, 180, 189, 191, 196, 198, 199, 201, 202, 203, 203, 208, 211, 230, 236, 249, 255, 264, 268, 269, 300, 332, 347, 356, 358, 364, 388, 433, 469, 476, 484, 548, 652, 698, 723, 748, 753, 807, 815, 1013, 1200, 1222, 1243, 1274, 1447, 1643, 1741, 2900, 3909, 4627]

Reason for the discrepancy

Two aspects have to be considered regarding the split:

Is the split done in a stratified manner? (it should)
Is the data shuffled? (it should)

The line

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)

splits the data in a stratified manner by default (see parameter stratify) and it does shuffle by default (see parameter shuffle):

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

The line

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')

also splits the data in a stratified manner (see parameter cv:

  For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

), but it does not shuffle. This causes the bad results for this line.

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

Solution

Option 1: Shuffle the data beforehand:

import sklearn
scores = cross_val_score(clf, *sklearn.utils.shuffle(feature_matrix, df.eClass, random_state=42), cv=5, scoring='f1_macro')

Option 2: use appropriate cross-validaton object

I also looked into using an appropriate cross-validation sheme by using a different object:

import sklearn
skf = sklearn.model_selection.StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in skf.split(X, y):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    clf.fit(X_train, y_train.values)
    print('--------------------------------------')
    predicted_train = clf.predict(X_train)
    predicted_test = clf.predict(X_test)
    print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
    print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
    print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
    print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
    print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))
    print('--------------------------------------')

see:

https://scikit-learn.org/stable/modules/cross_validation.html
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html


Choice of method for classification and/or duration prediction Let's consider an e-commerce problem.

I have data about users that almost place an order online : they give some information about themselves (name, age, etc.), but won't immediately validate the order.

Sometimes they'll come back to finalize it, but other times, they'll never be back (better offer/product at another place). I do have the data to know if a user came back or not, and if he came back, how long it took him to come back.

Thus, I am interested by 2 points :

Will the user come back ?
If yes, how long did it take him to do so ?

For the first problem, it is obviously a classification problem. And  I am considering a regression problem to answer the second question as it is a duration problem.

However, I was wondering if I should train these models separately, or if a specific method to take into account the link between these two was more appropriate ?

If you are interested in predictions, I would do two different models to get the most out of each single step.

If you are interested in causal effects, you may consider so called „hurdle models“.
https://www.stata.com/stata14/hurdle-models/


Polynomial regression vs. multilayer perceptron Polynomial regression and multilayer perceptrons have different structures and different learning procedures. What are these two algorithms pros and cons? Are there some situations where one should perform better than the other?

Polynomial regression can have multiple entries in the normal equation and it is not easy to say which polynomials you have to use in advance. Moreover, if you have lots of features you cannot handle memory errors most of the time. Nowadays people use MLPs and use batch normalization among layers for learning better. Those that you are referring to are a bit old algorithms but the former one is the logical mathematical solution for learning problems and the latter one is a beginning point for deep neural networks. I recommend taking a look at here and here.


Why are precision and recall used in the F1 score, rather than precision and NPV? In binary classification problems it seems the F1 score is often used as a performance measure. As far as I've understood the idea is to find the best tradeoff between precision and recall. The formula for the F1 score is symmetric in  precision and recall. However, (and that's what bothers me) there is an asymmetry between precision and recall. While recall is a property of the classifier that is independent of prior probabilities, precision is a quantity that does depend on the prior probabilities.

Can anyone tell me what's so special about the combination of precision and recall? Why don't we use precision (which is the positive predictive value) and negative predictive value?

F1 score weights precision and recall equally but there are easy generalizations to any case where you consider recall $\beta$ times more important than precision. See https://en.wikipedia.org/wiki/F1_score:

$F_\beta = (1 + \beta^2) \frac{precision \cdot recall}{\beta^2 precision + recall}$

F1 is just a harmonic mean. The simple mean doesn't quite make sense because precision and recall have the same numerators (true positive) but different denominators (test positive, condition positive). So only a harmonic mean makes sense. I don't know if there's more theory to it than that -- the simplest weighted mean that make sense.

I think I get the gist of your point, which I paraphrase to refer to the fact that precision has "test positive" in the denominator so is quite sensitive to how much the classifier marks positive. For this reason you don't so often see, for example, precision-recall curves. You see ROC curves which are recall-specificity curves (true positive rate vs false positive rate).

That's closer to what you suggest but you're suggesting PPV vs NPV. Sure that could be valid depending on your use case, but I think the argument tends to cut the other way, to recall-specificity instead, not precision-NPV.


Python Seaborn: how are error bars computed in barplots? I'm using seaborn library to generate bar plots in python. I'm wondering what statistics are used to compute the error bars, but can't find any reference to this in the seaborn's barplot documentation.

I know the bar values are computed based on mean in my case (the default option), and I assume the error bars are computed based on a Normal distribution 95% confidence interval, but I'd like to be sure.

Looking at the source (seaborn/seaborn/categorical.py, line 2166), we find

def barplot(x=None, y=None, hue=None, data=None, order=None, hue_order=None,
        estimator=np.mean, ci=95, n_boot=1000, units=None,
        orient=None, color=None, palette=None, saturation=.75,
        errcolor=".26", ax=None, **kwargs):

so the default value is, indeed, .95, as you guessed.

EDIT: How CI is calculated: barplot calls utils.ci() which has

seaborn/seaborn/utils.py

def ci(a, which=95, axis=None):
    """Return a percentile range from an array of values."""
    p = 50 - which / 2, 50 + which / 2
    return percentiles(a, p, axis)

and this call to percentiles() is calling:

def percentiles(a, pcts, axis=None):
    """Like scoreatpercentile but can take and return array of percentiles.
    Parameters
    ----------
    a : array
        data
    pcts : sequence of percentile values
        percentile or percentiles to find score at
    axis : int or None
        if not None, computes scores over this axis
    Returns
    -------
    scores: array
        array of scores at requested percentiles
        first dimension is length of object passed to ``pcts``
    """
    scores = []
    try:
        n = len(pcts)
    except TypeError:
        pcts = [pcts]
        n = 0
    for i, p in enumerate(pcts):
        if axis is None:
            score = stats.scoreatpercentile(a.ravel(), p)
        else:
            score = np.apply_along_axis(stats.scoreatpercentile, axis, a, p)
        scores.append(score)
    scores = np.asarray(scores)
    if not n:
        scores = scores.squeeze()
    return scores

axis=None so score = stats.scoreatpercentile(a.ravel(), p) which is

scipy.stats.scoreatpercentile(a, per, limit=(), interpolation_method='fraction', axis=None)[source]
Calculate the score at a given percentile of the input sequence.

  For example, the score at per=50 is the median. If the desired quantile lies between two data points, we interpolate between them, according to the value of interpolation. If the parameter limit is provided, it should be a tuple (lower, upper) of two values.

Parameters:
a : array_like
A 1-D array of values from which to extract score.
per : array_like
Percentile(s) at which to extract score. Values should be in range [0,100].
limit : tuple, optional
Tuple of two scalars, the lower and upper limits within which to compute the percentile. Values of a outside this (closed) interval will be ignored.
interpolation_method : {‘fraction’, ‘lower’, ‘higher’}, optional
This optional parameter specifies the interpolation method to use, when the desired quantile lies between two data points i and j
fraction: i + (j - i) * fraction where fraction is the fractional part of the index surrounded by i and j.
lower: i.
higher: j.
axis : int, optional
Axis along which the percentiles are computed. Default is None. If None, compute over the whole array a.
Returns:
score : float or ndarray
Score at percentile(s).

and looking in the source for scipy.stats.stats.py we see the signature

def scoreatpercentile(a, per, limit=(), interpolation_method='fraction',
                      axis=None):

so since seaboard calls it with no param for interpolation it is using fraction.

On a side note, there is a warning of future obsolescence in stats.scoreatpercentile(), namely

  This function will become obsolete in the future. For Numpy 1.9 and higher, numpy.percentile provides all the functionality that scoreatpercentile provides. And it’s significantly faster. Therefore it’s recommended to use numpy.percentile for users that have numpy >= 1.9.


What's the difference between feature importance from Random Forest and Pearson correlation coefficient I have following business domain. I have a product with three outputs/labels. The outputs are impacted by 1000 procedures, each procedure is digitized and measured. The customer wants to know what is the most influential procedures on the outputs.

1.
From Pearson correlation coefficient we could learn how two variables' relationship, say 1 is proportional, -1 is negative proportional and 0 is no relation. So I could find the biggest value of Pearson correlation coefficient to find more influential procedures.

2.
From Random Forest algorithm, I could know the top feature importance. So I could identify also the most influential procedures.

Which one is better?

Pearson correlations capture linear relationships between the input and target variables. Therefore this only makes sense for continuous inputs and a continuous target variable, and not continuous inputs with a binary/categorical output. Correlations essentially measure the positive/negative 'change' in one feature as you increase/decrease the other.

So it doesn't make much sense to compare the relationship between your input features and the categorical outputs this way. You may as well calculate the mean input for each feature and each label, and calculate the differences between those. I found this answer on Cross-Validated which explains this much better than I can.

Feature importance in tree based models is more likely to actually identify which features are most influential when differentiating your classes, provided that the model performs well. How this feature importance is calculated depends on the implementation, this article gives a good overview of how different tree based models calculate importance for features.


How to handle large number of categories in a dataset? I have one dataset of "Books" which contains 8 columns initially and out of which 3 of them contains text values which can be categorized. The 3 columns contains "Language-code", "Author Name" and "title" of the book. As sklearn LinearRegression don't take text as input so i decided to categorize these 3 columns by using "pandas_getdummies(...)" but after categorizing it the columns number exceeded to 20072 from 8 which is way too high.

The dataset url is: https://www.kaggle.com/jealousleopard/goodreadsbooks/downloads/goodreadsbooks.zip/6

So my queries are:

What to do with the title name? Categorizing it doesn't seems right.
What to with the rest 2 columns? If i leave the title name then the number of columns exceeds to 7646. Is there any other algorithm where i can directly feed the dataset without categorization?
How to handle these large number of features after categorizing?

Algorithm like Decision Tree, can also work well on ordinal values, i.e. without OneHotEncoding. You can try this one.

Also Im not sure the importance of Title Name, so you have to take decision based upon the requirements. But I have avoided in my use case.

  Also, generally when you have large number of categories, you can give
  a try by clubbing uncommon categories into one.


Classifying videos with varying length using ConvLSTM2D in tensorflow I have a collection of videos, where I would like to extract a frame for every second, and then feed them through a ConvLSTM2D for binary classification.

I was under the impression that a LSTM could take varying input sizes, but after many hours of googling it seems like I either need to:

Use padding and masking
Use ragged tensors
Actually use varying input length, but use batch size of 1

I'm not sure how to proceed from here, since I cant find any resources for padding and masking a sequence of images. Ragged tensors are confusing, and I cant find any examples for a sequence of images. When trying to use a batch size of 1, tensorflow still complains that the inputs are not the same size when using model.fit.
The length of the video is actually important, so thats the reason I'm using a variable amount of images, but I could possibly extract a fixed amount of frames.
Any code examples or suggestions appreciated

I found a solution that works for me.
Since I wanted to avoid using padding and masking, and didn't fully understand ragged tensors, I decided to continue with using varying input lengths.
My training data consists of a list of image stacks between 6-82 frames. When trying to use this directly with model.fit(x=x, y=y, batch_size=1) where x is a list of tensors, tensorflow will complain that the input dimensions have varying size. I thought this didn't matter since the batch size was 1, but apparently tensorflow tries to change the list of image stacks to a tensor.
A way around this, is to pass a training sample for every step using a generator:
class ArtificialDataset(tf.data.Dataset):
  def _generator(samples):
    for i in samples:
      yield (image_stack, output)

And then making sure that the steps per epoch is the size of the data set
training_data = ArtificialDataset(training_samples)
model.fit(training_data, epochs=epochs, steps_per_epoch=len(training_samples))

This way tensorflow never tries to create a tensor with varying inner dimensions. A drawback of this method is that no batching occurs, so training takes a while. In my case this doesn't matter much since the network size and input data already requires me to load very few samples at a time.
An optimization would be to batch same-length videos (or videos with same number of extracted frames). I might do this at a later time, since the implementation of varying batch sizes is too much work right now.


Question on ANOVA and Correlation/Association I've been working on examining statistical relationships between variable:

Pearsons, Spearman's for continuous variables
Kendall's Tau, Cramer's V for ordinal/nominal variables.

I know there's many more ways. Recently I read about ANOVA and hypothesis testing. It seems similar to measuring correlation and association. In fact, I can't tell if it is just another way of doing the same thing, or if it is something entirely different. Most explanations of ANOVA seem a bit more complicated than most explanations of correlation or association.

For example, I know that Pearson's R is a measure of covariance scaled by standard deviation. And ANOVA stands for Analysis Of Variance. So it appears to me that it's the same sort of thing. But I can't tell 100% for sure.

Will someone please shed some light on this technique, what it is used for, and how it contrasts with measuring correlation?

About what ANOVA is used for: it can answer whether the difference between the mean values for the data samples I have is due to randomness or is it statistcally significant. Then it is a significance-test that gives you an idea about whether your mean values are (statistically significantly) the same or not. A drawback is that it does not tell you which data sample/s differ from the rest or by how much (useful source).
You can think of the process as follows (as described in Practical Statistics for data scientists):

Combine all the data together in a single box
Shuffle and draw out n resamples of m values each (where n is the number of data samples and m the number of data points in each sample)
Record the mean of each of the n groups
Record the variance among the n group means
Repeat steps 2–4 many times (say 1,000)
What proportion of the time did the resampled variance exceed the observed variance? This is the pvalue.

On the other hand, the direct measure of correlation gives you a number, it tells you by how much two data samples vary linearly along with each other


One way to improve the efficiency of the prediction is to convert the dataframe type to float32. I got better result while converting the data type to float32.
df = pd.DataFrame(np.array([[45,1411.27],[47754,1596200.68],[105894,3750304.55],[372953,14368324.86],[389915,14899302.85]]),columns=['ROW_CNT','TOT_SALE'],dtype=np.float32)
def outlier_func(df):
    model = IsolationForest(behaviour='new',n_estimators=1000,  max_samples='auto',
    contamination='auto', max_features=1.0)
    model.fit(df[['ROW_CNT', 'TOT_SALE']])
    df['scores'] = model.decision_function(df[['ROW_CNT', 'TOT_SALE']])
    df['anomaly'] = model.predict(df[['ROW_CNT', 'TOT_SALE']])
    anomaly = df.loc[df['anomaly'] == -1]
    anomaly_index = list(anomaly.index)
    return anomaly

outlier_func(df)


Norm type in cost function of ANN I'm reading a tutorial about ANN. They use the following cost function:

As you can see this equation includes a norm. I'm new to the concept of norm.

My question is what kind of norm they use here (There are Absolute-value norm, Euclidean norm, Euclidean norm of a complex number, Taxicab norm, Manhattan norm, p-norm, Maximum norm, infinity norm, uniform norm, supremum norm, Zero norm, etc.)

Reading the article they state that

  The two vertical lines represent the $L^2$ norm of the error, or what is known as the sum-of-squares error (SSE)

Otherwise known as the Euclidean norm, Euclidean length, $L^2$ distance, $ℓ^2$ distance, $L^2$ norm, $ℓ^2$ norm, $2$-norm, or square norm.

Also speaking from my experience, most of the time when a norm is specified without a subscript then you can generally assume that it is the Euclidean norm.


How to convert unstructured texts to structured data? I would like to write an algorithm to convert unstructured texts (with contest descriptions) to structured data with the following fields:

contest start date (optional)
contest end date
main prize
additional prizes (optional)

I have hundreds of text examples, that could be used for model learning.

How to approach this task? Just in case this is important - the preferable language is Python. But I never worked on such tasks before.
What is the best practice to do that?

If I get you right, you would like to extract the mentioned entities from the unstructured text. This sounds as Named Entity Recognition (NER) problem for me. In your case I would define the following entities:

START_DATE
END_DATE
MAIN_PRIZE
EXTRA_PRIZE

Your text now can be tagged or annotated with those entities. You would like to build a machine which will automatically tag the encountered entities in the unstructured text (free text).

I would suggest to take a look at Conditional Random Fields (CRF), a classic approach to NER. There are also Recurrent Neural Networks (RNN) which are state of the art for sequence modeling but I am not an expert with those methods.

Let's begin the process of model building. You need to decide what will be a single training example. This can be a sentence and this can be also a paragraph.

Since you did not provide any example, let me invent two:

The contest started on <START_DATE>july 1st</START_DATE> and ended on <END_DATE>july 25</END_DATE>.

The winner got away with <MAIN_PRIZE>1 million dollars</MAIN_PRIZE> and <EXTRA_PRIZE>bunch of flowers</EXTRA_PRIZE>.

Notice the way how the text is structured. Every word now is either label (surrounded by tags) or not labeled at all (outside of any tag). This is one way how to structure the text. This would be also the desired output on some unstructured text.

The machine does not understand the text and annotation. You need to come up with features that you can extract from each word. It can be the word itself, the previous word, the next word, left bigram, right bigram, suffix of length 1,2,3, prefixes of length 1,2,3 and so on. I suggest you to look at Stanford NER features.

Once you got the features, you need to create a training set. The training set is your annotated text (see my example) converted to label + feature information. It may look like this

O   feature1 feature2 ...
O   feature1 feature2 ...
B-START_DATE   feature1 feature2 ...
I-START_DATE   feature1 feature3 ...
...

This basically says that the first two words are not labeled, but the last two are part of an entity START_DATE.

Each line is essentially a word followed by features for it. The label "O" is the way to say that the word is not labeled, the labels B- and I- for DATE mean the first and second words of a named entity (e.g. "july 1st").

The sentences can be separated by empty lines.

Once you've got the training set file ready, you can do the model training. With a model in hand, you can apply it on any unstructured data (extracted features) to get the desired entities extracted.

As a tool I suggest you CRFSuite (also available with Python bindings). I can detail more each step if you would like.


Approaches for implementing Domain specific Question answering System Given several wikipedia articles on different movies.
What are the different approaches to implement a QA system to answer different quires related to movies.

Dataset : Wikipedia articles

Input   : natural language query, eg : who directed terminator ?

Output  : The Terminator is a 1984 American science fiction action film directed by James Cameron

Different approaches available are :

1 IR Based approaches

In IR based approaches the similarities between query and passaeges are considered and best suitable passages are selected as answer (as Google does)

2 Ontology based

In Ontology based prior domain knowledge is required about the domain. And documents need to be mapped accordingly.

3 Machine learning based

In ML based approaches large training set of documents is needed for training.

What are the other cons and pros of each approaches?

Which of these approaches suits the best for this use case ? Also is there any other methods ?

While the actual "best system" depends heavily on a number of factors including your goals and your resources, it is possible to discuss some general pros and cons to each system.

1. IR Based approaches: Information retrieval approaches allow the algorithm to make effective use of what we might call "explicit knowledge" or hardcoded facts. They can work very well when the answer to the query exists in some form already and where the algorithm simply needs to find this answer amongst a set of all the other possible answers. Such systems typically rely on either an explicit dataset of possible answers or facts, or by treating the web as such a dataset and querying it in some way. IR systems are often the only reasonable choice when the answer cannot be generated from abstract principles. For example, if we built a system to answer trivia questions about movie stars, we would need to have explicit knowledge of which features are associated with which movie stars (i.e. Brad Pitt has brown hair and starred in Fight Club.) No machine learning algorithm can be expected to generate this without the knowledge already present (although machine learning systems with access to such knowledge may serve as effective information retrieval systems.) Due to the external nature of the data with respect to the algorithm, IR systems tend to be very effective for large and rapidly changing data sets, which would otherwise require retraining of the machine learning model or manual reconfiguration of the ontology to handle. It is also worth noting that more complex information retrieval systems may include other systems such as NLP engines to translate the data (ex. natural language text) into a form usable by the IR system.

2. Ontology based approaches: Ontology methods take the most time to implement and are usually the most finicky. They require the iterative hypothesizing of a structured representation of the domain, testing of the representation, and modification of the representation based on discovered flaws. For complex real-world domains with many possible contingencies, this process can take a very long time and be quite frustrating. Because everything is hard-coded, even the best ontology-based systems display a relative degree of inflexibility compared to their machine learning and IR counterparts. Ontologies can be effective, however, when you do not have access to a data set for information retrieval or machine learning training, as the domain knowledge are hard-coded into ontology systems by hand. They can also be useful when the domain is relatively well-structured and where it is important to understand and plan every decision made by the system such as in automated attendant phone systems.

3. Machine learning based approaches: As machine learning approaches extract patterns from data, they are relatively robust to subtleties present in complex domains that were not explicitly built into an algorithm, model, or ontology. They work best when a prediction needs to be made about which "class" or category a given input (or query in your case) belongs to, or when a quantitative prediction needs to be produced from data. For example, if I have a large collection of animal pictured labeled with the type of animal present in these images, I can train a machine learning model to predict which animal type is present in future images, as long as the model has seen enough pictures of this animal that it can extract the patterns in the image that correspond to the image likely representing a lion or a tiger or a bear, etc. This works because there are properties in the image itself that predict which animal it represents and these patterns can be extracted from labeled data (there are also machine learning algorithms that pick out patterns in unlabeled data called unsupervised algorithms, but these are not relevant for your domain of interest.)  Traditionally, machine learning systems are relatively poor at modeling domains where the answer depends not on recognizing a pattern, but having access to knowledge like in the movie stars example above. Machine learning systems are also great because they require relatively little manpower to implement. As you mentioned, they also require access to a dataset, which may need to be quite large depending on the complexity of the domain.

We can roughly summarize the advantages and disadvantages by ranking the methods according to a few criteria:

All of the best systems (ex: IBM Watson) use a hybrid approach, taking advantage of each method for its relative strengths and substituting other methods to address their weaknesses. Depending on the performance you want, a QA system can be built by a single person with some knowledge of any of the above, or may require a team of upwards of 100 engineers.


Roadmap to learn CNN in tensorflow from scratch I'm working in the medical field and I'd like to learn applications of CNN for image recognition and classification. All the (few) things I learned come from self-learning on the web or sparse books. I'm studying now Tensorflow for CNN implementation but I'm having trouble finding clear documentation for my actual level, so I think I'm missing the basic knowledge in order to understand this.

I'm at a basic level of python programming, I have better understanding of classical machine learning algorithms, which resources should I learn in order to get a good grasp of the argument? Is there such an ideal pathway to this?

I'm assuming you're talking about medical images classification, rather than localization.

Personally I recommend Kaggle, it has an awesome forum and people share their codes and opinions there.

You can start at Digit Recognizer, it's actually the well-known MNIST dataset(hand-written numbers). There's no relationship between MNIST and medical field. However there are some common techniques and tricks as they are both image recognition/classification problems.

If you encounter problems, read the other ones' codes at the kernel section, especially those with upvotes.

In the meanwhile, comment section is a good place to learn. I learned a lot there when I started to learn CNN.


Error loading tidyverse in R Studio - No broom package I'm getting numerous errors when trying to install packages, namely tidyverse and ggplot.

The error is of the form are always in the form:

> library(tidyverse)
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
  there is no package called ‘broom’
Error: package or namespace load failed for ‘tidyverse’

I have already tried installing the package broom independently with dependecies = TRUE per the below

Having trouble installing and loading tidyverse- No DIB package

I've also tried restart my R session as well as installing and running an updateR package/ function per:

How can I run tidyverse package?

Thanks,
Marc

It worth trying again because sometimes there is an issue with CRAN.

Try installing first

> install.packages("tidyverse")

Then loading

> library(tidyverse)


Why averaging the gradient works in Gradient Descent? In Full-batch Gradient descent or Minibatch-GD we are getting gradient from several training examples. We then average them out to get a "high-quality" gradient, from several estimations and finally use it to correct the network, at once.

But why does averaging the gathered gradient work? Each training sample ends up in a distant, completely separate location on the error-surface

Each sample would thus have its direction of a steepest descent pointing in different directions compared to other training examples. Averaging those directions should not make sense? Yet it works so well. In fact, the more examples we average, the more precise the correction will be: full-batch vs mini-batch approach

Each training sample ends up in a distant, completely separate location on the error-surface

That is not a correct visualisation of what is going on. The error surface plot is tied to the value of the network parameters, not to the values of the data inputs. During back-propagation of an individual item in a mini-batch or full batch, each example gives an estimate of the gradient in the same location in parameter space. The more examples you use, the better the estimate will be (more on that below).

A more accurate representation of what is going on would be this:

Your question here is still valid though:

  But why does averaging the gathered gradient work?

In other words, why do you expect that taking all these individual gradients from separate examples should combine into a better approximation of the average gradient over the error surface?

This is entirely to do with how the error surface is itself constructed as the average of individual loss functions. If we note cost function for the error surface as $C$, then

$$C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} L(x, \theta)$$

Where $X$ represents the whole dataset, $\theta$ are your model's trainable parameters, $L$ is an individual loss function for $x$. Note I have rolled labels into $X$ here, it doesn't matter for this argument whether loss is due to comparison of model output with some part of the training data - all we care about is finding a gradient to the error surface.

The error gradient that you want to calculate for gradient descent is $\nabla_{\theta} C(X, \theta)$, which you can therefore write as:

$$\nabla_{\theta} C(X, \theta) = \nabla_{\theta}(\frac{1}{|X|}\sum_{x \in X} L(x, \theta))$$

The derivative of the sum of any two functions is the sum of the derivatives, i.e.

$$\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$$

In addition, any fixed multiplier that doesn't depend on the parameters you are taking the gradient with (in this case, the size of the dataset) can just be treated as an external factor:

$$\nabla_{\theta} C(X, \theta) = \frac{1}{|X|}\sum_{x \in X} \nabla_{\theta} L(x, \theta)$$

So . . . the gradient of an average of many functions, is equal to the average of the gradients of those functions taken separately. Taking any completely random subset of $X$ will result in an unbiased estimate of the mean gradient, same as taking a random subset of any variable and taking its mean will give you an unbiased estimate of the population's mean.

This will not work if your samples are somehow correlated, hence why you will often see recommendations to shuffle the data prior to training, in order to make it i.i.d.

This will also not work if your cost function combines examples in any way other than addition. However, it would be an unusual cost function that combined separate training examples by multiplying their loss functions, or some other non-linear combination.


How does LightGBM deal with value scale? I understand that the loss metric can be used as linear, or log, or other things. This is documented at http://lightgbm.readthedocs.io/en/latest/Parameters.html?highlight=logloss#metric-parameters

I would like to understand how LightGBM works on variables with different scale. In other words, is it necessary for me to harmonize scale when running LightGBM? (I am used to linear regression where you need to get into linear scale.)

If I had inputs x1, x2, x3, output y and some noise N then here are a few examples of different scales.

$y = x1 + x2 + x3 + N $
$y = exp(x1 + x2 + x3 + N) $
$y = log(x1 + x2 + x3 + N) $
$y = sqrt(x1 + x2 + x3 + N) $
$y = log(x1 * x2 * x3 * N) $

Generally, in tree-based models the scale of the features does not matter. This is because at each tree level, the score of a possible split will be equal whether the respective feature has been scaled or not.

You can think of it like here: We're dealing with a binary classification problem and the feature we're splitting takes values from 0 to 1000. If you split it on 300, the samples <300 belong 90% to one category while those >300 belong 30% to one category. Now imaging this feature is scaled between 0 and 1. Again, if you split on 0.3, the sample <0.3 belong 90% to one category while those >0.3 belong 30% to one category.

So you've changed the splitting point but the actual distribution of the samples remains the same regarding the target variable.


How to measure F1 score and NMI for clustering task? I see the authors of this paper are measuring F1 and NMI scores to measure the clustering quality. However, I don't understand the algorithm of how they actually measure it. See the Evaluation Section.

Although I have looked at the code, I am not sure about the actual algorithm.

Use the original papers, and books, on F1 and NMI.

There is little reason to only use a 2016 arxiv (non-reviewed) paper that uses them, because they have been introduced long before.

I don't see F1 mentioned anywhere in the paper,and it would be an uncommon choice for clustering (see literature for details).


Reinforcement Learning in 2018, best tips and tricks? Putting aside things applicable to neural networks such as dropout, l2 regularization, new opitmizers - what are the cool things I should be adding to my Reinforcement Learning algorithm (Q-Learning and SARSA) to make it stronger, in 2018?

So far, I know these:

Experience Replay (gather experiences, then train at once on some of them)
Slowly making our network approach the Q-values of a Target Network. The target Network is only cloned sometimes.
If our function approximator is RNN and our Memory Bank stores sequences of several timesteps, after selecting few random sequences from Memory bank, only backpropagate a half of the sequence, without going back to its start

Have you looked at the Rainbow RL? It combines all improvements of RL. Apart from structural changes other improvements come from Reward Shaping, Curiosity-driven Learning, RL with Unsupervised Auxiliary tasks and RL with external memory (e.g. Neural Episodic Control, Model-free episodic Control). It's pitty that you leave out from your question Policy Gradients, they are lots of fun :) Happy reading!


My accuracy changes throughout every epoc but the val_acc at the end of each epoc stays the same I am training a transfer learning CNN on 161 pictures that have been augmented into 966 photos, 4 classes. I am training on a balanced data set, so there are 52 images of each class and also in the validation and test data, I am keeping the data balanced.

I am using MobileNet as a transfer learning network, and actually only using the first 8 layers as non-trainable layers, the rest are trainable and then I added another 3 dense layers with activation functions of relu and the last one softmax so as not to encounter the vanishing gradient descent.

Currently using the adam compiler, with a learning rate of 1e-6, beta_1=0.9, beta_2=0.999 decay=0.0000001 and amsgrad=True. Loss is categorical_crossentropy and metrics are accuracy.

Now, I am running 35 epochs, and with each epoch the accuracy changes from 0 to 40% (it changes every epoch, so far acc=0.4 was the highest that I saw and it doesn't always start from 0, it changes) but at the end of every single epoch the validation accuracy is 0.2 and the following epoch starts, it seems like, randomly. Why is that and how do I fix that? It's as if each epoch starts training it from the beginning. In addition, the actual loss hasn't changed much at all, it started, first batch first epoch with a loss of 1.3866 and it fluctuated max +-0.01 (not even).

Thanks in advance!

Your model complexity is high enough to learn features and give you results better than 20%. Although I would suggest to increase your learning rate to 1e-3 or 1e-4.

The accuracy in epoch refers to accuracy for the batch its training on so it might fluctuate. But is val accuracy and loss is same that means the model isn't learning distiguishable features. It could mean the problem is hard.


Text annotating process, quality vs quantity? I have a question regarding annotating text data for classification.

Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time to go through all the text samples, but at least a significant portion of them.

Should we focus on generating new samples for each annotator? (They never see the same text samples as any other annotator) (quantity approach).
Or should all annotators see the same samples and the annotator agreement is taken to account? (quality approach).

Thoughts,

will generate more unique samples than 2. (more training samples for a classifier) - and hoping that in the feature extraction part, the useful features will appear by themselves.
will generate less unique samples, but with the annotator agreement taken into account. (less training samples for a classifier, but with higher quality)

It depends on the context which you consider. For example, suppose there is a situation that all possible states can be covered by 10K different texts. It is trivial, if these texts are all annotated, then for 1000 tests, at least we can classify 500 of them truly (as we have two classes, and probability of wrong annotation for each text is at most 0.5).

Now, suppose 1K of 10K texts are annotated. Then, as annotations are exact, we can classify 1/10 of 1000 texts truly (because we have no idea about the other 9K possible states).

Therefore, in this situation quantity is more important than quality.

Also, we can consider these cases, when the possible states are 1K. It could be straightforward to show that in this case (if the power of annotators is same as the former case) quality can be more important than quantity. However, in the most cases this number is not realistic.

In sum, as in the most cases the variety of texts are more than the power of annotators, we prefer quantity to quality, as we can cover more text space and machine can learn more. Although the accuracy can be less, but for two classes classification is negligible.


Y-axis output is not order I am currently attempting to output some data based on user agent strings. The strings are too messy to place into a plot similar to the one shown below, so I plotted their index instead. Why is my plot outputting out of order? I would like to have my $y$-axis be in ascending order.

For context, I am plotting 6 plots with a for loop and none of them are producing the y-axis in order.

Edit: I added 'uas_id' dplyr code with solution.

gbuas_large<-gbuas_large%>%
              mutate(uas_id = sort(strtoi(rownames(gbuas_large))))%>%
              select(user_agent_string,uas_id,everything())

col<-names(gbuas_focus[,5:10])

for(i in col) {
  print(i)
  ua_p<-ggplot(gbuas_focus
              )+
    geom_point(aes_string(
                x=i,
                y='uas_id'),
                color="#70a6ff",
                size=3
               )+
    labs(x='Variable of Score',
         y='User Agent Strings',
         title="Understanding Misclassifying Variables"
        )+
    facet_wrap(~classification)
    print(ua_p)
}

It is in order, but not in the order that you want.

It is currently treating the indices as a string.

To get your desired index as numerics, here are some codes that hopefully can help you.

> c = c("1", "2", "13")
> c = sort(c)
> c
[1] "1"  "13" "2"
> c = strtoi(c)
> c
[1]  1 13  2
> sort(c)
[1]  1  2 13


Activation function between LSTM layers I'm aware the LSTM cell uses both sigmoid and tanh activation functions internally, however when creating a stacked LSTM architecture does it make sense to pass their outputs through an activation function (e.g. ReLU)?

So do we prefer this:

model = LSTM(100, activation="relu", return_sequences=True, input_shape(timesteps, n_features))
model = LSTM(50, activation="relu", return_sequences=True)(model)
...

over this?

model = LSTM(100, return_sequences=True, input_shape(timesteps, n_features))
model = LSTM(50, return_sequences=True)(model)
...

From my empirical results when creating an LSTM-autoencoder I've found them to be quite similar.

Simple explanation with images

We know that an activation is required between matrix multiplications to afford a neural network the ability to model non-linear processes.

A classical LSTM cell already contains quite a few non-linearities: three sigmoid functions and one hyperbolic tangent (tanh) function, here shown in a sequential chain of repeating (unrolled) recurrent LSTM cells:

Images borrowed from "colah's blog"

So far this is just a single LSTM layer, and here we see that the cell output is already the multiplication of two activations (a sigmoid and a hyperbolic tangent). In this case, you could agree there is no need to add another activation layer after the LSTM cell.

You are talking about stacked layers, and if we put an activation between the hidden output of one layer to the input of the stacked layer. Looking at the central cell in the image above, it would mean a layer between the purple ($h_{t}$) and the stacked layer's blue $X_{t}$. You will notice then, that the output in this case, like with the sequential output, has already been activated, as it is the exact same output as the black left-to-right arrow (know as $f_{t}$). What is more, the first thing the input would do in the stacked layer is to be passed through the sigmoids and hypoerbolic tangents of the forget/input/output gates.

So there are plenty of non-linearities being used, meaning it is unnecessary to add yet another between the stacked LSTM layers. You might like to think of it as simply applying two ReLU layers after a fully-connected layer. The results might be slightly different compared to just using one, but not much; as in your experiements with stacked LSTMs.

Documentation

If you look at the Tensorflow/Keras documentation for LSTM modules (or any recurrent cell), you will notice that they speak of two activations: an (output) activation and a recurrent activation. It is here that you can decide which activation to use and the output of the entire cell is then already activated, so to speak. PyTorch doesn't seem to (by default) allow you to change the default activations.

Real world stacked models

Common applications of recurrent networks are found in NLP, for example the ELMo model. If you look through the network design code, you see only basic LSTM cells being used, without additional activation laters. They only mention adding activations for the fully-connected layers (namely a ReLU) at the final output.

The first usage of stacked LSTMs (that I know of) was applied to speech recognition (Graves et. al), and the authors also do not mention the need for activation layers between the LSTM cells; only at the final output in conjunction with a fully-connected layer.


Reuters Dataset Labels description the list of Reuters dataset labels are provided with the dataset and also available in varous online resources. Here is an example.
But I couldn't find what each of these labels actually means.
I was able to find out 'acq' is short for 'Acquisitions'. But yet don't know these ones:
dlr, gnp, bop
Is there a description availbale about what these labels represent?

Apparently the paper introducing the dataset mentions a list of "online appendices" (table 1 p 363) which seems to contain details about the categories. However I wasn't able to find these appendices in the additional material.

[edit] Since it appears that there is no existing source for the original list of topics, I think your best bet is to rely on an acronym dictionary. Based on Acronym Finder, given that the data is mostly about economic news, I think the most likely meaning for the three topics you mention are:

DLR: dollar
GDP: Gross Domestic Product
BOP: Balance Of Payments

You could check whether these are correct by looking at a few documents labelled with these topics in the dataset.


When a dataset is huge, what do you do to train with all the images on i t? I'm using Python 3.7.7.
I'm trying to load a lot of NIFTI images using SimplyITK and Numpy from the [BraTS 2019 dataset][1].
This is the code I use to load the images into a numpy array.
import SimpleITK as sitk

def read_nifti_images(images_full_path):
    """
    Read nifti files from a gziped file.

    Read nifti files from a gziped file using SimpleITK library.

    Parameters:
    images_full_path (string): Full path to gziped file including file name.

    Returns:
    SimpleITK.SimpleITK.Image, numpy array: images read as image, images read as numpy array

    """
    # Reads images using SimpleITK.
    images = sitk.ReadImage(images_full_path)
    # Get a numpy array from a SimpleITK Image.
    images_array = sitk.GetArrayFromImage(images)

    # More info about SimpleITK images: http://simpleitk.github.io/SimpleITK-Notebooks/01_Image_Basics.html

    return images, images_array

This code works fine with smallest dataset but here I'm trying to load 518 nii.gz files with 155 images each file.
To run the code I'm using PyCharm latest version on a Windows 7.
How do you do it to train with all the images if all of them can't be in memory because memory limits?

A couple of options:

Rent a bigger computer on a cloud service.
Move to a distributed computing framework (e.g., Spark).
Use a data loading function that only loads the needed data.

Option #3 is the simplest. Most training does not need all of the data in memory at the same time.
Dask can be used for this type of image loading .


Is parsing files an application of machine learning? I presently receive files from a device in a semi-csv format. I have a written a simple recursive descent parser for getting information out of these files. Every time the device updates firmware, I have a new version of the parser for the changes the update brings.

Down the road, we will be taking data from other devices, which means another parser and more updates to firmware. I'm wondering if I could define a basic structure of "this is the data I need" and use a neural network to get the parsed data without having to write a parser for each new file type that comes in.

Is this a pipe dream or is it a valid application of machine learning? I'm much more of a software engineer than I am a data scientist, but I'm starting to dip my toes into the machine learning realm.

Thanks in advance.

I would answer the question at two levels. The first level is "can it be done using machine learning?" I would say that machine learning is essentially about learning. So given that you prepare sufficient examples of sample documents and the output to expect from those documents, you can train a network to learn the structure of documents and extract the relevant information. The more general form of extracting information from documents is a well-researched problem and is more commonly known as Information Retrieval. And it is not limited to just machine learning techniques, you can use Natural Language Processing tools as well. So, in its general form, it is actually being done in practice. Coming to the second level, "should you be doing it using machine learning?". I would agree to what @NeilSlater said. The better and more feasible approach would be to use good programming practices so that you can reuse parts of your parser as your dataset evolves.


Normalizing the data set I have two questions :

Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ?
When we normalize the training set we ought to normalize the target set too . Won't it affect the performance ? I mean won't the data set change completely because the model had different Ranges of features as compared to the ranges of features in target set .

I tried googling the questions but was not able to come to conclusion . Any help would be appreciated .

Thanks !

Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated)?

Theoretically, normalization does not influence the performance of the model. In order to understand this let us have a look at a standard linear regression.
$$y_i = \boldsymbol{w}^T\boldsymbol{x}_i + b+\varepsilon_i$$

If we Introduce the scaled independent variable $\boldsymbol{z}_i=\dfrac{1}{\sigma}\left[ \boldsymbol{x}_i - \bar{\boldsymbol{x}}\right] \implies \boldsymbol{x}_i=\bar{\boldsymbol{x}}+\sigma\boldsymbol{z}_i$. This will result in
$$y_i = \boldsymbol{w}^T\bar{\boldsymbol{x}}+\sigma\boldsymbol{w}^T\boldsymbol{z}_i+b+\varepsilon_i.$$

If we introduce $\tilde{b}=b+\boldsymbol{w}^T\bar{\boldsymbol{x}}$ and $\tilde{\boldsymbol{w}}^T=\sigma\boldsymbol{w}^T$ we can rewrite the equation into

$$y_i=\tilde{\boldsymbol{w}}^T\boldsymbol{z}_i+\tilde{b}+\varepsilon_i.$$

Hence, we see that the transformed independent variables just change the bias (if a translation is included) and the weights are scaled by $\sigma$. The significance of the parameters will not change only their specified values.

But if normalization does not really enhance the model, why do we still do it? The reason is more computationally inspired. If we had very large values the weights would need to be very small such that we can scale the output of the regression into a reasonable range. A large range of numerical values forces us to reserve more memory for our variables that we use. Hence, it is better to normalize our inputs, such that the parameters don't need to scale the inputs down to a reasonable amount that it fits the output.

  When we normalize the training set we ought to normalize the target set too. Won't it affect the performance? I mean won't the data set
  change completely because the model had different ranges of features
  as compared to the ranges of features in the target set.

Normalizing the output is not necessary, but it can also improve the numerical efficiency. You can just use the previous linear transformation on your dependent variable (output) and you will see that you can rewrite it to a standard linear regression in the new output. Just remember to transform your inputs and retransform your outputs if you want to use the original variables.

Does the significance of the parameters change?

In order to show that scaling the inputs by a constant factor $\sigma$ does not influence the significance of the parameters, we will calculate the $t$-value for a given coefficient. If the $t$-value stays invariant the $p$-value will also stay invariant.

For the linear regression $y_i = \boldsymbol{w}\boldsymbol{x}_i+\varepsilon_i$ (bias absorbed into the weight vector). The regression coefficients are given by

$$\hat{\boldsymbol{w}}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\boldsymbol{X}^T\boldsymbol{y}\quad \text{ , in which } \quad
\boldsymbol{X}=\begin{bmatrix}\boldsymbol{x}_1^T \\ \vdots \\ \boldsymbol{x}_N^T\end{bmatrix}$$

is the data matrix with an added $1$-column for the bias.

Additionally, we need the matrix
$$\boldsymbol{C}=\left[\boldsymbol{X}^T\boldsymbol{X} \right]^{-1}\quad \text{ and } \quad s_e = \sqrt{\dfrac{(\boldsymbol{y}-\hat{\boldsymbol{y}})^T(\boldsymbol{y}-\hat{\boldsymbol{y}})}{N-p-1}},$$

in which $N$ is the number of observations, $p$ is the number of predictors, $\boldsymbol{y}$ is the vector of outputs and $\hat{\boldsymbol{y}}$ is the vector of predicted outputs. We saw that the predicted values will stay invariant under scaling. Hence $s_e$ is invariant under scaling.

The $t$-value for a regression weight given by

$$t_i= \dfrac{\hat{w}_i-\mathbb{E}\left[\hat{w}_i\right]}{s_e\sqrt{c_{ii}}}.$$

The $c_{ii}$-values are the corresponding diagonal values of the $\boldsymbol{C}$ matrix.

If we scale our inputs by $\sigma$ (we ignore the 1 column of the data set which is only important for the bias) then we observe

$$\boldsymbol{X}'=\sigma\boldsymbol{X}
\text{ , }\hat{\boldsymbol{w}}' = \dfrac{1}{\sigma}\hat{\boldsymbol{w}} \text{ , } \mathbb{E}\left[ \hat{\boldsymbol{w}}'\right]=\dfrac{1}{\sigma}\mathbb{E}\left[ \hat{\boldsymbol{w}}\right] \quad \text{ and } \quad \boldsymbol{C}' = \dfrac{1}{\sigma^2}\boldsymbol{C}.$$
The last condition implies
$$\implies c_{ii}' = \dfrac{1}{\sigma^2}c_{ii} \implies \sqrt{c_{ii}'} = \dfrac{1}{\sigma}\sqrt{c_{ii}}.$$

By these observations, we see that the $t$-value stays invariant.

$$t' = \dfrac{\hat{w}_{i}'-\mathbb{E}\left[\hat{w}_{i}' \right]}{s_e\sqrt{c_{ii}'}}=\dfrac{\hat{w}_{i}-\mathbb{E}\left[\hat{w}_{i} \right]}{s_e\sqrt{c_{ii}}}=t$$

Hence, the significance of the regression coefficients didn't change either. In theory, the same should apply individually scaling the variables but the algebra gets more complicated.


How to find nearest neighbors in SMOTE I am reading the original paper by Chawla and others for SMOTE. I am trying to understand how to generate these synthetic examples for over-sampling the minority class. The paper says:

"Synthetic samples are generated in the following way: Take the difference between the feature vector (sample) under consideration and its nearest neighbor. Multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This causes the selection of a random point along the line segment between two specific features".

I understand the idea, take your sample, the nearest neighbor, pick a random point in between, what I don't understand is how these nearest neighbors are defined.

You need to compute the Euclidean distance between each point and sort theses distances.

That said, you can find an implementation in the imbalanced-learn toolbox. More precisely, you can see the different steps of the implementation that you mentioned: (i) fit a KNN, (ii) find the NN of each sample, (iii) generate a new samples.


What is my training score the mean_train_score or mean_test_score? I am using sklearn to train some models (random forest, decision tree). For the training I am using RandomsearchCV with Stratified k-fold as cross-validation. Then I make a predictions on the test set and calculate the test score.

However, I would like to compare the test score with the training score. I assumed I could use the mean_train_score of the cv_results_ report from the RandomseachCV as training score for the model, because I thought it would show the validation against the hould-out-fold from the k-folds.
However, I am not sure about this because there is also a mean_test_score.

I was looking for an explanation of the mean_train_score and mean_test_score. I know these scores exits also for the single folds. But how are these scores calculated? And is one of them my training score, which shows how my model during the training performed?

I found an approach of explanation, but it's too superficial for me: GridSearch mean_test_score vs mean_train_score

The mean_test_score is actually the mean score of the validation step for each fold. The "test" word is probably not well chosen by sklearn in that case, if you want to make the distinction between validation and test.

However, I don't think you are totally finished here, and you should therefore not compare mean_train_score with the test core.

Indeed, the cross-validation phase gives you the best set of hyperparameters (case with maximal "test" score, which is actually a validation score), but you should not keep the corresponding model, especially (but not only) if the training set has few observations or there are few folds.
You should instead re-train your model a last time, with this set of hyperparameters, but over the entire train set (not only the $\frac{k-1}{k}$ cases used during cross-validation). This trained model will give you the training score (over the whole training set) and the test score (over the test set).


Machine Learning vs Deep Learning I am a bit confused by the difference between the terms "Machine Learning" and "Deep Learning". I have Googled it and read many articles, but it is still not very clear to me.

A known definition of Machine Learning by Tom Mitchell is:

  A computer program is said to learn from experience E with respect to
  some class of tasks T and performance measure P, if its performance at
  tasks in T, as measured by P, improves with experience E.

If I take an image classification problem of classifying dogs and cats as my taks T, from this definition I understand that if I would give a ML algorithm a bunch of images of dogs and cats (experience E), the ML algorithm could learn how to distinguish a new image as being either a dog or cat (provided the performance measure P is well defined).

Then comes Deep Learning. I understand that Deep Learning is part of Machine Learning, and that the above definition holds. The performance at task T improves with experience E. All fine till now.

This blog  states that there is a difference between Machine Learning and Deep Learning. The difference according to Adil is that in (Traditional) Machine Learning the features have to be hand-crafted, whereas in Deep Learning the features are learned. The following figures clarify his statement.

I am confused by the fact that in (Traditional) Machine Learning the features have to be hand-crafted. From the above definition by Tom Mitchell, I would think that these features would be learned from experience E and performance P. What could otherwise be learned in Machine Learning?

In Deep Learning I understand that from experience you learn the features and how they relate to each other to improve the performance. Could I conclude that in Machine Learning features have to be hand-crafted and what is learned is the combination of features? Or am I missing something else?

As a research area, Deep Learning is really just a sub-field of Machine Learning as Machine Learning is a sub-field of Artificial Intelligence.

1) Unsupervised Feature Learning

Conceptually, the first main difference between "traditional" (or "shallow") Machine Learning and Deep Learning is Unsupervised Feature Learning.

As you already know, successfully training a "traditional" Machine Learning model (ex: SVM, XGBoost...) is only possible after suitable pre-processing and judicious feature extraction to select meaningful information from the data.
That is, good feature vectors contain features distinctive between data points with different labels and consistent among data points with the same label.
Feature Engineering is thus the process of manual feature selection from experts. This is a very important but tedious taks to perform!

Unsupervised Feature Learning is a process where the model itself selects features automatically through training.
The topology of a Neural Network organized in layers connected to each other have the nice property of mapping a low-level representation of the data to a higher-level representation. Through training, the network can thus "decide" what part of the data matters and what part of the data doesn't. This is particularly interesting in Computer Vision or Natural Language Processing where it is quite hard to manually select or engineer robust features.

(picture credits: Tony Beltramelli)

As an example, let's assume we want to classify cat pictures. Using a Deep Neural Net, we can feed in the raw pixel values that will be mapped to a set of weights by the first layer, then these weights will be mapped to other weights by the second layer, until the last layer allows some weights to be mapped to numbers representing your problem. (ex: in this case the probability of the picture containing a cat)

Even though Deep Neural Networks can perform Unsupervised Feature Learning, it doesn't prevent you from doing Feature Engineering yourself to better represent your problem. Unsupervised Feature Learning, Feature Extraction, and Feature Engineering are not mutually exclusive!

Sources:

http://deeplearning.stanford.edu/tutorial/
https://arxiv.org/abs/1404.7828
https://arxiv.org/abs/1512.05616 (Chapter 2, Section 2)

2) Linear Separability

Deep Neural Networks can solve some non-linearly separable problems by bending the feature space such that features become linearly separable.
Once again, this is possible thanks to the network topology organized in layers mapping inputs to new representations of the data.

(picture credits: Christopher Olah)

Sources:
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

3) Statistical Invariance

Lastly, Deep Neural Networks are surpassing traditional Machine Learning algorithms in some domains because some architectures are showcasing Statistical Invariance (ex: Spacial Statistical Invariance with Convolutional Neural Networks and Temporal Statistical Invariance with Recurrent Neural Networks)

Check this Udacity video for more details:
https://www.youtube.com/watch?v=5PH2Vot-tD4


PCA and maintaining relationship with target variable I'm rather new to PCA and was hoping to have some confusion cleared up. Lets say for example we have a feature matrix that's nx100 and I want to get it down to something a bit smaller, p-dimensions, without losing too much variance.

After applying PCA and receiving and new feature matrix nxp, I would use x_reduced to predict some target variable y.

My question is, after the transformation, the new reduced feature matrix has been rotated by the eigenvectors and is sitting on a new basis. Yet, our y has not changed relative to X_reduced.

I'm unsure about how y_original and x_reduced can be used for training since y has not changed with respect to x_reduced.

Is there a way to correct for this or am I not thinking about it correctly?

The short answer is that the y_original and x_reduced are still connected to each other, so it is safe to train your data using y_original and x_reduced.  While x_reduced is on a different scale, as you mentioned via eigenvectors, it still is representative of the data that was attached to that observation, just in a different format.  You lose a lot of interpretability as far as what the actual numbers mean which is why it may seem confusing, but it's just a transformed representation of the x_original that (hopefully) contains enough of the x_original variability to make it useful.


How to seperate text lines from txt files in python? I am playing with a .txt file in python for EDA and I want to seperate the lines from this fashion:

Corona Virus = Justin Piper

COVID19 = Piper, Justin
Fed Paper Says Companies May Automate More Jobs During Pandemic https://t.co/SUqrMfpNJY

Donate the money
Might put food on plate
And smile on face

to this fashion:

[['Corona Virus = Justin Piper'],['COVID19 = Piper, Justin
Fed Paper Says Companies May Automate More Jobs During Pandemic https://t.co/SUqrMfpNJY'],['Donate the money
Might put food on plate
And smile on face ']]

I tried .split('\t') but it is taking each line as a new row.

You can read the file line by line, discard blank lines, and wrap the lines that are left by a list to get the same as your desired result

with open(file_name, 'r') as f:
    print([[x.strip()] for x in f.readlines() if x.strip() != ''])

By doing [x.strip()] you clear the whitespace at the beginning and end of the file and then wrap it in a list, and the condition x.strip() != '' makes sure that the line actually contains content and is not just a blank line.


Model with rank as the dependent variable? I have a dataset with n records. The input features are f1, f2, f3 ... fm, and the dependent variable (output) is just the rank (not any type of scores).

What type of model should I build so that I can use the model to predict the rank of a new dataset? I suppose I could build a linear regression by assuming the dependent variable is just the rank (i.e. score = rank), but I am wondering if there is a better model or approach for this problem. Thanks!

Leaving aside any discussion of algorithms and statistics, you cannot predict a rank of an object without context. Are you supposed to rank it relative to your current rows, or in relationship to other new rows? For ranking algorithms based on features, you would usually want to assign an arbitrary score and then sort by it and assign ranks accordingly.


Python interface to Titan Database How can I connect to Titan database from Python ?

What I understand is that Titan (Graph database) provides an interface (Blueprint) to Cassandra (Column Store) and
bulb is a python interface to graph DB.

Now how can I start programming in python to connect with titan DB?
Is there any good documentation/tutorial available ?

You can use Rexster to expose a REST api and then you can use this api via Requests module (there are many other modules to do this task - though Requests is what I prefer)