BERT reasoning capabilities I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://towardsdatascience.com/how-does-bert-reason-54feb363211 that is able to compute the highest positive/negative attribution given to the words (the reason why I'm not able to use the approach presented in the aforementioned article is the price). In this way, I hope that I'll be able to better understand (and possibly correct) these misclassifications. I tried looking at the attention heads but I don't feel that I'm able to fully understand and draw conclusions based on this information.
Any help would be greatly appreciated!

BERT is free. Any Sentiment Analysis model will achieve the task.


GridSearchCV results are different to directly applied default model (SVM) I run a Support Vector Machines model on part of my train set with following result:

alg = sk.svm.SVC(probability=True, gamma='auto')
cv_results = model_selection.cross_validate(alg, X_pca, labels, cv =4)

but when I am trying to tune the parameters, with following method:

model=sk.svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0.1, 1, 'auto'],
      'probability' : [True]
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False).fit(X_pca, labels)
pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score', 'params']].sort_values(by='rank_test_score')

So not only all results looks scetchy because they are the same. but also in one of the rows I have C:1, gamma:auto and probability: True which is the same parameters as in first table.

I want to also say, that the same logic I am using for the rest of my 15 ML algorithms and only SVM showed this kind of weird behavior. Wondering that maybe I have some stupid mistake in how I create X_pca and labels data table, I copied code from other algorithm and just replaced second code but it gave the same results.

Can you spot something wrong?

I think the problem might be due to the data, as this code:

from sklearn import svm
from sklearn.model_selection import GridSearchCV
import pandas as pd
import numpy as np

X_pca = np.random.rand(100, 2)
labels = X_pca[:, 0] + X_pca[:, 1] > 0.5

model = svm.SVC()
params = {'C' : [0.01, 0.1, 1, 10],
      'gamma' : [0., 0.1, 1, 'auto'],
      'probability' : [True]
     }
clf =  GridSearchCV(model, params, cv=2, return_train_score=False)
clf.fit(X_pca, labels)

print(pd.DataFrame(clf.cv_results_).loc[:, ['mean_test_score', 'rank_test_score']] \
        .sort_values(by='rank_test_score'))

gives better output:

    mean_test_score  rank_test_score
10             1.00                1
13             1.00                1
14             1.00                1
15             1.00                1
11             0.95                5
0              0.83                6
1              0.83                6
2              0.83                6
3              0.83                6
4              0.83                6
5              0.83                6
6              0.83                6
7              0.83                6
8              0.83                6
9              0.83                6
12             0.83                6

(consider the fact that the results will depend on the seed of the rand function, but with other seeds they are similar and the mean test score changes using different values of the parameters.


Where to know the list of NLTK tagset? I need to know what the abbreviations mean, e.g 'VBD', 'NN', etc. I am learning text processing and now I want to get verbs in the sentences, so that I need to know which abbreviations represent any form of verbs.

No one answered the question, then I will answer it myself. Thanks to @user12075 for the link. I didn't find it when I was googling it.

https://stackoverflow.com/questions/15388831/what-are-all-possible-pos-tags-of-nltk

From the above link, I know that nltk uses The Penn Treebank's POS tags.

nltk.help.upenn_tagset() will give you the list.


Visualizing categorical data in scatter plot with Orange (version > 3.18) Why in Orange versions after 3.18 is not possible to visualize categorical data in Scatter Plot widget?

As a work around, you can change the data type in the file widget, from Categorical to Continuous. Then you can use scatter plot.


Trouble doing a simple Box Plot This is a problem that I keep having when trying to do a boxplot in R. From what I can see the code I am using should work, not only that but when compared to my notes and other classmates it appears to be the same. What am I doing wrong?

result <- read.csv("keough.csv", header=TRUE, sep=";", dec= ",")
head(result)

 BIOFILM.SERP
1        SL,61
2       SL,113
3       SL,123
4        SL,75
5        SL,75
6        SL,83

attach(result)
boxplot(BIOFILM~SERP, dat=result)

  Error in eval(predvars, data, env) : object 'BIOFILM' not found

result <- read.csv(“Keough.csv”, header=TRUE, sep=“,”, dec=“,”)


pandas: how to change the specific column as index and change index into various columns Hi I'm new to data science. Learning data science from course-era.
I'm having pandas data frame as follows,

   time   value

A   9      5
A   8      4
A   7      3
B   9      3
B   8      2
B   7      1
C   9      3
C   8      2
C   7      1

I want to convert this as ,

       A   B  C

9      5   3  3
8      4   2  2
7      3   1  1

As I start to write query for this, it is getting complicated. Is there any easy way to do this? Thanks for the help.

For me, when it comes to reshaping a dataframe(switching columns/indices/rows and such) its fairly intuitive using the pivot_table function.

my_df.pivot_table(index='time', columns=my_df.index, values='value')


Predict the probability that a user is active on a website I have been given two different arrays, which describe if a user is active on a site or not based on some features. The first array is for training. Each row of the array is a different user and the columns are different features. The first 896 columns describes the users using-patterns and the remaining 28 are lables (These 28 columns stands for 28 different time periods, they are labled 1 if the user is active in the time period, 0 otherwise.

The second array is for testing and has the same amount of columns, but here the last 28 rows are empty. My task is to predict the probability that a user will be active during each time period. I am a bit lost and are looking for (preferably quite simple) methods to solve this problem with.

I’ll throw out a couple of hints / clarifications.

Your ‘test’ set is of no use to you since you have the dependent variables missing, so just ignore it to begin with.
You will want to split you ‘training’ data into two sets to begin with, one for training a model and one for testing the model and computing the effectiveness of the model.
Hopefully you have lots of records in the training set because you have so many independent variables. Some models are very sensitive to low sample size and high dimensionality.
You will need to decide if the feature columns need to be encoded. You’ll have to pay particular attention to any categorical features because they need to be one hot encoded.
You will need to choose a model or ensemble of models. You may start by trying a binary classifier. You will need to creat one for each of your timeframes. Each model should take the ~890 features and produce one result with a 0 or 1.
You will need to figure out how to evaluate your models to understand how the choices you are making to tune each model are influencing the results.

This is not a comprehensive list, but hopefully it is helpful to get you started.


SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier.

I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 training examples for each category.

All looks good and the classifier is working with acceptable precision on unseen documents. But what I am noticing that often when classifying a new document, the classifier is often assigning a very high probability to one of the categories (the prediction probability is almost equal to 1), while the other categories receive very low probabilities (close to zero).

What are the possible reasons for this phenomenon?

I can think of one possible reason, which is all the words in some documents did not appear with some categories in the training dataset? but I am not totally convinced of that especially that for most of the results, there is always one category who has very high probability and all the others have very low probability.
Is there any other explanations or reasons?

I would like to add that in SPARK ML there is something called "raw prediction" and when I look at it, I can see negative numbers but they have more or less comparable magnitude, so even the category with the high probability has comparable raw prediction score, but I am finding difficulties in interpreting this scores.

The reason NB is called "Naive" is that is makes the assumption that the predictive variables are all independent.
This assumption usually skews the model scores (which, under the above naive assumption are unbiased probability estimates) towards 0 or 1.

In your case, e.g., the presence of words flower and petal indicate gardening category, but, because the presence of these words is not independent (if one is present, the other is likely to be present too), the model will over-value their appearance. Taking the extreme case, if words A and B appear only together, then

P(Category=X | A & B) = P(Category=X | B) = P(Category=X | A)

and thus one should multiply by the odds ratio of A&B once, not twice, as the Naive Bayes algorithm requires.

Your remedy is to use calibration (a separate model which maps model scores to probabilities).


Rolling average: when is it possible to consider it? I would like to know if I can consider rolling average to predict the future trend of sells.
I collected data from January 2020 to March 2020, day by day, on sells in a shop and I would like to run some analysis.
I was considering rolling average on multiple periods (5 - 10 days). Do you think it might make sense?

You can! But the implications can be huge.
If you choose a big number, you automatically shorten the dataset from which you can train your time-series or forecast model. Also, if things start to "change" in recent time, your model might take too long to react if it weights those rolling averages heavily.
If you go too short, it might overreact to recent changes, or you're just missing that information from which your model might be able to learn on. I usually make that decision based on how I want to use the forecast on unseen data. This is not an easy decision for which there is a solid rule-of-thumb.
Welcome to the real-world complexities of a time-series problem!


LSTM - How many times should I look back to predict next six hours -Multivariate Time-Series I am still finding confusing on look back topic when using LSTM for time-series analysis.
If I have hourly data and I want to predict next 6 hours with multiple predictors, should I look back up to 6 hours when I prepare my training set?

OR

Should I look back one hour (shift 1) and then predict next hour and take that predicted value and feed it back to predict the value after until next 6 hours?

This concept is still a little fuzzy for me in LSTM. Any thoughts would be appreciated.

The second option sounds fun but better don't do it: errors will sum up and it's much harder to track where errors come from. Like you have some error for the 1st hour, feed it back in the net, predict the 2nd hour get an error there and can't really tell how much of that is caused by the error of the first hour.

Besides that, I haven't seen any application which works with this feedback mechanism.


What is the best way to use word2vec for bilingual text similarity? I face a problem where I need to compute similarities over bilingual (English and French) texts. The "database" looks like this:

+-+-+-+
| |F|E|
+-+-+-+
|1|X|X|
+-+-+-+
|2| |X|
+-+-+-+
|3|X| |
+-+-+-+
|4|X| |
+-+-+-+
|5| |X|
+-+-+-+
|6|X|X|
+-+-+-+
|7|X| |
+-+-+-+

which means that I have English and French texts (variable long single sentences) for each "item" with either in both version (in this case the versions are loose translations of each other) or only in one language.

The task is to find the closest item ID for any incoming new sentence irrespective the actual language of either of the sentence in the "database" or of the incoming sentence (that is, the matching sentence in the "database" needn't necessarily be in the same language as the incoming sentence as long as the meaning is the closest). I hope this goal explanation is clear.

Originally I planned to build a word2vec from scratch for both languages (the vocabulary is quite specific so I would have preferred my own word2vec) and find similarities only for the corresponding language for each new sentence but this would omit all candidates from the items where the corresponding language sentences are missing.

So I wonder if generating a common word2vec encoding for the combined corpus is viable (the word2vec method itself being language agnostic) but I cannot figure out if such a solution would be superior.

Additionally, the number of the sentences is not very large (about 10.000) maybe word2vec generation from scratch is not the best idea on one hand, but there are really specific terms in the corpora on the other hand.

This paper from Amazon explains how you can use aligned bilingual word embeddings to generate a similarity score between two sentences of different languages. Used movie subtitles in four language pairs (English to German, French, Portuguese and Spanish) to show the efficiency of their system.

"Unsupervised Quality Estimation Without Reference Corpus for Subtitle Machine Translation Using Word Embeddings"


Decision tree and SVM for text classification - theory I used 4 classifiers for my text data: NB, kNN, DT and SVM. As for NB and kNN I fully understand how they work with text - how we can count probabilities for all words in NB and how to use similarity metrics with TF-IDF vectors in kNN I don't understand at all how decision tree and support vector machine work with text data. I implemented all algorithms in Python so all I need is some resource or explanation how the other two classifiers work with text...
I understand DT with non-text data - it seams logical for example nodes with checking if some data is more/less than some number. But with text I get confused. Does it operate on text or with numerical vectors? The same applies to SVM...

Similarly to NB or kNN, the DT and SVM algorithms work with the features which are provided as input. So whenever ML is applied to text it's important to understand how the unstructured text is transformed into structured data, i.e. how text instances are represented with features.
There are many options, but traditionally a document is represented as as a vector over the full vocabulary. A very simple version of this is a boolean vector: a cell $v_i$ contains 1 if the word $w_i$ occurs in the document and 0 otherwise. The DT training will generate the tree the usual way, so in this case the conditions at the nodes will be v_i == 1, representing whether the word $w_i$ is present or not. If the values in the vector are say TFIDF weights, the conditions might look like v_i > 3.5 for instance. Similarly for SVM: the algorithm will find the optimal way to separate the instances in a multi-dimensional space: each dimension actually represents a single word, but the algorithm itself doesn't know (and doesn't care) about that.


How to measure a almost (?) ordinal classification? I have a model where I predict classes to define instructions for a trader robot. The classes are -2, -1, 1 and 2 (strong sell, light sell, light buy and strong buy) and I'm using a simple confusion matrix to asses the performance of the model, however I would like to know if there are other ways that takes into account the "distance" between the classes.
I've found some content about ordinal regression, but I already made my classification in other way (not sure if this content applies to me though).
Any suggestions?

You could just calculate the mean absolute error.
However, this quality measure (and any other measure taking into account the distance) would not be consistent with the loss function you used while training your classifier I guess. So it would not be fair to your algorithm. It is like if your professor would change the grading system after you submitted your test.
Instead of changing the quality measure you might want to reconsider your approach. The suggestion to use the ordinal regression sounds good. In case if the cost of misestimation between neighbors is similar you could even use plain regression and then round.


Can Hadoop be beneficial when data is in database tables and not in a file system I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured data) as opposed to OLAP databases (Netezza)

'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (Seems like everyone has their own version of it these days)

To be honest though, if you're asking if it could be helpful, you probably don't need Hadoop (sorry in advance of that comes off harshly, it's not intended to). Lots an lots of places think they need Hadoop, but very few actually need Hadoop. There are of business that are down on the tech because they transitioned to it when the need didn't really exist. If you truly do need Hadoop or another distributed system it'd be almost impossible to determine which setup would be beneficial to your org without an intimate understanding of your data, and your specific business means.


Significant drop from validation accuracy to test accuracy I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I consistently achieve decent validation accuracy but significantly lower accuracy on the test set.

I've been performing validation like this:
1) Standardize the training data; store the mean and variance (in an sklearn Scaler object).
2) Hold out 10% training data for validation.
3) Perform 3-fold cross validation on the remaining 90% training data. Look at the errors.
4) Train the model on the 90% training data (with 10% of this subset used as validation during training).
5) Evaluate the performance on the 10% held-out training data. This is used as my final validation measure.

Then I test my model by:
1) Standardizing the test data using the mean and variance of training data.
2) Predict and evaluate

Depending on the model I test, I typically receive 10+% drop in accuracy from validation to test. Now, to test whether this is caused by the test set coming from a different distribution, I combined the test set (~2k) with an equal-sized random portion of the training set. Then I held out 10% of this as a new test set and performed training/validation as described above using the rest. So in this case the test set is no longer an independently collected dataset.

It appears that I'm in some way overfitting to the validation set, although as I describe above, the validation portion is held out until after training.

I think it is worth noting that my target variable and many predictors are exponential-like distributed (the target variable has a lower bound at 0 and increasingly rare large values). Heteroscedasticity is apparent because plotting predicted vs. observed output shows increasing error with larger values.

Here are results using an MLP regressor with the original training data and test data (Left: Validation, Right: Test):

Here are results using an MLP with equally combined train/test (~4k) as training, with 10% held out for testing (as described) (Left: Validation, Right: Test):

Here are results using a Random Forest with same train/test as for the figure directly above (Left: Validation, Right: Test):

I would greatly appreciate any ideas of the cause of this discrepancy. Thank you!

I don't think this is ususual, I experiance it quite frequently with regression problems. Generally I think it means the model is underspecified so instead of learning the actual relationship between X and y it overfits. Generally I find that I have to do a bit of feature engineering, and ensure I'm only using features that are really necesary - too many features or features with spurious relationships to the target can really hurt a RF model in particular in my experiance. Also make sure as part of your cross validation that you're cross validating parameters which directly influence model complexity i.e. tree depth, number of trees etc.


Result of Reinforcement learning I've started reading some literature about reinforcement learning and I can't understand what is the result of the application of RL. I'll be more specific: let's have a time series problem in continuous state space, finite numbers of actions and a linear approximator of the policy function. So I follow an algorithm to find the best policy, that is, in this specific case, the optimal values of the weights of the linear function I've considered. Now my doubt is here: the so-called best policy is the one found in the process of applying the algorithm or I have to take the final optimal values and, for each period, I have to use them to find which action maximise the action-value function? In other words, the result of RL is a classic function to (re)apply at each time step, as if it was a regression? I think the answer of this question is No, but I would appreciate if someone can confirm this.

(to better explain what I meant with "policy found in the process of applying the algorithm" let's consider this stupid consideration: the best policy also include those time steps of exploration)

Is the result of RL is a classic function to (re)apply at each time step?
In some manner yes, when using RL to find the best policy you end up with a policy that can be described as a function(classic or not) from possible states to possible actions.

As if it was a regression?
No, regression algorithms 'solves' the function between feature space and target space.
 In RL both of these spaces have no(very different) meanings.
In addition, RL algorithms take into consideration multi-step predictions(state transitions + rewards), which is not very straight-forward in regression problems.


What is the Most Efficient Tool in Python for row-wise manipulation of data? I'm doing a lot of work that requires operations to be performed across rows, using the data in that rows's columns on other columns in the row.
I recently had to do some processing on a 1.2 million row dataset (27 columns, 300 Mb) that required row-wise modification of ~ 300,000 rows. Some transactions were being split across multiple records and I wanted to aggregate them. This involved collapsing them upwards with lazy deletion, summing 5 columns and doing a weighted average on a 6th.
I at first attempted to use Pandas.
Pandas spent 12+ hours to go through ~ 80,000 rows (the machine I'm using at the office is trash).
I could maybe see that if I were dropping the rows as I went and it had to resettle 300 Mb of arrays in memory after each row, but it never even got around to deleting rows since that was set to happen at the end.
I've generally seen that the sentiment around Pandas seems to be "Don't iterate over a dataframe's rows."
I'm not asking for how to make it work in Pandas, the purpose of the above was to provide context as to why I believe that Pandas isn't a viable option for what I was trying to do.
I ended up going for a basic list implementation and it ran in 5-10 seconds.
In this case, the list implementation worked. Since I had the indices of the rows in the dataframe, I avoided the O(N) look-up time pitfall.
So, while Pandas is highly optimized for column-wise operations, it's evidently not really intended to perform granular row modification, en masse (Case in point, look at the big bolded warning in the iterrows documentation).
So here's the question:
Assuming you have a non-uniform row of data, with different sized columns and data types, what is the most efficient data structure/ library to perform a large number of row-wise modifications?
What about if all the columns were floats?

It depends on whether you can separate or know in advanced what rows are affected per operation.
In this instance, if indeed they involve collections then Dask Bags. For more non-structured data (as you suggest) but may be easy to run in parallel then Dask Delayed has worked for me.


Is there a way to put a separate line between clusters for k-Means Clustering? k-Means Clustering is a method of vector quantization, originally from signal processing, that is popular for cluster analysis in data mining.

here is a piece of code to perform a 2-d k-Means Clustering

from sklearn.datasets.samples_generator import make_blobs
X, y_true = make_blobs(n_samples=300, centers=3,
                       cluster_std=1.1, random_state=0)
plt.scatter(X[:, 0], X[:, 1], s=50);
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3)
kmeans.fit(X)
y_kmeans = kmeans.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=y_kmeans, s=50, cmap='viridis')

centers = kmeans.cluster_centers_
plt.scatter(centers[:, 0], centers[:, 1], c='black', s=200);

outputs a figure (fig_1)

this figure (fig_2) comes from wiki

is there a way to put a separate line shown in fig_2 on fig_1?

there are two answers to this question.

The first one is yes, you can do it with python code. From a Sklearn tuto, you can plot the decision boundary by using meshgrid:

# Step size of the mesh. Decrease to increase the quality of the VQ.
h = .02     # point in the mesh [x_min, x_max]x[y_min, y_max].

# Plot the decision boundary. For that, we will assign a color to each
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1)
plt.clf()
plt.imshow(Z, interpolation='nearest',
       extent=(xx.min(), xx.max(), yy.min(), yy.max()),
       cmap=plt.cm.Paired,
       aspect='auto', origin='lower')

plt.plot(X[:, 0], X[:, 1], 'k.', markersize=2)
# Plot the centroids as a white X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
        marker='x', s=169, linewidths=3,
        color='w', zorder=10)
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())
plt.show()

And there is a second answer, the analytical one:

The decision boundary is the group of points at which we can't decide the label. For the Kmeans it is the equidistant points from two centroides. A small calculation shows that this is a line. To find it's equation, you only need the coordinates of the two centroides.

After that you find the middle of the segment between those two, find an orthogonal vector to this segment going throw the middle and you have your equation. You just need to plot it and voila! Coding this don't take long, i'll try to update the answer if you want.


Getting GitHub repository information by different criteria New to the Data Science forum, and first poster here!

This may be kind of a specific question (hopefully not too much so), but one I'd imagine others might be interested in.

I'm looking for a way to basically query GitHub with something like this:

Give me a collection of all of the public repositories that have more than 10 stars, at
least two forks, and more than three committers.

The result could take any viable form: a JSON data dump, a URL to the web page, etc. It more than likely will consist of information from 10,000 repos or something large.

Is this sort of thing possible using the API or some other pre-built way, or am I going to have to build out my own custom solution where I try to scrape every page? If so, how feasible is this and how might I approach it?

My limited understanding, based on brief browsing GitHub API documentation, is that currently there is NO single API request that supports all your listed criteria at once. However, I think that you could use the following sequence in order to achieve the goal from your example (at least, I would use this approach):

1) Request information on all public repositories (API returns summary representations only): https://developer.github.com/v3/repos/#list-all-public-repositories;

2) Loop through the list of all public repositories retrieved in step 1, requesting individual resources, and save it as new (detailed) list (this returns detailed representations, in other words, all attributes): https://developer.github.com/v3/repos/#get;

3) Loop through the detailed list of all repositories, filtering corresponding fields by your criteria. For your example request, you'd be interested in the following attributes of the parent object: stargazers_count, forks_count. In order to filter the repositories by number of committers, you could use a separate API: https://developer.github.com/v3/repos/#list-contributors.

Updates or comments from people more familiar with GitHub API are welcome!


Why does np.linalg.eig produce an opposite-signed eigenvector? I am learning SVD by following this MIT course.  In this video, the lecturer is finding the SVD for

$$
\begin{pmatrix} 5 & 5 \\ -1 & 7 \end{pmatrix},
$$

which involves finding the eigenvalues for

$$
C^T C =
\begin{pmatrix}
26 & 18 \\ 18 & 74
\end{pmatrix}.
$$

In the example (at the time in the link above), the lecturer finds eigenvalues

$$\begin{pmatrix}-3/\sqrt{10} \\ 1/\sqrt{10} \end{pmatrix},
\begin{pmatrix} 1/\sqrt{10} \\ 3/\sqrt{10} \end{pmatrix}.$$

But np.linalg.eig produces the opposite vector to the second one:

w, v = np.linalg.eig(C.T*C)
v
matrix([[-0.9486833 , -0.31622777],
        [ 0.31622777, -0.9486833 ]])

Why?

Any scalar multiple of an eigenvector is also an eigenvector.  LAPACK (which np.linalg.eig uses under the hood) chooses to return unit-length eigenvectors (good for SVD!), but this still leaves two choices, and there doesn't seem to be a convention for which one to return; it's up to the underlying algorithm (which in turn may depend on the input data).

https://stackoverflow.com/questions/17998228/sign-of-eigenvectors-change-depending-on-specification-of-the-symmetric-argument


Data amount for a very simple chatbot For a project at my university I have to develop a simple chatbot. Since I am new to machine learning, it should be really simple and work like a customer support. The chatbot has to recognize about 10 intents related to some really simple topic. For example I thought that it could give information about a party, so the questions would be something like "Where is the party?, When does the party start? How much does the entry cost?"

The chatbot should classify the question, and out of the classification it should give back a prepared answer. If it cannot assign the question to the one of the 10 intents, it should do nothing or just answer that it does not understand the question. The same may happen if the question has spelling mistakes.

So it should be really simple, but be programmed with the deep learning approach.

My question is, how much data should I have, to train this bot to assign questions to the intents? Would I be able to generate the data by my own, if I reformulate each question on, lets say 20 ways? Or do I have to  feed the algorithm with a very large amount of data, even for such simple task?

Crowdsource your results.

Your problem is very similar to writing Amazon Alexa skills. Creating intents, utterances and slots for Alexa skills. One of the recommended methods for developing this section of Alexa skills is to simply ask your friends and peers how they would go about asking for a given outcome. Crowdsourcing your inputs will give you some robust results because other people may think quite differently than you. I would recommend hopping into whatever Slack/Discord/chat channels you belong to and asking there. There may be more formal sources for such information out there but I am not directly aware of them.


Strange Pearson Correlation Coefficient Given DataFrame I have a Python DataFrame that looks something like:

pageviews, type
100         0
50          1
...

When I run:

df.corr()

            pageviews   type
pageviews   1.000000    -0.009611
type       -0.009611    1.000000

So there's less than a 0.01 correlation between type and pageviews.  However, when I do:

df.groupby('type')['pageviews'].mean()

I get back:

type
0    1421.406621
1     885.092874
Name: pageviews, dtype: float64

So type 0 has almost twice the number of average pageviews but there's very little correlation.

How is this possible?

Thanks!

If two variables are independent, then their correlation will be zero. However, you cannot say the opposite. Zero correlation doesn't necessarily imply independence.

It's hard to answer your question without any plot or the raw data, but you need to seek information for non-linear dependency.

This is a great example from Wikipedia page to understand that Pearson cannot prove dependency


What if we input sequence data to feedforward network? One main advantage of RNN is the ability to take input of variable length like the case of sequences.
However, what if we neglected this case and assumed some applications that may accept some fixed window of input. For example, the weather prediction. We may input the temperature of the last 300 days for example to allow the network to predict the new temperature. We may fix this window of 300 all the time so that we have a fixed-length input.
My question now is what if we neglect the RNN in this case and input this window of 300 days into a feedforward network with 300 input dimensions and one output that is doing regression for it?
What will be the output as compared to RNN? In other words, why I should use RNN in this case?

A finite impulse response (FIR) RNN can be expressed as a directed acyclic graph (DAG), hence it can be represented as a FFNN. So, you could theoretically make an equivalent FFNN to an RNN in this situation.
In terms of performance between non-equivalent models, an RNN would probably be better because, as they are inherently designed for sequential data. In the case of weather stat prediction, the temporal nature of the sequence elements would need to be learned by a FFNN, whereas an RNN would have in-built knowledge of such relationships. The RNN would also be able to more easily take advantage of "memory" via the common gating mechanisms as seen in LSTMs and the like.


Sklearn - Override random_state=None by default Many scikit-learn and pandas objects/functions use random_state=None as a default parameter. How can it be overridden to random_state=100 by default for all objects without manually editing the random_state for each object?

make_blobs(n_samples=100, n_features=2, centers=3, cluster_std=1.0,
           center_box=(-10.0, 10.0), shuffle=True, random_state=None)

For scikit-learn can set np.random.seed(1), for example, and as long as nothing in your script is modifying the seed nondeterministically then you should get reproducible results.

This is described in the scikit-learn FAQ under How do I set a random_state for an entire execution?

However, I don't believe it is possible to do the same thing for pandas. See here for discussion.


Why do we need to use sysfunc when we call a SAS function inside a SAS macro I saw this piece of code in my project:

%let num = test;
%let x=%sysfunc(trim(&num));

Why could not  I write:

%let x= %trim(&num);

Why did I need to use sysfunc?

Under what circumstances can I call a function inside a macro without using sysfunc?

Without the sysfunc(), the expression will not be evaluated. You will not be assigning the value of the expression trim(&num) to the macrovariable, but rather the whole expression.

If you want to store the result of an expression, you need to execute that function with sysfunc()

http://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#z3514sysfunc.htm


How to improve accuracy of Named entity recognition (NER) tagger on local data? I am using NER from spacy. Its giving incorrect results for few words. Its trained on general dataset. How can I customize on my local data.

For example,

Person -  {'Mike Miller', 'Miller', 'Infantino', 'Gianni Infantino'}
Location -  {'England', 'UK', 'Europe', 'Telegraph'}

Here, "Telegraph" is incorrectly is assigned to location.

@Ravikm, excellent question. In Spacy, you can assign a word manually. For example, "Tesla" to ORG. Source: screenshot from Jose Portilla's NLP course on Udemy.


Can I expect necessarily higher accuracy when feature selection/dimension reduction is used to select a subset of features? The feature selection/dimension reduction is performed to eliminate irrelevant or redundant features so it will improve the computation efficiency (less computationally expensive). My question is that can we expect any changes in the accuracy of prediction when efficient features are used for classification vs when all features used?

Using all features would usually lead to over-fitting, that is your model would not generalise well to unseen data. To overcome this, while using as much information possible we resort to feature selection ( sometimes feature generation ), and dimensionality reduction. Dimensionality reduction techniques like Principal Component Analysis (PCA) tries to find new features that explain maximum variability in data, using the existing features. In this way, we ideally reduce the number of features in the model, while making sure most of the variability is explained.
So as a result:

This new model is computationally faster since we are no longer using all the features and burdening our model.
Generalises well since we are not over-fitting leading to greater accuracy on unseen data.


What are Laplacian scores and how do they effect feature selection? I was going through a paper related to feature selection wherein I constantly came across the term Laplacian Scores which I was not able to understand. Can anyone explain their importance in feature selection?

As @Spacedman said, the paper provides a very clear explanation of the algorithm on page 2.

If you're a little less comfortable with the math of the notation, here's the intuition/explanation in words.

Make a k-nearest neighbor's graph. That is, for each observation, define an edge in the Graph for that observation if another observation is one of its k-nearest neighbor's. If you're using a supervised algorithm you can define an edge if they share the same label.
If any two nodes (observations) are connected, define a weight matrix S measuring the similarity between those two nodes (using some distance measure).
For each feature define the Laplacian graph.
Compute the Laplacian score based on their equation.

Intuitively, you're using KNNs to define a network graph and assessing how similar the features are according to your distance metric. This is just as good of a measure of feature importance as any other but will also has its pitfalls, just like all of the others.


How to bin a distribution data reported with different frequencies ( salaries ), showing mixed linearity and non-linearity? I am researching on pay-scales, and wish to receive advise to treat data of salaries.
Objective
My interest is to approximate the salary corresponding to different hierarchical levels in an organisation.
Methodology
How to bin
I thought about using quantiles : knowing that there are, say, 10 levels in an organisation (e.g. President, Director, ..., Worker) I would like to estimate the average pay for the corresponding level.
I thought to use quantiles; am looking at the doc of pandas :

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html#pandas.qcut
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html#pandas.cut

Which choice would it be more appropriate ?
Describing observations
My distribution seems to be linear to a certain point, and then it seems to follow an exponential curve:

Could you advice best approach to approximate the quantiles ?
I am in doubt if calculating quantiles against the whole distribution, or if infer two distributions to best reflect observations.
I thought about extracting one distribution as a collection of data points lower than x standard deviations, and a collection of points greater than x standard deviations.
Respectively, I would have these distributions then:

(describing lower payscales, which seems to me linear)
and

(describing higher payscales, which seems to me exponential)
Advices for the analysis are much appreciated.
Considerations about the sample
Keep in mind each values are reported differently in their frequency (for lower payscales, there are more reports, which it makes sense because there is more turnover).
Should I keep into account to estimate an error ?

I'm going to just answer about the difference between the two functions qcut and cut because it's a very important difference:

the first qcut is indeed about quantiles, which means that it's about dividing the data into bins, each containing an equal number of points. For instance if you use deciles it means that there is the same number of people in the first decile as in the last (or any other), so the useful information is the range of values for each bin. For example if the first decile ranges from 0 to 500, then 10% of the people are paid less than 500.
the second cut is not about quantiles, it simply creates bins which all cover an interval of values of equal length. For instance the first interval would be 0-500, the second 500-1000, etc. In this case the number of points in every bin is usually different, and that's the useful information. Typically this is how a histogram is built: equal intervals but possibly different frequency (number of data points) in every bin.

[edit following OP's comments]

Data points are reported salaries for each company role (actually, it is an average value reported )

Ok so the data that you have contains a distribution of the salaries for every role.

The X axis, simply the id corresponding to each role

It's not a very good idea to transform an ordinal/categorical variable into a continuous one, in a case like this you could instead plot a bar for every role (with the name of the role on the X axis). Even more precise, you could plot a boxplot for every role, keeping the order of the roles by mean salary. This way it would be possible to visualize not only the mean (usually replaced by median in a boxplot) but how much variations there are for each role.
Another visualization idea would be to plot the full distribution of salaries with a difference color for every role. In this case X would usually be the salary, Y would be how many people are in this interval (bin) of salary, and stacked bars with different colors show the proportion by role. This would show not only the link between salary and role but also how many people are in each category.

how to bin the distribution (qcut or cut)

I suspect that this is the wrong question, because apparently you're not very familiar with these concepts and how to represent them. So I would suggest that you first play with the visualization ideas I mentioned: normally these don't require you to do any binning yourself, the library should take care of it (see examples in Python).
Knowing whether you need cut or qcut depends on what you want to do with the result. As I said above, quantiles are useful for solving questions such as "80% of the employees have a salary lower than X", whereas cut directly represents "X% of the employees have a salary between 100 and 150". Ultimately the two represent the same information, just under different perspectives. If the difference is still confusing, I'd suggest you start with regular bins with cut because quantiles are slightly more complex to interpret.


Is it feasible to use decision tree algorithms for sensor fault detection? The gist is me wanting to separate system faults from sensor faults given some dataset from a wireless sensor network using a machine learning algorithm.

For instance, if I have some temperature sensors in a given area and their corresponding readings from every sort of time interval, I would like to know whether an abnormal value is due to an actual fault, or due to a faulty sensor. Of course, it would be a given that the training set would have such entries tagged with either sensor fault or system fault.

I have thought of just using something like linear regression but I would like it to work even if the system could not be modeled like that. Decision tree seemed to me like a more appropriate algorithm for this.

Lastly, there is also some consideration for the time it takes for training and classification as I wish to see if it can be used for systems which respond really quickly to such anomalies.

Sorry if it's a bit wordy but I wasn't sure how much information I should put since this is my first time posting (I'm not even sure if this is the right stack exchange to post this). Anyway, thanks in advance for the answers!

You need to determine how to formulate your problem. I see it as having two aspects:
1. Detect an abnormal value (in temperature)
2. Determine whether abnormal value is due to sensor or system problem

The first is an anomaly detection problem and there is lots of literature on the topic, including tree-based methods on sensor data. One prominent approach is Isolation Forests, which is implemented in scikit-learn. http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html

The second is fault localization. This is an area also well described in literature, and tree-based models were used even before machine learning became a thing.
A key here is to only train on anomalous data, then this should become a relatively simple classification problem assuming your have relevant features.


Training, Testing and Validation Dataset I'm training a Unet model for tumor segmentation.
I have a dataset of 400 patients for that.
The used images are CT scans (3D images) that I divide into 2D images (for a total of 30k 2D images).
I am actually splitting the dataset into:
10% test data, 18% validation data, 72% actual training data.
I'm dividing the test and training data over patients (i.e. the patients used for testing are not the same as the one for training).
Afterwards, I shuffle the 2D images and split in training/testing dataset (i.e. the same patients can be found in training dataset and validation dataset but not same stack images).
I have two questions:

Should I split the train/validation dataset according to patients too ?
Are the division percentages in train/test/validation adapted for my problem ?

Generally numbers (percentages) do not matter.
What matters is that your Splitting (Train/test/Validation) does 2 things. Represent the real world sitatution and making sure the model can generalise given that ist evaluated on the holdout sets.
So what does that mean here exactly? You have 30k Images and 400 patients. Most likely patients(scans) will differ from each other so you should split according to patients also to make sure the model can generalise on slightly different distributions of images.
And according to percentages. You Need to make sure that Things you find in Train test and Validation represent your Problem. This can mean Splitting by Patient, Splitting by some other feature, checking the Distribution of data etc. but what it does not mean is that only cause you have 12% in one set you are sure.
What does that mean. Lets say you have 1000 rows of data. You split 90% 10% so in holdout you have 100 data Points. But in the Train set out of the 900, the majority of them are same similiar. And they differ from the 100 Points in holdout. Is this a good split? obviously not cause your model is learning Nothing.


Is there an algorithm for categorizing unlabeled samples into K classes? I am not sure if this would be considered unsupervised, or semi-supervised learning.  I am looking for an algorithm that will take N input arrays of features, and then cluster samples(not features) into a user-specified amount of classes.

For example, let's say we had data that we knew were images of circles, squares, and triangles.  In this case, I would be able to specify 3 classes, and the desired outcome would be that it would separate out the circles, squares, and triangles.

Also, if there is such algorithms, are they any good?  Thanks.

Best solution would be to use the inception pertained model in your case as it's very good with over 1000 classes. Also you can label some of the data and make a CNN in Tensforflow, and then use this data to classify other data.


Normalizing time data If I have a dataset with events occuring at certain times of day, Hour, how would I go about using this for, say, a classifier? Example:

Hour     | Event | Item
08:45:22 | Buy   | Apple
09:03:10 | Buy   | Orange
10:00:00 | Sell  | Apple

Would I convert the Hour timestamp into a numeric value, such as Unit Timestamp and then do normalization on it, like I would any other numeric value?

My answer is that I would just normalize 0 = 00:00:00, 1 = 23:59:59+1sec by counting seconds and divide by the number of seconds in a day.

I disagree with the sine/cosine transformation, it might treat 11:30 and 12:30 the same if you pick the wrong transformation.

Also, there might be patterns like 'last minute' or 'early rise' that do not correspond to the cyclic nature of the day.

It should be the responsibility of the machine learning algorithm to detect which transformation should be made, e.g. if your machine learning algorithm would be a neural network, it should/could have nodes with activation functions that may look like sine/cosine, amongst others. Those would detect and respond to such cyclic behaviors.

The normalized time data is also easy to pick up by tree based/svm algorithms. Maybe even easier than 'Hour' as category, e.g. if you have a shop that is open from 9:30 to 17:30, the hour category "9" would be ambiguous.


Localised filters for CNNs I was going through some material and the term "localized filters" was used, can someone please explain what it means and what are the advantages of having a localized filter.

EDIT-1
My question is in the context of the word used in NIPS 2017 tutorial on Geometric Deep Learning on Graphs and Manifolds at 7:23 it is said that "In many successful instances of CNN models, these filters are localized in space. And the intuition why they have to be localized can be understood from the geometric point of view, that if I apply my filters to the inputs that are slightly deformed then many of the filter responses are going to be stable." I am not able to understand the last line.
For this purpose, I have read stuff related to localization of rings and related things in Algebra but can't relate from there as well.

Attempt at a Conceptual Approach

I will attempt to answer this question focusing primarily on the concept itself, rather than the mathematics which have little explicit relevance. Therefore, correct me if I am wrong, but I believe that when you state "Localized Filter" you are referring to  what is also, commonly known as a "Convolution Layer" or "Neuron" even occasionally a "Kernel". In a manner, of which I hope will be suitable to your application, I would have one visualize this as a sort of mask, who's depth MUST equal that of the source image depth, that is to be masking the source input data, centered at some point (x,y,z), then sequentially shifted one position at a time in a manner such that, the area covered by the filter in the second position will be overlapping the first all but one column. To better understand this concept, imagine the way the data input to a CNN would be "seen" by the machine, the following images attached to this guide, show the general concept; although the picture above this paragraph is AN ACTUAL GRAPHICAL REPRESENTATION OF A NEURON, the same principle applies to both:

  Some array -> N1xN2xN3 of numbers.

After successfully shifting, the algorithm will then perform an element wise multiplication across the values in the original matrix(X0) that the "conv layer" currently Filters and its own numeric values(X1) then stored in some other array (A), referred to as the "Activation Map", that becomes some degree smaller than the original data in relation to the filter size:

  np.multiply(X0,X1)

Of which the result is stored in some Matrix, of numerals, Pt, position total. Upon completion of the sequence of the entire original input data, the algorithm
then computes the sum of all elements and stores them, so to gain the ability to pass to the next layer of the CNN:

  Ptfinal = sum(Pt)

The filter itself, being representative of some pattern, for example a circle, would then result in the outcome of a large number after completing the second step if there was another circle within its area as the pixels that have any data within them will result in an element of the original array having some positive number. As an example reference the below picture's and notice the difference in the resulting Integers for both instance's:

Where as the opposite is true, of an area's values of which do not match those of the "conv layer". This scenario will result in a very low number, relative to a confirmed match, or even a zero if the element happens to be empty of data other than zero. This method is what comprises the "secret layer" in a CNN, which generally alternates in a sequence between different variants, "Convolution", "ReLU" as well as "Pool" layers.

I hope that this explained better what is going on in the localized filters that you came upon in your research! If I was to ambiguous on any part of the explanation, please feel free to espouse and I will try to extrapolate.


Is it possible to predict using fewer number of features than the number of features that was used in training the model? I'm making a model using sklearn.svm.SVC that would predict machine performance (ErrorID).

For the training of the model, I'm using 6 features, namely, EmployeeID, JobID, MachineID, Speed, RunningDateandTime, Meters and passing in ErrorID as labels.

Now for the prediction, I only have RunningDateandTime because I want to predict the performance in the future. But the model won't accept it because the number of features at the time of training and the number of features at the time of prediction is not same.

Is there any way to force the model to predict using only 1 feature while being trained on 5 features?

Machine learning model is nothing but an equation (hypothesis), derived from the features (data set) used for model training. Using training dataset we optimize this equation by changing the coefficient(also known as weight) values with each iteration(or with each batch).

For example:

 where x1, x2, x3 are features and theta1, theta2, theta3 are weights...that we have to optimize with each iteration.

  In order to get the results from this function, we need some value for
  x1, x2,and x3.  This function wont work in case you feed just either
  of these values.

  Best we can do in your case is to feed some average values for rest of
  the missing features. But then wont expect much from the model.


Sentences language translation with neural network, with a simple layer structure (if possible sequential) Context: Many language sentences translation systems (e.g. French to English) with neural networks use a seq2seq structure:

  "the cat sat on the mat" -> [Seq2Seq model] -> "le chat etait assis sur le tapis"

Example: A ten-minute introduction to sequence-to-sequence learning in Keras, Python for NLP: Neural Machine Translation with Seq2Seq in Keras

I noticed that in all these examples the structure of the neural network is not done by using a Sequential structure with consecutive layers, but rather a more complex structure like this:

Question: are there successful attempts of doing sentence language translation with classic Sequential layers?

i.e.:

Input layer: word-tokenized sentence in english, zero padded: "the cat sat on the mat"
=> x = [2, 112, 198, 22, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]
Output layer: word-tokenized sentence in french, zero padded: "le chat etait assis sur le tapis"
=> y = [2, 182, 17, 166, 21, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]

What would you use as layers? I think we could start with:

model = Sequential()                                   # in  shape: (None, 200)
model.add(Embedding(max_words, 32, input_length=200))  # out shape: (None, 200, 32)
model.add(LSTM(100))                                   # out shape: (None, 100)
.... what here? ...

but then how to have a second Embedding for the output language and reverse it? from a 200x32 embedding (floats) to an integer list like this [2, 182, 17, 166, 21, 2, 302, 0, 0, 0, 0, 0, 0, 0, 0, ...]?

Also, how to measure the loss in this situation, mean squared error?

More generally, what is the simplest structure you can think of (even if it does not give the best results), working for language translation? (it's ok even if not sequential)

Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like attention and residual connections the performance was only comparable (not better than) statistical phrase-based machine translations [1].

I can only think of this paper as a successful attempt to use LSTMs in the encoder, decoder setting (8 encoder and 8 decoder layers) to get comparable results (there might be other attempts too). AWD-LSTMs [2] perform remarkably better than other models.

In the machine translation task, the model should understand proper relation between translated words and the words being translated and their positioning. This can only be achieved by using knowledge representation (word embeddings/encoding) from both the languages.

That's why we need to use both encoder and decoder layers.

If you ask me, I would say the following code (taken from link) is the simplest model structure possible using a simple LSTM/seq2seq model.

from keras.models import Model
from keras.layers import Input, LSTM, Dense

# Define an input sequence and process it.
encoder_inputs = Input(shape=(None, num_encoder_tokens))
encoder = LSTM(latent_dim, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
# We discard `encoder_outputs` and only keep the states.
encoder_states = [state_h, state_c]

# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None, num_decoder_tokens))
# We set up our decoder to return full output sequences,
# and to return internal states as well. We don't use the
# return states in the training model, but we will use them in inference.
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
                                     initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# Define the model that will turn
# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

From your question, it seems you only want to define the model using only Keras's sequential layer. If that is the case then you must know that variable encoder_states plays an important role in the aforementioned model's definition.

LSTMs are sequential models which means it works on a single word at a time and calculates hidden state for the next word in one iteration. The process is followed for all words in the input sequence (source language). Then the final hidden state is used in the decoder layer to compute the context for output sequence (destination language). That's why there is initial_state=encoder_states in decoder LSTM layer definition. Without encoder_states decoder LSTM won't know the context and your model will only give jibberish output.

are there successful attempts of doing sentence language translation with classic Sequential layers?

Sadly the answer is No.

You can only try to understand how machine translation works and get comfortable with the complexity of the machine translation model definition. As this is the simplest model possible.

For more information, you can go through these papers. 1 2 3

I hope it helps.


can accuracy rise while precision and recall drop? I am working on a model and running some experiments, I see that under some configurations, The accuracy rises while the recall and precision are much lower, what is the mathematical explanation? is the TN rate dropping?

The explanation is simple, assume you have the following values:

True Positives (TP) = 1
True Negatives (TN) = 998
False Positives (FP) = 1
False Negatives (FN) = 1

Accuracy = (TP + TN) / (TP + TN + FP + FN) = 999/1001 = 0.998
Precision = TP / (TP + FP) = 1/2 = 0.5
Recall = TP / (TP + FN) = 1/2 = 0.5

In summary you have an unbalanced dataset i.e. the number of samples of class is much larger than the number of the other classes. So predicting that every sample belongs to class results in a high accuracy value.


More layers in NN give worse result So I was working on a classification task with the help of a NN. The data-set was normalised, weights random between 0-1, and all the activations were sigmoid function.

Now, when I used a 2 hidden layer model the accuracy was 50%, whereas, when I used 1 hidden layer model the accuracy was 99%. Isn't this contrary to intuitive understanding about NN's. I knew more layers means better fitting even over-fitting, but apparently something different is happening in this case (maybe the values outputted by the second hidden layer is too small for the output layer to discern). So what exactly am I missing?

Maybe you are making a mistake, put your code here. But without seeing your code, these are possible points:

Vanishing problem, I don't think you this problem due to having a very shallow network. You can change your activation function to relu for avoiding that.
Covariat shift, What it means is that similar to input features which have to be normalized, the inputs of the deeper layers have to be normalized to. The normalization process is in a way that the inputs to each layer should have a special distribution that does not change. You can use batch normalisation for avoiding that problem.
Bug in coding, you may have fed the activations of each layer to the next layer in a wrong way or you may have updated the weights not simultaneously If you have not used vectorization. There are also numerous different reasons which may lead to bugs in your code.
The number of neurons may not be enough for each layer, try to increase them. For understanding the meaning of increasing the layers and the number of neurons in each layer, you can take a look at here.


how to compare different sets of time series data I am trying to do some anomaly detection between time#series using Python and sklearn (but other package suggestions are definitely welcome!).

I have a set of 10 time-series; each time-series consists of data collected from torque value of a tire (so 10 tires in total) and the sets may not contain same number of data points (set size differ). Each time-series data is pretty much just the tire_id, timestamp, and the sig_value (value from the signal, or the sensor). Sample data for one time-series looks like this:

tire_id        timestamp        sig_value
tire_1           23:06.1            12.75
tire_1           23:07.5                0
tire_1           23:09.0            -10.5

Now I have 10 of them, and 2 of them behave strangely. I understand that this is an anomaly detection problem, but most of the articles I read online are detecting anomaly points within the same time-series (aka if at some points the torque values are not normal for that tire).

To detect which 2 tires are behaving abnormally, I tried using clustering method, basically k-means clustering (since it's unsupervised).

To prepare the data to feed into the k-means clustering, for each time-series (aka for each tire), I calculated:

The top 3 sets of adjacent local maximum and local minimum with
highest amplitude (difference)
Mean of torque value
Standard Deviation of torque values

I also set the number of clusters to be only 2, so either cluster 1 or 2.

So my end result (after assigning clusters) looks like following:

        amplitude  local maxima  local minima  sig_value_std  \
tire_0     558.50        437.75       -120.75      77.538645
tire_0     532.75        433.75        -99.00      77.538645
tire_0     526.25        438.00        -88.25      77.538645
tire_1     552.50       -116.50        436.00      71.125912
tire_1     542.75        439.25       -103.50      71.125912

        sig_value_average  cluster
tire_0          12.816990        0
tire_0          12.816990        0
tire_0          12.816990        0
tire_1          11.588038        1
tire_1          11.588038        0

Now I have a question of what to do with this result ... so each tire has 3 rows of data, as I've picked the top 3 pairs of local max/min with 3 largest amplitudes, and that means each row can be assigned to a cluster, and sometimes they are assigned to different clusters for 1 tire even. Also the cluster size is normally larger than just 2.

My questions are:

How to do anomaly detection about "set of time-series" not just individual
data points?
Is my approach reasonable/logical? If it is, how can I clean up my result to get what I want? And if not, what can I do to improve?

Pretty interesting question!

First of all have a look at my edit as your question was a bit unclear according to the standard terminology. you have a set of Time-Series and you want to detect the outliers (abnormalities).

Your approach is pretty clear and logic and shows understanding of the problem and solution. The point is about the way you chose to apply it.
K-means is not the best way. I would like to point out that choice of 2 clusters are very smart here as you hope the clusters get formed based on a normal/abnormal structure. It just does not work well in practice if the features you extracted from your time-series are not inhibiting the abnormal behavior.
I assume Embedding algorithms are the right way to do so. Most probably if you apply a simple PCA you will see abnormal time-series somewhere further than others. Bellow I write the code. Try it and drop me a line if it didn't work so I go for more sophisticated solutions (e.g. you may construct a phase-space and see your data there or monitor the recurrence of time-series, etc.)

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

pca = PCA(n_components=2)
X_new = pca.fit_transform(X)
plt.figure(figsize=(10,10))
plt.plot(X_new[:,0],X_new[:,1],"*")
plt.show()

where X is a 10xN matrix in which each row is one time-series.

You may choose more components for PCA and check different PCs against each other.

Anyways the problem is not that difficult and if it did not work I will update my answer with another solution.

Hope it helps and good luck!


Choosing a set of CNNs for paper There are so many CNNs out there and I am trying to do a comparison between some of them in my paper. Which networks should I include? Resnet, VGG, and Inception are obvious, but I would like three or four others. Which ones should I choose? There are so many variations out there like Xception, resnext, and ... that I am confused. Which of these networks are most used in the academic papers?

It largely depends what kind of paper you are writing — you haven't provided too much information in that regard. You could discuss a selection of models that have scored the highest on ImageNet. Most of these networks have historical significance, and brought a new aspect to the table.


Expected Lifetime: Churn Formula vs. Experience Data I am analyzing data for a subscription based company. I.e they sell a service in exchange for monthly payment. I would like to conduct an analysis and come up with an estimate of the average lifetime (in months) of a customer. I have approximately 6 years of data including date of enrollment and date of cancellation. N is fairly large 70k, 80k, 100k, 115k, 135k, 161k enrolled clients in the 6 years, respectively.
I have seen articles such as this one that describe how to calculate average lifetime by taking $$\frac{1}{\text{churn rate}} = \frac{1}{\text{1-retention rate}}.$$
I don't entirely understand why this formula holds,but my understanding is that this is a problistic calculation. Someone on my team calculated an average lifetime of 71 months using this methodology.
Out of curiosity, I wanted to compare this to our experience, so I computed the following in R
data %>%
  filter(Cancelled == "TRUE") %>%  #--ignore active policies
  mutate(duration = 12*(as.yearmon(CancelRequestDate) - as.yearmon(EnrollDate))) %>%
  pull(duration) %>%
  mean()

> dataAvgPetMo
  24.52419

These two numbers are very different. Could anyone enlighten me as to why, and perhaps offer some guidance for how to interpret or refine this study to come up with a reasonable estimate of average lifetime.

new to this community. edits and advice welcome :)

The average lifetime is calculated simply by using the geometric distribution's mean formula. If probability of churn in any month (or simply percent churned in a month) is 0.2, then the expected life time of a customer is 1/0.2 = 5 months.
Here you are filtering the unchurned customer at the first step (filter(Cancelled == "TRUE")) and therefore you get the average lifetime of the churned customers in your sample rather than the average lifetime of any customer.


scikit-learn - Should I fit model with TF or TF-IDF? I am trying to find out the best way to fit different probabilistic models (like Latent Dirichlet Allocation, Non-negative Matrix Factorization, etc) on sklearn (Python).

Looking at the example in the sklearn documentation, I was wondering why the LDA model is fit on a TF array, while the NMF model is fit on a TF-IDF array. Is there a precise reason for this choice?

Here is the example: http://scikit-learn.org/stable/auto_examples/applications/topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-topics-extraction-with-nmf-lda-py

Also, any tips about how to find the best parameters (number of iterations, number of topics...) for fitting my models is well accepted.

Thank you in advance.

In the word sampling steps in LDA the word count is used as weights for the multinomial dist. Re-weighting the TF's by its IDF's would dispropotionally increase the chance of rare words being sampled, making them have a stronger influence in topic assignment. But rare words have equally low probabilities of showing up in all documents and topics, hence flattening out the topic distribution of each document.

That is a similar reason why it is recommended to remove stop-words before running LDA.


How do i get the loss function graph? I used Mini-batch gradient descent to train the model, but i am unable to get the proper loss graph. The loss graph is always showed as a straight line. I know there is something wrong but would anyone be able to guide me
from sklearn import metrics

error = []

for epoch in range(epochs):
    for i in range(0,x_train.shape[0],minibatch_size):
        x_mini = x_train[i:i + minibatch_size-1,:]
        y_mini = y_train[i:i + minibatch_size-1,:]

        #feed forward
        #layer 1
        in1 = x_mini@w1 + b1
        out1 = sigmoid(in1)

        #layer 2
        in2 = out1@w2+b2
        out2 = sigmoid(in2)

    if epoch > 10:
        error.append(0.5*np.power((y_mini-out2),2).mean())

z=np.arange(len(error))
plt.plot(z,error,label="train",color='red')
plt.legend(loc='best')
plt.show())

I wanted the graph for loss before training and after but cant seen get a proper one

The code you provided only performs the forward propagation of a neural network without using gradient descent to backpropagate the loss. This means that your network parameters do not change, i.e. your model is not learning anything. The model will always output the same value for an input in your training data and the loss will always stays the same.


Does data anonymization conflict with GDPR rules? There are GDPR articles which relate to a persons ownership about their data e.g. Art. 17 GDPR Right to erasure (‘right to be forgotten’) and Art. 20 GDPR Right to data portability. In case one would anonymize the data without a way to "restore" the relation between the person (name + e-mail adress) (which in turn would allow handling of the person specific data) I'd say this would conflict with these GDPR articles. Are there data anonymization techniques which allow to "restore" the relation between name + contact e-mail after the data has been anonymized? This would allow to satisfy these GDPR rules.

Formally speaking, this is clarified in the GDPR Recital 26, unofficially titled Not Applicable to Anonymous Data:

The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable.
This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes.

Informally speaking, the claim that data anonymization would violate the data subject's rights to data erasure and portability, hence we should seek to use reversible anonymization techniques, sounds awkward, and against the very spirit of GDPR; and the official interpretation regarding anonymization is very clear:

Effective data anonymization is made up of two parts:

It is irreversible.
It is done in such a way that it is impossible (or extremely impractical) to identify the data subject.

In other words:

if a subject's data have been effectively anonymized, they are no longer personal data, hence they are no longer governed by GDPR; consequently, Articles 17 & 20 are not applicable, and this does not constitute any conflict
if any personal data used as a source for the anonymized ones remain, they are subject to GDPR; data subjects can exercise their right to erasure and portability for these non-anonymized data
if the personal data used as a source for the anonymized ones are already erased (possibly in compliance to GDPR), then neither the right to erasure nor the right to portability are applicable any more, and this does not constitute any kind of conflict either.

Notice that the fact that ensuring effective anonymization is not as clear-cut as it may sound has also been legally recognized in the Opinion 05/2014 on Anonymisation Techniques of the Data Protection Working Party:

Thus, anonymisation should not be regarded as a one-off exercise and the attending risks should be reassessed regularly by data controllers.


How to transition between offline and online learning? I am training an RL agent on a time series (with TensorFlow in python) in the following way: to predict the quantity of interest at time period $t$, I feed a window of $W$ observations at time periods $[t-W,t)$. Throughout the training the window advances step by step until I have a minibatch of $M$ observations and rewards to train on. Repeat until you run out of historical data, that's one Epoch. I train on a few thousand epochs with small learning rate (the loss is very unstable).

Eventually, I want to start pulling live data from the environment to make predictions.
At this point, if I wanted to continue the training online, how should I deal with the epochs? There is no "running out of data" anymore.

Although RL algorithms can be run online, in practice this is not stable when learning off policy (as in Q-learning) and with a function approximator. To avoid this, new experience can be added to history and the agent learn from the history (called experience replay). You could think of this as a semi-online approach, since new data is immediately available to learn from, but depending on batch sizes and history size, it might not be used to alter parameters for a few time steps.

Typically in RL systems like DQN, you would train for some randomly sampled batch between each action, out of some window of historical data (maybe all historical data, maybe last N steps). The amount of training you perform between actions is a hyper-parameter of your model, as is any sampling bias towards newer data.

For example in the Atari game-playing paper by Deep Mind team, the agent sampled a mini-batch with 32 observations (state, action, reward, next state) to train the neural network on, in between each action, whilst playing the game online.

The concept of an epoch does not occur in online learning. If you are using each epoch to report performance metrics, and want to continue using comparable numbers, then you can pick some number of training iterations instead. There are no fixed rules for this, but you might want to consider reporting the same statistics on similar number of iterations as you trained on historical data - e.g. if you had 10,000 training samples in your history and are now training online with a mini-batch size of 50 per time step, then report progress every 10,000/50 = 200 time steps.


Faster Data Transfer? I'm doing a project which involves a small dataset (4GB). I'm trying to upload it to Paperspace to do some analysis but it's taking an absurd amount of time.

Using Gradient, it would upload maybe 40MB in 10min then stop uploading. This was done using the "upload" button in the jupyter notebook.
Using a P400 hourly computer I'm transferring at 800MBPS over SSH (well, through Nautilus), but it's still taking hours to transfer this small dataset.

What am I doing wrong? Is there a faster way?

Most cloud systems support the command line mv command to move files or folders.

Taking a command line approach allows for more control and visibility into tasks.


How can I find similarities in two graphs? Let's say that I have two 1 dimensional arrays, and when I plot the two arrays they look like this:

If you look at the top and bottom graphs, then you can see that the highlighted parts are very similar (in this case they're exactly the same). I need to find a way to find these sections using some sort of algorithm or method.

I've tried searching everywhere in the numpy and scikit docs, I even searched everywhere on stackexchange and couldn't find a solution for this problem. I don't think anyone published a solution for this yet.

Does anyone have any idea how I can find similar sections in two graphs? My dataset is a 1 dimensional data array for each graph, and I need a algorithm that tells me where the similar parts are. Just remember that the similar sections are never 100% the same, sometimes they're a little bit off and sometimes there's anomalies so a small part would be different but everything else will still look the same. Also you can ignore the curvature of the graph, that's irrelevant. Only the X and Y coordinates of the data points are important.

I can't read explanations that have a lot of maths inside of them and I also can't turn explanations that have a lot of maths into code, I'm still learning how to do that at University. But I'm really good at reading pseudo-code and other programming languages so please give me an answer with real code.

Well, you need to first define what your threshold for 'similar' is, and also what length of similarity is meaningful to you.

One way of achieving this is by taking a 'slice' of the first set of coordinates, and comparing them against each slice of the same size in the second set. If all values are within a certain threshold distance, bingo.

You can then repeat this with the next slice of coordinates from set #1.

e.g. here is an O(n2) implementation:

slice_len = 10
thresh = 2
overlap_x1 = []
overlap_x2 = []

for i in range(len(x1)-slice_len):
    for j in range(len(x2)-slice_len):
        # checking the y coords are all at most 'threshold' far away
        if max(abs(y1[i:i+slice_len]-y2[j:j+slice_len])) < thresh:
            # Adding the similar x-coords to the containers
            overlap_x1.append(x1[i:i+slice_len])
            overlap_x2.append(x2[i:i+slice_len])

# Converting arrays to ordered sets to remove duplicates from overlap
# Since they are x-coords, they are monotonic increasing, order is preserved
overlap_x1 = OrderedSet(overlap_x1)
overlap_x2 = OrderedSet(overlap_x2)


How gradients are flown back to Network in siamese architecture? How weights of all CNN models are same even when using different models TL;DR : Intuition behind the gradient flow in Siamese Network? How can 3 models share same weights? And if 1 model is used, hpw Gradients are updated from 3 different paths?
I am trying to build to a Siamese Network and as far as I can know, if I have to build a Triplet Loss based Siamese, I have to use 3 different networks. So for simplicity, let us say that my architecture is something like: Please correct the architecture if wrong
    I1 = Input(shape=image_shape)
    I2 = Input(shape=image_shape)
    I3 = Input(shape=image_shape)

    res_m_1 = ResNet50(include_top=False, weights='imagenet', input_tensor=I1, pooling='avg')
    res_m_2 = ResNet50(include_top=False, weights='imagenet', input_tensor=I2, pooling='avg')
    res_m_3 = ResNet50(include_top=False, weights='imagenet', input_tensor=I3, pooling='avg')

    x1 = res_m_1.output
    x2 = res_m_2.output
    x3 = res_m_3.output

    # x = Flatten()(x) or use this one if not using any pooling layer

    ##### ------- ---------------------------- --------- ########

    'NEED HELP AFTER THIS ONE; HOW TO BUILD ARCHITECTURE'

    ########### ------------------------------------ ###########

    siamese_model = Model(inputs=[I1,I2], outputs=final_output)

    siamese_model.compile(loss=some_triplet_loss,optimizer=Adam(),metrics['acc'])

    siamese_model.fit_generator(train_gen,steps_per_epoch=1000,epochs=10,validation_data=validation_data)

My Understanding and Question:: If there are 3 networks in the architecture, how come they can produce the output with the same weights? How come these networks are sharing weights?
Also, Let us suppose it is just one network (unable to assume, how, please help), then at the first epoch, it'll give outputs with default weights (if used ImageNet). But when the gradients flow back to the network, how are these updated? Because There are 3 different paths going from the same model and how the gradients will flow back to theese paths? Parallelly is not possible (I can't think how) and if sequentially, how that one either because outputs were provided sequentially but gradients can't flow back that way?

You only create ONE model for a siamese network that you pass your inputs to.
(You have created three models in the example)
So in your triplet case you would pass the three inputs seperatly to the network and compute a loss and backprop it.
The gradients are then updated just like training any neural net.


Interpreting DataFrame.where() documentation From examples outside of the documentation, I thought I understood the examples of the .where() method.  Basically, it seems to be a another way to filter a dataframe.
However, when I checked the documentation itself for an example of how to use .where(), it was counterintuitive.
The documentation provides this example:
 df = pd.DataFrame({'A': [1, 2, 3],
                    'B': [4, 5, 6],
                    'C': [7, 8, 9]})

df.where(lambda x: x > 4, lambda x: x + 10)

[output]:
    A   B  C
0  11  14  7
1  12   5  8
2  13   6  9

It seems to me that this code should filter for all values greater than four.  According to my logic,
df.where(lambda x: x > 4, lambda x: x + 10)

should add 10 to all values greater than 4, changing the output to
    A   B  C
0   1   4 17
1   2  15 18
2   3  16 19

Could someone please explain to me the error in my logic?

I assumed "where" meant "where condition is true, replace with X", but it's actually "Where this condition is true, KEEP the value".  This is odd to me but I found a different documentation page that more clearly explained the .where() method.

As it turns out, the .mask() method would return what I expected from .where().  Specifically:

df.mask(lambda x: x > 4, lambda x: x + 10)


How to get pixel location in after rotating the image? I'm trying to rotate some images with some boundary boxes, but I couldn't get the new bb. So if I have an image of 100x70 and I have a pixel at (19,39) and then I rotate the image with angle = 45,
how can I calculate the new position of this pixel?

You first need to define where you are rotating your image around.
If you are rotating it around (0,0) (i.e. your image is located in the positive x and y quadrant), you can apply a rotation matrix in its vanilla form:
$\left[\begin{array}{c}x' \\y' \\1\end{array}\right] = $
$\begin{bmatrix}\cos \theta &\sin \theta &0\\-\sin \theta & \cos \theta &0 \\0 &0 &1\end{bmatrix}$
$\left[\begin{array}{c}x \\y \\1\end{array}\right] = $
$\left[\begin{array}{c}x\cos\theta + y\sin\theta \\-x\sin\theta+y\cos\theta \\1\end{array}\right]$
where x and y is your starting pixel location, x' and y' is your new pixel location, and $\theta$ is the counterclockwise angle to rotate (you can ignore the 3rd row for now). Notice that because you are using sine and cosine functions, the new pixel location is unlikely to be an integer, so you will need to do some rounding.
However, if you are rotating around the center of the image (which is the more common use-case), you will need to first translate the "origin" of your image to the center, apply your rotation, then move your image back.
This alters the calculations to this:
$\left[\begin{array}{c}x' \\y' \\1\end{array}\right]$
$=\begin{bmatrix}1 &0 &0\\0 &1 &0 \\X/2 &Y/2 &1\end{bmatrix}$
$\begin{bmatrix}\cos \theta &\sin \theta &0\\-\sin \theta & \cos \theta &0 \\0 &0 &1\end{bmatrix}$
$\begin{bmatrix}1 &0 &0\\0 &1 &0 \\-X/2 &-Y/2 &1\end{bmatrix}$
$\left[\begin{array}{c}x \\y \\1\end{array}\right]$
$=\left[\begin{array}{c}(x-X/2)\cos\theta + (y-Y/2)\sin\theta + X/2 \\-(x-X/2)\sin\theta+(y-Y/2)\cos\theta + Y/2 \\1\end{array}\right]$
where capital X and Y are the size of your image in the x and y directions.

You can read more about image transformations on the mathworks page here.


Uneven dataset, how I should train the model if I can't load all pictures at once I have a dataset of multiple classes (About 50). The dataset doesn't have the same number of pictures per class, some have 300, some have 1000, and some more, and I've seen that this ruined my accuracy on my model

First of all, the data is too big for me to store in the RAM, so I have to split it into parts (I take 300 pictures of each class at once), train the network on that data then repeat the process until I have no more photos left.

The question I have is, is it a problem if I do this (I guess it is from my results)? I have for example from 100 classes of pictures, only 10 left and I train the model only on those 10,then on only 5,then on 3,etc.? Because I did that and after I went over the smallest classes, the overall accuracy went up, but those smaller classes accuracy went down, and in the end from 80% in the beginning for each class I got 0-2% on 48 classes, and 99% on 2 classes.

How can I solve this 'unevenness' (I don't know the word) then so I won't have this problem anymore?

From your tags, I see that you use keras. Keras offers you the class ImageDataGenerator which has the method flow_from_directory() (see here). This method loads the images in your training directory batch by batch from the hard drive and stores only the current batch in the RAM. This eliminates the bottleneck you currently face when loading the images.

To tackle the class imbalance, the recommended method is to use the class_weight argument of the keras classifiers. This argument assigns a weight to every class in your data, allowing you to give higher importance to images from the minority classes. This answer shows how you can calculate the class weights.

In the code below, I put that all together:

# Define constants - change them according to your requirements
BATCH_SIZE = 128
EPOCHS = 50
IMAGE_SIZE = 224

# Set up Image Data Generator
train_datagen = ImageDataGenerator(dtype=np.float16) # here you can also do some data augmentation

# Set up flow from directory
train_generator = train_datagen.flow_from_directory(directory="path/to/your/directory",
                                                    class_mode="categorical",
                                                    target_size=(IMAGE_SIZE, IMAGE_SIZE),  # resize the images if required
                                                    batch_size=BATCH_SIZE)

# Calculate class weights
counter = Counter(train_generator.classes)
max_val = float(max(counter.values()))
class_weights = {class_id: max_val/num_images for class_id, num_images in counter.items()}

#
# Here you set up your model ...
#

# After compiling the model, you fit it to your data using fit_generator
model.fit_generator(train_generator,
                    steps_per_epoch=train_generator.n // BATCH_SIZE,
                    epochs=EPOCHS,
                    class_weight=class_weights,  # use the class_weights as method parameter
                    verbose=1)


What is the difference between explainable and interpretable machine learning? O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes.

Christoph Molnar says interpretable ML refers to the degree to which a human can understand the cause of a decision (of a model). He then uses interpretable ML and explainable ML interchangably.

Wikipedia says on the topic "Explainable artificial intelligence" that it refers to AI methods and techniques such that the results of the solution can be understood by human experts. It contrasts with the concept of the black box in machine learning where even their designers cannot explain why the AI arrived at a specific decision. The technical challenge of explaining AI decisions is sometimes known as the interpretability problem.

Doshi-Velez and Kim say that that interpretable machine learning systems provide explanation for their outputs.

Obviously, there are a lot of definitions but they do not totally agree. Ultimatively, what should be explained: The results of the model, the model itself or how the model makes decissions? And what is the difference between interpret and explain?

I found this article by Cynthia Rudin which goes a bit more into the difference between the two terms that is in line with your source from O'Rourke.
At the core it is about the time and mechanism of the explanation:
A priori (Interpretable) vs. A posterio (Explainable)
I found this quote to be very helpful and inline with my own thoughts (emphasis mine):

Rather than trying to create models that are inherently interpretable, there has been a recent explosion of work on ‘explainable ML’, where a second (post hoc) model is created to explain the first black box model. This is problematic. Explanations are often not reliable, and can be misleading, as we discuss below. If we instead use models that are inherently interpretable, they provide their own explanations, which are faithful to what the model actually computes.

In short an interpretable model is able to output humanly understandable summaries of its calculation that allow us understand how it came to specific conclusions. Due to that a human would be able to actually create a specific desired outcome by selecting specific inputs.
A "merely" explainable model however does not deliver this input and we need a second model or mode of inspection to create a "Hypothesis about its mechanism" that will help explain the results but not allows to rebuild results by hand deterministically.


What open-source books (or other materials) provide a relatively thorough overview of data science? As a researcher and instructor, I'm looking for open-source books (or similar materials) that provide a relatively thorough overview of data science from an applied perspective. To be clear, I'm especially interested in a thorough overview that provides material suitable for a college-level course, not particular pieces or papers.

One book that's freely available is "The Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman (published by Springer): see Tibshirani's website.

Another fantastic source, although it isn't a book, is Andrew Ng's Machine Learning course on Coursera. This has a much more applied-focus than the above book, and Prof. Ng does a great job of explaining the thinking behind several different machine learning algorithms/situations.


How to extract features from long chemical names? I have an interesting problem that I am uncertain about how to even get started. I am working on a binary classifier that will take a chemical name, encoded as a string, and predict whether it is a 'good' or 'bad' name. I have had quite good success doing this by examining the structure of the chemical directly, but I would like to explore whether I can learn anything about the given name of the chemical itself (as the name can contain some structural information about the molecule that my encoding of the molecular structure is missing). I have been searching around trying to find anything built into sklearn to do text feature extraction. There is quite a bit, but it mostly seems to me like they are used for encoding whole sentences or paragraphs. My input would be very long, single words such as:

1-(aminoiminomethyl)-N'-[2,3,6-tri-O-benzoyl-4-O-(2,3,4,6-tetra-O-benzoyl-α-D-glucopyranosyl)-β-D-glucopyranosyl]-
2,4,5-trideoxy-2-[(16-mercapto-1-oxohexadecyl)amino]-1,3-O-(1-methylethylidene)-6-O-undecyl-
octahydro-7-hydroxy-1-[[2-O-(4-hydroxybenzoyl)-α-D-allopyranosyl]oxy]-7-methyl-

And as such, I'm not certain a bag of words or one-hot encoding of the strings would work. Could anyone perhaps point me in the right direction on methodologies or algorithms that might possibly be able to extract features from these strings such that I could train a binary classifier on them?

You can train an RNN with character embeddings. This can be done by splitting the name into sequences of chars and vectorize them numerically. If you are working with Keras, you can feed them into an Embedding() layer that will learn how to represent characters. RNN layers will then process their sequence. At the output node, your Network will perform a classification ('good'/'bad').


Why is it important to have sufficient number of instances in your dataset for each stratum? As per the figure 1, most of the median-income values are clustered around \$20,000-\$50,000, but some median incomes go far beyond \$60,000.

I didn't understand the explanation behind why housing['median_income'] has to be divide by 1.5

 housing['income_cat'] = np.ceil(housing['median_income'] / 1.5)

  Explanation - It is important to have a sufficient number of instances in your dataset for each stratum, or else the estimate of the stratum's importance may be biased.

Can someone help me to understand the explanation and also why is it only 1.5 ?. As per the explanation, why the estimate of stratum's importance will be biased when there is no adequate instances for each category ?

Lets take an example that you are trying to evaluate average income of a company. Assume company has 100 employees in total and they belong to two stratum (70 management and 30 technical executives) and management executives on average are paid higher compared to technical executives.

You are the surveyor:
Due to resource constraints, you have access to survey of any 20 individuals chosen at random.

Case 1: you did a random sampling from 100 employees and end up with 10 management and 10 technical executives. you will get an estimate for average income that is biased towards technical executives. In other words, you are giving more importance/ representation to income of technical executives than
actual representation in the population.

Similarly, in other attempt:
Case 2: you did a random sampling from 100 employees and end up with 17 management and 3 technical executives. you will get an estimate for average income that is biased towards management executives.

Both the above sampled cases have bias and are not representative of the population. Therefore, when we have the stratum identified before the sampling, stratified sampling should be done. For eg. random sample for 14 out of 70 management executives and  6 out of 30 technical executives.

In your given example of the housing and income data, income scale has been shrunk by a factor of 1.5, so as to create less no of income categories for the same bin width. Any other factor > 1 could also be used to ensure each stratum has considerable number of instances.