how to get the number of columns that have the same value between 2 rows I want to compare two records and get the number of columns that have the same value , for example
I have two rows, the first row is :

rank
rank1
rank2

1
1
3

and the second row is

rank
rank1
rank2

1
1
2

the result should be 2 columns have the same value. is there a function in python that is capable of performing something like this?

If your records are in form of a pandas dataframe, then you can use df.columns.duplicated() to know the columns which are duplicated.


Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data? I'm training a deep network (CNN-LSTM-CRF) for Named Entity Recognition. Is there a reason that increasing the number of parameters would make the network less able to overfit a small training set (~20 sentences), or does this indicate a serious bug in the code?

It is not necessarily a red flag. Of course, without seeing the code, it is impossible to say that for sure.

What is likely happening here is that adding parameters to your model, it makes it harder for it to converge to some minimum. More parameters roughly mean that your model is able to explain "more complex stuff". And since you have a small amount of data, the explanation should remain rather simple. Therefore, your model is trying to explain something simple in a complicated way, and it might not be easy to do so.

Also, are you using dropout or regularization? If yes, this might also be an issue as these are explicitly use to avoid overfitting.


How to arrange the image dataset in CNN? How do I arrange the image dataset in CNN? Should I put each image category in a separate folder? Or all of them in the same folder? Should the image name be the category name?

I would like to see an example for an image dataset (other than MNIST).

Thank you.

Directory structure like in dogscats/.(atleast I kept it this way)

dogscats

    |-- train

          |-- cats

                |-- catpic0, catpic1, …

          |-- dogs/

                |-- dogpic0, dogpic1, …

    |-- valid

          |-- cats

                |-- catpic0+x, catpic1+x, …

          |-- dogs

                |-- dogpic0+x, dogpic1+x, …

    |-- test

           |-- catpic0+x+y, catpic1+x+y, dogpic0+x+y, dogpic1+x+y

Becareful With the naming of the files also..

Also note that you will need a mapping of image names and classes as well like in a CSV or something...

Also there is no globally accepted directory structure, it completely depends on the API you will use...

What is a good train/validation/test split?(depends on your dataset size)

can do $80/20$ (train/validation)

if you have or are creating a 'test' split, use for (train/validation/test):

can do $80/15/5$
can do $70/20/10$
can do $60/20/20$

Remembering that sole aim is to generalize eventually on test sets...


Classify samples based on other sample probabilities I was wondering if there's a way to train a classifier or set up a way of classifying after that can classify certain samples as some relationship between the previous two.

I notice that, for example, when I use the predict_proba from scikit-learn's RandomForest, I can see the probability that a class was predicted like this: [0.3, 0.43, 0.27].  I want to do something like: if | p(class1) - p(class2) | < ε then class 3 should be boosted. Maybe this means using a binary classifier at first and then checking the relationship between the binary classes.

To describe the situation further, class 1 and class 2 are distinct and class 3 has a mixture of both features.

This is a case of multilabel/multioutput classification.  You have a corpus of data in which several classes can be true for a single sample.  Moreover - where one class is literally a mixture of the other two classes.  A much more common problem than a lot of us would wish it to be.

Note: I'll rename classes 1, 2 and three into classes 0, 1 and 2 respectively; since that is how sklearn enumerates them.

The sklearn's RandomForest supports multilabel classification out of the box, therefore instead of organizing your data as follows:

X                   | y
feature1  feature2  | label
--------------------+------
0.1       0.3       | 0
0.2       0.1       | 1
0.7       0.5       | 1
0.8       0.3       | 1
0.6       0.6       | 1 (but also 0 - so probably should be 0 and 1 - class 2?)
0.3       0.9       | 0
0.5       0.5       | 0 (but also 1 - so probably should be both as well- class 2?)

Organize the data in the following way:

X                   | Y
feature1  feature2  | class0?  class1?
--------------------+-----------------
0.1       0.3       | 1        0
0.2       0.1       | 0        1
0.7       0.5       | 0        1
0.8       0.3       | 0        1
0.6       0.6       | 1        1
0.3       0.9       | 1        0
0.5       0.5       | 1        1

In other words, make your label vector into a matrix - i.e. both $X$ and $Y$ will have two dimensions now.  sklearn's RandomForest will accept that inside it's fit() and inside it's predict() methods (and inside predict_proba() as well).

The only tricky bit may be the interpretation of the output of predict_proba() in multilabel/multioutput classification, for example (watch for typos, I'm doing this code from memory):

import numpy as np
from sklearn.ensemble import RandomForestClassifier

X = np.random.random((3, 3))
Y = np.array([[0, 1],
              [1, 0],
              [1, 1]])
model = RandomForestClassifier()
model.fit(X, Y)
model.predict(X)

    np.array([[0., 1.],
              [1., 0.],
              [1., 1.]])

model.predict_proba(X)

    [np.array([[0.6, 0.4],
               [0.7, 0.3],
               [0.1, 0.9]]),
     np.array([[0.9, 0.1],
               [1.,  0. ],
               [0.2, 0.8]])]

In summary, predict_proba did return a list of two elements: the first element is the probability of class 0 independently of class 1, whilst the second element in the list is the probability of class 1 independently of class 0.  Whether the is a high probability of class 0 and a high probability of class 1 then you have a prediction of [1, 1].


RandomForest surprisingly high accuracy I've been experimenting with Random Forests on Python after trying Naive Bayes which gave me lower accuracy than I expected, 62%. My csv file has around 14,000 records, I use 80% for the training set and 20% for testing set. I tried with different parameters like 100 trees, 500 and 1000, -1 for n_jobs and so on but during all those tests the accuracy never changed too much, it was always around 74% or 75%, almost 76% at times. I checked that the online tutorial which teaches how to implement got the accuracy of 73%, as my tests got higher than that, I wasn't worried, thought it's normal for this kind of algorithm.

However, my surprise came during my latest test, the only difference is I saved the model and vectorizer to file so I don't have to train the model each time. The new script loaded those files, applied the loaded (trained) model to the testing set and wow, I got the highest accuracy I've seen so far, 93.989%, almost 94%.

Anyone got a similar experience? Is this big increase related to having saved the model to a file and loaded it? The rest of the code is all the same. Or was I just too lucky and if I try more times I will go back to the usual around 75% accuracy?

You can be sure that saving the model to disk won't make the model have higher accuracy when loaded again. Saving the models to disk would only cause worse performance if parameters are truncated, for example.

To establish a fair and rigorous comparison method, you have to keep track of your training/test splitting result. In that sense, you can:

Make your code reproducible, using random seeds conveniently
Before saving the model, you can save your splitting results by either saving the data or returning the indices for the training set and the test set. For that purpose you have to store the data-set in a list (for example, a list with the file names…)


Removing correlation between independent variables If there are two continuous independent variables that show a high amount of correlation between them, can we remove this correlation by multiplying or dividing the values of one of the variables with random factors (E.g., multiplying the first value with 2, the second value with 3, etc.). We would be keeping a  copy of the original values of the independent variable whose values have been transformed for proper comparison of the original values with the prediction results received. Can this be considered as an alternative to dropping one of the highly correlated variables?

can we remove this correlation by multiplying or dividing the values
of one of the variables with random factors (E.g., multiplying the
first value with 2, the second value with 3, etc.)

No you can not decorrelate 2 variables by multiplying or dividing them by a random factors, you can use one of the following methods instead if you want to keep the 2 variables:

Divide a and b: new_a = a / b
Powering up a: new_a = a^2, a^3, ...
PCA
...


Clustering based on partial information? I'm open to suggestions on how to improve the title.

My problem is this, but I think it's a more general problem. In my context, I have a lot of data which has location data (Lat/Lon) as well as various characteristics of my objects.  I'm interested in clustering based on location in such a way that it controls for effects based on weather and other local effects.

So what I want is a clustering which, after trained, determines the cluster of a new object solely based on its GPS coordinates. However, I want the clusters not to just cluster based on locational clusters (there are lots of objects in this ball in Florida, this ball in California, etc.) but to make clusters so that within the clusters, objects are more similar than you would expect.

So for example, k-means would not accomplish this; it would just identify pockets of objects based on location clustering, or it would require more information to determine the cluster of a new object (neither are desirable).

Another way would be to run k-means a bunch of times, rate the results based on comparing homogeneity within clusters, and pick the best, but this seems both computationally annoying and unlikely to get good results (it would tend to identify circular clusters, rather than e.g. picking out things within 10 miles of a coast, or whatever turns out to be useful).

I feel like someone has thought of this before and come up with something.  What sorts of algorithms (classical or not) might help with this?

I don't mind re-implementing something on my own, and I don't mind stacking/etc. classifiers.  My dataset is fairly small for now, so even if there isn't a fast algorithm for this, it should be okay (even $O(n^2 \log(n))$ algorithms are doable, I've found).

You may need to treat this as an optimization problem.

Without a formal notion of what should be a cluster, you will not be getting far. A blackbox algorithm will not solve your problem, you need to open the box (understand the algorithm) and adjust it as necessary for your problem.

Once you have this notion, you can try to match it to algorithms. For example, if you can quantify similarity (e.g. if distance times attribute dissimilarity matches your desires) then you can try HAC. Or if you have thresholds (distance at most 100mi, attribute difference at most y) then you can run DBSCAN and merge neigh ors transitively into clusters.

But don't expect anything automatic.


Finding both count and average of a column in R data.table, after group by I have a data table of 5 columns. I want to do a group by on one column  and want to get the count of another column, id.

I do this in this way:  dt[,length(id),by=dt$cid].

If I also want to get sum of id, how do I proceed?
I tried this:

dt[,(sum(id) length(id)),by=dt$cid]

but it did not work. Could you please help out.

Thanks

If you want to get but the sum and length of id with data.table, you can do it like this:

dt[,.(total=sum(id), n=length(id)), by=cid]

You could even write this in shorter format with the .N operator of data.table:

dt[,.(total=sum(id), .N), by=cid]


Merging two different models in Keras I am trying to merge two Keras models into a single model and I am unable to accomplish this.

For example in the attached Figure, I would like to fetch the middle layer $A2$ of dimension 8, and use this as input to the layer $B1$ (of dimension 8 again) in Model $B$ and then combine both Model $A$ and Model $B$ as a single model.

I am using the functional module to create Model $A$ and Model $B$ independently.  How can I accomplish this task?

Note: $A1$ is the input layer to model $A$ and $B1$ is the input layer to model $B$.

I figured out the answer to my question and here is the code that builds on the above answer.

from keras.layers import Input, Dense
from keras.models import Model
from keras.utils import plot_model

A1 = Input(shape=(30,),name='A1')
A2 = Dense(8, activation='relu',name='A2')(A1)
A3 = Dense(30, activation='relu',name='A3')(A2)

B2 = Dense(40, activation='relu',name='B2')(A2)
B3 = Dense(30, activation='relu',name='B3')(B2)

merged = Model(inputs=[A1],outputs=[A3,B3])
plot_model(merged,to_file='demo.png',show_shapes=True)

and here is the output structure that I wanted:


Does Tensorflow has a keras module? everyone. I am a Geophysicist who has recently started learning Data Science. As part of this process I am going over some example source code included in the book "Machine Learning for the Oil and Gas Industry". The problem I have is that the source code has the following lines of code:
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential, load_model
from keras.wrappers.scikit_learn import KerasRegressor, KerasClassifier

The first two lines seem to run correctly, however a ModuleNotFoundError is raised for the third line. I use Anaconda individual distribution and installed Tensorflow using Anaconda Navigator in my base environment.
Is there any way to get this to run properly?

You can replace the third line with the following import
from tensorflow.keras.wrappers.scikit_learn import KerasRegressor, KerasClassifier
Hope this helps.


Unable to resolve Type error using Tokenizer.tokenize from NLTK I want to tokenize text data and am unable to proceed due to a type error, am unable to know how to proceed to rectify the error, To give some context - all the columns - Resolution code','Resolution Note','Description','Shortdescription' are text data in English- here is the code that I have written :

#Removal of Stop words:

    from nltk.tokenize import sent_tokenize, word_tokenize
    from nltk.corpus import stopwords
    from nltk.tokenize import RegexpTokenizer

    tokenizer = RegexpTokenizer(r'\w+')
    stop_words = set(stopwords.words('english'))

    tokenizer = RegexpTokenizer(r'\w+')
    dfclean_imp_netc=pd.DataFrame()
    for column in ['Resolution code','Resolution Note','Description','Shortdescription']:
        dfimpnetc[column] = dfimpnetc[column].apply(tokenizer.tokenize)
    for column in ['Resolution code','Resolution Note','Description','Short description']:
        dfclean_imp_netc[column] = dfimpnetc[column].apply(lambda vec: [word for word in vec if word not in stop_words])
    dfimpnetc['Resolution Note'] = dfclean_imp_netc['Resolution Note']
    dfimpnetc['Description'] = dfclean_imp_netc['Description']
    dfimpnetc['Short description'] = dfclean_imp_netc['Short description']
    dfimpnetc['Resolution code'] = dfclean_imp_netc['Resolution code']

My error output is attached below:

w_tokenizer = nltk.tokenize.WhitespaceTokenizer()

dfimpnetc[column] = dfimpnetc[column].apply(lambda x: [lemmatizer.lemmatize(w) for w in w_tokenizer.tokenize(x)])

Try the above code.

Mark as correct if this helps ;)


How to treat missing data for survival analysis I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, after two weeks and so on till after 3 months. Now some patients dropout in between the treatment stage. The dataset I have consists of around 50 columns(10 questions repeated over 5 times during the course of treatment), but there are missing data for some patients as they dropout from the treatment.

My questions are:

how do I handle the missing data as it is not filled by the patient?

Should I impute that with mean values or is there any other way?

P.S.: I am new to survival analysis. So any help will be appreciated. Thanks in advance.

id  age sex dropout s1_q1 s1_q2 s1_q3 s1_q4 s1_q5.... s5_q10
217 50  m   0       2     3     3     3     2         3
202 58  f   0       4     9     10    10    10        N/A
222 72  m   1       3     8     9     10    9         N/A
207 50  m   0       2     7     6     7     7         6
277 55  f   0       2     4     5     5     5         6
281 62  m   0       4     10    10    10    10        10

There are several different approaches here.

One (which you've already described) could be to impute missing values with their mean. You could then add an extra column which keeps track of whether that value was originally missing or not. So, in the example you provide, we would end up with

id  age sex dropout s1_q1 s1_q2 s1_q3 s1_q4 s1_q5.... s5_q10 s5_q10_missing
217 50  m   0       2     3     3     3     2         3      0
202 58  f   0       4     9     10    10    10        6.25   1
222 72  m   1       3     8     9     10    9         6.25   1
207 50  m   0       2     7     6     7     7         6      0
277 55  f   0       2     4     5     5     5         6      0
281 62  m   0       4     10    10    10    10        10     0

Another could be to get rid of rows that have missing values, if you have enough data (which in your case you probably don't).

A slightly more involved approach would be to predict the missing values using the non-missing data. So you could build a predictive model to predict s5_q10, train it on the non-missing data, and then use that model to predict s5_q10 for the rows where it's missing.


How do Data Scientists integrate Predictive Modeling with SQL? I've heard a lot about the importance of knowing SQL but how do you guys at the larger firms integrate your software for modeling/analysis with the server that holds the data?

Do you simply access some SQL Workbench and download a .csv file then read the file in your software? Or do you gain access to the server directly through your software?

This question is coming from the guy who learned SQL but never learned how to synchronize that with R.

There is no need to download data in CSV format, in most cases it is actually bad practice. Consider cases where data size > 1GB and is updated daily. This would add a considerable overhead and is not easy to automate.

Instead you can take a look at various R packages that exist to fetch data from your SQL database. Using RODBC, SQLite, sqldf or other available libraries you can import data to R, run queries, create tables, update tables and pretty much anything else you will need.


What are some good loss functions used to minimize extreme errors in regression and time series forecasting? E.g. In detriment of a smaller mean error, I want to have fewer big mistakes

I'm working on a time series forecasting task and in some specific cases I don't need perfect accuracy, but the network cannot by any mean miss by a lot.

Any suggestions of loss functions or other methods to solve this issue?

As you increase the harshness of big misses, you make the model less willing to miss big.

For instance, absolute loss considers missing by $2$ to be twice as bad as missing by $1$, but square loss considers missing by $2$ to be four times as bad as missing by $1$!

Imagine if you use cubic loss:

$$L_3(y,\hat{y}) = \sqrt[3]{\sum\big\vert y_i - \hat{y}_i \big\vert^3}$$

Imagine if you use quintic loss:

$$L_5(y,\hat{y}) = \sqrt[5]{\sum\big\vert y_i - \hat{y}_i \big\vert^5}$$

This gets at the general form of $L_p$ loss, which I'm guessing you can guess before reading what I type.

$$L_p(y,\hat{y}) = \sqrt[p]{\sum\big\vert y_i - \hat{y}_i \big\vert^p}$$

This relates to the idea of $L^p$ norms.

At the extreme of $L^p$ norms, you can set $p=\infty$, which is equivalent to just looking at the maximum. If you use $L_{\infty}$ loss, you minimize the greatest error.

I've wanted to play around with $L_{\infty}$ loss for a while and would be very curious what happens if you choose this loss function. I do, however, think this would be a very extreme loss function.

(Remember that $RMSE$, $MSE$, and $SSE$ all have the same $argmin$; by similar logic we are free to take the $p^{th}$ roots to get the cubic and qunitic loss functions I gave.)


Which of the NIPS 2014 papers are most significant, and why? As a newcomer to the field, I find many of the NIPS 2014 papers fascinating, but it is difficult for me to evaluate which ones represent real progress over current approaches.

Which papers do you think are most significant and are likely to have a major impact?

The awards went to "Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)" by Anshumali Shrivastava and Ping Li and "A* Sampling" by Chris J. Maddison, Daniel Tarlow, and Tom Minka.
Further Quora discussion can be found here.


What is the distribution of the goal feature in this dataset https://www.kaggle.com/kemical/kickstarter-projects
I was checking the distribution of the "goal" feature in this 2016 KickStarter data set. I figure goal would be more normally distributed but it wasn't. I checked the Kick Starter website for published statistics. It seems the sampling was random as certain categories  appear at the frequency Kick Starter Published but the data didn't meet the required assumptions for a T test. I check the Kaggle site for a summary of the goal data and the distrubtion looks the same but I'm very skeptical. I made a qq plot to check normality but this really can't be right.
What type of distribution is the column 'goal' in this data set? If the distribution should look like this why? Maybe I broke something or put in the wrong code and I wanted to be sure. My gut is telling me to take a serious look at the goal column.

It looks like it's probably pareto-distributed- meaning that most goals are small, some are medium, and a few are REALLY big. It's fat-tailed and not normally distributed at all.
A nice way to see it is to do a histogram plot with log=True and bump up the number of bins.


Best way to handle padding in time series data such as text I have a bunch of documents containing sequential data that I want to
use to train a neural network with. It is as a collection of letters each about a 2-3000
characters long. My task is, given an excerpt of such a letter I want
my network to output plausible text in the same style as the letter.

What would be the best way of presenting my data to the neural
network?

One method which I've seen in tutorials is to concatenate all letters
into one long text sequence and letting the network train on that. But
then some training sequences will contain text from multiple
letters. Suppose a letter is 500 characters long and the training
sequence length is 150 then the last training sequence in that letter
will contain 50 characters from that letter and 100 characters from
the subsequent one. I suspect that would confuse the network.

Another idea would be to insert an end of letter token between each
letter and then concatenate them. But again some training sequences
would contain the end of letter token and training on them would
perhaps confuse the network too.

A third idea would be to insert a very long sequence of padding tokens
between each letter so that no real training sequence contains text
from two different letters. But that seems very wasteful. For example,
if my training sequences are 256 characters long I would have to
insert 256 padding tokens between each letter.

Time series Analysis refers to the type of problems where we have to analyse an outcome based on time dependent inputs. Time series data is basically a sequence of data, hence time series problems are often referred to as sequence problems. Recurrent Neural Networks (RNN) have been proven to efficiently solve sequence problems. Particularly, Long Short Term Memory Network (LSTM), which is a variation of RNN, is currently being used in a variety of domains to solve sequence problems.

If we are trying to obtain more features, it is a good idea to utilise the idea of distributed representation of words, in which each input is represented by many features and each feature is involved in many possible inputs. In distributional semantics, word embedding models are used to map from the one-hot vector space to a continuous vector space in a much lower dimension than conventional bag-of-words model. we can learn the dependencies between characters and the conditional probabilities of characters in sequences so that we can in turn generate wholly new and original sequences of characters.

In such a perspective, Word2Vec word embedding model would be a good choice. It could be a good idea to convert the words into vectors using word embedding models. Then the word sequences in sentences can be feed as input to the LSTM to learn the long distance contextual dependency among the words.

Typically, sequence problems can be broadly classified into one-to-one, many-to-one, one-to-many, and many-to-many types. As per the descriptions, it seems the requirement is to analyse many sequences of textual letters as different many textual sequences itself. In these sequence to sequence analysis problems, you can either choose a single feature or with multiple features.

Long Short Term Memory (LSTM) neural networks are generally suitable for sequence prediction and sequence classification problems. If we are using LSTM for this problem, we need to model the text sequence accordingly. First we need to vectorise the text by turning each text into a sequence of integers or into a vector. It is important to limit the dataset to a top word limit. Also we need to have a maximum sequence length for each text.

Then we can tokenise the text applying this model. We can truncate and pad the input sequence such that they are all same length for modelling. The sequence will not be of same length in terms of content, but same length vectors need to be constructed.


Should I remove outliers if accuracy and Cross-Validation Score drop after removing them? I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I found many outliers. So I tried to delete them from the dataset.

The accuracy and Cross-Validation dropped by approximately 5%. I had 80% accuracy and an Cross-Val-Score of 0.8

After removing the outliers from the 3 most important_features (RF's feature_importance) the accuracy and Cross-Val-Score dropped to 76% and 77% respectively.

Here is a part of the description of my dataset:

Here is an overview of my data:

Here are the boxplots before removing the outliers:

Here are the feature importances before removing outliers:

Here is the accuracy and Cross-Val-Score:

Accuracy score:  0.808388941849
Average Cross-Val-Score:  0.80710845698

Here is how I removed the outliers:

clean_model = basic_df.copy()
print('Clean model shape (before clearing out outliers): ', clean_model.shape)

# Drop 'num_likes' outliers
clean_model.drop(clean_model[clean_model.num_likes > (1938 + (1.5* (1938-125)))].index, inplace=True)
print('Clean model shape (after clearing out "num_likes" outliers): ', clean_model.shape)

# Drop 'num_shares' outliers
clean_model.drop(clean_model[clean_model.num_shares > (102 + (1.5* (102-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_shares" outliers): ', clean_model.shape)

# Drop 'num_comments' outliers
clean_model.drop(clean_model[clean_model.num_comments > (54 + (1.5* (54-6)))].index, inplace=True)
print('Clean model shape (after clearing out "num_comments" outliers): ', clean_model.shape)

Here are the shapes after removing the outliers:

Clean model shape (before clearing out outliers):  (6992, 20)
Clean model shape (after clearing out "num_likes" outliers):  (6282, 20)
Clean model shape (after clearing out "num_shares" outliers):  (6024, 20)
Clean model shape (after clearing out "num_comments" outliers):  (5744, 20)

Here are the boxplots after removing the outliers (still have outliers somehow.. If I delete these too, I will have really few datapoints):

Here is the accuracy and Cross-Val-Score after removing the outliers and using same model:

Accuracy score:  0.767981438515
Average Cross-Val-Score:  0.779092230906

How come is removing the outliers drops the accuracy and F1-score? Should I just leave them in the dataset? Or remove the outliers that are to see in the 2nd boxplot (after removing the 1st outliers as shown above)?

Here is my model:

model= RandomForestClassifier(n_estimators=120, criterion='entropy',
                              max_depth=7, min_samples_split=2,
                              #max_depth=None, min_samples_split=2,
                              min_samples_leaf=1, min_weight_fraction_leaf=0.0,
                              max_features=8, max_leaf_nodes=None,
                              min_impurity_decrease=0.0, min_impurity_split=None,
                              bootstrap=True, oob_score=False, n_jobs=1,
                              verbose=0, warm_start=False,
                              class_weight=None,
                              random_state=23)
model.fit(x_train, y_train)
print('Accuracy score: ', model.score(x_test,y_test))
print('Average Cross-Validation-Score: ', np.mean(cross_val_score(model, x_train, y_train, cv=5))) # 5-Fold Cross validation

As a rule of thumb, removing outliers without a good reason to remove outliers rarely does anyone any good.  Without a deep and vested understanding of what the possible ranges exist within each feature, then removing outliers becomes tricky.  Often times, I see students/new hires plot box-plots or check mean and standard deviation to determine an outlier and if it's outside the whiskers, they remove the data points.  However, there exist a myriad of distributions in the world that if you did that, you would be removing perfectly valid data points.

In your example, it looks like your dealing with social media data.  If I were to sample 1000 users from a social media database and then plotted a box-plot to find "outliers" for number of likes a post gets, I can imagine that there could be a few so-called outliers.  For example, I expect my Facebook post to get a handful of likes on any given day, but when my daughter was born, my post related to that got into the hundreds.  That's an individual outlier.  Also, within my sample of 1000 users, let say I managed to get user Justin Bieber and simply like at his average number of likes.  I would say that he's an outlier because he probably gets into the thousands.

What outliers really mean is that you need to investigate the data more and integrate more features to help explain them.  For example, integrating sentimental and contextual understanding of my post would explain why on my daughter's birthday, I received hundreds of likes for that particular post.  Similar, incorporating Justin Bieber verified status, large following may help explain why a user like him receives a large number of likes.

From there you can move on to either building separate models for different demographics (average folks like me vs people like Justin Bieber) or try to incorporate more features.

TL;DR.  Don't remove outliers just because they are abnormal.  Investigate them.


To datawarehouse or not to data warehouse? I was wondering if you will be as so kind to assist me with a quick question (will to be happy to explain more if you are willing to...). I am researching and setting up a system to do a machine learning job (training) to find correlations between Social Media (or other digital trails from wearables etc.) information of a user and his scores on personality tests.

The scores are in my Postgresql (on AWS) and I need to decide on how to store the Social Media/Digital trails from wearables (unstructured and structured) information. I was thinking DynamoDB.

I was also thinking to integrate both databases under Amazon Redshift and to do the analytics (using RapidMinder) from there..... Does it all make sense? Do I really need a data warehouse for this? Will it be more sensible to use just a single DB (Postgresql or Dynamo) for all this without data warehousing? to I am talking about up to 100K records more or less (for the training).... Future data will in the millions.

I get so many conflicting answers and I hope and will appreciate your kindness and advice. Thank you so much in advance!!!

The main purpose of a datawarehouse is the ability to aggregate different types of data, and columns in a rapid way ( near realtime ) . Storage capability isn't the problem Datawarehousing is trying to solve. I can't really answer your question since i'm not quite aware of the volume of analytics you want to perform, but if it's for training a model over and over ( Online Learning ) , just set up a whole pipeline to apply the transformations you want ( the ETL part ) , model, train, and run your predictive method as many times as you want.


Data mining on reviews of the games I am trying to text mine the reviews of the games and want to find the interesting things from the reviews
example of the data is

'review': ['Simple yet with great replayability. In my opinion does "zombie" hordes and team work better than left 4 dead plus has a global leveling system. Alot of down to earth "zombie" splattering fun for the whole family. Amazed this sort of FPS is so rare.',
'Amazing, Non-stop action of blowing stuff to bits, Decapitation and shooting everything you see. With a combination of action, thriller and emmersive gameplay, as well as enviromental challanges (Jump physics). This game will really put your eyes to the test, can you see the enemys before they see you? Cause their are so many!This is the second level of the killing floor, I quote bill LF4D "Son we just crossed the street" But in reality they only moved up an elevator level on genes. What has yet to come as the game is slowly realsed with thrilling and horryfing creations, But who really cares, Let\'s just blow it up, I invite you to get on the GODAMN KILLING FLOOR, LET'S SHOOT  AND GET PAID!',......
]

I am interested to find the name of games and types of games(strategy, shooting, etc) from the reviews data.

I tried and created the regular expression such that getting the text which has data ( is a game )
so that I will get the reviews

Dying Light is a game.
Magicka is a game

I want a suggestion to improve the results for the above
and my interst is to text mine and find the game type such as a puzzle, shooting,etc

Suggestion to improve the results is to use NER. You can at very least extract names and types. Search a bit more maybe someone has also trained on different types/genres of game also, so you could discriminate on that also.


How to use CNN to deal with a 2D regression problem? I have seven measurements (Obs1-7), each measurement has the dimension of [x,y,t] where x and y are coordinates and t is time. Now I want to build a model that uses the first 6 measurements to predict the last one, saying Obs7=f(Obs1~6). I want to use CNN to distil this relationship f. I had some experience using CNN for classification, but have no idea how to deal with such 2D/3D regression. Could someone please give me some ideas? Thanks!

It should even be easier than classification: you do not need the final layer.

Your input layer should have 18 nodes $(x_i,y_i,t_i)|i=1..6$

Hidden layers as you see fit (experiment with it, depending on data and results).
I would expect the best results from a fully connected layer and a 'relu' layer.

The output layer has 3 nodes $(x_7, y_7, t_7)$

Train the model.

Evaluation is just feed forward of a set of 18 values, and retrieving 3 results from the output layer nodes.


keras plotting loss and MSE Can someone give me a tip on how I could incorporate MSE & loss plots? I have been following some machinelearningmastery posts to plot this but the application is classification and I am attempting regression. Also what is different in my script is I am defining the model thru calling a function, so I am curious if my script could be re-written without the function def wider_model() that defines the model.

This script below works except what is commented out on the bottom for the plt plots. In the machinelearningmastery post, someone does ask this question how to do for regression, and supposedly if you print print(history.history.keys()) two values are returned for dict_keys([‘mean_absolute_error’, ‘loss’])...

Any tips help, there isn't a lot of wisdom here... Thank you

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math

# load dataset
dataset = pandas.read_csv("prepdSPdata.csv", index_col='Date', parse_dates=True)

# shuffle dataset
dataset = dataset.sample(frac=1.0)

# split into input (X) and output (Y) variables
X = numpy.array(dataset.drop(['Demand'],1))
Y = numpy.array(dataset['Demand'])

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)

def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=11, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=1, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))

# list all data in history
#print(wider_model.wider_model.keys())

# summarize history for MSE
#plt.plot(history.history['acc'])
#plt.plot(history.history['val_acc'])
#plt.title('model MSE')
#plt.ylabel('MSE')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()

# summarize history for loss
#plt.plot(history.history['loss'])
#plt.plot(history.history['val_loss'])
#plt.title('model loss')
#plt.ylabel('loss')
#plt.xlabel('epoch')
#plt.legend(['train', 'test'], loc='upper left')
#plt.show()

cross_val_score does not return the history of the training. You can use fit instead:

history = model.fit( ...

See this example.

As you mentioned, the history object holds the results of the training for each epoch.

Here is the relevant bit:

history = model.fit(X, X, epochs=500, batch_size=len(X), verbose=2)
pyplot.plot(history.history['mean_squared_error'])
pyplot.plot(history.history['mean_absolute_error'])
pyplot.plot(history.history['mean_absolute_percentage_error'])
pyplot.plot(history.history['cosine_proximity'])
pyplot.show()

I was actually working on the same example that you referenced yesterday. I think it is hard to understand because it introduces many functions and concepts: estimators, StandardScaler, KerasRegressor, Pipeline, KFold and cross_val_score.

However, I did like the approach to creating and testing models, and Cross Validation would produce more robust models.


How to detect medicine name from the medicine wrapper I have got medicine wrapper ( Packaging ) of different medicines.

I want to detect medicine name out of it.

I'm using Google Cloud Vision to extract all the text from the medicine wrapper.

Text contains medicine mfg info, address, pin code, expiry date, ingredients and most importantly medicine name.

I want to extract only medicine name out of it.

What are the possible ways to get medicine name ?

Thanks!

The Python Library
Python-tesseract is an optical character recognition (OCR) tool for python. That is, it will recognize and “read” the text embedded in images. Python-tesseract is a wrapper for Google's Tesseract-OCR Engin
and please tell me where you find the medicine wrapper ( Packaging ) of different medicines.


Multi-ouput Classification + Sentiment Analysis I am facing some issues with a text classification problem and need your help to understand the best way to approach it.

The problem statement is as follows:
Assume I have a set of sentences describing a certain product (assume they are furniture items). Each of the sentences needs to be tagged with a class coming from a roster of 10 possible classes. The 10 classes refer to possible attributes of the item, such as softness, brightness etc.
On top of this, sentences contain sentiment information on each item (good, bad, neutral). Example: "I am satisfied with the performance of this couch, it is really soft".

The first question is: how would you approach the problem of labeling each sentence with both classes AND sentiment?

Now, to make things even more complex, assume each sentence can describe up to 3 items, with relative sentiment.
How would you approach the multi-ouput sentiment+class problem in this case?

Thanks a lot in advance for any help!

The "low hanging fruit" approach is "bag of words".

Lemmatize the sentence and use the lemma frequencies as features.

Build two models:

Sentiment (or reuse StanfordCoreNLP)
Topic (for this you need a training sample - a set of sentences labeled with the topic).

Now you can map every sentence to topic and sentiment and this will tell you whether the customer liked or disliked (sentiment) the softness (topic).

Multi-topic sentences

Usually a sentence has a single topic, but, if you do require multiple topics per sentence, you can use predict_proba or similar and use all classes with probability over, say, 0.3.

Class proliferation

The above approach fails on sentences like

  I liked the softness but hated the design.

because the total (average) sentiment of the sentence is probably neutral and we miss out softness-good and design-bad.

It is thus tempting to triple the number of classes (topic-sentiment) to accommodate reviews like

  The table combines pleasant design, flimsy build and neutral color.

The problem is that building models for very many classes requires a lot of training data. IOW, you will need to manually annotate a lot of reviews - and even then the model quality if unlikely to meet your expectations.

Real NLP

If you really need to handle multi-topic sentences, you might want to consider a "deeper" approach.

Specifically, take a look at the parse tree of the sentence.

split it into constituent phrases
use the topic model to identify the unique topic of the each phrase
use the sentiment model on the phrase


You say "I don't want to calculate slopes or averages by groups and cluster because the distributions don't seem linear, or normal" but it looks to me like each group makes a nice compact cluster which would be represented well by the centroid of the group. Of course, you have 5400 groups and I am only looking at four of them in your sample. Maybe the other groups are not as nice. Nevertheless, let's take a look at the data that you provided. Here is your data in four of the seven dimensions. The group centroids are marked with an orange circle with a black edge.

It certainly seems to me like the centroids provide a good representation of the group position. Based on this,  I would recommend calculating the centroids by group and then clustering the centroids. (Of course,  you will need to normalize the variables first because they have such different scales.)

Code

Centroids = aggregate(df[,2:8], list(df$Grp), mean)

par(mfrow=c(1,2))
plot(df[,c(2,7)], pch=20, col=df$Grp)
legend("bottomright", legend=levels(df$Grp), pch=20, col=1:4)
points(Centroids[,c(2,7)], pch=21, bg="orange")

plot(df[,c(4,6)], pch=20, col=df$Grp)
legend("topleft", legend=levels(df$Grp), pch=20, col=1:4)
points(Centroids[,c(4,6)], pch=21, bg="orange")


Creating Balanced Dataset Using Scikits I have a classic User-Item dataset where each row (i.e., (user, item)) indicates the action of a user clicking/selecting an item. Now, the dataset only provides positive samples and does not specifically indicate whether a user has disliked an item. In order to create a balanced dataset, I would like to create random negative samples (for instance randomly pick a set of items which the user has never clicked). Of course, I can achieve this by writing a program; however, I found this library in python Scikits called imbalanced-learn, which seems to provide various sampling techniques. Can someone provide a small code-snippet that can use this library (or some other python library) to achieve this?.

If I understood correctly, you want to invent new negative samples to have a balanced dataset, because your current dataset only have positive samples.

However, the library you mentioned only re-samples a dataset where all classes are present. The sample creation process is by definition data-specific, as there can be impossible attribute combinations, etc.

If you want to blindly create combination of attribute values, you can use the stuff in module random:

To create integer attributes:
def random_integer(min_val, max_val):
    return randint.randint(0, 9))

To create real-valued attributes:
def random_float(min_val, max_val):
    return random.uniform(min_val, max_val)

To create discrete attributes:
def random_discrete(value_list):
    return random.choice(value_list)

And then, with a loop, generate your negative samples in a loop:

    def generate_negative_sample():
        return {'attr1': random_discrete(['a', 'b', 'c']),
                'attr2': random_float(-10., 10.),
                'attr3': random_integer(0, 4)}

    random_samples = [generate_negative_sample() for _ in range(1000)]


RL Policy Gradient: How to deal with rewards that are strictly positive? In short:

In the policy gradient method, if the reward is always positive (never negative), the policy gradient will always be positive, hence it will keep making our parameters larger. This makes the learning algorithm meaningless. How do we get around this problem?

In detail:

In "RL Course by David Silver" lecture 7 (on YouTube), he introduced the REINFORCE algorithm for policy gradient (here just showing 1 step):

The actual policy update is:

Note that $v_t$ here stands for the reward we get. Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence $\theta$ will keep increasing! So how do we deal with rewards that never change sign?

Let's say we're playing a game where the reward is always positive (eg. accumulating a score), and there are never any negative rewards, the gradient will always be positive, hence θ will keep increasing! So how do we deal with rewards that never change sign?

This is true. However, in many policy functions and in most situations, the gradient part $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ will tend to zero as you reach a deterministic policy. This happens for a softmax action selection based on "preferences" (a matrix of softmax weights per action for each state) or as the output layer of a neural network. And it will counteract the tendency for the final layer preferences (or the logits of the neural network last layer) to grow uncontrollably.

You have identified a true weakness of REINFORCE. For instance, using the softmax action selection as an example and your always-positive returns:

When the agent selects a non-maximising action, this will result in a positive return, the agent will add to its preference for that action.
When the agent selects a maximising action, this will result in a larger positive return, the agent will add to its preference for that action more
REINFORCE works by increasing the preferences of better actions faster than preferences of worse actions.
This leads to a feedback process where better actions are chosen more often, increasing their preference values even faster.
Ultimately the preference for best actions will be so much higher than the alternatives that the softmax function will saturate. The gradient  $\nabla_{\theta} log \pi_{\theta}(s_t,a_t)$ for all actions will be close to zero.

This does not always happen quickly, and basic REINFORCE implementations can be numerically unstable for exactly the reason in your question. To improve stability (and often learning speed), you can use REINFORCE with baseline, which starts to address your concern by using offsetting values ($v_t - \bar{v}_t$ or similar). You can also then take that idea further and use Actor-Critic.


Clustering cartesian coordinates associated with 1 categorical feature I have a series of 2D coordinates X = {x, y}. Each are associated with one categorical variable W that can take 7 different values.

E.g:

coord  W
X1     3
X2     5
X3     7
X4     3
X5     2
X6     3
X7     2
...
X2000  5
...

I would like to get all the clusters that belong to a given set of my categorical variable within 2 pixels of each other. Say, for the triplet of W values (2, 5, 7), or a tuplet of all values (1,2,3,4,5,6,7), I want to have all the sets (if any) of coordinates that are within 2 pixels of each other. What would be the most appropriate methods?

Also, there are no singletons, all these coordinates have at least another nearest neighbour with a different W value. I only know how to do this to only find the sets of pairs of coordinates for 2 different W values (using pair-wise euclidian distance matrix), but to get clusters of more elements for more than 2 of my W values, I am confused about what clustering method to use (and if that actually falls in the realm of clustering at all...), as that seems rather basic and I keep reading about rather sophisticated approaches that look overkill (KNN, HDBSCAN, ...).

This isn't a clustering problem. Within each set (like W=3), you simply sort them by x coordinate and then y coordinate too. Find the points +/- 2 pixels in the x dimension, then in the y dimension, and take the points that are in both. This ought to be linear within a sorted list.


How to represent time based periodic data for use in ML I have a dataset of following format -

Number of Machines
CustId   Month0   Month-1   Month-2   Month-3   Month-4
abc      23       26        29        0         0
def      53       26        22        22        12
ghi      11       26        150       120       10

Size of data protected
CustId   Month0   Month-1   Month-2   Month-3   Month-4
abc      23       26        29        0         0
def      53       26        22        22        12
ghi      11       26        150       120       10

The data is Month-over-Month data. For simplicity I have used the same data in both tables, but for a given CustId data will be present in both the tables. Similarly there are tables for other parameters as well.

I want to use machine learning for some classification. What is the best way to serialize this MoM data for different parameters? Is there any standard practice for this?

Edit:

I am not sure of what are the distinctive patterns in the graphs of the parameters for different classification categories. By graph of the parameter I mean Number of machines plotted against Month.

One thing I can do is to combine all the tables with format something like -

    CustId   ----# machine---       -------data protected-----
             Month0   Month-1 ...   Month0   Month-1   Month-2 ...
    abc      23       26      ...   23       26        29      ...
    def      53       26      ...   53       26        22      ...
    ghi      11       26      ...   11       26        150     ...

But I don't want to use this format because in here all the columns (monthly data) would be treated as independent data points. I want the columns under '# machine" category to be treated as group and the columns under "data protected" as another group.

I came across this Python Library (tsFresh) which extracts information from the graphs. I am looking for something similar or any other way to approach this problem.

Answering the question JahKnows asked in the comment above will help with finding the suitable solution for your use cases.

If you are not exactly sure what you are after, here are a few things you can do with a dataset like this:

One of the basic things you can do is to find customers that behave similarly. With the data that you have, you can consider the number of machines used by each customer in each month as features. In this case, each customer is described with 5 features. I would run a clustering algorithm against this data (of course leave CustId field out) and see how many types (clusters) of customers you have.
Another thing to do is to run statistical analysis over the number of machines your customers use each month and find those that stand out from the rest of the population (customers).
Another analysis you could do is to analyze the trend for each customer separately and based on their past behaviour, determine if the number of machines they have used in the past month was in accordance with their historical behaviour. In this case, you can take proper actions based on the shift of behaviour. E.g. a drop in the number of machines could mean the customer is having technical difficulties or churning.


Does it make sense to apply clustering on aggregation of data? I was wondering if it makes sense to apply clustering techniques on an aggregation of data, like, I have three different sources of data such as S1 S2 and S3 where each of these sources share some common columns but the majority are not shared. Does it make sense to group all the sources with all the columns in a big dataframe and apply clustering techniques whereas some records will only have null values for the columns that are not part of its corresponding service.
Thank you.

Update:
The input is basically logs coming from different services with different columns where some of them are shared between all services. The output should be a cluster of records representing a user having the same behavior.
The logs are gathered by second corresponding of a user action. They are aggregated by hours for each user to derive other features (the granularity of seconds it according to me, too much).
The goal, is to detect anormal behaviors.
And the question is, can I regroup different dataframe having different features while some of them are shared and run a K-mean on it. Because my dataframe would look like this (s* for service, and common_ for common feature):

-------------------------------------------------------------
s1_f1 | s1_f2 | s2_f1 | s2_f2 | s3_f1 | common_f1 | common_f2
  1       OK     NULL    NULL    NULL     midday      less
  2       OK     NULL    NULL    NULL     midnight    more
 NULL    NULL     2       5      NULL     midday      less
 NULL    NULL     8       9      NULL     morning     less
 NULL    NULL    NULL    NULL    777      morning     more
 NULL    NULL    NULL    NULL    888      night       more

Since some features are missing for specific sources, the missing values are not missing-at-random but are systematically missing. In this situation, I'd advise against doing clustering on the combined data set with all available features. If missing values were occurring at random, you could have used some missing value imputation method before performing cluster analysis. However, since the values are systematically missing, imputation would be difficult to tackle. (You could try to predict those missing values, but I am afraid that will add a lot of unnecessary noise in the data.)

I'd recommend choosing from one of these two options:

Perform clustering on the combined data set, but use only those features that are non-missing across all sources.
Perform three different cluster analysis, one for each source. This way, you can ensure that you are using as many features (information) as possible. The determination of "abnormal" behavior can then be determined within each source. This can be an added benefit since it would allow you to be more specific about why a use might be abnormal, as you have more features that can be used to explain this. The results can also be then summarized across all sources to create one consolidated report.


Setting class weights for categorical labels in Keras using generator I'm working on with an imbalanced dataset in Keras, and would like to give a higher weight to the samples from my minority class. The fit() function has a nice sample_weight argument, however because of the size of my data I have to use fit_generator().

fit_generator() has a class_weight argument, which seems useful for this purpose and is already discussed in Another question. However, in this case the labels are not one-hot-encoded/categorical and I could not find whether using class_weight also allows for categorical data.

Can use the class_weight argument for one-hot-encoded/categorical labels and if so how? Or do I have to resort to a custom weighted loss function?

For categorical data, it is best to use sample_weight instead of class_weight argument. This can be done by simply giving all samples of a particular class the same weight. sample_weight works for categorical data because it takes a numpy array as its value as opposed to a dictionary (which won't work for categorical class labels) in case of class_weight.

See: Keras sequential model methods

  You can either pass a flat (1D) Numpy array with the same length as
  the input samples (1:1 mapping between weights and samples), or in the
  case of temporal data, you can pass a 2D array with shape (samples,
  sequence_length), to apply a different weight to every timestep of
  every sample. In this case you should make sure to specify
  sample_weight_mode="temporal" in compile().

The other way, as you mentioned is using custom weighted loss function. A detailed discussion on it can be found here.


Number of rows vs. number of variables I am new to the ML/DS field. I started a project where I need to predict the age of users. I'd like to build two models, one predicting the age group (I created 6 age groups), and the other one predicting the age explicitly (as a number). I cleaned the data, and did the correlation between the possible input variables and target variables.
What I get confused about is: how many variables is too little compering to the dataset, and how many variables is too much?
Some people are suggesting that "the more the better", but I am also aware that too many features can cause overfitting...
Is there a rule of thumb? ex: "if I have a dataset of 500k it's reasonable to have 50 variables..." or is this really depending from case to case, and I should figure out variable importance to the model, once it is done?
Please let me know if something needs to be clarified, I'd really like if someone could help me with their experience and/or direct me to an article/paper that deals with this subject.

There are general rules of thumb for different models, both in terms of the number of samples required as well as the number of features. I will try to just put some concrete numbers on things, with the caveat that is is indeed problem specific, and you usually have to make compromises.

If you are looking at something like a simple linear regression e.g. with 5 features, then for the basic tests to be considered statistically relevant, you need at least 40 samples.

In your case with 500,000 samples, 50 features is definitely acceptable - you could even have more. There are, however, two things I would be wary of:

Firstly, you make life hard for your model is many of the features are really describing the same thing. The technical term is multicollinearity, which, if your data has it, means that your features themselves are highly (linearly) correlated. It is hard for the model to know which feature to rely on, and your results may be improved by simply removing one. A simple example: if I predict your shoe size based on your height, and I have two features: your height in inches and your height in centimetres, you can see that I only really need one of these, as they are perfectly correlated measures.

Secondly, The more the better is theoretically true, as long as you have a model which is able to decide when to include features and when to ignore them - and that your data is good enough to facilitate this. There are ideas like Occam's razor, saying that a simple model should be selected over a complex model with the same performance; i.e. you should also consider interpretability. There are measures defined that quantify this, such as the Akaike Information criterion. It is a model selection tool - so you could fit the model using 10, 20, 30, and 50 of your features, compare the results of this creiterion and pick the model which scores best. Here is a recent question which touches on that topic.

There are functions that will remove features, based on a correlation threshold with other features in the dataset. It is explained well in the Caret package, for R, but the same thing can be done in Python fairly easily (not sure if there is a standard implementation).
[EDIT] - the Python equivalent looks to be the VarianceThreshold class.


How to train convolutional neural networks on unbalanced datasets of images? How can I train convolutional neural networks on unbalanced datasets of images?

My dataset has around 400 classes and the classes have different number of images..

By oversampling. When training a CNN, you generally use a mini-batch gradient descent method. The mini-batch method consists on taking some images at random and performing a gradient step with those images by backpropagating the loss of the model on those images. This random selection doesn't need to be uniform on all the images. Instead, you can make the images in the classes with few elements appear with more likelihood (oversampling them) than the other images. With this, you can help the CNN not only to learn about the most prevalent classes.

In addition, if you have very few number of images of some classes, it is recommended to do image augmentation, thus generate more images in order to avoid overfitting those classes.


How to feed key-value features (aggregated data) to LSTM? I have the following time-series aggregated input for an LSTM-based model:

x(0): {y(0,0): {a(0,0), b(0,0)}, y(0,1): {a(0,1), b(0,1)}, ..., y(0,n): {a(0,n), b(0,n)}}
x(1): {y(1,0): {a(1,0), b(1,0)}, y(1,1): {a(1,1), b(1,1)}, ..., y(1,n): {a(1,n), b(1,n)}}
...
x(m): {y(m,0): {a(m,0), b(m,0)}, y(m,1): {a(m,1), b(m,1)}, ..., y(m,n): {a(m,n), b(m,n)}}

where x(m) is a timestep, a(m,n) and b(m,n) are features aggregated by the non-temporal sequential key y(m,n) which might be 0...1,000.

Example:

0: {90: {4, 4.2}, 91: {6, 0.2}, 92: {1, 0.4}, 93: {12, 11.2}}
1: {103: {1, 0.2}}
2: {100: {3, 0.1}, 101: {0.4, 4}}

Where 90-93, 103, and 100-101 are aggregation keys.

How can I feed this kind of input to LSTM?

Another approach would be to use non-aggregated data. In that case, I'd get the proper input for LSTM. Example:

Aggregated input:

0: {100: {3, 0.1}, 101: {0.4, 4}}

Original input:

0: 100, 1, 0.05
1: 101, 0.2, 2
2: 100, 1, 0
3: 100, 1, 0.05
4: 101, 0.2, 2

But in that case, the aggregation would be lost, and the whole purpose of aggregation is to minimize the number of steps so that I get 500 timesteps instead of e.g. 40,000, which is impossible to feed to LSTM. If you have any ideas I'd appreciate it.

Sounds to me like you can reshape the input of a timestep as matrix s.t. $X_t \in \mathbb{R}^{n \times 2}$ (assuming you only have a and b as features) and use Convolutional LSTM layers:
$t=0$:
$X_0 = \begin{bmatrix}a(0,0)&b(0,0)\\a(0,1)&b(0,1)\\\vdots&\vdots\\a(0,n)&b(0,n)\end{bmatrix}$
$t=m$:
$X_m = \begin{bmatrix}a(m,0)&b(m,0)\\\vdots&\vdots\\a(m,n)&b(m,n)\end{bmatrix}$
Basically each sequential keys $y$ is a row in your input matrix. You can think of each matrix as some kind of image you feed to your LSTM network.
Hope that helps you any further.


Scaling DBSCAN clustering - minHash? Applying density based clustering (DBSCAN) on $50k$ data points and about $2k$-$4k$ features, I achieve the desired results.

However, scaling this to $10$ million data points requires a creatively efficient implementation since DBSCAN requires $O(n^2)$ to calculate the distance matrix and crushes my memory.

There must be some efficient sampling-based method to overcome this, ideally something similar to minHash - but I'm not sure how to approach this, and if there exists a  solution that can work on existing sklearn DBSCAN algorithm. Any ideas?

DBSCAN is O(n) times the cost of a neighbor search.

If you use an index like LSH that could answer neighborhood search in O(1) (assuming a very even data distribution, with O(1) neighbors per point) then DBSCAN can run in O(n) plus the time needed to build such an index.

Yes, minHash indexes could be used if they are appropriate for your data and distance.


ROC curve for different hyperparameters of `RandomForestClassifier`? I'm currently trying to train a RandomForestClassifier on a dataset consisting of 5000 instances with 12 (now) encoded features and a binary target label. Through GridSearchCV I found out, that

best_parameters = {
    'criterion':    'gini',
    'max_depth':    12,
    'max_features': 'log2',
    'n_estimators': 300
}

works best out of

hyperparameters = {
    "n_estimators": [9,10,20,30,40,50,60,100,150,200,300,1000],
    "max_depth":    [3,6,9,12,20],
    "criterion":    ["gini", "entropy"],
    "max_features": ["log2", "auto"]
}

And gives a mean_test_score of 0.8546 which is quite good already, as I think.

Now I would like to get some kind of visual interpretation like a ROC curve for each parameter. But does it actually make sense in the case of a RandomForestClassifier to create a ROC curve for each hyperparameter? Or are there other ways to tune my classifier?

Seems like you already tuned your algorithm based on the mean_test_score. You also could tune the AUC, which is the area under the ROC curve. Tuning this can even provide better mean_test_score as I show in my blog post: http://philipppro.github.io/Tuning_random_forest/


In which cases are non-linear learning methods preferred than logistic regression in classification problems? We know that neural networks and other learning methods can have better performance relative to logistic regression in some non-linear classification problems. But, it is known too that logistic regression can separate classes with a line that can be curvy if only we add more predictors that are a square, cube, etc of the given predictors (still considered a linear decision boundary).

So my question is, can we theoretically solve any classification problem through logistic regression, or are there limitations that I do not realize that make other non-linear learning methods mandatory to certain classification problems?

There are several assumptions that are needed to be satisfied in order for logistic regression to work. From personal experience the two most important assumptions:

Features need to be not (or a little) correlated with each other, because if they are correlated then the change of one feature indicates also the change of the other that's a problem when you try to solve the optimization problem
May some features not be ordinal

You can also check here for more information.

Other learning methods such as RandomForest require no such assumptions and may work better for some datasets. But keep in mind the no free lunch theorem, most of the times you need to get your hands dirty and test several learning models before you decide which works best for your data.


Creating trial number ascending list, for separate subjects in long format data frame, R? I have a very long data frame (over 300 000 rows), consisting of all subjects trials of a task, in long format. So about 300 rows is successive trials for one subject, followed by the other subjects underneath. What I want to do is create a new column with trial numbers for every subject.

Such as:

subject   trial_number
    101           1
    101           2
    101           3
    102           1
    102           2
    102           3

I am thinking I should somehow make R choose a subject number from column subject, and then create an ascending list and then loop this over all the subject numbers. But I have not been able to figure out how to loop over subject numbers, while also creating an ascending list in one and the same column over these subjects?

I don't have example code for failed attempts or so, as I haven't been able to figure out how to structure this. I'm thinking some kind of combination of  subset and within? Or are there better solutions my googling skills haven't helped me find yet?

If I understand your question correctly, and assuming that your initial data frame looks something like trial_df below, you can use the following code and the dplyr package.

library(dplyr)

# create data frame
trial_df <- tibble('subject' = c(rep('101', 3), rep('102', 3)),
                  'result' = c(32, 33, 64, 12, 18, 14))

# group by subject and add a row number
# assumes your data frame is ordered by trial number for each subject
new_df <- trial_df %>%
  group_by(subject) %>%
  mutate(trial_number = row_number())


Sort a data frame when column name is assigned in a variable I am new to R and hence my question is likely to be basic. I tried researching the answer before posting here however didn't get the answer I was looking for.

I am trying to order a data frame on the basis of certain columns. The column is dynamic and depends on a certain "outcome" which would be from a vector of "validoutcomes".

validoutcomes <- c("A","B","C")

if outcome is "A" then I am supposed to read data from column 11. For "B" its columns 17 and for "C" it is 23.

I renamed the columns so that it is easier for my readability and alignment with  validoutcomes. "input" is my source data from the CSV.

colnames(input)[11] <- validoutcomes[1]
 colnames(input)[17] <- validoutcomes[2]
 colnames(input)[23] <- validoutcomes[3]

I then tried to sort using

statelist <- statelist[order(statelist$outcome)]

Any ideas?

Your first attempt was more accurate. Since I do not have your data I will just use the iris data set,

 iris_ordered_on_Sepal_Length <- iris[order(iris$Sepal.Length),]

or if you use data.table,

iris_dt <- data.table(iris)
iris_dt[order(Sepal.Length)]

data.table will make renaming easier too,

setnames(input, c("colname_11, "colname_17", "colname_23"), validoutcomes[1 : 3])


Augmenting the validation set in Ensemble Model I have 8 models which I have trained on 90% of my set (training set) and tracked its performance on the loss of the validation set (10% of the original set). I want to generate an ensemble model by blending the different models with a dense layer. However, the validation set only consists of 64 samples and I am concerned that this might cause severe overfitting. I was wondering whether it would make sense to augment the validation data, so that I can increase the number of instances and help in preventing overfitting.

Have you considered freezing the ensemble models themselves and allowing only the weights in final layer to change? Then the overfitting shouldn't be so pronounced...
You might also want to consider cross validation procedure.


What are some good sources to learn fraud/anomaly detection in normal/time-series data? I would like to know more on fraud/anomaly detection. I am looking for good source or survey article/book etc out there which will give me some preliminary idea of the area.

Any suggestion is greatly appreciated.

Thanks

Chandola et al's survey is probably the best and most-widely cited survey in the anomaly detection field.


Why fully calculate softmax? Why is the softmax activation function used in the output layer for CNNs?
Why not just take the highest value of the units in the output layer?

If you are only interested in the most likely class, during inference you can skip the softmax. This is fairly common even, and the reason why TensorFlow has specific logit functionality. However, for optimizing your network you need a loss function that makes sense. You have normal labels, what would your loss function be if you only compute the linear combinations?


Over-sampling: is my model over-fitting? I would like to ask you some questions on how to consider (good or not) the following results:
OVER-SAMPLING
              precision    recall  f1-score   support

         0.0       1.00      0.85      0.92       873
         1.0       0.87      1.00      0.93       884

    accuracy                           0.92      1757
   macro avg       0.93      0.92      0.92      1757
weighted avg       0.93      0.92      0.92      1757

Confusion Matrix:
 [[742 131]
 [  2 882]]

I have a dataset with 3500 obs (3000 with class 0 and 500 with class 1). I would like to predict class 1 (target variable). Since it is a problem of imbalance classes, I had to consider re-sampling methods. The result shown above is from over-sampling. Do you think it over-fits and/or that it cannot be a good re-sampling method for my case?
I am looking at the f1-score column, since it is a text classification problem.

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification report should mirror the imbalance in your dataset.
From what I understand you have 3500 samples, then you did some oversampling (probably brought them to around 6000) and then took 1757 from these for testing. This evaluation scheme is wrong. Take a look at the illustration below to see a more correct scheme.
      |--- train --> oversample train set --> train model---|
set --|                                                     |--> evaluation on test set
      |--- test --------------------------------------------|


Numpy array from pandas dataframe I am new in using python for data science.
What is the difference between selecting a a column with: df['name'].values and df.iloc[:,1].values and df.iloc[:,1:2].values they return differnt types of numpy vectors. why?

Not entirely sure what you mean by "numpy vectors" but am assuming the question is why each of these methods return essentially (almost but not quite) the same output...
Reference: pandas docs.
df['name'].values is a "Series corresponding to colname". In other words, you're just calling the data from that column and putting the in an array by calling .values.
.iloc is a "Purely integer-location based indexing for selection by position". Same as above but you're calling the indexed location of the column where df.iloc[:, 1] is df.iloc[all rows, col 2]. Probably an easier method to call multiple consecutive columns in a DataFrame then writing out each individual column name.
df.iloc[:,1:2].values <-- creates an array of arrays where the main array is the column that you called (col2) and each row values is contained in a subarray. This is--I think-- because you're slicing the dataframe between column index locations 1 and 2 (rather than just calling loc 1 like above). This would mean that each row is being called individually so that a new array is created for each row that exists between column index locations 1 and 2 (which is the 'parent' array).


How to generate abbrevations from shortend words in medical records I have text files which contains medical history of a patients and would want to extract information out of it.

Basically what want is generate english text of abbrevations,semantic category region,location and relation also correct spelling mistakes if any by parsing medical history records.

I have google around for open source tools and found cTAKES & Metamap.However could not find python api for the same.

Can someone suggest how to make use of these tools or interact via python?and also is there a better way to extract information using regular epression..etc.

Sample sentence:
55 yr M comes to the ED with c/o of a chest pain that started 1 hr ago,
The cP are a/w SOB.

Expected o/p
55 yr(year) (M)male comes to the (ED)emergency department with (c/o)complaint of a chest pain that started 1 hr(hour) ago,The (cP)chest pain is associated with (SOB)short of breathing.

Apart from generating abbrevations,also need to cature temparatures,levels such as 5/4,semantic category region,location and relation also correct spelling mistakes in the clinical notes.

While am waiting for UMLS license,just wanted to check can above functionality be achieved through UMLS/cTaked/Metamap? or cutsom script needs to be written in conjunction with python/NLP/RE and open source clinical abbrevations dictionary(please suggest)?

First a warning: is your data anonymized? Even if it is, be extremely careful because medical history is super-sensitive personal information. There are legal requirements about how to handle this kind of data and what you can or can't do with it.

As far as I know there's no python API for cTakes or Metamap. Anyway I guess that such an API would boil down to a system call, so it wouldn't be very good.

As an alternative you could implement your own system using the UMLS Metathesaurus, which is a massive list of medical terms grouped by concept (MetaMap extracts UMLS terms/concepts).


Why do recurrent layers work better than simple feed-forward networks? On a time series problem that we try to solve using RNNs, the input usually has the shape $input features \times timesteps \times batchsize$ and we then feed this input into recurrent layers. An alternative would be to flatten the data so that the shape is $(input features \times timesteps) \times batchsize$ and use a fully connected layer for our time series task. This would clearly work and our dense network would be able to find dependencies between the data at different timesteps as well. So what is it that makes recurrent layers more powerful? I would be very thankful for an intuitive explanation.

The first reason is the number of parameters. The former case that you've mentioned, for each neuron there should be corresponding entries that would increase the number of training parameters. The other reason is that by employing simple feed-forward neurons you are somehow discarding the temporal information of your data which means you are discarding the sequence information in your data. This is somehow like the spatial data which is obtained by convolutional layers in CNNs.


What is the intuition behind decreasing the slope when using regularization? While training a logistic regression model, using regularization can help distribute weights and avoid reliance on some particular weight, making the model more robust.
Eg: suppose my input vector is 4 dimensional. The input values are [1,1,1,1]. The output can be 1 if my weight matrix has values [1,0,0,0] or [0.25,0.25,0.25,0.25]. L2 norm would give the later weight matrix (because pow(1, 2) > 4*pow(0.25,2) ). I understand intuitively why l2 regularization can be beneficial here.
But in case of linear regression l2 regularization reduces the slope. Why reducing the slope only provides better performance, is increasing the slope also an alternative?

Using regularization and shrinking the parameters, we reduce the sample variance of the estimates, and reduce the tendency to fit the random noise. The fitting to the noise is something that we wish to reduce. We can't increase the slope as we wan't to reduce overfitting.
L2 doesn’t necessarily reduce the number of features, but rather reduces the magnitude/impact that each features has on the model by reducing the coefficient value.
Shrinking can lead to positive effect when we have overestimated and negative when we have underestimated. But we are not shrinking everyone equally ,we are shifting with a factor that is larger if the estimate is larger away from zero.
Shrinking all the slopes towards zero will make some of them more accurate and some of them less accurate, but you can see how it would make them collectively more accurate.


t-SNE Python implementation: Kullback-Leibler divergence t-SNE, as in [1], works by progressively reducing the Kullback-Leibler (KL) divergence, until a certain condition is met.
The creators of t-SNE suggests to use KL divergence as a performance criterion for the visualizations:

you can compare the Kullback-Leibler divergences that t-SNE reports. It is perfectly fine to run t-SNE ten times, and select the solution with the lowest KL divergence [2]

I tried two implementations of t-SNE:

python: sklearn.manifold.TSNE().
R: tsne, from library(tsne).

Both these implementations, when verbosity is set, print the error (Kullback-Leibler divergence) for each iteration. However, they don't allow the user to get this information, which looks a bit strange to me.
For example, the code:
import numpy as np
from sklearn.manifold import TSNE
X = np.array([[0, 0, 0], [0, 1, 1], [1, 0, 1], [1, 1, 1]])
model = TSNE(n_components=2, verbose=2, n_iter=200)
t = model.fit_transform(X)

produces:
[t-SNE] Computing pairwise distances...
[t-SNE] Computed conditional probabilities for sample 4 / 4
[t-SNE] Mean sigma: 1125899906842624.000000
[t-SNE] Iteration 10: error = 6.7213750, gradient norm = 0.0012028
[t-SNE] Iteration 20: error = 6.7192064, gradient norm = 0.0012062
[t-SNE] Iteration 30: error = 6.7178683, gradient norm = 0.0012114
...
[t-SNE] Error after 200 iterations: 0.270186

Now, as far as I understand, 0.270186 should be the KL divergence. However I cannot get this information, neither from model nor from t (which is a simple numpy.ndarray).
To solve this problem I could:

Calculate KL divergence by my self,
Do something nasty in python for capturing and parsing TSNE() function's output [3].

However:

would be quite stupid to re-calculate KL divergence, when TSNE() has already computed it,
would be a bit unusual in terms of code.

Do you have any other suggestion? Is there a standard way to get this information using this library?
I mentioned I tried R's tsne library, but I'd prefer the answers to focus on the python sklearn implementation.

References
[1] http://nbviewer.ipython.org/urls/gist.githubusercontent.com/AlexanderFabisch/1a0c648de22eff4a2a3e/raw/59d5bc5ed8f8bfd9ff1f7faa749d1b095aa97d5a/t-SNE.ipynb
[2] http://homepage.tudelft.nl/19j49/t-SNE.html
[3] https://stackoverflow.com/questions/16571150/how-to-capture-stdout-output-from-a-python-function-call

The TSNE source in scikit-learn is in pure Python. Fit fit_transform() method is actually calling a private _fit() function which then calls a private _tsne() function. That _tsne() function has a local variable error which is printed out at the end of the fit. Seems like you could pretty easily change one or two lines of source code to have that value returned to fit_transform().


Database for a trie, or other appropriate structure for recommendation engine We are storing the information about our users showing interest in our items. Based on this information, we would like to create a simple recommendation engine that will take the items I1, I2, I3 etc of the current user, search for all other users that had shown interest in those items, and then output the items I4, I5, I6 etc of the other users, sorted by their decreasing popularity. So, basically, the standard "other buyer were also interested in..." functionality.

I'm asking myself what kind of a database is suitable for a realtime recommendation engine like this. My current idea is to build a trie of item IDs, then sort the item IDs of the current user (as the order of items is irrelevant) and to go down the trie; the children of the last trie node will build the needed output.

The problem is that we have 2 million items so that according to our estimation the trie will have at least 1E12 nodes, so that we probably need a distributed sharded database to store it. Before we reinvent the wheel, are there any ready-to-use databases or generally, non-cloud solutions for recommendation engines out there?

Have a look at Apache Mahout. Last version features also user-item-based recommenders.


Can Boosted Trees predict below the minimum value of the training label? I am using gradient Gradient Boosted Trees (with Catboost) for a Regression task. Can GBtrees predict a label that is below the minimum (or above the max) that was seen in the training ?
For instance if the minimum value the label had is 10, would GBtrees be able to predict 5?

Yes, gradient boosted trees can make predictions outside the training labels' range.  Here's a quick example:
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingRegressor

X, y = make_classification(random_state=42)

gbm = GradientBoostingRegressor(max_depth=1,
                                n_estimators=10,
                                learning_rate=1,
                                random_state=42)
gbm.fit(X,y)
preds = gbm.predict(X)
print(preds.min(), preds.max())

outputs -0.010418732339562916 1.134566081403055 (and make_classification gives outputs just 0 and 1).
Now, this is unrealistic for a number of reasons: I'm using a regression model for a classification problem, I'm using learning rate 1, depth only 1, no regularization, etc.  All of these could be made more proper and we could still find an example with predictions outside the training range, but it would be harder to construct such an example.  I would say that in practice, you're unlikely to get anything very far from the training range.
See the (more theoretical) example in this comment of an xgboost github issue, found via this cv.se post.

To be clear, decision trees, random forests, and adaptive boosting all cannot make predictions outside the training range.  This is specific to gradient boosted trees.


Hidden neuron representation of weights In an RBM, if we represent the weights learned by the hidden units, they show that the neural net is learning basic shapes. For example, for the mnist dataset, they learn features of the numbers they are trying to classify.

In a regular feed-forward net with one hidden layer, I can train the network to recognize digits, and it works, but when I try to visualize the hidden layer weights, I only see noise, no distinguishable feature. Why is that? Hasn't the network learned to recognize the digits?

It has learnt to recognize the digits, but it might have put too much weight on single pixels. Try to add different amounts of L2 regularization or dropout and compare the visualizations of the weights. Adding some kind of regularization should make the net rely less on single / independent pixels and more on the inherent structure of the digits, giving you smoother weights / visualization.


Why does a randomly initialised convolution kernel correspond to an edge detector? In this nice tutorial about CNNs, the authors build a single-layer CNN. The initial convolution weights are set randomly, according to a uniform distribution.

By the end of this scetion, the authors note that the randomly initialised kernel behaves very similar to an edge detector and give the following input and output as example.

Why does the randomly initialised kernel behave like an edge detector?

Take this with a grain of salt, but I think this is simply not true. You can evaluate it with the code I just wrote:

Only 2, probably 3 of the 25 look like edge filters to me. The result of an edge filter looks like this:


How can we model the class which maximizes the event probability? I’d like to ask about a case when we would like to predict the best class of some input variable, so that the probability of event will be maximized.

For example the advertisement type for a given customer, which will maximize the probability of purchase.
In collected data we have many various ad types, customer descriptors and informations if the purchase was made or not.

One solution which comes to my mind is to treat the ad type as an input variable, train regular probability model, then make predictions with all ad type configuration and then pick one giving the best estimation.

What are the other options?

In your method, you would build a model which maps the input features (ad type, customer descriptors) to the output (purchase made = 0/1). You can use a model such as a logistic regression, or a decision tree (which can model the interaction effects between the ad type and the customer descriptors.) This seems to be reasonable and is one of the approaches commonly used for ad selection.

A few other options based on customer similarity:

Near-neighbour based approach: Based on the customer descriptors, identify similar neighbours of the current customer. Compute the probability of conversion for each ad type for all the neighbours put together, and pick the ad type which has the highest probability of conversion.
Cluster based approach: Form clusters of customers based on the customer descriptors. Map the current customer into one of the clusters, and pick the ad type which maximizes conversion probability for that cluster.

With respect to designing a system of ad selection using such a model, it is also important for the system to explore new regions of the space, so some small randomization of predictions will help here.


How does a FC layer work in a typical CNN I am new to CNNs and NNs. I am reading this blog: CNN and I am confused about this part: What confuses me is the operation that will be performed on an input vector/matrix. Will we be using a typical ANN equation: "O = W.T * input"?. And then a sigmoid on top of it?

Yes, essentially a typical CNN consists of two parts:

The convolution and pooling layers, whose goals are to extract features from the images. These are the first layers in the network.
The final layer(s), which are usually Fully Connected NNs, whose goal is to classify those features.

The latter do have a typical equation (i.e. $f(W^T \cdot X + b)$), where $f$ is an activation function. Usually in the context of CNNs, $f$ is a ReLU, except for the activation function of the final layer, which is selected according to the nature of the problem. The most common cases are:

Sigmoid activation functions work for binary classification problems.
Softmax activation functions work practically for both binary and multi-class classification problem.
For regression problems, the final layer has no activation.

One final note I'd like to make is that before entering the first FC layer, the output of the previous layer is flattened. By this I mean that the (typically 3) dimensions of that tensor are layed out into one large dimension.
For example a tensor with a shape of $(5, 5, 32)$, when flattened would become $(5 \cdot 5 \cdot 32) = (800)$.


Shape of a distribution as a feature How can I use the shape of a distribution as a feature in machine learning ? Do I use something like the standard deviation ?

If this distribution is row specific (each sample has a different associated distribution) or category specific this is not a bad approach to encode more information in your features. It's unclear how you have these distributions, are there empirical samples or do you have a parameterized distribution? A few approaches you could take to encoding this could be:

Fit a distribution family (or if you have one already) and use the location/shape/scale parameters as features
Add a few moments and other statistics of the distribution
Or similarly take a few percentiles

Depending on the parameterization of the distribution, the first might be less direct but it should be easy to test.


Tools to explore various datasets TL:DR - Do you know of good automated tools to explore a dataset?

Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/scrips that could automatically round up the following answers for me:
Things I would be looking for:

datatype of a column?
dimension or measure?
histogram or count of values in a column
correlation between measures
unique key auto-find
import into a strongly typed data structure

Obviously I'm looking for something more than basic summary(cars). Something that goes through each column and spits out detailed analysis report

There is a very cool active Python package called pandas-profiling, is exactly what you want. With a simple pandas_profiling.ProfileReport(df) it returns a lot of important statistical information about your data, the official documentation says:

  For each column the following statistics - if relevant for the column
  type - are presented in an interactive HTML report:

  Essentials: type, unique values, missing values
  Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile    range
  Descriptive statistics like mean, mode, standard deviation,    sum, median absolute deviation, coefficient of variation, kurtosis,
  skewness Most frequent values
  Histogram
  Correlations highlighting of highly correlated variables, Spearman and Pearson matrixes

Look at their demo here.
I personally used it few times and it quite nice, BUT it dataset is large (mostly I mean in variable space) it takes a long time to give the statistics. I think there should be an option to even return all these statistics for a subset of features (columns) in case you do not want for all and it is very high dimensional!


Irregular time series classification I'm trying to classify about 5000 time series for patient admissions into two groups. which consists of large time gaps on nothing (months or years) followed by a short period of a significant number of data points, an admission. Each data point consists of a code (out of 6000 codes) or a code and a numerical value.

Not every time series contains every code, and the codes are quite sparse. I've tried to look at the distribution of values in each code, but got nothing. I've tried a naive Baysian approach of looking at the probabilities of codes in each set, after ignoring any values associated, and did a little better. I'm now out of ideas. I looked at RNNs, but thought the long gaps in data would affect their performance.

I looked at GLM's but the missing data and high dimensionality worried me.

If anyone could suggest a technique that might offer some traction, I would be very grateful.

If I understand correctly, a single patient would be one feature, so one column, and they had interations with hospitals over short periods; followed by no interations (and therefore no data) over longer periods.

Due to the sparsity of data along each single time-series, perhaps you could look into ways to encode the information a little more consicely. I think this approach may lead to fairly good results because you are only aiming to categorise the patients into one of two groups.

This could perhaps be done by converting your base data into more insightful statistics. For example, for each patient, you could compute some standard values:

Number of time periods with(-out) hospital interaction
Number of different codes (assuming the code refers to the ailment or something useful?)
Average length of gap between periods of activity
Average length of periods of activity

For the above, you'd need to probably heuristcally define what a "period of activity" is, e.g. 5 data points within 100 timesteps. Otherwise you're in a "gap" between such periods.

This would dramitically reduce the amount of datapoints and lend itself quite well and allow you to use simpler classification methods, such as GLMs, logistic regression (as you only have two classes) or perhaps a simple feed-forward neural network.

The idea is to extract as much of the temporal aspect from the data as possible during this feature engineering/pre-processing, such that formal time-series modelling would become unnecessary.

If you want to try normal time-series, such as regression models, you need to remember that they return a numerical value by default (regression problem versus classification), so you would need to decide on a threshold to put patients in one of the two groups.

Try trying to encode the missing data simply as a count of timesteps between datapoints would not work because of the fact that each patient has different timelines, so the resulting data would have strongly varying dimensions, making many models unusable.


Is it legal to scrape YouTube videos for training data? Is it legal, at least as long as I am not "selling" the video under my name, to scrape YouTube videos to train a neural network?

If it is not, is there a procedure to get permission for the above? I am in academia and require a massive amount of video data that YouTube would be the perfect source in.

Note: I couldn't figure out where to post this questions, as it is not technically technical. I am posting it here only because I believe someone here would know a bit or two. Would appreciate any suggestions regarding community this question might have a better chance in. What about Law?

It will depend on the rights of the videos themselves, although probably the terms of service of youtube wouldn't agree with it anyway.

But you have the YouTube 8M dataset, released by Google for research purposes.

  YouTube-8M is a large-scale labeled video dataset that consists of millions of YouTube video IDs and associated labels from a diverse vocabulary of 4700+ visual entities. It comes with precomputed state-of-the-art audio-visual features from billions of frames and audio segments, designed to fit on a single hard disk. This makes it possible to get started on this dataset by training a baseline video model in less than a day on a single machine! At the same time, the dataset's scale and diversity can enable deep exploration of complex audio-visual models that can take weeks to train even in a distributed fashion.

Edit: And Facebook and MIT just released SLAC (dataset to be released soon, as well as pretrained models for transfer learning).

  This project presents a novel video dataset, named SLAC (Sparsely Labeled ACtions), for action recognition and localization. It consists of over 520K untrimmed videos and 1.75M clip annotations spanning 200 action categories.