Same predictors in test set but I want different outputs I have a (training) dataset about what TV spectators are watching and for how long. The goal (at new set - the test set) is to predict for how long the TV spectators will watch a specific channel and show.

Specifically, I have the following predictors:

TV Channel (e.g. BBC, CNN etc)
Content (e.g. news, entertainment, business etc)
Starting time (e.g. 11:00, 14:00 etc)

and the following target:

End time (e.g. 12:00, 15:00 etc)

Obviously I am going to apply One Hot Encoding with the TV Channel and Content predictors and handle Starting time in a way (see more here: Encoding features like month and hour as categorial or numeric?).

However, in my training set I may have multiple observations with the same predictors' values (e.g. 'BBC', 'news', '20:00') but with different output. This is obviouly done because different users are starting to watch the same thing at the same time but they stop at different times.

Is this going to be a problem since also my test set includes observations like these?

Specifically, I do not want to receive the same output (end-time) for these observations but I want to receive different outputs which (ideally) follow the distribution of the respective observations in the training set. How can I achieve this?

Shall I simply add a new categorical variable for each user?

I hope I did not misunderstand your question, as otherwise, this may sound too trivial to you.

I don't think its per se a problem that the same X (predictors) values lead to different Y (end time). After all, if every combination of X would lead to a unique Y, what is the point of estimating a model? Think, of a simple algorithm may predict the majority, e.g. for the same 10 X, you may get

5 * Y1
3 * Y2
2 * Y3

so that your prediction is Y1 for this X.

That being said, I don't think your target variable needs to be categorical, I'd rather configure it continuous, the same with start time. And then, in the simple case, run a regression.

Further, I would not recommend grouping by the same X and add a categorical user type. This leads to a deterministic relationship (mapping), i.e.

User type 1 --> Y1
User type 2 --> Y2

because the way you derive user type is by finding the combination of X that leads to an unique Y.

I'd rather put this as a separate analysis, i.e. inferring user types from TV watching, where you basically cluster your data and say each cluster is a user type.

Edit

Based on your edited problem statement, recall that $Y$ is modelled conditionally on $X$, i.e.

$Y_i = E[Y_i|X_i] + \epsilon_i$.

So your estimated end time $\hat{Y}$ for a certain combination of $X$, e.g. $X_i = \{BBC, news, 20:00\}$, is the conditional expected value, i.e. mean estimate, but your predicted $\hat{Y_i}$ has a distribution (and if you would be more conservative, you could give a confidence interval for your prediction, which graphically speaking is a snippet from that distribution). This distribution is different for every $X_i$, see e.g. the figure in this doc

Now, to your request: what you could do is to sample from the distribution of $Y_i$ (the respective end time). Say you got 10 times the same $X_i = \{BBC, news, 20:00\}$, then sample 10 times from $Y_i$ and output that.

Yet, I don't think this to be the best solution. Graphically speaking again, I would provide the error bounds, i.e. as in the last graph here. In words, don't output 10 different predictions for the same input, but rather output that you predict the end time $Y_i$ for $X_i = \{BBC, news, 20:00\}$ is with 90% confidence between 20:55 and 21:10.


Does Non Negativity Constrains increases the estimation error I have been working with Tensor and matrix Non negative constrained algorithms. I have never seen a non negative constrained algorithm (ex. Non Negative Tucker Decomposition NTD) with error that is comparable to the non constrained algorithm (ex.High Order SVD HOSVD). Publications normally don't compare estimation error of unconstrained algorithms with constrained algorithms instead the comparison is always between different algorithms of same type.

By Testing, I found that the result of constrained algorithms with noise is around $10^{-7}$ compared to unconstrained ones is around $10^{-15}$.

Any reference that I can start from or any explanation would be appreciated.

Are you talking about train error or test error?
Non negativity constrains make your search space of possible solutions much smaller. This means that the train error would be always higher or equal when you add non-negativity constraint.
But higher train error can mean that you are overfitting. If the assumption of non-negativity make sense for the solution of your problem, then it is likely that this assumption would decrease the test error as you are limiting your function space and decrease the chance of overfitting.


When we should use binning to redure noise? or How we find out we have noise? I read several times which binning is helpful for reducing the noise of data. But how can we find out our data has noise? What if our data is clean and we reduce the accuracy of data?

Is there any method to measure noise of an attribute?
When should we do binning?

You can plot your datapoints and see if there are many outliers to the dataset. Making frequncy plots and heatmaps helps a lot in this case.Binning helps to remove smaller erroneous data points. There are other methods of reducing noise in a dataset like applying IQR.

If you want to reduce noise, apply Inter Quartile Range after applying binning only if you have a large dataset. : https://www.wikihow.com/Find-the-IQR


Quantify the amount of differences of one data set to another Basically I'm seeking the best method for my case scenario here. For example with the image shown below, the left one is poincare maps dataset #1, and the right is dataset #2.

What's the best method of quantifying the differences of the right one to the left? Standard deviation won't be of much help here, since these are two separate data sets. Appreciate any inputs, thanks in advance!

Just average (or pool, moree generally aggregate) some of the standard measures:

if you look closly python implementation is already general for some of them, meaning that you can apply them directly on dataset and not individually on the vectors

Examples: Cosine similarity, Eucledian distance, Manhattan/city-block distance, Chebychev distance, Minkowsky dustance. For computer vision, similarity measures are: Hausdorff Distance (the two data vectors will be considered close if every point of each set is close to some point of the other set), Bhattacharyya Distance (measure of similarity of two probabilty distributions), Bhattacharyya coefficient (measure of relative closeness of the two vectors) and Mahalanobis Distance (a specific case of Bhattacharyya Distance).


Can we use package "dplyr" on R base 3.0.2? Is there any way to use package dplyr on RStudio having R base 3.0.2 ?

I am not interested in plyr package.

dplyr 0.3 requires R 3.1+. If you're stuck on R 3.0.x, you have to use dplyr 0.2.x.


Algorithms and techniques for spell checking Can anyone suggest any algorithm and technique for spell checking?
After some googling, I found some interesting ones such as this one from Peter Norvig, http://norvig.com/spell-correct.html and few others. However, most of them were written many years ago. Therefore, I am trying to find out whether is there any newer/improved approach to tackle such problem.

Here is what I built...

Step 1: Store all the words in a Trie data structure. Wiki about
trie.
Step 2: Train an RNN or RNTN to get seq2seq mapping for words and
store the model
Step 3: Retrieve top n words with  levenshtein
distance with the.  Wiki about LD with the word that you are trying to
correct.

Naive Approach: Calculating the edit distance between the query term and every dictionary term. Very expensive.
Peter Norvig's Approach: Deriving all possible terms with an edit distance<=2 from the query term, and looking them up in the dictionary. Better, but still expensive (114,324 terms for word length=9 and edit distance=2) Check this.
Faroo's Approach: Deriving deletes only with an edit distance<=2 both from the query term and each dictionary term. Three orders of magnitudes faster. http://blog.faroo.com/2012/06/07...

Step 4: Based on the previous 2 or 3 or 4 words, predict the words
that are retrieved from Step 3 above. Select the number of words
depending on how much accuracy you want (Of course we want 100%
accuracy), and the processing power of the machine running the code,
you can select the number of words you wanna consider.

Check out this approach too. It's an acl paper from 2009. They call it - language independent auto correction.


The problem is the shape of y_train. You're defining an output layer of 2 neurons, and only passing labels with a shape of (n_samples, 1). You need to convert this to shape (n_samples, num_labels) and can use keras.utils.to_categorical to do so easily.

The number of neurons in your output layer has to match the second dimension of the label array, so you transform it from one dimensional like this:

[0,1,0,0,0,1]

To two dimensional like this:

[[1,0],
[0,1],
[1,0],
[1,0],
[1,0],
[0,1]]

When you make your predictions, softmax activation will give you a 'probability' for each class like so:

[[0.97,0.03],
[0.85,0.15],
...
]]


How to use pca results for linear regression I have a data set of 11 variables with allot of observations for each one. I want to make linear regression on the variables with the observed $\vec{y}=\alpha +\beta*\vec{X}$ when X is matrix. I'm trying to reduce my parameters so I activate pca algorithm on X. I get the "loading" data but i don't understand how to use it to get only four (for example) variables to estimate instead of 11.

somebody can help?

In general, I would suggest to use a regularization technique for reducing the dimensionality ofa data set in linear regression cases. Please refer to L1 regularization.

If you want to decrease the number variables using PCA, you should look at the lambda values that describe the variations in the principle components, then, select the a few components with the largest corresponding lambda values (eg the first four).

Note:

Do a scaling if necessary.
Sometimes the very first component is not very relevant and can be eliminated.


Question of using gradient descent instead of calculus. I checked previous questions there are still points to clarify First of all I checked http://stats.stackexchange.com/questions/23128/solving-for-regression-parameters-in-closed-form-vs-gradient-descent, http://stackoverflow.com/questions/26804656/why-do-we-use-gradient-descent-in-linear-regression, https://stats.stackexchange.com/questions/212619/why-is-gradient-descent-required but couldn't find my answer.

Gradient descent is: $w_{i}:=w_{i}-\alpha \frac{\delta }{\delta w_{i}}j(w)$ where w is a vector.

At his book "Pattern Recognition and Machine Learning" Bishop says:

"Because the error function is a quadratic function of
the coefficients w, its derivatives with respect to the coefficients will be linear in the
elements of w, and so the minimization of the error function has a unique solution..."

So if we take derivative of $j(w)$ with respect to $w_{i}$ and equal to zero, in the end it will give us the smallest $w$. Which is actually the first exercise.

In gradient descent we take the derivative too, so the problem can't be because of derivative. Such as an equation that its derivative can't be found. If we can find the answer with one iteration(making equation equal to zero for every feature) why do we iterate over and over again instead, which is the case of gradient descent.

If I understand correctly, your question is Why do we solve gradient descent problems iteratively rather than setting the derivative to 0 and solving for the weights which minimize the cost function?

We can't do this (in all but extremely simple problems such as linear regression) because there is no easy closed-form solution when you set the derivative equal to 0.

Let's take a look at a simple example: let's say you have a 1-layer neural net with no bias and a sigmoid activation function.  Your network output is $\hat{y} = \sigma(WX)$.  If you use L2 loss, your cost function would be $L = (\sigma(WX) - y)^2$.  (For simplicity we'll assume that we're only using a batch size of 1).

Let's take the derivative with respect to our weights $W$ and use the chain rule
:
$\begin{eqnarray*}
\frac{dL}{dW} &=& 2(\sigma(WX) - y)\frac{d}{dW}\sigma(WX)\\
&=& 2(\sigma(WX) - y)\sigma'(WX)\frac{d}{dW}(WX)\\
&=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))\frac{d}{dW}(WX)\\
&=& 2(\sigma(WX) - y)\sigma(WX)(1-\sigma(WX))X\\
\end{eqnarray*}$

At this point, we get a bit stuck.  If we set this derivative equal to 0, it still won't be easy to solve for $W$ since $W$ appears 3 times in this equation and can't easily be isolated by itself on one side of the equation.

Since we cannot solve for the cost-minimizing weights directly, instead we use gradient descent.  We take the derivative of the cost function to tell us which direction we should change each weight in order to most decrease the cost.  The actual size of the update is determined by the learning rate $\alpha$.


Alternative means of clustering streams of incoming facial recognition data I have a time-series dataset of incoming face data. Each data point is a facial-feature-vector of length 256 that represents the facial features of a person (it is generated by a modified RESNET). Features that are close together are deemed to belong to the same person.

I am (successfully) clustering the incoming face features by DBSCANing. I've recently switched to HDBSCAN also with good results.

My problem is this: DBSCAN and HDBSCAN require I have all the data together at one time. I often have >200,000 features which can be a very large download.

I would much prefer to be able to take every incoming f and assign it to a person without having to collect all the information at one time.

Is there an alternative to this (preferable with a Python implementation)?

You are describing incremental learning, input data is continuously used to extend the existing model's knowledge.
There is a Python implementation of incremental DBSCAN.
There is no current Python implementation of incremental HDBSCAN.


Understanding sklearn FeatureHasher Wanting to understand "the hashing trick" I've written the following test code:

import pandas as pd
from sklearn.feature_extraction import FeatureHasher
test = pd.DataFrame({'type': ['a', 'b', 'c', 'd', 'e','f','g','h']})
h = FeatureHasher(n_features=4, input_type='string')
f = h.transform(test.type)
print(f.toarray())

In the above example, I'm mapping 8 categories into 4 columns, and the output is:

[[ 0.  0.  1.  0.]<-a
 [ 0. -1.  0.  0.]<-b
 [ 0. -1.  0.  0.]<-c
 [ 0.  0.  0.  1.]<-d
 [ 0.  0.  0.  1.]<-e
 [ 0.  0.  0.  1.]<-f
 [ 0.  0. -1.  0.]<-g
 [ 0. -1.  0.  0.]]<-g

In the resulting matrix, I can see repetitions and some categories are represented the same way. Why is that? 8 categories can be mapped into 4 columns if I use a binary representation.

Can someone please explain the output of this technique and maybe elaborate a bit?

FeatureHasher assigns each token to a single column in the output; it does not do the sort of binary encoding that would allow you to faithfully encode more features to a smaller space.  Instead, the point is that you don't have to look up which column a given category maps to as you would in OneHotEncoder: you just have to apply the hash function to the value.  Using a lower-dimensional target space will guarantee collisions, but even a larger target space may produce a few collisions.


ML models: average of all versus average of averages? I have data for 20 different people and am training a model (e.g. a neural network with the same hyperparameters) on the data from each person; so this gives me 20 models.

I chose to use RMSE to assess the performance. However, since the training data is shuffled, the computed RMSE is nondeterministic and so oscillates. So I thought running each model 10 times and averaging the results, i.e. the RMSE's, would give me a better estimate of performance. But this is for a single person/model. How do I combine the performance of everything, i.e. all 20 models, into a single measure?

Run each of the 20 models 10 times for a total of 200 RMSE values, and average all? Or first compute the average for each person, and then the average of these averages?

Maybe a different method is better? The end goal is to compare a couple of models (e.g. NN vs SVM).

It shouldn't matter if you average all of them or take the average of the average for each person. It turns out that it will give you the same number. This is known as the law of total expectation. Essentially,

E(X)=E(E(X|Y))

Here's the wikipedia article for more details:

https://en.wikipedia.org/wiki/Law_of_total_expectation


Cluster documents based on topic similarity I have set of documents where I have assigned topics per each document.

E.g., Topics of document 1 -> 1.0 Science, 1.0 politics, 0.8 History, 0. 8 Information and Technology

Now I want to cluster these documents and find what are the documents that share similar like topics.

Can you please suggest/recommend suitable clustering technique for me to start with

PS: I am interested in something like this -> https://www.quora.com/How-do-I-cluster-documents-using-topic-models
However, the topics assigned for my documents varies

As @Emre suggested, if you already have the distribution of topics in each document, you can represent each document as a vector $x_d \in \mathbb{R}^N$, where $N$ is the number of the unique topics in your collection. For documents, not exhibiting specific topics just fill the specific cells in each feature vector with zeros. Then, you can use some clustering algorithm such as nearest neigbors, using those feature vectors.

Example usage code in python below:

import pandas as pd
import numpy as np
from sklearn.metrics import pairwise_distances

# Initialize some documents
doc1 = {'Science':0.7, 'History':0.05, 'Politics':0.15, 'Sports':0.1}
doc2 = {'News':0.3, 'Art':0.5, 'Politics':0.1, 'Sports':0.1}
doc3 = {'Science':0.8, 'History':0.1, 'Politics':0.05, 'News':0.1}
doc4 = {'Science':0.2, 'Weather':0.2, 'Art':0.6, 'Sports':0.1}
collection = [doc1, doc2, doc3, doc4]
df = pd.DataFrame(collection)
# Fill missing values with zeros
df.fillna(0, inplace=True)
# Get Feature Vectors
feature_matrix = df.as_matrix()

# Get cosine similarity (i.e. 1 - cosine_distance) between pairs
sims = 1-pairwise_distances(feature_matrix, metric='cosine')

# Get the ranking of the documents given document 0
# from most similar to least similar. Don't take into account
# the first document, because it will be the same that the query was about
ranking_1 = np.argsort(sims[0,:])[::-1][1:2]
print ranking_1

ranking_2 = np.argsort(sims[1,:])[::-1][1:2]
print ranking_2

This takes 4 documents with $N=7$ unique topics, fills missing values with zeros and creates a similarity matrix between all documents. Then querying for documents 1(Science=0.7) and 2(Art:0.5) the most similar other document in the collection, we surely get documents 3(Science:0.8) and 4(Art:0.6) correspondingly.

You can try more sophisticated approaches regarding clustering and other distance metrics.


Handling encoding of a dataset which has more than total 2000 columns Whenever we have a dataset to be pre processed, before feeding it to the model we convert the categorical values to numerical values for which we generally use LabelEncoding, One Hot encoding etc techniques but all these are done manually going through each column.
But what if are dataset is huge in terms of columns(eg : 2000 columns), here it wont be possible to go through each column manually, in such cases how do we handle encoding?
Are there any specific libraries available which deal with automatic encoding of variable? I know of category_encoders which provides with different encoding techniques but how do we do it at in the above mentioned condition.

To correctly encode your variable you have to know what those variables are about. An algorithm needs to somehow understand the type of your variable to automatically encode them. In such a case, you should have a dictionary for variables about the type of variable (sometimes, dataset guides or readme files, or some txt files contain that). Or you have to know that all the variables are monotype, so you can apply the same encoding. If you don't have these or similar information sources about the data, it is not possible (unless you have a perfect model to categorize the variables according to their type :)) ) to automatically encode them.
Although you cannot categorize the types of categorical variables automatically, it is possible to distinguish the continuous (and also discrete) and categorical variables. When I face a similar situation of having lots of variables, one of the very first things that I do is to have a count and percentage of distinct values for every variable. Thus, for example, if a variable with 200000 samples has ~154000 (unless there is a variable with 154000 categories, which is almost possible) distinct values, then it is a continuous (or discrete) variable. If a variable with 200000 samples has 13 distinct values then it is a categorical variable for sure. Using similar tricks you can identify categorical variables. However, thereafter, it is inevitable to analyze categorical variables one-by-one. When you categorize them within themselves e.g. rank variables, nominal variables, etc. you can altogether encode each variable type.


What does this expression from gradient descent mean? I am looking over some neural network theory and came across this equation, coupled with this description (gradient descent ball-valley analogy):

''let's think about what happens when we move the ball a small amount Δv1 in the v1 direction, and a small amount Δv2 in the v2 direction. Calculus tells us that C changes as follows:''

I don't understand where this expression has come from. It looks like the product rule? Could someone either explain what it means and/or state the area of calculus this belongs to, so I can read up on it? Many thanks!

After reading up on multivariate calculus, I have understood what the above equation means. It is a simplified version of the multivariate chain rule - a rule used to differentiate multivariate functions. Here is what it looks like:
$$\cfrac{dC}{dt}≈\cfrac{∂C}{∂v_1}  \cfrac{dv_1}{dt}+\cfrac{∂C}{∂v_2}  \cfrac{dv_2}{dt}$$

Time applies here because we are studying how the cost changes over time/over the course of training. Multiplying each side by $dt$ gives:
$$dC≈\cfrac{∂C}{∂v_1}  dv_1+\cfrac{∂C}{∂v_2}  dv_2$$

And since the differential operator $d$ means a 'small change', we can replace this with delta to represent the same thing, thus arriving at the equation from the question:
$$\Delta C≈\cfrac{∂C}{∂v_1}  \Delta v_1+\cfrac{∂C}{∂v_2}  \Delta v_2$$
To summarise, this equation is a simplified version of the multivariate chain rule, which although means the same as the original, does a better job in communicating that a 'small change' in $v_1$ and $v_2$ results in a 'small change' in $C$.


How to Identify p (lag order) for ARIMA Model in Python here is my auto correlation plot.
Generated by the following python code.

from pandas.plotting import autocorrelation_plot
from statsmodels.tsa.arima_model import ARIMA

def display_corelation(series):
    autocorrelation_plot(series)
    plt.show()

I know i can pass 1 or 2 in p by looking into the plot for the ARIMA Model.
My question is how i can generate the p value,

how to calculate the lag order from some series by pandas or any library instead of plotting currently at the moment ?

model = ARIMA(history, order=(1,1,0))

Got it by following code:

    import pandas as pd
    k=0
    highestCorr = 0
    for i in range(1,10):
        cor = pd.Series.autocorr(series, lag=i)
        if(cor > highestCorr):
            highestCorr = cor
            k=i


Help deciding the best statistic tool for my data I'm not from the data science area, and I need to perform some statistical analysis on my data, but I don't know which analytical tools I should use.
Number of variables - 6
Independent Variables 3 - (one is scalar (e.g., 2,4,6,8,10) and the other two are nominal (e.g., A,B,C and Lote 1, Lote 2, Lote 3, Lote 4, Lote 5)
Dependent Variables - 3 (all scalar variables and dependent on all three independent variables, they are basically performance metrics)
One question I would like to answer is how the dependent variables behave when the scalar independent variable increases, taking into account the nominal variables. The picture below shows a sample of my data.

You could start with plotting the relation between any relevant pair of variables, typically with a simple scatter plot and possibly using color to represent a third variable.
Pearson correlation coefficient is a simple but useful measure of association between two variables. By calculating it between two variables on the whole data and then on subsets based on specific conditions (e.g. third column is A) you can observe the effect of the condition, i.e. see if it increases or decreases the correlation level.


New parameters in final training I am training an Xgboost using 60% of my data and use 40% for testing.

In the 60% of data, I use 5-fold validation to find the best number of trees. I find that the optimal number of trees is around 150.

I evaluate my model in the 40% of the data that I have left and I am happy with the performance, so now I can deploy my model. However, before that, I want to do a final retrain to make use of all my data. I wonder if the number of trees obtained in cross-validation is going to be optimal when I train with the full dataset as well. My intuition says that I should use more than 150 trees as I have more data and I can overfit less.

Are there any sound decisions regarding the number of trees of the retrained model?

In my mind I have at least 3:

Use the full data to cross-validate and obtain the optimal number of trees (this is ok for sure but it is the slowest).
Use the same number of trees (this is the most conservative, we might be underfitting the data).
Use a heuristic, like final_trees = 150*100/60. I am very interested in a legit heuristic where I would not underfit and where I don't need to train models on cross-validation again.

Have you heard of any heuristic like that?

Note: this is not only specific to Xgboost, as any model with a parameter that controls regularization can also have the same issues.

Unpopular opinion: Second quickest way to overfit (next to data-leakage) is hyper-parameter optimization.

Why? You are assuming you wont have covariate-shift, while in most of the cases you can bet on it. Hence optimising too much on train (available data) will be ruin.

The most reasonable assumption (that we have to make sure it stands) is that 60% of the data is representative of the 40% and also approximately of the future unseen data hence all of the 150 trees should catch all of the neccesarry information.


Analysis of Real-Time Bidding I'm totally new to the topic of real-time bidding in which I know Machine Learning algorithms are used pretty often.

Can somebody explain me the system in a plain language i.e. a language for a non-technical person?

What is the bidding? Who bids on what? Where does Machine Learning get involve?

What is cookie matching mainly about?

A simplified example: suppose two companies: the webpage owner and a book store. And a customer named Jane, interested in reading.

The web page owner has some free space on its page, which can be sold for advertisement. The book store wants to place an advertisement on the same web page, in order to increase sales. Both companies meet eachother at an auction, where the web space is sold to the highest bidder in real-time.

So, the book store is bidding on the right to show an advertisement to Jane, who is visiting the page of the webpage owner.

The machine learning is done on the part of the book store, who receives information about the web page visitor. This can be all sort of information that the web page owner wants to release, and that could be of any use to the bookstore.

Based on this information, it is decided weather to make a bidding for the advertisement space or not, and the amount up to which the book store will want to make a bidding.

Without cookie matching, the webpage visitor Jane will probably not be identifiable by the bookstore, so the store must decide on bidding, based on parameters like geographical location of the customer, browser version (just to name a few ).

With cookie matching, each visitor/customer gets a unique identifier at the book store. Based on this identifier, the book store has more information of the visitor, like: what ads have been served before, and how long ago ? This visit to the web page can be linked to earlier visits and this information will likely ease the decision making process for doing the bidding on the advertisement space.

( There is more to it as there can be two more intermediary companies: one that holds the auction and one that delivers the ad )


How to interpret silouette coefficient? I'm trying to determine number of clusters for k-means using sklearn.metrics.silhouette_score. I have computed it for range(2,50) clusters. How to interpret this? What number of clusters should I choose?

They are all bad. A good Silhouette would be 0.7

Try other clustering algorithms instead.


Finding repeating string patterns in thousands of files I have files like the following: https://pastebin.com/5mkXY1aU

These are created by filling predefined forms so there are thousands of them that would match a pattern.

I will try to give more generic example for the sake of simplicity.

Let's say I have thousands of documents created using different patterns like `

- "sjkghkjfs <data> skjfs <data> kjskdfjsfkjs <data> sahkj";
- "tretyer erytewr fskjdf <data> trjk";
- "sdhfjsdhj <data> <data> <data> dsjadh";
- "<data> djfhsdk";` and so on.

<data> could be just any string.

I don't know what these patterns are beforehand. I want to find them out.

I can easily come up with some bruteforce solution but it would obviously not work on such amount of data.

It sounds to me like a problem that might have already been solved.

I wondered if there is some readily available software tool I could utilize directly to solve this or a library atleast ?

In case there isn't one how should I proceed to solve it in the most painless way ?

It is a one time task to find these patterns out, so I am not into the idea of spending days on implementing a solution from the scratch.

It looks like you want new word discover ?

Because thousand is not a big deal

Just build ngrams of filenames , count them would be fine .

You can use Trie to store string counts,  reduce memory cost , I can provide a dict way(in python):

from collections import defaultdict, Counter

# for memory effcient, you would need trie here
# t = Trie()
t = defaultdict(int)

filenames = ["sjkghkjfs <data> skjfs <data> kjskdfjsfkjs <data> sahkj",
            "tretyer erytewr fskjdf <data> trjk",
            "...."]

# some preprocess, tokenize to  sentence , to  words , filter useless one

def ngrams(s, start, end):
    for i in range(start, end+1):
        if len(s) > i:
            for j in range(i, len(s)):
                yield s[j:j+i]

# Suppose you need same string with length 4 ~ 6
for s in filenames:
    for word in ngrams(s, 4, 7):
        t[word]+=1

# Suppose you need most_common 5
print(Counter(t).most_common(5))

Further more

you can

tokenize the filenames at first.
Calculate tf-idf or word entropy to remove the useless words
then do something like above .

And

If you have a group of traget strings ,
word2vec may be a good tool, you can use it to search the strings in similar domain.


Classify if someone is home based on time I have a dataset with locations and a timestamp of a subject. For each location and timestamp I determined by comparing the location to the home address if the subject was at home or not (0/1) and added this value to the dataset.

Now, I want to train a model to learn based on the timestamp when it is most likely that the subject is at home. Thus, if you give the model some timestamp, it will classify if the subject was at home at this time. The model learns the "best time" for someone being at home so to say.

Obviously people are not at home at the same time every day but over a long period of time there should be some pattern and I want the model to classify based on this pattern.

What would be a fitting algorithm to do this?

This is an ideal case for feature engineering!

I did this same case for myself using the google takeaway data to predict whether I am at home or at work.

Instead of just using time I extracted the following features:

Work Day --> 1 / 0
Day of the Week
Month
Year
Time

I then trained a random forest classification model to tell me whether I am at home, at work or other place based on those five features.

As a successive step I used this model to actually identify dates where I "moved" or was "on holiday" because of the difference between prediction and actual labels.


I'm trying to do a time series model without a datetime field in python. Is this possible? I have a dataset with data like this:
Day       Revenue
1          1.2
2          1.5
3          1.1
4          1.34

I want to do a time series model on it, but am getting this error:
ValueError: view limit minimum -35.45 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units
When I plt, it assigns all of the date to 1/1/1970. I understand why, because it's not a date time field. Out of curiosity, I tried converting the Day column to a datetime, but it assigned every day to 1/1/70. Is there a way to either convert the column to a datetime field and have it assign a new date starting with a specific date (say 1/1/2017, 1/2/2017, etc) or is there a work around when you just have the day counts (1,2,3,4)?

You can use timedelta function from datetime to achieve this, starting from a known date.
Example code can be something like,
from datetime import datetime, timedelta

start = datetime.strptime('2021-01-01', '%Y-%m-%d')

all_dates = [start + timedelta(x) for x in range(10)]

print(all_dates)

Replace the range(10) above with the sequence of the actual days in your dataset.


Why is my neural net getting a poor accuracy? EPOCH=40
batch_size=50
mnist=input_data.read_data_sets("MNIST_data", one_hot=True)

tf.reset_default_graph()

input_X = tf.placeholder(tf.float32, shape=[None, 784])
input_y = tf.placeholder(tf.int64, shape=[None, 10])

input_layer=tf.layers.dense(input_X, 784, activation=tf.nn.sigmoid)
hidden1=tf.layers.dense(input_layer, 256, activation=tf.nn.sigmoid)
hidden2=tf.layers.dense(hidden1, 256, activation=tf.nn.sigmoid)
output=tf.layers.dense(hidden2, units=10)
output=tf.nn.softmax(output)

entropy=tf.nn.softmax_cross_entropy_with_logits(labels=input_y, logits=output)
loss=tf.reduce_mean(entropy)
step=tf.train.GradientDescentOptimizer(0.01).minimize(loss)

#correct=tf.nn.in_top_k(tf.argmax(output, y, 1) rank error

correct=tf.equal(tf.argmax(output, 1), tf.argmax(input_y, 1))
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

sess=tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for epoch in range(EPOCH):
  for i in range(1000):
    X, y=mnist.train.next_batch(batch_size)
    sess.run(loss, feed_dict={input_X:X, input_y:y})
  acc_train=accuracy.eval(feed_dict={input_X:X, input_y:y})
  acc_val=accuracy.eval(feed_dict={input_X: mnist.validation.images
                                  , input_y: mnist.validation.labels})
  print(epoch, "Train accuracy: ", acc_train, "\n Val accuracy: ", acc_val)

Results went something like this-

0 Train accuracy:  0.1
 Val accuracy:  0.0986
1 Train accuracy:  0.14
 Val accuracy:  0.0986
2 Train accuracy:  0.06
 Val accuracy:  0.0986
3 Train accuracy:  0.06
 Val accuracy:  0.0986

I have referenced part of the code from O'Reily textbook on Tensorflow.

EDIT- found this O' Reily guide online-https://github.com/ageron/handson-ml/blob/master/10_introduction_to_artificial_neural_networks.ipynb

They have a code similar to mine but their's works just fine.

Reason is you did not train the network.

You need to run the step op in your code. You defined it, but didn't use it anywhere later. Running the  loss only calculates the loss, but does not train your network.

after fixing your code (remove duplicate softmax, using adadelta, run step) you should get:


HOW TO: Deep Neural Network weight initialization Given difficult learning task (e.g high dimensionality, inherent data complexity) Deep Neural Networks become hard to train.
 To ease many of the problems one might:

Normalize && handpick quality data
choose a different training algorithm (e.g RMSprop instead of Gradient Descent)
pick a steeper gradient Cost function (e.g Cross Entropy instead of MSE)
Use different network structure (e.g Convolution layers instead of Feedforward)

I have heard that there are clever ways to initialize better weights. For example you can choose the magnitude better:
Glorot and Bengio (2010)

for sigmoid units: sample a Uniform(-r, r) with $r =
   \sqrt{\frac{6}{N_{in} + N_{out}}}$
or hyperbolic tangent units: sample a Uniform(-r, r) with $r =4
   \sqrt{\frac{6}{N_{in} + N_{out}}}$

Is there any consistent way of initializing the weights better?

As far as I know the two formulas you gave are pretty much the standard initialization. I had done a literature review a while ago, I copied it below if interested.

[1] addresses the question:

First, weights shouldn't be set to zeros in order to break the symmetry when backprogragating:

  Biases can generally be initialized to zero but weights need to be initialized carefully to break the symmetry between hidden units of the same layer. Because different output units receive different gradient signals, this symmetry breaking issue does not concern the output weights (into the output units), which can therefore also be set to zero.

Some initialization strategies:

[2] and [3] recommend scaling by the inverse of the square root of the fan-in
Glorot and Bengio (2010) and the Deep Learning Tutorials use a combination of the fan-in and fan-out:

for sigmoid units: sample a Uniform(-r, r) with $r=\sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).
for hyperbolic tangent units: sample a Uniform(-r, r) with $r=4 \sqrt{\frac{6}{\text{fan-in}+\text{fan-out}}}$ (fan-in is the number of inputs of the unit).

in the case of RBMs, a zero-mean Gaussian with a small standard deviation around 0.1 or 0.01 works well (Hinton, 2010) to initialize the weights.
Orthogonal random matrix initialization, i.e. W = np.random.randn(ndim, ndim); u, s, v = np.linalg.svd(W) then use u as your initialization matrix.

Also, unsupervised pre-training may help in some situations:

  An important choice is whether one should use
  unsupervised pre-training (and which unsupervised
  feature learning algorithm to use) in order
  to initialize parameters. In most settings
  we have found unsupervised pre-training to help
  and very rarely to hurt, but of course that
  implies additional training time and additional
  hyper-parameters.

Some ANN libraries also have some interesting lists, e.g. Lasagne:

Constant([val]) Initialize weights with constant value.
Normal([std, mean]) Sample initial weights from the Gaussian distribution.
Uniform([range, std, mean]) Sample initial weights from the uniform distribution.
Glorot(initializer[, gain, c01b])   Glorot weight initialization.
GlorotNormal([gain, c01b])  Glorot with weights sampled from the Normal distribution.
GlorotUniform([gain, c01b]) Glorot with weights sampled from the Uniform distribution.
He(initializer[, gain, c01b])   He weight initialization.
HeNormal([gain, c01b])  He initializer with weights sampled from the Normal distribution.
HeUniform([gain, c01b]) He initializer with weights sampled from the Uniform distribution.
Orthogonal([gain])  Intialize weights as Orthogonal matrix.
Sparse([sparsity, std]) Initialize weights as sparse matrix.

[1] Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

[2] LeCun, Y., Bottou, L., Orr, G. B., and Muller, K. (1998a). Efficient backprop. In Neural Networks, Tricks of the Trade.

[3] Glorot, Xavier, and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." International conference on artificial intelligence and statistics. 2010.


What is the appropriate statistical significance test for multi-class classification? I have a multi-class classification problem. I am primarily using macro-average F1 measure to evaluate the performance of models and want to verify if the results are statistically significant. I have the results of two classifiers on the same train/test-set (paired observations).
Some sources suggest to use McNemar’s test for binary classification task. However, is there any generalization of McNemar’s test for multi-class classification problem? If so, what would be the appropriate procedure to carry out these tests?

Generalisation of Mcnemars is called Cochran–Mantel–Haenszel test.
There is an implementation in R, but I suppose porting to Python should not be too hard. You can find the R version here.


Spark development on local machine with PyCharm I am wondering what is the best practice other devs are using for their python spark jobs.
I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to a standalone cluster and being able to run my code from PyCharm or at least from the command line onto the cluster. I am using Windows 8.

Any success stories with such set up? what is the recommended setup/practice?

In my opinion, I recommend to use the following approach to develop spark jobs for a big Data Context :

First, develop your spark in local mode on your computer. Use a simple subset of data from your final dataset. This way allows you to work with your favorite IDE (jupiter notebook, PyCharm...) and small computing latences. This is really suitable for spark job development.
Secondly, once your are done with dev in local mode, you can transfer your code on a cluster using SSH clients or whatever. You just have to change your spark context relative instructions and the input/output sources. Now test your code on a small dataset again and correct the few issues that could appear.
Finally, once your code is cluster suitable, you can now use it on your whole Big Data like dataset.

Note : On a Hadoop based cluster, I recommend to store your data on HDFS ORC format. You can then smoothly structure it with Hive and easily read or write these tables with Spark.

Note : Concerning the SSH clients, I recommend you using mobaXterm (a ssh based client for windows) to easily access cluster and drag-and-drop your code. To get some subsets of data for local mode, you can use "Toad for Hadoop" client which gives you the possibility  to run SQL-like queries directly on your hive tables and extract data in CSV or Excel format.

Note : Spark Standalone mode is not suitable for more that few machines. Prefer yarn or mesos clusters approaches for big datasets.


predicting probability distribution for time series I have time series of several variables. Just in one specific case one variable is linear combination of the rest.

I want to predict probability distribution (that is not only best estimate but estimates with probability of that happening) of future value of variables. I want to see when probability of small interval of possible outcome is high.

A priory I don't know rules of the game how variables evolve and inter-depend.

What is the tool to best do such prediction and how easy is it? Will scikit-learn do? Maybe neural networks?

ADDED based on answer:

As I've understood, time series theory data science assumes random walk, however I assume variables are at least partly moved by free will of players of the game under some a priory unknown to me constraints and goals.

Should statistical solution advised work in such case? Can data predict reversal of possible current trend?

Yes, you can use scikit-learn for this use case. You may find this tutorial useful.


Keras load pre-trained weights. Shape mismatch I have some trouble loading pre-trained weights with Keras.
Let's say I have a keras model model and that my weights are stored at my_weights.h5.

I try to load my weights as follow :

model.load_weights("my_weights.h5", by_name=True)

But this give me the following error :

Layer #1 (named "conv2d_1"), weight <tf.Variable 'conv2d_1/kernel:0' shape=(3, 3, 32, 64) dtype=float32> has shape (3, 3, 32, 64), but the saved weight has shape (32, 3, 3, 3).

So I tried to see what was the shape of my weights and my model structure :

for layer in model_body.layers :
    print(layer.name+" : input ("+str(layer.input_shape)+") output ("+str(layer.output_shape)+")")

print("__")
with h5py.File(weights_filepath, 'r') as f:
    for k in f.keys():
        for l in f[k].keys():
            for m in f[k][l].keys():
                print(k+ " : " + m + " : " + str(f[k][l][m].shape))

conv2d_1 : input ((None, None, None, 32)) output ((None, None, None, 64))
__
conv2d_1 : kernel:0 : (3, 3, 3, 32)

(I kept only the layer that appear in the error)

By seeing this, I don't understand why the shapes mismatch, and where the shape (3, 3, 32, 64) in the error come from). Am I missing something ?

I have solved the kind of issue as follows. Hope the solution would be helpful.
1. Delete "by_name=True"
# -model.load_weights(weights_path, by_name=True)
model.load_weights(weights_path)

2. Change the number of class
While it throws another throwing "ValueError: Shapes (1536, 1000) and (1536, 1001) are incompatible", I change num_classes from 1000 to 1001. And then it shows the correct model summary.
# -num_classes = 1000
num_classes = 1001


How design a autoencoder architecture I would like to build an autoencoder (CNN) to learn a representation of my data.

I never built such a network and I have some experience in supervised learning (classification).

I would like to know if some good practices in training a classifier is also right for an autoencoder:

Does reference architecture exists like ResNet/Inception or something? If not, should I design manually layers?
Does transfer learning/fine tuning works for autoencoder (or is it better to train from scratch)?

Yes, there are open source examples. Take a look at here and here.
About your second question, yes. There are numerous studies. For instance, take a look at Supervised Representation Learning: Transfer Learning with Deep Autoencoders.


Is Java or Python a better choice for an application involving data intensive algorithms employing natural language processing? I am a professional java developer with 9 years of experience. Never worked with Python but, I do not have any restraints learning it.

Still I need clarity on a couple of aspects. I hope if anyone can help, it will clear the cobwebs in my mind. I am looking to develop a machine learning project :

With a web interface to present interactive analytical graphs and views.
Employing a web crawler collecting data, both structured and unstructured.
A backend crunching the data collected using machine learning and artificial intelligence and natural language processing algorithms.

While exploring options available, I found both Python and Java have extensive support and frameworks/libraries/modules available to leverage machine learning implementations.

However, working with Java gives me a sense of strongly typed programming and it can leverage the underlying CPU cores via its robust multi-threading APIs. Python on the other hand is scripted language and on the face value looks to lesser efficient when working on complex logic crunching a huge amount of data pertaining to Natural Language Processing.

I might be utterly wrong as I am not very acquainted with Python ecosystem.

Can anyone help me understand on the choice between Java and Python if I keep in mind execution speed, maintainability and community support targeted toward development of a large, scalable and robust machine learning/artificially intelligent application ?

I work with python and java in big-data settings every day.

python is definitely my language of choice when researching, experimenting and fitting models.
python has a ton of very useful libraries such as spacy, nltk and sklearn that makes exploration very easy, especially in within a jupyter notebook.

When running the code in production, If performance is a consideration, we implement only the prediction phase in java.

java is more suited for production, has great support for packaging, tests, deployment.


How to determine number of leaves in decision tree analysis? Would be grateful if some expert on the forum can help me understand how to decide optimum number of leaves in a decision tree analysis.

I am using SAS and if I supply leaves=6 in my model then miss-classification rates for validation & training data sets are 18.6% & 18.8% respectively. And SAS lists 5 variables which are significant.

And if I don't supply leaves count in the code and let SAS decide it, then SAS after pruning takes 10 as leaves count and miss-classification rates for validation & training data sets are 17.5% & 16.9% respectively. And SAS lists 6 variables which are significant.

Now that the miss-classification rates have reduced & trees after pruning have increased from 4 to 10, is it a good thing or it indicates overfitting?

Looking forward to opinions of experts in this group.   Thanks

I'll assume that your test and validation datasets have been created appropriately (e.g. no observations are in both test and validation sets, both sets are of appropriate size, etc.).

Overfitting means that your model fits very well to your training data but does not generalise well on unseen data (i.e. will perform poorly on your validation dataset).

Your misclassification rate on the validation set (unseen data) is decreasing, and is therefore a good thing. However, if the misclassification rate on the validation set were to increase, that would indicate overfitting.


Regression - random error term When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where x is an input vector and $\varepsilon$ is the random error term.

Now, this random error term can have any probability distribution ?

$\varepsilon$ is referred as a noise term with 0 mean. The distribution is random in the real world, but you can make assumptions on its distribution.

For example the Gaussian Process machine learning suggests that it follows a Gaussian distribution i.e :  $\varepsilon \sim \mathcal{N}\left(0, \sigma^ 2 \right)$.

The variance $\sigma$ of the latter distribution can be seen as an hyperparameter that we obtaining by maximizing the likelihood function, or by having prior information on the data set.
You can find more information in this book Rasmussen, C. E., and C. K. Williams. "I (2006) Gaussian Processes for Machine Learning." (2006)

In some cases by looking at your a priori data, and estimating the possible error sources, you can a priori expect the type of the error distribution and try to assess this assumption a posteriori (mainly using data-driven method).


Initialize perceptron weights with zero I'm new to datascience so please just don't blast me.

In a text book i found:

  Now, the reason we don't initialize the weights to zero is that the
  learning rate (eta) only has an effect on the classification outcome
  if the weights are initialized to non-zero values. If all the weights
  are initialized to zero, the learning rate parameter eta affects only
  the scale of the weight vector, not the direction.

Now, why? Are you able to explain me?

If you choose zero initial weights, then the perceptron algorithm's learning rate $\eta$ has no influence on a neuron's predicted class label.

This is because the decision function used in the perceptron algorithm depends only on the sign of $z$:
$$
\phi(z) = \begin{cases} 1 &\text{ if } z \ge 0\\ -1 &\text{ otherwise.} \end{cases}
$$

Consider a perceptron algorithm with a single neuron.
(I will write ${\bf x} \cdot {\bf y}$ for the vector product ${\bf x}^T {\bf y}$ to avoid ugly double superscripts).

Pick an initial weight vector ${\bf w}^{(0)}.$
Plug in the first input vector ${\bf x^{(1)}}$ and predict the class label
$$
\hat{y}^{(1)}  = \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)}).
$$
This gives the weight update
$$
\Delta {\bf w}^{(1)} = \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)},
$$
where $\eta \in (0, 1)$ is the learning rate and $y^{(1)}$ is the true class label.
The new weights are ${\bf w}^{(1)} = {\bf w}^{(0)} + \Delta {\bf w}^{(1)}.$
Similarly, plug in our the input vector ${{\bf x}^{(2)}}$ and predict the class label
$$
\hat{y}^{(2)} = \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}).
$$
This gives the weight update
$$
\Delta {\bf w}^{(2)} = \eta (y^{(2)} - \hat{y}^{(2)}){\bf x}^{(2)},
$$
where $y^{(2)}$ is the true class label.

Notice that $\Delta{\bf w}^{(2)}$ implicitly depends on $\Delta{\bf w}^{(1)}$, which in turn implicitly depends on ${\bf w}^{(0)}$.
Let us unravel these dependencies by plugging in:
\begin{align*}
\Delta {\bf w}^{(2)} &= \eta \big(y^{(2)} - \hat{y}^{(2)}\big){\bf x}^{(2)} \\
&= \eta \big(y^{(2)} - \phi({\bf w}^{(1)} \cdot {\bf x}^{(2)}) \big) {\bf x}^{(2)} \\
&= \eta \big(y^{(2)} - \phi\big( ({\bf w}^{(0)} + \Delta {\bf w}^{(1)}) \cdot {\bf x}^{(2)} \big) \big) {\bf x}^{(2)}\\
&=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \hat{y}^{(1)}){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)} \\
&=  \eta \Big(y^{(2)} - \phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
\end{align*}

Now suppose we have initialized with zero weights: ${\bf w}^{(0)} = {\bf 0}.$
Since ${\bf 0} \cdot {\bf x}^{(1)} = 0$ and $\phi(0) = 1,$ the last line in this calculation simplifies to
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}
$$

Let us zoom in here and look at $\phi$ and its argument:
$$
\phi\Big(  \eta (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big).
$$
Since $\eta > 0$, it does not change the sign of $(y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)}.$ But the sign is all that matters to the function $\phi$.
So we can simply remove $\eta$ from the function argument without changing the result:
$$
\Delta {\bf w}^{(2)} = \eta \Big(y^{(2)} - \phi\Big( (y^{(1)} - 1){\bf x}^{(1)} \cdot {\bf x}^{(2)} \Big) \Big) {\bf x}^{(2)}.
$$

The same arguments hold for $\Delta {\bf w}^{(3)}, \Delta {\bf w}^{(4)}, \dots$

It follows that
$$\Delta {\bf w}^{(i)} = \eta * \text{[Something that does not depend on $\eta$]}$$
for every $i = 1, 2, \dots$
This is what your quote means by saying that the parameter $\eta$ "affects only the scale of the weight vector, not the direction."

For nonzero initial weights, this is different. Recall the last line of the long calculation $\Delta {\bf w}^{(2)} = \dots$ above and zoom in again on the function $\phi$ and its argument:
$$
\phi\Big( \big({\bf w}^{(0)} + \eta (y^{(1)} - \phi({\bf w}^{(0)} \cdot {\bf x}^{(1)})){\bf x}^{(1)}\big) \cdot {\bf x}^{(2)} \Big).
$$
If ${\bf w}^{(0)}$ is nonzero, then $\eta$ may affect the sign of the function argument, and therefore the predicted class label.


Sentiment Analysis: Train Separate Models or Use One for All We have been tasked with building an in-house sentiment tool and we are going to use it on a multitude of data sources; survey responses, reviews, social listening etc..

This may be obvious to some but I would like to hear some thoughts. So my question is, would you train separate models for each data source since the documents from each source would be quite different? We would then apply the appropriate model depending on the data source.

Or is it best practice to train one model using a sample of all the data sources and apply that model to everything?

I would suggest that you train a single model. The features correlated with positive or negative sentiment in a review are likely the same as those in a survey response. That is, “good” means good in both, “bad” means bad in both.

In general you should combine models and datasets when you think that the features in the different situations have the same meanings/interpretations. If they have contradictory or different meanings/interpretations, then you should consider training separate models.


Will I be qualified for data science roles with a PhD in Engineering and Machine Learning? Currently doing a funded PhD in Civil Engineering using Machine learning applications. In essence data driven modelling of a niche area of engineering (to do with some material deformation predictions etc.)

I have a years experience working as a Civil Engineer on site, and have a first class MEng in Civil & Structural Engineering.

I think I would really enjoy data science, and I am wondering if my PhD will provide a good stepping stone for this? And if so, what are some other things I can do to really make myself stand out as a data scientist?

Write blogs, papers, and so on. Take part in the open source community. This PhD does neither qualify you nor not. But when you are able to show your passion you have got a ticket.


Why does it has a constant val_loss:? I am working on somr dataset and am implementing a deep neural network. There are some typos that I am not familiar with. strong text

Change the last layer's Neuron count to 2.
model.add(keras.layers.Dense( 2, activation="softmax"))
OR
Change your last layer's activation to Sigmoid and keep y_train with single column
model.add(keras.layers.Dense( 1, activation="sigmoid"))


Book keeping of experiment runs and results I am a hands on researcher and I like testing out viable solutions, so I tend to run a lot of experiments. For example, if I am calculating a similarity score between documents, I might want to try out many measures. In fact, for each measure I might need to make several runs to test the effect of some parameters.

So far, I've been tracking the runs inputs and their results by writing out the results into  files with as much info about the inputs. The problem is that retrieving a specific result becomes a challenge sometimes, even if I try to add the input info to th filename. I tried using a spreadsheet with links to results but this isn't making a huge difference.

What tools/process do you use for the book keeping of your experiments?

I recently ran into a similar problem: How to manage extracting a variety of features from a large dataset, without knowing up front what all of them would be. (Even calculating mean values repeatedly would be computationally expensive.) Further, how would I manage predictions based on different feature sets? Meaning, if I added a new feature, how would I know which models to train on new features? It could quickly snowball into a huge mess.

My current solution is to track it all in a local NoSQL database (MongoDB). For example, I might have a collection features, each entry of which has a name, a description of how the feature was was calculated, the python file that ran the extraction, etc.

Likewise, a collection models includes models run on the data. Each entry might have a name, a list of features that were used to train the model, its eventual parameters, predicted values on a held-out test set, metrics for how the model performed, etc.

From my vantage point, this has a number of benefits:

By saving predictions, I can use them later in ensemble predictions.
Because I keep track of which features were used, I know which ones need retraining as I extract more features.
By saving model descriptions, I ensure that I always know what I've tried. I never have to wonder, "Have I tried LASSO with regularization parameters set by grid-search CV?" I can always look it up, and see how successful it was.

From your question, it sounds like you could adapt this approach to your problem's workflow. Install Mongo or another database of choice, and then save each experimental run, its inputs, its results, and anything else you might wish to track over the course of the project. This should be much easier to query than a spreadsheet, at the least.


How to teach neural network a policy for a board game using reinforcement learning? I need to use reinforcement learning to teach a neural net a policy for a board game. I chose Q-learining as the specific alghoritm.

I'd like a neural net to have the following structure:

layer - rows * cols + 1 neurons - input - values of consecutive fields on the board (0 for empty, 1 or 2 representing a player), action (natural number) in that state
layer - (??) neurons - hidden
layer - 1 neuron - output - value of action in given state (float)

My first idea was to begin with creating a map of states, actions and values, and then try to teach neural network. If the teaching process would not succeed I could increase the number of neurons and start over.

However, I quickly ran into performance problems. Firstly, I needed to switch from simple in-memory Python dict to a database (not enough RAM). Now the database seems to be a bottleneck (simply speaking there are so many possible states that the retrieval of actions' values is taking a noticeable amount of time). Calculations would take weeks.

I guess it would be possible to teach neural network on the fly, without the layer of a map in the middle. But how would I choose a right number of neurons on the hidden layer? How could I figure out that I'm loosing periously saved (learned) data?

You need to use some function approximation scheme. In addition, experience replay would be useful for two reasons: (1) you want to keep past memories (2) you need to decorrelate the way to teach your network.

Have a look at Deepmind's DQN on ATARI games. What you are describing is basically what they have solved. The paper is in their website: http://deepmind.com/dqn.html

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, et al. Human-level control through deep reinforcement learning. Nature. 2015 Feb 25;518(7540):529–33.

With respect to the network architecture, it will definitely require some experimentation. For an alternative, you can also have a look at HyperNEAT. They evolve the network topology:

Hausknecht M, Khandelwal P, Miikkulainen R, Stone P. HyperNEAT-GGP: A HyperNEAT-based Atari general game player. In: Proceedings of the 14th annual conference on Genetic and evolutionary computation p. 217–24. Available from: http://dl.acm.org/citation.cfm?id=2330195

For more strategic games. Maybe you can have a look at "Giraffe: Using Deep Reinforcement Learning to Play Chess" http://arxiv.org/abs/1509.01549


Good accuracy on train dataset with cross validation, but low accuracy on test dataset I'm using the adult dataset from UCI and trying to predict the native country of a person, based on workclass, education.num, race, gender and income. I'm using sklearn and KNeighborsClassifier.

With this, predicting the test split I get 0.917252 accuracy,  with K = 15. But, when I use the test dataset (given in the UCI link), that accuracy goes to less than 1%.

Where am I going wrong?

train = pd.read_csv('datasets/adult_data/adult.csv')
test = pd.read_csv('datasets/adult_data/adulttest.csv')

# Cleaning null values
train = train[train["workclass"] != " ?"]
train = train[train["occupation"] != " ?"]
train = train[train["native.country"] != " ?"]
test = test[test["workclass"] != " ?"]
test = test[test["occupation"] != " ?"]
test = test[test["native.country"] != " ?"]

category_col =['workclass', 'race', 'education','marital.status', 'occupation',
               'relationship', 'gender', 'native.country', 'income']
for col in category_col:
    b, c = np.unique(train[col], return_inverse=True)
    train[col] = c

for col in category_col:
    b, c = np.unique(test[col], return_inverse=True)
    test[col] = c

features = train.drop('native.country',axis=1)
target = train["native.country"]
features_test = test.drop('native.country', axis=1)
features_target = test["native.country"]

features_name = ['workclass', 'education.num', 'race', 'gender', 'income']

features = features[features_name]
features_test = features_test[features_name]

#spliting data into train and test data
X_train,X_test,y_train,y_test = train_test_split(features,target,random_state = 12)

k_values = np.arange(1,26)
scores = []

for i in k_values:
    clf = KNeighborsClassifier(n_neighbors=i)
    clf.fit(X_train,y_train)

    y_predict = clf.predict(features_test)
    # y_predict = clf.predict(X_train)

    scores.append(metrics.accuracy_score(features_target, y_predict))
    # scores.append(metrics.accuracy_score(y_train, y_predict))

print("Accuracy for {} is {}".format(np.argmax(scores),max(scores)))

plt.plot(np.arange(1,26),scores)
plt.title('Variation of accuracy with K value, with all features')
plt.xlabel('K values')
plt.ylabel('Accuracy')

plt.show()

I think you are using commented lines in the loop to calculate the cross validation accuracy.

# y_predict = clf.predict(X_train)

It should be

 y_predict = clf.predict(X_test)

And then you should check accuracy score with 'y_test' not 'y_train'.

You are predicting on the data KNN has learnt on. And since KNN algorithms kind of learns the data. You are getting good accuracy.


Training Machine Learning in Production Is there a way to train your machine learning model in the cloud? Or does it really have to be batch training? i.e. (Pull some data on SQL, then feed that to the model)

What i was thinking is implementing my own model from scratch, use Stochastic Gradient Descent to update the parameters for every row from the database.

There is a field of online machine learning where incremental updates can be made to a model, including a variation of stochastic gradient descent called incremental stochastic gradient descent.

Generally, several rows of data are batched together to perform an update to the parameters. Training on a single row of data at a time can lead to unstable updates.


Fisher Scoring v/s Coordinate Descent for MLE in R R base function glm() uses Fishers Scoring for MLE, while the glmnet appears to use the coordinate descent method to solve the same equation. Coordinate descent is more time-efficient than Fisher Scoring, as Fisher Scoring calculates the second order derivative matrix, in addition to some other matrix operations. which makes expensive to perform, while coordinate descent can do the same task in O(np) time.

Why would R base function use Fisher Scoring? Does this method have an advantage over other optimization methods? How does coordinate descent and Fisher Scoring compare? I am relatively new to do this field so any help or resource will be helpful.

The only way to be sure is by benchmarking, but for glm Fisher scoring should be faster than coordinate descent.  Fisher scoring is a special case of Newton Raphson, which has a faster rate of convergence than coordinate descent (Newton-Raphson is quadratically convergent, while coordinate descent is linearly convergent.)  So while the computation of second-derivative information means each step takes more time, it can require many fewer steps than coordinate descent.

For the lasso, the special form of the penalty term makes it a very special case (and in fact absolute value isn't differentiable anyway, though sometimes you can finesse this).  For this special problem, coordinate descent proves to be particularly fast.  There are many other optimization problems where in practice Newton-Raphson is faster.


GAN Loss Function Notation Clarification In the Generative Adversarial Network loss function, what do these mean?: $E_{x~p_{data}(x)}$ and $E_{z~p_{z}(z)}$ and how are they used in this context?

Generally, the notation $\mathbb{E}_{x\sim p}f(x)$ or $\mathbb{E}_{x\sim p(x)}f(x)$ refers to the expectation of $f(x)$ with respect to the distribution $p$ for variable $x$ (e.g. see the explanation in the notation section in "Pattern Recognition and Machine Learning" by Bishop).

In the context of GANs, it means that for an objective function such as
$$\mathbb{E}_{x\sim p_{data}(x)}[logD(x)] + \mathbb{E}_{z\sim p_z(z)}[log(1 - D(G(z)))]$$
the first summand is the expectation with regards to $x$ coming from the data and the second summand is the expectation with regards to $z$ which you sample from $p_z$ as an input for G.

Since the first summand stands for correctly classified real data (coming from $p_{data}$) and the second summand stands for correctly as fake classified images coming from G using $z$ sampled from $p_z$, D tries to maximize this expression. And G does the opposite.


Should I apply PCA on the entire dataset or just the nominal values? I have a data-set with 14~ attributes, roughly half of them nominal. I've used a binary vectorizer to convert these values to a number of attributes. The number of attributes, naturally, ballooned up; I'm sitting on roughly 50 at the moment. I've looked at using PCA to reduce this number.

As far as I can understand the things I've been reading, I should exclude my target variable from the analysis. But I'm not sure if I should perform PCA on the whole remaining data set (Including the values that were already numerical, like 'age') or just on the values I converted from Nominal to Numerical and then re-add those to the already numeric values.

To clarify, I've already converted this data-set from nominal to binary, and I'm not sure if I should apply PCA to just the binary columns generated, or the entire thing.

It is not advised for you to apply PCA on a dataset with nominal values. You can, but PCA translates variables in space. It is hard to find the relationship between nominal values in space. For example, how would one quantify the space between 'male' and 'female' or 'white' and 'red' or 'PC' and 'mobile phone'?

Some alternatives are

Use a tree model, like random forest, which will easily handle nominal values.
Use FactoMine in R.


dataset split for image classification I am trying to do image classification for 14 categories (around 1000 images for each cat). And i initially created two folders for training and validation. In this case, do I still need to set a validation split or a subset in a code? or I can use the whole files as train_ds and val_ds by deleting them
Folder names in the training and validation directory are same.
data_dir = 'trainingdatav1'
data_val = 'Validationv1'

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
                                                                data_dir,
                                                                validation_split=0.1, #is it required if I'm gonna use the whole folders and files for training?
                                                                subset="training",
                                                                seed=123,
                                                                image_size=(img_height, img_width),
                                                                batch_size=batch_size)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
                                                              data_val,
                                                              validation_split=0.8, #need to check
                                                              subset="validation",
                                                              seed=455,
                                                              image_size=(img_height, img_width),
                                                              batch_size=batch_size)

num_classes = 14

model = tf.keras.Sequential([
  layers.experimental.preprocessing.Rescaling(1./255, input_shape=(img_height, img_width, 3)),

  layers.Conv2D(16, 3, padding='same', activation='softmax'),
  layers.MaxPooling2D(),

  layers.Conv2D(32, 3, padding='same', activation='relu'),  #from renu
  layers.MaxPooling2D(),

  layers.Conv2D(64, 3, padding='same', activation='relu'),
  layers.MaxPooling2D(),
  layers.Dropout(.2),             #prevent overfitting

  layers.Flatten(),
  layers.Dense(128, activation='sigmoid'),
  layers.Dense(num_classes)
])

model.compile(optimizer='SGD', #adam
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])
model.summary()

epochs=50
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

Another question is the overfitting issue - validation accuracy is not over 0.4 and val_loss is around 2.xxx. Suggestions from Stacexchange are:

Reduce the layers of the neural network.
Reduce the number of neurons in each layer of the network to reduce the number of parameters.
Add dropout and tune its rate.
Use L2 normalisation on the parameter weights and tune the lambda value.
If possible add more data for training.

Are there any other suggestions?

If you have already split your training and validation sets into separate directories then there is no need to technically do the splitting in your code. However, the problem with a pre-defined validation set is that it can lead to overfitting more easily: the primary purpose of a validation set is to detect overfitting and if you keep tuning your hyperparameters for your model using a fixed validation set every time you train, then your model hyperparameters may be overfitting to that specific validation set.
Another problem with a fixed validation set is that it prevents you from exploiting approaches such as K-fold cross validation, where you split your data into K groups and each each group takes its turn as a validation set during training (prevents hyperparameter overfitting to a specific validation set as discussed above)...
With regard to your question on overfitting, your list is very comprehensive. However, I would add Early Stopping to that list - this you may already be doing - but good to mention explicitly.


Model selection metric for validation phase in deep learning I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This selection of parameters is done using the same loss function that we use for training, but applied to validation data instead of training data.

My question is, does the metric used in the validation phase really have to be the same loss function, or can it be some other metric, such as accuracy or average precision?

It doesn't have to be the same function and usually is not.

The point of the validation set is to measure how well our model is actually doing. This is only useful if we measure it on a metric that has some value to us.
For instance, in classification, no one is concerned about which model achieves the least cross-entropy, rather which one achieves the highest accuracy, which makes more sense from a business standpoint.

One clarification I'd like to make on your initial statement that doesn't have to do with the question itself.

  I have been taught that for each epoch in training, we perform a training phase, and then a validation phase where we decide whether the new set of parameters is better than the current best. This selection of parameters is done using the same loss function that we use for training, but applied to validation data instead of training data.

This is incorrect! An epoch is technically the point at which the model has seen the training dataset once. After the end of the epoch, we usually do a pass on the validation set to measure how well our model is actually doing. However, there is no hyperparameter tuning performed at this step, as the model hasn't been fully trained yet. A model typically takes multiple epochs to train, after which we can change the hyperparameters and start again. The only change we might make to the hyperparameters at the end of an epoch is something like a scheduled learning rate reduction.


LSTM number of units for first layer I'm trying to use LSTM (with Keras) for a time series problem. I would like predict the next value of the time series given its previous value. I'm using TimeseriesGenerator to create the training data as follows (setting length equal to 1 indicating the prediction is based on the previous value):

generator = TimeseriesGenerator(series, series, length=1, batch_size=10)

For the modeling I'm using the following:

model = Sequential()

model.add(LSTM(num_units, activation='relu', input_shape=(n_input, n_features)))

model.add(Dense(1))

model.compile(optimizer='adam', loss='mse')

model.fit_generator(generator, steps_per_epoch=len(generator)/n_batch, epochs=50)

for prediction:

yhat = model.predict(x_input, verbose=0)

when I set num_units = 1 (in the the first layer), the predicted values are much lower than the typical values in the time series (the typical values in the time series are 30-50 and the prediction is around 0.4). However, when I set num_units = 700, the predicted values become very close to the test values and the predictions seem to be overfitting. Why is that? What does the num_units in the first layer intuitively represent? If our input data is just one number, what does it mean to have a layer with 700 units? What is the intuition behind mapping a number to 700 neurons and what does one gain from it? In general, if in our time series, we're trying to make a prediction from the past n observations, how many units should be in the first LSTM layer?

num units is the number of hidden units in each time-step of the LSTM cell's representation of your data- you can visualize this as a several-layer-deep fully connected sequence of layers in which each layer also has a connection to a memory across the layers,even though that a analogy isn't 100% perfect. num units, then, is the number of units in each of those layers.

For your specific problem, and with length = 1, this reduces to a single layer- your model is not taking advantage of the memory capabilities of LSTM because there's simply nothing to remember beyond a single time step, because there's only a single time-step.

As to why it seems to perform better with a higher num units-- your model doesn't have enough expressive power at num units = 1 to describe the function it's trying to- there is too much information to encode in only a single unit. At num units = 700, I'd be very curious as to what the performance of the model is on out-of-sample data.

It's very likely that your network is simply over fitting, or memorizing your training data, with that much expressive power. If you can verify that that's happening- you'd see very good performance on the training data and poor performance on out-of-sample data- then you'll need to regularize the network in some way. Common ways to do this include implementing early stopping, introducing dropout layers, and/or reducing the number of hidden units in your layers.

Overall, if you don't have more than one time-step observation for a single entity, I would suggest that you change the LSTM layer to a simple fully connected layer to simplify the architecture. If you do or can get those sequential observations, then an LSTM layer might be a good solution for you.


How is bayesian risk computed to prune decision trees? I've been trying to follow this paper on Bayesian Risk Pruning. I'm not very familiar with this type of pruning, but I'm wondering a few things:
(1) The paper describes risk-rates to be defined per example. We have $R_k(a_i|x)=\sum\limits_{j=1,j \neq i}^{T_c} L_k(a_i|C_j)p_k(C_j|x)$. $L_k(a_i|C_j)$ is defined to be the loss of an example being predicted in class $C_j$ when the true class is $C_i$. $p_k(C_j|x)$ is the estimated probability of an example belonging to $C_j$.

Above is the decision tree that was produced from a C4.5 algorithm. Pruning occurs from left to right, bottom-up. My main question: How are the risk-rates found in the decision tree such as for Node 3?
(2) There's also conflicting statements here:

The first image states that if parent risk-rate exceeds the total risk-rate of the leaves, then the parent is pruned to a leaf. However, the second claims that pruning occurs if the leaf risk-rate exceeds the parent. To confirm, if the risk-rate of the parent is less than the risk-rate of the leaves under the subtree of the parent, then I would set the parent to be a leaf?
(3) From (1), loss would be 0-1 in the binary case. What could be a reasonable loss for multi-class output?
(4) From (1), would estimated probability of $C_j$ be the proportion of $C_j$ in the partitioned output class at a node? For instance, at node 3, we're looking at output = [No].
(5) From (1), would risk-rate be over all training examples?

I'm going to try to answer my question. To (4), I would say yes. For (5), I believe that for each node, there's a specific partition that falls into the node when following the branches down the decision tree. For instance, for Node 2, there are 3 instances (where a=1) that results in [Yes, No, No] for the partitioned output class at Node 2. For (1), I would calculate the risks over each of these relevant examples and sum them. Then at node 2: $\frac{2}{3}(1)+\frac{1}{3}(1)+\frac{1}{3}(1) = \frac{4}{3}$ is the bayes risk where I use 0-1 loss. Then I would use these estimated risks in the pruning algorithm. Can anyone corroborate?
(2) was answered in the comments above, and I'm assuming for (3) that cross-entropy can be used for multi-class loss.


Image Matching for similar products Is it technically possible to find out similar products online based on an given image?

Say the first link is a hoodie for women selling on Amazon. I want to do two things.

Based on the image, I want to generate features of the hoodie in addition to the ones provided from the product specs(color, size). For example, a logo in the middle and large pocket in the front
I want to find out the similar product(second link) based on the image(first link). Is it feasible to find similar images with at least this degree of matching(the more the better)  from a list of images? I will assume it's a failed match if finds me a women's hoodie with a big pocket in front but no logo.

https://www.amazon.com/GAP-Girls-Fleece-Pullover-Hoodie/dp/B075YPTLTQ/ref=sr_1_1_sspa?keywords=gap&qid=1552274273&s=gateway&sr=8-1-spons&psc=1

https://www.amazon.com/dp/B07F65W746/ref=twister_B07F62RWZL?th=1

Yes,  it is technically possibly. The easiest algorithm you can try out is siamese networks. Basically siamese networks try to learn a similarity measurement of two inputs (images, sounds, whatever) and work quite well for problems with little labeled data.
You can also find hands-on tutorials guiding you through the details like tutorial 1 or tutorial 2.

For more advanced check out prototypical networks.


Human Height Estimation using walking stride Are there any papers or research showing a correlation between walking stride and human height? My purpose is to estimate height from walking stride of a person.

I did not find scientific data, but interesting ratio is mentioned on several sports sites like this:

Women have ratio 0.413 and men 0.415 and it is used by multiplying height with ratio and we get the stride length.

My lenght 182cm,
ratio 0.415
stride length 182 x 0.415 = 75.53cm


How to ignore vectors of zeros (i.e. paddings) in Keras? I'm implementing a LSTM model with Keras. My dataset is composed by words and each word is an 837 long vector. I grouped the words in groups of 20 and to do this I padded them: initially I had groups of words of variable length and the maximum group length that I found was 20, this is why I padded all groups to 20.

For example, a group of 5 words is:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837]]

where xi is the i-th feature of the vector.

To pad this group to a length of 20, I added 15 vectors composed by 837 feature with value equal to zeros:

[[0.......0],
............
............
[0........0]]

So, at the end, my group is of the form:

[[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[x1,x2....x837],
[0...........0],
..............
..............
[0...........0]]

How could I ignore the vectors of zeros during training?

You can use Masking layer (with mask value of zero) before LSTM layer in order to ignore all timesteps with only zeros (i.e. zeros vector). You can find more information about this layer on its documentation. Here is an example from documentation which uses Masking layer:

  Consider a Numpy data array x of shape (samples, timesteps, features),
  to be fed to an LSTM layer. You want to mask sample #0 at timestep #3,
  and sample #2 at timestep #5, because you lack features for these
  sample timesteps. You can do:

  set x[0, 3, :] = 0. and x[2, 5, :] = 0.
  insert a Masking layer with mask_value=0. before the LSTM layer:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))


How to connect data-mining with machine learner process I want to write a data-mining service in Google Go which collects data through scraping and APIs.

However as Go lacks good ML support I would like to do the ML stuff in Python.

Having a web background I would connect both services with something like RPC but as I believe that this is a common problem in data science I think that there is some better solution.

For example most (web) protocols lack at:

buffering between processes
clustering over multiple instances

So what (type of libraries) do data scientists use to connect different languages/processes?

Bodo

The Data Science Toolkit is a powerful library (or collection of libraries, technically) which are available in a number of languages. For instance, I use the implementation called RDSTK in R.

In the case of your preferred language, Google Go, there's a list of web-related libraries here which looks very useful.


Neural Networks overfitting How the neural networks are overfitted for regression.Either it tries

to equal individual observation values or equals to the sum of all observations

Neural Networks basically act as a high memory-based machine learning algorithm. So for a given dataset the chance of it perfectly aligned with all the data at a given instance is high, as it most likely just ends up remembering every data point you give.

Overfitting occurs precisely because of this, when a new expansive data set is introduced there is no way it can adjust its fit to the new data, graphically it ends up missing more of the values than its supposed to fit.

In conclusion, it does not work well in the case where the scoring population is significantly different compared to training sample.


Is there a case that random forest data (a bunch of trees) consumes MB or GB of memory? Is there any case that random forest (a bunch of trees) consumes too much memory in practice?
I'm wondering why my scikit-learn consumes large amount of memory.

In theory, it could consume any amount of memory.
You could use an infinite amount of trees.


What does a conservative technique mean in the context of neural networks? I am reading this article on Layer-wise relevance propagation method and I can't understand this particular paragraph

  LRP is a conservative technique, meaning the magnitude of any output y is conserved through the backpropagation process and is equal to the sum of the relevance map R of the input layer. This property holds for any consecutive layers j and k, and by transitivity for the input and output layer.

What does this even mean? I can't understand what the author is trying to say. I understand what backpropagation is but not what is being said related to backpropagation!

In this layered graph structure, the
relevance conservation property can be formulated as follows: Let
j and k be indices for neurons of two successive layers. Let Rk be
the relevance of neuron k for the prediction f (x). We define R j←k
as the share of Rk that is redistributed to neuron j in the lower
layer. The conservation property for this neuron imposes
∑j R j←k = Rk.
Likewise, neurons in the lower layer aggregate all relevance coming
from the neurons from the higher layer:
Rj = ∑k R j←k
These two equations, when combined, also ensure a relevance conservation property between layers
Refer to this paper Methods for interpreting and understanding deep neural networks by Grégoire Montavon for more details


Cross validation plus oversampling? I am quite new to machine learning and python as well. I faced an imbalanced dataset and wanna use cross validation and oversamopling like the figure shown.

I realised the Python function below cannot be directly used for this purpose and please advice some codes for this task.

cross_val_score(model, X_train,np.ravel(y_train), cv=n_folds, n_jobs=1, scoring='roc_auc')

Adding to what Himanshu Rai said, you should be careful not to oversample before the StratifiedKFold, once you risk to put the same sample in both the training and testing folds check this, (where you took your image from) and that doesn't really evaluate your model's capacity to data that it never saw. What i did was to use StratifiedKFold, and then oversample or SMOTE (or whatever you want) all the training folds separately, by only looking at that fold's training data. Then use all those folds to validate your models.


How to find the residuals of a classification tree in xgboost So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on the residuals of the tree from the last round?

Say I have the following iris dataset:

And I get 3 error after running a classification tree. Then how do I fit another classification tree based on what I got wrong?

And after then, how would I weight them to form an ensemble?

There are 2 methods to "continue a model training":

Training from another xgboost model

If you have a pre-trained xgboost model, you can continue training using the xgb_model parameter in xgb.train.

  xgb_model  a previously built model to continue the training from.
  Could be either an object of class xgb.Booster, or its raw data, or
  the name of a file with a previously saved model.

Training residual

Using xgboost to training a residual or prediction of a model, you can following the demo provided in xgboost:
https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R

Simply speaking, you can use the "base_margin" element when preparing data.


What is the loss function defined by Mnih and Hinton in their paper “Learning to Label Aerial Images from Noisy Data”? In section 3.3 of the paper, they state that they use the cross entropy.
Then they define the probability for a label to be a false positive as $\theta_0$ and a false negative as $\theta_1$.

They use it to somehow modify the loss function but never actually state what this new loss is.

I was expecting something like

$f_\theta(\hat{m_i}, \tilde{m_i}) = \tilde{m_i} *g_{\theta_0}(\hat{m_i}) + (1-\tilde{m_i})*g_{\theta_1}(1-\hat{m_i})$

but $g$ is not given.

Section 3.3 simply gives the equation for the negative log-likelihood.
They say that it takes the form of cross-entropy (because it just looks like a cross-entropy equation, perhaps?), but mathematically it seems to come from the fact that they define the model in equation 2 to follow a Bernoulli distribution, which can be 0 or 1 with probabilities p or q respectively:

$$
\Pr(X=1)=p=1-\Pr(X=0)=1-q
$$

The probability mass function of the Bernoulli looks like this:

$$
f(x;p)=px+(1-p)(1-x)\!\quad {\text{for }}x\in \{0,1\}
$$

and the maximum likelihood for the Bernoulli looks like this:

$$
L(p) = \prod_{i+1}^n p^{x_1} (1 - p)^{1 - x_1}
$$

Taking the log (to get the log-likelihood that they mention), gives something of the form we see in equation 3:

$$
\log{L(p)} = \log{p}\sum_{i=1}^n x_i + \log{(1-p)}\sum_{i=1}^n (1-x_i)
$$

Then you can takes the sum out the front as it is over the image patches:

$$
\sum_{i}^{w^2_m} (\tilde{m}_i ln\hat{m}_i + (1-\tilde{m}_i)ln(1- \hat{m}_i))
$$
So here
This sums over all the pixel-wise output units in each image patch (map: m), hence why the summation goes to $w_m^2$; the width of the image patch squared. $\tilde{m}$ is a realisation (possible outcome) of $\hat{m}$, just as $x$ is a realisation of distribution $p$ in the above equations.

This looks like the binary cross-entropy loss function, which is why I think they say:

  For the model given in Equation 2 the negative log likelihood
  takes the form of a cross entropy between the
  patch $\tilde{m}$ derived from the given map and the predicted
  patch $\hat{m}$


Convert Numpy array with 'n' and 'y' into integer array of 0 and 1 I have a NumPy array of strings: 'n', 'y', wanna convert it into integer array of 0, 1, how to convert it?

imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
X = imp.fit_transform(X)

X

array([['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'y', ..., 'y', 'n', 'n'],
       ...,
       ['n', 'y', 'n', ..., 'y', 'n', 'y'],
       ['n', 'n', 'n', ..., 'y', 'n', 'y'],
       ['n', 'y', 'n', ..., 'y', 'n', 'n']], dtype=object)

It is quite trivial

(X=='y').astype(int)

Should do the trick. It simply converts your array to True or False according to your requirements and then astype will impose the required datatype. By default int will give you 1 for True and 0 for False.


Custom Named Entity Recognition using deep learning I have a dataset with two columns. First column has some text (news article) and the second column contains names of people (not exactly English names) present in those news articles (first column). I've tried to build a custom named entity recognizer using Spacy but it isn't working. Can I use deep learning approach to identify the names in unseen news articles ? (Test data)

The CNN and RNN are quite popular for custom NER. I will suggest you try to implement CNN first and then RNN. Also, when I did custom NER, I found out that for my dataset stacking is giving really good results. I used Random Forest, XGBoost and Linear Regression.


Extracting data and saving data from hdf5 I have a hdf5 file, and I want to extract a part of the data and save it as the same format.

The data type is

DATATYPE  H5T_IEEE_F32BE

So far I have successfully extracted the data using h5dump but it is in ascii format. I couldn't save it as the same hdf5 format. I saw there are several flags for saving as xml formats, but it doesn't work for me. I have tried using flags like -x or -d such as

h5dump -d variable -s 1 10 -X output.file  input.file

It creates the output file, but it is an empty file.

edit: I need to do it using h5dump.

You can always try using the h5py module in Python.

import h5py

# Load existing file
filename = "myfile.hdf5"
dataset = h5py.File(filename, "r")

# Show keys
[print(item) for item in dataset.items()]

# Assuming that you want to select the first 10 rows of key "A"
data = dataset["A"][:10]
dataset.close()

# Create the new dataset / file
dataset2 = h5py.File("myfile2.hdf5", "w")
dataset2.create_dataset("A", data)
dataset2.close()


Pros and Cons of Positive Unlabeled learning? I've been looking for papers that discuss the pros and cons of positive unlabeled learning but I haven't been able to find anything.
I'm looking to compare the general differences between creating a positive-unlabeled based problem vs a regression classification. I have a biological dataset where it's hard to definitively define a sample as negative but I can make rules that would find something as close as possible to negative - with my idea being that I can assign scores to samples (e.g. like 0.1 instead of 0 to imply the lack of certainty but the closeness of the sample  to being negative). However, I am trying to get an understanding of if I should consider positive unlabelled learning (in theory I could label my positive samples and ignore everything else even if other samples are capable of having a close to negative label/score) but I'm struggling to find information on pros and cons of trying positive-unlabelled learning.

I don't think it's possible to know for sure if PU learning would work in your setting or not. It's certainly relevant to cases like the one you describe, so it would be worth trying. But there are other valid options, and even within PU learning there are different approaches to choose from (you might be interested in this question).
In my opinion the alternative you propose with regression makes some sense and it might work, but it's not very "clean" in terms of design: first the choice of 0.1 is arbitrary (why not 0.2 or 0.05 or ...?). Second, it means that you're telling the regression algorithm that "this instance should have probability 0.1" for many negative instances and also for a few negative instances: this is different than saying "I don't know the target value for this instance".
Note that you could also consider one class classification in this kind of setting, (as part of PU learning or not).


Embedding of list of objects I have a dataset where each sample is a list of ordered items, lets say grocery list ,  and a label from 6 categories . each list can have up to 120 items but the mean items is 12 items in a list.
i would like to embed each sample in a similar way to word2vec and maybe average the vectors for each list to get 1 vector per sample.
You think that word2vec is the right approach for it ? since the "texts" are very short and the location of each item is not contextual but more like the number purchased

So, you want to make embeddings for sequences of items, and order doesn't make much sense. You don't have a specific objective, just want to retrieve embeddings with some conventional properties, such as natural clustering.
If you can train word2vec and take mean of item embeddings in a cart, you'll probably get noisy vectors. It doesn't work as well as expected, from my experience. I suggest that you try dimensionality reduction methods, such as NMF. Fit it with a sparse binary matrix of shape (num_samples, num_items). You'll get transformed vectors of desired dimensionality.
Then you might want to evaluate embeddings somehow. If don't have any quality metrics, you can either perform clustering to see how your embeddings form groups, or make nearest neighbour search queries by hand.
Good luck!