examples of semi structured data ? i read that Json or XML are unstructured data; Are Json or XML data or are they tools to tag the data?

I understand from this wikipedia page (https://en.wikipedia.org/wiki/Semi-structured_data) that semi structured data are data without a formal database structure, but still have some tags;

JSON and XML are not tools. They are data formats. For example, to find out more about JSON you can look up the RFC description that specifies how to format JSON files:

https://tools.ietf.org/html/rfc7159

It explains how to tag the data. Then there are tools that validate JSON, and XML files to see if they conform to the requirements, or there are tools that convert JSON into another format or read it into memory into some object or class.


How to include categorical fields to enhance a text classification I would have a question on how to add more categorical fields in a classification problem.
My dataset had initially 4 fields:
Date             Text                            Short_Mex                        Username        Label
01/01/2020       I am waiting for the TRAIN      A train is coming                Ludo       1
01/01/2020       you need to keep distance       Social Distance is mandatory     wgriws    0
...
02/01/2020       trump declared war against CHINESE technology      China’s technology is out of the games      Fwu32      1

...
I joined this dataset to a new one with labels, having values 1 or 0. This will need for classification.
However I have extracted also other fields from my original dataset such as number of characters, upper case words, top frequent terms, and so on.
Some of these fields may be useful for a classification, since I can assign more ‘weight’ based on a word in upper case rather than lower case.
So I would need to use a new dataset with these fields:
  Date             Text                            Short_Mex                        Username    Upper    Label
    01/01/2020       I am waiting for the TRAIN      A train is coming                Ludo    [TRAIN]       1
    01/01/2020       you need to keep distance       Social Distance is mandatory     wgriws       []      0
    ...
    02/01/2020       trump declared war against CHINESE technology      China’s technology is out of the games      Fwu32    [CHINESE]       1
...

I would like to ask you how to add this information (upper case) as a new info for my classifier.
What I am doing is currently the following:
#Train-test split
x_train,x_test,y_train,y_test = train_test_split(df['Text'], news.target, test_size=0.2, random_state=1)

    #Logistic regression classification
    pipe1 = Pipeline([('vect', CountVectorizer()), ('tfidf', TfidfTransformer()), ('model', LogisticRegression())])

    model_lr = pipe1.fit(x_train, y_train)

lr_pred = model_lr.predict(x_test)

print("Accuracy of Logistic Regression Classifier: {}%".format(round(accuracy_score(y_test, lr_pred)*100,2)))
print("\nConfusion Matrix of Logistic Regression Classifier:\n")
print(confusion_matrix(y_test, lr_pred))
print("\nCLassification Report of Logistic Regression Classifier:\n")
print(classification_report(y_test, lr_pred))

Scikit-learn has compose.ColumnTransformer which

allows different columns or column subsets of the input to be
transformed separately and the features generated by each transformer
will be concatenated to form a single feature space. This is useful
for heterogeneous or columnar data, to combine several feature
extraction mechanisms or transformations into a single transformer.

A demo of mixing numeric and categorical types is here. In your example, CountVectorizer is numeric and Label is categorical.


BIO tagging software I would like to label character data with BIO tags as part of an active learning process on unlabelled data. I am assuming there are open source GUI tools available which I can use to make this easier - i.e. present the string to be labeled and some way of tagging characters from a predefined set of tags (and probably allow new tags to be added).
I have not been able to find anything though - ideally cross-platform (Linux and macOS at a minimum). It could be a cloud service or AWS product but I'd prefer something which is locally installed.
Does anyone know of such a thing? Even something as simple as search term would help.

Another solution, without however the active learning support that prodigy offers, is doccano. It's open-source (free), easy to install and supports various applications, apart from NER (BIO tagging), also classification and translation.


How does Pytorch deal with non-differentiable activation functions during backprop? I've read many posts on how Pytorch  deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation functions during backprop. However I was not able to come up with a full picture as to what exactly happens.
Most answers deal with ReLU $\max(0,1)$ and claims that the derivative at $0$ is either taken to be $0$ or $1$ by convention (not sure which one).
But there are many other activation functions with multiple points of non-differentiability.

2 points

4 points
How does Pytorch systematically deal with all these points during backprop? Does anyone have an authoritative answer?

The function value is never exactly equal to those exact point because of numerical precision error.And again those functions in torch calculate left or right derivative which is defined in every case.So non-differentiability doesn't pose a problem here.


Is it normal when BLEU score on filtered data by length is greater than BLEU score on whole data I am creating 2 neural machine translation model (model A and B with different improvements each model) with fairseq-py. When I evaluate model with bleu score, model A BLEU score is 25.9 and model C is 25.7. Then i filtered data by length into 4 range values such as 1 to 10 words, 11 to 20 words, 21 to 30 words and 31 to 40 words. I re-evaluated on each filtered data and all bleu scores of model B is greater than model A. Do you think this is normal case?

The original BLEU scores 25.9 and 25.7 are very close, there might not even be any significant difference. It's totally possible that model B performs better than model A on the filtered data only by chance. It's also possible that model B actually performs better than model A on shorter sentences. And finally it's worth noting that BLEU score is based on the number of n-grams in common, so it's likely to be affected by the length of sentences independently from the model being tested.

Conclusion: based on the information provided, this difference seems perfectly reasonable.


Classifying whether a chess board square contains a piece or not? I have an image of a chessboard which I segment into its constituent 64 squares. I want to construct a binary  classifier which can detect which square contains a piece or not. I know how to do this via CNN's, but I feel like that might be a bit overkill. I was wondering what would the simplest approach be to this problem? Thank you.

If it's a screenshot of a computer game, where each constituent has purely that box, that is by default any given box is white or black. then, any simple matrix operation like, variance of color in that box could be a feature in determining if there is a piece or not.

If it's a hover cam image (Top view), then it's a different problem. you can simply do a kernel multiplication for edge detection, any empty box will have array of 0 as it's output. while it won't be the case if there is a piece.


How can I extract bootstrap generated datasets into individual dataframes? I am having a bit of trouble understanding Bootstrapping and what/how I can manipulate the bootstrap created dataset.

This is all in R

My original dataset is structured like this:

Time Animal_ID Treatment Volume
0     1          Control   1
2     1          Control   2
4     1          Control   6
0     2          T1        6
2     2          T1        6
4     2          T1        8
0     3          T2        21
2     3          T2        7
4     3          T2        8
0     4          T3        4
2     4          T3        3
4     4          T3        2

I am going to compute a Z-score,mean, and median statistic on the bootstrap generated data sets which is something I can do in the function. However, What I really need as a proof of concepts is a way to extract my bootstrap generated data sets into individual data frames or write them as excel files.

To clarify, I want to showcase a simple example of the statistic I am performing so if I create 5 bootstrap data sets I would need a way to explore them like they were unique individual data sets. I would need to show that this works in concept when looking at R=5 and then I would be able to confidently expand this to R = 100 no extraction required.

Does anyone know how to accomplish this? Or if there are other alternative?

One way is to use the infer package.

library(tidyverse)
library(infer)

df <- read_table(pipe("pbpaste"))

five_bootstraps <- df %>%
  generate(reps = 5, type = "bootstrap")

five_bootstraps

# # A tibble: 60 x 5
# # Groups:   replicate [5]
#     replicate  Time Animal_ID Treatment Volume
#         <int> <int>     <int> <chr>      <int>
#   1         1     4         3 T2             8
#   2         1     2         1 Control        2
#   3         1     0         3 T2            21
#   4         1     0         3 T2            21
#   5         1     2         2 T1             6
#   6         1     0         3 T2            21
#   7         1     0         4 T3             4
#   8         1     2         3 T2             7
#   9         1     0         2 T1             6
#  10         1     2         4 T3             3
# # ... with 50 more rows

So each "bootstrap" replicate is labeled in the replicate column. If you'd like to write each replicate to a separate .csv file, you could do:

five_bootstraps %>%
  split(.$replicate) %>%
  walk2(.x = names(.), .y = ., ~ write_csv(.y, paste0(.x, ".csv")))


Cosine similarity between query and document confusion I am going through the Manning book for Information retrieval. Currently I am at the part about cosine similarity. One thing is not clear for me.

Let's say that I have the tf idf vectors for the query and a document. I want to compute the cosine similarity between both vectors.

When I compute the magnitude for the document vector, do I sum the squares of all the terms in the vector or just the terms in the query?

Here is an example : we have user query "cat food beef" .
Lets say its vector is (0,1,0,1,1).( assume there are only 5 directions  in the vector one for each unique word in the query and the document)
We have a document "Beef is delicious"
 Its vector is (1,1,1,0,0). We want to find the cosine similarity between the query and the document vectors.

You want to use all of the terms in the vector.

In your example, where your query vector $\mathbf{q} = [0,1,0,1,1]$ and your document vector $\mathbf{d} = [1,1,1,0,0]$, the cosine similarity is computed as

similarity $= \frac{\mathbf{q} \cdot \mathbf{d}}{||\mathbf{q}||_2 ||\mathbf{d}||_2} = \frac{0\times1+1\times1+0\times1+1\times0+1\times0}{\sqrt{1^2+1^2+1^2} \times \sqrt{1^2+1^2+1^2}} = \frac{0+1+0+0+0}{\sqrt{3}\sqrt{3}} = \frac{1}{3}$


Adapting ZFNet on 2244x224 image using a filter 7X7 I am building a model based on ZFNet in Tensorflow 2.0. I am using the Petal images dataset. The images are of size 224x244x3.
So my question is when implementing the first layer (conv2d) with filter size = 7 and a stride of 3 and padding of 0. I am getting the output dimension of 109.5 using formula (n+2p-f/S + 1). So if I use the above-mentioned values what will be the dimension returned by TensorFlow in the first layer.
and secondly, how can I adjust the parameter values so it returns a whole number.
reference formula : (n+2p-f)/2 +1
reference calculations:
224+0-7/2 +1  = 109.5
Thanks.

As per the formula for the feature map dimension:
$$feature_{dim} = \frac{n+2p-f}{S} + 1$$
The values for :

n = 224
p = 0
f = 7
s = 3

$$feature_{dim} = \frac{224+2*0-7}{3} + 1 = 73.66 $$
As you've guessed this is not the size of the feature map.
Tensorflow takes it as 73.
If you're relying on the formula, you are missing out on a concept, that this should a process were in the kernel slides over the Input hence the feature map dimension should be an integer. So what happens is that the kernel while sliding with a stride of 3 leaves out the last few pixels and won't reach the other edge.
If you're trying to get a feature map that's an integer by keeping a constant filter size of 7. Then you're stride is :
$$S = 217 /(N - 1)$$
where N is you're desired output size.
If you choose N to be 8 or 32, you'll end up with a stride of 7 or 31. It's better to choose S = 7 to get most of the information. But still, it doesn't matter as TensorFlow has checks for the same to prevent errors.


How does inception decrease the computational cost? From the second paragraph of 3.1 Factorization into smaller convolution in the paper Rethinking the inception architecture for computer vision:

  This setup clearly reduces the parameter count by shar- ing the weights between adjacent tiles. To analyze the expected computational cost savings,

  ...

  This way, we end up with a net 9+9 × reduction of computation, resulting
  25 in a relative gain of 28% by this factorization

Apprantly this design decreases the number of parameters. But I can't understand why it decreases the computational cost?

For the case of using 2 3*3 convnets to replace 5*5, I think it increases (3*3*9+3*3)/5*5 = 3.6 times of computational cost.

What do I miss here?

However, in the overall scheme, sliding this network can be represented by two 3 x 3 convolutional layers which reuses the activation between adjacent tiles.

Since the replacement (two 3x3 instead of one 5x5) share weights, we don't have to calculate them twice. That is where the gain comes from.

Edit:

The gain comes from the sliding: using the following formula (n-filtersize+1) x (n-filtersize+1) that calculates the output of a filter on an n x n input

a more detailed answer can be found here:
Reducing Filter Size in Convolutional Neural Network
Thank you @Thomas W.


How do you determine the architecture of a model(keras)? I have solved quite a few kaggle playground problems lately, but I can't understand how to come up with good enough model architecture which gets 0.9+ validation accuracy and without overfitting.

Is there some formula, or is it some hit and trial method for determining filters and neurons.
And also I'm always using relu activation in hidden layers(read it somewhere) when will other activations be used in hidden ones.

Can you lay down some guidelines that I should follow?. Considering I mostly work with image classification problems.

And I you have some other tips as well you can share in your answer.

There is no theoretical understanding, that would take a problem and specify the optimal network architecture for you. So no, there is no formula, I'm afraid. What does exist, are strategies to arrive at good solutions.

Good strategies are to start simple and try to make iterative improvements. Define a single KPI (error metric) that you're trying to optimize, build the simplest model you can imagine that will hopefully work a little bit, then try to improve from there. You get improvements by trying to understand why your model does not perform optimally. You need to be able to distinguish between underfitting and overfitting to make the right modifications. Study the errors that your model makes on specific images and try to make improvements to address those mistakes with changes to the model.

I think you may find this course on Coursera interesting: Structuring Machine Learning Projects.


Efficient Decision Tree Pruning Is there an efficient way to handle pruning in Decision Tree with Python ?
Currently I'm doing that:
def do_best_tree(Xtrain, ytrain, Xtest, ytest):
    clf = DecisionTreeClassifier()
    clf.fit(Xtrain, ytrain)
    path = clf.cost_complexity_pruning_path(Xtrain, ytrain)
    ccp_alphas = path.ccp_alphas
    clfs = []
    for ccp_alpha in tqdm(ccp_alphas):
        clf = DecisionTreeClassifier(ccp_alpha=ccp_alpha)
        clf.fit(Xtrain, ytrain)
        clfs.append(clf)
    return max(clfs, key=lambda x:x.score(Xtest, ytest))

But it's super slow (as I create and fit a lot of trees).
Is there a more efficient way to do this with scikit-learn, or another library that handle this ?

You might benefit from random forests instead which aim to achieve the same objectives you are aiming for, i.e better generalization through pruning to remove overfitting.
scikit learn's random forest algorithm will let you specify how many or what proportion of variables you want to automatically drop across the many trees whose results will be averaged for even better generalization performance.


Theoretical bound - regression error The Bayes error rate is a theoretical bound that determines the lowest possible error rate for a classification problem, given some data. I was wondering whether an equivalent concept exists for the case of regression algorithms. My aim is to determine how far my regression algorithm's error is from that theoretical bound, as a way to assess how far am I from the best possible solution. Is there any way to obtain a bound of the lowest regression error for a given dataset?

I realize this question was asked more than a year ago, but I think one possibility is to use the bias-variance decomposition to calculate a lower bound on the error rate.

Essentially, the error rate is written as the sum of three terms, the bias, the variance, and the irreducible error. One good source for learning about these terms is An Introduction to Statistical Learning.

Assume that the true function ($f(x)$) lies within the family of functions that our machine learning model is capable of fitting, and take the limit as the amount of training data we have goes to infinity. Then, if our machine learning model has a finite number of parameters, both the bias and the variance will be zero. So, the actual error will simply be equal to the irreducible error.

As an example, suppose our true data is linear with Gaussian noise: $y \sim N(a + bx, \sigma^2)$. One of the the optimal estimators is obviously linear regression, $\hat{y} = \hat{a} + \hat{b}x$, and, as we add more training examples, the estimated coefficients $\hat{a}$ and $\hat{b}$ will approach $a$ and $b$, respectively. So, the best error (assuming squared loss) we could hope to achieve would be equal $\sigma^2$, the inherent error/irreducible noise associated with the data generation itself

In practice, computing the irreducible error is difficult (impossible?), since it requires knowledge of the true process for generating the data. But, this critique is also applicable to the Bayes error, since that requires knowledge of the true class probabilities.


Should input images be normalized to -1 to 1 or 0 to 1 Many ML tutorials are normalizing input images to value of -1 to 1 before feeding them to ML model. The ML model is most likely a few conv 2d layers followed by a fully connected layers. Assuming activation function is ReLu.

My question is, would normalizing images to [-1, 1] range be unfair to input pixels in negative range since through ReLu, output would be 0. Would normalizing images into [0, 1] range instead be a better idea?

Thank you.

As the other answers previously said, in practice it doesn't have much difference which of the two you choose. However, theoretically it's better to scale your input to $[-1, 1]$ than $[0, 1]$ and I'd argue that it's even better to standardize your input (i.e. $μ=0$, $σ=1$).

Let me explain why:

Deep neural networks, especially in their early days, had much trouble with backpropagation, as they suffered from vanishing/exploding gradients. A popular way to combat this is by initializing the weights of the network in smarter ways. Two initialization techniques have proven to be the most popular: Glorot initialization (sometimes referred to as Xavier initialization) and He initialization, both of which are variations of the same idea.

In order to derive their intialization they make a few assumptions, one of which is that the input features have a zero mean. You can read this post for simplified explanation, but I'd suggest reading the two papers mentioned above.


Is it possible to compress a sequence of numbers through an autoencoder? Specifically: I would like compress a set of coordinates, which map to the locations of 1's in a binary image, and then decode back to the original set. For instance, for a 16x16 image, the input might be something like the following:
[5, 4], [12, 5], [8, 7],....
I am not looking to recognize any spatial patterns, nor is this a time series problem because the input corresponds to just one static image. The trained autoencoder should be able to handle any array of arbitrary "coordinates", under the assumption that the data is scaled between 0 to 1 so the resolution (actual range of numbers) is inconsequential.  Is this doable? What would be a good way to train it?

This is doable, but the caveat is that a neural network might not give you a better result than a lossless compression algorithm (depending on what you want to use the compressed version for).
The most sensible way to do it in my opinion is to have a function at the start that converts the coordinates into an image and a function at the end to convert the recovered image to coordinates again.
The way you would train it is to gather a large number of training examples, pass them as 16*16 tensors containing 1s and 0s to your autoencoder network, then use the reconstructed image to extract the coordinates where the value is 1.
There’s a good article/tutorial here to get you started.
If you’re determined not to explicitly give your network a 16*16 tensor you could pass your coordinates as an ordered set of two dimensional vectors and treat the problem as a sequence transduction task. The network would need to learn an embedding for each set of coordinates and then predict each embedding based on surrounding embeddings. I expect you won’t get much joy from a transformer model as they are notoriously hard to train, but a recurrent model with attention like the ones in the article I’ve linked to might give interesting results.


Deep learning technique to recognize a person's silhouette I'm reading Nielsen's book on neural networks. In the first chapter you construct a neural net to recognize handwritten digits by training the network on 60,000 samples. I wonder how would you build a neural net to recognize a person by his/her silhouette? Is there some special technique to make it possible?

For this purpose you have to have a data-set that can be interpreted by the human. I mean an expert should label the data samples without hesitating. Then you can make a typical neural network and train it. In your case you have 10 classes and all 60 thousand images are labeled by an expert. Consequently, you are able to train a network. Your task is supervised and you need labeled data-set.


How to transform specific type feature to yield better prediction? I have a dataset with 5K records focused on binary classification problem. I have about 60 features.

Out of 60 features, around 45-46 features are of 'Min' and 'Max' type.

For example, minimum blood pressure, maximum old pressure, minimum heart rate, maximum heart rate, minimum potassium, maximum potassium etc.

Several other vital parameters like sodium, urea etc. follow this pattern of min and max.

Any suggestions on how can I transform them without losing info . currently when I use as-is,  I get only around 86% accuracy with recall value of minority class only around 70. Resamplin didn't help much.

Your suggestions and experience in how to transform this to yield better predictions would really be helpful.

ML has sometimes problems to find relations obvious for people, so I would help a little bit by creating a new feature:

relX = (maxX-minX)/minX

because some diseases are more correlated with jumps of some parameters than with their level, however, such relX may be too much correlated with target and may cause overfitting.
Usually, after creating such a feature, it is better to remove one of the original features, sometimes both.


Visualizing large number of points as a 3D density map The  result of my computational simulation is a (time-dependent) system of large number (~100k) of moving points in a confined space. Each point has its own Cartesian coordinates as well as a weight (w) in the form of $(x_i,y_i,z_i;w_i)$.

I'm looking for a software/app/package to create a snapshot of the 3D spatial density map of these points. (something like this).

Like you see in this figure, the points are not going to be displayed individually, but only a transparent cloud will be drawn whose local intensity is proportional with the local number of points.

The final goal is to create a movie of the changing 3D spatial density with time.

So far, I have tried R, Matlab, Origin, and ImageJ. But no success!

In Wolfram Language you may use ListDensityPlot3D.

First we create some data to plot using a 3D density function at three time points (0, .5, and 1) using Table.

p[t_, x_, y_, z_] := Cos[t z] + x y
densityClouds = Table[{x, y, z, p[t, x, y, z]},
   {t, 0, 1, .5}, {x, -1, 1, .05}, {y, -1, 1, .05}, {z, π/2, 3 π/2, π/40}];

densityClouds contains the 3 clouds but it is not the correct format for ListDensityPlot3D.

Dimensions@densityClouds

{3, 41, 41, 41, 4}

We can Flatten the two upper layers of each cloud (the x and y coordinates) to the third layer (the z coordinate) to get the correct format.

densityClouds = Flatten[#, 2] & /@ densityClouds;
Dimensions@densityClouds

{3, 68921, 4}

That we have data we can just plot the one of the clouds. The Last cloud is t = 1.

ListDensityPlot3D[Last@densityClouds,
 PlotLegends -> Automatic,
 AxesLabel -> {"x", "y", "z"}]

You may find other list 3D plots in the Data Visualization guide useful (like ListContourPlot3D) and their 3D equivalents in the Function Visualization guide (like DensityPlot3D).

Hope this helps.


Is my Lasagne/Theano neural network running too slow? So I'm a newcomer to the world of neural networks, and I've been getting a little familiar with the field and have started playing with my own networks.

I'm using Lasagne, and I'm finding that the training of my NN is taking an infeasible amount of time.  Maybe the problem I'm looking at is simply biting more than my computer can chew; I just don't know, because I'm not familiar enough with how these things go.

I'm trying to train a 1D convolutional NN, which has two 1D convolution layers, and two max-pooling layers, and two fully connected layers. All in all, the NN has around 95,000 parameters (weights + biases).  I'm running this on an Amazon EC2 GPU instance.  What I'm finding is that during training, it's taking 0.3 sec/sample.

By contrast, when I run the Lasagne MNIST CNN example, it's taking about 0.0003 sec/sample, 1000 times faster!  Now to be fair, my input samples are 18000x3 dimensional, and the MNIST samples are 28x28 dimensional; but the difference there is a factor of 68.  And the MNIST CNN actually has more parameters than mine, with 160,000 or so parameters.

Does this indicate anything wrong with what I'm doing?

Found the answer

I found out why it was going slowly, after doing some profiling and reading this:

It turns out that if you do a convolution with a stride != 1, then it doesn't use the GPU for that operation (or perhaps the gradient of that operation), and makes things run much slower.  I was only using a non-1 stride to save execution time anyway, but turns out I'm better off getting rid of the stride.

Also note that in your convoution layer, you have to completely get rid of stride=X.  Just having stride=1 is not sufficient.


Data with similar mean, min and max across all columns. What could I do to build a classifier I have a data with the following columns
    col1         col2       col3        col4        label
    7669.533073 7669.533073 7669.695497 7669.922593 1
    7669.533043 7669.533072 7669.695487 7669.922596 0

the mean across all the 50 columns are similar and also the min and maximum.
I am trying to build a classifier and the best model(random forest) is giving me a recall of .55 (doesn't seem so good), could there be anything I am missing in this?
I have thought about normalising the data but there seems to be no need as all columns have a similar mean and std.
Is there any statistics technique I could apply to the data to help get an improved result.
Note the data is from a simulated crypto price and I am trying to predict the price movement (up or down)

Your datapoints are too close to each other and hence it is really tough for any ML model to learn this inputs as it doesn't know how to differentiate almost same data to 1 and 0 label. That's why the result is random and you are getting around half accuracy.


How to extract keywords from a list of URLs? I have a bunch of URLs in a text file like-

https://www.mycustomer.com/marketing/technology/how-marketers-are-over-simplifying-b2b-buyer-behaviour
https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts
https://www.forbes.com/sites/forbesagencycouncil/2017/02/03/video-marketing-the-future-of-content-marketing
https://www.searchenginejournal.com/seo-content-marketing-strategy/258253
...

What's the best way to go about extracting top 10 keywords from just these URLs (no parsing the webpages)?

I'm aware of TF-IDF but that usually requires a title and a body, can I still use it here? Or are there any other approaches (e.g. TextRank) that would work better here?

urllib parse seems like the function for you. With this you are able to extract keywords from the net location and the path separately if you desire to process them separately or even if you want to join them back again later.

The result should look something like this:

from urllib.parse import urlparse

o = urlparse('https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts')

ParseResult(scheme='https', netloc='www.forbes.com:443', path='sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts',
           params='', query='', fragment='')

A second step will have to do with string parsing and string splits so something like

.split("/")
.split("-")

Will work just fine to split the path of the example's URLs into words. Also, remember to transform each word into lowercase.
After that stemming is a good idea so that all related terms are grouped into the same category. It's not necessary but it's a good idea.

Finally counting the occurrences of the words will give you the rank of the top "keywords" in those URLs.


Explain Binary Classification with output 0.5 (True) What is the interpretation of output 0.5 of a typical classifier?

I made a prediction and the probability of that data point being from the True class is 0.5.

It sounds like your model is saying there is a 50% chance that this data point could be in either class. Since this is a special case, prediction algorithms need to pick which class to choose when there is an equal probability for each class. In this case, it looks like your model defaults to the True class.

There are some models in sklearn like random forest and SVM that have predict_proba functions. You might find more information if you look there or check if the model you are using has documentation on how its probabilities are calculated.


Best practice for computing week-over-week change in MAU metric When we calculate week over week change for a metric like "Monthly active users", we often refer to a calculation of SUM(This week) - SUM(Last week)/SUM(Last week), which means we will run total of this week's number first, and then compare to the total from last week.
There is another situation when we can have change any date comparing 7 days ago(week), So each date during a week will have a change value.

Are there any best practice for this "week over week" concept. I've also heard that someone is using "weekly change" referring to the similar thing. Is there any difference? Which calculation I should chose?

Both of these options have value if interpreted properly. You are trying to answer  the question (I assume): 'are we doing better this week than last week?'

I don't know what industry you're in or how variable (or volatile) your metric is, but I'd advise caution in interpreting changes week to week. I might suggest that alongside your weekly (or whatever) metrics, you include some moving average. That way, you capture the overall trend while tempering your reaction to occasional ups and downs.


Pandas: How can I update dataframe values? I have two spreadsheets where one is updating the other.

How can I update this data using the pandas library?

Example, where 'b' updates 'a':

a = {'field': ['a', 'b', 'c'], 'value': ["", None, 1]}
b = {'field': ['a', 'b', 'd'], 'value': [1, 2, 1]}

Expected outcome:

c = {'field': ['a', 'b', 'c', 'd'], 'value': [1, 2, 1, 1]}

df_a = pandas.DataFrame(a)

df_b = pandas.DataFrame(b)

c = pandas.concat([df_a, df_b], ignore_index=True).drop_duplicates(subset=['field'], keep='last')


Rstudio using 2.5% of 250GB RAM. how to Increase it I am working on Rstudio on a server which has 250GB ram. But its taking too much time to handle a 2GB data file. how should i speed up my work?

You may or may not already know this, but here are a few basics about how R and Rstudio work and use resources.

Rstudio is a graphical user interface to R, not the interpeter/runtime environment.  There is a separate "R session" that actually executes your R programs and returns results for Rstudio to display.  Therefore, having Rstudio use more memory won't make any difference in the execution speed of your program.

Second, memory allocation and cleanup (a.k.a. "garbage collection") is handled automatically by the R runtime environment.  "R holds objects it is using in virtual memory". Virtual memory is a combination of physical main memory and secondary storage, the mixture determined by the server OS configuration.  You can use command line to change the amount of virtual memory allocated to processes. (Consult with your favorite Linux expert on this.)

Third, your speed of your program may or may not be limited by memory speed. You may be compute-bound.  Do some testing to find out what is constraining performance.

Fourth, you should first ask yourself whether your program implements an efficient algorithm.  Even if you aren't programming with loops and branches, the functions you call may use them, and maybe not efficiently for your application.  For example, there are dramatic performance gains to be had by switching to data.table from data.frame.

Fifth, once you have chosen an efficient algorithm, you can put effort into parallel execution.  The simplest way to do this is by using functions that automatically vectorize operations.  A bit more complicated is to recode your program use the packages doParallel and foreach.  With doParallel, you can specify the number of CPU cores to use, which on a server may range from 32 to 64 or more.  Finally, if your server has a Graphics Processing Unit (GPU), it's possible for some algorithms to reprogram using the GPU commands and get massive parallelism.  This option takes the most effort and has the most constraints.


Speed up Keras Model Prediction Load Times I am trying to create a prediction API using keras which loads the model predicts and closes the model. But initializing time in python is about 3-5 secs so each request takes around 5 secs to return the prediction irrespective of number inputs rows(predictions)
Is there any way to keep the model loaded and then stream the input data to get prediction. Like a Pre loaded model either through a socket or a port.
Similar to open office document converter
\program\soffice.exe -accept="socket,host=127.0.0.1,port=8100;urp;" -headless -nofirststartwizard -nologo

Keras Prediction Code
#!/usr/bin/env python3.6
import sys
import pandas as pd
from keras.models import load_model
model = load_model('model.h5')
X = pd.read_csv(sys.argv[1]).values
prediction = model.predict(X)
pd.DataFrame(prediction).to_json(sys.argv[2])

Script is called as
python3.6 predict.py input_scaled.csv output_scaled.json

The prediction time are as follows
#row    time
1       4.76 secs
10      4.49 secs
50      5.37 secs
5000    5.46 secs
50000   12.7 secs

I was able to get to work like this without flask or django. Just using default http.server in python
from http.server import BaseHTTPRequestHandler, HTTPServer
import logging
import sys
import pandas as pd
from keras.models import load_model
from urllib.parse import urlparse
model = load_model('model.h5')

class S(BaseHTTPRequestHandler):
    def _set_response(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()

    def do_GET(self):
        query = urlparse(self.path).query
        params = dict(qc.split("=") for qc in query.split("&"))
        X = pd.read_csv(params["input"]).values
        prediction = model.predict(X)
        pd.DataFrame(prediction).to_json(params["output"])
        self._set_response()
        self.wfile.write("Processed".encode('utf-8'))

def run(server_class=HTTPServer, handler_class=S, port=8080):
    logging.basicConfig(level=logging.INFO)
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    logging.info('Starting httpd...\n')
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        pass
    httpd.server_close()
    logging.info('Stopping httpd...\n')

if __name__ == '__main__':
    from sys import argv

    if len(argv) == 2:
        run(port=int(argv[1]))
    else:
        run()

Trigger Server using
python3.6 predict_server.py 8000

API like
http://ip/localhost:8000/?input=predict_scaled.csv&output=prediction.json


Validation data for multi-series stateful LSTM With stateful LSTM the network state is propagated to subsequent sequences and batches. I have multiple data files with data that I present to the network for training (making this multi-series). My question is how to create the validation data.

At the moment I take 20% of the data from either head or tail of each file and use that for the validation data, and the other 80% I submit to the NN.

However, because this is stateful, I wonder if that's not a good way to create my validation data. To elaborate; when training I potentially present several years or a decade of data, so when it comes to the final sequence the network, it has the state built up from previous years of data. If I present some validation data which is maybe just a week of data, then the network doesn't build up any prior state to use to generate the final output.  Am I right in thinking this isn't a good way to validate a stateful RNN/LSTM?

What I'm thinking instead is to put aside 20% of the data files I have, and use the entire data in those files to create the validation loss. Would that be better?

Using a stateful model, am I correct in assuming that you have time-series data?

If so, it would perhaps make sense (at least for your test accuracy) to always validate against the time-step that immediately follows a test batch. This is what you would actually want to be doing with the model once it is trained.

Have a look at this answer, where I explain the idea in a lot more detail. The main idea is to split data into a kind of rolling forecast pattern, whereby you would have a test batch e.g. with something like:

window_size = 100     #whatever makes sense for your data
val_size = 15

train_batch1 = data[0:window_size]
val_batch1 = data[window_size:window_size + val_size]

train_batch2 = data[1:window_size+1]
val_batch2 = data[window_size + 1:window_size + val_size + 1]

...

That is just pseudo code to make it as clear as possible. Here is the diagram I created for the post linked above, which visualises the same notion:

In the image, it just has train/test sets... you could of course add validation into the mix.

If this doesn't seem to make sense for you data, temporal relationships must not matter too much. In that case, perhaps a stateful model is also not entirely useful.


In layman terms, what is the meaning of fitting a model into a data set? When we fit any model into a data set for prediction, what exactly happens behind the scenes?

I am learning Regression and I am a little confused about how exactly it fits the classifier or regression into a dataset.

Your asking for a layman's explanation, and the phrasing of your question, suggests to me that you are getting tangled up in the semantics, and I totally get that.

So let's let go of the usual terms as much as we can, and speak in concepts.

You have a bunch of observations -- or maybe even only one observation. This is your data. Maybe you are one of the particularly inquisitive ancients, and you notice some sort of relationship between lunar patterns and tides. While tracking this data, you start getting a sense for what sort of forces might be at work that would explain the apparent cause-and-effect relationship. The set of rules that you eventually develop becomes your model. The more correct your rules, the better you will be able to predict the tide tomorrow, next month, and next year.

This concept you've developed -- probably something to do with gravity -- may be even more generally useful. If the general concept is re-used with different numbers, it can also help explain Earth's annual cycle. And why we see the Morning Star (Venus) with certain patterns.

In the cases where we are re-using the concept, we are "fitting the model to the data" -- we are testing whether the set of rules we've developed -- the model -- can explain the apparent forces at work in the observations -- the data -- that we have.

In the usual applications of regression (or any statistical method), we're not trying to formulate the laws of physics, we're trying to find rules that sufficiently govern complex systems -- often human behaviour -- and so we're unlikely to explain observations perfectly. But a model is useful to the extent that it successfully predicts future observations. Rather than waiting for the future, though, we typically perform analysis on part of a data set, and test for predictive power with the remainder of the data set.

A specific instance of regression analysis might be in trying to explain housing prices. There are many many possible factors -- square meterage (footage), number of floors, swimming pool or not, size of property, shape of property, age, likelihood of flooding, distance to neighbours, energy rating, economic conditions, quality of the local school, etc, etc, etc. Which factors are most important? Do one, two, three of the factors explain 95% of the variation in housing prices? Can we come up with an equation that will predict a house's selling price reasonably well, rather than relying on a real estate agent's recommendation -- and having to second-guess what his/her agenda is -- how they might be playing us?

And if you will humour a little editorializing:

There are countless examples of reliance on experts proving to be a mistake (Nate Silver's "The Signal and the Noise", Daniel Kahneman's "Thinking Fast, Thinking Slow", Philip Tetlock's "Superforecasting") -- Data Science is about finding ways to derive insight from the data, rather than experts' opinions. But many data scientists get caught up in the mechanics, and lose sight of the purpose. It's not just about razzle-dazzle infographics. As Hamming puts it: "the purpose of computing is insight, not numbers". So I applaud your effort to seek a layman's explanation, rather than technical speak, to understand regression.


Can we use ReLU activation function as the output layer's non-linearity? I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value.

Can I use ReLU as the activation of the output layer? I am afraid of trying, since it is generally used in hidden layers as a rectifier. I'm using Keras.

Yes, you can. Basically, for regression tasks, it is customary to use the linear function as the non-linearity due to the fact that it's differentiable and it does not limit the output. This means you can make any output using your inputs. People do not use tanh or sigmoid as the activation function of the last layers for regression tasks due to the fact that they are limited and cannot generate all numbers which are needed. In your task, you can use ReLU as the non-linearity. The concept of non-linearities in hidden layers is to add non-linear boundaries and for the last layer in regression tasks, it should make all possible choices. In your case, ReLU is the best.


How do i get to make colab read my images from my google drive? I havee about 2000 images, I have uploaded them to my Google drive. How do I make colab see and read these images. All the images are in one folder.

There are some commands you must run, designed specifically for Colab:

from google.colab import drive
drive.mount('/content/drive')

With this, an authentication page will open. They want you to explicitly allow an access to your Drive. Click on the link and get a key code, that you can insert into an input line. At that point, you Colab Notebook is ready to access the data within you Drive account.

Each file path commonly starts with:

"/content/drive/My Drive/ ... your folders.../"

Within Colab Notebook, linux shell commands are also working, and can be accessd by putting ! before them (as in Jupyter).

NB: google.colab commands do not work outside Google Colab Notebooks, therefore, if you download your .ipybn file, remember to change those line or it'll throw error.


copy data between two dataframes in python I have two different dataframes, both containing a common element (identifier). I want to copy data from df1 for each identifier in df2 (each identifier can repeat maximum 5 times in df1)
df1 ---data vertically
id_column         value1
identifier1       some data1
identifier1       some data2
identifier1       some data3
identifier2       some data2
identifier2       some data1
identifier3       some data3

While df2 looks like this and this is how the information should be copied from df1:
df2 ---data horizontally
id_column        new1          new2           new3

identifier1      some data1      some data2       some data3
identifier2      some data2      some data1
identifier3      some data3

Is there a way to do this with python?
Many thanks for any help

If you want to add the data from df1 to df2 you can use pandas.merge. Given that you want to keep all records from df2 and only add data from df1 for the identifiers in df2 you can use the following syntax:
import pandas as pd

pd.merge(df2, df1, how="left", on="id_column")


Training And Testing Error Curves caret package in r I am Running the following models

Logistic regression
Decision Trees
SVM
Naive Bayes
Random Forest
On the same data set.

I am using Caret package in r.

Its My dream to plot Training error and Testing error curves, I know those plots

are pretty beautiful as well as other performance measure curves using caret

package, any help would be really appreciated.

Here we are creating data to for a given model.

The main advantage with this code snippet is you are able to see how the model performers over various sizes of the training set. The learing_curve_data function can be found in the [6.0-7.1 version of caret][1]. Of course, learning curves are useful in machine learning for several reason which include comparison of algorithms, choosing model parameters, optimization, right data size to use for training.

set.seed(1412)
class_dat <- twoClassSim(1000)

set.seed(29510)
lda_data <- learing_curve_dat(dat = class_dat,
                               outcome = "Class",
                               test_prop = 1/4,
                               ## `train` arguments:
                               method = "lda",
                               metric = "ROC",
                               trControl = trainControl(classProbs = TRUE,
                                                        summaryFunction = twoClassSummary))

ggplot(lda_data, aes(x = Training_Size, y = ROC, color = Data)) +
geom_smooth(method = loess, span = .8) +
theme_bw()


Sports prediction using machine learning I am trying to predict soccer scores using past results. The dataset I have only consists of the home team, the away team, goals scored by the home team and goals scored by the away team in each match. How can I use the limited data on my dataset to model this using a random forest algorithm and the naive bayes algorithm?

You have two categorical features (team names) and two continuous features (goals scored). The continuous features will probably remain bounded at a lower value since soccer isn't typically a high scoring game, so no normalization will be necessary.

You may need dummy variables for the team names, as the text names will likely not play well as a feature for most classification libraries. Research the sklearn Ordinal Encoder for this task.

Look at the sklearn classification algorithms that are available and try implementing a few of those.

Your response variables will be the two predicted scores. One for home and one for away. Or you may decide that the response variable could simply be one 'spread
' feature for the expected spread of the score.


how to check the distribution of the training set and testing set are similar I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of the training set and testing set are similar.
And I search it and find a blog which check the similarity of distributions by converting it into a binary classification problem. If it gets a high AUC, the distribution of the training set and the testing set must be different. And the idea he gives as follows:

  If there exists a covariate shift, then upon mixing train and test
  we’ll still be able to classify the origin of each data point (whether
  it is from test or train) with good accuracy.

But I still can't understand why he can check the similarity of these two distributions in this way.
And are there other ways to check the similarity of it?
It will be appreciated, if anyone could help me.

It looks like the person that wrote the blog is combining the samples from the test set and train set into one dataframe and then predicting if each sample comes from the test set or training set (his y variable is called “is_train” which indicates whether the sample came from the training set or not). I think his point is that if you are able to accurately classify whether the sample comes from the test or training set then the predictor variables have different underlying distributions. That would signify that your original model will probably not work well on this test data. Also — he is using AUROC as the performance metric. A high AUROC means that the model is performing well, and in this case it means that there is a big difference in distributions of predictor variables between the training and test set. Ideally, the distribution of the predictors for the training and test set should be the same, so you would want to get an AUROC that is close to 0.5.

I think this situation would only be relevant in cases where you have your model deployed and you need to check if your model is still relevant over time. If you are building a new model you shouldn’t need to do something like this because the test data is randomly sampled from the dataset. Additionally, if you’re doing cross validation then there is even less reason to worry about something like that.


Dataset with disproportionately more of a single label than any other I'm using the data analysis software Orange to analyze rows of data with labels $\{H, T,L\}$. $T$ is the neutral state of the system I'm trying to model so data is almost always labeled with $T$.  This leads to an extremely high $97$% classification accuracy, that is untrue.  Almost no datapoint with an $H, L$ actually gets labeled with an $H, L$ respectively by the neural net.  The neural net is doing what's logical: flatten everything to $T$ since that occurs most of the time.  How can I repair this situation?

There are a few ways:

Oversample the under-represented class.
issue: Leads to very constrained boundaries (not smooth) around the under-represented class
Weigh the loss coming from under-represented class more than over-represented class. If you think about it, this is mathematically doing the same thing as 1
Use SMOTE (here) (i recommend this). Basically samples additional points randomly from lines joining nearest under-represented classes and increases the dataset size accordingly. scikit contrib implementation of smote

There are other ways but they offer very marginal improvement over SMOTE


Are there any interesting application of linear regression Linear regression is a widely used ML algorithm. So far I have only encountered 'boring' applications of it. (e.g predict sales for next quarter, predict housing prices for next year , predict population of a country by 2020 etc.)

What are some interesting/cool application of linear regression? Finding cool applications greatly helps in motivating myself to learn, hence the quest.

Cool as in: Application to stock trading, application to video games, application to astronomy, application to sports betting, application to airfare prediction

The way that you have phrased this question makes it tough for people to answer without first offering you some background on linear regression (LR).  Its great that you are interested in learning some ML and LR is a great place to start.

Linear regression is really just finding a line (or plain or hyperplain) that maps a relationship between two or more variables or features.  The important requirement is that the target variable be a continuous numerical value.  So its less about finding a problem for which linear regression "works" and more about finding some data that interests you and playing with it using linear regression.

I suggest you download some open data in the realms that you have mentioned.  There is plenty of open data in every topic that you mentioned that contain continuous numerical values that you can predict using other features of the data.

stock trading
video games
astronomy
sports betting
flight time prediction

Also think about taking some sort of online MOOC as this will help you gain some footing in the subject.  Andrew Ng's Coursera on Machine Learning is highly recommended as a starting point and include some linear regression during the first portion and scikit-learn is a great Python based library.


What does it mean by “t-SNE retains the structure of the data”? I was learning about t-SNE when I was told that t-SNE retains the structure of the data in the embeddings.

What exactly does this mean ?

How does the algorithm achieve this ?

So far I have understood this ->

t-SNE is an unsupervised learning algorithm that is used for dimensionality reduction and visualization of high dimentioanal data. The algorithm works by measuring the similarity between one point and all the other points using the t-curve. The width of the curve is dependant on the density of the cluster the point belongs to. t-SNE retains the structure of the initial data.

My question is what does it mean by retaining the structure of the data? Shouldn't there be some loss in the structure of the data seeing as it is transformed into a lower dimensional space ? Also what does "strucutre of the data mean"? Please ask for any further details that are required.

You should break this down one step further: retaining local structure and retaining global structure.

Other well-understood methods, such as Principal Component Analysis are great at retaining global structure, because it looks at ways in which a dataset's variance is retained, globally, across the entire dataset.
t-SNE works differently, by looking at locally appearing datapoints. It does this by computing a metric between each datapoint and a given number of neighbours - modelling them as being within a t-distributed distribution (hence the name: t-distributed Stochastic Neighbourhood Embedding). It then tries to find an embedding, such that neighbours in the original n-dimensional space, are also found close together in the reduced (embedded) dimensional space. It does this by minimising the KL-divergence between the before and after datapoint distributions, $\mathbb{P}$ and $\mathbb{Q}$ respectively.

This method has the benefit of retaining local structure - so clusters in the low dimensional space should be interpretable as datapoints that were also very similar in the high dimensional space. t-SNE works remarkedly well on many problem, however there are a few things to watch out for:

Because we know have some useful local structure retained, we essentially trade that off for ability in retaining global structure. This equates to you not being able to really compare e.g. 3 clusters in the final embedding, where 2 are close together and 1 is far away. This does not mean they were also far away from each other in the original space.
t-SNE can be very sensitive to its perplexity parameter. In fact, you might get different results with the three-cluster example in point 1, using an only slightly different perplexity value. This value can indeed be roughly equated to "how many points shall we inlude in the t-distribution to find neighbours of a datapoint" - it essentially gives the area which is encompassed in the t-distribution.

I would recommend watching this lecture by the author of t-SNE, Laurens van der Maaten, as well as getting some intuition for t-SNE and it's parameters using this great visual explanation.

There are also some good answers here on CrossValidated with a little more technical information.


Advantages and disadvantages of using classification tree I was working on a project and was trying to validate my decisions. I wondered why would I want to use a decision tree over more powerful algorithms like random forest or Gradient boosting machine which uses similar tree based architecture.

One main advantage of using Decision Tree is that you can visualize your prediction more precisely than any other classification approaches.

In addition,

What rules make any specific prediction? You can generate such rules
What feature/value is crucial for what decision? You can indirectly do the feature selection.

There are some shortcomings as well; sometimes, the tree might get quite large, so you have to reduce the size of the tree (pruning) with the cost of accuracy.


Can we automatically chose k value in k-means algorithm? Can we choose automatically the K value, trying every possible values (k=1,.., n) where n is the number of instances to be clustered. We then keep the value of K for which we obtained the minimum value by the method of sum of least squares.
Can this strategy work?

This is a known problem of automatic clustering, how to choose / adapt the number of clusters so it represents the "real" clusters.
Hierarchical clustering is more helpful in this regard. For algorithms like K-means this is not so easy and research has tried various approaches to determine the optimum number of clusters (eg employing Information-Theoretic criteria like Akaike information criterion - AIC).
An overview in given in wikipedia article and references therein.

Determining the number of clusters in a data set, a quantity often
labelled k as in the k-means algorithm, is a frequent problem in data
clustering, and is a distinct issue from the process of actually
solving the clustering problem.
[..]The correct choice of k is often ambiguous, with interpretations
depending on the shape and scale of the distribution of points in a
data set and the desired clustering resolution of the user. In
addition, increasing k without penalty will always reduce the amount
of error in the resulting clustering, to the extreme case of zero
error if each data point is considered its own cluster (i.e., when k
equals the number of data points, n). Intuitively then, the optimal
choice of k will strike a balance between maximum compression of the
data using a single cluster, and maximum accuracy by assigning each
data point to its own cluster. If an appropriate value of k is not
apparent from prior knowledge of the properties of the data set, it
must be chosen somehow. There are several categories of methods for
making this decision.


Understanding predict_proba from MultiOutputClassifier I'm following this example on the scikit-learn website to perform a multioutput classification with a Random Forest model.
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np

X, y1 = make_classification(n_samples=5, n_features=5, n_informative=2, n_classes=2, random_state=1)
y2 = shuffle(y1, random_state=1)
Y = np.vstack((y1, y2)).T

forest = RandomForestClassifier(n_estimators=10, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
multi_target_forest.fit(X, Y).predict(X)

print(multi_target_forest.predict_proba(X))

From this predict_proba I get a 2 5x2 arrays:
[array([[ 0.8,  0.2],
       [ 0.4,  0.6],
       [ 0.8,  0.2],
       [ 0.9,  0.1],
       [ 0.4,  0.6]]), array([[ 0.6,  0.4],
       [ 0.1,  0.9],
       [ 0.2,  0.8],
       [ 0.9,  0.1],
       [ 0.9,  0.1]])]

I was really expecting a n_sample by n_classes matrix. I'm struggling to understand how this relates to the probability of the classes present.
The docs for predict_proba states:

array of shape = [n_samples, n_classes], or a list of n_outputs such arrays if n_outputs > 1.
The class probabilities of the input samples. The order of the classes corresponds to that in the attribute classes_.

I'm guessing I have the latter in the description, but I'm still struggling to understand how this relates to my class probabilities.
Furthermore, when I attempt to access the classes_ attribute for the forest model I get an AttributeError and this attribute does not exist on the MultiOutputClassifier. How can I relate the classes to the output?
print(forest.classes_)

AttributeError: 'RandomForestClassifier' object has no attribute 'classes_'

Assuming your target is (0,1), then the classifier would output a probability matrix of dimension (N,2).
The first index refers to the probability that the data belong to class 0, and the second  refers to the probability that the data belong to class 1.

These two would sum to 1.

You can then output the result by:

probability_class_1 = model.predict_proba(X)[:, 1]

If you have k classes, the output would be (N,k), you would have to specify the probability of which class you want.


How to encode a column containing both string and numbers I have a column in my dataset which contains both number and strings as the value. I want to encode the string variable so to use it for predicting. What is the best way to do this?

Encoding is a way to transform categories to numerical variables, there are a lot of techniques.

The best technique depends on what is the information you want to encode and what is the model that you are going to use. Some models benefit more of one techniques than others.

You should ask yourself the following questions to try to find the best solution:
 - Why do you have numerical and categorical values in a column?
 - Does it make sense to have them?
 - Which model am I using?
 - What is the best way to feed them to this model?

From the bit that I get I would reccomend that either you do target encoding with everything or that you split that feature into two. And then you do target encoding in the categorical and even in the numerical. Just check which works better.


Create clusters based on specific keywords I am working on raw text data. I am using clustering to put together common words in the documents. My requirement is to create clusters based on a specific list of words i.e I want to get a group of words that are typically found with the user-given list of words. Visually, the clusters should look like below. Typically, the clustering techniques are focused on creating segregated clusters while I need segregated clusters with some overlap. The image shows the view of the expected results.
I have tried using k-means clustering, the Apriori algorithm, and PrefixSpan in Python. But my desired result is not achieved.
Any suggestion is appreciated.

Methods for generating fuzzy (or soft) rather than crisp clusters may be applicable to your problem. One implementation for soft clustering in Python is a variant of DBSCAN called HDBSCAN. This link explains soft clustering using this algorithm. In soft clustering each point's similarity to each cluster can be measured. If a point is sufficiently similar to multiple clusters, then the clusters may overlap in the feature space.
There are also algorithms for explicitly generating overlapping cluster assignments from the literature such as:

In Overlapping Correlation Clustering (Bonchi 2013) each point may be assigned to multiple clusters based on pairwise similarities between points in the input set.
In Model-based Overlapping Clustering (Banerjee 2005) a generative approach is used to modeling the clusters.

For a survey of overlapping clustering approaches, see Overlapping Clustering: A review (Baadel 2016).


How to improve results for clustering of words I have a list of words (names actually) on which I would like to apply some entity resolution.
My first guess was to create clusters of similar names so I could extract a representative entity from multiple name shapes.

I need to specify that I have no labelled data, and I am not working on a document analysis (this is different from Improve results of a clustering for example), only a raw list.

To do so, and based on what I could read, I attempted two approaches :

apply n-gram transformation on my names and use k-means clustering
apply n-gram transformation, compute a similarity matrix (cosine distance) and use it for affinity propagation

Both approaches give me interesting results, yet I can't understand some of the results. For example, I get the following clusters :

Geronese, Varonese, Veronefe, Veronese, Veronesse, ...

Cameroni, Veronèse, Veronèse P., Veronése, Veronêse

Why do I get two different clusters for shapes that look so similar (except for Cameroni which I don't know why it is in that cluster) ? Is this a problem in the k-means algorithm tuning ?

Also, I tried using the silhouette metrics to find the optimum number of clusters but I get the exact same value no matter what is the number of clusters (0.315 for what it's worth).

As for the affinity propagation approach, I get a lower silhouette score for my clusters, and I get some similar effects, like having this kind of cluster :

Birttetti, Laruette, Laruelle, Larvette, Laurette, ...

Any ideas how I could improve my results (if this is possible) ?
Or maybe any idea for a better approach than mine ?

I cannot write comments yet, so I will start with an answer that still contains some questions, feel free to complement/correct it!

A) The data

Without any contextual information, it looks like the only solution you have to link entities is their character similarity. I don't know which distance metric you use, but have you tried these metrics that are well designed for string data?:

Levenshtein distance (also called string edit distance): basically computes the number of operations needed to transform one string into another
Jaro-Winkler distance: close to levenshtein but favoring strings with a common prefix
Jaccard similarity: roughly computes the ratio of common elements between two sets, be it characters, ngrams, words..

You can try these metrics without the n-gram transformation first, but also on transformed data and use cosine distance afterwards.

You can also perform additional normalization but you need to assess whether it is relevant for your dataset, which I don't know, so I am just firing ideas, not advices: stripping accents, special characters, short tokens (like the 'P.' in your example), which will reduce the variance in your data.

B) The clustering

One thing that could explain your result is that with KMeans or Affinity Propagation, any data point must belong to a cluster, so names that should be alone are assigned to the cluster so that it minimizes the loss of the algorithm.

Have you thought of trying DBSCAN? It can label some data points as noise, and if you are using one of the string distances above, you can encode prior knowledge about the maximum distance between two potential matches through the epsilon parameter.

But as with any unsupervised method, you will never have the guarantee to get rid of noise.


"residual error" of LSTM during backprop vs usual "error" What does the residual error mean when we are talking about LSTM?

Taken from the middle of section 3 of this paper, where it says:

"...of the residual error $\epsilon$"

Where $s_0$ is the initial state of the RNN network.

Question: how is a residual error different to a usual error? Why to use such a term?

Residual errors are the errors that remain after a model has tried fitting to some data. It is the error which resides.

People use that term, along with just error or residuals interchangeably, but after a model has been tested, it just means how much of the data cannot be explained by the model.

The letter $\epsilon$ is commonly used to explain stochastic noise inherent in (co)-variates of a model, i.e. noise.error that we cannot explain with the given data.


What is the best approach for send time optimization? I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be your approarch?

Generally, the best method to find optimal send time is A/B testing. Send the same emails/notification at different times to different people. Then see if one group tends to open the emails more than another group.


How do I deploy a model when using Stratified K fold? I have used Stratified K fold for learning the model . Below is the python code:
>def stratified_cv_v1(X, y, clf, shuffle=True, n=10,):
>    stratified_k_fold = StratifiedKFold(n_splits=n,shuffle=shuffle)
>    y_pred_v1 = y.copy()
>    for ii, jj in stratified_k_fold.split(X,y):
>        X_train, X_test = X[ii], X[jj]
>        y_train = y[ii]
>        clf_v2 = clf()
>        clf_v2.fit(X_train,y_train)
>        y_pred[jj] = clf.predict(X_test)
>    return y_pred_v1

>print(classification_report(y, stratified_cv_v1(X, y, GradientBoostingClassifier)))

Now how do I use the model to deploy on a new data set where I need to predict ?

k-fold CV is meant to evaluate the model. Once the evaluation is done and one is ready to move to deployment, there's no point using CV anymore: the method has been tested and validated, so one can reasonably assume that from now on applying the same method to the same kind of data will lead to the same level of performance. Thus the usual process is:

Train a final model on the full dataset (no CV, no testing)
Apply the model to new instances


Keras' 'normal' LSTM uses the GPU? I'm running Keras' LSTM (not CuDNNLSTM) but I notice my GPU is under load. I need recurrent dropout, so I can only stick with LSTM. Is the 'normal' LSTM assisted by GPU? If so, how are LSTM and CuDNNLSTM different? I presume CuDNNLSTM uses the CUDNN API (and LSTM doesn't?

Similarly, is the normal LSTM supposed to be faster running on GPU or CPU?

Not sure if you figured this out, but I've been looking into it recently, and this is what I've found:

Is the 'normal' LSTM assisted by GPU?
is the normal LSTM supposed to be faster running on GPU or CPU?

Like @pcko1 said, LSTM is assisted by GPU if you have tensorflow-gpu installed, but it does not necessarily run faster on a GPU. In my case, it actually slowed it down by ~2x, because the LSTM is relatively small and the amount of copying between CPU and GPU made the training slower. I think with a larger network, it would speed things up. I also found that LSTM only used ~25% of the GPU, while CuDNNLSTM used ~35% of the GPU, but haven't done a thorough investigation to figure out where the difference comes from.

how are LSTM and CuDNNLSTM different?

This github issue talks about how to convert CuDNNLSTM layers to LSTM layers. I found it pretty illuminating in how CuDNNLSTM has 2x weights/biases as LSTM, and how to convert from one to the other.


Understanding missing values in dataset I have recently worked with a dataset of real estate transactions with missing entries for some features. For instance, GarageYrBlt (year when a garage was built) was sometimes null. Understanding what null means here could help me, I guess, decide what approach should I take to handle missing values for this column. Simple analyzis of the dataset has shown that it is always null, when GarageArea,  GarageCars are 0, which probably means no garage exists at all for a given estate:

GarageYrBlt GarageArea  GarageCars

        NaN          0           0
       1973         45           2

I was wondering if there is a generic approach to understand missing values semantics in a given dataset.

Some approaches I could think of:

Go through feature and find those with related names. For instance, all having the word Garage. Then filter samples where those features have 0, null or some negative number that doesn't make sense in this particular context. This doesn't scale well for large number of features.
Employ Apriori algorithm for finding associative rules like the one I found above:

{GarageArea ==0 AND GarageCars == 0} ==>  {GarageYrBlt == null}
or {GarageYrBlt == null} ==> {GarageArea ==0 AND GarageCars == 0}

(although I am not sure if Apriori supports finding the latter, which have multiple elements on the right side).

Anything else?

No generic approach in data handling. Role of DS is understanding data. As you know sometimes NULL value means something, you have to check all aproaches to handling data and I find out which is most suitable to yours data.
Usually I marked missing value and resampling datasets. So I getting one dataset without missing values and another one with marked missing value.

Another techniques
https://www.kdnuggets.com/2017/06/7-techniques-handle-imbalanced-data.html

Best


Confusion matrix in sklearn If you look at this:
>>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

I suppose fist row of array means "predicted ant" and first column is "actually is ant" second column is "actually is bird" etc.
So first row first col 2 i read like "predicted ant, is ant", first row second col 0 i read as "precited ant is bird" is 0 which fits, and third column is "predicted ant is cat" is 0 but should be 1.
What i am doing wrong while understanding the confusion matrix.
Another example is this
>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

Where is not even clear, what is the order of classes.
Source: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html
edit: Unless it is swapped. First row is "is ant" not "predicted ant". Only that on wikipedia the system is that row is the prediction.

You just confused the actual and predicted. Every row represents actual values of distinct elements in your array and columns represent predicted values of them. That is,

First row: There are 2 ants, and 2 samples are predicted as ant.
Seconds row: There are 1 bird and 1 sample is predicted as cat.
Third row: There are 3 cats, 1 sample is predicted as ant, 2 samples are predicted as cat.


Encoder-Decoder performance time I have two encoder-decoder models.

*First model:

*Second model:

When I check the performance of the models I get approximately the same performance time (First model ~ 42 sec, Second model ~ 40 sec). I train my model on GPU and check performance on CPU. I test it only on one large image where the size is 12348x12348.
I was expecting the larger model that has more parameters to train (second model) to give me longer run time. Anyone can help me to understand why it is not the case here? Am I doing something wrong?

Could you tell us a little bit more about those 2 models ?
If your small network is recurrent, the computation graph of the model is not very parallelizable. It could explain the latency.


what is the difference between multilabel and multilabel-multiclass classification? I am trying to classify news articles into their required category. However I am confused by the above(multilabel and multilabel-multiclass) terminologies. My dataset consists of 2 csv files. The first file contains content of the news article. The second file in that same corresponding row number consists of labels associated with it. Here is the sample entries in second file i.e. labels file:

So for row number 1 the first csv file contains news article which is labelled as sport and football. Similarly for row number 2, the article in first csv has labels sport and media. My question here is, is this dataset multi-label or multilabel-multiclass? what is the difference between them?

Multiclass classification means a classification task with more than two classes; e.g., classify a set of images of fruits which may be oranges, apples, or pears. Multiclass classification makes the assumption that each sample is assigned to one and only one label: a fruit can be either an apple or a pear but not both at the same time.

Multilabel classification assigns to each sample a set of target labels. This can be thought as predicting properties of a data-point that are not mutually exclusive, such as topics that are relevant for a document. A text might be about any of religion, politics, finance or education at the same time or none of these.

Multioutput-multiclass (or multilabel-multiclass) classification means that a single estimator has to handle several joint classification tasks. This is both a generalization of the multi-label classification task, which only considers binary classification, as well as a generalization of the multi-class classification task. The output format is a 2d numpy array or sparse matrix.

As each object of your data can be classified in one to multiple labels at the same time, it is a simple multilabel classification problem

Taken from : https://scikit-learn.org/stable/modules/multiclass.html


Grouping profiles strings having the same words, but occurring out of order I have a dataframe containing a column of profile types, which looks like this:
0                                    Android Java
1                  Software Development Developer
2                            Full-stack Developer
3                      JavaScript Frontend Design
4                          Android iOS JavaScript
5                             Ruby JavaScript PHP

I've used NLP to fuzzy match similar profiles, which returned the following similarity dataframe:
left_side                       right_side                  similarity
7   JavaScript Frontend Design  Design JavaScript Frontend  0.849943
8   JavaScript Frontend Design  Frontend Design JavaScript  0.814599
9   JavaScript Frontend Design  JavaScript Frontend         0.808010
10  JavaScript Frontend Design  Frontend JavaScript Design  0.802881
12  Android iOS JavaScript      Android iOS Java            0.925126
15  Machine Learning Engineer   Machine Learning Developer  0.839165
21  Android Developer Developer Android Developer           0.872646
25  Design Marketing Testing    Design Marketing            0.817195
28  Quality Assurance           Quality Assurance Developer 0.948010

While this has helped, taking me from 478 unique profile to 461, what I'd want to focus on are profiles like this:
Frontend Design JavaScript  Design Frontend JavaScript

The only tool I've seen which looks to address this problem is difflib?
My question is, what other techniques would be available so as to go through and standardize these profiles that consist of the same words, but out of order, to one standard string.
So desired output would be, taking a string containing "Design", "Frontend" and "JavaScript" and replacing it with "Design Frontend JavaScript".
Right now, I'm merging my original dataframe with the similarity dataframe to replace all occurrences of profile string on the right_side with the left_side, but that means I'm replacing the right_side below ("Java Python Data Science") with the left_side below ("JavaScript Python Data Science").
53  JavaScript Python Data Science  Java Python Data Science

Any help would be greatly appreciated!!!

"fuzzy match" is a vague term for many different kind of string similarity measures. There are two main categories:

Sequence-based methods, such as Levenshtein edit distance (and variants), Jaro-Winkler
Bag-of-something methods, such as Cosine-TFIDF, Overlap, Jaccard, ...

Traditionally the first category is used at the level of characters (e.g. to match "abcdefg" with "abdfeg") and the second at the level of words (e.g. to match "the management of the team" with "the team management"), however there is no technical restrictions to keep it this way. The main difference between the two categories is whether or not they take the order of the units (e.g. words or characters) into account. In fact there are also some hybrid measures such as SoftTFIDF which combine both approaches.
Conclusion:

For the problem of word order, a simple bag-of-words measure such as cosine or Jaccard would perfectly match two profiles.
If more flexibility is needed, Soft-TFIDF or other variants would work.

The last part of the question is about selecting a standard string. In general there's no way to know what should be the standard string from the different variants themselves. A simple approach would be to pick the most common variant.


Minimum numbers of support vectors I'm trying to understand the concept of SVM.
Consider linearly separable data $\{(x_i , y_i )\}_{i=1}^n , x_i \in \mathbb R^d , y_i \in \{−1, 1\}. \text{Let}\ \ \{x | w^T x + b = 0\}$ be the margin-maximizing hyperplane that separates the data.
Intuitively I understand that since there is a classification problem with at least 2 classes, there should be at least two support vectors (one for each class). But is there any formal proof of that (the minimum number of support vectors is 2)? And could there be more than two support vectors? If so, can you give an example?

There is an answer on this question: https://stats.stackexchange.com/questions/259290/theoretical-minimum-number-of-support-vectors

Also, follow the links there to see more...


How to set a class_weight Dictionary for Random Forest? I'm dealing with an unbalanced dataset, so I decided to use a weight dictionary for classification.
Documentation says that a weight dict must be defined as shown below:
https://imbalanced-learn.org/stable/generated/imblearn.ensemble.BalancedRandomForestClassifier.html
     weight_dict = [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}]

So, since I want to predict 12 classes which are located in the last column.
I assume that the setting would be like:
weight_dict = [{0: 1, 1: 5.77390289e-01}, {0: 1, 1: 6.48317326e-01},
               {0: 1, 1: 1.35324885e-01}, {0: 1, 1: 2.92665797e+00},
               {0: 1, 1: 5.77858906e+01}, {0: 1, 1: 1.73193507e+00},
               {0: 1, 1: 9.27828244e+00}, {0: 1, 1: 1.18766082e+01},
               {0: 1, 1: 8.99009985e+01}, {0: 1, 1: 6.39833279e+00},
               {0: 1, 1: 2.55347077e+01}, {0: 1, 1: 9.47015372e+02}]

Honestly, I don't clearly understand the notation of the first indicators, I mean the:
      0:1 of {0: 1, 1: 1}

or the:
 1: value.

Do they represent column position, label order?
What is the right way to set it?
I'll be grateful for your insights.

If you're just doing multiclass classification, you should specify the weights as a single dictionary, e.g. {0: 1.0, 1: 1.5, 2: 3.2} for a three-class problem. (Or use the convenience modes "balanced" or "balanced_subsample").
The list of dictionaries is used for multilabel classification (where each row can have multiple true labels).  In that case, each dictionary is for one of the outputs, the keys being the classes for that output and values the weights.


Question about the Silhouette number for K means clustering I have used K means clustering. In order to find the best value for K, I've looked at the changes of inertia value vs K and also changes of average Silhouette number vs K. The graph for inertia seems to indicate there are 5 clusters in the data. However, the average Silhouette number reaches a minimum at 5. So, how does one interpret this? If there are 5 clusters in the data, shouldn't the average Silhouette number have a maximum at 5 (not a minimum)?

Thanks

These are only heuristics.

The are not very reliable, and often fail. They will not detect if you did not preprocess your data well.

So instead of looking at some number, look at your data instead.


CNN or Viola-Jones for facial detection I was wondering since CNNs have dominated every image-related task. Is the Viola-Jones face detector still considered state-of-the-art, or have CNNs surpassed its performance?

According to the paper A Comparison of CNN-based Face and Head Detectors for
Real-Time Video Surveillance Applications CNN based neural networks are significantly better than Viola-Jones in terms of precision but need more computational power to calculate the results. The mean average precision error for CNN based networks is 5 times less than for VJ for FDDB data. Here is the precision comparison table from the paper:

But VJ allows real time recognition with 60 frames per second. Here is the speed comparison from the same paper:


CNN Multi-Class Model Only Predicts 1 class for all test images I am trying to build a CNN model to predict 42 classes. I used pre-trained models for this. I used Xception.
This is how I have imported my dataset:
train_datagen = ImageDataGenerator(rescale =1.0/255.0,
                                   zoom_range = 0.2,
                                   shear_range = 0.2,
                                   horizontal_flip = True,
                                  validation_split = 0.2)
training_data = train_datagen.flow_from_directory(train_path,
                                                  target_size = (299,299),
                                                  batch_size = 32,
                                                  class_mode = 'categorical',
                                                 subset = 'training')
validation_data = train_datagen.flow_from_directory(train_path,
                                                   target_size = (299,299),
                                                   batch_size = 32,
                                                   class_mode = 'categorical',
                                                   subset = 'validation')

I then built my model:
import keras
prior = keras.applications.Xception(include_top = False, weights = 'imagenet', input_shape = (299,299,3))
model = Sequential()
model.add(prior)
model.add(Flatten())
model.add(Dense(256, activation = 'relu'))
model.add(Dropout(0.1))
model.add(Dense(42, activation = 'softmax', name = 'Output'))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

My model performs quite well but I do not know why when I predict it on the test set, it predicts all the same class.
This is the epochs:
Epoch 1/3
2636/2636 [==============================] - ETA: 0s - loss: 1.9625 - accuracy: 0.4816
Epoch 00001: val_accuracy improved from -inf to 0.56412, saving model to xception.hdf5
2636/2636 [==============================] - 5041s 2s/step - loss: 1.9625 - accuracy: 0.4816 - val_loss: 1.6792 - val_accuracy: 0.5641
Epoch 2/3
2636/2636 [==============================] - ETA: 0s - loss: 1.4015 - accuracy: 0.6224
Epoch 00002: val_accuracy improved from 0.56412 to 0.64584, saving model to xception.hdf5
2636/2636 [==============================] - 5101s 2s/step - loss: 1.4015 - accuracy: 0.6224 - val_loss: 1.3240 - val_accuracy: 0.6458
Epoch 3/3
2636/2636 [==============================] - ETA: 0s - loss: 1.2084 - accuracy: 0.6747
Epoch 00003: val_accuracy did not improve from 0.64584
2636/2636 [==============================] - 4968s 2s/step - loss: 1.2084 - accuracy: 0.6747 - val_loss: 1.4000 - val_accuracy: 0.6373

However, when I predict its all one category:
['39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',
 '39',

Any help is appreciated. I would also like to know how I can use pre-trained models for these.

73 millions trainable parms

- When using Transfer learning we first freeze the base model
- Train it till you reach good accuracy
- Then unfreeze it and train for just few epochs. Keep LR small
Other probable issues -
- If your labels are not One-Hot coded, please use sparse_categorical_crossentropy
- Add validation_split in fit method
- Suggest you add a keras.layers.GlobalAveragePooling2D after the base model and before flattening it

"setting include_top=False: this excludes the global average pooling layer and the dense output layer"

Can use this code as guidance
base_model = keras.applications.xception.Xception(weights="imagenet", include_top=False)
model = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(n_classes, activation="softmax")(model)
model = keras.Model(inputs=base_model.input, outputs=output)

for layer in base_model.layers:
    layer.trainable = False

optimizer = keras.optimizers.SGD(lr=0.2, momentum=0.9, decay=0.01)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=15, validation_data=valid_set)

for layer in base_model.layers:
    layer.trainable = True

optimizer = keras.optimizers.SGD(lr=0.01, momentum=0.9, decay=0.001)
model.compile(loss="sparse_categorical_crossentropy", optimizer=optimizer, metrics=["accuracy"])
history = model.fit(train_set, epochs=5, validation_data=valid_set)

Code Ref - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition


How do we go about imbalanced data for prediction problem? As in classification we have imbalanced classes, we use up-sampling or down-sampling and other techniques, what do we do when we have imbalanced data in prediction problems, for example, I have distribution of outputs like 90% value1, 5% value2, 3%value3, 2% value4?

In addition to post linked by @Wes:

You should to check other metrics                than accuracy like
precision, recall (especially), F1 score or similar.
After a certain threshold (lower than the one provided in your
example definitely), it might be worthwile to use anomaly detection
and use your lesser present classes as such.

I would advise against downsampling as it's leaving informations about data, and IMO you would usually want to avoid that.


Why does test data need to be normalized on train data mean and std? I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data.

Why is it that we can not take the mean and std of the whole dataset combined (train and test data) and then normalize around these values?

The reason you split your dataset to Training and Test is to simulate real-world cases. What you actually do with the train-split validation is to evaluate your model in unknown data.

Imagine now that you have trained your model and you are on a production where new data keep coming for prediction. You might not get them in mass, but one by one such as in an API call. You don't have the mean and standard deviation of those "new" data. You only have the mean and std during the training process.

To sum up, train-test validation tries to be as close as possible to the real problem. And since you won't know anything about your upcoming data, you should not use any knowledge you get from the test data.


Importance of variable to implement in Logistic Regression model in Python sklearn? Logistic regression using Python sklearn

In Logistic Regression model we can implement variables which meet 3 conditions:

independent variables have to be correlated with dependent variable
independent variables have NOT to be correlated with each other
only relevant variables can be added to the model, otherwise it does not make sense

and both 1. and 2. point we can achieve using Pearson correlation for example:

-- correlation between independent variables and dependent variable (targetVariable

1. CORREL = data.corr().sort_values('TargetVariable')

--heatmap which shows lvl of Pearson correlation between independent variables (the closer 0 the lower the correlation)

2. sns.heatmap (data.corr(), cmap="coolwarm", annot=true)

But when I choose appropriate variables after 1. and 2. step how can I meet the 3. step and choose which variables (which are correlated with independent variable and not correlated with other independent variable) are important and I should use it in model and which are not important and I should out them? Maybe p-value and delete all with p-value higher than 0,05? But how?

Please give me the example code to meet 3. step.

The problem you are describing is feature selection. Welcome to a particularly sticky problem. Your method of Pearson correlation certainly can help - simply drop those features with low and or insignificant correlation to your dependant variable. This is univariate selection.  Since you are using a logistic regression, you can also inspect the model coefficients and odds ratios to determine the effect size and significance of each feature.

But what if some combination of your independent variables produces a significant effect within your model? Say the combination of: gender, age, income, and house value for a binary response related to likelihood to purchase. This is multivariate feature selection.

For logistic regression, and other models, you can use "interaction terms" to investigate the combined effect of input variables. In Data Science and Machine Learning literature this would fall under the practice of feature engineering. There are a number of other procedural and rule of thumb methods to select inputs for multivariate models. For instance, one can use a decision tree method to evaluate important features across repeat random samples of the data, or one can simply use domain and conceptual knowledge of the process being modeled. In most cases you should use a variety of techniques and choose the inputs that make both mathematical and theoretical sense.


Model to predict based on frequency of occurrence I have the following dataset

+-----------------------------------+
|  Passenger           |    Trip    |
+-----------------------------------+
| John                 | London     |
| Jack                 | Paris      |
| Joe                  | Sydney     |
| John                 | London     |
| John                 | London     |
| Jill                 | New york   |
| Jim                  | Sydney     |
| Jack                 | Paris      |
| James                | Sydney     |
+-----------------------------------+

And am trying to use scikit library to predict the likelihood of next possible trip of a passenger based on the frequency ( In this case John => London).
As a novice am unsure on which model / function to use.

Update 2:

If I have over 10 million records , how different should I approach this problem ?

For something like this, you could go with a simpler approach. One idea is to sample randomly among the cities that a given passenger has visited using the amount of times each city has been visited as probabilites.

Here's a way you could do so. I've added a few more examples to the dataframe so that the application is seen more clearly. Say you instead have:

     Passenger    Trip
0       John     London
1       Jack     Girona
2       Jack      Paris
3        Joe     Sydney
4        Joe  Amsterdam
5        Joe  Barcelona
6        Joe  Barcelona
7       John     London
8       John      Paris
9       Jill    Newyork
10       Jim     Sydney
11      Jack      Paris
12     James     Sydney

You could define a function like the folllowing in order to randomly sample from the existing data in the dataframe:

def random_sample(df, name):
    import numpy as np
    # group the dataframe by Passenger and count
    # the different trips
    g = df.groupby('Passenger').Trip.value_counts()
    # Make the probabilities add up to 1
    freq = g[name] / g[name].sum()
    # random destination based on
    # its probabilities
    random_name = np.random.choice(a=freq.index, size=1,
                     p = freq.values)[0]
    # return likelyhood of next randomly chosen
    # destination and destination
    return freq[random_name], random_name

 Usage

Say we want to select a a randomly samples destination for say Joe and also to know which is the likelihood. Considering that the destinations where Joe has been are:

Trip
Barcelona    2
Amsterdam    1
Sydney       1

We could get for example:

for _ in range(5):
    freq, dest = random_sample(df, 'Joe')
    print('Chosen destination {} with a probability of {}'.format(dest, freq))

Chosen destination Sydney with a probability of 0.25
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Sydney with a probability of 0.25