What does the "Loss" value given by Keras mean? I setup my neural net to use mean square error as shown below. To my understanding (and from reading the documentation) this means that if the correct result of a row is 0.7 and the net predicts 0.8 the contribution to the loss by this entry is (0.8 - 0.7) squared = 0.01

from keras.models import Sequential
from keras.layers import Dense

#...build up neural network layers here...

net.compile(optimizer = 'adam', loss = 'mean_squared_error')
net.fit(training_data, training_results, batch_size = 4, epochs = 100)

I get the following output.

Epoch 100/100
1190/1190 [==============================] - 0s 133us/step - loss: 0.0082

Wow the the loss is tiny, my little neural network is doing so well! However if I validate the result on my original training data

prediction = net.predict(training_data)
prediction_delta = (prediction - training_results)

Although some of the values in the prediction_delta are small overall the loss is way higher than 0.0082 with single values as high as 0.44. Note that this is for the same training data used to fit the net and is not the test data (which also shows similar results) so I would expect to get the value 0.082 back. How does Keras calculate this loss number?

In this particular case, the loss is Mean squared error
i.e. the mean of the squares of the differences of your model prediction and the real value of each sample.


How to pass 2 features to LSTM , one of them is one-hot-encoded with Keras? I have a very simple LSTM model

model = Sequential()
model.add(LSTM(64, input_shape=(seq_length, X_train.shape[2]) , return_sequences=True))
model.add(Dense(y_cat_train.shape[2], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, y_cat_train, epochs=100, batch_size=10, verbose=2)

The input X_train has 2 feature , one is categorical (values 1-4) and the other is numeric (values 1-100). There are 4 classes in y_test that I one-hot-encoded with keras's to_categorical .

Should I encode the categorical input feature as well ? If I do , how can I pass it along with the other feature ? (e.g. now a timestep looks like this for example: [1,44])
Later , I would like to take make a sampling , meaning I need to take the predicted y_hat<t-1> and pass it as x<t> . I will have to pass the second numeric feature (1-100) along with it. How can it be done ?

EDIT : note that I do not want my numeric feature to become categorical since there is importance to the values (meaning 2<10<90 etc)

(1) Yes, it is a common practice to encode the categorical feature by one-hot encoding, for example, encode [1,44] as [1,0,0,0,44], encode [2,44] as [0,1,0,0,44], etc.

(2) Same as (1), just concatenate the one-hot encoded categorical feature and the numerical feature(s).


Artificially Increasing Training data I am trying to predict sales quantity based on attributes of the item.Sales are aggregated by week wise and prediction is also done by week wise.I am having large number records with zero sales quantity than compared to positive sales(for 20 positive sales 250 zero sales records are there).I want to increase my training data by adding small values to sales quantity and combining with total records.
At present I am using RANDOM FOREST and NEURAL NETWORK .I am not getting any good results
Please correct me if anything is not considerable

I have tried SMOTE for categorical response variable scenario but never tried on Continuous response variable but I think that should not matter. You can run oversampling or undersampling based on the predictor variables. If you are using R, you can use DMwR library and use the SMOTE function. In Python you got to use imblearn.over_sampling.SMOTE


How to use fresh data when target prediction period is long? I'm using supervised learning on monthly activity data to predict when a customer buys a particular product. This product is typically bought infrequently and at the moment my target variable is whether the customer buys the product in the next twelve months.

Assume that for every customer I get a set of features every month, $x_1,x_2,\ldots,x_n$. The goal is to use these features to predict whether $y=0$ or $y=1$ ($y$ is 1 if the customer did buy the product in the next twelve months, otherwise it is zero).

However, this creates a dilemma. If I use this approach for $y$ then my freshest training data is twelve months old as I do not know the true value for $y$ for data that is younger than twelve months old. My main question is thus the following: Is there a way for me to make use of newer data in this setting?

Also, I should note that I have tried changing $y$ into: "Does the customer buy the product in the next month?". It works but not nearly as well as the other approach. My data is imbalanced so by allowing the target period to be composed of the following twelve months instead of a single month I get many more positive data points.

I am confronted with the same problem and I am afraid the answer is quite generally no. You'll have to wait the length of the time horizon to use new data in your calibration process. Similarly, for validating your prediction you will have to wait the time horizon.
Somehow building a target from some information you have would not yield interesting results. Because you will build the target based on some features, once you put your features trough a ML model, said ML model will just learn what rule you used to build the target.
As you mentionned the main solution is to build a shorter horizon target. A shorter horizon might help your main problem, but if it too short you will get stability problems. (And generally speaking a bad target, if for exemple, the sale usually happen 3 months after, using a 1 month horizon target won't help you learn the relationship between feature and the sale). Maybe you should try other target horizon between 1 month and a year.
Note that the time horizon you want to use might also depend on which frequency you observe your instances. Using a time horizon for a target that is longer than the time interval between your observation will create correlation between targets. This might get really bad for separating independent subsets (like for train and test set splitting or using cross validation).


Timeseries VAR vs VARMA model: issue in time to fit model I want to use VARMA model on a data of about 80000 samples with 10 features. I tried using VARMA model from statsmodels with p=50 and q=10 but it is taking too much time to build the model. I tested the model was running even after 12 hours. Then I tested VARMA using p=50 and q=0, this also was running even after an hour with maxiter=1. The code I am using is:

from statsmodels.tsa.statespace.varmax import VARMAX
modelVARMA = VARMAX(dff, order=(50,0))
resultsVARMA=modelVARMA.fit(maxiter=1)

I also tested VAR model with p=50 and it gave the model quickly.
Is there any way to run VARMA model fast as it is in case of VAR.

It's because 80,000 samples is probably too much training data. Usually, statistical forecasting models are not trained on the entire training set like ML models. Try reducing your training data significantly, and incorporate only a certain number of seasons as your training data.

Example:
If your data is hourly, one season can be 24 samples. Then, the training data can span 14 seasons - 24*14 samples.


How to match a user with another user based on their taste? Information available

Consider that there are N users on a platform. Every user adds items that they like on their profile. These items have static attributes that describe the product.

User A:
Row   | Attribute a | Attribute b | Attribute c
Item 1|    0.593    |    0.7852   |   0.484
Item 2|    0.18     |    0.96     |   0.05
Item 3|    0.423    |    0.886    |   0.156

User B:
Row   | Attribute a | Attribute b | Attribute c
Item 7|    0.228    |    0.148    |   0.658
Item 8|    0.785    |    0.33     |   0.887
Item 9|    0.569    |    0.994    |   0.374

User A has a list of items that he/she likes. Same goes with User B... User N. The items in the profiles of different users might or might not be the same but the items describe the User's taste for that particular item.

Goal

What I want to do is, match a User with another User if they have a similar taste in picking items. I don't understand how to achieve this. Any help is appreciated!

You can perform clustering of your customers based on a distance function.
Definition might look like this:

First, calculate euclidean distances between the first item of the first customer's basket and all of the items in the second customer's basket.
Then find out, what is the closest item from second customer's basket (minimum euclidean distance).
Perform the same operation for each item in first customer's basket.
Calculate mean of the minimum distances.
Do the same for the second customer.
Take maximum of means from the first and the second customer.


In Mini Batch Gradient Descent what happens to remaining examples Suppose my dataset has 1000 samples (X=1000) . I choose batch size of 32.
As 1000 is not perfectly divisible by 32 , remainder is 8.
My question is what happens to the last 8 examples. Are they considered? If they are, will they effect the efficiency of my model.
def next_batch(X, y, batchSize):
     for i in np.arange(0, X.shape[0], batchSize):
           yield (X[i:i + batchSize], y[i:i + batchSize])

This code is from a book and according to me this code is not considering the last remaining data points

It's an implementation-dependent point but there is no reason that the last few records should be left.
In Keras - It takes the remaining data points as the last step.
Addition of one extra elemtn increases the steps by 1.
Case-I - Data count is divisible by batch_size
epochs = 1
batch_size = 16
history = model.fit(x_train.iloc[:864], y_train[:864], batch_size=batch_size, epochs=epochs)

54/54 [==============================] - 0s 3ms/step

Case-II - Adding an extra data point
epochs = 1
batch_size = 16
history = model.fit(x_train.iloc[:865], y_train[:865], batch_size=batch_size, epochs=epochs)

55/55 [==============================] - 0s 3ms/step -

In your example too, same thing is happening
batch_size = 16
np.arange(0, x_train.shape[0], batch_size)

.....672, 688, 704, 720, 736, 752, 768,832, 848, 864])

When the last slice will happen, it will be a batch of 11 datapoints
len(x_train[864:880])  # Although x_train end at 875

11


What is the meaning of pct_trend_adjusted in FiveThirtyEight's polling data? I'm working on polling data from fivethirtyeight. What is the meaning of the pct_trend_adjusted column here? I understand that pct stands for percentage, but I don't get what the trend is that they are adjusting for, and how these figures are obtained.

Looking at the methodology provided by FiveThirtyEight, it seems as though this figure (i.e. polling figures on state level) is adjusted for trends in national polls. See differentiator 1 on the linked page for more info:

Basically, in states that haven’t been polled recently, we make
inferences about what’s going on there using national polls or polls
from other states that have been surveyed recently


Decision trees for anomaly detection Problem
From what I understand, a common method in anomaly detection consists in building a predictive model trained on non-anomalous training data, and perform anomaly detection using the error of the model when predicting on the observed data. This method requires the user to identify non-anomalous data beforehand.

What if it's not possible to label non-anomalous data to train the model?
Is there anything in literature that explain how to overcome this issue?

I have an idea, but I was wondering if someone has heard for something similar before, and could point me to the right direction (link papers/blogs, or explain existing methods).
Idea
I'd like to train a decision tree on a dataset $X$ with $N$ rows, $p$ columns, having a real valued target variable $Y$ (it's a regression problem). The dataset $X$ contains both anomalous and non-anomalous objects. The decision tree training process generates groups of objects, splitting the dataset iteratively along one dimension, at each iterations. The decision trees during prediction assigns an object to a specific leaf node. Each leaf node will have a certain distribution of values of the target variable Y. An anomalous object in this problem it's the one which doesn't perform well, more precisely an object for which Y is too low.

Can I use the distribution in a leaf node to perform anomaly detection?
Assuming that a bigger value for Y is preferred, can I say that the entities in the lowest 5th percentile of Y in a node are outliers?

Example
The decision tree assigns to the node j the objects following the rule: $2 <= x_1 < 4$ and $5 <= x_2 < 7$, where $x_1$ and $x_2$ are two columns of the dataset. If I run a prediction on the entire dataset, the values of Y in the node j have have a gaussian distribution (mean=25, std=3), and I consider the values with $Y < 20.5$ outliers. The idea is that objects with $x_1$ and $x_2$ within a range, should perform similarly, and never below a target value.
Considerations
*First of all, I can see that this problem can be generalized to other methods. However, I find decision trees easier to explain here, moreover, I find the hard clustering property of trees useful for my problem, as I also need to cluster objects together.
*Second, I can see the issue of training the model with both anomalous and non-anomalous data. Though, I'm wondering whether due to some property, with a big amount of data, the leaf node's distribution move's toward the optimal (value of Y of the non-anomalous objects). But I guess this depends on the ration between anomalous vs non-anomalous objects.
Any hint in the right direction would help.

I will try to refer to the points I have some opinion about:

"What if it's not possible to label non-anomalous data to train the model?"
In this case, you face unsupervised learning problem. there are plenty of reading material regarding this topic and plenty of approaches.
here is one for example:
https://towardsdatascience.com/unsupervised-learning-for-anomaly-detection-44c55a96b8c1

this article may be good as well:
https://www.researchgate.net/publication/224375498_Anomaly_Detection_by_Combining_Decision_Trees_and_Parametric_Densities

if you can distinguish anomalies by their distribution, it sounds that you do know the anomaly data, am I wrong?

the issue with defining anomalies lower than 5th quantile, that it means that 5 % of your data that examined by the leaf will be considered as anomaly, so you if run you algorithm frequently, there is a good chance you will see a lot of anomalies.
maybe try using confidence intervals.

From what I have read, decision trees are not the classic method for anomaly detection.
read about fraud detection, isolation trees, using SVM and there are more ways.


AUC ROC in keras is different when using tensorflow or scikit functions. Two solutions for using AUC-ROC to train keras models, proposed here worked for me. But using tensorflow or scikit rocauc functions I get different results.

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

and

def auc(y_true, y_pred):
    return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)

Based on the history, it looks like both are being applied to train and validation.
When I plot history metrics, tensorflow curve looks very smoothed compared to scikit.

Shouldn't I get about the same results using both functions?

No, you shouldn't have the same numbers. All depends on the additional parameters:

tf.metrics.auc(
    labels,
    predictions,
    weights=None,
    num_thresholds=200,
    metrics_collections=None,
    updates_collections=None,
    curve='ROC',
    name=None,
    summation_method='trapezoidal'
)

This means that this curve will have 200 points, so very smooth.

sklearn version doesn't have this kind of parameters:

roc_auc_score(y_true, y_score, average=’macro’, sample_weight=None, max_fpr=None)

The number of outputs depends on the curve and the number of points if I remember properly.


How to properly represent a tic tac toe board to a CNN? I'm figuring out how to manipulate convolutional neural networks (CNN) in python and I want to apply this kind of NN to an agent player that plays tic tac toe. I know that's weird and the problem doesn't need some elaborate and complex solutions like CNN, but I'm using this environment just to learn better and compare with the training of a MLP that I implemented.

So, I'm trying to think about how to represent the board correctly as an input to the CNN. Unlike the chess board which we can think the input as 8x8x6 (8x8 as the 2d array representing the board and 6 channels which one representing the different pieces of the game), the tic tac toe is a bit more complicated for me because there are just one kind of piece. Is it possible and correct to represent the X and O as different pieces? Has anyone implemented something like this?

I used a tic tac toe board to explain the value of convolutions in my master's thesis (https://apps.dtic.mil/dtic/tr/fulltext/u2/1046500.pdf checkout page 21-23 if you are interested).  I think this illustration (extracted from my thesis) might answer the question on how you could represent the board to the CNN and what you hope your filters might learn through training. Presumably you could train the network to recognize playable spaces and predict best X/O placement as you add more layers to your network.  While this doesn't tell you how to do this, I think my diagram will be useful in developing what type of input you should provide your network and some of the dimensions in order to answer your question.


How to make a decision tree with both continuous and categorical variables in the dataset? Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables?
Edit:
For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to be primarily with the continuous attributes. Let's say I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?

Decision trees can handle both categorical and numerical variables at the same time as features, there is not any problem in doing that.
Theory
Every split in a decision tree is based on a feature. If the feature is categorical, the split is done with the elements belonging to a particular class. If the feature is contiuous, the split is done with the elements higher than a threshold. At every split, the decision tree will take the best variable at that moment. This will be done according to an impurity measure with the splitted branches. And the fact that the variable used to do split is categorical or continuous is irrelevant (in fact, decision trees categorize contiuous variables by creating binary regions with the threshold).
Implementation
Although, at a theoretical level, is very natural for a decision tree to handle categorical variables, most of the implementations don't do it and only accept continuous variables:

This answer reflects on decision trees on scikit-learn not handling categorical variables. However, one of the scikit-learn developers argues that

At the moment it cannot. However RF tends to be very robust to categorical features abusively encoded as integer features in practice.

This other post comments about xgboost not handling categorical variables.
rpart in R can handle categories passed as factors, as explained in here
Lightgbm and catboost can handle categories. Catboost does an "on the fly" target encoding, while lightgbm needs you to encode the categorical variable using ordinal encoding.

Here's an example of how lightgbm handles categories:
import pandas as pd
from sklearn.datasets import load_iris
from lightgbm import LGBMRegressor
from category_encoders import OrdinalEncoder

X = load_iris()['data']
y = load_iris()['target']

X = OrdinalEncoder(cols=[3]).fit_transform(X)

dt = LGBMRegressor()
dt.fit(X, y, categorical_feature=[3])


Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)? I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose.

The presentable records of this dataset have a such structure:

+----------+------------+--------------------+--------------------+--------------------+
|         0|  07/29/2013|       Consumer Loan|        Vehicle loan|Managing the loan...|
|         1|  07/29/2013|Bank account or s...|    Checking account|Using a debit or ...|
|         2|  07/29/2013|Bank account or s...|    Checking account|Account opening, ...

After some preprocessing/data cleansing operations I would like to create and then obviously train a model that will classify issues (Issue) into some categories, that are still unknown. I have read some articles about TF-IDF, but not sure if this could be suitable for this case.

If you want to categorise your text using machine learning techniques, you have to get fixed length features from text to train any ML model. You can do
That using bag of words, tf-idf, averaging word vectors.
If you are using any deep learning based models, you can use LSTM with word vectors or CNN’s.


Forecasting using Python I have very less training observations (15). I need to predict 6 months into the future. What forecasting model is best suited for this scenario? Here is how my dataset looks
Month        | Response Rate       |% Promoters          |% Detractors        |%Neutrals
2019-01-01     | 5%                 |60%                 |30%                | 10%
2019-02-01
.....
2020-07-01
I need to predict Response Rate, % Promoters, % Detractors and % Neutrals all of which are numeric variables.
I am new to this forum, so pardon me if I have done any mistake while framing the question.

If you use any forecasting model, then it will do over fitting on your data set, as you have just 19 observations. At least 1000 observations are the good point to start applying any machine learning model.

I need to predict Response Rate, % Promoters, % Detractors and %
Neutrals all of which are numeric variables.

You can do data exploration where you start using bar charts, line charts, and try to find the pattern for each of those variables. Using this, if not accurate, but you would be able to understand lump sum forecasted value for upcoming months.
Even built in functions are there in excel. Check this out: https://www.excel-easy.com/examples/forecast.html


Which data sets would help to predict (exponential) market trends? Which kind of datasets do websites such as MeetGlimpse, trends.co, explodingtopics.com use to detect exponential market trends? I love them (not affiliated) and would like to better understand how they work.
I would assume they somehow buy access to clickstream data or something similar? Any hints?
Also, which algorithms/approaches do they use?

From a quick look, my guess would be that maybe they mine data from some predefined target websites, typically social media: twitter, facebook, etc. I could imagine that they capture trends in the sense of "what people are talking about" on social media platforms, but maybe they have other sources and methods.
There are Natural Language Processing methods which can be used: the basis would be some form of topic modelling, then there could be some more specific applications such as event detection. We can't know how much manual post-processing is involved though, there could be a manual filtering to remove irrelevant stuff.


Clustering by common elements in a list Suppose I have these elements:

a = [1, 6, 3, 4, 10, 32, 2, 54]
b = [20, 5, 14, 25, 18, 1]
c = [54, 3, 6, 12, 41, 1, 9]
d = [3, 4, 1]
e = [19, 20, 25, 5]

Each identified by a list of ids (that actually represent a string but we'll use numbers for simplicity).

How can I cluster these by finding those who have the most elements in common?
For example:
a and d have 3 elements in common
b and e have 3 elements in common
a and c have 4 elements in common

So the clusters I would like to have ar (a, c, d) and (b, e).
I would like to obtain this in a Python script

In order to do clustering, you only have to define a distance measure. When you have defined a distance, you can apply K-means, hierarchical clustering or other algorithms. In your case, I would define the following distance function:

$ d(a, b) = 1 - \frac{\text{number of common elements of a and b}}{\text{maximum between lengths of a and b}}$

As $d(a, a) = 0$ and $d(a, b) > 0$ hold, this can be a distance. I don't think the triangle inequality holds, so this will not be a well-defined distance. However, you can still try to work with this function, it might give good results for your aim.

Edit: sklearn implementation

In order to implement this using scikit-learn, the way to make it work is to use DBSCAN, set metric='precomputed' and pass the distance matrix of our data as an argument. We have to use the precomputed distance matrix because scikit-learn does not allow (now) to use custom distances for clustering.


Classifying one particular class of documents from the rest I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I would sample 'not a sports document' category as there can be anything - book, news article, resume, invoice etc. How would approach this problem?

I already tried training One class SVM classifier with my sample of sport documents, however the accuracy turned out to be awful - around 6%.

I also read about PU learning, do you think this is the way to go? Are there any other options?

Thank you.

Your problem is indeed a typical one-class classification problem, and as far as I know one-class SVM is usually a good option for that.

I think you should investigate what causes the poor performance:

Evaluating with accuracy is probably not informative enough, you would need to find out at least whether the errors tend to be mostly false positive or false negatives, thus using precision/recall.
You could look at what is happening at the level of features: I would expect some words specific to sports to be assigned a strong weight by SVM for instance. It could also be a problem with the dimensionality being too high, maybe you need to remove stop words or filter out rare words, etc.


ideal algorithms to demonstrate overfitting or underfitting When one tries to look up concepts such as overfitting and underfitting, the most common thing that pops up is polynomial regression. Why is polynomial regression often used to demonstrate these concepts? Is it just because it can be easily visualised like the graphs here:
https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html
But then most ml algorithm such as kmeans clustering can also be used. Then why is it usually polynomial regression only? Are there any other similar algorithms that could be used?

Another example you could use is the separation boundaries in decision trees classification problem. In the picture below, you can see that the training error continues to go down (lower the better) when max_depth increases, whereas the testing error is not as good. This is because the model has carved out specific pink areas (for 'x') as the separation boundary based on the training data. When these boundaries are applied to another set of data, they become poor separation boundaries. So these boundaries cannot be generalised to other testing data sets.


How can I encode a 'Name' so that similar names are represented by vectors close in n-dimensional plane? I want to encode names of people for similarity comparison between them such that a name like 'Sarah' is closer when represented in vector to a name like 'Sarah connor', something very similar to what word2vec does but it uses sentences to train but I only have list of words. Using string matching algorithms like Levenshtein distance and Jaccard Index, I can find similarity between them but this can't be used to derive vector embeddings of these words satisfying above conditions, or can they ? If not is there a way to encode these list of names such that the condition of similar names (based on characters and other conditions) are closer in n-dimensional space, n being the vector length of these embeddings.

You can do that. I propose the simplest one conditioned on the fact that number of data is not very large. In case you need more ideas please drop a comment.
In this case, you can use the idea of similarity encoding based on Fuzzy String Matching and get the spectral embedding. The amount of data is crucial here as you need to do order of $n^2$ comparisons to get affinity matrix for spectral embedding. Follow the code bellow (and get a wonderful idea from this paper)
data = ['sarah connor', 'sara jones', 'jack blabla', 'jackie jones', ' jakob blabla', 'sara conor']
n = len(data)

aff_mat = np.zeros((n,n)) # This is the S matrix in the paper
D = np.zeros((n,n))

for ii in range(n):
    for jj in range(n):
        name1 = data[ii]
        name2 = data[jj]

        surname1 = name1.split()[0]
        lastname1 = name1.split()[1]

        surname2 = name2.split()[0]
        lastname2 = name2.split()[1]

        aff_name1_name2 = fuzz.ratio(surname1,surname2) + fuzz.ratio(lastname1,lastname2)
        # Fuzz ratios are betweein 0 and 100 and we add 2 of them
        # so we normalize the whole score to 0 and 1 by dividing by 200
        aff_mat[ii,jj] = aff_name1_name2/200

for ii in range(n):
    D[ii,ii] = np.sum(aff_mat[:,ii])

L = D - aff_mat # This is Laplacian matrix

Having the Laplacian matrix, you simply calculate the eigenvectors, directly from the code inside the paper. Here I choose second and third eigenvector as the forst eigenvector is trivial. please not that there are tones of ways to calculate Laplacian matrix and what we did here is different that the one in the paper. So despite the paper which chooses first $k$ eigenvectors, we drop the first one. For more details on this you may refer to the literature.
# compute eigenvectors / eigenvalues of L
evals, evcts = eig(L)
# extract "smallest" 2 eigenvectors (ignoring first one)
sortedevals = argsort(evals)
U = evcts[:,sortedevals[1:3]]

Now U is your embedding in 2 dimensions. Just plot and see the result:
for (x,y), label in zip(U, data):
    plt.text(x, y, label, ha='center', size=10)
plt.xlim((-1,1))
plt.ylim((-1,1))
plt.show()

and this is the result:

Now it's up to you how you would like to query similar names. The main job is done.
PS: As obvious above, I assumed you are interested in similarity of both first names and last names. In case you want the same code only for first name just simply take the $lastname$ variable out.
Hope it helped. Good Luck!


For obejct detection, should I resize my custom images first and then start the annotation or it won't matter? I have my custom dataset images of size (1080 x 1920) and I am trying to use yolov3 for object detection. I noticed that yolov3 model accepts an input image size of 416 x 416. So I am in confusion if I should resize the image and apply zero-padding to save the aspect ratio and start my annotation after that OR should I annotate my custom images in original size? And will data argumentation affect the annotation while training? Thanks

It is best practice to resize the input data to the size a pretrained model expects. A pretrained model has feature detectors for relative differences.
Resizing should be done early in the process so subsequent steps use consistent data. Sometimes resizing is a non-linear process that can use interpolation.


How to retrieve images from a url in a pandas dataframe and store them as PIL object in a new column I'm trying to store as a PIL object in a new column of a dataframe pictures that are located in a column of the same dataframe in the form of URL's.

I've tried the following code:

import pandas as pd
from PIL import Image
import requests
from io import BytesIO

pictures = [None] * 2

df = pd.DataFrame({'project_id':["1", "2"],
                    'image_url':['http://www.personal.psu.edu/dqc5255/gl-29.jpg',
                                'https://www.iprotego.com/wp-content/uploads/google.jpg']})

df.insert(2, "pictures", pictures, True)

for i in range(2):
    r = requests.get(df.iloc[i,1])
    df.iloc[i,2] = Image.open(BytesIO(r.content))

df

I expected to get a dataframe with this structure but including both training examples:

    project_id                  image_url                                  pictures
0       1    http://www.personal.psu.edu/dqc5255/gl-29.jpg <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=400x300 at 0x116EF9AC8>

But instead got the following error:

OSError: cannot identify image file <_io.BytesIO object at 0x116ec2f10>
```

I just changed the User-agent in the for loop so that now the request line in the loop is: r = requests.get(df.iloc[i,1], headers=headers) with headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/XXX.XX (KHTML, like Gecko) Chrome/XX.X.XXXX.XXX Safari/XXX.XX'} and this solved the error. I also added a r.raise_for_status() to check the status before using the r.content

Final code:

import pandas as pd
from PIL import Image
import requests
from io import BytesIO

df = pd.DataFrame({'project_id':["1", "2"],
                    'image_url':['http://www.personal.psu.edu/dqc5255/gl-29.jpg',
                                'https://www.iprotego.com/wp-content/uploads/google.jpg']})

pictures = [None] * 2

df.insert(2, "pictures", pictures, True)

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/xxx.xx (KHTML, like Gecko) Chrome/xx.x.xxxx.xxx Safari/xxx.xx'}

for i in range(2):
    r = requests.get(df.iloc[i,1], headers=headers)
    r.raise_for_status()
    df.iloc[i,2] = Image.open(BytesIO(r.content))

df
````


Confusion Matrix three classes python I want to calculate:
True_Positive, False_Positive, False_Negative, True_Negative

for three categories. I used to have two classes Cat Dog and this is the way I used to calculate my confusion_matrix

y_pred has either a cat or dog
y_true has either a cat or dog

from sklearn.metrics import confusion_matrix
confusion_matrix_output =confusion_matrix(y_true, y_pred)
True_Positive = confusion_matrix_output[0][0]
False_Positive = confusion_matrix_output[0][1]
False_Negative = confusion_matrix_output[1][0]
True_Negative = confusion_matrix_output[1][1]

Now I have three classes 'Cat' 'Dog' 'rabbit'
Y_pred has Cat Dog rabbit
y_true has Cat Dog rabbit

How to calculate True_Positive, False_Positive, False_Negative, True_Negative?

Multi-class Confusion Matrix is very well established in literature; you could find it easily on your own. Anyhow, Scikit-learn can do it easily like:

from sklearn.metrics import confusion_matrix

y_true = ['Cat', 'Dog', 'Rabbit', 'Cat', 'Cat', 'Rabbit']
y_pred = ['Dog', 'Dog', 'Rabbit', 'Dog', 'Dog', 'Rabbit']

classes=['Cat', 'Dog', 'Rabbit']

confusion_matrix(y_true, y_pred, labels=['Cat', 'Dog', 'Rabbit'])

array([[0, 3, 0],
       [0, 1, 0],
       [0, 0, 2]])

You can even plot it nicely using the below function:

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    import itertools
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.tight_layout()

like this:

cnf_matrix = confusion_matrix(y_true, y_pred,labels=['Cat', 'Dog', 'Rabbit'])
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['Cat', 'Dog', 'Rabbit'],
                      title='Confusion matrix, without normalization')

More examples here and here.


How can I create a table from an existing table in SQL but using cells from the old table as columns in the new table? I have a table,

and I want to create a new table such as the one below (from the table above)

In SQL, I tried using the following commands. I am able to generate a table with only one column like this,
CREATE TABLE table2 AS
SELECT balance
FROM table1 WHERE balance='currency'

But if I try to do multiple WHERE clause's it doesn't seem to work.
I tried to do,
CREATE TABLE table2 AS
SELECT balance, category
FROM table1 WHERE balance='currency', category= 'date_valid_from'

I also tried using an AND statement but still no luck,
CREATE TABLE table1 AS
SELECT balance, category
FROM table2 WHERE balance='currency' AND category = 'date_valid_from'

This just returns empty columns of 'balance' and 'category' and not the specified Where clauses.

ALSO, the columns in the new tables, are the ones which I selected from the old table. i.e 'balance' and 'category' However I wan't the cells within those columns i.e 'currency' in 'balance' to be the new columns.

I tried this once, but due to some database settings I was not permitted to create a table with an SQL-command. Maybe something similar is happening to you? I had to copy the DDL code (Data Definition Language) for the table and create it, and then add the rows.
Maybe you could first copy and run the DDL. Should be available in whatever program you are using for the database.
https://www.educba.com/sql-ddl-commands/
Once the table is created, it will be empty. Then you populate it. Here is some example code.
INSERT INTO table2 (column1, column2, column3, ...)
SELECT column1, column2, column3, ...
FROM table1
WHERE condition1 AND condition2 AND condition3;


How to estimate GridSearchCV computing time? If I know the time of a given validation with set values, can I estimate the time GridSearchCV will take for n values I want to cross-validate ?

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search.

E.g. suppose you want to use a grid search to select the hyperparameters a, b and c of your pipeline.

params = {'a': [1, 2, 3, 4, 5],
          'b': [1, 2, 3, 4],
          'c': [1, 2, 3]}

cv = GridSearchCV(pipeline, params)

By default this should run a search for a grid of $5 \cdot 4 \cdot 3 = 60$ different parameter combinations. The default cross-validation is a 3-fold cv so the above code should train your model $60 \cdot 3 = 180$ times. By default GridSearch runs parallel on your processors, so depending on your hardware you should divide the number of iterations by the number of processing units available. Let's say for example I have 4 processors available, each processor should fit the model $ 180 / 4 = 45$ times. Now, if on average my model takes $10 sec$ to train, I'm estimating around $45 \cdot 10 / 60 = 7.5min$ training time. In practice it should be closer to $8min$ due to overhead.

Finally, because some parameters heavily affect the training time of that algorithm, I would suggest using the max_iter argument whenever available so that your estimation doesn't fall far off.


Heat map and visualization I want to create a heat map to visualize some production data, but without geolocation. I am finishing some experiments in a greenhouse, divided in different sectors. The idea is to make a heat map to watch in which areas we are harvesting more fruits, or less, in each season.

Which tool, software or language do you think is better for this purpose? I was looking for information, but the main tools are for geolocation.

I believe that the simplest is to use Seaborn. Check out the example below:

import numpy as np
import seaborn as sns

sns.set()
np.random.seed(0)

uniform_data = np.random.rand(10, 12)
ax = sns.heatmap(uniform_data)

And the output looks as follows:

This is taken from the seaborn documentation


metric accuracy for linear regression and non linear regression Is there a one size fit all metric to measure accuracy / error rate for both linear or non linear regression models?

For example adjust R2 is only for multi linear regression (or so they say).
RMSE seems the best choice for majority except for curve lines of best fit as it can be misleading.

How do I know which metric to use?

Normally the evaluation measure doesn't depend on the method used, it depends primarily on the task being carried out.

Of course there are standard evaluation measures associated with broad types of tasks, such as classification or regression. There are technical constraints to take into account, for example whether the output is categorical or numerical. It's a common mistake to overlook this question and/or treat it as purely technical, but the choice of an appropriate evaluation setup should be made based on how well it represents the quality of the output of the ML process.

To answer your question: if the task is the same, the evaluation measure should be the same whether the method is linear regression or non-linear regression. Otherwise one would be measuring only some technical aspect specific to the method, not estimating the quality of the output in a comparable way.


R Programming, how to replicate for districts in a city I am new to R Programming and just learned basics through codeschool.com

Our network spans the city, and it is divided into districts.

I would like to create a map that assigns a value (based on ratio of outdated software and new software) to each district.

This website has sample of 3-D maps that were created by R Programming, and I see one I am very interested in replicating, but for our city only.

But when I see the source code, I don't see any mention of latitude or longitude.

My head is spinning, trying to figure out how I will input this, i.e latitude and longitude of a district in our city, versus an assigned ratio, which I believe will be read from a spreadsheet.

Thanks for any guidance.

The latitude and longitude are stored in a text file that is read into a table, Dat.  It has the form:

lat long climate-group

For your particular case, you would calculate Dat[, 3].


Backpropagation in other fields Powerful techniques are sometimes rediscovered by various disciplines at different points in time, because the particular scientific fields do not overlap or interact.

A couple of ML researchers have pointed out, that the back-propagation (Rumelhart et al, 1986) algorithm (a solution to credit-assignment) has been rediscovered by many fields.

In what other fields is backprop used? Under which names and in which context?

Beyond its use in deep learning, backpropagation has been used in many other areas, ranging from weather forecasting to analyzing numerical stability. In fact, the algorithm has been reinvented several times in different fields. The general, application independent, name is "reverse-mode differentiation" [3].

The modern version of backpropagation, also known as automatic differentiation, was first published by Seppo Linnainmaa [1] in 1970. He used it as a tool for estimating the effects of arithmetic rounding errors on the results of complex expressions [2].

Gerardi Ostrowski discovered and used it some five years earlier in the context of certain process models in chemical engineering [2].

In the sixties Hachtel et al. considered the optimization of electronic
circuits using the costate equation of initial value problems and its discretizations to compute gradients in the reverse mode for explicitly time-dependent problems [2].

Others researchers that discovered it include Bernt Speelpenning. He arrived at the reverse mode via compiler optimization when asked to automatically generate efficient codes for Jacobians of stiff ODEs [2].

More info can be found in: Deep Learning in Neural Networks: An Overview  and in Who invented backpropagation? and references therein.

[1] Who invented backpropagation?

[2] Who Invented the Reverse Mode of Differentiation?

[3] Calculus on Computational Graphs: Backpropagation


Do we have to split our dataset into training & testing when using ARIMA model? I am working on a project where I predict the total quantities sold at the ITEM/DAY leve. As for the model, I decided to with an ARIMA model (I'm using R). For guidance, I decided to follow the two tutorials below:

The first one here

The second one here

The thing that I am confused about is that, in the second one, they split the data into training and testing and they fit the model on the training set and did the evaluation on the test set (all that makes sense). However, in the first article, they didn't do the splitting.

Can anyone guide me through which is the correct approach and provide me with an explanation?

Thank you.

Splitting in Train and Test sets or not depends from the purpose of your analysis. You can follow a statistical approach or a machine learning approach.

In the classical, statistical approach, you fit a model on the whole batch of data. Your goal here is to check the sign of the variables' parameters, and whether they are significant or not. Scientifically speaking, each of those parameters represents the test of an hypothesis.

In the machine learning approach, you want a model that is good at predicting data it has never seen before. You don't care whether a given variable has a positive or negative association with you dependend variable, you don't care whether your parameters are 95% significant or not, you just care that the model predicts the output as precisely as possible.

So, the answer to your question is: it depends! What do you need your model for?


Approaches to A/B testing when you can't randomize on the user level I am trying to find suggestions for different approaches to running tests when it isn't possible to randomly assign users to test and control buckets.

For example, if I own 10 shops and have a certain algorithm that I use to price items, and perhaps I want to test a new algorithm that would result in price changes. There are a few approaches I can see but it isn't clear to me if I'm missing others and would love to hear more.

Standard A/B testing:

If I were to randomly assign shoppers to test or control buckets then this would obviously be a bad experience for them (given that prices are visible).

Switchback experiment

Perhaps instead I decide to run  a switchback experiment, in which the control pricing algorithm runs for 6 hours, then the test pricing algorithm, runs for 6 hours, and so on throughout my test period. This is obviously better than the standard A/B approach but I feel like it still has problems (ie what if the price change results in a change to long term customer behavior but it takes some time for the behavior to change - perhaps the switchback wouldn't capture this).

Causal Impact approach

Another approach might be to use a Causal Impact style analysis, in which I set certain shops as 'control' shops, and other shops as 'test' shops, and then change the algorithm at the test shops and analyze my metric of interest, where we look for a causal effect in the time series at the test shops with respect to the control shops. The drawback here is that you seem highly dependent on how well the controls represent the test shops.

Does anyone have any other suggestions, or even somewhere I can read more about different experimental setups, beyond A/B testing?

I would recommend against switch back as the purchase data is likely to exhibit seasonality (e.g. 2pm - 6pm / tuesdays exhibit lower volume, December exhibit higher volumes, company running advertisements during certain months etc.), which would become an unnecessary exogenous factor that complicates the post implementation analysis.

I am inclined towards the causal impact approach. However, there are certain things that should be checked before selecting the stores as control vs. test (it is a kick-start list, please feel free to adapt to your business domain).

Decide on the KPI to be measured as the objective, is that revenue per customer? Profit per customer? Or total profit per store? Is the algorithm designed to increase per customer KPI or more customers?
Check that the stores selected have some history to give assurance that they were similar in terms of the selected KPI before the implementation of the tests. Remember to remove outlier customers / transactions.
Check that enough stores are selected in control vs. test buckets to detect a desired threshold increase that is statistically significant.

Hope this helps! :)


'DecisionTreeClassifier' object has no attribute 'importances_' I've this code in order to visualize the most important feature of each model:

dtc = DecisionTreeClassifier(min_samples_split=7, random_state=111)
rfc = RandomForestClassifier(n_estimators=31, random_state=111)
trained_model = dtc.fit(features_train, labels_train)
trained_model.fit(features_train, labels_train)
predictions = trained_model.predict(features_test)
importances = trained_model.feature_importances_
    std = np.std([trained_model.feature_importances_ for trained_model in
 trained_model.estimators_], axis=0)
 indices = np.argsort(importances)[::-1]
    for f in range(features_train.shape[1]):
        print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))
        plt.figure()
        plt.title("Feature importances")
        plt.bar(range(features_train.shape[1]), importances[indices], color="r", yerr=std[indices], align="center")
        plt.xticks(range(features_train.shape[1]), indices)
        plt.xlim([-1, features_train.shape[1]])
        plt.show()

Using RandomForestClassifier this code runs good but when I try it using Decison Trees classifier I get the following error:

std = np.std([trained_model.feature_importances_ for trained_model in trained_model.estimators_], axis=0)

builtins.AttributeError: 'DecisionTreeClassifier' object has no attribute 'estimators_'

Which attribute should I use see the most important feature of each model?

Do Checkout this Link

  To Visualise The Tree Itself

from sklearn.tree import export_graphviz
import graphviz

export_graphviz(tree, out_file="mytree.dot")
with open("mytree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)

OR

from sklearn.tree import convert_to_graphviz
convert_to_graphviz(tree)

OR

from sklearn.tree import convert_to_graphviz
import graphviz

graphviz.Source(export_graphviz(tree))

The Visualisation You Can Get Will be Whole Tree Itself..

  To Display Feature Importances

from sklearn.tree import DecisionTreeClassifier
classifier = DecisionTreeClassifier()
classifier.fit(features, labels)
for name, importance in zip(features.columns, classifier.feature_importances_):
    print(name, importance)
    ## Now You Can Do Whatever You Want(plot them using a Barplot etc)


What is the formula and log base for idf? To calculate tf-idf, we do:

tf*idf

tf=number of times word occurs in document

What is formula for idf and log base:

Log(number of documents/number of documents containing the word)
Log((1+number of documents)/(1+number of documents containing the word))
1+Log(number of documents/number of documents containing the word)
1+Log((1+number of documents)/(1+number of documents containing the word))

There a a number of variation how to calculate inverse document frequency. Have a look at the wiki page (Tf-Idf) or scikit-learn's TfidfVetorizer class.


Unsupervised feature learning for NER I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas.

Now I want to make the same NER for different language. The problem here is that I can't use POS tags and lemmas. I started reading articles about deep learning and unsupervised feature learning.

My question is:

Is it possible to use methods for unsupervised feature learning with CRF algorithm? Did anyone try this and got any good result? Is there any article or tutorial about this matter?

I still don't completely understand this way of feature creation so I don't want to spend to much time for something that won't work. So any information would be really helpful. To create whole NER system based on deep learning is a bit to much for now.

Yes, it is entirely possible to combine unsupervised learning with the CRF model.  In particular, I would recommend that you explore the possibility of using word2vec features as inputs to your CRF.

Word2vec trains a  to distinguish between words that are appropriate for a given context and words that are randomly selected.  Select weights of the model can then be interpreted as a dense vector representation of a given word.

These dense vectors have the appealing property that words that are semantically or syntactically similar have similar vector representations.  Basic vector arithmetic even reveals some interesting learned relationships between words.
For example, vector("Paris") - vector("France") + vector("Italy") yields a vector that is quite similar to vector("Rome").

At a high level, you can think of word2vec representations as being similar to LDA or LSA representations, in the sense that you can convert a sparse input vector into a dense output vector that contains word similarity information.

For that matter, LDA and LSA are also valid options for unsupervised feature learning -- both attempt to represent words as combinations of "topics" and output dense word representations.

For English text Google distributes word2vec models pretrained on a huge 100 billion word Google News dataset, but for other languages you'll have to train your own model.


Analytic Hierarchy Process - consistency ratio over 50% I have such data of pair-wise comparison. I calculated consistency ratio for this data of 56%. That's too high for relevant results.

Whether is possible to modify initial data to improve consistency if I have no other sources what factor should I fix?

I read some articles and came up to the solution when consistency is very high like 100% I need to reduce it to min 20% by playing with most dismised values.

I recommend online calculator for AHP
https://bpmsg.com/ahp/ahp-calc.php


Get values from column that appear more than X times (without pandas or external libraries) I have a .csv file that contains a row with various loan IDs. I would like to be able to output the IDs that occur 6 or more times.

CUST_ID,PROD,BAL,PER
1151612518-107,1151612518-107c,780.1,4
1151612518-1049,1151612518-1049a,344.72,2
1151612518-813,1151612518-813a,1376.1,6
1151612518-427,1151612518-427a,913.51,4
1151612518-1056,1151612518-1056a,1851.49,7

The data is in this format and I would like to have a way to analyze the first column and output every CUST_ID that occurs 6 or more times.

The "no pandas or other external libraries" instruction was included in the homework question? ;-)

You need a group-by the CUST_ID and then select on the count in each group.

Your question doesn't specify, but you tag with Python. Otherwise I would say to load the data into an SQL database or use the great CSV toolkit.

If you need to implement it using Python, the group-by is done by sorting on CUST_ID, and then going through the sorted data (with a for-loop, for example), counting the number of repeated CUST_ID's. When the CUST_ID changes, you check how often you have seen this CUST_ID. If greater than X, then you output this CUST_ID. Otherwise, reset your counter.


Gracefully removing observations with outliers in N fields I have a function.

remove_outliers <- function(x, na.rm = TRUE, ...) {

    #find position of 1st and 3rd quantile not including NA's
    qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)

    H <- 1.5 * IQR(x, na.rm = na.rm)

    y <- x
    y[x < (qnt[1] - H)] <- NA
    y[x > (qnt[2] + H)] <- NA
    x<-y

    #get rid of any NA's
    x[!is.na(x)]
}

Given a dataset(numbers) like this:

  x
  5
  9
  2
  99
  3
  4

The functioning is obvious

remove_outliers(numbers)

means I now have this:

  x
  5
  9
  2
  3
  4

However, what if I have an ID that I want to retain, such as:

number_id    numbers
12              5
23              9
34              2
45              99
56              3
67              4

How do I remove the outlier(99) with the remove_outliers function(or another, better suited function), to get this data:

number_id    numbers
12              5
23              9
34              2
56              3
67              4

(note the entire observation with the outlier has been removed)

And how can I scale this solution to handle n more variables?

I can do it very ungracefully by taking out each column separately and building a new data frame with loops, but it's hardly readable and a mess to debug. Is there a more graceful way?

This will achieve what you want. You can remove outliers from any column you wish, just pass that column number as an argument in the function.

id <- c(12,23,34,45,56,67)
num <- c(5,9,2,99,3,4)
prac <- data.frame(id, num)

remove_outliers <- function(x, col) {

  #find position of 1st and 3rd quantile not including NA's
  qnt <- quantile(x[ ,col], probs=c(.25, .75), na.rm = TRUE)

  H <- 1.5 * IQR(x[ ,col])

  x[ ,col] <- ifelse(x[ ,col] < (qnt[1] - H) | x[ ,col] > (qnt[2] + H), NA, x[ ,col])

  #get rid of any NA's
  x <- x[!is.na(x[ ,col]), ]
  x <- assign("dataset", x, envir = .GlobalEnv)
  return(x)
}

remove_outliers(prac, 2)


Diminishing returns in language identification data set size? Most problems have a curve whereby the results improve as data are added but level off at some point.

Are there research papers or industry results that discuss the correlation between data set size and prediction accuracy for natural language identification?

LIDE: Language Identification from Text Documents benchmarked the difference between using character and word n-gram information for natural language identification.

Word-length algorithm for language identification of under-resourced languages proposed new algorithms for languages with limited available data.


What recommendation engine for a situation where users can only see a fraction of all items? I want to add a recommendation feature to a document management system. It is a server on which most company documents are stored. Employees browse the web interface and click to download (or read online) the documents they want.
Each employee only has access to a subset of all documents:

My goal: Recommend to an employee the documents recently opened by their teammates, or the spreadsheet that serves as an annex to the document they just opened, or anything they might want to read.

There are many recommendation engines for publicly available data (all Netflix users can see all movies), but the situation here is special: Each employee only has the permission to a fraction of all documents, whereas in Netflix any user has access to all movies.

Example: Employee1 can read DocumentA but not DocumentB. Employee2 can read both and Employee3 cannot read any.

Of course, I must not recommend to an employee documents to which she/he does not have access. Furthermore, I guess I should consider the popularity of a document only in the context of the employees who have access to it. To make things even more complicated, employees sometimes move from a project to another, which impacts the documents to which they have access.

Is there a name for this kind of problem?
Can it be reduced without precision/efficiency loss to a more common kind of problem?
If not, what approach would work well for this kind of problem?

Note: A Netflix-like recommendation engine is not good enough. A document with 50 views should be prominent if only 10 employees (including me) have access to it, but not prominent if 100000 employees have access to it.

In case it is needed, here are a few data specifics: The average company has 1000 employees, about 10000 documents, an employee clicks about 5 documents per day. Each project has an average of 10 employees having access to it, and has about 100 documents. Each employee works on an average of 5 projects in parallel.

I feel that you need to address two things separately.

First, You need to have access control for the users in your system. You can have access tokens attached to each user and file. Filter the files database before you process.

Second, Ranking the documents I would suggest have some weight for a document weight and user weight which relative to the current browsing user.

For example I can think of document weight and user weight as follows but they can be much more complex as per your system-

DocumentWeight = Number of Views/ Number of Users can Access
UserWeight = ## Relative to browsing user- Users in similar project will have higher weights

DocumentScore = Sum over all viewed users{DocumentWeight x UserWeight}

You can rank the documents, this will statistically pull up the documents you need. I hope this will be some help.


Combining text (NLP), numeric and categorical data for a regression problem I have a dataset

data = {
    points: 3.765,
    review: `Food was great, staff was friendly`,
    country: 'Chile',
    designation: 'random',
    age: 20

}

I am looking for a way to use these features to build a model to predict points.

Description seems to hold alot of information about points.

How do i feed this data into the model and also which model?

Note I dont want to use word2vec (embeddings)

To predict the points variable, we have four features namely review, country, designation and age.

For the review feature,

You can use Doc2Vec to transform the review into a fixed size vector. Similar reviews will lie in a close vicinity.

For the country and designation features,

Assuming these variables have finite values, so you can treat the variable as categorical. We can use one-hot encoding.

For the age feature,

You can treat this variable as numeric.


GPU shows 0 utilization even when tensors and model are mounted on the gpu? I am trying to run some PyTorch scripts on a remote GPU server. While calling the script in the ubuntu terminal i start as:CUDA_VISIBLE_DEVICES=0(or whichever is available) python3 <script.py>. Also, used the following snippet in the code and used .to(device) on the model, input and target tensors.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

I have confirmed that my model and data and target tensors are mounted on the cuda device. But the GPU shows 0 percentage utilization all through the run. What could I be missing?

The problem was due to some modules I was importing without ensuring the tensors are mounted on cuda through that part of the pipeline. This had turned into the bottleneck blocking the GPU utilization.


Convert a pdf into a conditional pdf such that mean increases and std dev falls Let success metric(for some business use case I am working on) be a continuous random variable S.
The mean of pdf defined on S indicates the chance of success. Higher the mean more is the chance of success. Let std dev of pdf defined on S indicates risk. Lower the std deviation lower the risk of failure.

I have data,let's call them X, which affects S. Let X also be modelled as bunch of random variables.

P(S|X) changes based on X.
The problem statement is I want to pick Xs such that P(S|X) has mean higher than P(S) and std deviation lower than P(S).

Just to illustrate my point I have taken X of 1 dimension.
Scatter plot between X(horizontal) and Y(on vertical):

You can see that P(S|X) changes for different values of X as given in the below plot:

For X between 4500 and 10225, mean of S is 3.889 and std dev is 0.041 compared to mean of 3.7 and std dev of 0.112 when there is no constraint on X.

What I am interested in is given S and bunch of Xs... pick range of Xs such that resulting distribution of P(S|X) has higher mean and lower standard deviation... Please help me find a standard technique that would help me achieve this.

Also I don't want to condition on X such that number of samples are too small to generalise.I want to avoid cases such as on the left most side of tree where number of samples is 1.

Just apply an optimization to search for the X values that satisfy the criteria you're looking for. Here's a simple demo:

set.seed(123)
mu_x_true = 1e4
mu_y_true = 3.75
n = 1e2

x <- rpois(n, mu_x_true)
y <- rnorm(n, sqrt(mu_y_true))^2

plot(x, y)

# conditions:
# E[Y|X] > E[Y]
# std(Y|x) < std(Y)

mu_y_emp = mean(y)
sd_y_emp = sd(y)

objective <- function(par, alpha=0.5){
    if (par[1]>par[2]) par = rev(par)
    ix <- which((par[1] < x) & (x < par[2]))
    k <- length(ix)
    if (k==0) return(1e12)
    mu_yx <- mean(y[ix])
    sd_yx <- sd(y[ix])

    alpha*(mu_y_emp - mu_yx) + (1-alpha)*(sd_yx - sd_y_emp)
}

init <- mean(x) + c(-sd(x), sd(x))
test <- optim(objective, par=init)

ix <- which((par[1] < x) & (x < par[2]))

mean(y[ix]) > mean(y)
# TRUE

sd(y) > sd(y[ix])
# TRUE


Is there standard parameters that characterize how fast neural network learns to achieve specific error rate? I want to estimate how fast my model learn. Mb how many learn steps my model need to do to achieve specific error rate(precision/ recall/ MSE/ etc.). Any standard approach for measuring this?

Sometimes you can estimate that by smoothing out your val_loss curve and hoping that your training is repeatable and stable, but seems that word "unpredictable" was invented specifically to describe loss curves. :) Especially if you're using some sort of dynamic data generation or custom cost function.

Take a look at few species at http://lossfunctions.tumblr.com/


How important is the channel order in deep-learning computer vision tasks? I stumbled across this question while working with OpenCV, which stores color images in BGR order in memory, while most other libraries I know of use RGB order.

How important is this difference? Would I be able to use BGR images for an RGB-trained network? Obviously, a fire truck is red in most regions, and not blue. But don't CNNs look for texture rather than simple colors?

Even assuming there is a difference - could it still make sense to apply channel permutations as a kind of data augmentation during training? This would spread the spectrum of textures from each original channel to both others, again, the assumption being that colors are not all that important.

I would be curious to know if research like this already been done - I would be surprised if it had not.

Task dependent but could be important.

Lets find some arbitrary representation that separates luminance from chrominance (represented as a color 2D vector)? That way, you are detected bright objects independent of color, and colorful objects less dependent on brightness. Obviously under assumption that you are color agnostic. Check this paper where they applied exactly this augmentation


Columntransformer multiple columns with vector inputs This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this.

My question is related to the sklearn's ColumnTransformer class.
Considering 'description' corresponds to a text/string data column with 9508 rows. The following works as expected:

In [61]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), 'description')])

In [62]: X=transformer.fit_transform(df)

In [63]: X.shape
Out[63]: (9508, 5913)

However, the following (note the [] around 'description'):

In [64]: transformer = ColumnTransformer(transformers=[('text-features', CountVectorizer(), ['description'])])

In [65]: X=transformer.fit_transform(df)

does not work as expected:

In [66]: X.shape
Out[66]: (1, 3)

Note that there is no issue parsing the list of column names for other transformers such as OneHotEncoder(). This is making it difficult to programmatically configure transformers for dataframes containing heterogenous data type columns.

One possibility is to configure transformer for each column individually even if multiple columns require the same transformer but I was wondering if there is better way to deal with this?

See the section 6.1.4 in the documentation.

As per the documentation, whenever the transformer expects a 1D array as input, the columns were specified as a string ("title"). For the transformers which expects 2D data, we need to specify the column as a list of strings (["title"]).


Problem with sort by and group by in pandas I have a DataFrame with duplicate id's but have different dates.

For Example:

A          B
1    12.09.2015
1    14.08.2018
1    13.07.2012
2    01.09.2015
2    09.10.2011

I want to group them by the Id's and get each of their maximum and minimum dates and create a different dataframe with each line for one id like:

A  B(min)  B(max)

All you need is a groupby operation + aggregation on the min/max values.

df.groupby('id').agg(('min','max'))['date_column']

The output should be like this : different dataframe with each line containing an id and the min/max dates.


Clarification about Octave data size limit I'm just starting to work on a relatively large dataset after ML course in Coursera.
Trying to work on https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD.
Got an accuracy of 5.2 in training and test set with linear regression using gradient descent in octave.

I tried adding all possible quadratic features (515345 instances and 4275 features), but the code just won't stop executing in my HP Pavilion g6 2320tx, with 4GB RAM in Ubuntu 14.04.

Is this beyond the data size capacity of Octave ?

You have about 4GB of RAM on your machine and Octave is an in memory application.

If you want to work with 515345 instances and 4275 features, assuming that you are using double precision (i.e. 8 bytes), you would need a memory of 515345*4275*8/1000000/1024 bytes ~ 17.6 GB. Even if you were using 4 bytes for each data point, you would require at least 9 GB for the computation to go through.

This issue might not be the Octave memory restriction in this case. See here for further details on Octave's memory usage.


Prediction with tensorflow keras I'm trying to predict age from a given picture. I built the model below but the problem is that I'm getting very large loss value with low accuracy while fitting the model.

I think the problem is choosing the wrong loss function (here mean_squared_error). What can be the problem here?

import tensorflow as tf
from tensorflow import keras

X = X.reshape(-1, image_size[0], image_size[1], 1)
model = keras.models.Sequential()

model.add(keras.layers.Conv2D(32, (5, 5), activation='relu', input_shape=X.shape[1:]))
model.add(keras.layers.MaxPooling2D((2, 2)))

model.add(keras.layers.Conv2D(32, (3, 3), activation='relu'))
model.add(keras.layers.MaxPooling2D(2, 2))

model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))

model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(60, activation='relu'))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(1, activation='softmax'))
model.compile(optimizer='adam', loss=keras.losses.mean_squared_error, metrics=['accuracy'])

model.fit(X, Y, epochs=170, shuffle=True, validation_split=0.1)

As another question, are my layers correct to predict a number for a given picture?

This is the problem: model.add(keras.layers.Dense(1, activation='softmax'))

For predicting real-valued data such as age, it is customary to set the activation as linear or in this case, you can probably use relu.

To illustrate, softmax will create a distribution over the output, in this case the output of the model will be always 1. Since the age is positive definite continuous number, you need an activation that has a range of real numbers >= 0 which the relu activation satisfies.

  As another question , are my layers good to predict a number for given picture ?

The architecture seems reasonable so try changing the model's final activation first.


How are neural nets related to Fourier transforms? This is an interview question

  How are neural nets related to Fourier transforms?

I could find papers that talk about methods to process the Discrete Fourier Transform (DFT) by a single-layer neural network with a linear transfer function. Is there some other correlation that I'm missing?

They are not related in any meaningful sense. Sure, you can use them both to extract features, or do any number of things, but the same can be said about a many techniques. I would have asked "what kind of neural network?" to see if the interviewer had something specific in mind.


sagemath: compared to r.quantile, what is a faster way to find boundaries for a boxplot? I was using the r.quantile method in sagemath to find boundaries for a box plot.
The plot was taking a long time using r.quantile.

r.quantile took more than 20 seconds to find the quartiles for a data set that could be sorted and plotted point by point in less than half a second on the same machine.

What is a faster alternative?

The following (crude) code is at least 50 times faster than r.quantile:

def findFences (orderedList, outlierC = 1.5, farOutlierC = 3.0):
   """findFences: an ordered list of ints or floats -> tuple of 9 floats
   (aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax)
   keys: float for outlier constant [outlierC] and far outlier constant [farOutlierC]"""

   lenMod4, half, quarter = len(orderedList) % 4, int(len(orderedList)/2), int(len(orderedList)/4)

   aMin, aMax = orderedList[0], orderedList[-1]

   # find quartiles
   if not lenMod4: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, (orderedList[half] + orderedList[half-1])/2.0, (orderedList[half+quarter] + orderedList[half+quarter-1])/2.0
   elif lenMod4 == 1: Q1, Q2, Q3 = float(orderedList[half-quarter]), float(orderedList[half]), float(orderedList[half+quarter])
   elif lenMod4 == 2: Q1, Q2, Q3 = float(orderedList[half-quarter-1]), (orderedList[half] + orderedList[half-1])/2.0, float(orderedList[half+quarter])
   else: Q1, Q2, Q3 = (orderedList[half-quarter] + orderedList[half-quarter-1])/2.0, float(orderedList[half]), (orderedList[half+quarter] + orderedList[half+quarter+1])/2.0

   IQR = Q3 - Q1
   outDist = IQR * outlierC
   farOutDist = IQR * farOutlierC
   innerLoFence, innerHiFence = Q1 - outDist, Q3 + outDist
   outerLoFence, outerHiFence = Q1 - farOutDist, Q3 + farOutDist

   return aMin, outerLoFence, innerLoFence, Q1, Q2, Q3, innerHiFence, outerHiFence, aMax


What it means for a neural network to memorize data I've seen at many places that sometimes neural networks simply memorizes training data. What that means actually ?

Neural network consists of bunch of weights which gets trained and outputs based on input data. It'll output different thing for different input. Where does this memorization comes into play ?

Memorization is the same as overfitting.  The memory is implicitly represented by your weights. If your network does have enough parameters it will be able to memorize/overfit.

In order to understand what I mean by overfitting and memorization let us look at the polynomial regression

$$y_n=w_0+w_1x_n+w_2x_n^2+ \varepsilon.$$

We have three coefficients. If we only had three data points (which do not perfectly lie on a line) we could fit a quadratic regression equation without any error. Hence, the model would memorize the data by using three coefficients.
We would have a loss of zero, but as you know this result would also be very likely overfitting the model to the data.


Is it possible to use Jaccard similarity instead of Cosine similarity in gensim document similarity? I am using gensim library to compute similarity between documents but it only uses cosine similarity. I was wondering if there was a way to use jaccard similarity or any other similarity measure for that matter instead

If you have trained a gensim model, that object can act as a dictionary to give you the vector projection (via https://radimrehurek.com/gensim/models/word2vec.html)

$ model['computer']  # raw numpy vector of a word
array([-0.00449447, -0.00310097,  0.02421786, ...], dtype=float32)

So it is possible to manually implement any vector comparison that you choose.  Cosign similarity is typically chosen because it performs relatively well when compared to other methods of grouping high dimensional projections.

The way I could envision implementing Jaccard Similarity would be to identify a list of key words on a per document basis, and when comparing document, include words that are synonyms as intersections.

Based on reviewing the gensim document comparison text (https://radimrehurek.com/gensim/similarities/docsim.html) I don't believe there is a native implementation.


Training the Discriminative Model in Generative Adversarial Neural Network What I know so far in DCGAN is that a discriminator is trained using the labeled data (so maybe that occurs before training the generative model). Also, I know that there is race between the generator and the discriminator, so maybe training occur online. So I have some concerns here:

How many outputs the discriminator should have (Is it one output that describes the probability, ex: P(x))?
How do we chose its output when feeding fake data vs. real data?
Is the discriminator trained before using it with the DCGAN or the training is done online (It is mentioned in the Original Paper: Generative Adversarial Nets https://arxiv.org/pdf/1406.2661.pdf, that the whole network is trained using the back propagation), hence I think its online?

Any help is much appreciated!!

In normal GANs, there are no labels, the training is completely unsupervised.

The role of the discriminator is to tell apart samples generated by the generator from those taken from the training dataset. The training dataset is just a bunch of images. The discriminator is trained to output 0 for data generated by the generator (i.e. fake data) and 1 for real data (so the discriminator has a single output). This should answer points 1 and 2.

The training of discriminator and generator takes place alternatively in a loop: first we train the discriminator, then the generator, then the discriminator again, etc. It is possible (and common) to train the discriminator a few times per each time we train the generator. This should answer point 3.

It is also possible to use labels, but not in the way you were suggesting. When labels are used, we have Conditional GANs (https://arxiv.org/abs/1411.1784). In this case, the label is supplied as input to both the generator and the discriminator. The generator has to generate data that is associated to the supplied label. The discriminator has to tell apart fake data from real data, given the label.


What does the repeated message "No further splits with positive gain, best gain: -inf" mean? I am training a LightGBM classifier on a binary classification problem.

From time to time I get the following message repeatedly:

[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

It almost seems, that if it happens, the training is stuck in an infinite loop
Does someone know, what the reason is for this and if it is indeed an infinite loop or if for example this message is output each time the split of a tree node does not succeed and the algorithm continues with the next node letting the current one be a terminal (leaf) node. In the latter case I would maybe wait a bit longer before I terminate the process.

Unfortunately at present I cannot really say with which parameters the training was started since it is performed in a hyperopt loop, I would need to terminate it first.

The training parameters were:

{'n_estimators': 1730, 'max_depth': -1, 'num_leaves': 24579, 'min_child_samples': 450, 'reg_lambda': 0.0, 'silent': False, 'reg_alpha': 0.0}

It seems that for the unlimited max_depth in combination with the high number of num_leaves the algorithm frequently came to a point where it could not find a split with at least 450 samples (the total number of samples is less than 500'000).

I don't know the internal specifics about LGB behavior in this case but I don't think it is stuck in an infinite loop.
From Github's LightGBM issues we can find that:
What's the meaning of "No further splits with positive gain, best gain: -inf" message?

It means the learning of tree in current iteration should be stop, due
to cannot split any more. I think this is caused by
"min_data_in_leaf":1000, you can set it to a smaller value.

That is, LGB is trying to split the data in leaf but it can't.
The greater min_data_in_leaf is the more conservative the algorithm is.
Its optimal value depends on the number of training samples and num_leaves.


When I use SHAP for classification problem, it shows an output that is not 0 or 1. How can I overcome this? I'm using Pima Indians Diabetes Database(https://www.kaggle.com/uciml/pima-indians-diabetes-database). I made predictions using XGboost and I'm trying to analyze the features using SHAP.

However when I use force_plot with just one training example(a 1x8 vector) it shows that my output is -2.02. This is a classification problem, I shouldn't be seeing such a value. I'm new in SHAP and I don't know what the problem is.

Here is my code:

import numpy as np
import xgboost as xgb
import sklearn as skl
import shap

dataset=np.loadtxt("diabetes.csv", delimiter=",")
X=dataset[:,0:8]
Y=dataset[:,8]
seed=7
test_size=0.33
X_train, X_test, y_train, y_test=skl.model_selection.train_test_split(X, Y, test_size=test_size, random_state=seed)

shap.initjs()
model=xgb.XGBClassifier()
model.fit(X_train, y_train)

predictions=model.predict(X_test)
accuracy=skl.metrics.accuracy_score(y_test, predictions)
print(accuracy*100)

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_train)
shap.force_plot(explainer.expected_value, shap_values[0,:].reshape(1, 8), X_train[0,:].reshape(1, 8))

Accuracy of my model is: 77,95.

The default link function is the identity, so you are seeing log-odds rather than probabilities. To see the probability, try adding link='logit' to your call of force_plot like so:

shap.force_plot(explainer.expected_value,
                shap_values[0,:].reshape(1, 8),
                X_train[0,:].reshape(1, 8),
                link='logit')

You can read more at the SHAP documentation site.


static and dynamic data in clinical trials Hi everybody and thanks in advance for those who will help me for this problem.
I have multiple data regarding patients involved in a clinical trial and my goal is to predict their death/non death. These data are composed by two datasets , the first is static (age, gender ecc) and the second is dynamic (analysis results collected during multiple sessions like %hematocrit, %sodium, blood pressure and so on). I know that if I had only static data I would have used any ML classification algorithm and of course in case of only dynamic data the "right" way was to build a LSTM but with both of them I'm a bit confused.
Which kind of model can I build in this case? I was thinking at two models where the output of the first one (static) becomes the input of the second one (dynamic). Is it a good idea? Thank you.

What you are referring to is called a multi-input model and can be esaily built in most deel learning frameworks. The idea is to have both types of data as separate inputs, then use specific layers depending their types (recurrent layers to sequence data, CNN to images, and so on...) to later on concatenate them together.

If you can use Keras, there is the functional Api which is specialy well suited for the task at hand. An example of code (based the example given in the documentation) for your problem could be:

from keras.layers import Input, Embedding, LSTM, Dense, merge
from keras.models import Model

# headline input: meant to receive sequences of 100 integers, between 1 and 10000.
# note that we can name any layer by passing it a "name" argument.
main_input = Input(shape=(100,), dtype='int32', name='main_input')

# this embedding layer will encode the input sequence
# into a sequence of dense 512-dimensional vectors.
x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)

# a LSTM will transform the vector sequence into a single vector,
# containing information about the entire sequence
lstm_out = LSTM(32)(x)

#At this point, we feed into the model our auxiliary input data by concatenating it with the LSTM output:

auxiliary_input = Input(shape=(5,), name='aux_input')
x = merge([lstm_out, auxiliary_input], mode='concat')

# we stack a deep fully-connected network on top
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

# and finally we add the main logistic regression layer
main_output = Dense(1, activation='sigmoid', name='main_output')(x)

#This defines a model with two inputss:
model = Model(input=[main_input, auxiliary_input], output=main_output)

#Then compite and train
model.compile(optimizer='rmsprop', loss='binary_crossentropy')
model.fit([headline_data, additional_data], labels,
          nb_epoch=50, batch_size=32)

In your case, the dynamic data would be the headline_input and your static data the auxiliary_input. The model will take both, apply the recurrent layer to the former and concatenate them to pass the union through the dense layers.

Of course, many of these parameters will depend on your data, but at least this example will give you an idea on how to build such model.

There is also this interesting project conditionall RNN which is also meant for this purposes. Worth taking a look.


Clustering algorithms based on lists of most similar items? I'm a beginner to machine learning. I have a large dataset of items, each with a limited set of most similar items in the dataset, which I would like to create clusters of, preferably each having the same size, although this isn't necessary. Could someone point me in the right direction? Thanks.

You can go for K-means clustering to cluster the data based on the pattern.


Feature importance with Text features I would like to determine features importance in several models:

support vector machine
logistic regression
Naive Bayes
random forest

I read that I will need an agnostic model, so I have thought to use performance_importance (in python).
My features look like

Text (e.g., The pen is on the table, the sky is blue,...)
Year (e.g., 2019, 2020,...)
#_of_characters (e.g., 34, 67,...): this value comes from Text
Party (e.g., National, Local, Green, ...)
Over18 (e.g., 1, 0, ...) : this is a Boolean variable

My target variable is Voted.
In the pre-processing phase, I am using BoW and TF-IDF for Text, OneHotEncoder for Party, SimpleImputer for numerical.
Using the following:
from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt

result = permutation_importance(clf, X_test, y_test, n_repeats=5, random_state=42, n_jobs=2)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=X.columns[sorted_idx]);

I am getting a similar output like the below(I forgot to include Over18, but it is just to give an idea of the output):

Although I have difficulties in interpreting the results, especially circles and negative values, I would like to understand if, in case of text classification, it makes sense to have Text for the importance and not, for example, the single words (unigrams, bigrams,...).
For instance, in my example, I have ['The','pen','is','on','table','sky','blue']. Would it make more sense to understand how much each word contributes to the model instead of Text, or this is just considered in Text (where there are many words that contributes to the model), which is the most significant feature in the model?
UPDATE: for the different features I am using the following pre-processors:
categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')

numeric_preprocessing = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')
])

# CountVectorizer
text_preprocessing_cv =  Pipeline(steps=[
    ('CV',CountVectorizer())
])

# TF-IDF
text_preprocessing_tfidf = Pipeline(steps=[
    ('TF-IDF',TfidfVectorizer())
])

and then
preprocessing_cv = ColumnTransformer(
    transformers=[
        ('text',text_preprocessing_cv, 'Text'),
        ('category', categorical_preprocessing, categorical_features),
        ('numeric', numeric_preprocessing, numerical_features)
], remainder='passthrough')

clf_nb = Pipeline(steps=[('preprocessor', preprocessing_cv),
                      ('classifier', MultinomialNB())])

permutation_importance is considering the top-level features. It is permuting each one sequentially and learning the importance.
So, the inner encoding i.e. OHE/tfid is not visible to it.
To get the importance of components of the top-level feature, you should encode it separately and then pass the encoded data to the permutation_importance

Get the pre-processed data using preprocessing_cv.fit_transform(X_train)
Call your permutation_importance code on the above data and any model of your choice

Edit
Adding the snippet. I am excluding ColumnTranformer as it is causing some issue.
data = {'Number':[1,2,3], 'Text':['pen is table', 'sky is blue','Sun is kool'], 'Cat':['A','B', 'C']}
df = pd.DataFrame(data)

categorical_preprocessing = OneHotEncoder(handle_unknown='ignore')
numeric_preprocessing = SimpleImputer(strategy='mean')
text_preprocessing_cv =  CountVectorizer()

text_tfid = text_preprocessing_cv.fit_transform(df['Text']).toarray()
num = numeric_preprocessing.fit_transform(df['Number'].values.reshape(-1, 1))
cat = categorical_preprocessing.fit_transform(df['Cat'].values.reshape(-1, 1)).toarray()

data = np.concatenate((cat,num,text_tfid), axis=1)
cols =  np.concatenate((categorical_preprocessing.get_feature_names(), text_preprocessing_cv.get_feature_names(), ['Num'])) # New cols name

df = pd.DataFrame(data, columns=cols) # Encoded DataFrame with col name

from sklearn.inspection import permutation_importance
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier

clf = DecisionTreeClassifier().fit(df, [1,0,1])

result = permutation_importance(clf, df, [1,0,1], n_repeats=2, random_state=42)
sorted_idx = result.importances_mean.argsort()

plt.boxplot(result.importances[sorted_idx].T,
            vert=False, labels=df.columns[sorted_idx]);


cross_val_score meaning I'm studying the following code, which cross_val_score_ was used as well as .mean() and .std(). I read many documentation of the meanings, but didn't get what each of the above does.
import pandas as pd
import numpy as np
from sklearn import tree
import graphviz
from sklearn.model_selection import cross_val_score

#importing the dataset
d = pd.read_csv('student-por.csv', sep= ';')

d['pass'] = d.apply(lambda row: 1 if (row['G1']+ row['G2']+ row ['G3']) >= 35 else 0 , axis=1)
d = d.drop(['G1', 'G2','G3'], axis=1 )

#Doing one-hot encoding
d=pd.get_dummies(d, columns =['sex','activities','school', 'address', 'famsize','Pstatus','Mjob','Fjob','reason','guardian','schoolsup','famsup','paid','nursery','higher','internet','romantic'])

#shuffle rows
d = d.sample(frac=1)

#split traning and test
d_train = d[:500]
d_test = d[500:]

d_train_att = d_train.drop(['pass'], axis=1)
d_train_pass= d_train['pass']

d_test_att = d_test.drop(['pass'], axis=1)
d_test_pass= d_test['pass']

d_att = d.drop(['pass'], axis=1)
d_pass = d['pass']

t = tree.DecisionTreeClassifier(criterion ='entropy', max_depth = 5)
t= t.fit (d_train_att, d_train_pass)

#to export the tree
dot_data = tree.export_graphviz(t,out_file = 'students-tree.png', label ='all', impurity=False, proportion= True, feature_names=list(d_train_att), class_names=['fail', 'pass'], filled = True, rounded=True)

t.score (d_test_att, d_test_pass)
scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

in short this is what I need to know:
scores = cross_val_score(t, d_att,d_pass, cv=5)

print ('Acuracy %0.2f (+/- %0.2f)' % (scores.mean(), scores.std() *2))

one more thing, am I suppose to get the same score as in the original code publisher? because I didn't.

The source, around line 274 is where the default scoring for cross_validation_score gets set, if you pass in None for the scorer argument.  For classifiers, the usual default score is accuracy.  For regression, it's rmse, IIRC.  So, since you're applying a decision tree classifier, cross_val_score splits the data into 5 equalish sized pieces, trains on each combination of 4 and gives back the accuracy of the estimator on the 5th.  The mean and std of these accuracies presumably tells one something about the performance of the family of decision tree classifiers on your dataset, but I would take it with a grain of salt.


How Int8 (byte) operations can be useful for deep learning? Nvidia is planing to add hardware support for int8 operations to their titan card and target deep learning. I am trying to understood how its useful and what types of network will benefit from this.

I know that FP16 instead of FP32 is what should be useful for DL, but not sure how int8 could do. There are some research that you can train with full FP32 precision and then round it to one byte - but this does not speedup or reduce memory footprint for training.

Actually, recently people have been trying much lower precision in neural nets: 1-2-5 scheme (1 bit weights, 2 bit activations, and 5 bit gradients) seems to work well for easy datasets (MNIST and CIFAR-10). However, on ImageNet the results are significantly lower than those with full-precision (16 or 32 bit).
To achieve state of the art, convnets don't need more than 16 bits for training, but current RNNs might need more. For inference, on ImageNet, 4-5 bit weights (stochastically rounded from full precision) should be enough.


Should the input data be normalized using keras pre-trained models I want to use a pre-trained VGG16 in keras. My question is simple. Should I normalize the input image before predicting its label?

According to Very Deep Convolutional Networks for Large-Scale Image Recognition, which is the paper that first presented VGG: "...The only pre-processing we do is subtracting the mean RGB value, computed on the training set, from each pixel."- (Karen Simonyan, Andrew Zisserman).
I am certain that Keras provides a preprocess function based on the above mentioned principle.
Checkout: tensorflow.keras.applications.vgg16.preprocess_input
If it is in the TF version, its certainly in the other version.


How to implement LSTM using Doc2Vec vectors to get representation? Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about step-by-step guide to build it. How LSTM can make representation with input is list vector of sentence trained by Doc2vec.

Inputs of LSTMs are vectors and it does really matter what the vectors are, in signal processing there are signal windows, in NLP, these are usually word embeddings, here, they are document representations obtained from Doc2vec. Judging from the scheme, they probably use Doc2vec to obtain sentence embeddings. It is rather unusual, but definitely possible.

If the input vectors carry enough information, LSTM will certainly learn a reasonable representation.