You could use at least these two options:

PIL library:

from PIL import Image

img = Image.new('1', (width, height)) #width and height of the original image
data=[int(x) for x in x]  #x being your array
img.putdata(data)
img.save('my_handwritten_number.png')
img.show()

You can see here an example with the whole process, from reading a hand written image file through converting it to an array and displaying it again to an image:

import numpy as np

np_img=np_img.reshape(28900, )

width = np.sqrt(len(np_img))

from PIL import Image

img = Image.new('1', (int(width), int(width)))
data=[int(np_img) for np_img in np_img]
img.putdata(data)
img.save('my_hand_written_digit.png')
img.show()

getting as a result this reconstructed digit:

nevertheless, I see your array is a 1D array whose square root is not an integer, so it seems it does not come from an square image; you need to be able to convert it to a matrix (via the Image.new method).
If I do it with your array, this is what I get:

another option you have is with Keras:

import tensorflow.keras as keras

keras.preprocessing.image.array_to_img(np_img.reshape(170, 170, 1),
data_format=None, scale=True, dtype=None)


Dissimilarity Matrix of non-metric proximity data we currently have a coding exercise, where we are asked to implement Constant Shift Embedding (Paper). This in itself is not a big problem. For the algorithm, all you need is a symmetric non-zero diagonal dissimilarity matrix of some non-metric proximity data. With the algorithm you can then embed the information into a vector space and therefore you can use commonly known denoising and dimensionality reduction methods to improve the results of for example k-means clustering.

Given the E-Mail communications based on this data set, how would go about choosing a reasonable dissimilarity matrix?

The data is simply a list of unique pairs, where at least one e-mail has been sent from node A to node B. This gives rise to a graph of around 1000 nodes and 25000 edges.

Creating an adjacency matrix of this undirected graph might be a first step (which is also already provided in the framework).

I'm thankful for any pointers in the right direction.

EDIT: Over night I had an idea:

Let's say we only have 8 nodes. Now compare the proximity elements of two vertices. So if the prox. vectors would for example look like:

1 0 0 0 1 0 1 1

0 1 0 0 0 1 0 1

Their dissimilarity would be 5, since their vectors differ at 5 points.

Now just normalize w.r.t. the total number of nodes, therefore 5/8.

With this, we also incorporate the information of how many neighbors are shared instead of only looking at direct edges, and might therefore receive better results, when we later try to cluster the nodes.

Let me know what you think.

Maybe I did not completely understand your question, but I think the answer you are looking for is one of the following:

You may want to fill a n-by-n matrix with 1 if the person $i$ has sent e-mail(s) to person $j$, 0 otherwise
Maybe you want to fill the n-by-n matrix with the number of emails sent from person $i$ to person $j$.

Both measures are distances in the mathematical definition.

For clarity:

You could program the dissimilarity matrix as $M[i,j] = 1$ if the pair of people in your data exists.


K-means program in python for large excel database I have an excel database of 50 rows and some 28,000 columns. The database provides information of water parameters such as pressure, roughness, pipe length, diameter, valve, pump, flow and so on. I want to apply k-means code on this data to find outliers.

Code

from pandas import read_excel
import numpy as np
from numpy import arange
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode

dataset = read_excel('D:\M.tech\Dataset\Testing database1 New.xlsx')
#print(dataset.tail())
dataset.fillna(0)
cols=dataset.columns[1:]
#print(cols)
np.isnan(dataset).any()
dataset = dataset.fillna(method='ffill')

clusters=KMeans(n_clusters=4)
dataset["clusters"]=clusters.fit_predict(dataset[cols])
dataset.head(20)

#Principal component separation to create 2 dim picture
pca=PCA(n_components=2)
dataset['x']=pca.fit_transform(dataset[cols])[:,0]
dataset['y']=pca.fit_transform(dataset[cols])[:,1]
dataset=dataset.reset_index()
print(dataset.tail())

trace0= go.Scatter(x=dataset[dataset.clusters == 0]['x'],
                   y=dataset[dataset.clusters == 0]['y'],
                   name="Cluster1",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(15,152,152,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

trace1= go.Scatter(x=dataset[dataset.clusters == 1]['x'],
                   y=dataset[dataset.clusters == 1]['y'],
                   name="Cluster2",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(180,18,180,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

trace2= go.Scatter(x=dataset[dataset.clusters == 2]['x'],
                   y=dataset[dataset.clusters == 2]['y'],
                   name="Cluster3",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(132,132,132,0.8)",line=dict(width=1,color="rgb(0,0,0)")))

trace3= go.Scatter(x=dataset[dataset.clusters == 3]['x'],
                   y=dataset[dataset.clusters == 3]['y'],
                   name="Cluster4",
                   mode ="markers",
                   marker =dict(size=10,color="rgba(230,20,30,0.5)",line=dict(width=1,color="rgb(0,0,0)")))

data =[trace0,trace1,trace2,trace3]

iplot(data)

Above code displays no error but when I run the code in ipython console, it does not display any figure.

How can I do this?

First, you can read your Excel File with python to a pandas dataframe as described here: how-can-i-open-an-excel-file-in-python

Second, you can use scikit-learn for the k-means clustering on your imported dataframe as described here:KMeans


Weighted k nearest neighbor search I've searched quite a bit and haven't landed on any useful results.

The problem statement is:
Given a set of vectors, I wish to find its approximate k-nearest neighbors.
The caveat here is that each of my dimensions resemble a different entity and hence we cannot use the same weight for each dimension while computing the distance.
Thus, solutions like kd-tree don't work as is.

Is there any data-structure or any alternate algorithm that I can use to find such approximate weighted k-nearest neighbors.

Note: Multiplying the initial input data with their weights so as to get a uniform weight is not an option.

As per @an6u5's comment:
If you want to weight one dimension higher than others then I suggest you standardize all of your data so that the mean is zero and the standard deviation is one. Then you can multiply the less important dimensions by a factor (2-10) so that they appear farther away to the KNN distance metric and leave the most important dimension un-scaled. Note that both standardizing and scaling are completely reversible processes, so there is very little reason not to use this simple solution


Do I have to scale/normalize my training data for LSTM Classification, even if I only have one feature? I have a time-series data as follows:
# Time, Bitrate, Class
  0.2,  312,     1
  0.3,  319      1
  0.5,  227      0
  0.6,  229      0
  0.7,  219      0
  0.8,  341      1
  1.0,  401      2

I am using only the "Bitrate" column as a feature, and "Class" for the labels for an LSTM classification model. In case of multiple features, I need to scale my data of course, to prevent domination from one feature to another. However, in my case, do I still need to scale/normalize my data, considering there is only one feature?
Thanks!

Normalizing the features ensures that they take on reasonable values say between -3 and +3. This ensures that you don't run into a numeric overflow or under flow issues in your network. For e.g. just see what value np.exp(312) or np.exp(-312) takes on.
where, 312 is a value of the bitrate in your observation.
Certain activation functions such as the sigmoid might run into numerical precision issues if your data is not normalized. So, in this case it doesn't matter if your data contains only 1 feature.


why an advanced LSTM model produce the same results as a simpler one? I have implemented the model proposed in this article which is a text classification model that uses sentence representation rather than only word representation to classify texts.

model=tf.keras.Sequential()
embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
model.add(TimeDistributed(embeding_layer))
model.add(TimeDistributed(tf.keras.layers.LSTM(50)))
model.add(tf.keras.layers.Bidirectional(costumized_lstm.Costumized_LSTM(50)))
model.add(layers.Dense(6,activation='softmax'))
opt=tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
self.model=model

and I use a dataset with 40000 documents with 6 different labels to train it. (30000 for train and 10000 for the test). I uses a pretrained word embeding and the input for this model is (sample,sentences,words). it achieves 84% accuracy. the problem is that I can achieve this accuracy very easily with this simple model:

        model=tf.keras.Sequential()
    embeding_layer=layers.Embedding(self.vocab_size,self.word_vector_dim,weights=[word_embeding_matrix],trainable=False,mask_zero=False)
    model.add(embeding_layer)
    model.add(tf.keras.layers.Bidirectional(layers.LSTM(50)))
    model.add(layers.Dense(6,activation='softmax'))
    opt=tf.keras.optimizers.RMSprop(learning_rate=0.001)
    model.compile(optimizer=opt,loss='categorical_crossentropy',metrics=['accuracy',self.f1_m,self.precision_m, self.recall_m])
    self.model=model

this one is not based on sentence representation and the input for this model is (sample, words).
what is the first model ? is my implementation wrong? what should I do?

the training process for both models is as below picture. I also have used every trick to overcome overfitting but I haven't got any results. any suggestions please?

If you can achieve high results with a simpler model, it is great news! Always choose simpler models because they may be wrong on fewer things than complex ones (Occam's Razor).

Nothing to be worried about, this can happen.

Obviously, it is always possible that there is an implementation problem, so always make sure your code works better.

Reproducing results from papers is always difficult since many things can differ between your work and theirs:

Code might be different
Data might be different
Package versions may be different
Randomness might be controlled differently
Weights may be initialized differently
Data might be fed into the model differently
Hardware may be different which, especially with GPU can cause slight difference in results


How to do imbalanced classification in deep learning (tensorflow, RNN)? I am trying to do binary classification of News Articles (Sports/Non-Sports) using recurrent neural net in tensorflow. The training data is highly skewed [Sports:Non-Sports::1:9].

I am using cross-entropy as my cost function, which treats both classes equally.

What are the ways by which user can penalise one class? Or is there any other cost function suitable for this purpose?

This has already been answered both in stackoverflow and crossvalidated.

The suggestion in both cases was to add class weights to the loss function, by multiplying logits:

loss(x, class) = weights[class] * (-x[class] + log(\sum_j exp(x[j])))

For example, in tensorflow you could do:

ratio = 31.0 / (500.0 + 31.0)
class_weight = tf.constant([ratio, 1.0 - ratio])
logits = ... # shape [batch_size, 2]
weighted_logits = tf.mul(logits, class_weight) # shape [batch_size, 2]
xent = tf.nn.softmax_cross_entropy_with_logits(
  weighted_logits, labels, name="xent_raw")


Pearson correlation can be used for this purpose. The Pearson correlation between two entity shows that how mush the values of these two are  linearly related to each other.

  According to the Cauchy–Schwarz inequality it has a value between +1
  and −1, where 1 is total positive linear correlation, 0 is no linear
  correlation, and −1 is total negative linear correlation.

According to the values you reported here, there is a strong correlation between doc and mrr so the role of doc in the prediction of mrr is more important than others.

But on the other hand none of the features doesn't have any considerable linear correlation with ra. In this case you can test some other correlation methods.

For further information visit here. It can be helpful to you.

Conclusion: the most important feature in predicting an output is the most correlated one with it which has a considerable correlation.


Building a data frame and matrix for citation network I have a question about how to create a matrix that will take the lists defined below and make them the entries for the columns and rows. My data involves two lists: one lists originals sources (or papers) and the other lists those sources' individual citations.

The code is done in RStudio.

A sample of my original dataset looks like this:

 # List all the sources
 sources.vertices.levels <- cit.total$Sources
 sources.vertices <- (as.character(sources.vertices.levels))

 citation.edges.levels <- cit.total$Citations
 citation.edges <- (as.character(citation.edges.levels))

 # Break up the factors into character lists
 list.sources <- levels(sources.vertices.levels)
 list.citations <- levels(citation.edges.levels)

Below I have created an empty matrix (50X1497). I would like to make the row names be each individual source, and the columns be each individual citation.

Also I am lost on how to create a binary matrix that matches each individual source (from the rows) to the individual citations (to the columns).

 # Building an aggregate matrix
 matrix.network <- matrix(cit.citations,ncol=1497, nrow=50)

An example of what kind of matrix I want to do is below:

Are the 1s and 0s only to indicate that the source matches the citation? If so, try this :

library(tidyverse); library(magrittr)
cit.total %>%
mutate(score = 1) %>% # so that all 'matches' in your original data frame are tagged '1'
spread("Citations", "score", fill = 0) # each source and citation pair gets a number, 1 if the match exists, 0 if it doesn't


Data Science Noob - Customer Scoring based on conversion probability I work at a University and have a project to be able to score applicants based on their likelihood to enroll (convert), using their answers to application form questions.

The applications contain name, DOB, date & time of application, country, gender, course selected, English proficiency, funding availability and some other similar fields. There are also various free-text fields, but I think these will complicate things too much to begin with.

My initial thought is to use a regression model to do this, using R. But I am a complete noob - I studied regression at uni 10 years ago....

I have had a search around and I think once I know I'm on the right path I will be able to figure the process out but I am unsure where to begin, and do not want to start by going down the wrong avenue. My main concerns are:

Is a regression model the correct approach? If not what is?
Are categorical fields a problem - as opposed to continuous fields?
There is additional information which is only available for some applicants - can this be included, or do we need to use the same information for all applicants?

Your question could be closed for too broad, but let's give a try. You want the enrolment probability, this sounds like a logistic regression for me. Neither categorical nor continuous data type should present a problem. You may model the additional information, applications who don't have one simply be assigned a NA category. You can include the NA category in your model.

I recommend you read the book Applied Predictive Modelling, I think it has a section on credit-card applications, which is close to what you are doing. You should try to learn from it.


In Neural Networks and deep neural networks what does label-dropout mean If you take the following sentence from an article on deep neural networks

  to regularize the classifier layer by estimating the marginalized effect of label-dropout during training.

What does label-dropout mean

I ended up finding the paper you were referencing, maybe next time, add it to your post.

From what I can understand in this paper, label-dropout means that you are dropping the real labels and replace them with others.

The section of the paper you're referring to is explaining everything in great detail, so I'd try to read it carefully. In short, instead of using the ground-truth distribution (i.e. one 1 and a bunch of 0s) in the target layer, they are adding some uncertainty. That way, the model becomes less confident, and therefore is less prone to overfitting and becomes more adaptable.


How to split train/test in recommender systems I am working with the MovieLens10M dataset, predicting user ratings.  If I want to fairly evaluate my algorithm, how should I split my training v. test data?

By default, I believe the data is split into train v. test sets where 'test' contains movies previously unseen in the training set.  If my model requires each movie to have been seen at least once in the training set, how should I split my data?  Should I take all but N of each user's ratings for all the data and evaluate my performance on the held out NxUser_num  ratings?

Leave-one-out cross validation is probably the most straight-forward way to address this.  If you happen to be using a model that requires a lot of time to train, then leave n% out might be more appropriate.
The method involves eliminating one known rating and trying to predict it.  If you want to remove n percent of the ratings, just choose them randomly rather than choosing a specific number of every user's ratings.  And keep n pretty small - on the order of 10% or less.
Here is a good treatment of cross validation methods for recommender systems, An Evaluation Methodology for Collaborative Recommender Systems.  Generally:

Holdout is a method that splits a dataset into two parts: a training
set and a test set. These sets could have different proportions. In
the setting of recommender systems the partitioning is performed by
randomly selecting some ratings from all (or some of) the users. The
selected ratings constitute the test set, while the remaining ones are
the training set. This method is also called leave-k-out. In [17],
Sarwar et al. split the dataset into 80% training and 20% test data.
In [18] several ratios among training and test (from 0.2 to 0.95 with
an increment of
0.05) are chosen and for each one the experiment is repeated ten times with different training and test sets and finally the results are
averaged. In [13] the test set is made by 10% of users: 5 ratings for
each user in the test set are withheld.
Leave-one-out is a method
obtained by setting k = 1 in the leave-k-out method. Given an active
user, we withhold in turn one rated item. The learning algorithm is
trained on the remaining data. The withheld element is used to
evaluate the correctness of the prediction and the results of all
evaluations are averaged in order to compute the final quality
estimate. This method has some disadvantages, such as the overfitting
and the high computational complexity. This technique is suitable to
evaluate the recommending quality of the model for users who are
already registered as members of the system. Karypis et al. [10]
adopted a trivial version of the leave-one-out creating the test set
by randomly selecting one of the non-zero entries for each user and
the remaining entries for training. In [7], Breese et al. split the
URM in training and test set and then, in the test set, withhold a
single randomly selected rating for each user.
A simple variant of the
holdout method is the m-fold cross-validation. It consists in dividing
the dataset into m independent folds (so that folds do not overlap).
In turn, each fold is used exactly once as test set and the remaining
folds are used for training the model. According to [20] and [11], the
suggested number of folds is 10. This technique is suitable to
evaluate the recommending capability of the model when new users
(i.e., users do not already belong to the model) join the system. By
choosing a reasonable number of folds we can compute mean, variance
and confidence interval.

Hope this helps!


What would be the mse (mean squared error) of my scaled dataset on the original scale? I build an LSTM model on a standardized dataset using sklearn's MinMaxScaler. All values of the dataset are between 0 and 1. Features and target variables were standardized between 0 and 1.
I achieve an mse of around 0.02 .

Now this mse is valid for the standardized dataset. What would the mse be on the original scale?
Let's say the original scale is dollars. What would an mse of 0.02 be on the original scale in dollars?

TLDR: You need to multiply your scaled mse by $(max - min)^2$.

Let's give our values some names: Let $y_n$ denote the dependant variable, $m$ be the number of observations
\begin{align*}
& min := \min\limits_{n} y_n, \\
& max := \max\limits_{n} y_n
\end{align*}
and let $p_n'$ be the (scaled) predicted value for $y_n$. The scaled value $y_n'$ for $y_n$ is given by
$$ y_n' = \frac{y - min}{max - min}. $$
So scaled mse that you calculated is given by
$$ 0.02 = mse_{\text{scaled}} = \frac1m \sum\limits_n |y_n' - p_n'|^2 = \frac1m \sum\limits_n |\frac{y - min}{max - min} - p_n'|. $$
In order to get the true mse, you want to rescale $y_n'$ to $y_n$ and $p_n'$ to $p_n$ using
$$ p_n' =  \frac{p_n - min}{max - min}.$$
This gives us
\begin{align*}
mse_{\text{scaled}}
& = \frac1m \sum\limits_{n} |\frac{y_n - min}{max - min} - \frac{p_n - min}{max - min}|^2 \\
& = \frac1m \sum\limits_{n} |\frac{y_n}{max - min} - \frac{min}{max - min} - \frac{p_n}{max - min} + \frac{min}{max - min}|^2 \\
& = \frac1m \sum\limits_{n} |\frac{y_n}{max - min} - \frac{p_n}{max - min}|^2 \\
& = \frac{1}{(max - min)^2} \frac1m \sum\limits_{n} |y_n - p_n|^2.
\end{align*}
Rearraging this gives us
$$ mse_{\text{original}} = \frac1m \sum\limits_{n} |y_n - p_n|^2 = (max - min)^2 mse_{\text{scaled}}. $$

However, in general, it is better to first rescale your values and then take the mse because you lose interpretability and, as you see, it is more difficult to rescale the mse than to rescale the predicted values.


Should I Impute target values? I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas.

I achieved imputing missing values in independant variables. For numerical data by using the Imputer with the means strategy and for one categorical variable I used the Labelencoder and afterwards imputed with the mode strategy.

But now I face the issue that the dependant variable $y$ also contains missing values. Should I delete those lines or should I impute $y$ which is numerical.

For the missing data problem, one thing to be aware of, is the missingness mechanism. Depending of the dataset, the NA's (Missing Values) you have could be a result of a condition of the phenomenon and you shouldn't impute then using mean but maybe.

Besides, for the dependent variable, if you want to train a model with the independent ones to predict this, let's say Y, you wouldn't train a model using this observation with NA on the dependent (target?). Then, you would drop this lines or maybe using another technique which takes into account the dependence of the other variables.

I think a good start is to give look at this:
Missing-data imputation

It shows the limitations of using some approaches like yours and defines the mechanisms of missing data.


Regression using gradiant boosting - smoother predictions I have practical machine learning problem. I have trained a LightGBM model to predict house prices. Compared with other models I have tried, the loss (RMSE) is quite low and overall I'm quite happy with the predictions.

When investigating the predictions by tweaking features, I have noticed one problem. The trend is correct thus that when I increase size of house, the price generally goes up. However, in many cases a small change in a feature can result in prediction change in the wrong direction. Let's say all features remain the same but I increase house size a little and the predicted price goes down for example. If I increase it further it goes up again.

Are there any some techniques to handle this problem, and get "smoother", more continuous predictions?

Another approach could be to train several models and then simply take the average of their predictions - essentially using an ensemble. It will be necessary to initialise the models differently and ensure there is some randomness/stochasticity during training such that the models are indeed different.

This approach should allow your grupo of models to tolerate one or two models giving odd predictions in each new prediction, while still giving the most sensible result.

Studies show that this can dramatically smooth predictions. Have a look at this comparison of use cases and various ensemble technique from Armstrong. There are some guiding principles suggested, depending on saveral factors such as domain knowledge and variance of predictions.


Cosine similarity with arrays contaning NaN I am trying to calculate a cosine similarity using Python in order to find similar users basing on ratings they have given to movies. As it can be expected there are a lot of NaN values.
I am using movie dataset from Kaggle.

When I use np.dot() on two nd.arrays the outcome is:'nan'.
I have checked with np.nansum() that there are some other than 'nan' values.

I do not want to change all 'nan' values to '0' as it would mean that users have given 0 rating to the movies which lead to 'false' similarity between users.

Please, give me some advice regarding how to proceed with this problem.

Thanks in advance.

I think it's rarely meaningful to consider cosine similarity on sparse data like this, not just because of sparsity (because it's only defined for dense data), but because it's not obvious the cosine similarity is meaningful. For example a user that rates 10 movies all 5s has perfect similarity with a user that rates those 10 all as 1. Magnitude doesn't matter in cosine similarity, but it matters in your domain.

It's much more likely that it's meaningful on some dense embedding of users and items, such as what you get from ALS.

To answer the question, you either need to impute the missing ratings (don't assume 0, but a mean value or similar), or ignore dimensions that aren't defined in both.


What's the difference between probabilistic programming such as pyro and belief networks? I heard about ubers pyro and stumbled upon this Wikipedia article.

As I understand, a bayesian network is the same as a belief network according to this post.

Does someone know how these are related?

A probabilistic program and a Bayesian Network are both ways of specifying probabilistic models. Any model that can be specified as a Bayesian Network can also be specified by a probabilistic program, in fact by a probabilistic program that has no control flow. Roughly

Bayes Nets == Straight line Probabilistic Programs

For example consider the Bayes Net

This Bayes Net is equivalent to the probabilistic program (in Pyro)

def model():
    p_rain = pyro.param("p_rain", torch.tensor(0.2), constraint=unit_interval)
    p_sprinkler = pyro.param("p_sprinkler", torch.tensor([0.4, 0.01]),
                             constraint=unit_interval)
    p_wet = pyro.param("p_wet", torch.tensor([[0.0, 0.9], [0.8, 0.99]]),
                       constraint=unit_interval)

    rain = pyro.sample("rain", Bernoulli(p_rain))
    sprinkler = pyro.sample("sprinkler",
                            Bernoulli(p_sprinkler[rain.long()]))
    wet = pyro.sample("wet", Bernoulli(p_wet[rain.long(), sprinkler.long()]))

More generally, probabilistic programs can contain control flow (if, for, while) and recursion. Some of these extra features are expressed in extensions to Bayes nets, e.g. some for loops can be expressed as plates in Bayes Nets.


difference between scaling/normalizing data at a specific step I am using the MinMaxScaler normalization method, however I have seen various ways that this can be done, I want to know if there is any actual difference between the following:
1. Standardizing/Normalizing the data before splitting the data into train and test
Code 1
scaler = MinMaxScaler() #Normalization
#Transform X and Y values with scaler
x = scaler.fit_transform(x)
y = y.reshape(-1,1)
y = scaler.fit_transform(y)

# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

2. Standardizing/Normalizing the data after splitting the data into train and test and then scaling on train and test
# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

# created scaler
scaler = MinMaxScaler() #Normalization

# transform training dataset
x_train = scaler.fit_transform(x_train)
# transform test dataset
x_valid = scaler.fit_transform(x_valid)

3. Standardizing/Normalizing the data after splitting the data into train and test. Then fitting on the training set and then scaling on both train and test
# Split Data in train and validation
x_train, x_valid, y_train, y_valid = train_test_split(x, y, test_size = 0.25)

# created scaler
scaler = MinMaxScaler() #Normalization
# fit scaler on training data
scaler = MinMaxScaler().fit(x_train)

# transform training dataset
x_train = scaler.fit_transform(x_train)
# transform test dataset
x_valid = scaler.fit_transform(x_valid)

Case I -
With a single scaling step, you might leak the test info into the train. In this approach you have a common min/max otherwise it would have been two pairs
See, the plot for one of the Features of Iris dataset
Also, we don't scale the target. But I see this in your code.

Case II -
 This is fine but you should also consider the online cases, where you will not have a test set to scale the new test data.
Case III -
 This is a better and an agnostic approach.
Your code implementation incorrect as suggested in the comment


Best way to generate artificial handwritten text? I would like to train a generative model that generates artificial handwritten text as output. which architectures would you recommend to use?

Training input could be either images of handwritten letters, not words, or maybe sequences of points for each letter.
I thought of using some kind of combination of GAN+LSTM/GRU.
Already found:

http://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
https://distill.pub/2016/handwriting/

Would appreciate any further recommendations.

Found some implementation of lstm-based handwriter. Maybe I will use some parts.

https://github.com/sjvasquez/handwriting-synthesis


Feature extraction from pure text I have a dataset (~52k rows) with a column containing just pure sentences (upper and lowercase, with punctuation and stop words) in each row. What can I do to represent this data in a meaningful way for my model. I tried using a CountVectorizer but it's giving me a dataframe with way too many columns and the model is having a hard time using it. What other options do I have?

That is the real-world! in text analysis you usually end up with a extremely high-dimensional representation. First of all, I always recommend TF-IDF over original BoW (CountVectrizer).
To end up with a low-dimensional space, you may use this naive approach which is not written efficient in computational way but the approach gives you an idea where you need to go. It will be more clear when I add explanations to the notebook soon.


How do I get one overall prediction, where each data point has many pictures? My task is not a simple image -> category. I have between 5 and 10 images of an object, and I must classify it. The problem is that the category isn't "visible" in each picture, so I would need many images -> category.
For instance, let's say I'm trying to classify men vs. women. I have a number of pictures taken from random angles. I need to determine, based on the aggregated score of these pictures, which category the bunch belongs to.
All the pictures contain a little bit of unique information.

You are describing a problem of supervised learning with multiple inputs. That is not an uncommon task and you can find many tutorials about multiple inputs for neural networks out there. Using Tensorflow, I personally recommend Keras Functional API for this task, since it gives you more control on the layers while keeping the high-level simplicity.


Does click frequency account for relevance? While building a rank, say for a search engine, or a recommendation system, is it valid to rely on click frequency to determine the relevance of an entry?

Depends on the user's intent, for starters.

Users normally only view the first set of links, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive those are the best links, otherwise the clicks are most likely going to reflect placement, not relevance. For example, here's a click and attention distribution heat-map for  Google search results:

Further, using click frequency to account for relevance is not a direct measure of the resource's relevance. Also, using clicks is problematic, since issues like click-inflation, click-fraud, etc. will pop-up and are hard to counter.

That said, if you're interested in using user interaction to model relevance, I would suggest you attempt to measure post-click engagement, not how users respond to search results; see "YouTube's head of engineering speaking about clicks vs engagement" for more information, though note that the size itself of the content is a factor too.

Might be worth noting that historically Google was known for PageRank algorithm though it's possible your intent is only to review click-streams, so I won't delve Google ranking factors; if you are interested in the Google's approach, you might find a review of Google's Search Quality Rating Guidelines.


How to use Non Negative Matric Factorization (NMF)'s transform method to project topics on new text data I have created a NMF topic model in python the code snippet for which is as follows:

def select_vectorizer(req_ngram_range=[1,2]):
    ngram_lengths = req_ngram_range
    vectorizer = TfidfVectorizer(analyzer='word', ngram_range=(ngram_lengths), stop_words='english', min_df=2)
    #print("User specified custom stopwords: {} ...".format(str(custom_stopwords)[1:-1]))
    return vectorizer

vectorizer = select_vectorizer([2,5])
X = vectorizer.fit_transform(new_review_list)

clf = decomposition.NMF(n_components=20, random_state=3, alpha = .1).fit(X)
vocab = vectorizer.get_feature_names()
print_top_words(clf, vocab, num_top_words)

which created 20 topics like the following:

Topic #0:
[u'blocks available', u'delivery blocks available', u'notifications blocks', u'notifications blocks available', u'new blocks', u'know blocks available', u'new blocks available', u'know blocks', u'open blocks available', u'available work', u'zero blocks', u'like blocks', u'notification blocks', u'day blocks', u'slow blocks', u'10 blocks', u'option set', u'logged 10', u'notification blocks available', u'day blocks available']
Topic #1:
['something', 'something'] and so on..

Now I want to supply new text data to this fitted NMF model such that the model tells us which topic this new text belongs to. I have searched a lot, somebody suggested using "transform", but I do not know how to use it. For eg: if I have a new text, like:

new_feedback = "This is a new tester feedback"

Then how do I find which topic out of the 20 topics created earlier, does this new_feedback fall into? Please help!!

UPDATE: On trying to do what's mentioned in the below comment, I receive the following error:

clf.transform(vectorizer.transform(new_feedback))
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1380, in transform
    X = super(TfidfVectorizer, self).transform(raw_documents)
  File "/usr/local/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 884, in transform
    "Iterable over raw text documents expected, "
ValueError: Iterable over raw text documents expected, string object received.

This is how you should get your topics and corresponding words. You first get the NMF transformation of your new input and sort its components. The biggest components tell you which features to look at in nmf.components_, which links topics to words (your tfidf features). Finally, you print the k most important words of the topics. Topics and words scores are just calculated by dividing their value by the sum of all values.

    import numpy as np
    # Transform new_feedback to NMF space
    nmf_new_feedback = clf.transform(vectorizer.fit(new_feedback))
    # top 10 topics of the new feedback
    compo = np.argsort(nmf_new_feedback)[-10:][::-1]
    # Connection between indices and words of tfidf
    feature_names = vectorizer.get_feature_names()
    for topic_idx in compo:
        # Current nmf topic
        topic = nmf.components_[topic_idx]
        print("Topic number: ", topic_idx)
        print("Topic score: ", nmf_new_feedback[topic_idx]/sum(nmf_new_feedback))
        # Top 15 words of current topic
        for k in topic.argsort()[-15:][::-1]]
            print("Word: ",feature_names[k], "  Score: ", topic[k]/ sum(topic))


Data representation (NoSQL database?) for a medical study Problem description

I have a data set about 10000 patients in a study. For each patient, I have a list of various measurements. Some information is scalar data (e.g. age), some information is time series of measurements, some other information can be even a bitmap. The individual record itself can be quite thick (10kB to 10MB). The data is to be processed practically in two steps:

Preprocessing at the level of individual records (patients), i.e. to extract some features in raw data, store them. Calculate some slopes in time series etc. All this can be done on individual level and it can be very easily distributed.
On top of the preprocessed data (extracted features), I will need to calculate some aggregated things such as e.g. average age, but also some machine learning tasks.

The question

Obviously, this is very suitable to be addressed in Apache Spark (or any map-reduce architecture). At the most general level, my question is: what is the most appropriate NoSQL database for this situation?

So far, I have considered two basic options:

MongoDB - to take advantage of the document oriented storage where everything is on the same place. However, I am not sure about the performance on the larger binary data (pictures, time-series).
Cassandra -  this may have some better storage of binary data, but the joins will be necessary (even if optimized by indexing all data by "patient id").

I think you need to define the process and desired outcome a little more. It sounds like you need to:

Define what features you want.
Figure out how to extract those features.
Figure out how to store those features.
Pass that dataset to a ML model for training.

I would figure out what exactly you want for #3 in terms of the number and type of data elements and then choose a data storage method only then.

Unless you are going to be passing the unstructured documents directly to your model, you don't need MongoDB's capabilities. 10,000 records is small, but since you mentioned you want to calculate some aggregated statistics on your patient-level data, you could likely get by with something as simple as MySQL or SQLite.

Spark and Map/Reduce are actually competitors, with Spark stealing MR's spotlight lately. You might need one or the other for feature extraction but they're probably overkill for the rest of the stuff you described.


Intuitive explanation of Adversarial machine learning How would you explain Adversarial machine learning in simple layman terms for a non-STEM person? What are the main ideas behind Adversarial machine learning?

Consider a game being played between two people, for simplicity, we'll assume this game is distinguishing a true picture of a panda vs. a fake picture of a panda. The first player will take the painting and show it to the second player, if the second player guesses whether it is a fake correctly, they receive a reward, if not they do not. Both players are playing the game with the goal of maximizing their reward.
To go a little deeper but make it slightly more relevant to the context of Adversarial ML. We can further assume that both players start from 0 knowledge of pandas. You might imagine that player 1 just throws random colors at a canvas and tries to convince player 2, and player 2 just randomly guesses, slowly building their intuition for what a Panda is.
After several hours/days/years, we might find that both players are extremely skilled and drawing pandas and identifying fake pandas respectively.
This is really what adversarial ML is about in a nutshell. The goal is to have two agents with competing rewards, where their optimal solution is at some form of a mixed strategy Nash equilibrium.


Classifier using pytorch I'm writing a demo code to predict a 2-class classification for a dataset of 10-D inputs. Below, function  _data generates the data:
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

def _data(dimension, num_examples):
    # Create a simulated 10-dimensional training dataset consisting of 1000 labeled
    # examples, of which 800 are labeled correctly and 200 are mislabeled.
    num_mislabeled_examples = 20
    # We will constrain the recall to be at least 90%.
    recall_lower_bound = 0.9

    # Create random "ground truth" parameters for a linear model.
    ground_truth_weights = np.random.normal(size=dimension) / math.sqrt(dimension)
    ground_truth_threshold = 0

    # Generate a random set of features for each example.
    features = np.random.normal(size=(num_examples, dimension)).astype(
        np.float32) / math.sqrt(dimension)
    # Compute the labels from these features given the ground truth linear model.
    labels = (np.matmul(features, ground_truth_weights) >
              ground_truth_threshold).astype(np.float32)
    # Add noise by randomly flipping num_mislabeled_examples labels.
    mislabeled_indices = np.random.choice(
        num_examples, num_mislabeled_examples, replace=False)
    labels[mislabeled_indices] = 1 - labels[mislabeled_indices]

    return torch.tensor(labels), torch.tensor(features)

The code below shows my attempt, where predictor is the model and the loss function is chosen to be Hinge loss.
import math
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

dim = 10
N = 100
target, features = _data(dim, N)

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.f_1 = nn.Linear(10, 1)

    def forward(self, features):
        return self.f_1(features)

model = predictor()

optimizer = optim.Adam(model.parameters(), lr=1e-2)
loss = torch.nn.HingeEmbeddingLoss(margin=1.0, size_average=None, reduce=None, reduction='mean')

running_loss = 0
for _ in range(1000):
    optimizer.zero_grad()
    output = model(features)
    objective = loss(output, target)
    objective.backward()
    running_loss += objective.item()
    optimizer.step()

    print(running_loss)

My questions:

I see my loss increase from zero to 20 and then dives deep into negative realm. I was wondering if my implementation is correct.
I was trying to implement my predictor without using nn.Linear by defining the computations myself as:

class predictor(nn.Module):
    def __init__(self):
        super(predictor, self).__init__()
        self.weights = torch.zeros(dim, dim, requires_grad=True)
        self.threshold = torch.zeros(1, 1, requires_grad=True)

    def forward(self, features):
        return torch.matmul(self.weights, features) - self.threshold

but in the optimization process,
model = predictor()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

I get the following error:

ValueError: optimizer got an empty parameter list

I would appreciate direction or comments on how to fix these issues. Thanks.

Choose lr of optimizer something very small. It might be because of exploding gradient.

In self.weight use nn.Parameter() then pass your torch.zeros() to make it a model parameter.


How to optimally train deep learning model using output as new input I'm trying to train a network to predict the future. My current setup uses 5 time steps as inputs from the past, each consisting of 10 features, resulting in a [5, 10] input matrix (initially generated by another algorithm). Each of these time steps is a dt step further in time. The network outputs 1 output with again 10 features, a dt step in the future w.r.t the last time step from the inputs. This input replaces the oldest time step in the inputs matrix.

The network is trained using a generated input sequence as well as one correct output for that sequence. The MSE error reduces to about 1e-4 for a 100.000 samples training set.

When executing the network using the previously mentioned method. The solution blows up pretty quickly as the network makes an error in the first prediction and uses that as an input, removing a correct but older input. Now, a bigger error is made due a partially incorrect input sequence which it has never seen before. The next time step this becomes even worse and so on.

Instead of training with a completely correct input sequence and its correct output, i was thinking the network should train on its own predicted, not completely correct, inputs as well. I could only think of training the network as mentioned before. Use that to create more samples including its own prediction errors and train it again using that training set. That process could be repeated until a desired point.

Is there a more efficient way to do that training only on a single training set using Keras as my deep learning toolbox?

Edit

I will provide some more context for my problem. I'm trying to predict the future states for a few types (square, triangle and sawtooth) of travelling waves, moving with a certain speed to the left or right. The following figure gives an example of a random travelling wave I could be using during training.

These waves are discretized by 1024 x positions with a y value. Due to their relative high dimensionality, I use an autoencoder to create a latent space with a dimensionality of 10. The following figure gives an example of an autoencoded wave, including the input, encoded latent space and the decoded output.

The lower dimensional latent space can be advanced in time more easily for an RNN network. To do this, I create five time steps of the wave using a generation algorithm. These are encoded to their latent space using the encoder of the autoencoder. Now they are fed into an RNN which predicts the next time step of the latent space. This output is fed back to the inputs to replace the oldest time step input. The output can also be decoded using the decoder of the autoencoder, resulting in a high dimensional prediction of the next time step of the traveling wave.

I have trained the RNN network giving it examples of 5 latent inputs and the correct next time step latent space output, reaching an mse of about 1.5e-4. A result for a new prediction of this RNN can be seen in the following figure.

The first prediction are pretty accurate for most of the inputs I give the network. However, when I use its own outputs as new inputs to keep on predicting next time steps, the error increases. The next figure gives an example of the same wave depicted before but now at prediction step 20.

For the RNN i'm currently using the following Keras setup:

inputLayer = Input(shape = (RNN_numInputs, AE_latentSize))
x = CuDNNLSTM(RNN_units[0], return_sequences=False)(inputLayer)
x = Dropout(0.2)(x)
x = RepeatVector(RNN_numOutputs)(x)
x = CuDNNLSTM(RNN_units[1], return_sequences=True)(x)
x = Dropout(0.2)(x)
x = Conv1D(AE_latentSize, RNN_outputKernelSize, activation='linear', padding = 'same')(x)

rnn = Model(inputLayer, x)

Where RNN_numInputs is 5, AE_latentSize is 10, RNN_units is [200, 400] and RNN_outputKernelSize is 3.

I think that when I can decrease the error of the prediction even further, the long term prediction results can become better.

Does anyone have an idea how I can increase the perfomance of this algorithm?

The first point I would make (and you are likely aware of this given you specified "lstm" as part of your tags), is that you should use a long-short term memory network for this purpose.

Given that you are working with time series data, the risk with a regular neural network is that it treats all the observations as independent when this is not the case with a time series.

  Instead of training with a completely correct input sequence and its
  correct output, i was thinking the network should train on its own
  predicted, not completely correct, inputs as well.

This is somewhat of a slippery slope, because one must assume that you are using actual outputs. The model might be falsely trained to believe that the predicted values (which ordinarily would not yield the correct output) do in this case, and this skews your results. The model is only as good as the data that is being fed into it - if the data is wrong then the results will be as well.

In general, your error reading looks quite good, although I am not familiar with your data or model.

If you wanted to generate dummy data, one possibility could be to use a bootstrap simulation of sorts. For instance, suppose you have a time series with 10% mean and 20% volatility. You then choose to generate a random time series based on these two readings. The neural network could then be further trained on such data, and may potentially improve accuracy.

However, I would be cautious of mixing inaccurate inputs with correct outputs - as this has the potential to skew your results.


Does Feature Normalization affect Gradient Descent | Linear Regression am new to datascience and i want to learn linear regression so i coded linear regression from scratch and performed gradient descent to find the best $w_\theta$ and $b_\theta$ values using a tutorial. And it went just fine i was able to find the best $w_\theta$ , $b_\theta$ values and i ploted the line-of-best-fit (below).

And the gradient descent code i used to find the $w_\theta$ , $b_\theta$ values is below .

def step_gradient_descent(data,m,b,learning_rate=0.0001):
        b_gradient= m_gradient = 0
        N=float(len(data))
        for i in range(len(data)):
                [x,y]=data[i]
                y_=(m*x)+b
                m_gradient+= - (2/N)*(x*(y-y_))
                b_gradient+= -(2/N)*(y-y_)
        #print("m ={}, b ={}".format(m_gradient,b_gradient))
        m_new = m-(learning_rate*m_gradient)
        b_new = b-(learning_rate*b_gradient)
        return (m_new,b_new)

def perform_gradient_descent(data,m,b,lr=0.0001,epochs=1000):
        m_array=b_array=[]
        for i in range(epochs):
                if(i % 100 == 0 ):
                        print("Running {}/{}".format(i,epochs))
                (m,b) = step_gradient_descent(data,m,b,lr)
        return (m,b,m_array,b_array)

and then i performed features normalization / standization on $x$ , $y$ below

def featureNormalize(data):
    mean = np.mean(data , axis=0 )
    std = np.std(data , axis=0 )
    norm = ( data - mean ) / std
    return norm

and then after that i plotted the line-of-best-fit , which was different from the above plotted one.

So things i tried to get back the previous line-of-best-fit as before are

changing learning rate ( $r$ )
changing epochs ( $n$ )

which didnot work out. So my understanding is that feature normalization / standization should not have any effect on the gradient descent but that didnot happen in this case. Just curious to figure out what's happening in my case.

Link to my notebook

Thanks in Advance.

Standard normalization let gradient descent converge faster. That's why most cases normalization is applied (of course it also eliminate effect of different scale features values).

In this quora topic it is explained well.

Your learning rate is very small. I think there is not enough number of iterations to coverge local minimum. This graph explains well the effect of learning rate to gradient descent:

You may want to check my another answer for it.


How is calculated the error with multiple output neurons in neural network? Machine Learning books generally explains that the error calculated for a given sample $i$ is:
$e_i = y_i - \hat{y_i}$
Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:
$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$
The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).
What does change in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?

You are mixing various concepts:

$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$ is used only for regression problem and not for binary classification because MSE fits very well when your target distribution is normal
You can use the latter formula for binary classification but will works really bad because your target data distribution is a Bernoulli, not Normal. Remember that the choice of the right imply a prior assumption on the target data distribution. For this reason the right formula is binary crossentropy (aka negative log likelihood of a Bernoulli) $$ L = - \sum_i y_i \log \hat{y_i} (1 - y_i) \log(1 - \hat{y_i})   $$
For multi classification problem there is a generalized formula of binary crossentropy which is called categorical crossentropy. If $\hat{y}$ is a vector of C element, one for each class and the true class $y$ is encoded as integer (e.g 0, 1, 2 ...) then the loss is $$ L = - \sum_i \log(\hat{y_i}[y]) $$


What is the difference between fasttext and DANs in document classification? I came across two interesting papers that describe promising approaches for document classification using word embedding.

1. The fasttext algorithm

Described in the paper Bag of Tricks for Efficient Text Classification here.

(With further explanation here).

2. DANs

Described in the paper Deep Unordered Composition Rivals Syntactic Methods for Text Classification here.

Question:

What is the difference between both approaches?

Are they essentially the same as they both seem to average word embedding and pass it through an MLP or am I missing something crucial?

The first most important difference consists in the fact that when using fasttext you are training a language model, i.e. your own embedding vecotrs, while DAN is an architecture (not a language model) that require either a random initializion of embedding layers (which are then trained along with the other layers) or to use pre-trained embeddgings like GloVe (or even fasttext vectors!).
DAN is something that has become popular in some sense (even though I never saw this paper before now). Aveaging embedding vectors of single words before feeding them to a dense layer is a common practice if you need to perform some task at a paragraph or document level.
Just to add some peculiarities of fasttext embeddings, their are trained not for single words, but for n-grams. So during the preprocesing of the corpus from which the embedding are learnt, words are splitted into several chunks of character. For example:
'matter' would become [ma, mat, att, tte, ter, er]
and a unique embedding is then learnt for each chunk like 'ma' or 'mat'. The training follow the same logic of word2vec vectors, which means that from every chunk the model tries to predict context chunks. The advantage of learning embeddings for each chunk relies on the ability of these vectors to learn specific morphological features that classic token-level embeddings usually miss.
If it might help, for a good survey on word embeddings I suggest to take a look at this.


NLP: find the best preposition for connecting parts of a sentence My task is to connect 2-3 parts of the sentence into one whole using a  preposition

the first part is some  kind of action. Ex. "take pictures"
the second part is an object  that can consist of  only one noun or a noun with  adjectives and additions dependent on it. Ex. "juicy cherry pie", "squirrel"
the third part is a place. Ex. "room", "London"

To solve this task I've already tried some options  such as generation using GPT-2 (or other networks like LSTM) and using grammar rules, but both of them didn't work as well as I want.
Using GPT-2 needs  a lot of resources and performance, which is very expensive on the server. Ordinary neural networks aren't good enough and often generate bad results. And it is also impossible to describe all cases using only grammar rules.
Are there any approaches how you can implement a solution to this problem without using transformers or grammatical rules, so that the model isn't too heavy and at the same time effective?
There may be several options for prepositions, the main thing is that they are used.
P.s. a good option was to use n-grams, but it is not clear how to work with it when the second part is long.
Examples:
[Pour, juice, boiler room]     ->  Pour the juice into the boiler room
[Go, theater]                  ->  Go to the theater
[Learn the history, ballet]    ->  Learn the history of ballet

The traditional approach for this kind of problem would be an n-gram language model. The language model is trained on a large corpus, then it's reasonably simple to calculate the most likely missing tokens for any incomplete sentence. SRILM was one of the most common toolkits, but there are probably many other libraries.


Dataset - Sample pdfs for text processing? I'm looking for a rather large amount of pdf files for testing my text processing program. Tried looking for an open site to get like some thousand pdfs, but wasn't able to find anything. I don't really know if that is the right place to ask (probably not) but maybe one has a good tip for me.

Thanks in advance.

A GitHub repo with ~1,000 pdfs is here.

Another GitHub repo has a corpus of pdf examples, including edge-cases, is here.


How to achieve SHAP values for a CatBoost model in R? I'm asked to create a SHAP analysis in R but I cannot find it how to obtain it for a CatBoost model. I can get the SHAP values of an XGBoost model with
shap_values <- shap.values(xgb_model = model, X_train = train_X)

but not for CatBoost.
Here is the reproducible code for my CatBoost model:
library(data.table)
library(catboost)

train_example <- data.table(categorical_feature = c("a", "b", "a", "a", "b"), payment = c(244, 52352, 4235, 3422, 535), age = c(34, 27, 19, 40, 92), target = c(0,0,1,0,1))
label_values_train <- train_example$target
train_X <-train_example[, - "target"]
train_X$categorical_feature <- as.factor(train_X$categorical_feature)
# datapool for the train dataset:
pool <- catboost.load_pool(train_X, label = label_values_train, cat_features = 1)
fit_params_report <- list(iterations = 1000,
                          loss_function = 'Logloss',
                          eval_metric='F1',
                          border_count = 42,
                          depth = 8,
                          learning_rate = 0.04,
                          l2_leaf_reg = 0.001,
                          task_type = 'CPU')

model <- catboost.train(pool, params = fit_params_report)

How can I proceed to get the SHAP values here?

catboost::catboost.get_feature_importance(model, pool = pool, type = "ShapValues")


How to avoid resampling part of pipeline on test data (imblearn package, SMOTE) I am using the imblearn package to resample some data before applying other transformation/prediction techniques.  Specfically, I am using SMOTE in a slightly unconventional way, as a data augmentation technique to upsample all classes rather than just the minority.

I would like to do a pipeline of SMOTE -> PCA -> SVM on data separated into training and test (most likely K-fold).  However, consider the following snippet of code from the imblearn pipeline doccumentation:

pipeline = Pipeline([('smt', smt), ('pca', pca), ('knn', knn)])
X_train, X_test, y_train, y_test = tts(X, y, random_state=42)
pipeline.fit(X_train, y_train)
y_hat = pipeline.predict(X_test)

In the last line, it seems the test data will be resampled, which doesn't make sense -- only the training data should be resampled using this technique, or you will be testing on artificially generated data.  The documentation for pipeline.predict() implies samplers are applied during prediction as well.  Is there a way to avoid this, besides simply removing SMOTE from the pipeline?

Checking the code on GitHub, it seems that a 'transform' performing the resampling of data is not applied (cf : 'if hasattr(transform, "fit_sample"'): pass). So, from my understanding only modifications that do not resample the test set will be applied.


Does image's background matter for detector training (CNN)? Does an image's background matter for detector/localisation in the training part (using CNN)?

For example, if I want to make a face detector, which one is better as training dataset?

Faces cropped dataset
Faces in a global scene dataset

Does it matter?

Of course, it matters.

Which one is best completely depends on your problem. The golden rule for machine learning problems is that you want the data you train on to be as representative as the data the model will encounter once live.

Typically, faces cropped dataset will be an easier problem because the CNN can directly focus on the face, but if your use case will use images of faces that are not cropped, your model might not perform well.

So use whatever data you expect to encounter later on. The only thing you could do is add a face-cropping algorithm before your network, and then you could train on the faces cropped dataset right away.


How to reshape data for LSTM training in multivariate sequence prediction I want to build an LSTM model for customer behaviour. It's the first time for me working on a timeseries, so some concepts are not clear to me at all.

My prediction problem is multidimensional, meaning that I also want to predict many informations associated to an action for each customer.

The dataset is currently shaped as a list of 2d padded arrays of one-hot encoded features (customer actions + other informations), for example:

   customer_id               encoded_features
0   25464205      [[0,1,0],..,[1,1,1],[1,0,1],..,[1,0,1]]
1   56456574      [[0,1,1],..,[1,0,1],[1,0,1],..,[1,1,1]]

where each element in the encoded_features entries represents a specific timestep.

My idea here is to use keras input shape

(n. customers, n. timesteps, length of features encoding)

In the example above it would be (2,#timesteps,3).

I have two main questions:

Is this whole setting rigth for the prediction of next single customer action? I would like to simply give a new sequence of features for a certain customer and predict all features in the next timestep.
I am thinking about splitting the data (according to a certain ratio) into sequential training and test sets, in order to test the trained model on unseen feature vectors. In the example above it would be:

  customer_id       X_train          y_train
0   25464205       [[0,1,0],..]      [1,1,1]
1   56456574       [[0,1,1],..]      [1,0,1]

  customer_id       X_test           y_test
0   25464205       [[1,0,1],..]      [1,0,1]
1   56456574       [[1,0,1],..]      [1,1,1]

Notice that X_train and X_test will generally contain all Train/Test events, except for the last one which has to be predicted.
Is this a correct interpretation?

This makes sense. It should work for input and first couple of layers. For output layers, you can have a softmax if you need to generate only next record in sequence.

Following Keras code has an example that :

Accepts multi-dimensional inputs  (Each sample is a Sequence of video frames)
Predicts next few frames of video ( Multi dimensional since each pixel is a feature)

https://github.com/keras-team/keras/blob/master/examples/conv_lstm.py


How would you optimize this python/pandas code? In [1]: df.head()
Out[1]:

   uf    city
0  mg    araguari
1  mg    uberlandia
2  sp    sao-paulo
3  go    goiania
4  mg    belo-horizonte

In [2]: len(df)
Out[2]:

982

In [3]: indicators.head()
Out[3]:

     city               uf         population
 0   abadia-goias       go         6876
 1   abadia-dourados    mg         6704
 2   araguari           mg         19000
 3   abaete             mg         22690
 4   uberlandia         mg         700000

In [4]: len(indicators)
Out[4]:

5554

Now here I'm trying to include 'population' in df, using the corresponding uf and city. But this next code is too slow.

df['population'] = np.nan

for city, uf, index in zip(df.city.values, \
    df.uf.values, df.index):

    df['population'].iloc[index] = \
    indicators.population[indicators.uf == uf] \
    [indicators.city == city].values[0]

It seems you just need to merge two DFs:

df = df.merge(indicadores[['estado','cidade','population']],
              leftt_on=['address_locality','address_region'],
              right_on=['estado','cidade','city'],
              how='left')

NOTE: usually you don't need to specify left_on and right_on if the "joining" columns have the same names in both DFs (Pandas will use all common (with same names) columns
 from both DFs for joining/merging). But as i've seen from comments and from your question before you've edited it - it's not your case. So you would need to specify correct column names for left_on and right_on.

UPDATE: demo (using updated sample data sets):

In [7]: df
Out[7]:
   uf            city
0  mg        araguari
1  mg      uberlandia
2  sp       sao-paulo
3  go         goiania
4  mg  belo-horizonte

In [8]: indicators
Out[8]:
              city  uf  population
0     abadia-goias  go        6876
1  abadia-dourados  mg        6704
2         araguari  mg       19000
3           abaete  mg       22690
4       uberlandia  mg      700000

In [9]: df = df.merge(indicators,
   ...:               left_on=['city','uf'],
   ...:               right_on=['city','uf'],
   ...:               how='left')
   ...:

In [10]: df
Out[10]:
   uf            city  population
0  mg        araguari     19000.0
1  mg      uberlandia    700000.0
2  sp       sao-paulo         NaN
3  go         goiania         NaN
4  mg  belo-horizonte         NaN


Using Orange3 to predict image class I used logistic regression to classify 3 classes of images: cars, dogs, cows. The accuracy is 100%, perfectly separated in clusters (scatter plots).
When I try to use the learner to predict 3 images (1 dog, 1 car, 1 cow), the predictor simply assign the probability of 1/3 to each image. How can I fix it?

It means that the model you are using fails at classifying the images. Logistic regression may not be the good one for your classification. Try using another model.


How to predict an approximate weekly/monthly number, when the Unique Daily Visitors for that week/month are already known I am trying to come up with a formula or machine learning algorithm using which I can approximately predict the weekly or monthly users.

What to keep in mind is that I already have counts for the unique visitors per day for the week/months that I would like to make a near accurate prediction. Here, simply summing the daily unique users would not work, as they can be unique on one day but not on two days as they can have a session lasting over 2 days.

This method is to serve as an alternative to running a Spark job on the whole week/month data in order to save time and resources - Is this possible?

I have looked at Time Series and Linear Regression, but need more clarification on the possible approaches and also on any work-arounds?

To calculate the (estimated) value of the monthly/weekly unique visitors based on the per day count you may use the HyperLogLog algorihm.

This will give you the historical values and you can use any time series approach to predict future counts


Cross validation while preserving a column (not the target ) distribution So i'm doing cross validation and then i'm predicting using all the data on a test set ( a hold-out set ). My hold-out set has the same ratio on a column than the train ( seems thats how the test set was generated, a function that sampled it and tried to preserve the ratio for the target classes, and a particular column ) . My local CV is a bit lower than my score on the test set, and i think the problem is stemming from the fact that i'm using stratification only for 'y'.

Can lack of stratification of that feature be the reason of Cv & test scores aren't really close?

And if so how can i perform stratification for the target and a feature!
Thanks

Edit : i'm already doing stratification on the target since my data is imbalanced.

One idea would be to combine the two columns (one predictor and the target) and then stratify using the combined column.

Example: say for some observations, the target and the column take the following values: target = [0,1,0] and column = [A,A,B]. Then the combined column could look something like [A0,A1,B0] and could be used for stratification.

I'm obviously assuming that the predictor column is categorical - you might have to take a different approach for a continuous variable.


activation functions in multiple layers in CNNs An activation function (say sigmoid) is necessary on the final fully connected layer. But why is an activation function applied on the convolution layer too? As I understand it, the activation function is needed to apply only once, on 1 final layer.

Not only for Convolutional Neural Networks (CNNs), also for DNNs (Deep Neural Networks) and RNNs (Recurrent Neural Networks), we use activation functions at every layer. Sigmoid (for binary classification), softmax (for multiclass classification) or some other types are usually used at the final output layer, each specific for the kind of labels that we have to compare with the predictions.

However, other neurons require activation functions as well, especially for nonlinearity purposes; most popular ones are ReLUs(Rectified Linear Units), Leaky ReLU, tanh,.. and so on. We nearly always use an activation function for every neuron in Deep Learning. For a detailed insight, have a look at:

https://www.youtube.com/watch?v=Xvg00QnyaIY

Also specificly for Convolutional Nets:

https://www.youtube.com/watch?v=jPOAS7uCODQ&list=PLkDaE6sCZn6Gl29AoE31iwdVwSG-KnDzF&index=7

Hope I could help, please do not hesitate to ask more. Good Luck!


Objective function for multi-label classification The customary objective function for multi-label (e.g. M labels) classification is binary cross-entropy. The problem is, if we use binary cross-entropy, we are assuming that the output labels are independent of each other, turning the problem to M independent binary classification problems. Is there any suitable objective function that makes the output labels to be dependent on each other?

What you're looking for is called "cost-sensitive classification". Most methods however don't work with label similarities, but rather with relative penalties for different types of misclassifications.


Cross-Validation: Repeated K-Fold/Group K-Fold Repeated K-Fold vs Group K-Fold

As per my understanding from sklearn docs

Repeated K-Fold:

RepeatedKFold repeats K-Fold n times. It can be used when one requires to run KFold n times, producing different splits in each repetition.

Repeated Stratified K-Fold cross validator:

Repeats Stratified K-Fold n times with different randomization in each repetition.

Group K-Fold:

GroupKFold is a variation of k-fold which ensures that the same group is not represented in both testing and training sets.

  Can somebody explain in-detail, When would one use Repeated K-Fold over Group k-fold?
  What are the advantages/disadvantages of using Repeated K-Fold over Group k-fold?

Group k-fold is sufficiently specialized that the comparisons you ask for don't really make sense.  "Repeated" really does just mean to remake the splits multiple times; you could easily make a "repeated group k-fold" splitter.

You use group k-fold when you have groups you don't want split across the training and test sets.  For example, if your data includes multiple rows for each customer (but it still makes sense to train on individual transactions/rows), and your production use-case involves making predictions for new customers, then testing on rows from customers that also have rows in your training set may be optimistically biased.

You use repeated XYZ when the data and models have high variability and you want more performance datapoints with which to make statistically sound statements.  (Likely this is when your dataset is small, in which case happily your model building should be faster so that you can afford the computational cost of repeating the cross-validation procedure multiple times.)

https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html
esp. the image:

(but now I'd like to see one with groups not lining up with classes like that...)


Reading a CSV in TensorFlow RNN I am just starting off with TensorFlow and trying to implement an RNN for a dataset which consists of 11 Features (all numeric). These features will be used to predict the output of another column.

I am currently lost on where to start and tho I am able to understand how a RNN functions all the tutorials I could find were mainly related to image and text datasets.

There is no fundamental difference between using RNNs for text vs. for numeric values. In fact, text is more difficult because you have to preprocess the text to convert into numeric values first.  In any case you might have a look at this tutorial:

https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

I find this site to be pretty useful in general for examples of working with Keras.


tensorflow mnist tutorial i have a question about the tutorial of tensorflow to train the mnist database
how do i create my own batch without using next_batch() , the idea is to train with a batch of 50 ,then 100 and so but it has to be in order

here is the code

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y_conv), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
sess.run(tf.initialize_all_variables())
for i in range(20000):
    batch = mnist.train.next_batch(50)#revisar aqui
    if i%100 == 0:
        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
        print("step %d, training accuracy %g"%(i, train_accuracy))
    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

In a nutshell
If you have a look at what mnist.train is you'll find there are two numpy arrays in it:
mnist.train.images (shape (55000, 784))and mnist.train.labels (shape (55000, 10)).Then all you have to do is iterate on these.
Example
This example is voluntarily a bit complicated so that it is robust to whatever sizes you chose but keeps the idea of fixing a number of batches as primary information on how much you want to train on the dataset :
X_train = mnist.train.images
Y_train = mnist.train.labels
number_of_examples = X_train.shape[0]
batch_size = 1000
number_of_batches = 200
epoch = 0

for i in range(number_of_batches):

    j = (i - epoch) * batch_size % number_of_examples
    k = (i - epoch + 1) * batch_size % number_of_examples

    if (k < j):
        k = number_of_examples
        batch_x = X_train[j:number_of_examples, :]
        batch_y = Y_train[j:number_of_examples, :]

        print('Shuffling data to retrain on dataset')
        data = numpy.concatenate((X_train, Y_train), axis=1)
        np.random.shuffle(data)

        # redefine X_train and Y_train from shuffled dataset
        X_train = data[:, :784]
        Y_train = data[:, 784:794]

        epoch = i + 1
    else:
        batch_x = X_train[j:k, :]
        batch_y = Y_train[j:k, :]

    # [...]
    train_step.run(feed_dict={x: batch_x, y_: batch_y, keep_prob: 0.5})

j is the row of the dataset which will be the batch's first row
k is the last one, so j-k=batch_size examples per batch, as expected.
Except if you've reached the end of the dataset (k<j)
in which case you just go to its end and then shuffle the
dataset so that batches won't be the same.
You count i - epoch and not just i so that you're sure you'll always
start at the 1st row of the dataset even when you've already gone
completely through it.
For instance if number_of_examples = 55555, batch_size = 1000 and
number_of_batches = 200:  i = 56 is the first time you go again
through the (shuffled) dataset but 56 * 1000 % 55555 = 445 so you'd
start at row 445 and not 0.
Also you HAVE to concatenate, otherwise the pairs (example, label) will be lost.

i, j ,k % Whaaat?
Run the toy code below to see how the iteration index behaves:
number_of_examples = 55555
batch_size = 1000
number_of_batches = 200
epoch = 0

for i in range(number_of_batches):

    j = (i - epoch) * batch_size % number_of_examples
    k = (i - epoch + 1) * batch_size % number_of_examples

    if (k < j):
        print('Overlap : ', j, number_of_examples)
        epoch = i + 1
    else:
        print(i, '   ', j, k)


Model ensemble with Spark or Scikit Learn I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.

Here is a reproducible of what I would like my model to do with the Spark api

from sklearn.datasets import load_diabetes
import pandas as  pd
import pyspark
from pyspark.ml.feature import VectorAssembler, SQLTransformer
from pyspark.ml.classification import LogisticRegression
from pyspark.ml import Pipeline
# Query diabetes data
diab = load_diabetes()
df = pd.DataFrame(diab.data, columns=diab.feature_names)
df['is_male'] = df.sex > 0
df.drop('sex', inplace=True, axis=1)
df['label'] = diab.target
# Model made with Spark

spark = pyspark.sql.SparkSession.builder.master('local[10]').appName('A random spark context').getOrCreate()
def create_gender_model_male():
    return Pipeline(stages=[SQLTransformer(statement='SELECT * FROM __THIS__ WHERE is_male'),
                                VectorAssembler(inputCols=['age', 'bmi', 'bp', 's1'],outputCol='features'),
                                LogisticRegression(featuresCol='features', labelCol='label', maxIter=100,
                                                   elasticNetParam=1, regParam=0.)
                                ])
def create_gender_model_female():
    return Pipeline(stages=[SQLTransformer(statement='SELECT * FROM __THIS__ WHERE not is_male'),
                                VectorAssembler(inputCols=['age', 'bmi', 'bp', 's1'],outputCol='features'),
                                LogisticRegression(featuresCol='features', labelCol='label', maxIter=100,
                                                   elasticNetParam=1, regParam=0.)
                                ])
df = spark.createDataFrame(df)
class MixedModel():
    def __init__(self):
        self.models = {'male': create_gender_model_male(), 'female': create_gender_model_female()}
        self.fitted_models = {'male': None, 'female': None}
    def fit(self, df):
        self.fitted_models['male'] = self.models['male'].fit(df)
        self.fitted_models['female'] = self.models['female'].fit(df)
    def predict(self, df):
        return self.fitted_models['male'].transform(df).union(self.fitted_models['female'].transform(df))
mm = MixedModel()
mm.fit(df)
mm.transform(df)

Here, for example I have one logistic regression per sex but I would also like to be able to have prediction with a tree for males and prediction with Logistic regression for females if I want.

In a perfect world there would be a function:

ModelAggregation(('is_male is true, male, model_for_male), ('is_male is false', model_for_female)))

which would return me an object like my model aggregation

As you said in your question, there is no way to do that with the baseline algorithms provided in MLLib.

Two ways you could do that is by either :

Creating a function to generate a Pipeline
Creating a Meta Estimator that would take your base learners and the forking column.

The first one is what you have stated in your question and the second one has been explained by Brian Spiering.

As you said making a custom Estimator and Transformer will make it works nicely with all the MLLib Tools as Tuning Tools.

If you want a really precise example how to do so, there is my library which implements meta algorithms for Ensemble Learning with Spark.

There is a way to train multiple estimators in parallel quite simply in the Stacking Classifier, and for all of them you have an example on how to ask for an estimator as the parameter (to chose either your logistic regression or decision tree).

You can pick ideas out of that !

Edit : Similar question on StackOverflow


Entity recognition with context/relation Is there a way to get a specific entity based on the context where it is found?
For example:

The temperature today is 35°C.

Store risperidone tablet at 20°C.

Both are talking about temperature. For the first sentence I would want the temperature to be "WeatherTemperature" entity. The second sentence I would want the temperature to be "DrugTemperature". What model could I use to train for this behavior?

With Wolfram Language you may use TextCases or TextContents from the Text Analysis guide.  These are both experimental functions so may change a bit before they are finalised in a future version.  Both have the TargetDevice option so you can run them on one or more GPUs if you have them.

Reference the Text Content Types guide for the list of entities these functions have been trained on.

I think TextContents is an optimised call to TextCases for multiple cases so I will only use TextContents below.

Use TextContents to locate all trained entities in your sentences.

res1 = TextContents["The temperature today is 35°C.", TargetDevice -> "GPU"]

res2 = TextContents["Store risperidone tablet at 20°C.", TargetDevice -> "GPU"]

TextContents returns a Dataset object that can be Query'ed for specific cases.

For example for your "DrugTemperature"

res2[ContainsAll[{"Chemical", "Quantity"}], "Type"]

True

and the quantity units

res2[
  SelectFirst[#["Type"] == "Quantity" &],
  "String" /* SemanticInterpretation /* QuantityUnit]

"DegreesCelsius"

An Entity of type "Chemical" has a wealth of properties so additional information can be obtained if needed.  For example,

res2[
 SelectFirst[#["Type"] == "Chemical" &],
 "String" /* Interpreter["Chemical"] /* (#["MoleculePlot"] &)]

Hope this helps.


What optimization algorithms are used in scikit-learn? Gradient descent is one of the well-known optimisation algorithms. However, are the regression algorithms in scikit-learn implemented with gradient descent or some other techniques?

There are multiple approaches to optimization in scikit-learn. I will focus on generalized linear models, where a vector of coefficients needs to be estimated:

LinearRegression and Ridge use closed-form solution $\beta=(X^TX+I\lambda)^{-1}X^TY$, but Ridge can also use stochastic gradient descent or method of conjugate gradients
Lasso and ElasticNet use coordinate descent
OrthogonalMatchingPursuit uses a greedy algorithm with the same name, that has $L_0$ penalty on coefficients
ARDRegression and BayesianRidge use something like EM algorithm
SGDRegressor and PassiveAggressiveRegressor use guess what! Stochastic gradient descent.
HuberRegressor uses BFGS (a second-order optimization method)


Are non-relu activations better for small/ dense datasets? Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.

Why is ReLU used as an activation function?
Why ReLU is better than the other activation functions

Am I better off using a smoother activation like tanh if I have a smaller, dense dataset?

There is no reason to to predict tanh would be better if for a smaller, dense dataset.
The effect of activation function varies from dataset to dataset. You could it treat like an empirical question - find the best activation function for a specific data set like any other hyperparameter.


Predicting Customer Activity Absence Could you please assist me with to following question?

I have a customer activity dataframe that looks like this:

It contains at least 500.000 customers and a "timeseries" of 42 months. The ones and zeroes represent customer activity. If a customer was active during a particular month then there will be a 1, if not - 0. I need determine those customers that most likely (+ probability) will not be active during the next 6 months (2018 July-December).

Could you please direct me what approach/models should i use in order to predict this? I use Python.

Thanks in advance!

First issue in your model building method is that you have data in the binary form for each month and you are trying to predict into a 6 month time period. You gotta define what will be considered active in the next 6 months?

Active in 6 months  = Active in each individual months?
or
Active in 6 months = Active in any given month?

If you want to first predict for individual months, you can use logistic regression. It is useful for a binary classification of this kind.

Use LogisticRegression from sklearn.linear_model to train and fit the data.
Then, use confusion matrix and classification_report from sklearn.metrics to test the performance of your model.

After having predictions for next 6 months, you can create a new column that checks if there are any 1's in the last 6 months and stores 1 else stores 0


Feature importance ratio I trained a Random Forest classifier (sklearn) and consequently computed the feature importance and consequently ranked them. The forest has 100 estimators. My top 5 features with their importances are as here:

f1 = 0.91
f2 = 0.04
f3 = 0.013
f4 = 0.007
f5 = 0.004

To avoid over-fitting, I did the evaluation using cross-validation and learning curve.

My question is that the importance for f1 seems significantly higher than other features. Does it imply incorrectness (over-fitting?) of any sort? Should I do feature selection in some other way to generalize the model better?

In fact, this means that the quality of the feature f1 is very high. Usually, you should be worried if you get all the features giving the same importance level, and that level is low.

It is very important to understand that random forest does two level of randomization: at the data level and the feature level and it is very difficult to over fit.

Since you have this large number of trees in the forest, I do not think you have over fitting problem.


How can I infer no target in a target classification problem based on deep learning? Let's take the MNIST dataset (My application is different) with a lot of noise, I am going to train a deep NN to classify the letters.
What's the right way to infer, there's no letter possibility? or a letter not included in the training set?
Do I have to add the class "not a letter"  and try to pick or simulate the part of the dataset that has no letters but noise,
Or, I can do it without adding the extra label?

What you are looking for is anomaly detection, or novelty detection.

This cannot always be solved by labeling some images as "no letter", since all kind of images having no letters may not be available for labeling, or maybe too costly to exhaustively label.

This link has some directions you can look at for a start.


How do I use Convolutional Neural Nets to classify if there are rain drops or soil on the camera lens? I have certain videos for which the frames are labeled either as dirty (meaning the camera lens is occluded by soil or rain) or as clean.
The goal is to test a convolutional neural net on this data to evaluate how well it can classify if a frame is dirty or clean.

One idea is to use the first layers of an existing network and train last (fully connected) layers using the available data.

But one issue might be that basically all available networks are trained for object classification and might not be very suitable for tasks like soil and rain detection.

Do you have any recommendations about networks or models which might be suitable for this task?

So neural nets can be quite powerful for end-to-end training. So if you have a dataset that contains both clean and dirty images, and also the classes you are trying to detect in the first place, it should be fairly straightforward to try to train you net.

I'd proceed as follows.

adjust the network to produce a logits vector that outputs not only the object classification but also a binary classifier for whether your image is clean or dirty.
then adjust the labels for each image to indicate the classification label and the cleanliness of your image.
I'd then train and test on the dataset to see if a model emerges.

The convolutional parts of your network should converge to features that are good at
A. classifying the subject of the image, and
B. detecting whether the camera is dirty or not.

Its a good idea and I might use it myself!


help finding research discussion on HTS classification My question is about the theory of this problem, and not necessarily syntax.

I'm wondering if anyone here has experience with automating HTS (Harmonized Tax Schedule) classifications, specifically training a classifier to predict the HTS code that should be assigned to a product? There are rules for these designations that are stored as descriptive text data that 'explain' the product. This seems (to me) to make predicting how a part will be classified very difficult.

For context, it would seem that NLP is heavily involved in this process, but with the high cardinality involved in these descriptions and the high amount of overlap from one description to another, are there any best practices for success in making such predictions? Similarly, I'm having a hard time finding any public research on the matter. It appears that almost all advancement in this focus was developed behind the closed doors of companies. I understand why they'd make that decision, but I am a bit surprised how little public discussion exists on the topic considering how much business it affects across the globe.

things I am considering/researching:

Word vectorizing (Word2Vec): I got the idea from another user here, but given that it is typical to train a vectorizer with a large public corpus, I am not sure how that would affect the vectoring process, as the products I am classifying are domain specific.

n-gram: I suppose using 2, 3, or 4 would be sufficient for classification, given that most descriptions that I can see are succinct. But as I said, a lot of these descriptions will have overlap, for example:

    HTS CODE             HTS Description
------------------     ------------------
11111 = 'football'     An inflated ball (used for kicking)
11112 = 'basketball'   An inflated ball (used for dribbling)
11113 = 'ball'         An inflated ball

In this case (if my understanding is correct), n-gram 6 would be needed for the first two products but only n-gram 3 is needed for the third product. Is the n-gram value something we can 'soft code' to adapt to each record or will I need to somehow find the appropriate number for n?

Thanks

I would approach such a problem with a method a bit similar to record linkage: trying to match every product description with the most relevant HTS description. The traditional approach would be to use textual similarity measures such as cosine TF-IDF, but many variants can also be considered, e.g. with embeddings or other ways to take semantic similarity into account. At the end the code corresponding to the most similar HTS description is predicted.

Initially this could be done with vectors of words (unigrams), but it's true that comparing vectors of $n$-grams is likely to be more precise. However $n$-grams don't work the way you describe: in a case like your example you would have for instance $n=2$ and extract all the sequences of 2 consecutive words. This "bag of $n$-grams" is what the vector represents. Combining different lengths of $n$-grams is possible but not in the same representation: for example you could measure cosine similarity over 2-grams vectors only, then measure cosine similarity of 3-grams vectors only, and take the mean of the two scores (or even build a regression model using different similarity scores as features).

In case it helps, here is an example of computing a very simple similarity score with bigrams (one could certainly find better examples online).


Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks? I understand that there are rarely general recipes in field of machine learning and the many results can be achieved only by trial and error, and are task specific as well.
My question is, if the model doesn't give a desired quality, but one may expect, that the model of this or similar architecture can achieve a reasonable result, based on some prior assumptions or knowledge, what is, in general, better strategy, more successful in majority cases:

Tuning of the optimizer parameters of the optimizer. (playing with the learning rates, random seeds, weight decay, momentum, e.t.c). hoping that it converges to the better optimum
Changing some architecture properties of the model (number of hidden layers, dropout rate, width, number of filters in CNN)

Here I assume, that the current network is sensible choice - not 1-hidden layer narrow MLP - for solving some complicated task.
Minimization of loss function doesn't always lead to improvement of the quality of the model on the test data (well known problem of overfitting), but here let us assume, that one has lot enough data, and the training and test set are organized in a such way, that good score on the training examples would also lead to good accuracy on the test data.

The short answer, is that you want the fastest way to reach the performance you expect/desire. This would mean first playing with some hyperparameters like learning rate, initialisation strategy, trying different optimizers, etc.
Having said that, if you managed to write your training process such that it is trivial to swap out the model completely for another one, you can also give that a go.

If you have a model that is known to work well for a given type of task, but is really under-performing (e.g. not converging or a large difference to results in the literature), then I would first check for some common issues in the training process:

data quality: are the inputs scaled correctly? Should you add some
data volume: if you don't have many samples, can you get more or use augmentations?
loss function: are the outputs of the model being correctly consumed and the losses correctly computed? are you even using the right loss formulation?

This is essentially debugging your training setup.
I would convince myself as far as possible that none of these exist before building a different architecture or a radically different approach. The main reason for this, is that I assume you did some research into the problem and the type of models that are known to perform well - so if this doesn't work, what makes you think another architecture will?  :)


Always drop the first column after performing One Hot Encoding? Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to always drop the first column after performing One Hot encoding, regardless of the algorithm of choice ?

This question, in a slightly different form, was discussed herein earlier.
You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you want to drop at the very beginning?
In fact in pandas.get_dummies there is a parameter i.e. drop_first allows you whether to keep or remove the reference (whether to keep k or k-1 dummies out of k categorical levels). Please note drop_first = False meaning that the reference is not dropped and k dummies created out of k categorical levels! You set drop_first = True, then it will drop the reference column after encoding.


String classification beginner question I analyse log files, very often there are strings that only differ on few places and are same every where else, I am trying to find the generic string that they most likely belong to e.g. :

UHGUYGUYGUYGUY id = U1234 UYAG*&^T*#@G*(&G@ id2 = 8767 ib97y79yh0978
UHGUYGUYGUYGUY id = Z1D#34 UYAG*&^T*#@G*(&G@ id2 = 98h ib97y79yh0978
Sss3ug87g87g78ghs837g8 obj { 876t7g }937hs937hs973h97sh397 jh7897y98h
Sss3ug87g87g78ghs837g8 obj { 98u2 }937hs937hs973h97sh397 ZN7897y98h

to me the only difference is between the two the value of ids, so a generic forms/groupings will be

UHGUYGUYGUYGUY id = * UYAG*&^T*#@G*(&G@ id2 = * ib97y79yh0978
Sss3ug87g87g78ghs837g8 obj { * }937hs937hs973h97sh397 *7897y98h

I am not sure what in machine learning I should be looking up for this problem, or even if this problem has a name.

of course this is a very simplified example, the number, location of id's can vary for different generic cases, that is why I can not write old fashioned code for this, too many things can change.

Is there anything in machine learning to help find groups for such strings? if yes, what is it called?

That sounds like k nearest neighbor clustering because a large portion of the string is similar. This answer to a similar question from 2014 has code you could try in Python.

https://stats.stackexchange.com/a/158090