Hindsight Experience Replay, how to define a partially-known End-Goal One of the requirements of the Hindsight Experience Replay is supplying the DQN with a state and a goal (the desired end-state) that we hope to end up in:

<currState, goal>    <-- For Hindsight Experience Replay;
<currState>          <-- This would be an input for a usual DQN;

This paper allows to quickly learn when the rewards are sparse. In other words when the rewards are uniform for most of the time, with only a few rare reward-values that really stand out.

Question:

Let's say I want to have the player be killed by monsters in my game. Thus, my "goal state" must include a value of 0 for player's hit-points. However, the state-vector also includes his position (xyz coordinate), rotation vector, IDs of equipped items:

inputVec = <hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID>

I don't want to impose a specific position of a player, etc - I just want him dead. I only know what his 'hp' should be (should be zero), I don't care about the other values.

Therefore, I can't provide a perfectly well-defined goal vector - does this mean I can't use Hindsight Experience replay?

Edit:
my understanding is that components of currState and goalState must have identical components. We can't have these 2 vectors be of different sizes or store different things

Edit after accepting the answer:

As @lfelipesv mentioned, page 4 tells us:

  We assume that every goal $g ∈ G$ corresponds to some predicate $f_g : S → \{0, 1\}$ and that the agent’s
  goal is to achieve any state s that satisfies $f_g(s) = 1$. In the case when we want to exactly specify the
  desired state of the system we may use $S = G$ and $f_g(s) = [s = g]$

  The goals can also specify only
  some properties of the state, e.g. suppose that $S = \mathbb{R} ^2$
  and we want to be able to achieve an arbitrary
  state with the given value of x coordinate. In this case $G = \mathbb{R}$ and $f_g((x, y)) = [x = g]$.

I've got an answer from the Author:

  The simplest solution would be to have 2 types of goals - a) specific hp and position; and b) specific hp, arbitrary position. I haven't tried anything like that though.

Based on the response I reason it might work if we do the following:
use a placeholder value, perhaps for the last component of the input vector.
This way the DQN will notice that when this "last bit" is on, only health matters.

<hp,  x1,x2,x3,  q1,q2,q3,  chestID, handsID, headID, feetID,   placeholder0or1>

Edit: if you know a better way, please share it as an answer!


Need help with entity tagging I need to design a system which can identify movie and production company names in a sentence.

The approach that comes to my mind is to train a NER Named-entity recognition system on labeled data so that it identifies the corresponding entities. But what about new entities (movie or production company name) which trained system hasn't seen, how can we tag them. Re-training the model every time with new released movies won't be feasible.

  Labeled data: Sentences with the position of words that corresponds to movie or production company name

I am a beginner in NLP any help would be appreciated

But what about new entities (movie or production company name) that
  trained system hasn't seen how can we tag them. Re-training the model
  every time with new released movies won't be feasible.

A NER model should not have to be retrained to tag a new text it has not seen before. If trained successfully it will use information it learned from the labeled data and be able to apply it on new data. For your use case this could be information such as:

Capitalization - common for movie titles and production company names
Words used - frequently used words for either movie titles or company names
Where in text - movie titles perhaps appear early in texts

spaCy is a good library to get started with NER. Here is an example on how to train one using your own data.


PyTorch MultiLayer Perceptron Classification Size of Features vs Labels Wrong I am getting the following error:

/usr/localnfs/Compiler/python/Anaconda3-2019.07/lib/python3.7/site-packages/torch/nn/modules/loss.py:431:
UserWarning: Using a target size (torch.Size([30, 1])) that is different to the input size (torch.Size([30, 10])).

This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

I think I know why this is happening but I don't know how to fix it. I am using the loss function torch.nn.MSELoss(), with no inputs. My model is the multilayer perceptron, and this is how I'm doing it:

class MLP(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MLP, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.fc1 = nn.Linear(self.input_size, self.hidden_size)
        self.fc2 = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, features):
        out = self.fc1(features)
        out = self.fc2(out)
        return out

My input size is 100, my hidden size is 50, and my output size is 10, because my dataset has 10 columns with 30 rows (the size of my mini batch). The size of my labels is 30x1.

 predict_label = mlp_model(features.float())
 loss_batch = loss_func(input=predict_label, target=label.float())

I am somehow supposed to predict labels for my 30x10 dataset, and compare them to my 30x1 label vector, and I don't know how to do that. The error shows up when I execute the loss_batch line. Please help, I'm very new to this.

You have stated that:

Batch size: 30
Input dimensionality: 10
Label dimensionality: 1

With this in mind, the output dimensionality of the MLP must be the same dimensionality as the labels, which is 1. Therefore, the output size of the MLP cannot be 10, but 1.

Apart from that, please note that concatenating 2 linear transformations (without any non-linearity in between) is equivalent to a single linear layer. Your MLP should have a non-linearity (e.g. ReLU) applied after fc1 and before fc2.


Minkowski distance with Missing Values Im currently doing a subject for data science, and have the following point that im trying to understand.

We are looking to calculate distance in data sets where values may not be present. Now i know that R does this by default, but we are learning the "how" behind the what.

Literature we are give states.
"The idea is to normalise the inner sum by the number of valid (non-missing) terms, so distances computed from different amounts of terms are commensurable. Otherwise, distances computed with fewer missing values tend to be artificially larger."

Given the following data set.

We have the following sample formulas for Euclidean and Manhattan

Euclidean distance: d(x1, x2) = √(4/3) *( (2 – 7)2 + (1 – (-4))2 + (0 – 8)2 ) = √(4/3)*114 = 12.328

Manhattan: d(x3, x4) = (4/2) * ( |3 – 10| + |2 – 5| ) = (4/2)*10 = 20

Assuming the normalization section fro euclidean is number of non missing terms of each row divided?

How do you derive the normalization section on the Manhattan formula?

The way you type formulas is a bit confusing, but here's a go at interpeting it.

Euclidean distance is defined by:

Manhattan distance is defined by:

Handling missing terms is an independent issue. The way it is handled in the example is by taking the average of the present features (dividing by their number) and multiplying by the total number of features as a way to bring the data to a scale comparable to datapoints without missing features.

Now let's have a closer look at the example. The total number of features is 4. To calculate a distance between datapoints you can only use features that are present in both:

The first (Euclidean) formula finds 3 features present in both x1 and x2. Thus your coefficient is 4/3.
The second (Manhattan) formula finds two features present in both x3 and x4. Thus the coefficient is 4/2.


How do I send the results of a convolutional layer and non-deep-learning features into a dense layer in Keras? I understand that I can set up a convolutional network for 1-dimensional sequence/time series.

model = Sequential()
model.add(Conv1D())
model.add(GlobalMaxPooling1D())
model.add(Dense())

Let's say I'd like to use "regular" (non-deep-learning) features too in my model, how should I best combine the two at a dense layer?

Concretely, let's assume that, for each row of my dataset, there are 1k points in the time series, along with 100 "regular" features.

To generalize my question, let's say there are now two kinds of time series plus regular features for each row in my dataset. If I would like to have a separate convolutional block for each time series, how do I combine all three?

This can be done with Keras functional API.

In this example, "merge_1" layer gets input from :

Output of LSTM Layer (lstm_1)
aux_input layer

More info : https://keras.io/getting-started/functional-api-guide/


Prepare data for an LSTM -I want to make a python program using the LSTM model to predict an output value that is 1 or 0.
-My data is stored in a .csv file of the form: (Example of the line)
Date              time         temperature     wind  value-output
10-02-2020        10:00            25           10         1
-I found several courses, several examples of LSTM but I don't find my classification problem to do the same thing, there are many examples on translation.
-I am stuck on how to prepare my program my data to give them to the LSTM model.
-I want to take into consideration my temperature and wind inputs in addition to the time to predict the output value.

(I have already made a python program based on a simple ANN to predict my output value by following a tutorial), but for the LSTM I find it difficult.

Thanks in advance for your help.

You have to prepare your data as a numpy array with the following shape:

( Number of observations , Input length , Number of variables )

Assuming you are working with Keras, the input of the LSTM() layer is as above, but you don't need to report the number of observations: input_shape = (Input length , Number of variables). Input length is an hyperparameter of your choice.

I pushed this Notebook on GitHub that contains a function to preprocess your dataset for RNNs:

def univariate_processing(variable, window):
    ''' RNN preprocessing for single variables.
    Can be iterated for multidimensional datasets. '''
    import numpy as np

    # create empty 2D matrix from variable
    V = np.empty((len(variable)-window+1, window))

    # take each row/time window
    for i in range(V.shape[0]):
        V[i,:] = variable[i : i+window]

    V = V.astype(np.float32) # set common data type

    return V

def RNN_regprep(df, y, len_input, len_pred): #, test_size):
    '''
    RNN preprocessing for multivariate regression. Builds multidimensional
    dataset by iterating univariate preprocessing steps.
    Requires univariate_processing() function.
    Args:  df, y:                X and y data in numpy.array() format
           len_input, len_pred:  length of input and prediction sequences
    Returns:  X, Y matrices
    '''
    import numpy as np

    # create 3D matrix for multivariate input
    X = np.empty((df.shape[0]-len_input+1, len_input, df.shape[1]))

    # Iterate univariate preprocessing on all variables - store them in XM
    for i in range(df.shape[1]):
        X[ : , : , i ] = univariate_processing(df[:,i], len_input)

    # create 2D matrix of y sequences
    y = y.reshape((-1,))  # reshape to 1D if needed
    Y = univariate_processing(y, len_pred)

    ## Trim dataframes as explained
    X = X[ :-(len_pred + 1) , : , : ]
    Y = Y[len_input:-1 , :]

    # Set common datatype
    X = X.astype(np.float32)
    Y = Y.astype(np.float32)

    return X, Y

Let me know if that's what you were looking for.


A few ideas to parse "events" from a text document I'm working with lots of press releases (mostly PDF or DOC documents, i.e. text). I would like to automatically parse them into a CSV file, 1 line per event.

Input:

Output:

datetime;location;title;description
2016-06-02 18:30;Public Library;About Picasso's last years;Lorem ipsum...
2016-06-03 17:30;Public Library;1001 stories;For children, 3 to 6 year old

I could write a parser for each precise layout of document (e.g. date always in bold, description always in italic) or detect keywords (example: january, february, etc. => assume the nearby words are a date, etc.). But this would require lots of work for every different press release.

I'm looking for more clever solutions, maybe involving machine learning and/or neural networks. I'm open to take a MOOC on this subject (I have a math PhD background and programming skills).

What general techniques do you see to address such a big (and difficult topic) concern ?
Are there open-source projects dedicated to that?
Some Python libraries that would make this task easier?
Any general idea?

These documents can be very unstructured and machine learning rather takes perfectly structured data like a single clean table of numbers. The real hard part, how to translate all the interesting information (font size, bold, position, order, ...) into a clean table, isn't solved by machine learning. You see it's hard to create a sensible, purely numeric table from a document.

I think your best bet is to use a natural language processing toolkit and do named entity recognition. See http://www.nltk.org/book/ch07.html "5 Named entity recognition". Or see the recommendation here: https://stackoverflow.com/questions/11333903/nltk-named-entity-recognition-with-custom-data

With this auxiliary information, I'd rather spend time handcoding rules, which work on most documents. You can create a nested data structure and try to match patterns like "headline, enumitem1, enumitem2, ..."

The rules would throw errors on documents where they are unsure. For these documents you will have to manually check what's going on. But this way you don't lose information.

I'm no expert on the very advanced methods, but in theory, with a lot of labeled data, a very clever encoding schema and a fair bit of luck an approach like RNN might be successful. But even if all the mentioned requirements hold, the training of those RNNs require a lot of experience that is hard to learn from references.


Decision Trees split in scikit I am working with the titanic dataset and using decision trees for analyzing the age covariate. I'd like just to see whether kids are more likely to survive than adults. I implemented my own Gini coefficient and I had plot the coefficient by age: dataset here titanic ds

import pandas as pd
import seaborn
import matplotlib.pyplot as plt
from sklearn import tree
import graphviz
import numpy as np

def gini_by_age(df, t):
    df['age_group'] = df['age'].apply(lambda row : 0 if row <= t else 1)
    kids = df[df['age_group'] == 0]
    kids0 = kids[kids['survived'] == 0]
    kids1 = kids[kids['survived'] == 1]
    adults = df[df['age_group'] == 1]
    adults0 = adults[adults['survived'] == 0]
    adults1 = adults[adults['survived'] == 1]
    gk = 1 - (len(kids0)**2 + len(kids1)**2)/float(len(kids))**2
    ga = 1 - (len(adults0)**2 + len(adults1)**2)/float(len(adults))**2

    return gk + ga

def plot_gini_by_age(df):
    ages = range(2,25)
    y = [gini_by_age(df, a) for a in ages]
    plt.plot(ages, y)
    plt.show()

def use_tree(df):
    X = np.array(df['age']).reshape((len(df['age']),1))
    y = df['survived']
    clf = tree.DecisionTreeClassifier(max_depth=1).fit(X,y)
    dot_data = tree.export_graphviz(clf, out_file=None)
    graph = graphviz.Source(dot_data)
    graph.render("age")

titanic_df = pd.read_csv("titanic_ds.csv")
ages_cov = titanic_df[['age', 'survived']].dropna()
plot_gini_by_age(ages_cov)
use_tree(ages_cov)
print gini_by_age(ages_cov, 5)
print gini_by_age(ages_cov, 8.5)
print gini_by_age(ages_cov, 15)

output:
    0.925844132419
    0.937732003001
    0.963875889772
I see from the plot that gini coefficient has local minima at roughly 5, 8 and 15 years old and the best is at 5. But scikit gives me 8.5 years old as the best split. What is wrong here?

I got it thanks to the scikit team, I put the answer here for the people to come. The split used in scikit uses weights in calculating the Gini coefficient, just add the following lines before returning:
    ....
    gk *= len(kids)/len(df)
    ga *= len(adults)/len(df)


How to use lists in Tensorflow? I have a number of lists, such as [1,2,3,4], [2,3,4], [1,2], [2,3,4,6,8,10], whose lengths are obviously unsame.

How can I use this as input of placeholder in Tensorflow?

As I have tried, the following setting will raise error.

tf.constant([[1,2],[1,2,3]...],dtype=tf.int32)

So I guess placeholder cannot be set by the upper input of lists.

Is there any solution?

Edit:

The following is my example. How to make it run without errors?

When you create a Numpy array like this:

x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]])

The internal Numpy dtype is "object":

array([[1, 2], [4, 5, 6], [1, 2, 3, 4, 5, 6]], dtype=object)

and this cannot be used as a Tensor in TensorFlow. In any case, Tensors must have same size in each dimension, they cannot be "ragged" and must have a shape defined by a single number in each dimension. TensorFlow basically assumes this about all its data types. Although the designers of TensorFlow could write it in theory make it accept ragged arrays and include a conversion function, that kind of auto-casting is not always a good idea, because it might hide a problem in the input code.

So you need to pad the input data to make it a usable shape. On a quick search, I found this approach in Stack Overflow, replicated as a change to your code:

import tensorflow as tf
import numpy as np

x = tf.placeholder( tf.int32, [3,None] )
y = x * 2

with tf.Session() as session:
    x_data = np.array( [[1,2],[4,5,6],[1,2,3,4,5,6]] )

    # Get lengths of each row of data
    lens = np.array([len(x_data[i]) for i in range(len(x_data))])

    # Mask of valid places in each row
    mask = np.arange(lens.max()) < lens[:,None]

    # Setup output array and put elements from data into masked positions
    padded = np.zeros(mask.shape)
    padded[mask] = np.hstack((x_data[:]))

    # Call TensorFlow
    result = session.run(y, feed_dict={x:padded})

    # Remove the padding - the list function ensures we
    # create same datatype as input. It is not necessary in the case
    # where you are happy with a list of Numpy arrays instead
    result_without_padding = np.array(
       [list(result[i,0:lens[i]]) for i in range(lens.size)]
    )
    print( result_without_padding )

Output is:

[[2, 4] [8, 10, 12] [2, 4, 6, 8, 10, 12]]

You don't have to remove the padding at the end - only do this if you require to show your output in the same ragged array format. Also note that when you feed the resulting padded data to more complex routines, the zeros - or other padding data if you change it - may get used by whatever algorithm you have implemented.

If you have many short arrays and just one or two very long ones, then you might want to consider using a sparse tensor representation to save memory and speed up calculations.


predicted bounding boxes that stretch beyond grid cell (Andrew NG CNN course)? I was following Pr. Andrew Ng course on Course about Convolutional neural network and I have a doubt regarding one of the points he mentions in the Yolo algorithm.

In one of the slides he mentioned two key points:

1) For each grid in our $3 \times 3$ grid image will have 2 predicted bounding

2) And each of these bounding boxes will be bigger than the size of the grid.

I couldn't get why there will be $2$ predicted bounding boxes? Is it because we consider two anchor boxes?

Also, how can the bounding boxes bigger than the size of the grid? Because we know that each object can belong to one grid only based on the midpoint

1) Exactly. You have two anchor boxes in Andrew's current example, so the algorithm is going to output two predicted bounding boxes for each grid cell.

2) Your statement below is not true:

  "Because we know that each object can belong to one grid only based on the midpoint"

I don't remember that being said on the course. The center of the object belongs to a single cell.

TO CLARIFY: The fact that an object spans at a region greater than the grid cell it is assigned to has nothing to do with the grid cell's size itself. The object can and will be larger than its assigned grid cell. However, the output assigns each object to one grid cell because that grid cell contains its midpoint.

Anyway, the receptive field of the neurons is much larger than the single cell they process (i.e. they cover the whole image). Anchors are initialized on a certain width and height but will be resized during inference, based on the identified object size using the final feature map. So one might consider that yolo predicts the occurrence of an object as well as its size.

More on the receptive field of CNNs

Check the: "A regressor rather than a classifier" part

  A taste: For every positive position, the network predicts a regression on the bounding box precise position and dimension

This has been also asked here


Literature on selecting specific dimensions in a word embedding vector I am aware that the different dimensions in the word embedding represents different information and algebraic operations can be performed between two embeddings for example.
Can anyone point me to literature on selecting specific dimensions from a word embedding vector. I don't mean dimensionality reduction, but papers following the theory that all those dimensions are not important for all the tasks and that specific ones will be more important for a specific task. For example: Sentiment Analysis would benefit from dimension 5th to 55th and 250th to 300th instead of using the whole 300 dimensions.
Let me know if this theory isn't true.

There is not much theoretical work on the topic. The number of dimensions for an embedding is an empirical question depending on the corpus and the task.
The "GloVe: Global Vectors for Word Representation" paper by Pennington  showed that there are large gains in accuracy by adding dimensions up to couple of hundred then diminishing returns.

The most common number of dimensions is either 128 or 256. 128 and 256 are both powers of 2 which can speed up training time.


Is loss the same thing as variance? In Keras, the first object returned in the score list is loss.

score = model.evaluate(X_test, y_test,verbose=1)
print(score)

[1] [0.025217213829228164, 0.99487179487179489]

Is this the same thing as variance?

No. Loss measures the error between your predicted values and true values in a given train set. Whereas variance mesures how your model performs (i.e. how the loss changes) on different training sets. Also, variance is an atribute of a given model whereas loss depends on your dataset.

In other words, we can also say that loss on different sets is used to measure variance of a given model.


Name of ML discipline for trainable clusterer - supervised clustering, or group ID assignment Is there a name for a machine learning algorithm, that learns 'clustering approach' from examples of clustered datapoints (with different numbers of clusters each time)?

Imagine the training set as an ensemble of sets with their clusters assigned.

Has anyone please encountered a similar problem in any literature? My goal is to read something more about this problem.

Example #1:
Underlying rule: "cluster words to the same group if they do exist on the same page of an imagined storybook"
Training dataset:
1) Source values: [banana,car,rock,orange] -> Target groups: [banana and orange],[car],[rock]
2) Source values: [kiwi,audi,folswagen] -> [kiwi,audi,folswagen]
...) .. and so on...

... And that we do not know the original story, but only instances of (datapoints -> their groupping). And also we do not know the rule :)

Example #2:
Underlying rule: "be evil - cluster like kmeans, but if there seems to be a cloud of points looking like a smiley, do not cluster at all!"
Training dataset:
1) Source values: [bunch of points in a shape of two circles] -> [each circle is a cluster]
2) Source values: [bunch of points and a cloud of points in a shape of smiley] -> []
...) and so on ...

Yes it can be solved by detecting if there is the smiley, but we do not know the rule, we want the algorithm to learn it from the data.

Edit:
maybe the word clustering is not a wise choice from me. The quality of output will not be evaluated by user, but on a big validation set to see if the algorithm did learn from samples it has seen. There also supposedly does exist one best solution, from the nature of our problem (the underlying rule is always present, although not known).

Original, unclear, description before editing:
The problem is as follows:
Each row of data is a shuffled list/set of datapoints (source variable) and their assignment to groups (target variable).
It would seem as a classification task, BUT the groups make sense only to each specific datarow (and their number also varies).
It would seems as a clustering task, BUT the underlying principle needs to be applied again and again to each different row.

Perhaps you are looking for some combination of Meta clustering and Ensemble clustering?

Meta clustering

  A sampling-based approach that searches for distance
  metrics that yield the clusterings most useful to the user

...

  Generate many good, yet qualitatively different, baselevel
  clusterings of the same data.
  Measure the similarity between the base-level clusterings
  generated in the first step so that similar clusterings
  can be grouped together.
  Organize the base-level clusterings at a meta level (either
  by clustering or by low-dimension projection) and
  present them to users.

Ensemble Clustering

  In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble consists of only a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.


what is difference between Logistic regression and SGDClassifier with log loss OR SVM and SGDClassifer with hinge loss? Can we just use SGDClassifier with log loss instead of Logistic regression, would they have similar results ?

From a practical point view, yes you could use both options seamlesly with in general similar results (check this scikit learn functionality), but:

SGD (Stochastic Gradient Descent) is an optimization algorithm among others
log loss/hinge loss... are the loss functions used in the selected optimization strategy (SGD in your case) to find the optimal weights to fit such linear models, but another optimization solvers  could be used with those algorithms. See this post for logistic regression with sklearn, where you have the solver parameter, which can be, among others, 'sag' (stochastic average gradient descent), but others like 'lfbs' are also accepted.


Difference between coefficient of determination and least squared error? Where and when should I consider R^2 as a measure of goodness-of-fit for regressions?
Usually I choose the least squared model as the best model.
Is it possible that the least squared model does not have the highest R squared?
Should I use both as the evaluation of regressions?
Could you give me an intuitive example?

R squared is one of many model diagnostics. One calculation of it is (1-var(residuals)/var(response variable)). Since the response variable is constant, you want to minimize the average squared residuals (the variance), which is identical to least squares.

The issue with using only R-squared is that if you have a lot of variables in a model, you will likely get a high R-squared, but that doesn't mean the model is good.  In fact, if you have 100 rows of data, 100 independent variables and 1 response variable, there's a good chance you will get an R-squared of 1, but the model will be completely overfit.

There are 2 common solutions to this problem:

Add a penalty term that reduces (or increases, depending on what is optimal) the "score" of a model when it has more variables.
Use a form of validation to see if the model performs well on data it wasn't trained on.

Oftentimes both of these are used to varying degrees, but here are a few examples:

AIC, or Aikaike Information Criterion, is -2*log(P(Y|X)) + 2*p, where P(Y|X) is the probability density of Y given X under the current model and p is the number of predictor (X) variables.  The goal is to minimize AIC.
In K-fold cross-validation, where K is the number of subsets you divide data into, you train the data on K-1 of the sets combined and then test it on the remaining set and report the error.  You repeat this process for each combination of K-1 training sets and 1 remaining set and combine the K results (usually by averaging error).

The first method is usually a safe bet for linear regression models, but oftentimes, especially when you are using more complex algorithms, you will want to validate the algorithm to make sure it isn't overfitting the training data.  Also, using validation allows comparisons between different models (especially non-parametric ones) to be made much more easily.

For more ways of evaluating models, I would suggest reading into different regularization techniques and different validation techniques.


How to use k-means outputs (extracted features) as SVM inputs? l have a dataset of images with their labels.
l put them into a k-means algorithm (as a feature extractor).
Now, l would like to use this new representation of images (features extracted from k-means algorithm) as SVM classifier inputs.
How can l do that ?
Number of cluster k=400 and numbers of images=1000.

However, l just have the vectors of centroids (400 centroids)

l need to get the representation for each image with respect to the centroids.

EDIT1

package update

from sklearn import mixture
gmm = mixture.GMM(n_components=6).fit(X)

Now l would like run k-means with different k=range(50,500), how can l get the distances for each k ?
Is is correct to do the following :

K=range(50,500)
KM=[KMeans(n_clusters=k).fit(X) for k in K]
distances = [np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in C.cluster_centers_]) for C in KM]

'Prediction' of k-means algorithm for each observation is just the corresponding centroid. So you can take vector of predicted centroids and use it as a categorical feature (maybe one-hot encoded).

But it is just one feature. With little coding you can do better. For example, you can find for each sample its distance to each of $k$ cluster center, and so create $k$ new features. A Python example:

from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.svm import SVC
import numpy as np
iris =  load_iris()
X = iris['data']
y = iris['target']
kmeans = KMeans(n_clusters=6).fit(X)
distances = np.column_stack([np.sum((X - center)**2, axis=1)**0.5 for center in kmeans.cluster_centers_])
svm = SVC().fit(distances, y)

Another (and maybe simpler way) is to fit a gaussian mixture model (e.g. by scikit-learn). It is similar to k-means, but for each observation produces a probability distribution over clusters, instead of a single cluster label. These vectors of predicted cluster probabilities may be used as features as well.

from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=6).fit(X)
proba = gmm.predict_proba(X)
svm2 = SVC().fit(proba, y)


Handwritting Recognition moving from character level to word level Given the experience on MIST, I try this problem as a character level.

I have a handwritten text and I want to "OCR" it. Even though I made progresses with openCV (on the image pre-processing, before a DNN classifier).

I think that the best approach would be to move to word level (into RNN). I am writing this in Python. But I don't know how to change the last layer of the DNN into an input layer for RNN.

Is there any code for dummies to see how is this done (for images as input of the first DNN layer)? Besides I see that you could put a language model on top (this would be a dictionary right?). Advice?

I do not think that you would need a recurrent neural net here. This will be much slower to train than a ConvNet. Additionally, your data is images, correct? I think you could use a deep convolutional net to a fully-connected net. This should be fast and perform well. Please look at the following archived blogs from keras.
Building powerful image classification models using very little data
How convolutional neural networks see the world


Should I apply 1D or 2D CNN on binary text classification? I am trying to train a text classification model. For all sentence examples, I limit them up to 32 words, and if there are not exist 32 words, I am creating zero pad arrays. To convert each word to vector, I used a pre-trained word2vec model.

In the final setting, the shape of my data is :

x_train: 15000 samples and each sample has 32 vectors in which each vector size is 100. (15000, 32, 100).

y_train: 15000 binary targets (15000, 1)

So my question is that, should I apply 1D CNN on my x_train or 2D CNN? I think that I can do it in both ways, but is there the main correct approach to this kind of problem? I read some stuff about 1D CNN on text classification but also there are some examples with 2D CNN. What are the cons and pros?

The standard way would be to apply 1D convolution.

While technically there is nothing preventing you to implement a 2D convolution over your textual representations, they would be "less expressive" than normal 1D convolutions:

In a normal 1D convolution, the kernel would have depth 100 and the width you choose (e.g. 3, 5). The kernel is slid through the "time" dimension and is computed over the whole channel dimension:

In order to compute a 2D convolution over your text, the 2 dimensions would be time and channels. You could have the kernel height be 100 and therefore take the whole channel dimension, but this would be analogous to a 1D convolution, no there is no point in doing so. Therefore, for having a 2D convolution, you would have a kernel that does not cover the whole channel dimension at a time:

Therefore, while it is possible to have a 2D convolution over text, it would be less expressive that the 1D convolution kernel, which covers the whole channel dimension and can, therefore, express any computation a 2D convolution can

So the answer is that you should use a 1D convolution. This is what makes sense and what people do.

(Images were taken from here and here)


Cost function - ideas I build xgboost model for regression problem. By the default xgboost optimize $(y - y_{pred})^2$, so the RMSE will be the best eval metric to measure performance. But my task is to build the best model for evaluation metric which check if predicted value is in range $-/+10%$ of true value i.e. $y_{pred} \in [0.9*y,1.1*y]$. Do you have any idea for this or maybe MSE is still the best option?

In my experience, it is very difficult to come up with good working custom objective functions for xgboost.

Custom objectives need to be continuous and need to have a convex gradient and non zero hessian, which is often not the case for custom loss functions.

One simpler method you could use is to define a custom validation metric based on your range +/-10, that can be used in conjunction with early stopping to optimize your hyperparameters of the xgboost model. So the model will not directly optimize for this, but will select hyperparamters that will minimize your custom loss. However, I would suggest to stick with the RMSE objective for this problem.


Get elements from lists in pandas dataframe I have the following column of a data frame:

I get it by doing dataFrame['opcodes].

and I would like to consider only the first 20 and the last 20 elements of each list. Is there a way to do this?

I have tried to do the following:

dataFrame['opcodes_modified'] =  dataFrame['opcodes'].apply(lambda x: x[:20].append(x[-20:])

but i get the following error message:

SyntaxError: unexpected EOF while parsing

can somebody help me? Thank's in advance.

You logic is correct, it's just the lambda function that is slightly wrong. .append() adds a single element to the end of the list, so in your case I guess it adds an element which is a list of 20 elements. You could either use .extend() instead, which will do what you want or you can simply write

lambda x: x[:20] + x[-20:]


CUDA_ERROR_OUT_OF_MEMORY I have a large network that is somewhat similar to Wavenet. Although (it seems) that my GPU has enough memory, I get an out of memory error on fitting (see logs below).

Any idea?

How can I troubleshoot these kind of CUDA driver issues?

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.6575
pciBusID 0000:01:00.0
Total memory: 11.00GiB
Free memory: 10.71GiB

2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-12-22 23:32:05.288986: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-12-22 23:32:05.429386: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device
: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)

2017-12-22 23:32:06.131386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 10.17G (10922166272 bytes) fro
m device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:06.599386: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 9.15G (9829949440 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
2017-12-22 23:32:07.332586: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorf
low\stream_executor\cuda\cuda_driver.cc:924] failed to allocate 8.24G (8846954496 bytes) from
device: CUDA_ERROR_OUT_OF_MEMORY
built graph

Did you try to start your training with a smaller data set and that worked?

How about using generators to gradually input your data? For me that worked quite well (in Keras : model.fit_generator(...))


Site packages location not shared between Python and Jupyter Lab I appreciate the fact that Jupyter runs in an isolated mode. I read several posts about it by now.

What I don't understand is why the JUPYTER_PATH variable is ignored as well as appending manually (as a proof of concept) the path of the current site packages from my brewed Python dir.
I couldn't find any documentation specific for the Lab so I assumed this should have worked out of the box.

Any idea on how to avoid installing all the necessary packages via a !{sys.executable} -m pip install .... from within Jupyter?

Of course modifying the sys.path from within the environment is one option, but this has to be done on a per notebook basis, which is not ideal.

I'd like to install things from command line via pip and be able to use them immediately.

Thanks

A new version of Python was released, and yet again I lost all my libraries by the the time Brew dropped in the new release.

I, therefore, spent some time to look into the best options out of there, and I believe the best is to use a Python Environment management package.

I decided to stick on PyEnv.

A simple %pip -V demonstrates how the command is now run from the shim's directory, which makes me confident whatever I'm now going to install will persist in the 3.8.2 folder and will be movable at a later stage.

Guess that moving packages will be not as easy as 1-2-3, but at least I may not  necessarily have to download all of them rather than copy/move. Any more expert advice here is welcome.


Loss function when the output is a single probability I have a regression problem where the output y is a single probability, i.e. real number that varies in the interval [0, 1]

While using L1 or L2 loss will very likely work well, I feel that they are not the most appropriate options considering that the range [0, 1] is already well defined.

Is Binary Cross Entropy (BCE Loss in pytorch) the most appropriate in this case?

At first I was going to say:

  It doesn't make sense to use use cross entropy loss in a regression
  problem!

  See explanation here.

But then I realised that if you are really trying to do regression on probabilities it could have some sense.

But still, why would you use it instead of L1, L2? So maybe try it and let me know if it works better!


Combining Different Models I have N models, each of them are used to predict on a set of data.
I am currently combining their predictions by averaging across rows.
Need suggestions on combining their predictions. End goal is to classify the samples.

Part of the problem lies in how much data you have. To create a second level of complexity, you ideally want to use a holdout data set to decide the right combination for the model predictions. If you use the training data from the models themselves to also combine the model output, you risk over fitting your final model. If you have a small data set, trying to build an ensemble on top of it might lead to worse performance in deployment since the model is only memorizing the training data. In that case, a simple average might do you better than trying to anything more sophisticated.

But, assuming you have more data to use, or if you want to use the data you trained on anyways, you can either design a weighted average or create a second model.

If you have an idea of which model performs better, you might want to try manually assigning weights for each model's output. A simple average assigns an equal weight to each output, but you can experiment with shifting the weight around a bit.

For an example, given two models, a basic average yields

new_pred =  0.5 * mod_1  + 0.5 * mod_2

However, if you see the performance on model one does better overall, you might shift the average to

new_pred =  0.9 mod_1  + 0.1 mod_2

If instead you want to create a second model, you can try using any machine learning model suitable for classification.  I would probably stay away from larger models like Random Forests, but doing something like using the predictions as features into a Logistic Regression model can work well. But again, I would emphasize only trying to use the predictions from a holdout data set into the logistic regression model to make sure you aren't over fitting.


Normalize matrix in Python numpy I've an array like this:

 array([[ 0,  1],
        [ 2,  3],
        [ 4,  5],
        [ 6,  7],
        [ 8,  9],
        [10, 11],
        [12, 13],
        [14, 15]])

I want to make normalize this array between -1 and 1. I'm currently using numpy as a library.

nazz's answer doesn't work in all cases and is not a standard way of doing the scaling you try to perform (there are an infinite number of possible ways to scale to [-1,1] ). I assume you want to scale each column separately:

1) you should divide by the absolute maximum:

arr = arr - arr.mean(axis=0)
arr = arr / np.abs(arr).max(axis=0)

2) But if the maximum of one column is 0 (which happens when the column if full of zeros) you'll get an error (you can't divide by 0).

arr = arr - arr.mean(axis=0)
safe_max = np.abs(arr).max(axis=0)
safe_max[safe_max==0] = 1
arr = arr / safe_max

Still, this is not the standard way to do this. You're trying to do some "Feature Scaling" see here

Then the formula is:

import numpy as np

def scale(X, x_min, x_max):
    nom = (X-X.min(axis=0))*(x_max-x_min)
    denom = X.max(axis=0) - X.min(axis=0)
    denom[denom==0] = 1
    return x_min + nom/denom

X = np.array([
    [ 0,  1],
    [ 2,  3],
    [ 4,  5],
    [ 6,  7],
    [ 8,  9],
    [10, 11],
    [12, 13],
    [14, 15]
])
X_scaled = scale(X, -1, 1)
print(X_scaled)

Result:

[[-1.         -1.        ]
 [-0.71428571 -0.71428571]
 [-0.42857143 -0.42857143]
 [-0.14285714 -0.14285714]
 [ 0.14285714  0.14285714]
 [ 0.42857143  0.42857143]
 [ 0.71428571  0.71428571]
 [ 1.          1.        ]]

If you want to scale the entire matrix (not column wise), then remove the axis=0 and change the lines denom[denom==0] = 1 for denom = denom + (denom is 0).


Batch Normalization and Dropout together causing incorrect segmentation results So, I've been running a test to see how well a number of networks can perform road segmentation on a particular customer's dataset.  I am testing UNET, RDRCNN, and Tiramisu against each other.

UNET reliably trains, however RDRCNN and Tiramisu cannot.  Immediately during training, the validation loss and accuracy plateau.  After looking at the results, it looks as if the network is guessing a single pixel value; leading to inferences that are just a square of all one color.

I have tried turning the learning rate down, using a simple optimizer like SGD, and making sure I am loading the data with the same exact loader.  I have also tried using a dice/jaccard loss to account for class imbalance.  None of these have worked.

Has anyone ever experienced this before? Any help would be greatly appreciated.  Thank you!

The differences between UNET and the deeper networks like RDRCNN and Tiramisu is that the deeper networks have Batch Normalization layers.  They also include dropout.

I thought it might be the bias terms in the convolutional layers, but the bias for all those layers was set to 'False'.  I tried regularization, changing the momentum value of the Batch Norm layers, and removing the Batch Norm layers closest to the output.  However, the only thing that worked was removing all the Batch Norm layers entirely or removing all the dropout entirely.

It seems this is a data issue on my end.  Certain images are exacerbating a common problem with using batch normalization and dropout together.  For some, this may introduce too much noise into the network, leading to incorrect predictions.


How to change Linear model in SGDClassifier scikit learn? The SGDClassifier of scikit learn defines it as "Linear classifiers (SVM, logistic regression, etc.) with SGD training.". I understand from this that any Linear classifier can be used here. I want to know that how can we define a specific linear classifier to be used in the case of SGDClassifier.

The specific linear classifier can be defined with the loss function argument. The options are { ‘hinge’, ‘log’, ‘modified_huber’, ‘squared_hinge’, ‘perceptron’}. For example, hinge loss is equivalent to a linear SVM and log loss is equivalent to Logistic Regression.


Pandas - how to make further sub-groups after groupby( ) and get_group( ) Sample data in .csv format

| No.|   IP     |      Unix_time     |    # integer unix time
| 1  | 1.1.1.1  |     1563552000     |    # equivalent to 12:00:00 AM
| 2  | 1.1.1.1  |     1563552030     |    # equivalent to 12:00:30 AM
| 3  | 1.1.1.1  |     1563552100     |    # equivalent to 12:01:40 AM
| 4  | 1.1.1.1  |     1563552110     |    # equivalent to 12:01:50 AM
| 5  | 1.1.1.1  |     1563552150     |    # equivalent to 12:02:30 AM
| 6  | 1.2.3.10 |     1563552120     |

Here's the current working code using pandas groupby( ) and get_group( ) functions:

data = pd.read_csv(some_path, header=0)
root = data.groupby('IP')

for a in root.groups.keys():
    t = root.get_group(a)['Unix_time']
    print(a + 'has' + t.count() + 'record')

You will see the results below:

1.1.1.1 has 5 record
1.2.3.10 has 1 record

Now, I want some changes.

For the same IP value (e.g., 1.1.1.1), I want to make further sub-groups based on a maximum time interval (e.g., 60 seconds), and count the number of elements in each sub-group. For example, in above sample data:

Start from row 1: row 2 Unix_time value is within 60 seconds, but row 3 is beyond 60 seconds.

Thus, row 1-2 is a group, row 3-4 is a separate group, row 5 is a separate group. In other words, group '1.1.1.1' has 3 sub-groups now.

How to make it?

You can do that by using a combination of shift to compare the values of two consecutive rows and cumsum to produce subgroup-ids.

So the code looks like this:

# define a function that assigns subgroups
def get_time_group(ser):
    # calculate the time difference between
    # each time and the time of the previous
    # time
    # the backfill has the effect, that the first
    # row gets diff=0
    time_diff= ser - ser.shift(1).fillna(method='backfill')
    # now assign the id which is increased any time the
    # time difference between two records is greather than 60
    return (time_diff > 60).cumsum()

# group by the IP to compare the times only for the same IP
# and call the get_time_group from transform to assign the
# new group to each row (the row will be unique in combination
# with the IP
grouped= df.groupby('IP')
df['sub_group']= grouped['Unix_time'].transform(get_time_group)

After execting this code the df now looks like this:

           IP   Unix_time  sub_group
No.
1     1.1.1.1  1563552000          0
2     1.1.1.1  1563552030          0
3     1.1.1.1  1563552100          1
4     1.1.1.1  1563552110          1
5     1.1.1.1  1563552150          1
6    1.2.3.10  1563552120          0

Now you can use the sub_group as required. E.g. if you want to have the start-end of each subgroup and the number of records per subgroup, you can do:

result= df.groupby(['IP', 'sub_group']).agg({'Unix_time': ['first', 'last', 'count']})
result.columns= ['Group_start', 'Group_end', 'Num_entries']

# the result looks like this:
                    Group_start   Group_end  Num_entries
IP       sub_group
1.1.1.1  0           1563552000  1563552030            2
         1           1563552100  1563552150            3
1.2.3.10 0           1563552120  1563552120            1

The test dataframe for the output above is from your sample data, you get it by executing:

import io
import pandas as pd

raw=\
"""| No.|   IP     |      Unix_time     |    Comment
| 1  | 1.1.1.1  |     1563552000     |    # equivalent to 12:00:00 AM
| 2  | 1.1.1.1  |     1563552030     |    # equivalent to 12:00:30 AM
| 3  | 1.1.1.1  |     1563552100     |    # equivalent to 12:01:40 AM
| 4  | 1.1.1.1  |     1563552110     |    # equivalent to 12:01:50 AM
| 5  | 1.1.1.1  |     1563552150     |    # equivalent to 12:02:30 AM
| 6  | 1.2.3.10 |     1563552120     |    #"""

df= pd.read_csv(io.StringIO(raw), sep='\s*\|\s*', dtype={'Unix_time': 'Int64'}, engine='python', index_col=0, usecols=['No.', 'IP', 'Unix_time'])


Use text similarity (cosine) instead of machine learning to classify companies into industries I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry.

I took 2000 companies and assigned them into 15 industries. Then I trained a few models based on the company's description. All of them performed extremely bad.

So I wanted to get more data (indicated by the learning curves), i.e. more companies. But getting the classification right is difficult. Thus, I decided to try a simple similarity calculation.

For each industry I select words that fit best to this industry. Then I calculate the similarity between each industry and a new company and select the one that fits the best.

For the three industries I tested it worked great. But I am unsure if this will makes sense down the road. I think that the similarity approach is great because:

I can clearly define (from a business perspective) what is what
It scales nicely

Can anybody tell me if this approach makes sense? All other classification system are build using some model, so I am unsure.

EDIT:
I want to build an industry classifier that can classifier "all" existing companies. By all I mean companies that fit the NACE industries (NACE is the industry standard classification system used in the European Union)

The 2000 companies that I used are arbitrary. They do not represent all the industries from NACE. The 15 industries I ended up with are mostly based on NACE. However, I grouped some of them together. Also I made up some industries based on the most frequent keywords across certain companies (sometimes that is very obvious. Example: "dating companies" like Tinder).

The company descriptions come directly from the company (mostly parsed from their website).

I have tried to cluster the companies but I didn't find any combination of clusters that make sense based on NACE.

Yes, I have filtered stopwords (and applied other normalization such as only keeping text characters and stemming).

Regarding objectivity (thanks to @skiddles): I think this is actually another pro-argument for similarity. If I extract all the keywords that are used to describe the industries in the NACE specification I end up (it seems) with at least one recognized standard.

So, I believe, my question should be:
I have 20 industries. Each industry is described with 100 distinct words. I get a new company with a description. The words of this company description can be found in the descriptions for the industries. Should I classify the companies into industries based on similarity or use a classification model?

This is an interesting approach.  For me though, it raises a few questions that might impact the effectiveness of the approach:

Were all the descriptions created by the same organization, or performed for the same purpose? e.g. Descriptions are taken from SEC filings.
Have you tried any unsupervised approaches to grouping these companies based on their descriptions?
How did you come up with 15 industries, not 14, not 16?
Have you applied stop words and removed very infrequent words?

Not being critical, but how confident are you that someone else would apply the same label to each of the 2000 instances you labeled?  Could that be one reason why the models are performing poorly?

Ultimately, to answer your question, there is no reason why domain knowledge cannot be used to inform your key words, but it seems like you will struggle when you get to industries that are "similar but different."  The good results that you have received using this method may just be indicative that you created both the labels, and the key words for the industries, which may well be what you want to achieve.

From a defense perspective, your classifier would only represent your opinion.  In the long run, you may be better off giving up a bit of performance to gain objectivity.  Company classification is difficult because of vertical and horizontal integration, and the practice of using holding companies to manage diversified businesses.  If you can take your opinion out of the process, it may be "better".  If someone disagrees with your model, they are just disagreeing with you.  If they disagree with a more traditional ML model, they are disagreeing with the dispassionate similarity of the descriptions.

HTH


How to combine two Deep learning model weights into one Suppose I have these two models (model1 and model2) trained from same structured data, but different datasets:

# create and fit the LSTM network on dataset1
model = Sequential()
model.add(LSTM(4, input_shape=(look_back, 6)))
model.add(Dense(1))
optimizer = Adam(lr=0.001, decay=0.00001)
model.compile(loss='mean_squared_error', optimizer=optimizer)
model.summary()
model1 = model.fit(trainX, trainY, epochs=100, batch_size=64, verbose=2)
model1.save_weights("Model1.h5")

# create and fit the LSTM network on dataset2
model = Sequential()
model.add(LSTM(4, input_shape=(look_back, 6)))
model.add(Dense(1))
optimizer = Adam(lr=0.001, decay=0.00001)
model.compile(loss='mean_squared_error', optimizer=optimizer)
model.summary()
model2 = model.fit(trainX, trainY, epochs=100, batch_size=64, verbose=2)
model2.save_weights("Model2.h5")

How do I combine Model1.h5 and Model2.h5 to make suppose Model3.h5 which has all the attributes of Model1.h5 and Model2.h5 ?

Any help will be highly appreciated.

Thanks,

Merging models is not as simple as it seems.
In my opinion, I see multiple possibilities :

Don't merge models, merge datasets and retrain : this is in my opinion the most reliable solution, models are fitted from a dataset which represent a certain distribution of data and features. If you can retrain : retrain, especially if datasets are differents.
Create a model over models : use your models as features extractors (cut final part) and build a siamese network (which compute features in parallel from two submodels), merge features obtained and add top layer which will classify from these new set of features (just retrain final part, freeze your models's layers). With this approach, you can retrain a new model which will keep both models's logic without having to retrain a full network. This approach is better than the first if retrain a model from scratch is too constraining.
Play around models's weights : you can access to weights of models and create a third by taking mean of weigths's layers. In my opinion, you can try this but I highly doubt of this approach, instead take the best model and use it.


Why does Feature Importance change with each iteration of a Decision Tree Classifier? After applying PCA to reduce the number of features, I am using a DecisionTreeClassifier for a ML problem

Additionally I want to compute the feature_importances_. However, with each iteration of the DecisionTreeClassifier, the feature_importances_ change.

Iteration #1

Iteration #2

Why would it change? I thought the initial split was made on a feature to "produce the purest subsets (weighted by their size)". Acting on the same training set, why would that change?

Thanks in advance for any help.

From sklearn.tree.DecisionTreeClassifier help:

  The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data and max_features=n_features, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

Also, you might want to have a look at my critique on feature importance.


Actual problems in Data Science/Machine Learning connected with music I'm on my $4^{th}$ year now and searching for a theme for my diploma in mathematics & computer science specialization.
Is there are interesting problems or fields for researches that i can explore? Probably i'd wanted to work in fields connected to art - music reccomendations e.g. (but we have Spotify sure).
I was suggested to take theme "Authorship attribution in classical music", but i think this theme is rather explored.

You could begin your journey by picking up one of the topics present in the magenta project started by google. It's open-source so you could pick up some great ideas there.
Link to Magenta: https://magenta.tensorflow.org/
Another aspect you could go ahead with is trying to combine to genres of music and produce something new. There are also surveys in recommendation for music. Spend a day getting to know what's there and if you could take up a certain drawback from a paper and go ahead with it!
Finally, if nothing clicks, you could refer to chapter 8 from here: https://arxiv.org/abs/1709.01620
This paper is a survey on deep learning for music generation! The 8th chapter focuses on current problems and future prospects.


How to select random data for two different recommender systems? The business problem: We have two different vendors that offer personalized recommender engines and want to do A/B testing with them. The recommendation will give the user a personalized offer via a push message on the phone. During the testing period, we should give each provider a dataset with different details regarding the customers (purchase history, in-app events, etc). Each vendor will receive a dataset with identical info but from different clients.

What is the best method to choose the two datasets so that they would be similar in terms of client behaviour?

I assume that giving them random data from our database wouldn't be a rigorous method so one idea that I have in mind is applying dbScan clustering on our database and further randomly picking clients from each cluster - I don't know if this is the best approach. The full database has 200k clients and each dataset should contain 5k clients.

Example: After dbScan clustering there are k=10 clusters so I randomly pick elements from each cluster and split them into Dataset01 and Dataset02.

Any suggestions?

Welcome @Remus Raphael :) - Your approach is a sound option.

More specifically, if a density-based algorithm was already working for you, I'd recommend the HDBSCAN clustering algo, which should have a better performance and has a unique built-in cluster validation (based on the DBCV algorithm).

Then your general pipe line could be:

An optional pre-processing
An optional NLP / TFIDF for meaningful text features
An optional dimensionality reduction (I found TSNE and TruncatedSVD to
work nicely with HDBSCAN with textual processing)
HDBSCAN tuning for different params and distance metrics
Finally, when you're satisfied with the clustering quality - you can simply start with the 2 largest clusters for your AB testing

I'd love to hear you feedback on your actual data :)


Precision, recall and accuracy metrics significantly different between training/validation and actual predictions I have two sequential models built with Keras that train on data from a CSV file.
This is how they are built
# First model

model = Sequential()
model.add(Input(shape=X_1.shape))
model.add(Dense(128, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model.add(Dropout(0.7))
model.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

# Second Model

model_2 = Sequential()
model_2.add(Input(shape=X_1.shape))
model_2.add(Dense(64, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dropout(0.5))
model_2.add(Dense(32, activation='relu', kernel_initializer='he_uniform'))
model_2.add(Dense(1, activation='sigmoid'))
model_2.compile(loss='binary_crossentropy',
              optimizer=Adam(learning_rate=0.001),
              metrics=[recall, prec, ba])

My dataset is split in half, one for training/validation and one for testing, also I have about 47k rows of data across 19 columns. Most of them are categorical, just one is numerical, all of the categorical data is one hot encoded and the numerical is normalized using MinMaxScaler,
When training the model I use the built-in metrics from Keras for Recall, Precision, Accuracy and I get decent numbers of above 0.7 for these. I calculate the F1 score of my results manually.
I then test the trained model on the resting data (the second half of the dataset) and calculate the metrics using Sklearn as I wasn't able to find some built-in functions for that in Keras. I use the below code.
# predict crisp classes for test set
yhat_classes = (model.predict(X_2) > 0.5).astype("int32")

# Calculate F1 score
accuracy = accuracy_score(y_2, yhat_classes)
precision = precision_score(y_2, yhat_classes)
recall = recall_score(y_2, yhat_classes)
f1 = f1_score(y_2, yhat_classes)

On these metrics, I get considerably lower numbers across all of them. Again my validation numbers from the .fit method of the model are exactly what I need. I also use EarlyStopping on my model, watching val_recall when training.
Why are my resulting numbers from the prediction so different than training and how can I improve them?
Is there a difference between how Sklearn and Keras calculate Precision and Recall?

Your model is overfitted. It means you are like "memorizing" from training data, when you should "learn". A good model must have similar scores for Training and validation data, but it's not your case, scores are super different. To avoid that:

Use more data to train. Why you use only 50% for training? I will
recommend you use about 90% almost. Anyway, you will be keeping
around 5000 row to test your model, more than enough. With more data, more difficult to overfit. Moreover you can use techniques like data augmentation to obtain more data.
Increase complexity of the model. Adding more layers or convolutions.
You can try also to use some extra techniques like drop out or regularization.
Decrease the value of learning rate, maybe you can improve your model
with this step.

If you use a combination of these point I commented, you should see an improvement on your validations scores.


Correlate an array of categorical features to binary outcome I have a data set that looks like this:

target,items
1,[i1,i3]
1,[i4,i5,i9]
0,[i1]
...

The variable target is 0-1 outcome. The feature "items" is a set of items (variable length). Each item is a categorical variable (one of: i1, i2, .., i_N). There's no order/relationship between the items. A business example would be "set of products in a cart, outcome whether the customer abandons cart".

The size of data is approx. 1,000,000 by 5,000 (I have ~1 million examples, and N is approximately 5,000)

I want to do the following analysis. I want to find the items that influence (or lead to) target = 1. I don't have extra features to add. What is the type of statistical analysis or machine learning modelling technique that I should use?

How large is N?
Can you reshape your data into something like:

target   i1  i2  i3  i4  i5 ... i9 ... iN
     1    1   0   1   0   0 ...  0 ...  0
     1    0   0   0   1   1 ...  1 ...  0
     0    1   0   0   0   0 ...  0 ...  0

Once you have fit everything into a data frame, you can use any two-class supervised classification algorithm to build your model. There is no "best" model in general, but try a few to see which one works the best with your data.

Sorry for posting as an answer; comment doesn't allow preformatted text.


Dealing with extreme values in softmax cross entropy? I am dealing with numerical overflows and underflows with softmax and cross entropy function for multi-class classification using neural networks. Given logits, we can subtract the maximum logit for dealing with overflow but if the values of the logits are quite apart then one logit is going to be zero and others large negative numbers resulting in 100% probability for a single class and 0% for others. When loss is calculated as cross-entropy then if our NN predicts 0% probability for that class then the loss is NaN ($\infty$) which is correct theoretically since the surprise and the adjustment needed to make the network adapt is theoretically infinite.

I know this can be dealt with normalizing the data and choosing weights and biases from standard normal distribution but this is a what-if scenario where the data is mean preprocessed but not standard deviation processed, I believe this can also occur even after preprocessing both mean and stddev.

Where exactly in the computations are these underflows manifesting? See here for a brief explanation around the extremes of the softmax.

Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.

In any case, the softmax shouldn't have problems with your input, as the final activations are exponentiated:

$$
\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}}
$$

This means all values will now be in the range [0, 1].

The cross-entropy equation should also be able to deal with the output of this.

If none of this helps your specific issue, could you share a specific example of your problem?


What is the difference between nDCG and rank correlation methods? When do we use one or the other?

My use case:

I want to evaluate a linear space to see how good retrieval results are. I have a set of data X (m x n) and some weights W (m x 1). I want to measure the nearest neighbour retrieval performance on W'X with a ground truth value Y. This is a continuous value, so I can't use simple precision/recall.

If I use rank correlation, I will find the correlation between retrieved Ys and retrieval rank. If I use nDCG, I will use sorted Y to compute IDCG.

I would like to compare this to the correlation value I get when I change Y also. (For example, Y could be head pose angle in one case and age in another)

A central problem in ranking is to design a measure for evaluation of ranking functions. In this paper we study, from a theoretical perspective, the Normalized Discounted Cumulative
Gain (NDCG) which is a family of ranking measures widely used in practice. Although
there are extensive empirical studies of the NDCG family, little is known about its theoretical properties

NDCG has two advantages compared to many other measures. First, NDCG allows each retrieved document has graded relevance while most traditional ranking measures only allow binary relevance. ... Second, NDCG involves a discount function over the rank while many other measures uniformly weight all positions.

More: Info with Theoretical & Mathematical


Machine learning classification with time-domain signals how to ignore signal arrival time? I am training a Tensorflow classifier model with signal data (converting signals to the spectrograms).
I want the model to be insensitive to the arrival time of the signal within the fixed window.
Moreover the model should classify both of these signals as the same.

Ideas

Train the model using the same signal at different arrival times
Preprocess the signal before providing it to the model

I want to avoid 1. because it will mean I have to generate a huge amount of signal data which will slow down training.
Are there any commonly applied preprocessing steps for this problem?

A common approach is what you suggest in 1. - apply time-shift as a Data Augmentation strategy. The augmentation is generally beneficial with deep learning models, and GPUs are fast so the compute time is rarely a big problem.
Another strategy, less common, would be to make sure that the event is always located at the same position inside the analysis window during training. At prediction time one uses a high degree of overlap between analysis windows, to ensure that the model has a good chance of seeing the signal in the correct position.
If you have high quality labels, aligned time-stamps for each event, this is rather straight forward.
A third approach would be to use a combination of a detector model and a classification model. The detector job is to take the input signal stream, and detect the presence of a possible event - then possible events are passed to the classifier. A well designed detector would have a rather predictable trigger time, so the classifier does not have to be time-shift invariant. This is easy enough for signals that have a very clear event vs not-event signatures, but hard for more complex signals or low SNR conditions.


Recommend System AB test metric events I build personal recomendation system for choosing games. In website on main page on special place there is collection of personal games recomendation. And after AB test(between 2 recommend system)  I don't understand, what events I should collect. Only events after click on recomend icon or all events(recommend events plus events without choosing recommend game-user can choose game on other places such as finder)?
For example, one of the metric is sum payment per user per game. Should I collect payments only on recommend games, which was clicked, or all payments? As I understand, in theory, recomend system can give negative effect for site(user see advises and not to click for them).  And If I collect only recommend payments, I can skip watching negative effect. On other hand, I build  recommend system, and want to see metrics only for players, who click on advice icon.
What way is more correct?

This all comes down to your objectives that come to mind which you could use:

We care about the entire website.  The new recommender system will boost total payments either directly or indirectly.  We do not want the recommender to hurt total payments.
We care only about the recommender, perhaps we believe
this is the future of our business and we want to eventually replace
the entire website.  It's critical the recommender directly drive payments.

Either way, I would recommend tracking recommender-associated payments and non-associated payments.  It's important to understand direct and ecosystem effects.  It's possible the recommender is exceptionally successful at driving clicks and finding users great games, but also finding them games which do not monetize successfully.  It's also possible the recommender is so awful that it scares people away from the website.  You will want to measure and track both direct and indirect effects, but how you make your decision will depend on the objective you are trying to achieve.


checking model stability - Performance for different class I tried to do multi-class classification problem. The goal is to predict whether the match will be won by HomeTeam, AwayTeam or Draw. I did feature engineering from the attributes and finally came up with final data to train a classifier. I make sure that the data is balanced for all the 3 class.

To train a classifier I did XGB Classifier, Logistic Regression, SGD Classifier and Normal DNN(Tensorflow Estimator). I checked the metrics for all the classifiers and I am picking out the best one from the classifier.

Linear SGD Classifier Performance on Validation Set

     Class, Precision, Recall,    spe,       f1,      geo,      iba,      sup

      A       0.58      0.69      0.79      0.63      0.74      0.54       275
      D       0.51      0.61      0.66      0.55      0.63      0.40       338
      H       0.81      0.50      0.94      0.62      0.69      0.45       315
   Avg/mean   0.63      0.60      0.79      0.60      0.68      0.46       928

Model Performance  for Test Dataset

              pre       rec       spe        f1       geo       iba       sup

      A       0.87      0.55      0.97      0.67      0.73      0.51        84
      D       0.43      0.69      0.66      0.53      0.67      0.45        83
      H       0.80      0.69      0.86      0.74      0.77      0.58       139

We can see this model is stable over the class A and H but the precision is so poor for class D. I think because of a lack of feature the model is not performing well for class D. Though, I did several EDA and Feature Engineering to increase the recall for class D.

My question is, Is this model is considered stable?

A consideration: I dont think A is stable since it has a huge difference between validation and test results.

Some questions before answer:

Are you using cross validating with how many folds? The best result are the mean between folds results? What is the stddev? If stddev is big, it can tell us a history.
Your folds are shuffled? I see lot of people folding when using scikit learn without shuffle since its not the default behavior.
Is your test data unbalanced? if so, its normal the difference between test and validation results.
How many examples are you using for test and how many fot test? Sometimes the stability of the model requires more examples.


In Keras, how to get 3D input and 3D output for LSTM layers In my original setting, I got

X1 = (1200,40,1)
y1 = (1200,10)

Then, I work perfectly with my codes:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 1), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense(10)))

Now, I further got another time series data same sizes as X1 and y1. i.e.,

X2 = (1200,40,1)
y2 = (1200,40)

Now, I stack X1, X2 and y1, y2 as 3D arrays:

X_stack = (1200,40,2)
y_stack = (1200,40,2)

Then, I try to modify my keras code like:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 2), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense((10,2))))

I want my code work directly with the 3D arrays X_stack and y_stack without reshaping them as 2D arrays. Would you give me a hand on how to modify the settings? Thank you.

Add another dimension to represent the number of time steps.

From Keras documentation on recurrent layers:

  if return_sequences: 3D tensor with shape  (batch_size, timesteps,
  units).

Something like this:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 10, 3), return_sequences=True))


Alternative distance to Dynamic Time Warping I am performing a comparison among time series by using Dynamic Time Warping (DTW). However, it is not a real distance, but a distance-like quantity, since it doesn't assure the triangle inequality to hold.

Reminder:d:MxM->R is a distance if for all x,y in M:

1 - d(x,y) ≥ 0, and d(x,y) = 0 if and only if x = y
2 - It is symmetric: d(x,y) = d(y,x)
3 - Triangle inequality: d(x,z) ≤ d(x,y) + d(y,z)

There is any equivalent measure that ensures the condition of distance in a matemathical sense? Obviously, I am not looking for a Euclidean distance, but one that ensures the proper classification of my series in a future clustering.
If so, there is any solid implementation in a R or Python package?

Like suggested in one answer on this SO question, you could use elastic matching with Levenshtein distance to your task. Levenshtein distance obeys triangle inequality and is therefore a metric distance.

Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data.

There is an implementation of elastic matching and Levenshtein distance calculation in Python.

To put them together you most probably need to build your own implementation.


Abbreviation in Orange's contingency table What do the scores ARI and AMI mean in Orange's contingency table?

ARI stands for adjusted Rand index and AMI stands for adjusted mutual information.

They are metrics for clustering.

Remark:

ARI might take negative values.
The AMI takes a value of 1 when the two partitions are identical and 0 when the MI between two partitions equals the value expected due to chance alone.


What is "position" in CNN (im2latex) for Positional Encoding? I'm trying to build a model that maps images of math
formulas into LaTeX markup.
I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder architecture that solves this problem.
In encoder after the work of CNN it's proposed (page 3) to add following signals to generated feature maps "in order to retain the spatial locality information":
$$PE(x,y,2i)=sin(x/10000^{4i/D})$$
$$PE(x,y,2i+1)=cos(x/10000^{4i/D})$$
$$PE(x,y,2j+D/2)=sin(y/10000^{4j/D})$$
$$PE(x,y,2j+1+D/2)=cos(y/10000^{4j/D})$$
Here the authors refer to positional encoding part of Transformer model "tailoring the 1-D positional encoding technique to 2-D".
In original Transformer model (from article "Attention Is All You Need") the position encoding is calculated like this:

Where "pos" is just an index number of word in given sentence (correct me if I'm wrong).
And here comes the question: what is "x" and "y" in first formula?

In the original transformer model, one input 'pos' was required as input was 1 dimensional (words). With encoder being CNN, the input feature vector will be 2D, so x and y are just the indexes of an 2D feature vector. By using those indexes, PE functions can produce those positional encoder images that you can find in the paper.


Python Script using pandas to plot histograms between the features I am working around with data from Kaggle : Titanic Competition dataset.

Script

#cols : List of all column header
cols = list(df.columns.values)
#script to plot histograms of all columns with respect to column: "Survived"
for val in cols:
       df.groupby('Survived')[val].hist(alpha = 0.2)

Code of importance (COI)

df.groupby('Survived')['Age'].hist(alpha = 0.3)

Output

The COI works but the Script is not working. The script will help to plot all the histograms in one go without manually inserting the column in the COI.

The script is running indefinetly.
The COI also runs indefinetly but once I execute the following code:

df.groupby('Survived').Age.hist(alpha = 0.2) #This runs flawlessly

COI runs perfectly. (Not able to figure out why).

The scripts are written in Kaggel Kernel.

Given the above details, please help me figure out a way to achieve the above script either by making improvements to the Script or an alternate way of doing the same.

cols = list(df.columns.values)
for val in cols:
        df.groupby('Survived')[val].hist(alpha = 0.2)
        plt.show()

just add plt.show() in the end it would create histogram for all columns


Correlation Matrix for non-numeric features Currently, I have dataset with numeric as well non-numeric attributes. I am trying to remove the redundant features  in the dataset using R Programming Languages. Note: Non-numeric attributes cannot be turned into binary.

The Caret R package provides the findCorrelation which will analyze a correlation matrix of your data’s attributes report on attributes that can be removed. However, It only works numeric values of 'x'. I have been unable to find a package which does it for non-numeric attributes.

Is there a function in Caret R Package that does that for non-numeric attributes as well? If not, any method/package that would help me achieve the same?

There are measures of association for categorical variables. If you are looking at two ordinal variables you may use Spearman's correlation coefficient. There are also many measures for association for purely categorical variables, such as gender and race. Yule's Q and Crammer's v are popular choices.


Difference between Gradient Descent and Normal Equation in Linear Regression Hi I am new to Linear Regression.
I want to know

  what is the difference b/w Gradient Descent and Mean Square Error in
  Linear Regression using machine learning?

And

  When to use Gradient Descent and Mean Square Error in Linear
  Regression using machine learning?
  Or

  When to use which algorithm in Linear Regression.?

Can anyone explain.?

To train a model, two processes have to be followed. From the predicted output, the error has to be calculated w.r.t the real output. Once the error is calculated, the weights of the model has to be changed accordingly.

Mean square error is a way of calculating the error. Depending upon the type of output, the error calculation differs. There are absolute errors, cross-entropy errors, etc. The cost function and error function are almost the same.

Gradient descent is an optimization algorithm or simply update rule, used to change the weight values. Some of the variations are Stochastic gradient descent, momentum, AdaGrad, AdaDelta, RMSprop, etc. More about Optimization algorithms


VlookUp: how to loop it? I am not an experienced user of Excel because I usually use python but I am obliged to use it this time.

This is the big picture of my case.

The problem is:

I have a sheet with a column with some words, another column with the translation of these words in a non standard written way because they are written using numbers (such as "sale" will be "5413") and they are aligned with the corresponding standard written words (i.e. "sale").

In another sheet I have the same non standard written words (first column) that are sorted from 0 to 9 and there are also some numerical values associated to them.

My goal is:
take the values of the second sheet and put them in the order of the first sheet so that I can have the values of the non standard written words in the same order of the standard written words.

Someone suggested me to use the VlookUp function.

=VLOOKUP($D2;$'leet_nopunct (2)'.$2:$1000;**97**;0)

It works! But I have to change the second last number (here 97) every time for each column of the second sheet and it is a very long process.

I know that Excel can use some macro: is there a way to loop this number in order to be automatically incremented it?

If you use a combination of INDEX and MATCH, you can let the desired look up array vary automatically as you drag the function across your columns, which is an alternative to manually fixing the look up column in VLOOKUP.

If you wish to continue with VLOOKUP, you could potentially add column numbers to the top of your data table, and reference these numbers instead of the hard-coded 97.


Assigning new items to existing similarity based clustering Given some clusters created from similarity measures between items, is there a recommended way to assign a new item to an existing cluster based on similarity alone? (i.e. avoiding re-clustering)

Measuring the similarity of a new item to all other items is fairly cheap, so I'm looking for a way of using this to assign it to the cluster it's most likely to belong to. It's also important for it to take cluster size into account (i.e. doesn't unfairly weight towards or against larger clusters).

Basically, I'm trying to sacrifice some clustering accuracy in exchange for avoiding a complete re-clustering when the occasional new item is added.

I suggest you think about this in terms of "data set" and "training set" (technically, it is also recommended to have a separate test set). Once you have your clusters defined on the training set, your can start using them to classify any amount of new data without recalculating, by simply measuring similarity to cluster centroids, for example.

(This doesn't prevent you from deciding to enlarge your training set and data set later, just try to not do that selectively to avoiding overfitting.)


How do we pass data to a RNN? Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:

Scheme 1

Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss)words from each article (for the sake of simplicity let us assume batch size = m)
Set the initial hidden state H0 = $[0,0,..,0]$
Calculate loss and gradient on this batch and update the parameters
Move a word forward and get next S words from each article (hence S2, S3, ... , Ss are the same words as in the previous batch)
Use H1 calculated from previous step and use it as H0 in this iteration
Do this to the end

*In this scheme we would have to use zero padding on the last batch (at the end of articles)

Scheme 2
Same as Scheme 1 but in step 5 we reinitialize H0 to a vector of Zeroes

Scheme 3
Same as scheme 1 but in step 4 we move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration

Scheme 4, 5, 6
Same as Scheme 1, 2, 3 but instead of taking s consecutive words we take first sentence from each article and zero pad them to the length S

What is the right way to go through the data in feeding it to a RNN, Please give reasons as well. I think it should be Scheme 1 but it could be really slow

Based on how much i got to know i propose the correct scheme would be third one i.e.

Take the first batch of data as first S (Number of time steps) (S1, S2, .., Ss) words from each article (for the sake of simplicity let us assume batch size = m)
Set the initial hidden state H0 = [0,0,..,0]
Calculate loss and gradient on this batch and update the parameters
We move s words forward to next non overlapping s words in each article and initialize H0 to Hs from last iteration
Do this to the end

Why we wont take any but Scheme 1 and Scheme 3 is because:
H0 should only be initialized with a vector of zero when there is no context available (if you have some additional information, why not use it?!). The objective is to maximize the probability of each article and the sentences are not independent. Also, there is no reason to not treat period "." as a step in itself. We do calculate its word embedding too

Why not Scheme 1 - Because of catastrophic interference ( As the network in Scheme 1 fed in same input again and again ) combined with slow learning it would result in


How to convert photo to a vector drawing I am using the following code to convert photo to a drawing:
import cv2
import numpy as np

img = cv2.imread("adventure.jpeg")

# 1) Edges
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray = cv2.medianBlur(gray, 5)
edges = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 9, 9)

# 2) Color
color = cv2.bilateralFilter(img, 9, 300, 300)

# 3) Cartoon
cartoon = cv2.bitwise_and(color, color, mask=edges)

cv2.imshow("Image", img)
cv2.imshow("Cartoon", cartoon)
cv2.imshow("color", color)
cv2.imshow("edges", edges)
cv2.waitKey(0)
cv2.destroyAllWindows()

the result is as follows:

However I do not want the above result, what I want is this:

How do I do it using opencv / deep learning ?

This problem is known as "style transfer", and deep learning can achieve some pretty amazing results.

Here's a fantastic GitHub repo with an easy-to-use implementation of neural style transfer (usage instructions).  I think you'll have better luck using a pre-built model like this rather than training your own.

If you want to train your own, this repo might be a good starting point.  It has a training script included, you'll just need a bunch of training images and a style image.


Random Forest - Explanation Parameter I got some question about the "standard" parameter from a random forest. Following I write my understanding about these parameters. I would be glad if I could confirm my understanding or correct it. :)
For your information, I'm using scikit-learn.

max_depth: Thats the depth of a tree. In case of execution-performance it is good to set a limit. And for the model-performance it is also good, because of overfitting.

max_features: With this parameter I specifie der number of features with which tree is build.
My question: If the parameter is set (for example 0.75), each tree includes 75% of all features, but every tree is build with different 75% feature?

min_samples_leaf: The minimal number of samples per leaf. Is this an important feature and why?

n_estimators: The number of trees. Is there a good default value for this parameter? Here are usually more trees better, right?

Are these the most important parameters or have I forgotten some?

You need to be very careful with your assumptions about "important" parameters. All the parameters have their function, that is precisely why they are implemented on the first place. If with importance you mean potential to affect your outputs/efficiency, perhaps there are some more relevant than others but most likely this relevance will also depend on the nature of your task and you data.

Having said that, I recommend you navigate through the scikit-learn documentation which is usually helpful and it contains a description including all these parameters:

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Moreover, they usually include nice and simple usage examples that can be helpful: here and here.

It is important that you practice and practice and you will start to get a feeling about the importance of each parameter and its relevance to the task at hand.

Finally, there are hundreds of blogs with nice tutorials that will give you a little bit more of insights about parameter tuning and "importance" ;)

You can try this once for example.

Good luck!


Model confusing classes I'm training a bidirectional lstm with attention on a dataset with text data and six target classes.

The F1 score on the test set by class is about 0.7 for four of the classes, and about 0.35 for the other two. Not, only that, the model is confusing those two classes and I can't find the pattern in the data by which it's making the prediction for them.

I'm using
fasttext
pretrained word vectors.

I couldn't find a proper solution for such a problem.

So, my question is: Is there a research on this topic that you could guide me to, or a solution that I could use?

One option is to print the confusion matrix to see the rate of misclassification by class.

Another option is to read through the misclassified documents to see if a pattern is apparent.


Performance of Triplet loss network vs multiclass classification I am training a triplet loss based classification network and a normal multiclass classification network on some image data. In my case, triplet loss network performs poor than multiclass network. I have tried changing layers, neurons, margin etc for triplet loss network but multiclass network performs better. Are there any cases where triplet loss network can perform worse than normal multiclass classification. If no...what are the possible things I can  look at improve or change in triplet loss network for it to work.

It is 5 class classification with some given features

For me triplet loss function (as mentioned by Neil Slater as well), is used for object recognition i.e. identify the similar object. Face-recognition is one of the use case.

  This function is based upon Siamese Network, which will provide
  us the feature vector as an output. During recognition, we compare the
  feature-vector of the new image with the feature-vector of the
  training data. Select the one which has closer matching.

In order to improve Triplet-loss function, you can try different similarity approaches or different threshold value's. Please do share the output.

But again for me it's not comparable with object-classification use case.


AWS EC2 Segmentation Fault (core dumped) issue with fast-style-transfer I'm trying to utilize AWS EC2 p2.xlarge instance to convert images using style-transfer code given in this git repo:https://github.com/lengstrom/fast-style-transfer.git, yet when the input file becomes large, I keep running into this error:

2018-09-12 02:55:25.797741: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports
instructions that this TensorFlow binary was not compiled to use:
SSE4.1 SSE4.2 AVX AVX2 FMA
2018-09-12 02:55:25.797880: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread
pool with default inter op setting: 2. Tune using
inter_op_parallelism_threads for best performance.
Segmentation fault (core dumped)

This happens when the file size of the input is bigger than 130kb...yet works fine when the input is smaller than a certain size around 130kb. This defeats the purpose of using AWS, as any file of that small size would convert quite quickly on my computer too.

If anyone wants the files that I've used for testing, I could also upload it on the question.

I had similar issue.Just use tensorflow-gpu=1.5.0.

Follow these steps:

$ pip uninstall tensorflow-gpu
$ pip install tensorflow==1.5.0
$ pip install numpy==1.14.0
$ pip install six==1.10.0
$ pip install joblib==0.12

Hope this helps!


How does real world machine learning production systems run? Dear Machine Learning/AI Community,

I am just a budding and aspiring Machine Learner who has worked on open online data sets and some POC's built locally for my project. I have built some models and converted into pickle objects in order to avoid re-training.

And this question always puzzles me. How does a real production system work for ML algorithms?

Say, I have trained my ML algorithm with some millions of data and I want to move it to production system or host it on a server. In real world, do they convert into pickle objects? If so, it would be huge pickled file, isn't. The ones I trained locally and converted for 50000 rows data itself took 300 Mb space on disk for that pickled object. I don't think so this is right approach.

So how does it work in order to avoid my ML algorithm to re-train and start predicting on incoming data? And how do we actually make ML algorithm as a continuous online learner. For example, I built a image classifier, and start predicting the incoming images. But I want to again train algorithm by adding the incoming online images to my previously trained data sets. May be not for every data, but daily once I want to combine all received data for that day and re-train with newly 100 images which my previously trained classifier predicted with actual value. And this approach shouldn't effect my previously trained algorithm to stop predicting incoming data as this re-training may take time based on computational resources and data.

I have Googled and read many articles, but couldn't find or understand to my above question. And this is puzzling me every day. Do manual intervention is needed for production systems as well? or any automated approach is there for it?

Any leads or answers to above questions would be highly helpful and appreciated. Please let me know if my questions doesn't make sense or not understandable.

This is not a project centric I am looking for. Just a generic case of real world production ML systems example.

Thank you in advance!

Yes, it is typical to have some persistent representation of the model that is uploaded, and yes, it is typically very big as a file/files. Using pickle is one way to do it, commonly used with scikit-learn, for example. Deep learning frameworks typically have their own formats, but nothing stops you from using pickle with them as well, except that it is more complicated and less efficient as an approach.

I'm not sure I understand the second part of the question, but if you want to modify your model online, nothing stops you from creating a new pickle. It is advisable that you run a new training as a separate batch process in the background, to avoid blocking your site or web service, especially if your web application server is single-threaded. Furthermore, you need to be wary of utilisation of server resources, so you would be better off running batch updates like that in low-traffic periods. This could be on over the weekend, or if your user base is predominantly regional, at night or on public holidays.