Match an image from a set of images : Combine traditional Computer vision + Deep Learning/CNN In the application I am developing, I have about 5000 product label images.(One label per product).

One functionality of my application is that user can take a picture using his camera and get a possible match(es) against the product labels registered the system.

Since initially, my system only has one sample per product, I decided to go with traditional Computer Vision techniques. I managed to implement this using Feature extraction and Descriptor matching.(using OpenCV SIFT and FLANN techniques referring this: https://github.com/kipr/opencv/blob/master/samples/cpp/matching_to_many_images.cpp)

Now I am thinking how to improve the accuracy by combining with CNN or Deep Learning techniques since when users approve matches, it gradually add more label samples for a product.

Is it possible to build a hybrid image matching system combining Computer Vision techniques and CNN/Deep Learning techniques?

Are there any similar services already available as services?

In my opinion, if you want a hybrid of Convolutional Neural Networks and the classic feature extraction techniques that would be redundant. Mainly because the architecture of a Convolutional Neural Network is composed of convolutions. I won't go much into detail of the whole architecture but these convolutions actually extract the good features for you and then those convolutions are connected to a classic Neural Network that does the classification task. Hence, extracting features using SIFT and using Convolutional Neural Networks would be redundant. In addition, the features that CNN will extract are better as compared to simple SIFT features. Though if you want to push through with combining SIFT and Deep Learning, you can instead substitute convolutions and use SIFT for feature extraction and then feed it into a Neural Network. That would work as well.


After running K-means on 12 features, I get an array containing clusters for each row. What is the next step after this? So I used the elbow method to identify the optimal number of clusters, i.e. 4 in this case. After running K-means on dataset with 12 features, I get an array with the cluster number each row belongs to, like this:

Feature1    Feature2  .... Feature12   Clusters
<data>      <data>         <data>      2
<data>      <data>         <data>      0
<data>      <data>         <data>      0
<data>      <data>         <data>      3
<data>      <data>         <data>      1     and so on.

What should be the next step now? Should I individually check the differences between each cluster to identify the similarities and differences? If yes, how exactly?

You can run your K-Means for different values of K (e.g. k=2, 3, ..., N) and for each iteration you can compute the Silhouette metric, which provides a measurement on "how compact" your clusters are. You can plot your Silhouette score and find what is the K that makes the score minimum, or stable. Since this is a clustering operation and K-Means is a simple method (nothing bad about that), I would recommend using a second clustering method (e.g. t-SNE, DBSCAN) and apply a similar procedure to see whether there is some agreement between the models. This is a general framework in data science, just applied to a clustering problem.


K-Prototype for anomaly detection I have logs of the form (e.g. from a gym login).. the representational case is so:
UserName, Login time, timeSpend_on_weights, time_spent_on_elliptical
Ava, 5jan 12pm, 10 mins, 20 mins,
Bob, 5jan 2pm, 30 min, 20 mins,
Cecila, 6jan 10am, 40min, 0 mins
...

Now I've converted the above time column to HourOfDay and day of month to get:
UserName, DOM, HOD, #weights, #elliptical
Ava, 5, 12, 10, 20
Bob, 5, 14, 30, 20
Cecilia, 6, 10, 40, 0
..

I treat the first 3 columns as categorical data and the last two as numerical, and I run K-Prototypes with N=2 (anomalous or non-anomalous). The final predictions I get can be filtered on each user to find anomalies specific to the username. The anomalous cluster is the one with lesser elements.
However, for some of the users, the cluster partitions on the Login time (HOD/DOM).. E.g. everything before 12am is one cluster and everything after 12am is another one. That doesn't convey any information.
What is the best way to handle these scenarios?
Is there a better way to do anomaly prediction on these kinds of dataset?
Update:
Type of anomalies I'm looking for:

Ava did 20 mins in elliptical, that she never used before. This individually can be done simply by using some form of outlier analysis, or K-means (on dataset filtered by 'Ava')
Ava did elliptical on Monday morning (Samething as above but filtered on Ava & time of the day).

Individually I can create models for each dataset with reasonable success, but how do I create one model that handles both of them.
If I use an actual clustering algorithm like DBSCAN/HDBSCAN, how do I not have it partition on the time? (or some other categorical variable)

The problem is that you want to do supervised learning without having labels. You have some target in mind, but it might not be how the underlying data is split. Without any information on what is anomalous or not, your clustering algorithm won't try to work out that information. It will give you groups of users that have different behaviors (people that prefer weight vs. elliptical), but not necessarily the splitting that interest you.
First you need to think about and define what would be an anomaly for that problem : an unbalanced value ? an extreme value ? an impossible one ? An anomalous session vs. an anomalous user ? It's not clear from your question and generally speaking, it might help you work trough your problem, eventually by labelling your data.
To solve your ML problem, after working on your dataset (see below), you might consider different approaches :

If you want to continue on the unsupervised path : augment the N number, it might provide you more clusters of which some of them might correspond to what is anomalous to you. Then you might consider techniques like hdbscan that find outliers outside of the mains clusters.

Label the data then consider some fully supervised technique (a small random forest might be enough). You might need to define exactly what is anomalous to you (someone that come very often ? someone that badges multiple times thus giving 0 minutes sessions ? someone that stay the whole day ?).

Label the data for those you consider some sort of anomaly and use a semi supervised algorithm, like self organised map, that will help you retrieve those anomaly you missed in your labelling process.

Generally speaking it seems you also need to work on your dataset beforehand:

It seems you only have too few predictors. As is, it is unlikely your algorithm find meaningfull anomalies, that you couldn't find / work out without ML. There is two path : try do to some explanatory data analysis (frequency of training, distribution of time spent), that might help you understand what are your anomalies and design a non-ML solution. Or try to get more variables, that will help you detect anomalies and give them meaning.

You consider each entrance separately, based on the user name. First it may cause some problems if two user have the same identification. You might want to consider a unique identifier for user identification. Then, if you want to identify anomalies about user, not sessions, you might want to group entrance by user. That would require you to build other features, like frequency, average time spent... etc.


Keras - no prediction probability for multiple output models? I have built the following model:

def create_model(conv_kernels = 32, dense_nodes = 512):

    model_input=Input(shape=(img_channels, img_rows, img_cols))
    x = Convolution2D(conv_kernels, (3, 3), padding ='same', kernel_initializer='he_normal')(model_input)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Convolution2D(conv_kernels, (3, 3), kernel_initializer='he_normal')(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Dropout(0.25)(x)
    x = Flatten()(x)

    conv_out = (Dense(dense_nodes, activation='relu', kernel_constraint=maxnorm(3)))(x)

    x1 = Dense(nb_classes, activation='softmax')(conv_out)
    x2 = Dense(nb_classes, activation='softmax')(conv_out)
    x3 = Dense(nb_classes, activation='softmax')(conv_out)
    x4 = Dense(nb_classes, activation='softmax')(conv_out)

    lst = [x1, x2, x3, x4]

    model = Model(inputs=model_input, outputs=lst)
    sgd = SGD(lr=lrate, momentum=0.9, decay=lrate/nb_epoch, nesterov=False)
    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

    return model

When I do a prediction:

model.predict(X_test)

it works properly. However, when I want to get prediction probability like this:

model.predict_proba(X_test)

my model has no predict_proba function. Why not? Is it because of the multiple-output nature of the model?

As you can see here Keras models contain predict method but they do not have the method predict_proba() you have specified and they actually do not need it. The reason is that predict method itself returns the probability of membership of the input to each class. If the last layer is softmax then the probability which is used would be mutually exclusive membership. If all of the neurons in the last layer are sigmoid, it means that the results may have different labels, e.g. existence of dog and cat in an image. For more information refer here.

For more information as stated here in the recent version of keras, predict and predict_proba are the same i.e. both give probabilities. To get the class labels use predict_classes. The documentation is not updated. (adapted from Avijit Dasgupta's comment)


Image segmentation - handcrafted features vs DNN? Currently working on a project that requires multi-class image segmentation to identify distinct anomaly types in various sheet metals.

Have had moderate success with various NN segmentation architectures, however inference performance is far from ideal given the native resolution of the images (even troublesome with extensive down-sampling).

Have also attempted to hand-craft several features based on filter banks of various convolutional kernels and then subsequently applying clustering algorithms to build a sort of 'anomaly database'. I can readily apply this defined filter bank to extract features and compared to the features to those in the labeled database.

Shown in the above image, the center horizontal line needs to be identified as an anomaly. Intuition would lead me to believe simple shadow filters would be able to detect its presence, however given the inherent inconsistencies in the sheet metal - it doesn't appear to be simple (despite human identification being very simple).

I'm not necessarily looking for a perfect solution to solve this, however if anyone was able to provide potential avenues for investigation. I want to exhaust all possible (promising) options of segmentation methods.

You have an object vs background task, i.e. two classes and instance (or m.b. class segmentation problem). If you have a labelled dataset, you can finetune one of the publicly available networks (e.g. FCN, MaskRCNN) to it to produce an image mask (i.e. an array size of the input image where every pixel is labelled as either object or background). So yeah, in 2018 feature engineering can't beat a (fully convolutional, 140M parameter, 16-layer) ConvNN.


Can I use survival analysis model to predict probability of an item sold I am building a model to calculate probability of items being sold( at least within a reasonable amount of time).
I know when the item hit the market and when/if the item sold.
What is the best approach here?
Nearly 70% of items in the dataset were sold and the best classification model I have come up with so far only has 72% accuracy.
Can I turn this into a survival analysis problem and output a probability of items being sold?

You can. You should be able to create a survival function for each type of item to give a breakdown of the probability of it being sold by that point in it's lifecycle.
An issue you might run into is that the interpretation of the survival function is not the same as your outputted probability in the classification problem. In the classification problem you are given a window: "I predict that this item will be sold within X days" (determined by your "reasonable amount of time" window). The survival analysis curve however will only tell you the probability of your item being sold/not-sold at this point in time relative to it hitting the shelves. If someone asks you "when will this item be sold?" or "will this item be sold this week" you might have an easier time answering this question via the classification framework in comparison to the survival analysis approach.
In my experience: if you're interested in a predictive model the classification framework has more direct business application (depending on your selected window size). If you are simply looking to describe the lifecycle of your products & communicate a point regarding shelf life to some target audience you can probably get by with survival analysis.


Why do we need to add START  + END  symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models? In the Sequence-to-Sequence models, we often see that the START (e.g. <s>) and END (e.g. </s>) symbols are added to the inputs and outputs before training the model and before inference/decoding unseen data.

E.g. http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

SOS_token = 0
EOS_token = 1

class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

Is there a technical definition or academic explanation of why that is necessary?
Or does that need to add the END symbol only applies to Natural Language Processing task where the sentence generation needs to end?
But what does the START symbol do? Other than to take the initial state where the trained network will start inferrring.

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the embedding, and we don't want the encoder to have to guess as to whether or not the input sentence is complete or not.

The EOS token is important for the decoder as well: the explicit "end" token allows the decoder to emit arbitrary-length sequences. The decoder will tell us when it's done emitting tokens: without an "end" token, we would have no idea when the decoder is done talking to us and continuing to emit tokens will produce gibberish.

The start-of-sequence (SOS) token is more important for the decoder: the decoder will progress by taking the tokens it emits as inputs (along with the embedding and hidden state, or using the embedding to initialize the hidden state), so before it has emitted anything it needs a token of some kind to start with. Hence, the SOS token.

Additionally, if we're using a bidirectional RNN for the encoder, we're definitely going to want to use both SOS and EOS tokens since the SOS token will signal to the reversed-input layer when the input is complete (otherwise, how would it know?).


Paramaeter estimation in noisy conditions with Machine Learning, possible? Let's take two constants, $\alpha$ and $\beta$, both are given by two functions $f_1(\vec{\theta})$ and $f_2(\vec\theta$) (the model). These functions are known: we have an analytical closed expression, that for a given set of values $\vec\theta$, $\alpha$ and $\beta$ are retrieved.

PROBLEM:

The components of $\vec\theta$ are noisy measurements, so, the variance of $\alpha$ and $\beta$ is high: for different measurements, we have quite different values for them... though our model defines them as constants. Furthermore, the model is not actually realistic, and obviously it imposes some assumptions.

formulation:

The observed values are $\vec\theta = [\beta, \dot\psi, \delta, A, B, C]$

\begin{equation}
\alpha = \frac{\dot\beta + \dot\psi}{-2A\beta + B\dot\psi A\delta}
\end{equation}
\begin{equation}
\beta = \frac{\alpha K\beta + \alpha C\dot\psi + \alpha D\delta}{\ddot\psi}
\end{equation}

(The derivatives are computed using finite differences)

NOTE: the observed values of $\vec\theta$ are not independent since they come from the same sensing unit (which little do we know about), so I think assuming statistical independence won't work here...

QUESTIONS:

Are there common approaches in Machine Learning to mitigate the effect of the noise? Can Machine Learning help to get the true values? Can Machine Learning model the variance of the estimators? By Machine Learning I mean algorithms that are not as known as Kalman filtering.

MOTIVATION AND CONSIDERATIONS:

I was asked this question by a colleague at my job office, since I work on Machine Learning. I personally think that a better approach is to face it from the estimation theory (ref.). For instance, modeling the noise of the sensors and try to get Maximum Likelihood estimators, and if possible, derive a bound for their variance (CRLB) or a closed expression for the bias. For handling the noise, I suggested u-Kalman, Particle Filtering... But other terms that came to my mind were: "fuzzy", "GMM", "EM"...

EDIT:

I think what my colleague wants is a "data-driven/black-box" approach that can help him skip the math, since he has not the enough background...

Statistical approach

This question is more related to statistics than data science, perharps you would have better answers on Cross Validated. As you suggested in the question, a purely statistical approach should work for sure (though it requires some assumptions).

As i understand your problem, your prior knowledge is that $\alpha$ (or $\beta$) should be always the same, but you get noise on the realizations of $\alpha$, and you suppose this noise comes from the input ($\theta$), not the process ($f_1$).

Let us assume that the true $\alpha$ corresponds to the mean value of $\theta$, i.e. we have $\alpha_{real} = f_1(\bar{\theta})$. Just taking $\bar{\alpha}$ is generally wrong as you are assimilating $f_1(\bar{\theta})$ with $\overline{f_1(\theta)}$: those are not the same if the problem is not linear.

I suppose you have no idea what the distribution of $\theta$ looks like. But the good news is that you know the $f_1$ and $f_2$ processes, so you can infer the distribution parameters from your data (by maximum likelihood estimation). For instance, you can assume $\theta \sim \mathcal{N}(\mu, \sigma^2)$ (or whatever distribution seems likely), and determine which $\mu$ and $\sigma$ explain the realizations of $\alpha$ and $\beta$ best. Having two different outputs is comfortable, this makes the approach more robust.

Then, your true $\alpha$ and $\beta$ would simply be $f_1(\mu)$ and $f_2(\mu)$.

An incomplete "data-driven" idea

If we make the assumption that $\theta$ is evenly distributed and you have enough samples, its median is the same as the mean. So basically if we find the median, we get a good estimator of the true $\alpha$ and $\beta$ values (again assuming that the true value is the one at the mean of $\theta$).

If either $f_1$ or $f_2$ process is monotonous, this is straightforward, because you just have to sort $\alpha$ or $\beta$ realizations. In the same way, if one process is easily invertible (analytically or numerically), the problem becomes almost trivial. If both are highly non-monotonous and non-invertible, you will need to find a way to sort them in a machine-learning fashion. In the following, I will assume that both processes are at least continuous.

First option: try to perform a binary clustering of your data ($\alpha$, $\beta$). You will need to use the kernel trick, and this will only work if:

you have enough samples to map your processes in a continuous way
there is no "crossing" zone if both processes, i.e. $\forall \theta, \theta^*, [f_1(\theta^*) = f_1(\theta)\ and\ f_2(\theta^*) = f_2(\theta)] \iff \theta = \theta^*$

This may work if your processes are not too "complex".

Second option (which should work better on complex processes): start with the samples in a random order, and compute the euclidian distance (or any other relevant distance) from one sample to the next one, then sum: this gives you the total distance of the pathway going through all samples. Then, use an optimizing procedure (simulated annealing for instance) interverting the index of two samples, until you find an order that minimizes this metric.

This second option is basically solving the travelling salesman problem.

On both cases, I cannot guarantee that it will work, but I think it's worth trying, just for the fun of using standard methods that are not at all suited to the initial problem!


Multi-Class Neural Networks | different features This may be a wrong question or something so feel free to correct me :).

I have been studying neural networks for weeks now. I came across the multi-class classification model that uses neural networks.

As we see in this picture, the model allows you to classify your input into different classes. But this also assumes your input always use the same number features, right ? Meaning if I want to recognize handwritten digits, I should have images with the same dimension (for example 24x24) so I can use the same number of features (in this case 24x24=576). But what if one class, for example number 6, requires a different number of features (like the dimension of the handwritten digit 6 is 30x30 pixels)

I know that the logical way to do this, is to have two different neural networks, but Is there any way to simultaneously train a multi-class classification model where inputs might use different features? What does research say about this?

PS: If you have good reading materials about this, please feel free to link them too.

Bests,

It is essential for all input patterns to have the same number of features. The reason is that each input feature is connected to specified neurons and they have specified trained weights. A typical solution is to resize your input by reserving its aspect ratio. You should consider that if you want to have good accuracy, you have to have such an operation while training the network too. Even in convolutional networks, it is essential to have the same input size. Although the convolutional layers won't bother you, the connection of fully connected layers and flattened layers used after convolutional layers will have a problem if you don't use correct input shape; the dimensions won't match.

About recent studies, I have not seen yet, but a typical solution can be employing PCA and using let's say its top-10 features as the input of a fully connected network, although for input patterns with a huge difference in the number of input dimensions I guess it is not logical.


Word embedding/Word2vec for POS tagging I am building a entity detection and relation classification method using deep learning approach which requires vector representation of POS tags and entity label. I am familiar with word-embedding method but I don't know the answer of following questions:

How to convert pos tags into vector representation(say 20 dim)
How to combine both word embeddings and pos embedding together to build the classifier.

Similar question was asked in Cross validate community but I couldn't find the answer. Here is the link to question:
https://stats.stackexchange.com/questions/238016/deep-learning-word-embedding-with-parts-of-speech

Research paper link:
https://arxiv.org/abs/1601.00770

Word Embedding for Pos tags can easily be trained using pos tags sequence. There are lot of ways that you can get the trained model. I did it by gensim's word2vec api. Here is the link to it:
https://radimrehurek.com/gensim/models/word2vec.html

Also if you want memory efficient solution, radim(creater of gensim) provides a great tutorial: https://rare-technologies.com/word2vec-tutorial/

You just need to pass pos sequence of your training data, resulting vector size, min frequency count etc. You can view api's documentation for more detail.


How do you apply SMOTE on text classification? Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it possible to apply it on text classification problem? Which part of the data do you need to oversample? There is already another question about it, but it doesn't have an answer. Where can I possibly learn to get started with this?

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons:

SMOTE works in feature space. It means that the output of SMOTE is not a synthetic data which is a real representative of a text inside its feature space.
On one side SMOTE works with KNN and on the other hand, feature spaces for NLP problem are dramatically huge. KNN will easily fail in those huge dimensions.

So I can propose you two approaches:

Do not care about the real text representation of new synthetic samples which I assume should be fine. You need to balance the distribution for your classifier not for a reader of text data. So apply SMOTE as traditional (however I usually use the solution 2 bellow so I do not gaurantee the result!) with some Dimensionality Reduction step.

  1) Lets assume you want to make your data samples from minor class double using 3-NN. Ignore the major class(es) and keep only minor class samples.

  2) For each sample point in feature space choose 5 nearest neighbors. Then choose 3 of them randomly (isn't it unnecessary complicated? if I didn't want to explain the original algorithm I would say just choose 3 neighbors!)

  3) For each dimension calculate the distance between sample and neighbors and multiply it in a random number between 0-1 and add it to the original value of sample in that dimension. (this complicated paragraph simply means for each dimension choose a random value between the original sample and that neighbor!)

But I usually do another oversampling which is on the text (so more intuitive) and is kind of SMOTE.

  1) Ignore the major class. Get a length distribution of all documents in minor class so that we generate new samples according the the true document length (number of words/phrases). We assume we want to make the size of class triple (so producing $k=2$ synthetic documents per original document)

  2) Generate a sequence of $n$ random integers according to that distribution. It's used for determining the length of new synthetic document.

  3) For each document: Choose one integer from random length sequence and $m$ random document whose length is close to the integer. Put tokens of all $m$ documents in a set and randomly choose $n$ tokens $k$ times. these are your $k$ new documents.


Running multiple random forest and combining them I am trying to build a random forest model in R (RStudio). My training dataset has around 2 million rows and 38 variables. When I tested 5000 rows from this dataset I was able to build the random forest but when I run on the whole dataset I get the following error:

Error in randomForest.default(m, y, ...) :
  long vectors (argument 24) are not supported in .C

Can anyone please suggest, apart from removing the number of rows, how can I fix this? Can I run multiple random forests and then combine them into one? If yes, can someone please recommend how can I try this?

Many thanks in advance.

I'm very surprised you're running into this error with only 2mil rows and 38 variables. I would encourage you to have a go doing this in python using SKLearn and see if you run into the same issue. More generally though, if you have too much data (imagine you had 200 million rows), the right thing to do would not be to build multiple forests, but to build each tree with a smaller fraction of the data.

I don't have a suggestion for where to find such an implementation (there might be some way of combining sklearn and dask, I've seen such things for XGBoost), but the general principle would be that you wouldn't hold all of your data in memory, you'd read a subset of it from disk, train a tree, read a new subset, etc.


Clustering customer dataset to find customer patterns Say I have a dataset, which consists of n customers. A customer is represented by a name, an order, an orderAmount and a timestamp (time of day) for when the order was placed. It could look like this:

name     |order|orderAmount|timestamp
customer1|tv   |1          |08:30
customer2|hifi |1          |12:00
customer3|hifi |3          |12:30
customer4|tv   |2          |09:30
customer5|cd   |10         |10:00
customer1|tv   |2          |11:30
...      |...  |...        |...

What I'm interested in is clustering these customers, so it would be possible for me to clearly differentiate between them. By looking at the above set, it's clear that there's a difference in what, how much and when they buy, but I would like to automate this process, and I assume clustering is one way to do it, but please correct me if I'm wrong.

One thing in particular I'm not sure of is how I would represent this dataset in a "customer-matrix". I know, for example, a distance-based clustering method such as KMeans requires some sort of normalized input, but how would I get started on that with a dataset such as this?

When/if I can produce good clusters, I imagine I could assign labels to these clusters and use these labels to train a classifier?

I'm a beginner to data science, so there might be a step or two missing in my process so please bear with me.

Suggestion: use an indicator variable for your "order"+"orderAmount" data. The table would look like this, which seems more suitable for distance metrics:

name      | tv | hifi | cd | timestamp
customer1 | 1  | 0    | 0  | 8:30
customer2 | 0  | 1    | 0  | 12:00
customer3 | 0  | 3    | 0  | 12:30
customer4 | 2  | 0    | 0  | 9:30
...

If you do so, then I'd also suggest to normalize those indicator variables (make sure they are in the interval [0, 1], otherwise you could have a order of 1000 "cd" completely dominating the maximum of 10 "tv" for example).

Another thing to watch out is the possibility of binning. If the time of the day doesn't matter much, you could group all the orders from the same day and customer in a single row.

Also, if you can manually label some examples, you could use a semi-supervised algorithm, and maybe it would have better performance than completely unsupervised clustering. Some possible algorithms are HMRF-KMeans, co-training variants, and Spy EM.


How to transform raw data to fixed-frequency time series? How to transform raw data to fixed-frequency time series?

For example I have the following raw data in DataFrame

                       A    B
2017-01-01 00:01:01    0    100
2017-01-01 00:01:10    1    200
2017-01-01 00:01:16    2    300
2017-01-01 00:02:35    3    100
2017-01-01 00:02:40    4    100

I'd like to transform it into a time series:

1 minute frequency
column A should have sum of values in time interval
column B should have mean of values in time interval
possibly other functions over other columns

Note: Raw data is not periodic.

Transformed data should be:

                       A    B
2017-01-01 00:01:00    3    200
2017-01-01 00:02:00    7    100


Multicolinearity & accurate weights of predictors Let’s suppose that the stock value of various companies is the target of my models.

I have some “internal” predictors e.g. yearly sales of each company, sum of salaries at each company etc.

I have some “external” predictors e.g. geographical position of each company (latitude & longitude), population in the area in which each company operates etc.

Therefore, each observation at my dataset is about the stock value of a company along with its internal and external predictors.

The purpose of my project is to understand how each of a company's internal predictors affect in a very specific way the stock value of the respective company.

In simpler words, I want to get some accurate weights for the internal predictors which show me how exactly they affect the stock value of the respective company.

However, because there is relatively high multicolinearity between some of the internal predictors as a result I am not really getting very accurate weights for each one of them.

There may also be between the internal and the external predictors but I do not consider this as a problem because I think that you should account for all external predictors when taking the weight of the internal predictors.

However, I am not sure that I have to put all the internal predictors at the same model together because for example I do not want the weight of the yearly sales of a company to be modified because of the presence of other internal predictors such as the sum of salaries at this company.

In this regard, I am starting to think that the best way to go is to have multiple different models where each one of them has to do with one internal predictors separately but at every case all the external predictors.

Does this make sense?

Do you have any better idea?

P.S.
I just found a post which is quite similar to my line of reasoning: https://www.researchgate.net/post/Is_building_separate_models_a_solution_to_multi-collinearity.

If you build separate models, you are making the internal predictors/features independent of each other. This will cause a lot of your internals predictors to get really high weights which would probably not be the case if you added them together in the same model. An obvious method is to remove correlated predictors and then see what weights you get.


Reasons that LIME and SHAP might not agree with intuition I'm leveraging the Python packages lime and shap to explain single (test-set) predictions that a basic, trained model is making on new, tabular data. WLOG, the explanations generated by both methods do not agree with user intuition.

For example, when leveraging the methods in a healthcare setting, they might list the presence of a comorbidity (a disease that frequently co-occurs with the outcome disease of interest) as a factor that decreases a patient's risk of an adverse event.

Intuitively, such behavior is incorrect. We shouldn't see that history of heart attacks lowers the risk of adverse events, for example. What are some reasons that we might see these inconsistencies?

Some of my ideas

Class label imbalance: tried balancing the dataset, did not solve the issue
Kernel width for LIME: working on tuning this, but cursorily, no benefit
Relationship to training data: for tabular data, both lime and shap require the training dataset as input to build the explainer class. If there are instances in which a feature such as history of heart attacks were associated with a no adverse event outcome, such instances would "confuse" the methods, so to speak. However, I'm not sure I have the intuition correct there.
Error in understanding on my part: there may be nuances in intuition here that I've missed. Specifically, I am trying to make sure I correctly understand the relationship between the generated explanations and the training dataset used to build them.

As pointed by Sean Owen, this is probably linked to correlation in the input variables.

Concerning LIME : you can read here : https://christophm.github.io/interpretable-ml-book/lime.html (paragraph 5.7) why a correlation between inputs could pose problem, more specifically due to an implementation problem. (gaussian sampling ignore features correlation).

Shapley values are designed to take correlation into account by construction (more specifically trough the symetry condition, hinting that the more variables are similar the more they will have a similar effect). However, as stated in https://arxiv.org/abs/1705.07874 (paragraph 4.2), SHAP rely on the independence of input to approximate Shapley values.

So both techniques you used are subject to caution when used on data with important correlations.


What kind of model is this? Can anyone help me identify what kind of Architecture is behind this Application? Is it a "simple" Classification Network? If so how are the heatmaps generated?

https://www.youtube.com/watch?v=c_giaOUzB_8

Thanks

This technique is called saliency maps [1][2].

From the documentation of Keras-Vis:

  The idea is pretty simple. We compute the gradient of output category with respect to input image. This should tell us how output category value changes with respect to a small change in input image pixels. All the positive values in the gradients tell us that a small change to that pixel will increase the output value. Hence, visualizing these gradients, which are the same shape as the image should provide some intuition of attention.

  The idea behind saliency is pretty simple in hindsight. We compute the gradient of output category with respect to input image.

  $\frac{∂output}{∂input}$

  This should tell us how the output value changes with respect to a small change in inputs. We can use these gradients to highlight input regions that cause the most change in the output. Intuitively this should highlight salient image regions that most contribute towards the output.


Why does momentum need learning rate? If momentum optimizer independently keeps a custom "inertia" value for each weight, then why do we ever need to bother with learning rate?

Surely, momentum would catch up its magnutude pretty quickly to any needed value anyway, why to bother scaling it with learning rate?

$$v_{dw} = \beta v_{dw} +(1-\beta)dW$$
$$W = W-\alpha v_{dw}$$

Where $\alpha$ is the learning rate (0.01 etc) and $\beta$ is the momentum coefficient (0.9 etc)

Edit

Thanks for the answer! To put it more plain: momentum controls "how well we retain" the movement, and learning rate is "how fast do we reGain" the movement

To answer the first question about why we need the learning rate even if we have momentum, let's consider an example in which we are not using the momentum term.  The weight update is therefore:

$ \Delta w_{ij} = \frac{\partial E}{\partial w_{ij}} \cdot l $

where:

$ \Delta w_{ij} $ is the weight update
$ \frac{\partial E}{\partial w_{ij}} $ is the gradient of the error with respect to the weight
$ l \space $ is the learning rate coefficient

Our weight update is determined by the gradient of our current error with respect to the weight at node $ ij $.  Therefore, our prior weight deltas are not factored into our weight update equation.  If we were to eliminate the learning rate, our weights would not update.

Now let's consider an example using the momentum term in its derived form:

$ \Delta w_{ij} = (\frac{\partial E}{\partial w_{ij}} \cdot l) + (\mu \cdot \Delta w^{t-1}_{ij}) $

where:

$ \mu $ is the momentum coefficient
$ \Delta w^{t-1}_{ij} $ is the weight update of node $ ij $ from the previous epoch

Now we are factoring the previous weight delta in our weight update equation.  In this form, it is easier to see that the learning rate and momentum are effectively independent terms.  However, without a learning rate, our weight delta would still be zero.

Now you might ask: what if we remove the learning rate after getting an initial momentum value so that momentum is the sole influence of the weight delta?

This destroys the backpropagation algorithm.

The objective of backprop is to optimize the weights to minimize error.  We achieve this minimization by adjusting the weights according to the error gradient.

Momentum, on the other hand, aims to improve the rate of convergence and to avoid local minimas.  The momentum term does not explicitly include the error gradient in its formula.  Therefore, momentum by itself does not enable learning.

If you were to only use momentum after establishing an initial weight delta, the weight update equation would look as such:

$ \Delta w_{ij} = (\mu \cdot \Delta w^{t-1}_{ij}) $

and:

$ \lim_{t \to \infty} \Delta w^t_{ij} =
\begin{cases}
0 & | \space \mu < 1 \space \lor \space (\mu = 1 \space \land \space \Delta w^{t=0}_{ij} < 1) \\
1 & | \space \mu = 1 \land \space \Delta w^{t=0}_{ij} = 1\\
\infty & | \space otherwise
\end{cases}
$

Although there exists a scenario where the weight delta approaches zero, this descent is not based on the error gradient and is in fact predetermined by the momentum coefficient and the initial weight delta: this weight delta does not achieve our objective to minimize the error and is therefore useless.

TL;DR:

The learning rate is critical for updating the weights to minimize error.  Momentum is used to help the learning rate, but not replace it.


var = (0,833)
new_var = np.zeros(1000)
for i in var:
    new_var[i] = 1

I would need your sparse matrix datatype to give you a more precise answer.


Trouble in calculating the covariance matrix I'm trying to calculate the covariance matrix for a dummy dataset using the following formula, but it's not matching with the actual result.

Let's say the dummy dataset contains three features, #rooms, sqft and #crimes. Each column is a feature vector, and we have 5 data points. I'm creating this dataset using the following code:

matrix = np.array([[2., 3., 5., 1., 4.],
                   [500., 700., 1800., 300., 1200.],
                   [2., 1., 2., 3., 2.]])

Let's normalize the data, so the mean becomes zero.

D = matrix.shape[0]
for row in range(D):
    mean, stddev = np.mean(matrix[row,:]), np.std(matrix[row,:])
    matrix[row,:] = (matrix[row,:] - mean)/stddev

Now, I can write a naive covariance calculator that looks at all possible pairs of features, and that works perfectly.

def cov_naive(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(D):
        for j in range(i, D):
            x = X[i, :]
            y = X[j, :]
            sum_xy = np.dot(x, y) / N
            if i == j:
                covariance[i, j] = sum_xy
            else:
                covariance[i, j] = covariance[j, i] = sum_xy
    return covariance

But, if I try to implement the formula mentioned in the beginning, the result is incorrect. The method I am trying out is as follows:

def cov_naive_2(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(N):
        x = X[:, i]
        covariance += x @ x.T
    return covariance / N

What am I doing wrong here?

Expected output:

array([[ 1.        ,  0.96833426, -0.4472136 ],
       [ 0.96833426,  1.        , -0.23408229],
       [-0.4472136 , -0.23408229,  1.        ]])

Actual output from cov_naive_2

array([[3., 3., 3.],
       [3., 3., 3.],
       [3., 3., 3.]])

Your approach is mathematically right, your problem comes from the fact that numpy matrix multiplication defaults to inner product when providing vectors, independently from them being transposed to row or column.

Try modifying the penultimate line like this to force outer product.

def cov_naive_2(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(N):
        x = X[:, i]
        covariance += np.outer(x, x)  # <---- here
    return covariance / N
```


Assign a unique cluster based on a dataframe column with KMeans Algorithm I have the following df

    x1      x2    x3   x4
   1000   5000    0.8  restaurant1
   2000   7000   0.75  restaurant1
    500   1000    0.5  restaurant2
    700   1400    0.6  restaurant2
   1000   5000    0.8  restaurant2
    100    600    0.9  restaurant3
    200   1200    0.9  restaurant3
     50   1000    0.9  restaurant3

applying a Kmeans Algorithm for 2 clusters what happens is that y:

    x1      x2    x3   x4             Y
   1000   5000    0.8  restaurant1    1
   2000   7000   0.75  restaurant1    1
    500   1000    0.5  restaurant2    2
    700   1400    0.6  restaurant2    2
   1000   5000    0.8  restaurant2    1
    100    600    0.9  restaurant3    2
    200   1200    0.9  restaurant3    2
     50   1000    0.9  restaurant3    2

Possible Desired Outputs:

     x1      x2    x3   x4             Y
    1000   5000    0.8  restaurant1    1
    2000   7000   0.75  restaurant1    1
     500   1000    0.5  restaurant2    2
     700   1400    0.6  restaurant2    2
    1000   5000    0.8  restaurant2    2
     100    600    0.9  restaurant3    2
     200   1200    0.9  restaurant3    2
      50   1000    0.9  restaurant3    2

or

     x1      x2    x3   x4             Y
    1000   5000    0.8  restaurant1    1
    2000   7000   0.75  restaurant1    1
     500   1000    0.5  restaurant2    1
     700   1400    0.6  restaurant2    1
    1000   5000    0.8  restaurant2    1
     100    600    0.9  restaurant3    2
     200   1200    0.9  restaurant3    2
      50   1000    0.9  restaurant3    2

I would like to set this boundary: a restaurant must belong to 1 and only 1 cluster.

I understand why there is this output, but how could I avoid and fix it?

Below the code that I used in my notebook:

  #Converting float64 to numpy array
  x1=df['x1'].to_numpy()
  x2=df['x2'].to_numpy()
  x3=(df['x5']/df['x2']).to_numpy()
  x4=df_joint_raw['x4'].cat.codes.to_numpy()

  X=np.stack((x1,x2,x3,x4),axis=1)
  #Getting clusters
  y_pred=KMeans(n_clusters=2, random_state=0).fit_predict(X)

Very interesting question! I try my best:

It depends a bit on the number of clusters and number of restaurant but in general I explain a bit.

If the number of restaurants and clusters are the same, then, theoretically, your question has just one trivial answer: "each restaurant is a cluster". You even don't need any algorithm. I go a bit deeper on it.

Most of ML algorithms solve an optimization problem to find the answer. Sometimes optimization problems are subject to some constraints.

Example:

Cluster restaurants such that all similar restaurants are necessarily assigned to the same cluster.
Cluster restaurants such that the density of same restaurants in same clusters is maximum.

The first one has the trivial answer I gave before but second one can be solved. You run several clustering methods (or just k-means but with several initial conditions) and accept the one in which higher number of similar restaurants are in identical clusters. For this you need to convert "density of same restaurants in same clusters" to a mathematical formulation and use it as criterion of choice. If you need help on it just drop a comment so I update the answer.

In any case, you change the output of the clustering and you don't let it "naturally" find the clusters as you push a criterion which is not normally considered in the algorithm. But don't worry! The good thing is that at least you have a criterion for "goodness" of your clustering which does not exist normally in clustering problem.

UPDATE

Let's try $\chi^2$ test first. It is pretty simplified but try it and if it didn't work we can think of something else. For know how, I prepared it for you in a simple way so you don't get confused with different tutorials on the net.

Imagine you have 4 restaurants and you want 4 clusters. You will end up with such a frequency table which says how many restaurants of which type fall in which cluster:

Then in Python, you simply calculate $\chi^2$ statistic which tells you if clusters and restaurants "are correlated or not".

from scipy.stats import chi2_contingency
obs = np.array([[10,1,2,1], [1,11,0,1], [1,2,8,1], [0,2,2,12]])
chi, p, _,_ = chi2_contingency(obs)
print('The chi-square statistic of {} with p-value of {}'.format(chi,p))

P-value, as you know, tells you if the statistic is significant. There theoretical consideration in this solution but I am not going to confuse you with that. I apologize as I did not go through your proposal in the comment. Will answer accordingly as soon as I find time to have a look at that.

Good Luck!


Best ML technique to suggest predictor variables In the following dataset, the first 4 columns are predictor variables and the engine running index is the response variable.
   O2 level | Cylinder pressure | Fuel Flow | Engine temp | Engine running index
       5              15               3           31                7
       2              31              18            1               88
       1              22              66            4               31
      ...            ...              ...          ...             ...

Situation: An engineer comes with a similar data set and asks a question: "What should be the setting of my O2 level, cylinder pressure etc to get the best running index?"
Now one could take a statistical approach and try figure out the answer, but I was looking to employ some machine learning techniques. The ones I've tried, i.e. Regression, are used more to predict what the Engine running index will be, rather than suggest what the settings should be, to get the best running index. The other one I was looking at was PCA, but not sure that it will give me an answer as well.
Note: the possible max value for Engine running index is not known.
Question:
Are there ML techniques to help me answer the engineer's question?

STEP 1

When you do Regression on this dataset, what you obtain is:

$\hspace{60mm}f(x_1,x_2,x_3,x_4) = ERI$

where your $x_1,x_2,x_3,x_4, ERI$ are O2 level, Cylinder Pressure, Fuel Flow,, Engine Temp, and Engine Running Indexrespecetively.
In case of Multiple Regression, you will get 1 coefficient per feature (total 4 in coef_, {$w_1,w_2,w_3,w_4$}) and one intercept_ ($w_0$) term for the intercept. Your equation would then come to look like:

$\hspace{40mm}ERI = w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$,

where $w_0$ - $w_4$ are fixed values. Your objective then is to find the best $x_1$ - $x_4$ which maximize $ERI$.

STEP 2

The above is better expressed by the optimization problem:

$\max_{(x_1,x_2,x_3,x_4)} w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0$
with constraints that $x_1,x_2,x_3,x_4 \geq 0$

I added those constraints by default as all those variables would be positive only. You could have upper limits on each of them as well. It is important to add these constraints because you don't want any of your value to reach $\infty$ or even some unrealistic values like Engine Temp = $10^6$.

The above function is the objective function which you want to maximize, OR you can minimize the negative of the same. Both of them are equivalent. Thus your objective becomes:

$\min_{(x_1,x_2,x_3,x_4)} -(w_1 x_1 + w_2 x_2 +w_3 x_3 +w_4 x_4 +w_0)$
with constraints that $x_1,x_2,x_3,x_4 \geq 0$

From here, you can use Scipy's Minimization function with the Objective function we just came up with.


Loss function in GAN Since the aim of a Discriminator is to output 1 for real data and 0 for fake data, hence, the aim is to increase the likelihood of true data vs. fake one. In addition, since maximizing the likelihood is equivalent to minimizing the log-likelihood, why are we updating the discriminator by ascending its stochastic gradient as mentioned in Algorithm 1 in https://arxiv.org/pdf/1406.2661.pdf. Shouldn't we update the discriminator by descending its stochastic gradient?

Any help is much appreciated!

In algorithm 1 of the original GAN article (https://arxiv.org/pdf/1406.2661.pdf), the discriminator is said to be updated by "ascending its stochastic gradient". This is referring to equation 1:
$$
\min_G \max_D V(D, G)=
\mathbb{E}_{x\sim p_{data}(x)}[\log D(x)]
+ \mathbb{E}_{z\sim p_z(z)}[\log(1 - D(G(z)))]
$$

When we want to minimize something, we do grandient descent. When we want to maximize something, we do gradient ascent. In this context, we want to maximize $V(D, G)$ with respect to the discriminator $D$, that is, the $\max_D V(D, G)$ part from equation 1.

I recommend you have a look at NIPS 2016 GAN tutorial video and text. They are very enlightening.


overfitting of regression tree from the beginning I am trying to build a regression tree to model insurance claim frequencies.
I have 36000 observations and 9 covariates.

My model overfits right from the beginning!, ie as the cost complexity goes down,
the relative validation error goes up.
I am using this command in R:

rpart_model <-rpart(cbind(Duration,Nbclaims) ~ Gender + DriverAge +
  CarAge + Area + Leasing + Power + Fract + Contract, data = motor,    method="poisson",parms=list(shrink=1), control=rpart.control(cp=0))

and obtain these results:

        CP nsplit rel error xerror     xstd
1   1.9019e-03        0     1.00000   1.0002   0.022911

2   1.3854e-03        1     0.99810   1.0004   0.022973

3   1.1587e-03        2     0.99671   1.0057   0.023185

4   9.1009e-04        5     0.99324   1.0069   0.023269

5   9.0852e-04        6     0.99233   1.0134   0.023526

6   8.8781e-04        7     0.99142   1.0134   0.023531

7   8.5397e-04       10     0.98872   1.0150   0.023603

8   8.5119e-04       11     0.98786   1.0153   0.023621

I have done my research to try and figure this issue out but am completely stuck for the moment, could anyone help with this?

Complexity Parameter CP describes a threshold $T$. If branch provides improvement less than $T$ it is deleted from the tree. You are using cp=0 so you told the algorithm to DO NOT prune any branches independly on their results! CP tested by printcp are very small also.

The bigger value of CP is, the more branches would be pruned so your tree should be smaller and should generalize better. Therefore you can try with highers values of CP at start and then print cross-validation results for getting suggested CP value and finally use prune method to reduce your tree for example:

model_pruned <- prune(rpart_model, cp=0.01)

You can compare your trees' structures using rpart.plot(tree_object) method. I think the original tree is way too complex and that is why it overfitts at start.


When are weights updated in CNN? In CNNs when do we update the kernel parameters using back propagation? Suppose I have batch size of 50 and training data of 1000. Do I back propagate after each batch has been presented to network or after each data sample?

Whenever you train the network using batch means that you have chosen to train using batch gradient descent. There are three variants for gradient descent algorithm:

Gradient Descent
Stochastic Gradient Descent
Batch Gradient Descent

The first one passes the whole data through the network and finds the error rate for all of them and finds the gradients with respect to all the data samples and updates the weights after passing the whole data-set. That means for each epoch, passing the whole data-set through the network, one update occurs. This update is accurate toward descending gradient.

The second one, updates the weights after passing each data which means if your data sample has one thousand samples, one thousand updates will happen whilst the previous method updates the weights one time per the whole data-sample. This method is not accurate but is so much faster than the previous one.

The last one tries to find a trade-off between the above approaches. You specify a batch size and you will update the weights after passing the data samples in each batch, means the gradients are calculated after passing each batch. Suppose you have one thousand data sample and you have specified a batch size with one hundred data sample. You will have 10 weight update for each epoch. This method is more accurate than the second approach and is more faster than the first approach.

  Do I back propagate after each batch has been presented to network or after each image?

Your method is the last one. Consequently, after passing the entire batch, you would update the weights.

Based on the comments of one of our friends, the above approaches are named as follows, respectively:

[Batch] Gradient Descent (batch size = all training samples)
True SGD (batch size = 1 - weights update for each training sample)
Mini-batch SGD (batch size = m out of n training samples).


Error when trying to run RandomForestClassifer with Pipieline and GridSearch I am trying to run a RandomForest Classifier using Pipeline, GridSerach and CV

I am getting an error when I fit the data. I am not sure how to fix it.

Will appreciate any help on this.

My code is:

 column_trans = make_column_transformer((OneHotEncoder(), ['CategoricalData']),
 remainder='passthrough')
 RF = RandomForestClassifier()
 pipe = make_pipeline(column_trans, RF)

 # Set grid search params
 grid_params = [{'RF__criterion': ['gini', 'entropy'],
 'RF__min_samples_leaf': [5,10,20,30,50,80,100],
 'RF__max_depth': [3,4,6,8,10],
 'RF_min_samples_split': [2,4,6,8,10]}]

 # Construct grid search
 gs = GridSearchCV(estimator = pipe,
 param_grid = grid_params,
 scoring='accuracy',
 cv=5)

 gs.fit(train_features, train_target)  ----This is where I get an error

 ValueError: Invalid parameter RF for estimator Pipeline(steps=[('columntransformer',
             ColumnTransformer(remainder='passthrough',
                               transformers=[('onehotencoder',
                                              OneHotEncoder(),
                                              ['CategoricalData'])])),
            ('randomforestclassifier', RandomForestClassifier())]). Check the list of available parameters with `estimator.get_params().keys()`.

In addition to adding a '_' I had to replace RF by randomforestclassifier in all grid params - Please see link below.
https://stackoverflow.com/a/62166539/9592484


Pre-trained models for finding similar word n-grams Are there any pre-trained models for finding similar word n-grams, where n>1?
FastText, for instance, seems to work only on unigrams:
from pyfasttext import FastText
model = FastText('cc.en.300.bin')
model.nearest_neighbors('dog', k=2000)

[('dogs', 0.8463464975357056),
 ('puppy', 0.7873005270957947),
 ('pup', 0.7692237496376038),
 ('canine', 0.7435278296470642),
 ...

but it fails on longer n-grams:
model.nearest_neighbors('Gone with the Wind', k=2000)

[('DEky4M0BSpUOTPnSpkuL5I0GTSnRI4jMepcaFAoxIoFnX5kmJQk1aYvr2odGBAAIfkECQoABAAsCQAAABAAEgAACGcAARAYSLCgQQEABBokkFAhAQEQHQ4EMKCiQogRCVKsOOAiRocbLQ7EmJEhR4cfEWoUOTFhRIUNE44kGZOjSIQfG9rsyDCnzp0AaMYMyfNjS6JFZWpEKlDiUqALJ0KNatKmU4NDBwYEACH5BAUKAAQALAkAAAAQABIAAAhpAAEQGEiQIICDBAUgLEgAwICHAgkImBhxoMOHAyJOpGgQY8aBGxV2hJgwZMWLFTcCUIjwoEuLBym69PgxJMuDNAUqVDkz50qZLi',
  0.71047443151474),

or
model.nearest_neighbors('Star Wars', k=2000)
[('clockHauser', 0.5432934761047363),
 ('CrônicasEsdrasNeemiasEsterJóSalmosProvérbiosEclesiastesCânticosIsaíasJeremiasLamentaçõesEzequielDanielOséiasJoelAmósObadiasJonasMiquéiasNaumHabacuqueSofoniasAgeuZacariasMalaquiasNovo',
  0.5197194218635559),

First off, there aren't, to my knowledge, models trained specifically to generate ngram embeddings. Although, it would be very easy to modify the word2vec algorithm to accommodate ngrams.
Now, what can you do?
You could compute the ngram embedding by summing up the individual word embeddings. Potentially, you could apply weights based on tfidf for instance, but not required. Once you have 1 embedding, simply find a nearest neighbor using cosine distance.
Another approach, though more computionally expensive would be to compute the Earth Mover's Distance (also called Wasserstein) between ngrams and find nearest neighbors this way.


How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis? np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on function_name(mean,sigma,skew,kurtosis,size).
I tried scipy.stats.gengamma but I don't understand how to use it. It takes 2 parameters - a,c and creates a distribution. But it is difficult to interpret for what values of a & c, the function will give a particular value of skewness and kurtosis.
Can anyone explain how to use gengamma or any other way to create such a distribution in python, even from scratch by writing mathematical equations?
Edit: By Gaussian, I mean that I want the distribution to be normal with some skewness or kurtosis as well. It need not be a standard normal distribution.

The Gaussian distribution is fully described by its mean and variance.
Gaussians have fixed values for Skewness (0) and Kurtosis (3) - so you can't really change them if you have made the Gaussian assumption for your model.


What is the best practice to classify category of named entity in sentence I have 1-4 gram text data from wikipedia for 14 categories, which I am using for NE classification.
I feed named entity from sentence to lucene indexer which searches named entity from these 14 categories.
Issue I am facing is, for single entity I get multiple classes as a result with same score.
like while search titanic, indexer gives this result

Score    - 11.23
Title    - titanic
Category - Book

Score    - 11.23
Title    - titanic
Category - Movie

Score    - 11.23
Title    - titanic
Category - Product

now problem is which class to be considered?

I already tried with classifiers (NB,ME in nltk,scikit learn), but as it consider each entity from dataset as feature, it works as indexer only.

Why lucene?

I'm not sure I fully understand your question, but it seems to me that you're trying to determine the category of the string/entity "titanic" out of context. Your data tells you that "titanic" could be a book, a movie, or a product, and you want to figure out which one is correct -- is that what you're trying to do?

If so, the problem is that you've dropped the context in which the string/entity "titanic" appears in your original text. For example...

In the sentence "I couldn't stop reading Titanic," the word "titanic" refers to a book.
In the sentence "Titanic was one of the highest-grossing films of all time," the word "titanic" refers to a movie.
In the sentence "The Titanic was the world's largest ocean liner," the word "titanic" refers to a product.

Without that context, there's no way to know which is the correct category. I'd suggest looking into how named entity recognition tools like Stanford NER work -- that will help you better understand how to do something like this. You'll see that the input to an NER tool generally needs to be a sentence, in order to take advantage of the context to properly categorize the extracted entities.


Searching for a 3D Dataset, segmented by 2 or more Experts We are looking for data sets with 3D images, preferably from the medical field. It is important that they have been segmented by more than one person/expert. An example of this is the BraTS Challenge dataset (https://www.med.upenn.edu/sbia/brats2018/data.html) or the LIDC IDRI Dataset (https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI), but we need more datasets for our research.

Many thanks in advance!

You can look into the MICCAI dataset. This contains 35 3D MRI scans of brains. The brains have been parcelated into 134 distinct brain regions by professionals.


Model with 2 datasets: combine time series data and statistics I am new to data science modelling so apologies if using wrong terminology in advance.
I have a standard time series dataset of historical prices which is used to train/test a simple Random Forest classifier model which predicts the returns direction (+/-).
I also have a few general statistics for 'day of the week direction' eg. frequency counts: Monday UP=120, Monday DOWN=90, Tuesday UP=67, Tuesday DOWN=50, Friday UP=55, Friday DOWN=181.
How can I combine the results from the time series RF model with the bias from the statistics into a new model? eg. give more weight to positive/up predictions on Monday, and more weight to negative/down predictions on Friday?

There are multiple approaches. Two of them are:

Convert time series data to structured data
Convert your statistic data to time series data

This article is a good introduction.
For your second question about giving more weight to some features, please read this post.
Hope it helps.


How to know for sure if we can learn from a given data or not? I want to know that given a set of data and a target, how we can know for sure whether we can learn from that data to make any inference or not?

how can we know for sure

We can't.
A toy example to show why even humans can not do this for sure:

Assume you get the number 2, 4, 8, 16, 32, ?? and want to extrapolate to the next number ??. A natural extension of the series would be 64, but we can not take this for granted. The next number can just as well be 0. You can not be sure.

Only given the data and without additional assumptions about what you would expect to see, you can not learn a correct model per-se. You always have to be critical about your data.


Is there a relationship between LDA, linear SVMs and Perceptron? LDA (linear discriminant analysis), SVMs with a linear kernel, and perceptrons are linear classifiers. Is there any other relationship between them, e.g.:

Every decision boundary that can be found by LDA can be found by linear SVM
Every decision boundary that can be found by linear SVM can be found by LDA.
Every decision boundary that can be found by LDA can be found by a perceptron
Every decision boundary that can be found by linear SVM can be found by a perceptron.
Every decision boundary that can be found by a perceptron can be found by LDA
Every decision boundary that can be found by a perceptron can be found by an SVM with a linear kernel

Always on the same data, of course.

For example, I think the linear SVM can find more decision boundaries than a perceptron due to slack variables. While the perceptron finds just an arbitrary hyperplane which separates the data linearly (if such a hyperplane exists), the linear SVM will always find the same hyperplane due to the optimality criterium.

The brief answers are:

1, 2. No. They depend on different subsets of examples.

3, 4. Yes, but only if it separates classes linearly and you are extremely lucky. Otherwise no.

5, 6. No, because SVM and LDA find only one solution, but perceptron can find many.

Now let me explain.

Decision boundaries of classicl SVM and LDA are calculated offline based on the whole training sample at once. Thus, SVM and LDA will indeed always find exactly one hyperplane, regardless of the order of learning examples.

But these hyperplanes of LDA and SVM are not generally guaranteed to be different. As its name suggests, SVM solution is based only on the support observations, which usually constitute a small fraction of the learning sample. LDA solution, however,  is sensitive to all the learning examples, because it is explicitly based on their classwise means. Thus, you can shift one non-support point a little, and SVM solution will not change, but LDA will. And it means that you cannot expect that SVM and LDA will find the same boundary.

Perceptron, in contrast, is updated online, so its decision boundary depends on the order of learning examples. Moreover, its solutions depends on the initialization of coefficients. Namely, if $w_n$ is vector of coefficients after seeing $n$ learning examples, then:
$$
w_n = w_0 + \lambda \sum_{i=1}^{n} e_i x_i
$$
where $e_i$ equals 1 if $i$'th example was false negative, -1 for false positives, and 0 otherwise, and $\lambda$ is the learning rate. By varying $w_0$ and order of feeding $x_i$ into perceptron, you can achieve many different hyperplanes.

For example, if (by luck!) you set $w_0$ equal to solution of SVM or LDA, and this solution linearly separates your classes, then perceptron will never change this solution, so it will be equal to solution of SVM or LDA.

However, if the solution of SVM or LDA does not fully separate the classes (for LDA, this is may be the case even if the classes are separable), then perceptron will change it with the next misclassified example, so its solution will diverge from SVM or LDA.


splitting into train test by train_test_split of float values? How to split into train test by train_test_split of float values ?
I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2., what is the better solution?

 X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=seed )

    from imblearn.under_sampling import RandomUnderSampler
    from collections import Counter

    rus = RandomUnderSampler(random_state=seed, return_indices=True) # (50:50)

    from sklearn import preprocessing
    lab_enc = preprocessing.LabelEncoder()
    training_scores_encoded = lab_enc.fit_transform(y_train)
    #print(training_scores_encoded)

    X_res, y_res, idx_res = rus.fit_sample(X_train, training_scores_encoded)

    from sklearn.preprocessing import StandardScaler

    scalerT = StandardScaler().fit(X_train[:,[0]])
    #scalerT = StandardScaler().fit(X[:,[0]])

    X_train[:,[0]] = scalerT.transform(X_train[:,[0]])
    X_test[:,[0]] = scalerT.transform(X_test[:,[0]])
    X_res[:,[0]] = scalerT.transform(X_res[:,[0]])

    scalerA = StandardScaler().fit(X_train[:,[idx_Amount]])
    #scalerA = StandardScaler().fit(X[:,[idx_Amount]])

    X_train[:,[idx_Amount]] = scalerA.transform(X_train[:,[idx_Amount]])
    X_test[:,[idx_Amount]] = scalerA.transform(X_test[:,[idx_Amount]])
    X_res[:,[idx_Amount]] = scalerA.transform(X_res[:,[idx_Amount]])

    from sklearn.linear_model import LogisticRegression

    models = []
    models.append(('LR', LogisticRegression()))

    # Default parameters for each model
    models

    from sklearn.model_selection import cross_val_score
    from sklearn.model_selection import StratifiedShuffleSplit

    XX = X_res
    yy = y_res

    #StratifiedKFold is used by default for cv integer
    #cv=10
    cv = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=seed)
results = []
names = []

for name, model in models:
    cv_results = cross_val_score(model, XX, yy, cv = cv, scoring = 'accuracy')
    results.append(cv_results)
    names.append(name)
    msg = "%s: %.2f (+/- %.2f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)

fig = plt.figure()
fig.suptitle('Accuracy')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()

results = []
names = []

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedShuffleSplit

XX = X_res
yy = y_res

#StratifiedKFold is used by default for cv integer
#cv=10
cv = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=seed)

for name, model in models:
    cv_results = cross_val_score(model, XX, yy, cv = cv, scoring = 'recall')
    results.append(cv_results)
    names.append(name)
    msg = "%s: %.2f (+/- %.2f)" % (name, cv_results.mean(), cv_results.std())
    print(msg)

The issue is your data violates the requirements of StratifiedShuffleSplit, specifically it's not possible to do a 70:30 split of the data and maintain the same number of distinct y values in the test and train set - most likely because you have a y value which only occurs 1 time?

Perhaps just use ShuffleSplit?


Multi layer back propagation Neural network for classification Can someone explain me, how to classify a data like MNIST with MLBP-Neural network if I make more than one output (e.g 8), I mean if I just use one output I can easily classify the data, but if I use more than one, which output should I choose ?

Suppose that you need to classify something in K classes, where K > 2. In this case the most often setup I use is one hot encoding. You will have K output columns, and in the training set you will set all values to 0, except the one which has the category index, which could have value 1. Thus, for each training data set instance you will have all outputs with values 0 or 1, all outputs sum to 1 for each instance.

This looks like a probability, which reminds me of a technique used often to connect some outputs which are modeled as probability. This is called softmax function, more details on Wikipedia. This will allow you to put some constraints on the output values (it is basically a logistic function generalization) so that the output values will be modeled as probabilities.

Finally, with or without softmax you can use the output as a discriminant function to select the proper category.

Another final thought would be to avoid to encode you variables in a connected way. For example you can have the binary representation of the category index. This would induce to the learner an artificial connection between some outputs which are arbitrary. The one hot encoding has the advantage that is neutral to how labels are indexed.


Why replacing null values with outliers? I have been watching a tutorial on stock price prediction with multivariate linear regression and the tutor replaces missing value data, NaN, with the outlier -99999.
Why and how do replacements like that do not skew the data and provide a biased or incorrectly-trained classifier?

-999999 is pretty common when you work with trees or forests (you imputatte missing values with the value that differs much from other values to perform better splitting).

Maybe (!) in some rare cases imputation by -999999 may be useful (for example, I used it by clustering) though.

Mean/median as advised are really good-working approaches. Including, for example, boolean column that indicates missing values (=1; before imputation) and non-missing (=0).


How to get nearest 5 points mean with Nearest Neighbours? In a dataset of longitude, latitude and price (of houses) I'm using sklearn's KNearestRegressor to get the 5 nearest neighbors mean price for each point. The problem is I want to do this for the whole dataset and each point is taking its own price into consideration since KNearestRegressor is a supervised algorithm, and I'm predicting the training set. How can I get the 5 nearest neighbors mean price for each point, not taking into consideration that point's price?

I have tried sklearn's unsupervised NearestNeighbours but it only outputs the indexes of the nearest neighbors, not the mean price.

First of all you train/ fit a model on the training set, may be cross validate using a training set. But it is not meant to predict itself. That's what test set is for.

You can achieve your objective by predicting each observation in a Leave One Out manner (also used for cross validation):

For every observation (i) in your dataset, fit a KNN regressor using all data except (i), evaluate your prediction for (i).


Classification vs Regression Model what should I choose? I am working on a problem like 'customers next month revenue prediction'. Here revenue will be the target variable. Again we actually segment the customers based on there revenue(like if they give less than 200 they will be in category 'A' else 'B'). I have to predict both(provable revenue + category). What will be the right approach, choose a regression model and predict the revenue and then categorized or i should choose separate model(regression for revenue, classification for category).

The segmentation sounds rather arbitrary if it is simply a reflection of the revenue you are trying to predict rather than having been derived from other bjective data that then predicts revenue it is unlikely to add very much (though obviously past revenue is a good predict of future revenue - the question is does the sementation improve the prediction vs just using the past data.

So if it were me I would be inclined to do the following:

(1)  try some dimension reduction and/or clustering to explore whether or not you really have meaningful customer segments rather than just an arbitrary cutoffs by revenue.

(2) if you have a good segmentation try creating a customer classifer to predict segment memberhip

(3) if your classifier has some value use the (non-linear) segment prediction as a additional input to a regression to predict revenue.

(4) validate and/or test the model against some reserved data not just the main data set.

With any regression though correct model specification is vital. So the first thing you need to do is explore the dataset (plot lots of stuff against lots of other stuff, look at some distributions, etc) to try and understand its structure before you specify a model.

Particulary, if your primary data source is the time series of past sales you probably need to look at including mean-reversion and momentum terms in the specification (see ARM, ARIMA etc models ).

(5) It would also be fun to try a Recurrent Neural Network, though I have never done this myself.


How to interpret Hashingvectorizer representation? I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams per document and as such one can compute (dis)similarity between the representation vector.

But how could one imagine the arbitrary number of features in case of Hashing?
Are these features purely synthetic features with no correlation of the original words?
Is it able to learn from some training set (by "fit") and apply this learned features to new sentences (by "transform") like other vectorizers?
Does Hashing represent any semantic distance like word2vec for words? How?

Basic Background

Imagine the process of count vectorizer: you first create a vocabulary which maps each word (or n-gram) to an integer index (index in the document term matrix). Then, for each document, you count number of times a word appears and set that value at appropriate index to build vector representation for the document.
This can potentially create a very large number of features since each n-gram/token is one feature.
Even if you want to limit the total number of features by using some trick like top-N words by occurrence, you still need to calculate and hold in memory the map of all word-counts. This can be potentially prohibitive in some applications.
Similar problem happens for TfIDf, where you additionally store the mapping of word to document occurance for calculating the IDf part.
Either way, you are doing multiple passes over the data and/or potentially large amount of memory consumption.
The problem is also with bounds or predictability: you do not know the potential memory usage upfront in first phase.
Hashing vectorizer can build document representation for all documents in one single pass over the data and still keep memory bounded (not necessarily small, the size depends on size of hash-table).
In a single pass, you calculate hash of a token. Based on the hash value, you increment the count of particular index in the hash-table (the array underlying the hash table implementation). You get representation of current document without looking at every other document in the corpus.
This gives rise to problem with representation accuracy. Two different tokens may have hash collision.

So you are in effect trading [representation accuracy and explanatory power] Vs. [space (bounded predictable memory usage) and time (no multiple passes on the data)].

Answers to your specific questions

There is a (sort) correlation between input words and features: through the hash function. But this correlation is potentially defective (hash-collisions) and there's no inverse transformation (you can't say what word is represented by feature number 207).
There's no fit and transform. For a fixed hash-function, no dataset specific learning is happening (ala word2vec).
There's no semantic interpretation of the distance. Two words semantically similar words may not be close to each other in the representation. As long as two almost (syntactically, based on tokens) similar documents are close enough, it will work on text classification.

Why Would It Work?

Given these information, you are right in being skeptical: why on earth this should work? The answer is empirical: a randomized representation like hashing works reasonably well in practice (the benefits from exact count based representation are not that great). There might be some theoretical explanation too but I don't know it enough. If curious, you can probably read up this paper.


How do I find the minimum value of $x^2+y^2$ with a genetic algorithm? I want to find $(x,y)$ which minimizes $x^2+y^2$ with GA to apply it for another function.

Does anyone know any example of GA with deap (Python) like that?

Genetic algorithm

This consists in 4 crucial steps: initialization, evaluation, selection and combination.

Initialization

Each individual in the population is encoded by some genes. In our case the genes represent our $[x, y]$ values. We will then set our search range to [0, 1000] for this specific problem. Usually you will know what is naturally possible based on your problem. For example, you should know the range of possible soil densities in nature. We will create 100 individuals in our population.

Evaluation of the fitness

This step simply asks you to put the $[x,y]$ values into your function and get its result. Pretty standard stuff.

Selection

There are many ways with which you can select parents. I will always keep the alpha male. The best individual in the population, he will be cloned to the next. Then I will use tournament selection. We will repeat the following until the next generation population is full. Pick four parents at random, take the best individual from the first two and the best from the last two. These will be the two parents which will gives us our next offspring.

Combination

From the two parents we will build the new genome for the child using the binary values of the $[x,y]$ values of the parents. The resulting binary value for each codon in the genome of the child is selected from the two parent genes by uniform random.

The code

class Genetic(object):

    def __init__(self, f, pop_size = 100, n_variables = 2):
        self.f = f
        self.minim = -100
        self.maxim = 100
        self.pop_size = pop_size
        self.n_variables = n_variables
        self.population = self.initializePopulation()
        self.evaluatePopulation()

    def initializePopulation(self):
        return [np.random.randint(self.minim, self.maxim, size=(self.n_variables))
                           for i in range(self.pop_size)]

    def evaluatePopulation(self):
        return [self.f(i[0], i[1]) for i in self.population]
        #return [(i[0]-4)**2 + i[1]**2 for i in self.population]

    def nextGen(self):
        results = self.evaluatePopulation()
        children = [self.population[np.argmin(results)]]

        while len(children) < self.pop_size:
            # Tournament selection
            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p1 = self.population[randA]
            else: p1 = self.population[randB]

            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p2 = self.population[randA]
            else: p2 = self.population[randB]

            signs = []
            for i in zip(p1, p2):
                if i[0] < 0 and i[1] < 0: signs.append(-1)
                elif i[0] >= 0 and i[1] >= 0: signs.append(1)
                else: signs.append(np.random.choice([-1,1]))

            # Convert values to binary
            p1 = [format(abs(i), '010b') for i in p1]
            p2 = [format(abs(i), '010b') for i in p2]

            # Recombination
            child = []
            for i, j in zip(p1, p2):
                for k, l in zip(i, j):
                    if k == l: child.append(k)
                    else: child.append(str(np.random.randint(min(k, l),
                                                             max(k,l))))

            child = ''.join(child)
            g1 = child[0:len(child)//2]
            g2 = child[len(child)//2:len(child)]
            children.append(np.asarray([signs[0]*int(g1, 2),
                                        signs[1]*int(g2, 2)]))
        self.population = children

    def run(self):
        ix = 0
        while ix < 1000:
            ix += 1
            self.nextGen()
        return self.population[0]

Then you can use the code by

f = lambda x, y: (x)**2 + y**2
gen = Genetic(f)
minim = gen.run()
print('Minimum found      X =', minim[0], ', Y =', minim[1])

  Minimum found      X = 0 , Y = 0

f = lambda x, y: (x-6)**2 + y**2
gen = Genetic(f)
minim = gen.run()
print('Minimum found      X =', minim[0], ', Y =', minim[1])

  Minimum found      X = 6 , Y = 0


Extra output layer in a neural network (Decimal to binary) I'm working through a question from the online book.
I can understand that if the additional output layer is of 5 output neurons, I could probably set bias at 0.5 and weight of 0.5 each for the previous layer. But the question now ask for a new layer of four output neurons - which is more than enough to represent 10 possible outputs at $2^{4}$.
Can someone walk me through the steps involved in understanding and solving this problem?
The exercise question:
There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first 3 layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least 0.99, and incorrect outputs have activation less than 0.01.

The question is asking you to make the following mapping between old representation and new representation:

Represent    Old                     New
0            1 0 0 0 0 0 0 0 0 0     0 0 0 0
1            0 1 0 0 0 0 0 0 0 0     0 0 0 1
2            0 0 1 0 0 0 0 0 0 0     0 0 1 0

3            0 0 0 1 0 0 0 0 0 0     0 0 1 1
4            0 0 0 0 1 0 0 0 0 0     0 1 0 0
5            0 0 0 0 0 1 0 0 0 0     0 1 0 1

6            0 0 0 0 0 0 1 0 0 0     0 1 1 0
7            0 0 0 0 0 0 0 1 0 0     0 1 1 1
8            0 0 0 0 0 0 0 0 1 0     1 0 0 0

9            0 0 0 0 0 0 0 0 0 1     1 0 0 1

Because the old output layer has a simple form, this is quite easy to achieve. Each output neuron should have a positive weight between itself and output neurons which should be on to represent it, and a negative weight between itself and output neurons that should be off. The values should combine to be large enough to cleanly switch on or off, so I would use largish weights, such as +10 and -10.

If you have sigmoid activations here, the bias is not that relevant. You just want to simply saturate each neuron towards on or off. The question has allowed you to assume very clear signals in the old output layer.

So taking example of representing a 3 and using zero-indexing for the neurons in the order I am showing them (these options are not set in the question), I might have weights going from activation of old output $i=3$, $A_3^{Old}$ to logit of new outputs $Z_j^{New}$, where $Z_j^{New} = \Sigma_{i=0}^{i=9} W_{ij} * A_i^{Old}$ as follows:

$$W_{3,0} = -10$$
$$W_{3,1} = -10$$
$$W_{3,2} = +10$$
$$W_{3,3} = +10$$

This should clearly produce close to 0 0 1 1 output when only the old output layer's neuron representing a "3" is active. In the question, you can assume 0.99 activation of one neuron and <0.01 for competing ones in the old layer. So, if you use the same magnitude of weights throughout, then relatively small values coming from +-0.1 (0.01 * 10) from the other old layer activation values will not seriously affect the +-9.9 value, and the outputs in the new layer will be saturated at very close to either 0 or 1.


Unsupervised text clustering using a driving list I want to apply unsupervised clustering on a set of short texts, which I need to divide into 2 clusters.
Also I know that one of my clusters is likely to contain some words (non-exhaustive list) and I would like to use them for driving my machine learning.

I did not really find any simple way to do it (from any algorithm, library, tool...). Do you know one or have any solution to help me doing this?

I was thinking about applying some kind of k-means on my DTM matrix, using a specific metric e.g the euclidian distance between 2
texts which I would "artificially" reduce when the 2 texts words match my primary list of words which are likely to appear.

I did not test it yet and I was wondering if this solution could be relevant or not.

Your direction of using k-means on term-document matrix is correct. To go further, you can

Use Weighted k-means

You can also try artificially boosting the weight for certain columns/variables (corresponding to your pre-specified list of words). This will force the similarity calculation to be dominated by those words.

Use Constrained k-means

You can read up more about constrained k-means algorithm: you can force certain points to be in the same cluster. In other words, if there are two documents containing words from the pre-specified list, they are forced to be in same cluster.

Alternative: Classification

If you know that there are only two clusters to be formed, why stick to only unsupervised learning? Depending on the application and availability of labelled data, you may think of it as a binary classification task using nearest neighbor or similar algorithm.


Getting answers to bullets (numbered items) from text via NLP This is related to information extraction. In real world data, documents are written in bullets/numbered items form. For example,

How to create a website:

 - Get A DNS
 - Get a Hosting
 - Deploy wordpress or some site ...

above is sample of a structured data. Take another example where content is semi structured,

While sandeep was going to home there was a road on the way he saw a

 - Car
 - 2 wheeler
 - cart

and he carefully crossed the road

If I have to find out the "steps to create a website" or "what is on the road"  .. is there an established method using NLP? As some data is semi structured, so simple classification may not work in this case.
Maybe deep learning network or some pretrained model? I have seen google is able to return such results when we search something like "what are steps create a website".

If we want to search for What are the steps to create a website, all you need to do is the find which set of bulleted points is going to answer that. One thing that you can try is that you can vectorize your question using some pretrained model (w2v, GloVe etc) and also encode the descriptions before the bulleted list ( For eg : How to create a website). Once you have the query vector $q$ and the descriptions vectors $d1, d2, d3,..., dn$ , you can perform cosine similarity of the query vector with all the description vectors and choose the description with the maximum similarity. Following this, the corresponding bullet points will be the answer.


I think brute force is the only method. There might be some complicated reductions but that would be outside the scope of the tutorial.

Brute force isn't unreasonable here since there are only 46656 possible combinations. Even python should iterate through it in a couple of seconds.

This is straight forward tree-search problem, where each node's values is a conditional probability. The hardest part of it is having to manually type all the conditional probabilities in.


Cost Function on non linear noisy data I was curious to see if one can use a cost function on a set of data points to find the "optimial minimum" solution for any given set of data.

I know for a regular set of data that is clustered symmetrically following a straight regression line is easy to find the proper cost function

Example:

but what if the data makes a funky shape like that of x^2 or 3*x^3 or etc...

As you can see that the the second graph is a lot more noisy than the first.

Then what do you do? is it the same process or is it different?
Can you find an optimal Minimum for any set of data points?

I know the cost function is using the residuals to form its best fit but I was just curious if it can do with any set of data.

I don't think the second picture necessarily has more noise, it's just less linear than the first one. If you are doing regression (which is the case in both your problems) you can use the same loss function to optimize your family of functions that you are using to fit your data. The only difference is that with certain function families gradient descent might not get to a global optimum but will get stuck in a local one. The loss function only determines how to penalize certain errors (based on the residuals).


Random Forest Prediction Let's say I want to classify if the employee will churn or not. In my random forest, I have 6 estimators where 3 of them predict the employee to churn and the other estimators predict the employee to retain. In short, 3 estimators = predicted to churn and 3 estimators = predicted to retain.

I know that the random forest works by getting the most votes for each class or by getting the mode.

The question is how does the random forest predict in this case?

Seems like it predicts the first class. Sklearns Random Forest implementation generates probabilities for each class by averaging the probability predicted by each estimator into an array of shape (n_samples, n_classes), and then uses np.take(np.argmax()) to select the highest probability, something akin to this:

# Pretend "a" is our averaged predictions for the forest. So the first sample is predicting 78% probability
class 0, 22% class 1. The second has the probabilities reversed and the third is 50/50 split.

a = np.array([[0.78, 0.22], [0.22, 0.78], [0.5, 0.5]])

np.argmax(a, axis=1)

The output is array([0, 1, 0], dtype=int64). These are the indices of the highest value in each sample of the array, and for the sample with an even split you can see it's picking class 0.

edit:

If you want to see it yourself, the relevant piece of code is line 540 in forest.py of sklearn:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/forest.py


How to modify the resnet 50 in Pytorch (pre-trained) to give Multiple Outputs for Multi-Label Classification class ResNet(nn.Module):
def __init__(self):
    super(ResNet, self).__init__()
    resnet = models.resnet50(pretrained=True)
    modules = list(resnet.children())[:-1]
    self.resnet = nn.Sequential(*modules)
    self.fc=nn.Linear(2048,10)

def forward(self, x):

    x1 = self.resnet(x)
    x1=self.resnet(x)
    x1 = x1.view(x1.size(0), -1)
    x1=self.fc(x1)

    x2 = self.resnet(x)
    x2= x2.view(x2.size(0), -1)
    x2=self.fc(x2)

    return x1,x2

A softmax layer helps:

class ResNet(nn.Module):
def init(self):
    super(ResNet, self).init()
    resnet = models.resnet50(pretrained=True)
    modules = list(resnet.children())[:-1]
    self.resnet = nn.Sequential(*modules)
    self.fc=nn.Linear(2048,10),

def forward(self, x):

x1 = torch.softmax(self.resnet(x), dim=-1)
#x1=self.resnet(x)
#x1 = x1.view(x1.size(0), -1)
#x1=self.fc(x1)

x2 = torch.softmax(self.resnet(x) dim=-1)
#x2= x2.view(x2.size(0), -1)
#x2=self.fc(x2)

return x1,x2

To know more about the dim attribute, corss reference here: https://stackoverflow.com/questions/52513802/pytorch-softmax-with-dim

I hope this what you were looking for.


Regex in R as a list for Quanteda R newbie here. I'm doing some text analysis using the package quanteda. Basically, what I'm trying to do is put all the words follow the regex pattern child|(care) basically to capture any text which includes any of the words "child" or "care".
To do this, I can create a list and then use the dictionary function:
childcare_list <- c("child","care")
word_dict <- dictionary(list(childcare = childcare_list)).
However, how could I incorporate regex and do this for other patterns which would be tedious to type up manually as in the first line? For example, I may want to capture something like
\bC\w?V\w?D\-19 which captures possible typos of "COVID-19" e.g. "CiVID-19", "CpVID-19".
I could of course do covid_list <- c("CiVID-19", "CpVID-19", ...) but that would be too manual. As well, it doesn't use the \b anchor.
Basically, asking if it's possible to make it so a list contains all possible combinations of a regex.

This doesn't seem like a great task for regex--even your pattern would miss very close typos like COWID-19 or potential OCR mistakes like C0VID-I9. Instead, I'd suggest using the stringdist package to do fuzzy matching, perhaps stringdist::afind to find approximate matches of "COVID-19". You can read a bit about it here.
This will let you select from a variety of string distance algorithms and set a maximum distance. You could then, e.g., correct matches to "COVID-19" and proceed with your analysis.