deep learning output data in keras fit method if I have following data set and I want to predict all three feature How should I prepare the output data ?

 a      b    c
2265.0  8   21.3515625
2264.5  8   21.3515625
2268.0  8   21.3046875
2266.5  8   21.328125
2260.5  8   21.328125
2259.0  8   21.328125

I know Keras takes 3d input. Therefore the input needs to be reshape. There is a timestamps as well.So the output/label would be all except the time stamp in the dataset ?

If you are talking about time-series analysis, your test data should be the three columns you want to predict, but shifted by some amount. So the first batch (e.g. first 5 rows) predicts the 6th row. Here is a small diagram showing what I mean:

Notice that the timestamp column not in the target, and that we predict the three features in the following row, given an input of 5 rows.


AttributeError: module 'keras.backend' has no attribute 'backend' Hi I have tensorflow installed of version 1.14.0 on my ubuntu machine. I am trying to run a code with

import keras

and I get an error

AttributeError: module 'keras.backend' has no attribute 'backend'

i have tried installing specific version of keras(2.0.8) because i have seen an answer which says keras 2.0.8 is required for tensorflow 1.14.0 . But still I get the error.

Can someone help me with this. Really stuck at a vey basic point

Do you have multiple versions of keras installed? Did you try reinstalling? Sorry for posting as awnser but I don't have enough reputation to comment

pip uninstall keras
pip install keras


In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document and word representations? In Latent Dirichlet Allocation (LDA), is it reasonable to reconstruct the original bag-of-words using the document-by-topic and topic-word inferred matrices?

I understand that I will not get frequencies by reconstructing the original matrix, but is the non-zeros after reconstruction valid?

It is possible to produce a corpus from the learned LDA parameters ($\theta$ and $\phi$) according to the generative model of LDA but it is not realistic to expect that you would recreate the original documents (in bag-of-words form). To be more specific, it is possible - but highly improbable - that you would generate the bag-of-words documents corresponding to the input corpus.


K-Means anomaly detection not clustering anomalies K-means anomaly detection scatter plot

The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset.

import pandas as pd
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import seaborn as sns

X=pd.read_csv('C:/Files/dataset.csv', sep=';', encoding='latin1' )

#Adding the anomalies

for i in range(0, 50):
    X.append(X.my_column.max() * (10 + pd.np.abs(pd.np.random.normal())))
X = pd.np.array(X)

clf = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=1)
clf.fit(X.my_column.values.reshape(-1, 1))
X_prd = clf.predict(X.my_column.values.reshape(-1, 1))

plt.scatter(X.index, X.my_column, c=X_prd)

The picture bellow shows the results and I was expecting outlier cluster to be clear compared the normal data.

Why so ?

Because for creating the anomalies I took the maximum value of my_column which was  9689.

I am stuck here and I don’t know where to do from here, so I would appreciate some help.

The goal is that K mean to detected these added anomalies.

By multiplying the maximum by 10 in a loop, you are repeatedly multiplying by $\sim 10$, so that the final point is $\sim 10^{53}$ (hrm, the plot actually doesn't go quite that far?).  Hence the last added point is so far away from the rest, it becomes too costly to include any other points in that cluster.  (Doing so drags the mean of the cluster very far away, and then the cost function blows up; see the sklearn manual.)


What would cause a Hierarchical Cluster to look like skewed (if this is the right term to use!)? I am surprised about this hierarchical cluster! For me, it looks somehow abnormal. Or, maybe normal but I am not able to identify why it looks like this. Any idea why data would be clustered in such a structure?

This happens if:

There is no clear way to separate the clusters, i.e. there is only one large group of instances and the instances which are far from this group are too far from each other to form their own cluster.
The linkage criterion plays a big role: this is more likely to happen with single linkage clustering , because as the main cluster grows it becomes more likely to be the closest point to any other instance, therefore all the instances are "attached" to this cluster one by one. This is less likely to happen with complete-linkage clustering, but this option has other biases.


Simple Binary Classification Example in Python I'm not sure the correct place to ask, but I'm trying to develop a simple function/algorithm that outputs a predicted number from a sequence of numbers (I have a background in Python, but little to no experience working with data prediction). More specifically, when there is only binary choice. I looked around online, but I'm really lost and couldn't quite find something discussing it:

Suppose there's a bunch of 1s and 0s in sequence, like:

0100010010111011010010001011010

Suppose there is some complex math pattern that is determining the sequence, even though it is unbeknownst to us observers. However we know for certain that this data is not random, and contains no anomalies.

What sort of function can be used to predict if the next digit is a 1 or 0?

For example:

01010101 -> the next digit predicted should be 0.

001100110011 -> the next digit predicted should be 0.

Is there some simple Python function (using numpy, pandas, etc.) that can predict the next 1 or 0, no matter the pattern?

The problem you are describing is a type of sequence prediction.

I think this is too complicated for any numpy or pandas library to solve. Maybe sklearn has something close, but I think a neural network would give you the most accurate predictions.

Before deciding, some of the points to consider are:

Is there a limit on max length for the string pattern you are
looking for?
Do you only want one prediction, based on one pattern that is
considered most likely, or do you want to consider multiple
patterns?
Others

Examples:
001001001 -> next digit should be 001
01001010010 -> there are 2 subpatterns and another larger pattern. next digiti could be 50% chance of 0 (01 pattern) and 50% chance of 1 (001 pattern)

I think the best approach is to let an LSTM find any patterns and predict the next digit based on the model it built. The model will contain all the information mentioned above without you needing to define them.

Depending on how simple or complicated you want to make this, you could start with a simple LSTM model and go from there.

https://machinelearningmastery.com/sequence-prediction/

or

https://machinelearningmastery.com/models-sequence-prediction-recurrent-neural-networks/


What is the best machine learning model to predict a continuous variable where the predictors include categorical, numerical variables and a text? The variables include categorical variables like (contains video, author) and numerical variables like (average word length) and a text (combination of words).
I am confused about this because from what I understand only regression can be used to predict continuous variable. However, I was thinking of forming a bag of words array for the text and title and I do have other categorical variables so i am not sure which model would be ideal. Which ML model would be the best to predict a continuous variable?

I believe this is what you want to know, the approach to handle your text data. You can do one hot encoding for your categorical data if they are not ordinal or label encoding for ordinal. As for model, I am also a newbie but I think you can't define a rule of thumb that this model is best for regression. It depends on the data, which model will work.


Should the data be shuffled on a translation dataset As reference to Why should the data be shuffled for machine learning tasks I was curious if this is the case for neural-machine-translation.

I would like to analyze why I think this is the case for the 3 components (train, test, val) parts where the data (as a reference dataset think of the europarl-dataset) is split:

for the training part, when doing mini-batch gradient descent it is better that for each update we collect data that are closer semantically, so that the learning gets focused each time on a certain direction. For example is better to have a lot of topics for let's say coffee together for a single update, than topics about drugs or coffee, dogs etc in a single update
for the test part, we would like to see how the model behaves in a big text of continuous sentences, as the goal of translation is that of documents. The same should hold for validation

I would like your view on that if you have any empirical or theoretical insight of why this is not the case for neural machine translation.

Yes, in NMT data is always shuffled.

For training, if each batch contains the same "type of content" (e.g. domain, register), then your model will be biased toward the type of content of the last batches. You don't want that. Apart from that, in NMT data is normally bucketed by length to avoid wasting batch space with padding.

For the test data, you would like to have data that follows a similar distribution to the one to be used in production. If the test is biased to only one domain, then you won't be able to assess correctly the expected performance of the model.

Document-level translation has little to do with the shuffling of the data. In document-level MT you normally have context sentences that are either ingested with an auxiliary encoder or as part of a single, longer sentence. This may reduce the number of sentences you can fit in the batch, but each sentence (+context) is independent from the rest.


What is a better input for Word2Vec? This is more like a general NLP question.
What is the appropriate input to train a word embedding namely Word2Vec? Should all sentences belonging to an article be a separate document in a corpus? Or should each article be a document in said corpus?
This is just an example using python and gensim.

Corpus split by sentence:

SentenceCorpus = [["first", "sentence", "of", "the", "first", "article."],
                  ["second", "sentence", "of", "the", "first", "article."],
                  ["first", "sentence", "of", "the", "second", "article."],
                  ["second", "sentence", "of", "the", "second", "article."]]

Corpus split by article:

ArticleCorpus = [["first", "sentence", "of", "the", "first", "article.",
                  "second", "sentence", "of", "the", "first", "article."],
                 ["first", "sentence", "of", "the", "second", "article.",
                  "second", "sentence", "of", "the", "second", "article."]]

Training Word2Vec in Python:

from gensim.models import Word2Vec

wikiWord2Vec = Word2Vec(ArticleCorpus)

The answer to this question is that it depends. The primary approach is to pass in the tokenized sentences (so SentenceCorpus in your example), but depending on what your goal is and what the corpus is you're looking at, you might want to instead use the entire article to learn the embeddings. This is something you might not know ahead of time -- so you'll have to think about how you want to evaluate the quality of the embeddings, and do some experiments to see which 'kind' of embeddings are more useful for your task(s).


Why my Decision Tree Model is not further bifurcated in to Age I've created a Decision Tree Model for expected candidates that buyed Villa.

Though the Drilling down Green highlighted box, show data has Age in it.

Question- But why my model is not bifurcating it further into Age?

Deposit & Age both are having content type as Discretized as DicretizationBucketCount as 5 & 10 respectively.

Source Data Link

You have your Decision Tree set to a default expansion of 3 levels. The Decison Tree works with the most significant features first, and it has not deemed age as important. Increase the default expansion to more levels to see the bifurcation of age.

A word of caution: deep decision trees are prone to overfitting.


Filters in convolutional autoencoders I have a question regarding the number of filters in a convolutional Autoencoder.
As far as I have understood, as the network gets deeper, the amount of filters in the convolutional layer increases.
A typical pattern would be to $16, 32, 64, 128, 256, 512 ...$

Right now I am looking into Autoencoders and on the Keras Blog I noticed that they do it the other way around.
They start with $16$ filters in the first layer, then the number of filters is decreased:

https://blog.keras.io/building-autoencoders-in-keras.html

Now I am wondering if this is usual when working with Autoencoders or does it depend on what kind of features the network should learn?

Thanks in advance,

Cheers,

Michael

Autoencoders are meant to reduce the dimensionality of your data. Increasing the number of filters would do the opposite.


Boxplots or violinplots? This is quite a general question, perhaps somewhat opinion-based.

In most papers people use boxplots to visualize a certain distribution, yet violinplots are able to give more information. Violinplots are made by performing a kernel density estimation on your distribution.

Are there objective arguments to use one over the other? Or maybe there are specific situations in which one would prefer one over the other?

An example of both can be found here:

Hugely dependent on both user and audience preference (violin plots being more unusual could throw people), so it is mostly up to you.

One main reason to go for a violin plot is to give more detail about the distribution, as box plots just give hard stops at the mean, stddev and 2 stddevs. Therefore if you think there is interesting information contained in the distribution between those points go violin.

The other main reason is they're more eye catching on Kaggle ;)


What is difference between leave one subject out and leave one out cross validation What is the difference between leave one subject out cv and leave one out cross validation (loocv)? are they same or different?. I have images of 24 subject and according to literature, leave one subject out is best cross validation for pain expression detection because  of its subjective nature.
is there any function for leave one subject out cv in sklearn like loocv? or I have to manually hold out one subjects images  (one subject out of 23) and train for rest 23 subjects? is it right approach ?

Leave one out subject makes it sure that you don't have subject bias.

The fact that you have the same subject in your training and your testing datasets will make the model know more about your subject than it should. With a brand new subject, the model will probably perform poorly because it never trained on the subject before.

There is no such tool available in sklearn, as it's very easy to build your own if the subject id is a column in your dataset.


What is the reason behind having low results using the data augmentation technique in NLP? I used Data augmentation technique on my dataset, to have more data to train. My data is text so the data augmentation technique is based on random insertion of words, random swaps and synonyms replacement.

The algorithm I used performs well in other datasets, but in mine, it gives lower accuracy results comparing to the original experiment. Are there any logical interpretations?

Text data is at the same time:

very structured, because swapping only a few words in a sentence can make it complete gibberish,
and very flexible, because there are usually many ways to express the same idea in a sentence.

As a consequence, it's very hard to have a text sample which is representative enough of a "population" text, i.e. which covers enough cases for all the possible inputs. But augmentation methods are practically sure to fail, because either they are going to make the text gibberish or just cover minor variations which don't improve the coverage significantly.

That's why a lot of the work in NLP is about experimental design and preprocessing.


Tensorflow F-RCNN first stage input and output strides Trying to optimize performances on Tensorflow's faster_rcnn_resnet50 (from the model zoo), I'm currently working on understanding the full .config file they provide, and I'm having a hard time with the strides.

Here's the relevant part of the file, with bolded parts being things I'm unsure about:

image_resizer {
  keep_aspect_ratio_resizer {
    min_dimension: 1134
    max_dimension: 2016
  }
}
feature_extractor {
  type: "faster_rcnn_resnet50"
  first_stage_features_stride: 16
}
first_stage_anchor_generator {
  grid_anchor_generator {
    height: 28
    width: 31
    height_stride: 14
    width_stride: 15
    scales: 0.242
    scales: 0.621
    scales: 1.0
    scales: 1.739
    scales: 2.478
    aspect_ratios: 0.386
    aspect_ratios: 0.693
    aspect_ratios: 1.0
    aspect_ratios: 1.464
    aspect_ratios: 1.929
  }
}

What I would like to know is if I should be computing the feature map stride according to the anchors' dimensions and strides or if they're completely unrelated ? Also is there any restriction on the strides themselves, for example keeping them in powers of 2 like the original f-rcnn ?

Let me try to explain what exactly stride means generally, then you'll be able to address your specific problem.
Let's say we have an images of size 7x7. Let's take a kernel of size 3x3. When you slide the kernel over image with :
1. stride=1

2. stride=2

Essentially stride means how much gap you should leave between two kernel position while applying convolution operation.
Now this concept is generic and applies to any operation where we need to slide a kernel over an input.

For Faster-RCNN model, features and anchors are two different things, so probably they're unrelated. Confirm this with official paper though.
For stride size, there is no restriction, you can take it whatever number you want, just make sure it should be less than or equal to half of the size of image(you can think why logically).


Text representation using TFIDF .toarray() freezes my computer. Too many features to handle? I'm new to Data Science, so hopefully this question makes sense.

I have a dataset with ~50,000 rows. It consists of one column that has item category in it and one column with item description in it.

I'm trying to get bigrams from the item description with the highest correlation to each category.

i'm using TfidfVectorizer to do this.

I compress the item descriptions into an array using this:

features = tfidf.fit_transform(df.ItemDescription).toarray()

In Spyder, when i even click on the features under variable explorer, the memory immediately shoots up to 99% and my computer freezes.

If I try to call features at all, it will also freeze.

Here is how i'm calling features later in my code after making it into an array:

from sklearn.feature_selection import chi2
import numpy as np
N = 2
for UNSPSC, category_id in sorted(category_to_id.items()):
  features_chi2 = chi2(features, labels == category_id)

The array size is 50000X12000

Is that too big? Is there anything I can do to get the bigrams without my computer freezing? As a last resort, I will trim down my data. But i feel like that will reduce the accuracy of any model i try to train later.

Please let me know if you need full code.

If your downstream algorithm can work with sparse matrices, you can proceed with sparse matrix rather than using toarray.
Alternatively, try to minimize the number of features which are created. Check documentation of TfidfVectorizer (parameters like min_df, max_df, stopwords) to filter out un-important words.


batch_size in neural network When NN is construsted, batch size is not defined and place holder is used and its summary(tensorfow) shows the batch size as None.
This is useful because you can change batch size later.
In case of a simple model with 10 input features, 1 hidden layers with 10 neurons and output layer, the shape of the hidden layer would be (None, 10), which means if the batch size is 20, the hidden layer would have the shape of (20, 10).
When the model is used to predict for a single output, with shape(10, 1), how does the math work?

I guess you have a confusion here. The None part represents the number of samples.
For example if you have a neural network with architecture 100-50-10, it means that you have

(None,100) : input layer shape
(100,50) : shape for weights connecting input to hidden layer
(None,50): shape for hidden layer given by (None,100)*(100,50) matrix multiplication
(None,50): shape after nonlinearity application.
(50,10):  shape for the shape matrix between hidden and output layer
(None,10) : output layer shape (None,50)*(50,10) matrix multiplication

So if youre feeding a single input sample the shapes would be:
(1,100)[Input] => (1,100)(100,50) = (1,50)[Hidden Layer] => (1,50)*(50,10)=(1,10)[Output Layer]


Why perceptron does not converge on data not linearly separable This is how I understand the perceptron algorithm.

The perceptron loss function is the hinge loss $\ell(w,x,y) = \max(0, -yw\cdot x)$.
Suppose the data set is $D = \{(x_1,y_1),\dots,(x_n,y_n)\}$ with $y_i \in \{-1,1\}$.
The perceptron criterion is
$$
E(w) = \sum_i \ell(w, x_i, y_i) = \sum_i \max(0, -y_iw\cdot x).
$$
The perceptron algorithm tries to minimize $E(w)$ using gradient descent.
The gradient of $E(w)$ with respect to $w$ is
$$
\nabla E(w) = \sum_i \left\{
\begin{array}{ll}
\mathbf{0} & \text{ if } \ell(w,x_i,y_i) = 0\\
-y_ix_i & \text{ otherwise }
\end{array}\right..
$$
So the perceptron updates the $w$ according to
$$
w^{(t+1)} = w^{(t)} - \eta \nabla E(w^{(t)}) = w^{(t)} + \eta \sum_{i\in M(t)} y_ix_i
$$
where $M(t)$ is the set of points misclassified by $w^{(t)}$. Since we can define another equivalent sequence of weight vectors $v^{(t)} = (1/\eta)w^{(t)}$, we can just take $\eta = 1$.

I know that $w^{(t)}$ will converge to a seperating hyperplane if one exists and $E(w)$ will converge to $0$.

My question is about when the data set is not linearly separable. In that case, does $E(w)$ converge to some minimum?

If $E(w)$ attains some minimum, its gradient there has to be $\mathbf{0}$. One possibility is that all the terms in the sum of $\nabla E$ are 0, which means that the dataset is linearly seperable and this is contradiction. The other possibility is that the sum of $-y_ix_i$ over the misclassified points is somehow $\mathbf{0}$.

The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by the simple scaling you alluded to.
With appropriately small learning rates though, it seems you are guaranteed convergence to some local minimum, if you avoid certain degenerate situations that would give e.g. a constant loss surface.  Exactly what you describe happens at these minima: the loss to the misclassified points of either class equal each other.
I put together a short demonstration in this colab notebook (github link).  Below are some animations of the evolution of the decision line during the gradient descent, starting at the top with a large learning rate and decreasing it from there.  In this situation, there should be two local minima, the two "tangents" to the "blobs" (possibly pulled in a little, misclassifying a couple of extra points if it can decrease the loss on the central points a little more than the cost for the newly misclassified points).


Dealing with multiple distinct-value categorical variables So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values.

For instance, one column have more than one million unique value, it's an IP address column in case anyone is interested. Someone suggested to split it into multiple other columns using domain knowledge, so split it to Network Class type, Host type and so on. However wouldn't that make my dataset lose some information? What if I wanted to deal with IP addresses as is?

Nevertheless, the domain knowledge solution might work on the IP column, however, I've got other columns that have more than 100K distinct values, each value is a constant-length random string.

I did work with Embedding Layers before, I was dealing with max thousands of features, never worked with 10K++ features, so I'm not sure if that would work with millions.

Much Regards

Have you heard of CatBoostClassifier?

https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_catboostclassifier-docpage/

It is type of Boosting classifier developed to deal specifically with categorical features. It has achieved state of the art results and the package developed by the authors have excellent support and even GPU portability. Take a look, this can be your solution.


Predicting High-School test scores after a disciplinary action I'm somewhat new to machine learning and have learned to apply many of the basic regression and classification methods using python and various packages.  However, approaching this problem has me stumped.  To illustrate the problem, I created a fictitious scenario where a guidance counselor wants to predict test scores for a student after disciplinary action.  Suppose they have data available like the mock-up below:

Column definition:
Student - Student Identification #
Gender - Male/Female
Age - Current Age
Athlete - Sport that student plays (0-None, 1-Basketball, 2-Football, 3-Soccer)
Online - Student takes classes online (0-No, 1-Yes)
Before_Disciplinary_Action_Scores - Sequence of their last n test scores prior to discipline (in date order)
Discplinary_Action - Action Counselor Took (0-None, 1-Assigned Tutor, 2-Guardian Meeting, 3-Study Program, 4-Ineligible for game)
After_Disciplinary_Action_Scores - Sequence of their next n test scores after discipline (in date order)

Assume the fictitious school system is quite large and there are about 80k records in total.  All students have a total of 12 test scores from Before/After discipline action, but the number of test scores varies based on when the disciplinary action was given.  For simplicity, you can assume the scores are given on a weekly basis for a single course.

I can compute mean before/after scores and create a fairly good classification model, but I'd like to take it a step further and predict the post-discipline scores.

I've tried using Prophet and LSTM models to predict the after scores from a time series approach with poor results and the varying number of scores makes it difficult.  I can see that the features around Athlete, Transfer, and Online are important components and I tried adding them as regressors, but that too has failed.  I appreciate any guidance you can give.

If your data is correct then i see a factor of gender also. I hope that is taken into account and also take Disp_action as categorical variable (one hot encoding).
Also, try to club pre and post action scores in one fields as output variable. Another field can simply tell if the output score is pre action or post action.
This feature engineering, I hope will give better results.


Splitting train/test sets by an identifier? I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, the random seed will reset with each updated dataset and take a different train/test split. Doing this, your ML algos will eventually cover the whole dataset, defeating the purpose of the train/test split because it'll eventually train on too much of the whole dataset over time.

The book I'm reading (Hands-On Machine Learning with Scikit-Learn and Tensorflow) gives this code to split train/test by id:

# Function to check test set's identifier.
def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

# Function to split train/test
def split_train_test_by_id(data, test_ratio, id_column):
    ids = data[id_column]
    in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
    return data.loc[~in_test_set], data.loc[in_test_set]

And it says when there's no ID column given, to create one either by indexing the rows or creating a unique index from one of the variables.

My questions are:

What is the 3rd line doing:

crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32
What is the anonymous function doing in the 2nd to last line?

lambda id_: test_set_check(id_, test_ratio)
In practice, do you commonly split datasets by id in this manner?

Thanks,

Greg

In Auriel Geron's book, there is a short description of the approach:

you   could compute a hash    of  each    instance’s  identifier, keep    only    the last    byte    of  the hash,   and put
the   instance    in  the test    set if  this    value   is  lower   or  equal   to  51  (~20%   of  256).   This    ensures
that  the test    set will    remain  consistent  across  multiple    runs,   even    if  you refresh the dataset.
The   new test    set will    contain 20% of  the new instances,  but it  will    not contain any instance
that  was previously  in  the training    set.

While a full explanation of what exactly happens and why is probably best placed on StackOverflow, I can try to answer your questions, first with some background info.
The method uses a cyclic redundancy check, which is a method of checking that the raw blocks of memory have not been damaged/changed. It is a way to ensure data integrity, e.g. in network traffic - checking if a message way altered between being sent and received.
For train/test splits, it is checking the unique identifier of each sample. We have a column that gives each sample an ID - this should never be changed! Don't delete rows, only append to the end with new unique IDs.
In this part: test_ratio * 2**32, the part $2^{32}$ represents the largest integer of a 32-bit system.

0xFFFFFFFF is a large number; it's the hexadecimal representation of $2^{32}-1$

To answer your questions:

What is the 3rd line doing:

crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

Based on the information I gave above, we see the crc32 function finds the checksum value in memory (of the unique identifier). If we know the unique ID has never changed, then We ensure that crc32(np.int64(identifier)) & 0xffffffff will always return exactly the same numeric value, across all Python versions and platforms.
Imagine we give IDs in the range 0-80 for train, and 81-100 for test. No we want to make sure a sample'd s ID falls in the first bucket. We check its ID is simple less that 81, right? Well the numeric value we made above is checked to be less than our test_ratio * 2**32, where 2**32 is the largest 32-bit number. It checks that the sample's ID is within the range of train data, not in the test bucket:: > test_ratio * 2**32.

What is the anonymous function doing in the 2nd to last line?

lambda id_: test_set_check(id_, test_ratio)

This simply applies our test_set_check function to each sample's unique identifiers. Using the apply methd on a Pandas Series object (here it is one column of a Pandas DataFrame).

In practice, do you commonly split datasets by ID in this manner?

Not really... Scikit-Learn's train_test_split is often good enough. I think there are many other ways to remove bias and errors from your models before worrying too much about the impact of random splits.
For example, the snoop bias, whereby you analyse the entire dataset yourself before deciding on a model architecture/pipeline, thereby incorporating knowledge of the entuire distribution, which is inherently biasing our model.
There is also bias in overfitting e.g. in sequential imaging data (think frames of videos) such that the background is consistent, even though the objects you might want to detect are not. Your model will learn what to expect based on the background, which is not robust! Here you might look into using a geographical split (not random at all).

On a side note, there is also a slightly robuster way of setting random seeds in Python (instead of using NumPy's random seed generator). Have a look here for some differences.

Helpful resources:

https://stackoverflow.com/questions/36819849/detect-int32-overflow-using-0xffffffff-masking-in-python
https://pynative.com/python-random-module/
https://stackoverflow.com/questions/30092226/how-to-calculate-crc32-with-python-to-match-online-results
https://stackoverflow.com/questions/49331030/bitwise-xor-0xffffffff/49332291#49332291


How to set parameters to search in scikit-learn GridSearchCV I want to use scikit-learn's GridSearchCV to optimise a BaggingClassifier that uses a support vector classifier (SVC). I want the grid search to search over parameters for both the BaggingClassifier and the SVC.

I have tried this setup:

svc_pipe = Pipeline([
     ('svc', SVC(probability=True)),
])
pipe = Pipeline([
     ('bag', BaggingClassifier(svc_pipe, no_estimators=50)),
])

params = {
    'bag__bootstrap_features' : [True, False],
    'bag__svc__kernel': ['linear', 'rbf'],
    'bag__svc__decision_function_shape': ['ovo', 'ovr']
}

rnd_search = GridSearchCV(pipe, param_grid=params)

but I get this error:

ValueError: Invalid parameter svc for estimator BaggingClassifier(base_estimator=Pipeline(memory=None,
     steps=[('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=True, shrinking=True,
  tol=0.001, verbose=False))]),
         bootstrap=True, bootstrap_features=True, max_features=1.0,
         max_samples=1.0, n_estimators=50, n_jobs=-1, oob_score=False,
         verbose=0, warm_start=False). Check the list of available parameters with `estimator.get_params().keys()`.

Can someone show me what I have done wrong?

There is a typo in pipe, no_estimators should be n_estimators.

To address your problem, if you run the following piece of code:

for param in rnd_search.get_params().keys():
    print(param)

This will show you how the parameters are passed to different parts of the pipeline, the parameters of interest are:

bag__base_estimator__svc__kernel
bag__base_estimator__svc__decision_function_shape

So you were almost there, you were just missing base_estimator__ in the svc pipeline parameters. All you need to do is change the svc parameters like so:

params = {
    'bag__bootstrap_features' : [True, False],
    'bag__base_estimator__svc__kernel': ['linear', 'rbf'],
    'bag__base_estimator__svc__decision_function_shape': ['ovo', 'ovr']
}


Need suggestions on customer segmentation I have been tasked with performing customer segmentation for a Business to business use case based on customer purchase history. Can experts provide me inputs on how do I proceed with customer segmentation based on the following dataset
Dataset details which have been provided to me

Hierarchy 3,4,5 define the categories under which the product falls

Edit:
Also need inputs on how do i select features for my clustering algorithm?

So the question is about how to before customer segmentation on this data.
When I do any customer segmentation, I firstly think to myself, do I know how many segments prior to the analysis or not.
If I do,
Then I would use a clustering method like K-means clustering (https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1), where k refers to number of customer segments.
If I do not,
Then I would use something like agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/).
When it comes to data representation, you would represent a customer and their (purchasing) behaviours as a vector of values (I will refer to as customer vector).
If the variables are numerical (e.g. number of items purchases), then you can put the numerical values in the customer vector. If the variables are categorical (e.g. products purchased), then we concatenate a one-hot encoded vector (https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) of that variable to the customer vector.


Fuzzy name and nickname match I have a dataset with the following structure:

full_name,nickname,match
 Christian Douglas,Chris,1,
 Jhon Stevens,Charlie,0,
 David Jr Simpson,Junior,1
 Anastasia Williams,Stacie,1
 Lara Williams,Ana,0
 John Williams,Willy,1

where each predictor row is a pair full name, nickname, and the target variable, match, which is 1 when the nickname corresponds to the person with that name and 0 otherwise. As you can see, the way the nickname is obtained from the full name doesn't follow a particular pattern.

I want to train an ML algorithm that, given the pair full name, nickname, predict the probability of match.

My baseline is just trying to see the number of carachters that match, and features like that. However, I am thinking about an NLP approach using deep learning. My question is wether there are neural network architectures that are specific of this problem.

I had a similar problem in my last job. My solution was to build features via (transformation(s) + comparison) * many combos and feed to models, then aggregate and model, i.e. 2 layer model. The key is encoding and similarity scores as features.

Transforms: remove vowels (great for certain roots), remove end vowels, remove double characters, convert to phonetic string (IPA, soundex, https://pypi.org/project/Fuzzy/), replace characters that either sound similar or have different sounds in other languages ($J$ in East Europe sounds like $Y$ in US, $C$ can sound like $K, D~T, T\sim TH$, etc), ... The strategy is to handle lots of weirdness/irregularity in people's names.

Comparisons (similarity and difference): try [character level, block/root/[pre/suf]fix level, word level (may not apply to you)] similarity and difference scores. Try Dice's coefficient, Levenshtein, Needleman–Wunsch, Longest common (non)contiguous substring, character histogram similarity, # characters matching, not matching (each left and right), etc. You could try using an RNN/LSTM and have it learn similarity for each transform. Use the output of the trained model(s) as another feature.

Experiment with different combos of the above and select a few that seem to have value. You could simply take all the scores and fit with Logistic Regression (or Neural Net) or you could build statistical models and output percent rank based on a small training set to normalize it. Another way to preprocess the raw scores is to use calibration encoding via logistic function. Then add summary stats from the normalized scores as additional features. Push all this into the final model.

Will you handle names that are derived from Arabic, Spanish, French, etc names? This is just extra, but consider downloading the Social Security and US Census name stats data to enhance your project with more name variations. I'll leave the how to you, but it helps knowing about the likely possibilities. Be aware that simply using Levenshtein doesn't work so well with William->Bill, Dianne->Di, Larry->Lawrence, Mohammed->Muhamed and Hamed, Danielle->Daniela, Thomas->Tom, and Jimmy->James. The strategy I mentioned should help you with all the variation.

Additional resources to explore:
https://github.com/jamesturk/jellyfish
https://nameberry.com/list/276/If-You-Like-Danielle-You-Might-Love
https://pypi.org/project/phonetics/


Difference usage of document level, sentence level and aspect level in sentiment analysis I'm studying about Sentiment analysis. What is the purpose of using Document Level, Sentence Level and Aspect Level?

I read this definition in Web Data Mining book by Bing liu.

In this book it was stated that,

Sentence Level sentiment analysis is to classify a sentence to negative, positive, neutral class.

Document level sentiment analysis to classify a document.

Aspect Level sentiment analysis is to classify each aspect of entity mentioned in a review.

Thanks.

I am building on Toros91's answer. He has said:

  Use case of all these 3 different levels are:

Sentence Level: Used to classify a sentences for instance, whether a tweet is positive or negative or neutral.
Document Level: Used to classify the whole document level to say whether that document sentiment is positive or negative or neutral.
Aspect Level: Used to select an aspect and give the outcome of that respective Aspect, for instance let us consider Iphone X as an

  aspect and mine tweets(this is an example you can use documents and so
  on) and carry out Sentiment Analysis.

Now, to answer your question further about how it can be beneficial, just take an example of some review. I am just forming my own review on something as an example, I do not know the specifics, so please do not quote me here. This has some business logic implied as well.

  I feel the latest laptop from Mac is really good overall. It has
  amazing resolution. The computer is really very sleek and can slide
  into bags easily. However, I feel the weight is a let down. The price
  is a bit expensive given the configurations. I expect an SSD storage.
  However, the processor seems really good.

At a document level, I can say the entire paragraph looks more positive than negative. So, the review, if has to be labeled as one thing, is positive.

At a sentence level, I can break the document into 6 sentences and classify each of them as positive, negative or neutral. I get a bit more specific here.

Finally, at an aspect level, I get information about price, storage, processor and so on.

Given N customer reviews, you can know how well different aspects of a particular product are received. Say, 10 customers feel the price is fine, but 4 feel it is expensive. 13 of your 14 customers really love the resolution while 1 feel it is okayish. A lot of products in day to day life have multiple things to be considered. And this is where fine grained labeling help analysts or scientists.


Why transform embedding dimension in sin-cos positional encoding? Positional encoding using sine-cosine functions is often used in transformer models.
Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the embedding size. This positional encoding layer encodes $X$’s position $P \in R^{l\times d}$ and outputs $P + X$
The position $P$ is a 2-D matrix, where $i$ refers to the order in the sentence, and $j$ refers to the position along the embedding vector dimension. In this way, each value in the origin sequence is then maintained using the equations below:
$${P_{i, 2j} = \sin \bigg( \frac{i}{10000^{2j/d}}} \bigg) $$
$${P_{i, 2j+1} = \cos \bigg( \frac{i}{10000^{2j/d}}} \bigg)$$
for $i = 0,..., l-1$ and $j=0,...[(d-1)/2]$
I understand the transormation across the time dimension $i$ but why do we need the transformation across the embedding size dimension $j$? Since we are adding the position, wouldn't sin-cos just on time dimension be sufficient to encode the position?
EDIT
Answer 1 -  Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.
I still don't understand how the embedding dimensionality will be reduced to 1 if the same positional vector is added. Say we have an input $X$ of zeros with 4 dimensions - $d_0, d_1, d_2, d_3$ and 3 time steps - $t_0, t_1, t_2$
$$
\begin{matrix}
& d_0 & d_1 & d_2 & d_3\\
t_0 & 0 & 0 & 0 & 0\\
t_1 & 0 & 0 & 0 & 0\\
t_2 & 0 & 0 & 0 & 0\\
\end{matrix}
$$
If $d_0$ and $d_2$ are the same vectors $[0, 0, 0]$, and the meaning of position i.e time step is the same, why do they need to have different positional vectors? Why can't $d_0$ and $d_2$ be the same after positional encoding if the input $d_0$ and $d_2$ are the same?
As for the embedding dimensionality reducing to 1, I don't see why that would happen. Isn't the embedding dimensionality dependent on the input matrix $X$. If I add constants to it, the dimensionality will not change, no?
I may be missing something more fundamental here and would like to know where am I going wrong.

First, let's reason why positional embeddings are needed at all:
A multi-head attention layer of the Transformer architecture performs computations that are position-independent. This means that, if the same inputs are received at two different positions, the attention heads in the layer would return the same value at the two positions.
Note that this is different from LSTMs and other recurrent architectures which, apart from the input, receive the state from the previous time step.
The role of positional embeddings is to supply information regarding the position of each token. This allows the attention layer to compute results that are context-dependent, that is, two tokens with the same value in the input sentence would get different representations.

Second, let's clarify why having a fixed formula to compute the positional embeddings:
Positional embeddings can be handled as "normal" embedding matrixes and
therefore can be trained with the rest of the network. These are "trainable positional embeddings". With this kind of positional embeddings, after each training step, the positional embedding matrix is updated together with the rest of the parameters.
However, we can obtain the same level of performance (translation quality, perplexity, or whatever other measure being used) if, instead of training the positional embeddings, we used the formula proposed in the original transformer paper.
This saves us from having to train a very big embedding matrix.

Now, about why using the "embedding size dimension" in the formula:
We need different values in each position of the embedded vector. Having the same value in each position of the vector would leave us with an "effective" embedding size of 1, as we are wasting the other $d-1$ positions.
In order to compute different values for each position of the embedded vector, we need an independent variable that we use to compute the value at each position based on it. We don't have any other suitable variable but the position itself. That's why it is used in the formula.

Old answer:
Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.
The formula uses the embedding size dimension to be able to provide different values within each embedded vector.


PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a sentence, the result is a tuple with the form below:

[('__label__positif', '__label__négatif', '__label__neutre', 0.8947999477386475, 0.08174632489681244, 0.023483742028474808)]

but when I tried to apply it to the column "text" I did this :

from pyspark.sql.types import *
from pyspark.sql.functions import udf, col
import fasttext

schema = StructType([
    StructField("pos", StringType(), False),
    StructField("neg", StringType(), False),
    StructField("ntr", StringType(), False),
    StructField("pr_pos", DoubleType(), False),
    StructField("pr_neg", DoubleType(), False),
    StructField("pr_ntr", DoubleType(), False)
])

udf_label = udf(lambda words : predictClass(words), schema)
df = df.withColumn("classe", udf_label(col('text')))

df.select('classe').show()

I get this error:
PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

In FastText Users FB page a certain Maksym Kysylov answered me " It's not a FastText problem. It's a Spark problem :) When you apply function to Dataframe (or RDD) Spark needs to serialize it and send to all executors. It's not really possible to serialize FastText's code, because part of it is native (in C++). Possible solution would be to save model to disk, then for each spark partition load model from disk and apply it to the data. Something like: df.rdd.mapPartitions(func). And func should: 1. load the model; 2. for record in partition: yield ft.predict(record['text'])."   It works for me and I thank him very much !!!


Why decreasing the number of convolutional layers inside a CNN increases the number of parameters? I am building a CNN from scratch, and I am trying to change the number of convolutional layers to see what happens.
I have noticed that decreasing the number of convolutional layers increases the number of parameters.
This fact really surprised me, since I was expecting to see a decrease in the number of parameters.
Why decreasing the number of convolutional layers inside a CNN increases the number of parameters?

Generally removing the number of convolutional layers indeed decreases the number of parameters in a network, but given your situation I suspect you are using fully connected layers after the convolutional layers in your network. When removing some convolutional layers it means that the image/tensor size before being passed to your fully connected layers is larger and contains more pixels. Since each neuron in the fully connected layer is connected to all pixels in the layer before it the increase in the number of parameters caused by the larger image size is larger than the decrease in the number of parameters caused by the removal of the convolutional layers (leading to an overall increase in the number of parameters in your network).


Imputation missing values other than using Mean, Median in python I heard that Mean, Median isn't the best way to impute the missing values, why would that be?

In my scenario, I have data like this

Brand|Value
A|2,
A|NaN,
A|4,
B|8,
B|NaN,
B|10,
C|9,
C|11

if using mean imputation the data would be

Brand|Value
A|2,
A|7.3,
A|4,
B|8,
B|7.3,
B|10,
C|9,
C|11

which does make sense for brand B to be 7.3 but doesn't make sense if brand A 7.3 because the value of Brand A has its tendency somewhere around 2 and 8
is there any other way to fill the missing values based on the Brand?

This is an example of data with only 2 features, with 1 feature that may has pattern for missing values, what if there are like 20 features, and there would be more than one features that may have pattern to better define the missing values.

How to apply this in Python?

So if you want to impute some missing values, based on the group that they belong to (in your case A, B, ...), you can use the groupby method of a Pandas DataFrame. So make sure your data is in one of those first.

import pandas as pd
df = pd.DataFrame(your_data)              # read documentation to achieve this

Then, it is just a case of chaining a few steps together:

df["Value"] = df.groupby("Brand")["Value"].transform(lambda x: x.fillna(x.mean()))

df.groupby simply groups the dataframe into sub-dataframes (groups), such that each group only contains one Brand
transform() will apply a function to a dataframe - so to each of the individual groups created in groupby
the nameless function (a lambda function) calls the DataFrame's fillna() method on each dataframe, using just the mean() to fill the gaps

You can simply substitute the mean() method for anything you like. You could also create a more complicated function, ifyou need it, and replace that lambda function. It would simply need to take a dataframe as input and return a dataframe with a comparable index.


How were auto encoders used to intialize deep neural networks? In a document on deep learning about auto encoders, it is said that these networks were used back from 2006 to 2010 for deep neural networks initialization.

Can somebody explain how this was done?

There were a few different techniques. One popular one was stacked autoencoders, where each layer was trained separately.

Essentially this was done by progressively growing the autoencoder, two layers at a time (one encode layer, plus equivalent decode layer), followed by complete training at each step of growth.

If learning from a fixed training set, you could store the encoded representation of the whole dataset so far as input into next stage of training, saving some computation when building up the layers.

After training each encoder layer separately you could use the weights of the encoder section of the autoencoder as the starting weights of the deep NN. Intuitively this made sense as you would have a representation of the input that you knew could be used to reconstruct it, and that typically was compressed, so should in theory have extracted salient details from the training data population. On top of these pre-trained layers, you may add one or two new layers that implemented whatever classification or regression task that you needed the final NN to perform. Then you would train with the labelled data - this is similar to fine-tuning networks and transfer learning that is still done nowadays.

The results from this pre-training stage could be worthwhile. It is still a valid technique if you have a lot of unlabelled data, and a relatively small amount of labelled data. However, the introduction of ReLU and careful controls on weight initialisation meant that deep networks could often be trained more directly. Recent additions such as skip connections and batch normalisation have further improved more direct training approaches.

Here is an example with code, using TensorFlow.


How does tf.random.log_uniform_candidate_sampler work while generating negative samples? I was trying to build a simple negative sampler for a Word2Vec model using TensorFlow by following the tutorial here.
From what I understand, the tf.random.log_uniform_candidate_sampler() takes as input the correct class to be exluded from sampling, the number of samples to return, and the size of the distribution to sample.
In the tutorial, the true_classes is 1, range_max is 8 and num_ns is 4. Which translates to, "Return 4 samples from the distribution [0, 8), but skip 1, since it's a positive class."
However, in the tutorial, the sampler returns the following tensor: tf.Tensor([2 1 4 3], shape=(4,), dtype=int64).
My question is, why does the sampler return the class 1, even after being told to exclude it, since it doesn't constitute a negative sample?
Furthermore, in the following section of the tutorial:
target_index    : 7
target_word     : sun
context_indices : [1 2 1 4 3]
context_words   : ['the', 'wide', 'the', 'shimmered', 'road']
label           : [1 0 0 0 0]

We can see clearly that the first instance of class 1 (index 0 in context_indices) has been flagged as a positive skip-gram, whereas the second instance of 1 (index 2 in context_indices) has been labelled as a negative skip-gram.
I'd love some clarification or a correction in my understanding of this model or tutorial.
Thanks!

This parameter isn't used in this tutorial, is works other return values. you can understand this by read the 122ed line of the source code .
source code


Addressing polysemy in NLP tasks Looking for modern algorithms using NN Language Model implementations addressing polysemy in NLP tasks, including text classification, question answering and topic modeling. Transfer/Zero-short learning methods are most interesting to find. Any working solutions with BERT and Hugging Face Transformers libraries?

Maybe this article will help you How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings.
Talks about contextual word embeddings like BERT and GPT how they can capture various polysemous concepts rather than the static word embeddings which create a single representation for each word, such as GloVe.


How can I make ROC and compute AUC? I created a boosting tree and got the probability for each tuple in my testing set. But I'm confused on how to combine each probability. Can someone tell me how to combine the probabilities?

Is an interesting question.

You are solving a optimization problem

$max$
$AUC(\alpha_1*X_1 + \alpha_2*X_2,Y)$

$s.t.$
$\sum^2_{i=1}\alpha = 1$

You are maximizing the AUC using the models 1 and 2 (which are $X_1$ and $X_2$) subject to the sum of weights is 1.

You may any optimization algorithm, even Excel's solver to do this.


Stanford parser Python : Combine NER and POS tags Hi I am experimenting with stanford parser and NER with python

 Input = "Rami Eid is studying at Stony Brook University in NY"

Parser Output:

NER Output :

[(u'Rami', u'PERSON'), (u'Eid', u'PERSON'), (u'is', u'O'), (u'studying', u'O'), (u'at', u'O'), (u'Stony', u'ORGANIZATION'), (u'Brook', u'ORGANIZATION'), (u'University', u'ORGANIZATION'), (u'in', u'O'), (u'NY', u'O')]

Now can I combine NER results with Parser result ?

So that

(u'Rami', u'NNP'), (u'Eid', u'NNP') ==> u('Rami EID', u'PERSON')

(u'Stony', u'NNP'), (u'Brook', u'NNP'), (u'University', u'NNP') ==> (u'Stony Brook University',u'ORGANIZATION')

will get replaced in the graph

How this can be done ?

This question has already been answered. See Alexis' answer at https://stackoverflow.com/questions/30664677/extract-list-of-persons-and-organizations-using-stanford-ner-tagger-in-nltk

If you want to use chunking NER without the Stanford library see alvas' answer at https://stackoverflow.com/questions/31836058/nltk-named-entity-recognition-to-a-python-list


Best way to deal with realistically imbalanced dataset for Regression problem I have a dataset where each object has a label between 0-1. Objects with label = 1 are very common but those with label = 0 are very rare. I am interested in predicting the label in unseen data.

NOTE: The labels are continuous between 0 and 1 but binned for visual purposes.

These objects are from a simulation which accurately predicts the population numbers which we see in reality. (Hence the "realistically imbalanced" title)

Unfortunately, an object with label = 0 exhibits random looking motion and therefore the number of ways in which you can construct these objects tends to infinity..whereas those with label = 1 are highly ordered objects with not many variations in their structure and so have a limited number of ways in which they can appear.

To me this means that, in order to explore all possible variations of these objects in their domain would require a dataset who's "number of each class" would be the complete opposite as we see in the figure above. i.e. you need to observe exponentially more label ~ 0 objects than label = 1 objects in order to explore the objects' domain.

Obviously this is impossible and impractical. I'm curious as to what people think is the best method of approaching this issue?

I have thought about:

Making this an object detection method whereby you discriminate between objects with label greater than or below a threshold such as 0.8. (I think this would be difficult given that the labels are continuous and so you would get muddying at the boundary).
Joining two CNNs: one which models features that describe the label such as velocity data, and one which models properties that somehow link the level of physical randomness in the images to the corresponding labels.

I have already tried upsampling with a CNN classifier but with so few label ~ 0 images, the CNN quickly overfits and defaults to predicting them with a higher label.

You said that "an object with label = 0 exhibits random looking motion...whereas those with label = 1 are highly ordered objects". If you can see that, why should your model not be able to? Just train it and see if it works.

However, your dataset does not seem to be very large. You should think about transfer learning from some pre-trained model if your images/objects are not too strange.

There is no a priori need to handle an imbalance of labels, just be sure to compare your loss to a proper baseline, e.g. the label mean.

You said you have additional features, append them after flattening the convolutional layers. If the additional features are also 2D, just add them as a channel to your Input.


Carlification of the MSE loss sum symbol So I have a question regarding the MSE loss on the application of a Neural Network.
Loss function: $\text{MSE} = \frac{1}{2} \sum_{i=1}^{n} (Y_i - \hat{Y_i}) ^ 2$
I am wondering for what the $\sum_{i=1}^{n}$ stands.

Do I sum over the loss of all training examples for each output node in my Neural Network?
Or do I use a single training example and sum over all Neural network output nodes?
Or do I both and sum over all training examples and over all output nodes?

I want to use the MSE loss later than for updating my weights in the Neural Network. What would I do for that?

I think it depends on what you're doing.

If you want to find the MSE of just one training example, then you do number 2. sum all in one example.

If you want to see if it's stable or converging to stability, you could calculate MSE on all examples independently, and then average them to give you a sense of overall behavior


How do i create a classifier on sensor data? I am working on a indoor localization based on magnetometer.
I have 9 separate time-series datasets of sensor readings taken from coordinates 00, 01, 02, 10, 11, and so on until 22. Basically I am using my own coordinate system and gathered data. The coordinate system looks like this:
0,0 | 0,1 | 0,2

1,0 | 1,1 | 1,2

2,0 | 2,1 | 2,2

The dataset has columns timestamp, X, Y, Z and Magnitude. Here's what all my dataset looks like:

Edit:
I collected a sensor readings data at one place with coordinate say 0,0 and saved the file as  (coor00.csv) then at another place with coordinate say 0,1 and saved the file as  (coor01.csv) and so on.
What I want to predict is coordinates. I want to build a simple classifier that predicts the coordinate/place/area based on sensor readings.
Since I don't have any labels/tags so I thought of creating one myself by simply adding a column named label. This column will be my target variable. So in coor00.csv file I will add a column label which will have details of what coordinates the sensr readings were taken from and I will do the same for other datasets. Then I will combine 9 datasets into a dataframe and then run a classifier after splitting the dataset into train and test.
There are plenty resources out there, but I just want to know how/where to start. I want to know if this is the right way.
I plan on using RandomForest classifier but I would appreciate any suggestions on what kind of classifier algorithms should be used?

By looking at your datasets, it resembles a multivariate time series problem , not sure why you are opting for Random Forest classifier ?

I would suggest you to start implementing simple statistical
algorithm first  here is the link

Then explore on complex Deep Learning algorithm based on the data
set size , please find the referenced link

Thanks,
Durga


How to interpret correlation matrix? I have built a correlation matrix to check multicollinearity in a regression model. But how to interpret this?

Can we say that there is a certain correlation value from which the independent variables related to this correlation should be removed from the model? Is this the value over 0.5?
Are there any better solutions than removing the conflicting variables (when correlation value is too big)? I tried normalization but it didn't help.

Regarding 1) Usually you would check this based on "tests" like the VIF (see this post for details). I personally would not be too worried with a correlation of 0.5.
Regarding 2) Not really since you would need to "decorrelate" the variables and you can only do linear transformations (when you want to preserve the variable(s)). What you can do is to make projections, e.g. by principle components, but this will totally change the variable set.


Grouping of similar looking text I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same test but, due to minor changes in the titles, titles are shown as unique.

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]

labels = ['title', 'description']
import pandas as pd
labtest = pd.DataFrame.from_records(values, columns = labels) # Create data frame

This is how the data frame looks like. [The actual dataset has many such titles, this is only for the purpose of this question]

Title                       Description
Complete blood test         AB
COMPLETE Blood test\        AB
Blood glucose               AB
Complete blood picture      AB

And this is what I would the like the data frame to look like:

Title                       Description
Blood test                   AB
Blood test                   AB
Blood test                   AB
Blood test                   AB

I would like to search for the word "blood" in each title and if its true, then change the whole title with "Blood test". Is there a way to do this?

One possible solution is the following:

import re
import pandas as pd

pattern = re.compile('blood', re.IGNORECASE)

def change(text):
    if pattern.findall(text):
        return 'Blood test'
    else:
        return text

values = [('Complete blood picture', 'AB'), ('Complete BLOOD test', 'AB'), ('blood glucose', 'AB'), ('COMplete blood Profile', 'AB')]
labels = ['title', 'description']

# Create data frame
labtest = pd.DataFrame.from_records(values, columns=labels)
labtest['title'] = labtest['title'].apply(change)

print labtest

The output was:

        title description
0  Blood test          AB
1  Blood test          AB
2  Blood test          AB
3  Blood test          AB

The first line import the regex (regular expression) module of Python. The line:

pattern = re.compile('blood', re.IGNORECASE)

creates a regex that finds the word blood ignoring case. The function change, replace the input text with 'Blood test' in case the string 'blood' was found. Finally you used the apply method from pandas DataFrame to transform the column. Finally the apply method, as the name suggests, 'applies' the function change to every value in the 'title' column.

More info on regular expressions using Python and pandas apply method can be found here and here. If you want to know more about text processing in Python I would recommend you take a look at the pointers in this question.


What are default keras layer weights I build a classification model on keras (Vanilla MLP),It went quite well by using a tutorial.But what are the default weights of layers because I didn't mention any in the code.It has a feature of intializing weights, but what if we don't initialize.

It uses Xavier initialization. If you don't initialize it, by default it uses this method for initialization. If you want to know how to perform that you can use here. In the first link at the end, link of the paper is also available.

This Xavier initialization (after Glorot’s first name) is a neat trick that works well in practice. However, along came rectified linear units (ReLU), a non-linearity that is scale-invariant around 0 and does not saturate at large input values. This seemingly solved both of the problems the sigmoid function had.


How many run should we implement on the machine learning model? Theoretically, we can implement fix seed on the machine learning model to get the same results every run  (reproducible)but it may leads to bias. So, in order to prevent bias, I gonna run the model for several times without fix the seed/state. How many time should I run the machine learning model or normally done by researchers to get the result (10 runs,20.. or more)?

It depends on your problem, start with a big number (32 ?) then decrease if the model is stable / when you deal with stability.
You need to know that having multiple runs may be part of the solution: you can create an ensemble of models to get more stable models. Here again, the way to create an ensemble and the number of models you need to run to build an ensemble is determined experimentally. The main alternative to add stability to your model is trough regularisation (that really depend on the model you want to use).


Machine Learning: Predicting target based on a feature I have a df looks as follow:

-It is very likely that the same feature1Xfeature2Xfeature3 combination will appear multiple times.
-type values are in the range of 1..4.
I want to train a model, that for each feature1Xfeature2Xfeature3 combination as input, will predict a type s.t this type will match to the lowest price. e.g, if there are two rows with the same feature1Xfeature2Xfeature3 = [5.0, 3.6, 1.4] combination, one has price of 5000 with type 3, the other has price of 2000 with type 1, then inputting the model with feature1Xfeature2Xfeature3 = [5.0, 3.6, 1.42] (a combination which is very close to the above one) will likely output 1 since it has lower price.
I've got some experience with models and learning using python and sklearn, but it sums to training a model with one target, which is not based on a specific feature.
The final goal is to try and fit several algorithms on this problem(KMeans, DecisinTree, NN..), and check who did the best.
Any help will be appreciated.
Thanks

One quick thought is to use 1-NN (k-Nearest-Neighbors with k=1).
This does not need any training (it is a lazy algorithm) and for each new sample you look up what is the nearest datapoint in your dataset.
In your case, when you try to classify a new sample you can set its price to 0.
If you use Euclidian distance as a distance metric for the 1-NN, then in case there are multiple points with the same feature values, the nearest one will be the one with the smaller price (assuming that prices are in the range [0, inf) to avoid complications with negative prices).
In summary, what you need to do to classify a sample:

Set the sample's price to 0.
For each data point in your dataset:
3. Calculate the Euclidian distance between the sample and the data point.
Use the type of the data point with the minimum distance from the sample as the classification value for the sample.


identify similarities in a dataset I have a dataset of customers:

age  height  weight eye_colour
30    174    74       Nan
20    191    71       Nan
28    165    56       Brown
...

I would like to identify similarities between customers where the eye_colour is Nan in order to help identify the cause of this missing value.

Is there any method to help identify similarities?

In general identifying similarities is done with clustering, but in this case what you're looking for is a potential pattern in the data leading to a missing value in a specific column, right? So I would try to train a decision tree using all the columns except the target one as features, and using a binary class indicating whether the target column has a missing value or not. Visualizing the decision tree obtained after training  should show the pattern if there is one.


How to do class balancing? I am working with a really imbalanced dataset ($\approx$ 1% of positive cases) for a classification problem. I know that class balancing is an important step in this scenario.
I have two questions:

Considering that I don't want to put the 0/1 label, but just to order the record according to the output score (it is always a calibrated probability of being in the positive class), is it still a good idea to do class balancing or, considering the specific output required, it is useless?
Basically, I do not care about the cut-off point, but I just sort the record in order to identify the one with a higher probability of being positives.

Considering the really small percentage of positive cases, is it better to do over/under sampling? Is there any rule-of-thumb to decide the proportion of resampling?

Thank you in advance!

With such a heavy imbalance and two classes (it seems) you could treat this as more of an outlier detection problem. You should read up on models and algorithms in that direction!
If you go forward with a traditional classification you need to balance the data set, consider methods such as SMOTE.
Depending on the size of your data I would generally recommend downsampling the majority class which avoids producing "synthetic" cases but advanced methods such as SMOTE basically take care of this decision for you.
Can you elaborate what you mean with your first question as well? A classification algorithm needs 0/1 labels and therefore the output score cannot be ordered in the way that you mention. Some classification algorithms put out a probability score instead of predicted label so if this is what you mean I can tell you that the imbalance will still be a problem.


Suggestions on how to explain 'models' & 'algorithms' I guess other members of this Stack have ran in to this before, but I may be wrong: Have you ever been approached and asked to explain the difference between models and algorithms? This happened to be recently and, while I feel that I explained it well, there is a colleague whom disagrees with my assessment..

How I see the two:

Algorithms: Used to train a model, that is, give it instructions and process the input.

Model: A diagram of sorts, that incorporates utilizing algorithms to train inputs. Can be reused on similar data.

Pre-Trained Model: A model, usually trained on larger data sets and can be utilized to build [your] own models with other data.

Thoughts? I'd appreciate them to see if I am just completely off base here.

I think your definitions are mostly right....I would add that model is a specific implementation of algorithms designed to handle a specific issue or group of issues
Regarding pre-trained model, my understanding is that it can be used to build other models indeed, but the main use case is simply using it to predict or classify on new data, having already been trained previously


Ideal aggregation function for Partially Connected Neural Network (PCNN) I am building a Python library that creates Partially Connected Neural Networks based on input and output data (X,Y). The basic gist is that the network graph is arbitrarily updated with nodes and edges in the hidden layers.

Example Data:

X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float)
Y = np.array([[0],[1],[1],[0]], dtype=np.float)

Example Graph:

I am currently using the sum product aggregation function to calculate each layer's values:

$$
\sum_{i=0}^n w_{ij}x_i
$$

The synapse weights are denoted by two subscripts $ij$. Subscript $i$ represents the previous neuron and subscript $j$ represents the current neuron under consideration. The sum product for current neuron $j$ is computed as:

$$
s_j = w_{0j}x_0 + w_{1j}x_1 + ... + w_{nj}x_n = \sum_{i=0}^n w_{ij}x_i
$$

Is sum product an ideal aggregation function for PCNNs? Dot product won't work as the tensor sizes are almost always incompatible.

Update: As Bence Mélykúti suggested, my solution was to use a weight vector $w$ that is padded with zeros and reshaped for dot product. Here is how I accomplished it in my Python library:

from functools import reduce
import numpy as np
import math

# An example of some input/sensor data
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=np.float16)
# From the graph, two inbound synapses in layer 1, here randomly generated
W = np.random.random((2,))

# Enumerate the element count in each array
xElements = reduce(lambda x, y: x*y, list(X.shape))
wElements = reduce(lambda x, y: x*y, list(W.shape))

# If previous layer has more elements than the weights in the current layer
if xElements > wElements:
    # Create a list of zeros of shape X.shape
    A = np.zeros(X.shape).flatten().tolist()
else:
    # Otherwise, create a list with the larger number of elements in W
    A = np.zeros(W.shape).flatten().tolist()

# If the list is odd, add one element to ensure it is even
# Done so the array can be reshaped to fit the data and dot product
if len(A) % 2 != 0:
    A.extend([np.float16(0.0)])

# Fill the list with the elements from W
for ix,ele in enumerate(W):
    A[ix] = ele
A = np.array(A)

# Convert list to an array and reshape to be compatible with dot product
A = A.reshape(X.shape[-1],(math.floor(A.shape[0]/2)))

result = np.dot(X,A)

The result of the dot product looks like:

print(result)
array([[0.        , 0.        , 0.        , 0.        ],
       [0.        , 0.        , 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ],
       [0.45510922, 0.79058734, 0.        , 0.        ]])

Thanks Bence!

What you call sum product is the same as a dot product of two vectors of equal length, $(w_{0j},w_{1j},\dots,w_{nj})$ and $(x_0,x_1,\dots,x_n)$.

Based on your diagram, your difficulty is that you want to be able to define connections between neurons that are not in consecutive layers (in other words, the connections might skip layers). I assume that there are no directed edges (synapses) that would go in the opposite direction (from layer $n$ to layer $m$ for some $m<n$). In this setting, you have to forward propagate information to layer $n$ from input $X$ and layers $1,2,\ldots,n-1$. I don't know when and how often you want to 'update' your network topology. Do you want to change it during training?! (Otherwise you'd fix a partially connected NN and you wouldn't write update in your question.)

For forward propagation in a fixed network topology, I would associate to neuron $j$ a vector of weights $(w_{ij})_{i\in I_j}$, where $I_j$ is the set of neurons that bring information to $j$ (those $i$ where there exists a directed edge from $i$ to $j$). Then you could either cherry-pick the neurons $i$ from the set of all neurons (by maintaining a list of indexes to find them in the vector of all neurons) and put them into a vector $x$, which is now as long as $(w_{ij})_{i\in I_j}$, and you can use dot product on the two vectors. (With numpy for Python, numpy.dot would work.) You would need to do this separately for all $j$ in the given layer. As these are receiving information from differing numbers of inputs and neurons, they indeed cannot be arranged into a matrix/tensor, not even receiving neurons of the same layer.

Or you could force all potential preceding neurons into one vector. Neuron $j$ of layer $n$ can only receive information from neurons $x=(<$input nodes$>$, $<$neurons of layer $1>$, $\ldots$, $<$neurons of layer $n-1>)$. This is a vector, and you can define a vector $w$ of equal length for $j$, which would indeed have many zeros, and $x$ and $w$ can be multiplied. For all neurons of layer $n$, you could stack these $w$ as row vectors on top of each other into a matrix/tensor of size (the number of neurons in layer $n$) times (the size of input $+$ the number of neurons in layer $1$ $+\ldots+$ the number of neurons in layer $n-1$) and use np.dot between this matrix and the $x$.

How to do training (backpropagation) with such topologies is something you have to consider a bit. I suspect it will work without much adjustment compared to a fully connected neural network without layer skipping. In the second option, you would only need to zero out the weights which must be kept zero after each backpropagation update. This indeed carries overhead if you're not careful. But if you maintained a binary ($0$ or $1$) 'mask' matrix of which weights are live (which correspond to existing connections), then you would only ever update these weights in the backpropagation.

Additionally, you may want to introduce a bias term $w_{-1j}$ (also known as intercept term, a constant offset) as if $x$ had an extra dimension that is kept constant $1$. After the aggregation with the sum product, you should also pass the result through a non-linear activation function before entering the value into the neuron in the next layer.


Predictions with arbitrairy sequence length for stateful RNN (LSTM/GRU) in Keras I have time series data of the following properties:

input shape:  (num_timesteps, num_features)
output shape: (num_timesteps, num_outputs)

I reshape it to batch form:

input shape:  (num_batches, num_timesteps_in_batch, num features)
output shape: (num_batches, num_timesteps_in_batch, num outputs)

I have a stateful RNN in Keras:

modelinput = Input(batch_shape=(num_batches,None,num_features))
prediction = GRU(10,return_sequences=True,stateful=True)(inputs)
model = Model(inputs=modelinput,outputs=prediction)

After trainig (which works fine) I would like to predict on a sequence without cutting the data, so input shape (num_timesteps, num_features). How can I do that?

I thought about having a second model that shares the weights with the RNN and that has dynamic input shapes. Is that possible?

I found a possible solution: You can save the weights, create a complete new model and load the weights again.

Original Stateful Model

modelinput = Input(batch_shape=(num_batches,None,num_features))
prediction = GRU(10,return_sequences=True,stateful=True)(inputs)
model = Model(inputs=modelinput,outputs=prediction)

Save Weights after Training

model.fit(...)
model.save_weights('weights.h5')

Create New Model (same structure, just not stateful and dynamic batch size)

modelinput_pred = Input(batch_shape=(None,None,num_features))
prediction_pred = GRU(10,return_sequences=True,stateful=False)(inputs_pred)
model_pred = Model(inputs=modelinput,outputs=prediction_pred)

Load Save Parameters and Predict

model_pred.load_weights('weights.h5')
model_pred.predict(...)

Nevertheless I hope for a better solution through sharing weights between a stateful GRU and a standard GRU. Is that possible?


Target encoding with cross validation I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation.

  It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

  Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

  So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

  It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

As far as I understand, the motivation of this approach is that: target encoding requires the knowledge of output, which is not available on the test set. So if we use the means obtained from the whole train set and apply on test set, that may cause overfitting. So instead, we will use other values derived from its subset.

I found some discussions from this post but have trouble of understanding the following points:

1) It seems to me that the his second-level CV is nothing but taking the average of the whole #2-#20 fold. So basically, this is just one-level cross validation, where instead of using the mean of #1 fold, we use that of #2-#20 fold as the mean value for #1 fold. Am I missing something here?

2) Once we obtain the means of all 20 folds, what will we do next? If we average, this is again nothing but taking average of all train set.

Your question 1:

You are close but not exactly.

Each time you take the mean, you include only the samples with the given value, say, A, of the categorical feature.

You partition #2-#20 folds into 10 equal folds; however, once you consider only the samples with the category value A, their numbers may be different in these 10 folds.

Say, these numbers are n1,...,n10.

Then the mean of all but the first fold assigns the weight of 0 to the samples in the first fold and the weight of 1/(n2+...+n10) to each sample in all other folds.

The mean of all but the second fold assigns the weight of 0 to the samples in the second fold and the weight of 1/(n1+n3+n4+...+n10) to each sample in all other folds, etc.

Taking the average of these means yields the weighted average of the target with the weights of the samples being the averages of the above weights.

These weights will be slightly different for samples in different folds.

Thus, we get the weighted average of #2-#20 folds but with slightly different weights obtained by this random procedure.

You probably want to understand what is the point of this: how is it better than just taking the plain average over folds #2-#20?

This process adds randomality to the values of the encoding.  (The randomality can be measured, e.g., by the standard deviation of the encoding values for the category value A.)

This randomality of the encoding values may prevent the algorithm from learning the relationship between the encoding and the target in case this relationship holds only in the training set, i.e., reduce overfitting.

This randomality depends on two parameters:

The width of the distribution of the target value for the samples having the category value A.  (The width may be measured, e.g., by the standard deviation.)  Wider distribution in these target values results in greater randomality (wider distribution) of the encoding values.
The number of samples having the category value A: more samples result in smaller randomality of the encoding values.

One can compare this way with other ways of increasing the randomality of the encoding values.  The simplest way is to do single cross-validation but with less than 20 folds.  This will increase the randomality of the encoding values but it will mainly depend on the width of the distribution of the target values and less on the the number of samples in the category A.  This could be the reason for this double cross-validation.

Your question 2:

As you said, the point is to use for each sample the average of the target values that does not include this sample.

For a sample in the fold #1 with the category value A, the value of the mean target encoding is the (weighted) mean of the target values over all samples in the folds #2-#20 with the category value A (calculated as above).

This means that the values of the encoding of A will be different in different folds.

The post you quoted, does not mention the test set.  Other posts suggest that on the test set you take the average of the target values of all samples that have category A in the entire train set.


How to deal with broad and narrow variance within classes in classification tasks Let's say I'm doing an animal image classification task (it doesn't have to be image classification - this is just my example), and the training and test data is balanced across classes. The classes might be ['gorilla', 'giraffe', 'dog', 'donkey']. Now we all know that there is relatively a lot of variance within the 'dog' class compared to the other three classes.

So, is there any way one would treat this problem vs another problem where all classes have about the same amount of variance (where I might replace 'dog' with 'sheep' for instance)?

You will want to have many diverse examples of the high variance classes and pick a model that has a high learning capacity. In other words, a large number of samples and a big model to capture the properties of the data.

After training the model, look at the confusion matrix by class for the hold-out dataset to see if the model is learning to generalize for all classes, including the high variance classes.


Difference between validation and prediction As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction.
To keep it simple, I will refer to train, val and test:

Training data: Train model, especially find hyperparameters through GridSearchCV or similar
Validation data: Validate these hyperparameters on "new" data?
Test data: Make prediction on unseen data

My status so far:

Split data: 60 % Training - 20 % Validation - 20 % Test
Find hyperparameters on training data
Fit again with best parameters on training data by using .fit(X_train, y_train, validation_data(X_val, y_val)).
Check model on unseen data through .predict() or .evaluate().

Is this correct? Though using GridSearchCV do I have to split train manually into training and validation?

In the GridsearchCV function, there is a parameter, cv that tells you how many times cross-validation is performed. If you're performing cross-validation on the train set, all you need is a train/test set since the hyperparameters are being tuned without seeing the test set.

If you want to use your own validation set, see the link below. The link below that also has a good description of how K-Fold CV works in GridSearchCV.

Finally, gridsearch optimization is pretty slow and there are more advanced methods for hyperparameter tuning now, in case performance is an issue.

See this previous answer:

Difference between test and validation set

Preselected Validation Set

Gridsearch CV and KFold CV


Can we calculate iou and mean iou with only images and labels and without train and test csv file Are train and test CSV files necessary for calculating IoU (Intersection over Union) and Mean IoU. Can I calculate IoU with just images and labels?

The IoU is a metric that measures the similarity of two sets. In the context of object detection or semantic segmentation, the IoU compares the similarity among the ground truths samples and their counterpart predictions. Thus, what you need (in a general approach, you don't give any details) is:

Samples and their ground truth values (labels) for a given set (train, test, validation...)
The predictions for the samples of the given set
Obvious: in order to get the predictions you need a pre-trained model

Typically, you get the prediction $\hat{y_i}$ of a given sample $x_i$ in a given set, and compare it with the ground truh label $y_i$ with $IoU(\hat{y_i},y_i)$. Then, you can average the IoU over the whole dataset.

Answers

  Are train and test CSV files necessary for calculating IoU (Intersection over Union)

At least one of them, but IoU does not compare the train with the test set. It compares the prediction for a sample with its ground truth.

  Can I calculate IoU with just images and labels?

No, you need the predictions


Skin Detection From a Face Image I have photos of people's faces, and I want to detect each person's skin color for later use. I don't want to use predetermined color ranges, instead I want to determine each person's color and then use it to detect occlusions in this person's later photos. The training face images are taken under the same conditions, but testing will be with photos of different conditions. My first thought is to use color histograms of these images, and then the peak will be this person's skin color. Is there a smarter way to do this? I'm using Python and OpenCV, but I can use other tools if needed.

It'd be good idea to extract face segment first. In OpenCV there're predefined HaarCascades for face detection. Though, you can run into some problems using them. Photos really need to be well-exposed. They're also prone to false positives as hell.

https://realpython.com/face-recognition-with-python/

Slightly better, but still vanilla, there's dlib.
Worst problem with them is you'll only get bounding box. Background may creep in and mess your color analysis.

To get seminal-like scores you gotta go DNN. Here's some example.

I would propose a bit different approach. Take your data set. Extract faces using existing tool and label skin colour manually. Then train CNN. Just be careful, when you augment data set. Some operations might be damaging, like swapping channels. Color-based analysis is tough. You need to think of covering different exposure values, contrast, white-balance, saturation etc.


Inception/ResNet doing worse than SIFT in feature extraction We are doing our Thesis on multimodal retrieval. it's basically searching different modalities (multimedia ex: text, video, images ...) with other modalities. i.e. searching a database of images with a text query.

For any modality we need first to map it to a space where it has a constant number of features, and those feature must be somehow expressive of the data.

For images, papers we evaluate utilizes SIFT feature extraction, we use NUSWIDE for evaluating different methods, it already exists in SIFT format so most papers we evaluate uses these existent dataset.

We tried to improve on this feature extraction mechanism by using Inception or Resnet and taking layers exactly before softmax as our features. however, they perform way worse than SIFT. we used tensorflow and keras to extract features.

so any idea why would resnet/inception perfrom worse than SIFT?

After examining the dataset, we found that the problem was in NUSWIDE dataset itself. Almost half of the dataset didn't have supervised labels (81 labels entered by humans). Also we weren't training on the whole dataset, rather, we were taking random samples because some methods required that the whole dataset was loaded in memory which wasn't feasible.
After knowing that, we deleted the tuples that didn't have supervised labels, extracted features and done PCA and got a comparable results to SIFT (improved, in some cases).
So my advice if you face a similar problem is:

Make sure that your data labels are valid.
Use multiple layers for feature extraction, not just the deeper ones in the network.
Using PCA instead of random sampling dataset worked very well for us.


Remove outliers from a noisy curve Allow me to present some images so that I may explain my problem. The images on the left contain a smooth curve surrounded by lots of outliers/noise. The image on the right depict the desired curve.

Is there any approach that would help in eliminating the noise and retrieving the original curve. Please note, this can be any smooth curve (the image on the right) and equation is not known. No prior knowledge of the curve equation is available.
Please note that I had originally published this question on Match stackexchange. Posting to data science based on feedback.
Thanks,
Sau

Expanding my comment:

One can use a regression method (eg linear, non-linear, symbolic, ..) to try to fit a curve to the data.

Outliers can be removed in 1 or 2 steps:
2.a. Remove some points before applying regression, eg by testing how much away they are wrt to std.
2.b. Remove outliers after fitting the curve and measuring distance of point from curve and discard if above a certain threshold.

[UPDATE]
In curve fitting using regression, one rarely has knowledge of the actual curve, instead one assumes the curve belongs to some class of functions and tries various options.
Furthermore, there are what are called symbolic regression methods, which assume even less about the actual curve and can even find functions in implicit form, although more tedious.
Finally, some pre-processing to remove outliers before applying regression (eg discard points very "far", density-based approaches, ..) can certainly improve the outcome based on reasonable domain-based assumptions.
The above will work for most of the cases presented in question. This is an ill-posed inverse problem, so one should make some assumptions and have reasonable expectations.
[UPDATE 2]
If your data is only 2D image data and you simply need to get the curve as an image (and not need to infer any analytical form of the curve), then doing a combination of blur/lowpass filtering and edge detection, can be an option. OpenCV, for example, has methods to achieve that.
References:

An edge detection based noise removal algorithm

We present an image denoising method using the edge map of an image.
The denoised image is considered as a linear combination of the
observed image and its average value, where the coefficients are
controlled by a local edge detector. The parameters are set on
suitable values related to noise energy computed by the curvature norm
of the original image. Implementation can be done in a single
iteration and the speed of the process is reasonably high. Noise
reduction quality of the introduced method is compared with Wiener and
Total Variation based filters for some images. The method appears to
be easy, fast and useful for very noisy images. The differences
between our method and the patent 6229578 “Edge Detection Based Noise
Removal Algorithm” are explained.

How to remove the noise without destroying the main edge?

Canny Edge Image - Noise removal