Text classification analysis based on similarity I have been reading a lot of literature regarding text classification and different approaches/models, especially using Python language, but probably I am still missing something on how to build the models and the steps involved.

I have multiple datasets, each of them on a particular topic. These datasets include news and fake news, manually flagged at the moment.
I have collected texts from different sources about similar topics (using keywords) and now I would like to try to build a model that can allow me to classify a news as real or fake automatically.

I have thought that it could have been useful to study the frequency of words and punctuation, and also similarity, trying to group similar texts based on same conditions (for example, similarity between texts as in plagiarism).
I have used similarity (Jaccard or cosine) for comparing texts rather than words, but I do not know if this is the right approach and how I should create a model based on that.

Probably the best method would be to build a logistic regression model using a binary variable, but I also read a lot of naive bayesian modes and nn models.

Do you have any example on how I should apply text similarity for classification or further relevant information about that?

I shall give you some hints here.

It depends on your data, my advice is that you try several algorithms.

If you have binary classification you can use both classification (try Logistic regression, SVM, RandomForest) or regression (try XGBoost or any gradient boosting algorithm, there are several in scikit learn), and then seek for a boudary in order to decide wheter a value is 0 or 1. Depending on your data, it might work very well (I've got 0.8 fscore on text classification using RandomForest and performing an important pre-processing before). Do not forget to do an appropriate preprocessing. Concerning pre-processing, in scikit-learn you have CountVectorizer and TfIdfVectorizer, very useful.

Do not use Naive Bayes, not on text and NLP.

Well, the best method for that is using Sequence models (especially LSTM), parsing text as an succession of words or characters. You can use in this case Sequences available in Keras.


Why do people import weights for DenseNet when Keras includes them? In several Kaggle kernels I've seen that people often import their weights into Keras' DenseNet. In the following case, I believe the weights are sourced from this github repo and contain the pre-trained weights on the ImageNet dataset.

densenet = DenseNet121(
    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',
    include_top=False,
    input_shape=(im_size,im_size,3)
)

Why do people load weights from a file when you can use the ImageNet weights within Keras by specifying weights='imagenet' in the following manner?

keras.applications.densenet.DenseNet121(include_top=True, weights='imagenet', input_tensor=None, input_shape=None, pooling=None, classes=1000)

Is there a difference that I should be aware of or is it simply a matter of personal preference?

Because it must have been a kernel only competition, with:  'the following conditions must be met:"Internet must be turned off".'

When we use  " weights='imagenet' " , the notebook will connect to internet to download the weights. Since the user cannot connect to internet for that particular competition it is better to upload the weights as data source, and then import it for use.


Target Encoding: missing value imputation before or after encoding I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values.
Let's say I have a few continuous features, Cnt1-Cnt5 (without NA's) and two categorical features, Cat1 and Cat2, with Cat2 having missing values. Let's also assume that I want to use Random Forest as an imputation method. Which approach would be the correct one?

Impute Cat2 treating Cat1 and Cnt1-Cnt5 as predictors in RF and then perform target encoding on categorical variables.
Target encode Cat2 for non missing and Cat1, build RF and impute missings for Cat2 (which is now numeric, not categorical).
Any other approach?

We can generalize this question and ask whether we should impute missings for any kind of variable (including continuous) before or after target encoding?

I see at least one benefit of imputation after target encoding - if there are unseen levels of categorical variable present in the test data (which will result in NA's in the test set after performing target encoding), those would be easily imputed by RF built on training data, without any potential error due to new levels.

If you want to do TargetEncoder you have to impute the missing values first.

First of all you should convert your categorical features into int, using LabelEncoder or OrdinalEncoder. I used a huge numeric value (my choice : 8888) in order to fill the NaN values, before running OrdinalEncoder. Then transform your matrix to int, it will be more efficient.
For the imputation of missing values, you may use different strategies :

2.1. . Fill with the most frequent value in each features (column) (if you did the previous stage you do not need this). If you perform TargetEncoding, imputing with most frequent generally works well.

2.2. . Fill using IterativeImputer. You can provide in the argument estimator the kernel you want to use (I tested RandomForestRegressor - for numeric features). Warning: IterativeImputer uses only regression kernels. If you did the previous stage, do not forget to add the argument : missing_values=8888.

You can apply your TargetEncoder now.


What is the definition of precision for no positive classifications? The precision is defined as

$$\text{precision} = \frac{\text{true positive}}{\text{true positive} + \text{false positive}}$$

Is there any definition what this value should be if there is no positive classification (but of course positive elements)?

I just found that sklearn.metrics.precision_score handles it like this:

>>> sklearn.metrics.precision_score(y_true, y_pred)
/home/moose/.local/lib/python2.7/site-packages/sklearn/metrics/classification.py:1074:
  UndefinedMetricWarning: Precision is ill-defined and being
    set to 0.0 due to no predicted samples.
    'precision', 'predicted', average, warn_for)
0.0

So they give a warning and set it to 0.


Use of cross validation for Polynomial Regression I've two text files which contains my data.
One text file on X axis another text file on Y axis
Then using scatter function from python I did the data visualization
After that, I used polyfit function from python to get the curve which will fit my data
In that polyfit function we need to write degree of the polynomial we want eg. 2 or 3
Now let's consider I got 4 such a equations of degrees 2,3,4,5 respectively.
Now here comes my problem.
Among all those equations I got, how can I select the best equation which fits my data. I want to use cross validation here. Any high level library function can be use. My language is Python.

I think that you want this:

K-fold

If you want say MSE of each check out section 3.1.1 here:

cross validated metrics


Visualizing Customer journeys Objective

I want to visualize a typical customer's journey using Python or R. As usual, customers buy different products at different points of time.

I checked every customers journey and chose product A as a starting point. 10,000 customers bought product A as their first product. Now I want to show the "flow" of their  journey. After 10 minutes, 2,500 of these 10,000 customers,  bought product B as their second product. Product C was bought by 7,500 people after 25 minutes.

I am looking for a more complex sankey chart. You start of with a bar chart with the height of 10,000 on the left. There are two bar charts on the right and you see a connection between the left and right charts. The longer it took the customers to buy the next product, the further away are the bars on the right from the left one. All the bars have titles with their product names. Quick infos on the charts would also be nice.

The final chart would visualize a longer customer journey (more than two products).

What I have tried

I used the networkD3 R package to visualize a sankeyNetwork, however I could not place the bar charts according to my desires.

Do you have any recommendations for Python or R packages, or is there a way to draw it by myself (in a reasonable amount of time)?

This may be too simple for what you had in mind, but perhaps it will help:

Example data

library(tidyverse); library(magrittr) # load R libraries
sample_data = data.frame(Customer = 1:100, A = 0, B = rep(c(10,NA), c(25,75)), C = rep(c(NA,25), c(25,75)))

Bar chart

sample_data %>% gather("Product", "Time", -Customer) %>% ggplot()+geom_bar(aes(x=Time, fill=Product))+geom_label(aes(x=Time, label=Product), y=-Inf, vjust=0)

Point plot

The use of this is that you can map the customers (since the y-axis is not the count but the Customer ID), i.e., you can add lines connecting them but they would be too many, so I haven't (they can be grouped though, for A->B, B->C, A->C, etc)

sample_data %>% gather("Product", "Time", -Customer) %>% ggplot()+geom_point(aes(x=Time, y=Customer, color=Product))+geom_label(aes(x=Time, label=Product), y=-Inf, vjust=0)


How can I see a long string in my dataframe? I have a column in my dataframe in which there are sentences which are too long. I want to see them as a whole but every time I perform even a simple iloc operation i get output like 'i am going to...'. How can I remove the ... and see the whole sentence ?

At first, here is the Data.

data = {
    'Capacity' : [
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze"
    ]
}
df = pd.DataFrame(data)
print(df['Capacity'])
# 0  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 1  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 2  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 3  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...

You can set pd.options.display.max_colwidth to increase the number of characters you want to display.

pd.options.display.max_colwidth = 20
print(df['Capacity'])
# 0    qsfjvanzicgbaefz...
# 1    qsfjvanzicgbaefz...
# 2    qsfjvanzicgbaefz...
# 3    qsfjvanzicgbaefz...

pd.options.display.max_colwidth = 100
print(df['Capacity'])
# 0    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 1    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 2    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 3    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...

# This seeks the maximum length of the current column 'Capacity'.
pd.options.display.max_colwidth = int(df['Capacity'].str.len().max()) + 1
print(df['Capacity'])

Or you can use the .values method instead, so that you are watching the 'raw' data.

#   Reset the option.
pd.options.display.max_colwidth = 50
print(df['Capacity'].values)
# ['qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze']

Or simply select one row from the column you're interested in.

print(df['Capacity'][2])
# qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze


k means clustering when k = n I want to find the minimum value of the objective function if we set K equal to the number of samples.

I know the objective function is $J=\Sigma_{n=1}^{N}\Sigma_{k=1}^{K}r_{nk}||x_n-\mu_k||^2$
And we take the derivative to get $\mu_k=\frac{\Sigma_{n=1}^{N}r_{nk}x_n}{\Sigma_{n=1}^{N}r_{nk}}$

However, when plugging that back in and set K=N, it seems unable to be simplified:
$J=\Sigma_{n=1}^{N}\Sigma_{k=1}^{N}r_{nk}||x_n-\frac{\Sigma_{n=1}^{N}r_{nk}x_n}{\Sigma_{n=1}^{N}r_{nk}}||^2$

Any help?

Well, when k = n, the obvious global minimum occurs where $x_n = \mu_k$ for all cluster centers. This falls out from your derivate because $r_{nk} = 1$ iff $n = k$.


What is the difference between concept learning and classification? I've recently read about concept learning in a machine learning class. They defined a concept as (translated from German):

  Concept:

  Describes a subset of objects or events defined on a bigger set.
  Boolean function defined on bigger set

  Given: Examples which are labeled as members or not-members

  Searched: Automatically infer the definition of the underlying concept

  Definition of concept learning: Infering a boolean-valued function from training data of its input and its output.

I would call this a 2-class classification problem. Is there a difference?

The objective is rather different.
For classification problems, we want to know: GIVEN X, WHAT WILL BE Y?
For concept-learning problems, we are asking: WHAT KIND OF X WILL GIVE US Y?

It may seem similar, but for classification problems, we may use a model that is unable to generalize and answer "WHAT KIND OF X WILL GIVE US Y" until you plug X into the model (e.g. K-nearest neighbor).

On the other hand, concept-learning problems, the main issue is to generalize and answer what can be X specifically.


How do I provide input and output for such a network structure in keras I am trying to create a CNN network for classification purposes, the network with both input and output is illustrated as such:

The input the image is separated into sections, each section is given to a certain model in which a convolution and pooling is applied. This is done as such to ensure full weight share does not occur, as I am not interested in the complete image shares the same weight.  After the convolution an pooling, is all the output of models concatenated into one, and fed into a fully connected layer, which then performs classification and outputs a class.

The input I have is the image (either the complete or separated into sections) and the output being the class, but I am not sure how I should train the model, given these input and output.

The way I've constructed the model is as such:

#Convolution for different section conv2d_[convolution#]_[model#]
conv2d_1_1 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_1)
conv2d_3_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_1)
conv2d_4_1 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_1)
conv2d_4_1_flatten = Flatten()(conv2d_4_1)

conv2d_1_2 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_2)
conv2d_3_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_2)
conv2d_4_2 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_2)
conv2d_4_2_flatten = Flatten()(conv2d_4_2)

conv2d_1_3 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_3)
conv2d_3_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_3)
conv2d_4_3 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_3)
conv2d_4_3_flatten = Flatten()(conv2d_4_3)

conv2d_1_4 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_4)
conv2d_3_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_4)
conv2d_4_4 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_4)
conv2d_4_4_flatten = Flatten()(conv2d_4_4)

#Merging the output of the convolution
merge = Merged([conv2d_4_1_flatten, conv2d_4_2_flatten,
                conv2d_4_3_flatten, conv2d_4_4_flatten] mode = 'concat')

#Connecting the merged layer to the fully connected layer
dense1 = Dense(activation = 'relu')(merge)
dense2 = Dense(activation = 'relu')(dense1)
dense3 = Dense(output = 1 ,activation = 'softmax')(dense3)

This (I guess) CNN model should now be connected, but how do I train it given the input and outputs I have? How do I parse it?

My input data is stored as list of numpy.ndarray. Each row of each ndarray, is the input for each input.  How do I parse each row to each input? And the output is a vector of classes. I have in total 145 classes, and output is vector length 3 with the true labels extracted from 4 input channels?

You can pass in a list to the input parameter when you create the model.

model = Model(input=[conv2d_1_1, conv2d_1_2, conv2d_1_3, conv2d_1_4], output=dense3)

Make the input x a list of four numpy arrays of shape (3,6,3)

Make the output y an array of one-hot vectors corresponding to your 145 classes.

Then call

model.fit(x, y)

You probably want to change your last layer to have 143 dimensions in the output

num_classes =143
Dense(num_classes, activation='softmax', activation = 'softmax')


A suitable feature vector for images I have a set of images of various products from different websites. I want to cluster the images based on the product shown in the image. How can I generate a suitable feature vector for an image for this purpose?? I just need to know how to generate a feature vector given an image. I tried NetVLAD, but it is very slow. I would like something that is fast and gives high accuracy for clustering in the scenario I have described. Please help me.

Feature extraction is basically reducing the amount of resources required to describe a large set of data. Analysis with a large number of variables (like images) generally requires a big amounts of memory and computation power.
You can start from the simplest and slowly work your way up (in computational requirements) until you reach your desired accuracy.

Start from:

ORB (oriented FAST, rotated robust independent features) -> SURF (speed up robust feature) -> SIFT (scale invariant feature transform) -> OverFeat (Convolutional Network) -> vgg16 ... (There are many more, any neural network model can be used for this)


Is it possible to reuse my CNN trained model over a new dataset with different number of classes? I had used a trained CNN model (VGG16) over a large dataset with 6 number of classes in FC layer it gave me a good accuracy ( for example over Testing data: loss=0.59, accuracy=0.829).
When I had applied the same model over a dataset with less samples and labels (4 classes) it reduced the accuracy.
Here my question is : is it possible to use model 1 over the 2nd dataset (bec I had saved the attained weights) or it is not possible due to the difference in classes number?
Also if I rerun my model 1 does the accuracy would be improved or no?

The accuracy will downgrade since the model was trained on a dataset with 6 classes but is being tested on 4 classes.
If you want a good accuracy for the dataset with 4 classes, first train it on the data of 4 classes and then test it on some test data with 4 classes.
Hope this is of help.


Getting a best k in KNN Algorithm So, i was learning the KNN Algorithm and there i learnt cross Validation to find a optimal value of k.Now i want to apply grid search to get the optimal value.I found an answer on stack overflow where both standardScaler and KNN are passed as estimator.

 pipe = Pipeline([
        ('sc', StandardScaler()),
        ('knn', KNeighborsClassifier(algorithm='brute'))
    ])
    params = {
        'knn__n_neighbors': [3, 5, 7, 9, 11] # usually odd numbers
    }
    clf = GridSearchCV(estimator=pipe,
                      param_grid=params,
                      cv=5,
                      return_train_score=True) # Turn on cv train scores
    clf.fit(X, y)

My questions

I am already applying the standardscaler to standardize the data before passing to KNN. So here do i still need to pass the standardscaler in the estimator?
why X and Y are passed instead of x_train and y_train assuming x and y are independent and dependent variable and x_train,y_train are formed after train_test_split operation ?

Any example of such code will be appericiated.

Looking into the linked answer, it appears that they are directly training on X and y since they're using a GridSearchCV, which already includes a k-fold cross validation (5 fold by default). So basically you'll already have a score for the classifier by calling GridSearchCV with the defined pipeline.

That being said I'd argue that it is never really the recommended approach to directly do this without a final test step, to asses the performance of the trained model on unseen data. So even if you do a k-fold cross validation, it is advisable to leave a test set to get a final score, specially when the k-fold process involved a hyper-parameter tuning, as in this case. In such cases you  need anther validation step that is independent of the tuning.

And in relation to the second point, no, you don't need to include a StandardScaler if the data is already normalised. Though, since you're using pipeline, you might as well include all transformation logic in the pipeline, for the sake of simplicity.


Zero Padding in Convolutional neural network We use Convolutional neural network because it by design learns features that generalize over spatial location , so when using conv operation it reduce image size and that what we hope to have so we can have more simplicity in the following layers , but when doing zero padding the image return to its original size , so what is the benefit of doing this ? the same problem return back since image size will be still large . Also, CNN works well for classify images; so how it can be used in games to detect best next move, I mean how it decide the Q values of the actions from the image itself even though understanding image doesn't give any indication of the next move  ?

For padding in CNN's there is a useful answer in this link:https://stats.stackexchange.com/questions/246512/convolutional-layers-to-pad-or-not-to-pad
In Q-learning an action is taken in current state, and a next state is obtained. Then state-action value of the current state-action pair is updated by using best state-action value of obtained next state. If you have discrete state-action space, or a few states you can simply record each state-action pair in a table, and then update it by Q-learning, but if the space is continuous, or there are so many state-action pairs then you should use function approximation methods like neural networks. For example by extracting current state's visual features by CNN, and then updating weights to better approximate value of this perceived state, and taken action pair.


Approaches to Bag-Of-Words Information Retrieval I'm interested in an overview of the modern/state-of-the-art approaches to bag-of-words information retrieval, where you have a single query $q$ and a set of documents which you hope to rank by relevance $d_1,...,d_n$.

I'm specifically interested in approaches which require absolutely no linguistic knowledge and rely on no outside linguistic or lexical resources for boosting performance (such as thesauri or prebuilt word-nets and the like).  Thus where a ranking is produced entirely by evaluating query-document similarity and where the problems of synonymy and polysemy are overcome by exploiting inter-document word co-occurence.

I've spent some amount of time in the literature (reading papers/tutorials), however there is so much information out there it's hard to get a bird's eye view.  The best I can make out is that modern approaches involve some combination of a weighted vector space model (such as a generalized vector space model, LSI, or a topic-based vector space model using LDA), in conjunction with pseudo-relevance feedback (using either Rocchio or some more advanced approach).

All these vector space models tend to use cosine-similarity as the similarity function, however I have seen some literature discussing similarity functions which are more exotic.

Is this basically where we currently are in attacking this particular type of problem?

While reading blogs and papers is helpful to identify the latest and greatest, having a solid foundation helps a lot, too. But I assume you already have gone over Manning's great (and free in e-book form) book on IR, right?

http://nlp.stanford.edu/IR-book/

It contains information on creating your own thesaurus from your document collection to solve synonymy problems, LSA for polysemy, etc..

As for similarity measures, you will see there that Okapi BM25 (Robertson et al.) is considered superior to cosine similarity (but more expensive to implement and run).
Regarding the current state of the art, there was a small emergence of Bayesian Network-based classifiers in the early nineties (starting with Turtle & Croft), but that went quiet for a while.
However, right now, using BNs for IR is again finding some revival, particularly in  biomedical IR.
In that respect, I think most ongoing work is directed towards using Bayesian models incl. topic models and deep learning for word-sense disambiguation (WSD) and semantic similarity.
Here is a pointer to a recent paper with good references on the topic.

http://arxiv.org/abs/1412.6629


Binary Classification Probabilities I'm using keras in R to predict financial time series. I guess that ANN converges.

But when I do predictions I can see that probabilities don't sum up to 1

I'm using relu as final activation and loss = "binary_crossentropy".
Does that mean the net doesn't converge or do I miss something else?

Even if input to a neural netwrk are scaled or normalised, the raw output values can still go outside of that range.

In your case, the output values are being interpreted as to make a binary YES/NO decision, but the raw values cannot necessarily be interpreted as raw probabilities! They are merely the final activations of the network.

To get what you expect, the final activations are usually passed through a softmax function, which essentially squashes the values you see in your table to sum to 1 on each row - this allows us to treat them as probabilities to make the final classification.

In practice, this means simply adding the softmax activation to your final Dense layer in Keras (activation="softmax") and then compile the model using:
 loss="categorical_crossentropy"


Combine results from multiple models I am using chunks of 100000 rows at a time from the CSV file to train the a simple LASSO model.

  How do i combine all of these models trained from these different chunks? I would like to use all these trained models for prediction

  I am familiar with DASK and other alternatives but i would like to use Pandas.

pipelines = {
    'lasso' : make_pipeline(StandardScaler(), Lasso(random_state=123))
}

for key, value in pipelines.items():
    print( key, type(value) )

# Lasso hyperparameters
lasso_hyperparameters = {
    'lasso__alpha' : [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10]
}

hyperparameters = {
'lasso' : lasso_hyperparameters
}

# Create empty dictionary called fitted_models
fitted_models = {}

# Create cross-validation object from pipeline and hyperparameters
model = GridSearchCV(pipeline, hyperparameters[name], cv=10, n_jobs=-1)

def train(X_train, y_train):
    # Fit model on X_train, y_train
    model.fit(X_train, y_train)

    # Store model in fitted_models[name]
    fitted_models[name] = model

    # Print '{name} has been fitted'
    print(name, 'has been fitted.')
    print ("__________________________________")
    print (model.cv_results_)

for df in pd.read_csv('train_V2.csv', chunksize=100000):
    df = df.dropna()
    df = pd.get_dummies(df, columns=['matchType'])
    df_train = df.drop(['Id', 'groupId', 'matchId'], axis = 1)
    y = df_train.winPlacePerc
    X = df_train.drop('winPlacePerc', axis=1)
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=1234)
    X_train = np.asarray(X_train)
    X_test = np.asarray(X_test)
    y_train = np.asarray(y_train)
    y_test = np.asarray(y_test)

    train(X_train, y_train)

What you are looking for is called "stochastic optimization". You don't need to fit separate models and then combine them.


What will happen if almost constant values for features? In a problem of an epidemiology dataset, is it desirable to keep the features that have almost constant values?
For example, In case of the feature, type_of_residence Large for 97 percent and Small for 2.7 percent of subjects. Is it okay to keep this feature?
My target variable is outcome of patients and this data set is unbalanced. Like oversampling and under sampling techniques in class imbalance problem, is there any process in ML to address other predicting features?
I am not doing feature selection using ML at this point for knowing the importance of this feature. But would like to know is there any general rules regarding keeping or not keeping these kind of features in dataset.

If a feature is truly constant across all the instances, then it's useless for classification and it can be removed.
But in any other case, as in your example, a feature can be useful even if constant with most instances. In your example it might turn out that the target class is strongly correlated with type_of_residence, so it would be a mistake to get rid of it since your model can take advantage of it.
Unless you have a good reason to get rid of a feature (expert knowledge), keep all of them. If you must reduce the number of features (for instance to avoid overfitting), then the selection should be based on information about which features are the least informative (e.g. using some correlation measure).
Ideally the best way to know which features are useful is to try to train a classifier and evaluate it with every possible subset of features (but it's rarely possible in practice)


How to do face recognition without using any kind of Neural Networks like CNN? Is/was there any way to perform face recognition, instead of using the Convolution Neural Network which uses the technique of mapping(encoding) the face using 128-D vector and then using classifier (like knn/SVM) on it?

Before the invention of the CNN approach, what did we use for face recognition?

From a comment by @rahulreddy:

Custom object detection is possible using Voila-Jones Haar cascade or using the histogram of oriented gradients (HOG) technique or now by CNN.

As it stands, NN-based approaches are the current state of the art. It outperforms the method described below, but in the spirit of answering the question, here it goes...

The trick for not using CNNs is to find a good representation of images with faces on them. CNNs are great because they learn good features. If you don't want to use it, you can use CNNs, you can use Eingenfaces as your features.

The main assumption is that most face images lie on a low-dimensional subspace determined by the first k (k<d) directions of maximum variance.

You can Use PCA to determine the vectors or “eigenfaces” u1, ... ,uk that span that subspace.

Then you can represent all face images in the dataset as linear combinations of eigenfaces.

For example, given these images for training:

You can learn the following eigenfaces:

These eigenfaces highlight different features in a person's face. If you average all of them you find a mean face, which would be the most common features among all people in your training set.

Then, this face:

Can be reconstructed by a linear combination of the mean face and some other eigenfaces:

Now, the weights of each component can be used as features for your classification algorithm. Which at this point can be anything, like KNN or SVM.


How to interpret confusion matric for Extreme learning machine ? why it has decimal points in it? ELM for phishing URL detection.
Dataset is divided into 70/30. 30% is for testing.
Dataset size: 11,000 Entries.

The confusion matrix shows the percentage of observations that fall into a combination of the actual and the predicted value. These percentage add up to 100% row-wise (0.3419 + 0.6581 = 1). This means that of all observations that had the label 'Phishing', 34.19% were correctly classified as 'Phishing' and 65.81% were incorrectly classified as 'Legitimate'.


How come same cluster category be separated? I have these 200 vectors which were clustered using K-means clustering based on keywords weight similarity that was given by TF-IDF (Term Frequency - Inverse Document Frequency). The vectors were clustered with respect to the vectors in four cities which are Amsterdam, Rotterdam, The Hague and, Utrecht. I have chosen k-cluster centroid = 6, which means I have cluster 0 to cluster 5. On each cluster, I also calculated the average number of keyword's numerical weight so that then I got the most relevant and least relevant set of keywords just like the picture below:

Both relevant keywords and least relevant could help the interpretation of what is the cluster about. For example, cluster 0 is related to rail transportation because in the most relevant keywords include tram, line, tramway, station, and rail. And the least relevant keywords emphasized the interpretation of cluster 0 where the keywords include photography, bicycle, iphoneography, green, nature, and flower.
And I have the cluster map of all six clusters in the Amsterdam city shows in this picture:

The problem is that in Amsterdam city, there is no cluster 0, which related to rail transportation. In my analysis opinion, it is because all the vectors that relate to rail transportation were clustered to cluster 3, which is also related to rail transportation (based on my interpretation on most relevant and least relevant keywords on both clusters). Cluster 3 also related to rail transportation because in the most relevant keywords include tram, line, tramway, station, and rail. And the least relevant keywords emphasized the interpretation of cluster 0 where the keywords include photography, bicycle, iphoneography, green, nature, and flower.

There is also the evidence that cluster 3 could not be found in Rotterdam and The Hague city because all vectors related to rail transportation in both cities were all clustered to cluster 0. Below you could find the pictures of the cluster map in both cities:

My question is whether my analysis could be justified? But how come two clusters with the same theme could be separated? why they do not cluster together?

This KMeans Clustering is a representative of points in the space based on 200 Features and their length. I think there is some gap in your perspective and the actual clustering.
 Other than "Tram", most of the other relevant features are uncommon in both the Rail Cluster. So, two different blobs are created in space.
See this image, data are at the same place on the Tram dimension but other Features have created two different groups.

 These are the approaches, you may take -
 - Clean all the features for duplicate/similar meaning in the domain context e.g. rail, railway
 - Try different cluster count e.g. 5 and see if these two merges
 - Even if these are not merging then there is some message and you should figure it out. e.g. gvg, combino, streetcar, city


EfficientNet function composition or Hadamard In the page 3 of the paper of EfficientNet, there is a equation
$$\mathcal{N} = \bigodot_{i=1...s} \mathcal{F}_{i}^{L_i} \big(X_{\langle H_i, W_i, C_i \rangle}\big)$$
where $\mathcal{N}$ is the conv net and each $\mathcal{F}_i^{L_i}$ is the $i$th-stage layer operator that has length $L_i$.
What I don't understand is, what is this $\odot$ in this equation? Does the author refer to the Hadamard product or does he refer to the function composition? He previously mentioned that $\mathcal{N} = \mathcal{F}_k \odot ... \odot \mathcal{F}_1 (X_1)$, where k is the depth of the net. So I thought it means that $\odot$ is just function composition. But EfficientNet has skip connection. In the keras implementation it uses layers.merge.Multiply() so it can also means that the input data $X$ is multiplied with the transformed $\mathcal{F}(X)$, and $\odot$ maybe means the Hadamard product.
Does anyone knows the answer? Thanks.

It probably represents repeated function composition.
Key reasons:

The authors' wording before this equation is: "a list of composed layers".
A Hadamard product could not be applied to layers of different sizes, whereas function composition of course can.
Yes, EfficientNet has skip connections, but I think they try to validate function composition via a distinction between "layers" and "stages". They say that "$F_i^{L_i}$ denotes layer $F_i$ is repeated $L_i$ times in stage $i$". They also explain that "all layers in each stage share the same architecture. Therefore, I think they put skip-connections in the form of function composition, but it is confusing. I think it's their way of compactly expressing the (often) repeated architecture of skip-connected layers.


Minimum number of samples to train XGBoost without overfitting When using Neural Networks for image processing I learned a rule of thumb: to avoid overfitting, supply at least 10 training examples for every neuron.
Is there a similar rule of thumb for classifiers such as XGBoost, presumably taking into account the number of features and estimators?
And, considering the 'curse of dimensionality' shouldn't the rule of thumb be that n_training is geometric in n_dimensions, and not linear?

It's completely true that the number of examples should be related to features. But it is not only the number of features because the range of a number(max-min and count of different number) is also important. On the other hand, if you have noise you need more examples, so it's related to your dataset.


Different learning rates for each dimension I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent.

One thing that got me wondering is that we use a pre-defined set of learning rates for each of our dimensions. Now I believe if we don't normalize them we would need to keep our learning rate small due to different scales of the features, but that got me thinking, can't we have different learning rates (suited to each feature dimension scale) to compensate for not doing the normalization?

The answer to your question is yes, and it is already implemented due to popular approaches which are introduced in academic papers. For instance, Adam optimisaiton approach is an optimiser which assumes it is not valid to multiply each dimension, feature, with a specified value. It uses the idea of momentum and geometric average and a bunch of other methods to employ the idea that you should not go down the hill with the same speed for different dimensions. What it says is that you may be in a location which each dimension has different slope, and due to this fact you should not go downhill with the same learning rate. In that approach, although you specify the same learning rate for the optimiser, due to using momentum, it changes in practice for different dimensions. At least as far as I know, the idea of different learning rates for each dimension was introduced by Pr. Hinton with his approache, namely RMSProp.


How should I sample from a mixture distribution? Let's say we have a mixture distribution, defined by density $f(x)= w_1 p_1(x) + w_2 p_2(x)$, where $w_i$ is a scalar weight. Furthermore, we have efficient methods to evaluate the pdf and cdf/icdf for the distribution $D_i$ corresponding to density $p_i$. I would like to sample from such a distribution.
The method I currently employ is an implementation of rejection sampling. I construct a proposal function $M*u(x)$, where $u \sim \text{Unif}(lb,ub)$ ($lb,ub$ are constructed such that at least 99% of the cdf of each $D_i$ is contained within, using icdf) and $M$ is constructed by finding the $\max_{x \in [lb,ub],i} p_i(x)$. Because such proposal function envelopes $f$, I am able to sample $f$ by choosing $x \in X \sim \text{Unif}(lb,ub) \times \text{Unif}(0,M)$ and rejecting if $\pi_2 x > f(x)$ [$\pi_2 x$ being the second coordinate of $x$].
However, doing this is quite slow. Not only is the proposal function inopportune (it is scaled uniform, which likely will lead to many rejections), but the construction of $M$ is very slow, as maximizing a function is a non-trivial task. Is there a more efficient way to sample such a distribution? I had considered icdf sampling, but constructing the icdf for $f$ seems non-trivially difficult. Is this impression incorrect? Or perhaps is there some other effective method? If it is helpful, I am implementing this in python and am currently using the scipy and pytorch libraries.

The mixture distribution can be obtained in the following way.
Let $f(x)=w_1p_1(x) + w_2p_2(x) + ... + w_np_n(x)$, where $p_i$ are density functions and $w_i>0$. Note that $f(x)$ is a density function if the sum of all weights is one.
Then, we use the following two-stage process.
Stage 1. Draw a random variable $X$ (selector, if I remember correctly), such that $P(X=i)=w_i$ for $i=1,2,...,n$.
Stage 2. Return a random variable drawn accordingly to $p_X$, where $X$ is the index obtained in the stage 1.


Learning cost function for linear regression (Me: Never learned calculus or advanced math and I started Stanford openclasses for machine learning. I know basic matrix calculations.)

One chapter of my course is about cost function. I have been trying to find any example calculation of it with numbers. Googling only finds the same formula everytime, and also on Octave. But I want to do the same thing first with pen+paper and without it, I cannot understand. Please give me a very simple example of using the formula with numbers. Thanks a lot.

I require a cost function calculation example for following sample dataset:

#Rooms = Rent
1 = 4000
2 = 10000
3 = 22000
4 = 30000

There is a worked out example in the Wikipedia page for "simple linear regression"

Just for the sake of it, let me plug in your example into the formulas:

The fitted model should be a straight line with parameters $\alpha$ (value at $x = 0$) and $\beta$ (the slope):

$$f(x) = \alpha + \beta x$$

The values for these parameters that minimize the distance between line and data points are called $\hat{\alpha}$ and $\hat{\beta}$.  They can be computed out of the data point values by using these formulae, derived here:

$$\begin{align}
\hat{\beta} & = \frac{ \overline{xy} - \bar{x}\bar{y} }{ \overline{x^2} - \bar{x}^2 } ,
\\
\\
\hat{\alpha} & = \bar{y} - \hat{\beta}\bar{x}
\end{align}
$$

where an expression with an overline $\overline{xy}$ means the sample average of that expression: $\overline{xy} = \tfrac{1}{n} \sum_{i=1}^n{x_iy_i}$.

Here are the values I find for the datapoints you have listed in your question:

$$
\begin{align}
\overline{xy} &= \frac{1}{4} \sum{<(1 \times 4000), (2 \times 10000), (3 \times 22000), (4 \times 30000)>} \\
              &= \frac{1}{4}(4000 + 20000 + 66000 + 120000) = 52500, \\
\overline{x} &= \frac{1}{4} \sum{<1, 2, 3, 4>} = 2.5 ,\\
\overline{y} &= \frac{1}{4} \sum{<4000, 10000, 22000, 30000>} = 16500 , \\
\overline{x^2} &= \frac{1}{4} \sum{<1^2, 2^2, 3^2, 4^2>} = 7.5 , \\
\overline{x}^2 &= 2.5^2 = 6.25
\end{align}
$$

and the fitted line should be:

$$
\begin{align}\
\hat{\beta} &= \frac{52500 - 2.5 \times 16500}{7.5 - 6.25} = \frac{41250}{1.25} = 33000 , \\
\hat{\alpha} &= 16500 - 33000 \times 2.5 = - 66000 , \\
\Rightarrow f(x) &= - 66000 + 33000 x
\end{align}
$$

Therefore, the model would predict, for a house with 10 rooms, a rent of:

$$ f(x) = -66000 + 33000 \times 10 = 264000 $$


How to train a classification algorithm with normalized data set using scikit-learn python I have a dataset in CSV format, 6 columns and 1877 rows. The full dataset can be viewed at ShareCSV.

The first five columns are characteristics and the final column is a binary result, I want to create a classification network to predict result using the five inputs as seen in the CSV above.

I use the following code to normalize the data with pandas, then split into testing and training groups.

from sklearn import preprocessing
import pandas as pd

df = pd.read_csv(r"D:\path\data.csv", sep=",")

df=(df-df.min())/(df.max()-df.min())

X = df[["Title words", "Hour posted", "Author age", "Is Text", "Subreddit age"]]

Y = df['Front']

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

I now need to pass this data to scikit-learn and select a classification algorithm, however this is where I am unsure what would be optimal, if anyone could recommend the best algorithm for my data and a rough implementation that would be great.

I would suggest you to scale your data using standard scaler and do it before you split it into X and Y here in your case. Why ? Please check this answer on stats sc.
Also, keep the target variable (Front in your case) as it is. So, according to me, the right choice looks like this, however you can try and experiment with min max scaler too:

from sklearn.preprocessing import StandardScaler
import pandas as pd
df = pd.read_csv(r"D:\path\data.csv", sep=",")
X = df[["Title words", "Hour posted", "Author age", "Is Text", "Subreddit age"]]
Y = df['Front']
scaler = StandardScaler()
scaled_X = scaler.fit_transform(X)
scaled_X = pd.DataFrame(scaled_X, columns = X.columns)
X_train, X_test, y_train, y_test = train_test_split(scaled_X, Y, test_size=0.2, random_state=42)

To start, you can try with logistic regression and move on to svm, or maybe neural networks, if your data is big enough.

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression(class_weight='balanced', C = 0).fit(X_train, y_train)
y_pred = lr.predict(X_test)
precision, recall, fscore, support = score(y_test, y_pred, pos_label= 1, average='binary')

Now, you can choose a good performance metric for the same, looking at the distribution of the class label, whether they are balanced/imbalanced etc.


String harmonization in Python Just landed on a Entity Resolution (Matching) project.
I am working in python and I was wondering if there's an off-the-shelf implementation of string harmonization functions, including:

removal of trailing spaces
removal of multiple spaces
removal of special characters (eg. ü)
...

All this is easily done with string and Regex operations but I do not want to reinvent the wheel and tap into something already ready (and perhaps learn from that, shall I need to move on to something custom later).
Are there readily available libs for that? Tried nltk, which suggests the use of regex expression, textblob, and googling around but found nothing really helpful so far - probabily because I did not get to use the right keyword.

As an example, it would be useful to have a function that replaces special characters from foreign languages with morphological equivalents - for example "ß" in German with "ss" ("straße" -> "strasse")

Edit: clarification
In my question, I am not really specifying the problem at hand. This is because what I am trying to get to is really a "grammar" that would let me and the other ppl in my team define workflows of harmonization/canonicalisation operations.

I am at the moment Working on this as a chain of functions f: string -> string. Whether some steps are part of it, it will depend on the problem at hand. Workflow will be likely tested empirically (we'll pick the workflow that leads to the best performing algo)

Note: this is more frequently called canonicalisation than harmonisation.  URLify and URL-safe are also terms that can help you in your search.

whitespace
Combining the built-in functions string.split + string.join is a solid way to canonicalise intermediate and trailing whitespace, including tabs, carriage returns and so on.

non-ASCII chars
ASCIIification is actually somewhat contrary to canonicalisation.  That is, you should think hard about whether you really want to be converting 'München' to 'muenchen', rather than 'muenchen' to proper 'München'.

If you do want to go in that direction, you should give more details on the use case and the desired conversions.  Here are some examples of possible input:

  Where is Münsterstraße?
  ısparta'daki istanbul'dakı ve gülenciler!
  Isparta'daki İstanbul'dakı ve Gülenciler!
  ¿Cómo está?
  Это вообще фарс...
  Биће белајааа, чорба
  Biće belajaaa, čorba
  You can now resume your résumé.

Converting to ASCII will be lossy and there are some decisions to make, often dependent on the source and target language.

If you define the likely input, scope and desired behaviour then others can make recommendations.


Wrangling data for CNN I am using convolutional nets for a physics application. I am trying to figure out how to structure my raw data as an image for input into the CNN.

I have $N$ samples. Each sample consists of the following:

(1) Temperature values at discrete times and at discrete spatial locations.
I.e., temperature is a function of time, $t$, and space, $(x,y,z)$.

(2) TBD, but it should consist of some other spatial and temporal data. I haven't figured this out yet. For this question and for simplicity, we could just limit it to working with (1) only.

The examples that I've looked at deals with images, and they're inherently 3D. e.g., 100 pixels wide by 50 pixels high by 3 RGB channels. So the input can be structured as a 3-D tensor.

In my case, temperature depends on 4 independent variables (3 for space and 1 for time). What would be the best way to wrangle it for use as input into a CNN? A 4-D tensor? Or would it be better to map the 4-D tensor to a lower-order tensor, say, 3-D or 2-D?

Also, somewhat of a side note, I am a little bit confused on how an input image relates to the number of nodes in the input layer of the NN. Say, we have an image that is 100 pixels wide by 50 pixels high with 3 channels. Does this mean there should be 100x50x3=15,000 nodes in the input layer?

I'm relatively new to this, so I don't really know what I'm doing. Any insight would be greatly appreciated.

-----------------Edit 1---------------

After some thinking, this is one format that I'm considering. This format represents my temperature values in an image-like matrix:
$$
\begin{bmatrix}
T(x_1,y_1,z_1,t_1) & \cdots & T(x_1,y_1,z_1,t_n) \\
\vdots             & \ddots & \vdots \\
T(x_m,y_m,z_m,t_1) & \cdots & T(x_m,y_m,z_m,t_n) \\
\end{bmatrix}
$$
where $m$ is the total number of locations and $n$ is the number of discrete time intervals. Each column is constant in time, but varying in space. Each row is constant in space, but varying in time. The matrix elements closest to each other (i.e., closer in space and time) have temperature values that are similar, and elements further from each other have temperature values that are less related.

In this format, the number of channels would be 1. And my input into the first convolution layer would be $m \times n \times 1$. Does this format fit the definition of a 2-D topological order? It seems to me that it does since this matrix abstraction is structured such that elements close to each other should be more similar than the elements further apart (see bolded statement above).

Another format I am considering is similar to the one suggested by Antonio in his answer. In this format, the number of channels would be equal to the number of discrete times, so $n$, and my "image" becomes a 1-D column (or I guess it could be a row vector too):
$$
\underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_1)  \\
\vdots              \\
T(x_m,y_m,z_m,t_1)  \\
\end{bmatrix}}_{\text{channel 1}},
\  \ \underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_2)  \\
\vdots              \\
T(x_m,y_m,z_m,t_2)  \\
\end{bmatrix}}_{\text{channel 2}},
\ldots
,\underbrace{\begin{bmatrix}
T(x_1,y_1,z_1,t_n)  \\
\vdots              \\
T(x_m,y_m,z_m,t_n)  \\
\end{bmatrix}}_{\text{channel n}},
$$
So each sample input is now $m \times 1 \times n$, and for each channel we have a $m \times 1$ column vector.

When you use a 2D-convolutional layer, you assume that there is some 2D topological order in your data. E.g. images have 2D topological order: it's important that some pixel is at position (x, y) - if we move that pixel to some other position, the image would be wrong. The number of channels can be arbitrary in images (1 for grayscale, 3 for RGB), but we still consider this as a 2D topological order.

Check the Table 9.1. at chapter 9.7. Data Types (Deep Learning book by Ian Goodfellow and Yoshua Bengio and Aaron Courville).

So, since your data doesn't have 2D topological order, but 1D, you should use 1D-convolutional layer with multiple channels.

You could have 4 channels: 1 for time, and 3 other for space coordinates.


Understandable and explainable machine learning model I want to find formula for best financial portfolio.

Inputs: Historical fundamental data for last 15 years. For 3000 companies for every quatal we have things like market cap, cash flow, liabilities, assets, revenue etc, every one of these inputs is a time series with data points for every quartal (i.e. 4 times in a year).

Goal: I want to find a formula that given the 3000 companies would allow me to rank (sort) it from best to worst. And then I can pick let's say top 10 of those as the best portfolio.

Example, how human can solve this task: study those historical financial data for couple of years, and came up with something. Like this very simple formula rank = revenue[last]/total-assets[last] + revenue[last]/market-cap[last] (looking for highly profitable and not too much overpriced companies).

Machine Learning way to solve it: feed those data points to Neural Net, run it and hope it will learn something after a while.

The problem is that we can't analyse the the neural network result because it's a black box. And I want to see the final formula in more or less compact form and be able to understand what it does.

What other technics other than Neural Nets can be used? The model has to be sophisticated enough to be able to work with the time series as inputs.

Neural networks are convenient for automatically building features from the data, but as far as I know this is not compatible with interpretability since we don't know what the features represent. So the only way I'm aware of to obtain predictions that can be explained and analyzed is to use traditional models, and this would certainly require a bit of feature engineering in this case. I would recommend decision trees which are completely transparent in their decision process and can be understood even by non-experts. With this kind of model the idea would be to provide the features which represent the evolution across time, e.g. min/max/average over the last N years, difference current-N, etc., possibly for several values of N.


Classifier Threshold I am designing a classifier for an Imbalanced Data set. I have a queries regarding choosing the threshold for a classifier, currently I am using mean of the predicted probabilities as the threshold and I am maximizing the recall on positive classes. Is this a correct way of choosing a threshold (I.e. mean of predicted probabilities) or I should try doing something like this book by using recall as a metric on KFold cross validation.

Any suggestion would be highly appreciable. Thanks :)

The threshold you choose depends on the specifics of the problem you are trying to solve.  More specifically, it should be based on how you weigh false positives vs. false negatives, i.e. how bad each of these are relative to each other. You mention that you are trying to maximize recall on the positive class, but if that were true you would should just classify everything as a positive class, and get a recall of 1.0.  Based on the domain you are working in, you should decide how much a false positive 'costs' vs. how much a false negative 'costs'.  Once you decide this, you can find the threshold that minimizes the function

total cost = false negative count x FN cost + false positive count x FP cost


Web Framework Built for Recommendations I'm wondering if there is a web framework well suited for placing recommendations on content.

In most cases, a data scientist goes through after the fact and builds (or uses) a completely different tool to create recommendations. This involves analyzing traffic logs, a history of shopping cart data, ratings, and so forth. It usually comes from multiples sources (the web server, the application's database, Google Analytics, etc) and then has to be cleaned up and processed, THEN delivered back to the application in way it understands.

Is there a web framework on the market which handles collecting this data up front, as to minimize the retrospective data wrangling?

I haven't seen anything like that and very much doubt that such frameworks exist, at least, as complete frameworks. The reason for this is IMHO the fact that data transformation and cleaning is very domain- and project-specific. Having said that, there are multiple tools that can help with these activities in terms of partial automation and integration with and between existing statistical and Web frameworks.

For example, for Python, the use of data manipulation library pandas as well as machine learning library scikit-learn can be easily integrated with Web frameworks (especially Python-based, but not necessarily), as these libraries are also Python-based. These and other Python data science tools that might be of interest can be found here: http://pydata.org/downloads. Specifically, for cleaning and pre-processing tasks, which you asked about, pandas seem to be the first tool to explore. Again, for Python, the following discussion on StackOverflow on methods and approaches might be helpful: https://stackoverflow.com/q/14262433/2872891.

Consider an example of another platform. The use of pandas for data transformation and cleaning is rather low-level. The platform that I like very much and currently use as the platform of choice for data science tasks is R. Rich ecosystem of R packages especially shines in the area of data transformation and cleaning. This is because, in addition to very flexible low-level methods of performing these tasks, there are some R packages, which take a higher-level approach to the problem, which may potentially improve developer's productivity and decrease the amount of defects. In particular, I'm talking about two packages, which I find very promising: editrules and deducorrect. You can find more detailed information about these and other R packages for data transformation and cleaning in my another answer here on Data Science StackExchange (paper that I reference in the last link there could be especially useful, as it presents an approach to data transformation and cleaning that is generic enough, so that could be used as a framework for this on any decent platform): https://datascience.stackexchange.com/a/722/2452.

UPDATE: On the topic of recommender systems and their integration with data wrangling tools and Web frameworks, you may find my other answer here on DS SE useful: https://datascience.stackexchange.com/a/836/2452.


Why don't we gradually update the activation parameters in RNN from one activation to the next as the network is learning more? I'm very new to (unidirectional, vanilla) RNN and sequence modeling in general, and all I understood about the motivation on having the connection between two successive hidden layers/activation is that: this connection is needed to reuse the information learnt from the $t$-th part of the $i$-th sequence, $x_i^{<t>}$ to learn about the $x_i^{<t+1>}$, i.e. the $(t+1)$ -st part of the (same) $i$-th sequence.

Correct me if I'm wrong, but I fail to see the motivation behind using the same activation-to-activation parameter set $\theta_{aa}$ for every connection between two successive hidden states, except of course that: we have less number of parameters to estimate while we minimize the cost. I can't help thinking that $\theta_{aa}$ should be gradually updated with each hidden state, as more information (=part of the same sequence, i.e. words in case of translation) comes in. See the example below.

Let's consider the example of machine translation from English to French for example: (EN) "I am a man" to (FR) "Je suis un homme". Here intuitively, the RNN should try to learn that "am" occurs with certain probability after "I" in English, and correspondingly in French, "suis" occurs with certain probability after "Je"; but given that it's already learnt that, the (conditional?) probability of the occurrence of "un homme" after "je suis" can be more effectively estimated when we know the probability of "a man" occurring after "I am". So intuitively, the RNN should be "better informed" when it knows more parts of the given sequence than lesser part of it, and hence the activation parameters should be gradually updated accordingly.

I must be missing something, but not sure what it is? I've only motivated myself using the machine translation example, but examples from other areas would also be appreciated.

The motivation to use RNN is that the length of the sequence or position info is random in the data.

For example we could use the sample trained RNN model to translate the following setences:

I am a man
I am a woman and you are a man

In RNN, we do not consider the position of the words, only consider the relation between words. Thus the different activation parameters trained for different positions are useless.

Moreover to make RNN better informed (using previous words / next words) we could use Gated recurrent unit (or LSTM) and Bidirectional RNN.


Classification with two 1-NN classifiers I have two images, let's say old and new. In the old one there are 1:M objects, in the new one 1:N. I need to label the objects in the new one based on some metric (yes, I realize object tracking is a whole complicated area, but I need some   entry point right now). So I've obtained a metric based on the distance to the objects in the previous image, sort of 1-NN. I've also obtained a different metric, also sort of 1-NN, based on pixel intensity (color). Of course, I want to use both of these features somehow. For example, based on the distance metric object 3 in the new is closest to object 5 in the old, and based on the pixel intensity metric - closer to object 9 in the old image.

Any suggestions on how to combine these two? Perhaps some kernel, but I'm not sure.

There are multiple solutions:

Do linear scalarizing. E.g. If you have $d_0$ and $d_1$ then you could do:

$$D = w_1 d_1 + w_2 d_2$$

You could use the Pareto optimal front. The Pareto optimal front is the set of all distance pairs where one of the distances cannot be dominated without deteriorating the other one. This solution is probably not that well suited for your problem.
If you have a training set of linked object you could learn a model that predicts whether or not the objects are the same based on the distances. E.g. A logistic regression with the two distances as input. You could use a kernel or other non-linear tools to increase performance.

More ideas on this wiki about multi-objective optimization.


In Naive Bayes classifier how is P(sneezing,builder|flu) = P(sneezing|flu)P(builder|flu)? Please refer to this literature:

According to Naive Bayes classification algorithm:

$P(sneezing,builder|flu) = P(sneezing|flu)P(builder|flu) $

where sneezing and builder are independent events.

How do they arrive at the above conclusion mathematically?

Is it something like:

$P(sneezing,builder|flu)$
$=P(sneezing \cap builder| flu)$
$= \frac{P((sneezing \cap builder) \cap flu)}{P(flu)}$

This is just a basic property of conditional independence. If two events A and B are conditionally independent, given event C then:

$$Pr(A \ and\ B \ |\  C) = Pr(A \ |\ C) * Pr(B \ | \ C)$$

or equivalently,

$$Pr(A \ | \ B \ and\  C) = Pr(A \ |\  C).$$

That is,

$$ Pr(A  \ and \  B | \ C) $$
$$ =\frac{Pr(A \ and \  B \ and \  C)}{Pr(C)} $$

(definition of conditional probability)

$$= \frac{Pr(A \ | B \ and\ C) * Pr (B \ and \ C)}{Pr(C)}$$
(by Bayes rule)
$$= Pr(A \ | \ C) * Pr(B\ | \ C) $$

(independence and definition of conditional probability again)

The independence assumption is what makes Naive Bayes "naive" because it is a bold assumption in general to think your variables are all conditionally independent, given a class. In Naive Bayes, we seek to find an expression for:

$$Pr(class \ r \ | x_1, x_2, x_3..., x_k)$$ where $x_1, x_2, x_3, ... x_k$ are the features/variables/predictors that we have. Applying Bayes rule;

$$Pr(class \ r\ | x_1, x_2,...,x_k) = \frac{Pr(x_1, x_2, ...x_k\ | class \ r) * Pr(class  \ r)}{\sum_{i=1}^m Pr(x_1, x_2, ...x_k\ |\ class \ i) Pr(class \ i )}$$

where m = number of classes.

The expression $Pr(x_1, x_2, ...x_k\ | class \ r)$, the conditional joint distribution of the predictors, is essentially impossible to estimate directly. So we make the assumption of conditional independence in the predictors, given a class (and that the distribution of $(x \ | \ class \  i)$ follows Gaussian, typically, but not always)

$$Pr(x_1, x_2, ...x_k\ | class \ r) = \prod_{j = 1}^{k}Pr(x_j \ | \ class \ r)$$

Hence:

$$Pr(class \ r\ | x_1, x_2,...,x_k) = \frac{\prod_{j = 1}^{k}Pr( \ x_j | \ class \ r) * Pr(class  \ r)}{\sum_{i=1}^m \left[\prod_{j = 1}^{k}Pr(x_j | class \ i) \right]Pr(class \ i )}$$.

We typically estimate the prior probabilities, $Pr(class \ i )$, using the MLE = sample proportion; # of training examples in class i / # of total training samples.


Is correlation needed when building a model? Some papers report the correlation between features when building a model and some don't. Is there a need to check the correlation between features and target feature? It won't be easy if the number of features is high.

Not really, no. Sort of. It depends on how complex your model/data is.

It's entirely possible to have a situation where a feature taken in isolation will not be correlated with the target variable, but multiple features considered together will. This is why univariate correlation is unreliable for feature selection. A trivial case that demonstrates this is a bivariate model performing a binary classification where the positive class is bounded by the right upper and left lower quadrants, and the negative class is bounded by the left upper and right lower quadranta (i.e. the "XOR" pattern):

If the input features have the same sign (x>0 & y>0 or x<0 & y<0), it's the positive class, else it's the negative class. But either feature in isolation is completely useless and uncorrelated with the target.

Additionally, modern models like deep neural networks are effectively capable of "learning their own features", i.e. constructing extremely complex features by developing abstractions from the raw inputs. The "final" features learned by such a model will likely be correlated with the target, but the input features need not be.

For example, if you consider the imagenet task (classifying a photo as a member of one of 10,000 classes), I'd be very surprised to learn that there's any correlation between the values of specific pixels and any target class. That's just not how we interpret photos. The value of the pixel in position [25, 10] should not have any correlation with whether or not the picture is a photo of a dog. But, if we think of the entire network before the output layer as a feature engineering system (such that the "classifier" is just the output layer), then the "features" provided by the penultimate layer (the inputs to "The Classifier") probably have some correlation with the target.

TL;DR: If a feature is correlated with the target, it probably contains information about the target that will be useful for modeling. But that does not mean uncorrelated features are useless. Reporting correlation when it's there is a simple way to demonstrate that there's a signal in that variable. But lack of correlation doesn't necessarily mean you should throw those features away. In fact, correlation doesn't even mean you should necessarily use that feature either: you can have multiple features correlated with the target that are highly correlated with each other, in which case you would probably only want one or a handful of that group in your model.


Is it possible to classify documents of corpus using labels? I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, not topics).
So I followed a 2-step approach:

Synthetically generate labeled data (using a rule-based labeling approach, obviously the recall is very low, ~ 1/8 documents are labeled)
Somehow, use this labeled data to identify labels for other documents.

I have attempted the following approaches for step 2:

Topic modeling on data classified using rules to extract significant terms and using significant terms to label the remaining documents.
Finding significant terms using sentence embedding
Using sentence embedding as features for my classifier

But I haven't been successful in getting good results for my document classifier. Are there any other methods that can be used to classify the documents?
All help is greatly appreciated.

You could simply encode the documents using BERT and cluster the documents based on their content provided they sufficiently different in terms of the kind of content they contain.

Another approach would be to train a document segmentation model which would segment documents based on their semantic structures and then classify the documents based on their masked skeletons. This however would require a large dataset to train. Fortunately you can find one online called PubLayNet. Augment that with a few representations of your documents for better generalization over the test set.

I've read about the second approach being implemented to classify patents, legal documents, research papers etc. With good results. However it would take a long time to train.
I'd recommend simply clustering the documents based on their text embedding (point 1) and then naming the clusters. If that doesn't work satisfactorily, try the deep learning method for document semantic masking.


Sanity check: low PPV but high AUC scores? I have two algorithms running on a piece of data, both of which perform differently.

One of them (call it A) consistently gets a positive predictive value of about 0.75-0.78. Looking at the AUC of the Receiver operating characteristic it has a score of about 0.82

The second algorithm (let's call it B) consistently get's a lower predictive value of about 0.72-0.75. However this gets a higher AUC of the Receiver operating characteristic value of about 0.85.

Does this definitively indicate an error of some kind as AUC is so tightly associated with the positive predictive value? Or is this entirely reasonable, subject to other factors?

AUC of ROC is not affected by PPV (precision), but by recall (true positive rate) and fall-out (false positive rate).

As per your results, model A gets a better PPV hence a lower recall and a lower ROC AUC. Inversely, model B gets a better recall hence a better ROC AUC and a lower PPV.

So this is entirely reasonable due to the definition of the two metrics.


experimental design in R project I want to know of any repositories that contain complete experimental design in R covering basic test and analyses?
I want to take a top-buttom approach to learn step by step through a real project how that works. Do you know of any places to find that, such as an R notebook?

I suggest you take a look at the TidyTuesday repo, where every week they post a raw dataset, a chart or article related to that dataset, and ask you to explore the data. The repo also contains other resources, like data science books.
Together with the repo, I suggest the TidyTuesday videos by David Robinson, where he creates screencasts of complete data analyses in R for the TidyTuesday datasets.


Removing duplicate rows in orange3 In Orange3, I have a table with row duplicates. How can I simply remove the duplicated rows to keep only one copy of each?
I feel like Orange3 is great but cannot do simple tasks. I hope I just did not find the right widget.

Alteryx user, I was looking for the Unique or Summarise tools...
My work around is to use the Pivot Table widget
Particularly useful when you want to merge data and overcome the cartesian block of Orange.


How to run unmodified Python program on GPU servers with scheduled GPUs? Say I have one server with 10 GPUs. I have a python program which detects available GPU and use all of them.
I have a couple of users who will run python (Machine learning or data mining) programs and use GPU.
I initially thought to use Hadoop, as I find Yarn is good at managing resources, including GPU, and YARN has certain scheduling strategies, like fair, FIFO, capacity.
I don't like hard-coded rules, eg. user1 can only use gpu1, user2 can only use gpu2.
I later find Hadoop seems to require the program written in map-reduce pattern, but my requirement is to run unmodified code as we run on Windows or local desktop, or modify as little as possible.
Which knowledge should I look at for running and scheduling python programs on a machine with multiple GPUs?

A popular solution used for job management on GPU environments is SLURM.
SLURM allows specifying the resources needed by a job (e.g. 2 CPUs, 2Gb of RAM, 4 GPUs) and it will be scheduled for execution when the needed resources are available.
A job can be any program or script.


Data manipulation in R I have a long data frame having 96 consecutive observation belonging to same variable.
I have to two such variables. One variable is Energy denoted by "param2(kWh)" and another is demand denoted by "param2(kW)". So data frame is like---

first row has variables described as "Interval" and "param2(kWh)" and then 96 observations belonging to Energy variable, followed by 98th row having variables described a "Interval" and "param2(kW)" and then 96 observations belonging to Demand variable.

Again after that there is row described as "Interval" and "param2(kWh)" and then 96 observations belonging to Energy variable. After that again there is row describing variables "Interval" and "param2(kW)" and 96 observations belonging to Demand variable and such intertwining of these two variables i.e. Energy and Demand continues for long.

The objective is to have two separate data frames one for Energy i.e. "param2(kWh)" and one for Demand i.e. "param2(kW)".
How these two data frames be formed from the above single data frame?

I am attaching the internet site of the data frame.

https://1drv.ms/f/s!Apm2LEjQkhz3aNBBoUygHPMz2_E

Here is one way to generate two data frames based on the data.

# read data
dat <- read.csv("sample csv1.csv", stringsAsFactors = FALSE, header = FALSE)

# find rows containing "Interval"
idx_int <- dat[[1]] == "Interval"

# calculate groups
group <- cumsum(idx_int) %% 2

# remove indices for "Interval" rows
group2 <- group[!idx_int]

# split data frame into two data frames
dat_list <- split(dat[!idx_int, ], group2)

# generate two separate data frames
dat_energy <- dat_list[[2]]  # Energy
dat_demand <- dat_list[[1]]  # Demand


L2 regularization increase the loss rate of the deep learning model When I add L2 regularization to my deep learning model the training and validation loss rate is  increased.

Why ????

Suppose a neural network with a regular loss function.

$$
\sum_{i=1}^N L \left( y_i, \; \hat y_i \right)
$$

Here, $y_i$ is label for the $i$-th example, while $\hat y_i$ is the model's prediction for the same. The loss function $L$ is a function that compares the actual and the predicted output and outputs a value indicating how close the prediction was to the actual output.

L2-regularization adds a norm penalty this loss function and as a result to each weight update.

$$
\sum_{i=1}^N L\left( y_i, \; \hat y_i \right) + λ \cdot \|W\|_2^2
$$

This penalty counters the actual update, meaning that it makes the weight updates harder. This has the effect of actually increasing the output your loss function.

What you should be looking for, by adding regularization to a model, isn't a reduction in training loss but a reduction in the validation loss. This would indicate that the regularization is successful in reducing your model's overfitting, which was its goal.


How to improve identification of outliers for removal I have many datasets where the measured value is either "normal" (i.e. the process is running" or abnormal (i.e. process is not running).

Unfortunately, I don't have a measurement that clearly indicates that the process is running, so I have to make the determination from the measured value data itself. Visually, the outliers are obvious (see below).

I have had success with several different techniques

IQR based outlier identification
3 sigma rejection
simple cutoff
modified z-score

These all work some of the time, but I am looking for a better solution for my data.

If you look at the image below, the outliers are all of the points near zero. In this specific case a threshold of 0.01 easily eliminates all of the bad data. Unfortunately, the threshold required for a given dataset can vary significantly.

I would appreciate any thoughts.

You can try a quantile-quantile plot: you fit your data to a given distribution and plot your empirical distribution quantile against the theoretical quantiles.

This can help you infer outliers in your sample as outliers for the fitted distribution.


Latent Dirichlet Allocation vs Hierarchical Dirichlet Process Latent Dirichlet Allocation (LDA) and Hierarchical Dirichlet Process (HDP) are both topic modeling processes. The major difference is LDA requires the specification of the number of topics, and HDP doesn't. Why is that so? And what are the differences, pros, and cons of both topic modelling methods?

HDP is an extension of LDA, designed to address the case where the number of mixture components (the number of "topics" in document-modeling terms) is not known a priori.  So that's the reason why there's a difference.

Using LDA for document modeling, one treats each "topic" as a distribution of words in some known vocabulary.  For each document a mixture of topics is drawn from a Dirichlet distribution, and then each word in the document is an independent draw from that mixture (that is, selecting a topic and then using it to generate a word).

For HDP (applied to document modeling), one also uses a Dirichlet process to capture the uncertainty in the number of topics.  So a common base distribution is selected which represents the countably-infinite set of possible topics for the corpus, and then the finite distribution of topics for each document is sampled from this base distribution.

As far as pros and cons, HDP has the advantage that the maximum number of topics can be unbounded and learned from the data rather than specified in advance.  I suppose though it is more complicated to implement, and unnecessary in the case where a bounded number of topics is acceptable.


When should I normalize data? I often see that numeric values in machine learning is scaled to 0-1 range. Why is it better?

I have some temperature values in my training set.
What if I will have some values to predict that will be outside training set values?

I mean that eg in training set I will have range of temperatures like 5-20 and MinMaxScaler will fit to these values, and then I will have 25 to predict.

It is actually depends on the algorithm you are using.
For example, for random forests the ranges don't matter, since one feature is never compared in magnitude to other features. It's only the range of one feature that is split at each stage.

But on the other hand SVM or Logistic regression will probably do better if your features have roughly the same magnitude, unless you know apriori that some feature is much more important than others, in which case it's okay for it to have a larger magnitude.


in binary classification where class labels are {-1, 1} is preprocessing needed? In machine learning we convert labels using LabelEncoder to convert string ex:{"malignant", "benign"} -> {0, 1}
I am wondering if converting Labels to any other numbers matter, in my scenario to {-1,1}.
If it matters, reason provided will be helpful (or direction of helpful resource)!
Thanks in advance!

You need {0,1} or {-1,1} labels depending on the output of your model. If you have a Sigmoid output use {0,1}, while TanH outputs work with {-1,1}.
No label choice is inherently right or wrong, as long as it's compatible with your model architecture and gives you good results.

EDIT:
In case of logistic regression you must use {0,1}, that is because this class of models has a Sigmoid output. Sigmoid function is always bounded in [0,1] and it can't take values outside of that range. It could never reach -1 and training won't work.


LSTM forecasting on multivariate time series I'm new to RNNs and LSTM and would like some direction with a problem I have. I have a data set containing system metrics (like CPU utilization, disk operations, memory use) of an AWS EC2 instance with a total of 7 columns and around 8000 rows. Each row also has a timestamp with a 5 min interval between each row.

I want to build a LSTM model to forecast the performance for let's say the next hour based on my data. What would be the best approach for solving a problem like this? I know this can be done in many different ways but I would really appreciate some input how to go about this.

LSTM should be a good fit for this problem(as you mentioned in question).  It can identify patterns in sequence of data (For example : high memory usage is a leading indicator of  higher disk usage [due to higher usage of swap partition]).
This is a tutorial for similar use-case.


How to reduce the error in linear regression In linear regression, how can we minimize the error term?

Answer based on the comment:

  Intention here is to reduce the error by all means and considerations

Choice of model: First let's address the obvious assumption: linear regression is a model which requires the response variable to be expressed as a linear combination of the independent variables. In order to improve performance in general one must make sure that these constraints are satisfied. If not it's often worth looking into other models or finding a way to "linearize" the data.
Data cleaning: depending on the size of the data, linear regression can be very sensitive to outliers. If it makes sense for the problem, outliers can be discarded in order to improve the quality of the model. Of course one shouldn't remove points just because it decreases the error, this has to be done sensibly with respect to the characteristics of the task.
Feature engineering: it's worth analyzing/experimenting which of the independent variables actually help obtaining a good model. For instance redundant variables may decrease performance. It's also possible that variables can be expressed differently or transformed in a way which improves performance.


How is the Beeswarm plot better than a histogram? I have read about Beeswarm and Histogram plots. But I did not understand the difference. In a Histogram, if the bins number is increased then its intrepretation is changed. So how can a beeswarm plot help to understand the data?

They are quite similar but :

Histograms results of aggregation into bins (loss of information due to aggregation: high-resolution point (2.324) is reduced to bins resolution [2,3])
Beeswarm plot displays the exact value of points on the axis. It is a 1D-scatter plot.

However, if two points are very close to each other they will overlap if the point size is big enough. To avoid this, the beeswarm plot will slightly shift one point to the side so it does not overlap.

They will often look similar because the more points you have locally, the higher the probability of overlapping. Then, high-density areas will often result in "peaks" as in histograms.

The beeswarm plot is a more faithful representation

Gap in the data: depending on the binning (both placement and size), the histogram may not represent it (as on point 1 in the figure).
Min and max: Imagine values are limited at 5 but the histogram last bins continues to 6, it might give the impression that the max value of the variable is 6.

In both cases, you can see that beeswarm is a more faithful representation.

Because of this, it is not highly scalable (it has to represent every single point with no overlap). On the seaborn.swarplot function docs :

  This function is similar to stripplot(), but the points are adjusted (only along the categorical axis) so that they don’t overlap. This gives a better representation of the distribution of values, but it does not scale well to large numbers of observations. This style of plot is sometimes called a “beeswarm”.


Random accuracy results on validation set (Binary classification) I am configuring a CNN model for a classification problem of human action in Python using TensorFlow.

My data is video frames representing human body joints. Every 3 consecutive columns represent x,y,z coordinates of one joint.

I set the placeholders in tf to be of shape (None, l, j*3) where l=fixed number of frames and j= number of joints. Now the model is giving me very random results (pretty much noise) on validation.

This is the code I used to compute loss and validation and plot results.

for e in range epochs:

# Loop over batches
        for x,y in random_batches(X_train, Y_train, batch_size):

            # Feed dictionary
            feed = {X : x, Y : y, keep_prob_ : 0.5, learning_rate_ : learning_rate}

            # Loss
            loss, _ , acc = sess.run([cost, optimizer, accuracy], feed_dict = feed)
            train_acc.append(acc)
            train_loss.append(loss)

            # Print acc & loss every 5 iter.s

            if (iteration % 5 == 0):
                print("Epoch: {}/{}".format(e, epochs), "Train acc: {:.6f}".format(acc))

            # Compute validation loss every 10 iter.s

            if (iteration%10 == 0):
                val_acc_ = []
                val_loss_ = []

                for x_v, y_v in random_batches(X_valid, Y_valid, batch_size):
                    # Feed
                    feed = {X : x_v, Y : y_v, keep_prob_ : 1.0}

                    # Loss
                    loss_v, acc_v = sess.run([cost, accuracy], feed_dict = feed)
                    val_acc_.append(acc_v)
                    val_loss_.append(loss_v)

                # Print
                print("Epoch: {}/{}".format(e, epochs), "Validation acc: {:.6f}".format(np.mean(val_acc_)))

                # Store
                validation_acc.append(np.mean(val_acc_))
                validation_loss.append(np.mean(val_loss_))

            # Iterate
            iteration += 1

    ## Plotting:

t = np.arange(iteration-1)

plt.plot(t, np.array(train_loss), 'r-', t[t % 10 == 0], np.array(validation_loss), 'b*')

So I thought maybe if I tell the system to consider that every 3 consecutive columns are representing just one joint it would solve this randomness on validation.

Not sure how to do this as tensorflow accepts arrays. And arrays don't have headers unlike dataframes.

0  0.30467  0.45957 -0.95414  1.74687  1.42338 -0.03860
1  0.27331  0.59293 -1.00874  1.74135  1.32004 -0.00701
2  0.30348  0.88129 -1.05517  1.75090  1.65138 -0.03112

What I would want the system to understand is that the first 3 columns represent the same joint and the 2nd 3 columns represent the 2nd joint.

I tried shaping the data to be (None, l, j, 3) but that also gave me random results on validation as well. And when I looked up similar works, they were feeding tf data such as (None, l, j*3).

Another area where I am doubting is wrong (although less likely to be wrong) is the shuffling of the data. I have my inputs data (X) and I have constructed the labels (Y) from the filenames. Below is a snippet of the get batches function (which does the shuffling):

# 1- Shuffle (X, Y)
m= X.shape[0]

    permutation = list(np.random.permutation(m))
    shuffled_X = X[permutation,:,:]
    shuffled_Y = Y[permutation,:]

    # 2- Partition (shuffled_X, shuffled_Y). Minus the end case.

    num_batches = m // batch_size # number of mini batches of size batch_size in the partitionning
    X, Y = shuffled_X[:num_batches*batch_size], shuffled_Y[:num_batches*batch_size]

    # Handling the end case (last mini-batch < mini_batch_size)
    if m % batch_size != 0:
        X = shuffled_X[num_batches * batch_size : m,:,:]
        Y = shuffled_Y[num_batches * batch_size : m,:]

    # Loop over batches and yield
    for b in range(0, len(X), batch_size):
        yield X[b:b+batch_size], Y[b:b+batch_size]

I am a beginner in deep learning and tensorflow. Appreciate any advice or help here.

Ah, I think I've found the problem with the discrete values at least.  In your random batching, you overwrite X and Y to be the remainder of the dataset here:

if m % batch_size != 0:
    X = shuffled_X[num_batches * batch_size : m,:,:]
    Y = shuffled_Y[num_batches * batch_size : m,:]

These datasets only have size $200-3*64=8$.

It's surprising that the training accuracies work out nicer then; is the training dataset perhaps a multiple of 64?

(After you get this fixed up, your original question is still interesting; maybe it should be posted as a separate question?)


How to combine categorical and continuous input features for neural network training Suppose we have two kinds of input features, categorical and continuous. The categorical data may be represented as one-hot code A, while the continuous data is just a vector B in N-dimension space. It seems that simply using concat(A, B) is not a good choice because A, B are totally different kinds of data. For example, unlike B, there is no numerical order in A. So my question is how to combine such two kinds of data or is there any conventional method to handle them.

In fact, I propose a naive structure as presented in the picture

As you see, the first few layers are used to change(or map) data A to some middle output in continuous space and it is then being concated with data B which forms a new input feature in continuous space for later layers. I wonder whether it is reasonable or it is just a "trial and error" game. Thank you.

There's three main approaches to solving this:

Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input
Concating all the data into a single vector/tensor as a preprocessing step and then train a simple single input NN
The multiple input NN architecture you proposed

The ensemble approach is the most straight forward option and will yield decent results, however, it will not work as well as the option you proposed as because the ensemble network only receives class probabilities from the two networks as an input and will in comparison to your approach miss out on more complex relationships between the data types.

The second approach is in theory not that different from your proposed approach, with the difference being that it assumes that the network will on it's own figure out that the input consists of two types of data (as they are both in the same vector/tensor). It will take a lot of training time for the network to learn that and you might get stuck in a local minima before that even happens.

Based on my personal experience, the network you proposed is the best option as and it's likely to have shortest train time and once you get the architecture right you will find it very easy to train and maintain (retrain) the network in production as well as you will only ever have to retrain one single model.


Python Machine Learning Experts I'd like to apply some of the more complex supervised machine learning techniques in python - deep learning, generalized addative models, proper implementation of regularization, other cool stuff I dont even know about, etc.

Any recommendations how I could find expert ML folks that would like to collaborate on projects?

You could try some competitions from kaggle.

Data Science courses from Coursera, edX, etc also provide forums for discussion.

Linkedin or freelance sites could be other possibilities.


Does this type of classification exist? Im fairly new to data science and trying to see if a type of classification exists for my needs.

I understand that a classification into 2 categories will look something like this:

You have 2 desired outcomes and you try to build a model that classifies as 0 or 1. If these models are not 100% accurate then you will:

a) Miss some true values (outside edges of circles)
b) Get some of the wrong values in each category (overlaps between circles)

However, I am looking for something more like this:

In this case, I want to predict only 1, and I dont mind if some 0s are included but want to make sure that as many as possible 1s are predicted.

In my mind, this is effectively just widening the orange circle (the classification of 1s) in the picture.
How can i achieve this?

Instead of formulating the problem with Venn diagrams you could also look at a simple two by two table. Usually the problem is formulated graphically in a different way (see picture below from Wikipedia page). If you are interested in just predicting the occurrence of the value 1 you are just focusing on the sensitivity (or true positive rate) of your classification algorithm.
This is rater straightforward within a ROC analysis framework, you could just select a minimun value for the classifier threshold. However, this comes at the cost of very low specificity. You should also consider the cost-benefit ratio of your sensitivity/specificity results.

https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity


Is it possible to implement a Recommender System without having a ratings/previous purchases similar data? I'm trying to implement a recommender system for a website that hosts a wide variety of softwares and you can search the website to find what you need. The need is to implement a recommender system to better recommend softwares to the users except that there is no ratings system and users don't really buy the softwares from the website...
A first approach would be like this: let's imagine a user clicks on software A, we can search for the most clicked on softwares by all the users who also clicked on software A. But this would lack meaning since these users that clicked on software A may just search the category of this software and would click on the first suggestion from a list that may be sorted alphabetically.
Also the users don't really interact with the softwares. They can't bookmark it or post a comment so we can't just take these actions and give them a weight to simulate a ratings system.
I don't really find ideas about solving this issue and I don't have any experience with user tracking data so any idea would be really helpful.

In the complete absence of user feedback, you can still build a (non-personalized) recommender system based on the products' descriptive features, e.g. by employing some distance metric.
In your case, there are at least two behavioral signals: clicks and search terms. Both can be used to personalize recommendations. I would start with getting familiar with session-based recommendations, which take into account user's actions in the context of the current session, and refine the recommendations incrementally (a kNN algorithm would be a good starting point).


Algorithms to generate a rating system based on history I am a beginner in data science. I have a data set of drivers that has the following attributes available-

Time stamp
Speed
Acceleration
GPS co-ordinates

I need to build a driver rating system to rate drivers on a scale of 1-100 on parameter of speeding. The data set has the data from many drivers and also their past data.
Which is the best algorithm that I can use in this situation?

You need to define what speeding means and how to rate the drivers for amount of such. As you note that you have speed in your dataset this should be rather trivial. Not sure at which stage you want to involve machine learning? Seems like you don't need any.
Machine learning is not the same thing as learning about cars and their drivers.


Correlation with Multi-Dimensional Clustering I have a dataframe with multiple features, where I'm selecting 3 features to cluster on. Ex.
Col1 | Col2 | Col3
-------------------
A    |  1  |   i
B    |  1  |   j
C    |  2  |   k
D    |  3  |   j

I'm trying to cluster such that If there is a matching value in any two columns, it will add that row to the same cluster, for all matches within the values in a cluster.
So from this table, my intended output for a cluster would be:
A    |  1  |   i
B    |  1  |   j
D    |  3  |   j

Is there a clustering algorithm that I can use to achieve this type of clustering?

That is not clustering since the rows are not being grouped together.
It is a filtering problem where the threshold to keep a row is based on frequency in each column.