Predict part of the input for a fixed target I dispose of a data set composed of 6 features all of them are numeric and a binary target(taking 0 and 1). How should I proceed in order to predict the values of 2 features knowing the target and the remaining fixed 4 features.

Thanks in advance.

I am not sure what is the scale of your dataset, but I guess what you're looking for conceptually is Multi-task learning. You can use this concept to carry out multiple tasks, or in your case regression predictions using a single model.What you can try doing is use the four features, and the target variable as input features to predict the two variables.

If you have a large dataset, you can look into deep learning multi-taking approaches here. If you are familiar with python and sklearn, you can try some basic multi task regression algorithms available as a starting point. This algorithm might be helpful, its easy to use. If you are looking for a more deep learning approach, there are many architectures available online that you can try as starting point.


Neural Net Accuracy: Test Set vs Real World Data Neural Net accuracy is high on test set but low on new real world image examples.

Looking for advice regarding what generally causes this scenario and how to fix it.

Sampling basis? Training/test set is not representative of real world data? Obtain more training/test data?

Many resources teach the process of splitting data into training, validation and test sets. This is what you want to do for "closed" datasets where it's not possible to get additional data.

This assumption of a closed dataset is often not true in the real world, where it may be feasible to collect more data. Statistically speaking, it is a lot more desirable to define a test set as a new data sample that was collected separately from your training data. This might be more representative of how the model will behave in production, but sometimes even this is not enough:

A few weeks back I built an image classifier for cars. I trained it using a mix of existing datasets and the results of a web scrape. Ultimately, it was deployed through an iOS app where it was supposed to do make predictions in real time. In this case, it was not enough to just create a test split or collect a new sample from the web. We needed to shoot our own images that were representative for the use case in order to make realistic assumptions about the app's performance.


Predictive output with your own model built I would need to better understand how can be created a machine learning algorithm from scratch using an own model developed based on boolean values, for example # of words in a text, # of punctuation, # of capital letters, and so on, to determine if a text is formal or informal.
For instance: I have
Text
there is a new major in this town
WTF?!?
you're a great person. Really glad to have met you
I don't know what to say
BYE BYE BABY

I created some rules to assign a label on this (small) train dataset, but I would need to understand how to apply these rules to a new dataset (test):

if there is an upper case word then I;
if there is a short expression, like don't, 'm ,'s, ... , then I;
if there are two symbols (punctuation) close to each other, then I;
if a word is in list of extra words, then I;
otherwise F.

Suppose that I have a dataframe to test and to assign these labels (I or F):
FREEDOM!!! I don't need to go to school anymore
What are u thinking?
Hey men!
I am glad to hear that.

how could I apply my model to this new dataset, adding labels?
Test                                                  Output
FREEDOM!!! I don't need to go to school anymore       I
What are u thinking?                                  I
Hey men!                                              I
I am glad to hear that.                               F

Update after mnm's comment:
Would it be considered a machine learning problem the following one?
import pandas as pd
import numpy as np
data = { "ID":[1,2,3,4],
        "Text":["FREEDOM!!! I don't need to go to school anymore",
    "What are u thinking?",
    "Hey men!","
    I am glad to hear that."]}

# here there should be the part of modelling
df['upper'] = # if there is an upper case word then "I"
df['short_exp'] = # if there is a short exp then "I"
df['two_cons'] = # if there are two consecutive symbols then "I"

list_extra=['u','hey']
df['extra'] = # if row contains at least one of the word included in list_extra then 'I'

# append cols to original dataframe
df_new = df
df_new['upper'] = df1['upper']
df_new['short_exp'] = df1['short_exp']
# and similar for others

It is not clear, however, the latest part, that one based on condition. How can I predict the new values for the other texts?

What you are proposing is a heuristic method, because you define the rules manually in advance. From a Machine Learning (ML) point of view the "training" is the part where you observe some data and decide which rules to apply, and the "testing" is when you run a program which applies these rules to obtain a predicted label. As you correctly understood, the testing part should be applied to a test set made of unseen instances. The instances in the test set should also be manually labelled (preferably before performing the testing in order to avoid any bias), so that you can evaluate your method (i.e. calculate the performance).
Technically you're not using any ML approach here, since there is no part where you automatically train a model. However heuristics can be useful, in particular they are sometimes used as a baseline to compare ML models against.

[addition following comment]

I think most of common pre-processing approach requires to convert text into lower case, but a word, taken in different contest, can have a different weight.

This is true for a lot of tasks in NLP (Natural Language Processing) but not all of them. For example for tasks related to capturing an author's writing style (stylometry) one wouldn't usually preprocess text this way. The choice of the representation of the text as features depends on the task so the choice is part of the design, there's no universal method.

how to train a model which can 'learn' to consider important upper case words and punctuation?

In traditional ML (i.e. statistical ML, as opposed to Deep Learning), this question is related to feature engineering, i.e. finding the best way to represent an instance (with features) in relation with the task: if you think it makes sense for your task to have specific features to represent these things, you just add them: for instance you can add a boolean feature which is true if the instance contains at least one uppercase word, a numeric feature which represents the number of punctuation signs in the instance, etc.
Recent ML packages propose standard ways to represent text instances as features and it's often very convenient, but it's important to keep in mind that it's not the only way. Additionally nowadays Deep Learning methods offer ways to bypass feature engineering so there's a bit of a tendency to forget about it, but imho it's an important part of the design, if only to understand how the model works.


Binary classification problem Im new to ML, I have a data set for Music sales info for Vinyls, the data set contains:

Author
Album Title
Genre
Country
RevenueGenerated
AverageRevenueGenerated

My goal is to create a Model which I can help me understand which Music may generate a lot of revenue (Boolean). I created a field AverageRevenueGenerated which is the average of all Revenue generated for all artists.
Im looking for a tool that can help me associate or generate insights based on input signals above. This cold be automatically or a specific guide that allows me to say for example if:

UK + Industrial
IT + Opera
Daft Punk + Electro

Will be potential high revenues.

I found house prices example: https://yalantis.com/blog/predictive-algorithm-for-house-price/ is it the same type of problem?  I'm looking which input signals may be the highest revenue. Any insights or pointers will be helpful.

Yes, you can run a regression using the features you have, and predict the revenue but you will need more features to run an effective analysis. Maybe adding features for the genre and wether the music is trending or not. To turn it into a binary classification problem you will have to decide on a cut-off and label everything above that as 1 and below it as 0, the median might be a good cut-off but solely depends on your data. This entire problem will boil down to feature engineering, which in my view is one of the hardest thing to do. Also you will need a lot of data for this. You can create features wether the song was featuted on the billboard. See, try to think of it like a human and add such features to your model. Like as a person I think the music that will generate more revenue will be :

Popularity and Reputation of the artist.
Wether the genre is trending these days.
Has the song won any awards.

Stuff like this. So you wilk have to find features that quantify these things. Try to find some open source datatset for this kind of problem and work from there.


How to use testing data set to measure recommender system algorithm I am new to recommender systems and am trying to build one using item-to-time CF. Currently, I am trying to evaluate/measure results using MAE. I have one step which is unclear (after I managed to split training dataset and testing data set and computed item-to-item similarities matrix). I understand that I can now do predictions for a given item for a specific user. Let's say from the testing data set, there is User A with 10 items already rated. What should I do with that user? Randomly pick one item I and pretend I didn't know the score of User A given to item I. Then calculate the predicted score using a weighted sum of rest of the 9 items?

Or, should I actually enumerate all 10 items that User A has ratings for. Then each time take out only the target item and using rest of the 9 items to do predictions?

I guess the right answer might be just predict all 10 items User A has ratings for. And compute MAE using all of them.

Update: Just learnt that approach above is called "Judging Model Quality by Residuals", there are others like Cross Validation


How can I fix this error : Python Error ""ValueError: could not broadcast input array from shape (4) into shape (1000)"" I have the following python code:

from keras.applications import InceptionResNetV2
conv_base=InceptionResNetV2(weights='imagenet',
                            include_top=False,
                            input_shape=(150,150,3))
#conv_base.summary ()
# Extract features
import os, shutil
from keras.preprocessing.image import ImageDataGenerator

datagen = ImageDataGenerator(rescale=1./255)
batch_size = 16

def extract_features(directory, sample_count):
    features = np.zeros(shape=(sample_count, 3, 3, 1536))  # Must be equal to the output of the convolutional base
    labels = np.zeros(shape=(sample_count))
    # Preprocess data
    generator = datagen.flow_from_directory(directory,
                                            target_size=(150,150,),
                                            batch_size = batch_size,
                                            shuffle=False,
                                            class_mode='categorical')
    # Pass data through convolutional base
    i = 0
    for inputs_batch, labels_batch in generator:
        features_batch = conv_base.predict(inputs_batch)
        features[i * batch_size: (i + 1) * batch_size] = features_batch
        labels[i * batch_size: (i + 1) * batch_size] = labels_batch
        i += 1
        if i * batch_size >= sample_count:
            break
    return features, labels

train_features, train_labels = extract_features(train_dir, 384)  # Agree with our small dataset size
validation_features, validation_labels = extract_features(validation_dir, 80)
test_features, test_labels = extract_features(test_dir, 86)

I get the following error:

  File "<ipython-input-31-6bdd0a78c5f7>", line 26, in <module>
    train_features, train_labels = extract_features(train_dir, 384)

  File "<ipython-input-31-6bdd0a78c5f7>", line 20, in extract_features
    labels[i * batch_size: (i + 1) * batch_size] = labels_batch

ValueError: could not broadcast input array from the shape (16,4) into shape (16)

Your data generator retrieves your labels as categorical and based on the error, I assume you have 4 classes. However, in your extract_features function, you are first initializing a labels array of shape=(sample_count) instead of shape=(sample_count, 4). In your for loop, you are getting a labels_batch from the generator which apparently has a shape of (16, 4) and you are trying to save it as part of your initialized labels array which has a shape of (sample_count) instead of (sample_count, 4) and that's why it complains.

In short, you should change:

labels = np.zeros(shape=(sample_count))

to:

labels = np.zeros(shape=(sample_count, 4))


Unstable results when I train a CNN I'm currently training a CNN to do a binary classification.
I'm getting fairly good results, but unfortunately the training is very unstable.
Just by changing the seed the relative error changes by 20-30%.

What can be the cause of this and how can I prevent it?

Other info:

The amount of data is rather small.
I'm starting from imagenet snapshot.

Understand a data
Find the most suitable metrics
handle missing values
normalize, scale etc your variables
feature selecting
use cross validation(kfold for example)
try oversampling or undersampling
trying other models


Appropriate loss function for multi-hot output vectors I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four values that are equal to 1:

[0 0 0 1 0 0 1 0 1 0 0]

These kinds of vectors are sometimes called "multi-hot embeddings".

I am looking for an appropriate loss function for outputs of this kind. Is there a published equation I should check out? Or should I implement a custom loss function? Any suggestions others can offer on this question would be greatly appreciated!

I have also been pondering on this question and trialled loss function on this sort of problem.
For these type of classification tasks, the loss function which seems most appropriate is Binary Cross Entropy Loss : https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a


How do you define the steps to explore the data? I'm falling in love with data science and I'm spending a lot of time studying it. It seems that a common data science workflow is:

Frame the problem
Collect the data
Clean the data
Work on the data
Report the results

I'm struggling to connect the dots when comes to work on the data. I'm aware that step 4 is where the fun happens, but I don't know where to begin. What are the steps taken when you work on the data? Example: do I need to find the central tendency or the standard deviation? Is machine learning needed?

Ps: I know these are broad questions, so please answer it within your own domain expertise.

That is a very good framework of solving a question you have. According to me, it has multiple answers. I'll give you the one which I relate to.

After cleaning the data or rather while cleaning it, we have to be clear about the task ahead and about our results.
The work on the data follows the below steps mostly:

Feature Detection
Training using the above features(there are many Machine Learning/Deep learning models to do this) like Classification (this depends on the task)
Check the trained model on validation test if needed and later on test set
The features depend on the dataset. Standard deviation or to find central tendency isn’t always a criteria. Machine Learning is needed to train on the dataset in most of the cases.


Value error array with 0 features in linear regression scikit My input and output data are written in an 6xn row-column excel file,thatI read them using pandas

using this code :

from sklearn import linear_model
import os
import pandas as pd
import numpy as np
import openpyxl as pyx
import numpy as np
import matplotlib.pyplot as plt

path = r"E:\"
os.chdir( path )

data1 = pd.read_excel("input.xlsx", header=None,index=False)
data2 = pd.read_excel("target.xlsx", header=None,index=False)

# print df.head()

le= 20;lg=len(data1)
x_train=[];x_t=[];y_train=[];y_t=[];x_test=[];x_ts=[];y_test=[];y_ts=[];
for i in range(le):
    x_t = data1.iloc[i,:]
    x_train.append(x_t)
    y_t = data2.iloc[i,:]
    y_train.append(y_t)
    if i > le :
        x_ts = data1.iloc[lg-i,:]
        x_test.append(x_ts)
        y_ts = data2.iloc[lg-i,:]
        y_test.append(y_ts)

ols = linear_model.LinearRegression()
model = ols.fit(x_train, y_train)

print (model.predict(x_test)[0:5])

I get this error :

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
  line 880, in runfile
      execfile(filename, namespace)

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py",
  line 102, in execfile
      exec(compile(f.read(), filename, 'exec'), namespace)

  File "E:/Scikit-Learn.py", line 46, in
      print (model.predict(x_test)[0:5])

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\base.py",
  line 268, in predict
      return self._decision_function(X)

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\linear_model\base.py",
  line 251, in _decision_function
      X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\sklearn\utils\validation.py",
  line 424, in check_array
      context))

  ValueError: Found array with 0 feature(s) (shape=(1, 0)) while a
  minimum of 1 is required.

The error says that the array you feed into predict has shape=(1,0) meaning that it must be an empty iterable.

And by looking at your code, it's easy to see why. x_test starts of as empty, and in your code, the only way to add things to it is if i > le. But since your loop is defined as for i in range(le), by definition i will never be greater than le since the last value of range is le-1


When to use missing data imputation in the data analysis problem? I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use the missing value imputation to complete the dataset. I have finished the univariate analysis for each variable in the raw dataset, and I found there are three continuous variables and two categorical variables with lots of missing data. I want to use the missing data imputation to complete the dataset after processing with the bivariate analysis and graph exploration of every variable. But I am not sure if that's a correct order to do?

Should I use missing value imputation to complete the dataset before bivariate association analysis or should I do it after that?

In addition, if I want to examine the distribution of the outcome variables to find proper transformation, should I do it after impute the missing data as well?

Thank you!

Generally speaking you have two options:

impute the missing data
discard the missing data

Due to the fact that ML models perform better with more data, the former is usually preferred. However, you should keep in mind that the imputed data should not affect the distribution of the feature.

This is especially the case with features with a high percentage of missing values. If a feature has, let's say, 90% of its values missing, then by imputing it you are dictating what you want its distribution to be like (because all the data in that feature will be artificial). In this case it is better to discard the feature altogether.


How would I approach training a model and encoding this categorical data So I have the following data:
I have one series where each word has a value that describes the average review score that would get.
For example, if the word "excellent" showed up in reviews with a score of 2,3,5,4 it would gain a value of 3.5.
I also have a list of the words contained in a review, and the review scores of each of those written reviews.
For example,
Unique_words
["good","clean","hotel","enjoyed","stay","here"]

score
4

(These are ofc simplified examples, my actual data is a lot longer)
I also have the original reviews, from which the unique_words are taken from.
The question is, how would I use this data in order to train a machine-learning algorithm to predict what score a review would get, given the unique words contained inside it.

You need to encode categorical variables as dummies.
This means to create new features for each type of category and then assigned either a 1 (where a record has that category) or 0 (to each record that doesn't have that category). With some examples, this should look something like this
                        good  clean  hotel  enjoyed  stay  here
I am good               1     0      0      0        0     0
My face is clean        0     1      0      0        0     0
This is a clean hotel   0     1      1      0        0     0
I enjoyed a good meal   1     0      0      1        0     0
Don't stay here         0     0      0      0        1     0

In python you can use the pandas function pd.get_dummies()


Python Code to find the number of hapax legomena in a Text or Words_List In corpus linguistics, a hapax legomenon is a word that occurs only once within a context, either in the written record of an entire language, in the works of an author, or in a single text. The term is sometimes incorrectly used to describe a word that occurs in just one of an author's works, but more than once in that particular work. Hapax legomenon is a transliteration of Greek ἅπαξ λεγόμενον, meaning "(something) being said (only) once"

Hapax_legomenon
enter link description here

You can use nltk's freq dist, there is a built in method there for that.
from nltk.probability import FreqDist
from nltk import Text

self.text = Text(self.tokens)
self.fdist = FreqDist(self.text)
hapaxes = fdist1.hapaxes()

if by any chance you cannot use the library, feel free to manually calculate it:
text = 'your text ... '
local_hapax = list(set(text.split(' ')))
weighted_local_hapax = local_hapax/len(text)


My CNN produces volatile validation_loss and does not converge in python(Keras) My model is experiencing wild and big fluctuations in the validation loss and does not converge.

I am doing an image recognition project with my three dogs i.e. classifying the dog in the image. Two dogs are very similar and the 3rd is very different. I took 10 minute videos of each dog, separately. Frames were extracted as images at each second.

This block of code is responsible for augmenting and creating the data to feed the model.

randomize = np.arange(len(imArr)) # imArr is the numpy array of all the images
np.random.shuffle(randomize) # Shuffle the images and labels
imArr = imArr[randomize]
imLab= imLab[randomize] # imLab is the array of labels of the images
lab = to_categorical(imLab, 3)

gen = ImageDataGenerator(zoom_range = 0.2,horizontal_flip = True , vertical_flip = True,validation_split = 0.25)
train_gen = gen.flow(imArr,lab,batch_size = 64, subset = 'training')
test_gen = gen.flow(imArr,lab,batch_size =64,subset = 'validation')

This picture is the result of the model below.

model = Sequential()

model.add(Conv2D(16, (11, 11),strides = 1, input_shape=(imgSize,imgSize,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Conv2D(32, (5, 5),strides = 1))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(3,3),strides = 2))
model.add(BatchNormalization(axis=-1))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(BatchNormalization(axis=-1))
model.add(Dropout(0.3))

#Fully connected layer
model.add(Dense(256))
model.add(Activation('relu'))
model.add(BatchNormalization())
model.add(Dropout(0.3))

model.add(Dense(3))

model.add(Activation('softmax'))

sgd = SGD(lr=0.004)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])

batch_size = 64
epochs = 100
model.fit_generator(train_gen, steps_per_epoch=(len(train_gen)), epochs=epochs, validation_data=test_gen, validation_steps=len(test_gen),shuffle = True)

Things I have tried.

High/low Learning rate  ( 0.01 -> 0.0001)
Increase Dropout to 0.5 in both Dense layers
Increase/Decrease size of both Dense Layers ( 128 min -> 4048 max)
Increased number of CNN layers
Introduced Momentum
Increased/Decreased Batch Size

Things I have not tried

I have not used any other loss or metric
I have not used any other optimiser.
Have not adjusted any parameters of the CNN layers

It seems that there is some form of randomness or too many parameters in my model. I am aware that it is currently overfitting, but that should not be the cause of the volatility(?).
I am not too worried about the performance of the model. I would like to achieve about a 70% accuracy. All I want to do now is to stabilise the validation accuracy and to converge.

Note:

At some epochs, the training loss is very low ( <0.1 ) but validation
loss is     very high ( > 3 ).
The videos are taken on different backgrounds, but +- the same amount on each background for each dog.
Some images are a bit blurry.

It seems that there is some form of randomness or too many parameters in my model. I am aware that it is currently overfitting, but that should not be the cause of the volatility(?)

It is quite plausible that this behavior is caused by a combination of the model's excessive capacity and the data that you are feeding into it.

As you have pointed out, overfitting is obvious here - the model cannot generalize well and fails on unseen data. If you reduce model capacity (better yet, start off with the bare minimum and incrementally build up from there), then the network will be forced to learn instead of performing memorization. Since you mention that you have already tried tuning rates/layers/sizes, the model alone cannot be the cause, although maybe someone with more expertise in convolutional layers can provide useful insight.

The lack of convergence can be an indicator that there is simply not enough data to learn from - neural networks aren't guaranteed to converge when there is insufficient data. The required amount depends greatly on the specifics of the task.

  my data set consist of about 1800 images...600 of each dog

That's quite a small dataset, considering that:

  Two dogs are very similar and the 3rd is very different.

So roughly two thirds of your images can cause confusion. The more they look alike, the more samples you will need to capture the details.

  The videos are taken on different backgrounds, but +- the same amount on each background for each dog.

The model could then erroneously try to discriminate between the two similar-looking dogs based on the background alone (think worst-case scenario).

In your case, I would look at the data more closely, investigate which images are consistently the easiest/hardest for the model to classify and take full advantage of ImageDataGenerator's image augmentation to improve generalization capabilities. See how the model responds if you place similar-looking dogs under the same label, or have one of them appear exclusively in training, the other in validation. Maybe there are some answers there.


Linear Discriminant Analysis (LDA) before or after k-fold cross-validation? I have features extracted from a small dataset, would like to reduce the dimensions by using LDA. Also want to do a SVM classification with k-fold cross-validation.

My question is: What would be the best practice: to do LDA before the CV, or to do LDA within the CV (i.e. to each train and test fold)?

It depends on what you want to achieve.

If you want to visualize the results of your SVM classification, then you should do it after.

If you want to reduce noise, speed up training ... or whatever reason you want to reduce the dimensionality of your problem. You should do it before.

One idea here that can be useful, is to do LDA inside a pipeline and choosing the best hyperparameter within the CV.

In this example you can check a PCA and then a logistic regression. Your case will be similar but with LDA and SVM.


How to implement Python's MLPClassifier with gridsearchCV? I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code:

parameters={
'learning_rate': ["constant", "invscaling", "adaptive"],
'hidden_layer_sizes': [(100,1), (100,2), (100,3)],
'alpha': [10.0 ** -np.arange(1, 7)],
'activation': ["logistic", "relu", "Tanh"]
}

clf
= gridSearchCV(estimator=MLPClassifier,param_grid=parameters,n_jobs=-1,verbose=2,cv=10)

Though,I am not sure if hidden_layer_sizes: [(100,1), (100,2), (100,3)] is correct. Here, I am trying to tune 'hidden layer size' & 'number of neurons'. I would like to give this 'tuple' parameter for hidden_layer_sizez: 1, 2, 3, and neurons: 10, 20, 30,...,100.

But
I do not know if it is the correct way to do it. Therefore, I am choosing default neurons to be 100 in each layer.

Can
anyone advise please?

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer.

If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$.

$(100,1)$ would mean that the second hidden layer only has one neuron.


Neural Nets: unordered sets of ordered tuples as features of data I'm working on a very small scale pet project in which inputs are essentially sets of (x, y) pairs, and are to be classified into categories, using deep learning, specifically using Keras (I know this may not be the best for this, but it's more of a proof of concept / I want to try it out).

However, I'm not sure how to go about representing the data.

I'm starting with a simple classification problem (i.e. if (a, b) is a feature of a sample, and a and b are both within 5% of a certain (c, d), then they are a positive example, and not otherwise), but I'm not sure how to represent the data such that the network can learn this.

I was thinking of doing one-hot encoding, but then the dimensionality of the data may grow immensely (x and y both take on values in a continuous interval), and I'm worried that it would not generalize well to data augmentation (I would augment the data by adding noise to each x, y).

Any ideas?

Your features are already numerical, so you can simply pass your array of numbers into a MLP.  For instance, if you have 5 sets of (x,y) coordinates per sample, your MLP will have 10 inputs.

One-hot encoding is used to convert categorical features into numerical values.  This image from Kaggle does a great job of illustrating how you might use one-hot encoding:

If you were to directly convert categorical values to numerical values, that would imply that one color is greater than another, so we need to use a boolean array instead.

As for the variable length feature vector, it is difficult to give an exact answer without knowing more details about your problem, but here are general possible solutions:

use a recurrent neural network
use distance-based regression/classification algorithms such as k nearest neighbors
transform your data into feature vectors of fixed length (which may not be reasonable depending on your data and what it represents)

A quick high level introduction to neural networks might be helpful.


Computational aspects are typically ignored by statisticians In the introductory chapter of "Process Mining: Data Science in Action" (2016 - Van der Aalst, pag 11) the author says that :

  Although data science can be seen as a continuation of statistics, the
  majority of statisticians did not contribute much to recent progress
  in data science. Most statisticians focused on theoretical results
  rather than real-world analysis problems. The computational aspects,
  which are critical for larger data sets, are typically ignored by
  statisticians. The focus is on generative modeling rather than
  prediction and dealing with practical challenges related to data
  quality and size.

The bold phrase is not clear to me. In fact, since a generative model is the model that generates the data, once we obtain it we can do predictions. So, to me generative modeling and prediction are not opposing concepts. What do you think ?

I think what the author is speaking about is the time/memory complexity of algorithms that statisticians may don't care about. Make a model which is mathematically well proven may be more important to statistician eyes than making approximation to render a model feasible in real life.

I encourage you to look at the complexity of frequent mathematical operations.

Often operation like Singular Value Decomposition, Matrix Inversion, Matrix transpose are used and their cost is way higher than the "upper bound scalable time complexity limit" $O(n.log(n))$ which prevent any utilization on massive datasets.

As an example, you can easily imagine than we are rapidly limited with time complexity because we are not able to increase computational power -- or wait -- more than approximately linearly with the increase of problem size. Take a very common $O(n^2)$ complexity, you can't afford to wait $1000000$ more longer -- or multiply your computational power by $1000000$ to keep same app duration -- when you multiply your dataset size by $1000$ (except if your operations are really fast...)


Best approach to create a text classification model with two inputs? I'm looking to train a model with two text inputs (sentences) and a binary classification. Essentially, for 2 given sentences, are they paraphrases or not.
I want to use the Microsoft research Paraphrase Detection dataset as I feel this would be a good place to start.
What would be the best approach to do this?
Thanks in advance, and sorry for my naïvete.

All current pre-trained Transformers (BERT, RoBERTa, ...) can take two sentences as an input, simply by concatenating the sentences and separating them by a special token. You can then finetune the model on the paraphrase dataset.
The Microsoft Research Parallel Corpus is part of the GLUE benchmark for sentence representation evaluation. It has a public leaderboard, so you can have a look and get an idea, how the best models look like.


Do convolutions "flatten images"? I'm looking for a good explanation of how convolutions in deep learning work when applied to multi-channel images. For example,  let's say I  have a 100 x 100 pixel image with three channels, RGB. The input tensor would then have dimensions 100 x 100 x 3.

If I apply a convolution with N filters and a stride of one, will the output dimension be:

100 x 100 x 3 x N ?

or

100 x 100 x N ?

In other words, does the convolution that is applied "flatten" the image, or is the convolution applied on a channel by channel basis?

In all the implementations for CNNs processing images that I have seen, the output in any layer is

Width x Height x Channels

or some permutation. This is the same number of dimensions as the input, no additional dimensions are added by the convolutional layers. Each feature map channel in the output of a CNN layer is a "flattened" 2D array created by adding the results of multiple 2D kernels (one for each channel in the input layer).

Usually even greyscale input images are expected to be represented as Width x Height x 1 so that they fit the same pattern and the same layer model can be used.

It is entirely feasible to build a layer design which converts a standard 2D+channels input layer into a 3D+channels layer. It is not something I have seen done before, but you can never rule out that it could be useful in a specific problem.

You may also see 3D+channels convolutions in CNNs applied to video, but in that case, the structure will be some variation of

Width x Height x Frames x Channels


How to apply class weight to a multi-output model? I have a model with 2 categorical outputs.
The first output layer can predict 2 classes: [0, 1]
and the second output layer can predict 3 classes: [0, 1, 2].

How can I apply different class weight dictionaries for each of the outputs?

For example, how could I apply the dictionary {0: 1, 1: 10} to the first output,
and {0: 5, 1: 1, 2: 10} to the second output?

I've tried to use the following class weights dictionary
weight_class={'output1': {0: 1, 1: 10}, 'output2': {0: 5, 1: 1, 2: 10}}
But the code fails with an error.

My script also runs normally when i remove the class_weight parameter

Code Example

I've created a minimal example that reproduces the error

from tensorflow.python.keras.models import Model
from tensorflow.python.keras.layers import Input, Dense
from tensorflow.python.data import Dataset
import tensorflow as tf
import numpy as np

def preprocess_sample(features, labels):
    label1, label2 = labels
    label1 = tf.one_hot(label1, 2)
    label2 = tf.one_hot(label2, 3)
    return features, (label1, label2)

batch_size = 32

num_samples = 1000
num_features = 10

features = np.random.rand(num_samples, num_features)
labels1 = np.random.randint(2, size=num_samples)
labels2 = np.random.randint(3, size=num_samples)

train = Dataset.from_tensor_slices((features, (labels1, labels2))).map(preprocess_sample).batch(batch_size).repeat()

# Model
inputs = Input(shape=(num_features, ))
output1 = Dense(2, activation='softmax', name='output1')(inputs)
output2 = Dense(3, activation='softmax', name='output2')(inputs)
model = Model(inputs, [output1, output2])

model.compile(loss='categorical_crossentropy', optimizer='adam')
class_weights = {'output1': {0: 1, 1: 10}, 'output2': {0: 5, 1: 1, 2: 10}}
model.fit(train, epochs=10, steps_per_epoch=num_samples // batch_size,
          # class_weight=class_weights
          )

This code runs successfully without the class_weight parameter.
But when you add the class_weight parameter by uncommenting the line
# class_weight=class_weights than the script fails with the following error:

Traceback (most recent call last):
  File "test.py", line 35, in <module>
    class_weight=class_weights
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1536, in fit
    validation_split=validation_split)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 992, in _standardize_user_data
    class_weight, batch_size)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1165, in _standardize_weights
    feed_sample_weight_modes)
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1164, in <listcomp>
    for (ref, sw, cw, mode) in zip(y, sample_weights, class_weights,
  File "venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training_utils.py", line 717, in standardize_weights
    y_classes = np.argmax(y, axis=1)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 1004, in argmax
    return _wrapfunc(a, 'argmax', axis=axis, out=out)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 62, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "venv/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 42, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
numpy.core._internal.AxisError: axis 1 is out of bounds for array of dimension 1

Edit

I've also opened an issue in the Keras github page, but i wanted to ask the same question here to see if perhaps i'm missing something and doing something wrong.

I wansn't able to use the class_weight parameter yet, but in the mean time i've found another way to apply class weighting to each output layer.

Current solution

In this keras issue they have supplied an easy method to apply class weights via a custom loss that implements the required class weighing.

def weighted_categorical_crossentropy(y_true, y_pred, weights):
    nb_cl = len(weights)
    final_mask = K.zeros_like(y_pred[:, 0])
    y_pred_max = K.max(y_pred, axis=1)
    y_pred_max = K.reshape(y_pred_max, (K.shape(y_pred)[0], 1))
    y_pred_max_mat = K.cast(K.equal(y_pred, y_pred_max), K.floatx())
    for c_p, c_t in product(range(nb_cl), range(nb_cl)):
        final_mask += (weights[c_t, c_p] * y_pred_max_mat[:, c_p] * y_true[:, c_t])
    return K.categorical_crossentropy(y_pred, y_true) * final_mask

where weights is a CxC matrix (where C is the number of classes) that defines the class weights.
More precisely, weights[i, j] defines the weight for an example of class i which was falsely classified as class j.

So how do we use it?

Keras allows to assign a loss function for each output.
so we could assign each output a loss fucntion with the correct weights matrix.

For example, to satisfy the request i made in the question we could suggest the following code.

# Define the weight matrices
w1 = np.ones((2, 2))
w1[1, 0] = 10
w1[1, 1] = 10

w2 = np.ones((3, 3))
w2[0, 0] = 5
w2[0, 1] = 5
w2[0, 2] = 5
w2[2, 0] = 10
w2[2, 1] = 10
w2[2, 2] = 10

# Define the weighted loss functions
from functools import partial
loss1 = partial(weighted_categorical_crossentropy, weights=w1)
loss2 = partial(weighted_categorical_crossentropy, weights=w2)

# Finally, apply the loss functions to the outputs
model.compile(loss={'output1': loss1, 'output2': loss2}, optimizer='adam')

And that accomplishes the request :)

Edit

There is a small edition that needs to be made.
The loss functions must have a name, so we can supply this with the following:

loss1.__name__ = 'loss1'
loss2.__name__ = 'loss2'


Model to detect specific semantic content without labeled data I want to build a model that can detect sentences that discuss requests for communication - like 'email me', 'phone us', 'contact us', etc. However, I do not have any labeled data which I can use to simply train a neural network.
How can I go about solving this problem? On way I was thinking was to create an initial labeled dataset by detecting above mentioned phrases. But I am not sure how to generalize and detect sentences expressing the similar sentiment using some other phrase not in my list.
I am open to any other ideas as well. Thank you.

First, this is not sentiment analysis, which is about quantifying "affective states", i.e. something along the lines of "like / don't like". This would rather correspond to a specific application of general text classification: predicting whether a sentence contains a contact request.
Now about the specific problem: you're right, it could be a good idea to start by identifying some clearly positive sentences using pattern matching. There are many approaches which can be used to expand this initial set of positive sentences:

semi-supervised learning
Using semantic similarity between words (for instance with word embeddings) to find more candidate terms for pattern matching
Using similarity between sentences to find potential new positive cases, then label these manually (this could be done iteratively)

Note that all the above approaches could work but they are biased in favor of the known positive cases, i.e. they risk missing positive cases which have little similarity with the original ones. The only option to avoid this issue is to take a random subset of sentences and label all of them as either positive or negative.


How to add a new column with labels in a dataframe? I have thousands of sentences that I would need to label based on their sentiment.
An example is

Testo
"Può dirmi quale fotocamera della Sony fa delle belle foto?"
"Che macchina fantastica! Ha smesso di funzionare dopo due giorni"
"Quel pallavolista ha un fisico pauroso"
"Questo iPad è fantastico"

I have tried using Fasttext (Python Package) but I do not know how to add the sentiment label in a new column. Also, I got the wrong sentiment (below it is shown the expected output).

Testo                                                            Sentiment
"Può dirmi quale fotocamera della Sony fa delle belle foto?"         0
"Che macchina fantastica! Ha smesso di funzionare dopo due giorni"  -1
"Quel pallavolista ha un fisico pauroso"                             1
"Questo iPad è fantastico"                                           1

How can I do it?

Do you have a working method to get the sentiment label for an individual sentence?

If so, I would think you could just create a method to get the sentiment given a sentence and use it's return value as the row entry.

The apply function from pandas will apply a function to every row if you specify axis=1

def getSentiment(row):
    # Do some sentiment operations here on row['Testo']
    return sentiment_label

df.apply(getSentiment, axis=1)


Interpreting the Root Mean Squared Error (RMSE)! I read all about pros and cons of RMSE vs. other absolute errors namely mean absolute error (MAE). See the the following references:

MAE and RMSE — Which Metric is Better?
What's the bottom line? How to compare models
Or this nice blogpost, or this question in stats.stackexchange containing interesting responses, and this one in datascience.stackexchange

Still I can not get my head around something about RMSE:

Scenario: Let's say we have a regressor for predicting house prices with a MAE of 20.5\$ and a RMSE of 24.5\$. Based on MAE, I can certainly interpret that the average difference between the predicted and the actual price is 20.5\$. How can I interpret RMSE? Can we still safely say the predicted and the actual price are off by 24.5\$ at the same time base on RMSE (upper-bound of prediction error)?

In the first medium post, it says:

  RMSE does not describe average error alone and has other implications
  that are more difficult to tease out and understand.

It confuses me a little. And I could not find any reliable reference to also clearly state that one can safely interpret RSME as one does MAE. Is RMSE is simply a only mathematically more convenient for optimization etc., and we are better off with MAE for the interpretation?

Any detailed explanation is highly appreciated.

How can I interpret RMSE?

RMSE is exactly what's defined. $24.5 is the square root of the average of squared differences between your prediction and your actual observation. Taking squared differences is more common than absolute difference in statistics, as you might have learnt from the classical linear regression.

  It confuses me a little. And I could not find any reliable reference
  to also clearly state that one can safely interpret RSME as one does
  MAE. Is RMSE is simply a only mathematically more convenient for
  optimization etc., and we are better off with MAE for the
  interpretation?

I think this post should help you. I'll answer your questions directly:

RMSE is easier mathematically and also practically. Have you heard of derivative? The derivative for MAE is undefined when your prediction is equal to observation but well defined everywhere for RMSE. In machine learning, a well defined gradient function is generally better.
Both RMSE and MAE are useful, but they are two very different metrics. In regression, it's generally about choosing between linear regression and quantile regression. They are two very different models!
As stated in the link, if you don't want your residuals affect your model too much, MAE could be better. Otherwise, if your data set is well defined (not many residuals), RMSE could be better.

There is no right or wrong on which one is better. Think like two different algorithms for doing modelling.


Where to find height dataset, or datasets in General Hi there smart people,

I am new to data  Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest.

For example, I wanted to build a simple program that takes a person's height and predicts the likely height of his/her child.
(greetings from Galton)

But even for this relatively famous topic I am unable to find data.

Are there any good websited or something that list a lot of datasets for specific topics?  A database of datasets, so to speak.

I mean, I am on the Internet, it should not be that difficult to find some data ;)

The Galton Height dataset seems to be exactly what you're looking for.

There are a ton of repositories of open datasets online.  For beginners, I often see the UCI Machine Learning Repository.  It has quite a few datasets which are easy to work with.

Some other resources for datasets:

Kaggle - hosts data science competitions, but you can download all the datasets without participating in a competition.
Data.gov - You can freely download data from US govt agencies
AWS Open Data - You need a AWS account, but these datasets are free-to-download


R Studio like editor for Python? I hope this question is okay for the forum. I want to ask for your experiance with Python editors.

Currently, I use VS-Code to work with Python. However, in R Studio I really appreciate that it holds data frames in the memory and makes it easy to view/inspect dataframes and other items. I'm "closer to the data" in R Studio. Also line-by-line/blockwise execution of code is really helpful.

So my question: Is there anything like R Studio for Python (preferably non-browser based)? Can someone recommend an editor for Python with similar features like R Studio (especially wrt data handling)?


Result of uniform weight initialization in all neurons Background
cs231n has the question regarding how to initialize weights.

Question
Please confirm or correct my understandings. I think the weight value will be the same in all the neurons with ReLU activation.
When W = 0 or less in all neurons, the gradient update on W is 0. So W will stay 0. When W = 1 (or any constant), the gradient update on W is the same value in all neurons. So W will be changing but the same in all neurons.

I'll make this as explicit and non handwavy as possible, at the risk of being boring.
Let us take a small toy example: a network with two inputs $x_1$ and $x_2$, two hidden layers with two units each and one output. For simplicity, we will assume that the activation function is the identity function (i.e. no activation) and that the objective function is the output itself (not very realistic, but for our purposes this could mean our cost function is meant to drive the output to the zero function).

Below are the outputs of each unit expressed as a function of their inputs.
$$y = w_9y_3 + w_{10}y_4$$
$$y_3 = w_5y_1 + w_7y_2$$
$$y_4 = w_6y_1 + w_8y_2$$
$$y_1 = w_1x_1 + w_3x_2$$
$$y_2 = w_2x_1 + w_4x_2$$
Take a look at $w_9$: $\frac{\partial y}{\partial w_9} = y_3 = w_5y_1 + w_7y_2 = w_5(w_1x_1 + w_3x_2) + w_7(w_2x_1 + w_4x_2)$. The update of $w_9$ depends on $w_1$, $w_2$, $w_3$, $w_4$, $w_5$ and $w_7$: if we consider the diagram above to be a directed acyclic graph, the weights of all the edges on paths leading to $y_3$.
In our toy example, updates are simple: $w_{i, n+1} = w_{i, n} - \eta \frac{\partial y}{\partial w_i}(w_{i, n})$, where $i$ denotes the weight in the diagram and $n$ denotes the iteration.
If all weights are initialized to a constant $a$, this amounts to $w_9 = a$ and $\frac{\partial y}{\partial w_9} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$.
Let us also take a look at $w_{10}$: $\frac{\partial y}{\partial w_{10}} = w_6(w_1x_1 + w_3x_2) + w_8(w_2x_1 + w_4x_2)$. In this case, $w_{10} = a$ and $\frac{\partial y}{\partial w_{10}} = a(ax_1 + ax_2) + a(ax_1 + ax_2)$.
As can be seen, the initial value of both $w_9$ and $w_{10}$ is the same and their updates are the same too. On the first iteration, they will have changed after the update, but they will still be equal to each other. Not necessarily a bad thing, but if they're equal, you might just as well keep only one of them in memory. If you're keeping both, they should be different.
What happens on the next iteration? Looking at the derivatives with respect to $w_9$ and $w_{10}$ previously derived, we notice that if $w_5 \neq w_6$ or $w_7 \neq w_8$ the two updates will be different, so let's take a look at those then!
$$\frac{\partial y}{\partial w_5} = \frac{\partial y}{\partial y_3} \frac{\partial y_3}{\partial w_5} + \frac{\partial y}{\partial y_4} \frac{\partial y_4}{\partial w_5} = w_9y_1 + w_{10} \cdot 0 = w_9(w_1x_1 + w_3x_2)$$
$$\frac{\partial y}{\partial w_6} = w_{10}y_1 = w_{10}(w_1x_1 + w_3x_2)$$
$$\frac{\partial y}{\partial w_7} = w_9(w_2x_1 + w_4x_2)$$
$$\frac{\partial y}{\partial w_8} = w_{10}(w_2x_1 + w_4x_2)$$
We can see that for the same reasons as before, after the first iteration, $w_5$ and $w_6$ will be the same, just like $w_7$ and $w_8$ will be the same. This, combined with the fact that $w_9$ and $w_{10}$ were equal after the first iteration, means that $w_9$ and $w_{10}$ will still be equal after the second iteration.
We can already say that $w_5 = w_6$, $w_7 = w_8$ and $w_9 = w_{10}$ ad infinitum.
Finally, let us take a look at the weights of the first hidden layer, $w_1$, $w_2$, $w_3$ and $w_4$.
$$\frac{\partial y}{\partial w_1} = (w_9w_5 + w_{10}w_6)x_1$$
$$\frac{\partial y}{\partial w_2} = (w_9w_7 + w_{10}w_8)x_1$$
$$\frac{\partial y}{\partial w_3} = (w_9w_5 + w_{10}w_6)x_2$$
$$\frac{\partial y}{\partial w_4} = (w_9w_7 + w_{10}w_8)x_2$$
Here, $x_1$ and $x_2$ don't have to be equal, so $w_1$ and $w_2$ can be different from $w_3$ and $w_4$. However, if we initialize all weights to $a$, $w_1 = w_2$ and $w_3 = w_4$ after the first iteration, which means that $w_5 = w_6 = w_7 = w_8$ after the second iteration (not just the first).
To conclude, we can say that $w_1 = w_2$, $w_3 = w_4$, $w_5 = w_6 = w_7 = w_8$ and $w_9 = w_{10}$ ad infinitum!
Generalizing this toy example, we can state that if we initialize all weights to the same value $a$, the weights of a layer will all be equal forever, except for the first layer, which keeps a semblance of normalcy. In other words, using only one weight per layer is enough.
A special case of initializing all weights to the same value is initializing them all to 0. If we look once more at the partial derivatives (which we use to update our weights), we can see that they will all be zero, always! In this case, the weights never get updated at all, so our model becomes entirely useless.
A very pretty visualization for the effects of different basic weight initialization techniques can be viewed here.


How to interpret metrics of a model after scaling the data I have a GradientBoostingRegressor from scikit-learn which I trained. Afterwards, I obviously would like to know how good the model is. So, on a non-scaled dataset I would just use the mean_squared_error function from scikit and it would output a certain value that made sense (regarding the dataset).

Now, I scaled (/ transform) the targets in my dataset using the scikit QuantileTransformer(output_distribution='uniform'). The target is now scaled between 0 - 1. This is fine during training etc.

After training the model, I ran the following code to get a few metrics:

test_pred = gb.predict(X_test)
mse_test = mean_squared_error(y_test, test_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(y_test, test_pred)
print("MAE on Test:", mae_test)

Because the target values are scaled, the output is something similar to this:

RMSE on Test: 0.23563730007705744
MSE on Test: 0.05552493718760521
MAE on Test: 0.19235478752773819

I assumed that I could get the 'actual' non-scaled metrics back by applying the QuantileTransformer.inverse_transform function to the output.

So then I got:

RMSE on Test: 2231.21330222
MSE on Test: 807.28588575
MAE on Test: 1888.23406628

Which doesn't seem very correct to me. Normally, the RMSE would be smaller than the MSE, but that isn't the case. If you get that the sqrt of a (MSE) value under 1. Also, the MAE should probably be smaller than the MSE.

My question is, how do you interpret those scaled metric values? Is the inverse_transform output correct? How do I get correct, non-scaled values for the metrics?

I'd appreciate some help on this.

Thanks.

Edit:
The QuantileTransformer is only an example. The question also applies to the MinMaxScaler and other scalers in general.

I solved this issue with the help of @Franziska W. Thanks!

I currently reverse transform y_test and y_pred and then calculate the metrics as following:

# qt_y is a QuantileTransformer instance, gb is a GradientBoostingRegressor
y_pred = gb.predict(X_test)
inv_y_pred = qt_y.inverse_transform(y_pred.reshape(-1, 1))
inv_y_test = qt_y.inverse_transform(y_test.reshape(-1, 1))

mse_test = mean_squared_error(inv_y_test, inv_y_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(inv_y_test, inv_y_pred)
print("MAE on Test:", mae_test)


Which Loss function is correct for binary mapping? I have built a 3 layer neural network to perform a binary mapping (2016 inputs, 288 outputs.) I am getting decent results with mean square error and stochastic gradient decent. My question is: Is there a more appropriate loss function for regression when the output is binary?

Yes, use binary cross-entropy loss. In case you are using Keras, this has been already implemented as a standard loss function for binary outputs.


How does the meta Random Forest Classifier determine the final classification? I am trying to understand exactly how the meta random forest classifier determines the final prediction, I understand that there is a voting system and an aggregation from the decision trees is used to find the final prediction, I have read from here: Classification Random Forests in Python:

"Random forest is an ensemble decision tree algorithm because the final prediction, in the case of a regression problem, is an average of the predictions of each individual decision tree; in classification, it's the average of the most frequent prediction"

I read in the RFC source code:
Ensemble/Forest:

"The predicted class of an input sample is a vote by the trees in the forest, weighted by their probability estimates. That is, the predicted class is the one with highest mean probability estimate across the trees"

Does this mean that for example:

We have 3 Classes (A, B and C) with 4 estimators, each tree has a certain amount of final leaf nodes with a class prediction;

Probabilities of Class A, B or C are a result of the number of the class prediction from the leaf nodes,
i.e Tree1: Class A is predicted 3 times out of 10 possible times, the other 7 times a different class was predicted.

Class A: [0.30, 0.25, 0.30, 0.25] | mean = 0.275
Class B: [0.10, 0.40, 0.10, 0.40] | mean = 0.250
Class C: [0.70, 0.65, 0.70, 0.65] | mean = 0.675

So the Meta Classifier would predict Class C as it has the highest mean probabilities. Is this correct? Or am I looking at this completely the wrong way?

Each data point $x$ that you are predicting necessarily only lands in one leaf node for each tree. So if I have N trees, and $K_A$ is the number of trees in which $x$ ends up in a leaf node which predicts A, then we can say $x$ is class $A$ with "probability" $K_A/N$, and similarly for other classes. Then you can just make an argmax prediction for which class $x$ belongs to based on the "probabilities" (I keep putting this in quotes because it isn't an actual probability, but it's sometimes ok to think of it this way.)


Why does a filter need to be applied to the output of the input gate before cell state is added to? In a neural network there are 4 gates: input, output, forget and a gate whose output performs element wise multiplication with the output of the input gate, which is added to the cell state (I don't know the name of this gate, but it's the one in the below picture with the output C_tilde).

Why is the addition of the C_tilde gate required in the model? In order to allow the input gate to subtract from the cell state, we could change the activation function that results in i_t from sigmoid to tanh and remove the C_tilde gate.

My reasoning is that the input gate already has a weight matrix W_i that can is being multiplied to the input gate's input, hence it already does filtering. However, when C_tilde is multiplied with i_t that seems to be another unnecessary filter.

My proposed input gate would then be i_t = tanh(W_i * [h_t-1, x_t] + b_i) and i_t would directly be added to C_t (C_t = f_t * C_t + i_t rather than C_t = f_t * C_t + i_t * C_tilde_t).

Here is my hypothesis : The $i_t$ can add explainability to the model, as the value of the sigmoid function can give an idea as to how important a particular word is to altering the cell state $C$. This is because $i_t$ lies between 0 and 1. Having single $W$ do both filtering as well as a feature transform of $[h_{t-1},x_t]$ not only puts more stress on the matrix (has to do two things at once), but also no longer has this explainability factor.

Example: 2 vectors may require the same transformation $W*v$, but unless you allow seperate sigmoid function to give each an importance, their contribution to the cell state will remain same.


How Natural language processing and elasticsearch are related I got call from a recruiter for an elasticsearch job. I am not sure how these 2 are related. Can anyone suggest and provide example use case?

Natural language processing (NLP) is a field of computer science, artificial intelligence, and computational linguistics.
ElasticSearch is a search engine based on Lucene.

ElasticSearch uses parts of NLP (e.g., tokenization and stemming). ElasticSearch also includes software engineering elements to ensure a search solution is performant.
Since NLP is entire discipline, there are many parts of NLP that are not related to ElasticSearch.


txt file into pandas I have a txt file where values are randomly separated, sometimes there is a single space, and sometimes there are many. I am importing the file into pandas with pd.read_csv, but I get only one column in my DataFrame.  How can I define the sep parameter to read the values properly?

Do you only need to differentiate between one or more spaces?
If so, maybe you can make use of a regular expression separator as listed in the pandas docs.
Something like this:
pd.read_csv(... sep='\s+', ...)

Also, consider adding the parameter delim_whitespace as such:
pd.read_csv(..., delim_whitespace=True, ...)


How to tell if the “clusters” I see in my pair plots are statistically significant or occurring by random chance? I have a data set with one row per subject. Some variables include laboratory parameters for blood chemistry, hematology, etc. I also have some flag variables: any = 1 if the subject experienced an adverse event, 0 if not; and ser_flag = 1 if the subject experienced a serious adverse event, 0 if not.

There doesn't seem to be any difference in the distribution of laboratory parameters between subjects who experienced an adverse event (any=1) and subjects who did not (any = 0). When I do a pair plot of all the lab parameters against each other and color by the any flag, there doesn't seem to be any clustering or separation of subjects.

However, when I do the same pair plot and color by ser_flag - I notice that the 20 subjects who experienced a serious adverse events seem to be clustered together in many of the plots.

What test (if any) can I use to determine if these clusters I think I am seeing are occurring randomly, by chance...or if they statistically significant?

The easiest test is equality of means of two distributions. Student's t-test


Instance segmentation performance depending on the number of classes I have a setting in which I have approximately 10 classes, one of which I would need to put an extra emphasis on. Whilst training a mask-rcnn instance segmentation network (detectron2) I found out that I can do a lot better if I separate this one class and train a completely different network for it and then afterwards combine the predictions. Is this true in general for these networks? It would seem reasonable that the network would perform better with fewer classes to predict, but I couldn't find any good writings about this.

The process you have used it spot on. Your problem: Give more importance to one class than the rest will have to be hard-coded in. There will be no pre-trained networks for that class especially.
One thing to note could be that assuming you have to segment roads in a road scene problem. You can create a function which can bias the weights towards the road specifically. This is just a thought, I haven't implemented it myself!


Create tensor of a sequence with a defined shape using TensorFlow Create a tensor Y of shape (2,3) having a sequence of numbers 12,15,18,...., 150

I tried using the below code

tf.constant(np.arange(12,153,3),dtype=tf.dtypes.float32,shape=[2,3])

I got an error saying the tensor with 47 elements cannot be converted into 6 elements.

I added the following third dimension in shape and it worked.

tf.constant(np.arange(12,153,3),dtype=tf.dtypes.float32,shape=[8,2,3])

How to proceed as the given question wants the sequence tensor to be in shape [2,3]?

Please help.

As given in tf.constant documentation:

  If value is a list, then the length of the list must be less than or equal to the number of elements implied by the shape argument (if specified).

In your code, the value is the returned array from np.arange(12,153,3) containing 47 integers.

Now, when you specify shape = [2,3], it implies that the length of the list you have provided as value is 2x3 = 6 which is not true because the array generated by np.arange(12,153,3) has the length = 47. In this case, you will get an error saying:

  ValueError: Too many elements provided. Needed at most 6, but received 47

On the other hand, when you specify shape = [8,2,3], it implies that the length of the list you have provided as value is 8x2x3 = 48 which is also not true because the actual length is 47. It does not raise an error because the length of the list can be less than the number implied by the shape argument as it says in the documentation:

  In the case where the list length is less than the number of elements specified by shape, the last element in the list will be used to fill the remaining entries.

So, it is not possible to shape a sequence of 47 elements to [2,3]


Is there an overview over recommender system architectures? I want to learn more about the recommender system topic.
I am very interested in the usage of different database systems for this use case.
My problem is that I cannot find a good overview of different architectures of recommender systems, especially with the focus on the database part.
Can someone help me out with a good reference or some own thoughts?
Thanks a lot. As interesting as this topic is for me as hard it seems to get some concrete information about it.

A great source is the personal page of Xavier Amatriain (former Head of Engineering at Netflix and Quora). There are several links at publications detailing the system and architecture of recommender systems. In particular you could take a look at this Building industrial-scale real-world recommender systems, there is also this talk Lessons Learned from Building Real-Life Recommender Systems and this blog post System Architectures for Personalization and Recommendation


Target encoding before split on skewed data Hi My data is distributed like follow:

And I only have categorical variables on many many levels.
As I need to make a regression task I thought about doing leave one out encoding on my categories. However, while my model has a very good performance on my validation (0.98 R2) on my test is kind of terrible (0.5 R2).
My colleague did the encoding before the split, reaching an R2 of 0.97 on test.
His reason is that this to cover unseen new categories and to cover all the possible targets during the encoding.
Isn't this a bad practice? Or for this highly skewed data is a possible behavior to adopt?
Thanks

R2 and encoding before split are both can be misleading. At first, target encoding, I wouldn't recommend to do it before the split - this is basically target leak. There're at least two major downsides of doing target encoding before split:

1) Overfitting - putting knowledge about your target into features during training always tend to overfit.
2) Impossible to do the same in production. Handling unseen categories is the thing which should be supported in production as well.

Running well-done N-fold cross-validation can help you check how are you handling unseen categories "in average".

There's a good example of target encoding I always share if anyone asks me about this technique. Also for highly skewed data, I can suggest using log(y) when you encoding your feature, that may help as well.

Also, I would recommend reading about R2 (from my experience it's very bad metrics for non-linear problems):
 - Why R Squared is Useless?
 - another post about R2

I would rather use MAPE/MAE or some metric which fits your task. Looking at your distribution, I guess it shouldn't be sensible to outliers.


Are GPTs close to real intelligence or just another Data In -Data Out -Data Permutation and Combinations? Use cases and Solutions surrounding GPT's have taken NLP world with storm and started the GPT-Best vs GPT- Not So Best war on the internet. There are solutions been derived from API's provided by HF. What would be the impact and how feasible would be GPT development and deployment in the real world use case across domains?

To answer that we Need to have a formal Definition of real intelligence. You got multiple definitions, but which one satisfies you?
Lets say you Claim real intelligence is Awareness. Than if you ask a machine can you prove to me that you are Aware you wont be satisfied. But can you prove you are Aware.
I mean at the end of the day it does not even matter. What matter is we are solving some specialised takes with GPTs a lot better than with other language models. Thats it. GTP(3) has a lot of flaws, just Google it.


Problem in implementing CNN I am using the fashion MNIST dataset to try to work this out:

I am using the data from the links:

Training :
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz

training set labels:
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz

test set images
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz

test set labels
http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz

I use the code to open the dataset:

def load_mnist(path, kind='train'):
    import os
    import gzip
    import numpy as np

    """Load MNIST data from `path`"""
    labels_path = os.path.join(path,
                               '%s-labels-idx1-ubyte.gz'
                               % kind)
    images_path = os.path.join(path,
                               '%s-images-idx3-ubyte.gz'
                               % kind)

    with gzip.open(labels_path, 'rb') as lbpath:
        labels = np.frombuffer(lbpath.read(), dtype=np.uint8,
                               offset=8)

    with gzip.open(images_path, 'rb') as imgpath:
        images = np.frombuffer(imgpath.read(), dtype=np.uint8,
                               offset=16).reshape(len(labels), 784)

    return images, labels

label = ['T-shirt/top',  'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt',
         'Sneaker', 'Bag', 'Ankle boot']

data_dir = './'
X_train, y_train = load_mnist('D:\book', kind='train')
X_test, y_test = load_mnist('D:\book', kind='t10k')

X_train = X_train.astype(np.float32) / 256.0
X_test = X_test.astype(np.float32) / 256.0

I am trying to build a Convolutional Neural Network with the following architecture:
- Convolutional Layer with 32 filters with size of 3x3
- ReLU activation function
- 2x2 MaxPooling
- Convolutional Layer with 64 filters with size of 3x3
- ReLU activation function
- 2x2 MaxPooling
- Fully connected layer with 512 units and ReLU activation function
- Softmax activation layer for output layer
For 100 epochs using the SGD optimizer

My Code is:

X_train = X_train.reshape([60000, 28, 28, 1])
X_train = X_train.astype('float32') / 255.0
X_test = X_test.reshape([10000, 28, 28, 1])
X_test = X_test.astype('float32') / 255.0
model = Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=[28,28,1]))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
model.summary()
y_train = keras.utils.np_utils.to_categorical(y_train)
y_test = keras.utils.np_utils.to_categorical(y_test)
model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=100)

But it is taking a lot of time for execution. It is like 30 minutes per epoch. I think I am doing something wrong in my code. Can someone help me figure that out?

The main problem in your code is that you are passing the whole data at once ,instead of that you should use a batch size of 500 or some other value .There is a parameter of batch size in model.fit()
as well. And you should also reduce the number of epochs.


How to arrange the sets to predict y on x in time series? I'm performing my first NN with my own data and while I was already tuning the parameters I stumbled over an aspect which confuses me now such that I'm not sure what is right and what is wrong..

Given this (head of the) data df_train1_raw:

                     timestamp       A_phsA
2018-02-05 14:00:00 1.517839e+09    856.436487
2018-02-05 15:00:00 1.517843e+09    859.653339
2018-02-05 16:00:00 1.517846e+09    836.635463
2018-02-05 17:00:00 1.517850e+09    801.097284
2018-02-05 18:00:00 1.517854e+09    794.855960
                       (...)

The timestamp column is basically the index just converted into numeric so I can use these time information for the nn model.

Goal: Predict A_phsA on timestamp

First, I create the train and test sets:

# Prepare data
X_train1_raw = df_train1_raw.values
y_train1_raw = X_train1_raw

# Split data into appropriate sets
## Standardize and scale data

scaler = StandardScaler()
tscv = TimeSeriesSplit(n_splits = 5)
pyplot.figure(1)
index = 1

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

This looks reasonable. When I plot the loss and val_loss values of the nn later it also looks reasonable.

But what struggles me actually is this line at the beginning:

y_train1_raw = X_train1_raw

I can't tell if it is plain stupid or if I can't get my head around it anymore. The reason is, when I look for example at KFold:

X =  list(range(10))
print (X)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
y = [x*x for x in X]
print (y)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

kf = KFold(n_splits=5)
X = np.array(X)
y = np.array(y)
for train_index, test_index in kf.split(X):
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    print(“X_test: ", X_test)

they have different X and y. Which makes sense I would say.
But when I adjust my code accordingly

X_train1_raw = df_train1_raw.iloc[:, 1].values
y_train1_raw = df_train1_raw.iloc[:, 2].values

I get an error when performing

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
    X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index]), scaler.fit_transform(X_train1_raw[test_index])
    y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index]), scaler.fit_transform(y_train1_raw[test_index])
    pyplot.subplot(510 + index)
    pyplot.plot(X_train1[:, 1])
    pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
    index +=1
pyplot.show();

  ValueError: Expected 2D array, got 1D array instead:
  array=[1.5178392e+09 1.5178428e+09 1.5178464e+09 1.5178500e+09
  1.5178536e+09

And I don't understand why. Which part is not correct?

edit: Is it scaler.fit_transform() ?

fig, ax = plt.subplots(1, 1, figsize=(24,7))
for train_index, test_index in tscv.split(X_train1_raw, y_train1_raw):
     X_train1, X_test1 = scaler.fit_transform(X_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(X_train1_raw[test_index].reshape(-1,1))
     y_train1, y_test1 = scaler.fit_transform(y_train1_raw[train_index].reshape(-1,1)), scaler.fit_transform(y_train1_raw[test_index].reshape(-1,1))
     pyplot.subplot(510 + index)
     pyplot.plot(X_train1[:, 1])
     pyplot.plot([None for i in X_train1[:, 1]] + [x for x in X_test1[:, 1]])
     index +=1
pyplot.show();

This should fix your code and this happens because the Standard Scaler expects inputs as samples where each sample is an array of n_features to transform. The reshape(-1,1) does that for you if you have got a single feature to scale.


Clustering algorithm to sort filenames A client would like to sort out his filesystem (~ 1,000,000,000 files), which has been fed by numerous workers over the years, each with their own unknown naming convention, e.g.:

[DATE]-[CLIENT]-[FILENAME]
[TYPE]-[CLIENT].[DATE][FILENAME]
...

Here are four examples (out of ~1,000,000,000 files) to make things clearer:

JPM_TPD0001662_2009124012302000451.pdf

JPMF_STA_1712010832_18001_LUX_approval.pdf

CHACN05CTRP_00111.001.pdf

CHACN63CJO1_00018.001.pdf

The purpose is to find out patterns in the naming conventions, but I can't use regular expressions, since the conventions are unknown a priori.

I was wondering whether there was a kind of clustering algorithm to be able to group files according to their naming conventions.

Any K-Mean philosophy applied to strings?

This is not at all a typical clustering problem, so I doubt any of these algorithms will help. If you want to try clustering, you will need to do appropriate feature extraction. Don't expect things to work on the raw data. But I guess once you have good features, the problem will already be solved.

Instead of trying to frame this as a clustering problem, look at it either from a sequential pattern view point, or even better: look at the few questions on how to learn regexps from a set of strings.


Predicting parallel time series with multiple features I am trying to predict sales for 2 departmental stores which share similar demographic properties. My goal is to make a single LSTM model to predict sales from these parallel time series having multiple features.

My input features for training would be

+----------+-------+--------------+-------+
|   Date   | Store | DayOfTheWeek | Sales |
+----------+-------+--------------+-------+
| 1/1/2019 | A     |            2 |   100 |
| 1/2/2019 | A     |            3 |   200 |
| 1/3/2019 | A     |            4 |   150 |
| 1/1/2019 | B     |            2 |   300 |
| 1/2/2019 | B     |            3 |   550 |
| 1/3/2019 | B     |            4 |  1000 |
+----------+-------+--------------+-------+

and my output for training would be

+----------+-------+--------------+-------+
|   Date   | Store | DayOfTheWeek | Sales |
+----------+-------+--------------+-------+
| 1/4/2019 | A     |            5 |   220 |
| 1/4/2019 | B     |            5 |   700 |
+----------+-------+--------------+-------+

Problem is that LSTM takes input as 3D i.e (n_sample, n_timesteps, n_features) and I can pass a single time series for a specific store (e.g. A)

If I had a univariate mutiple time series I can reshape my input data as follows and pass it to LSTM.

+----------+---------+---------+
|   Date   | A_Sales | B_Sales |
+----------+---------+---------+
| 1/1/2019 |     100 |     300 |
| 1/2/2019 |     200 |     550 |
| 1/3/2019 |     150 |    1000 |
+----------+---------+---------+

But I need to identify how can I predict parallel multivariate time series? Is there any other way to define in Input LSTM layer that there are 2 time series  with 2 features each i.e (2*2).

Here are two ways to model the problem. The first one is simpler, the second one is more complex but closer to your original statement of the problem.

Store as an input feature

You can consider the store as a feature to pass to your LSTM. With two different stores, just add a binary input feature "store" where store A is 0 and store B is 1, for example.

Then, you can implement it this way.

from keras.models import Sequential
from keras.layers import LSTM, Dense

timesteps = 20
n_features = 5
model = Sequential()
# add +1 to n_features for the store identifier
model.add(LSTM(32, input_shape=(timesteps,n_features + 1), return_sequences=True))
model.add(LSTM(32))
model.add(Dense(1,activation="relu"))
model.compile(optimizer="rmsprop", loss="mse")

One sample of your data at a given timestep will be a vector of the form (feature_1, feature_2, ..., feature_n, store).

Multivariate time-series prediction

Here we input both time series and aim to predict next values of both stores. So you have a shared-LSTM processing store separately, then concatentate both produced embeddings, and compute the predicted values.

from keras.models import Model
from keras.layers import LSTM, Dense, Concatenate, Input

timesteps = 20
n_features = 5

# Store A and B time-series inputs
a_inputs = Input(shape=(timesteps, n_features))
b_inputs = Input(shape=(timesteps, n_features))

# Stacked LSTM
lstm_1 = LSTM(32, return_sequences=True)
lstm_2 = LSTM(32)
# Stacked LSTM on Store A
a_embedding = lstm_1(a_inputs)
a_embedding = lstm_2(a_embedding)
# Stacked LSTM on Store B
b_embedding = lstm_1(b_inputs)
b_embedding = lstm_2(b_embedding)

# Concatenate embeddings and define model
outputs = Concatenate()([a_embedding, b_embedding])
outputs = Dense(64)(outputs)
outputs = Dense(2, activation="relu")(outputs)
model = Model([a_inputs, b_inputs], outputs)
model.compile(optimizer="rmsprop", loss="mse")

Inspired by section 7.1.5 Shared weight sharing, in Deep Learning with Python by F. Chollet


What does embedding mean in machine learning? I just met a terminology called "embedding" in a paper regarding deep learning. The context is "multi-modal embedding"

My guess: embedding of something is extract some feature of sth,to form a vector.

I couldn't get the explicit meaning for this terminology and that stops me from fully understanding the author's idea and model mechanism

I check the dictionary and search on line,but the explanation is based more on the real life meaning rather than meaning as a machine learning terminology.

And that raise a more generalized and frequently met question, when you find some machine learning terminology/word that you can't understand well, where can you get the solution, some specific way to google? join a machine learning group? raise a question in stack exchange?

In the context of machine learning, an embedding is a low-dimensional, learned continuous vector representation of discrete variables into which you can translate high-dimensional vectors. Generally, embeddings make ML models more efficient and easier to work with, and can be used with other models as well.

Typically, when I stumble upon jargon I'm not familiar with I first turn to Google, and if it can't be found I ping my colleagues and data science forums.


What do the symbols in this image mean in relation to autoencoding? (which concepts do they represent and why?) I'm completely new to neural nets, and am trying to get a grasp of autoencoding. I understand the function of the decoder and encoder, but what does the arg min section and beyond mean? Why is it equal to both phi and psi? Feel free to be abundant in your explanation. Thanks!

The question seems more about notation than autoencoders:
$ a,b = argmin_{a,b}f(a,b)$ simply means that $a$ and $b$ are chosen to minimize $f$.
w.r.t. autoencoders, the argmin statement is saying that we want the decoding of the encoded data $X$ to be as similar as possible to the original $X$.
To represent "decoding of the encoding", we can use function composition: $\psi\circ\phi X$ which means $\psi(\phi(X))$, hence the decoding of the encoding of $X$.
So overall, we are saying that we choose the encoding and decoding functions to minimize the delta between input $X$ and encoded/decoded $X$.

Where the question asks:

what does the arg min section and beyond mean?

The latter half "and beyond" should be more specific.


What are the differences and advantages of TensorFlow and Octave for machine learning? I have been exploring the different libraries and languages you can use in order to implement machine learning. During this, I have stumbled upon a library TensorFlow and Octave(a high-level programming language) as both are intended for numerical computations.

What are the differences and advantages of using either?

Octave is a great language for prototyping and experimenting with ML algorithms, as it has built-in support for numerical linear algebra such as matrix and vector calculations. Octave is optimized for rapid calculations, which is very useful in Machine Learning. It is also quite easy to do matrix multiplications in Octave as Matrices are first-class objects in Octave.

Tensorflow is indeed a versatile platform for machine learning with an ever-expanding list of packages and frameworks getting built.

Octave is a good tool for learning the essentials and internals of mathematics of machine learning and Tensorflow is a good platform for building industry solutions for machine learning projects. Hence both are good for their own purposes.


How to handle features with very broad range I have a long list of continuous values like in the image below:

The plot looks like this:

How to handle such features? If I train the model with this, the model will not have the best precision, because there are a lot of outliners.

I suggest to try a log transformation. This has two potential benefits:

The range of x values becomes smaller
Your transformed data might be closer to resemble a normal distribution (only relevant for some models, e.g. not for trees)

Here are two toy examples to illustrate:

Toy example 1

s = np.random.lognormal(3, 1, 1000)
plt.hist(s, 100)
plt.show()

plt.hist(np.log(s), 100)
plt.show()

As you can see from the second plot the range of x values has become smaller and the transformed distribution resembles a normal distribution. Of course this is a highly artificial example since the non-transformed distribution is log_normal.

Toy example 2

s = np.random.geometric(0.2,100000)
plt.hist(s, 100)
plt.show()

plt.hist(np.log(s), 100)
plt.show()

This one does not look as nice (i.e. normally distributed) anymore as example 1 but still your range of x values has become more compact.

In case your data includes x values of $0$ you can use np.log(x+1). If your data included negative values it would become trickier (approaches for this case include signed log as described here or simply adding a constant value, like the minimum, to your data).


Lasso Regression Doubt I was trying to solve one problem based on Linear Regression (Predicting the sales which is a continuous variable). I have used Linear Regression for the problem but there is one suggested solution using Lasso Regression. However, the user has used Train function. I am unable to understand why train function has been used. Is it the syntax for Lasso? Similar function has been used for Ridge Regression.

my_control = trainControl(method="cv", number=5)

Grid = expand.grid(alpha = 0, lambda = seq(0.001,0.1,by = 0.0002))

lasso_linear_reg_mod3 = train(x = Train[, -c(1,2)], y = Train$Item_Outlet_Sales, method='glmnet', trControl= my_control, tuneGrid = Grid)

The user has divided it's original dataset into two sets: train set and test set.

The model i.e., lasso regression should be trained using train set. Then, the trained model is used to test the model on the test set.

In fact, when you train your model you are trying to find the optimal hyperparameters such as C and regularization (in your code, Grid) via cross validation (in your code, cv).

After finding the optimal hyperparameters, you should train your recently constructed model on the whole train set to find the coefficients (weights) of regression.

Finally, the model is tested on the test set.


When to enhance a dataset of images with flips and rotations of the images? I am a beginner in machine learning, so I'm sorry if my question is a bit trivial.

Suppose I have a dataset of images and which I want to classify, say using a neural network. It makes sense to me to try to enhance my dataset by e.g. flipping and rotating the images, so to obtain more training observations.

At some point, I want to split my dataset in a training set and a test set, and maybe also to additionally subdivide the training set for cross-validation. My question is: when should I enhance my dataset with the flipped/rotated images? If I do it before I split my dataset in training and test sample, then the test sample will contain e.g. rotations of observations that also are in the training sample, so I have the feeling it might be "contaminated" and under-represent the test error. Same thing with the split for cross-validation. Is there a consensus on how to proceed? Also, is this a valid approach?

Those operations should  be performed on the training data part. We are introducing varieties into the training data set.

If you were to do the rotation and flipping data and then you split the data set with the possibility of the rotated image of the test data being included in the training procedure. You are giving the model a glimpse of the target that you are going to test later, which defeat the purpose of testing it on new unseen data.

The cross validation and test data should be examples of real life data that your model works on. Suppose your test data are flipped in the real life regularly (for example, a car facing different way), then perhaps it can be justified but those flipped images of the test data shouldn't be included in the training procedure.


Decision trees With branching Factor? How to construct a Binary Tree from a decision tree with branching factor greater than 2 (b>2)?

I think the following issue is related to your question:
https://stats.stackexchange.com/questions/12187/are-decision-trees-almost-always-binary-trees

Let's say you have for each node, 3 children with three different conditions:

n1 -> cond a -> n2
   -> cond b -> n3
   -> cond c -> n4

For binary trees, however, each node holds only one condition and it is evaluated to either True or False, thus meaning you go to the right or the left child. For making the previous simple tree binary, you just need to chain the conditions of a node in the binary tree. For example:

n1 -> cond a -> n2
   -> not cond a -> _n2 -> cond b -> n3
                        -> not cond b -> _n3 -> cond c -> n4


Attention model with seq2seq over sequence On the official tensorflow page there is one exmple of a decoder (https://www.tensorflow.org/tutorials/text/nmt_with_attention#next_steps):

class Decoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):
    super(Decoder, self).__init__()
    self.batch_sz = batch_sz
    self.dec_units = dec_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(self.dec_units,
                                   return_sequences=True,
                                   return_state=True,
                                   recurrent_initializer='glorot_uniform')
    self.fc = tf.keras.layers.Dense(vocab_size)

    # used for attention
    self.attention = BahdanauAttention(self.dec_units)

  def call(self, x, hidden, enc_output):
    # enc_output shape == (batch_size, max_length, hidden_size)
    context_vector, attention_weights = self.attention(hidden, enc_output)

    # x shape after passing through embedding == (batch_size, 1, embedding_dim)
    x = self.embedding(x)

    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

    # passing the concatenated vector to the GRU
    output, state = self.gru(x)

    # output shape == (batch_size * 1, hidden_size)
    output = tf.reshape(output, (-1, output.shape[2]))

    # output shape == (batch_size, vocab)
    x = self.fc(output)

    return x, state, attention_weights

The output here is of shape (batch_size, vocab). I would like to have an output with shape (batch_size, max_length, vocab), where max_length is the output sequence length. I suppose there can be something done with the TimeDistributed layer but I tried several things and nothing realy worked out. Is there any work around to obtain this?

One way would be:

self.attention = layers.TimeDristributed(BahdanauAttention(self.dec_units))

but since BahdanauAttention has two inputs I do not know how to fix it, since timedistributed layer does not work with multiple inputs easily.

The output of an Attention layer  - the Context - is typically the SUM of the weighted inputs. Each of the input is diminished or magnified by the attention weights based on how relevant it is at that time-step.
So the context will have the same shape as the input. This is typically (batch_size, Encoder_Embedding_dimension). You need to generate this context at every timestep of the decoder. So you will automatically get max_length context's. So the attention layer logic is correct above, you now need to call this in the decoder in a loop.
See "def train_step(inp, targ, enc_hidden):" in the link you have shared where this is done.


Sample selection through clustering I have a biased set of samples going into a binary classification sklearn pipeline, white and black samples.

It is easy for me to fetch as many black samples as required, while whites are a bit harder to get.
I suspect the black set is unbalanced and there for I have an incentive to preprocess as many black samples as my hardware permits, and pick the most representative samples.

The feature set (after dropping completely invariant features) is about 28k features.

The best approach I came up with is to cluster my black samples to a percentage of my white samples. For example say I have 500k white samples, I would then cluster the black samples to 100k clusters and pick five samples (randomly?) out of each cluster.

My questions are the following:

How can I have a transformer that will cluster the samples and mask out the unneeded samples in sklearn? Is there something ready for such tasks or should I build my own transformer based on the clustering algorithm?
Can you point at any potential issue with what I plan to do? I think one potential risk here is not clustering the white samples, but since I'm not getting enough of those anyway...
What clustering algorithms might be a good fit? I'm looking a different clustering algorithms that accept the number of clusters as parameter right now I plan to experiment with most and see which ones work best, but a direction will be appreciated.
Additionally, I'd like to read suggestions about other approaches to sample subsampling in my case.

Clustering will be much too expensive for your purpose (most are O(n^2), and the good ones like HAC may even be O(n^3) - you won't be able to run them on 300k instances). Also beware of the prerequisites of the various algorithms - they may not be applicable to your data, or the results may be as bad as random! Also, clustering algorithms may behave rather unpredictable, and will profuce unbalanced data sets. It would not be unusual if 99% of your data ends up in the same cluster, while other clusters are almost empty (or even empty!)

I would consider first to remove all duplicates and maybe near-duplicate records (ignoring identifiers, of course). These algorithms are much more scalable and tolerate more data types. That may already be enough to achieve your goal of data reduction.


svm KKT condition violation that alpha choice in SMO when I was read the paper about SMO　for SVM.
John C. Platt paper

choosing the alpha which violated KKT condition is very confusing me. I do not very understand the code as below.

if ((r2 < -tol && alph2 < C) || (r2 > tol && alph2 > 0))

why does it choose r2 < -tol when alph2 < C, not r2 < tol?

Equation $12$ of the paper states the $KKT$ condition:

\begin{align}
\alpha_i = 0 &\iff y_iu_i\geq1 \\
0<\alpha_i < C &\iff y_iu_i = 1 \\
\alpha_i = C &\iff y_iu_i \le 1
\end{align}

Hence if $\alpha_i < C$, we are suppose to have $y_iu_i \geq 1$.

To violate $y_iu_i \geq 1$ means $y_iu_i < 1$,

$$y_iu_i < 1 = y_i^2$$

is equivalent to

$$y_i(u_i-y_i ) < 0$$

Using the notation of the paper, it is $r_2=y_i E_i < 0$.

Numerically, we consider $y_iE_i$ to be negative if it is smaller than $-\epsilon$ for some small positive number $\epsilon$.


Is AUC a good metric for evaluating the performance of a multi-class classification? Considering the definition of AUC (Area Under Curve), is that a reliable performance metric for a multi-class (30-40 classes) classification problem?

No - AUC (Area Under Curve) can not be used directly to assess the performance of multi-class classification.
If you want to use AUC, it is necessary to binarize the output. Either each class as to be compared against each other class or 1 class versus the rest.