How should I approach such classification problem where the input is an array of integers? I am training a model for predicting a number between 0 to 10 in this case. These are the number of roots of a polynomial. The input array for the number of polynomials is the coefficients of that polynomial from $x^n$ to constant. Even though I prepared a balanced dataset in which all number of roots (from 0 to 10) are equally existed, my input is an array of array of integers.

For example:

23 43 -545 34 45 -34 234 -434 234 434 -2344334

And the output of this is the number of roots of that polynomial (as I said before).

I tried many combinations of layers, but the accuracy never gets higher than 50% percent. By accuracy, I mean the number of correct predictions (I count the biggest probability as the prediction).

My Keras code for modeling:

model = Sequential()
model.add(Dense(32, input_dim=degree+1, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(degree+1, activation='softmax'))

model.compile(loss="categorical_crossentropy",
              optimizer='adam', metrics=['accuracy'])

model.fit(X, y, epochs=500, batch_size=50, verbose=0)

Is there something that I am doing wrong? Is this a good model for this kind of problem? I am new to deep learing so if the answer is obvious, I am sorry.

Thanks.

Your information is not discriminative enough

Why? Coefficients of polynomials dont give (alteast partially) discriminative information about roots of the polynomial. In other words different coefficients could give same roots.

It does not matter how complex your network is it cant catch what is not there to be catched in the first place.


What is the relation between input into LSTM and number of cells? I want to train an LSTM network for time-series predictions, and want to get to the bottom of LSTM's.
In my understanding, the number of cells in a single LSTM layer can vary. However, since each cell takes an input at time-step t, wouldn't the number of cells need to be equal to t?

For example (from TensorFlow tutorial):

 t=0  t=1    t=2  t=3     t=4
[The, brown, fox, is,     quick]
[The, red,   fox, jumped, high]

words_in_dataset[0] = [The, The]
words_in_dataset[1] = [brown, red]
words_in_dataset[2] = [fox, fox]
words_in_dataset[3] = [is, jumped]
words_in_dataset[4] = [quick, high]
batch_size = 2, time_steps = 5

Wouldn't the maximum number of cells in each layer be 5? since after the 5th input we no longer have any other information to input. However, I've seen many networks with a higher number of cells than that. Therefore, why is this a possibility?

Consider RNN networks as a simple MLP which for each time span t, takes the inputs of that time and the outputs of the previous step. Actually each time, you unroll the network. Consequently, the number of cells does not have any relation to the input size or the length of the time series. Take a look at here for a better understanding.


Ways to increase recall in SVM I am training an SVM on UCI's Bank Marketing Data Set, the bank additional-full.csv. As the data is skewed I am also interested in recall. I am getting accuracy of about 87.95% but my recall is around 51%. I want to know ways to increase recall without decreasing accuracy so much using SVM only.

My code:

from sklearn.svm import SVC

svm_clf = SVC(gamma="auto",class_weight={1: 2.6})
svm_clf.fit(X_transformed, y_train_binary.ravel())

Additional info:

I have not created any new feature (i.e combining features) and considered unknown as label.
I have also removed Duration attribute as suggested by attribute information

I have tried different class_weights, so I can increase recall upto 75.32% but then my accuracy drops to 68%
How can I increase recall in SVM models without decreasing accuracy so much?

Duplicating is RandomOverSampling, not help much in OverSampling.
I quickly did a RandomUnderSampling.The score looks good for a baseline to improve.
I have not done anything for the model improvement

 Code as it is from my Google Colab -

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank-additional.zip'

from urllib.request import urlretrieve
urlretrieve(url, "data.zip")

from zipfile import ZipFile
file_name = "/content/data.zip"
with ZipFile(file_name, 'r') as zip:
    zip.extractall()

import numpy as np,pandas as pd
data = pd.read_csv("/content/bank-additional/bank-additional-full.csv",delimiter=";")
X = data.iloc[:,:-1]
y = data.iloc[:,-1]
y.value_counts()

X_cat = X.select_dtypes(include='object')
from sklearn.preprocessing import LabelEncoder
lbe = LabelEncoder()
for colname in X_cat.columns:
    X_cat[colname] = lbe.fit_transform(X_cat[colname])
    X[colname] = X_cat[colname]
y = lbe.fit_transform(y)

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X,y,test_size=0.20,random_state=201, stratify=y)

from imblearn.under_sampling import RandomUnderSampler
rand = RandomUnderSampler(sampling_strategy=.6)
x_train, y_train = rand.fit_resample(x_train, y_train)

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=200,max_samples=0.05)
model.fit(x_train, y_train)
from sklearn.metrics import accuracy_score
y_pred_train = model.predict(x_train)

####Metrics on train
from sklearn.metrics import confusion_matrix
tn, fp, fn, tp = confusion_matrix(y_train, y_pred_train).ravel() # (55148, 1715, 8,90)
print("Training",fp/(tn+fp),fn/(fn+tp), accuracy_score(y_train, y_pred_train), tn, fp, fn, tp)

####Metrics on test
y_pred = model.predict(x_test)
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel() # (55148, 1715, 8,90)
print("Test",fp/(tn+fp),fn/(fn+tp), accuracy_score(y_test, y_pred), tn, fp, fn, tp)

from sklearn.metrics import recall_score
print("Test-recall",recall_score(y_test, y_pred))

Next -
- Try SMOTE and Combo of Over and Under
- Work on Feature Engg and Dimensionality reduction
- Check other models


calculate distance between each data point of a cluster to their respective cluster centroids I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this

token_dict="wrist. overlapping. direction. receptacles. comprising. portion. adjacent. side. hand. receive. adapted. finger. comprising. thumb. ..............................."

By using k-means clustering, I clustered this data by using k=3. Now, I want to calculate the distance between each data point in a cluster to its respective cluster centroid. I have tried to calculate euclidean distance between each data point and centroid but somehow I am failed at it. My code is as follows:

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
style.use('ggplot')
token_dict = []
import glob
path = 'E:\\Project\\*.txt'
files=glob.glob(path)
for file in files:
    f=open(file, 'r')
    text = f.read()
    token_dict.append(text)
vectorizer = TfidfVectorizer(max_df=0.8, max_features=10000, min_df=2, use_idf=True)

#print(X)

km = KMeans(n_clusters=3)
#labels = km.fit_predict(vectorizer)
#print(labels)
X = vectorizer.fit_transform(token_dict).todense()
km.fit(X)
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)
# =============================================================================
# cluster_0=np.where(X==0)
# print(cluster_0)
#
# X_cluster_0 = data2D[cluster_0]
# print (X_cluster_0)
# =============================================================================

# =============================================================================
# def euclidean(X1, X2):
#     return(X1-X2)
# =============================================================================

# =============================================================================
# distance = euclidean(X_cluster_0[0], km.cluster_centers_[0])
# print(distance)
# =============================================================================
# =============================================================================
#
# km.predict()
# =============================================================================
order_centroids = km.cluster_centers_
centers2D = pca.transform(order_centroids)
labels = km.labels_
colors = ["y.", "b.","g."]
for i in range(len(X)):
    plt.plot(data2D[i][0], data2D[i][1], colors[labels[i]], markersize=10)

plt.scatter(centers2D[:, 0], centers2D[:, 1], marker='x', s=200, linewidths=3, c='r')
plt.show()

Can someone see where I went wrong?

def k_mean_distance(data, cx, cy, i_centroid, cluster_labels):
        distances = [np.sqrt((x-cx)**2+(y-cy)**2) for (x, y) in data[cluster_labels == i_centroid]]
        return distances
clusters=km.fit_predict(data2D)
centroids = km.cluster_centers_

distances = []
for i, (cx, cy) in enumerate(centroids):
    mean_distance = k_mean_distance(data2D, cx, cy, i, clusters)
    distances.append(mean_distance)

print(distances)

Using this function I have solved my problem


Dimensionality reduction and prediction when all columns have approximately same variance I have a dataset of 25 columns where the goal is to predict the value of the 25th column based on the previous 24 columns.

The dataset is quite big that's why I initially thought to proceed with PCA before doing any prediction. The problem is that PCA did not produce any good results in the sense that it outputs a 4% variance explained on each PC.
I suspect that this can be attributed to the fact that the variance of all columns is approximately the same (say from 90-91% for each column).

I am wondering what can be done in such a case to reduce dimensionality and what data science algorithms are most suited to address this problem.

I have already tried OLS, Random Forests, SVR and Gradient Boosting regression but their scores seem quite disappointing at the moment, letting aside the fact that the computational time is quite large.

Concerning the dimensionality reduction
I encourage you to check non-linear dimensionality reduction if PCA does not give satisfying results. It can happen that some low dimensionality manifold hides behind a much higher number of features. The excellent sklearn's guide explains manifold learning in detail.
Concerning the algorithm
Almost any algorithm can manage 15 features with similar variance. It is always the case when you standardize your data before feeding the algorithm with it.
You might want to train any model on a sample of the data and check the results. It can highly reduce the computation time without hurting too much the performance. (there is a saturation of performance with respect to data for a given complexity (features + model's parameters)).
Feature Engineering importance
When performance seems to ceil when changing models, you should consider improving your feature engineering. (products of features, polynomial features, ...)


What is the gradient descent rule using binary cross entropy (BCE) with tanh? Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE).

So we have

$$
\Delta \omega = -\eta \frac{\delta E}{\delta \omega}
$$

Now we have BCE:

$$
E = −(ylog(\hat{y})+(1−y)log(1−\hat{y}))
$$

Considering my output is $\hat{y} = tanh(\omega .x)$, $x$ is my input vector and $y_i$ is the corresponding label here.
$$
\frac{\delta E}{\delta \omega} = \frac{\delta −(ylog(tanh(wx))+(1−y)log(1−tanh(wx)))}{\delta \omega}
$$

Now on this website they do something similar for the normal sigmoid and arrive at (eq 60):

$$
\frac{σ′(z)x}{ σ(z)(1−σ(z))}(σ(z)−y)
$$

Could we use that and continue there? We can get the derivative like this and get:

$$
\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x-xtanh(wx)^2}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x-x\hat{y}^2}{\hat{y}(1−\hat{y})}(\hat{y}−y)
\\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}
$$

Wherever I look, I don't find this :)

Update

Given the first answer that gives $(1 + \hat{y})(1 - \hat{y})$, we arrive at the same

$$
\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y)
\\= \frac{x(1 + \hat{y})(1 - \hat{y})}{\hat{y}(1−\hat{y})}(\hat{y}−y)
\\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}
$$

Let 'a' be the output from an activation function like sigmoid or tanh.

Therefore, the derivative of sigmoid is a*(1-a) whereas the derivative for tanh is (1+a)*(1-a).

Just follow the derivation of sigmoid except replace the derivative of sigmoid with that of tanh.


Depending upon how I download, I get two different files I am downloading the data set for the Kaggle competition on the titanic.
If I use the following code :
if (!file.exists("data")){
  dir.create("data")
}
fileUrl <- 'https://www.kaggle.com/c/titanic/download/train.csv'

download.file(fileUrl, destfile='./data/train.csv')

I get a 14kb file, however,
Paste this Url directly in your browser and you will download the correct file about 60kb.

This code works for sites where you don't need to be logged on. The Kaggle link only gives you the file when you are logged on to Kaggle. The file that is created with the code only contains html / javascript code of the Kaggle page.


Perform several different torchvision.transforms on ImageFolder object I am using pytorch to build DCGAN which i aim to train on custom dataset.

I have already posted a question on training DCGAN on small dataset, and of course answer was data augmentation.

But i have a problem. I am coding in PyTorch and i want to perform several different transfomrations on existing ImageFolder object which represents my loaded dataset.

Idea was to perform, lets say around 5 different transformations, and after performing each transformation i want to expand my dataset by adding the newly transformed images to it. Here is pseudo code:

list_of_transformations = ['random_crop','center_crop','color_jitter','mirror','grayscale']
dataset = ImageFolder('path_to_dataset')
for t in list_of_transformations
       dataset += dataset.t()

I am guessing this might not be the best idea, or is even considered 'stupid', but i am new to Deep Learning and Python in general. Any help would be appreciated! Thank you in advance!

I've just found out about the class torch.utils.data.ConcatDataset([datasets]) . Its easy to use and documentation is here : http://pytorch.org/docs/master/data.html


Why is batch size limited by RAM? The parameters of the network are changed to minimize the loss on the mini-batch, but usually the loss on the mini-batch is just the (weighted) sum of losses on each datum individually. Loosely, I would represent this as
$$ dT = \frac{1}{\text{batch_size}} \sum_{i \in \text{batch}} dT_i$$
Where $dT$ is the update of the net parameters for the batch and $dT_i$ is only for one training example.
Why can't $dT$ be calculated 'on-line' then, where the only RAM needed is on the partial sum for $dT$ and whichever $dT_i$ you are working on at that moment?

Something similar to what you describe is frequently used in some domains and it is called gradient accumulation. In layman terms, it consists of computing the gradients for several batches without updating the weight and, after N batches, you aggregate the gradients and apply the weight update.
This certainly allows using batch sizes greater than the size of the GPU ram.
The limitation to this is that at least one training sample must fit in the GPU memory. If this is not the case, other techniques like gradient checkpointing can be used.


Is it possible for a ML model to perform worse than random classification? I implemented different machine learning algorithms on a matrix with binary data to predict a univariate target with two classes.

random forest (accuracy = 62.01)
Neural Network(acc= 58.9)
svm-radial kernel (accuracy = 58.02)
linear discriminant analysis(accuracy = 57.9)
logistic regression(accuracy = 57.6).

My baseline accuracy is 52.55. But in case of Naive Bayes in same setting gives only 48.5 accuracy that identifies only one class in y. predict.

Is it possible for a machine learning model to behave worse than a random classification?

Yes, it's possible.

Just means the model is adjusting to noise, so it's valuing the "wrong" features.

As an analogy, if you were randomly guessing basketball game outcomes, you'd probably perform better than someone who thinks less points is better and is guessing based on previous games' scores


Specifying number of threads using XGBoost.train When using the xgboost.train() function, all the threads are used. I would like to use a specific amount. Unfortunately, this function does not accept the parameters nthread nor n_jobs. How can I control the number of threads being used?

Thanks.

// Edit

It seems that I found a solution. In contrast with the method, how one provides the nthread (or n_jobs) parameter to XGBClassifier of XGBRegressor, by adding this parameter directly to the brackets as xgb.XGBRegressor(nthread=n) then as indicated on xgboost document (page 46), I added an additional parameter parameters["nthread"] = number_of_threads to the parameters (a dictionary) I am using. After testing with different numbers, the number of threads being used reported in htop was the same as the number_of_threads parameter provided. Can anyone confirm this to be the right method?

You can set the number of threads by nthread parameter in XGBClassifier or XGBRegressor

import time
import numpy as np
from sklearn.datasets import load_boston
import xgboost as xgb
num_threads = [1,2,3,4,5,6,8,16,32,64]
for n in num_threads:
    start = time.time()
    model = xgb.XGBRegressor(objective='reg:squarederror',nthread=n)
    model.fit(X, y)
    elapsed = time.time() - start
    print(n, round(elapsed,3))
    results.append(elapsed)

The output after execution of this code is

    1 0.059
    2 0.071
    3 0.063
    4 0.094
    5 0.075
    6 0.078
    8 0.09
    16 0.099
    32 0.157
    64 0.235


Improving Validation Loss and Accuracy for CNN I am new to CNNs and need some direction as I can't get any improvement in my validation results.

I am trying to do binary image classification on pictures of groups of small plastic pieces to detect defects. Unfortunately, I am unable to share pictures, but each picture is a group of round white pieces on a black background. One class includes pictures with all normal pieces, the other class includes pictures where two pieces in the picture are stuck together - and therefore defective.

I have a small data set: 250 pictures per class for training, 50 per class for validation, 30 per class for testing. The pictures are 256 x 256 pixels, although I can have a different resolution if needed.

Here is my CNN architecture:

classifier = Sequential()
classifier.add(Conv2D(32, (7, 7), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Conv2D(64, (5, 5), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Flatten())

classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dense(units=1, activation='sigmoid'))

classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Treatment done to images:
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
validation_datagen = ImageDataGenerator(rescale=1./255)

train_batch_size = 10
val_batch_size = 10
num_epochs = 100
train_images = 250
val_images = 50

classifier.fit_generator(training_set,
                             steps_per_epoch=train_images // train_batch_size
                             epochs=num_epochs,
                             validation_data=validation_set,
                             validation_steps=val_images // val_batch_size)

Here are the results:

It's overfitting and the validation loss increases over time. The validation accuracy is not better than a coin toss, so clearly my model is not learning anything.

I have tried different values of dropout and L1/L2 for both the convolutional and FC layers, but validation accuracy is never better than a coin toss.

I understand that my data set is very small, but even getting a small increase in validation would be acceptable as long as my model seems correct, which it doesn't at this point.

Update:
Switching from binary to multiclass classification helped raise the validation accuracy and reduced the validation loss, but it still grows consistenly:

Any advice would be very appreciated. Thanks in advance!

Your data set is very small, so you definitely should try your luck at transfer learning, if it is an option.

Other than that, you probably should have a dropout layer after the dense-128 layer. Also, it is probably a good idea to remove dropouts after pooling layers. Instead, you can try using SpatialDropout after convolutional layers.


Linear and non-linear dependence in a single DS model I have a dataset with parameters (features) a,b,c, etc. We need to develop a model to predict a (our target).
b is correlated with a significantly (85%) and I suspect linear dependence. c is a measurement of b in another depth, so it has a high correlation with b and good correlation with a. Also, there are a bunch of other parameters (numerical features) whose dependence of a in non-linear (from background knowledge and box plot analysis). These features don't have a high correlation with a but have effect on a.
I am building a model to predict a based on the knowledge above. I think that a multi linear regression cannot be a good idea because of non-linear dependecies. On the other hand, b is a strong predictor of a and depend on a linearly. Therefore, I need to ensemble both linear regression and other methods (e.g., random forest) in a single model to have advantages of both.

The simple answer is to experiment.
You did a quite detailed analysis of the relations between your features and response variable and that's definitely a good idea, but don't be afraid to experiment with various models, even those which don't seem perfectly adapted to the task. Why? Because the one thing that such an analysis by individual features doesn't show is the patterns which happen with combinations of features, and that's what most ML algorithms are usually good at finding. This is why one often finds surprisingly good results with methods expected to perform poorly.
So my advice is to start with a range of simple methods known to be quite robust, for instance Random Forest, SVM, logistic (or even linear) regression. A good strategy is to start with simple methods and then try to improve on that based on analyzing the results. For example a more advanced approach would be to use ensemble learning: train different types of models and then a meta-model which relies on their predictions.


Why do we do citation analysis? Pardon my question if it seems too general... I'm new to networks in general, but recently I came across a type of network called citations network. However, I couldn't figure out why we do citation analysis or what kind of special information we can get out of it.

One simple example of an application of citation analysis is ranking. Imagine you do a keyword search over a collection of scientific articles, How do you rank the result set from most relevant to least? A measure one could use is the amount of citations an article has or the quality of the citations it has and also the quality of the articles the paper refers to. This problem is closely related to the problem of ranking web pages according to its hyperlinks.


What is the difference between GPT blocks and Transformer Decoder blocks? I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same?
In the original Transformer model, Decoder blocks have two attention mechanisms: the first is pure Multi Head Self-Attention, the second is Self-Attention with respect to Encoder's output. In GPT there is no Encoder, therefore I assume its blocks only have one attention mechanism. That's the main difference I found.
At the same time, since GPT is used to generate language, its blocks must be masked, so that Self-Attention can only attend previous tokens. (Just like in Transformer Decoders.)
Is that it? Is there anything else to add to the difference between GPT (1,2,3,...) and the original Transformer?

GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:

For GPT-2, this is clarified by the authors in the paper:

There have been several lines of research studying the effects of having the layer normalization before or after the attention. For instance the "sandwich transformer" tries to study different combinations.
For GPT-3, there are further modifications on top of GPT-2, also explained in the paper:


Advance Methods of Understanding Significance of Customer Behaviors I currently own a couple of websites and lately I've been implementing some feature changes - I've noticed some changes in website traffic and I was wondering what some of the more sophisticated ways are to measure the significance of the change.

I have 2 scenarios:

I changed one of my website's background from black to orange, this actually increased the site traffic by 17%.
I moved the check out button to a more obvious position on a different site, this specific site unfortunately experienced decreased traffic the day after the implementation by 8%.

I have three questions:

I know the somewhat traditional method of measuring significance, basically run a t-test (look up the z-score in a table and see the probability of this happening by chance, and if its <5% then the change is significant, etc). But is there a more advanced (data-science-y) way of measuring the significance of the change? If it's too much to explain, pointing me to a general direction to look is greatly appreciated!
Speaking of a more data-science-y approach, I've been thinking about triple-exponential smoothing. I can break the day down by hour and build a model of usage by hour, then I can compare the actual hourly usages to the predicted usages and compare the two. Is this a more sophisticated way to go about this or am I just being dumb?
Right now I'm testing out features on different sites in order to minimized interference of the new features I'm implementing. If I implement both these changes to the same site, what are some of the ways I can separate and measure the significance of these two features (kind of similar to partial correlations where I take control of certain variables).

Thanks for reading! Please let me know your thoughts. Any advice would be greatly appreciated.

Welcome to DS stack exchange! Some of your questions/statements are not too clear, I'll try to answer to the best of my abilities but try to ask more precise questions in the future to get better answers (and avoid being down-voted).

To your points 1. and 2.: I would say it's highly unlikely that you'd be able to attribute a small fluctuation like 8% to just moving a button, moreover on such a short time span. And the question you should ask yourself is: "Is this the only thing that you've changed?". The fluctuation in point 2. could easily be attributed to effects like it being the weekend. Two things you'll want to look at to get you started are A/B testing and ARMA models.

Use A/B testing to show some of your customers e.g. a button on a new position or a new background color. Now check how likely they are to return vs. the batch of customers who saw the old design. If the new one is significantly better, then you can be confident that you should roll out the new design.

Set up an ARMA model to predict daily aggregate traffic. I'd advice against smoothing of any kind, it would likely be a bad idea. You first need to set a baseline which you do by comparing e.g. a month's worth of predictions with the actual counts. From this you can estimate the probability distribution of the errors you'd expect from your model. Now when you roll out new features/contents, you can see if the predicted traffic (which is based on data for the old website) is significantly lower that the actual traffic. If so, your new feature/content is a success.

Hope this helps!


how to convert timedelta64[ns] to int? I want to convert timedelta64[ns] to int

timedelta64[ns] example :
01:00:00 , 02:00:00 , 03:00:00 , 04:00:00

I want to convert as : 1, 2 , 3 , 4

Help me please T^T

You can do:

 t, _ , _ = "01:00:00".split(":")

and then:

int(t)

That will return you: 1.

Putting this together as a function would be:

def time_to_integer(time):
    t, _,_ = time.split(":")
    return int(t)

Please note that you should pass the time variable as string. For example:

time_to_integer("02:00:00")

Would return: 2


Converting a Keras model to PyTorch I have a Keras h5 file that I want to load into the same model but this one is created using PyTorch. Is ONXX a viable intermediary option? What else can I use?

I'm sorry that this is not an exact solution to your problem but a work around I did, since I also tried to convert the models, but I didn't find any magic library to make it happen and I can't spend much time on this.

If you already trained the model you will also need to convert the weights.
Keras and TF weights are in hdf5 format, while pytorch weights are pickle, so you need to convert the weights and import the model.

Check out this post: How to convert my tensorflow model to pytorch model?. There you can see how you should add the weights one by one for each layer. It seems too much trouble.

Another option for the weights is this one (to be implemented while you train the model in keras): Pickling Keras Models ... but you will still need to convert the ANN architecture to Pytorch.

On the other hand, implementing models in keras is pretty easy and nice to get started with ANN architectures, so you could give it a go.

To sum up: I just found a keras library that implements the ANN backbone model that I needed, in order to train the model and make inference later, which is pretty straight forward.

Please let me know if you find a better option.


Linear SVM in matlab and python giving different results I have a particular dataset on which I am getting different results when using a linear SVM in matlab and sklearn toolbox.
The data has been normalized in matlab and imported into python from a mat file.
The codes used in Matlab is
acc = 0;
for i = 1:10
   [train,test] = crossvalind('HoldOut',Y,0.2);
   mdl = fitcsvm(X(train,:),Y(train),'KernelFunction','linear');%,'BoxConstraint', 10,'KernelScale',0.001);
   predictions = predict(mdl,X(test,:));
   C = confusionmat(Y(test),predictions);
   acc(i) = (C(1,1)+C(2,2))/((C(1,1)+C(1,2)+C(2,1)+C(2,2)));
end
acc = sum(acc)/10;

The code used in python is
clf_opt = svm.SVC(C=10,gamma=0.001,kernel='linear',random_state=0, tol=1e-5)
clf_opt.fit(X,y)
cvs_svm = cross_val_score(clf_opt,X,y,cv=StratifiedKFold(10)).mean()

For matlab SVM I am getting an accuracy of around 77% and in python around 60%. The choice of parameters of C=10 and gamma = 0.001 was reached after doing a GridSearchCV in python.
I went through existing posts in google for reasons of difference in LinearSVM in matlab and python but none of them worked out. I also tried out X = StandardScaler().fit_transform(X) in python but changed accuracy by 0.5 %.
I am getting comparable classifier accuracies on standard datasets (eg.IRIS) but the results  are differing in this dataset only. The dataset is attached in the link below
https://ufile.io/qs7jy
The link is a compressed file in 'rar' format and  contains three files

Python_Dataset_X  - Can be loaded with pickle
Python_Dataset_Y  - Saved as np array
Matlab_Dataset.mat - Contains the X matrix as table and Y array.

Any assistance would be appreciated.

In Matlab, you are separating a train-test (HoldOut Validation) type of data separation. You get the accuracy of the test set.

In Python, you are making a 10-fold Cross Validation where you get the resulting accuracy of the 10-fold, not using any seperate test set.

This two methodologies are definitely not the same, you need to have the same train-test structures to compare the two fairly.


Sentence similarity prediction I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the dataset. An example would look like:

New sentence: "I opened a new mailbox"

Prediction based on dataset:

Sentence                       | Similarity
A dog ate poop                   0%
A mailbox is good                50%
A mailbox was opened by me       80%

I've read that cosine similarity can be used to solve these kinds of issues paired with tf-idf (and RNNs should not bring significant improvements to the basic methods), or also word2vec is used for similar problems. Are those actually viable for use in this specific case, too? Are there any other techniques/algorithms to solve this (preferably with Python and SKLearn, but I'm open to learn about TensorFlow, too)?

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model.

Doc2vec solution
You can train your doc2vec model following this link. You may want to perform some pre-processing steps like removing all stop words (words like "the", "an", etc. that don't add much meaning to the sentence). Once you trained your model, you can find the similar sentences using following code.

import gensim

model = gensim.models.Doc2Vec.load('saved_doc2vec_model')

new_sentence = "I opened a new mailbox".split(" ")
model.docvecs.most_similar(positive=[model.infer_vector(new_sentence)],topn=5)

Results:

[('TRAIN_29670', 0.6352514028549194),
 ('TRAIN_678', 0.6344441771507263),
 ('TRAIN_12792', 0.6202734708786011),
 ('TRAIN_12062', 0.6163255572319031),
 ('TRAIN_9710', 0.6056315898895264)]

The above results are list of tuples for (label,cosine_similarity_score). You can map outputs to sentences by doing train[29670].

Please note that the above approach will only give good results if your doc2vec model contains embeddings for words found in the new sentence. If you try to get similarity for some gibberish sentence like sdsf sdf f sdf sdfsdffg, it will give you few results, but those might not be the actual similar sentences as your trained model may haven't seen these gibberish words while training the model. So try to train your model on as many sentences as possible to incorporate as many words for better results.

Word2vec Solution
If you are using word2vec, you need to calculate the average vector for all words in every sentence and use cosine similarity between vectors.

def avg_sentence_vector(words, model, num_features, index2word_set):
    #function to average all words vectors in a given paragraph
    featureVec = np.zeros((num_features,), dtype="float32")
    nwords = 0

    for word in words:
        if word in index2word_set:
            nwords = nwords+1
            featureVec = np.add(featureVec, model[word])

    if nwords>0:
        featureVec = np.divide(featureVec, nwords)
    return featureVec

Calculate Similarity

from sklearn.metrics.pairwise import cosine_similarity

#get average vector for sentence 1
sentence_1 = "this is sentence number one"
sentence_1_avg_vector = avg_sentence_vector(sentence_1.split(), model=word2vec_model, num_features=100)

#get average vector for sentence 2
sentence_2 = "this is sentence number two"
sentence_2_avg_vector = avg_sentence_vector(sentence_2.split(), model=word2vec_model, num_features=100)

sen1_sen2_similarity =  cosine_similarity(sentence_1_avg_vector,sentence_2_avg_vector)


How can learn machine learning with Statistics background? I have statistical background now I want to learn machine learning. I have experience in R, SPSS, Stata. Necessary resources and tutorials which are effective not only waste my time are needed for my early research career. I have searched several tutorials but I need effective one.

I would recommend using Kaggle or a similar website. There are a lot of competition so you probably can find one that motivates you.

One point that is imho often overseen is their very large collection of non-competition datasets. I am rather confident that you will find one that meets a personal interest of yours. You can then start to explore it and discuss results with others.

I really think that machine learning like other programming skills is best learned on practical examples. You will fail, learn from your mistakes only to fall into the next trap and this is very important as these experiences will help you later on!


How can I get my trained model ready for production I successfully trained my model using the sklearn's multiple linear regression. This is the code I used:

import pandas as pd

dataset = pd.read_csv('C:\\mylocation\\myfile.csv')
dataset2 = pd.get_dummies(dataset)
y = dataset.iloc[:, 31:32].values
dataset2.pop('Target')
X = dataset2.iloc[:, :180].values

#Split the dataset
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2, random_state = 0)

#Feature Scaling
#from sklearn.preprocessing import StandardScaler

#sc_X = StandardScaler()
#X_train = sc_X.fit_transform(X_train)
#X_test = sc_X.transform(X_test)

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

#Predicting the Test set results
y_pred = regressor.predict(X_test)

According to validation results my y_pred is a reasonable predictor. Now, I would like to take into production this model and I am wondering what are reasonable steps to apply this model to the whole dataset I have stored and future datasets if needed.

You should first cross validate your pipeline, making sure that you get an homogeneous y_pred result.

Then you can retrain a model with the same parameters on your full dataset. Pickle the model as well as any preprocessor tools, and reuse them to predict on new data.


Feature engineering decrease my cross validation I'm currently working on a fraud detection data set. I'm evaluating my training data with a 10-skfold roc auc and an estimator of default param LightGBM. But, the problem is every time I try to create a new column by calculating a ratio of 2 column, my CV always drop even though the new column is theoretically will highlight a difference between a fraud or not.

The other problem is when I find 2 good (increasing CV) separate new feature by feature engineering when evaluating them independently, my CV actually decrease when I combine both of the feature on the same evaluation.

Is there anything wrong with my way of feature engineering? Right now my step is:
1. Create a new column based on other column
2. Evaluate CV with 10-skfold default param
3. If the CV is increased (relative to original data) then it's good feature else no.

Any help is appreciated.

You might be dealing with overfitting, since your model does not generalize that well. It should be noted that Light GBM is sensitive to overfitting in small datasets.

To solve this, you can check for features with correlation and drop some of them. Another solution can be the implementation of a feature selection method. Lastly, since GBM is a tree algorithm, you can try an early termination.

If you are dealing with highly imbalance data, you can consider using a stratified method rather than CV. In sklearn, there are StratifiedKFold and StratifiedShuffleSplit, which ensure that relative class frequencies are approximately preserved in both train and validation phases.


How to use ensemble of models in FM or FFM? I am using Factorization Machines ( libfm) and also the Field Aware Factorization Machines (libffm) for a kaggle competition.  I am currently using the single models of each respectively for prediction.
I came to know that we can use ensemble of models for both FM and FFM.
Can someone explain how  this is done ?

Most ensembling methods are independent from the internals of individual models, e.g. something like majority voting, bagging or model stacking can be directly applicable to any model.

Note: this was originally a comment to the OP.


What is context window size? I am trying to implement a recurrent neural network machine translation system, and I am just learning the things.

I am creating a word embedding matrix. In order to do that, I should know my vocabulary size, dimension of the embedding space, and context window size?

What is context window?

A context window applies to the number of words you will use to determine the context of each word. Like of your statement is "the quick brown fox"  a context window of two would mean your samples are like (the,quick) and ( the, brown). Then you slide one word and your samples become  (quick, the ), (quick, brown) and (quick fox) and so on. I would suggest reading up this word 2vec tutorial to understand the training method and terminology.


Constructing graph of crypto financial instruments 2016-2017 Set of financial instruments represent the set of vertices of the graph. For any pair of vertices $i$ and $j$, an edge connecting them is added to the graph if the corresponding correlation coefficient  $C_{ij}$ based on the price  fluctuations of instruments $i$ and $j$ is greater than or equal to a specified threshold $\theta, (\theta ∈ [ − 1; 1])$.

(Reference)

Finding correlation
Degree of distribution
Analysis of cliques and independent sets in the market graph

I am wondering how to achieve the above? I am willing to understand the internal structure of the market.

Finding correlation - It has been described in the provided reference
Degree of distribution - Every decent package in which you build the network will have a function that will calculate the empirical distribution of the vertex degrees.
Analysis of cliques and independent sets in the market graph - This is a problem of Graph Clustering. There are many methods available. Start by reading this.


How to interpret PCA rankings in Weka I am struggling to understand what the rankings in Weka are representing. I.e. the coefficients for each attribute in the rank. What is the output in the Weka program for PCA telling me with these rankings? And how does this help me feature select attributes? Because right now its making no sense how they are ranked.

My data set is 31000 rows with 13 attributes. My class is Income_brack > or <= 50k

The ranking is derived form the size of the eigenvalue of the principal component (largest on top) and the scores represent 1 - cumulative sum of the variance.


Improve performance for known unlabelled test set I'm training a machine learning model from a training set of $1,000$ samples (but around $23,000$ features). I'm fairly content with my cross-validated results, but would like to improve it further by using information I have about the test set I will be using.
To explain that some more: I have an unlabelled test set with $500$ unlabelled samples which I know is the one my model needs to perform on. I was wondering if there are some techniques which allow one to weight features based on some criterion, for example whether they are significant in the test set (or alternatively, find which samples in the train set are "close" to the ones in the test set, and try to fit them with a higher importance than the other, more distant samples). In case there is such a technique, I'd also be interested in a way to do this in sklearn in particular (and preferably as part of a pipeline so that I could use it in cross-validation as well, since it is very important that I be able to gauge the accuracy of my model).

You are asking something different than standard machine learning. In standard machine learning, a test set is a dataset with labels that is evaluated a single time. The result of the test set evaluation is never used to change the model.
Another way to frame your question is - "I have high quality unlabeled data that I would combine with my labeled data to create a model."
The answer to that question is semi-supervised learning.


How to feed data to Machine Learning Model? I'm working on an SVM model as my college project. And the goal is to identify whether a tumor is benign or malignant. I'm implementing the model in Python. I found the data set from Gene Expression Omnibus.

Link to data set. The data set is related to Thyroid cancer. As you scroll down you can see a table. My problem is how can I feed that data to my model.

I don't have much idea on dealing with the data set. Could you tell me more about the data set from the mentioned website? I didn't understand it properly.

I'd be happy if you help me out with this. I'm open to all kinds of suggestions and ideas. If you know about any another data set repos, let me know.

Have a great day.

you can feed data as SVM(train_X,train_Y)
before feeding the data to model,do required proprocessing of data like having dummies or one-hot encoding for categorical variables and scaling numerical variables when mean of numerical column values are 10fold or more apart


The validation loss < training loss and validation accuracy < training accuracy I have a binary classification problem. I get the following results: a val_loss (far) lower than the train_loss, but the accuracy is also lower for the validation compared to the training set. How is that possible ?

Epoch 1/10
10708/10708 [=======] - loss: 0.6356 - acc: 0.8289 - val_loss: 0.4981 - val_acc: 0.7760
Epoch 2/10
10708/10708 [=======] - loss: 0.6243 - acc: 0.8248 - val_loss: 0.5075 - val_acc: 0.7609
Epoch 3/10
10708/10708 [=======] - loss: 0.6204 - acc: 0.8302 - val_loss: 0.5152 - val_acc: 0.7694
Epoch 4/10
10708/10708 [=======] - loss: 0.6215 - acc: 0.8307 - val_loss: 0.4981 - val_acc: 0.7824
Epoch 5/10
10708/10708 [=======] - loss: 0.6180 - acc: 0.8318 - val_loss: 0.4942 - val_acc: 0.7848

This is the Keras model I'm using:

model = Sequential()
model.add(LSTM(16, input_shape=(1, 1)))
model.add(Dense(2, activation='softmax'))

adam = Adam(lr=0.001)
model.compile(loss='categorical_crossentropy',
          optimizer=adam,
          metrics=['accuracy'])

How did you split your training and test sets? My guess is that the minority class is under-represented in your test set. Consider the case where the minority class has frequencies of 25% and 5% in the training and test sets, respectively. If my model predicts everything is the majority class, the recall for the minority class is 0% on both sets but we get 75% training accuracy and 95% validation accuracy. Now let's say we train the model a bit more and  achieve 50% recall while still correctly labeling all of the majority class samples: now we're up to 87% training and 97% accuracy. Because the training and test sets exhibit significantly different class frequencies, the test set will basically always have a higher accuracy than the training set (up to the point where you overfit, at which point the validation accuracy could be be lower than training accuracy but not necessarily).

The solution here is to either use stratified sampling to make sure your classes have equal representation between your different datasets, or to monitor other evaluation metrics like precision and recall. Plotting the taining and validation accuracies could also be helpful, since it should be visually pretty obvious if your model starts to overfit, regardless of class imbalance between your training and test sets.


What does $\mathbf{w^Tx}+w_0$ graphically mean in the discriminant function? I found a post explaining the discriminant function very detailed. But I am still confused about the function $g(\mathbf{x})=\mathbf{w^Tx}+w_0$ in 9.2 Linear Discriminant Functions and Decision Surfaces. What does it represent graphically? Could anyone explain it, probably with figure 9.2?

Does it mean the distance between the origin and the hyperplane?

Indeed, there is a lot to understand from the $g(\mathbf{x}) = \mathbf{w}^T \mathbf{x}+w_{0}$.
So let us break down what can be geometrically interpreted:
The set $\{\mathbf{x} \mid g(\mathbf{x}) = 0\}$ is a hyperplane, if $\mathbf{w} \neq \mathbf{0}$. Then, the dimension of the hyperplane is $n-1$, if $\mathbf{x}  \in \mathbb{R}^{n}$. So for $n=2$ a hyperplane is a line, for $n=3$, the hyperplane is a plane.
The vector $\mathbf{w}$ is the normal vector. It is orthogonal to the hyperplane.
So if you take vectors $\mathbf{x}',\mathbf{x}''$ of  the hyperplane, the vector $\mathbf{x}''-\mathbf{x}'$ will be orthogonal to $\mathbf{w}$.
Thus in your figure, $\mathbf{w}$ defines the orientation of the hyperplane.
Now $w_{0}$ translates the hyperplane by a corresponding vector $\mathbf{t}$ ( see here for more details).
In the following, lets assume $||\mathbf{w}|| = 1$, which does not change the hyperplane.
We can always find a basis $\mathbf{w},\mathbf{y}_{2},\ldots,\mathbf{y}_{n}$ of $\mathbb{R}^{n}$ such that $\mathbf{w} \bot \mathbf{y}_{i}$ for all $i$, that is $\mathbf{w}$ is orthogonal to all $\mathbf{y}_{i}$.
Now given $\mathbf{x}$, there are coefficients $\lambda_{1},\ldots,\lambda_{n} \in \mathbb{R}$ such that
$\mathbf{x} = \lambda_{1} \mathbf{w} + \lambda_{2} \mathbf{y_{2}}+ \ldots + \lambda_{n} \mathbf{y_{n}}$.
Therefore $ g(\mathbf{x})-w_{0} = \mathbf{x}^{T} \mathbf{w} = \langle \mathbf{x},\mathbf{w} \rangle = \langle \lambda_{1} w+ \sum_{i = 2}^{n}\lambda_{i} \mathbf{y}_{i},\mathbf{w} \rangle = \lambda_{1} \langle w,w\rangle = \lambda_{1}$
and thus $ g(\mathbf{x}) = \lambda_{1}+w_{0}$.
The geometric conclusion is therefore: $g$ depends on how you move $\mathbf{x}$ in direction of $\mathbf{w}$.
(1) Given a vector $\mathbf{x}$, and you consider $\mathbf{x}+\mathbf{u}$ for some $\mathbf{u} \bot \mathbf{w}$, then $g(\mathbf{x}) = g(\mathbf{x}+\mathbf{u})$.
So if you move $\mathbf{x}$ "along" the direction of the hyperplane, $g$ does not change its value. So given $x',x''$ of the hyperplane, if you move $\mathbf{x}$ by $\mathbf{x}'-\mathbf{x}''$, you will have $g(\mathbf{x}+(\mathbf{x}'-\mathbf{x}'')) = g(\mathbf{x})$.
(2) If you move $\mathbf{x}$ further away from the plane (along the line that contains $\mathbf{w}$), the value $g(\mathbf{x})$ will either increase or decrease, depending in which direction you move. So if you move $\mathbf{x}$ by $\mu \mathbf{w}$, you will have $\mu+\lambda_{1}+w_{0} = g(\mathbf{x}+\mu \mathbf{w}) \neq g(\mathbf{x}) = \lambda_{1}+w_{0}$, if $\mu \neq 0$.
(3) $g(\mathbf{x}) = 0$ means $\lambda_{1} = -w_{0}$.
In your image, you therefore have:
(1) A point $\mathbf{x}$ on the plane has value $g(\mathbf{x}) = 0$.
(2) Each point $\mathbf{x}$ above the plane has value $g(\mathbf{x}) > 0$.
(3) Each point $\mathbf{x}$ below the plane has value $g(\mathbf{x}) < 0$.


How do convolutional layers get connected to each other? I'm trying to analyze the architecture of a CNN and I was expecting it to be shaped:

Input Image: 28 x 28
1st conv.: 28 x 28 x nb_filters
2nd conv.: 28 x 28 x nb_filters x nb_filters

But apparently it goes:

Input Image: 28 x 28
1st conv.: 28 x 28 x nb_filters
2nd conv.: 28 x 28 x nb_filters

I thought that every filter in the second layer would be applied to every activation map in the first layer resulting in:

Activation maps after second layer = nb_filters_1st_layer x nb_filters_2nd_layer

If it's not working that way, how do I know what filters are applied to what activation maps?

There is more than one way to do this in principle, but most CNNs, and most CNN libraries will do the following:

Each layer has a target number of feature maps, which is what you have labelled nb_filters.
Each feature map is derived from all previous layer's feature maps.
The weights of the locally connected kernels do follow a width x height x nb_input_features x nb_output_features structure similar to your first example (with width and height being the convolutional kernel size), because each output feature in layer b is derived as the sum of convolutions over every feature in layer a.

So if your output feature map with (channel) index $j$ in layer $k+1$ is $I^{(j, k+1)}$, and you are looking at connection to all input feature maps $I^{(i, k)}_i$ in layer $k$ with filters connecting each map $F^{(i,j,k)}$, then the maths looks a bit like this:

$$I^{(j,k+1)} = f(b^{(j,k)} + \sum_i I^{(i,k)} \ast F^{(i,j,k)})$$

. . .where $f(x)$ is the transfer or activation function e.g. ReLU. The bias is $b^{(j,k)}$ and the $\ast$ represents the convolution function. Summing over $i$ means iterating over all the input channels. NB I have indexed the items all with a superscript like this to more clearly show that all the variables (apart from $b$) are matrices after indexing - i.e. this is not the equation for a specific pixel, but for a whole feature map.

You can see that the kernel $F$ is indexed by $(i,j,k)$ i.e. there are nb_input_features times nb_output_features filters connecting each layer.

In comparison, your expected scheme would keep the derived features separate layer-by-layer, and create an exponential growth in number of feature maps as the layers increased, with each feature "type" being handled separately. Whereas what happens in practice is each feature map is a "re-mix" of all the feature maps in the previous layer, convolved each with its own filter then summed up.

This cross-referencing of features between layers helps create richer complex feature maps - e.g. it can combine edge and corner detectors to detect corners with specific angles. Whilst a scheme that refined each feature into multiple separate sub-features would not be able to do this mixing between features as freely (until the later fully-connected layers) - although it might do well at other tasks - e.g. detecting variations of extended structures of the same type longer lines, curves etc (NB this is just a guess on my part, I don't know for certain).

  If it's not working that way, how do I know what filters are applied to what activation maps?

You cannot say something like "this feature map in layer N is derived only from this other feature map in layer N-1, using this single filter", because each feature map in layer N is derived from all feature maps in layer N-1 and results of the filters are combined in a way that you cannot easily reverse.

However, the weights array will be arranged so that you can find which filters are connecting each input feature map to a specific output feature map. Which dimension you would use to isolate those depends on the library. If your library stores weights in a [kernel_width, kernel_height, input_feature_map_id, output_feature_map_id] form, then your selection (using Numpy-style syntax might be) [:,:,:,n] where n is the id of the output feature map that you want to fetch the filters for.


How can I prove bottleneck layer of my CNN auto encoder contain useful information? I am using CNN autoencoder to create a state representation layer which I will later be feed into my Reinforcement Agent. So I trained my CNN autoencoder and it is giving nice state representations. But I have following questions,

Can my autoencoder layer be overfitted
If there's a overfit will it cause rubish information in my bottleneck layer?

Yes to both of your questions. Your autoencoder can overfit and this will cause your bottleneck to store useless information (besides any useful information it already stores).

Some ways to prevent this is:

Find a larger dataset, or augment the current.
Add noise to the input (see de-noising autoencoders).
Regularization (e.g. early stopping, sparsity constraints)


Logistic Regression Cost Function Error with regards to the Logistic Regression cost function of:

And hypothesis:

Is there a way to tell the +/- of the error for how "confident" the hypothesis is?

E.g. if the +/- of the error was 0.1, I would know that if my hypothesis predicted 0.4 it could be 0.1 greater (0.5) or 0.1 less (0.3)

This is for binary classification

Logistic regression is just a generalized linear model, so there is a linear regression lurking in here whose errors are supposed to be normally distributed, and from which you can get confidence intervals in the usual way.

That is you have a linear predictor (the -theta' * x part), transformed by the inverse of a link function (the logistic function). You can transform prediction +/- confidence interval using this inverse link to get your +/- in probability terms. It won't be symmetric.

Nice writeup of how to do this in R: https://stackoverflow.com/a/14424417/64174


Selecting threshold for F1 Score When selecting a probability threshold to maximize the F1 score prior to deploying a model (based on the precision-recall curve), should the threshold be selected based on the training or holdout dataset?

Ideally, the threshold should be selected on your training set. Your holdout set is just there to double confirm that whatever has worked on your training set will generalize to images outside of the training set.
This is the reason why hyperparameters tuning like GridSearch and RandomizedSearch in python has a cv parameter to cross-validate between different folds of your training set instead of allowing to choose the best parameters based on metric measured using the holdout set.


How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning ) How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )?
I want to replace the DNN section of Qlearning with a RF or SVR but the problem is that there is no clear training data that I can put in my code at tensorflow or keras! How we can do this?

You would need to train the RF-QN or SVR-QN on a very large batch/sample generated in the same way as a mini-batch in the DQN version. The input data would be the states and actions visited whilst simulating or running the environment for the batch, and the label would be the TD Target $r + \gamma \text{max}_{a'}\hat{q}(s', a')$ using an old copy of the RF or SVR model to calculate $\hat{q}$ with a loss function of MSE.

The reason you would need a very large batch (and why you generally don't see this done, because it will be far slower than NN) is because neither RF nor SVR in their basic form can be made to run online. However, if you manage to find online versions of the algorithm (I know there are some for RF), then you can use them almost identically to a DNN.

Wherever you see $\hat{q}$ in the pseudo-code, you know you need to run the model forward to estimate Q values. Typically you need to do this to make action selections (by finding best Q value for possible actions) and to generate TD Targets for further training.

Here's how you would need to change a DQN implementation in general terms:

You need two copies of your SVR or RF model: a "target model" and a "learning model". You would start with a basic target model that can predict either random values or fixed values for $Q(s,a)$ - those will clearly be wrong, but it should not matter. Each model will train using output from the previous one, plus the real data about transitions and reward. On each training session, the model becomes more accurate, and allows for better selection of actions.
Wherever you see a call to predict or train the DNN, replace with same call to predict or train the SVR or RF. When you see the current network being cloned to the "target network" do the same with the SVR or RF model. Pay attention to when the code uses the learning network or the target network, and use the appropriate SVR or RF model.
If you are using the default full batch training of RF or SVR, then you will need to generate a large dataset to train at once, and you should clone the resulting newly trained SVR or RF model direct to the target model immediately. Also, as the learning model in SVR or RF does not work online, you will need to use the target network to select actions in your case - that should be fine, although it may slow down learning further. Depending on your RL problem, without online versions of the algorithms, your dataset could easily need to be 100,000 or more records per training and cloning step.


Under what circumstance is lemmatization not an advisble step when working with text data? Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data?

For example, would lemmatization be something that is not done when building a context-aware model?

For reference, lemmatization per dictinory.com is the act of grouping together the inflected forms of (a word) for analysis as a single item.

For example, the word 'cook' is the lemma of the word 'cooking'. The act of lemmatization is, for example, replacing the word cooking with cook after you have tokenized your text data. Additionally, the word 'worse' has 'bad' as its lemma, and as the previous example replacing the word 'worse' with 'bad' is the action of lemmatization.

NLP tasks that would be harmed by lemmatization:

1) Tense classification

      sentence        |  tense
------------------------------------
He cooked a nice meal |  past
He cooks a nice meal  |  present

The sequence of characters at the end of verbs can help in this task.
The verbs cooked and cooks differ at the last characters ed
 and s repectively.

With lemmatization, this information is lost. Both verbs become cook, making both sentences seem (in this case) in the present tense.

2) Author identification

Given

a set of documents $\mathcal{P}$ written by author $a$,
a set of documents $\mathcal{Q}$ written by author $b$,
a set of documents $\mathcal{S}$ written by either author $a$ or $b$,

classify if a document $s\in\mathcal{S}$ is written by author $a$ or $b$.

One way to achieve this is by looking at the histogram of words present in $s$ and compare it to documents from $\mathcal{P}$ and $\mathcal{Q}$ and select the most similar one.

This works because different authors use certain words with different frequencies. However, by using lemmatization, you distort these frequencies impairing the performance of your model.


Detecting abnormal 'cat' behaviour via Supervised Learning A few work colleagues and I were looking through a recently replaced 'cat', we had in the workplace. For those of you which are curious, the 'cat' in this context, refers to a specific type of pump known as a cat pump. It's affectionally called the 'kitty' as we only have one of these pumps.

After one routine inspection, I noticed a number of periods where kitty wasn't performing well. This is labelled in the dataset as Cat Anomaly.

(Accessible from here if anyone wishes to analyse further:

https://drive.google.com/open?id=1e0OhzhaSZP9_A3QdaK9-BvWXbRPZacED)

This is what it looks like when it is abnormal:

Essentially, it looks similar to a sinusoidal pattern and is slightly curved.
I also noted that the cluster of points seem to be quite tight when it is an abnormal state.

Using Python's Sklearn Libraries and some good ol fashioned signal processing, I decided to remove some of the noisy clusters so I can see what the signal looks like 'without' the noise. To do this, I have performed a Savitzy-Golay Filter on the data, which has smoothed this out as per below.

I have then used a feature to subtract the values from the 'normal' cat values - 'smoothed' cat values which gives me a rough indication of when the 'clusters' appear.

This has worked acceptably (~90% accuracy) and I can pick up abnormalities to an extent when I feed this feature into my random forest model, but I feel the 'cluster' approach I have used is very clunky and there are much better ways to capture the 'noisiness' of the cat signal as a variable.

So now I'm looking for a better way of recognizing and separating 'abnormal' cat behaviour.

So, as a better feature or variable to capture 'noisy' clusters, does anyone else have any ideas on improvements on how to capture this?

At what resolution are you performing you classification? At sample level?

If so, what does your baseline model look like? Did you try for example just feeding the last 5-10 raw samples as features and see what accuracy you get that way?

Also, you could try seeing what happens if you do some kind of "smoothing" at the output - e.g. if you assume that one "good" sample, surrounded by 10 "bad" samples should be reclassified as "bad" also -- because the system cannot jump so quickly between normal and faulty modes of operation.

Finally (or should be firstly), what is the accuracy that is actually necessary for your model to be useful in practice?


Is there a relationship between learning rate and training set size? I have a large dataset to use for training a Neural Network model. However, I don't have enough resources to do a proper hyperparameters tuning on the whole dataset. Therefore, my idea is to tune the learning rate on the subset of data (let's say 10%), which won't obviously give as good estimate as the whole dataset would give, but since it's already a significant amount of data I would expect it might give the estimate which is sufficient enough.
However, I wonder if there is some relationship between learning rate and the training size, other than the fact that the estimate would be noisier when subsampling. And by the relationship I mean some rule of thumb, for example, increase your "optimal" LR, when you're increasing the training set size. I don't see any (apart from the relationship between LR and batch size, but it's actually other topic), but would love to make sure that I'm not missing anything.

I wonder why would you tune the learning rate (too much) at all? On many platforms you can include something like a learning rate scheduler. It will decrease the learing rate when no more learning progress is made. So you "tune" the learning rate "on the fly" while training.
With Keras this could look like:
early_stopping = EarlyStopping(monitor='val_loss', patience=50, mode='auto', restore_best_weights=True)
reduce_on_plateau = ReduceLROnPlateau(monitor="val_loss", factor=0.9, patience=20, cooldown=5, verbose=0)
checkpoint = ModelCheckpoint("xy.hdf5", monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')
...
model.fit(...,callbacks=[early_stopping, reduce_on_plateau, checkpoint]

The callbacks above introduce "early stopping" (to stop if no more progress is made), "reduce on plateau" (decrease learning rate when required), and "checkpoint" (save model results on each epoch).
Start with some reasonable standard value for the learning rate and decrease if necessary.


Bayes Optimal Decision Boundaries for Gaussian Data with Equal Covariance I am drawing samples from two classes in the two-dimensional Cartesian space, each of which has the same covariance matrix $[2, 0; 0, 2]$.  One class has a mean of $[1.5, 1]$ and the other has a mean of $[1, 1.5]$.  If the priors are $4/7$ for the former and $3/7$ for the later, how would I derive the equation for the ideal decision boundary?

If it turns out that misclassifying the second class is twice as expensive as the first class, and the objective is to minimize the expected cost, what equation would I use for the best decision boundary?

When class-conditional distributions are gaussian with equal covariance matrices, the optimal decision boundary is a hyperplane.  This is the core concept behind Linear Discriminant Analysis (LDA).

For any data point $x$, the probability that $x$ comes from class $\omega_1$ is:

$P(x|\omega_1) \sim N(\mu_1,\Sigma) = (2\pi)^{-1}|\Sigma|^{-1/2}\exp\left\{ (-1/2) (x-\mu_1)'\Sigma^{-1}(x-\mu_1)
\right\}$

Similarly, $P(x|\omega_2) \sim N(\mu_2,\Sigma)$.

Let's denote the prior probability of class 1 and 2 as $P(\omega_1)$ and $P(\omega_2)$, respectively.

Equal Misclassification Cost:

If the cost of misclassification is equal, we want to assign new data points such that the probability of misclassification is minimized. This decision rule assigns points to class 1 when $x$ satisfies:

$P(x|\omega_1)P(\omega_1) > P(\omega_2)(P(x|\omega_2)$

We can establish a similar rule for assigning points to class 2.

o find the decision boundary, we need to find the values of $x$ that satisfy

$P(x|\omega_1)P(\omega_1) = P(x|\omega_2)P(\omega_2)$

The values of $x$ that satisfy this equality here lie on a line (or hyperplane for higher dimensions).

Unequal Misclassification Cost:

For the second question, you will need an additional term.  Let's call $C(\omega_1)$ the cost of making an error when the data point was actually from class 1 and $C(\omega_2)$ the cost of making an error when the data point was from class 2.

To minimize the cost of an error, your new decision rule would be to assign $x$ to class 1 when $x$ satisfies:

$P(x|\omega_1)P(\omega_1)C(\omega_1) > P(x|\omega_2)P(\omega_2)C(\omega_2)$

As before, to find the decision boundary, you need to solve for $x$ when:

$P(x|\omega_1)P(\omega_1)C(\omega_1) = P(x|\omega_2)P(\omega_2)C(\omega_2)$

This decision boundary will still be a line / hyperplane, however it may have a different offset or orientation from the solution in the case of equal misclassification cost.


How to interpret a drastic accuracy loss while training a neuronal net (CNN)? How can one interpret a drastic accuracy loss after ~38 epochs? Maybe more dropout should be added to the CNN network?

(x-axis shows the number of epochs)

There could be several reasons:

Numeric stability issues + overfitting. Some neural activation, or some weights, continue to increase/decrease monotonically until some function (e.g. softmax) encounters a numerical stability issue. This could happen when your network is memorizing instead of properly learning.
Improper learning rate. A well defined learning algorithm usually gradually decrease its learning rate as training proceeds. If that didn't work properly, or you are using a too large constant learning rate, some activation or weights could be exploding.

For the above two reasons, you may want to print the min and max values for the activations and weights in each layer during your training. If something looks like exploding (i.e. reaching something like 10E+38) you need to check on that. Also monitor if any values become NaN or infinity.

A bug in your code for calculating accuracy or plotting. You may want to manually inspect the predicted labels and the ground truth labels to confirm whether your algorithm is performing poorly after epoch 38.


How Retail Analytics Project Can Be Done? I have a project and I couldn't understand what I have to do because I am new to retail analytics.
They said "Our goal is to measure the effects of promotion on sales" and "Your goal is to model the effect of promotion on products and stores. In order to answer questions, you should divide products and stores into 3 clusters each. (High, Medium, Low)"
I have two datasets, let's say;
1 - data.csv -> Date, Store, Product, QuantityOfSales
2 - promotion.csv -> PromName, StartDate, EndDate
There are questions like that:

What are your criteria for separating High, Medium, and Low Stores?
Why?
Which items experienced the biggest sale increase during promotions?
Are there stores that have higher promotion reactions?
What is the biggest effect explaining sales change during promotions?
Is there any significant difference between promotion impacts of the
High versus Low items?

Also, they gave me the same dataset but different time intervals to measure how well my model has worked on this new data.
Actually, I am stuck because I don't know what exactly I have to do. Can anybody show me the path I should follow or show samples?

Start by plotting some of your data in various ways in order to get more familiar with it and understand the big picture. For example plot the total amount of sales for every store, for every product, look at the different by store for the same product, difference promotion/not promotion for the same product, etc.


Which learning algorithms to use in what order - dimensionality reduction, bayesian network structure, regression? The data is a huge set of observations of dozens of variables, all (potentially, somehow) related to a dichotomous outcome variable, and all (potentially) correlated to each other, or to unknown / unobserved things.

I think that "merely" applying logistic regressions or any related extension or improvement (such as Classification And Regression Trees etc.) might not be sufficient, because there is a "basic" bayesian network structure that seems to exist (based on expert opinion) - this means that the observed variables are somehow describing distinct nodes in a bayesian network, but the exact formula HOW they describe these nodes is unclear, as is the eventual predictive power of these nodes towards the outcome.

Here is this exact problem in a highly simplified example:

We have two runners who are going to compete in a 400m sprint against each other. We have a dataset of observations of thousands of such competitions with two runners, each observation containing, for each of the two runners, (a) age, (b) size, (c) number of training runs in the past 6 weeks and (d) average mileage per training run in the past 6 weeks.

Now, we believe that (a) and (b) will determine (e) "talent for sprinting" and (c) and (d) will determine (f) "current form", and that (e) and (f) together will give the probability of winning - but we do not know HOW (a) and (b) determine (e) or how (c) and (d) determine (f) (e.g., is it "age times size" or is it "age times 2 while age lower than 45" etc.). Further, we also do not know how (e) and (f) together determine the probability of winning...

Is there any "known" approach to such a problem, i.e. a way to determine the formula that connects the initial variables to the later variables that determine the outcome, and then further determining the predictive power of the resulting network?

Or is that unnecessary (i.e. using the original variables only in some algorithm - which one? - should give the same result)? Or is it impossible (too many unknown things)? What should I do / read about / learn to solve this problem?

And in addition, is there a way to CHECK whether my original assumption (that (a) and (b) determine (e) etc.) was correct, e.g. via some sort of clustering algorithm, PCA, ...?

What would you do to approach such a question?

Thank you very much indeed in advance for helping a rather new data science learner by pointing me into the right direction!

Is there any "known" approach to such a problem, i.e. a way to
  determine the formula that connects the initial variables to the later
  variables that determine the outcome, and then further determining the
  predictive power of the resulting network?

This is kinda a question about modelling approach as well as different types of models. I think that it is always best to start with simpler models, such as simple logistic regression. In this case you are trying to predict which runner will win, so that is a 0/1 outcome. So you can put all of your variables into a logistic regression model and see how it performs in terms of accuracy. If you have a lot of predictors, you can add a penalty term like L1 or L2. The simplest model serves as a baseline by which to compare more complex models, which might not always work better.

The formulation you describe seems to like a very common one. These are often called Latent Variable models or others might prefer to call them Bayesian network models. The idea is that we cannot measure a particular quantity directly, but we can use indirect measure to estimate them. So if your sense is that "talent for sprinting" and "current form" are better predictors for predicting who wins the race, then you can use a method for Latent Variables. I have not run those in a while but there are common algorithms like the EM algorithm or Markov Chain Monte Carlo sampling.

So by fitting the two different models you can see how much accuracy you gain by adding the latent variable structure versus making no assumptions about the structure. There are methods to try and estimate the bayesian network structure, but they often require a lot of data. In a case like this you might not have enough data to a neural net or something.

I don't think dimensional reduction would necessarily help in this case. You do seem to have time-series data, which you could include directly into the logistic regression equation. You might even be able to try an LSTM or sequence model if you have enough data.

Those are some ideas off the top of my head, but I think the idea of starting with simple models, benchmarking them, and then trying more complex models is the best way to learn and to avoid getting lost. Haha. Happy running.


Why does my KNeighborsClassifier graph look like this? I'm new to data science/ml and working on using the sklearn libraries to classify data. I'm currently using the KNeighborsClassifier with 5 fold cross validation whilst tweaking the k value but its producing a graph that looks quite strange.

I have my training data and test data in 2 different CSV files and load them in like this:

trainData = pd.read_csv('train.csv',header='infer')
testData = pd.read_csv('test.csv',header='infer')

I then separate the classifiers (Y is name of the column in my dataset that's the classification):

trainY = trainData['Y']
trainX = trainData.drop(['Y'],axis=1)

testY = testData['Y']
testX = testData.drop(['Y'],axis=1)

I use sklearn KNeighborsClassifier with 5 fold cross validation whilst tweaking the k value from 2 to 20:

trainAcc = []
testAcc = []

for i in range(2,20):
    clf = KNeighborsClassifier(n_neighbors=i, metric='minkowski', p=2)
    trainScores = cross_val_score(estimator=clf, X=trainX, y=trainY, cv=5, n_jobs=4)
    testScores= cross_val_score(estimator=clf, X=testX, y=testY, cv=5, n_jobs=4)
    trainAcc.append((i, trainScores.mean()))
    testAcc.append((i, testScores.mean()))

I then print the graph:

plt.plot([x[0] for x in trainAcc],[x[1] for x in trainAcc], 'ro-', [x[0] for x in testAcc],[x[1] for x in testAcc], 'bv--')

But I get something weird like this:

Can anyone explain where I went wrong and why my graph looks the way it does.

Thanks.

EDIT: It is indeed weird because when I run it without doing the cross-validation, I get a more normal graph like this:

clf.fit(X=trainX, y=trainY)
predTrainY = clf.predict(trainX)
predTestY = clf.predict(testX)
trainAcc.append(accuracy_score(trainY, predTrainY))
testAcc.append(accuracy_score(testY, predTestY))

To summarize from the comment thread: there are two "weird" things going on here.

1. The zig-zag.

As I addressed in the comments, and @BrianSpiering in an answer, this is probably a parity effect, arising from tied votes among the nearest neighbors when $k$ is even.

2. Training accuracy not decreasing (toward test accuracy) with increasing $k$.

This was caused by an incorrect usage of cross_val_score.  You ran that function twice, separately on the training and test sets.  But that means your results were the out-of-fold score on the training set (not the actual training score!), and scores coming from models fitted on (folds of) the test set (not scores on the test set from models trained on the training set).

Probably the cleanest way to approach this is to use GridSearchCV to search over the different values of $k$.  You can then pull the results from cv_results_ for plotting.  Otherwise, if you don't need them for anything else, you can remove either the cross-validation (just score on the train and test set) or the test set (just do cross-validation and get scores on the train and test folds).


Objects Localization Through CNN I am new to deep learning and tensor flow and I am trying to train a CNN at localizing digits in the Street View House Numbers data set. To this end I have an input set of 32x32 images and, since I want to recognize up to 5 digits, I am using as labels vectors of 20 elements like this

[top_x_digit1,top_y_digit1,widht_digit1,height_digit1,top_x_digit2, etc..]

0,0,0,0 when there is no digit

As far as I understand, after (let me say) 3 layers of convolution and pooling I can add 5 (parallel) fully connected layers aimed at extracting each the box features of a different digits (when present, 0 0 0 0 otherwise).

Is my approach correct?

Technically this is possible but most approaches combine localization with some sort of classification so that the network can get better at scoring regions, and also removes the problem of having to use the [0, 0, 0, 0] vector to encode a null region since if the classifier associated with the region gives a low probability for that object you can ignore the bounding box. That way the bounding box regression can focus on finding regions that look like they contain objects, not their type. Note that the classifier needs to be trained so that it recognizes a non-object region (ie, it must have a null/garbage/background class).

See for example:

Object localization and detection

Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

(I don't have enough reputation to post more links but you should also google the R-CNN paper "Rich feature hierarchies for accurate object detection and semantic segmentation")


In a binary classification, should the test dataset be balanced? I have a dataset with 4519 samples labeled as "1", and 18921 samples labeled as "0" in a binary classification exercise. I am well aware that during the training phase of a classification algorithm (in this case, a Random Forest) the number of 0/1 samples should be balanced to prevent biasing the algorithm towards the majority class.

However, should the test dataset be balanced as well?

In other words, if train my model with 1000 random samples of "0" class, and 1000 random samples of "1" class, should I test the model with the remaining 3519 samples of "1" class, and randomly select another 3519 samples of the majority "0" class, or I can go with the remaining 17921?

What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Thanks

The answer to your first question:

  should the test dataset be balanced as well?

is, like many answers in data science, "it depends."

And really, it depends on the audience for, and interpretability of the model metrics, which is the thrust of your second question:

  What is the impact of an imbalanced test dataset on the precision, recall, and overall accuracy metrics?

Personally, if the metrics will just be used by you to evaluate the model, I would use the sensitivity and specificity within each class to evaluate the model, in which case, I care less about the balance of the classes in the test data as long as I have enough of both to be representative. I can account for the prior probabilities of the classes to evaluate the performance of the model.

On the other hand, if the metrics will be used to describe predictive power to a non-technical audience, say upper management, I would want to be able to discuss the overall accuracy, for which, I would want a reasonably balanced test set.

That said, it sounds like your test set is drawn independently of the training data.  If you are going to balance the training data set, why not draw one balanced data set from the raw data and then split the training and test data?  This will give you very similar class populations in both data sets without necessarily having to do any extra work.


Confusion on result of K-Fold Cross Validation and Independent Test set I am relatively new in Machine Learning. I am using Random Forest and SVM for a project. Where I did a 10-fold cross-validation on the train set. which gives the following K-Fold Average Score:

Random Forest: 0.8716 &
SVM: 0.8665

On the other hand, when I tested with the independent test set it gives the following accuracy:

Random Forest: 93.63%
SVM: 90.47%

I am confused is it ok? I mean can the test accuracy be greater than K-fold? Is it what underfitting called. Please help. TIA.

Absolutely, I've had this happen to me. As Ben Reiniger pointed out in the comment, check your individual scores of the folds. I'd expect you to have some folds that score worse than your reported k-fold score and some that score better, maybe even better than the independent test set.

Some things I consider when this happens:

When you test on the independent test set, have you trained on the full training set?
What accuracy do you get if you predict on the training set after using it for training? Better or worse than the accuracy you got on the independent test set?

Lastly, consider the balance of classes in your training set, training folds, and your independent test set. Accuracy scores are sensitive to class imbalance, and may cause such a discrepancy in scores between sets if the balance shifts. If you suspect class imbalance is an issue, you could either balance it (e.g. undersample the test set) or use a different metric.


How to extract tabular data from pdf? I have a requirement where I need build a solution to extract information from multiple pdfs.
The information is stored in tabular format in pdf files. The tables are not very well structured and each pdf document has different format of tables.
The pdfs consist of a lot of pages with text and tables.

I tried Tabula, but it didn't produce a satisfactory result. Thought about using pdfminer as well, but it looks very cumbersome to me.

Can you suggest more ideas and tools to solve this problem?

For some literature on the problem you may want to look at https://medium.com/sap-machine-learning-research/chargrid-77aa75e6d605 and https://arxiv.org/abs/1809.08799

For a commercially available solution Textract and Tika come to mind.


Is there a different between one hot encoding and labelling categorical as scalar value and then normalise the batch? Let's say I have label = ['Male', 'Female', 'Undefined'].
Is there a difference if I label it as:

[[1, 0, 0], [0, 1, 0], [0, 0, 1]]

compare to labeling it as

[0, 1, 2] and then normalize the data later?

On the topic of this, what if I have another feature age, which of course only has the length of 1. How would I input Age and Gender in the case I choose one hot encoding, since the length of Gender now becomes 3? I'm using Keras.

The second option is wrong. If you label Male, Female, Undefined as [0, 1, 2] you are treating categories as numbers. That is not correct. The Network in this way would learn something such as Undefined > Female, which doesn't make sense.

You need one hot encoding to treat those variables: they are not a numerical values that you can put on a scale.

About the Age question: if that's a numerical variable (expressed in years, I guess?) then you can feed it directly into the Neural Network without problems.


Python plot for confusion matrix similar to confusion wheel? I have a confusion matrix with 7 classes and would like to represent the matrix in a graph. Something similar like a confusion wheel.

Mainly I need to show, the correct observations in each class and incorrect ones with other classes in a pie chart.

Any python implementations would be great.

http://www.cvast.tuwien.ac.at/ConfusionAnalysis/

I would strongly recommend against using a confusion wheel to visualize your confusion matrix. As impressive and fancy as they look, confusion wheels are visually complicated and unintuitive to read. Good data visualizations summarize information in a way that is simple, clear, and intuitive. Confusion wheels have none of those properties.

Unfortunately, there really isn't a great way to visualize a large confusion matrix. The best ones I've seen are a grid-based plot with either color-coding or size-coding representing the value in each table cell. Funnily enough, the visualization in Fig. 1(a) of the link you included is significantly easier to read and more informative than the confusion wheel in Fig 1(b). A grid graph like the one in the link's Fig. 1(a) is what I would recommend.


Use embeddings to find similarity between documents I need to find cosine similarity between two text documents. I need embeddings that reflect order of the word sequence, so I don't plan to use document vectors built with bag of words or TF/IDF. Ideally I would use pre-trained document embeddings such as doc2vec from Gensim.
How to map new documents to pre-trained embeddings ?

Otherwise what would be the easiest way to generate document embeddings in Keras/Tensorflow or Pytorch?

There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called infer_vector which allows to you map embeddings.

You can also use bert-as-service to generate sentence level embeddings. I would recommend using Google's Universal Sentence Encoder (USE) model to generate sentence embeddings if your goal is to find some sort of similarity between sentences or documents. There are ways you can combine sentence level embeddings to document level, the first step to try would be to take the mean, or you could generate sentence embeddings for a sliding window over the document and take the mean of that.

The reason I recommend USE over BERT is that, USE was trained specially for sentence similarity tasks whereas BERT, even though can be applied to any NLP task was original trained to predict words in a sentence or complete a sentence. You might find this link helpful, it draws a great comparison between USE and BERT, and why it is important to choose a model based on task.


Decision tree and random forest over fitting I am working on a real state data set to predict the price of buying a house in Dubai based on area, no.of bedrooms, number of baths and the town which the house is in.

All variables are numerical except the town variable which I transformed, using one hot encoding and target transform.

Then I applied linear regression, decision tree, random forest and I got the same results for both transforming methods.

Algorithm            Train score        Test score

linear regression    0.50               0.45
decision tree        0.93               0.79
random forest        0.94               0.77

From the results, I can read(but not so sure) that the data has information and has a high correlation scores but the model is over fitting.
I used grid search to optimize the hyper parameters of the decision tree but the result did not improve.

So, the question is, what am I doing wrong?

Here are some questions that might help:

Do you have a balanced dataset? How does the distribution look like?
Is the distribution of the training and test set similar?
Did you try using cross validation?
What is your accuracy metric?
Are the features correlated among themselves?
I would also suggest using xgboost and compare its results.


Naive Bayes implementation: why Laplace smoothing is different from theory? Let's have a Naive Bayes Bernoulli classifier with $n_C$ classes and $n_F$ features.

According to the formula in here and here and almost every theory book I could see, Laplacian smoothing means that we take

$$P(x=X|c=C)=\frac{\#\{x=X,c=C\} +\alpha}{\#\{c=C\} +n_F\alpha }$$

but in the implementations of sklearn and this book the formula is actually

$$P(x=X|c=C)=\frac{\#\{x=X,c=C\} +\alpha}{\#\{c=C\} +2\alpha}$$

why is the number of features taken as two? Is there a theoretical reason?

I think the answer is just that in the case of the Bernoulli model, your variables are binary variables which means the only values they can take are 0 or 1. Therefore the "number of features" also called "size of the vocabulary" in the case of text classification is limited to 2.


How can I minimize features of the trainded model? I have real technological process, that explained with complex model (xgboost). I.e. current mass of a product (y) depends on current temperature (x1), pressure (x2) and so on. I would like to solve optimization task: which minimal values of the features can be selected, that mass of a product can reach the maximum?
It looks like simple optimization task: ||y-y0||^2 where y - equation of the model process and y0 - maximum or some of the closest to maximum values. But it is impossible to get weighted coefficients of the xgboost, so I can`t use skopt and even if I can get the coefficients, the real equation will be very difficult. Only decision that I have right now is sort out all possible values for all possible features, make predictions for this features and choose optimal, if y will reach maximum or close to it.
Could you give an advice, how can I solve this problem?

There are several algorithms which can help you in a smart way.

Usually, those algorithms are used to tune the hyperparameters of a model, so this is what you will find in the tutorials/examples. In your case, you have to find a good set of features instead of a good set of hyperparameters, but the principle is the same.

My suggestions:

1) SMAC. This is based on Bayesian optimization. It's an iterative process where a proxy function is built and maximized:

the function to be optimized (your XGBoost model) is evaluated in a point (in the feature's hyperspace) where the optimizer believes it can find the maxima (or, in the very first iteration, in a point given by the user);
the result is added to the set of all the evaluation points, and this set is used to build the proxy function;
the proxy function is maximized, and the coordinates of that maximum are believed to be the same where the original function will have a maximum too.

Those three steps are repeated as much as you want. So, repeat from the first step;

It works both for continuous and for categorical features, and you can also impose some constraints between features.

Here an example for your case, in Python (code not tested):

from smac.configspace import ConfigurationSpace
from ConfigSpace.hyperparameters import UniformFloatHyperparameter, UniformIntegerHyperparameter
from smac.scenario.scenario import Scenario
from smac.facade.smac_facade import SMAC

#a continuous feature that you know has to lie in the [25 ~ 40] range
cont_feat = UniformFloatHyperparameter("a_cont_feature", 25., 40., default_value=35.)

#another continuous feature, [0.05 ~ 4] range
cont_feat2 = UniformFloatHyperparameter("another_cont_feature", 0.05, 4, default_value=1)

#a binary feature
bin_feat = UniformIntegerHyperparameter("a_bin_feature", 0, 1, default_value=1)

#the configuration space where to search for the maxima
cs = ConfigurationSpace()

cs.add_hyperparameters([cont_feat, cont_feat2, bin_feat])

# Scenario object
scenario = Scenario({"run_obj": "quality",   # we optimize quality
                     "runcount-limit": 1000,  # maximum function evaluations
                     "cs": cs,               # the configuration space
                     "cutoff_time": None
                     })

#here we include the XGBoost model
def f_to_opt(cfg):

    #here be careful! Your features need to be in the correct order for a correct evaluation of the XGB model
    features = {k : cfg[k] for k in cfg if cfg[k]}
    prediction = model.predict(features)

    return prediction

smac = SMAC(scenario=scenario, rng=np.random.RandomState(42),
        tae_runner=f_to_opt)
opt_feat_set = smac.optimize()

#the set of features which maximize the output
print (opt_feat_set)

2) dlib optimisation. This converge much faster than the previous. As disclaimer, I have to say that this is an algorithm which in principle works only with functions that fulfill a certain criteria, and XGBoost models as functions do not. But in the reality it turns out that this procedure works also for less stringent functions, at least in the cases I tried it. So maybe you want also give a try.

An example code:

import dlib

#here we include the XGBoost model. Note that we cannot use categorical/integer/binary features
def f_to_opt(cont_feat, cont_feat2):
    return model.predict([cont_feat, cont_feat2])

x,y = dlib.find_max_global(holder_table,
                           [25, 0.05],  # Lower bound constraints on cont_feat and cont_feat2 respectively
                           [40, 4],    # Upper bound constraints on cont_feat and cont_feat2 respectively
                           1000)         # The number of times find_max_global() will call  f_to_opt


How can I extract the reason of the legal compensation from a court report? I'm working on a project (court-related). At a certain point, I have to extract the reason of the legal compensation. For instance, let's take these sentences (from a court report)

Order mister X to pay EUR 5000 for compensation for unpaid wages

and

To cover damages, mister X must pay EUR 4000 to mister Y

I want to make an algorithm that is able from this sentence to extract the motive of legal compensation. For the first sentence

Order mister X to pay EUR 5000 for compensation for unpaid wages

the algorithm's output must be "compensation for unpaid wages" or "compensation unpaid wages ".

For the second sentence, the algorithm's output must be "cover damages". Output can be a string or a list of string, it doesn't matter.

As I'm not an NLP expert (but I have already worked on a project on sentiment analysis, so I know some stuff about NLP), and there are so many articles, I don't know where to start.

I'm working on French texts, but I can get away with working on English texts.

Can you clarify how consistent the structure of these sentences will be?

If the court reports contain very consistent phrasing, you might be able to get away with using a rule-based approach (or perhaps even regular expressions).
But, if the sentences would require human-level language abilities to parse, you would probably need to invest a lot of energy into training a model to accomplish this, which would also require a large(ish) set of labelled training data.

That said, you may be able to apply parts of speech (POS) tagging to isolate the right class of word in each sentence. I'm sure other options also exist, but I know NLTK accomplishes this task.