How can I measure the similarity between 2 IP addresses? Is there any code to re-use? I need to measure the similarity of 2 IP addresses. I could not find any sample code in scala or other languages to find the distance between 2 IPs.

In Python there is the ipaddress module which can convert an IP address to an integer, then simply subtract to find the difference.

from ipaddress import ip_address
int(ip_address(‘10.0.0.1’))


Are there any unsupervised learning algorithms for time sequenced data? Each observation in my data was collected with a difference of 0.1 seconds. I don't call it a time series because it don't have a date and time stamp. In the examples of clustering algorithms (I found online) and PCA the sample data have 1 observation per case and are not timed. But my data have hundreds of observations collected every 0.1 seconds per vehicle and there are many vehicles.

Note: I have asked this question on quora as well.

What you have is a sequence of events according to time so do not hesitate to call it Time Series!
Clustering in time series has 2 different meanings:

Segmentation of time series i.e. you want to segment an individual time series into different time intervals according to internal similarities.
Time series clustering i.e. you have several time series and you want to find different clusters according to similarities between them.

I assume you mean the second one and here is my suggestion:
You have many vehicles and many observations per vehicle i.e you have many vehicles. So you have several matrices (each vehicle is a matrix) and each matrix contains N rows (Nr of observations) and T columns (time points). One suggestion could be applying PCA to each matrix to reduce the dimenssionality and observing data in PC space and see if there is meaningful relations between different observations within a matrix (vehicle). Then you can put each observation for all vehicles on each other and make a matrix and apply PCA to that to see relations of a single observation between different vehicles.
If you do not have negative values Matrix Factorization is strongly recommended for dimension reduction of matrix form data.
Another suggestion could be putin all matrices on top of each other and build a NxMxT tensor where N is the number of vehicles, M is the number of observations and T is the time sequence and apply Tensor Decomposition to see relations globally.
A very nice approach to Time Series Clustering is shown in this paper where the implementation is quiet straight forward.
I hope it helped!
Good Luck :)

EDIT
As you mentioned you mean Time Series Segmentation I add this to the answer.
Time series segmentation is the only clustering problem that has a ground truth for evaluation. Indeed you consider the generating distribution behind the time series and analyze it I strongly recommend this, this, this, this, this and this where your problem is comprehensively studied. Specially the last one and the PhD thesis.
Good Luck!


sentence classification with RNN-LSTM - output layer i have read a few  blogs and papers on the IMDB exercise w.r.t sentiment classification using LSTM's (and at times in conjunction with CNN) but there the output layer can contain just 1 neuron with a sigmoid since the sentiment can either be good or bad. But if i need to use the same technique to classify, say, 30,000 sentences into 20 different labels then what should my output layer look like ? if i am not mistaken i should have the same number of neurons in my output layer as the number of labels i am training the data for (20 in this case) each of which has the same sigmoid. Can you please let me know if this makes sense ?

When doing a multiclass classification problem, in which the goal is predict exactly one class label for each input, it is standard to use the softmax function (a normalized exponential) as the activation function for the last layer. However, if your problem is a multilabel classification (in which the classes are not mutually exclusive), then using a sigmoid as the activation function would be appropriate. In both situations, you would have as many output units as labels (20 in your case).


When should you balance a time series dataset? I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?

If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Are parametric method and supervised learning exactly the same? Parametric methods make explicit assumptions about the functional form of $f$, and supervised learning are pretty the same right? It's a way to build a statistical model for predicting, or estimating, an output based on one or more inputs.

If they are the same, why would they have to create different names. Is there a reason behind this?

Parametric methods in simple terms follow a particular distribution. The most common example would be that of Normal Distribution, where 64 percent of the data is situated around +-1 standard deviation from the mean. The essence of this distribution is the arrangement of values with respect to their mean. Similarly, other methods such Poisson Distribution etc have their own unique modeling technique.

Parametric Estimation might have laid the foundation to some of the most vital parts of Machine Learning but it is an absolute mistake to think supervised learning is the same thing.

Supervised Learning may include approaches to fit the aforementioned parametric models but this might not always be the case. More often, the data scatter is quite spread out. It might not just be fitting one parametric model but a hybrid of more than one. Supervised Learning also takes into account the error which most parametric models don't consider unless incorporated manually. You could say that supervised learning is an evolved and more robust version of parametric methods which is highly flexible.


How to determine semantic differences in NLP I would need to determine the difference in meaning between the following two sentences:
I am at home
I am not at home
I am at the office

the first two sentences differs in verb, which changes the meaning of the sentences (to negative); the second one, with the first one, differs because of the place.
I have thought of word2vec, but I am not completely sure if this is the best tool to analyse sentences like the above ones. Also cosine_similarity could be a solution, but I would have not information about the meaning. I think it is more about semantic meaning...

Out of the box, something like Google's Universal Sentence Encoder (USE) may work for your use-case. Many of the common NLP embedding techniques nowadays work on individual words and so creating sentence-level embeddings means averaging multiple word-level vectors together. USE was built to operate at the sentence level, so you may find it better.
The original paper can be found here:
https://arxiv.org/abs/1803.11175
An example blog post leveraging USE:
https://medium.com/@gaurav5430/universal-sentence-encoding-7d440fd3c7c7


Assuming that each invoice id has only 1 unique customer id something like this should work:
df["CustomerID"] = df.groupby("InvoiceNo")["Customer ID"].apply(lambda x: x.ffill().bfill())

The code above simply makes groups based on the customer id and forward an backward fills all NA values.


Training a model for object detection I am new to Machine Learning.  Need a little direction on how to proceed with training a model for object detection.  I have a complete training dataset and test dataset of cars with color images, annotations and bounding boxes.  This data set has 196 classes.

All google search mention use of pre-trained models.  However I want to train the model on this dataset to detect and predict bounding box values.  Did not find any on StackExchange either.

Appreciate any pointers.

Thanks

You could try a "Mask R-CNN" The github page contains a keras model that could be trained by yourself. And there is also a link to an article how to train the model yourself step-by-step.


Can we train a neural network to tell if an object is present or not in an Image? I am new to machine learning, working on object detection, but not interested in the location of the object in the image, so I just want to know is it possible to train such a neural network, if yes, how? (I just want a list of objects present in the image). I am not sure what kind of dataset I would require to build such a network.

edit: I will be more clear on my question. First I know about CNN and the models like VGG16, Inception, Resnet etc. My doubt is that these networks can be trained to identify an object i.e. it will predict which object is present in the image and give the probability. What if I want to train a network to tell whether particular objects are present in the image or not i.e. The output should be yes if any of the objects is present no if not

A standard computer vision and deep learning dataset for this problem was developed by the Canadian Institute for Advanced Research (CIFAR). The CIFAR-10 dataset consists of 60,000 photos divided into 10 classes (hence the name CIFAR-10). Classes include common objects such as airplanes, automobiles, birds, cats and so on. The dataset is split in a standard way, where 50,000 images are used for training a model and the remaining 10,000 for evaluating its performance. Other datasets are MNIST, CIFAR-100 STL-10, SVHN, ILSVRC2012 etc

State of the art results are achieved using very large Convolutional Neural networks. You can learn about state of the are results on CIFAR-10 on Rodrigo Benenson’s webpage. Model performance is reported in classification accuracy, with very good performance above 90%.

Here is a sample tutorial on convolutional neural network with caffe and python


CSVs: how to output missing data to make processing easier? If supplying data (that can be either string or numeric) via CSVs, what's a good strategy for marking that a value is missing?

Some non-empty sentinel, like NA
Some non-empty sentinel, but quoted like "NA"
The empty string, e.g. in the CSV the value will be 0 characters long.
The quoted empty string ""
Something else / depending on case?

If it makes a difference, would like the CSVs to behave reasonably when read using R readr, Python Pandas, and Excel.

Both pandas and readr allow flexibility on the marker used to indicate missing values, allowing to specify values that should be treated as missing when reading csvs:

readr https://readr.tidyverse.org/reference/read_delim.html
pandas https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

given that, maybe you can handle missing data in your code rather than replacing values in the original csvs.

If you get to choose, maybe NA (no hyphens) is a sensible default as it's handled as missing value by both libs by default.


Hyperparameter tuning does not improve accuracy? I am working on titanic dataset, I achieved 92% accuracy using random forest. However, the accuracy score dropped to 89% after I tuned it using Gridsearch. Now, I was wondering if it caused by imbalanced dataset, since it has only 342 out of 891 passenger survived the disaster. Would appreciate the clarification.

Welcome! You haven't given us enough information to be able to diagnose this issue completely, but you should check your grid search code to see how each cross-validated model is being trained and note which parameters are different from those used with the 92% model.

If it has something to do with the unbalanced data, it's because you're not stratifying your sampling of the dataset with respect to the labels (making sure training and validation sets have the same ratio of classes (in this case, 38% survive, 62% don't survive.))

I would guess that something is going on with your cross validation process.  If I had to guess specifics (and again, we can't say for sure given what you've posted here) , I would say that something you're doing in the CV probably results in those models not using as much training data as the 92% model.


Implication of a dominant Principal Component in PCA analysis I need help, are there any practical implications of a dominant principal component. For example, if of three PCs, PC1 explains almost 100% of the variance in this dataset, What does this mean in practical terms? or what does it say about the dataset?

Any help is highly appreciated. Thanks!

The principal components describe the amount of the total variance that can be explained by a single dimension of the data.

This is equivalent to the spread of the datapoints in a given dimension. The dimensions are (of course) direction that are orthogonal i.e. at 90 degrees to one another.

Have a look at this example of data points, where the red lines show the breasth of the data in two dimensions:

The dimensions don't have to be in X and Y - they could be pointing in any direction, but must be orthogonal. (more detail in this answer)

We can clearly see a bigger spread in the horizontal X-dimension, so I might expect it to account for 80% of the variance in the dataset. The vertical Y-dimension has less variance, less spread, so explains a smaller amount of the total variance. In this simple 2d example, it would explain the remaining 20% of the variance (it must sum to 100%).

In practical terms, if principal components have all very similar values, you might expect the data to form a circle (in 2d), and this means there is little directionality in the feature-space. You might like to think in terms of correlation between the features; a movement in one direction of the space does not guarantee a movement of a certain direction in the second feature. The opposite would be true if e.g. the first component had a normalised value of ~1 i.e. explained approximately 100% of the variance.

I say normalised, because the raw values that come out of PCA do not necessarily between 0 and 1 - so you can normalise them to help interpretation.

In a higher dimensional space, say with 10 variables (so 10d feature space), PCA computes eigenvectors and eigenvalues, which look for orthogonal dimensions that explain the variance of the data points, but these are not all all constricted to the dimensions of your features themselves! This means that you cannot just say that the first component (e.g. with a value of 0.6) is there because of a feature X, i.e. not due to a single feature, but a mixture of the features.

About the dataset

If the first PC explains nearly all of the data's variance, it is likely that you can express you data a lot more succinctly. PCA is often used as a dimensionality reduction method - so in this extreme case, you can reduce a high-dimensional space and convert it to a lower dimensional space without (theoretically) losing much of the explanatory power i.e. a model should be able to learn as much about the feature space from the one predominant PC, compared to the rich high-dimensional space. This of course saves compute time for any model.

I said *theoretically) above, because using PCs to model will make the interpretation of the resulting model itself - in terms of the original input features - more difficult.

For a more thorough introduction, read through a blog post like this one.
For some practical examples, check out the Sci-kit learn documentation.


Issue with predict generator keras I'm new with keras with tensorflow backend and I'm trying to do transfer learning with pretrained net.
The problem is that the accuracy on validation set is very high, around the 90% , but on test set the accuracy is very bad, less that 1%.
I solved the problem using opencv to read and resize image, but I'd like to understand why with keras methods I've this problem. I paste my code below.

from keras.preprocessing.image import ImageDataGenerator
from keras.applications.xception import preprocess_input
import keras

train_val_datagen = ImageDataGenerator( validation_split=0.25, preprocessing_function=preprocess_input)

train_val_generator = train_val_datagen.flow_from_directory( # subset di allenamento
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'training',
    seed=17)
val_train_generator = train_val_datagen.flow_from_directory( # subset di validation
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    subset = 'validation',
    seed=17)
final_train_generator = train_val_datagen.flow_from_directory( # set finale di allenamento con tutti i dati
    directory="./image-dataset/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=32,
    class_mode="categorical",
    shuffle=True,
    seed=17)

As you can see i used Xception as pretrained-net and I choose to resize my images to adapt them to the net.

After the training I created a new Iterator for test data as follow:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory(
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=1, # predico una alla volta
    shuffle = False,
    class_mode=None # non ce alcuna classe di riferimento
    )
test_generator.reset()

where the preprocess function is exactly the same.

The predictions are made with the following code:

predictions = model_xcpetion.predict_generator(test_generator, 6104, verbose = 1 )

where 6104 is the number of images in test folder.
After this i've generated a csv with images name with relative categorical probabilities:

import pandas as pd
import numpy as np
df = pd.DataFrame(predictions)
cols =[('probability of' + str(i)) for i in list(range(1, 30 )) ]
df.columns =  cols
df['images'] = imNames
df.to_csv('predictions_xception_all_data.csv', sep=',')

where the columns represent the labels (1 to 29) and imNames are obtained with filenames attribute of test_generator.
Finally I generated the csv with the labels with highest probabilities value and i compute the accuracy obtaining the value that I wrote before.

The code that I used to solve is the same but I read and resize images with the following code:

width = 299
height = 299
dim = (width, height)
images = []
# for each img
resized = cv2.resize(img, dim, interpolation = cv2.INTER_AREA)
images.append(resized)

where "img" are all the imageas read with "imread_collection" of skimage.io

Thanks in advance for your help.

EDIT1: the images resized with opencv have not been processed with the preprocessing function

I solved the problem following the advices in the comments of this discussion. I paste here my code:

dizionario = dict({'1': 0,
 '10': 1,
 '11': 2,
 '12': 3,
 '13': 4,
 '14': 5,
 '15': 6,
 '16': 7,
 '17': 8,
 '18': 9,
 '19': 10,
 '2': 11,
 '20': 12,
 '21': 13,
 '22': 14,
 '23': 15,
 '24': 16,
 '25': 17,
 '26': 18,
 '27': 19,
 '28': 20,
 '29': 21,
 '3': 22,
 '4': 23,
 '5': 24,
 '6': 25,
 '7': 26,
 '8': 27,
 '9': 28})

as you can see I created a dictionary to map classes label to index . I used the output of final_train_generator.class_indices to create this dictionary.
After predict_generator I created the csv of prediction with following code:

predicted_class_indices = np.argmax(predictions, axis = 1)
final_predictions = []
for element in predicted_class_indices:
  final_predictions.append(list(dizionario.keys())[list(dizionario.values()).index(element)])
df = pd.DataFrame()
df['class'] = final_predictions
df['imnames'] = imNames
df.to_csv('predictions_xception_all_data_bon.csv', sep=',')

I paste here also the code that I changed following the discussion link in the comments:

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_generator = test_datagen.flow_from_directory(
    directory="./TEST/",
    target_size=(299, 299),
    color_mode="rgb",
    batch_size=20,
    shuffle = False,
    class_mode = "categorical",
    )
test_generator.reset()
imNames = test_generator.filenames
predictions = model_xcpetion.predict_generator(test_generator, steps=len(test_generator), verbose = 1 )


Time series analysis on small dataset I am new to time series and therefore trying to feel my way around. I am required to do some analysis and possibly come up with a model. However, I have at most 2 years worth of quarterly data (at most 8 datapoints). Can I build any meaningful model based on the data? What possible model apart from linear regression with time as regressor can be built?

Thanks

There is no valid answer. If you have a stationary process with 0 variance, then the forecast horizon has no limit.

More realistically, you have the follow rule of thumb (which is totally from experience with absolutely no theoretical base): the horizon forecast may be half the historical base. You have 8 data points, so you can forecast 4 points. You have 2 years, so you can forecast 1 year.

The key point is : check this after your forecast is done.


Random Forest with 2D features I try to predict the position of a specific point (crest) in a 1D signal (elevation profile). Until now, I computed gradient at every point of my signal and combined that with additional features or heuristics to find approximate position of the expected output (position of the crest).
But there are some limits of this approach and I've found that ML techniques, and especially Random Forest classifiers could perform well in this kind of situation.
I would like to train my RF to find the most probable point (point_index) being the "output" based on a profile input.
Yet, I only found examples of training RF models with 1D inputs (like a time series). In my case, I have 2D input data (one signal is composed of N points with 2 features associated to each point) like the following dataframe :
   profile_index  point_index         z             z'        crest
0              0            1 -0.885429             0          false
1              0            2 -0.820151          0.02          false
2              0            3 -0.729671          -0.1           true
3              0            4 -0.649332           0.1          false
4              1            1 -0.692186             0          false
5              1            2 -0.885429           0.1           true
6              1            3 -0.820151         -0.05          false
3              1            4 -0.649332           0.2          false

I can map my data to split the dataframe for every profile, and get the output point_index as a feature, but how do I manage the fact that 2 of my features are arrays ?
Edit: here is another representation for my data
   profile_index               points_z         points_z_prime    crest_index
               0     [-0.05, ..., 2.36]        [0, ..., -0.01]            150
               1     [-0.02, ..., 4.41]        [0, ..., -0.02]            162

(this is probably irrelevant regarding the method, but I work with Python and scikit-learn)

If the number of points is constant in your array you can flatten your array and use each element as a feature in your RF. I worked on a similar problem (If I understood your problem correctly) where I predict the return of a stock based on his return on a given window of a fixed number of days and I have used the RF this way and it performs pretty well.
If your number of points isn't fixed then I suggest that that you use LSTM Neural Network where you can introduce a sequence of data (could be arrays) and it can predict the output that you are looking for.


How to do encode this target vector containing strings Consider a target vector like ["x", "x", "y", "z", "z", "z", "x", "y", "y", ...] containing the names ("x", "y", "z", ...) of some writers. For examples, assume that we have some quotes and we want to predict who has quoted that.
I want to encode these target names to numerical values. If I use frequency encoding, then, if two writers appear the same number of times in the list, they will get the same value. This is not what I want because in this case, a machine learning algorithm considers those two different persons (labels) as one person (label). If I use ordering then I will have a huge difference in the labels. Because I actually have about 80,000 different authors. "Get dummies" also doesn't work because generated data frame doesn't fit into memory. So what is the best way to encode this target column in a data frame?
My second question is, do you use multi-class classification for this task? As I said there are 80,000 different writers so there will be 80,000 labels and each label appears, in average, about 50 times in the list. Some labels appear only one time and some about 400 times!!.
In general, what supervised learning algorithm will you use for a task like this?

To answer your first question, I think OneHotEncoder from category_encoders should be up to the job, just set return_df=False to get a numpy array which you could wrap around a sparse matrix. I don't see how that would lead to a memory error unless you were operating in a very limited space.
If that is the case maybe you could simply try LabelEncoder, though this may be a less optimal solution.


Has anyone heard of a model similar to a random forest which fits a linear regression model in its leaf nodes? That is, each leaf node in each decision tree learns a linear model.
Anyone heard of this kind of model? Even better, anyone know of implementations?

M5P Model Trees are the closest thing that I'm aware of.  In these trees, the leaf nodes are linear models.  The difference is that the leaf nodes learn a multi-dimensional linear model instead of single-dimensional.
(If you had a single-dimensional model at the leaf, how would you know which feature to use in the regression?  What is the advantage over multi-dimensional?)
Of course a model tree is just a single tree, not a forest.  But you could easily build a random forest composed of model trees.


Prediction: plugin a corelation table (neuron) into a Time-Series Neuron in Keras/ TF i am adding more details

I have a time series of Babies (1,2,3) showing how many problem they have each week (Born week 1 to week 80) and in which organ (14 organ).

There is a separate numeric time-free correlation tables for babies/organs. showing Xtype blood cells per organ per baby -> Causes problems

My aim is to predict how many problem will NewBaby (Baby4) have, in which organ, per week

Predict this: Week1: 1 problem | Stomach Week1: 1 problem | Liver Week2: 1 problem | Stomach ..

TIME_SERIES_INPUT ------> LSTM (RNN?) -------\
-------------------------------------------------------------------> MERGE --->
AUXILIARY_INPUTS --> [do something] ----------/*

How can I model this problem with Keras/TF? To be specific firstly should I modify the dataset to include correlation tables. Or can I plug a model into LSTM? I have seen some examples on this board but they are mostly CNN that does not apply to my data set Ps I can insert whole dataset if needed

Thanks

DataSet Looks like

Week Baby   Age Organ   Date    IssueFound Problem Type
2   Baby2   A0  Organ2  06/24/16    1   F
2   Baby3   A0  Organ11 05/01/17    1   F
2   Baby3   A0  Organ11 05/02/17    1   F
2   Baby3   A0  Organ13 05/03/17    1   F
2   Baby3   A0  Organ13 05/05/17    1   F
3   Baby2   A0  Organ12 06/29/16    1   F
3   Baby2   A0  Organ2  07/01/16    1   F
4   Baby3   A0  Organ13 05/18/17    1   F
4   Baby3   A0  Organ2  05/19/17    1   F

time-free Co relation Table looks like

New X-Type Blood Cell per Organ per baby

Organ   Baby1   Baby2   Baby3   NewBaby
Bladder Organ1  288 247 97  253
Brain   Organ2  2548    1972    992 251
Ears    Organ3  32  254 162 155
Eyes    Organ4  111 0   459 -89
Heart   Organ5  42  1214    0   142
Kidney  Organ6  4   188 186 -87
Kidneys Organ7  806 1764    796 153
Liver   Organ8  658 657 -20 0
Lung    Organ9  0   484 0   0
Nose    Organ10 3328    9940    4435    3223
Pancreas    Organ11 11  389 207 194
Skin    Organ12 664 766 164 -145
Stomach Organ13 246 -71 0   -70
Tongue  Organ14 1403    0   33  0

If you just have a normal correlation table (Which I am not sure about) you can do it like this: For every row you can add a correlation of the organ in a row to other organs. So you will have

Week Baby   Age Organ   Date    IssueFound Problem Type Organ1 Organ2 Organ3 ...
2   Baby2   A0  Organ2  06/24/16    1      F            0.5    0      0.3    ...

If you do not have a row for a new baby you should then format the data this way together with other parameters. The Corr_Organ1 an aggregation of all correlations that lead to Organ1 failure.

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         Yes    06/24/16    No     nan         0.2         0           ...

Then you make iterations: First forecast everything with zero correlations. Then when you calculate the first forecast and Organ 1 failed you update all other correlations.

Starting data:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         nan    nan         nan    nan         0           0           ...

After first iteration:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         no     nan         0           0.8         ...

Then you predict again based on correlations and get the following result:

Baby  X-Type_cells Organ1 Organ1_time Organ2 Organ2_time Corr_Organ1 Corr_Organ2 ...
Baby1 1024         yes    xxx         yes     nan         0.2         0.8         ...

You repeat those iterations until you do not change anything.

The aggregation of correlations could be just a sum or something more complicated.


What predictive model to use to impute Gender? My data looks like this:

birth_date has 634,990 missing values
gender has 328,849 missing values

Both of these are a substantial amounts since I have 900k entries, so I can't discard empty rows. For birth_date someone recommended using Multivariate imputation by Chained equation (MICE). I don't know what predictive model I should use for gender. Of the non-missing data, there are 5x more males than females.

Can someone tell me what would be best practice here? What would be the best way to fill in the missing values for gender ?

I'm using the data to predict bike-ride duration and final destination (I know they're shown on the table above)

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often better than any imputation done in preprocessing.

The way they achieve this is by using the splitting of the decision trees they are built on. The missing values are ignored when finding the best split, then once it is found they are allocated to whichever side will minimize the loss function most.


How important is it for each row of data to have the same number of features? I'm using decision tree learning to try and classify a device based its components. Different devices have a different number of components and the location of these components within the device is important.

Device 1 might have components 9, 3, 8, 4, and 1 in that order.
Then device 2 might have components 4, 3, 2, 6, and 7 in that order.

The issue with this is I'm finding it difficult to convert these into features.

I would have the features of each row be the locations and the value be the name of the component in that location but with some devices having few (<10) components and some with many more (>60) this could lead to many blank values.

I'm worried that when training if most of the training data is on devices with for example < 40 components, when trying to classify a device with > 40 components I'm assuming it would go terribly wrong, but you also can't just use devices with more than 40 components because then it will never learn to work for simple devices with smaller numbers of components.

If there is a technique or way to get around this please let me know! Or if I've come at this problem the wrong way and need to rethink let me know that as well!

A lot of blank values in row isn't a problem, you can use sparse matrices.

I suppose you have a dataset to work with. So you can use validation and check some your suspicions. For example: fit model using dataset with <40 components and test it on another part of data.

Trying to consider order of components within device is challenging task. Maybe you can add pairs of components as features...

This is just my suggestions, I'm not an experienced data scientist =)


A2C Continuous for Pendulum-v0 working implementation, negation for loss and entropy calculation very good implementation of A2C continuous for Pendulum-v0

Code has snippet to stop execution when mean of last 10 or 20 is higher than -20 but the results look like:

episode: 706   score: [-13.13392661]
episode: 707   score: [-12.91221984]
episode: 708   score: [-50.38036647]
episode: 709   score: [-74.58410041]
episode: 710   score: [-138.1596521]
episode: 711   score: [-87.3867222]
episode: 712   score: [-63.28444052]
episode: 713   score: [-0.37368592]
episode: 714   score: [-13.28473712]
episode: 715   score: [-117.78089523]
episode: 716   score: [-25.65207563]
episode: 717   score: [-0.36829411]
episode: 718   score: [-50.81750735]
episode: 719   score: [-0.33565775]
episode: 720   score: [-0.47168285]
episode: 721   score: [-0.35240929]
episode: 722   score: [-0.40577252]
episode: 723   score: [-0.37114168]
episode: 724   score: [-25.73963544]
episode: 725   score: [-37.70957794]

Even with the reward/10 line, still pretty good. However, I don't understand these lines regarding negation of loss and why the entropy equation looks different from what I saw in Packt Publishing Deep Reinforcement Learning Hands-On per picture below:

The code:

 def actor_optimizer(self):
        #placeholders for actions and advantages parameters coming in
        action = K.placeholder(shape=(None, 1))
        advantages = K.placeholder(shape=(None, 1))

        # mu = K.placeholder(shape=(None, self.action_size))
        # sigma_sq = K.placeholder(shape=(None, self.action_size))

        mu, sigma_sq = self.actor.output

        #defined a custom loss using PDF formula, K.exp is element-wise exponential
        pdf = 1. / K.sqrt(2. * np.pi * sigma_sq) * K.exp(-K.square(action - mu) / (2. * sigma_sq))
        #log pdf why?
        log_pdf = K.log(pdf + K.epsilon())
        #entropy looks different from log(sqrt(2 * pi * e * sigma_sq))
        #Sum of the values in a tensor, alongside the specified axis.
        entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.))

        exp_v = log_pdf * advantages
        #entropy is made small before added to exp_v
        exp_v = K.sum(exp_v + 0.01 * entropy)
        #loss is a negation
        actor_loss = -exp_v

        #use custom loss to perform updates with Adam, ie. get gradients
        optimizer = Adam(lr=self.actor_lr)
        updates = optimizer.get_updates(self.actor.trainable_weights, [], actor_loss)
        #adjust params with custom train function
        train = K.function([self.actor.input, action, advantages], [], updates=updates)
        #return custom train function
        return train

Again, the entropy equation coded was this:  entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above.

Also, why is the loss a negation? actor_loss = -exp_v?

Is it negated because it is gradient ascent rather than gradient descent of the objective function for a policy gradient?

Again, the entropy equation coded was this:  entropy = K.sum(0.5 * (K.log(2. * np.pi * sigma_sq) + 1.)) which looks different from what's given in the textbook photo above.

They are the same after simple algebraic manipulations.
The entropy of a single variable Gaussian distribution with pdf $p(x|\mu, \sigma)$ is
\begin{align}
  \mathcal{H}(p) & = ln(\sqrt{2 \pi e \sigma^2}) \\
                 & = 0.5 \cdot ln(2 \pi e \sigma^2) \\
                 & = 0.5 \cdot (ln(2 \pi \sigma^2) + 1)
\end{align}
In policy gradient, we assume these $\sigma$s are isotropic. Thus the total entropy is the sum above $\mathcal{H}$s.

  Also, why is the loss a negation? actor_loss = -exp_v? Is it negated because it is gradient ascent rather than gradient descent of the objective function for a policy gradient?

Yes. In policy gradient you would like to maximize the likelihood log_pdf of getting higher episode returns advantages. At the same time we prefer a policy of high entropy. Deep learning frameworks usually only provide optimizer for minimizing the loss. Therefore you negate exp_v as loss.


Is multi-class classification the same with categorical classification? I am working on a project with image classification with 5 different classes and I was wondering if categorical classification can be used as a synonym for multi-class classification.

PS. Forgive me for the naive question

Yeah, the answer is as simple as :

Binary Classification means only 2 targets to find.
Multilabel / Multiclass / Multinomial Classification means more than 2 targets to find.

In Supervised Learning, you will have :

Classification (class of 0, 1 etc..)
Regression (continuous values 84.2, 105.8 etc..)

Categorical and class means the same and are the opposite of Regression which are continuous variables

You can find example on https://scikit-learn.org/stable/supervised_learning.html#supervised-learning


Split Dataset in two Explainable Subsets is there a general approach for splitting a dataset into two or more subsets so that a classification problem can be solved with higher accuracy for the resulting separate subsets?

I have a very inconsistent dataset which I can only classify with a 50% accuracy. I spit the set in two subsets based on an assumption of what predictor should correlate with a output variable. In my first subset, I only included samples from the dataset were the predictor correlates with the output variable. The remaining samples go in the second subset. Afterwards, both subsets (1k samples/2k samples) performed better, from 50% accuracy to 95%/65%.
In this specific example, I have a strong suspicion that the class labels used for tagging were interpreted differently by different persons which led to conflicting tags. Manually "enforcing" one interpretation helped separating this. This is supported by the fact that correlations between this output variable and other output variables drastically swap signs between one and the other subset.

I was wondering if there is a general approach on how this could be achieved for my other output variables. I.e., find a separation of my dataset so that both sets separately can be classified accurately with some machine learning algorithm.

Most classical machine learning algorithms assume targets are labeled completely correctly. If you think your labels are noisy, it would make sense to apply techniques to handle that issue.
As far as splitting a dataset into subgroups to improve performance, you are describing any tree-based machine learning algorithm. The goal of any tree-based learning algorithm is learn to split the data into more homogenous groups in the feature space based on target labels.


De-normalization in Linear Regression I have implemented a Linear regression model on a dataset of 7 independent variable and 1 target  with the below 2 approaches

1) Without normalization of the data, resulted in a Mean squared error of 36530921.0123 and R2  value 0.7477

2) With normalization of the data, resulted in a Mean squared error of 5.468490570335696e-10 and R2 value 0.92750882996584. But in this case even the Target variable is also normalized.

Which is the better approach adding new features to the 1st case or considering the 2nd case and it has a better R2 and MSE value.

If we are considering the 2nd case, what is the steps need to be taken to denormalize the target variable.

Thank you.

Unless you normalize the MSE in scenario 1 (or denormalize the MSE in scenario 2), comparing two MSE with two different scales is irrelevant.

You can have data with values varying from 10 to 30 millions, centered then normalized to -1/+1. Suppose you have a MSE of 1000 in the first case, and 0.1 in the second, you will easily see that the second MSE is way more impacting than the first one after (de)normalization.

That said, if you want to retrieve the target from scenario 2, you need to apply the reverse operations to what has been done to get the "normalized" target. Assuming for instance that you centered / reduced the target.

$$
Z = \frac{Y-\bar{Y}}{\bar{Y}}
$$
$$
Z = \alpha X_1 + \beta X_2 + \gamma X_3
$$

Where $Y$ is your initial target, $\bar{Y}$ its average, $Z$ your normalized target and $X_i$ your predictors.

When you apply your model and get a prediction, say $\hat{z}$, you can calculate its corresponding value $\hat{y}$ after applying the reverse transformations to your normalization.

$$
\hat{z} = \alpha x_1 + \beta x_2 + \gamma x_3
$$
$$
\frac{\hat{y}-\bar{y}}{\bar{y}}  = \alpha x_1 + \beta x_2 + \gamma x_3
$$
$$
\hat{y} = \bar{y} *(\alpha x_1 + \beta x_2 + \gamma x_3 ) + \bar{y}
$$

Now in this particular case, as Ankit Seth pointed out, you need not normalizing your target. The linear model would have adjusted its coefficients automatically, because you have been using linear operands.

However, if you proceed to non linear operations for target normalization, the same logic should apply. Say for instance that your model predicts the logarithm of your target.

$$
log(Y) = \alpha X_1 + \beta X_2 + \gamma X_3
$$
The reverse operation would be to apply its reciprocal function, e.g. exponential.

$$
Y = exp(\alpha X_1 + \beta X_2 + \gamma X_3)
$$


mean and variance of a dataset I have a simple question. Please see the below screenshot :

It is from a midterm exam from a university : https://cedar.buffalo.edu/~srihari/CSE555/exams/midterm-solution-2006.pdf
My questions is how the means are postive ? I am asking because the class samples are all negative so I would expect that the mean is also negative ?

It is a typo in the solution.
The author corrects it by correctly computing the Bayes discriminant in the next step.


Coloring labels using scatterplot3d in R I am trying to visualize data using R and scatterplot3d.

I have loaded data and used:

colors <- c("#999999", "#E69F00", "#56B4E9" )
scatterplot3d(output$X2,output$X6 , output$X7 , color=colors, pch="X9")

X9 is label column in my dataset. it contains 3 categories : A , B , C.

By documentation:

  color : colors of points in the plot, optional if x is an appropriate
  structure. Will be ignored if highlight.3d = TRUE.

  pch: plotting  "character", i.e. symbol to use.

Yet I still get this error

  Error in scatterplot3d(output\$X2, output\$X6, output\$X7, color =
  colors,  :    length(color) must be equal length(x) or 1

I assumed I had to put color for every collumn in dataset, but creating such array where:

length( colors ) = length( num of columns )

yields the same error.

What is the right way to add colors?

For the colors  argument you need to assign a vector with the same length as your data points.

You don't explain how colors must be assigned to values of your output columms . In any regard, this would assign 4 different colors randomly:

cols = sample(1:4, nrow(output), replace=TRUE)

plot it with

scatterplot3d(output$X2,output$X6 , output$X7 , color=cols, pch=19, cex=1)

You could create a new column in your dataframe, e.g. rgbcolor and assign that:

scatterplot3d(output$X2,output$X6 , output$X7 , color=output$rgbcolor, pch=19, cex=1)

pch is the plotting character id, usually 1, 19 or 20 (circle or larger dot or smaller dot. (see for yourself: plot(1:25, 1:25, pch=1:25))

Tune the cex param to make it larger or smaller (1 means size = 100%)


Machine learning on classifying speech So, I have 9k of 1 second wav files of a person speaking. These are labeled by whether the person speaking is wearing a face mask or not. I am supposed to come up with a machine learning model to classify these criteria.

So far I tried using KNN on the mfcc feature of the audio. This gets around 56% accuracy on test data.

I also tried to convert the wav files to jpg spectograms and apply a CNN. This one gets 60% accuracy on test data.

But I don't have that much experience, I am not sure which features of audio would best help to this problem.

Also, if you could recommend a machine learning model for this particular problem.

If you have 4,500 examples of each category, you’re doing better than random guessing. This sounds like a hard problem where the examples will have only subtle differences, so that’s an accomplishment. (I assume you’re doing some kind of out-of-sample testing.) Consider how you do in classifying perhaps 20 examples of each. I would be curious to hear how you do. This is quite a different problem if you get 38/40 than 20/40.

The way to apply a CNN, though, would be to convert to a spectrogram and then run the CNN on the spectrogram array. Converting to a literal picture is unnecessary and might harm performance. There are many ways to convert your signal to two dimensions of time-frequency space. You’ve probably tried a Fourier transform. Check out wavelets.

Since you have time series data, consider recurrent neural nets and long short-term memory. These can be combined with CNN. You will find examples of CNN/LSTM code on GitHub, I’d imagine.

You have a fairly small data set, however, for deep learning approaches. Consider simpler models like logistic regression. Your in-sample performance may suffer while improving out-of-sample performance.

Finally, consider a proper scoring rule like Brier score. Frank Harrell has written much about this on the statistics Stack, Cross Validated. Shamelessly, I will mention that you may be interested in a question of mine on CV where I somewhat challenge the idea of proper scoring rules when there is a hard decision to be made: https://stats.stackexchange.com/questions/464636/proper-scoring-rule-when-there-is-a-decision-to-make-e-g-spam-vs-ham-email. My example comes from NLP, but there could be a speech situation (say text dictation, where, at some point, you have to decide to print “dad” or “bad” or “bicycle”).


'RandomForestClassifier' object has no attribute 'oob_score_ in python I am getting:

AttributeError: 'RandomForestClassifier' object has no attribute 'oob_score_'.
But I can see the attribute oob_score_ in sklearn random forest classifier documentation.

param = [10,15,20,25,30, 40]
# empty list that will hold cv scores
cv_scores = []
# perform 10-fold cross validation
for i in tqdm(param):
    clf = RandomForestClassifier(n_estimators = i, max_depth = None,bootstrap = True, oob_score = True)
    scores = clf.oob_score_
    cv_scores.append(scores)

ERROR

AttributeError                            Traceback (most recent call last)
<ipython-input-37-d6c9c54f299e> in <module>()
     15 for i in tqdm(param):
     16     clf = RandomForestClassifier(n_estimators = i, max_depth = None,bootstrap = True, oob_score = True)
---> 17     scores = clf.oob_score_
     18     cv_scores.append(scores)
     19

AttributeError: 'RandomForestClassifier' object has no attribute 'oob_score_'

This kaggle guide explains Random Forest.

Setting warm_start to True might give you a solution to your problem. The default value is False.


NER with LSTM - How to recognize person names that are not part of the vocabulary? I am learning Named Entity Recognition and going through posts similar to this one:
Named-Entity Recognition (NER) using Keras Bidirectional LSTM
So the sentences are fed into the model as a sequence of integers - every int corresponding to the index in the vocabulary - from what I understand this is how the embedding layer works.
My question is - does that mean the model would not be able to recognize a person's name if it doesn't exist in the vocabulary?
For example, from the sentence:
"John Doe went for a walk"
given John Doe is in the vocabulary, it will be recognized as a person name but the sentence:
"Unknown Name went for a walk"
will not be properly tagged if Unknown Name is not in the vocab?
To me, this would be a little strange as Unknown Name is in the same context as John Doe so I was hoping to be able to somehow tag it properly.
I'm obviously lacking knowledge here so I'd be very grateful for any suggestions and reference materials.

With word-level vocabularies, unknown words are normally encoded with a special token <unk>. If in the training data there were enough examples of person names, then the bidirectional LSTM may have learned to identify the person name from the context, not just the word itself. In that case, and provided that in the specific input sentence there is enough context, the model may be able to identify it. If the model were a normal LSTM instead of a bidirectional one, it would be more difficult for the model to identify it, as it would only be able to use the context to the left of the unknown word.
If the training data did not contain such examples, the it won't be able to identify the person from the context.
One option to improve the handing of this problem  would be to force this kind of examples in the training data, by replacing person names with unknown words with certain probability.
Another option would be to use a subword-level vocabulary, like a byte-pair encoding (BPE) one, eliminating altogether the out-of-vocabulary (OOV) word problem.


Understanding clusters after applying PCA then K-means I have a dataset grouped by customer level, and the rows are sum_mexico, sum_uk, ... etc to indicate if the customer has spent money at stores in those countries..similarily counts for these as well. I end up with 200 columns.
I would like to cluster this data to observe the spending habits and see if i can group them by these features. I'm unsure what clustering method would be best but i would like to provide meaningful results to business.
I've read about using PCA then k-means. E.G. you can carry out K-means on your Principcal componenets. What i don't understand is how i can then interpret the results. e.g. say i have the below situation from https://365datascience.com/tutorials/python-tutorials/pca-k-means/:

i can visually see 4 clusters, but how can i get the characteristics of each cluster.. What can i say about cluster 1 for example, are they charactertized by high spend in mexico ? (assuming i got these results myself using my own data and steps from link!) My end goal is to understand what charcterizes each cluster e.g. it may be that certain customer IDS have high spends in spain and wine etc.

PCA removes the connection with the original features,so the interpretation of the visualisations in the principle component space is therefore not very meaningful.
E.g. cluster A has higher values of PC1, where cluster B has higher values of PC2.
If you can clearly see that PC1 is only representative of Feature X, then fine, but this isn't often the case.
Instead use PCA to discover which features best represent the data, and use this to remove unnecessary (unrepresentative) features so that you can visualise the original data using its most important original features. And therefore describe the clusters with meaningful differences.
You might want to experiment with a scatter matrix, to explore which features spaces have the cleanest distinction between clusters.


Is there "Attention Is All You Need" implementation in Keras? Has anyone seen this model's implementation using Keras?

inb4: tensorflow, pytorch


Is there any way to plot ROC curve for Ensemble hard voting classifier? I am working on a multi-class text classification problem and performing an Ensemble learning for text classification. I chose hard voting as ensemble technique. I tried to plot ROC curve for my ensemble method but it didn't work by showing the error that ROC curve can't be plotted for Hard voting ensemble metho. So my question is there any possible way by which I can plot ROC curve for my hard voting ensemble classifier in python.

For an ROC curve, you need the model to produce an ordering of the test points (ordered by likelihood of being in the positive class).  A few approaches:

A hard voting classifier usually just outputs the final class prediction.  This is an ordering, just with nearly all ties.  That will produce an ROC curve of two straight line segments, from the square's corners to a single point representing the model's specificity and sensitivity.  Not the most informative, but there it is.

While you probably are using a majority vote to produce the final class prediction, you can also interpret it slightly differently: produce a probability score by averaging the votes (as binary) from the base models.  The final class prediction is the same as thresholding this score at 0.5.  So now you have at least fewer ties, and you can produce an ROC curve that looks a little smoother.  This will work best if you have lots of base models, and in fact is pretty much exactly what you do when building random forests.

If your base models come with probability scores, then for each threshold you can get the classifications from all of the base models, and have those vote to produce classes.  Varying that threshold then produces an ROC curve that's just as smooth as any of the base models' ROC curves.  The catch is whether one threshold represents the same thing for all the base models: you need all the base models to be well-calibrated for this to really make sense.  You also lose insight into the potential power of setting different thresholds for each of your base models.


Keras - understanding ImageDataGenerator dimensions I'm trying to implement custom object detection by taking a trained YOLOv2 model in Keras, removing the last layer and retraining it with new data. I'm confused about how to feed the data to Keras, though. I have annotated a bunch of pictures with bounding boxes using the YOLO annotation, and put them in two separate folders (images where the .jpgs reside and annots where the .txt annotations are).

I also removed the last layer from the model and added a custom one (I'm trying to predict bounding boxes for 2 classes).

I'm trying to pass my data with an ImageDataGenerator, as my dataset is quite small.

I have the following input objects:

np.shape(train_images) # this contains RGB data from 79 pictures
(79, 1, 608, 608, 3)
np.shape(train_y)
(79,)

I'm trying to pass these to the ImageDataGenerator, but I get an error:

train_datagen = ImageDataGenerator(
      rotation_range=20,
      width_shift_range=0.2,
      height_shift_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest')

train_generator = train_datagen.flow(
        train_images,
        train_y)

ValueError: `x` (images tensor) and `y` (labels) should have the same length. Found: x.shape = (1, 608, 608, 3), y.shape = (79,)

I don't understand what the problem is. Somehow the first dimension of my images data is completely gone and thus does not match... What's wrong with it?

I think your train_images array should have shape (79, 698, 608, 3). The generator works through each of the first dimensions of those arrays, so is passing a batch of 4d numpy arrays, instead of a batch of 3d numpy arrays.

You can try seeing if that helps, using numpy.squeeze(), like this:

In [1]: import numpy as np

In [2]: a = np.random.randint(0, 10, (2, 1, 10, 10, 3))

In [3]: np.squeeze(a, axis=1).shape
Out[3]: (2, 10, 10, 3)

So be sure to set the axis=1 argument. Then just pass the updated array as you were doing.


Dealing with outliers and z-scores I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.

I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.

I have a few questions:

Which normalization function to use and when? I am currently using the z score.
If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?
After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?

Just some quick fixes.
Normalization takes care of scale of data across the columns. e.g.
If one data column is in the ranges of 1000s and other in 10s then normalization will work.

But if you want to take care of the skew-ness one way is to simply take log of the data. It pulls in the outliers. as log increases very slowly for positive values. Do it before the normalization (Z score in your case)

Other ways are taking nth root of the data.


Looking for a strong Phd Topic in Predictive Analytics in the context of Big Data I'm going to start a Computer Science phd this year and for that I need a research topic. I am interested in Predictive Analytics in the context of Big Data. I am interested by the area of Education (MOOCs, Online courses...). In that field, what are the unexplored areas that can help me choose a strong topic? Thanks.

As a fellow CS Ph.D. defending my dissertation in a Big Data-esque topic this year (I started in 2012), the best piece of material I can give you is in a link.
This is an article written by two Ph.D.s from MIT who have talked about Big Data and MOOCs. Probably, you will find this a good starting point. BTW, along this note, if you really want to come up with a valid topic (that a committee and your adviser will let you propose, research and defend) you need to read LOTS and LOTS of papers. The majority of Ph.D. students make the fatal error of thinking that some 'idea' they have is new, when it's not and has already been done. You'll have to do something truly original to earn your Ph.D. Rather than actually focus on forming an idea right now, you should do a good literature survey and the ideas will 'suggest themselves'. Good luck! It's an exciting time for you.


How to build confusion matrix , when predicted value and actual value is in sentence? I am building some model, which predict on basis of  highest probability from history and I am assuming this is best action.
I am comparing this with real action.

Predicted : process resumed somedate2

Actual :  process resumed at somedate1

As both are similar and assumed this is correct prediction.

I have applied cosine similarity for same and doing comparison.

Is there any better way to compare same and to check accuracy ?

Any help will be appreciated.

You could create 2 new columns (1 for the truth, 1 for the prediction) which have a more generalised value like PROCESS_RESUMED. You can then use those columns to create the confusion matrix.


attach "1" for the bias unit in neural networks. what does it mean? In the image below the instructor says attach a "1" for the bias unit in neural networks. what does a "bias unit"mean in neural network?

It allows you to account for some static offset in your learning process. To illustrate, consider the output of your first hidden layer without the bias unit.

h = sigma(W * x)

Here W is the matrix of your layer weights, and W * x is the matrix-vector multiplication between these weights and your input. sigma is your nonlinearity operating on each element of the result. Now consider what this looks like With the "bias unit".

h = sigma(W' * [x, 1]) ~ sigma(W*x + b)

Here, W' is your weight matrix with new entries associated with the "bias unit", and your new input is your original input with a 1 concatenated to it. You can think of this as being equivalent to your original matrix multiplication W * x plus some bias term b.

I will have to dig for the sources on this, but conventional wisdom is that you don't need the bias unit in modern architectures. If it's for a class though, I'd say use it if that's what you are instructed to do.


How to improve OCR (Scanning) results?" Below is text output obtained after ocr image to string of medical discharge summary report.

XXXXXXXX  T D.0.A'. 20.05. 2017
13.0,? ; 20.05.2017
AGE / sax; 43 YEAR(S] / MALE
CODE: IP1’7- 14041] FHL33350709 D.0.D:22.05.2017
ROOM NO: 1309F
CONSULTANT: XXXXXXXXX [CARDIOLOGIST]
CO v CONSULTANT: XXXXXXXX [GEN . PHYSICIAN]
DIAGNOSIS : ACUTE CORONARY SYNDROME
UNSTABLE ANGINA
MILD CORONARY ARTERY DISEASE
PLAN : MEDICAL MANAGEMENT

The output I obtained is not accurate enough as the input image.I want to cleanse the text output and extract the values of the fields such as DOA,DOD,AGE/SEX,CODE,ROOM NO,CONSULTANT,DIAGNOSIS etc.
For ex , i need the extract the value of the field name "consultant" as "XXXXXXXXX [CARDIOLOGIST]" as output in excel format.

Below is my attempt :

import pytesseract as pt
import re
import pandas as pd
import cv2
import numpy as np
from matplotlib import pyplot as plt

from PIL import Image

img1 = Image.open('/root/regex_try/DS_Sample.jpg')

text = pt.image_to_string(img1)

print(text)

Fieldnames_dict = {'Hospital Name':['Hospital Name', 'Hospital_Name',
'hospital name'],
'Hospital Address':['Address', 'city', 'Hospital Address'],
'Patient Name' : ['Name', 'Patient_Name', 'Patient Name', 'patient name'],
'Address' : ['Patient Address', 'Address','Amy“'],
'Age' : ['Age', 'age'],
'Sex': ['Sex', 'sex', 'gender', 'Gender'],
'Doctor Name' : ['Doctor Name', 'doctor name', 'doctor', 'treating doctor',
'Consultants','CONSULTANT','Consultant Name'],
'Inpatient Number' :['Inpatient Number', 'IP Number'],
'Admission Date' : ['Date of Admission', 'Admitted Date', 'D.0.A'],
'Discharge Date' : ['Date of Discharge', 'Discharged Date', 'D.0.D'],
'Diagnosis' : ['Diagnosis', 'Final Diagnosis', 'Principal/Secondary
 Diagnosis'],
'Treatment Given' : ['Treatment Given', 'On Examination', 'Examination'],
'Follow up' : ['Follow Up', 'Follow up', 'Review & Advise', 'Condition on
 Discharge', 'Advise on Discharge'],
'Summary' : ['Summary', 'Past Treatment Given'],
'Bed No' : ['Bed Number', 'Bed / Ward Details'],
'SS Number' : ['SS Number', 'SS No.'],
'UHID Number' : ['UHID Number', 'UHID No'],
'MR Number' : ['MR No', 'MR Number','MRNO '],
}

def preprocess_formtext(field_dict,text):
   values_dict=field_dict
   values_dict=values_dict.fromkeys(values_dict, '')
 for i in field_dict.keys(): #reaching the keys of dict
    for x in field_dict[i]: #reaching every element in tuples
        q=0
        p=[x.upper()+r":(.*) ",x.upper()+r" : (.*) ",x.upper()+r": (.*)",x.upper()+r": (.*)"]
        for j in p:
            #print(j)
            match = re.search(j, text)
            if match:
                result = match.group(1)
                q=1
                values_dict[i]=result
                break
            else:
                result = ""
        if q==1:
            break
return values_dict

values_dict=preprocess_formtext(Fieldnames_dict,text)

formdata_df=pd.DataFrame([values_dict])

Build a dictionary of common words that frequently appear in these documents (e.g., MEDICAL, SEX, AGE, etc).  Then, for each word in the output from OCR, check whether it is similar to a word in your dictionary; if so, then replace it with the dictionary word.  "Similar" might be defined as "edit distance <= 1".

For example, your sample output has the word "sax".  If your dictionary contains the word "SEX", "sax" would be detected as a misspelling of "SEX" (edit distance = 1).

This post-processing will help you detect some of the common words and help you identify the fields.  What it won't do is clean up errors in the OCR of codes, as those can be anything.  There's probably no good way to handle that.

Also, do a Google search on how to use Tesseract.  There are some best practices that seem to improve its output (e.g., convert to greyscale TIFF format, deskew text, binarize, and more).

Finally, some commercial OCR software is significantly better than Tesseract or any other free OCR.  e.g., Abbyy seems to be well-regarded.


Random Forest - ValueError: Input contains NaN, infinity or a value too large for dtype('float32') I'm trying to apply the RandomForest method to a dataset and I get this error:

ValueError: Input contains NaN, infinity or a value too large for dtype ('float32')

Could someone tell me what I can modify in the function for the code to work:

def ranks_RF(x_train, y_train, features_train, RESULT_PATH='Results'):
    """Get ranks from Random Forest"""

    print("\nMétodo_Random_Forest")

    random_forest = RandomForestRegressor(n_estimators=10)
    np.nan_to_num(x_train)
    np.nan_to_num(y_train)
    random_forest.fit(x_train, y_train)

    # Get rank by doing two times a sort.
    imp_array = np.array(random_forest.feature_importances_)
    imp_order = imp_array.argsort()
    ranks = imp_order.argsort()

    # Plot Random Forest
    imp = pd.Series(random_forest.feature_importances_, index=x_train.columns)
    imp = imp.sort_values()

    imp.plot(kind="barh")
    plt.xlabel("Importance")
    plt.ylabel("Features")
    plt.title("Feature importance using Random Forest")
    # plt.show()
    plt.savefig(RESULT_PATH + '/ranks_RF.png', bbox_inches='tight')

    return ranks

You should imput missing data in your dataset, or delete thos rows if they are not many.

Alternatively, if the values are too big to be represented by float32 datatype, try to convert them to float64 (takes more RAM).


Is it possible to use a generative model to "share" private data? Let's say we have some data set, with lots of instances $X$ and a target $y$. If it is of some importance, you may assume that it is a "real life" data set : medium sized, with important correlations, y is unbalanced...etc.

Let's also say this data set is of relative interest : the field of research is rather active, but there is no (or only a few) available data set. So we are considering publishing our data set.

However, a direct publication wouldn't be possible for privacy concerns. Basic approaches have been considered (pseudonymisation, grouping instances to achieve statistical disclosure control, publishing an old dataset), but mostly set aside.

The research field concentrate on discriminative approaches for learning $y$, based on $X$. However, I recently came to read about generative approaches. So I am curious for (1) if there are generative techniques that could be used to share information about $X$x$y$ without being able to recognise any individual instances.

I think the usefullness of sharing a generative model (/the output of a generative model) to people that are trying to build discriminative models may be limited by the inherent performance of the generative model. Are there more to it than just calibrating a SOTA discriminative model twice (once on real data, once on generated data) and compare performances ? Wouldn't it be better to just share the SOTA algo calibrated on our dataset ?

Unfortunately I don't think a generative model could prevent from leaking private information from the original dataset.

Like any other kind of model, the generative model is based on the values obtained from the training data. The idea of using such a model in "generation mode" is indeed interesting since it would make it difficult to reverse-engineer the instances it generates back to the real individuals. It would be difficult sure, but not impossible: by re-connecting pieces of information together or exploiting rare (distinctive) cases, somebody could acquire at least partial personal information from the instances.

Additionally the design of the generative model itself introduces a huge bias in the data: the distribution of the instances would be modeled after this design, which might or might not accurately represent the real distribution. This issue significantly lowers the interest in exploiting the generated instances, since it's essentially artificial data.

For the record I think I've seen research about using distributed ML methods in order to overcome the privacy issue. As far as I understand the idea is to keep every individual/institution in control of their data, allowing specific automatized methods to read it in some kind of safe way.


Basis expansion for regression using neural network? I am trying to approximate a nonlinear function using a neural network. There are 3-4 input units. The network is struggling a bit to generalize the function outside the vicinity of the training data set.
I asked someone and he suggested that basis expansion might help. Can someone please provide a reference for the same, I am not able to find any. Also, he suggested "basis expansion using kernel method".

I guess expanding basis is somehow looking at the problem using linear algebra perspective. For explaining it using a trivial example, in linear algebra, for the 3D space, there are three bases namely, $i$, $j$ and $k$. It means for referring to any vector in 3D space you just need a linear combination of these three vectors to construct any desired vectors. Each basis for each space has a number of properties like being orthogonal or not or maybe being linearly independent and some other important properties. By adding extra features, you increase the number of features if they are linearly independent.

In machine learning, there are situations that comparing each feature with the output directly may not lead to a good decision boundary while using some features alongside each other can make decision boundaries which can lead to a better decision boundary. All of that means that if you change the input space to space with more number of features which the features are extracted using the current feature space, you can find a space that the data can be separated better. Finding a feature space such with properties can be done using feature extraction methods. Kernel methods are some of them.

Kernel trick is a kind of mathematical approach for reducing the difficulties of transforming the data in the current feature space to the appropriate feature space. Suppose you have one million rows of raw data and the feature space belongs to $R^{3}$. Suppose you find out that a feature space that can have a good  generalisation belongs to the space which belongs to $R^{95}$. Trying to transform the data in hand to the desired feature space is a time-consuming task if possible. Instead of transforming the data directly, the kernel trick tries to calculate the outcome of the transformed data without transforming the data into the new space.


Hyperparameter Tuning in Random Forest Model I'm new to the machine learning field, and I'm learning ML models by practice, and I'm facing an issue while using the machine learning model.
While I'm implementing the RandomForestClassifier model with hyper tunning it's taking too much time to predict output. And I'm also using GridSearchCV on it. so it's take much time.
Is there any way how can I solve this problem.
OR, Is Google Colab or Kaggle Notebook editor can perform better than Jupiter Notebook ?

You can access the GPU by going to the settings:
Runtime> Change runtime type and select GPU as Hardware accelerator.


How to train-test split and cross validate in Surprise? I wrote the following code below which works:

from surprise.model_selection import cross_validate

cross_validate(algo,dataset,measures=['RMSE', 'MAE'],cv=5, verbose=False, n_jobs=-1)

However when I do this: (notice the trainset is passed here in cross_validate instead of whole dataset)

from surprise.model_selection import train_test_split
trainset, testset = train_test_split(dataset, test_size=test_size)
cross_validate(algo, trainset, measures=['RMSE', 'MAE'],cv=5, verbose=False, n_jobs=-1)

It gives the following error:

AttributeError: 'Trainset' object has no attribute 'raw_ratings'

I looked it up and
Surprise documentation says that Trainset objects are not the same as dataset objects, which makes sense.

However, the documentation does not say how to convert the trainset to dataset.

My question is:
1. Is it possible to convert Surprise Trainset to surprise Dataset?
2. If not, what is the correct way to train-test split the whole dataset and cross-validate?

EDIT: It seems I misunderstood the task at first, so here's my correction. Hope it works this time

It seems like what you're trying to do is similar to what is in the documentation under examples/split_data_for_unbiased_estimation.py (or this github issue which seems to be exactly what you want)

The code manually splits the dataset into two without using any sort of function call. Then sets the internals of the data variable to be only the train split.

import random

from surprise import SVD
from surprise import Dataset
from surprise import accuracy
from surprise import GridSearch

# Load your full dataset.
data = Dataset.load_builtin('ml-100k')
raw_ratings = data.raw_ratings

# shuffle ratings if you want
random.shuffle(raw_ratings)

# 90% trainset, 10% testset
threshold = int(.9 * len(raw_ratings))
trainset_raw_ratings = raw_ratings[:threshold]
test_raw_ratings = raw_ratings[threshold:]

data.raw_ratings = trainset_raw_ratings  # data is now your trainset
data.split(n_folds=3)

# Select your best algo with grid search. Verbosity is buggy, I'll fix it.
print('GRID SEARCH...')
param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005]}
grid_search = GridSearch(SVD, param_grid, measures=['RMSE'], verbose=0)
grid_search.evaluate(data)

algo = grid_search.best_estimator['RMSE']

# retrain on the whole train set
trainset = data.build_full_trainset()
algo.train(trainset)

# now test on the trainset
testset = data.construct_testset(trainset_raw_ratings)
predictions = algo.test(testset)
print('Accuracy on the trainset:')
accuracy.rmse(predictions)

# now test on the testset
testset = data.construct_testset(test_raw_ratings)
predictions = algo.test(testset)
print('Accuracy on the testset:')
accuracy.rmse(predictions)

PS: If you feel like this seems a bit hacky and weird... then the core-developer of Scikit-learn that wrote this code also agrees with that sentiment.


Is recall more important than precision for mass mailings? Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-customers.

For the mailings, we select the top 100,000 scoring (highest predicted probabilities) records. We are expecting a low response rate but we want to make sure we send mail pieces to all of the highest predicted prospects.

We bin the model scores into deciles and only select from decile 1 (highest scoring) which we'll say is >= 0.10 predicted probability. The model has a 95% recall and 5% precision at this threshold.

Does it make sense in this scenario to get the highest recall possible? Or would precision be important here and I should optimize on that or just optimize them both together.

The goal is to get the most customer conversions within that 100k and we already know there will be a good amount of false positives but we want as many true positives as possible.

Neither.  What you really need is for the probability estimate to be accurate.  It's not enough to optimize just one of them (and you can't optimize for two metrics simultaneously, since better precision tends to lead to worse recall, and vice versa).

I suggest you minimize the cross-entropy loss function.  This is designed to help produce accurate estimates of the probability.  In some sense, that will optimize both precision and recall and trade them off in a particular way.  The specifics will depend on what classifier you use.


seaborn heatmap not displaying correctly For some reason, my heatmap is not displaying correctly anymore. It was working just fine even with 6 classes. Since the last time I used it, I've installed many packages (including plotly). I don't know what exactly has caused this. How can I make the annotations and the x/y labels centered again? In both images the exact same code is used.
    import matplotlib.pyplot as plt
    import seaborn
    conf_mat = confusion_matrix(valid_y, y_hat)
    fig, ax = plt.subplots(figsize=(8,6))
    seaborn.heatmap(conf_mat, annot=True, fmt='d',xticklabels=classes, yticklabels=classes)
    plt.ylabel('Actual')
    plt.xlabel('Predicted')
    plt.show()

Current version of matplotlib broke heatmaps. Downgrade the package to 3.1.0
pip install matplotlib==3.1.0
matplotlib/seaborn: first and last row cut in half of heatmap plot


Multi-Source Time Series Data Prediction I was wondering if anyone has experience with time series prediction for data from multiple sources. So for instance, time series $a,b,..,z$ each have their own shape, some may be correlated with others. The ultimate goal is to have a model trained such that the value at time $t+1$ for any given data source can be predicted.

I personally have two solutions that in theory could work, but was wondering if anyone knew of other frequently used methods.

Multi-task learning with LSTM
Use feature engineering to model properties of each time series source as features along with usual features and use these with LSTM

I'm only aware (and using) a RNN which gets multiple time-series in it's first layer and then mixes those in the following layers. Let me know if you have a question on this approach.


Machine Learning with sometimes missing data I'm trying to do an indoor locationing system based on my RSSI signal on my routers, I'm sniffing my network so I know what's the RSSI of my phone related to my routers antennas (I have 5 antennas all over my house).

The iPhone is not always broadcasting probe requests so I only get know in-real time the RSSI signal related to the router the device is connected to and no the other routers. To make it simple to understand: I sometimes have the RSSI of a device related to the 5 routers and sometimes only to 3 and in the worst case escenario only to 1 so my data sometimes looks like this:

room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: -55, rssi4: unknown, rssi5: unknown

or:

room: 1, device_id: 1, rssi1: -80, rssi2: -95, rssi3: -55, rssi4: -102, rssi5: -96

or:

room: 1, device_id: 1, rssi1: -80, rssi2: unknown, rssi3: unknown, rssi4: unknown, rssi5: unknown

It is room based, so I'm not triangulating. I am trying to forecast in which room I am inside the house just by looking on the behaviour of the signals.

Machine Learning would be perfect, but if the data was always there and not some values missing sometimes. What kind of data algorithm should I use for this scenarios when sometimes I have all the data and sometimes just a few.

There are couple of ways to deal with missing data.

Replace missing values by mean/median. If the missing values is very
less, then this method would be apt. Also depends on how skewed
your data is.
Imputation. Build a linear regression model to predict
the missing values based on other parameters. KNN could also be used
to predict the missing value

Also there are other methods to deal with missing data such as mimicking the parameters, removing missing data, etc.


How do standardization and normalization impact the coefficients of linear models? One benefit of creating a linear model is that you can look at the coefficients the model learns and interpret them. For example, you can see which features have the most predictive power and which do not.
How, if at all, does feature interpretability change if we normalize (scale all features to 0-1) all our features vs. standardizing (subtract mean and divide by the standard deviation) them all before fitting the model.
I have read elsewhere that you 'lose feature interpretability if you normalize your features' but could not find an explanation as to why. If that is true, could you please explain?
Here are two screenshots of the coefficients for two multiple linear regression models I built. It uses Gapminder 2008 data and statistics about each country to predict its fertility rate.
In the first, I scaled features using StandardScaler. In the second, I used MinMaxScaler. The Region_ features are categorical and were one-hot encoded and not scaled.
Not only did the coefficients change based on different scaling, but their ordering (of importance?) did too! Why is this the case? What does it mean?

When you have a linear regression (without any scaling, just plain numbers) and you have a model with one explanatory variable $x$ and coefficients $\beta_0=0$ and $\beta_1=1$, then you essentially have a (estimated) function:
$$y = 0 + 1x .$$
This tells you that when $x$ goes up (down) by one unit, $y$ goes up (down) by one unit. In this case it is just a linear function with slope 1.
Now when you scale $x$ (the plain numbers) like:
scale(c(1,2,3,4,5))
           [,1]
[1,] -1.2649111
[2,] -0.6324555
[3,]  0.0000000
[4,]  0.6324555
[5,]  1.2649111

you essentially have different units or a different scale (with mean=0, sd=1).
However, the way OLS works will be the same, it still tells you "if $x$ goes up (down) by one unit, $y$ will change by $\beta_1$ units. So in this case (given a different scale of $x$), $\beta_1$ will be different.
The interpretation here would be "if $x$ changes by one standard deviation...". This is very handy when you have several $x$ with different units. When you standardise all the different units, you make them comparable to some extent. I.e. the $\beta$ coefficients of your regression will be compareable in terms of how strong the variables impact on $y$ is. This is sometimes calles Beta-Coefficients or Standardised Coefficients.
A very similar thing happens when you normalise. In this case you will also change the scale of $x$, so the way how $x$ is measured.
Also see this handout.


How does attention mechanism learn? I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding.

I have this question because I’m tackling a NLP task using attention layer. I believe it should be very easy to learn (the most important part is to learn alignments). However, my neural networks only achieve 50% test set accuracy. And the attention matrix is weird.
I don’t know how to improve my networks.

To give a example:
English: Who are you?
Chinese: 你是誰？

The alignments are
‘Who’ to ‘誰’
‘are’ to ‘是’
‘you’ to ‘你’

How does attention learn that?

Thank you!

From the Amazing Blog - FloydHub Blog- Attention Mechanisms

Attention Mechanisms

Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the words of another sentence form the rows, and then it makes matches, identifying relevant context. This is very useful in machine translation.

When we think about the English word “Attention”, we know that it means directing your focus at something and taking greater notice. The Attention mechanism in Deep Learning is based off this concept of directing your focus, and it pays greater attention to certain factors when processing the data.

In broad terms, Attention is one component of a network’s architecture, and is in charge of managing and quantifying the interdependence:

Between the input and output elements (General Attention)
Within the input elements (Self-Attention)

Let me give you an example of how Attention works in a translation task. Say we have the sentence “How was your day”, which we would like to translate to the French version - “Comment se passe ta journée”. What the Attention component of the network will do for each word in the output sentence is map the important and relevant words from the input sentence and assign higher weights to these words, enhancing the accuracy of the output prediction.

Weights are assigned to input words at each step of the translation

I recommend having a read through this article - Attention Mechanism

More at - Attention Mechanisms and Memory Networks


reinforcement learning: Decompose a policy gradient I am studying the policy gradient through the website: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f

Couldn't figure out how the first equation becomes the second equation?

In the second equation, why the first Expectation value have only s_0, a_0, s_1, a_1 ... s_t, a_t but no reward involved? Also, the second Expectation Value has only r_t+1, s_t+1, ... r_T, s_T, but no action involved? Could anyone please share the thoughts/intuition behind this? Thank you!

The Medium post has brackets in the wrong place . . . the second expectation must be inside the sum to make sense, otherwise $t$ is not defined. You can see a couple of steps later that $Q(s_t, a_t)$ gets magically moved back inside the sum.

I cannot see a way to fix the first expectation though that uses $t$ before it is defined!

The expectations are trying to show which parts of the distribution of trajectories each calculation explicitly depends upon. I don't think you need to read into it anything more than that.

However, I would suggest you find another article which does not have these errors. There are similar derivations of Policy Gradient in Reinforcement Learning: An Introduction


.h5 file format does not close properly import h5py #added
hf = h5py.File('../images.h5', 'w') #added
hf.close() #added

h5_file = tables.open_file("images.h5", mode="w")

I also tried:

h5py.File.close(hf)

the error that pops up in both cases is:

ValueError: The file 'restricted_images.h5' is already opened.  Please close it before reopening in write mode.

I've also tried:

if isinstance(obj, h5py.File):   # Just HDF5 files
    obj.close()

while

In[]: hf
Out[]: <Closed HDF5 file>

, the file is not closed yet.

You may want to use something like the following snippet:

with h5py.File("some_path.h5") as f:
   f["data1"] = some_data

import h5py #added
with h5py.File('../images.h5', 'w') as f:
    h5_file = tables.open_file("images.h5", mode="w")
    # do what ever you want to do, it will be closed by itself.


Automating Logistic Regression I have $3$ datasets I'd like to run a logistic regression on split into

$X_1, y_1$
$X_2, y_2$
$X_3, y_3$

How can I run a loop so that I can run an automated logistic regression using the X_train, X_test split function from SkLearn and in doing so printing 3 separate accuracy results for each dataset?

To run on one data set of X, y is as follows:

X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size=0.25, shuffle=False)

logreg = LogisticRegression()
logreg.fit(X_train,y_train)
y_pred = logreg.predict(X_test)

print('Accuracy:',metrics.accuracy_score(y_test, y_pred))

Having $(X1,y1), (X2,y2), (X2,y2)$  saved in a dictionary:

dictionary = {X1:y1, X2:y2, X3:y3}

accuracies = []
for k,v in dictionary.items():
    X_train, X_test, y_train, y_test = train_test_split(k, v,
    test_size=0.25, shuffle=False)
    logreg = LogisticRegression()
    logreg.fit(X_train,y_train)
    y_pred = logreg.predict(X_test)
    accuracies.append(metrics.accuracy_score(y_test, y_pred))

But it's not really a good way especially when the datasets are too big.

So:

accuracies = []
for i in range(1,4):
    X = 'X{}'.format(i)
    y = 'y{}'.format(i)
    X_train, X_test, y_train, y_test = train_test_split(vars()[X], vars()[y],
    test_size=0.25, shuffle=False)
    logreg = LogisticRegression()
    logreg.fit(X_train,y_train)
    y_pred = logreg.predict(X_test)
    accuracies.append(metrics.accuracy_score(y_test, y_pred))

Note:
in order to access the varibale names which are different in a small part, like here for X1,X2,..., we can use the mentioned method:

import numpy as np
X1= np.arange(1,10)
y1=[i**2 for i in X1]
X2= np.arange(-5,5)
y2=[i**2 for i in X2]
for i in range(1,3):
    X = 'X{}'.format(i)
    y = 'y{}'.format(i)
    print('X_{}'.format(i) , vars()[X])
    print('y_{}'.format(i) , vars()[y])

Output:

X_1 [1 2 3 4 5 6 7 8 9]
y_1 [1, 4, 9, 16, 25, 36, 49, 64, 81]
X_2 [-5 -4 -3 -2 -1  0  1  2  3  4]
y_2 [25, 16, 9, 4, 1, 0, 1, 4, 9, 16]


What does a predicted probability really mean, without considering the accuracy of the underlying model? Say I've built a (completely unrealistic) classification model in Keras that gives me 1.00 accuracy.

And next, I would like to use my model on some new, unseen data, and use model.predict_proba to get a probability that the observation belongs to class "A". Say this returns to me a 0.75.

Am I interpreting this correctly in English:  "100 percent of the time, the model is confident that this new observation is 75 percent likely to be class A" ?

If this is correct, then let's consider if my model was not totally perfect, like in real life, and instead it gave me a 0.40 accuracy. Say my predict_proba is still 0.75.  Then, is this correct:

"40 percent of the time, the model is confident that this new observation is 75 percent likely to be class A." ?

If so...this makes it seem like predict_proba() is not tell a complete story.

I could mislead someone (say a journalist...or a judge, whoever) by saying, "There's a 75 percent chance this unseen observation belongs to class A"...and that might sound great, if I fail to reveal that this statment was based on a model that had a low accuracy like  0.40.

Am I stating this correctly, and does my apprehension have validity?

Accuracy is measured in classification model by comparing the predicted labels to the actual known labels.
The predicted labels are a function of both the predicted probabilities for each class and a predefined threshold(binary classification usually is 0.5)
So if sample A got predict_proba of {0: 0.2, 1: 0.8} it will be labeled as 1(since 0.8 > 0.5).
Accuracy is measure of classification correctness and predict_proba is a direct result of the model underlining function.


Exporting R model to OpenCV's Machine Learning Library I'm wonder if it's possible to export a model trained in R, to OpenCV's Machine Learning (ML) library format? The latter appears to save/read models in XML/YAML, whereas the former might be exportable via PMML. Specifically, I'm working with Random Forests, which are classifiers available both in R and OpenCV's ML library.

Any advice on how I can get the two to share models would be greatly appreciated.

Instead of exporting your models, consider creating an R-based interoperable environment for your modeling needs. Such environment would consists of R environment proper as well as integration layers for your third-party libraries. In particular, for the OpenCV project, consider either using r-opencv open source project (https://code.google.com/p/r-opencv), or integration via OpenCV C++ APIs and R Rcpp package (http://dirk.eddelbuettel.com/code/rcpp.html). Finally, if you want to add PMML support to the mix and create a deployable-to-cloud solution, take a look at the following excellent blog post with relevant examples: http://things-about-r.tumblr.com/post/37861967022/predictive-modeling-using-r-and-the.


how to read emotion label of MSP-IMPROV dataset samples? I need to read MSP-IMPROV dataset audio file's to conduct research. I read the help file that says 15th character of the file name, shows the emotion label of each utterance.
For example:
MSP-IMPROV-S01A-M01-P-FM01  intended Angry emotion.
But when I run the code that follows this rule, I can't reach the other emotions that are labeled O as others and WA as without agreement.
Moreover, I must extract

Anger     (792)
Happiness (2644)
Sadness   (885)
Neutral   (3477)

But I find out the number of each emotion is

Anger     (2511)
Happiness (2267)
Sadness   (2035)
Neutral   (1625)

Can anyone help me to find out where I made a mistake?
Thanks in advance.
*for more information about this database, you can find it in this link and this page

I asked this question from Prof. Busso as the leader of the MSP-Improv dataset team. He answered:
"
The intended emotion is included in the file name. The emotion annotated by the workers is provided in the file Evalution.txt. You should not use the intended emotion as the actual emotion.
Example:
MSP-IMPROV-S01A-M01-P-FM01
intended emotion was anger, but if you see the evaluation you will see the following information:
UTD-IMPROV-S01A-M01-P-FM01.avi; N; A:3.666667; V:3.166667; D:2.833333 ; N:4.166667;
51o11nal985p2oq49qmnnqivm2-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:4.000000; N:4.000000;
6lrk33e18tetg934je64ale7r6-p3b; Neutral; Neutral; A:4.000000; V:3.000000; D:3.000000; N:4.000000;
cv970rs8rn8v23p3f3tec5bgb4-p3b; Happy; Happy,Neutral; A:5.000000; V:4.000000; D:3.000000; N:5.000000;
n76ifqflrdaao10o0bg8js05o3-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:2.000000; N:4.000000;
puumgduqdt8hfiffuvts8m3241-p3b; Neutral; Neutral; A:4.000000; V:3.000000; D:2.000000; N:4.000000;
qlvg8teulu3c7mcgvds07eqte1-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:3.000000; N:4.000000;
This file was annotated as neutral.
"
so I write a code for this purpose that reads the file name and finds it in the Evaluation file and read real annotated emotion.
this is the main function (please put MSP database  in "DBs" folder):
 function [DB, samplesN, fs]= mspDBread(sesNum)
%%
emoEvl  = 'DBs/MSPEvalution.txt';

rootdir     = 'DBs/MSP-IMPROV';
sesList    = dir(rootdir);
sesList    = sesList(3:end);
samplesN =1;
for iSes= sesNum
session = sesList(iSes).name;
sentenceRoot = [rootdir,'/',session];
sentences = dir(sentenceRoot);
sentences = sentences(3:end);

for iSentence = 1:numel(sentences)
    scenarioRoot=[sentenceRoot,'/',sentences(iSentence).name];
    SentName = sentences(iSentence).name;
    scenario = dir(scenarioRoot);
    scenario = scenario(3:end);
    for iScen = 1:numel(scenario)
        fileRoot = [scenarioRoot,'/',scenario(iScen).name];
        fileList = dir(fileRoot);
        fileList = fileList(3:end);
        for iFile = 1:numel(fileList)

            emoList={};
            i=1;
            textFileID = fopen(emoEvl);
            tline = fgetl(textFileID);

            while ischar(tline)

                tline = fgetl(textFileID);
                if numel(tline)~=0
                    if strcmp(tline(1),'U')
                        words = strsplit(tline);
                        destLineHeader = words(1);
                        destLineHeader = destLineHeader{1,1};
                        destLineHeader = destLineHeader(5:end-5);
                        waveFileName = fileList(iFile).name;
                        if strcmp(destLineHeader,waveFileName(5:end-4))

                            emotion = words(2);
                            emotion = emotion{1,1};
                            emotion = emotion(1,1);

                            compRoot= [fileRoot,'/',waveFileName];

                            [x, fs] = audioread(compRoot);

                            DB{samplesN,1} = samplesN;                          % sample Number
                            DB{samplesN,2} = str2double(session(end));          % session number
                            DB{samplesN,3} = str2double(waveFileName(13:14));   % scentence number
                            DB{samplesN,4} = scenarioMap(waveFileName(21));     % scenario type
                            DB{samplesN,5} = labelMap(emotion);                 % emoton label
                            DB{samplesN,6} = genderMAP(waveFileName(17));       % speaker gender
                            DB{samplesN,7} = str2double( waveFileName(18:19));  % speaker ID
                            DB{samplesN,8} = genderMAP( waveFileName(23));      % listener gender   1:male 0:female
                            DB{samplesN,9} = genderMAP( waveFileName(24));      % seaker gender     1:male 0:female
                            DB{samplesN,10} = str2double(waveFileName(25:26));  % turn number
                            DB{samplesN,11} = waveFileName;                     % complete name  of voice
                            DB{samplesN,12} = x;                                % data

                            disp ([num2str(samplesN), '   session : ',session(end), '     label:',waveFileName(15),'  detect: ',emotion,'   EmoIndex: ',num2str(labelMap(emotion)),'    ',  waveFileName])
                            samplesN =samplesN+1;

                        end
                    end
                end
            end
            fclose('all') ;
        end
    end
  end
end

other functions that are used in the above function is listed below:
1- scenario mapping function:
function [code]=scenarioMap(character)

if character=='T'
    code = 0;
elseif character=='S'
    code = 1;
elseif character=='P'
    code = 2;
elseif character=='R'
    code = 3;
else
    code = 4;
end

2- label mapping function:
function [code]= labelMap(character)

if character=='A'
    code = 0;
elseif character=='H'
    code = 1;
elseif character=='S'
    code = 2;
elseif character=='N'
    code = 3;
elseif character=='O'
    code = 4;
    disp('--------zzzzzzz-----')
elseif character=='W'
    code = 4;
    disp('--------zzzzzzz-----')
else
    code = 4;
end

3-Gender mapping function:
function [gender ]= genderMAP(str)

if strcmp(str,'F')
    gender = 0;
elseif strcmp(str,'M')
    gender = 1;
end