Handling features with multiple values per instance in Python for Machine Learning model I have a dataset which contains medical data about children and I am developing a predictive machine learning model to predict adverse pregnancy outcomes. The dataset contains mostly features with a single value per child, e.g. gender = ["Male", "Female].
However, I have some features that have multiple values per child, such as the abdominal circumference which has been recorded multiple times per child, as such:
    ChildID     abdomcirc
0   1           273
1   1           267
2   1           294
3   2           136
4   2           248

So in the above table child 1 has 3 values for abdomcirc and child 2 has two values for abdomcirc. Adding this feature to the remaining dataset (comprised of single observational features) will result in nearly duplicate rows, apart from the different values for abdomcirc, like so:
    ChildID     gender  diabetes  birthroute  abdomcirc
0   1           Male    No        Normal      273
1   1           Male    No        Normal      267
2   1           Male    No        Normal      294
3   2           Female  Yes       csection    136
4   2           Female  Yes       csection    248

I am unsure what the best way to deal with these features is, without merging the data and having near-duplicate rows. I have considered the following:

Using python list type for abdomcirc. However, I do not know if a machine learning model can handle this data type. So my data will look something like this
      ChildID     gender  diabetes  birthroute  abdomcirc
 0    1           Male    No        Normal      [273, 267, 294]
 1    2           Female  Yes       csection    [136, 248]

Transforming abdomcirc into a single observational feature by calculating the mean (although I am not sure how useful this information would be for my predictive model) like so:
      ChildID     gender  diabetes  birthroute  abdomcirc
 0    1           Male    No        Normal      278
 1    2           Female  Yes       csection    192

I have tried looking for resources to help me with this but have not been very successful, maybe because I am not typing the correct keywords or something. So, I would appreciate your opinions and helpful resources. Many thanks!

A possible resource is featuretools, they do feature engineering on data that has many records. Their examples are not from medical cases but I think it should work for you too.
You can also manually build several features. For instance, given a list of abdomcirc, you can compute its:

mean
maximum
minimum
variance
difference from minimum to maximum
last value (if they are sorted by date)
number of unique values

These features would get most of the information of the abdomcirc list, and this should help your modelling.
I wouldn't go for the first approach of giving lists to the algorithm, although it is possible, I think it is a relatively advanced thing and I wouldn't go for it unless the simpler approaches don't work.


Why is the learning rate for the bias usually twice as large as the the LR for the weights? I've noticed in a few caffe models I've been working with that the learning rate for the bias is often set to be twice that of the one for the weights.
Another user mentions that this is the case in this thread but does not elaborate on what the reason for doing so is.
Is there some justification for having different learning rates for the bias as the weights? Did I miss some paper providing a theoretical or experimental justification of this?

I can't find any literature discussing setting a fixed learning rate differently for the bias specifically than for the other weights, but if you asked the researcher responsible for that model their answer would probably be something like:

  "The network wasn't training properly and I noticed the error for the bias term wasn't converging as fast as I'd like. When I tried doubling the learning rate for just that term it seemed to fix the problem. I dunno man, it just worked."

As far as I can tell, training neural networks is still more art than science and the amount of skill, patience, and care taken in training a neural network (depending on the architecture) can have as much impact on the model's performance as the network topology. Consider issues like mode collapse in GANs, vanishing gradient with ReLUs, or just getting stuck in local optima. Neural networks often need to be "babysat" during training, and consequently trial and error may contribute significantly to certain training process decisions. I'm sorry if that's not a satisfactory answer, but I have a strong suspicion that's what's going on here. If you post some specific examples, it might be easier to figure out concretely what's going on, but otherwise my guess is that this was just the result of trial and error.

Regarding using parameter-specific learning rates generally, most modern optimizers do this, including AdaGrad, RMSProp and Adam.


Correlation between nominal categorical variables I have two arrays, whose values are nominal categorical variables. Each element represents a zone of a city: in the first vector we have the class each zone belongs to (so these might also be seen as ordinal, since values span from 0 to 3, with 3 being the upper class -let's say richest- and 0 the poorest, but I am not sure about this). The second vector is made of names: each item is the name of the candidate who won the Presidential elections in that particular zone.

I would like to calculate the correlation between the two vectors, to find whether there is some kind of relationship between the class of the zone and the winning candidate (i.e. candidate X systematically won in the poorest zones), but I am not sure on how to calculate correlation between nominal variables. I found this question somewhat helpful, but the example provided in the answer does not match with my case.

The Chi-Squared test of independence (and subsequent Cramer's V test) give an indication of the relationship between two categorical variables.


Detecting Spam using Machine Learning The most online tutorials like to use a simple example to introduce to machine learning by classify unknown text in spam or not spam. They say that this is a binary-class problem. But why is this a binary-class problem? I think it is a one-class problem! I do only need positive samples of my inbox to learn what is not spam. If I do take a bunch of not spam textes as positiv samples and a bunch of spam-mails as negativ samples, then of course it's possible to train a binary-classifier and make predictions from unlabeled data, but where is the difference to the onc-class-approach? There I would just define a training-set of all non spam examples and train some one-class classifier. What do you think?

Strictly speaking, "one class classification" does not make sense as an idea. If there is only one possible state of a predicted value, then there is no prediction problem. The answer is always the single class.

Concretely, if you only have spam examples, you would always achieve 100% accuracy by classifying all email as spam. This is clearly wrong, and the only way to know how it is wrong is to know where the classification is wrong -- where emails are not in the spam class.

So-called one-class classification techniques are really anomaly detection approaches. They have an implicit assumption that things unlike the examples are not part of the single class, but, this is just an assumption about data being probably not within the class. There's a binary classification problem lurking in there.

What is wrong with a binary classifier?


Which classifier is efficient in dealing with test query which belongs to no trained class? Suppose classifier trained with 5 class, and input query content does not belong to any of the trained class data.

Naive bayes provides and random class as a result here. Which classifier deals best in such scenario?

You have trained a model to recognize or discriminate between several particular classes. So when having a new test sample (that according to your knowledge) does not belong to any of this class, the model will fit it to the most similar class.

This is, off course, a very general way of putting it. The behavior and things to consider wether you are using a probabilistic graphical model (like Naive Bayes), linear or non linear classifiers, etc, will vary. However, the principle is the same: the model has learnt the relation among the features of the training data you have used to match particular classes.

More specifically, if you are using a probabilistic approach you can  use the probability of belonging to a class and define a threshold. So if a probability higher than certain value (let's say 65%) is not given, then your confidence on the result will be low, and you might say "I cannot say wether it belongs to any of this 5 classes with certainty".

Other non probabilistic methods have interesting approaches for scoring a new example in terms of probabilities. Check this link for SVM (https://stats.stackexchange.com/questions/55072/svm-confidence-according-to-distance-from-hyperline), which is actually the approach used in Python scikit-learn: http://scikit-learn.org/stable/modules/svm.html#scores-and-probabilities

If you use Random Forests you could use the voting of each tree to define a rate of confidence in a similar way (https://stats.stackexchange.com/questions/94845/how-to-estimate-confidence-level-for-svm-or-random-forest)

A similar thing can be done with K-Nearest-Neighbors by analyzing the distance of the K or J (where J < K) nearest neighbors and just trust the result if they are close to your test sample in some ratio (very generally said, this can be done in many ways).

Now, methods based on deep learning are trying to learn features in an unsupervised way, so your problem could solved in a more interesting way. However, it would not be strict classification anymore, nor you would have the amount of data and servers required to even try it (but just saying :) )


BERT Masked Language Model question I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word from the context given. The loss function will be the lowest for output embeddings which are closest to the original masked word embedding. Wouldn't it be that using this loss funtion does not guarantee that BERT will output word embeddings with context and instead could just output an embedding closest to the original masked word embedding without the relevant context?
Thanks.

In this context, masking means replacing the token with a special [MASK] token. The network does not have the information of what the original token was, the only way how it could potentially figure out what it was is by looking at the context.
It is not the loss function that guarantees that the model learns something meaningful, it is architecture design that does not allow the workaround that you suggest.


How is Stochastic Gradient Descent used like Mini Batch gradient descent? As I know, Gradient Descent has three variants which are:
1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent.
2- Stochastic Gradient Descent: processes one training example per iteration. Hence, the parameters are being updated even after one iteration in which only a single example has been processed.
3- Mini Batch gradient descent: which works faster than both batch gradient descent and stochastic gradient descent. Here, b examples where b < m are processed per iteration.
But in some cases they use Stochastic Gradient Descent and they define a batch size for training which is what I am confused about.
Also, what about Adam, AdaDelta & AdaGrad, are they all mini-batch gradient descent or not?

But in some cases they use Stochastic Gradient Descent and they define
  a batch size for training, I am confused about that?

If you defining a batch size, then you are performing mini-batch gradient descent.

Stochastic Gradient Descent ( SGD ) and Mini Batch Gradient Descent are often used interchangeably in many places. The idea is that if the batch size is 1, then its totally SGD.

If the batch size is equal to the number of samples in the dataset, then its Batch Gradient Descent.

  Also, what about Adam, AdaDelta, AdaGrad, are they all mini-batch
  gradient descent? or not?

Adam, Adadelta and AdaGrad are all different in which we update the parameters.  The main concept of GD is the calculation of gradient of the loss function which is fundamental to all the above optimizers.

The difference is where on how much samples from a dataset are the gradients of the loss function calculated.
After the gradients are calculated, the different optimizers mentioned use modified methods to update the parameters using the calculated gradients.

Finally,

  SGD, Mini batch/Batch GD are methods determining on how much samples are gradients calculated.
  AdaGrad, Adadelta etc. update the parameters differently based on
  calculated gradients.


Why model.fit_generator in keras is taking so much time even before picking the data? In Keras, I am training a model as in below

model.fit_generator(data_generator(),
                samples_per_epoch = count,
                validation_data=(x_val, y_val),
                nb_epoch=50,
                callbacks=getCallBacks(),
                verbose=1)

In data_generator function, I am printing few debugging statements.

On running above fit_generator function, it takes really long time to print what I had put in the data_generator().

Is there any series of steps which keras takes before picking out the data batch for the training which makes the process so slow or there is some other caveat?

I have noticed this when training on one or more GPUs. I think it is due to Tensorflow having to acquire the resources (in the background it blocks the entire GPU memory). Perhaps some amount of data e.g. the first batch is copied over to the GPU.

I didn't find a way to reduce the waiting time dramatically, but you could try the tensorflow options to allocate less memory and allow memory growth as required. Check out the official docs and this issue for the Keras version.

Here is the latest recommendation form that issue: setting Tensorflow options befor eimporting Keras.

-------------------------- set gpu using tf ---------------------------
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
-------------------  start importing keras module ---------------------
import keras.backend.tensorflow_backend as K
import keras......


Need help with python code as part of a data analysis project I am new to python programming. As part of a data analysis project, I am trying to get a scatter plot of Salary vs Wins for each year of 4 consecutive years (so I am trying to get 4 different scatter plots, one for each year). I am using the following code:

teamName = 'OAK'
years = np.arange(2000,2004)
for yr in years:
    df = joined[joined['yearID']==yr]
    plt.scatter(df['salary']/1e6,df['W'])
    plt.title('Wins vs Salaries in year' + str(yr))
    plt.xlabel('Total Salary (in millions)')
    plt.ylabel('Wins')
    plt.xlim(0,180)
    plt.ylim(30,130)
    plt.grid()
    plt.show()

However, I am only getting one plot corresponding to 2003.

Can anyone point out the mistake ?

Thanks

All of your plots are appearing on top of each other.  You need to invoke plt.subplot(xxx) before you create each plot.  For info on how the xxx command works, go to the MATLAB documentation.

You might end up with multiple figures - see this page for info about that.


Test data from ImageNet Here's the description about the data usage for ILSVRC 2016 of ImageNet. I've interpreted it with the table as follows,

Data          Image        Label    Changed    Release_Time
-----------------------------------------------------------------
training      1,200,000    1,000    no         always public
validation       50,000    1,000    some       upon registration
test            100,000        0    some       final evaluation

My question is specific about the test data. Is it chosen from the same set of 150,000 images since ILSVRC 2012, although randomly ? If so, most of the test data has been made public before the final evaluation phase of the challenge, how the result could remain fair enough ?

It has the same images since ILSVRC 2012. The test set images have never been released.


How to pass linear regression weights to Xgboost regressor? I'm trying to build an xgboost regressor or a catboost regressor for a task. I have a working linear regression model. I also trained an xgboost regressor model for the task but it was worse than the linear regression model. I am wondering if there is a way to pass the linear  regression weights (model parameters) as an initial set of parameters to the xgboost (or catboost) model to ensure performance gain?

E.g. if $w0*x0 + w1*x1 + w2*x2 + w3*x3 = y$ is the linear regression model, is there any way to tell xgboost to start by the same equation (and get better while training)?

Answer is NO
Why? weights are hyperparameters of the linear regression model and they are not the same as the ones for xgboost or catboost.
What you can do is combine models (if you really want to use xgboost or catboost additionally) SEE this
NOTE just because model is more powerfull it does not mean that it will beat linear regression on every dataset ;)


Predict the salaries of customers in 2019 from 2016 data I have a spreadsheet of data with ages and salaries for 2016. For example, in 2016 there were 12988 customers who were 18 years old and earned 20k.
I need to predict the salaries of customers for 2019. All I have are their ages but not their actually salary.

Since I do not have a lot of data (just 2016), I was thinking of simply making an estimate based on percentages. For instance, if 1% of all customers earned 20k, then I assume that 1% of the current customers also earn 20k
Should that be enough or is there a better way?

Welcome to the site! First off, I will start by saying that you might be better off asking this in a statistics SE because this isn't really data science. But I will still try to help you.

I will assume that age and salary are the only items you have and that salary is the only thing you are trying to predict. If those assumptions are correct then there really is no algorithm, you can just do some distribution "tricks" to get you there.

You're on the right path of thinking about things on a proportional basis and I would proceed accordingly. However, I wouldn't think of it in whole numbers (like your 20k example), I would think about using some scaling. So, take the max value of all your salaries and create scales with something like salary / max salary. From there, you should should know how many people will have X salary (in a scaled, decimal format).

Now, you need to decide how much salary inflation (or deflation) has happened in the 3 years since your data was available. Take that percentage and assign it back to your scaled values and now you can have a pretty good idea of how much each person makes in today's dollars (otherwise, you'd end up with salaries in 2016 dollars).

Good luck!


Using predictive modelling for temperature data set I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on size of data?

To be specific to the title, I have to predict missing values in a given temperature data set and i am unaware of anything that i can use. Could someone guide me through?

I am not able to understand how and what modelling techniques do we
  use?

Every data science workflow has the follwing steps:

Pre-processing (data cleaning and wrangling)
Exploratory analytics
Model selection
Prediction and testing.  (And re-iteration)
(Optional) Reporting the workflow

  Does it depend on the data type?

Yes, the entire workflow is dependent on the type and features of the data.

  Does it depend on size of data?

Size of data makes a difference in the tools and sometimes(very rarely) the algorithms used.

  I have to predict missing values in a given temperature data set and i
  am unaware of anything that i can use

There is a lot of material and algorithms on how to impute missing data, which you can refer to and use them accordingly depending on the type of data and the problem statement.


How are deep-learning NNs different now (2016) from the ones I studied just 4 years ago (2012)? It is said in Wikipedia and deeplearning4j that Deep-learning NN (DLNN) are NN that have >1 hidden layer.

These kind of NN were standard at university for me, while DLNN are very hyped right now. Been there, done that - what's the big deal?

I heard also that stacked NN are considered deep-learning. How is deep-learning really defined?

My background of NN is mostly from university, not from jobs:

studied applications of NN in industry
had about 5 courses on artif. intel. & mach. learn. - though maybe 2 of them on NN
used NN for small, simple project on image recognition - used 3 layer feed-forward NN
did not do real research (as in doctor thesis) on them

You are right in that the basic concept of a deep NN hasn't changed since 2012.  But there have been a variety of improvements to the ways in which deep NNs are trained that have made them qualitatively more powerful.  There are also a wider variety of architectures available today.  I've listed some developments since 2012, grouped by training improvements and architecture improvements:

Improvements to training deep NNs

Hardware: The most obvious change is just the inexorable progression of Moore's law.  There is more computing power available today.  Cloud computing also makes it easy for people to train large NNs without needing to buy a huge rig.
Software: The open source software for deep learning is really enormously improved from 2012.  Back in 2012 there was Theano, maybe Caffe as well.  I'm sure there are some others, too.  But today we also have TensorFlow, Torch, Paddle, and CNTK, all of which are supported by large tech companies.  This is closely related to the hardware bullet point since many of these platforms make it easy to train on GPUs, which drastically speeds up training time.
Activation functions: The use of ReLU activation functions is probably more widespread these days, which makes training very deep networks easier.  On the research side, there is a wider variety of activation functions being studied, including leaky ReLU, parametric ReLU, and maxout units.
Optimization algorithms: There are more optimization algorithms around today.  Adagrad and Adadelta just been introduced in 2011 and 2012, respectively.  But we now also have the Adam optimizer and it's become a very popular choice.
Dropout: In the past few years, dropout has become a standard tool for regularization when training neural networks.  Dropout is a computationally inexpensive form of ensembling for NNs.  In general, a set of models trained on random samples of the dataset will outperform a single model trained on the entire dataset.  This is difficult to do explicitly for NNs because they are so expensive to train.  But a similar effect can be approximated just by randomly "turning off" neurons on each step.  Different subgraphs in the NN end up getting trained on different data sets, and thereby learn different things.  Like ensembling, this tends to make the overall NN more robust to overfitting.  Dropout is a simple technique that seems to improve performance in almost every case, so it's now used de rigueur.
Batch normalization: It's been known for a while that NNs train best on data that is normalized --- i.e., there is zero mean and unit variance.  In a very deep network, as the data passes through each layer, the inputs will be transformed and will generally drift to a distribution that lacks this nice, normalized property.  This makes learning in these deeper layers more difficult because, from its perspective, its inputs do not have zero mean and unit variance.  The mean could be very large and the variance could be very small.  Batch normalization addresses this by transforming the inputs to a layer to have zero mean and unit variance.  This seems to be enormously effective in training very deep NNs.
Theory: Up until very recently, it was thought that the reason deep NNs are hard to train is that the optimization algorithms get stuck in local minima and have trouble getting out and finding global minima.  In the last four years there have been a number of studies that seem to indicate that this intuition was wrong (e.g., Goodfellow et al. 2014).  In the very high dimensional parameter space of a deep NN, local minima tend not to be that much worse than global minima.  The problem is actually that when training, the NN can find itself on a long, wide plateau.  Furthermore, these plateaus can end abruptly in a steep cliff.  If the NN takes small steps, it takes a very long time to learn.  But if the steps are too large, it meets a huge gradient when it runs into the cliff, which undoes all the earlier work.  (This can be avoided with gradient clipping, another post-2012 innovation.)

New architectures

Residual networks: Researchers have been able to train incredibly deep networks (more than 1000 layers!) using residual networks.  The idea here is that each layer receives not only the output from the previous layer, but also the original input as well.  If trained properly, this encourages each layer to learn something different from the previous layers, so that each additional layer adds information.
Wide and deep networks: Wide, shallow networks have a tendency to simply memorize the mapping between their inputs and their outputs.  Deep networks generalize much better.  Usually you want good generalization, but there are some situations, like recommendation systems, in which simple memorization without generalization is important, too.  In these cases you want to provide good, substantive solutions when a user makes a general query, but very precise solutions when the user makes a very specific query.  Wide and deep networks are able to fulfill this task nicely.
Neural turing machine: A shortcoming of traditional recurrent NNs (whether they be the standard RNN or something more sophisticated like an LSTM) is that their memory is somewhat "intuitive".  They manage to remember past inputs by saving the hidden layer activations they produce into the future.  However, sometimes it makes more sense to explicitly store some data.  (This might be the difference between writing a phone number down on a piece of paper vs. remembering that the number had around 7 digits and there were a couple of 3s in there and maybe a dash somewhere in the middle.)  The neural Turing machine is a way to try to address this issue.  The idea is that the network can learn to explicitly commit certain facts to a memory bank.  This is not straightforward to do because backprop algorithms require differentiable functions, but committing a datum to a memory address is an inherently discrete operation.  Consequently, neural Turing machines get around this by committing a little bit of data to a distribution of different memory addresses.  These architectures don't seem to work super well yet, but the idea is very important.  Some variant of these will probably become widespread in the future.
Generative adversarial networks: GANs are a very exciting idea that seems to be seeing a lot of practical use already.  The idea here is to train two NNs simultaneously: one that tries to generate samples from the underlying probability distribution (a generator), and one that tries to distinguish between real data points and the fake data points generated by the generator (a discriminator).  So, for example, if your dataset is a collection of pictures of bedrooms, the generator will try to make its own pictures of bedrooms, and the discriminator will try to figure out if it's looking at real pictures of bedrooms or fake pictures of bedrooms.  In the end, you have two very useful NNs: one that is really good at classifying images as being bedrooms or not bedrooms, and one that is really good at generating realistic images of bedrooms.


Optimization of pandas row iteration and summation i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result.

What i am trying to obtain is a summation of IDs in one table (player table) based on each row of a second table (UUID). Functionally each row needs to sum the total of the players table rows that are contained in its Active row and assign the UUID as the index to that row.

My initial thought was to loop row by row and calculate out my results but that has produced quite a slow result, and i suspect is not the optimal way that this could be accomplished. In the version below my estimate total time for the full dataset would be around 66 minutes. Running on a subsample of 10,000 takes around 20 seconds.

Would anyone have a better solution to calculating these results?

Thanks in advance!

UUID Table

This is a subset of the whole table

shape = (2060590, 2)

Player ID Table

This is a subset of the whole table

shape = (39,8)

Final Table

Code

# executes in ~20 seconds
df = None
for ix, i in enumerate(uuid_df[["UUID", "Active"]].sample(10000).itertuples(index=False)):
    # Get UUID for row
    _uuid = i[0]
    # Get list of "Active items" (these are the ones that will be summed)
    _active = i[1]

    # Create new frame by selecting everything from points table where the ID is in the Active List.
    # Sum selected values, convert to a dataframe with UUID as index and tranpose
    _dff = points_table_df.loc[points_table_df.index.isin(_active)].sum().to_frame(_uuid).T

    # Check if first dataframe, if not concat to existing one
    if df is None:
        df = _dff
    else:
        df = pd.concat([df, _dff])

This could actually be done quickly and intuitively using linear algebra.

So consider your player as label binarized array (can be done with MultiLabelBinarizer) so you would expect an array of size (2060590, 39) containing 0 an 1, rearrange the columns similar as how you order the your player table (or the other-way around which ever is easier), basically such that first column of your new matrix correspond to the same player on the player table. Finally just apply matrix multiplication, and done.

This is an example using generated sample, but hopefully you get the idea of doing this.

import numpy as np
import pandas as pd
from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()

sample_active = pd.Series([[100,50,150,200],
                           [100,50,150],
                           [100,50],
                           [100]])
sample_df = pd.DataFrame()

sample_df['id'] = ['fadfsadsa', 'dsafsadf', 'dfsafsda', 'dasfasdfsaf']
sample_df['active'] = sample_active
## sample_df should look close to your original df

classes = [50,100,150,200]

player_df = pd.DataFrame({cl : np.random.uniform(0,1,size=5) for cl in classes}).T
player_df.columns = ['A','B','C','D','E']

sample_transformed = mlb.fit_transform(sample_active.values) ##apply multilabel binarizer

output = sample_transformed.dot(player_df.loc[mlb.classes_]) ##matrix multiply and get your required answer, use loc so the order will be similar as your binarized matrix.

new_df = pd.concat([sample_df['id'], pd.DataFrame(output)], axis = 1)
new_df.columns = ['id'] + list(player_df.columns)

For your case I think this should work :

mlb = MultiLabelBinarizer()
active_transformed = mlb.fit_transform(uuid_df['Active'])
output = active_transformed.dot(points_table_df.loc[mlb.classes_])
df = pd.concat([uuid_df[['UUID']], output], axis = 1)
df.columns = ['UUID'] + list(points_table_df.columns)

Try it!


How to deal with memory exceptions while processing large amount of data? Am trying to develop a python script which reads a large CSV file (approx 1.2 GB historical data) in chunks and performs following steps:

Take backup of the file
Extract new records of previous day transactions,append to original/base CSV file and store the data in dataframe.
Perform mathematical operations on the big dataframe
Convert big dataframe to CSV and store in same location for nex day processing.

Similar process runs next day as well and so on...

I'm getting memory exception error while processing the step 3 above (probably step 1 & 2 would have consumed most of the memory max-3GB limited space). Even if I extend the space in the server, I foresee a problem as my input file size is going to be increase daily.

I need to dip through historical data daily for mathematical operations along with daily transactions so can't avoid storing/accessing of base CSV file which contains historical data and can't afford to buy space in cloud as well.

I have used pd.read_csv for reading records from CSV file in chunks, for mathematical operations pandas and numpy has been used.

The script runs without any issue on my local machine, however I have the memory issue while processing on the server. So, as far as the code goes, it seems to be good.

I believe that if I can change the processing in steps 1 to 4 to be more efficient, the memory exception can be solved and with limited space my script can be executed.

Can someone suggest the best way to handle the above steps?

I had also faced similar issues when my python script was processing my 12GB and 35GB files. It used take weeks for processing and many times failed.

You need to check your mathematical operation, if it can work on subset of data. In my case, I finally was able to split the file in multiple pieces and run the mathematical operation on individual pieces. It helped improving the speed as well as better exceptional handling.

If you cant split your file, then check if your code can work in batches. Then you can read your file in chunks (in pd.read_csv you can supply chunk size) and then feed to your formula piece by piece.


Tensorflow 2.0 - Layer with fixed input I'm trying to use Tensorflow to optimize a few variables to be used in a KNN algorithm, however, I'm running into an issue where I'm unable to have a layer work properly if it is not connected to an Input layer.

What I'm trying to do below is pass in [[1]] as a static tensor, and then using the Lambda layer to coerce the weights into a usable shape.  Once trained, I would just retrieve the weight value from the first Dense layer.

ones = tf.ones(shape=(1,1)) # trying to use this to take the place of dynamic inputs

theta_layer = layers.Dense(1, activation="linear", use_bias=False, trainable=True)(ones)
theta_layer = layers.Lambda(lambda x: tf.ones(shape=(self.batch_size,1)) * x)(theta_layer)
print(theta_layer)
# tf.Tensor(
# [[-1.151663]
# [-1.151663]
# [-1.151663]], shape=(3, 1), dtype=float32)

concat_layer = layers.Concatenate()([theta_layer, features_input])

model = models.Model(inputs=features_input, outputs=concat_layer)
model.compile(optimizer="adam", loss="mse") #, metrics=["mae"])#, bias_regularizer=None)

The problem with the above strategy is that theta_layer does not show up in model.summary() and the weights don't show up in model.get_weights().

Also, when I try to directly set theta_layer as the model output, I get this message:

AttributeError: Tensor.op is meaningless when eager execution is enabled.

This is probably the giveaway but frankly I'm just not knowledgeable enough about Tensorflow.

I know that I can fix this by passing in a separate tf.ones input to the model (not just into specific layers), but it would complicate the code seemingly unnecessarily (I have to do this for multiple layers/variables).  Is there a way that I can alter the tf.ones passed to the layer or the alter the layer itself to eliminate this issue?

This feels like a bit of a hack, but I was able to infer something from an answer on another question: https://stackoverflow.com/a/46466275/6182971

It works if I change the first line above from:

ones = tf.ones(shape=(1,1))

to:

ones = layers.Lambda(lambda x: tf.ones(shape=(1,1)))(features_input)

Even though the Lambda layer is returning a constant, passing in features_input, which is the main training data connects the tf.ones constant to the network inputs, which seems to be sufficient.


Does switching the positive and negative case make a difference? I believe that in most cases, the positive case is arbitrarily defined to be the case with a lower frequency. If I instead define the negative case as the one with the lower frequency, what differences would this entail for the model?

For all of the algorithms/procedures that I know, it makes no difference between a positive or negative interpretation of a class.

The fact that one category has a lower frequency (by that I am assuming that you mean a low probability) is insignificant. It is just a matter of communication, by maintaining the general designations in a specific field so that others will be able to read your work more easily.

As an example, in anomaly detection, one designates the cases with lower frequency as anomaly, but even so, technically it does not matter how you label your cases, they are interchangeable.


GPU Accelerated Data Processing for R in Windows I'm currently taking a paper on Big Data which has us utilising R heavily for data analysis. I happen to have a GTX1070 in my pc for gaming reasons. Thus, I thought it would be really cool if I could use that to speed up some of the processing for some of the stuff my lecturers have me doing, but it really doesn't seem easy to do this at all. I've installed gpuR, CUDA, Rtools, and a few other bits and bobs, and I can get it to create gpuMatrix objects from genomic expression data, for example, but I have yet to find a function which both works with the gpuMatrix objects and also provides any noticeable difference in performance. Perhaps this just relates to limitations inherent to the gpuR package - some other packages do seem to talk about having functions which sound like they would be more like the sort of thing I'm looking for, which brings me to the question:

Almost all of those packages are exclusively for Linux, is it particularly hard to implement GPU support for R in windows? Or is there some other reason that there are so few packages available to do this in Windows? In some sense I'm just curious, but it would also be very cool to really get it working. It surprises me that there is so little available for Windows, usually it's the other way around.

From my experience setting up GPU processing for R is hard, setting it up on a Windows machine is even harder. Additionally, GPU processing can only be used for very specific types of calculations.

If you just want to setup GPU processing for the sake of it, then my answer is quite useless.

If you however care about general performance optimization of your system and your code, I advise to check the following steps:

Use Microsoft R Open instead of Base R because it automatically enables multicore processing on your machine.
Vectorize your code
Use libraries such as data.table instead of dataframes
Avoid growing objects

In general, R's performance depends strongly on your code quality. A very good summary on what you can and should do is provided in R Inferno by Patrick Burns.


Logistic Regression optimal threshold is a negative value I run the code below:
import pandas as pd
import numpy as np
from sklearn.model_selection  import train_test_split
from sklearn import linear_model
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from numpy import sqrt
from numpy import argmax
from sklearn.metrics import roc_curve
from sklearn.preprocessing import StandardScaler

def standardize(variable):
      return (variable - np.mean(variable)) / np.std(variable)

def normalize(x):
    return (x-x.min()/(x.max()- x.min()))

data.columns = np.arange(len(data.columns))

trainX, testX, trainy, testy=train_test_split(X,y,test_size=0.5,random_state=2, stratify=y)

# fit a model
model = LogisticRegression(solver='lbfgs')
model.fit(trainX, trainy)

#yhat = model.predict_proba(testX)
yhat = normalize(testX.values)
yhat = yhat[:, 0]
print(yhat)

# calculate roc curves
fpr, tpr, thresholds = roc_curve(testy, yhat)

#print(thresholds)
# calculate the g-mean for each threshold
gmeans = sqrt(tpr * (1-fpr))
# locate the index of the largest g-mean
ix = argmax(gmeans)
print('Best Threshold=%f, G-mean=%.3f' % (thresholds[ix], gmeans[ix]))
# plot the roc curve for the model
pyplot.plot([0,1], [0,1], linestyle='--', label='No Skill')
pyplot.plot(fpr, tpr, marker='.', label='Logistic')
pyplot.scatter(fpr[ix], tpr[ix], marker='o', color='black', label='Best')
# axis labels
pyplot.xlabel('False Positive Rate')
pyplot.ylabel('True Positive Rate')
pyplot.legend()
# show the plot
pyplot.show()

The optimal threshold score is: Best Threshold= -0.049752, G-mean=0.889
Why is the optimal threshold a negative number? What does it mean? And why am I getting a negative number?

I'm not familiar with the way you are obtaining the optimal threshold, but It might be a little bit easier.
What you are looking for is the leftmost point in the x-axis (false positive rate) and the rightmost point in the y-axis (true positive rate) So by calculating the difference between the two you will have so.
from sklearn.metrics import roc_curve
yhat = best_model.predict_proba(X_train)[:,1]

fpr, tpr, thresholds = roc_curve(y_train, yhat)
optimal_idx = np.argmax(tpr - fpr)
optimal_threshold = thresholds[optimal_idx]

This threshold will give you the lowest false positive rate and the highest true positive rate
EDIT
I just notice that you are passing the $P(Y = 0| X)$ i.e yhat = yhat[:, 0] try passing yhat = yhat[:, 1]


Does policy optimization learn policies to make better actions with higher probability? When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients.

I would like to know is it correct to say: Policy Optimization learns policies to make better actions with higher probability?

Also, what is the location of  Proximal Policy Optimization in the picture?

The image in your question looks to me like a loose hierarchy explaining how various Reinforcement Learning methods relate to each other. At the top are broad categories of algorithm based on whether they are value-based or policy-based, towards the bottom are more specific methods.

There is more than one way to categorise and split RL algorithms, and it would be messy to try and include all the ways that they relate to each other. Just bear in mind that it is a very rough guide.

The main difference between value-based and policy-based methods is:

Value-based methods learn a value function (by interacting with the environment or a model of it), and consider the optimal policy to be one that takes actions with maximum value.
Policy-based methods learn a policy function directly, and many can do so without considering the estimated value of a state or action (although they may still need to calculate individual values or returns within episodes).

Note Actor-Critic is linked to both headings, as it learns both a policy function (the "actor") and a value function (the "critic").

  I would like to know is it correct to say: Policy Optimization learns policies to make better actions with higher probability?

Yes that is broadly correct, although you don't define "better". A policy function typically returns some probability distribution over possible actions for any given state. When learning, it will tend to increase probabilities of actions that resulted in better returns (discounted sums of rewards) and reduce probabilities of actions that did not. This may be a very random, high variance process though, depending on the environment.

There are exceptions. Some policy-based methods learn a deterministic policy $a=\pi(s, \theta)$, and adjust the value based on adding some noise to this function to explore and adjusting the function based on the results of the different action. These don't behave like your statement (because there is no probability to make higher).

  Also, what is the location of Proximal Policy Optimization in the picture?

Proximal Policy Optimization is definitely a policy-based method, and it does use an estimate of a value function too for updates (in this case a value called advantage which you also see used in Advantage Actor-Critic).

In the diagram I would probably place it under Actor-Critic Methods in a new row as a specific example of one. However, it does have some significant variations from "vanilla" Actor-Critic, based on how it restricts large changes to the policy function.


What is the minimum requirement for the dataset for time series forecasting? I have a dataset of patients where, for each patient, a measurement is taken 3 times per day. For example, patient 1 has recordings at 7.30 am, 12.30 pm and 8.30 pm. Patient 1 has a collection of 30 days with such data recorded thrice a day.(altogether 3 recordings per day * 30 days = 90 data points).

Is this type of data suitable for time-series forecasting?

Thanks in advance for the help.

Sure! A time-series is just a set of data points indexed temporally, and that's exactly what you have for each patient.

You can certainly build a forecaster using this data.  You may run into a couple wrinkles since your data is not spaced evenly, but that's not an insurmountable barrier.

If you want to quickly prototype a forecasting model, FB Prophet is incredibly easy to use and often works well out-of-the-box.


Confusion Matrix - Get Items FP/FN/TP/TN - Python After run my python code:

print(confusion_matrix(x_test, x_pred))

I get this:
[100 32
211  21]

My question is how can I get the following list:

True positive = 100
False positive = 32
False negative = 211
True negative = 21

Is this possible?

Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them

def perf_measure(y_actual, y_pred):
    TP = 0
    FP = 0
    TN = 0
    FN = 0

    for i in range(len(y_pred)):
        if y_actual[i]==y_pred[i]==1:
           TP += 1
        if y_pred[i]==1 and y_actual[i]!=y_pred[i]:
           FP += 1
        if y_actual[i]==y_pred[i]==0:
           TN += 1
        if y_pred[i]==0 and y_actual[i]!=y_pred[i]:
           FN += 1

return(TP, FP, TN, FN)

Alternatively, if confusion matrix is a 2x2 matrix (named cm), you can use

TP = cm[0][0]
FP = cm[0][1]
FN = cm[1][0]
TN = cm[1][1]


How to input sets as features Need advice on the best way to represent the below data to be fed into an ML algorithm (yet to decided on)
This is from the online order processing domain. An order consists of a set of variable number of items. Each item can be located in different warehouses, again this is a variable number. The entire order with multiple items and items with multiple warehouses per item, needs to be processed as one training sample. The goal is to learn a function that outputs the warehouses from which the items can be picked under some rules/conditions to minimize processing costs.
The number of items can run in millions and stores in 1000's.
I've been looking at representing these as permutation invariant sets - is there a simpler way or is that the right way to go about it ?

In ML you really need good examples and then things for which you don't know the outcome. You learn from the good examples and then apply this "knowledge" to the examples for which you wish to know the outcome.
I agree that Mathematical Optimisation would probably be a better route to take in a problem such as this.
Alternatively, if you want to imply some kind of connection between sets, you could create a categorical (dummy) variable that designates such. If I understand you correctly here is an example
         item1    item2   basket1  basket2  basket3  GroundTruth (target)
order 1  Banana   Apple   True     False    False    warehouse1
order 2  Toy      Carrot  False    True     False    warehouse2
order 3  Picture  Shoe    False    False    True     warehouse3

You could also include the items in the warehouses in a similar way though if you have lots of different items, lots of warehouses, lots of baskets and not relatively enough training example this is going to get pretty sparse pretty quickly.


How can I specify I want output of which units in LSTM? I see this code concept(with Keras library) in most code examples of LSTM:

model.add(LSTM(X))
model.add(Dense(Y))

But I don't really know if I have 10 time-steps in input side and need the last time-step in output is the following code true?

model.add(LSTM(10))
model.add(Dense(1))

Or if I have 10 input time-steps and need 10 output time-steps(I mean I need all outputs from all units) is the following code true?

model.add(LSTM(10))
model.add(Dense(10))

How should I specify if I have 1 layer LSTM with 10 units, I need all of 10 outputs? Or I need 1 last output from unit 10? Or the last 5 outputs from unit 5 to 10?

If you use:

LSTM(10 , return_sequences = True)

you will only get as output something of size (batch_size, num_timesteps, 10).  The "10" for LSTM unit-size is a hyper-parameter you should be tuning.  However, what you want is something the size of (batch_size, num_timesteps, num_features).  This is why you want to put a Dense layer after the LSTM layer.  That being said, you will also want to add a TimeDistributed wrapper around your Dense layer, so that a Dense layer is applied to every time-step (1):

TimeDistributed(Dense(num_features))

Now the output of the TD dense layer will be of size: (batch_size, num_timesteps, num_features), which I'm assuming is what you want.  You will have to set up your Y data to be of the appropriate size for this to work.  It will have to be of size (batch_size, num_timesteps, num_features).  You mentioned that you are curious to know how to use just part of it.  At prediction time, post training, you would call your model.predict and it will return something the size of (batch_size, num_timesteps, num_features).  You can just use numpy array slicing to access what it is that you want.


Training on derived features that won't be present in a test set So I have an odd real-world problem in which the data that's going to be fed to a categorical predictive model has (a few) certain features, and when building a model I can bolt on additional datasets that add context to some of those features, but when it's going to be running live I won't be able to use any of that additional data - the model itself needs to already "know" what's important about specific values of the original data and be able to operate just on those attributes.

The major features here are time(/date/DOW) and location (lat/long) within a city.  So, the additional datasets are context about that location (zoning, demographics, transit) or time (traffic, commercial activity, etc).  But for performance reasons (join cost, etc) we can't use them live, we only have the time and location numbers (plus easily parse-able derivatives).

I'm imagining a process whereby the model uses those additional features to learn which subsets or demarcations of time-and-location actually matter, and can then continue to use those weightings and rules or whatever even in the absence of the additional features, just relying on the numerical values.  But it's also possible that there's no such approach.  Can anyone suggest an algorithm or approach that could let us retain some of that value?

Thats not how it works, there is not juice to be extracted if the data is missing from the test set.

You will have features in train that might be discriminative, but missing in test. Then when you try to predict there wont be any inputs that you can map to the outputs.

This beeing said I should mention that NaiveBayesclassifier will not use missing features, so there would be no harm in including them but you wont get any additional information, just more unnecessary complexity.


Why MLP only learns bias for unbalanced binary classification? I just started training a MLP model on a data set which has the following statistics. Notice that both train and validation sets are unbalanced (88.4% negatives).

Note: The unbalance isn't caused by sampling and is because that's the natural ratio of the classes. E.g. when I look back in a 3 month period, I see I have eaten 88 apples and 12 oranges; and looking forward, I might still favor apples more than oranges by the same ratio. I may be wrong here but I think I shouldn't give class weights nor resample to balance the two classes, because that's their natural ratio.

Train data:
    Shape (features): (891473, 122)
    Shape (labels): (891473,)
    # of classes: [(0, 788118), (1, 103355)]
    % of classes: [(0, 0.88406266931247501), (1, 0.11593733068752503)]
Validation data:
    Shape (features): (251141, 122)
    Shape (labels): (251141,)
    # of classes: [(0, 222009), (1, 29132)]
    % of classes: [(0, 0.8840014175303913), (1, 0.1159985824696087)]

I'm using three 8-sized dense layers each following a ReLU layer, finally a sigmoid since it's binary classification.

However, the training process converged after only 2 epochs and basically only learned the bias term and and ended up predicting everything to be negative. The confusion matrix stays unchanged like this since the 2nd epoch till the end (20 epochs):

Train confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884
Validation confusion matrix:
           P = 0   P = 1 | Total
    L = 0  0.884   0.000 | 0.884
    L = 1  0.116   0.000 | 0.116
    ----------------------------
    Total  1.000   0.000 | 0.884

I have tried to increase the size of the dense layer (e.g. from 32, 128, 512, 1024). They all yielded the same result.

I'd expect that with 3 x 1024 neurons the model would be complex enough to learn something beyond a single bias term. So I'm quite confused as what I am doing wrong to make it not learning anything other than the bias.

Question:

What could be the possible reasons for the model to learn not much?
What should I try and/or what kind of info should I be looking for at this point to move this forward?

Thank you for your time in advance~!

The following is a snippet of my code to show what I was trying to do:

learning_rate = 1e-4
batch_size = 10000
epochs = 5

DENSE_SIZE = 1024

train = (training_features.values, training_labels.values)
validation = (holdout_features.values, holdout_labels.values)
test = (eval_features.values, eval_labels.values)
eval_sets = [
    ('Train', train),
    ('Validation', validation),
    ('Test', test),
]

model = Sequential()
model.add(Dense(DENSE_SIZE, activation='relu', input_shape=train[0].shape[1:]))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(DENSE_SIZE, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# initiate optimizer
opt = RMSprop(lr=learning_rate)

# Let's train the model
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

model.fit(
    train[0],
    train[1],
    batch_size=batch_size,
    epochs=epochs,
    validation_data=validation,
    shuffle=True,
    verbose=2,
    callbacks=callbacks,
)

EDIT:

The three answers as of writing all suggest that the inefficiency in learning is associated w/ the imbalanced classes and I should try to balance the classes by either using class weights or resampling.

While I'm happy to see an apparent agreement of what the problem is what , could anyone provide more insights about why imbalanced classes cause difficulty even for a complex model? In theory a complex enough model can at least memorize all the rarely positive samples and achieve a higher-than-bias accuracy on the training set.

Though you know that dataset is naturally balanced but your model will still consider it as unbalanced dataset. If you take a example financial fraud detection, number of fraudulent transactions are always less than the ligit translations, to handle such such cases data scientists will have to balance the classes for better results.
You will have to balance your dataset to get the better results from model.


What type of optimization problem is this? We have a deformable mirror controlled by 40 actuators that have input voltages from -1V to 1V.
Prior to hitting the mirrors, a pulse of light enters a diffraction grating and then the light is spread over the mirror by wavelength. Each actuator applies a difference phase shift to each wavelength. In the end, the light gets reconverged and read by a CCD.

Our goal is to maximize the integral of the spectrum read by the CCD with respect to the 40 variable voltages. This optimization will be done in matlab. I mostly just need to know what type of problem this is...

The program will do something like:

Measure a spectrum and integrate
Input the new spectrum to the optimization toolbox.
Output a new set of voltages to try
Go back to 1

There is likely multiple local maxima and the positions will change from day to day as our setup is always changing. Any help on where to look for more information would be much appreciated!

Some flavor of evolutionary algorithm may suit your problem nicely, since:

The gradient of the objective function is unavailable or cannot be computed.
The objective function itself is somewhat expensive to compute.
The objective function is may have a large number of local maxima.

In short, evolutionary algorithms do optimization by generating candidate solutions and iteratively:

Checking their "fitness" (the value of the objective function)
Selecting the best candidate solutions
Generating new candidate solutions, somehow based on the best candidates from the previous iteration

Naturally, the devil is in the details of how the best candidate solutions are selected and combined to generate new candidate solutions. This gives rise to a large number of evolutionary algorithm variants. Two popular variants are genetic algorithms and evolution strategies.

Genetic algorithms

Standard genetic algorithm implementations require you to encode your input variables in a binary fashion. This may be fine for categorical inputs, but your input variables take real values in the interval $[-1,1]$. It would be possible to implement a genetic algorithm for your problem, but it wouldn't be ideal. For your problem, I'd take a look at...

Evolution strategies

On the other hand, evolution strategies handle real-valued inputs by design and, after getting used to the jargon, are fairly easy to implement. You mentioned that you plan to use MATLAB for optimization. I found this toolbox on the FEX which implements evolution strategies, but I haven't taken it for a test drive myself. Whatever the case, I think an evolution strategy would be a good option to try, given the dynamic nature of your problem.


How feed a numpy array in batches in Keras I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays.
To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
    indices = np.arange(len(X))
    batch=[]
    while True:
            # it might be a good idea to shuffle your data before each epoch
            np.random.shuffle(indices)
            for i in indices:
                batch.append(i)
                if len(batch)==batch_size:
                    yield X[batch], Y[batch]
                    batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
    batch=[]
    while True:
            np.random.shuffle(ids)
            for i in ids:
                batch.append(i)
                if len(batch)==batch_size:
                    yield load_data(batch)
                    batch=[]

Your loader function may look like this:

def load_data(ids):
   X = []
   Y = []

   for i in ids:
     # read one or more samples from your storage, do pre-processing, etc.
     # for example:
     x = imread(f'image_{i}.jpg')
     ...
     y = targets[i]

     X.append(x)
     Y.append(y)

   return np.array(X), np.array(Y)


Is it alright to split a GridSearchCV? Is it ok to split a GridsearchCV?
At first, I would try estimators from 100-300 (100 steps) for a random forest regressor and some other parameters and after that, I would start the GridsearchCV with the same parameter and just change the estimators from 400 - 600.
Is there any aspect that would disagree with that logic?

First my understanding of your problem. You want to find the best hyperparameters for a Random Forest.

For that, you want to first adjust n_estimators parameter and then the rest of parameters in different runs.

Before answering to your question, you will only want to do a thorough search of hyperparameters when you want to have an improvement of around 1%. So it will be a small improvement. If you want to improve your model, probably feature engineering or data engineering will give you a better improvement. Or even a different algorithm.

You can print the results of your GridsearchCV with

pd.DataFrame(clf.cv_results_)

The answer to your question:

No you shouldn't run a GridsearchCV in different runs you have to explore the whole parameters if you want to find the global minima. A small change in one parameter can affect other. At the end you are exploring a search space.


How to handle sparsely coded features in a dataframe I have a dataset that contains information regarding diabetes patients, like so:
    id   diabetes   diet   insulin   lifestyle
    0    No         NaN    NaN       NaN
    1    Yes        Yes    Yes       NaN
    2    No         NaN    NaN       NaN
    3    Yes        NaN    NaN       NaN
    4    Yes        Yes    NaN       Yes
    5    Yes        Yes    Yes       Yes

Features diet, insulin and lifestyle have a high percentage of missing data (around 95% each). So initially, I excluded these features from my dataset. However, after taking a closer look at the data, I found the values for diet, insulin and lifestyle to be associated with the value for diabetes feature. This makes sense as diabetes patients would be recommended treatment relating to diet, insulin intake and lifestyle changes.

So in cases where diabetes='No', values for features diet, insulin and lifestyle are missing.

And in cases where diabetes='Yes', I have found that in most cases, at least one feature from diet, insulin and lifestyle to have a value of 'Yes', and the remaining values are missing.

After some reading, I believe features diet, insulin and lifestyle are Missing at Random (MAR), and clearly not missing completely at Random (MCAR) as is explained here.
Anyway, so my question is, should the nature of the missing data here change my decision to remove these features from the dataset due to their high percentage of missing values. Or, should I impute the data for these features, by filling in missing values with "No", like so:
    imputer = SimpleImputer(strategy='constant', fill_value='No')
    x[:, 2:5] = imputer.fit_transform(x[:, 2:5])

The first question you have to answer is whether these are actually missing or simply sparsely coded!
E.g. if the variables are "supposed" to only show doctors recommended diet, insulin or lifestyle change then we could naturally conclude that any NaN is actually a "No" in which case you do not even have to impute the data but rather straight-up replace with "yes = 1 and NaN = 0.
How do you differentiate this case (sparsely coded) from the other case: actual missing values? Besides applying your domain knowledge you should also test whether the following rules apply:

If diabetes == No then all treatment variables are missing
If diabtes == Yes then at least one treatment variable is "Yes"
Treatment variables are always only "Yes" or NaN

If this is the case your dataset is likely sparsely coded, otherwise you have actual missing values.
Take note however that should the values be actual missing values (e.g. because you identify some cases where non-diabetics have a treatment or some cases are labeled "No" in the treatment variables) you can assume they are MAR but not MNAR. In this case I would recommend to remove these variables because imputation as "No" or o does not make sense if you do not have any information what the NaN actually means.


Should the minimum value of a cost (loss) function be equal to zero? We know optimization techniques search in the space of all
the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or Categorical Cross Entropy, has a global minimum value equal to zero, in the ideal case.

For example, the Gradient Descent, $\theta_j \leftarrow \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta)$, updates parameters based on the derivation of the calculated cost function value, $J(\theta)$.

I was wondering what will happen if we design a cost function that has a non-zero global minimum in its ideal case. Does it make a difference, e.g. in the convergence rate or other aspects of the optimization process, or not?

Saying that the well-known loss functions, like MSE or Categorical Cross Entropy, has a global minimum value equal to zero is flawed . The idea behind loss function is to measure how near the model predictions are to the actuals(in case of a regression). Now ideally , you would want your model to predict exactly equal to the actuals . In that case only , we get loss equal to zero. Otherwise , loss is non zero almost all the time . If you remember the loss function for a linear regression setting ,

We need to minimise  so that the predictions  can be as close to the actuals as possible . For that the derivative of  should be zero . It doesn't matter if   is zero or non zero . Graphically , for a cost function like this
 , you want to reach the point  where the derivative is zero.


Weights in neural network So I am newbie in deep learning, I came across activation functions which gives an output and compares it to label, if it's wrong, it adjusts its weight until it gives the same output as labelled data for that particular input in training data set.

 x1          x2           x3          y
 10          15           20          0
 20           7           10          1
 5           10           4           0

So imagine this is an example training set, we send these inputs to activation function, and for the first input it returns correct output (0).

But for second output it again returns 0, so the weights are adjusted until the activation function returns 1.

So now my doubt is, if the new updated weights returns the wrong output for the third input, its weight gets changed again, but will there be a situation where these weights will not satisfy for the previously tested inputs, like for example the first input in this case.

Is there a chance that new weights will return 1 for the first input, which is wrong?

In parametric models such as linear regression, logistic regression and multi-layers perceptrons, weights are updated with regards to the "difference" between the output of your model and the real label.

More precisely, weights are updated using the gradient descent / backpropagation procedure. It is composed of two parts : the forward pass and the backward pass.

For a given observation (or a set of observations), the forward pass is about feeding the model with observations and output a result a. This output "a" is then compared with the real value, the label y. Using some cost function metrics (such as Absolute Error or Square Error for regression purposes,  cross-entropy for classification purposes...), we can then compute j(y,a) which is the error between the output and the real value.

We can now run the backward pass which is about computing the derivative of the cost function with regards to any weight / bias coefficient in the logistic regression / neural network. We can the update coefficients such as :

Where alpha is the learning rate. So to answer your question, weights are not updated until they reach the expected value. We are just trying to reach the minimum in cost function surface by running gradient descent procedure.


Overcoming "too many indices for array" error when loading file into Orange I am attempting to load the Stack Overflow 2017 developer survey results into Orange. The dataset has ~51.4K rows with 156 features. It is comma separated and is about 91MB in size.

When I attempt to load this file into Orange, I get the following error:

Cannot parse dataset H:/survey_results_public.csv: too many indices for array

How can I overcome this error?

This error basically means that your data is incorrectly formatted and Orange cannot read it. Somewhere there is either a redundant column delimiter or it might be absent. Either way, you should carefully check that all your columns are correctly separated by tabs or any other consistent separator.


Kmeans using silhouette_score I am using silhouette_score to find the optimal k value. So I am running a for loop with a range of possible k values. I have added my code below. this program takes a very long time to run. Could you suggest some improvements for a more efficient run time?

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from sklearn.cluster import KMeans
from sklearn import metrics

data=np.load(filename)

coeffs=[]

for i in range(2,8):

    clusters=KMeans(n_clusters=i)
    clusters.fit(data)
    labels = clusters.labels_
    sil_coeff = metrics.silhouette_score(data, labels,metric='euclidean')
    coeffs.append(sil_coeff)

coeffs=np.array(coeffs)
k=np.argmax(coeffs)+2

Do some measurements to identify your bottlenecks.

Here, I suggest to not use Silhouette. Because it is much slower than k-means. Silhouette need O(n²) distance computations every run!

The obvious was to speed this up would this be to compute and store a distance matrix just once. This won't help k-means, but it will make multiple Silhouette runs a little bit faster (it will still be O(n²), but only array lookups instead of distance computations now). Nevertheless, this will not be scalable to large data.

The better approach is to use a cheaper heuristic to guess the "best" value of k. None such measure is perfect anyway.


Finding the top K most similar sets I have a database containing sets of words. So for example, I have a database that has:

{happy, birthday, to, you}
{how, are, you}
...

Given a query set, lets say {how, was your, birthday}, I want to find the top K sets in the database that is most similar to my query. Similarity metric used can be something like Jaccard's Index. Right now, I go through the database one by one and calculate the Jaccard Index and keep track of the top K scores found so far. I was wondering if there are any data structures or methods that would allow me to more efficiently find the top K scores. Right now it's a linear search. Thanks

Do you have any information about your data set? Is it sparse, will most similarities be zero? Is the total dictionary small? You could consider a inverted index. For example

word  query_id
W1    [1, 3, 6]
W2    [2, 5]
W3    [1, 3, 4]
W4    [2, 3, 4]
W5    [2, 3, 6]

query_id  query
1         W1 W3
2         W2 W4 W4
3         W1 W3 W4 W5
4         W3 W4
5         W2
6         W1 W5

Here W_i is a word, e.g. birthday and query_id is the id of the query in the database. e.g. {how, are, you} might have id 22. Now you get a query {W1 W3 W5}. Aggregate counts on the inverted index. W1 was seen in queries 1, 3, and 6. W3 in 1, 3, and 4, etc.

query_id  count
1         2
2         1
3         3
4         1
6         2

The count will the number of words in common with the incoming query, this is the numerator of the jaccard similarity. So, to find the top k you can start with the queries with the highest count. query_id 3 has the highest count and its similarity is 3/4.

If you have a massive database there are techniques like locality sensitive hashing which will basically reduce the search space into a small bucket. The incoming query gets hashed and lands in a bucket. You can then do a linear search with all the queries in this bucket to find the nearest k.


Parsing data from a string I think this is something that experienced programmers do all the time. But, given my limited programming experience, please bear with me.

I have an excel file which has particular cell entries that read

[[{"from": "4", "response": true, "value": 20}, {"from": "8", "response": true, "value": 20}, {"from": "9", "response": true, "value": 20}, {"from": "3", "response": true, "value": 20}], [{"from": "14", "response": false, "value": 20}, {"from": "15", "response": true, "value": 20}, {"from": "17", "response": false, "value": 20}, {"from": "13", "response": true, "value": 20}]]

Now, for each such entry I want to take the information in each of the curly brackets and make a row of data out of it. Each such row would have 3 columns. For example, the row formed from the first entry within curly brackets should have the entries "4" "true" and "20" respectively. The part I posted should give me 6 such rows, and for n such repetitions I should end up with a matrix of 6n rows, and 4 columns ( an identifier, plus the 3 columns mentioned).

What would be most efficient way to do this? By "doing this" I mean learning the trick, and then implementing it. I have access to quite a few software packages(Excel, Stata, Matlab, R) in my laboratory, so that should not be an issue.

If you have R is quite simple

Copy the lines into a file, let's say: "mydata.json"
Be sure you have installed the rjson package

 install.packages("rjson")

Import your data

 library("rjson")
 json_data <- fromJSON(file="mydata.json")


How can I implement deconvolution on CNN (TensorFlow)? I'm new in tensorflow and machine learning. Could you explain me how implement deconvulation on this CNN model(http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/) for text classification?

CNN model:

class TextCNN(object):
"""
A CNN for text classification.
Uses an embedding layer, followed by a convolutional, max-pooling and softmax layer.
"""
def __init__(
  self, sequence_length, num_classes, vocab_size,
  embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0):

    # Placeholders for input, output and dropout
    self.input_x = tf.placeholder(tf.int32, [None, sequence_length], name="input_x")
    self.input_y = tf.placeholder(tf.float32, [None, num_classes], name="input_y")
    self.dropout_keep_prob = tf.placeholder(tf.float32, name="dropout_keep_prob")

    # Keeping track of l2 regularization loss (optional)
    l2_loss = tf.constant(0.0)

    # Embedding layer
    with tf.device('/cpu:0'), tf.name_scope("embedding"):
        self.W = tf.Variable(
            tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_chars = tf.nn.embedding_lookup(self.W, self.input_x)
        self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)

    # Create a convolution + maxpool layer for each filter size
    pooled_outputs = []
    for i, filter_size in enumerate(filter_sizes):
        with tf.name_scope("conv-maxpool-%s" % filter_size):
            # Convolution Layer
            filter_shape = [filter_size, embedding_size, 1, num_filters]
            W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
            conv = tf.nn.conv2d(
                self.embedded_chars_expanded,
                W,
                strides=[1, 1, 1, 1],
                padding="VALID",
                name="conv")
            # Apply nonlinearity
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
            # Maxpooling over the outputs
            pooled = tf.nn.max_pool(
                h,
                ksize=[1, sequence_length - filter_size + 1, 1, 1],
                strides=[1, 1, 1, 1],
                padding='VALID',
                name="pool")
            pooled_outputs.append(pooled)

    # Combine all the pooled features
    num_filters_total = num_filters * len(filter_sizes)
    self.h_pool = tf.concat(pooled_outputs, 3)
    self.h_pool_flat = tf.reshape(self.h_pool, [-1, num_filters_total])

    # Add dropout
    with tf.name_scope("dropout"):
        self.h_drop = tf.nn.dropout(self.h_pool_flat, self.dropout_keep_prob)

    # Final (unnormalized) scores and predictions
    with tf.name_scope("output"):
        W = tf.get_variable(
            "W",
            shape=[num_filters_total, num_classes],
            initializer=tf.contrib.layers.xavier_initializer())
        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
        l2_loss += tf.nn.l2_loss(W)
        l2_loss += tf.nn.l2_loss(b)
        self.scores = tf.nn.xw_plus_b(self.h_drop, W, b, name="scores")
        self.predictions = tf.argmax(self.scores, 1, name="predictions")

    # CalculateMean cross-entropy loss
    with tf.name_scope("loss"):
        losses = tf.nn.softmax_cross_entropy_with_logits(logits=self.scores, labels=self.input_y)
        self.loss = tf.reduce_mean(losses) + l2_reg_lambda * l2_loss

    # Accuracy
    with tf.name_scope("accuracy"):
        correct_predictions = tf.equal(self.predictions, tf.argmax(self.input_y, 1))
        self.accuracy = tf.reduce_mean(tf.cast(correct_predictions, "float"), name="accuracy")

I know about tf.nn.conv2d_transpose, but I don't understand how can I get original data (not embedded_chars)

Deconvolution have very simple structure: unpooling → deconv
like this:

# Unpooling
Ps = (tf.gradients(pooled, h))[0]
unpooled = tf.multiply(Ps, P)

# Deconv
batch_size = tf.shape(self.input_x)[0]
ds = [batch_size]
ds.append(self.embedded_chars_expanded.get_shape()[1])
ds.append(self.embedded_chars_expanded.get_shape()[2])
ds.append(self.embedded_chars_expanded.get_shape()[3])
deconv_shape = tf.stack(ds)
deconv = tf.nn.conv2d_transpose(
    unpooled,
    W,
    deconv_shape,
    strides=[1, 1, 1, 1],
    padding='VALID',
    name="Deconv"
    )


Will images modification get me a better machine learning model? Will images modification get me a better machine learning model?
I have the following scenario. Camera is fixed and does photos of a process. The process has a few states. Now I want to train a model given a photo to classify to which state, does the photo belong. I.e. what is the current state of the process.
For the model training I was given only one photo for each of the process state.
Now my question is whether I should train my model with one photo in each class or should I take that one photo and do random modifications to the photo so that to get a few photos for each class?
I am thinking that it is better for the training to have just one but real photo. Instead of a lot modified. Am I right?
I use the ML.NET plugin for a Visual Studio to train the image classification model.

One can never 100% say that a data preprocessing approach will yield positive results. So, if you are trying different things, always test and use the metrics to see what works best.
With that said, what you described is often referred to data augmentation, namely the generation of more data points, typically from existing data points. It is very common practice and so I encourage you to try it on your problem. Common techniques include flipping, shifting, or rotating the original images.
The problem of having few examples per class is called zero- or few-shot learning. Specific techniques such as siamese networks have been devised to tackle these. Here is a good blog to use as reference.


Reaching 100% accurray in Data Mining I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results.

If I insepct the results of my Topic Model, the overall Topic assigment for each document is pretty good, but some documents got an assigned topic which does not fit very well.

So overall I got an accuraccy of approx. $85\%$.

Is it possible to reach $100\%$ accuracy in Data Mining?

In theory it's of course possible to reach perfect performance: if the algorithm can find what it needs in the features to correctly distinguish between classes (or clusters), then it will perform perfectly.

In reality however it's very rare that performance is perfect, because:

Text data is noisy and extremely diverse
Most of the time when there is a way to obtain perfect performance, there is also a simple heuristic which can do the same job more efficiently than using ML. Basically ML is used precisely because the task is hard and/or the data is complex, so it's not surprising that there are errors.

In the case of your problem, I notice that you have labels for your data but you use an unsupervised topic modeling method, right? If this is the case you might want to try using a supervised method, since the system would have more clues to find the right answer. Also you use accuracy for evaluation, so be careful: accuracy can be misleading since it doesn't give any detail about the different classes.


Increasing samples increases variance I've been running kfold cross validation with 10 folds and comparing it against a test set. Logging the score and the stdev along the way.

Once I wad happy with my model I then run the estimator through the cross validation with the full data (including the test set) but ... The standard deviation of the scores increases and the score decreased????

Is this normal? I assumed MORE data decreases variance ? Or is it just due to the randomness in the splits ?

The score decreasing and the standard deviation of the scores increases, as sample size increases, indicates overfitting to the smaller sample size.


How to further Interpret Variable Importance? Forgive me if this is a duplicate question, I haven't found anything that answers my question specifically after searching for a while.

I have a dataset which I'm using to predict mobile app user retention, using the RandomForestClassifier in the SciKit Learn package. I'm pleased with the accuracy I'm getting and I'm planning on including a number of other metrics including precision, recall, Matthew's Corr Coefficient etc. I'm pretty sure the model is good.

The key thing that I'm interested in here are the features themselves. I want to know what is contributing to my user churn. I have extracted the feature importances and plotted a nice looking graph, but now I'm stuck. I'd ideally like to know how each variable influences the churned/not churned outcome. The problem with GINI feature importances is that I can see which ones are most influential, but for example with continuous variables I want to know at which value the RF found best to split on. I don't need to see this for every feature as I have 70+ only the most 'important' ones.

I saw a very nice decision tree plot here, but cannot find any way of reproducing something similar using scikit learn. I'm open to other suggestions. Thanks in advance :)

The documentation offers a couple options. To plot the individual trees in your forest, one can access them like model.estimators_[n].tree_ and then plot them with export_graphviz as explained in the documentation, or you can follow this example that directly prints the structure in text format.

However, I would say this is not the best idea, because a feature can occur in different trees and nodes with different split points. You probably get a better intuition about your features from partial dependence plots that try to isolate the effect of one variable on your response variable.

As a bonus, here is a good article about more alternatives to gain insight into your model (not all applicable to random forests).


K-Means Clustering for data points with multiple attributes I'm very new to K-Means clustering. Every example that I have seen has a two-dimensional data set.

I am working to classify recipes of varying ingredient composition into families. Each recipe is composed of a number of ingredients, and I want to group together the recipes that are similar to each other. Each recipe on average has 8 ingredients.

Is there any way I can use K-Means clustering to group these recipes together? Or would you suggest another unsupervised machine learning method?

In k-means clustering, the "k" defines the amount of clusters - thus classes, you are trying to define.

You should ask yourself: how many different groups (=clusters) of recipes am I looking for?

In your case, your data points (features) (=recipes), are of variable dimensions (attributes) (avg 8 dimensions).

What you must realize is that the dimension of each datapoint must be the of the same length. (So the amount of recipes for each ingredient should be the same).
This is because k-means calculates the distance of each datapoint to each centroid (k centroids).

What you can do to create datapoints of the same dimension, you can introduce "DTW".
More about this here:

https://stats.stackexchange.com/questions/26100/how-to-run-k-means-clustering-on-data-points-of-varying-dimensionality

An easier solution would be that you populate your dataset with a value for each ingredient present in the dataset, and set the value to "0" when the ingredient is absent.


what indicators can be used to classify a group of stocks? I am working with a dataset contains the daily return time series of 50 different stocks, I want to divide these stocks into several different groups.
My idea was to make a new dataset contains some indicators of these 50 stocks, and use a cluster algorithm to classify them, for example Hierarchical Clustering Method and plot dendrogram to decide how many groups make most sense. The indicators I am using are: Mean, Standard Deviation, Skewness and Sharpe ratio.
First of all, my economic background is weak and I was wondering if there are any other well used indicators could be added to this dataset to make the classification make more sense.
Secondly, too many indicators can make it difficult to interpret the stocks being classified.
Can someone share some ideas?

Question 1:
Geograhic location, market sector, it all depends on what you're trying to achieve with the clustering.
Question 2:
Yes. Adding too many indicators will cause your data to overfit and at some point you'll end up with every cluster having 1 member. I would use a regularisation penalty term to counter this. Either L1 (Lasso Regression) or L2 (Ridge regression) would work for this. Or you can use Elastic Net which would use a portion of both.


Appropriate algorithm for string (not document) classification? I am trying to classify a large-ish number of small strings (millions) into about 10 disjunct categories. Examples of classes and strings for each class include:

email: "foo@bar.com"
phone: "55", "22334455"
personName: "John", "Q.", "Public"
organizationName: "Reuters", "IBM"
date: "Dec.", "22.10.2010"
nameAndEmail: "Larsonlarson@gmail.com" (a last name has been concatenated with an email address.)
phoneAndEmail: "55larson@gmail.com" (part of a phone number has been concatenated with an emal address)
separator: ",", "and", "or"
other: stuff that does not fit in other categories.

Carrying out classification via a set of heuristics seems tedious, so I have been experimenting with an SVM. I'm not able to get more than about 80% accuracy without doing a lot of work manually encoding features like "has an ampersand", "is uppercase", "is mostly numeric", "has a colon in the middle", etc., which sort of defeats the purpose.

I'm wondering whether part of the challenge with using an SVM is that the "salient features" of a given string, like the '@' character, are not in a fixed position in the string, so it will not get the same feature index for each string.

Does anyone have suggestions for a more appropriate approach to this task, or can anyone recommend further reading?

You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings.  With N = 4 or greater, you would capture things like ".com" in emails, "##.#" in dates, etc.

It might also help to encode all single digits as one reserved number-only-character.

An easy way to to this might be to create all the n-gram substrings for each string in your dataset, then simply treat that list of words as a document.  Then you could use term frequency or tf-idf vectors in your supervised step.

For example to create the substring uni-, bi-, and tri-grams for "whatever@gmail.com":

a = "whatever@gmail.com"
b = set([a[x:x+y] for y in range(0,4) for x in range(0,len(a))])

set(['', 'co', 've', 'ai', 'eve', 'r@', 'at', '.co', 'gm', 'ev', 'tev', 'er', '@gm', 'ver', '@g', 'r@g', 'ail', 'il.', 'gma', '.', 'te', 'hat', '@', 'wha', 'om', 'wh', 'er@', 'mai', 'ma', 'ha', 'l.c', 'a', 'c', 'ate', 'e', 'g', 'i', 'h', '.c', 'm', 'l', 'o', 'l.', 'r', 't', 'w', 'v', 'com', 'il'])


Reversed Naive Bayes - likelihood and parameter estimation What happens if we flip the arrows in a Naive Bayes classifier? To clarify - from what I have found naive Bayes is defined for the following network structure:

I'm interested to understand what happens if instead from y->x the child will be y. As shown in the following:

I'm not entirely sure if we get the same results as a regular naive Bayes classifier (intuitively we don't), and if not how to estimate both the likelihood of the network and the parameters?

For further clarification: lets say there is a training set of size n; each data point consisting of (x,y) where x is a vector of size m of binary values and y is the class and is also binary. E.g. (0,1,0,1,1) where the last index is the class. I'm trying to figure out the likelihood of the network (where y is the child) and how to estimate the parameters in the case of y being the child.

To elaborate on my comment, the first image depicts conditional dependence of $X_i | Y$ for $i \in 1,...,m$.
Therefore, your likelihood for this model would be

$$
\begin{aligned}
P(Y = y) \times P(X_1 | Y = y) \times ... \times P(X_m | Y = y)
\end{aligned}
$$

The second image (bottom) depicts $Y | X_i$, therefore the likelihood would be

$$
P(X_1) \times P(X_2) \times ... \times P(Y | X_1, X_2, ..., X_m) \\
\ = P(X_1) \times P(X_2) \times ... \times P(Y | X_1) \times ... \times P(Y | X_m)
$$

Assuming that both scenarios do not have conditional independence, the two are not equivalent!


Data Visualisation Techniques for Multi Labelled data I am new to data science and am trying to figure out how to visualize my multi labelled data using graphs. I am using a dataset to classify music by emotion based on their acoustic features (such as: pitch, amplitude etc.).

So some have multi labelled emotion labels.

This is a snapshot of my dataset:

Please tell me any techniques for multi label classification visualization techniques. I searched all over the internet, but all of them are related to single label classification.

Are you looking for a Python or R package? Or would a tool like Tableau suffice?

If you feel comfortable using a Python package, seaborn has the ability to visualize multi-label classes.

Take a look at this link: https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff

Here, there's a good explanation of how to get this done.

However, if you want to use a tool to get your data visualized, look no further than Tableau. However, if you wish to do a lot of machine learning, this wouldn't be a right fit. Else, Tableau can visualize multi-label classification data. All you have to do is plug in your dataset as a data source and navigate to whichever chart preview suits your dataset. Also, Tableau neatly sorts your attributes into dimensions and facts; sometimes erroneously  might I add. In that case, all you have to do is a simple right-click and convert it to a dimension or vice versa.


Should we denormalize our data after normalization? If we use sklearn library's preprocessing.normalize() function to normalize our data before learning, like this:

preprocessing.normalize(training_set)
model.add(LSTM())

Should we do a denormalization to the result of LSTM to get predicted result in a true scale? If yes, how to denormalize?

This can be done in Python using scaler.inverse_transform.

Consider a dataset that has been normalized with MinMaxScaler as follows:

# normalize dataset with MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)

# Training and Test data partition
train_size = int(len(dataset) * 0.8)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t-50 and Y=t
previous = 50
X_train, Y_train = create_dataset(train, previous)
X_test, Y_test = create_dataset(test, previous)

# reshape input to be [samples, time steps, features]
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

Upon generating the predictions from LSTM, the same can be converted back using scaler.inverse_transform as follows:

model = Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X_train, Y_train, epochs=100, batch_size=1, verbose=2)
trainpred = model.predict(X_train)
testpred = model.predict(X_test)
trainpred = scaler.inverse_transform(trainpred)
Y_train = scaler.inverse_transform([Y_train])
testpred = scaler.inverse_transform(testpred)
Y_test = scaler.inverse_transform([Y_test])
predictions = testpred


how can LSTM output ever be negative with 0 initialization? From the lstm equations, e.g. as they appear on p406 of the Deep Learning Book, it looks to me like initializing with zeros (as is common practice), must always produce a strictly positive output. When $s_0 = 0$, then the $i$th output unit at the first time step can be written as
$h_i = \sigma(A)tanh((\sigma(B)(\sigma(C))$ where $A$, $B$ and $C$ are linear functions on the current input and the previous output. But isn't then $h_i > 0$ for all real numbers $A$, $B$ and $C$?

Cannot find the equations in your reference, so I take them from wikipedia. At first time step the following equation holds: $$h_{i} = \sigma(A)\tanh((\sigma(B)(\tanh(C))$$
where $A, B$ and $C$ are values calculated from linear functions. So it can be negative if $C$ is so.


Which classifier should I use for sparse boolean features? I have training data that is classified into 2 categories: X and Not X

Each piece of training or experimental data has a variable number of boolean features. Each piece of data may have ~100 features, but the total number of possible boolean features may be in the tens or hundreds of thousands.

I want to train a classifier so that given an observation (which has a set of boolean features), it will output the probability that it is X.

Could you recommend which ML algorithms could be used for a large number of sparse boolean features?

Follow-up question:

I also have a large amount of training data that would be best described as Maybe X (I haven't verified is X, but there's a good chance it is X). I could further segment those Maybe X's into Likely X, Maybe X, Probably Not X. Is there a way I can train with Maybe X or it's subclassifications? In the end, what I still only care for is the probability that an observation is X.

Mapping your inputs into a sparse matrix and using logistic regression with an $\ell_1$ penalty should work well. In R this is very straight forward with the glmnet and matrix libraries.

Additionally, if you have boolean features with a low frequency, you can filter some of them by taking a subset with at least some number of observations.

Here's a quick simulation in R of a lasso logistic regression with sparse features.

library(glmnet)
library(Matrix)

n <- 1e5
set.seed(420)

# These are all converted to boolean variables since they're arbitrary categories
ls <- data.frame(cat1=sample(letters, n, replace=TRUE),
                 cat2=sample(letters, n, replace=TRUE),
                 cat3=sample(letters, n, replace=TRUE))
ls$cat1cat2 <- factor(paste(ls$cat1, ls$cat2, sep="-"))
xs <- sparse.model.matrix(~.-1,data=ls)
betas<- rnorm(dim(xs)[2])
error <- runif(n)

ytrue <- xs %*%betas
ylogit <- 1/(1 + exp(- (ytrue + error))
ylabel <- ifelse(ylogit>0.5,1,0)

trainfilter <- runif(n)<=0.3
sparsemodel <- cv.glmnet(x=xs[trainfilter,],
                         y=ylabel[trainfilter],
                         alpha=1,
                         nfolds=3,
                         family='binomial')

preds <- predict(sparsemodel, xs, s='lambda.min', type='class')
print(table(ylabel, preds))

The classifier doesn't matter as much in this case as the underlying algorithm's ability to handle sparse matrices.

It's also worth noting that you can handle sparse boolean features by mapping them to arbitrary numeric indices and running them through a tree based algorithm (e.g., Random Forest or Gradient Boosting). In practice, this seems to work well.