Why is correlation between my independent variables helping my linear regression model? I am working with PUBG data and developing a linear regression model for the same ! Now there were three features in my original dataset, ridedistance, swimdistance, walkdistance. I combined the three with a new feature : distance covered which is the sum of the above mentioned three features. When putting it in a linearregression model, when I use the three features and the fourth one as well, I get a better score as compared to using the three featues only or using only the fourth feature. I have read that correlation between features when developing a model should not be there. But when all features (4 of them) having correlation are used to develop a model, the model has a better square (R-square). Why is this happening ?

It seems that you are dealing with the problem of Multicollinearity. Multicollinearity happens when your predictors are correlated with other predictors in the model.

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret.

You can use adjusted R-squared to see if the new added variable is actually helping your model to better explain the variance.


How to get Real life Social Network Data? I am trying to build a case study for Insurance fraud detection using SNA. I have already looked around and build my model using Stanford's SNAP and other similar databases available on the internet for research. However, I am just wondering, what are the data sources from which I can get Social Network Data apart from the Facebook, Linkedin, Twitter connections.
Can I get (buy) phone call records, or any other social network dataset?
Also, do FB, linkedin sell their connections data?

Take a look at these links:

https://opendata.stackexchange.com/questions/1670/any-cdr-call-data-record-dataset
https://stackoverflow.com/questions/26229570/dataset-needed-for-cdr-analysis
http://vlado.fmf.uni-lj.si/pub/networks/doc/seminar/nicta02.pdf


Trained CNN individually on multiple images to classify them, how can I now classify a related "set" of these images that correspond to one object? I have a N object classification examples, each example consisting of a set M individual images of the object at different angles. I've trained M CNNs with the dataset of one particular image angle each and their corresponding label. (Thus I have M sets of model parameters I've discovered for each angle)

Now given this information, what is a good approach to classifying a new single object based on it's set of M individual image angles? (i.e I can classify with some decent accuracy what each individual image is in the set, but is there a method I can use the feature-dependency of images that belong to the same unique example set to make an overall "better" prediction than just taking the average of all the individual probabilities?)

As far as I understand your problem is related to ensemble learning: You ensemble is a set of models trained on detecting the class of an object on images with the same angle on the object, and now you want to classify an object and choose the model voting of the model trained on the correct angle?
In that case, I would choose the prediction of the model which has the most unambiguous output before outputting a binary label.
So it's probably related to these topics:

Majority Voting in Ensemble Learning
Ensemble Learning in Machine Learning | Getting Started


CNN: How do I handle Blurred images in the dataset? I have 30% blurred images in each classes. I have a total of 10 classes. I'm not allowed to drop these blurred images. How do I train the model to get better accuracy for both blurred and nonblurred training dataset ? Currently, I'm at 11% accuracy.
The images were blurred using a Gaussian blur.
I have used a Wiener filter, but not able to restore the image from blurred images.
Please can anyone suggest a good way to train the model.

I will suggest using data augmentation approaches to even out your data distributions. It will make your blurred images more usable to the model
The data distribution of 30% of your images deviates from the rest because they are blurred. Experiment with training using random blur with appropriate min-max ranges in the data augmentation pipeline (on the images that aren't blurred). This will help the model to smoothly generalize across blurred images. If you don't have labels of which images are blurred, use blur detection algorithms to determine a threshold after which you want to augment.
After doing this, it may be important that you do test-time data augmentation as well.


Alternative to Flesch for a readability score algorithm Someone I know has been using the Flesch-Kincaide readability score algorithm to try to gauge how well students write as a predictor of their academic success. They've been mostly unsuccessful.

While taking a UCSD Coursera course I recently learned about this algorithm and I believe that their lack of success is just a basic misunderstanding of how the algorithm works. A low score does not imply poor writing -- anything written for college level reading ability, such as U.S. law and classic literature, receives a low readability score. High readability scores imply more simplistic writing, but not that the writing is more or less correct. So it's not really measuring much in terms of proficiency.

All that is to ask this -- is there some other algorithm or metric to gauge writing quality, which would be more useful? Something that perhaps measures proper grammar and/or punctuation?

I have been and will continue to search for this myself, but if anyone has any knowledge they can share it would be greatly appreciated.

ARI and Coleman-Liau are basically the same thing with different numbers.

Gunning Fog and Dale-Chall are similar in that they're simple functions of how "complicated" the words and sentences are, though at least they appeal to a notion of "complicated words", though that just pushes the problem somewhere else.

There are more and more like this, and I know you're not looking for this.

It may be stating the obvious, but if I were building such a measure, I'd look at things like misspellings and grammar errors. These are fairly possible to check automatically.

Given the success of RNNs in reproducing text in the style of given input, I suspect you could quite successfully apply deep learning to learn what about a text makes it readable, or at least find whether models trained on high or low readability texts seem to find a given new input more probable.


Distribution plot of word embeddings I have a list of sentences and a 10 dimensional embedding for each of the sentence.   I am trying to visualize these embeddings so that i can see if several sentence embeddings form a cluster such that these have similar embeddings which in turn mean similar context's. How can i do this in python? Also, i am wondering if there's any good metric or method using which i can form cluster's or calculate similarity between sentences that can tell me how close two sentences are?

One method to visualize high-dimensional data in lower dimensions is t-sne, which has a python implementation in scikit-learn (here)

To calculate similarity between two sentences, having their embeddings, its common to use the cosine similarity


Loss function for optimising precision & recall / sensitivity & specificity? I've been using precision and recall as my metrics, as per keras-team/keras/pull/9393/files

Sensitivity & specificity is what I want to optimise for. Every epoch I output it:

class SensitivitySpecificityCallback(Callback):
    def on_epoch_end(self, epoch, logs=None):
        if epoch:
            x_test = self.validation_data[0]
            y_test = self.validation_data[1]
            predictions = self.model.predict(x_test)
            output_sensitivity_specificity(epoch, predictions, y_test)

def output_sensitivity_specificity(epoch, predictions, y_test):
    y_test = np.argmax(y_test, axis=-1)
    predictions = np.argmax(predictions, axis=-1)
    c = confusion_matrix(y_test, predictions)
    print('Confusion matrix:\n', c)
    print('[{:03d}] sensitivity'.format(epoch), c[0, 0] / (c[0, 1] + c[0, 0]))
    print('[{:03d}] specificity'.format(epoch), c[1, 1] / (c[1, 1] + c[1, 0]))

On my 203rd epoch I get a great result, then it's all downhill—in the wrong direction!—from there.

How do I optimise for sensitivity and specificity? - Thinking to update the weights, and/or develop a custom loss function…

The comments raise two important points:

you can't directly optimize for sensitivity or specificity
you should save the model at the best epoch and use that one

Below I explain something that is not directly related, but might be interesting for you:
You can add the weights to the loss of some examples. For example, you can increase the weight of positive examples which means that the model will focus more on correctly classifying positive examples at the potential expense of negative examples. This is especially useful if your data is suffering from class imbalance.


Are time series algorithms immune towards collinearity? I have a time series dataset with 63 features and a univariate dependent variable. This is my first major time series project, so I was wondering if algorithms like ARIMA and LSTM are immune towards variables that have collinearity, similar to gradient boosted methods such as xgboost that is actually seen as a robust method. Below is the image of the correlation matrix. I have used xgboost with the dataset and have not got satisfactory results, even after removing some of the variables that are in the red category, so I'll proceed with the next step by using some time series methods.

The image can always be zoomed to get a clearer description of features.

The fact that some variables may be correlated among each other cannot be the source of your unsactisfactory results. Think about it: let's say that I'm predicting the wheather, having as variables v1 the number of fingers in my hand v2 in my foot (highly correlated variables, right?). Removing v2 from the model may help a little bit because my model won't consider a noisy variable to predict the wheather.

If you are asking about colinearity, I would say that it's better to remove correlated variables, but it's not mandatory. If you are looking for better results in your time series predictor, you may want to understand how forecasting models, such as ARIMA, works in-depth. Often, this kind of models uses the past (with time series "lags", i.e. previous observations) to predict the future. Try to model lag variables (also referred to autoregressive components), moving averages (trends) and seasonality components. Although they can be very correlated between each other (look for ACF plots), they are also suitable variables for forecasting models.


Can I save prediction value in same csv file as a another column using panda python I have csv data file and I design LSTM model to predict values. Then I want to save that prediction value in same csv file. Can I do that? I tried using one code then in my csv file only had prediction values and delete other columns. Can anyone give me a suggestion for that.

import pandas as pd
import numpy as np
data = pd.read_csv('data1.csv')
data = pd.DataFrame(data,columns=['x','x1','x2','y'])
data.columns = ['x', 'x1', 'x2','y']
pd.options.display.float_format = '{:,.0f}'.format
data = data.dropna ()
d = ['y']
y=data['y'].astype(int)
cols=['x', 'x1', 'x2']
x=data[cols].astype(int)
scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
x = np.array(x).reshape ((len(x),3 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)
print("row",len(y))
n = data.shape[0]
p = data.shape[1]
fill_missing(data.values)
train_start = 0
train_end = int(np.floor(0.65*n))
test_start = train_end+1
test_end = n
x_train = x[np.arange(train_start, train_end), :]
x_test = x[np.arange(test_start, test_end), :]
y_train = y[np.arange(train_start, train_end), :]
y_test = y[np.arange(test_start, test_end), :]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
seed = 20
np.random.seed(seed)
fit1 = Sequential ()
fit1.add(LSTM(
output_dim = 10,
activation='relu',
input_shape =(3,1)))
fit1.add(Dense(output_dim =1))
fit1.add(Activation(linear))
batchsize = 10
fit1.compile(loss="mean_squared_error",optimizer="adam")
fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =10,   shuffle=True)
print(fit1.summary ())
pred1=fit1.predict(x_test)
pred1=fit1.predict(x_test)
real_test = scaler_y.inverse_transform(np.array(y_test).reshape ((len(y_test), 1))).astype(int)
pred1 = pd.DataFrame(pred1, columns=['pred1']).to_csv('data1.csv')

if you want this column in the same dataframe just do

data['pred'] = pred1
data.to_csv('data1.csv')

The first line automatically adds a column called 'pred' to the dataframe with values coming from pred1.

Hope it helps. Good luck!


How to train NER LSTM on single sentence level My documents are only a single sentence long, containing one annotation.
Sentences with the same named entity of course are similar, but not context-wise.
NER training examples (afaik) always has documents sequentially related, aka the next document is context-wise related to the previous document. Consider the example below. The first sentence is about the US, with location annotations. The second sentence is about an organisation but still related to the previous.

The United States of America (LOC), commonly known as the United States (U.S. or US).
The Bank of America (ORG) is a multinational investment bank.

My dataset for example would be:

The United States of America (LOC), commonly known as the United States (U.S. or US) or America (LOC).
The Netherlands (LOC), informally Holland, is a country in Western Europe.
Peter (PER) works at the harbour.

The sentences are not related. When considering a bi-lstm, should I somehow separate the two sentences during training? so that it doesn't think that the annotation of the current sentence is related to the previous sentence too?
Take for example the dataset previewed in this Kaggle notebook. It has a sentence_idx to separate sentences by ID but other than that its just one gigantic long list of words (with features). What happens when a (bi)-lstm finds itself in a completely different context, where the current sentence has absolutely no relation to the previous.
Normally I suppose this isn't a problem because documents are very large, but mine are just a single sentence. Not sharing a context, except that, for example, sentences with a LOC annotation are of course about locations, but not a specific context.
I had a very difficult time describing my problem, questions and edits to make it more clear are very welcome. I believe a similar question is: How to train NER LSTM on single sentence level

Looks like this is done by the hyper-parameter called timestemps or sequence length. The steps/length indicates when the cell state is reset. Which can be set to a sentence length.

Just have to figure out whether this can be a variable length or the sentences should perhaps be padded?


Emotion detection on audio I'm trying to apply emotion detection on audio. Basically, given an audio file, it must generate emotional labels for a segment of any durations, say every 5s chunks, like happy, sad, angry, etc.
Is there a working example with pretrained models available to test? I was looking for a sample on GitHub, but had problems running it. I need a relatively accurate model. Any working sample would be helpful.

Sure, here is the github and here is the explanantion.


Is my dataset unbalanced? So I'm trying to implement Nvidia's end to end driving paper to simply have an agent in Carla follow the lanes. I'm trying to predict the steering angle of the car based on the RGB images from the front camera.

I'm getting and r_score of about 1.20 %

Every image in the dataset has a steering angle.

So here are plots for the distribution of steering angles.

I've also augmented the images but when I run it, the car still fails to follow the lane.

I'm being wrong or should I take another approach? Thanks :)!

To answer your question no.

The term "imbalance" usually refers to classification problems. For your case, i.e. a regression problem you can only look at the distribution of your target variable.

If by "balance" you mean them having a uniform distribution, you could argue that they are, if fact imbalanced. However, I'd argue that this is not the problem here. When steering you rarely need an extreme angle, I think this dataset represents this well.

Your goal is also a bit unclear. Are you trying to "predict the steering angle" or  "follow the lane"? If it is the first, I'd first suggest changing your metric. Use one that tells you actually how well you are doing, e.g. MSE, MAE.

A lot of things could be to blame for a poor performance. I don't thing the distribution of the angles is one of them.


How to estimate the mutual information numerically? Suppose I have a sample {$z_i$}$_{i\in[0,N]}$ = {($x_i,y_i$)}$_{i\in[0,N]}$ which commes from a probability distribution $p_z(z)$. How can I use it to estimate the mutual information between X and Y ?

$MI(X,Y) = \int_Y \int_X
                 p_z(x,y) \log{ \left(\frac{p_z(x,y)}{p_x(x)\,p_y(y)}
                              \right) }$

where $p_x$ and $p_y$ are the marginal distributions of X and Y:

$p_x(x) = \int_Yp_z(x,y)$

$p_y(y) = \int_Xp_z(x,y)$.

Binning

One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that fall within each bin to obtain a density. So, the calculation would be:

$$
    \sum_{b_x \in Bins_x}
    \sum_{b_y \in Bins_y}
    \frac {\#(b_x, b_y)} N \log \frac {\frac {\#(b_x, b_y)} N}
      {{\frac {\#b_x} N} {\frac {\#b_y} N}}
$$

where $\#(b_x b_y)$ is the number of samples where $X \in b_x$ and $Y \in b_y$, $\#b_y$ is the number of samples where $Y \in b_y$ and $\#b_x$ is the number of samples where $X \in b_x$.

Entropy and Density Estimation

Another method is to note that the mutual information can be represented as

$$
    I(X, Y) = H(X) + H(Y) - H(X,Y)
$$

so if you can estimate entropy you can estimate information. Now, in order to find the expectation of a function over your data you can just use the plugin estimator and do

$$
    E[g(X)] = {\frac 1 n} \sum_i g(X_i)
$$

The problem here is that we want to estimate the function $\log p(x)$ where $p$ depends on our data.

So one can use an estimate of the density of $p$ for each point and then use the plugin estimator. Kernel density estimators are one approach and nearest neighbor estimators are another. Both methods are non-parametric.


How to represent relation between users as a feature? I'm developing a model for unsupervised anomaly detection. I have a dataset representing communications between users (each example represents a communication): there are many features (time, duration, ...) and the ids of sender and receiver. My question is: how to represent the link between those two users?
I have several ideas, but each of them seems to have serious drawbacks:

Use id as is. Drawback: even if ids are integers, they have no numerical sense (id 15 is not 3 times id 5) and I think this may mislead the system
Use sort of vectors: for example, with 3 users: user1 = (0 0 1), user2 = (0 1 0), user3 = (1 0 0). Drawback : the number of users may vary over time, thus the number of features would vary as well and I would have to re-train my model.
Graph theory: I've heard of that way of representing data, which could fit perfectly my data model. Drawback: I've absolutely no knowledge in graph analysis
Assign each user a id which is a prime number. That way a communication could be represented in an unique way as the product of the 2 ids. Drawback: as for point 1, ids do not have a "numerical sense"

What do you think may be the better way to represent these relations?

There's a couple of approaches you can take depending on the nature of your data. It sounds like you're trying to detect social anomalies in your data so you need to model the communication boundaries between them which leads to some sort of graph representation.

If you don't have too many users in the system (say $n$) then you can create an $n\times n$ matrix, $M$ over a time period that represents the communications between users. The component $M_{ij}$ could be either $1$ or $0$ if user $i$ and $j$ communicated or the number of times that they communicated.

If you have more data then you would want to represent the data in terms of nodes and edges. Nodes would be the users and edges would be the presence of a communication. This can be done manually or by using a library such as NetworkX.

Here's a Python tutorial on getting started with graph network analysis.

If you're doing this at a large scale then you might want to use a graph database such as Neo4J.


Reducing sample size I have a large dataset (around $ 10^6 $ samples) and an algorithm that will surely choke on that much data.

Suppose that I have removed duplicates and near-duplicates. What are the well-known techniques for reducing sample size without losing too much of the information possibly encoded in the initial dataset?

I thought about using some clustering algorithm (which scales well with respect to number of clusters, possibly BIRCH) and use the resulting clusters to find $ N $ nearest points to cluster centroid. However this feels somehow wrong.

One method would be to take many subsets of your dataset, i.e.
bootstrapping, build your models, perform cross-validation and calculate the average performance.  This is a good explanation of how the amount of data affects the model outcomes: https://stackoverflow.com/questions/25665017/does-the-dataset-size-influence-a-machine-learning-algorithm

Play around with the size of your subsets until you start getting stable results.


How to stem plural words properly? I'm looking for a way to avoid removing ending s when s isn't a suffix. In order to do that, I first check if a word exists in my index, if it does, I don't remove the ending s but If it doesn't, I go on and remove the ending s and add it to the index. But the problem is what to do when starting to build the index.
Imagine we encounter books, I remove s and add book to my index. On the other hand, I may encounter dangerous for the first time, since it doesn't exists in my index yet, I remove s and add dangerou which is obviously wrong. What should I do?
Specifically I'm looking for ways to properly detect if suffixes and prefixes are indeed one or part of the original word. one way that comes to my mind is using a formal dictionary and instead of my own index, check the words in that dictionary.
P.S: I'm not working on English docs. It's a college/prototype thing Therefore I'm looking for general, good ideas with good accuracy. I'm not looking for advanced stuff with superb accuracy and considerable complexity.

It seems that you are collecting the lemmas in your docs. For that, you need a lemmatizer.
If available for your language, you should use an external lemmatizer. Some packages supporting lemmatization for different languages are StanfordNLP (or its equivalent for Python, Stanza), Spacy or NLTK.
Depending on the language, the approach to get a good lemmatization varies, but many times it involves expressing the language morphological knowledge as rules.
If no lemmatizer or stemmer is available in the language you are working with, another approach would be to use unsupervised approaches to segment words into morphemes, and use your linguistic knowledge of Georgian to devise some heuristic rules to identify the stem among them.
This kind of approach consists of a model trained to identify morphemes without any labels (i.e. unsupervisedly). The most relevant Python package for this is Morfessor. Also, there is a Python package called Polyglot that offers pre-trained Morfessor models in different languages, so you should check if yours is included.


Plotting two pandas data frame columns against each other I have a data frame like this:
timestamp                  vote_count
00:12:30                   350
05:05:00                   50
01:10:35                   200
05:05:00                   175
..............................
..............................
00:12:30                   900

Now I want to visualize the vote_count for the timestamps and do some analysis on that further. How can I plot the two columns against each other using matplotlib or seaborn?
Note: The timestamp is in 24hr format. Also, at any timestamp, there can be multiple vote counts.

You need to groupby to deal with multiple vote counts:

  df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')


How to use the finite difference to compute gradient for very complex loss function I need to put the prediction into a complex function to calculate loss.
This means that i can't build the loss function by tensorflow's operator and can't get  gradient from Automatic Differentiation.

I want to get gradient from finite difference in my training step.

You may consider using a score function estimator (also known as REINFORCE), which defines an estimator of the gradient of a scoring function that does not need to be differentiable. This can be achieved thanks to the log derivative trick.

You may find its mathematical foundations in the original REINFORCE article by Williams or in Ian Goodfellow's seminal book on Deep Learning (section 20.9.1 Back-Propagating through Discrete Stochastic Operations).


Why don't companies use on-device ML to serve ads to protect privacy? Eg: The ad inventory can be stored in the cloud. Using on device ML, a process then matches the ad to the user in order to serve the best ad to the right user.

Are there any limitations to this? (I am planning to make an app that serves ads using TFLite and CoreML so the user privacy could be protected.)
Can models in TFLite be updated with user data on-device, helping models stay relevant to user behavior without compromising privacy like Apple's CoreML?

There are several problems:

The problem with serving ads on server side is that they process information about the users, so it can interfere with their privacy, or even break some privacy laws in some cases.
The suggestion to move the algorithms to client side (e.g. mobile app) solves this issue, but produces the opposite problem: now you need to upload all the data about the adds to client-side app. This may be problematic because this could expose the corporate secrets of your company (who are your customers, what kind of adds they want to show to users, the target group for the ads). Your customers, who paid for the ads may not like it. Your competitors may potentially abuse this to gain advantage over you.

Another problem is that serving ads is about matching users to ads. On client side you could only match ads to user. Imagine that your customer pays you for showing 10K ads to the users, if you assigned ads on user side, you would not have direct control on showing exactly 10K ads, so you could loose money by showing the ads that were not paid for. Sure, there are ways around this, but this is a problem that needs consideration.
In real-life this gets even more complicated, because the ads are usually served in terms of betting (who pays more, gets their ad shown), where different customers bet to make ads and algorithm decides about most optimal share, such that it maximizes the gain from showing the ads. In such case, not only you don't have control over global gain, since you are making only local optimization decisions, but you would need to send the bets to client-side app, so again share sensitive information about your customers (how much do they pay for the ads).

Third thing is that even if the client-side app made a decision, it would probably still need to query ads database to download the user-matching ad. I guess, you could backward-engineer this to learn the characteristics of the user given the ads that were assigned to them by the algorithm, so in the end, this might not protect privacy.
As alternative, you could send all the ads to user and store them in the app database, but then you would need user to store potentially large amounts of your data just that you could show them your ads. I doubt users would like this.

Finally, I'm not sure if ad selling businesses is the most concerned about user privacy, i.e. if they would care.


Dealing with training set of questionable quality Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into the training set is often questionable (I don't even know how to start separating good from bad ones). I have spent much more time trying to pre-process the files in order to increase the validity of the data, than actually building the prototype.

I have only dumped a part of the data that is claimed to be better. There are still data from other sources to complement the missing part in the first set of training data. Given the amount of data, it is practically impossible for me to spend too much time on them.

So the question is, how do people deal with really horribly, inconsistent data in real life (assuming I am working with addresses, where people often write the wrong postcode due to bad city planning or lack of clear instruction, misspell the name of neighbourhood, plus data got "autocorrected" by excel such that "1-2" becomes "1-Feb", etc. etc.)? Specifically, how do I ensure the quality of the train classifier given the training set (and cross validation set) has questionable quality?

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online.  Those are super polished and refined.  But, the real world data is not so.

The step where one does the cleaning and scrubbing is called the data pre-processing step.

So, some nice data cleaning techniques, in addition to @jknappen's excellent answer are:

Elimination of zero variance columns/predictors: These columns are not important, and they cause the model and the fit to crash and leak errors. So, eliminating them would make complete sense.
Correlated Predictors: Reducing the level of correlation between the predictors would be a very nice step in the pre-processing process.
Scaling: You must be knowing why scaling is important during pre-process.
Predictor Transformations

A nice reference from Kaggle forums where the pre-processing and cleaning of data sets is discussed.


Groupby date and find number of occurrences of a value a in another column using pandas My Dataset is looking like this

I would like to get the output something like this.

I need to group by date and find the occurrences if each feedback. In this case, i have 5 feedback types (namely 5,4,3,2,1)

This can work:

import pandas as pd
import numpy as np

unstacked_df = stacked_df.unstack()

The next time you post, you could post a reproducible code so we could help you a little more.


Is removing poorly predicted data points a valid approach? I'm getting my feet wet with data science and machine learning. Please bear with me as I try to explain my problem, I haven't been able to find anything about this method but I suspect there's a simple name for what I'm trying to do. To give some idea about my level of knowledge, I've had some basic statistics training in my university studies (social sciences), and I work as a programmer at the moment. So when it comes to statistics I'm not a complete layperson, but my understanding leaves a lot to be desired.

I need to make predictions on a certain dataset, all features and the target are continuous variables (interval/ratio level), and I'm using a regression multi-layer perceptron with a single hidden layer. The $R^{\textrm{2}}$ is relatively low when fitting on the entire dataset, so I was hoping to improve the predictive power by doing cluster analysis and fitting multiple regressors, on each cluster separately. This didn't work for me, the highest cluster $R^{\textrm{2}}$ was the same as for the entire set, so I dropped that approach. I've tried multiple clustering algorithms, haven't noticed much of a difference among them, but I haven't done an exhaustive search, so I may have missed something.

What I ended up doing is, I fit the model on the training subset, identify the data point for which the prediction error in the test subset was greatest, and remove that one data point. The train-test split is done randomly, so the probability of some "bad" data point remaining by chance is pretty low. I repeat this process until I get an $R^{\textrm{2}}$ I'm satisfied with, after which I designate the remaining data points as belonging to e.g. "Group 1". This whole process is then repeated on the entire dataset minus Group 1, and ultimately all of the data should be divided into groups inside which reasonably reliable predictions can be made. To give some idea about the data: $R^{\textrm{2}}$ on the entire set of about 11000 data points hovers around 0.7. In Group 1, I kept 7000 data points for which I can get up to 0.9. The remaining groups also have acceptable (for my standards) $R^{\textrm{2}}$s.

After all of the data is split into groups in this way, I expect to be able to train a classifier on the features, to predict the group that a certain data point belongs to, and use the appropriate regression model for that group to predict the final target variable. My question is, is there some methodological flaw in this approach, specifically the removal of certain data points based on  prediction errors? Am I introducing some artefacts into the data, or something like that? As far as I can tell there's no leakage of information about the target. What this looks like to me is, some roundabout way of doing cluster analysis, as it seems to exclude "outliers" in a certain sense, but I couldn't be more precise, and that may not be the case at all.

Note: all mentions of $R^{\textrm{2}}$ refer to predictions, i.e. the score I get on the test set, not training.

EDIT: I remove data points only from test subset (which changes randomly in each iteration), but now that I think about it I guess there's no reason to limit exclusions to the test subset, as a worse prediction may happen in the training subset. I'll update when I try it. Also, I haven't yet tried fitting the classifier, so I suppose I may not get any better results with this final model; regardless of the results I get, I'm interested in the validity of this approach. Also, if someone knows of a theoretical limitation here, if there's a reason why I couldn't get better results in principle with this approach, I'd like to know.

The process you described is not valid, it is almost certainly overfitting. However the overall idea may very well be valid. What you should do instead is take a portion of your dataset as the testing which is not looked at during algorithm development, and split the rest between training and validation sets however you please.

Looking only at the training and validation sets, develop your entire end to end algorithm, including the classifier!

Once you are happy with the algorithm, try running it on the test set to get a more accurate measure of performance.


How to customize word division in CountVectorizer? >>> from sklearn.feature_extraction.text import CountVectorizer
>>> import numpy
>>> import pandas

>>> vectorizer = CountVectorizer()
>>> corpus1 = ['abc-@@-123','cde-@@-true','jhg-@@-hud']
>>> xtrain = vectorizer.fit_transform(corpus1)
>>> xtrain
<3x6 sparse matrix of type '<class 'numpy.int64'>'
    with 6 stored elements in Compressed Sparse Row format>

>>> xtraindf = pd.DataFrame(xtrain.toarray())
>>> xtraindf.columns = vectorizer.get_feature_names()
>>> xtraindf.columns
Index(['123', 'abc', 'cde', 'hud', 'jhg', 'true'], dtype='object')

I see that the special characters(-@@-) are omitted and "abc" and "123" are considered seperately. But, I want "abc-@@-123" to be treated as a single word. Is it possible to achieve? If yes, how?

Any help would be much appreciated.

It's possible if you define CountVectorizer's token_pattern argument.

If you're new to regular expressions, Python's documentation goes over how it deals with regular expressions using the re module (and scikit-learn uses this under the hood) and I recommend using an online regex tester like this one, which gives you immediate feedback on whether your pattern captures precisely what you want.

token_pattern expects a regular expression to define what you want the vectorizer to consider a word. An example for the string you're attempting to match would be this pattern, modified from the default regular expression that token_pattern uses:

(?u)\b\w\w+\-\@\@\-\w+\b

Applied to your example, you would do this

vectorizer = CountVectorizer(token_pattern=r'(?u)\b\w\w+\-\@\@\-\w+\b')
corpus1 = ['abc-@@-123','cde-@@-true','jhg-@@-hud']
xtrain = vectorizer.fit_transform(corpus1)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = vectorizer.get_feature_names()

Which returns

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud'], dtype='object')

An important note here is that it will always expect your words to have -@@-  nested in your tokens. For instance:

corpus2 = ['abc-@@-123','cde-@@-true','jhg-@@-hud', 'unexpected']
xtrain = vectorizer.fit_transform(corpus2)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = vectorizer.get_feature_names()
print(xtraindf.columns)

Would give you

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud'], dtype='object')

If you need to match words that don't have that exact special character structure, you can wrap the string of special characters in a group and use the non-matching group modifier ?:

more_robust_vec = CountVectorizer(token_pattern=r'(?u)\b\w\w+(?:\-\@\@\-)?\w+\b')
xtrain = more_robust_vec.fit_transform(corpus2)
xtraindf = pd.DataFrame(xtrain.toarray())
xtraindf.columns = more_robust_vec.get_feature_names()
print(xtraindf.columns)

Which prints

Index(['abc-@@-123', 'cde-@@-true', 'jhg-@@-hud', 'unexpected'], dtype='object')

I hope this helps!


Sparse connections in feedforward network tensorflow or pytorch? I want to create sparse feed-forward networks in Pytorch and Tensorflow, i.e., say each node is only connected to k number of neurons of the next layer where k is strictly less than the total number of nodes in the next layer. But all the tutorials/examples I have seen so far are for fully connected feed-forward networks. Is there any way to construct such sparse networks in these tools?

We don't use sparse connections in feedforward to make use of an efficient matrix product operations.
But If you want to model that aspect, see dropout


how to input the data set in to a word2vec by keras? I am new in using word2vec model, as a result, I do not know how I can prepare my dataset as an input for word2vec?
I have searched a lot but the datasets in tutorials were in CSV format or just one txt file, but my dataset is in this structure: 2 folders one of these is blood cancer and the other one is breast cancer.
each folder contains 1000 txt files which contain 40 sentences.
I do not have any idea about I can create a vocabulary as an input for the word2vec model in keras with tensorflow backend?
I use python 3.5 in ubuntu 17.10
Any guidance will be appreciated.

I have already searched for the solution and found this statements which are proper for this kind of datasets:
at first, you should concatenate 2 folders in one folder to apply the code below:

import os
import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',      level=logging.INFO)

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = './Dataset#2_BinaryClassClassification/lupus_alldataset/'

    def __iter__(self):
        for fname in os.listdir(self.dirname):
           for line in open(os.path.join(self.dirname, fname)):
               yield line.split()

sentences = MySentences('./DatasetBinaryClassClassification/alldataset/') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences,
    size=200,
    window=10,
    min_count=2,
    workers=5)

I found the 'class MySentences' in this site:enter link description here

I hope it is helpful.


Value Function of Generative Adversarial Network My question concerns the notation used in the Value function of  a GAN

Does "x ~ p_data(x)" mean  $$ E_{x\sim p_{data}}[log(D(x)]= \sum_{x} logD(x)(x)p_{data}(x)$$  ??

Many thanks,

Your notation is a little confusing, but I suspect this is because you're not reading the original equation exactly right. $\mathbb{E}_{x \sim p_{data}(x)}$ means "the expectation over $x$ drawn from the distribution described by $p_{data}(x)$". It looks like you're trying to multiply the expected value of $x$ times $\log(D(x))$ for all $x$, which isn't what's going on here and doesn't quite make sense in this notation.

The way to understand this formula is to relate it to the goal of the GAN paradigm. The first term represents "the expected ability of the discriminator to tell that real data is real". The second term means "The expected ability of the discriminator to tell that the generated data is generated". The goal is to maximize with respect to $D$ and then minimize with respect to $G$; this means "we want our discriminator to be really good at telling fake (generated) data from real data" and then "we want our generator to be really good at fooling the really good discriminator".

Hope this helps, good luck!


Plot dataframe with two columns on the x axis how would I plot the data below with the x axis as Year & Month? Each Year-month combination has a unique monthly count (Y). I am unsure how to proceed, given than they are in different columns, should I combine them first?

Using matplotlib, you could create a custom tick formatter to show the right ticks. The year and month can either be fetched from the dataframe via the index (df.iloc[value]['Month']) or just be calculated.

Here is an example. You can also read the month name in the status bar when you hover over a position in the plot. The xticks (the positions where to have a tick with a label) can either be set automatically or manually.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

month_name = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']

def format_func(value, tick_number):
    value = int(value)
    year = 2001 + value // 12
    month = value % 12
    # return f"{month+1}-{year}"
    return f"{month_name[month]}-{year}"

N = 229
# create a dataframe resembling the example
df = pd.DataFrame({'Year': np.repeat(range(2001, 2021), 12)[:N],
                   'Month': np.tile(range(1, 13), N // 12 + 1)[:N],
                   'monthlycount': np.random.binomial(5000, .7, N)
                   })

plt.plot(df['monthlycount'], marker='o', c='crimson')
plt.xlim(0, N - 1)
plt.xticks(range(0, N, 12), rotation=20) # set major ticks every year
# plt.gca().set_xticks(range(0, N, 12), rotation=30)
plt.gca().set_xticks(range(0, N), minor=True)
plt.gca().xaxis.set_major_formatter(plt.FuncFormatter(format_func))
plt.tight_layout()
plt.show()


Can theano work on mapreduce or on spark? I am not sure whether the Theano library can be used to write parallelized code in map reduce or in spark. Any expert opinion is welcome. A discussion was on at:
Theano-dev

I don't think theano have spark support yet, however there are at least 2 deep learning libraries that are good with that. The first one is mxnet, they have support for spark, R, python and C++. Another option is deeplearning4j, made in Java with direct access to Spark.


Dimensionality reduction to correlate large number of variables So I have this dataset with about 750 variables (columns) and 50,000 rows of entries. I would like to reduce the dimensionality of the dataset to say 25-50-100 dimensions and then compute a correlation matrix between these dimensions. I have continuous and discrete columns. I would also like to be able to know which dimensions represent which bundle of single variables in the original dataset. I should add that I have no output variable, I am just trying to find correlation in the data. Is there a way to do this in Python?

Yes, in scikit-learn, you can find the correlation between the elements using LedoitWolf Estimator. For dimensionality reduction, I assume you will use PCA but then, you want to backtrack the reduced to the original data, for that I don't have a solution, since, PCA transforms your data and computes the largest variance direction. But, yes both of these won't require any labels.


Linear Regression finding best fit I am trying to fit a LR model with an obvious objective to find a best fit. model which can achieve lowest RSS.

I have many independent variable so i have decided to yous Backward selection (We start with all variables in the model, and remove the variable with the largest p-value—that is, the variable that is the least statistically significant. The new (p − 1)-variable model is fit, and the variable with the largest p-value is removed. This procedure continues until a stopping rule is reached.) to fit the model.

This is a preview of my model fit

after fitting my model i started eliminating all the variables with with high p values.

The Adjusted R squared and RSE are almost the same in both cases indicating little to no improvement.

How should i approach it further??

Solution: Residual Plots

 What is R2

The definition of R-squared is fairly straight-forward; it is the percentage of the response variable variation that is explained by a linear model.

R2 = Explained variation / Total variation

R2 is always between 0 and 100%:

0% indicates that the model explains none of the variability of the response data around its mean.
100% indicates that the model explains all the variability of the response data around its mean.

Limitations

R2 value has limitations. You cannot use R2 to determine whether the coefficient estimates and predictions are biased, which is why you must assess the residual plots.

R2 does not indicate if a regression model provides an adequate fit to your data. A good model can have a low R2 value. On the other hand, a biased model can have a high R2 value!

Interpreting Residual Plots

A residual is a difference between the observed y-value (from scatter plot) and the predicted y-value (from regression equation line).

A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis. If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data; otherwise, a non-linear model is more appropriate.

 Good Fit

 Bad Fit

An unbiased model has residuals that are randomly scattered around zero. Non-random residual patterns indicate a bad fit despite a high R2.

High R2 and Bad Fit Example

Refer fitted line plot and residual plot below. It displays the relationship between semiconductor electron mobility and the natural log of the density for real experimental data.

Here R-squared is 98.5%. However, look closer to see how the regression line systematically over and under-predicts the data (bias) at different points along the curve. You can also see patterns in the Residuals versus Fits plot, rather than the randomness that you want to see. This indicates a bad fit. Always check residual plots!

 Source and References

Stattrek.com. (2010). Residual Analysis in Regression. [online] Available at: https://stattrek.com/regression/residual-analysis.aspx.
Roberts, D. (2019). Residuals - MathBitsNotebook(A1 - CCSS Math). [online] Mathbitsnotebook.com. Available at: https://mathbitsnotebook.com/Algebra1/StatisticsReg/ST2Residuals.html.
Coursera. (2018). Model Evaluation using Visualization - Model Development | Coursera. [online] Available at: https://www.coursera.org/learn/data-analysis-with-python/lecture/istf4/model-evaluation-using-visualization [Accessed 9 Jan. 2020].
Minitab Blog Editor (2013). Regression Analysis: How Do I Interpret R-squared and Assess the Goodness-of-Fit? [online] Minitab.com. Available at: https://blog.minitab.com/blog/adventures-in-statistics-2/regression-analysis-how-do-i-interpret-r-squared-and-assess-the-goodness-of-fit.
Frost, J. (2019). Jim Frost. [online] Statistics By Jim. Available at: https://statisticsbyjim.com/regression/interpret-r-squared-regression/.


Dealing with NA values while using Rotation Forest in R I am using Rotation Forest in r.
I have a very big list (which I can't give a sample) that has 50 attributes and 4000 rows.
for every row there is a few missing values in the attributes, it supposed to be like that and I cant fill the missing values.

this is my code:

y <- list[,1]
x <- list[,2:ncol(list)]
rF <- rotationForest(x,y,K = round(ncol(x)/3),L=15)

this is the error:

Error in cov.wt(z) : 'x' must contain finite values only

from what I gathered the problem is with NA in relevance to PCA in the rotation forest.

any suggestion on what I can do?

Are your variables categorical? If they are, then you do not need to remove them. You could just make new columns like NA_Variable1, which will indicate whether a NA value exists.


What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)? I have a data set I'm trying to create a predictor model for.  The 5 features and outcome are all categorical data.  One of the features contains 5,000 unique levels. While the other 4 are all under 100 levels.  The output variable in my dataset contains 400 unique levels.

How do I handle this scenario for modeling?

My approach was going to be:

Convert the 5,000 level feature to a sequential number list via a lookup table.
Convert the remaining 4 features using One Hot Encoding.
Convert the output variable to a sequential number list via a lookup table.
Use xgboost algorithm.

I don't think I can bin the 5,000 level feature because they are user id's.  So to me, there is no way to bin them.

Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner?

When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users which have good eyesight will play basketball even if it rains.

So try to think like that.


Python3 Tensorflow: Difference Between truncated_normal and uniform_normal when creating weights So, I am new to using Tensorflow, and I am trying to create a neutral network, and in all the guides I saw one of the below lines of code was used.

Weight1 = tf.Variable(tf.truncated_normal([-,-]))

or...

Weight1 = tf.Variable(tf.random_uniform([-,-]))

Both of them work in the examples, but I want to know is what is the difference between them and is one better for certain situations than the other?

A simple Google search would show you the differences.

Truncated Normal outputs random values from a truncated normal distribution. The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.
Random Uniform outputs random values from a uniform distribution. The generated values follow a uniform distribution in the range [minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.

Regarding when is it better to use one or the other, we don't know. Weights just need to be random, and there is no conclusive research that shows that one or another distribution is better for specific scenarios. That being said, there are better ways to initialize weights that completely randomly. These include the very popular Xavier initializer.


Perform fuzzy lookup to partially match the address column I have two separate datasets (eg. dataset1.xlsx and dataset2.xlsx). Dataset1 has 2 columns, serial number and service address. Similarly dataset2 has 2 columns service address and customer number. The problem is that the address in both these datasets have spelling errors, for example one of the address in dataset 1 is 790 spring ln, york while the dataset 2 has the same address as 790 spring lane, york. So there is a difference in spelling in both columns. I am trying to match the address in dataset 1 with address in dataset 2. I want the above to addresses to match but because of the spelling error in the word lane it wouldn't match.
The dataset structure is as follows:
DATASET1:
S.NO| SERVICE ADDRESS
1| 101 SUTTON RD,ABBOTTSTOWN
2| 106 E KING ST,ABBOTTSTOWN
3| 430 W KING ST, YORK
4| 130 BEAVER AV,ALIQUIPPA
5| 2601 DUSS AV,AMBRIDGE
DATASET2
S.NO| SERVICE ADDRESS
1| 430 W KING STREET, YORK
2| 2601 DUSS AVENUE,AMBRIDGE
3| 130 BEAVER AV,ALIQUIPPA
4| 106 E KING ST,ABBOTTSTOWN
5| 101 SUTTON RD,ABBOTTSTOWN
When I match the service address columns from dataset 1 and dataset 2, I only get a match between row 4 of dataset 1 and row 3 of dataset 2. I am expecting to get a match between all the rows because they have similar addresses expect for some minute spelling errors.

This problem is called record linkage. This process usually involves comparing records using an approximate string similarity measure.
Here are a few related questions on the site that I'm aware of:

About string similarity measures, you might find some info in these questions.
About record linkage, I'm aware of these questions.

(Note that there are certainly many other questions/resources available, I just mentioned the ones I remember)


Stuck on deconvolution in Theano and TensorFlow I'm captivated by autoencoders and really like the idea of convolution.  It seems though that both Theano and TensorFlow only support conv2d to go from an array of 2D-RGB (n 3D arrays) to an array of higher-depth images.  That makes sense from the traditional tensor-product math, c_ijkl = sum{a_ijk*b_klm}, but means it's hard to 'de-convolve' an image.

In both cases, if I have an image (in #batch, depth, height, width form), I can do a conv to get (#batch, num_filters, height/k, width/k).  I'd really like to do the opposite, like going from (#batch, some_items, height/k, width/k) to (#batch, depth, height, width).

TensorFlow had the hidden deconv2d function for a while (in 0.6, I think, undocumented), but I'd like to know if there's a math trick I can use to get a bigger output in the last two dimensions after a convolution than the input.  I'd settle for a series of differentiable operations, like conv -> resize, but I want to avoid just doing a dense matrix multiplication -> resize like I've been doing so far.

EDIT: As of today (2016/02/17) TensorFlow 0.7 has the tf.depth_to_space method, which helps greatly in this endeavor.  (https://www.tensorflow.org/api_docs/python/tf/depth_to_space)  I would still love a Theano based solution, too, to complete my understanding of the material.

Maybe have a look at this post. You can do a convolution which produces an output of similar size and then "unpool" those feature maps.


NLP CounterVectorizer (sklearn), not able to get it to fit my code I was starting an NLP project and simply get a "CountVectorizer()" output anytime I try to run CountVectorizer.fit on the list.  I've had the same issue across multiple IDE's, and different code.  I've looked online, and even copy and pasted other codes with their lists and I receive the same CountVectorizer() output.
My code is as follows:
cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)

**output ----->  CountVectorizer()**

I'm really stumped on this issue.  Any advice would be greatly appreciated.  Thanks.
Update:  This seems to be a local issue as I am able to get it to fit on Colab.  If anyone can suggest what might be going on I'd be estatic.

This is normal, the fit method of scikit-learn'sCountverctorizer object changes the object inplace, and returns the object itself as explained in the documentation. This holds true for any sklearn object implementing a fit method, by the way.
When you run:
cv.fit(messages)

The cv object learns the transformation (here it is simply about counting words) and stores all the necessary parameters in order to be able to later apply the transformation to any new data point.
If you want to transform the data, you need to use first the fit method (to learn the transformation as said before) and then the transform method.
In your case if you want to transform your list of sentences you should do as follows:
cv = CountVectorizer()

messages = ['This is a good product', 'It was a bit pricey for what it does', 'I found good value here']

cv.fit(messages)
vectorized_messages = cv.transform(messages)

vectorized_messages

This will output:
<3x14 sparse matrix of type '<class 'numpy.int64'>'
    with 15 stored elements in Compressed Sparse Row format>

So you will have successfully transformed your list of strings to a a matrix of token counts.
Note that sklearn often also implement a fit_transform method that does bothe at the same time:
cv = CountVectorizer()
cv.fit(messages)
vectorized_messages = cv.transform(messages)

#is the same as:
cv = CountVectorizer()
vectorized_messages = cv.transform(messages)


How to store a trained Random Forest model in python or matlab as a matrix Is it possible to store a random forest model in python (or matlab). Then use that trained model in a C program?

I am trying to do this because I am making a myoelectric prosthetic, and you can only write to the chip we are using (PIC32MK0512MCF064) with C code.

My first thought on how to attempt this was to store the model as a matrix but I dont know if that is even possible?

Any clues? anything that could help would be greatly appreciated

Not sure about a matrix, but here's a repo for porting sklearn models into other languages, including C.

https://github.com/nok/sklearn-porter


Use text classifier on unseen data I've trained a few models to classify between two categories of text. Logistic regression was the best. Now how can i test it on unseen data?
I tried this:

def train_model():
 classifier.fit(feature_vector_train, label)
 predictions = classifier.predict(feature_vector_valid)
 joblib.dump(classifier, url+name)
...

load_model =joblib.load('my_model.pkl)
result = load_model.score('testx')

It tells me i need a y input. However, if it's new i don't have the label. WHat am i missing?

Welcome to the forums.

My understanding is that you're wanting to use the previously trained model to label new data points? If so, you'll be wanting to use .predict(X). From sklearn's documentation they say.

  All supervised estimators in scikit-learn implement a fit(X, y) method to fit the model > and a predict(X) method that, given unlabeled observations X, returns the predicted
  labels y. (Source)

Another note, is that you can't pass direct strings to a model - you'll need to preprocess your data like you did for your training set. Here is a good example of building a classifier and using it to predict new points.

Let me know if you have any questions of I've misunderstood.


Training a classifier when some of the features are unknown I am training a classifier in Matlab with a dataset that I created.
Unfortunately some of the features in the dataset were not recorded.

I currently have the unknown features set as -99999.

So, for example my dataset looks something like this:

class1: 10 1 12 -99999 6 8
class1: 11 2 13 7 6 10
...
class2: 5 -99999 4 3 2 -99999
class2: -99999 16 4 3 1 8
...
class3: 18 2 11 22 7 5
class3: 19 1 9 25 7 5
...

and so on, where the -99999 are the places where the features werent able to be measured. In this case, each class has 6 features.

I don't want to bias my classifier with the unknown features so I thought it would be a good idea to set the unknowns to -99999 so it would be way out of the range of normal features.

I tested the classifier with the -99999's and it was 78% accurate.
Then I changed the -99999 to 0's and tested the classifier again, this time it was 91% accurate.

So my question is, what is a general rule for training a classifier when some of the features were not recorded? Was I right to assume setting the unknowns to a very high negative value? But why was it more accurate when I set the unknowns to 0s?

Thanks for reading!

Welcome to Data Science SE!

Well, we say that most of our jobs is to wrangle with data, and that is because data is usually trying to deceive us... jokes aside:

You have a missing data problem

that means your have to clean your data and fill those missing values. To perform this cleaning process your must take the most classic statistician inside of you and ask:

Why is this data missing?
How much data is missing?

There are many reasons for a specific information to be unavailable. This will demand you to make assumptions and decide how to deal with this.

Jeff Sauro posted at MeasuringU: 7 Ways to Handle Missing Data, some which I list here:

Delete corrupted samples:

If you have a large dataset and there is not much data missing, you can simply remove those corrupted data points and go on with life

Recover the values:

Some problems will allow you to go back and get missing information.

We usually ain't that lucky, then you can

Educated Guessing:

Sometimes, you can infer what would be the feature value by simply looking their pears. That is a bit arbitrary but it might work.

Average:

This is the most common approach, simply use the average of that value whenever it is missing. This might artificially reduce your variance but so does using 0 or -9999... for every missing value.

Regression Substitution:

You can use a multiple regression to infer the missing value from the available values for each candidate.

Some references on missing data are:

Allison, Paul D. 2001. Missing Data. Sage University Papers
Series on Quantitative Applications in the Social Sciences.
Thousand Oaks: Sage.
Enders, Craig. 2010. Applied Missing Data Analysis.
Guilford Press: New York.
Little, Roderick J., Donald Rubin. 2002. Statistical Analysis
with Missing Data. John Wiley & Sons, Inc: Hoboken.
Schafer, Joseph L., John W. Graham. 2002. “Missing Data:
Our View of the State of the Art.” Psychological Methods.

About your experiment:

Adding -99... is creating outliers and that bit of information is heavy (numerically speaking, it is huge) and will affect parameter tuning. For example, suppose you have this data:

| Feature1 | Feature2 |
|----------|----------|
| 0        | 8        |
| -1       | 7        |
| 1        | -        |
| -        | 8        |

And you try filling the missing values with -99, now try to fit a linear regression trough the data. Can you see that you don't be able to fit it properly?

The line won't fit, and this will yield bad performance.

Adding 0 values on the other hand will give a slightly better line:

It is still not good, but slightly better since the scale of the parameters will be more realistic.

Now, using average, is this case will give you even better curve, but using regression will give you a perfect fitting line:

Note: I need to remake those images, but these should do until I have the time for it.


Why do we use different image processing at train and test time for an image classification tasl I would like to ask why we use different image processing at train and test for an image classification task. For example, this pytorch tutorial uses RandomResizedCrop at train time and then Resize + CenterCrop at test time.

I just read this paper called Fixing the train-test resolution discrepancy and it says this has been the best practice.

However, I'm unable to figure out why nor find any documentation online as to why this is the case. Please help me shed some light on this. Thank you.

The key difference is that for training the preprocessing serves the purpose of data augmentation: it increases variance in the training data and thereby helps to better generalize. That is why you will find random cropping, flipping, rotating and zooming here.

But during inference that is not required and these random preprocessing steps would not be helpful. Therefore, you only do the preprocessing steps required  and eliminate randomness (e.g. crop images to the correct input size with centercrop instead of randomcrop).


Calculating RMSE AND R-squared from the confusion matrix I have my confusion matrix as C.mat

8263    20    39
2       3826  14
43       7    4431

My predicted class labels are Ypred and actual labels are Ytest. Ypred size is 16000*1 and Ytest 16000*1.

I am trying to calculate the R-squared and RMSE. Is there a way to directly calculate RMSE and R-squared from the confusion matrix?

I tried this:

RMSE = sqrt(immse(Ypred, Ytest))

However, it didn't work.

I can use  either R or Matlab.

Any advice will be appreciated!

The confusion matrix suggests that you are performing classification rather than regression. RMSE and R-square are measures associated with continuous variables; For categorical variables, I'd suggest using Accuracy / Recall / Precision / F1 score to measure the performance of the model.

https://www.quora.com/How-is-root-mean-square-error-RMSE-and-classification-related


Need equations for some of weight initializers in tensorflow? For my CNN model I tried some of the weight initializers such as "truncated normal initializer","random normal initializer","glorot normal initializer","glorot uniform initializer", my questions are:

1- I need the equations for each of them, for the two Xavier initializers I found more than one type of equation for that I confused which one is true but for other two I didn't found any equation?

2- Is the range of values are between 0 and 1 for all of them? are they considered a gaussian?

2 - Not necessarily between 0 an 1. Example: If you see the normal glorot formulae, The sigma_i value will entirely depend on fan_in and fan_out. So, it need not be centred at 0.

EDIT 1:

EDIT 2:

Research paper - weight initialization by Xavier

Credits: Also, check out this and this
Hope it helps.


Normalizing dependent feature by one of the independent ones I have a data set with three different features (x1, x2, x3) and I am going to use a regression model to predict y based on the features.

x3 is the total amount of money that a customer invest and y is the interest for that customer. So, for all cases y < x3. x1 and x2 are other information about the customer.

Rather than predicting y, I tried to predict y/x3 and I got much better results.

In my new model, I have tried to predict y/x3 = f(x1, x2, x3) using XGB model. The reason that I get better results is because of the reduction in the skewness in the data.

Do you think that this leads to data leakage? Please note that I did a lot of feature extraction based on x1, x2, x3, and use them for both models.

I think it's complete alright. In fact, the second model mathematical expression is given by
y=x3f(x1, x2, x3),
which is just like the first model but with some specific feature engineering. I don't see any possibility for data leakage.


Sequential Modelling: Multiple Sequence to One or Sequence to Sequence Suppose I have a single sequence of $x_1, x_2, ..., x_n$ and corresponding labels $y_1, y_2, ..., y_n$.

An example would be a person makes website visits $x_i$ and the label $y_i$ tells us if there was a purchase in any of the previous visits.

Is there a difference between training multiple sequence to one or a single sequence to sequence?

Multiple sequence to one

This approach gives us $n$ data points. Additionally it seems that the model will see $x_1$ n times, while $x_n$ will be only fed in once.

$[x_1] , [y_1]$

$[x_1, x_2] , [y_2]$

...

$[x_1, x_2, ..., x_n], [y_n]$

Single Sequence to Sequence

This approach gives us just 1 data point.

$[x_1, x_2, ..., x_n], [y_1, y_2, ..., y_n]$, with no repetition in data.

Do these yield the same outcome in RNN (e.g. LSTM) training? It seems that when your data is imbalanced, you would greatly benefit from the 1st approach, since you get more observations.

My understanding is that, since even a single data point is technically a valid data point in my case, I should be using the multiple sequence to one approach. But in cases of NLP (where the first word alone has no context), obviously we need the whole sequence.

As you are talking about RNN, I'm going to change the parameters of your question a little bit.

The multi sequence to one will be:

$[x_1] , [y_1]$

$[x_1, x_2, y_1] , [y_2]$

...

$[x_1, x_2, ..., x_n, y_{(n-1)}], [y_n]$

as the previous state will always be used.

This is typically a Markov process with a state containing a n input vector. It has known properties and that's what is used by RNN to be able to create a sequence one element by one element.

On the case of one sequence to one sequence, there is no such state, and we don't have the concept of one word following the other. It's basically n values in the input and n value as the output. But there is no constraint about "one after the other" in such a pseudo sequence anymore, so the outputs could be scrambled and it would still be a sequence to sequence output.

Now, for NLP, the fact that you don't have a state before time 0 is not a problem. You can discard the first n-1 elements even and keep the Markov property of the system.

This is exactly what I'm doing in chapter 8 in Building Machine Learning with Python and it works well if you start retrieving words when the system is in a proper state.


Extremely high MSE/MAE for Ridge Regression(sklearn) when the label is directly calculated from the features Edit: Removing TransformedTargetRegressor and adding more info as requested.

Edit2: There were 18K rows where the relation did not hold. I'm sorry :(.  After removing those rows and upon @Ben Reiniger's advice, I used LinearRegression and the metrics looked more saner. The new metrics are pasted below.

Original Question:

Given totalRevenue and costOfRevenue, I'm trying to predict grossProfit. Given that it's a simple formula totalRevenue  - costOfRevenue = grossProfit, I was expecting that the following code would work. Is it a matter of hyperparameter optimization or have I missed some data cleaning. I have tried all the scalers and other regressions in sklearn but I don't see any big difference.

# X(107002 rows × 2 columns)
+--------------+---------------+
| totalRevenue | costOfRevenue |
+--------------+---------------+
| 2.256510e+05 | 2.333100e+04  |
| 1.183960e+05 | 2.857000e+04  |
| 2.500000e+05 | 1.693000e+05  |
| 1.750000e+05 | 8.307500e+04  |
| 3.905000e+09 | 1.240000e+09  |
+--------------+---------------+

# y
+--------------+
| 2.023200e+05 |
| 8.982600e+04 |
| 8.070000e+04 |
| 9.192500e+04 |
| 2.665000e+09 |
+--------------+
Name: grossProfit, Length: 107002, dtype: float64

# Training

import numpy as np
import sklearn

from sklearn.compose import TransformedTargetRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=13)

x_scaler = StandardScaler()

pipe_l = Pipeline([
        ('scaler', x_scaler),
        ('regressor', Ridge())
        ])

regr = pipe_l

regr.fit(X_train, y_train)

y_pred = regr.predict(X_test)

print('R2 score: {0:.2f}'.format(sklearn.metrics.r2_score(y_test, y_pred)))
print('Mean Absolute Error:', sklearn.metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', sklearn.metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(sklearn.metrics.mean_squared_error(y_test, y_pred)))

print("Scaler Mean:",x_scaler.mean_)
print("Scaler Var:", x_scaler.var_)
print("Estimator Coefficient:",regr.steps[1][1].coef_)

Output of above metrics after training(Old Metrics with 18k rows which did not confirm to the relation)

R2 score: 0.69
Mean Absolute Error: 37216342513.01034
Mean Squared Error: 7.601569571667974e+23
Root Mean Squared Error: 871869805169.7842
Scaler Mean: [1.26326695e+13 2.14785735e+14]
Scaler Var: [1.24609190e+31 2.04306993e+32]
Estimator Coefficient: [1.16354874e+15 2.59046205e+09]

Ridge(After removing the 18k bad rows)

R2 score: 1.00
Mean Absolute Error: 15659273.260432156
Mean Squared Error: 8.539990125466045e+16
Root Mean Squared Error: 292232614.97420245
Scaler Mean: [1.57566809e+11 9.62274405e+10]
Scaler Var: [1.20924187e+25 5.95764210e+24]
Estimator Coefficient: [ 3.47663586e+12 -2.44005648e+12]

LinearRegression(After removing the 18K rows)

R2 score: 1.00
Mean Absolute Error: 0.00017393178061611583
Mean Squared Error: 4.68109129068828e-06
Root Mean Squared Error: 0.0021635829752261132
Scaler Mean: [1.57566809e+11 9.62274405e+10]
Scaler Var: [1.20924187e+25 5.95764210e+24]
Estimator Coefficient: [ 3.47741552e+12 -2.44082816e+12]

(To summarize the comment thread into an answer)

Your original scores:

Mean Absolute Error: 37216342513.01034
Root Mean Squared Error: 871869805169.7842

are based on the original-scale target variable and are between $10^{10}$ and $10^{12}$, at least significantly smaller than the mean of the features (and the target)?  So these aren't automatically bad scores, although for a perfect relationship we should hope for better.  Furthermore, a 0.69 R2 values is pretty low, no scale-consciousness needed.

That both of the model's coefficients came out positive is the most worrisome point.  I'm glad you identified the culprit rows; I don't know how I would have diagnosed that from here.

Your new ridge regression still has "large" errors, but significantly smaller than before, and quite small compared to the feature/target scale.  And now the coefficients have different signs.  (I think if you'd left the TransformedTargetRegressor in, you'd get largely the same results, but with less penalization.)

Finally, when such an exact relationship is the truth, it makes sense not to penalize the regression.  Your coefficients here are a little bit larger, and the errors drop away to nearly nothing, especially considering the scale of the target.


including Zeros counts categories with pandas value_counts() What i want is including zero counts categories while generating frequencies for categorical variables
example:

col1   col2
a       x
c       y
a       y
f       z

what i want is to generate a frequency table with counts and percentages including zero counts categories
results
  Counts   Cercentage
a  2          50.0%
b  0          0.0%
c  1          25.0%
d  0          0.0%
e  1          25.0%

what i have done is generating the frequency table with counts and percentages but i need to include also the zero counts categories like b and d as illustrated above
here is what i have tried
pd.concat([df.col1.value_counts(dropna=False),
           df.col1.value_counts(normalize=True,
                                dropna=False).mul(100).round(1).astype(str) + '%'],
          axis=1,
          keys=('Counts','Percentage'))

any help please

I got the answer use of Series.reindex with a list of all categories and if not match replace to 0
  categories = ['a', 'b', 'c', 'd', 'e']
  pd.concat([df.col1.value_counts().reindex(categories[::-1], fill_value=0),        df.col1.value_counts(normalize=True).reindex(categories[::-1], fill_value=0).mul(100).round(1).astype(str) + '%'],axis=1, keys=('Counts','Percentage'))

This worked for me


MemoryError when concatenating a large data-frame I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.

My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.

I split the data-frame into chunks and complete the get_dummies function on each chunk and store it into a list. This works without any issues.

I then try to concatenate the entire list, as you can see in the code below.

I also delete the data-frames and the list of chunks to save up on memory.

dummies = []
columns = self.df[self.selectedHeaders]
del self.df
chunks = (len(columns) / 10000) + 1
df_list = np.array_split(columns, chunks)
del columns

for i, df_chunk in enumerate(df_list):
    print("Getting dummy data for chunk: " + str(i))
    dummies.append(pd.get_dummies(df_chunk))

del df_list
dummies = pd.concat(dummies, axis=1)

As you can see from this code, I store the columns I need and split them into chunks. Then I run the get_dummies function on each chunk and store them within a list.

When I run the concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.

You should use sparse matrices

When you are categorizing a column, you are creating many columns that mostly hold zeroes, and since dense data structures keep track of all the zeros, the memory usage explodes. Sparse matrices on the other hand only keep track of non-zero values and their indexes. There are many version of sparse matrices optimized for different use cases.


Time series analysis vs linear regression I am working on developing an algorithm which will predict the future traffic for the restaurant. I am confuse that which of the two: Linear regression or time series analysis I should use as the base for my algorithm. The features I am using are: Day,whether there was festival,temperature,climatic condition , current rating,whether there was holiday,service rating,number of reviews etc.

Please guide me how should I proceed . Also how can I optimize my algorithm so that it can learn with time.

The data you are having is panel data which is a combination of both cross sectional data and Time series.

You can try with regression models by giving time stamp to your data .Like maintaining one feature based your weekday (1 to 7).or if you have trends and seasonality in your data you can go to giving week number as feature like (0 to 53) weeks.


Pandas get_dummies() rows dropping after joining back with X I'm having an issue that I can't explain and am hoping I am missing something simple.
I have a large dataset of shape(45Million+, 51) and am loading it in for some analyses (classifiers, deep learning, basically just trying a few different things as some research for work).
I take a few steps when I load it in:

dropna() to get rid of all rows with an na (only about 6K out of the 45M)
Use pandas get_dummies() to change a categorical variable with about a dozen classes into dummy variables (have also used sklearn's onehotencoder for this and had the same issue outlined below)

When I would run a RandomForest on a subset of the data (about 4 million rows, made using train_test_split) I would get the following error:
<ValueError: Input contains NaN, infinity or a value too large for dtype('float64').>

Since I dropped NAs at the start, this confused me, so I went back and checked the length of each output.
When I drop NAs, I get a length of 45356082
When I split off the categorical variable and one hot encode it, it has a length (and every variable within it does) of 45356082. We'll call this Dummy.
Here's where it gets weird-- when I join Dummy back to my original X as Xnew, Xnew has a length of the same as above, but the dummy variable columns now have length 45351726.
The join process is dropping like 4500 rows from the dummy columns.
Any idea why this would happen?
Here's the code I'm using:
choice_data_sub = pd.read_csv("predData.csv")
# Drop NAs
choice_data_sub = choice_data_sub.dropna()

X = choice_data_sub[["Columns1", "Column2", "Column3", "Categorical"]]
y = choice_data_sub[["NextPurchase"]]

choice_data_sub = choice_data_sub.reset_index()

gametype_df = pd.get_dummies(choice_data_sub.Categorical, prefix="Game")

# merge with X
gametype_df = gametype_df.reset_index()
# X = X.reset_index() <-- This breaks it in a different way, was tried as a fix
X = X.join(gametype_df)

Also! Just discovered that the following code works, but I'd still like to know why this didn't.
X = X.reset_index()
X = X.merge(gametype_df, left_index=True, right_index=True)

Can you post your code when you join the datasets? It's possible you're joining in a different way than you had intended. See the following helpful tutorial for figuring out exactly how you want to join datasets: https://pandas.pydata.org/docs/user_guide/merging.html
Additionally - I would check that the X and dummy dataframes have the same indexes. It's possible that in your join these forgotten entries simply don't share the same index in both dataframes.


Right Way to Input Text Data in Keras Auto Encoder I have several thousand text documents and I am currently working on obtaining the latent feature representations of words and generate sentences using variational auto encoder. The main obstacle I am facing is “how to input such large textual vectors into VAE (or say even in a simple auto encoder)". I know there are other techniques such as adding layers in the form of LSTMs, but I want to keep it simple and keep the model as just VAE. One way is to use one-hot-encoded vectors or bag of words, but again, this is not the most efficient way since for a vocabulary of 100K unique words, each document will have a 100K input vector. Additionally, we loose the sentence structure. For small datasets however, there is no problem in training an autoencoder using this type of input. Another method is to use word embeddings using a pre-trained Word2Vec. This is what I have been trying to do and the python notebook which can be DOWNLOADED HERE uses this technique. The code is too long and has multiple pre-processing steps, so I am unable to embed this code in my post. The following are my questions:

Now, each sentence (or document) will have different number of words. So the number of word embeddings for each  document will have different lengths. Unfortunately, keras requires all the inputs to be of same length ( if I am right). So, how to handle such cases of varying input lengths?. Currently, in the fifth block of the python notebook, you can see the statement data = [x for x in vect if len(x) == 10]. That is, I only consider documents that have exactly 10 words to overcome this problem. Ofcourse, this is not practical. Can we pad 0 vectors?
The VAE example shown in Keras blog uses MNIST data as exmple. Therefore, they use a sigmoid activation in the final reconstruction layer; consequently, “binary_crossentropy” as the loss function (along with the KL divergence). Since my inputs are word embeddings, where there are even negative values in the embeded vectors, I believe I should not use the activation as “sigmoid” in the final reconstruction layer.  Is that right?. Additionally, I have also changed the loss as  “mean_squared_error” instead of “binary_crossentropy” in the attached code.

It will be great if someone who has worked on VAE and autoencoders for text data can provide their inputs regarding the questions mentioned above.

Note: the attached code is a simplified version of the code in THIS LINK

Here's my take on your questions.

Yes, you can zero-pad vectors. However, I would strongly recommend you use an LSTM as part of your encoder if you're using sentences as input. An LSTM can take variable-length input - that's part of its charm!
The loss function depends entirely on your goal. It sounds like you're trying to create a sequence-to-sequence VAE. I would go about this by putting a softmax function on your output layer, to generate the next most probable word in the sentence. So, I would use categorical_crossentropy as the loss function, and I would use softmax instead of sigmoid, because you're choosing the next most likely word from a set vocabulary (this is not a binary issue but rather a categorical issue).

Hope that helps!


Iterate in Keras custom loss function I know that is better avoid loop in Keras custom loss function, but I think I have to do it.
The problem is the following: I'm trying to implement a loss function that compute a loss value for multiple bunches of data and then aggregate this values in an unique value.

For example I have 6 data entry, so in my Keras loss I'll have 6 y_true and 6 y_pred.
I want to compute 2 loss value: one for the first 3 elements and one for the last 3 elements.

Example of hypothetical code:

def custom_loss(y_true, y_pred):
    start_range = 0
    losses = []
    for index in range(0,2):
        end_range = start_range + 3
        y_true_bunch = y_true[start_range:end_range]
        y_pred_bunch = y_pred[start_range:end_range]
        loss_value = ...some processing on bunches...
        losses.append(loss_value)
        start_range = end_range
    final_loss = ...aggregate loss_value...
    return final_loss

Is it possible to achieve something like this? I need to process the whole dataset and calculate loss for multiple bunches and then aggregate all bunches value in a single value.

Assuming 2 partitions since, for index in range(0,2) and a fixed bunch size of 3 since, start_range + 3
The shape of your y_true should be (nb_samples, 2), where the first vector of 2nd dimension is the ground_truth and the other is partition_index
import numpy as np
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.losses import mean_squared_error

y_true = tf.Variable(np.array([[1.5, 0],[1.2, 0],[1.3, 0],[1.6, 1],[3.0, 1],[2.25, 1]]), dtype=tf.float32)
y_pred = tf.Variable(np.array([[1.35],[1.24],[1.69],[1.55],[1.24],[1.69]]), dtype=tf.float32)

def get_two_splits(y_true):
    child_y = y_true[:, 1]
    child_y = tf.expand_dims(child_y, 1)
    y_true = y_true[:, 0]
    y_true = tf.expand_dims(y_true, 1)
    return y_true, child_y

def some_processing_on_bunches(y_all):
    y_true, y_pred = get_two_splits(y_all)
    return mean_squared_error(y_true, y_pred)

def custom_loss(y_true, y_pred):
    n = K.shape(y_true)[0]
    y_true, partitions = get_two_splits(y_true)
    partitions = tf.reshape(partitions ,[n, ])
    partitions = tf.cast(partitions, tf.int32)

    y_elements = tf.dynamic_partition(tf.concat([y_true, y_pred], 1), partitions, 2)
    losses = tf.vectorized_map(some_processing_on_bunches, tf.stack(y_elements))
    return tf.reduce_mean(losses, axis=1)

K.eval(custom_loss(y_true, y_pred))
>>>array([0.05873336, 1.1379    ], dtype=float32)


Algorithms, techniques, papers for regression with vector output I have a regression problem that has relatively low dimensional input (say 8 initial relevant features, without the engineered ones), but very high dimensional output vector (not a single value for the output, a vector of reals). Basically for each sample during training, a low dimension of features and 2500 reals as the output go into the model, then testing samples take the same types of input features in to predict the 2500 reals (a curve).

              inputs                               outputs
 _________________________________________ _______________________________________
| a | b | c | ... | (engineered features) | 2500 reals as output (y's of a curve) |

Since we are not machine learning experts, we had some trouble in the beginning even finding an algorithm that supported output vectors as opposed to a single point.

I'm not looking for generic data science 101 suggestions like: cleaning the data, picking the relevant features, understanding the data, etc. We've done that well enough I believe, and we've even had some limited success already learning these curves with about a predication rate of ~70% (which sounds rough, but isn't too bad given the nature of this data). However, we are not experts as I said, and I'm still convinced that we are not even considering all of our options in terms of algorithms, frameworks, feature engineering techniques, etc. for this specific type of problem; I'd like to squeeze everything we can to get the best possible predictions. Given this data, there definitely isn't a 100% relationship between the input and output. But I do think we can get it predicting at a rate of higher than 70%. Maybe. Sometimes the predictions are near perfect, other times they have the "general shape" but are sometimes warped in places we can't explain.

We are using Python/SciPy/scikit-learn. We are doing 20-fold cross validation and our dataset has ~22000 samples.

How would you approach this type of problem - which algorithm, or which papers, have you come across / used / are relevant for this type of problem? What should we be keeping in mind for this specific type of problem?

We are very happy with this:

Not bad:

Not good:

Interesting:

Perhaps you could frame this as predicting a distribution (I hope I haven't mis-understood your question!).

You could try using Markov Chain Monte Carlo (don't be scared by the name!). What this essentially does, it to sample a posterior distribution from your input data and see how likely it was that is is related to your prior distribution (your data). If it thinks the sample fits with your data, it keeps puts it records that sample, otherwise it discards it. It keeps searching like this until some stop-criterion is reached. All the accepted points that were recorded then represent your target distribution, which can be N-dimensional (and so hopefully matches your use-case).

Here is a quick video introduction to the topic in general, and here is a deeper video explanation. Here is a more thorough text-based introduction.

You can get started doing this with packages like:

PyMC3 (code and docs) if you use Python. There are interactive examples here, e.g. the PyMC3_tips_and_heuristic notebook too.
If you use R, then there is the mcmc package.


Face Recognition with Video File I have built a face recognition model based on Pyimagesearch Tutorial

It works great on images but the issue I am facing is with videos.

I am recognizing the faces frame by frame(5fps). It gives me people present in the particular video frame.

But the same person is identifed with different names in different frames.

How can I restrict it to identify same name with throughout the video?

Sometimes it also recognized two different people with same name.
Eg. Sam and Dean are present in the frame but it identifies both as sam.

The pyimagesearch face recognition implementation used knn classifier to recognise the faces. I have worked with both svm and knn classifiers, from my experience you can do a couple of thinks to improve the face recognition performance.
1. use svm with higher C value (1 to 10) over knn classifier.
2. Maintain relatively equal number of images in all classes


Feature selection with L1 regularization on sklearn's LogisticRegression I'm using sklearn's LogisticRegression with penaly=l1 (lasso regularization, as opposed to ridge regularization l2). Lasso is causing the optimization function to do implicit feature selection by setting some of the feature weights to zero (as opposed to ridge regularization, which will preserve all features with some non zero weight). Is it possible to extract this weight information (feature selection) from sklearn somehow?

Yes, as demonstrated in the documentation: http://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_path.html

Welcome to DataScience.SE


Can GLM( generalized linear method) handle the collinearity between the predictor variables in a regression-analysis? I'm a beginner in Machine learning and I've studied that collinearity among the predictor variables of a model is a huge problem since it can lead to unpredictable model behaviour and a large error. But, are there some models (say GLM) that are perhaps 'okay' with collinearity unlike the classic linear regression? It is said that the classic linear regression assumes there is no correlation between its independent variables.
This question arises because I was doing a project that said "If the input features are correlated with each other, it is better to use a generalized linear model since they would perform better than linear regression."
Can someone explain this ?

“It is said that the classic linear regression assumes there is no correlation between its independent variables”
Depending on your goal of doing the regression, this common statement is false.
Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is the solution to the OLS optimization.
Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is minimum variance linear unbiased estimator from the Gauss-Markov theorem.
What the Gauss-Markov theorem requires is that the error terms not be correlated. This is commonly confused for saying that the predictors are not correlated, but that is indeed a mistake.
There can be numerical instability when you do math on a computer, particularly as you approach perfect multicollinearity ($X^TX$ is close to singular, singular in the extreme case of perfect multicollinearity or correlation of $1$ between variables), but there is no inherent problem with multicollinearity if your goal is to predict.
Where multicollinearity can hurt is when you want to do inference on the parameters, which is rarely a goal in machine learning. When you have multicollinearity, the parameter standard errors are inflated, sapping away your power to tell that they are not zero. Philosophically, it also makes it hard to attribute an effect to a particular predictor if it is correlated with others. (Imagine a hospital wanting to know if it pays its neurosurgeons as much as its heart surgeons and sees that the heart surgeons make way more but also sees that the heart surgeons have much more experience. Do they make more because of their specialty or because of their experience?)
Multicollinearity also can mean that you might be able to use a smaller amount of variables to get almost as much information as the entire set of variables. For instance, if two predictors are highly correlated, it might not be worth including both; you might be better off leaving out one for the sake of model parsimony and having fewer parameters in your regression, but that is an empirical problem and up to the model-designer’s judgment.
Getting to the full GLM framework, the Gauss-Markov theorem does not apply, but the idea remains that there is no inherent issue with multicollinearity when your goal is to predict instead of to do parameter inference, which is the typical goal in machine learning.


pytorch dataloader tensor modification T=tensor([101,123,414,463][234,903,313,341]...)
train=TensorDataset(T)
train_dataloader=Dataloader(train)

Now I would like to update tensor T[0] i.e tensor
T[0]=tensor([101,123,567,463])

for this i have tried as follows:
train_dataloader.Dataset[0].index_copy_(0,tensor([2]),tensor([567])

is it possible to modify this way or not?
Any kind of reference is helpful

Pytorch DataLoader is a generator, it will generate new batches when iterated through. Hence, as per the best of my knowledge, you can only change the data on the fly. For eg, if you want to replace the first element of your dataset with second element of the datase, you can do something like -
T = torch.tensor(([101,123,414,463],[234,903,313,341]))
train = TensorDataset(T)
train_dataloader = DataLoader(train)

for idx,i in enumerate(train_dataloader):
  if idx == 0:
    i = i[0] # Get the first and the only instance of the batch
    i = train[1][0]

Otherwise, you will need to change your dataset (train variable) and then again create a dataloader for this new dataset.


Improving SVM binary classification model on new dataset I am working to create a SVM binary classifier for classification of Tweets based on news class "Crime" and "Non Crime". I have downloaded a dataset of 6400 rows from various sources and training my model on it. While I am able to achieve over 95% accuracy, what concerns me is about its performance on new datasets. The system will be deployed on live streaming tweets, so how can I put in a feedback system such as my model is continuously updates itself? As in, say for new crime type news source which were not covered in training model, how to incorporate them further?

I am asking a broad methodology question, not some specific program related question as I would like to figure out the implementation myself :)

Online learning for SVMs is supported only for certain SVM options (a linear SVM with SGD can easily update for example).

There are implementations out there to address the issue of online learning for all SVMs.

https://stats.stackexchange.com/questions/26041/can-svm-do-stream-learning-one-example-at-a-time

If your model isn't terribly large it also probably isn't too bad to train a new one on all data periodically.


Does it make sense to train multiple classifiers for multiple partitions of data? Let's say I have a dataset consisting of 100 features and a binary target variable. On exploring the data, I see that Feature 10 which is binary, seems to split the data in an 80:20 ratio with respect to the target variable. Does it then make sense to partition the data into "Feature 10 is 0" and "Feature 10 is 1" and try to build separate classifiers for both cases? Or is this what models like Random Forests do under the hood?

Yes, there is no need for you to manually split the dataset into two parts and train separate classifiers for them.

That's something that decision trees (eg. Random Forest), will do, as you say, under the hood.

The manipulation that you could do on your dataset is probably the dimensionality reduction - but, this is also done, informally, by decision trees. See "Random Forests / Ensemble Trees" section here.