batched CrossEntropyLoss in pytorch I'm wondering how to implement this with pytorch built-ins. I've got a 3 dimensional input of uints called policy. Most of the entries are zero, and if I were to L1 normalize this I would have a (target) probability distribution.

I've also got the output of a linear layer, called 'logit', with the same shape as 'policy'. I must turn this into a probability distribution by taking the softmax, but only over the entries where policy is non-zero.

The loss is then -sum(log(logit_masked_softmax) * policy_normalized))

I've implemented this manually with the nn.functional module using boolean indexing. The problem is that I want to do this in batches, where a 4 dimensional tensor represents the batches of 3 dimensional inputs. I am convinced that there must be a built-in way to achieve this and it probably is also faster and more numerically stable.

I solved this by using torch.where to give the irrelevent entries a really big negative value, so that they vanish after an exp. I also took advantage of the log of CEL trick.

loss = torch.zeros(1).type(dtype)

states = torch.cat([states_[_] for _ in idc[:self.batch_size]]).type(dtype)
scores = torch.tensor([scores_[_] for _ in idc[:self.batch_size]]).type(dtype).view(-1, 1)
policies = torch.cat([policies_[_] for _ in idc[:self.batch_size]]).type(dtype)

values, logits = self.network(states)
value_loss = mse(values, scores)

p = torch.nn.functional.normalize(policies, dim=[1,2,3], p=1)
logit_exp = torch.exp(torch.where(p > 0, logits, inf))
s = torch.log(torch.sum(logit_exp, dim=[1,2,3])).view(-1, 1, 1, 1)
log_q = logits - torch.mul(ones, s)
policy_loss = torch.sum(torch.mul(p, -log_q))/self.batch_size

loss = value_loss + policy_loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
idc = idc[self.batch_size:]


How to prevent neural network from overfitting on small subset of features I'm trying to predict the win probability for a team in a basketball game using a neural network with a single sigmoid output. The input layer consists of a one-hot representation of the players, number of goals(hoops?) for each player, free throws, penalties, etc. For each game I have about 200 training examples each capturing the game state at a particular point in time. My training data spans multiple seasons, so players don't always play for the same team. My hope was that this way I could feed the network a snapshot of the current game state and it would give me the win probability for either team.

Unfortunately, during training the network immediately overfits to the team compositions and memorizes the outcome of that particular game and pays no attention to all the other features. Validation accuracy is abysmal. To prevent the network from fingerprinting the team compositions I tried using dropout on the team inputs only. Whereas in my experience keep_prob around 0.8-0.9 usually yields the best results, here I saw the best results with keep_prob 0.35. This kind of works but it doesn't feel great to basically withhold 65% of player information from the network.

What's the best way to prevent this sort of "cheating" where the network overfits to a small subset of features and looks up memorized results?

I believe that using drop-out is the best solution for such a problem.

I also suggest ensembling methods such as RandomForest instead of a neural network because RandomForest is robust w.r.t overfitting. Also, as you mentioned, your data is not that much complex. So, it would be a good idea to use RandomForest.


You have a very small number of observations (32?) and a non-trivial number of predictors.

It is known that the cross-validation function possesses multiple local minima/maxima. If you increase the number of multistarts to, say, nmulti=100 (add this option to your call to npregbw()), you ought to see that the same results occur on each invocation of the optimization process.

Note you can do this all in the call to npreg() and skip the bandwidth call for convenience (npreg() will call npregbw() automatically but accept the arguments intended for npregbw()).

Also, you will get the same results if you restart R each time you run the routine (seeds are set automatically to ensure this).

model <- npreg(power ~ temperature
                     + prevday1
                     + prevday2
                     + prev_instant1
                     + prev_instant2
                     + prev_2_hour,
                regtype="ll",
                bwmethod="cv.aic",
                nmulti=100,
                data=new_tr_dat)

To see whether things are stable with respect to the number of multistarts, look at the value of the cross-validation function and summary provided. Also, you can look at partial regression plots along with resampled variability bounds via plot().

summary(model$bws)
plot(model,common.scale=FALSE,plot.errors.method="bootstrap")

Also, I note that your predictor `temperature' is discrete, so you might consider using

+ ordered(temperature)

(i.e. use a discrete support kernel). Doing so reveals that there is little signal in this model, but the same holds for a simple parametric model (adjusted r-squared is negative).

model.lm <- lm(model$bws$formula)
summary(model.lm)

Hope this helps!


AxisError: axis 1 is out of bounds for array of dimension 1 I've used svm classifier. Now I need to construct the confusion matrix. Here is the code that I have used.

from sklearn.svm import SVC

svm = SVC(kernel='rbf')

svm.fit(feat_train,np.argmax(Y_train,axis=1))
svm.score(feat_train,np.argmax(Y_train,axis=1))
svm.score(feat_val,np.argmax(Y_val,axis=1))
y_pred = svm.predict(feat_val)
y_pred.shape
(950,)
y_class = np.argmax(y_pred, axis = 1)
y_check = np.argmax(Y_val, axis = 1)

cmatrix = confusion_matrix(y_check, y_class)
print(cmatrix)
print('accuracy Score :'),accuracy_score(y_check, y_class)

print('Report : ')
print(classification_report(y_check, y_class))

I'm getting the following error

---------------------------------------------------------------------------
AxisError                                 Traceback (most recent call last)
<ipython-input-102-67006c959068> in <module>
----> 1 y_class = np.argmax(y_pred, axis = 1)
      2 y_check = np.argmax(Y_val, axis = 1)
      3
      4 cmatrix = confusion_matrix(y_check, y_class)
      5 print(cmatrix)

~\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in argmax(a, axis, out)
   1101
   1102     """
-> 1103     return _wrapfunc(a, 'argmax', axis=axis, out=out)
   1104
   1105

~\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     54 def _wrapfunc(obj, method, *args, **kwds):
     55     try:
---> 56         return getattr(obj, method)(*args, **kwds)
     57
     58     # An AttributeError occurs if the object does not have

AxisError: axis 1 is out of bounds for array of dimension 1

I think the problem is with y_pred as the dimension is given as (950,). Please help me solve this

y_class = np.argmax(y_pred, axis = 1)

here axis=1 represents check max value index column wise
you are facing the error because you are passing vector (size,)
y_class = np.argmax(y_pred, axis = 0)

use above code to remove the error


How To Motivate A Neural Network Suppose a training dataset contains the following inputs:
company size
number of employees
turnover
average salary
country
years of operation

...and outputs:
budget allocated to marketing
budget allocated to sales
budget allocated to R&D
budget allocated to training
budget allocated to shareholders

It also contains the column:
profit

...which is what I want to optimise against (i.e. the primary motivation of the model).
For any given set of inputs (company size, number of employees, turnover, average salary, country, years of operation), I want to be able to predict values for the 5 outputs which are most likely to achieve the highest 'profit'.
In this example, would 'profit' be an input, an output, or neither?
I don't believe it's an input, as profit won't be available to input into the model, and I don't believe it's an output, as am not trying to predict profit for a given set of inputs. Rather I am trying to predict the outputs that are most likely to generate the highest profit for that set of inputs (based on previous training data examples).
Is it possible to therefore specify some sort of 'motivation' variable which the model strives to maximise?

I think your input features are lacking. Firstly, The budget allocation is not simply done based on the previous year turnover and size of the company, there are myriad other factors involved which directly correlate towards a budget allocated to any department.
Secondly where is the relation of profit vs budget allocation? For maximizing profit according to budget allocation, your training set needs to reflect that.
Thirdly you could simply make target profit an input feature and model your data accordingly to allocate budget keeping in mind the targeted profit as well. Provided you incorporate features highlighted in the first point.


Why do we use a softmax activation function in Convolutional Autoencoders? I have been working on an image segmentation project where I have created a convolutional autoencoder. I saw this image and implemented it using Keras.

At the output layer, the author has used the softmax activation function. Shouldn't it be ReLU?

  According to me, this seems to be a regression problem, where we need
  to predict the continuous values for the segmented image pixels. If so, why are we using a softmax function, instead of ReLU or a linear function?

Since its a multi-class classification problem, each class will have its own probability value between 0 and 1 and sum of all probabilities of classes will equal to 1.


Is there a way to standardize latitude and longitude to be used as predictors in KNN algorithm? The predictors are latitude and longitude, and target variable is region.

Rather than modifying your data, use great circle distance for those features' conmponent of the cost function.


Matplotlib: plot indicating the precision of measurements I have some data with some margin of error and am using numpy arrays to plot that data. Assuming the precision is, let's say, 0.05 or 0.03grams, then the data will be something like

[4.0 $\pm$ 0.05, 3.05 $\pm$ 0.03]

How can I show this on my plot (either on the x axis or the y axis. If it's both, even better)?

I use pyplot and my OS is macOS

One can use the matplotlib.axes.Axes.errorbar class.

From this example in the documentation:

# example error bar values that vary with x-position
error = 0.1 + 0.2 * x

fig, (ax0, ax1) = plt.subplots(nrows=2, sharex=True)
ax0.errorbar(x, y, yerr=error, fmt='-o')
ax0.set_title('variable, symmetric error')

Acknowledgments: Thanks to Emre!


Contrasting logistic regression vs decision tree performance in specific example I have a set of 10,000 integers, and another set of 100.
The integers in the first set are mapped to integers in the second set according to some rules (not mathematical rules, think of these values as codes for naming certain items, it is some categorical mapping).
The mapping is not necessarily 100 to 1, in some case I may have just 30 or so integers from the first set mapped to an integer in the second set, in other cases 300, but on average of course it is 100 to 1.
Using sklearn, I created a decision tree that was able to get over 99% accuracy, as I would expect.
When I tried logistic regression, though, accuracy was just 45%. The training sample is about 100,000 example, so, it should be enough to learn.
What is going on?
Is there something inherently different in the logistic regression method that I am missing?

A few assumptions I'm making:

I believe you're mapping your input set of integers to another integer, which means your output is discrete, so I think you're using [multinomial logistic regression] (https://en.wikipedia.org/wiki/Multinomial_logistic_regression).
You're treating your discrete inputs (integers) as continuous--i.e., you're not using a one-hot-encoding for each integer.
Your underlying function is something to the like of if $\forall j=2,..,10$ if $x \in (x_{j-1}, x_j)$ then $y = c_j$, correct?

If the above three are true, then it's clear why Logistic won't work and why the Decision Tree will. The Logistic will approximate the relationship between your input and output with a $continuous$ linear relationship, which is not exactly what you have here because you have a discontinuity. It's worth mentioning that Decision Trees will perform better at these points, which means the more discontinuities you have the worse your performance will be.

But after a quick simulation, the magnitudes you quoted are too large to be coming from this. So I'm guessing you're using $binary$ logistic regression, which is just the wrong choice of model (since it's mapping everything to a single class instead of many classes).

So, it sounds like you're just using the wrong model.

Below is a quick demo in R.

# Loading packages
require(nnet)
require(rpart)
set.seed(024)
buildData <- function(n){
  x <- 1:n
  x_j <- data.frame(vals=quantile(1:n, probs=1:10/10))$vals
  c_j <- 1:10
  y <- rep(0, n)
  for(i in 1:length(c_j)){
    if(i==1){
      y <- ifelse(x <= x_j[i], c_j[i], y)
    } else {
      y <- ifelse( x >= x_j[i-1] & x <= x_j[i], c_j[i], y)
    }
  }
  print(table(y))
  xdf <- data.frame(x, y=factor(y))
  return( xdf)
}
# Building the data
n <- 1e4
xdf <- buildData(n)
# Random 90-10 split
trnflt <- runif(n) <= 0.5

# Multinomial Logistic Model
multinommod <- multinom(y~x, data=xdf)
xdf$preds_mult <- predict(multinommod, xdf, type='class')

# Binary Logistic Model
logisticmod <- glm(y~x, data=xdf, family='binomial')
xdf$preds_log <- ifelse(predict(logisticmod, xdf, type='response') <= 0.5, 0, 1)

# Decision Tree Model
treemod <- rpart(y~x, data=xdf[trnflt,], method='class')
xdf$preds_tree <- predict(treemod, xdf, type='class')

# Multinomial Logistic confusion Matrix
print(with(xdf[trnflt,], table(y, preds_mult )))
print(with(xdf[trnflt==F,], table(y, preds_mult )))

# Binary Logistic confusion Matrix
print(with(xdf[trnflt,], table(y, preds_log )))
print(with(xdf[trnflt==F,], table(y, preds_log )))

# Decision Tree confusion Matrix
print(with(xdf[trnflt,], table(y, preds_tree)))
print(with(xdf[trnflt==F,], table(y, preds_tree)))

# Here's the performance of all 3
print(c(paste('Multinomial Logistic Regression Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_mult )))) / sum(trnflt==F)),
        paste('Logistic Regression Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_log )))) / sum(trnflt==F)),
        paste('Decision Tree Accuraccy =', sum(diag(with(xdf[trnflt==F,], table(y, preds_tree )))) / sum(trnflt==F))))


Why there are no tutorials on calculating Gini impurity in a three-valued categorical feature variables? Trying to learn here so please go easy on me if I asked dumb questions.

As the title says, I was searching for a tutorial that calculates the Gini in a CART algorithm for a three-valued feature variable. (i.e., assuming we have a feature variable named, "Blood pressure" and it has three levels (low, medium, and High)). However, I didn't find one. A perfect example of that is the Play Tennis problem, we can have a solution using the entropy method, but not with the Gini method.

There are no categories, you encode these columns as integer ones and proceede as usual.
Gini is there just to help you make decision on how and where you split. Gini,Entropy,variance are all replaceable components.


Prediction from data series with varying features I'm looking into a problem where the data points have unequal features.

Each instance represents the progression of an item throughout the system. A number of them have progressed to their end point, others are still at intermediary stages. The number of stages is known (10 in total). I have the time when it enters each stage. The point is to make predictions as to when they will reach their final stage.

There are a number of ways to handle this. It seems that a decision tree might be able to handle this properly. Even if a stage's entry time isn't known, it should still be able to make a reasonable prediction from the data it does have.

Can someone point me in the right direction?

If you need to impute each missing value you could consider multiple imputation or interpolation methods for Time Series.  e.g. https://stackoverflow.com/questions/49308530/missing-values-in-time-series-in-python

If the goal is to simply predict the entry time to Stage 10 and not worry about when they hit the intermediate stages, you can treat is a regression problem i.e. just predict time to Stage 10.  The simplest way might be this - just treat it like a normal supervised learning problem with some missing data. The only extra step is you need to create some missingness in your training/test data to match the data you will score.

Your train/test data will consist of all records with the full data up to stage 10.

Use cross-validation or a train/test split within this data to create a predictive model to predict time to Stg10. Within this data randomly blank out data between Stg1 and Stg9 to simulate the scoring dataset but keep Stg10 as you need a label to predict.  Use a tree based method such as Random Forests as they are OK with the dependence between the variables and will deal with missing data.

Use this model to score the unlabelled data (i.e. where there is no Stg10 info).  The remaining problem is the level of completeness within the unlabelled dataset.  If they are filled up to Stg9 you have no problem.  If they all go to Stg1 you have a problem! So you will have to see if you have enough completeness in your data to support this method.

But it's a valid place to start and may turn out to be sufficient for your purpose..


How many observations in a neural networks dataset? I started to study and programming in neural networks for a little while now, but I never read about the minimum number of observations one must collect in a dataset to get robust results. Of course, more observations better results, but, Does exist an empirical or theoretical relationship between variables and observations number?

I mean, neither in econometrics you can compute the minimum number of observations, but it does exist some rule of thumbs that relies the number of exogenous variables to the target variable.

I wonder if there is something similar to that in neural networks too, but, till now, browsing on the internet, I did not find anything of useful.

Any ideas, advises or hint will be appreciated.

A neural network is nothing but a set of equations. And the basic rule of any set of equations is that you must have as many data points as the number of parameters.

The parameters of any neural network are its weights and biases.

So that means that as the neural network gets deeper and wider, the number of parameters increase a lot, and so must the data points.

This being said, the more proper and detailed way to know whether the model is overfitting is to check if the validation error is close to the training error. If yes, then the model is working fine. If no, then the model is most likely overfitting and that means that you need to reduce the size of your model or introduce regularization techniques.


Feature selection on n different values I have a .csv file with data in the following form:

moment_1;moment_2;moment_3;force_x;force_y;force_z;...
-0,02131267;-1,6032766088;5,9906811787;5,40010285;0,0203;86,44227467;...
2599;-1,70091039344;-1,3044809;-0,0406673590;-2,60896180797;43,2334;...

The file is very large and I need to put it in an interactive visualization, that's why I need to reduce the data points without changing the overall structure too much.

Many data points are very close to each other as seen in the following image:

My approach was to define a threshold and filter all points which have a distance to the previous point lower than the threshold. But I think that's not an optimal solution because, when I remove one index, I need to remove it from the other data array too, otherwise the structure is changed.

Are there better approaches?

Instead of filtering single points I would suggest that you smooth your data using established techniques, e.g. Savitzky–Golay filter. Another option would be to employ Kernel Density Estimation, where you can then visualize the curves using a reduced, regular set of supporting points.


Can features that are the same in every sample contribute to learning? For simplicity, let's say that I am monitoring 4 sensors for an ongoing metric.
The first column is the sensor ID and the second column is the sensor type.
[
  [
    [0, 0, 0.123],
    [1, 0, 0.456],
    [2, 1, 0.789],
    [3, 1, 0.555]
  ],
  [
    [0, 0, 0.987],
    [1, 0, 0.654],
    [2, 1, 0.321],
    [3, 1, 0.666]
  ],
  [
    [0, 0, 0.591],
    [1, 0, 0.824],
    [2, 1, 0.760],
    [3, 1, 0.888]
  ]
]

If the first two columns are always the same values, will a CNN or an LSTM be able to learn from these columns or are they just redundant?
In my mind, the sensor ID could correspond with a postion on the map where different metrics are observed. Or the sensor type could correspond with some sort of sensitivity in the metric. But am I just kidding myself if they are the same in every sample?
I don't want to provide unnecessary dimensionality to the model.

It has no value if it is same for all the training set.
Let's say, you are using a global health dataset for Life expectancy then country code can be a useful feature. It might contain hidden information.
But if you are doing the same analysis for one country e.g. India, keeping a feature country which has only one value e.g. India, will be of no use.
It will not show any variance, now the variance is shifted to States instead of Country compared to the last example.
In you data, I can see multiple Sensor Id. If very dataset will always have these 4 values and there is no explicit assumption for 3rd column which is dependent on 1st, you can remove the feature


How can I weigh observations differently that were provided for a time horizon? I have 623 observations with one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal), defined based on researcher experience and literature review.
I considered planning to do several regression analysis to estimate the dependent variable and study the predictive factors (if they are positive, negative, and their magnitude) on it. Data are provided for 10 years. Since the latest observations are more important, I’m interested in using weighted observations. How can I approach this problem and validate my approach?

Therefore, the year of the observation was considered as a predictor variable which had a positive impact on the dependent variable. However, the year itself doesn’t have any effect on the dependent variable per se; rather, it is the other factors occurring in the same time period which result in improvements.

This sounds like a challenge for trees with sufficient interaction depth, as you've found that the year interacts with other factors that results in improvements. Ordinary least squares regression here do not capture that type of interaction well.

I would suggest setting up the following regression model:

Transform the date column into (CurrentYear - YearOfDateStamp) AS NumOfYearsAway. I'd recommend leaving this as a numerical feature rather than a categorical one. This would allow the tree based model to select cuts like NumOfYearsAway >= 5.5 instead of NumOfYearsAway in (6,7,10). This could also be more helpful when NumOfYearsAway = 0 occurs in your scoring dataset, where you might not have training data for current year data set.
Fit a tree based model, I'd pick XGBoost, with the usual CV to tune the hyperparameters such as interaction depth.

The drawback of picking XGBoost in your application however, is that the interpretation of the impact of a particular variable on the target variable is not obvious. You'd need partial dependence plot to observe how the target variable vary with the bespoke input variable. If interpretability is very important, one could pick a single tree regression model like rpart.


How to find average lag time with variance & confidence of two time series I have two variables as time series, one a consequent of the other, I would like to find the average time delay it takes the dependent variable to act on the independent variable. Additionally, I would like to find the range of variance that is associated with the lag time and its respective confidence level. I am unsure how to go about this in a statistically valid way, but I am using Python.

Currently I have used np.diff(np.sign(np.diff(df))) to isolate the relative max & mins of the time series to then try to find the time gap between the subsequent pairs of mins & max but that doesn't seem too valid to me -- thoughts? The out put of the mins & maxes return an array like [0, -2,  0,  2,  0,  0, -2] where -2 is the relative min & 2 is relative max.

Methodological pointers would be greatly appreciated.

Thank you for your time & stay safe!

All the best,
RS

In a first approach, What you are looking for might be lagged correlation. You shift the trailling time serie by some variable time t and you look at correlation with the first time serie while varying the lag. The value of the lag which yield the biggest correlation is a good candidate for an estimation of time delay betweeen the two.
If you want more advanced statitics like variance (meaning that you have a variable lag) it might get more complex as you would have to identify which peak in the first time serie correspond to wich one in the second. Depending on your format of your time serie it might be easy or not. I would suggest to try a simple approach first : for each peak of your first time serie try to find the corresponding one in the second (first peak that appears after it for exemple). It might get complicated if some peaks overlap or the delay is too variable. Another option is to use more complicated things like Dynamic Time Warping Algorithms.


How to transform and cummulate data in python I would like to transform dataframe and cummulate them using pandas.

year country value
1999 JAPAN   10
2000 KOREA   15
2000 USA     20
2001 USA     13
2002 JAPAN   30

* I want to transform dataframe and cummulate value for each country

year country value
1999 JAPAN   10
1999 KOREA   0
1999 USA     0
2000 JAPAN   10
2000 KOREA   15
2000 USA     20
2001 JAPAN   10
2001 KOREA   15
2001 USA     33
2001 JAPAN   40
2001 KOREA   15
2001 USA     33

I need your help.

Thank you.

I think the following lines should give the output you are looking for:

# Create pivot table
df = pd.pivot_table(df, values="value", index="country", columns="year")
# Calculate cumulative sum and forward fill NaN
df = df.cumsum(axis=1).fillna(method="ffill", axis=1)
# Reshape data back into long format
df = df.reset_index().melt("country")

I first create a pivot table with the countries in the rows and the years in the columns to get all the years that may be missing in the long format data. I then calculate the cumulative sum, after which I forward fill the values over the columns. To finally reshape the data back into the original long format I reset the index and melt the dataframe with the country as the id column. Given your input, this returns the following dataframe:

   country  year  value
0    JAPAN  1999   10.0
1    KOREA  1999    NaN
2      USA  1999    NaN
3    JAPAN  2000   10.0
4    KOREA  2000   15.0
5      USA  2000   20.0
6    JAPAN  2001   10.0
7    KOREA  2001   15.0
8      USA  2001   33.0
9    JAPAN  2002   40.0
10   KOREA  2002   15.0
11     USA  2002   33.0


Which algorithm Doc2Vec uses? Like Word2vec is not a single algorithm but combination of two, namely, CBOW and Skip-Gram model; is Doc2Vec also a combination of any such algorithms? Or is it an algorithm in itself?

Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context words and the target word, these variants arised.

Note: the name of the model maybe confusing

Distriubted Bag of words is similar to Skip-gram model
Distributed Memory is similar to Continuous bag of words model


Can I have a neural network output values >100? All the samples and articles i have seen are all having outputs of 1 or less. Is there a hidden reason why no one is using NN to produce higher value integers?? My situation is that I want NN to predict the exact outcome value rather than a -1

If I have a NN with say 4 inputs and 1 output with
Round 1 Input: 51,22,35,43 Output: 847
Round 2 Input: 25,31,46,29 Output: 761
I want
Round n Input: 51,33,19,27 Output: ??? (expected value)

I have read that GA is better in training NN but few examples have been provided to guide beginners like me. Am yet to see a detailed article on how to get this done. Using same example above, if i get the (near) best weights from first evolution using Round 1 error value, do i add the weight as parents in the next generation for Round 2?? How do i ensure that the best chromosome from previous rounds don't die after that generation.

Links to existing articles and detailed explanation will be very much appreciated. Thanks.

Is sounds like you want to use Neural Networks to do a regression problem instead of classification.

This post gives an example of using a neural network to do regression.


Does it make sense to combine PCA with an artificial neural network? I have a Dataset of around 200 features. Most of them are categorical and only a few are numerical. It seems that an artificial neural network with an Autoencoder has some problems with that kind and amount of features. Therefore, I thought to use PCA to reduce the dimensions and then apply the Autoencoder afterwards.

Does the combination of a PCA before neural nets makes sense as the neural nets also reduces the information in internal layers? Does anybody has experiences with such a combination?

Edit: There is also an interesting answer in the stats forum.

Neural networks are actually extremely effective at performing dimensionality reduction. A great example is word2vec, which applies a shallow neural network to reduce inputs on the order of several million features (i.e. unnormalized text) to a 30-150 dimensional vector via a process that is mathematically analogous to matrix factorization (which is the class of techniques PCA belongs to). Autoencoders function very similarly to word2vec: if you're planning on using an autoencoder to learn an embedding of your data, I wouldn't expect you'd gain anything from applying PCA first rather than using an autoencoder (or something similar) to learn a better embedding for your data that isn't constrained by the assumptions of PCA.


Choose binary classification algorithm I have a binary classification problem:

Approximately 1000 samples in training set
10 attributes, including binary, numeric and categorical

Which algorithm is the best choice for this type of problem?

By default I'm going to start with SVM (preliminary having nominal attributes values converted to binary features), as it is considered the best for relatively clean and  not noisy data.

It's hard to say without knowing a little more about your dataset, and how separable your dataset is based on your feature vector, but I would probably suggest using extreme random forest over standard random forests because of your relatively small sample set.

Extreme random forests are pretty similar to standard random forests with the one exception that instead of optimizing splits on trees, extreme random forest makes splits at random. Initially this would seem like a negative, but it generally means that you have significantly better generalization and speed, though the AUC on your training set is likely to be a little worse.

Logistic regression is also a pretty solid bet for these kinds of tasks, though with your relatively low dimensionality and small sample size I would be worried about overfitting. You might want to check out using K-Nearest Neighbors since it often performs very will with low dimensionalities, but it doesn't usually handle categorical variables very well.

If I had to pick one without knowing more about the problem I would certainly place my bets on extreme random forest, as it's very likely to give you good generalization on this kind of dataset, and it also handles a mix of numerical and categorical data better than most other methods.


Cardinality vs width in the ResNext architecture I was recently reading the paper Aggregated Residual Transformations for Deep Neural Networks.

One thing the author mentions in Section (5.1) is that increasing the cardinality (or, the number of branches), decreases validation error more than increasing the bottleneck width or increasing the depth. I understand the depth part, but I'm a bit confused about the width. Isn't the cardinality of a residual block the same as the bottleneck width? If not, what is the difference?

Thanks!

Like in InceptionNet, which increases width, the different paths are depth-concatenated as well as each path is different (i.e different filter sizes for different paths).

While in ResNeXt, which increases cardinality, the different paths are merged by adding them together, and each path is similar.

Read this article for more information An Overview of ResNet and its Variants


Modeling Grocery Store Transactions I'm fiddling around with some data that represent grocery store transactions. The data are in the following form:

Each row represents a final transaction by a customer, with a column for user ID, timestamp of purchase, and basket contents. For example:

| ID | Timestamp  |              Basket           |

| 12 | 2016-04-02 | ['Celery', 'Beets', 'Cheese'] |

The question I'm trying to answer is "How do I suggest an ideal basket to an individual customer?"

Problems I'm dealing with:

I can't seem to think past just suggesting the most frequently ordered items to a customer. Is there some technique that I'm missing here?
Since we have a customer ID, we can have multiple baskets for a customer over time. Because of this, I can't use apriori or eclat as those assume each transaction is independent.
How can I take into account seasonality with whatever technique I choose?

I'd really appreciate some general direction here, I'm having some serious analysis paralysis.

I think that you can approach this problem in a better way. To do so you need some Customer Demographics. Even if you try doing some recommendation using customer ID it won't give you ideal Basket/Recommendation, because from Customer ID alone you can't decide anything.

Is there any possibility to get Transaction Id, so that you can unique baskets to apply Aprori Algo/ any association mining techniques by which you can give good recommendation using Lift and Support.

In the time stamp do you have time? As your sample record doesn't show time in it(just for confirmation), Let us consider a scenario you have it. Now ['Celery', 'Beets', 'Cheese'] - transform as Basket-1, ['Celery', 'Bread', 'Cheese'] as Basket-2 and so on... Now you see which Baskets are sold the most WRT timestamp. This is to see if there is any seasonality in the data and you can even find some trends, if there are any.

This is just one way of looking at the problem but this is not the concrete solution.


Multilabel user classification based on multiple posts I'm currently doing a course project for a course at university and I'm tasked on doing multilabel user profile classification based on their posts in social media, for each user there's a set of posts on social media which combined make up the user's occupation, gender etc.

In the past I've done simpler classification, where each post would have its own label and you would have to predict only the post's class, whereas in my current task I would have to somehow take into account all the posts of a user, since I only have the labels for the user, and not for each individual post.

My first idea would be to concatenate all the posts for each user and create a tf-idf vector which I would use for a classifier, but that way I would lose information from each individual tweet.

My question would simple concatenation suit my needs for this task, or are there more elaborate techniques which would take into account each individual tweet's importance?

My first idea would be to concatenate all the posts for each user and create a tf-idf vector which I would use for a classifier, but that way I would lose information from each individual tweet.

This is the way I would do it too, I think there is no way to preserve the individual information of tweets. But it's more valuable to have the summarised information of tweets for an individual person.


How to automate the encoding process? I am working on the Boston challenge hosted on Kaggle and I'm still refining my features.  Looking at the dataset, I realize that some columns need to be encoded in binary, some encoded in decimals (ranking them out of a scale of n) and some need to be one-hot-encoded.  I've collected these columns and categorized them in distinct lists (at least based on my judgement on how their data should be encoded):

categorical_columns = ['MSSubClass', 'MSZoning', 'Alley', 'LandContour', 'Neighborhood', 'Condition1', 'Condition2',
                       'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'MasVnrType', 'Foundation', 'Heating',
                       'Functional', 'GarageType', 'PavedDrive', 'SaleType', 'SaleCondition']

binary_columns = ['Street', 'CentralAir']

ranked_columns = ['LotShape', 'Utilities', 'LandSlope', 'ExterQual', 'ExterCond', 'BsmtQual', 'BsmtCond',
                  'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2', 'HeatingQC', 'KitchenQual', 'FireplaceQu',
                  'GareFinish', 'GarageQual', 'GarageCond', 'PoolQC', 'Fence', 'MiscFeature']

One fellow stackexchange user suggested that I use pandas.get_dummies() method to one-hot-encode categorical variables like MSZoning and attach it to a variable like this:

OHE_MSZoning = pd.get_dummies(train['MSZoning'])

I'd like to know how I can automate this process using functions and control-flow statements like for-loop.

I'm the fellow Stackexchange user, hi! I wrote the function that iterates the one-hot encoding on all your categorical_columns:

def serial_OHE(df, categorical_columns):

    # iterate on each categorical column
    for col in categorical_columns:

        # take one-hot encoding
        OHE_sdf = pd.get_dummies(df[col])

        # drop the old categorical column from original df
        df.drop(col, axis=1, inplace=True)

        # attach one-hot encoded columns to original dataframe
        df = pd.concat([df, OHE_sdf], axis = 1, ignore_index = True)

    return df

So you can call it like this:

df = serial_OHE(df, categorical_columns)

Let me know it there are any problems.


How can I detect if an image was photoshopped? I would like to check JPG files if they were manipulated to change the content.

What I consider NOT photoshopped:

Cropping
Rotating
(Scaling)
Image resolution
Automatic changes smartphones might make

What I consider photoshopping:

Adding a new image on top of parts of the old image
Changing text of a part of an image

How can this automatically be checked?

(And: Are there ready-to-use libraries for it?)

Error Level Analysis as described Error Level Analysis, found at https://github.com/afsalashyana/FakeImageDetection seems to be one seems to be one way:

You exploit that local compression ratios might be different. And it seems to be possible to train neural networks on it.

I didn't find a paper which says how well this works so far


Difference between rand index and adjusted rand index? I am unable to understand what is the adjusted term in ARI. The expected index term in the ARI is from a prior. Kindly explain

The adjustment is simply

(Rand index - Expected value)/(Optimal value - Expected value)

The purpose is to scale it in an interpretable way. 0 is "as good as random", less than 0 is worse, and close to 1 is good.

The problem with the non adjusted Rand index is that a random result on certain data sets can achieve a high score otherwise.


Variance in statistics vs machine learning In basic statistics, variance is a measure the variability of the data about its mean. In machine learning, variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data.

Since machine learning is mostly refurbished statistics, how does these two concepts relate to one another

Variance in machine learning appears in many places, but in all cases it is of course simply an application of the mathematical definition. For instance, people often worry about the variance of stochastic gradients $\mathbb{V}[\nabla \mathcal{L}]$ (especially when using score function/likelihood ratio gradient estimators), which refers to how much the gradient (treated as a random variable) varies, either over time OR as a function of the minibatch. If you took the same model, with many different minibatches, ideally you'd hope that the variance of the resulting gradient estimate would be low (sometimes true with large enough batch sizes).

For the specific case you are talking about

  variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data

This is the variance in model performance with respect to changes in the training data.

Here's a simple way to look at this. Let $\mathcal{D}=(\mathcal{X},\mathcal{Y})$ be the space of possible datasets and $P_D$ some distribution on that space (so $D\sim P_D$ where $D=(X,Y)\in\mathcal{D} $ is a random training dataset).
A specific training example is $(x,y)\in D$ with $x\in {X}$ and $y\in {Y}$.
We write $x,y\sim D$ to mean $D\sim P_D$ and $x,y\sim\mathcal{U}(D)$ where $\mathcal{U}$ denotes the discrete uniform distribution.
Now, let $f_\theta : X \rightarrow Y $ be a learned function parametrized by $\theta\in \Theta$.
Suppose $\mathcal{L}: \Theta\times \mathcal{D}\rightarrow \mathbb{R}$ is some loss function defined on the space of learned functions. Note that it is dependent on the dataset! For example, one possible definition of $\mathcal{L}$ could be
$$ \widetilde{\mathcal{L}}(\theta,D) = \frac{1}{|D|}\sum_{(x,y)\in D} L_s(f_\theta(x), y) $$
where $L_s : Y\times Y\rightarrow\mathbb{R}$ is some loss function for specific examples. This is one possible form, but a specific one is not necessary.

What is the variance then? We can consider $\mathcal{L}$ as a (random) function of $D$ by first assuming that we can compute $\theta$ given $D$, presumably by some stochastic training algorithm $A$, so that $\theta \sim A(D)$. Presumably, this looks something like stochastic gradient descent. Then our expected loss is
$$ \mu_\mathcal{L} =
\mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \mathcal{L}(\theta, D) \right] \right]
$$
meaning the variance in the loss as a function of the training set is given by
$$
\mathbb{V}[\mathcal{L}] = \mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \left(\mathcal{L}(\theta, D) - \mu_\mathcal{L}\right)^2 \right] \right]
$$
Essentially, for a fixed (random) training procedure, we compute the variance in the model performance as the training set varies.

This is pretty oversimplified and honestly non-standard, so
if you are interested in the formalities of this kind of thing, look into Probably Approximately Correct (PAC) Learning Theory and Empirical Risk Minimization, where this is formulated properly (meaning, rigorously). :)

The comments above discuss (very appropriately) an important special case of this, namely the Bias-Variance Decomposition for squared error.
A common simplification for this case is to assume that $P_D$ samples from a fixed set of paired values $(X\in\mathcal{X}, Y\in\mathcal{Y}=\mathbb{R})$, related by
$$y = h(x) + \xi, \;\;\; x\in X, \, y\in \mathbb{R},\, \xi\sim\eta(\mu_\xi,\sigma_\xi^2)$$
where $\mathbb{E}[\xi]=\mu_\xi$, $\mathbb{V}[\xi]=\sigma_\xi^2$, and $\xi$ independent from $x,y$.
We still therefore denote $D\sim P_D$ to mean sampling a random training set (meaning, draw $n=|D|$ random points $x_i\sim X$ and then perform  $y_i=h(x_i) + \xi_i$).
Note that $\theta$ (and $f_\theta$) a random variable, dependent on the random variable $D$, computed by some (potentially stochastic) procedure like
$$\theta^*(D) \leftarrow \arg\min_\theta \mathcal{L}(\theta,D) =
\arg\min_\theta \frac{1}{|D|}\sum_{(x,y)\in D} (y - f_\theta(x))^p + R(\theta) , $$
where $p\geq 1$ and $R$ is a regularizer, for instance.

Our expected squared error on a single fixed pair $(x,y)$ is then
\begin{align}
\mathbb{E}_{D\sim P_D}\left[ (y - f_\theta(x))^2 \right]
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 + 2h(x)\xi + \xi^2 \right]\\
   &\;\;\;\;\;\;\;- 2 \left( \mathbb{E}_{D\sim P_D}\left[ h(x)f_\theta(x) \right] +
               \mathbb{E}_{D\sim P_D}\left[ \xi f_\theta(x) \right] \right) \\
    &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}\left[ f_\theta(x)^2 \right] \\
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] \\
    &\;\;\;\;\;\;\;+ \underbrace{\sigma_\xi^2}_{\mathcal{V}_\xi} + \underbrace{\mu^2_\xi + 2\mu_\xi\mathbb{E}_{D\sim P_D}\left[ f_\theta(x) - h(x) \right]}_{\mathcal{B}_\xi} \\
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] + \mathcal{V}_\xi + \mathcal{B}_\xi \\
&= \underbrace{\mathbb{E}_{D\sim P_D}[h(x)]^2}_{h(x)^2} + \underbrace{\mathbb{V}_{D\sim P_D}[h(x)]}_0 - 2h(x)\mathbb{E}_{D\sim P_D}[f_\theta(x)] \\ &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}[f_\theta(x)]^2 + \underbrace{\mathbb{V}_{D\sim P_D}[f_\theta(x)]}_{\,\mathcal{V}_f} + \mathcal{V}_\xi + \mathcal{B}_\xi \\
&=
\underbrace{\left( h(x) - \mathbb{E}_{D\sim P_D}[{f_\theta(x)}] \right)^2}_\mathcal{B_f}+
\mathcal{V}_f +
\mathcal{V}_\xi + \mathcal{B}_\xi \\
&= \mathcal{V}_f + \mathcal{B}_f +
\mathcal{V}_\xi + \mathcal{B}_\xi
\end{align}
where we used $\mathbb{V}[r] = \mathbb{E}[r^2] - \mathbb{E}[r]^2$ for any random variable $r$, $\mathbb{E}[uv]=\mathbb{E}[u]\mathbb{E}[v]$ for independent $u$ and $v$, and the fact that $h(x)$ is deterministic (meaning $\mathbb{E}[h(x)]=h(x)$ and $\mathbb{V}[h(x)]=0$), since $x$ is fixed and we are varying the epxectation over $D$ (which only affects $\theta$).
Note that the "classical" bias-variance decomposition (as on Wikipedia) is a special case when $\mu_\xi = 0$.

In any case, we get that the expected error of our regressor on a fixed input (where we consider the expectation over possible datasets) is a sum of four terms (two variance and two bias):

The bias $\mathcal{B}_\xi$ and variance $\mathcal{V}_\xi$ caused by the inherent noise $\xi$ in the relation between $x$ and $y$.
The bias $\mathcal{B}_f$ which is how far the average prediction of $f_\theta$ is from the true value.
The variance $\mathcal{V}_f$ which is how "noisy" the output of $f_\theta$ is. Note this does not depend on the true value, nor on the noise in the true relation between $x$ and $y$. It is the inherent noise in the learning processes, measuring how much a change in the training set alters the resulting model.

So anyway, I got a little sidetracked, but all of these derivations were merely just to show that the variance in ML is in fact literally a simple mathematical variance. :)


Can an ordinal variable with a wide range be considered as a continuous integer variable in regression? Just as a trivial example:

I have a sample of 25 children and I have gathered some information example: weight, age etc - to use as predictors.

Now, I don't have the height of the children, but I have an ordinal variable with the shortest child having a value of 1 and tallest child having the value of 25 - "classification wrt height" variable

If I wish to build a regression model, with the "classification wrt height" variable as the response variable, shall I consider it as ordinal or continuous integer variable please (to select the type of regression needed to be applied accordingly)?

Thank you!

Think carefully before you do this. You have no idea what the underlying height distribution is. Here are four possibilities.

If you were building a regression model, each of these sets of height data would be interpreted differently. However, if replaced by your ordinal variable, they all look numerically equivalent. If you use this variable as a continuous variable, you are assuming a level of precision and information not present in the data.

Keep your ordinal variable as an ordinal variable. It may even be advantageous to bin this variable into fewer categories (for example "low", "low-middle", "middle", "middle-high", "high"). There are two reasons to do this.

This is a variable that has a unique value for each observation, which may lead to it looking disproportionately important. When you bin it like this, you remove that problem.
Think about future data that your model will be used to evaluate. Will those height classes in the new data reference the original classes? What if the new set has 100 observations? Now the new data has a range larger than what the model originally considered. Binning this variable is equivalent to normalizing a continuous variable.

Ultimately, your choice should be based on the performance of your regression model. If it behaves better with the variable as a continuous numerical variable, then do that. If the model behaves better with the variable as an ordinal variable in bins, do that.


Pandas dataframe with multiple hierarchical indices I have a data frame which looks like this
FRUIT       ID      COLOR       WEIGHT
Apple       142     Red         Heavy
Mango       231     Red         Light
Apple       764     Green       Light
Apple       543     Green       Heavy

And I want the following result:
FRUIT                 COUNT
Apple       COLOR
            Red         1
            Green       2

            WEIGHT
            Heavy       2
            Light       1

Mango       COLOR
            Red         1
            Green       0

            WEIGHT
            Heavy       0
            Light       1

I tried different variations of set_index, groupby() and unstack() on the dataframe in combination with ['ID]'.count() and .size(), but my grouping works only on the first level (Fruit).
On the second level I am not able to get color and weight separately, I always end up with weight as another subindex of color so it looks like
FRUIT
Apple       COLOR    WEIGHT   COUNT
            Red      Heavy    1
                     Light    0

            Green    Heavy    1
                     Light    1

How can I get my desired result?

I got it working with below simple line of code

df.set_index('fruit').stack().groupby(level=[0,1]).value_counts().unstack(level=[1,2]).fillna(0)


fbprophet - adding regressor During linear regression classes in academia, it is taught that including trivial/irrelevant features to the model decreases its ability to predict more accurately.

In fbprophet, there is this function, add_regressor(), which allows us to add additional regressors to the model.

I want to know:
- Is there a way to check whether added parameter/feature actually improves the model or it's actually trivial?
- What should I look for during the process of adding regressors to fbprophet?

Of course I prefer a more intuitive, smart way rather than simply checking the evaluation metric with and without the added regressor, since the improvement can simply be noise and cause overfitting, decreasing the ability to generalize on unseen data.

No, there is nothing really better than recalibrating the model and see if it improve performance. Most techniques will be approximation of what you are doing with your main model. If you happen to find a simpler model that perfectly explain which variable plays a role in your main model and how, you should probably just use the simpler model.

Basic a priori techniques (looking at correlation with output, with other variables) will fail in the context of complex machine learning models.

One practical way to go nowadays is to look at feature importance. Those techniques depends on the model you use and will provide an importance value for each feature of each instance. If a feature has a low importance for all instances you should try removing it. (you can do that by batches of k least important features).


Are there any differences between Recurrent Neural Networks and Residual Neural Networks? I'm currently studying the former and have heard of the latter, and right now I'm thinking that they're the same. Are they?

Definitely they are different. Very deep nets have exploding/vanishing gradient problem. The authors of ResNet paper had seen that by stacking many layers of convolution and dense layers, the learning did not increase although they used ReLU activation and batch normalization. They used a concept named skip connection which helped the nets to learn whether the input to a typical layer should be preserved or it should be transformed by that layer. Using this concept allowed them to increase the number of layers without hesitating whether they would have vanishing/exploding gradients. The concept of residual nets was originally this. The paper uses this concept for spatial data but recently I've seen people debating using them in temporal cases too __time series data.

Recurrent nets are used in temporal domains. Tasks like sequence classification are examples of their usage. In this domain the net should know the information of previous seen data. Well known examples of these nets are LSTMS. Early recurrent nets had vanishing/exploding gradient problem too. but after years LSTMs get popular amongst deep-learning practitioners. They defined a concept named gates which could learn when to forget and when to keep the previous data.


Neural net for server monitoring I'm looking at pybrain for taking server monitor alarms and determining the root cause of a problem. I'm happy with training it using supervised learning and curating the training data sets. The data is structured something like this:

Server Type A #1

Alarm type 1
Alarm type 2

Server Type A #2

Alarm type 1
Alarm type 2

Server Type B #1

Alarm type 99
Alarm type 2

So there are n servers, with x alarms that can be UP or DOWN. Both n and x are variable.

If Server A1 has alarm 1 & 2 as DOWN, then we can say that service a is down on that server and is the cause of the problem.

If alarm 1 is down on all servers, then we can say that service a is the cause.

There can potentially be multiple options for the cause, so straight classification doesn't seem appropriate.

I would also like to tie later sources of data to the net. Such as just scripts that ping some external service.

All the appropriate alarms may not be triggered at once, due to serial service checks, so it can start with one server down and then another server down 5 minutes later.

I'm trying to do some basic stuff at first:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer

INPUTS = 2
OUTPUTS = 1

# Build network

# 2 inputs, 3 hidden, 1 output neurons
net = buildNetwork(INPUTS, 3, OUTPUTS)

# Build dataset

# Dataset with 2 inputs and 1 output
ds = SupervisedDataSet(INPUTS, OUTPUTS)

# Add one sample, iterable of inputs and iterable of outputs
ds.addSample((0, 0), (0,))

# Train the network with the dataset
trainer = BackpropTrainer(net, ds)

# Train 1000 epochs
for x in xrange(10):
    trainer.train()

# Train infinite epochs until the error rate is low
trainer.trainUntilConvergence()

# Run an input over the network
result = net.activate([2, 1])

But I[m having a hard time mapping variable numbers of alarms to static numbers of inputs. For example, if we add an alarm to a server, or add a server, the whole net needs to be rebuilt. If that is something that needs to be done, I can do it, but want to know if there's a better way.

Another option I'm trying to think of, is have a different net for each type of server, but I don't see how I can draw an environment-wide conclusion, since it will just make evaluations on a single host, instead of all hosts at once.

Which type of algorithm should I use and how do I map the dataset to draw environment-wide conclusions as a whole with variable inputs?

In my opinion you're looking into the wrong methods to solve your problem.

You have strictly no numeric data.

Statistics based machine learning has a very hard time with such problems.
Your problem sounds more like one of the problems which should be solved with rule based systems. My first instinct would be to try to understand the rules and put them into code leading to a classification.

There are however methods for learning such rule based systems based on logic. They are only quite unfashionable for machine learning today. https://en.wikipedia.org/wiki/Rule-based_system


How Mllib in Spark select variables in logistic regression I have a question about MLlib in Spark.(with Scala)

I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but I now have to do it on Spark to be able to analyze Big Data.

How is the variable selection done? Is there any try of different variable combinations in LogisticRegressionWithLBFGS or LogisticRegressionWithSGD? Something like a test of significance of variable one by one? Or a correlation calculation with the variable of interest? Is there any calculation of BIC, AIC to choose the best model?

Because the model only returns weights and intercept...

How can I understand those Spark functions and compare to what I'm used to with SAS or R ?

First, the spark programming guide for LogisticRegressionWithSGD recommends using L-BFGS instead, so perhaps focus on the one.  As for variable selection, the model description on the MLLib page for regressions has a nice explanation of how models are constructed and selected, but it does not address variable selection.  This leads me to believe that it considers all variables, and simply chooses the model with the best fit.


Need help understanding xgboost's approximate split points proposal background:

in xgboost the $t$ iteration tries to fit a tree $f_t$ over all $n$ examples which minimizes the following objective:

$$\sum_{i=1}^n[g_if_t(x_i) + \frac{1}{2}h_if_t^2(x_i)]$$

where $g_i, h_i$ are first order and second order derivatives over our previous best estimation $\hat{y}$ (from iteration $t-1$):

$g_i=d_{\hat{y}}l(y_i, \hat{y}) $
$h_i=d^2_{\hat{y}}l(y_i, \hat{y}) $

and $l$ is our loss function.

The question (finally):

When building $f_t$ and considering a specific feature $k$ in a specific split, they use the following heuristic to assess only some split candidates: They sort all examples by their $x_k$, pass over the sorted list and sum their second derivative $h_i$. They consider a split candidate only when the sum changes more than $\epsilon$. Why is that???

The explanation they give eludes me:

They claim we can rewrite the previous equation like so:

$$\sum_{i=1}^n\frac{1}{2}h_i[f_t(x_i) - g_i/h_i]^2 + constant$$

and I fail to follow the algebra - can you show why is it equal?

And then they claim that "this is exactly weighted squared loss with labels $gi/hi$ and weights $h_i$" - a statement I agree with, but I don't understand how does it relate to the split candidate algorithm they are using...

Thanks and sorry if this is too long for this forum.

I won't go into details but the following should help you grasp the idea.

They use Quantiles (Wikipedia) to determine where to split. If you have 100 possible split points, $\{x_1, \cdots, x_{100}\}$ (sorted), you can try the $10$-quantiles split points $\{x_{10}, x_{20}, \cdots, x_{90}\}$ and have a good approximation already. This is what the $\epsilon$ parameter is doing. They consider a split point when the split has $\sim \epsilon N$ more points under it than the last split point. If $\epsilon = 0.01$, you will end up with $\sim 100$ split points, being larger than $\{1\%, 2\%, ..., 99\%\}$ of the other points. They do not consider a new split when "the sum changes more than $\epsilon$" but when the number of points under the current point is larger by $\epsilon$ than the last one.

Now, if you have a lot of continuous points very that are already well classified, it might be useless to split between them. You want to split the parts of your data set that are very wrong, the ones that are difficult to learn. To do so, they use weighted quantiles. This is where the weights play a role. The first $10$-quantile will not be the first point that is larger than $10\%$ of the points, but the first point that is larger than $10\%$ of the weights.


Type-II error in Hypothesis Testing I am new to Statistics and was reading about Decision errors in Hypothesis Testing. My question is that why is Type-II error an error at all? From what I understand, it arises when we fail to reject a false null hypothesis. When we fail to reject null hypothesis, it simply means that we do not have strong evidence to reject it. We are not making any comment about which of the two hypothesis is true (or false) . Either can be true. We are not saying that the null hypothesis is true. Then, why is such a conclusion called an error?

Basically, it's called an error because you're making the wrong decision. Like for instance, let's say your null hypothesis is something like: The patient does not have cancer. And the Alternative: The patient does have cancer. A type 2 error in this regard would be like saying "Oh, this patient does not have cancer" while they actually do have cancer. We are FAILING to REJECT a FALSE null. So I guess the most simple way to explain this is, we aren't catching a mistake, and letting it slide by. Of course this isn't all there is to it, but it should help kind of show what a type 2 error is and why it's something you want to avoid.


Extract canonical string from a list of noisy strings I have thousands of lists of strings, and each list has about 10 strings. Most strings in a given list are very similar, though some strings are (rarely) completely unrelated to the others and some strings contain irrelevant words. They can be considered to be noisy variations of a canonical string. I am looking for an algorithm or a library that will convert each list into this canonical string.

Here is one such list.

Star Wars: Episode IV A New Hope | StarWars.com
Star Wars Episode IV - A New Hope (1977)
Star Wars: Episode IV - A New Hope - Rotten Tomatoes
Watch Star Wars: Episode IV - A New Hope Online Free
Star Wars (1977) - Greatest Films
[REC] 4 poster promises death by outboard motor - SciFiNow

For this list, any string matching the regular expression ^Star Wars:? Episode IV (- )?A New Hope$ would be acceptable.

I have looked at Andrew Ng's course on Machine Learning on Coursera, but I was not able to find a similar problem.

As a naive solution I would suggest to first select the strings which contain the most frequent tokens inside the list. In this way you can get rid of irrelevant string.

In the second phrase I would do a majority voting. Assuming the 3 sentences:

Star Wars: Episode IV A New Hope | StarWars.com
Star Wars Episode IV - A New Hope (1977)
Star Wars: Episode IV - A New Hope - Rotten Tomatoes

I would go through the tokens one by one. We start by "Star". It wins as all the string start with it. "Wars" will also win. The next one is ":". It will also win.

All the tokens will ein in majority voting till "Hope". The next token after "Hope" will be either "|", or "(" or "-". None of the will win in majority voting so I will stop here!

Another solution would be probably to use Longest common subsequence.

As I said I have not though about it much. So there might be much more better solutions to your problem :-)


How do I fix space allocation error on a machine with plenty of space? I'd like to run a model on RStudio Server, but I'm getting this error.

  Error: cannot allocate vector of size 57.8 Gb

This is what my data looks like and it has 10,000 rows.

   latitude longitude                 close_date close_price
1  1.501986  86.35068 2014-08-16 22:25:31.925431   1302246.3
2 36.367095 -98.66428 2014-08-05 06:34:00.165876    147504.5
3 36.599284 -97.92470 2014-08-12 23:48:00.887510    137400.6
4 67.994791  64.68859 2014-08-17 05:27:01.404296    -14112.0

This is my model.

library(caret)
training.samples <- data$close_price %>%
  createDataPartition(p = 0.8, list = FALSE)
train.data  <- data[training.samples, ]
test.data <- datatraining.samples, ]

model <- train(
  close_price~., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 1),
  preProcess = c("center","scale"),
  tuneLength = 1
)

My EC2 instance has more than 57 GB available. This is the memory.

             total       used       free     shared    buffers     cached
Mem:      65951628     830424   65121204         64      23908     215484
-/+ buffers/cache:     591032   65360596
Swap:            0          0          0

And it has enough storage space, too. This is the hard drive space.

Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs        32965196      64  32965132   1% /dev
tmpfs           32975812       0  32975812   0% /dev/shm
/dev/xvda1     103079180 6135168  96843764   6% /

And these are details on the machine.

R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Amazon Linux AMI 2018.03

It doesn't matter that your instance has more than 57.8GB. R is asking for another 57.8GB on top of whatever it is already using. Not to mention any operating system overhead.

Shrink your dataset and see if it what you are doing works at small scale before trying to do it at big scale. Maybe it will work on 9,917 rows and that's good enough. Or it might fail on 500 rows in which case you need to rethink what you are doing.


Proper way to make Train/test split on Time-Series I want to create a model with LSTM to predict a user the next purchase value. For this I have used I used a user's purchase history. I have created the model and it works well, but honestly, I don't know I do the Train/Test split on the proper way or not.
To do this, I have used (univariate) user's purchase history.(X-purchase history values, y-target purchase value) As a first step, I have created a sliding-window process that creates new data. (As you can see in the pic) In the original dataset, I had 1000 users with 2820 timestamps and 1 feature (purchase values), with the Sliding-window process I got 1000*2320 users with 500 timestamps and 1 feature.
X.shape -> OriginalDataShape (1000, 2820, 1)
X.shape -> ModifiedDataShape (2320000, 500, 1)

# model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(500,1))) model.add(Dense(1)) model.compile(loss='mae', optimizer='adam')

# train
repeats = range(3)
scores = list()
for i in repeats:
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
    model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5, batch_size=32)
    pred = model.predict(X_test)
    score.append(metrics.mean_squared_error(pred,y_test))
print('Final score (Mse):')
print(score)

My questions are: It is the proper way or not? If it is not, do you have a suggestion or GitHub link for the solution?

The problem here is that you're shuffling the time-series before splitting it.
This way, every time-step in the test set might have a time-step close to it in the train set.
To avoid this, you can set shuffle=False in train_test_split (so that the train set is before the test set), or use Group K-Fold with the date as the group (so whole days are either in the train or test set).
You can read more in this question in Cross Validated


Multi-class text classification with LSTM in Keras I'm quite new to Deep Learning and trying to solve the problem of Multi-Class, multi-label text classification using Deep Learning.

https://github.com/fchollet/keras/blob/master/examples/imdb_cnn_lstm.py .
I've another dataset. int form of a csv file ("text","classifier"), on which i want to perform text classification task. I've tried a few ways to pass my training text to keras but couldn't so I'm stuck at this point. Can anyone suggest me how should I pass my "train.csv" and "test.csv" file to the X_train, y_train and X_test, y_test?

Typically stuck in this line.

(X_train, y_train), (X_test, y_test) = imdb.load_data(nb_words=max_features)

'train.csv' has this format:

"Job Description:An ideal fitment would apply his/ her advanced analytics expertise at a cutting edge Industrial Analytics specialized Data Science organization; primarily, in any of the following areas- automotive/ energy/ oil & gas/ aerospace/ marine/ chemical.  Experience in statistical modeling, predictive modeling, Random forests, Decision tree, Linear Regression, Correlation, Time- series.  BE / MS/ PhD in Mechanical/ OR/ IE/ computer science/ EE/ chemical.    Mentor/ Lead a small team of data scientist",Business Analytics

'test.csv' has same format that is "job_description","category"

After I read the source code, I find out that keras.datasets.imdb.load_data doesn't actually load the plain text data and convert them into vector, it just loads the vector which has been converted before.

As for your problem, I assume you want to convert your job_description into vector. Maybe you can try sklearn.feature_extraction.text.CountVectorizer.

I'm no expert on NLP, but I've encountered problems (e.g. Two Sigma Connect: Rental Listing Inquiries) which need such technique.

In the meanwhile, there are other word2vec/embedding techniques you may try. Here is a tutorial from keras which shows a detailed example on word embedding.

References

https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py

http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

https://www.kaggle.com/sudalairajkumar/two-sigma-connect-rental-listing-inquiries/xgb-starter-in-python


How to set batch_size, steps_per epoch and validation steps I am starting to learn CNNs using Keras. I am using the theano backend.

I don't understand how to set values to:

batch_size,
steps per epoch,
validation_steps.

What should be the value set to batch_size, steps per epoch, and validation steps if I have 240,000 samples in the training set and 80,000 in the test set?

batch_size determines the number of samples in each mini batch. Its maximum is the number of all samples, which makes gradient descent accurate, the loss will decrease towards the minimum if the learning rate is small enough, but iterations are slower. Its minimum is 1, resulting in stochastic gradient descent: Fast but the direction of the gradient step is based only on one example, the loss may jump around. batch_size allows to adjust between the two extremes: accurate gradient direction and fast iteration. Also, the maximum value for batch_size may be limited if your model + data set does not fit into the available (GPU) memory.
steps_per_epoch the number of batch iterations before a training epoch is considered finished. If you have a training set of fixed size you can ignore it but it may be useful if you have a huge data set or if you are generating random data augmentations on the fly, i.e. if your training set has a (generated) infinite size. If you have the time to go through your whole training data set I recommend to skip this parameter.
validation_steps similar to steps_per_epoch but on the validation data set instead on the training data. If you have the time to go through your whole validation data set I recommend to skip this parameter.


which metric is better for boosting methods I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one.

I want to know in this case which metric should I use it and WHY?

Accuracy ??
AUC Test value ??
RMSE  ??
LogLoss ??

Depends.

The first thing that has to be clear is that you are running an experiment, which means you need to measure both with the same metric.

Which one? Depends on which underlying problem you are solving, if what you are doing is to determine which algorithm is better, your conclusion will only be applicable to your specific dataset

Accuracy: Is possible to measure the accuracy as the comparing metric, but it becomes trivial if your dataset is unbalanced, which means you have much more positives than negatives or viceversa. The accuracy is used when the dataset is balanced and if is equally bad to have a mistake on positives and negatives. Also, it has the problem of being too dependant on the threshold for defining positives/negatives.
Area Under the Curve: The AUC is one of the most robust metrics for measuring the ability of a model to separate positives and negatives, is unsensitive to threshold and immune to unbalanceness. I would use this.
RMSE: I only know RMSE for continuous regression, not for classification.
LogLoss: Its use is in multinomial classification


Does adding a model complexity penalty to the loss function allow you to skip cross-validation? It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

There is no free beer in such matters so "automatically takes care of the danger of overfitting the data" is not true. When you optimize for a loss error function and also for simplicity the immediate question is how do you trade off simplicity on loss error. There is no answer to that, other than in very specific cases (but those cases are so constraint so they can't provide a generic answer).
This is the reason why many models which allows you to do this trade off will give you a parameter or more to use as knobs for this trade off. Examples are logistic regression with regularization, neural nets with dropouts, decision trees with max tree levels, boosting models with learning rates, and how many more. Even if the model does this trade off automatically, it does not mean it is the best trade off, mostly it is some trade off incorporated into the model structure and its assumptions.
The second aspect of your question is that you do not consider the whole scope of cross validation and of similar medthods in general. CV is used to assess the performance of different models, either if they have the same structure but with different parameters, either they are from different families. CV allows you to have a reliable hind regarding which will be the error of a model if used on similarly distributed new data. This is not covered by regularization at all.


Linear Regression Model I'm taking a course on Supervised Learning in R: Regression. There is a section where I'm supposed to predict blood pressure given age and weight. This is was MY approach

# Create the formula and print it
fmla <- lm(blood_pressure ~ age + weight, data=bloodpressure)
fmla

# Fit the model: bloodpressure_model
bloodpressure_model <- fmla

# Print bloodpressure_model and call summary()
bloodpressure_model
summary(bloodpressure_model)

It was an incorrect submission. The message error message read -  "The contents of the variable fmla aren't correct."

DataCamp's solution was this

# bloodpressure is in the workspace
summary(bloodpressure)

# Create the formula and print it
fmla <- blood_pressure ~ age + weight
fmla <- lm(blood_pressure ~ age + weight, data=bloodpressure)

fmla

# Fit the model: bloodpressure_model
bloodpressure_model <- lm(fmla, data = bloodpressure)

# Print bloodpressure_model and call summary()
bloodpressure_model
summary(bloodpressure_model)

Both of these models had  the same diagnostic results. What's the issue with MY approach?

It should be a bug in their server. The fmla variable should have the same contents in your code and in theirs. This is because the last assignment on both scripts is

fmla <- lm(blood_pressure ~ age + weight, data=bloodpressure)


How to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV? Could you please let me know how to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV?
# Use scikit-learn to grid search the batch size and epochs
from collections import Counter
from sklearn.model_selection import train_test_split,StratifiedKFold,learning_curve,validation_curve,GridSearchCV
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
import numpy as np
from sklearn.model_selection import GridSearchCV
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import classification_report
import pandas as pd
from sklearn.pipeline import Pipeline

# Function to create model, required for KerasClassifier
def create_model():
    # create model
    model = Sequential()
    model.add(Dense(12, input_dim=20, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)
# load dataset
X, y = make_classification(n_classes=2, class_sep=2,weights=[0.95, 0.05], n_informative=3, n_redundant=2, flip_y=0, n_features=20, n_clusters_per_class=1, n_samples=1000, random_state=10)
print('Original dataset shape {}'.format(Counter(y)))

ln = X.shape

X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
st=StandardScaler()

# create model
model = KerasClassifier(build_fn=create_model, verbose=0)
pipeline = Pipeline(steps=[('scaler', st),
                               ('clf', model )])
# define the grid search parameters
batch_size = [20, 40, 60, 80, 100]
epochs = [ 50, 100]
param_grid = dict(clf__batch_size=batch_size, clf__epochs=epochs)
cv = StratifiedKFold(n_splits=5, random_state=42)

grid = GridSearchCV(estimator=pipeline, param_grid=param_grid,cv=cv,scoring="f1")
grid_result = grid.fit(X_train, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
# Predictions
ypred = grid_result.predict(X_train)
print(classification_report(y_train, ypred))
print('######################')
ypred2 = grid_result.predict(X_test)
print(classification_report(y_test, ypred2))

grid_result = grid.fit(X_train, y_train, clf__class_weight={0:0.95, 1:0.05})

FYI, per the docs fit_params should no longer be passed to the GridSearchCV constructor as a dict, but should be passed directly to fit as above.

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html


A way to init sentence embedding for unsupervised text clustering, better than glove wordvec? For unsupervised text clustering, the key thing is the init embedding for text.
If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model.
BERT can not get good init embedding.
If we do not use deep model, is there better way to get embedding better than glove wordvec?

Generally, combining word vectors in a single sentence / document representation does not work extremely well, although the average embedding has been used in fastText and pooling in this paper.
You can also use autoencoders to try and predict the word distribution, similar to a bag-of-words approach, like here.


Count the frequncy of words in a cell of a column in a series I want to calculate the frequency of the words in obama['text'] (obama is the variable where i have stored this series element ) in a dictionary  and store it in another column . Without using Counter library , how do i do that . The data is in this format :

 URI                                      |              name                |  text
<http://dbpedia.org/resource/Barack_Obama>           Barack Obama               barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureateduring his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief

The output should be in the format in a new column obama['word count']:

{ 2009:4 , the :40 , chicago :10
and so on  }

Why shouldn't you use the Counter class; it's exactly what you need?

from pandas import Series
from collections import Counter
text="barack hussein obama ii brk husen bm born august 4 1961 is the 44th and current president of the united states and the first african american to hold the office born in honolulu hawaii obama is a graduate of columbia university and harvard law school where he served as president of the harvard law review he was a community organizer in chicago before earning his law degree he worked as a civil rights attorney and taught constitutional law at the university of chicago law school from 1992 to 2004 he served three terms representing the 13th district in the illinois senate from 1997 to 2004 running unsuccessfully for the united states house of representatives in 2000in 2004 obama received national attention during his campaign to represent illinois in the united states senate with his victory in the march democratic party primary his keynote address at the democratic national convention in july and his election to the senate in november he began his presidential campaign in 2007 and after a close primary campaign against hillary rodham clinton in 2008 he won sufficient delegates in the democratic party primaries to receive the presidential nomination he then defeated republican nominee john mccain in the general election and was inaugurated as president on january 20 2009 nine months after his election obama was named the 2009 nobel peace prize laureate during his first two years in office obama signed into law economic stimulus legislation in response to the great recession in the form of the american recovery and reinvestment act of 2009 and the tax relief"
df = Series(text).to_frame()
newdf = df.assign(word_count = lambda x: x[0].str.split(' ').apply(Counter)[0])
newdf['word_count']

  0    {'44th': 1, 'born': 2, 'november': 1, 'running...


Visualizing a large number of variables in python I have a large number of variables (2000) and want to visualize them using Python. Also, I would like to see the summary statistics of these variables. It seems that drawing box plots for all of these variables would not be feasible. What are some methods that could be used to visualize this data?

You can load these variables to pandas dataframe and use dataframe.describe() for summary statistics.


Sentence to word similarity Is there a way to know how much a sentence is related to a word/topic?
For instance the following dataframe and the topics/attributes Romantique, Feminine, ...:
    comments
0   Très contente de mon achat. Je cherchais ce parfum depuis un temps en magasin et je suis heureuse qu’il soit disponible en ligne il sent tellement bon !! En plus en promo, génial ! \r\nLivraison très rapide !
1   J’adore les parfums de cette marque car je trouve qu’ils sont captivant et surtout ils tiennent toute la journée ! Ils ont des odeurs originales et que l’on ne retrouve pas partout ! Je conseil fortement
2   Le parfum ideal pour porter pendant toutes les saisons du matin à nuit !!!
3   Très bon parfum floral, envoûtant au note de Jasmin qui reste toute la journée\r\nCorresponds aux personnes qui aiment les parfums florales assez imposante
...

As a start I thought about doing a jaccard_similarity distance ...
>>>from collections import Counter
>>>Counter(df['comments'].apply(lambda x: x.split(' ')).apply(lambda x: jaccard_similarity(x,['féminin'])))
Counter({0.0: 1344, 0.025: 21, 0.05: 21, 0.0625: 21})

But is there a better way to see how much a sentence relate to a targeted word?
Update
My main goal is to compare the proportion of people who used some topics in their comments of a product with the presence or absence of these topics in the description of the product. I used a model which use the synonyms:
d = {}
for product in collection.find():
  d_product = {}
  name = product['q0']['Results'][0]['Name']
  description = product['q0']['Results'][0]['Description']
  comments = short_comments_df(product['q2'])['comments']
  #for every attributes
  for attribut in attributs:
    consumers_approved = 0
    # Is the attribut, or its synonyms, in the comments?
    try:
      # if the attribute or it synonyms are in the description then the product has the attribut
      product_approved = presence(synonymes[unidecode.unidecode(attribut)], description)
      # we test every comment to see if they talked about the attribute
      for comment in comments:
        # We only take the nouns and the verbs
        lemmatized_comment = lemmatize_pos_filtering(comment)
        # if the attributes are in the comments then we increment the consumer approved counter
        consumers_approved += presence(synonymes[unidecode.unidecode(attribut)],lemmatized_comment)
      # we take the proportion of people who used the attribute, but shouldn't we normalize it?
      proportion_approved = consumers_approved/len(record['q2']['Results'])
    except IndexError:
      print("IndexError: ",attribut)
    # we use the difference between if we found it in the description and the % of people who found it as well
    d_product[attribut] = product_approved - proportion_approved
  d[name] = d_product

df = pd.DataFrame(d)

It produces the following graph:

It's weird because it shows that for most products the difference between the presence/absence of any topic compared to its presence/absence in the description is the same for most topics, but different from 0! Everything that is above zero means that at least the description has it and none of the comments, but everything below zero means that the comments have it but not the description. What strikes me is these straight lines below zero. It means that the presence/absence of a given attribute in absciss is the same for every comment...

If you have a list of words for every topic you can indeed try to directly measure the similarity of this list against a sentence, but it's likely that a sentence doesn't always contain one of the topic words so it might not work very well.
A more advanced method would be to obtain a semantic representation for every topic (or topic word) from an external corpus. Any large corpus can be used for that, it doesn't have to be related to your input data. The traditional way to do that is to extract a context vector for every target word by counting the co-occurrences of the target in the corpus: for every occurrence of the target word, take for instance a window of 5 words to its left and 5 words to its right. Then count how many times every context word appears in the window of the target across the whole corpus. This way the final context vector contains the distribution of the context words, i.e. a representation of the meaning of the target word. Comparing this context vector against a sentence is likely to produce a more accurate semantic similarity score.
There are many variants about the exact definition of the context vector: usually stop words are removed but one could also use TF-IDF and/or other kinds of normalization. The more modern version of this method is probably to use word embeddings, but I'm not knowledgeable enough about this.


Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'? Tom White's 'Hadoop: the Definitive Guide' has become a popular guide to the entire Hadoop ecosystem and earned a reputation as providing both a broad survey, as well as covering individual aspects of Hadoop in decent depth. Has anyone thus far attempted to provide the Spark equivalent?

Learning Spark: Lightning Fast Big Data Analytics is a fairly comprehensive book covering the core concepts as well as the higher level components involved in the Spark stack. This is the book recommended by Databricks for their Spark Developer Certification as well.

If you are interested more about the use cases built using Spark, I suggest Advanced Analytics with Spark


Why would 2 sets of similar training samples take significantly longer to train? I've built a fully-connected feed-forward neural network to recognize handwritten digits. I used MNIST and another very similar dataset (containing Arabic digits - same training set and test set count as those of MNIST).

For a network with exactly the same architecture (1 hidden layer, same learning rate, same weight sampling), I tested on a sub-sample that used 100 images for training. The test accuracy was very low, but what I noticed was that for the Arabic digits, the network took significantly longer to train (double the amount of time than for English digits - 33ms and 15ms, respectively).

What could be the reason for this?

There could be multiple factors that could cause this to happen:

If the data types are different for the two images - np.float32 vs np.float64
Your computer might be performing some compression and decompression to optimize memory usage. If most of the pixels are very different from each other in the Arabic dataset, then the compressed size may be larger and require more time to decompress.
Your computer happened to have other competing processes running at the same time and distorted the execution results for one of your runs.

I would suggest you to use methods like timeit that do multiple runs to verify that this behaviour is consistent.


how to evaluate feature quality for decision tree model Most of the tutorials assume that the features are known before generating the model and give no way to select 'good' feature and to discard 'bad' ones.

The naive method is to test the model with new features and see how the new results change compared to the previous model but it can be complex to interpret when the tree is complex.

Is there an academic way to select good features and to discard bad ones?

(ressources appreciated)

The main reasons for seeking an efficient feature selection are the machine learning algorithm get faster training, reduces the complexity of a model, facilitates interpretation and improves the accuracy of a model.

Look for Filter Methods , Wrapper Methods and Embedded Methods to learn more about your issue.

Filter methods are generally used as a preprocessing step. The selection of features is independent of any machine learning algorithms. Instead, features are selected on the basis of their scores in various statistical tests for their correlation with the outcome variable.  Here you have to look for Linear discriminant analysis, Pearson’s Correlation, Chi-Square.

Some common examples of wrapper methods are :

Forward Selection: Is an iterative method in which we start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.

Backward Elimination: Here, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.

Recursive Feature elimination: It is a greedy optimization algorithm which aims to find the best performing feature subset. It repeatedly creates models and keeps aside the best or the worst performing feature at each iteration. It constructs the next model with the left features until all the features are exhausted. It then ranks the features based on the order of their elimination.

Embedded methods combine the qualities’ of filter and wrapper methods. It’s implemented by algorithms that have their own built-in feature selection methods.

Some of the most popular examples of these methods are LASSO and RIDGE regression which have inbuilt penalization functions to reduce overfitting.

Other example of embedded methods that could fits you is Regularized trees.

follow the link with some of these algorithms in sklearn.

sklearn - Feature Selection

I hope this could help you to start.


I get this error Expected Ptr for argument '%s' while resizing image in open cv2 This is my code:

# loading image from folder
folders = glob.glob('path to folder ')

imagenames_list = []

for folder in folders:
    for f in glob.glob(folder+'/*.jpg'):
        imagenames_list.append(f)

read_images = []

for image in imagenames_list:
    read_images.append(cv2.imread(image, cv2.IMREAD_GRAYSCALE))

re_size = cv2.resize(read_images, (50,50), interpolation=cv2.INTER_LINEAR)

I get this error

Expected Ptr<cv::UMat> for argument '%s'

What does this mean and how can I fix it?

I changed the data type from float16 to float32, solving the problem:

images.astype(np.float32)


What is a good method for detection of rare occurencies of speech in noisy audio data? I have some audio recordings (with relatively static but noisy background, e.g., wind in an open area) with small number of short occurrences of speech (~1% of the total audio duration).

What would be a good method to detect the speech occurences in an unsupervised manner?

I have tried simple thresholding on a spectrogram, but this is problematic since:

The intensity of the background can wary with time (i.e. noise is louder sometimes)
Different speech segments need not to be similar to each other
Often, speech is too quiet (compared to the average loudness of the background) and is overlaid by noise

This may seem like quite a hard task, however I can easily notice the speech segments by listening to the audio/looking at the spectrogram, since spectrogram of speech has some distinct structure (although it is non-trivial to rely on the structure for detection as it is still quite non-regular).

Note that I just want to detect intervals with something sounding like human speech (or, say, something distinct enough from the background, since data typically has no other sound sources besides background/speech).

Detecting speech specifically is a well known problem, generally called Voice Activity Detection (VAD).
The simplest mechanisms just calculate a time-averaged ratio of energy in the speech frequencies compared to total energy, many implementations on Github of this idea. The audio codes for speech also tend to include a VAD component. A modern example can be found in libvad, based on the VAD from WebRTC project.

More generally you can try Anomaly Detection using algorithms like Isolation Forest or LocalOutlierFactor. What is critical is to use good features as input. A good starting point would be to calculate mel-spectrogram or MFCC. The spectrogram frames should be normalized, typically by subtracting the median/mean and dividing by RMS energy.
Run the anomaly detection on individual frames of 20-50ms and then do some filtering of classifications to reduce false triggering. A simple method is to require N consecutive frames to be classified as an anomaly to consider it a real anomaly. Or to require at least P percent of frames within a larger rolling window (say 200-1000ms).

In Python, you can use librosa for audio feature extraction and scikit-learn for anomaly detection

Note that for evaluating your method you will need to collect and label a test-set.


Big Data and a fwf files reading with filters I have searched at all possible areas and haven't been able to find the solution to my problem. I am hoping that someone here would be able to help me out. If you feel that this question is already answered, I would appreciate the link.

I am using R3.2.2 64 bit on a windows 7 machine. I have a fixed width file of size 1.9 GB and has approx 18 million records in it. I need to read the file based on a filter condition RecordID = "Cast". All my attempts to read the file.dat has failed due to memory constraints. Any help on how to use package ff for this purpose?

I am open to other methods also.

Try the LaF package. See this:
https://stackoverflow.com/a/24716798/1427069

Once you create a laf object, you can create a data frame or data table using your row filter condition.

library(LaF)
laf <- laf_open_fwf("foo.dat",
                    column_widths = colWidths,
                    column_types=rep("character", length(colWidths)),
                    column_names = colNames)
castData <- laf[RecordID == "Cast", ]