what happens to the sample points on pruned leaves I think I understand what pruning is (in concept) in a decision tree. What is not so clear to me (in reading) is what happens to the pruned observations. What do you do with them? Are they just simply not used and forgotten?
What bothers me is the "hole" left behind by the pruned node. What if a future $Y$ needs to be predicted at that node? Do you go to the parent node and treat that parent node as an end node and predict $Y$ by the data at that parent node (even though some data at the parent node continue to be divided in sub-nodes, other than the pruned node)?

Once you prune nodes, they get removed from the tree altogether, so you wouldn't be able to predict based on them.
Typically, hyperparaneter tuning is used to ensure that the right nodes are pruned. Obviously, there is always a trade-off.


How to Normalize & Scale a Single Data Point I do understand the concept of normalizing & scaling the training/test data; it does help with the converging of the cost function. It is a great helper for many of the machine learning algorithms.

I train and validate my model with the normalized data (MinMaxScaler) and save my model.
A new input data comes in and I want to use my saved model to make predictions.
I no longer have access to the training/test data at this point. All I have is the new single row of input data.
How will I normalize this single vector of input data so that it can be fed to my model?

The only normalization that I can think of simply linear transformation of data (e.g. simply map values from [40, 70] range to [-1, 1]). I need a normalization technique that doesn't depend on the full range of data.

Thanks!

You need to normalise the input in the same way that the training data was normalised -- however, you don't need access to this training data during predictions of new data. If you have used a MinMaxScaler for example, then you can re-use this to transform your new data point:

scaler = MinMaxScaler()
X_normalised = scaler.fit_transform(X)

# do other stuff here like train & validate your model

# now we have a new test data point from somewhere, we scale
# it using the scaler we fitted on the training data
new_test_point_normalised = scaler.transform(new_test_point)

# now we can classify it with our model!

You can serialise the scaler to disk using pickle and re-load it when predicting new data points if necessary (i.e., the model training and testing happen in two different scripts).


Change values from nominal to numeric I want to change the values of the class labels from nominal into numeric.
e.g if the values of a class are {iris-setosa,iris-virginica,iris-versicolor} i want to make them {0,1,2} so the instances will have as a value at the class label the form (0,0,1,2,0,1,1,2,0).
Any idea?

It's called Label Encoding

In python with the help of scikit-learn you can do following:

  le = preprocessing.LabelEncoder() le.fit(["paris", "paris", "tokyo",
  "amsterdam"])

  list(le.classes_)

  le.transform(["tokyo", "tokyo", "paris"])

  list(le.inverse_transform([2, 2, 1]))

Documentation for more info: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html

PS: Label Encoding might be responsible for unintentional ordering of possible values. I.E. model might think that nominal value with associated numerical value 2 is more important than the nominal value with associated numerical value 1. The solution is One Hot Encoding.


Imbalanced Dataset (Transformers): How to Decide on Class Weights? I'm using SimpleTranformers to train and evaluate a model.
Since the dataset I am using is severely imbalanced, it is recommended that I assign weights to each label. An example of assigning weights for SimpleTranformers is given here.
My question, however, is: How exactly do I choose what's the appropriate weight for each class? Is there a specific methodology, e.g., a formula that uses the ratio of the labels?
Follow-up question: Are the weights used for the same dataset "universal"? I.e., if I use a totally different model, can I use the same weights or should I assign different weights depending on the model.
p.s.1. If it makes any difference, I'm using roBERTa.
p.s.2. There is a similar question here, however, I believe that my question is not a duplicate because a) the attached question is about Keras where my question is about Transformers, and b) I'm also asking about general recommendations of how weight values are decided where the attached question is not.

I am not sure about the model you are using but I might explain what the procedure is for ML in general. You have three "vanilla" solutions for coping with unbalanced supervised dataset.

Reweighing class label so that there is the same number (calculated as sum of weights for given label) of samples per label. For example if a label with maximum number of samples has $n_{max}$ samples, and some other class has $n_i$ samples, then you would assign a weight $w_i=\frac{n_{max}}{n_i}$.
Undersampling - a basic procedure that gets rids of all the additional samples so that you end up with a balanced dataset.
Oversampling - creating copies of the unbalanced classes (with less samples than the $n_{max}$)
Hope that helps,

Max


Loading own train data and labels in dataloader using pytorch? I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader?

I have a dataset that I created and the training data has 20k samples and the labels are also separate. Lets say I want to load a dataset in the model, shuffle each time and use the batch size that I prefer. The Dataloader function does that. How can I combine and put them in the function so that I can train it in the model in pytorch?

Assuming both of x_data and labels are lists or numpy arrays,

train_data = []
for i in range(len(x_data)):
   train_data.append([x_data[i], labels[i]])

trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100)
i1, l1 = next(iter(trainloader))
print(i1.shape)


Different activation function in same layer of a Neural network My question is that what will happen if I arrange different activation functions in the same layer of a neural network and continue the same trend for the other hidden layers.

Suppose I have 3 relu units in the starting and 3 tanh after that and other activation function I the same hidden layer and for the other hidden layers I am scaling all the nodes with the same scale (decreasing/increasing) and the arrangement and order of the activation function is not changing.

At a higher level, using multiple activation functions in a NN may not work as these activation functions operate differently for the same inputs. For an input of -1.56, ReLU will a 0, sigmoid will give a 0.174 and Tanh will give -0.91. These differences will not allow the gradients to flow uniformly during backpropagation.

You may try using two different activation functions in the same layers. I can think of some issues which may arise,

Talking about the combination of Tanh and ReLU functions, we can face the problem of exploding gradients. ReLU functions provide the same inputs as outputs if they're zero or positive. On the other hand, Tanh function provides outputs in the range [ -1, 1 ]. Large positive values will pass through the ReLU function unchanged but while passing through the Tanh function, you'll always get a fully saturated firing i.e an output of 1 always.
Therefore, while using ReLU in a classification problem, we use softmax instead of the sigmoid function. If the sigmoid function is used, you'll probably get an output of ones only.

Instead of the ReLU-Tanh combination, you may want to use the Sigmoid-Tanh combination.

Here, the sigmoid may give an output of 0 ( or a number close to zero ), if given a negative value. These negative values may be supplied by the Tanh function in the preceding layers. You may face overfitting here.


How do I find the relevant features out of 11,000+ possibilities? While working on Kaggle Competition, I ended up with 11,726 columns which are mostly "dummies" (one hot encoding). Is this too many?

I know that we need to find out which features are relevant, but not sure how to do this.

Your solution will depend on a couple of factors. One is what type of model you are using. If you are using something that automatically calculates feature importances then you could simply look at these (or take a more balanced look using permutation importances).

While you could look at feature importances, with ~11,000 possibilities this is going to be pretty difficult. The main focus should be to cut down these features into something more manageable, do you really need one hot encoding? Without knowing more about the dataset I can't provide much more advice.


Filling missing values with pyspark using a probability distribution I want to fill missing values in my dataframe.

 In [1]: df = spark.createDataFrame([[1],[1],[2],[3],[3],[None],[3],[None],[3],[2],[None],[1],[4]], ['data'])
 In [2]: df.show()
 +----+
 |data|
 +----+
 |   1|
 |   1|
 |   2|
 |   3|
 |   3|
 |null|
 |   3|
 |null|
 |   3|
 |   2|
 |null|
 |   1|
 |   4|
 +----+

I know I can use pyspark.ml Imputer to fill with the mean / median, or use this method to fill with the last valid value. These are fine options, but I would like to impute with a random sample from the data distribution. For example, in the data provided, nulls will be filled according to these probabilities:

 P(1) = .3
 P(2) = .2
 P(3) = .4
 P(4) = .1

What would be the best way to fill these values from a random sample?

You could try modeling it as a discrete distribution and then try obtaining the random samples. Try making a function p(x) and deriving the CDF from that. In the example you gave the CDF graph would look like this

Once you obtained your CDF you can try using Inverse Transform Sampling. This method allows you to obtain random samples for the distribution specified.

This link would be helpful for your understanding of that technique. These random samples can fill those missing values as per your requirement of probabilities.

Note:

There are other techniques as well, you could search and explore along the lines of random sample generation from discrete distributions. It might be the case that your actual data might fit for example something like Poisson's distribution etc. For all such cases, you can find various examples on how to generate random samples for those particular distributions


Increase number of iterations in a logistic regression I have achieved 68% accuracy using glm with family = 'binomial' while doing logistic regression in R. I don't have any idea on how to specify the number of iterations through my code. Any suggestions on it? Will application of stochastic gradient descent work for epoch. If so, how do I apply it in my code? I want to increase the accuracy of my model. Following is my code:

rm(list=ls())
library(dplyr)
data1 <- read.csv("~/hj.csv", head`enter preformatted text here`er=T)
train<- data1[1:116,]
VALUE<-as.numeric(rownames(train))
testset<- data1[1:116,]
mylogit <- glm(VALUE ~ POINT1 + POINT2 + POINT3 + POINT4 , data = data1, family ="binomial")
testset$predicted.value = predict(mylogit, newdata = testset, type="response")
for (i in 1: nrow(testset)){
  if(testset$predicted.value[i] <= 0.50)
    testset$outcome[i] <- 0
  else testset$outcome[i] <- 1
}
print(testset)
tab = table(testset$VALUE, testset$outcome) %>% as.matrix.data.frame()
accuracy = sum(diag(tab))/sum(tab)
print(accuracy)
print(tab)
table(testset$VALUE, testset$outcome)

Logistic regression in R uses the iterative re-weighted least squares algorithm. You can specify the maximum iterations and accuracy with:

m <- glm(..., family = "binomial", control = list(maxit = 2, epsilon=1))

Please read the documentation for glm.control here.

You can check the actual number of iterations used in fitting:

m$iter

BUT... The default settings should be enough.

> glm.control()

$epsilon
[1] 1e-08

$maxit
[1] 25

$trace
[1] FALSE

The default maximum number of iterations is 25, and I **doubt** you will get anything by changing it to anything larger. The accuracy is 1e-08, which is already very small.

You shouldn't blindly adjust the iteration number, most likely it won't help. Try this:

> glm(admit ~ gre + gpa + rank, data = mydata, family = "binomial", control = list(trace=TRUE))

Deviance = 458.875865177 Iterations - 1
Deviance = 458.517518881 Iterations - 2
Deviance = 458.517492476 Iterations - 3
Deviance = 458.517492476 Iterations - 4

I got the dataset from http://www.ats.ucla.edu/stat/r/dae/logit.htm (because I don't have your data file).

Do you see I have trace=TRUE? I print off the deviance for each iteration. R stops because the deviance difference between iteration 3 and 4 is too small. My model only requires 4 iterations.

My recommendation:

Print your deviance for each iteration and convince yourself you don't need more than 25 iterations
Check your model properly. Do you need more independent variables? Do you need to transform your variables to improve prediction accuracy?
Ask yourself is 60% accuracy enough? In many fields 60% is a huge success.
Does your model require interaction? Note a saturated model will give you prefect accuracy, but it has all the interaction terms. Your model assumes the effects of your independent variables are additive in predicting the log-odd. Is this a fair assumption?
Do you want to run automatic stepwise model selection algorithm in R? You might get a better model than manually done it yourself.

Some useful links you might want to consider:

https://stats.stackexchange.com/questions/5354/logistic-regression-model-does-not-converge
https://stats.stackexchange.com/questions/60958/logistic-regression-fishers-scoring-iterations-do-not-match-the-selected-itera


Does the score output of a classification model has a global meaning? The scores-output layer contains the class scores that the model generated for the current sample and it is passed thru the softmax layer to get the final output in the form of a probabilities vector.

My question is whether these scores have any global meaning, when compared with scores of other samples from the dataset?

  The way I see it, a standard classification model is trained on the
  output of the softmax layer, so the model only "cares" about the
  relative scores within each sample. So it has no mechanism for
  decreasing the global scores of a sample - a scores vector of
  [101,102,103] gets the same treatment as [1,2,3]. Am I correct in my
  understanding?

The output of the final layer, before softmax activation, is the log odds of the classes. Softmax isn't a linear function, so [10,20,30] going into softmax won't give the same answer as [1,2,3].


Using Cross Validation technique for a CNN model I am working on a CNN model. As always, I used batches with epochs to train my model. When it completed training and validation, finally I used a test set to measure the model performance and generate confusion matrix. Now I want to use Cross Validation to train my model. I can implement it but there are some questions in my mind:
1- Why most CNN models don't use Cross Validation?
2- If I use Cross Validation, how can I generate the confusion matrix? Can I split dataset to train/test and then do cross validation on train set as train/validation (i.e., doing cross validation as train/validation except for the usual train/test) and at last use test set the same way? Or how?

Question 1: Why do most CNN models not apply the cross-validation technique?

$k$-fold cross-validation is often used for simple models with few parameters, models with simple hyperparameters and additionally the models are easy to optimize. Typical examples are linear regression, logistic regression, small neural networks and support vector machines.
For a convolutional neural network with many parameters (e.g. more than one million) we just have too many possible changes in the architecture. What you can do is to do some experiments with the learning rate, batch size, dropout (amount and position) and batch normalization (position). Training a convolutional neural network with a huge dataset takes quite a long time. Doing hyperparameter optimization would just be total overkill. Often in papers, they try to improve the results of other research papers. It is not the goal to get better results by improving the chosen hyperparameters but rather to come up with new ideas to solve the given task but with better accuracy or less computational effort.

  Question 2: If I use cross-validation how can I generate confusion
  matrix? can I split dataset to train/test then do cross-validation on
  train set as train/validation (i.e. doing cross-validation as
  train/validation except for the usual train/test) and at last use test
  set the same way? or how?

In order to do $k$-fold cross validation you will need to split your initial data set into two parts. One dataset for doing the hyperparameter optimization and one for the final validation. Then we take the dataset for the hyperparameter optimization and split it into $k$ (hopefully) equally sized data sets $\mathcal{D}_1,\mathcal{D}_2,\ldots,\mathcal{D}_k$. For the sake of clarity let us set $k=3$. Then for each possible hyperparameter combination that we want to test we use  $\mathcal{D}_1$ and $\mathcal{D}_2$ to fit our model and we use $\mathcal{D}_3$ to validate our model. Then we do the same with $\mathcal{D}_2$ and $\mathcal{D}_3$ and use $\mathcal{D}_1$ for validation. Then we do the same with $\mathcal{D}_1$ and $\mathcal{D}_3$ and use $\mathcal{D}_2$ for validation. We will get $3$ confusion matrices for every possible hyperparameter configuration. In order to derive a metric from these three results, we take the mean of these confusion matrices. Then we can scan through all averaged confusion matrices so select the hyperparameter configuration that was the best (you have to define what parts of the confusion matrix are important for your problem). Finally, we pick the 'best' hyperparameters and calculate the prediction performance on the final validation set. This performance metrics are the ones that you report.


The central idea behind SGD Pr. Hinton in his popular course on Coursera refers to the following fact:

Rprop doesn’t really work when we have very large datasets and need to perform mini-batch weights updates. Why it doesn’t work with mini-batches? Well, people have tried it, but found it hard to make it work. The reason it doesn’t work is that it violates the central idea behind stochastic gradient descent, which is when we have small enough learning rate, it averages the gradients over successive mini-batches. Consider the weight, that gets the gradient 0.1 on nine mini-batches, and the gradient of -0.9 on tenths mini-batch. What we’d like is to those gradients to roughly cancel each other out, so that the stay approximately the same. But it’s not what happens with rprop. With rprop, we increment the weight 9 times and decrement only once, so the weight grows much larger.

As you can see, the central idea behind SGD is that the successive gradients in mini-batches should be averaged. Does any one have any valid formal source for this? is there any justification? I've not encountered to any proof till now.

Influence of the data generating distribution
To see this, first we have to mention that, neither by using Batch gradient descent (using the whole dataset to compute the gradient) nor using mini-batch gradient descent, we are computing the true (exact) value of the gradient.
To compute the true value of the gradient we would have to use the set of all possible values of the features, $x$, (and thereby the outputs $y$).
More formally, and refering to the quantity that we want to minimize as the expected value of the per-example loss function ($J(x,y,\theta)$, where $\theta$ are the parameters) w.r.t all the possible $x,y$ values, the true gradient $g$ is given by:
$$g = \frac{\partial}{\partial \theta}\mathbb{E}_{x,y\sim p_{data}}(J(x,y,\theta)) $$
And if we assume certain conditions We have that:
$$g = \mathbb{E}_{x,y\sim p_{data}}\left(\frac{\partial}{\partial \theta}J(x,y,\theta)\right) $$
Where $p_{data}$ is the data generating distribution (the distribution from which the values of $x$ and $y$ are drawn). However, this data generating distribution is usually unkown. We just know the dataset that we are given.
Because of this, to update the parameters using all the information given (the training set), we instead use the empirical ditribution defined by the training data ($\hat{p}_{data}$) which puts a probability of $1/m$ on each of the $m$ samples $(x^{(1)}, y^{(1)}), \,(x^{(2)}, y^{(2)}),\,...\,,(x^{(m)}, y^{(m)})$ of the dataset. So the gradient is approximated by:
$$ \begin{aligned}
\hat{g}&=\frac{\partial}{\partial \theta}\mathbb{E}_{x,y\sim \hat{p}_{data}}(J(x,y,\theta))\\&=\frac{\partial}{\partial \theta}\left(\sum_{i=1}^m \frac{1}{m}J_i(x^{(i)},y^{(i)},\theta)\right)\\
&= \frac{1}{m}\sum_{i=1}^m\frac{\partial }{\partial \theta}J_i(x^{(i)},y^{(i)},\theta)
\end{aligned}
$$
Ending up with the Batch gradient descent.
But what happens with mini-batches?
By using mini-bath updates, we are continuously seeing new data (assuming that we compute just one epoch). So in this case, using mini-batches, we are using the data generating distribution.
This means that on each mini-batch update, by sampling this data generating distribution, we end up with an estimation ($\hat{g}$) of the true gradient ($g$) that is unbiased i.e. $\mathbb{E}_{x,y\sim p_{data}}(\hat{g})=g$. To see this, and considering $\text{s-sized}$ mini-batches:
$$\begin{aligned}
\mathbb{E}_{x,y\sim p_{data}}(\hat{g})&=\mathbb{E}_{x,y\sim p_{data}}\left(\frac{g^{(1)}+...+g^{(s)}}{s}\right)\\
&=\frac{1}{s}(\mathbb{E}_{x,y\sim p_{data}}(g^{(1)}+...+g^{(s)}))\\
&=\frac{1}{s}s\,\,g=g
\end{aligned}
$$
Thereby, making succesive mini-batch updates we would be tending in average (as shown by $\mathbb{E}_{x,y\sim p_{data}}(\hat{g})$) to updating our parameters with the true value of the gradient. And this is what I think the authors refer to in the quote of the question.

Great references:
Deep Learning book, Ian Goodfellow et. al Chapter 8.1
Answers from here


Prediction Algorithm for Data with high Randomness I have data for the orders of the previous year containing the product and the seller who sold the product. I have an information product, product category, seller, delivery address price etc. Database size is more than 100,000 rows. Now, I have to suggest a seller for a totally new product based on the data I have. I tried using TF-IDF to find similar products of the same category to suggest the sellers and I got an accuracy of 70%. Then, I tried a random forest algorithm and sadly I got an accuracy of just 40%. I am unable to find a suitable approach for my use case. How can I approach this problem statement?
The Product and Seller Mapping table is like this

productId
sellerId
price
purchase Date
deliveryAddressId

1
4
100
9-01-2012
4

2
12
400
1-08-2020
4

1
1
123
4-09-2020
1

2
3
450
24-12-2020
1

3
4
150
14-05-2020
2

5
3
430
12-02-2020
2

Product has the following information

productId
name
categoryId

1
AC
1

2
TV
1

3
Food
2

4
Toy
3

5
Car
3

6
Book
4

Seller has the following information

sellerId
sellerName
totalTransactions

1
A
81

2
B
111

3
C
200

4
D
42

You can use seller mean buying price, std buying price, max buying price, min buying price, median buying price PLUS include recently user buying power to suggest the totally new product to the user given the current data that's best I can recommend although extensive data can lead to better suggestions.


ML packages in R: caret v e1071 I've looked and surprisingly have not found too much discussion on the relative strengths of the caret and e1071 package.  From my understanding, these packages perform many of the same ML algorithms.  With that in mind, I'm interested in what those practitioners who have experience with both think these packages relative strengths and weaknesses are.  What do you think about readability of code, ease of coding, methods, speed, etc...

In particular I'm interested in multi-class SVM

I think that caret and e1071 serve difference purposes. First lets discuss caret, its closest competitor is the mlr package. Both are meta packages that allow to optimize models across parameters. Take for example a problem where you are not sure whether you would want to use Lasso or Ridge to create a model. As explained here caret allows to choose the optimal lambda based on different type of cross validations.

The e1071 package instead is a bag of functions developed by the TU Wien. It is probably one of the most popular packages for support vector machines which are used by caret. But its goal is very different than caret, it implements learning algorithms and other functions whereas caret seeks to find the best parameter in learning models.


When should we start using stacking of models? I am solving a Kaggle contest and my single model has reached score of 0.121, I'd like to know when to start using ensembling/stacking to improve the score.

I used lasso and xgboost and there obviously must be variance associated with those two algorithms. So stacking should theoretically give me better output than my individual algorithms.

But how to idenfity if stacking is worth it and that we've reached dead end to accuracy of a particular model?

Stacking is going to help most when individual models capture unique characteristics of the data. It is often the case that different architectures perform similarly, if somewhat differently, on the same data. In those cases, ensembling/stacking will only offer slight incremental benefits. In the limit, of you only care about prediction, you can wire up as many different approaches as you can think of. However if interpretability is key, each additional component model will further complicate things.

Your specific question of when to know if it’s worth it or if you’ve reached the limit can be treated like anything else - is your incremental r-square/error/classification accuracy significantly better versus a simpler approach?


How to use class_weights with one-hot encoding in keras? Is it possible to use class_weights with a one-hot encoding?

I've tried sparse_categorical_crossentropy and, for some reason, it's significantly worse than my classic categorical_crossentropy with one hot encoding model.

This is how I'm calculating the class_weights with sparse:

unique_class_weights = np.unique(labels)
class_weights = class_weight.compute_class_weight('balanced', unique_class_weights, labels)
class_weights_dict = { unique_class_weights[i]: w for i,w in enumerate(class_weights) }

Training like:

full_model.compile(loss='sparse_categorical_crossentropy',
                   optimizer='rmsprop',
                   metrics=['accuracy'])

full_model.fit_generator(line_generator(data_train, labels_train),
                         validation_data=line_generator(data_test, labels_test),
                         validation_steps=1,
                         steps_per_epoch=len(data)/GENERATOR_BATCH_SIZE,
                         class_weight=class_weights_dict,
                         epochs=1)

I'm not allowed to comment, but have you tried using the numpy array that you get from class_weight.compute_class_weight(), rather than converting it to a dict? I've always skipped that part, and in your case I would say class_weight=class_weights. Sorry if I'm suggesting something you already ruled out. Good luck.


Low accuracy in multi-class classification despite all data being generated from rules I have a well defined data where i have cleaned up my data to final form which has 20 features mapping to a number between 1 to 100. Upto 5 features are enabled(value set to 1) for each row. The data looks something like below

 Result|f1|f2|...f19|f20
 45    |0 | 1|... 1 | 0
 92    |0 | 0|... 1 | 1

I'm trying to build machine learning models that can give me good accuracy and preferably models which can handle warm_start since each iteration generates 1 row that i need to fit into existing build model.

below are 2 classifiers that i tried to set some baseline

randclf = RandomForestClassifier(n_estimators=50)
decclf = DecisionTreeClassifier(criterion = "gini", random_state = 100,max_depth=3, min_samples_leaf=5)

However even with 100,000 records i'm getting very poor result with accuracy around 15-20%. considering how predictable data is(data is generated based on finite set of rules) i was expecting very high accuracy.

I'm i doing something wrong, i want get the high accuracy in classifying data(predicting Result) based on features given, can you suggest some models that might work well this kind of data. what about tensorflow and neural network approach?

data:

https://github.com/sachinhegde6/machinelearningdata

Update:
Data imbalance is something i cant help as they are generated based on rules.

I think neural network will be computationally intensive and would require you to have a good GPU along with good amount of training data.
You can try running a clustering algorithm like k-means or logistic regression using warm_start


multivariate clustering, dimensionality reduction and data scalling for regression I have a dataset with approximately 20000 observations consisting of 40 independent and 1 dependent variable. My initial objective is to develop a model that will predict the dependent variable. I have tried several models and applied linear regression and other algorithms such as Random Forests, of course by splitting the dataset into training and testing sets.

Unfortunatelly I cannot get any meaningful results; I have very large errors. I believe there is something "messy" with the dataset, so I have decided to do some clustering first and then apply regression within each cluster. Considering that my dependent variable may exhibit a lot of variation I believe I should do clustering with all variables (dependent and independent), so as each cluster will have similar values of my dependent variable. I have tried to apply Kmeans and I faced several problems. First of all, it seems I cannot identify the right number of clusters. The "elbow" method gives an unclear number and when I use it with less data (about 2000 observations) I get something like this:

I also had similar problems with hierarchical clustering. I have already tried to apply regressions within the clusters identified, but the results are still very poor.

Right now I believe I should possibly use some kind of "weight" to my data, in order to put more weight on the dependent variable when I do clustering, since I believe that this is the problem. Hence, my questions here are:

is there a way/algorithm where I adjust weights in the variables to be clustered?

Moreover I am confused with two more issues:

data scaling:
is it necessary to scale the data before clustering? does this guarantee more accurate results? when do we scale the data?
dimensionality reduction:
I have read a lot about principal component analysis and dimensionality reduction, but I am still confused. Again; is this necessary? how many variables are too many to consider applying PCA? are 10 variables too many? or maybe 20? or 50? when should we apply dimensionality reduction? a problem with PCA is that I would still need my original variables to extract the coefficients after regression, while to my understanding with PCA I cannot do that.

This question is more about discussion in order to understand some particular concepts and find a solution to my problem and does not refer to coding issues. Any example and/or references would be appreciated though. I am coding in R.

Great question, I will try to answer the aspects related to dimensionality reduction mentioned above.
$Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would still ask you to do a correlation analysis of the variables.

Pca removes these correlated features and gives you a set of features which amount to explain most of your data. One advantage of dimensionality reduction is in regression. Correlated features often are a cause of multicollinearity. Doing a dimensionality reduction helps us get rid of this problem. Also once we have a reduced set of features we can apply the cluster analysis. The reason is K-means calculates the l2- distance between data points. in very high dimensions the concept of euclidean distance becomes less useful because of the curse of dimensionality. (probably the reason of the problems with your elbow curve.)

Hence bringing down the number of features using pca and clustering later will give a better idea of the groupings of the data.
The problem which you mention about PCA can be solved in fact it is not an issue at all. You can do regression with the features obtained by the data and only take the coefficients of the reduced feature set.Some other techniques which you might want to look at are regularization in regression.
For example L1-regularization helps in feature selection and helps with correlated variables.


How to make this intersection chart? I want to prepare the following chart. Say, I have three datasets of different lengths, for example:

A = [c,d,e,g,h,o,m,n,o,x,y,z]
B = [d,e,f,g,h,i,j,k,l]
C = [a,b,c,d,e]

Any dataset might contain items already there in other 2 datasets. Now I want to show the size of each dataset as a circle. More items, bigger circle. But all three circles also overlap each other depending on how much items are common. As demonstrated below.

What this chart is actually called? And what library can I use which might have this kind of chart available?

Thanks to @Peter. It's called Venn Diagram and here's my needed Javascript Library. https://www.benfrederickson.com/venn-diagrams-with-d3.js/


KMeans clusterization on documents Whether correct or not, I'm not able to judge being myself in the early days of the Data Science.

However, I have applied a Kmeans on a corpus where some random documents (very short sentences) have been added.
These have been vectiorized so to be suitable.

With clusterization results at hands, I was somehow expecting the vectors (keyword) to fall only in one cluster at a time (and no more than that).
This is not the case.

In some circumstances, I have a vector falling in two clusters and I wonder why this is the case.

Is this because of the inappropriate usage of Kmeans on vectors made from documents?
Is this normal as the way Kmeans works (moving the centroids, but de facto assigning objects to the nearest cluster by distance)?
Is this overlap due to the fact that in analysing my results I assess the whole group of items within a cluster and not just (say) the top X near to the center?

--
Example:

corpus = [
'The car is driven on the road.',
'The truck is driven on the highway.',
'The train run on the tracks.',
'The bycicle is run on the pavement.',
'The flight is conducted in the air.',
'The baloon is conducted in the air.',
'The bird is flying in the air.',
'The man is walking in the street.',
'The pedestrian is crossing the zebra.',
'The pilot flights the plane].',
'On the route, the car is driven.',
'On the road, the truck is moved.',
'The train is running on the tracks.',
'The bike is running on the pavement.',
'The flight takes place in the sky.',
'Birds don''t fly when is dark',
'The baloon is in the water.',
'The bird flies in the sky.',
'In the road, the guy walks.',
'The pedestrian is passing through the zebra.',
'The pilot is flying the plane.',
'This is a Japanese doll.',
'I really want to go to work, but I am too sick to drive.',
'Christmas is coming.',
'With the daylight saving time turned off it''s getting dark soon.',
'The body fat may compensates for the loss of nutrients.',
'Mary plays the piano.',
'She always speaks to him in a loud voice.',
'Wow, does that work?',
'I don''t like walking when it is dark',
'Last Friday in three week’s time I saw a spotted striped blue worm shake hands with a legless lizard.',
'My Mum tries to be cool by saying that she likes all the same things that I do.',
'Mummy is saying that she loves me being a pilot when in reality she is scared all the time I take off.',
'Where do random thoughts come from?',
'A glittering gem is not enough.',
'We need to rent a room for our party.',
'A purple pig and a green donkey flew a kite in the middle of the night and ended up sunburnt.',
'If I don’t like something, I’ll stay away from it.',
'The body may perhaps compensates for the loss of a true metaphysics.',
'Don''t step on the broken glass.',
'It was getting dark, and we weren''t there yet.',
'Playing an instrument like the guitar takes out the stress from my day.']

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(analyzer='word',
                         max_df=0.8,
                         max_features=50000,
                         lowercase=True
                        )

X = vectorizer.fit_transform(corpus)

from sklearn.cluster import KMeans

num_clusters = 11
kmean = KMeans(n_clusters=num_clusters, random_state=1021)
clusters = kmean.fit_predict(X)

--

If you explore the clusters variable, you will notice the overlaps I am talking about.
For instance the keyword baloon appeara in both cluster 10 and 0.

There are 12 overlaps, which on a 33 unique keywords dataset represents 1/3, so I won't say something I could be happy with.

Any advice is appreciated.
Thanks

Let us assume that your corpus has n distinct keywords. For a k-means algorithm, each keyword is an axis in n dimensional space. Document is a point in that n dimensional space.

K-means algorithm will allocate each point (a document) to a single cluster. When you say a keyword is appearing in two clusters, it probably implies: that particular dimension/keyword is important for both clusters.

Let us take hypothetical example: if you have a patient's blood pressure, cholesterol levels and bunch of other medical parameters. Let us say you discretize blood pressure to 2 or 3 levels. If you run k-means on this data, each patient will be assigned a unique cluster. But it is quite possible that two (or even more) clusters all have patients with > 120 systolic blood pressure.

You need to probably read the results of the k-means more carefully.


Why does decreasing the SGD learning rate cause a massive increase in accuracy? In papers such as this I often see training curves with this kind of shape:

In this case SGD was used with a factor of 0.9 and learning rate decreasing by a factor of 10 every 30 epochs.

Why is there such a large decrease in error when the learning rate is changed?
Why does the validation error begin to increase after the initial drop, whereas the training error continues decreasing?
Can the same results be obtained by moving the 2nd and subsequent learning rate changes closer together? That is, why the delay in doing further drops?

With a higher learning rate, you take bigger steps towards the solution. However, when you are close, you might jump over the solution and then the next step, you jump over it again causing an oscillation around the solution.
Now, if you lower the learning rate correctly, you will stop the oscillation and continue towards the solution once again. That is, until you start oscillating again.
To keep in mind is that a larger learning rate can jump over smaller local minima and help you find a better minima, which it can't jump over.
Also, it is generally the training error that becomes better, and the validation error becomes worse as you start to overfit on the training data.


How to convert input numpy data to tensorflow tf.data to train model in tensorfow? I am working on an image classification problem using TensorFlow. I have converted my input image dataset and label into NumPy data but it takes more time and more ram to load all the data into memory because I have 90K images. I would like to use TensorFlow data API using tf.keras.preprocessing.image_dataset_from_director
Here's my current code to train NumPy data. I want to convert this code into tf.data(tf.keras.preprocessing.image_dataset_from_director) to train my huge dataset.
INIT_LR = 1e-4
EPOCHS = 20
BS = 32

# grab the list of images in our dataset directory, then initialize
# the list of data (i.e., images) and class images
print("[INFO] loading images...")
imagePaths = list(paths.list_images(args["dataset"]))
data = []
labels = []

# loop over the image paths
for imagePath in imagePaths:
    # extract the class label from the filename
    label = imagePath.split(os.path.sep)[-2]

    # load the input image (224x224) and preprocess it
    image = load_img(imagePath, target_size=(224, 224))
    image = img_to_array(image)
    image = preprocess_input(image)

    # update the data and labels lists, respectively
    data.append(image)
    labels.append(label)

# convert the data and labels to NumPy arrays
data = np.array(data, dtype="float32")
labels = np.array(labels)

# perform one-hot encoding on the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
labels = to_categorical(labels)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels,
    test_size=0.20, stratify=labels, random_state=42)

# construct the training image generator for data augmentation
aug = ImageDataGenerator(
    rotation_range=20,
    zoom_range=0.15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.15,
    horizontal_flip=True,
    fill_mode="nearest")

# load the MobileNetV2 network, ensuring the head FC layer sets are
# left off
baseModel = MobileNetV2(weights="imagenet", include_top=False,
    input_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(128, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(2, activation="softmax")(headModel)

# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)

# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
    layer.trainable = False

# compile our model
print("[INFO] compiling model...")
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
    metrics=["accuracy"])

# train the head of the network
print("[INFO] training head...")
H = model.fit(
    aug.flow(trainX, trainY, batch_size=BS),
    steps_per_epoch=len(trainX) // BS,
    validation_data=(testX, testY),
    validation_steps=len(testX) // BS,
    epochs=EPOCHS)

# make predictions on the testing set
print("[INFO] evaluating network...")
predIdxs = model.predict(testX, batch_size=BS)

# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)

# show a nicely formatted classification report
print(classification_report(testY.argmax(axis=1), predIdxs,
    target_names=lb.classes_))

# serialize the model to disk
print("[INFO] saving mask detector model...")
model.save(args["model"], save_format="h5")

You could try using the flow_from_directory() method on your ImageDataGenerator class, which does the augmentation - only a small change is necessary:
H = model.fit(
    aug.flow_from_directory(trainX, trainY, batch_size=BS),
    ...)

If you start using a tf.data.Dataset directly, you will get more control over how the data is read from disk (caching, number of threads etc.), but you will lose the easy image augmentation you get with the ImageDataGenerator.


Feeding large dataset to neural network using tf I have a csv file of size ~11GB contains 2 million rows located in my local disk as my training data. What's the best way to feed the data to a neural network model? I use tf.keras to build the neural network and my RAM doesn't allow me to import as a single np array. Thank you.

You can use generator to read the data if your model uses "Mini-Batch SGD"-like optimization method which just uses a small batch of samples each step. For example,

df_iterator = pandas.read_csv(your_data,chunksize=batch_size,iterator=True)
for small_batch in df_iterator:
#feed it to your model's input


Reducing a page of content to a short paragraph I remember years ago, Yahoo detailed how they were able to reduce a webpage down to a short paragrah of text succently summarising the content in sentences, as opposed to a list of keywords. What is this called? Are there any open / free code to do this?

Look into TextRank algorithm, here is the paper paper. You can find a neat python implementation here textrank implementation If I am not wrong gensim also offers an implementation.


Weights in scikit-learn metric functions Most of the scikit-learn metric functions have an option to take into account sample weights. What is the purpose or use case to use this options? Unbalanced classes?

Unbalanced classes is definitely one use case, another purpose is cost sensitive learning, where the cost of misclassifying classes is different for each class. In particular scikit-learn give weights for each sample so it allows a greater flexibility.


How do I use the model generated by the R package poLCA to classify new data as belonging to one of the classes? For example, in the election example from the documentation, if I create a new set of answers to the questions, how can I use the poLCA model to tell me what class (cluster) it's most likely to be in?

There doesn't appear to be a function to do this, though the model has a df within it that lists the probabilities of class membership for each value of each manifest variable. I'm tasked with converting some sql code that takes a second dataset and classifies the patients there as members of the clusters created from a first. Superficially this is a programming question. It seems like a function to do this would be a reasonable addition to the package. More deeply, if indeed there isn't such a function, it would become a question about how to use the table of probabilities to classify new data.

If readers aren't familiar with the R package poLCA, it's an LCA package that works with discrete/categorized data.

(full disclosure: I asked on cross-validated and a shorter version of this question was put on hold.)

Using carcinoma data available in the poLCA package and a 4 latent classes solution:

 library poLCA
 data(carcinoma)
 f <- cbind(A, B, C, D, E, F, G) ~ 1
 lc4 <- poLCA(f, carcinoma, nclass = 4)

The following line give the classification in terms of predicted probabilities

 lc4$predclass

They could be usefully binded to the original data for further visualizations or analyses

 carcinoma.predclass <- cbind(carcinoma, "Predicted LC" = lc4$predclass)
 head(carcinoma.predclass)

You could have a new data frame with the same columns/variables used in the previous analysis.

new.data <- data.frame(A=c(1,2,1), B=c(2,2,1), C=c(1,2,1), D=c(1,1,1), E=c(1,2,1), F=c(2,2,1), G=c(1,2,1))

  A B C D E F G
1 1 2 1 1 1 2 1
2 2 2 2 1 2 2 2
3 1 1 1 1 1 1 1

A simple method can be link the observed data patterns in the new data frame with the estimated latent class probabilities in the previous data. In fact, the first pattern has missing prediction because it wasn't in the training data.

left_join(new.data, unique(carcinoma.predclass))

Joining, by = c("A", "B", "C", "D", "E", "F", "G")
  A B C D E F G Predicted LC
1 1 2 1 1 1 2 1           NA
2 2 2 2 1 2 2 2            1
3 1 1 1 1 1 1 1            4

The best method is to use the posterior of poLCA. From the parameters estimated by the latent class model, this function calculates the probability that a specified pattern belongs to each latent class. This function can calculate posterior class membership probabilities for new data, observed or not in the training data.

new.lc4.posterior <- poLCA.posterior(lc4, new.data)

And bind the predicted Latent Classes (the classes which the highest posterior probability) to the new data.

cbind(new.data, "Predicted LC" = apply(new.lc4.posterior,1, FUN=which.max))

  A B C D E F G Predicted LC
1 1 2 1 1 1 2 1            2
2 2 2 2 1 2 2 2            1
3 1 1 1 1 1 1 1            4


CNN training on Different Datasets I currently want to train a CNN but I have two small datasets that are slightly different because of the camera setup that captured the images. I'm interested in ultimately tuning the neural network to only one of the camera setups and using the network for that one setup only from here on out.

Since I'm not going to be able to expand my dataset during this time, I realize I'll need to use both training datasets to get enough of variability. So I'm wondering what approach to this might be better -- should I combine the two datasets and train the CNN, or should I train on one dataset and finetune it to the second dataset? Or is there another method I should be using?

I suggest you to use as much data as possible. If the images are coming from two different cameras, it could be an original way to fight overfitting. I would use both datasets, and operate a train-validation-test split that is transversal to both.

Additionally, since the dataset is small, I strongly enourage you to use lots and lots of data augmentation, which is a very powerful technique to artificially increase the size of your dataset and fight overfitting.


storing a huge dataset in h5py file format I work on preparing the luna16 dataset for feeding into the CNN model, after reading all '.mhd' files and the labels(0, 1) in the annotated CSV file, I get a memory error, I know the problem is because of the data size is need a lot of RAM that I do not have on my computer, is there any solution to that?

You can use h5py files but due to the fact that they store the dimensions, they take even more disk to be stored and while fetching you have to supply enough memory which is not possible in your case, so you have to read them chunk by chunk. There is a simpler approach. One of pandas's inputs is chunk. try to use it.


Training vs test data set for supervised learning in real life scenario In the tutorials, I have noticed only similar data has been used with models training and prediction.

I was wondering how cases where you can't find training data that is similar to your final use case(test data)?

What happens if I am building a recommendation system for house rentals but the closest training data I can find is for movies. And it has fewer features than I would have in the house scenario.

How common are scenarios like this and how are they generally handled?

One important assumption of data for a lot of Machine Learning algorithms is that the data from the training set and test set are i.i.d (independent and identically distributed) and come from the same probability distribution. More explanation here. That said, you cannot train a model for movies recommendation and use for house rentals. Or not that I'm aware of.

However, it's possible to train a model for house rentals recommendation if you have a lot of data for movies recommendation and a small amount of data for house rentals recommendation, using a technique called Transfer Learning. The idea is simple: If two tasks are similar, we should be able to use what we learn from one task to learn the other task faster and better.

Following is the general process (for neural networks):

Using movies data, train a model for movies recommendation
Fine-tune this recommendation system using house rentals data: Freeze the most part of the movies recommender, retrain a few last layers with house rentals data.

This is just a direction about how it can be done, there's quite a lot of related literature for you to learn more. I would say this is a not very uncommon scenario.


For a multi-class classification problem, how to transform the target variable to a form that is usable by sklearn algorithms? I recently tried to create a model for predicting what class a sample belongs to out of 160 possible classes. These classes of the target variable are just simple strings describing workouts like "Push-ups", "Sit-ups" etc. I used sklearn's method LabelEncoder to transform the target variable, and it assigned every class a number between 0 and 159. I know that the reason why we are supposed to do one-hot encoding for categorical features is to avoid telling an algorithm something nonsensical like "blue is greater than red". My question then becomes, doesn't LabelEncoder tell the sklearn algorithm that "push-ups" is less than or greater than "sit-ups" if both of them are assigned distinct numbers between 0-159? Why or why not? This feels like a very stupid question, but I had real trouble finding an answer anywhere online.

Welcome to the site! Do a google search for "one hot encoding" and all will become clear.

In a nutshell, don't think of it as numbers "assigned" of 0-159. It's more about a "slot" for each category and using one hot encoding to let the algorithm know which slot a particular record belongs to.


Random Forest Target/prediction maximum minimum This is my current situation, I currently have a regression random forest that is targeting a continuous variable (sales amount) is there any configuration for the classifier that allowed me to set a maximum and a minimum for the target.

Example: Predicted sales_amount should be between 10000$ and 20000$

thank you!!

Random Forest Regressor doesn't have such an option, usually, if you have enough data and a correct training process, predictions should be in an acceptable range. Of course, they might go above and beyond your range, but the error might be still acceptable.

If you are looking for a model that is able to utilize prediction boundaries, I suggest you look at Facebook Prophet as it may be useful for your problem at hand.
Python Prophet Quick Start

For range-predictions, you can specify "Forecasting Growth":

  When forecasting growth, there is usually some maximum achievable point: total market size, total population size, etc. This is called the carrying capacity, and the forecast should saturate at this point.

Here is a link for this part
Forecasting Growth and Saturating Minimum

And another note - Prophet proves to be quite useful in predicting any seasonal related time series, so you can achieve even better performance predicting sales compared to Random Forest.


Counting non-overlapping objects in a semantic segmentation prediction mask What is a good way to count the number of roofs detected by a CNN in the following output on the right (produced using keras/tensorflow):

I need to count the discrete shaded areas and estimate their area in pixels. I'm a little stumped. It is not strictly an image classification problem.

This could be one of the approaches (if you are looking for the exact solution):

I assume that the image (your prediction) consists of only two values: 0 - black - roof, and 255 - white - other.

Then, this becomes an easy problem of counting the components of a graph:

Start a DFS from some black node which is not visited yet. Traverse all neighboring black nodes which are not visited yet. While moving, keep count of the number of black nodes you visited, and mark them as visited (so that you don't go back ...). Repeat the whole process if there are black nodes to start from (they are not already visited). This way you get the total number of roofs (this is the number of times you started a DFS from non-visited black node) and the total area of roofs (the number of black pixels you visited).

That is classical DFS problem. Here is an example in Python: https://eddmann.com/posts/depth-first-search-and-breadth-first-search-in-python/

This will work well, if you predictions are good. Actually, it will do what I have described, even if your predictions are bad, because it is an exact algorithm. But you have to check if your predictions are fine some other way.

If you are looking for different type of solution (i.e. not exact solution), the interesting approach would be to teach a model to count the number of roofs and estimate their area from an image. In order to do that, you would have to create a dataset with labeled images: for each image the label could be a total number of roofs on the image and the area of all roofs. So, that would be a regression problem.


Orange3 summarizing data, grouping data values Is there a simple way in orange3 (not writing a Python script) to summarize data and group similar data values?

For example, instead of plotting a scatter with lots of data points, I would like to plot just the average y value at every x value. In pandas, this is easily done with groupby().mean().

Is there a similar widget/feature I'm overlooking in orange?

Unfortunately, not that I know of.
You could try Feature Constructor and define custom ranges or simply create a column with mean, then plot it. But that is not so different from Python Script.


Stratified selection based on the y response creates a bias in information (Berkson's bias)? I have a database which has individuals in different months and a target variable which indicates wheter an event happened or not, let's say:

Id: 1, Month: 1, Event: 0
Id: 1, Month: 2, Event: 0
Id: 1, Month: 3, Event: 1
Id: 2, Month: 1, Event: 0
Id: 2, Month: 2, Event: 0

As my sample is so big (think of the number of combinations), I would like to choose a sample n much lesser than N (one month per individual).

For the individuals who had the event, the selected month is always the one which had the event. Always. There is no randomness.

Instead, for individuals which did not have the event, any month could be randomly selected.

These facts...

Selecting one month per individual.
Difference between selection methods for positives and negatives (I think is called Berkson's bias).

...affect the performance of the classification model?

It would be better to frame your problem as survival analysis, rather than classification. The goal of survival analysis is predict the duration until an event happens.


Treat similar observations in a classification problem I have a dataset of about 200k rows and I'm working on a classification problem.
Grouping the dataset by a key variable, I noticed that some rows, with the same key value, have similar values in other columns (not equal). It may be useful to calculate the distance between observations and then deleting the closest ones prior to modelling?
I tried to split the dataset by key and then I calculated the euclidean distance between observations, but now I have a list of distance matrices and I don't know what to do next.
What do you think? I'm on the right way? Do you know some literature about it?

Wouldn't remove similar looking observations unless you have a strong reason to do so. By deleting similar looking observations you may be adding bias into the underlying distribution responsible for generating the data.
Your model may be misled into learning a biased distribution and that may affect final performance.
To start with use the entire dataset as is. If the size is a concern, then sample a random subset of the data to build your model.


What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn? I've found two different approaches online when using the Elbow Method to determine the optimal number of clusters for K-Means.

One approach is to use the following code:

distortions_2.append(sum(np.min(cdist(data
                              , kmeanModel.cluster_centers_
                              , 'euclidean')
                              , axis = 1)) / data.shape[0])

Another is to use inertia_ from sklearn.cluster.KMeans:

distortions_3.append(kmeanModel.inertia_)

When I plot the results (using the same random states) both give different results but I'm not sure what the differences are, can anyone help?

K-means does not minimize Euclidean distances, but squared Euclidean distances. This is not the same.

The nearest center is the same for both, but the mean only optimizes the squares. You can find the counterexample on my earlier answers here.

The proper value to use for the basic elbow approach is the "inertia", which is the sum of squares. But some evaluation measures may use other distances. For example the Silhouette index can be used with other distances.

Don't forget that these approaches are just heuristics for trying some k. Always also consider other values.


Consequence of Feature Scaling I am currently using SVM and scaling my training features to the range of [0,1].
I first fit/transform my training set and then apply the same transformation to my testing set. For example:

    ### Configure transformation and apply to training set
    min_max_scaler = MinMaxScaler(feature_range=(0, 1))
    X_train = min_max_scaler.fit_transform(X_train)

    ### Perform transformation on testing set
    X_test = min_max_scaler.transform(X_test)

Let assume that a given feature in the training set has a range of [0,100], and that same feature in the testing set has a range of [-10,120]. In the training set that feature will be scaled appropriately to [0,1], while in the testing set that feature will be scaled to a range outside of that first specified, something like [-0.1,1.2].

I was wondering what the consequences of the testing set features being out of range of those being used to train the model? Is this a problem?

This was meant as a comment but it is too long.

The fact that your test set has a different range might be a sign that the training set is not a good representation of the test set. However, if the difference is really small as in your example, it is likely that it won't affect your predictions. Unfortunately, I don't think I have a good reason to think it won't affect a SVM in any circumstance.

Notice that the rationale for using MinMaxScalar is (according to the documentation):

  The motivation to use this scaling include robustness to very small
  standard deviations of features and preserving zero entries in sparse
  data.

Therefore, it is important for you to make sure that your data fits that case.

If you are really concerned about having a difference range, you should use a regular standardization (such as preprocessing.scale) instead.


Deploying models on bigdata platforms like Hadoop and Spark Where exactly bigdata platforms fit in to a data science/ machine learning projects ?

Say I have a large dataset for a binary classification problem - cats and dogs.

Now I need to create a model for real time classification

Here is my question.

1 Since dataset is huge I can make use any distributed platform for faster computation and model creation right?

2 Once the model is ready, then there is no need of these distributed platform right ? or are they needed for feature extractions ?

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right?

Yes, that is what distributed platforms are for.

  2 Once the model is ready, then there is no need of these disributed
  platform right ? or are they needed for feature extractions ?

Yes for data extraction, feature generation for scoring and scoring your dataset (the actual reason you trained at model for!), you would still need the platform as that data is also likely to be huge.


Create an API from EDA or ML outcome? I have the following sample dataset (the actual dataset is over 10 million records)

     Passenger    Trip
0       Mark     London
1       Mike     Girona
2    Michael      Paris
3        Max     Sydney
4     Martin  Amsterdam
5     Martin  Barcelona
6     Martin  Barcelona
7       Mark     London
8       Mark      Paris
9     Martin   New york
10       Max     Sydney
11       Max      Paris
12       Max     Sydney
...
...
...

And I wanted to get the destination frequently travelled by a passenger !

I was playing around in Jupyter and got the expected data with the following approach

series_px = df_px_dest.groupby('Passenger')['Trip'].apply(lambda x: x.value_counts().head(1))

df_px = series_px.to_frame()

df_px.index = df_px.index.set_names(['UName', 'DEST'])

df_px.reset_index(inplace=True)

def getNextPossibleDestByUser(pxname,df=df_px):
    return df.query('UName==@pxname')['DEST'].to_string(index=False)

While the response is fine. I have few doubts now

1) What's the best way to expose the method (say in this case getNextPossibleDestByUser) as a API (pass customer name as input and get the destination as output) ?

2) Whenever the API is being called , does that mean all the 10 million records gets processed each time ? Are there anyway to optimise that ?

3) Rather than dataframe (pandas) query approach can I consider some ml models or utility functions from say scikit to solve the same problem ?

What's the best way to expose the method (say in this case getNextPossibleDestByUser) as a API (pass customer name as input and get the destination as output) ?

Use flask. Pretty easy to build an API.

  Whenever the API is being called , does that mean all the 10 million records gets processed each time ? Are there anyway to optimise that ?

It depends on the exact application but for you case you can take the passenger as an input from the call to your API as POST body or in the arguments and only iterate over the subset of data for that passenger.

  Rather than dataframe (pandas) query approach can I consider some ml models or utility functions from say scikit to solve the same problem ?

If the problem is -- as you illustrated -- simple frequency calculations then better to stick to pandas, in my opinion. If your problem is solved by the simple approach it would not make sense to go for a complicated approach.


Data scientist light? Im a 26 year old guy with an MBA and I work as an ERP system administrator.
I have been interested in the field of data science for a while now. I've always liked statistics and various analytical tasks.

I would really like to try and work within this field, but it feels a bit overwhelming. People mention that you have to learn Python, R, SQL, Machine Learning, advanced algebra, data modeling, big data (hadoop etc), predictive analytics, various business intelligence tools, VBA, Matlab etc etc.

As of today, I have some SQL knowledge and have a general understanding of BI, big data, good excel skills etc. I am willing to learn some of the aforementioned areas, but I don't have the money or time to go back to full time university studies again.

So here is my question: is there any recognized "light version" of data scientist on the market? What are they usually called? What skills do they need to master? What should I learn in order to work with big data sets and analytics without having to study full time for another 5 years?

I live in Scandinavia so the job market is probably different here, but I thought it would be interesting to hear some answers.

Business intelligence is perfect for you; you already have the business background. If you want to become a bona fide data scientist brush up on your computer science, linear algebra, and statistics. I consider these the bare essentials.

I don't know about Scandinavia, but in the U.S., data science covers a broad spectrum of tasks ranging from full-time software development to full-time data analysis, often with domain expertise required in various niches, such as experimental design. You have to decide where your strengths and interests lie to pick a position on this spectrum, and prepare accordingly. Useful activities include participating in Kaggle competitions, and contributing to open source data science libraries.


1) Your code :

Expected inputs for confusion matrix

sklearn.metrics.confusion_matrix expect $y_{pred}$ and $y_{true}$ to be $(n\_samples,)$ shape. :

  y_true array-like of shape (n_samples,)
    Ground truth (correct) target values.

  y_pred array-like of shape (n_samples,)
    Estimated targets as returned by a classifier.

Results of argmax on your matrix :

The argmax function on your test and prediction array is :
for each row, where is the first (max element) of the row.
So let's take the first four rows of your test set :

TestArray = np.array(
[
[1,0,0,1,0,1,1,0,1,0,1,1,0,0,1,1,1,0,0,1],
[0,1,1,0,1,0,0,1,0,0,0,1,0,1,0,1,1,0,1,1],
[1,0,1,1,1,1,0,0,1,1,1,1,0,0,1,0,0,0,0,0],
[0,1,1,1,0,0,0,0,0,1,0,0,1,0,0,1,0,1,1,1],...]

Argmax returns :
[0, 1, 0, 1, ...]
Wich is the locations of the first 1 in each row. I imagine this is not what you expected.

2) What I propose :

If you want to get a confusion matrix that sums up to $(n_{rows}$ x $n_{columns})$ you have to ravel the inputs :

cm=confusion_matrix(TestArray.ravel(), PredictionArray.ravel())
TN = cm[0][0]
FN = cm[1][0]
TP = cm[1][1]
FP = cm[0][1]

print(TN,FN,TP,FP)

returns (sum up to 200) :

74 28 73 25


ML algorithm where variable importance depends upon other variables - specifying conditionality I am designing an algorithm that can detect cheating in a game of chess.

I have a database of players who have been flagged by moderators as cheating, along with a number of game-level characteristics such as move times and move accuracy (as determined by a chess engine) for both black and white.

Let's specify a simple regression:

cheatflag ~ whiteMoveTime + blackMoveTime + whiteMoveAccuracy + blackMoveAccuracy

Let's say cheatflag is a factor-level variable that is 0 if no cheater, 1 if white is the cheater, 2 if black is the cheater.

The issue is that the predictive value of variables like whiteMoveTime and whiteMoveAccuracy depends crucially on who is doing the cheating.  If white is cheating, those variables should have different coefficients than if black is doing the cheating.  As far as I know, running most blackbox algorithms will derive coefficients that average across all the cases of cheating without understanding the hierarchical nature of the variables.

This problem has a Bayesian flavor with some kind of conditional assumption -- if black is cheating, then these variables matter; if white is cheating, these variables matter, and so on.

How can I deal with this issue?

A simple place to get started is logistic regression for white outcomes first and then for black outcomes. Set cheatflag to 0 if there's no cheating and 1 if white cheated, then run the regression. Do it again, but set cheatflag to 0 if there's no cheating and 1 if black cheated.


Overfitting problem in model I am making a project on prediction cars price given its features. I was able to scrape over 13000 examples. After cleaning and manipulating the data, I left with a little more than 11000 examples, I used 12 features of car, like mileage, year, brand and all other important stuff. After model selection, I decided to use Random forests to predict model. There are several questions, that got me interested:

1) When tuning model, the best I can get on test set is mean absolute error more than 2200. R_2 score is 0.92
And training set error is 900-1000. R_2 score is 0.98
I couldn’t eliminate this overfitting, if it was, what I think it is.
I tried grid search on bagging, forests even, on boosting (I know it reduces bias, not variance, but I was desperate) with different parameters, but the test error and train error with best parameters were always approximately same.
I used simple estimators too, but they had too much bias in train and test.
I know that there is a way to get rid off this overfitting, if only I had more data, but this data is maximum I can get from the page I scraped.
So the first questions is: are there situations when overfitting cannot be solved without proper and big dataset? Am I stuck in this situation?

2) And the second optional question is:
Some of the features in data are categorical, so after creating dummy variables I had something like 120 features. And interesting thing is, that after scaling, which I needed for PCA, the variance I retained with arbitrary n_components was completely different, than variance I retained from data scaled just on continuous features, and not on dummy. I know that there is no meaning in scaling dummy variables, and it’s not handful. But after scaling dummy variables, in order to retain 99 percent of variance, I needed something like 110 features (originally 120), and after scaling just continuous features, to retain 99 percent of variance I needed less than half of the features (~55).
It’s strange for me, I cannot understand this behavior, is it okay at all?

Overfitting is when the test score is way below the training score, and when the latter is high. Here your test score is 0.92 which is not bat at all, so according to this score there is no overfitting here. To see that, you can dedicate a part of your dataset to be a validation set. So that, you will be able to evaluate training score, test score and validation score.

In the same idea you can use cross-validation (if you use scikit here), which will split you dataset into k-folds, k-1 of them will be used for the training and 1 for the test. This process in then repeated so that each of the folds is used as the test set.

If the score varies at each iteration, it could possibly mean that you overfit, if not then you can say that your model is robust

For the second question what do you mean by continuous feature ?


Which algorithm to use to identify clusters with a similar value? Here, an example of my problem:
10000 observations of people with several features [age, gender, region, number of sons, ...] and a value to predict "income".
There is not a general relationship between features and income, therefore a normal regression has poor results.
Nevertheless, I want to identify specific patterns where this relationship exists. For instance:
[young, woman, 2 son] -> high income
[young, man] -> small income
...

Maybe doing a clustering on the features, and then a regression on each cluster?
Or pattern recognition? topic modeling?

Thank you in advance

What you are describing is called ordinal regression. The target variable (income) is divided into discrete groups where the relative ordering between different groups is preserved (low, middle, high).

Binning a continuous variable as the advantage of better handling noisy data.


Standard Deviation of Classifier Models I am coming from the world of linear/logistic regression where residuals are assumed to be normal and the output includes some sort of standard error of residuals that one can use to make forecasts. Admittedly, I have very little experience in fitting and interpreting multinomial models..

Anyways, I am thinking about a specific problem:

Let's say we have lots of data on voters that have already cast a vote. With this data we wish to train a model to forecast future elections.

From the world of normal regression, there seems to be standard errors in forecasting each vote that could potentially be summed to create a random variable for the sums of votes for each candidate. These random variables could then be used to try to forecast the probability of each candidate winning.

My question is, what is the standard way of assessing the "standard error" of classifier models using tools like Stochastic Gradient Descent Classifiers for example. Or any of the more modern tools for classification. Some sort of bootstrapping approach?

Classifiers are commonly scored on the cross entropy or some other divergence, since we're talking about distributions. A multi-class classifier will give you a "score" associated with each class (option, outcome). It is usually easy to transform this score into a probability, so that the options add up to one. Furthermore, it is possible to associate with each class a confidence distribution. I don't know if this is what you are asking about, but here's a great blog post on it.


Catboost multiclassification evaluation metric: Kappa & WKappa I am working on an unbalanced classification problem and i want to use Kappa as my evaluation metric. Considering the classifier accepts weights (which i have given it), should i still be using weighted kappa or just use the standard kappa? I am not entirely sure of the difference to be honest.
model = CatBoostClassifier(
    iterations = 1000,
    learning_rate = 0.1,
    random_seed = 400,
    one_hot_max_size = 15,
    loss_function = 'MultiClass',
    eval_metric = 'WKappa', # weighted kappa
    #ignored_features = ignore_list,
    class_weights= weights,
    od_type = "Iter",
    od_wait = 50,
    task_type = 'GPU',
    border_count = 124,
    depth = 12
)

I posted the same question on the Catboost github (issues) page and got an answer.
The link can be found here: https://github.com/catboost/catboost/issues/1447
Answer:
Class weights and weights in WKappa are different.
Calculation of WKappa metric consists of two steps:

Confusion matrix calculation - here object and class weights are used.
WKappa calculation using confusion matrix - here values from confusion matrix multiplied by squared difference between classes. (i-j)^2.
In other words, weight of class 0 has the same effect when object of class 0 was classified as class 1 and when classified as class 5. But WKappa weight for misclassification 0->5 will be 25 times more than for misclassification 0->1.
So you can use both class weights and WKappa if your classes are ordinal.


Dealing with feature vectors of variable length How does one deal with a feature vector that can vary in size?

Let's say per object, I calculate 4 features. In order to solve a certain regression problem, I may have 1, 2, or more of these objects (no more than 10).  Thus, the feature vector is 4*N in length. How is this normally addressed?

The objects represent physical objects (e.g. other people) w.r.t. an observer. For a time slice, an object can be placed laterally, longitudinally, have some speed and some heading (4 features). Trying to solve: where should a person feel most comfortable. In some cases there is only 1 object, but there can be 2 or more.

Disclaimer: I have limited knowledge on ML approaches. I had classes in college years ago and took Andrew Ng's ML course online as a refresher but otherwise not up to speed. A starting place to look is appreciated.

First of all welcome to the community!

About the question I would say there are misunderstandings. You say you calculate 4 features "per objects". It means that every data point (object) is described with 4 features. So the length of features are not different. Please note that I say this because "you calculate features for your samples/data points/objects" so every object here is a point in 4-dimensional space. If you mean something else please correct me.

In other words, you have a data matrix with some rows (number of objects) and 4 columns.

More general about the whole concept. No, this is not a valid approach to create standard data. In standard data the number of columns are always the same because you can not study objects if they are defined in different spaces. Many ML algorithms also work with standard data which means you can not use them if the size of feature vectors vary. More conceptual so to speak, the size of features you calculate for objects can not vary. How? I want to extract features from faces. I say eye color, distance between eyes and distance between ears. I have 3 features and for any person who comes to my study I calculate the same set of features. How can the size be different if the feature extraction process is standard?

But in non-standard data you may end up with such a case. In this case you may standardize the features. E.g. as a baby example, imagine different graphs with different number of nodes and edges. You can describe each graph with its average degree, average path length and number of nodes. Then each graph is described with 3 features and you can feed it into an algorithm.

But if you don't want to standardize the data (because it usually has loss of information) you need to find tailored analytic methods for which you need to explain more about your data.

I hope it helped!

Good Luck!


Gower Distance Formula for KNN I'm trying to add Gower distance for my KNN implementation but I can't find out the formula for Gower distance. What is the formula for Gower distance?

The formula is defined in section two of Gower's A general coefficient of similarity and some of its properties.

  Abstract
  A general coefficient measuring the similarity between two sampling units is defined. The matrix of similarities between all pairs of sample units is shown to be positive semidefinite (except possibly when there are missing values). This is important for the multidimensional Euclidean representation of the sample and also establishes some inequalities amongst the similarities relating three individuals. The definition is extended to cope with a hierarchy of characters.


is there big difference between data Science , big Data and database? is there big difference between data Science , big Data and database? i am confused in these three can anyone help me to out of this confusion?

Well, they are absolutely different things but that are somehow linked. I gonna go through each of them.

Data Base

Think of a data base (DB from now) like a computer which only purpose is to store data accesable to be read. By data, and focusing only in SQL-like DB, I mean basically tables of information like and excel file with columns and rows. You can think of a SQL DB like an ecosystem of excel tables which share some common field. So basically a DB is a hardware infrastructure which allows to write and read a given amount of information within it (in the very beggining they were plain computers, of course with the rise of internet specialized hardware appeared). You can build your own DB in your personal computer.

Big Data

" An intellect which at a certain moment would know all forces that set nature in motion, and all positions of all items of which nature is composed ... for such an intellect nothing would be uncertain and the future just like the past would be present before its eyes." P.S.Laplace

Laplace did not think over it very deeply before formulate his sentence, obviusly if god had come to him and give what he wanted soon or later he would have realized all that information was indeed useless for him. Where could him store all that? From where should him start to read? What can he do with such amount of information he could never end to compute? In case he could read everything what should he calculate first?

These all are questions Big Data tries to answer and find a solution to. Big data appeared together with the huge websites internet gave birth, such as Amazon or Google. At some point they need to store so many information that it was imposible to store it in a single computer, not even in a big one, so they need to use a set of computers for which previous standard technologies for DB didnt work anymore. This fact was also the seed for No-SQL DB.

More about Big Data and non-sql here:
http://www.kdnuggets.com/2016/07/seven-steps-understanding-nosql-databases.html

Data Science

Finally Data Science is a statistical science, which aim is to extract order out of chaos as any other science, however meanwhile the rest of the sciences are focused in a single "narrow" piece of knowledge, such as biology, chemistry, etc; data science is on the other hand multidisciplinar science that could face problems from a broaden origin. Examples would be marketing or business oriented, cosmology, etc.

So data science uses mathematical and computer science algorithms to provide some useful information out of a disordered set of data. And here is where the link with Big Data comes, actually is in the question is set before: What can he do with such amount of information he could never end to compute? So Data Science and Big data are a very usual marriage in most IT companies nowadays and in more specific fields day after day. But, data science is apply a set of mathematical algorithms to data (like apply a calculation in an excel file to create a new row) and big data is the technology to have a huge amount of excel files (I use the word "excel files" here is just to make it easier to grasp).


Reinforcement learning: Discounting rewards in the REINFORCE algorithm I am looking into the REINFORCE algorithm for reinforcement learning. I am having trouble understanding how rewards should be computed.

The algorithm from Sutton & Barto:

What does G, 'return from step t' mean here?

Return from step t to step T-1, i.e. R_t + R_(t+1) + ... + R_(T-1)?
Return from step 0 to step t?, i.e. R_0 + R_1 + ... + R_(t)?

What does G, 'return from step t' mean here?

Return from step t to step T-1, i.e. R_t + R_(t+1) + ... + R_(T-1)?
Return from step 0 to step t?, i.e. R_0 + R_1 + ... + R_(t)?

Neither, but (1) is closest.
$$G_t = \sum_{i=t+1}^T R_i$$
i.e. the sum of all rewards from step $t+1$ to step $T$.
You are possibly confused because the loop for REINFORCE goes from $0$ to $T-1$. However, that makes sense due to the one step offset from return to the sum of rewards. So $G_{T-1} = R_T$ and $G_{T} = 0$ always (there is no future reward possible at the end of the episode).


Image feature extraction Python skimage blob_dog I am trying to extract features from images using:

def process_image(image_fp):
    image_ = imread(image_fp)
    image_ = resize(image_, (300, 200,3))
    image=equalize_hist(rgb2gray(image_))
    edges = skimage.feature.blob_dog(image)
    return edges.reshape(edges.size).tolist()

where image_fp is an image path.

I am having a problem due to the different sizes of the return. In general, the reshape guarantees the same size in the others algorithms.

Is there a way to get always the same size?

I only see a way: to truncate the lists (the stupid way).

Please read through the scikit documentation that is found here and I am assuming that you have gone through the method through which it calculates blobs in the images. If not the link is here. It return the 2D array of arrays with 3 values in each array, giving coordinates and std.deviation of Gaussian of the blob found. The length of the array returned is the number of blobs it has found in the image. So eliminating elements from that array by resizing it, is a wrong way of approach and may eliminate the features which you might find important.


Help with my training data I'm working on my first NN following a tensorflow tut and trying to use my own data.
After about 80 attempts of formatting my data and trying to load it into a dataset to train I'm throwing the towel.

Here is how my data currently looks

syslog_data = [
[302014,0,0,63878,30,3,1], [302014,0,0,3891,0,0,0], [302014,0,0,15928,0,0,2], [305013,5,0,123,99999,0,3],
[302014,0,0,5185,0,0,0], [305013,5,0,123,99999,0,3], [302014,0,0,56085,0,0,0], [110002,4,2,50074,99999,0,4],

In this the last item in each list is the label.
If you can tell me if I need to reformat my data and how or just how to get it loaded into a dataset properly.

Thanks for any help or advice you can give

Here is the full code:

import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
from . import syslog

print(tf.VERSION)
print(tf.keras.__version__)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation='relu'))
# Add another:
model.add(layers.Dense(64, activation='relu'))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation='softmax'))

model.compile(optimizer=tf.train.AdamOptimizer(0.001),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

dataset = tf.data.dataset.from_tensor_slices(syslog)

model.fit(dataset, epochs=10, steps_per_epoch=30)

There are a couple of problems and things you might want to add to your existing script.

Below I separate your example data into two NumPy arrays:

input values x
labels y

It is also important to make sure they are of type float32, because Tensorflow will complain if you pass it integers (as they otherwise would be interpreted).

The following works for me, the model trains to completion:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

syslog_data = [
    [302014, 0, 0, 63878, 30, 3, 1],
    [302014, 0, 0, 3891, 0, 0, 0],
    [302014, 0, 0, 15928, 0, 0, 2],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 5185, 0, 0, 0],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 56085, 0, 0, 0],
    [110002, 4, 2, 50074, 99999, 0, 4],
]

print(tf.VERSION)
print(tf.keras.__version__)

x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32)
y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation="relu"))
# Add another:
model.add(layers.Dense(64, activation="relu"))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation="softmax"))

model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(x, y, epochs=10, steps_per_epoch=30)


LSTM for time series - which window size to use I have a LSTM based network which inputs a n-sized sequence of length (n x 300) and outputs the next single step (1 x 300).

The "raw" data consists of a few thousand semi-processed sequences of variable length where each step is (obviously) 1 x 300. Hence X is (n x 300).

The way I am generating the training and test data, from this semi-processed original sequences is:

drop all original sequences shorter than K=9
apply a sliding window with stride 1 and length K=9 to each original sequence kept
shuffle the generated data
separate train/dev test/test data

Now all training/testing data is [9 x 300] and Y is [1 x 300]

The resulting network starts overfitting around epoch 10 which led me to 1. This is itself is not a problem since the results are of good enough quality.

If I try sequences as short as 6 (as in 6 x 300, that were dropped during the first phase) it gives proper enough information for us in y.

Hence the questions are about good practices and improving the network.

Technically, I am using Keras and using Google Cloud ML to execute the hyper parameter search.

The questions are:

should I stick to generating data with sliding windows since it is working properly?
if sliding window, k=9 is kind of a best guess, should I hyperparameterize this number K and search for the minimum loss with it in mind? I am afraid of doing so because hyperparameters typically affect the network itself, not the training data set, but note that K defines the data set length, would I be generating undesired bias by hyperparameterizing it?
if sliding window, should I be generating y shaped [9x300] with y=[x2, x3,...,x9, x10], instead of the current y generation? if I try both output structures and pick the one that lowers the loss in the dev test, I would check against a validation set?
if no sliding window, should I be training with one sequence per original time series input? and use attention or bidirectional lstm? 2
shuffling generates batches with shuffled partial sentences instead of batches from the same original sequence. but since shuffling is done prior to separating data for training and testing, it seems like I am not doing it the best way (or even properly). Should I first separate the original long sentences into train/each test phase, and then break them with the sliding windows (if applicable)?

References:

1 sliding window leads to overfitting in LSTM?
2 RNNs for time series prediction - what configurations would make sense

Three simple steps:

Make data stationarity (remove trends and seasonality).
Implement PACF analysis on the label data (For eg: Load).
Interpret the PACF plots to choose an optimal lag value for sliding
window

Concise tutorial for PACF analysis: http://rinterested.github.io/statistics/arima.html


Why Gradient methods work in finding the parameters in Neural Networks? After reading quite a lot of papers (20-30 or so), I feel that I am quite not understanding things.

Let us focus on the supervised learnings (for example).
Given a set of data $\mathcal{D}_{train}=\{(x_i^{train},y_i^{train})\}$ and $\mathcal{D}_{test}=\{(x_i^{test},y_i^{test})\}$ where we assume $y_i^{test}$ are unknown,
the goal is to find a function
$$
f_\theta(x), \qquad \text{such that} \quad f_\theta(x_i^{test}) \approx y_i^{test}.
$$
To do this, we need a model for $f$. Typically, neural networks are frequently employed. Thus we have
$$
f_\theta(x) = W^{(L+1)}\sigma(W^{(L)}\sigma(\cdots \sigma(W^{(1)}\sigma(W^{(0)}x+b^{(0)})+b^{(1)})\cdots )+b^{(L)})+b^{(L+1)}
$$
where $\theta = \{W^{(i)},b^{(i)}\}_{i=0}^{L+1}$.
Then $f_\theta$ is a neural network of $L$ hidden layers.
In order to find $\theta$, typically, one define a loss function $\mathcal{L}$.
One popular choice is
$$
\mathcal{L}(\mathcal{D}_{train}):= \sum_{(x_i^{train},y_i^{train})\in \mathcal{D}_{train}} \left(f_\theta(x_i^{train}) - y_i^{train} \right)^2.
$$
In order to find $\theta^*$ which minimizes the loss function $\mathcal{L}$,
a typical (or it seems the only approach) is to apply the gradient method.

As far as I know, the gradient method does not guarantee the convergence to the minimizer.

However, it seems that a lot of research papers simply mention something like

  We apply the standard gradient method (e.g., Adam, Adadelta, Adagrad, etc.) to find the parameters.

It seems that we don't know those methods can return the minimizer. This makes me think that it could be possible that all the papers rely on this argument (utilizing the parameters found by gradient methods) might be wrong. Typically, their justifications are heavily on their examples saying it works well.

In addition to that, sometimes, they mentioned that they tuned some parameters to run gradient methods. What does that mean ``tune"? The gradient method high depends on the initialization of the parameter $\theta^{(0)}$.
If the initial choice were already close enought to the minimizer, i.e., $\theta^{(0)} \approx \theta^*$, it is not surprising that it works well.
But it seems that a lot of trials and errors are necessary to find a proper (good and working well) initialization. It sounds to me that they already found the good solution via trials and errors, not gradient methods. Thus tuning sounds to me that they already found a parameter which already closes to $\theta^*$.

I start to think that there may be something I am not aware of as the volume of such researches is huge. Did I miss something? Or can we just do research in this manner? I am so confused...
I am not trying to attack or critize a specific paper or research. I am trying to understand.

Any comments/answers will be very appreciated.

If you have a convex cost function, gradient descent will find the global minimizer. Typically, the cost functions associated with neural networks are not convex - technically. However, if you read this question on Stats.SE, you find that the cost function of many neural networks is "close enough" to convex so that gradient descent still works. You'll find, technically, a local minimizer, but neural networks are often invariant to node-switching (as mentioned in the answer to the question to which I linked), so that you're still finding a very good minimizer.


How to Interpret the ROC Curve? i plotted the ROC curve for  RandomForest Classifier and this is what i get :

The shape looks weird to me , can somebody help me to make sense of it , and is this shape 'common' to not say normal?

thank you.

Imagine you have a blue line drawn from the point (0,0) to the point (0, 0.51). Do you see a common shape now?

That happens because you have few or no predicted probabilities above 0.9 or something close to it, so the point with 0 false positives is never touched.

The ROC curve you are watching is perfectly normal.


Which Algorithm or combination of Algorithms to use to develop supervised Video Event detection? I have to develop a video event detection tool in ticketing counter.The tool must take photographs of persons who jumb over gate without taking tickets.I have a set of videos in which people jumbing over gate.So with that data in hand How can i implement a Video event detection tool?I am new to video analytics.I don't know where to start.Where can i find some good tutorial about supervised video event detection.?

I recommend using a 3D convoluted neural network. Traditional 2D convoluted neural networks excel at identifying objects in images, and 3D convolution is used for identifying objects or movements also across multiple frames (i.e., time). Tensorflow offers this capability.


Ideas on cost function [If this is the wrong place to ask this question, please let me know where I can receive better answers:)]

So I'm building an internal tool for an digital media organization. The tool aims to track the number of hours (or any other measure of effort) spent on each brand by different people-roles (Eg: Designers, Content Writers, etc). Using the data collected for the first few months, the organization aims to achieve a better estimate for future quotes given to new clients.

I have thought about the following roles and their metrics:

Designers: Hours
Content Writers: Hours
SEO Guys: Hours
Client-Facing people. (Who actually talk to all the clients): Effort on a scale of 1-10

Whenever somebody starts work on a client (brand), they will start their timer and when done, they'll stop the timer. The tool will log the amount of time spent. Also, the current quotes to the different brands will be entered into the system initially.

I can think about this as a regression problem with the hours put in by different roles as the features and the 'quote' as the output. But I don't think there will be enough data collected in the first 2 months to give a realistic estimate for quotes to new clients.

What can my cost function be?

This will be my first project of this kind. Please give me some ideas. Even new approaches are welcome.

Thanks :)

In that great post, Joel Spolsky use Monte-Carlo simulation to give an estimation of the end of a project.

Thinking about a similar concept whould be to replace the notion of velocity by efficiency.

$$efficiency = \frac{quote}{hours} $$

The more discrete the scales of the quotes, the better.

It should work even with a few weeks of data.