which forecasting models could be chosen? I'm new for data analysis. I got some data from the regional environmental center.

Measurements:
Datetime, PointID, SubstanceID, Value (substances concentrations in air), MeteoID ,NextValue

Meteorological data:
MeteoID, Datetime, temperature, wind speed, wind diretion,humidity, pressure, precipitation.

Substances(8 substances(CO,Cl2,NO2, etc.), 5 of them have about 1.2 million records):
SubstanceID, Name, MaxVal (maximum allowable concentration value).

Points(9 static monitoring stations):
PointID, Adress, Longitude, Latitude.

Measurments table contains about 8 million records of substances concentratios in air(local mysql). I linked the measurement data and the meterologicala data (closest in time), and added the next value to measurement record. time between measurements 20 min( in average, for some periods data are missed).

I want to make a predictive data analysis and get short-term forecasts for substances concentrations depending on the previous value and weather data(may be for few hours). I am still thinking which methods, techniques can be chosen. I want to consider different methods. which toolkit is most suitable for this case? Should the data be considered as a multidimensional time series? I'm currently looking into the direction of some kind of neural network and implementation in python (but I still don’t know which package).

Should the data be considered as a multidimensional time series?

This depends on whether the target variable (the one that you want to predict) depends on the others. If not, there is no point in doing it. A fast way of checking if the variables are linearly dependent and, therefore, multidimensional forecasting is meaningful is by checking the linear correlation of the variables. Then select only the variables that have high correlation (>0.5) with the target variable to include them in your prediction model.

  I am still thinking which methods, techniques can be chosen.

The model that I recommend for time series forecasting is a Recurrent Neural Network. This is because of its inherent ability to store previous timesteps in its memory and to incorporate them into future predictions. This is very important, because it is among the few approaches that exploit the temporal dependencies between samples.

  I'm currently looking into the direction of some kind of neural
  network and implementation in python (but I still don’t know which
  package).

The most convenient way of implementing a recurrent neural network in Python is by utilizing the Keras framework. Please go carefully through this tutorial, as it will definitely be a very good first step to attack your problem.


Why random sample from replay for DQN? I'm trying to gain an intuitive understanding of deep reinforcement learning. In deep Q-networks (DQN) we store all actions/environments/rewards in a memory array and at the end of the episode, "replay" them through our neural network. This makes sense because we are trying to build out our rewards matrix and see if our episode ended in reward, scale that back through our matrix.

I would think the sequence of actions that led to the reward state is what is important to capture - this sequence of actions (and not the actions independently) are what led us to our reward state.

In the Atari-DQN paper by Mnih and many tutorials since we see the practice of random sampling from the memory array and training. So if we have a memory of:

$(action\,a, state\,1) \rightarrow (action\,b, state\,2) \rightarrow (action\,c, state\,3) \rightarrow (action\,d, state\,4) \rightarrow reward!$

We may train a mini-batch of:

[(action c state 3), (action b, state 2), reward!]

The reason given is:

  Second, learning directly from consecutive samples is inefficient, due to the strong correlations between the samples; randomizing the samples breaks these correlations and therefore reduces the variance of the updates.

or from this pytorch tutorial:

  By sampling from it randomly, the transitions that build up a batch are decorrelated. It has been shown that this greatly stabilizes and improves the DQN training procedure.

My intuition would tell me the sequence is what is most important in reinforcement learning. Most episodes have a delayed reward so most action/states do not have a reward (and are not "reinforced"). The only way to bring a portion of the reward to these previous states is to retroactively break the reward out across the sequence (through the future_reward in the Q algorithm of reward + reward * learning_rate(future_reward))

A random sampling of the memory bank breaks our sequence, how does that help when you are trying to back-fill a Q (reward) matrix?

Perhaps this is more similar to a Markov model where every state should be considered independent? Where is the error in my intuition?

The de-correlation effect is more important than following sequence of trajectories in this case.

Single step Q-learning does not rely on trajectories to learn. It is slightly less efficient to do this in TD learning - a Q($\lambda$) algorithm which averages over multiple trajectory lengths would maybe work better if it were not for the instability of using function approximators.

Instead, the DQN-based learning bootstraps across single steps (State, Action, Reward, Next State). It doesn't need longer trajectories. And in fact due to bias caused by correlation, the neural network might suffer for it if you tried. Even with experience replay, the bootstrapping - using one set of estimates to refine another - can be unstable. So other stabilising influences are beneficial too, such as using a frozen copy of the network to estimate the TD target $R + \text{max}_{a'} Q(S', a')$ - sometimes written $R + \text{max}_{a'} \hat{q}(S', a', \theta^{\bar{ }})$ where $\theta$ are the learnable parameters for $\hat{q}$ function.

It might still be possible to use longer trajectories, sampled randomly, to get a TD target estimate based on more steps. This can be beneficial for reducing bias from bootstrapping, at the expense of adding variance due to sampling from larger space of possible trajectories (and "losing" parts of trajectories or altering predicted reward because of exploratory actions). However, the single-step method presented by DQN has shown success, and it is not clear which problems would benefit from longer trajectories. You might like to experiment with options though . . . it is not an open-and-shut case, and since the DQN paper, various other refinements have been published.


Association rule learning for multi-classification suggestions The task is the following: given a training set of medical symptoms and an associated diagnosis, output a list of the most likely diagnosises for a combination of symptoms.
As of now, a solution exists which makes use of association rule learning methods: we find rules on the attributes of the training dataset and infer the likely diagnosises and their probability from the confidence we have in said rules to hold true.
However, this method doesn't seem to be able to scale for large datasets because of the sheer number of different attributes ($10^4$) and possible classes ($10^3$). Hence my question: are association rules a viable solution for this problem? Are there any alternatives?

Assuming there are many repetitions of each attribute and class, an embedding space can be learned. Attributes and classes that co-occur together will be projected into nearby space. One example is affinity weighted embedding.

Then prediction becomes approximate nearest neighbor search. For a given set of attributes, find the nearest class(es). One example is locality-sensitive hashing.

The strategy of combining learned embedding spaces and approximate nearest neighbor search scales very well for prediction. Search time can remain constant if you are willing to accept a looser definition of "approximate" as the data grows.


CNN 3D line angles prediction regression - results of training of phi depend on theta I am a beginner in "deep learning". What I am trying to do, is to predict two angles of a 3D line projected on a 2D image. The toy model is that I create a line going out from the centre of 48x48 array. The angle phi is the angle of the line in the image plane, angle theta is the angle of the line in the plane perpendicular to the image. Theta is just used to calculate each line point distance from the image centre and thus the line brightness at each point, by scaling line value with 1/R^2.

The problem is, that with theta fixed to 0 I can achieve ~3 deg precision of phi determination. However, if theta is allowed to go between 0 and 90 deg, the phi training results drop significantly. Of course, prediction of phi is worse for theta approaching 90 deg, because for theta=90 deg the phi is undefined. However, the results of phi training depend on the theta range allowed. I will have much better results of phi training if I allow theta in the range of 0-45, than if I allow theta in the range of 0-90. I would be grateful if someone could tell me, why this is the case and how to improve the results.

As for implementation. I am using Chainer based CNN. What I do with my layers does not seem to affect results much. At the moment I have 6 2D convolutional layers then 2 linear layers, the last one giving 3 variables. The variables are sin(phi), cos(phi), theta. Using sin/cos it is easier to cover whole 360 deg and take into account circular nature of angle. I output variable as is (linear). The loss function calculates phi as arctan2(cos(phi),sin(phi)), then uses mean_squared_error to compare its phi,theta with the input.

In this case changing CNN to a simple MLP with 3 layers changed the behaviour of the network completely. It becomes fast to train, accurate and more importantly, phi results do not depend on the range of theta used during the training (although they depend on particular theta value, as expected). This does not explain why in CNN there is dependency. I guess that convolution filters can't separate parts of the phase space, but I don't have intuition why.


How to train neural network a math multiplication table? I am trying to train neural network (brain.js) a multiplication table. It is not going too well: requires lots of hidden layers, iterations and very small error threshold, and the results are still incorrect. What am I doing wrong?
console.clear();
let net = new brain.NeuralNetwork({hiddenLayers: [128]});
let training_data = [
    {input:[2,2], output:[4/81]},
    {input:[2,3], output:[6/81]},
    {input:[2,4], output:[8/81]},
    {input:[2,5], output:[10/81]},
];
net.train(training_data, {
    errorThresh: 0.0000001,  // error threshold to reach before completion
    iterations: 500000,   // maximum training iterations
    log: true,           // console.log() progress periodically
    logPeriod: 10000,       // number of iterations between logging
    learningRate: 0.01    // learning rate
});
for(let i = 2; i < 3; i++){
    for(let j = 2; j < 10; j++){
        console.log(i + " * " + j + " = " + net.run([i,j])[0]*81);
    }
}

The output is:
2 * 2 = 4.265779517591
2 * 3 = 5.825156696140766
2 * 4 = 7.789406754076481
2 * 5 = 10.152319103479385
2 * 6 = 12.860703930258751
2 * 7 = 15.819618478417397
2 * 8 = 18.90912254154682
2 * 9 = 22.006048411130905
After receiving the answer from SRJ I have changed the neural net so, that it now has 4 hidden layers and receives 10 000 inputs:
console.clear();
let net = new brain.NeuralNetwork({hiddenLayers: [4]});
let training_data = [
];

for(let i = 0; i < 100; i++){
    for(let j = 0; j < 100; j++){
        training_data.push({input:[i,j], output: [i*j/10000]});
    }
}
net.train(training_data, {
    errorThresh: 0.001,  // error threshold to reach before completion
    iterations: 20000,   // maximum training iterations
    log: true,           // console.log() progress periodically
    logPeriod: 100,       // number of iterations between logging
    learningRate: 0.0001    // learning rate
});
let random_numbers = [Math.floor(Math.random()*50), Math.floor(Math.random()*50)]
console.log(random_numbers[0] + " * " + random_numbers[1] + " = " + net.run([random_numbers[0], random_numbers[0]])[0] * 10000);

Still does not work. The output: "29 * 42 = 728.9011776447296". (while in reality should be 1218).

As your function is linear and you have a hidden size of 128, Your model might be suffering from overfitting. Try to lower the hidden layer to 2 or 4 and try to regularize the model using dropout. And also neural net requires a lot of data to train. With small data neural net does not perform well. So if you are unwilling to provide extra data ,you should go for linear model like linearRegressor from sklearn.It will do your task just fine with proper hyperparameter. But if you want to train a neural net then you have to provide a huge amount of data and also choose proper hidden size and regularizer.


Fixed target values but more ID for clafssification I have a data set where there the target column as values between 1-15. Now I need to predict that based on the features. the feature values are have unqiue IDs. but the target column is the same value range.
one row could looks like

1,0.5,2.3,1.23,5 where first entry is the id and last entry is the target. In that why I have total 500 uniq IDs which has different feature values but same target range.

Now I would like to ask should I take the ID in the traning process ? or take features ? Or what would be the best approch for this classification ?

so one

I believe all id's are unique(no repetitions/duplicates). You can delete this variable and proceed for analysis. The variables like this(ids) and the variables which have the same values (zero variance) are not at all able to draw patterns to predict the target variable. You can delete these type of variables without a doubt.


After plotting my predicted values against the true label values, I didnt quite get the answer I was looking for I downloaded data on wine quality and tried to run a regression model to predict the quality, However I did not receive the plot I was expecting. The mean absolute error for the wine quality was approximately 0.5, as a result of this I thought that the True vs predicted value plot would look very similar however I got a graph that Isn't really what I was expecting. Is this what happens when you work with discrete parameters?

As I'm new to this, I was under the impression that this final plot would always resemble your standard linear graph if your predicted and true values were very similar.

Here is the link to my code if it helps: https://colab.research.google.com/drive/1mxRIx5ufVsA0ljdTpL0Ud0qm2y39eyGX

The true quality is 4,5,6... the predicted quality deviates from the true one. This deviation is expressed by the variation of the dots on the x axis. The line in the plot shows, that there is positive correlation between your prediction and true states (which is good and expected).

Think about the magnitude of mae 0.5 in your case. Means „half a class wrong“. You likely have some concentration of predicted values around the true value (not good to spot in such a figure) and some outliers, viz relatively bad predictions, which are far away from the true class. You can also see that your prediction is bad for low and high classes.

I guess this is a linear regression? Try multinominal logit or the like. In you linear regression the prediction can be any rational number. What you would like to have as a prediction is a class, so 4,5,6 or whatever the class is called.


Combine RepeatedStratifiedKFold and crossval Is it okay to combine RepeatedStratifiedKFold and cross_val_score?

The result in the example below are 30 accuracy values (3x10 splits).
How to calculate the final accuracy value for the 3 times 10 fold cv?

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=rand)
rf = RandomForestClassifier(n_estimators=e, random_state=rand)
scores = cross_val_score(rf, data, Y, scoring='accuracy', cv=cv, n_jobs=-1)

I found a recent paper dealing with repeated cross-validation:

On estimating model accuracy with repeated cross-validation

It states: If $A_i$ is the result returned by a single k-fold crossvalidation, then
the cross-validation result for the whole repeated cross-validation (n repeats) is: $A = \frac{\sum_i A_i}{n}$.

So to calculate the final score, we need a "chunks" function: (Code from stackoverflow)

def chunks(lst, n):
    means = []
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        means.append(np.mean(lst[i:i + n]))
    return means

Then the final score can be calculated by:

np.mean(chunks(scores, n_splits))


Matplotlib WIndow Not responding when loop is initialized I am initializing empty figure when executing plt.figure(). It openes fine. But as soon as I have the following code implemented opened graph gets not responding. Any idea why this window is getting not responding?

import matplotlib.pyplot as plt
plt.figure()   # It opens graph fine

with open("data.txt") as file_data:

    while True:
        pass

The while True: loop will be causing the python interpreter to think there's something more to do. It will remain unresponsive until it 'finishes' that loop, which it will never do. Remove the Loop or replace pass with break


Time Series Machine Learning Feature Selection Problem I have to solve a time series model that can take one of two shapes. It can probably take more but here are the two I'm going to ask about. If you have other ideas they are of course welcome.

First Possible Model -

  X(Dependent variable 'Spending') = X(lag1)...X(lagN) + X(Dummy
  variable when US) + X(Dummy Variable when Mexico) ... + Error term

Or Make a separate model for each Country like -

  X(Total Spending in the US only) = X(lag1)...X(lagN) + Error term

  X(Total Spending in the Mexico only) = X(lag1)...X(lagN) + Error term

  … for Each country

Mathematically I can't decide what approach is better. I will use an F-Statistic, dickey fuller statistic to check the auto regression for stationarity and then compare the two models but I wanted to see what others thought of the theory and if you should ever include the dummy variables.

I'm looking more for an answer that includes mathematical reasoning.

I will give you some other idea of it, but without mathematical.

Like horaceT's mentioned, it's all about feature construction and feature selection, then you have many machine learning method which can do the regression.

So, just look it as regression model, not just the time series model. Then you will get the following model:

y(Dependent variable 'Spending') = X1(moving average term)...X2(lastquarter moving average term) + X3(country id) + X4(Fourier term) ... + Error term

The moving average can catch the dependent variable's trend, Fourier term can catch the dependent variable's cycle. Maybe you can construct other feature.

Finally, you can selection the features using tree-based model, such as xgboost.


Why replay memory store old states and action rather than Q-value (Deep Q-learning) Here is the algorithm use in Google's DeepMind Atari paper

The replay memory D store transition (old_state, action performed, reward, new_state)

The old_state and the performed action a are needed to compute the Q-value of this action in this state.
But since we already compute the Q-value of action a in state old_state in order to choose a as the best action, why don't we simply store directly the Q-value ?

But since we already compute the Q-value of action a in state old_state in order to choose a as the best action, why don't we simply store directly the Q-value ?

That is because calculating a relevant TD Target e.g. $R + \gamma \text{max}_{a'}Q(S', a')$ requires the current target policy estimates for action value $Q$. The action value at the time of making the step can be out of date for two reasons:

The estimates have changed due to other updates, since the experience was stored
The target policy has changed due to other updates, since the experience was stored

Storing the $Q$ value at the time the experience was made should still work to some degree, provided you don't keep the experience for so long that the values are radically different. However, it will usually be a lot less efficient, as updates will be biased towards older less accurate values.

There is a similar, but less damaging, effect from the experience replay table even with $Q$ recalculations. That is because the distribution of experience may not match what the current policy generates - something that most function approximators (e.g. neural networks used in DQN) are sensitive to. However, there are other factors in play here too, and it can be beneficial to train on a deliberately different distribution of experience - for instance prioritising experiences with larger update steps can speed learning and keeping older experiences available can reduce instances of catastrophic forgetting.

Note that if you were using an off-policy Monte Carlo method, you could store the Monte Carlo return in the experience replay table, since it does not bootstrap by using current value estimates. However, the early parts of older less relevant trajectories would stop contributing to updates in that case once the target policy had changed significantly during learning.


Best fit sinusoidal wave in r I am working with the following data.

x <- 1:36
y <- c(6.62,6.57,6.57,6.55,6.52,6.56,6.52,6.55,6.54,6.55,6.55,6.62,6.78,6.73,6.6,6.59,6.57,6.57,6.51,6.55,6.39,6.4,6.15,6.22,6.3,6.27,6.27,6.24,6.24,6.23,6.22,6.26,6.23,6.22,6.2,6.14)

The values of y represent monthly totals spanning 3 years beginning with January. Relatively large increases of y tend to occur in January (at values of 1, 13, and 25 for x) and they fall gradually throughout the year. I would like to represent this behavior with an asymmetric sinusoidal wave that best fits the pattern. Per this thread, I experimented with the following R code.

mylm <- lm(y ~ x + sin(2*pi/24*x) + cos(2*pi/24*x))
plot(x, y)
lines(x, mylm$fitted.values, col = "blue")

The approximation is decent, but the symmetry of the fitted wave does not capture the sudden rises in January and the gradual declines across the remaining months.

I don't know the proper vocabulary to fully articulate what kind of wave I am seeking. Perhaps a smooth triangular wave? Thanks.

The pattern you describe is not obvious, or strong. Nevertheless, to answer your question, use Fourier series like this:

library(forecast)
y <- ts(c(6.62,6.57,6.57,6.55,6.52,6.56,6.52,6.55,6.54,6.55,6.55,6.62,
          6.78,6.73,6.6,6.59,6.57,6.57,6.51,6.55,6.39,6.4,6.15,6.22,
          6.3,6.27,6.27,6.24,6.24,6.23,6.22,6.26,6.23,6.22,6.2,6.14),
        frequency=12)
mylm <- tslm(y ~ trend + fourier(y, K=3))
plot(y, type='p')
lines(fitted(mylm), col='blue')

This is called a harmonic regression. See https://otexts.org/fpp2/useful-predictors.html#fourier-series for the details.


why we have run make.sh file initially in darknet and YOLO object detection? In have seen a couple of text and objection detection algorithm wher the first step everyone dose is to install cython and run a make.sh file . why we have run make.sh file initially in darknet/YOLO object detection ?

Below are the 2 links im referring to build yolo algorithm using darknet and im not able to understand why cython is needed and why it is necessary to compile make.sh file for the first step to build yolo model  ?

How to compile on Linux (using cmake)

How to compile on Linux (using make)

I know what is cython and make (but I never use YOLO!)

Cython is a C-extension for python. It allows you to write code C/C++ in a python script. (use for very fast program execution)

Make is command which executes your makefile. You can consider makefile is a build script to create/tune the necessary things like environment/folders/.. etc.


ValueError: bad input shape (111, 3) import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset = pd.read_csv('iris.csv')
X = dataset.iloc[:,:4]
y = dataset.iloc[:,4]

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(sparse=False)
y=encoder.fit_transform(y.values.reshape(-1,1))

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

from sklearn.naive_bayes import GaussianNB
classifier = GaussianNB()
classifier.fit(X_train, y_train)

I am getting these small errors which hinder my projects. Can you please provide a solution to this error?
Also how can I avoid these mistakes?

You are getting error because of below-

y=encoder.fit_transform(y.values.reshape(-1,1))

This is changing the shape of the y. try print(y.shape)

You might want to handle you data once its labels and transformed. I am not sure if you are trying to "learn" to apply OneHotEncoding. If yes, you might wanna look at the documentation in detail. here-https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

Let me know if you have tried it and still have problem.


Deep Learning Network decreasing in accuracy In order to familiarize myself with semantic segmentation and convolutional neural networks I am going through this tutorial by MathWorks:

Semantic Segmentation Using Deep Learning

I did not use the pretrained version of Segnet since I wanted to test on my custom data set. All code is the same, however I have different classes, and fewer labels. Below image shows the label name and amount of pixels associated with each.

To make up for the low pixel data for class 2, median frequency balancing was performed.

imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount
classWeights = median(imageFreq) ./ imageFreq

I proceed to train the network using the code provided in the example with the options and lgraph unchanged. The  SegNet network is created with weights initialized from the VGG-16 network.

Unlike the example, I get a much lower global accuracy:

To gain further insight I plotted the Mini-batch accuracy and Mini-batch loss against each iteration.

It is clearly seen that the accuracy fluctuates wildly and ends up worse than it started, so the network learned absolutely nothing! However the loss decreased gradually.

A possible solution I propose would be to use inverse frequency balancing. However, in the example above, median frequency balancing was already performed, so I doubt how much this would help.

Is the terrible performance related to simply not having enough training data? Can anything be be done to improve performance with existing data?

Any suggestions are greatly appreciated.

Possible tricks which can improve the accuracy of semantic segmentation models:

try lowering the learning rate
try variating the batch size
train longer (more epochs)
train the model on larger images
increase the dataset size


Is always converting a input vector into matrix and apply cnn's good idea? I know the benefits of using cnns(reduced size weight matrices). Is it a good idea to convert a input vector(which is not a image) into a matrix and apply cnn's. What I understand is that it should not be done because this would enforce some relationship between input vector value which actually doesn't exist.
Am I correct or there is some way we can apply cnn's to reduce computation?
If cnn's can't be applied what could be the method for reducing computation for very high dimensional input

This is a job for dimensionality reduction techniques. PCA is a simple and intuitive method that you should probably try first. If you find it is insufficient or ineffective, you could try using an autoencoder.
As you said, using a CNN would imply a locational relationship between variables. If such a relationship does not exist, the CNN could have difficulty finding suitable weights. It is possible that forcing the CNN to link seemingly unrelated variables could help the model generalize better or reveal underlying structure that you cannot perceive, but there are other problems that you will run into. For example, CNNs generally use pooling layers as a form of nonlinear dimensionality reduction. However, if your variables are completely independent and unrelated, you would simply be throwing away possibly critical information using this down-sampling technique.
So while it might be feasible to use a CNN, you should probably use the right tool for the job.


Training a convolutional neural network for image denoising in Matlab I am currently trying to train CNNs to remove Poisson noise from images. The software I am using is Matlab 2018b, however the results I am getting are poor.
I have followed the steps provided in the following link. Here it is stated that we need to make a denoisingImageDatastore, which holds patches of our training images, and applies Gaussian noise to them. This isn't what I am looking for, but for the sake of practice I decided to try it out on my of PASCAL VOC dataset. I will now roughly outline the steps in my matlab code for this.

I first created an imageDatastore holding 45 of our images used for training.
Fifteen of those images will be used for validation
Now we create denoisingImagedatastores for both the training and validation set. There will be 60 patches per image, with the patch size 50x50
Then we specify the training options
I then specified the network layers using dnCNNLayers function from matlab
The next step is to train the network using the trainNetwork function

Here is a screen shot from the end of the training process.

I have done only 100 iterations, but the results are not too bad.
Here we can se the pristine image, the noisy image and the denoised image using this network.

On the following link is the code I used for this. If you have any suggestions how the RMSE could be further lowered, it would be appreciated.
Please note that I have previously trained the same network on a much larger dataset and with nearly 10000 iterations, and the best rmse it could achieve was about 2.3. So obviously some parameters need to be changed.

So, I can point a few things that may cause a sub-optimal result:

Your model may be too simple. Although using CNNs, which are pretty much the state of the art in Neural Networks based image processing, you need to pay attention to the archietcture you are using. For instance, you can take a look into that paper, where the authors have used CNNs + residual learning to denoise images
Your dataset isn't complex enough/isn't big enough. As far as you said, you have pre-trained the network on a much larger dataset, however, the notion of "large dataset" isn't precise enough. Maybe you don't have enough data to train the model. In that sense, it may be useful use a well documented existing architecture where you can compare your results with those reported, and see if you indeed have the data needed to train it.
Another concern is about the noise on train/test data. If your model is to complex, and your data too poor, it may be memorizing the mechanics of denoising gaussian noise (which seems to be the noise component in your training data), and hence does not generalize good on Poisson noise. To solve this, you may consider change the noise component on the train data (if it was generated artificially), or getting a different dataset.
Change losses, take a look on other metrics. Loss choice have a great impact on the final model, since they determine how the gradient is calculated. For instance, following Goodfellow, Bengio and Aaron reasoning on the book "Deep Learning",

  For example, mean squared error applied to the pixels of an image implicitly
  specifies that an underlying cause is only salient if it significantly changes the
  brightness of a large number of pixels. This can be problematic if the task we
  wish to solve involves interacting with small objects.
  Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.

Hence, despite looking into RMSE, I may recommend Peak to Signal Noise Ratio (PSNR) or Structural Similarity (SSIM).


PAC Learnability - Notation The following is from Understanding Machine Learning: Theory to Algorithm textbook:

Definition of PAC Learnability: A hypothesis class $\mathcal H$ is PAC learnable
if there exist a function $m_H : (0, 1)^2 \rightarrow \mathbb{N}$ and a learning algorithm with the
following property: For every $\epsilon, \delta \in (0, 1)$, for every distribution $D$ over $X$, and
for every labeling function $f : X \rightarrow \{0,1\}$, if the realizable assumption holds
with respect to $\mathcal H,D,f$ then when running the learning algorithm on $m \ge m_H(\epsilon,\delta)$
i.i.d. examples generated by $D$ and labeled by $f$, the algorithm returns
a hypothesis $h$ such that, with probability of at least $1 - \delta$ (over the choice of
the examples), $L_{(D,f)}(h) \le \epsilon$.

1) In the function definition $m_H : (0, 1)^2 \rightarrow \mathbb{N}$; what does a) 0 and 1 in the bracket, b) the integer 2, and c) $\rightarrow \mathbb{N}$ refer to?

The explanations are as follows:

$m_H:(0,1)^2 \rightarrow \mathbb N$ is a similar notation to $f:R^n\rightarrow \mathbb N$ which means it takes a n-dimensional input consisting of real numbers only. In the case of PAC learning the input is 2 dimensional consisting of numbers between $0$ and $1$ only which stand for values of $\epsilon, \delta$ respectively.
The integer $2$ as explained above is the dimension of the input vector.
$\rightarrow \mathbb N$ means mapped to natural numbers. In case of PAC learning that for each value of $(\epsilon, \delta)$ a function $m_H$ maps it to natural numbers, simply put $m_H(\epsilon, \delta) = $


How to import trained keras models I have some trained models that I could import and compare, but then the same code gave:

"WARNING tensorflow:From 'where its saved' colocate with (from tensorflow.python.framework.ops)
  is deprecated and will be removed in a future version.
Instructions for updating:
  Colocations handled automatically by placer."

As well as 2 other deprecation warnings.

The code that I am using to import the models is:

"model_1 = load_model("filepath\\filename.h5")"

Restating my computer fixed the issue, but I'm curious as to why this happened, if anyone can explain.

I am afraid if you had saved the model in your drive rather google collab directory as it will lost after run-time session interrupt.

model_new = load_model('BestModelWeights.hdf5')  # will load model , if BestModelWeights file  exist in collab directory --which i feel lost in your case that could be the reason you are getting "where it saved" error.

I would advise use google drive to save weights and directly load weight file when you need.For this you need to follow two steps

Mount google drive and authenticate collab.

from google.colab import drive
drive.mount('/content/drive')
Load model

model_new = load_model('/content/drive/My Drive/BestModelWeights.hdf5')

and you can play with your new loaded Model.

hope it helps.
JSH


What does BNG stand for When i look at the available datasets in https://www.openml.org i often see a BNG dataset with no further information about it.

Can someone explane what BNG means in this context?

I am especially interested in this dataset: https://www.openml.org/d/1389

Has anyone more information about where this data set comes from?

The Bayesian Network Generated (BNG) datasets are a set of artificially generated datasets openly available on OpenML. These datasets were generated to fill the need for a large heterogeneous set of large datasets. This paper describes the BNG generator best:
Algorithm Selection on Data Streams.

Small quote from the paper about the BNG data generator:

  The generator takes a dataset as input, and outputs a data stream containing
  a similar concept, with a predefined number of instances. The input dataset is
  preprocessed with the following operations: all missing values are first replaced by the majority value of that attribute, and numeric attributes are discretized using Weka’s binning algorithm.

A personal note: For general Machine Learning studies, I would refrain from using BNG (or any other kind of artificially generated) datasets, as the concept is generally simpler than the original dataset. Instead, it is recommendable to use a per-defined benchmark suite, such as the OpenML-100.


Random Forest Classifier Probabilities My dataset has 140k rows with 5 attributes and 1 Attrition as target variable (value can either be 0 (Customer churn) or 1 (Customer Does not churn)). I divided my dataset in 80% training and 20% testing. My dataset is heavily imbalanced. 84% of my dataset has 0 as target variable and only 16% has 1 as target variable.

The feature importance of my training dataset is as follows:

ColumnA = 28%, ColumnB = 27%, AnnualFee- 17%, ColumnD - 17% and ColumnE - 11%

I initially wanted to do a very simple check of my model. After creating a Random Forest Classifier I tested the model on a dataset with just 5 rows. I kept all variables constant except Column AnnualFee. Below is a snapshot of my test dataset:

 Column A   Column B    AnnualFee   ColumnD ColumnE
 4500       3.9          5%         2.1      7
 4500       3.9          10%        2.1      7
 4500       3.9          15%        2.1      7
 4500       3.9          20%        2.1      7
 4500       3.9          25%        2.1      7

I expected that as annual fee increases the probability of customer churn also increases. But my rf.predict_proba(X_test) seems to be all over the place. I am not sure why this is happening:

I tried two different codes but the anomaly seems to be happening on both the codes:

Code 1:

rf = RandomForestClassifier(n_estimators = 400,random_state = 0,
min_samples_split=2,min_samples_leaf=5,
                      class_weight = {0:.0001,1:.9999})
rf.fit(X_train, Y_train )

Code 2: Not My Code - Got it Online

from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV
clf_4 = RandomForestClassifier(class_weight = {0:1,1:5})
estimators_range = np.array([2,3,4,5,6,7,8,9,10,15,20,25])
depth_range = np.array([11,21,35,51,75,101,151,201,251,301,401,451,501])
kfold = 5
skf = StratifiedKFold(n_splits = kfold,random_state = 42)

model_grid = [{'max_depth': depth_range, 'n_estimators': estimators_range}]
grid = GridSearchCV(clf_4, model_grid, cv = StratifiedKFold(n_splits = 5,
random_state = 42),n_jobs = 8, scoring = 'roc_auc')
grid.fit(X_train,Y_train)

I would really appreciate any help on this!

What do you mean by 'all over the place'? Have you tried actually using scoring measures such as accuracy scores, AUC, etc? These will play a vital role in determining how your model is performing. I'd suggest ROC AUC as to start off since accuracy will be pretty misleading due to the class imbalance.
The class weights you used make little sense. You usually would use something like n_samples / (n_classes * np.bincount(y)) for your weighting, but since sklearn provides automatic balancing of classes, you should probably use that by setting class_weights='balanced': http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html


Is it valid to shuffle time-series data for a prediction task? I have a time-series dataset that records some participants' daily features from wearable sensors and their daily mood status.

The goal is to use one day's daily features and predict the next day's mood status for participants with machine learning models such as linear regression.

I think cross-validation could be a good way for me to evaluate the performances. However, would shuffling the data randomly be fine?

Someone told me that because I am using a time-series dataset and I am trying to do a prediction task, shuffling the data randomly will cause some mix-up of future and past, which makes my model meaningless.
However, I think I can still use the strategy of randomly shuffling the dataset because the learning model is not a time-series model and, for each step, the model only learns from exactly 1 label value instead of a series of labels.

It depends on how you formulate the problem.

Let's say you have a time-series of measurements X and are trying to predict some derived series of values (mood) Y into the future:

X = [x0, x1, x2,.....]
Y = [y0, y1, y2,.....]

Now, if your model has no memory, that is, it is merely mapping xN -> yN, then the model does not care what order it sees x and y in. Feed-forward neural networks, linear regressors etc. are memory-less models.

But, if your model has memory, that is, it is mapping a sequence of features to the next days moods xN-2, xN-1, XN... -> yN-2, yN-1, yN..., then the ordering matters. The model in this is keeping track of what it has seen before. The model's internal parameters are changing and persisting with each new example it sees. The current prediction depends on the last prediction. Recurrent neural networks have memory, so order matters.

You can get around the memory requirement by restructuring your dataset. You can concatenate consecutive features and map them to next day's mood instantaneously (xN-2, xN-1, xN) -> yN. In this way, your input feature will incorporate information about the past, but the model won't care about the order since all the temporal information is encoded in the new feature, and not the model. Your new dataset will look like:

Xnew = [(x0, x1, x2), (x1, x2, x3), (x2, x3, x4),...]
Y    = [         y2,           y3,           y4,...]


Finding the relation between two dimensions in a multi-dimensional problem I have a collection of data points. Each point has 6 dimensions (x1, x2,...x6). I want to find a relation between two dimension (e.g. x1 vs x2). What I have been doing so far is look for points where the other dimensions (x3 to x6) are relatively constant, by defining a band. This way I would get several groups of data points where only the two dimensions of interest would change.

I was wondering if there is a better way of analyzing the relationship between these two dimensions. I looked at PCA, but I have a feeling that it does not help me much. If I reduce the problem to two dimensions the axes are basically meaningless.

Can you guys give me some directions to look at?

You might take a look at canonical correlation analysis. It tries to find correlation between two sets of data. I guess you could adapt it to explore your data.


Accuracy is lower than f1-score for imbalanced data For a binary classification, I have a dataset with 55% negative label and 45% positive labels.
The results of the classifier shows that the accuracy is lower than the f1-score.
Does that mean that the model is learning the negative instances much better than the positive ones?
Does that even make sense, to have accuracy less than the f1-score?

I'll try to answer this with a couple examples:
Say we have 100 instances (55 negative, 45 positive). Let's say we predict 1/45 positives and 55/55 negatives correctly. Then our accuracy is 0.56 but our F1 score is 0.0435.
Now suppose we predict everything as positive: we get an accuracy of 0.45 and an F1 score of 0.6207.
Therefore, accuracy does not have to be greater than F1 score.
Because the F1 score is the harmonic mean of precision and recall, intuition can be somewhat difficult. I think it is much easier to grasp the equivalent Dice coefficient.
As a side-note, the F1 score is inherently skewed because it does not account for true negatives. It is also dependent on the high-level classification of "positive" and "negative", so it is also relatively arbitrary. That's why other metrics such as Matthew's Correlation Coefficient are better.


Cost sensitive learning and class balancing I am facing a classification problem with classes that are really imbalanced (more or less 1% of positive cases). In addition, the "cost" of a False Negative (FN) is much higher than the cost of False Positive (FP).
Considering so, I decided to insert the weights into my classification model. Which is the best way to compute these weights? Ideally, the weights have to take into account both the data unbalancing and the miss-classification cost.
I am not interested in putting a 0/1 label to each record, but just in ordering the test dataset according to the output score. The idea is to contact the records with the highest score to offer a product.
I would like to use these weights using, for example, the sample_weights parameter available in most of scikit-learn classification algorithm (here the documentation).
Is it a good idea to oversample/downsample the data and then use the weight only to control the miss-classification cost? Or would it be better to use weights that are able to consider the whole situation? Are there any known ways to compute these weights?

Not clear if you are saying the cost of a FN or a FP is higher, you only mention FN in your statement. Think that you mean a FN is more costly and that a positive means a 1.
In general, if an incorrect prediction for the minority case is more costly (FN), you should sample that minority case higher or the majority case lower so the ratio is closer to 1:1. Balancing will help increase the accuracy of your model when predicting the minority case. Accuracy in predicting the majority case will already be higher as there are more samples to use for that case. Undersampling, oversampling and SMOTE are all useful ways to accomplish this balancing of samples, and each has their own strengths and weaknesses.
However, doing this sample balancing will quickly increase the number of FP, so even though the cost may be lower for a FP, the cost will add up quickly. For example, every 1 FP that you decrease, you may get 10 or 20 more FN
After doing this balancing, you can start to adjust the weights to get the best ratio of FN to FP, trying to get the total cost as low as possible.
minimizing: total cost = FN x cost_of_fn + FP x cost_of_fp
Not sure if there is a mathematical equation to solve this, but you can run this iteratively, change weight ratios for the 2 classes, and calculate the total cost using a confusion matrix to get FN and FP, and graph the results for cost (y) vs weight ratio(x), looking for a minima. I would start with a ratio that is equal to the ratio of your costs.
Example: If the cost of a FN is 10 dollars and the cost of a FP is 1 dollar, then the ratio should be 10:1 for minority:majority class
A straightforward calculation shows how to derive the desired mix of class i: μ(i) in terms of the calculated weight w(i) and number of examples in the training set (before any up or down sampling) n(i). This formula is given below:

Reference: https://medium.com/rv-data/how-to-do-cost-sensitive-learning-61848bf4f5e7


Why is deep learning used in recommender systems? I am currently reading a lot about recommender systems (RS) and came across that many RS are based on deep learning.
However, I never find a good scientific article why deep learning is used in RS and why it is more successful compared to other methods.
Now my question is:

Why is deep learning used in RS?

Why would it be better to use deep learning in RS?

How does deep learning work in RS?

I would be very happy if you could help me to answer my unanswered questions. I would be very happy if you would add a source (that is scientific, no medium etc.) that I can read more about it.

"Recommender Systems" is a very broad area and can be approached from different optics: latent variable models, graph models, etc.
"Deep learning" is an umbrella term for gradient-based optimization of deep differentiable models, and has been used to model all sorts of supervised learning problems, including graphs and latent variable models. Therefore, it has been applied to many different areas, with a great degree of success.
Multiple deep learning approaches have been applied to recommender systems because deep learning performed well in many other areas, so why not expect it to perform well for recommender systems.
Neural approaches have proven to be effective also for recommender systems. That is why they are used nowadays.
Also, different inductive biases have been applied to neural networks in other areas and, in some cases, such inductive biases are also useful for recommendation, like the sequential bias for recurrent networks, or the structured bias for graph neural networks.
If you want to explore the academic publications related to neural recommender systems, I would suggest that you start with the article Neural collaborative filtering and the survey Deep learning based recommender system: A survey and new perspectives.
Also, in order to get a broad view of the state of the art in recommender systems, you may have a look at papers with code, where you can see the best-performing systems for different benchmark datasets, with links to the associated articles and source code repositories.


How do I choose a discount factor in Markov Decision Problems? I'm referring to the gamma in the Value function:

This is the typical value function of Reinforcement Learning. The discount factor evaluates the importance of the accumulated future events in your current value. The smaller the number, the less important are the future events in the current action.

Usually this number is selected heuristically. I usually select 0.9. If I don't want any discount then I would select 1.


Calculating Median and Mode of numerical variables for different subgroups in R I have customer call data and I want to get the median and mode for the call success rate for different subgroups.

My variables are: Customer ID, Employment Status (Retired, Employed, Unemployed), count of calls, count of successful calls, call success rate (successful calls/number of calls)

So far I worked in Excel to get the average call reachability. Mode and median are not easy to calculate with Excel, because it can only do it for up to 255 recods I believe. I know that the summary command in R provides you with summary statistics for each variable but what if you want to get the median and mode of each subgroup. Meaning: Median and Mode of call succcess rate for Retired, Employed and Unemployed customers separately? Why I want to calculate the Median and Mode, although I have the average already? Because both are more resistant to outliers. But I am open for discussion if you think it is not necessary :)

assuming your data is stored in an object named df you can do:

tapply(df$S_Calls, df$Emp_Stat, median)

As for the mode, oddly enough R does not have a built in function for that. You could define one yourself using:

mode_stat <- function(x) {
  ux <- na.omit(unique(x))
  ux[which.max(tabulate(match(x, ux)))]
}

and then do in a similar fashion:

tapply(df$S_Calls, df$Emp_Stat, mode_stat)


What is Reinforcement Learning? I am familiar with the concepts of Supervised and Unsupervised Learning but recently Reinforcement (Reinforced?) Learning also popped up before me a couple of times. Could anyone give a hint what is it, what is the main difference from the other two (or from the Unsupervised type, specifically), with some good example? Is it a promising alternative worth to explore or just some hyped niche curiosity?

Reinforcement Learning uses a simple logic of learning in which the network tries to learn from the feedback it obtains. This tries to optimise the overall reward in the long run instead of the current reward.

This is one of the best platform to read about it. It also contains some useful links.

As stated by the wiki,
The basic reinforcement learning model consists of:

a set of environment states S;
a set of actions A;
rules of transitioning between states;
rules that determine the scalar immediate reward of a transition; and
rules that describe what the agent observes.

The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition. In many works, the agent is also assumed to observe the current environmental state, in which case we talk about full observability, whereas in the opposing case we talk about partial observability. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).


Web Scraping - a scientific database I am searching a scientific database for abstracts of papers containing the words project management. Here is the link:

For getting abstracts, I need to click on any paper and open a new page. How can I do that for 68 papers? I program in R and bash.

try RSelenium. with phantomjs since the date is requested and filled in by ajax calls. so any static web scraping tools wont work.

I managed to get the list on the first page.

http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-headless.html

sample of what i managed to pull.

remove( mopub, m, run , rx, x , first1)
library(RSelenium)
pjs<- phantom( pjs_cmd="C:/Users/bhavin.patel/Downloads/phantomjs-2.0.0-    windows/bin/phantomjs.exe")
Sys.sleep(5)
remDr <- remoteDriver( browserName = 'PhantomJS')
dsurl <- "http://en.journals.sid.ir/SearchPaper.aspxstr=project%20management"
remDr$open()
remDr$navigate(dsurl)
allt3 <-remDr$findElements('id', 'Table3')
lapply( allt3 , FUN=function(dst){ dst$getElementText(); })

[[1]]
[[1]][[1]]
[1] " 1 :   EFFECTIVE FACTORS ON RURAL PEOPLE’S NON-PARTICIPATION OF     MAHABAD’S DAM CATCHMENT IN WATERSHED MANAGEMENT PROJECTS\nAuthor(s): RASOULIAZAR SOLEIMAN*,FEALY SAEID\nJournal: INTERNATIONAL JOURNAL OF AGRICULTURAL MANAGEMENT AND DEVELOPMENT (IJAMAD)\nNumber: MARCH 2015 , Volume  5 , Number  1 ; Page(s) 19 To 26.\nKeyword(s): NON-PARTICIPATION, CATCHMENT, WATERSHED MANAGEMENT, MAHABAD TOWNSHIP, IRAN\nReference(s):  (0)      Citation(s):  (0) FullText:"


How to use a CNN to predict free space in a 2d environment with obstacles? When given a image(300 x 300) with obstacles(Basic 2D shapes) I want to predict 10 coordinates values in the image that will not collide with the obstacles using a CNN.

For now I am giving the coordinates of those 10 points(as train labels) like following
[[x11,x12, ..., y11,y12..], [x21,x22,... , y21,y22...]]
And I trained the nw but predicted label values are not close to the original one.

IS There a different approach I can take to find the coordinates of those 10 points with a CNN?

Instead of predicting free space, you can detect shapes/obstacles with a detector  or a mask builder and after determine free spaces locations.

In my opinion, this is a better approach to predict where are obstacles instead of predict where there are no obstacles (free space).


Overfitting in Linear Regression I'm just getting started with machine learning and I have trouble understanding how overfitting can happen in a linear regression model.
Considering we use only 2 feature variables to train a model, how can a flat plane possibly be overfitted to a set of data points?
I assume linear regression uses only a line to describe the linear relationship between 2 variables and a flat plane to describe the relationship between 3 variables, I have trouble understanding (or rather imagining) how overfitting in a line or a plane can happen?

In linear regression overfitting occurs when the model is "too complex". This usually happens when there are a large number of parameters compared to the number of observations. Such a model will not generalise well to new data. That is, it will perform well on training data, but poorly on test data.
A simple simulation can show this. Here I use R:
> set.seed(2)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
>
> (m0 <- lm(Y ~ X)) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.2393     1.8568  -0.129    0.909
X             1.0703     0.6780   1.579    0.255

Residual standard error: 1.516 on 2 degrees of freedom
Multiple R-squared:  0.5548,    Adjusted R-squared:  0.3321
F-statistic: 2.492 on 1 and 2 DF,  p-value: 0.2552

Note that we obtain a good estimate of the true value for the coefficient of X. Note the Adjusted R-squared of 0.3321 which is an indication of the model fit.
Now we fit a quadratic model:
> (m1 <- lm(Y ~ X + I(X^2) )) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -4.9893     2.7654  -1.804    0.322
X             5.8202     2.5228   2.307    0.260
I(X^2)       -0.9500     0.4967  -1.913    0.307

Residual standard error: 0.9934 on 1 degrees of freedom
Multiple R-squared:  0.9044,    Adjusted R-squared:  0.7133
F-statistic: 4.731 on 2 and 1 DF,  p-value: 0.3092

Now we have a much higher Adjusted R-squared:  0.7133  which may lead us to think that the model is much better. Indeed if we plot the data and the predicted valus from both models we get :
> fun.linear <- function(x) { coef(m0)[1] + coef(m0)[2] * x  }
> fun.quadratic <- function(x) { coef(m1)[1] + coef(m1)[2] * x  + coef(m1)[3] * x^2}
>
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

So on the face of it, the quadratic model looks much better.
Now, if we simulate new data, but use the same model to plot the predictions, we get
> set.seed(6)
> N <- 4
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

Clearly the quadratic model is not doing well, whereas the linear model is still reasonable. However, if we simulate more data with an extended range, using the original seed, so that the initial data points are the same as in the first simulation we find:
> set.seed(2)
> N <- 10
> X <- 1:N
> Y <- X + rnorm(N, 0, 1)
> ggplot(data.frame(X,Y), aes(y = Y, x = X)) + geom_point()  + stat_function(fun = fun.linear) + stat_function(fun = fun.quadratic)

Clearly the linear model still performs well, but the quadratic model is hopeless outside the orriginal range. This is because when we fitted the models, we had too many parameters (3) compared to the number of observations (4).

Edit: To address the query in the comments to this answer, about a model that does not include higher order terms.
The situation is the same: If the number of parameters approaches the number of observations, the model will be overfitted. With no higher order terms, this will occur when the number of variables / features in the model approaches the number of observations.
Again we can demonstrate this easily with a simulation:
Here we simulate random data data from a normal distribution, such that we have 7 observations and 5 variables / features:
> set.seed(1)
> n.var <- 5
> n.obs <- 7
>
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
>
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.6607     0.2337  -2.827    0.216
V1            0.6999     0.1562   4.481    0.140
V2           -0.4751     0.3068  -1.549    0.365
V3            1.2683     0.3423   3.705    0.168
V4            0.3070     0.2823   1.087    0.473
V5            1.2154     0.3687   3.297    0.187

Residual standard error: 0.2227 on 1 degrees of freedom
Multiple R-squared:  0.9771,    Adjusted R-squared:  0.8627

We obtain an adjusted  R-squared of 0.86 which indicates excellent model fit. On purely random data. The model is severely overfitted. By comparison if we double the number of obervations to 14:
> set.seed(1)
> n.var <- 5
> n.obs <- 14
> dt <- as.data.frame(matrix(rnorm(n.var * n.obs), ncol = n.var))
> dt$Y <- rnorm(nrow(dt))
> lm(Y ~ . , dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.10391    0.23512  -0.442   0.6702
V1          -0.62357    0.32421  -1.923   0.0906 .
V2           0.39835    0.27693   1.438   0.1883
V3          -0.02789    0.31347  -0.089   0.9313
V4          -0.30869    0.30628  -1.008   0.3430
V5          -0.38959    0.20767  -1.876   0.0975 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7376 on 8 degrees of freedom
Multiple R-squared:  0.4074,    Adjusted R-squared:  0.03707
F-statistic:   1.1 on 5 and 8 DF,  p-value: 0.4296

..adjusted R squared drops to just 0.037


file name pattern matching and identifying I have a automation system which manages files sent by multiple client inside S3. It so happens client sends file on a regular basis. Let's not consider the content of the file for time being as it can be something of new data or append on old data.

Typically my files are arranged on a pattern:

i) client_a/data_type1/20180610/file_name1_{some_random_text}.csv
ii) client_a/data_type1/20180611/file_name1_{some_random_text}.csv
iii) client_a/data_type1/20180610/file_name2_{some_random_text}.csv
iv) client_a/data_type1/20180611/file_name2_{some_random_text}.csv
v) client_b/data_type2/20180610/file_name3_{some_random_text}.csv
vi) client_b/data_type2/20180611/file_name3_{some_random_text}.csv
vii) client_b/data_type1/20180610/file_name4_{some_random_text}.csv
viii) client_b/data_type1/20180611/file_name4_{some_random_text}.csv

This is a sample of the file management. Now I want to be able to be able to identify unique files. For example case i and ii are same files received on different date. But may be separated by some_random_text (usually date or something).
The primary assumptions here are :

File name length varies as per client and datatype
Random text maybe at the first or end of file name for different files but   for not same file. i.e file_name1 will always have random text after the name
File name extension will vary on different files.

So I want to build a system which takes realtime input of the files received and do an analysis of the name and path to determine if newer version of he same file is received and place it accordingly (same in the sense data may be added to the file).

I have very limited understanding of machine learning algorithms as of whole and not sure which set of algorithms this requires. I am thinking of looking at clustering algorithms. I just want to sure i start in the right direction. Suggestions will be highly appreciated.

This does not sound like a machine learning problem. It seems to me that you could come up with a list of rules that determine whether two files belong together.

As S van Balen said, you should take a look at the glob and re packages in Python. (If you are on Linux, a Bash script would probably be much more elegant than a solution in Python).

The following links might be helpful:

Walk through a directory and its subdirectories to get a list of all files: https://stackoverflow.com/a/954522/4743630.
Group strings in a list based on partial matches (for instance: turn ['file_name1_xyz', 'file_name2_xyz', 'file_name1_abc', 'file_name2_def'] into [ ['file_name1_xyz', 'file_name1_abc'], ['file_name2_xyz', 'file_name2_def'] ]): https://stackoverflow.com/questions/40167651/python-iterate-through-a-list-of-strings-and-group-partial-matching-strings.

You mention that files may have different file types. Be careful when converting between types: Make sure for instance that separators and quote characters are treated correctly.


online detection of plateaus in time series I need to detect plateaus in time series data online. The data I am working with represents the magnitude of acceleration of a tri-axis accelerometer. I want to find a reference time window that I can use for calibration purposes. Because of that, the system must not move and hence only gravity should influence the system.

How can I find such plateaus or is there even a more principled approach that I can take?

Why don't you use a Shewhart Control Chart (more modernly referred to as 6-sigma) with a moving variable length window to understand the process that you have.

As you find your process variability decreases markedly, by having too many results within $±1σ$ then you have a "process change" and that might be defined as a plateau. You will need to decide how many observations you require to establish a plateau.

The as you find your variability increases and your $±3σ$, you know you again have a process change and need to recalculate your limits.


Correctness of a ROC Curve I've built a Decision Tree Classifier to practice with the sklearn library. My first task was to shuffle the iris dataset and split it keeping only the last 10 elements for the test. Then, after the training part I predicted the class of these elements and printed other useful metrics to understand what I'm doing. My doubt comes after building the roc curve, since I find it is really different from the other I'v seen in some example. This is the code:
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.metrics import f1_score
from sklearn.metrics import roc_curve, auc
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
from sklearn import tree
import numpy as np
import graphviz

iris = load_iris()

clf_ex1 = tree.DecisionTreeClassifier(criterion="entropy",random_state=300,min_samples_leaf=5,
                                      class_weight={0:1,1:10,2:10})

np.random.seed(0)

indices = np.random.permutation(len(iris.data))
indices_training=indices[:-10]
indices_test=indices[-10:]

iris_X_train = iris.data[indices_training]
iris_y_train = iris.target[indices_training]
iris_X_test  = iris.data[indices_test]
iris_y_test  = iris.target[indices_test]

clf_ex1 = clf_ex1.fit(iris_X_train, iris_y_train)

predicted_y_test = clf_ex1.predict(iris_X_test)

print("Predictions:")
print(predicted_y_test)
print("True classes:")
print(iris_y_test)

# print some metrics results
acc_score = accuracy_score(iris_y_test, predicted_y_test)
print("--------")
print("Accuracy score: "+ str(acc_score))
print("--------")
f1=f1_score(iris_y_test, predicted_y_test, average='macro')
print("F1 score: "+str(f1))
print("--------")

scores = cross_val_score(clf_ex1, iris.data, iris.target, cv=5)
print("Cross validation scores: "+str(scores))

# Confusion Matrix
print("--------")
print("Confusion matrix:")
print(confusion_matrix(iris_y_test, predicted_y_test))

# Building the ROC Curve
y_test_prob = clf_ex1.predict_proba(iris_X_test)

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

# Building the Plot for each class
plt.figure()
lw = 2
plt.plot(fpr_cl0, tpr_cl0, color='darkorange',
         lw=lw, label='ROC curve class 0 (area = %0.2f)' % roc_auc_cl0)
plt.plot(fpr_cl1, tpr_cl1, color='cornflowerblue',
         lw=lw, label='ROC curve class 1 (area = %0.2f)' % roc_auc_cl1)
plt.plot(fpr_cl2, tpr_cl2, color='aqua',
         lw=lw, label='ROC curve class 2 (area = %0.2f)' % roc_auc_cl2)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

And these are the results:

Are they consistent with the predictions?
If I change the weights of the classes in my DecisionTreeClassifier, but I get the same predictions, is it normal that the final plot do not changes?

The first issue seems to be in the following block of code:
# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

You are feeding y_test_prob[:,1] to each roc_auc calculator (when you should be feeding column 0, 1 and 2). In this case the predict_proba method outputs one column per class. Additionally you are feeding iris_y_test into this calculator which is numerically encoded with distinct values 0, 1 and 2. This function wants one column with the probabilities of each class on it's own. See a possible alteration:
import pandas as pd

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 0], y_test_prob[:, 0])
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 1], y_test_prob[:, 1])
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 2], y_test_prob[:, 2])
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

Secondly - looking at your output probabilities they are all either distinctly 0 or 1. Your decision tree seems to have "perfectly" classified each example into a class with 100% certainty.
The ROC AUC curve will only show interesting results if there are multiple possible thresholds to classify your samples into different classes. In your case - any threshold you pick between 0 and 1 exclusively will yield the exact same False Positive and True Positive rate. The ROC AUC curve helps to determine thresholds for classification - and in your case selecting thresholds is quite trivial.


How to define user churn I'd like to define our app user churn,  Generally, people will define their app user churn by some simply way, like: if the user do not logon continually 7 days or 1 month, we will define them churn user. So is there a better way?

The churn rate is users the ratio of active users that stop being active after a given period.

The churn rate for application user is exactly the one used for other cases, like employees.
However, when dealing with employees it is clear whether the employee is active,
The problem with application users is the identification that they are no longer active.

What you can do is to identify a period of in activity after which, the vast majority of these user never return.

For example, take the users that were active during January of the previous year and were not active for 7 days since then. If 99% of them were not active for the rest of the year than 7 days are a good indication of being in active.

The suitable period is usually determined by the domain (e.g., some applications are used in periods longer than 7 days) and the required confidence level (99%, 90%).
Note that you don't want to be too safe because if you'll decide that a user is inactive only after 2 years, this definition will prevent you from identifying users that are practicality lost and take actions to reduce the churn rate.


How to identify new clusters that the training data has never seen I have to identify the different operational states of a server. I have readings related to the different sensors of the server ( like temp sensor,fan speed sensor,job load sensor etc).The data I have information about some operational states (normal, high temp , high temp and high fan speed) etc. What ML algorithms should I use to identify if any other operational state,state which the training data has not seen, comes up?

I have used several clustering algorithms. I expected Gaussian Mixture models to work well, but they fail to indicate a new operational state of it comes up.

I used LSTM and looked at the residuals, but had to look at the residuals of each parameter to identify diff states.

A possible way to do this is to learn some compressed latent representation of your training data, and check how well new data matches this representation.

In practice, you could try training an autoencoder on your training data, by minimizing the reconstruction error between its input (your raw features) and its output (the reconstruction).

After training, you can check whether a new operational state comes up by checking if the reconstruction error is larger than a threshold. The intuition behind this is that if the new data belongs to a known or normal operational state that the model learnt from the training data, the model will be able to reconstruct it properly and the error will be low.


Feature selection inside Random Forest I understood random forest is building a model with multiple decision trees,

Row sampling is based on bootstraping

My question is how feature selection is happening for each tree ?
Any help would be appreciated.

A random subset of features than using the best split logic as done in normal Tree

The Random Forest algorithm introduces extra randomness when growing trees; instead of searching for the very best feature when splitting a node, it searches for the best feature among a random subset of features. The algorithm results in greater tree diversity, which (again) trades a higher bias for a lower variance, generally yielding an overall better model.

$\hspace{3cm}$Image credit - SE( https://stats.stackexchange.com/a/438384/256691) - Ashish Anand
Extra-trees -
More randomness by getting the feature using a random Threshold instead of searching the best split.

When you are growing a tree in a Random Forest, at each node only a random subset of the features is considered for splitting. It is possible to make trees even more random by also using random thresholds for each feature rather than searching for the best possible thresholds (like regular Decision Trees do)

I used Extra-tree to explain the point to have a contrast. It helps in catching the point.
Please also read the SE link of the image
[Blockquote ref] - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition


Is there a limit in the number of layers for neural network? I heard the neural network has a problem with vanishing gradient problems even though the ReLU activation function is used.

In ResNet(that has a connecting function for reducing the problem), there is limit of maximum layers of 120~190(I heard).

For the complete AI performance(or general AI with strong intelligence) I believe that the limit of the number of layers must be solved.

Is there is any possibility that we find a new activation function that does not limit the number of layers? (maybe we could use exhaustive search... checking the train performance in a neural network of 200~500 layers)

The maximum number of layers that a network can have depends on many factors, such as the model architecture, activation function, optimization method, and others. For example, as you alluded to, ResNets is huge in increasing the number of layers possible, even up to 1000 layers for CIFAR-10. Another example is that when ReLU functions became more popular, Kaiming initialization allowed deeper networks to be trained by keeping the variance of each layer's activations at around 1, as the other popular method of Xavier initialization is more suitable for sigmoid activation functions.

So in general, there's no known limit except to keep experimenting and coming up with new techniques, including but not limited to finding better activation functions.


Are my features enough? I am trying to fit a regression model on a non linear data.
The features I have are around 12 and around 800 samples.
With the help of PyCaret, i tried to fit the data on to around 22 model, and then selected the best one (Ada Boost) and then tried further to tune it to get better result.
However, none of the models gave a positive R2 score, Ada Boost was the least worst performing algorithm.
This is the test (red) and predicted test output (green) from the selected algorithm.

After trying all various techniques and still not getting a decent result, can we infer that the features are not enough to account for the variation of the target variable ?
In other words the provided features don't explain best the target variable.
It may sound silly but am a beginner in Data Sciences, so please dont mind.

I don't know much about R2 score, but having it negative all the time seems pretty strange to me. Maybe you should try to use AUC as a metric ( <0.5 classifier is worse than a random classifier, and the closer you're to 1, the best is your algorithm).
If it appears that you still can't find a model giving decent results, the direct conclusion is not that your features are not enough, because there can be plenty of other reasons.
I'd suggest you to try SMOTE, which will create new data based on the ones you already have, and try applying your models again. Sometimes, this is a way to tackle the issue of not having enough data


corpus development for plagiarism detection There are many simple plagiarism detection algorithms that work on search engines like google etc. I want to have a index of corpus of the whole internet to serve as a back-end database for my plagiarism detection software.
What should be the approach to build such database? Are there any opensource or collaborated live repositories?

somewhere i read instead of having local database of the entire internet, one can index and use it for faster search.

I know Elastic Search seems to be usable. Anyone has tried before?

I want to have a local database of corpus of the whole internet

Are you Google? If not storage might be an issue ;)

The PAN series have run various tasks related to plagiarism detection in the past: https://pan.webis.de/tasks.html#task-originality. I think they provide annotated datasets and they used to provide a live search engine.


Can Neural Networks be trained to smooth values / output the average? Let's say we have a neural network with one input neuron and one output neuron. The training data $(x, f(x))$ is generated by a process

$$f(x) = ax + \mathcal{N}(b, c)$$

with $a, b, c \in \mathbb{R}^+$, e.g. something like

feature  | target
-----------------
0             0.0
0             1.0
0             1.5
0            -1.2
0            -0.9
...

I know that neural networks can deal pretty well with labeling errors in classification problems. Meaning if you have a large dataset and a couple of examples have the wrong label, they get basically ignored.

But for this kind of problem I'm not too sure. A first experiment indicates that they do smooth values.

Are there choices in architecture / training which help the smoothing / averaging / removal of noise?

What I tried

I created a network which can solve this kind of regression problem without noise. It gets a MSE of about 0.0005. When I add a bit of noise to the training set only, I get an MSE of 0.001:

#!/usr/bin/env python

# core modules
import random

# 3rd party modules
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import train_test_split
import numpy as np

def main(add_noise=True):
    # Get data
    xs, ys = create_data_points(10000)
    x_train, x_test, y_train, y_test = train_test_split(xs, ys, test_size=0.20)

    # Add noise to training data
    if add_noise:
        noise = np.random.normal(0, 0.1, len(x_train))
        x_train = x_train + noise

    # Create model
    model = create_model()
    model.compile(optimizer='rmsprop',
                  loss='mse',
                  metrics=['mse'])

    # Fit model to data.
    model.fit(x_train, y_train, epochs=10, batch_size=32, verbose=1)

    # Evaluate
    y_pred = model.predict(x_test, batch_size=100).flatten()
    print("MSE on test set:")
    print(((y_pred - y_test)**2).sum() / len(y_test))

def create_data_points(nb_points):
    xs = []
    ys = []
    for i in range(nb_points):
        x = random.random()
        xs.append(x)
        ys.append(2 * x)
    return np.array(xs), np.array(ys)

def create_model(input_dim=1, output_dim=1):
    model = Sequential()
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(200, input_dim=input_dim, activation='relu'))
    model.add(Dense(output_dim, activation='linear'))
    return model

if __name__ == '__main__':
    main()

Outliers

In an earlier version of this question I wrote "outlier" when I meant "label noise". For outliers, there is:

The Effects of Outliers Data on Neural Network Performance

In general simpler models are more robust to noise in the input.  The strength of neural networks is also their biggest 'gotcha' - they are extremely expressive. This means they easily overfit and can be sensitive to noise in inputs.  The strategy of simplifying a model to make it more robust to noise is called regularization.  There are many types:

use smaller hidden layers (i.e. 20 instead of 200 nodes)
use dropout, where during training inputs are randomly set to 0 during training (makes the network more robust to noise overall)
stop training earlier - 'early stopping'
use L1 or L2 regularization, which imposes a cost on the weights

All of these can be done from with Keras.  You want to make your network more robust to noise without decreasing your validation quality.  To do this, I would try the above suggestions in order.  You can measure overfitting by looking at the difference between predictive accuracy on your training data and validation data.  If they are very different - your model has learned structure in your training data that is not in your validation that is NOT what you want.  Try to fiddle regularization nobs until (1) your train-validation AUC or accuracy is very similar, (2) your validation AUC/accuracy is still sufficiently high.


Variable Importance Random Forest on R I am currently using a random forest model for classification, however I am unsure how the feature selection technique "varImp" works on R. I understand the context of variable importance, however when I implement it in R it doesn't seem to produce the results I expect.

When removing the most important variable (of 31 features), the model's accuracy does not decrease. I would expect it to as it should be contributing the most to the model's ability to classify.

Could someone please explain what this function is doing?

What the function is doing for each variable?

Record the Out-Of-Bag (OOB) accuracy for each tree.
"shuffle" or permute the values of that variable.  This means you take all the values of that variable in the data, and assign those values randomly back out to the observations, which is a way of introducing noise and getting rid of the signal that that variable provided.
Now it finds the OOB accuracy again, but this time the values for that variable are incorrect since we permuted it.  By introducing noise where your model expects signal, you should see a decrease in performance.
Compare the original accuracy in (1) to the accuracy in (3) for each variable.  If the model performance decreases a lot for a variable in step (3) compared to (1), then it is deemed to have greater importance.

Why does removing the most important variable not have a negative affect on accuracy? (my guess)

Probably because that important variable is correlated with some other variable(s) you have.  Your model can capture the information contained in that missing important variable by using a few other variables to make up for it.  When you drop the important variable, which other variables see a notable gain?


Regularization - Combine drop out with early stopping I'm building a RNN (recurrent neural network) with LSTM cells. I'm using time series to perform anomaly detection.

When training my RNN I'm using a dropout of 0.5 and I'm early stopping with a patience of 5 epochs when my validation loss is increasing.

Does it make sense to use early stopping in combination with dropout?

Avoid early stopping and stick with dropout. Andrew Ng does not recommend early stopping in one of his courses on orgothonalization [1] and the reason is as follows. For a typical machine learning project, we have the following chain of assumptions for our model:

Fit the training set well on the cost function

↓

Fit the dev set well on the cost function

↓

Fit the test set well on the cost function

↓

Performs well in the real world

And what we want are tools that can target one of these four objectives and not the others in order to keep improving our models more efficiently, and this concept is called "orthogonalization." An analogy would be the different options in photo editing apps such as brightness, contrast, and saturation adjustments, which are independent of, or "orthogonal" to one another. Examples of orthogonal optimization strategies for the four goals are listed below:

Fit the training set well on the cost function (e.g. bigger neural network; Adam optimization)

↓

Fit the dev set well on the cost function (e.g. regularization; bigger training set)

↓

Fit the test set well on the cost function (e.g. bigger dev set)

↓

Performs well in the real world (e.g. change the test set; change the cost function)

Because early stopping both fits the training set less well and improves the dev set performance at the same time, it is not orthogonal and Ng advises us not to use it.

Reference:
[1] Week 1 of Course 3: Structuring Machine Learning Projects of Coursera Deep Learning Specialization


What is the difference between CountVectorizer token counts and TfidfTransformer with use_idf set to False? We can use CountVectorizer to count the number of times a word occurs in a corpus:

# Tokenizing text
from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)

If we convert this to a data frame, we can see what the tokens look like:

For example, the 35,780th word of the 3rd document occurs twice.

We can use TfidfTransformer to count the number of times a word occurs in a corpus (only the term frequency and not the inverse) as follows:

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)

Converting this to a data frame, we get:

We can see the representation is different. The TF is shown as 0.15523. Why is this different than the token count using CountVectorizer?

Actually, the documentation was pretty clear. I'll keep it posted in case someone else searches before reading:
The TfidfTransformer transforms a count matrix to a normalized tf or tf-idf representation. So although both the CountVectorizer and TfidfTransformer (with use_idf=False) produce term frequencies, TfidfTransformer is normalizing the count.


Is stratified sampling necessary (random forest, Python)? I use Python to run a random forest model on my imbalanced dataset (the target variable was a binary class). When splitting the training and testing dataset, I struggled whether to used stratified sampling (like the code shown) or not. So far, I observed in my project that the stratified case would lead to a higher model performance. But I think if I will use my model to predict the new cases which would highly probably differ in the distribution of target class with my current dataset. So I inclined to loosen this constrain and use the unstratified split.
Could anyone can advice to clarify this point?

train,test=train_test_split(myDataset, test_size=0.25, stratify=y)

If the number of values belonging to each class are unbalanced, using stratified sampling is a good thing. You are basically asking the model to take the training and test set such that the class proportion is same as of the whole dataset, which is the right thing to do. If your classes are balanced then a shuffle (no stratification needed here) can basically guarantee a fair test and train split.

Now your model will be capable or at least enough equipped to predict the outnumbered class (class with lesser points in number). That is why instead of just calculating Accuracy, you have been given other metrics like Sensitivity and Specificity. Keep a watch on these, these are the guardians.

Hope this helps.


Historical weather data with machine learning? My company gave me a task to build some weather forecasting. I have now historical weather data for 10 years (temperature, precipitation in mm, humidity and etc. more than 30 features total). We need to know what probability for rainfall in South-Asian countries in particular time period (week, month)?

Now I'm able to calculate approx probability based only on monthly averages (like if for 10 years 1st of Dec was rainy 8 times, so the probability is 80%). Of course, it's not a scientific approach at all, but sometimes we need to know rainfall probability for 3 or more months period in advance.

I was thinking to try building a neural network model, but don't understand how it might work in production? What data to give to this model after training it? Does anybody have any ideas?

For time-series modelling, one would typically look to LSTMs (or their more recent improvements). Such models are known for their long-time-memory, which is necessary here because of long term seasonality and similar effects.

As for production, there are several methods to serve neural nets, for example Tensorflow Serve.

Specifically about training once in production - you could choose to simply not do it. If your dataset has 10 years of data, and your model will be in production for more than a year, you could think about retraining it (with all the newly collected data + old) periodically, every year or so. It could be completely automated too.

There are other methods (look for "online training and inference" approaches, which are traditionally in the RL domain). But based upon your description, the extra time you would spend to build such capability doesn't seem to be worth the gain.


Word representation that gives more weight to terms frequent in corpus? The tf-idf discounts the words that appear in a lot of documents in the corpus. I am constructing an anomaly detection text classification algorithm that is trained only on valid documents. Later I use One-class SVM to detect outliers. Interesting enough the tf-idf performs worse than a simple count-vectorizer. First I was confused, but later it made sense to me, as tf-idf discounts attributes that are most indicative of a valid document. Therefore I was thinking of a new approach that would weight words that always appear in documents more, or rather assign a negative weight for the absence of such words. I have preset dictionary of words, so there is no worry that irrelevant words such as(is, that) will be weighted.
Do you have any ideas about such representations? The only thing I could imagine would be subtracting the document frequency from the attributes that are zero in a certain document.

I'm not aware of any standard representation which increases the importance of document-frequent words, but IDF can simply be reverted: instead of the usual
$$idf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \in d|}\right)$$
you could use the following:
$$revidf(w,D)=\log\left(\frac{N}{|d\in D\ |\ w \notin d|}\right)$$
However for the task you describe I would be tempted to try some more advanced feature engineering, typically by using features which represent how close the distribution of words in the current document is from the average distribution.


Add noise to an example set I'm trying to study the effect of curse of dimensionality in classification algorithms.

I have a simple dataset with only 1 feature. So i can represent the set in a 1 dimension.
If now i want to introduce some noise in this dataset, is it correct to add another feature with random values to my dataset ?

In addition to what Lupacante conceptually and nicely showed such that the added feature(s) has(have) to be informative for the model otherwise it can get ignored by majority of models (perhaps easily by regularized models), I would like add that you also increase the dimensionally of the feature space synthetically using many simple mathematical expressions as well. Concertedly, let's say your only feature (column) is:

x

You may easily construct other features like (it is common practice actually in physical sciences):

x$^2$, x$^3$, x$^{0.5}$, $sin(x)$, x$^2$sin(x),...

Till you hit the so-called the curse of dimensionality for your exercise. Although I am not yet sure what features/num_samples ratio exactly causes the curse of dimensionality! What I have gathered so far about the curse of dimensionality has been very subjective.


How to avoid overfitting in random forest? I want to avoid overfitting in random forest. In this regard, I intend to use mtry, nodesize, and maxnodes etc. Could you please help me choose values for these parameters? I am using R.
Also, if possible, please tell me how I can use k-fold cross validation for random forest (in R).

Relative to other models, Random Forests are less likely to overfit but it is still something that you want to make an explicit effort to avoid. Tuning model parameters is definitely one element of avoiding overfitting but it isn't the only one. In fact I would say that your training features are more likely to lead to overfitting than model parameters, especially with a Random Forests. So I think the key is really having a reliable method to evaluate your model to check for overfitting more than anything else, which brings us to your second question.

As alluded to above, running cross validation will allow to you avoid overfitting. Choosing your best model based on CV results will lead to a model that hasn't overfit, which isn't necessarily the case for something like out of the bag error. The easiest way to run CV in R is with the caret package. A simple example is below:

> library(caret)
>
> data(iris)
>
> tr <- trainControl(method = "cv", number = 5)
>
> train(Species ~ .,data=iris,method="rf",trControl= tr)
Random Forest

150 samples
  4 predictor
  3 classes: 'setosa', 'versicolor', 'virginica'

No pre-processing
Resampling: Cross-Validated (5 fold)

Summary of sample sizes: 120, 120, 120, 120, 120

Resampling results across tuning parameters:

  mtry  Accuracy  Kappa  Accuracy SD  Kappa SD
  2     0.96      0.94   0.04346135   0.06519202
  3     0.96      0.94   0.04346135   0.06519202
  4     0.96      0.94   0.04346135   0.06519202

Accuracy was used to select the optimal model using  the largest value.
The final value used for the model was mtry = 2.


R - Error in KNN - Test and training differ I am trying to classify set of text files whether it belong to category A or category B using KNN algorithm

Here is the error that i get

> knn.pred <- knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand)

  Error in knn(tdm.stack.nl, tdm.stack.nl_test, tdm.cand) :      dims of
  'test' and 'train' differ

 > dim(tdm.stack.nl)
    [1]  184 1599
    > dim(tdm.stack.nl_test)
    [1]   1 992

As obvious the number of words used by training dataset is different from train. How can I avoid this?

I want to mention that this error i get when i divide the training and test dataset manually to (70:30) ratio. However i dont get this error if i divide the training and test dataset using samlpling method.

> # create hold-out
> set.seed(500)
> train.idx <- sample(nrow(tdm.stack), ceiling(nrow(tdm.stack) * 0.7))
> test.idx <- (1:nrow(tdm.stack))[-train.idx]
>
> # create model - knn clustering
> tdm.cand <- tdm.stack[, "targetcandidate"]
> tdm.stack.nl <- tdm.stack[, !colnames(tdm.stack) %in% "targetcandidate"]
>
> # set up model
> knn.pred <- knn(tdm.stack.nl[train.idx,], tdm.stack.nl[test.idx,],    tdm.cand[train.idx])
> dim(tdm.stack.nl[train.idx,])
[1]  129 1599
> dim(tdm.stack.nl[test.idx,])
[1]   55 1599

However this method will not work for me as I want to classify a document in a real time using the model that was built before.

Please help. Kindly let me know if you need any other info.
Arun

You are avoiding the error with sampling purely by chance (the words are common). Simplest way would be to merge training and test, construct a tdm on a joined set, separate into training and test again and then purge some columns (the constant ones in the train set, as they correspond to words occurring in test only => useless for training).


Deriving the update rule for mean in k-means clustering I have a homework question that ask me to derive the update rule from the distortion metric:

It then says: "The update rule is derived by computing the gradient for each element of the k-th mean and solving for the value where the gradient is zero." Can somebody tell me what this question means? I already understand how k-means clustering works, but I need some clarification about what it means by "k-th mean" and "gradient".

The k-th mean just means the mean of the k-th cluster. We want to minimize the sum of squares of each of the clusters, which we do by minimizing this so-called distortion metric. The gradient is a multivariate generalization of derivatives and the gradient of this distortion metric is the zero-vector when this is optimized, and after this you update your clusters.


Merge two or more ML models with different size of output togetger I have several models that classify the input (word embedding) into several classes. My problem is that I need to train these models separately and need to merge the output of these models together to get a label.
For simplicity assume only two models:
Model 1: predicts A, B or C
Model 2: predicts D or E

Then, I need to classify the input X so that I get the joint probability over A, B, C, D and E.
I have tried this code in Keras/Python, in which I define neural networks and remove the softmax activation function after training:
inputs = tf.keras.Input(shape=dim_in)
x = layers.Dense(dim_out)(inputs)
outputs = layers.Activation(activation="softmax")(x)

model = tf.keras.Model(inputs=inputs, outputs=outputs, name='keras_model')
model_without_softmax = tf.keras.Model(inputs=model.inputs[0], outputs=model.layers[-2].output, name='keras_model', trainable=False)

model.compile(optimizer=tf.keras.optimizers.Nadam(),
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['sparse_categorical_accuracy'])

Then I get final prediction with softmax over joint arrays of predictions:
prediction = softmax(np.append(prediction_1, prediction_2))

It wasn't working because the output values from the neural networks were on a different scale when classifying into 2 or 3 classes. Thus, I tried dividing into a number of classes but without success:
prediction = softmax(np.append(prediction_1/len(idx2label_d1), prediction_2/len(idx2label_d2)))

Can you point me to right direction for the general formula of merging the neural network's outputs together?

Maybe you can use the Concatenate layer
outputs = tf.keras.Concatenate()([model1, model2])
full_model = tf.keras.Model(inputs=inputs, outputs=outputs, name='full_model')

This will simply concatenate the two softmax output into one.