How can I find if it is an overfitting problem? I am new in Machine learning, and I want to detect emotions from the face.
Preprocessing: I used equalizeHist to equalizes the histogram of grayscale images (JAFFE database with 213 images), in the goal to normalizes the brightness and increases the contrast of the image.
Feature extraction: I extract features with Gabors filters from images, and I get a matrix of 213x120. I split data: 60% train data and 40% test data, and I normalize it.
Train and test: for training the model, I use SVM classifier with an RBF kernel. With grid-search, I select the best couple C, gamma (using 10-fold cross-validation). Then, I test the performance of the model on the test data (unseen data), and I get 89% accuracy.
The problem is when I want to predict emotion from a new input face, I get a false result.. Is it an overfitting problem?
UPDATE 1: feature extraction
gf = GaborFilter(ksize=(11, 11), freq_nbr=5, or_nbr=8, lambd=4, sigma=8, gamma=5, psi=5*np.pi/6) #Gabor filter with *cv2.getGaborKernel*

data = []
for j in range(len(roi1_images)): # len(roi1_images) = 213
  vect1 = feature_extraction_roi_avr(roi1_images[j], gf.kernels) # feature vect from left eye
  vect2 = feature_extraction_roi_avr(roi2_images[j], gf.kernels) # feature vect from right eye (40 features)
  vect3 = feature_extraction_roi_avr(roi3_images[j], gf.kernels) # feature vect from mouth (40 features)
  vect = np.concatenate((vect1, vect2, vect3), axis=None) # feature vect of one face (40 features)
  data.append(vect)
data = np.array(data) # Data matrix (213, 120)

UPDATE 2: Learn and test model

clf = GridSearchCV(estimator=SVC(kernel='rbf'), param_grid=svm_parameters, cv=10, n_jobs=-1) # grid search with 10fold cross validation
scaler = StandardScaler()

# Split data
X_train_, X_test_, y_train, y_test = train_test_split(data, data_labels, random_state=0, test_size=0.4, stratify=data_labels)

X_train = scaler.fit_transform(X_train) # Normalize train data
clf.fit(X_train, y_train) # fit SVM model
X_test = scaler.transform(X_test) # Normalize test data
score = clf.score(X_test, y_test) # calculate mean accuracy
print(score) # score accuracy = 0.8953488372093024

UPDATE 3: predict new input
    landmarks, gs_image = detect_landmarks(path_image)
    roi1_image = extract_roi(gs_image, landmarks, 1) #extract region 1 (eye 1)
    roi2_image = extract_roi(gs_image, landmarks, 2) #extract region 2 (eye 2)
    roi3_image = extract_roi(gs_image, landmarks, 3) #extract region 3 (eye 3)
    vect1 = feature_extraction_roi_avr(roi1_image, gf.kernels)
    vect2 = feature_extraction_roi_avr(roi2_image, gf.kernels)
    vect3 = feature_extraction_roi_avr(roi3_image, gf.kernels)
    vect = np.concatenate((vect1, vect2, vect3), axis=0)
    vect = np.reshape(vect, (1,-1))
    vect = scaler.transform(vect)
    class_ = clf.predict(vect)[0]
    print(class_,end=" ")
```

These are some ways by which you can improve your model's accuracy and reduce overfitting.

Split your dataset into training, validation, and testing.
Add dropout.


optimal combination of hyper parameters and model selection This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc.

When we tune the hyper parameters of a deep learning model every possible combination of hyper parameters results in a different model. And we select an optimal combination based on the loss curves. What exactly is an optimal combination of hyper parameters?

My question is exactly this: There is an infinite number of combinations of hyperparams possible. We know that there are many possible configurations of hyperparams possible that give similar generalization error. What should the model selection decision be based upon?
And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?

"What exactly is an optimal combination of hyper parameters?" The combination of hyperparameters that produces the lowest possible error on unseen data for that model architecture.

"What should the model selection decision be based upon?" The best way to estimate error on unseen data that I know of is k-fold cross-validation. I examine the mean and standard deviation of the error across the k-folds; in most cases, I select the model with the smallest standard deviation among the models with the best mean error.

"And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?" As far as I know, one can never know what combination of hyperparameters will produce the lowest possible error on unseen data. In my experience, a good search strategy will get you close enough to the optimal combination so that further search is not worth the effort.


How much imbalance in a training set is a problem? In a simple binary classification problem, at what point does majority class to minority class become significant become significant? Intuitively, I would expect a 3:1 ratio to not be an issue, maybe not even a 10:1 ratio. But a 100:1 ratio certainly does require some action. What might be this cutoff?
As a followup, what might be come potential solutions beyond undersampling and oversampling?

There is no strict threshold at which a dataset is considered imbalanced. Accordingly, in Foundations of Imbalanced Learning Gary M. Weiss writes:

There is no agreement, or standard, concerning the exact degree of
class imbalance required for a data set to be considered truly "imbalanced."
But most practitioners would certainly agree that a data set where the most
common class is less than twice as common as the rarest class would only be
marginally unbalanced, that data sets with the imbalance ratio about 10:1
would be modestly imbalanced, and data sets with imbalance ratios above
1000:1 would be extremely unbalanced. But ultimately what we care about
is how the imbalance impacts learning, and, in particular, the ability to learn
the rare classes.

A pragmatic approach could be to fit your models on the imbalanced dataset and check if the imbalance leads to large differences in performance between the classes. But keep in mind that the ultimate goal is to minimize misclassification cost. So performance difference between classes are not necessarily a problem if classes have equal misclassification cost (usually for imbalanced datasets the underlying assumption is that misclassification cost are not equally distributed, i.e. misclassification cost for minority classes are assumed to be higher).
To handle imbalanced datasets sampling-based approaches are most common but they are not limited to under- and over-sampling. There are also hybrid (e.g. SMOTE+Tomek) and Ensemble-based methods (e.g. BalancedRandomForest).
Moreover, there are algorithmic approaches which include cost-sensitive learning (e.g. Weighted Random Forest) and skew-insensitive learning (e.g. Naïve Bayes).
Finally, you can adapt the performance metric to be used. AUROC and ROC Curves are commonly used for imbalanced datasets. However, they can be biased too. Which is why some authors suggest to use Precision-Recall-Curves and AUC-PR instead. Moreover, Precision, Recall and F1 score are common as well.
Note however, that simply using a different performance metric for model selection and model evaluation will not make your models optimize for these internally when being trained. Which is why the other approaches and especially sampling are so frequently used.


How to manage missing data in meteorological time series? How to know the type of missing data is what it is: MCAR, MAR or NMAR, knowing that I'm working on time series multivariate, and is that going to help me deal with the missing data, and what is the best techniques of processing missing data in time series, knowing that I work on meteorological data?

It is a question related to the domain of your project. You should know the cause of the missingness.

If some values are missing, because there is no applicable measure then this values are a special case, therefore they're missing not at random. In such case you can impute missing values with the mean value of the non-missing data and add another feature, which indicates special cases.

On the other hand, if the value is missing, because some sensor temporarily stopped working, then it is missing at random, so the measured value is just not known. In this situation you can perform linear regression (or any other regression, but you should start with the linear one) between non-missing data.

In case your problem has mixed types of missingness you should also perform some regression.


What does coefficient mean for a binary independent variable in multiple linear regression? Let's say I have a multiple linear regression model where my dependent variable, Y, is an integer. And, one of my independent variables --x1-- is binary --let's say either 0 or 1.

We know that sign of the coefficient for x1 in the model, positive or negative, demonstrates its correlation with Y. My question though is, is there anyway for us to know that what the correlation of x1 values independently, 0 or 1, is with the dependent variable? For instance, do we know if that 0 is the reason for the sign of coefficient and the more 0 will lead to more Y (not sure if this is the correct language but hope you understand what I am saying.)

A real-world example, I have a multiple linear regression model to find the correlation of a set of numeric indexes in tweets, as independent variables, with the number of retweets as dependent variable. One of the independent variables in this model is the tweet's fact checking label, "true" or "false" which we call it "truth_label." I have trained a multiple linear regression model and the coefficient of the truth_label is positive meaning that it has a positive correlation with the number of shares. But, I would like to know if that positive correlation is because of the "true" values or "false" values. Hope this example made my question a bit more clear.

Your question is a little bit confusing but I will try to answer it:

You are asking if the coefficient sign equals the correlation sign? You want to know if a positive sign in the regression leads to a positive correlation between the $Y$ and the binary $X$s.

Depends.

This is true if you are running a regression with only one THAT regressor.

The equation $Retweets=\beta_0+\beta_1*truth.label$ complies with this.

The model $Retweets=\beta_0+\beta_1*truth.label.1+\beta_2*truth.label.2$ does not necesarily complies with the statement.

Why? When you have two or more variables in a model, what they do to each other is to complement between themselves, i.e. one finds what the other variable misses (in terms of errors). So you can't state anything about coefficients and correlations.

This is true, does not matter the type of variable: Binary, Integer, Continuos.


Decovolution function I have an image (for example (7x7x3) and a filter (3x3x3)). I convolved the image with the filter and it became a (3x3) output. If I want to do the inverse operation and want it to become the image from the output and the filter. How can I implement this operation in Python with Numpy?

I don't know which operation I should use with the filter (inverse or transpose)?

Here is my code for the Deconvolution:

import numpy as np

def deConv(Z, cashe):

'''

deConv calculate the transpoe Convoultion between the output of the ConvNet and the filter

Arguments:
    Z-- Output of the ConvNet Layer, an array of the shape()
'''
# Retrieve information from "cache"
(X_prev, W, b, s, p) = cashe

# Retrieve dimensions from X_prev's shape
(m, n_H_prev, n_W_prev, n_C_prev) = X_prev.shape

# Retrieve dimensions from W's shape
(f, f, n_C_prev, n_C) = W.shape

# Retrieve dimensions from Z's shape
(m, n_H, n_W, n_C) = Z.shape

#create initial array for the output of the Deconvolution
X_curr = np.zeros((m, n_H_prev, n_W_prev, n_C_prev))

#loop over the Training examples
for i in range (m):

    #loop over the vertical of the output
    for h in range(n_H):

        #loop over the horizontal of the output
        for w in range(n_W):

            #loop over the
            for c in range (n_C):

                #loop over the color channels
                for x in range(n_C_prev):

                    #inverse_W = np.linalg.pinv(W[:, :, x, c])
                    transpose_W = np.transpose(W[:,:,x,c])
                    #X_curr[i, h*s:h*s+f, w*s:w*s+f, x] +=  Z[i, h, w, c] * inverse_W
                    X_curr[i, h*s:h*s+f, w*s:w*s+f, x] +=  Z[i, h, w, c] * transpose_W
                X_curr[i, h*s:h*s+f, w*s:w*s+f, :] += b[:,:,:,c]

X_curr = relu(X_curr)

return X_curr

This is probably irreversible operation unless the pre-convolution data was not full rank. But note that you reduced the dimensions of your signal, so the convolution was probably cropped to "valid" information (which doesn't need padding)

If you have at least the image or the filter, recovery might be possible, since convolution can be inverted using deconvolution but note that:

Convolution is defined as
$$ f(x) \circledast g(x) =  h(x)  = \int_{-\infty}^{\infty}f(x-t)g(t)dt $$
The some integral transforms (such as Fourier and Laplace) have the property that:

$$ T\{f(x) \circledast g(x)\}(s) = T\{h(x)\}(s) = T\{f(x)\}(s) \times T\{g(x)\}(s) $$

This is true for the Discrete Fourier Transform and Discrete Convolution as well, so to find your image $i$ from the filtered image $j$ by a filter $h$ given $I$,$J$ and $H$ as the Discrete Fourier transform of $i$,$j$ and $h$, respectively. Let $F\{.\}$ denote the discrete fourier transform and $F^{-1}\{.\}$ denote its inverse transformation:

$$ i = F^{-1}\{I\} = F^{-1}\{\frac{J}{H}\} $$

The discrete fourier transform is implemented efficiently by SciPy(signal module), FFTW, NumPy(fft module) and probably Theano. Having a lot of wrappers arround.

Note 1: Is important to notice that you will need at least an estimation of your filter, there are a lot of algorithms to do so.

Note 2: Deconvolution is very sensitive to noise, you can check on this class on Digital Image Processing to understand image filtering, mainly the part on Wiener filters.

Note 3: Image Deconvolution is implemented on scikit-image (e.g. Unsupervised Wiener) and on OpenCV using many algorithms (also on matlab in Image Processing Toolbox).


How to randomly split the data set into multiple different sets:(training 70%:validation 10%(optional):testing 20%) in R? I have a dataset with 4 predictor variables X1, X2, X3, X4, and one response variable Y. I have been asked to check the correlation between these variables and see how they are related and then use a linear model to fit them.

No split of training set: test set is given. I have one data set with 10000 samples. I was planning of splitting this data set in the ratio 80:20 for training and testing respectively.

I'd like to know how to do the same in R programming.

Also in general, we will split it into multiple combinations of training:testing set right? or? Please correct me if am wrong. I am a newbie to ML, so kindly help me out here.

Usually you choose the ratio between training and test data by how many samples you have, for example if you do not have that many you would go to something like 90:10, but with a larger one you can even co sider something like 75:25, although, 80:20 sounds good for your case.

For the random splitting that you are talking about, you should search and learn a lit k-fold cross-validation. It's a method with which you split your data in training and test sets k times in order to validate your model on different training and test sets each time.

About the R implementation i cannot really help you, but im pretty sure it should be quite easy to find a guide on how to do that.

Good luck.


How to deal with missing continuous data So I have a dataset where I have a continuous variable for only about 10% of the entries. How would you incorporate this in a model. Imputing does not make much sense to me, because there are so few values, however for those entries where I do have it, it is quite an important feature.

Where ever you don't have an entry just make that to a null value or a random unique value. an then create another column, make that column data equal to 1 if data is present 0 otherwise so, the model may learn about that unique imputation and data using another column we are adding.


Closed form of Weighted Ordinary Least Squares calculation of the trend line I would like to know if there is a closed form version of this equation:

$\beta = \frac{n\sum{xy}-\sum{x}\sum{y}}{n\sum{x^2}-(\sum{x})^2}$

But for weighted data, where the weight $w_i$ is the value of the importance I want the individual $i$ in my database to have.

We found the answer:

The formula is really close to the original OLS closed formulation:

$\beta = \frac{n\sum{(wx)(wy)}-\sum{xw}\sum{wy}}{n\sum{(wx)^2}-(\sum{wx})^2}$

There is something that should be warned:
When the sample is small, the method will simply don't work at all, resulting in absurd estimations of the $\beta$.

Make sure your sample is big enough!


GRU/LSTM models - Train/Test split I drove myself into a corner with this, can someone please explain?

I feel I'm missing something obvious...

If, for LSTM, each layer is trained with inputs from t and t-1, than that'd mean that if I've got a training set of a 10 000 observations, the network is trained to get 10 000 observations and produce a result as a function of all of them. If I use it on a test set of say 1 000 observations, why would it work?

Or if I want to make a prediction, from a single observation, whz would that work at all?

Should, in the case of LSTMs, the train test (in the toy example above) be 10 000 observations long (i.e. 9 000 old 'train' observations and 1 000 new 'test' ones)?

LSTM is often used for Sequence Prediction problems, for example, when the dataset is a time series.

In that kind of dataset, you don't split the dataset the normal (a.k.a random) way to avoid look-ahead bias. We should train with "old observations" and test with "new observations" as you said.

  Should, in the case of LSTMs, the train test (in the toy example above) be 10 000 observations long (i.e. 9 000 old 'train' observations and 1 000 new 'test' ones)?

This sentence is a little bit unclear to me. If you have a dataset with 10 000 observations, you can train with 9000 old observations and test with 1000 new observations. The training set shouldn't contain the test data in any case.

Here's an example of how to split a time series dataset:
We have data from 01-01-2000 to 31-12-2004, we can choose 01-01-2004 as a split day:

Training set: data from 01-01-2000 to 31-12-2003
Test set: data from 01-01-2004 to 31-12-2004

In this case, no knowledge from the future is used in the training phase and we can use the model to predict the data in the test set.


Softmax activation predictions not summing to 1 I am a beginner with rnns, consider this sample code

from tensorflow import keras
import numpy as np

if __name__ == '__main__':
    model = keras.Sequential((
        keras.layers.SimpleRNN(5, activation="softmax", input_shape=(1, 3)),
    ))
    X = [
        [1, 2, 3],
        [4, 5, 6]
    ]
    y = [
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0]
    ]
    X = np.array(X)
    X = np.reshape(X, (2, 1, 3))
    y = np.array(y)
    # print(X)
    # print(y)
    print(model.summary())
    model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.01),
                 loss="categorical_crossentropy")
    model.fit(X, y, epochs=100)
    p = model.predict(X)
    print(p)
    p = list(np.squeeze(p))
    print(p)
    print(np.sum(p,axis=1))

I am using a simple rnn with batch size=2, 3 input features and 1 timestep,as the activation is softmax the last line prints [1,1] as the sum of predictions of a softmax is 1.
But when when I change the layer from a SimpleRNN to

keras.layers.LSTM(5, activation="softmax", input_shape=
                 (1,3),recurrent_activation="softmax")

The sum of predictions is no longer 1, why is that?

Softmax doesnt work as an LSTM activation. You have to add a dense layer using a softmax activation after the LSTM layer. I wouls suggest using another activation in the LSTM like relu.


Traditional Predictive Analytics vs Machine Learning Methods What is the difference between traditional predictive analytics done using statistics and its tools and, one using machine learning and deep learning?
 How are we leveraging machine learning and deep learning to make predictive models better?
How to decide the boundary for when to use traditional models and, when to use ML and DL?
We have a lot of efficient statistical tools available Stata, SPSS. They are easier to use. Also, computationally efficient.

Thanks

I am unable to find some satisfactory and convincing answer with strong points on google.

This question is very hard to answer because ML is very new and not as semantically and academically defined as statistics.

However, it helps to see the problem a bit different by focusing on tools and methods as well as goals and use cases.

Per se machine learning uses statistical and mathematical algorithms to solve computational problems and specifically problems of prediction. This is true for traditional predictive analytics and tools as well.

A random forest model is not intrinsically traditional or ML and indeed I can fit it in SPSS as well as in python.

So what if any are the differences:

First, we have to understand that ML comes from a different domain than classical statistical analysis. For many cases, there will be no real difference but a CS student may call it ML while a Sociologist will call it predictive analysis.

So let's look at more specific questions of you:

Why not use STATA and SPSS?

You can! Indeed STATA and SPSS do have ML capabilities themselves and could be used to model modern ML algorithms. However, for many specific use cases in the ML domain, STATA and SPSS lack computational power and the ability to model large data sets efficiently.

Why not use traditional models like linear regression, etc. all the time?

Not every problem and every type of data fit classical methods. Linear regression, for example, cannot cope with sparse data and large instances of NAs, it cannot predict nonlinear relations and collapses really fast if we use a lot of predictors (a simple image can be converted to 786 predictors for example).

Additionally ML methods are just updating classical predictive analytics by implementing them using state of the art technology. ML is different from traditional statistics mainly in usage, where we have stuff like incremental learning and self-optimization, etc.

Also please understand that there is a huge difference between classical ML algorithms like boosted trees, random forest, Naive Bayes, etc. and deep-learning which is really, really different.


How to perform a reggression on 3 functions using a Neural Network I am currently building a neural network using Keras to perform a regression.

I have 4 independent variables W,X,Y,Z. They are used to predict 3 different functions f1(W,X,Y,Z), f2(W,X,Y,Z), f3(W,X,Y,Z).

Should my output layer have 1 or 3 neurons? Also, should I be using a relu or linear activation function for the output layer? I'm currently using MSE for my loss function and adam for my optimizer.

For my metrics, should I use 'accuracy' or 'r2'?

Any suggestions?  I'm sorry, I'm new to deep learning...

Should my output layer have 1 or 3 neurons?

The easiest thing to do is to create 3 separate networks, one for each function you want to approximate.

While it may certainly be beneficial if you combine all 3 outputs into the same model via a multi-task framework, I'd suggest starting off with separate networks, which is more intuitive. Then, if you want to see if combining them under one network improves your performance, you could check out how to create models with multiple outputs in keras.

  Also, should I be using a relu or linear activation function for the output layer?

Linear, definitely.

  For my metrics, should I use 'accuracy' or 'r2'?

Accuracy is a metric for classification, not regression. $r^2$ can be used, but for certain problems it isn't very reliable. You could try $r^2$, as well as MSE, MAE, MSLE (MSE of the $log$s of the predicted/actual values), etc.


How to determine precision level of outcome in ML models? I understand there are lot of ML products that can help predict the time to event. For ex, customer will purchase in the next 30 days or Patient has chances of re-admission in next 60 days etc.
But why do you think why don't people predict to the level of date?
ex: For ex, customer will make his next purchase on Feb 2nd, 2016 or patient has chance of readmitting on Mar 23rd, 2018.
Don't know from an ordinary layman perspective, I feel dates might be more useful. Of course, all our predictions are reported based on confidence Intervals.
For ex: If we know the date when the course will end, we can plan our schedule accordingly to do other tasks/sign up for new courses etc. But if we are told that your course might end anytime in next 30/60/90 days, we are at risk of wasting few days before we enroll for another course etc. Hope my example makes sense.
Can experts here suggest me on when
a) Prediction to date level might be a good thing?
b) Prediction to date level might be a bad thing?

The reason you get predictions without an exact time is because that is how models are trained. They are not trained to predict the exact time but a window for it.
The reason a model is not trained to predict exact time is because it introduces a lot of problems starting with data imbalance and the huge number of classes it would introduce. Also, DL/AI is not God or Laplace's demon, so to predict the exact is not going to happen.


Extractive text summarization, as a classification problem using deep networks I understand that this is broad, but I merely require a few pointers.

I wish to implement a text summarization system. I have already tried out an abstractive approach, now I am trying an extractive approach. The current extractive approaches out there(Textrank, Lexrank etc) are not giving optimal performance. It was suggested to me that I take it as an unsupervised learning task, using Autoencoders/RBM's or K means or any other unsupervised learning algorithm to classify sentences as important or not.

My questions are:

Is something like this feasible? (I know that nothing can be said for sure in data science unless tried out, but is it worth the shot?)
What kind of features should I feed into the classifier? I already know of word frequency distribution(TF-IDF), sentence position, co-occurance stats, but will these be enough? What additional features should I consider? Or should I consider directly feeding in word vectors?

I have done my bit of work on the same and have read several papers, but none of them offer clarity in terms of feature computation, but only explain network architecture.

Any form of help will be appreciated, thank you!

TL;DR:

  Is something like this feasible? (I know that nothing can be said for sure in data science unless tried out, but is it worth the shot?)

Yes

  What kind of features should I feed into the classifier? I already know of word frequency distribution(TF-IDF), sentence position, co-occurance stats, but will these be enough? What additional features should I consider? Or should I consider directly feeding in word vectors?

Feed in word vectors. Deep learning has it's way of figuring out good features like tf-idf,co-occurence etc.

Longer Version:

Input:

Convert all the words to word vectors with a word embedding algorithm like Word2Vec or Glove.
This is usually a good idea since this allows you to represent words in a better semantic (dog, cat, cow,etc will be close) and syntactic (November,December,etc will be close) sense.

(Using tf-idf as a feature hasn't worked for me. I used it to supervise extractive summarization with a neural net.)

How to manage inputs:

This depends very much on how your data is. (Please let me know which dataset)
I assume you have something like [sentence1, sentence2,...] as inputs and an output like [True, False, ...] which implies sent1 is in the summary and sent2 isn't.

If your data isn't in this form, you should convert it into this form.

You haven't specified whether it's Unsupervised or Supervised.

for Supervised:

Use an LSTM. Feed the LSTM the sentence word by word. Once the sentence has ended (marked by a token say  or a full stop) make the LSTM make a prediction whether the sentence should be in the summary or not. Train it based on this.

This is kind of similar to how you would do sentiment analysis: feed in the sentence and then ask if the sentiment is positive or negative.

You can experiment with stacking LSTMs or changing hyper parameters to get better results.

for Unsupervied:

This is a bit difficult.
You could convert the sentences to vectors (Sent2Vec) and hope that important sentences cluster together because they contain important words.

Or you could train an RNN to predict the next character and hope that one of the neurons learns to predict important words/sentences as done by OpenAI's unsupervised sentiment neuron which learnt to predict the sentiment.

The general "Deep" architecture followed these days is : Embed, encode, attend, predict

Check out the links I have posted (in blue), they might work for you.

Note: If possible, please mention your dataset and also how you managed to make abstractive summaries (considering that it is a lot harder than extractive)


Structure the dataset for financial machine learning I am trying to construct a dataset to apply MLP in forecasting financial returns. The main idea is that I want to predict future equity returns (1 month ahead, but the horizon can vary, just to give an idea) using fundamental data.

My dataset is made of n features for N companies. The main issue is that the features are not recorded at the same point in time, so that some of them would be a monthly time series and some other a daily time series. I am trying to understand the general way to approach this problem because it's my first ML application.

What should I do if my features are time series of data that differs in this sense? I have red a lot of publications, but no one gets in deeper about how the dataset is composed (before train/test splitting). Can someone recommend some useful source?

EDIT

Suppose I have a variable amount of companies over 10 years of monthly data, mainly because some of them doesn't exist anymore, but it is important for the problem of survivorship bias.

For each of these N companies I have n features that represents fundamentals of these companies. I would be able to regress the monthly return, so I am trying to solve a supervised learning problem.

What I cannot understand is the feasibility of the problem using a simple Multilayer perceptron instead a more complex structure like RNN or even LSTM. I wouldn't want to switch to one of these more complex architectures for some specific reason related to the project I am building.

Probably I am missing the reasoning behind the training of the network. How should I provide input data to the architectures in order to perform backpropagation and GD? The only thing I am sure is that I should shift the monthly return series of a time period to recast the problem as supervised.

You are going to have to consider three different factors:

1 - What data are you going to have available when you run your predictions? Are you going to have to pre-process that data? Are you going to have the time to do it? You should be setting your focus on this and work backwards from what runtime predictions look like

2 - When it comes to time series, you have to think of it in terms of (1) lookback windows, or how many periods prior are you considering and (2) time shifts, or how many periods forward are you predicting? This can result in an amazing number of combinations for you to model. You should end up with data sets where you have n features for X time periods and your target variable (your labeled data) is the result Y time periods from X where you are creating a prediction.

3 - The cardinal sin of time series is that you should never model on data that was not available at the specific runtime date. It's a common beginner mistake where you mix up your time periods and end up taking in data that was not actually available when it was released; in other words, it will not be available to you (or at least not correct) when you go to make a prediction. This can be common in financial data where entities can come back and "re-state" numbers at a future date. You have to make sure you are modeling on data as it stood on the target date you are modeling.

With these thoughts in mind, you should be on your way to prepping your data in a way that makes sense.


Classification technique for unsupervised data? I have unsupervised data (i.e this data doesn't have any target variable through which I can learn it's prior behaviour) it is a mix of continuous and categorical data. Now I want to classify the test data into three categories on basis of my unsupervised data.

The approach I took is to first do the clustering of unsupervised data, use this categorised data as a base data for preparing a new  model that predicts on top of it.

I want know whether this approach correct or not or is there better way for classifying test set? Particular algorithm I need to follow for this?

I am doing this in R.

The approach is to modify the training set data so that this can be used to properly predict the test data. Here target variable is missing in train and test set.

You have many options of algorithms to use for classification of unsupervised data.

This is a very broad topic, but if you need a specific algo recommendation, try to see if self-organizing maps (SOM) may help with your specific problem. In R, try the kohonen package.

K-means is another popular clustering algorithm.

No matter which method you use, consider converting your categorical data to numerical data for clustering, as it may alleviate some of your mixed data-type woes.


How can I detect events on fuel tank Hi guys.
I'm not a data analyst and I need some direction in this. I'm looking for a way to know the events of the fuel during a range of time, could be a day or a month, etc. If the consumption was like the picture above would be easy.

The problem is that the data I have is like this picture.

I need to take the fueling and the possible thefs, that considering that I have false data, that could go up or down. I can see that those abrupt changes are fake because it always comes to the original value, the main problem is that I don't know how long they are gonna stay fake. Could be just one point or n points of bad points.

So, how can I detect these events (fueling and possible theft) whitout counting the errors as events?.
What algorithms or formulas can I apply here?

Welcome to SO! It looks like you have a time series problem. Typically the first step when dealing with time series is to consider the difference. Let us define $f(t)$ as the fuel level at time $t$. You would want to calculate $diff_{\text{fuel}}(t) = f(t) - f(t - 1)$.

After this step you will likely see that the spikes that you identify as bad data are outliers. You could detect these by for example looking at all the data below and above your 2.5 percentile or 5 percentile. Typically this requires some careful analysis work to ensure that you do not delete too much. Once you have identified what the best workable percentile is you can could Winsorize your data.

Lastly, you would look at the data points in the bottom of your resulting distribution. These will likely be the points you identify as theft.

I hope this helps.


Do word embeddings help with out of vocab tokens? I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load existing weights from Glove but am still facing a few problems. The main one being that there are certain "negative" words I know of but that do not appear in the vocab. Because of this, when i try examples with word that do NOT appear in the vocab (like the word "rubbish") the network does not know that this contains a negative sentiment.

Is there a way to use Word2Vec / Glove / etc to pass in the word rubbish, find the similarity to the word garbage, and then pass that known word into the network instead? And if so is that handled by the "Embedding" layer or is it an additional step I need to perform during pre-processing?

Additionally, how can I handle misspelled words? For instance, how can i associate "rubbbbbish" with "rubbish"?

I am new to text classification and would really appreciate a bit of guidance!

By definition an out of vocabulary word (OOV) is a word which haven't been seen in the training data, so it's virtually impossible to know which other word it is similar to since this would have to be determined with some training data. In the example that you mention (synonym), what you could use is a resource such as WordNet which tells you which word it's similar to, then you can use the other word embedding.

Cases of misspelled words is a different story: you could pass your OOV words through string matching techniques, or maybe use a character-based NN which would recognize the word despite the misspelling.

Both cases could be part of a pre-processing stage.


How can I get an algorithm to have an evalutation metric based on aggregate predictions? Let's say I have a model that makes a prediction per individual. An example data set is below. Normally, evaluation metrics (for example within the XGBoost algorthim), are used at the individual observation. However, I don't care whether individual predictions are accurate, I just want predictions in aggregate to be accurate.

In other words, using the sample data set below, and aggregating predictions by Gender, I want to minimize the RMSE between (1000+3000) and (900+3100) for males, and (2000+4000) and (1900+4100) for females; this is as opposed to minimizing the RMSE in the normal sense per each observation.

Is there a phrase for this type of evaluation metric? Is there a way to imlement this into popular algorithms like XGBoost?

Note: There is no reasonable way to aggregate the data prior to training, and ultimately I do need predictions per individual.

I don't care whether individual predictions are accurate

If this is true (which I don't think it is), then you could just take the group mean of the target for males and females to minimize the RMSE and make this your prediction for all male and female individuals, respectively. (Basically a model where Gender is the only feature if you insist on using xgboost.) One could argue that this one-feature model might not generalize as well to new data than a model with more features, but if Gender is not in any meaningful way related to the target you can't optimize the RMSE towards Gender anyway.

Based on your "Note" I assume you somehow want both, good predictions for individuals and on the group level. In this case minimizing the normal "individual-level" RMSE is already the best way to go.


Partial derviative of prediction (sigmoid applied) with respect to weight I am very confused as to where a seemingly "extra" term is included in the above mentioned calculation in my Udacity course.

The above is taking the derivative of a sigmoid so why isn't it just

$$=\sigma(Wx+b)(1-\sigma(Wx+b)$$
but rather has $\frac{\partial}{\partial w_j}(Wx+b)$ tacked on the tail?

Recall that for chain rule, we have $$\frac{d}{dw}h(g(w))=h'(g(w))g'(w)$$

For the context of your question, $h(t)=\sigma(t)$ and $g(w)=Wx+b$,

hence that is why we have one more term.


In backpropagation, scale is also important? I think backpropagation is needed to find the direction of gradient decent method.

I also wonder, the scale is also important?

I heard some issue of vanishing(or exploding) gradient problem.
If the direction of the backpropagation is remained,
we still could apply gradient decent method(cause we still know the direction to update) and could finally get optimal answer.

If I'm correct, there is no real limit in deep learning?
I mean, although the speed may be slow, we can always finish the train work of neural network?

In the process of backpropagation,first of all we feed forward the model.Each node receives the input from its previous layer,calculate the weighted suma and the apply the activation function to the weighted sum and pass this result to next layer as input. When we got the result.

Lets say we are classifying animals images,the each output node will correspond to the different category more or less and node with the largest output will be considered as the output of the model.

Then the loss is calculated on the result,lets say we are using SGD for the loss function.for SGD the loss is calculated as "d(loss)/d(weights)".Here Backpropagation comes into play.

Since the output is calculated with the weight and input,to change the result we need to change these two values(weight and input).Since we can't change the input ,we will change the weights.These weights are updates as

                "new weight=old weight - (gradient*learning_rate)"

Here is the role of scale

For sake of simplicity,lets say for a particular connection weight is initialized to 0.833 and its optimal value is 0.437

Also,Gradient w.r.t to a particular weight in a network is the product of some components that resides later in the network(These components are gradients w.r.t. to weights)
Therefore,the earlier the weight ,more number of components will be required to calculate its gradients.

1.Vanishing Gradient:

If we set our learning rate(or considered as scale) to 0.0001.

                              "gradient*learning_rate"

This value will be very small.
It will take a lot of time to reach the optimal state and at some point will be stuck in a point where the there is almost no change in the weight.

2.Exploding Gradient:

If we set our learning rate(or considered as scale) to 0.01.

                              "gradient*learning_rate"

The scale will be larger enough to reach the optimal value for weight and therefore the optimal value will be skipped.
for simplicity lets say gradient is 1

                "new weight=old weight - (gradient*learning_rate)"

new weight=0.833-0.01=0.823

new weight=0.823-0.01=0.813

.

.

.

new weight=0.453-0.01=0.443

new weight=0.443-0.01=0.433

new weight=0.433-0.01=0.423

It can be seen that the optimal value is skipped.
That is why the scale is important and should be used carefully.


How does one define the possibility space of valid priors (models)? When one trains a model on data of any complexity one inevitably ends up with a one particular model among a vast many possible models that would make similar (or even, the same) predictions.

For example, we can both train a neutral net on mnist and the model I produce will have different weights and biases from yours, even if they make the same predictions.

But are there any algorithms that don't produce one model, but instead produce a description of the space of all possible models that could describe the data?

Does that even make sense? In some way, as near as I can tell, a model represents a set of priors (biases and weights). Is this sort of thing not done because it's too expensive?

Take the following riddle as a conceptual example, '''A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?'''

You might have a prior that says, "most surgeons are men" and another prior that says, "most parents are heterosexual." You have a space of two models that would explain the data, and of course they are mutually exclusive: either the boy has two fathers, or the surgeon is his mother.

The key is, both of these models would work to explain the data and you don't know which model is true until you get more data. Also, just as importantly, you can map each model's relationship to your prior beliefs so when you learn which one is valid you can update you're priors accordingly.

A technique or algorithm that produces a probability space of all possible models seems like a potentially powerful tool. Yet I can't find this kind of thing anywhere.

Are there any algorithms or machine learning techniques that do this? And furthermore how do you even go about mathematically describing a possibility space of all possible valid models according to their likelihood, according to your priors?

Thanks for your help!

From a theoretical perspective, I think this question is a variant of "given a problem, is there a way to determine the absolute best learning algorithm for it?". Why? Because as OP correctly suggests, if there was a way to represent all the possible models/priors, then assuming that an infinite stream of data is provided we would be able to eventually determine the absolute best learning algorithm. Unfortunately the No Free Lunch theorem is a theoretical result which states that there can be no such "absolute best" learning method. (to be honest personally I'm out of my depth with this kind of thing, but Wikipedia gives a nice short explanation of the implications of the theorem for machine learning.)

From a practical perspective, this is a matter of design, i.e. how one chooses to represent the problem. It's easy to overlook how much simplification is made when one represents a problem. In the example proposed by OP, one could imagine an infinity of "alternative worlds": maybe the word "father" doesn't have its usual meaning; maybe it happens in a future where people can have any number of parents; etc. My point is: whenever a problem is formally stated, a vast number of assumptions are made anyway, most of them unconsciously. It's not only a matter of cost (it's important, of course), it's a matter of "targeting" the exact problem: if we tried to leave the space of possibilities completely open, then we cannot even start to solve the problem, in the same way that people have to agree on a common language in order to understand each other. Thus choosing a specific space of priors/models for a particular problem is akin to fixing the language of the problem, and when doing it it's important to take into account the limitations/assumptions associated with it.


Which combination of 3 hyperparameters to combat overfitting of a convolutional neural network? I have a small dataset with which I want to train a CNN by using Data Augmentation. Since the CNN is overfitting due to the small data set, I would like to optimize some hyperparameters.
However, since I would like to use GridSearchCV from Scikit-Learn for this and I would therefore like to optimize only 3 hyperparameters due to reducing computational time. Here the question arises which combination of hyperparameters should I use for grid search?

My current approach would be to optimize the learning rate, the dropout layer rate and the number of epochs.

I choose the learning rate because the book "Deep Learning" by Goodfellow recommends to always optimize the learning rate. But I'm not sure if my combination of hyperparameters for tuning is really good.

What combination would you recommend?
Many thank for every hint

My previous architectur is as follows:

model = Sequential()
model.add(Conv2D(32,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv2D(32,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2))
model.add(Dropout(0.2))

model.add(Conv2D(64, (3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Conv2D(64,(3,3))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2))
model.add(Dropout(0.2))

model.add(Flatten())

model.add(Dense(512))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.4))
model.add(Dense(10))
model.add(Activation('softmax'))

As optimizer I use Adam.

If you are asking specifically about overfitting, I would only keep dropout rate from your list of three. Other two can be any chosen from: number of filters in the convolutional layers, number of convolutions, number of dense layers (if any), number of neurons in dense layers.

Learning rate should be optimized but not for the purpose of combatting overfitting, at least the way I understand it. Using Keras, you can start with some learning rate - not necessarily very fine tuned, and slowly reduce it over time once the training reaches a plateau using Learning Rate Scheduler in Callbacks. Also, there are ways to find an optimal learning rate such as learning rate finder, so it would probably be a misuse of resources to optimize for learning rate using grid search (I never used them though!).

You can also use the Model Checkpoint present in the above Callbacks page to save the model as the validation loss improves only, and ignore the later epochs where overfitting becomes an issue. This way, you won't need to include number of epochs in your search.


Looking for a good package for anomaly detection in time series Is there a comprehensive open source package (preferably in python or R) that can be used for anomaly detection in time series?
There is a one class SVM package in scikit-learn but it is not for the time series data. I’m looking for more sophisticated packages that, for example, use Bayesian networks for anomaly detection.

I know I'm bit late here, but yes there is a package for anomaly detection along with outlier combination-frameworks.
The package is in Python and its name is pyod. It is published in JMLR.
It has multiple algorithms for following individual approaches:

Linear Models for Outlier Detection (PCA,vMCD,vOne-Class, and SVM)
Proximity-Based Outlier Detection Models (LOF, CBLOF, HBOS, KNN, AverageKNN, and MedianKNN)
Probabilistic Models for Outlier Detection (ABOD and FastABOD)
Outlier Ensembles and Combination Frameworks(IsolationForest and FeatureBagging)
Neural Networks and Deep Learning Models (Auto-encoder with fully connected Neural Network)

Finally, if you're looking specifically for time-series per se, then this github link will be useful.
It has the following list packages for timeseries outlier detection:
datastream.io
skyline
banpei
AnomalyDetection


Out-of-bag error in Orange3 There is some way to see the out-of-bag error in Orange3? The widget Test and Score only reports mean squared error, root mean square, mean absolute error, coefficient of determination and root-mean-square deviation.

Orange3 doesn't calculate the OOB. This must be implemented a separate code. More information about how to calculate the OOB error in scikit-learn.


Confused between optimizer and loss function I always thought the SGD was a loss function then I read this on a notebook
model.compile(loss="sparse_categorical_crossentropy", optimizer=keras.optimizers.SGD(lr=1e-3), metrics=["accuracy"])

now I am confused what's the difference between loss and optimizer ? are they both used at the output layer to calculate the loss?
or is the optimizer something used in each layer?

Think of loss function what to minimize and optimizer how to minimize the loss.
loss could be mean absolute error and in order to reduce it, weights and biases are updated after each epoch. optimizer is used to calculate and update them.


Why GA convergence curves continue as two parallel lines? I'm working on a optimization problem and using GA algorithm (in MATLAB, ga function).

As you know MATLAB plots GA result with two curves, one for the best values and other to show the mean values and when this two curves touch each others it means algorithm has been converged.

On my cause these two curves don't get along and continue as two parallel lines until the end of max Generations number and finally a premature convergence,What makes it happen?

I have tested with other parameters value but same result. On pre-processing I used Missing value,Normalized and smoothing and tested data set belongs to UCI.

GA Parameters:

MutationFcn      : @mutationadaptfeasible: 0.03
CrossoverFcn     : @crossoverheuristic : 1.2
maxGenerations   : 200;
populationSize   : 180;
Chromosome length: 33
Test Instances   : 71

As you know MATLAB plots GA result with two curves, one for the best values and other to show the mean values and when this two curves touch each others it means algorithm has been converged.

More accurately when the two curves touch each other it means that all the individuals in the population perform exactly the same, because that's the only way for the best and the mean to be equal (and usually this happens because all the individuals are identical). That can happen only by chance and/or if the mutation probability is very small, because as soon as there is an individual with a mutation it's unlikely that this individual performance will be equal to the best.

To me in a plot like yours successful convergence was reached around 60-70 iterations. The residual difference is due to the random mutations.


Need explanation of a matrix multiplication I'm reading the Deep Learning book by MIT.
On the page 172, there's a part like this:
$$
f^{(1)}(x)=h=W^Tx  \tag{1}
$$
$$
f^{(2)}(h)=h^Tw \tag{2}
$$
Substitute (1) into (2), they got:
$$
f(x)=w^TW^Tx
$$
Since I'm not so familiar with linear algebra stuff, I infer that something like below is valid:
$$
A^TB=B^TA \tag{3}
$$
So what is the property of (3) is called?

Let's take it step by step.

$$
f^{(1)}(x)=h=W^Tx  \tag{1}
$$

$$
f^{(2)}(h)=h^Tw \tag{2}
$$

We substitute h.

$$
f(x)=(W^Tx)^Tw   \tag{3}
$$

To make it work, we'll make a little trick.

  The transpose of a transposed matrix is the original matrix.

$$
(A^T)^T = A
$$

We substitute w.

$$
f(x)=(W^Tx)^T (w^T)^T  \tag{4}
$$

Now we use following:

$$
(AB)^T=B^T A^T
$$

  The transpose of two matrices multiplied together is the same as the product of their transpose matrices in reverse order.

et voila

$$
f(x)=w^TW^Tx
$$


Is there a thumb-rule for designing neural-networks? I know that a neural-network architecture is mostly based on the problem itself and the types of input/output, but still - there's always a "square one" when starting to build one. So my question is - given a input dataset of MxN (M  is the number of records, N is the number of features) and a C possible output classes - is there a thumb-rule to how many layers/units should we start with?

This question has been answered in detail on CrossValidated: How to choose the number of hidden layers and nodes in a feedforward neural network?

However, let me add my own two cents:

There is no magic rule for choosing the best neural network architecture, but if you can find an architecture someone has used to solve a similar problem this is often an excellent starting point.

The best places to look are official or unofficial examples using popular neural network libraries  such as Keras, PyTorch, or Tensorflow, and architectures described in academic literature. keras/examples on github is a great resource.

These architectures were likely chosen after lots of trial and error, so most of the work will have been done for you.


What is the opposite of baseline? I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data), so I would like to know how I can call it (antonym of baseline) in the research paper.

The term is Oracle.
Some references:

SO question describing the term
Scientific articles related to machine learning using the term


Do neural networks have explainability like decision trees do? In Decision Trees, we can understand the output of the tree structure and we can also visualize how the Decision Tree makes decisions. So decision trees have explainability (their output can be explained easily.)

Do we have explainability in Neural Networks like with Decision Trees?

I disagree with the previous answer and with your suggestion for two reasons:

1) Decision trees are based on simple logical decisions which combined together can make more complex decisions. BUT if your input has 1000 dimensions, and the features learned are highly non linear, you get a really big and heavy decision tree which you won't be able to read/understand just by looking at the nodes.

2) Neural networks are similar to that in the sens that the function they learn is understandable only if they are very small. When getting big, you need other tricks to understand them. As @SmallChess suggested, you can read this article called Visualizing and Understanding Convolutional Networks which explains for the particular case of convolutional neural networks, how you can read the weights to understand stuff like "it detected a car in this picture, mainly because of the wheels, not the rest of the components".

These visualizations helped a lot of researchers to actually understand weaknesses in their neural architectures and helped to improve the training algorithms.


Forecasting non-negative sparse time-series data I have a time-series dataset (daily frequency) representing the sales of a product to a customer over time. The sales is represented as the following:

$$[0, 0, 0, 0, 24, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 17, 0, 0, 0, 0, 9, 0, ...]$$

in which each number represents the sales of the product in a day.

The problem is that time-series forecast methods (ARMA, HoltWinters) work well for "continuous" and "smooth" data, but is not producing good results in this case.

I want to make a forecast of that series, with attention to 2 points: (1) assuring non-negative values and (2) sparse/ non-continuous data. Anyone knows how to approach this problem? What methods/ technique?

Thanks!

I have two ideas here, maybe they will be helpful.

Idea 1: Model time between events

You might think of your data as being generated by two processes: the first is a distribution over time intervals, and the second is a distribution over purchase amounts. So to model your data you could create one distribution (gaussian?) over the nonzero values in your dataset, and another over the lengths of sequences of zeros (poisson?).

Idea 2: Model customer inventory

Even though the sales events in your dataset are sparse, you could spend a little time to come up with a model of why the customer is making purchases when they do. In one possible model, the customer has an inventory that shrinks over time, and they make purchases when their inventory crosses some minimum threshold. You could use your sales data to fit the slope (for linear shrinkage) or rate (for exponential shrinkage) as well as the threshold.

This could get arbitrarily complex, since the customer under this model might have different thresholds or shrink rates at different times ... but for starters it could be a useful approach to get a sense of things.


Causes of vanishing and exploding gradient I know the concept between the vanishing and exploding gradient. I would like to know the possible causes of these phenomena. I think I read somewhere on the internet about the activation functions. Could someone please clarify? Any help will be greatly appreciated.

Gradient vanishing and exploding depend mostly on the following: too much multiplication in combination with too small values (gradient vanishing) or too large values (gradient exploding).

Activation functions are just one step in that multiplication when doing the backpropagation. If you have a good activation function, it could help in reducing these bad effects. One such activation function is ReLu. See this tutorial.

Other sources:

A Gentle Introduction to Exploding Gradients in Neural Networks
Deep Learning book by Ian Goodfellow at el.: This book has a lot of examples with gradient problems (eg. Chapter 8.2.4: Cliffs and Exploding Gradients)


Data visualisation for big data sets Working on some voluminous data sets, I have been interested in methods for dimension reduction and plotting. I have stumbled upon this novel technique : UMAP (https://arxiv.org/pdf/1802.03426.pdf), which allow to reduce the dimension of a data set to plot it in 2 dimension. It seems fast and efficient.

In the main paper authors even provide some beautifull plots. As for exemple :

I have been unable to reproduce this kind of graphics in R with ggplot2 (computer crashing because of too many points, overlaping points...). How would one go about building a similar plot ?

Ggplot2 is great for simple visualisations but doesn't handle huge datasets very well, as you found out. Running out of memory is also often an issue with huge sets, and calculating prime divisibility on 30 million integers definitely counts as huge.

If you have access to more powerful systems or cloud platforms that should work a lot better. Some of the AWS and Azure etc. solutions aren't that expensive so you could try one of them.


What the good general regression technqiue for a problem with 50 independent varaibles I am a newbie to data science and statistics. I came across this problem, which has 50 independent variables and one dependent variable and trying to identify the good regression technique to start with. The following is the flow chart that I executed:

Data Exploration -> Correlational matrix -> dimension reduction -> PCA (Dimension reduction) -> Basic Linear Regression technique.

Can someone guide me, if there is any other better technique or procedure.

In no way is this going to be an exhaustive answer, but it will definitely give you a starting point in Python -

Data Exploration

Start with Pandas Profiling. It will give you HTML reports of your variables. If the quality of the data is good, it will provide some insights into the fill rate, depending upon the variable type some statistics for each variable

Correlational matrix

The pandas profiling report includes the coorelation matrix. But if you are looking to compute by hand, use pd.corr(). You can vary parameters to get different correlation metrics like ‘pearson’, ‘kendall’, ‘spearman’

Dimension Reduction -> PCA (Dimension reduction)

There are many ways to do this. Keep in mind if you are looking for accuracies only and don't care about how X is influencing y, (1) is an optional step (applies to (2) as well).

Analyse the correlation matrix and use VIF to dump variables with high correlation
Factor Analysis / PCA for dimensionality reduction
Use LASSO to fit a model, check the coefficients and the ones that are 0 or going to 0 can be thought of as weak indicators and can be eliminated.
Keep all 50, and use Ridge Regression and vary the alpha parameter to fine-tune accuracy (or whatever metric you are trying to optimize)
If the model still doesn't seem to be stable, try to cook non-linear features with sklearn's Polynomial Features, regularize and repeat.
Probably the most important in the real world, ask the domain expert on what he/she thinks might be the important variables

Basic Linear Regression technique

Playing with hyperparamters to get good cross-validation/test score is the key here for a basic Linear Regression model.
Try as many techniques as you can from here and here


Division of numbers from contours in opencv in python for cnn I want to separate numbers in suppose 7638 into different images which can be predicted individually using cnn.
By finding contours how can I divide each contour into separate image in python.
To be more clear: How can I divide https://www.letter2word.com/products/americana-numbers into 0,1,2,3..9 into individual images using the concept of contours in opencv in python so that the resulting images can be predicted by cnn classifier and later all the results written together to give the number in image(initial) as output.
Also, is there any other method to do so.
PS.-I tried searching in google but everywhere it is given to extract 1 particular part of the image while I need more than 1 parts of the image for correct prediction

After browsing for long hours I have found out that taking the contours as ROI is a considerable option


Resampling for imbalaced datasets: should testing set also be resampled? Apologies for what is probably a basic question but I have not been able to find a definitive answer either in the literature or in the Internet.

When dealing with an imbalanced dataset one possible strategy is to resample either the minority or the majority class to artificially generate a balanced training set that can be used to train a machine learning model.

My doubt stems from the fact that a testing set is supposed to be a representation of what real-world data is going to look like. Under this assumption, my understanding is that the testing set is not to be resampled, unlike the training set, since the imbalance we are trying to deal with is present in the live data in the first place.

Could someone please clarify if this intuition is right?

You are right! As you said, the final goal of the model is to work with real life data distribution, therefor the test data has to stay unchanged.

The goal of resampling is not to have the same number of objects from all classes, but to fit model to work with your data. So an intuitive example might be: if you have k objects of the first class and 5*k of the second, but your model learn data well, then resampling is not necessary. It's the reason, why we shouldn't use it during test (we have already trained model)

One more thing that could be useful to remember is that F1 score could be calculate in multiclass task in different ways (it could account label imbalance), here is an example from sklearn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html


How does keras (TF) know how to differentiate my custom loss? Suppose I have this custom loss:

def custom_loss(y_true, y_pred):
            out = K.clip(y_pred, 1e-8, 1-1e-8)
            log_lik = y_true*K.log(out)
            return K.sum(-log_lik*advantages)

How does keras (with TF as backend) know how to differentiate in terms of the input specifically and ignore the 'advantages' simply as coefficient?
Does it do it numerically? If so is that the same as with it's own loss functions or only with custom?

keras will differentiate automatically by looking at the graph of operations you use as part of your custom loss function. Since you are using only operations that keras "knows about" aka that exist as operations in TensorFlow keras will automatically a graph of operations to backpropagate against.

If you however have a custom loss function that uses an operation TF doesn't know, you'd have to write a custom operation so that TF knows how to backpropagate against it


Help with reusing glove word embedding pretrained model When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?

I was stuck in a similar problem while working with glove. Assuming that you have a dataset in text form, from which you want to collect the topmost 100000 words, you'll have to make a list of those words. In the glove file, each embedding is on a separate line, with each line starting with the word itself and then the embedding. You'll have to write a code to compare your list of words with the words in glove file and extract the lines which make a hit. Have a look here for example code.


Visualizing items frequently purchased together I have a dataset in following structure inserted in a CSV file:

Banana  Water   Rice
Rice    Water
Bread   Banana  Juice

Each row indicates a collection of items that were purchased together.  For example, the first row denotes that the items Banana, Water, and Rice were purchased together.

I want to create a visualization like the following:

This is basically a grid chart but I need some tool (maybe Python or R) that  can read the input structure and produce a chart like the above as output.

I think what you probably want is a discrete version of a heat map.  For example, see below.  The red colors indicate the most commonly purchased together, while green cells are never purchased together.

This is actually fairly easy to put together with Pandas DataFrames and matplotlib.

import numpy as np
from pandas import DataFrame
import matplotlib
matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)
import matplotlib.pyplot as plt

####
# Get data into a data frame
####
data = [
  ['Banana', 'Water', 'Rice'],
  ['Rice', 'Water'],
  ['Bread', 'Banana', 'Juice'],
]

# Convert the input into a 2D dictionary
freqMap = {}
for line in data:
  for item in line:
    if not item in freqMap:
      freqMap[item] = {}

    for other_item in line:
      if not other_item in freqMap:
        freqMap[other_item] = {}

      freqMap[item][other_item] = freqMap[item].get(other_item, 0) + 1
      freqMap[other_item][item] = freqMap[other_item].get(item, 0) + 1

df = DataFrame(freqMap).T.fillna(0)
print (df)

#####
# Create the plot
#####
plt.pcolormesh(df, edgecolors='black')
plt.yticks(np.arange(0.5, len(df.index), 1), df.index)
plt.xticks(np.arange(0.5, len(df.columns), 1), df.columns)
plt.savefig('plot.png')


Mathematically prove why sparsity leads to model overfitting With respect to the stackoverflow post here: https://stackoverflow.com/a/59566478/9130959

I can't quite get why the logic stands: when # features increases, the hypothesis space is expanded, leading to sparse data, thus easily overfit. Is there a way to mathematically explain all these?
Thx in advance.

I don't think sparsity in itself cause the model to over-fit, but it can increase the chance of it.
Overfitting is not a state in itself. What I meant is that if I say that my training accuracy is 92%. Can you predict if it's over-fit?
 It's a relative state i.e. you can only know it if I tell the accuracy on test/new data.
It means, farther the new data from the training data more is the chance that the model will Over-fit.
In a high dimension space, let's say a 10 features dataset -
To fill the space evenly you need 1010 record which is 10Bn. Normally you might have 1Mn.
So, it's almost 9999/10000 chance that you will get a new data outside the training data.
Definitely the model will have a very high chance to fail(Assuming no Regularization in place)(I am not saying that it will fail 9999 times)
But
 This is completely based on the assumption that every feature will have values spanning the full space.
 Let's say 6 out of 10 features have only 2 possible value. Then we will have only 6.4Mn datasets required to fill the space evenly. In this case, there is very less chance of the model to be over-fit compared to the last case.
Although the model will be very complex and wiggly(a lot of branches/leaves), but not a lot over-fitted


face landmark detection cnn loss not converging in tensorflow I am trying to build face landmark detection model using simple regression.I used celeba dataset which has 5 points hence 10 output units.I used grayscale and normalized image as input.
Here is my model
    self.model = models.Sequential()
    self.model.add(layers.Conv2D(32, (4, 4), activation='relu', input_shape=(218, 178,1)))#l1
    self.model.add(layers.Conv2D(64, (4, 4), activation='relu'))#l2
    self.model.add(layers.Conv2D(32, (4, 4), activation='relu'))#l3
    self.model.add(layers.Conv2D(16, (4, 4), activation='relu'))#l4
    self.model.add(layers.Conv2D(16, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(16, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(32, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    self.model.add(layers.Conv2D(128, (3, 3), activation='relu'))
    self.model.add(layers.AveragePooling2D((3, 3)))
    self.model.add(layers.Flatten())
    self.model.add(layers.Dense(16))
    self.model.add(layers.Dense(10))

Here is my loss function
model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.001),loss='mse')

Here loss is stuck around 18 and doesn't go below that. I tried various configuration of CNN architecture like adding and removing layers and I also tried with changing learning rate  but no use.
Please anyone point me in right direction. How can I debug this network.
(For this I only used first 100 images of dataset)

Use a standard pretrained network like ResNet, Effnet or even VGG16 for feature extraction and put a regression layer in the end.
Now, talking about your network. FIrst of all, the image size is not square, make it! Don't use even valued kernels (3 and 5 are mostly used). Number of filters increase with depth, yours don't follow a pattern.
Lastly, 100 images are nothing for these behemoths, give it more data.
Better stick to a pretrained model.


whats wrong with the graph I wanted to plot a bar graph which shows covid cases reported in different regions in different regions of USA.
So here is my code:
import pandas as pd
import matplotlib.pyplot as plt
datainput = pd.read_csv("MD_COVID-19.csv")
fig = plt.figure()
ax = fig.add_axes([0,0,1,1])
rgn1=list(datainput['Allegany'])
rgn2=list(datainput['Anne_Arundel'])
rgn3=list(datainput['Baltimore'])
rgn4=list(datainput['Baltimore_City'])
rgn5=list(datainput['Calvert'])
rgn6=list(datainput['Caroline'])
rgn7=list(datainput['Washington'])
region=['Allegany','Anne_Arundel','Baltimore','Baltimore_City','Calvert','Caroline','Washington']
rgn1s=0
rgn2s=0
rgn3s=0
rgn4s=0
rgn5s=0
rgn6s=0
rgn7s=0
for item in rgn1:
 rgn1s +=item
for item in rgn2:
 rgn2s +=item
for item in rgn3:
 rgn3s +=item
for item in rgn4:
 rgn4s +=item
for item in rgn5:
 rgn5s +=item
for item in rgn6:
 rgn6s +=item
for item in rgn7:
 rgn7s +=item
tc=[]
tc.append(rgn1s)
tc.append(rgn2s)
tc.append(rgn3s)
tc.append(rgn4s)
tc.append(rgn5s)
tc.append(rgn6s)
tc.append(rgn7s)
ax.bar(region,tc)
plt.ylabel("Number of Cases")
plt.show()

My bar graph is shown as above. Why are only three bar graphs coming?
Where are the other four? Also how can I draw pie chart?

The reason the other bars are missing is because of your method of summing the values and the missing values in your dataset. The way you are adding the values together means that if even one value is missing (NA) the total for that column will be missing as well, and as a result, will not be in your final plot. It is better to use pandas built-in methods (for conciseness, speed, but also to avoid errors such as the one you encountered) to add up the values as follows (using random values):
import pandas as pd
import matplotlib.pyplot as plt

datainput = pd.read_csv("MD_COVID-19.csv")
fig = plt.figure()
ax = fig.add_axes([0,0,1,1])
regions = ['Allegany', 'Anne_Arundel', 'Baltimore', 'Baltimore_City', 'Calvert', 'Caroline', 'Washington']
# select all region columns and calculate the sum column-wise
total_cases = datainput[regions].sum()
ax.bar(regions, total_cases)
plt.ylabel("Number of Cases")
plt.show()

Which gives the following result:


Encoding features like month and hour as categorial or numeric? Is it better to encode features like month and hour as factor or numeric in a machine learning model?

On the one hand, I feel numeric encoding might be reasonable, because time is a forward progressing process (the fifth month is followed by the sixth month), but on the other hand I think categorial encoding might be more reasonable because of the cyclic nature of years and days ( the 12th month is followed by the first one).

Is there a general solution or convention for this?

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through.

(More Info)


Compare cross validation values of Bernoulli NB and Multinomial NB I'm testing the Multinomial NB and Bernoulli NB on my dataset and I'm using the cross validation score to better understand which of the two algorithms work better. This is the first classifier:
from sklearn.naive_bayes import MultinomialNB
clf_multinomial = MultinomialNB()
clf_multinomial.fit(X_train, y_train)
y_predicted = clf_multinomial.predict(X_test)
score = clf_multinomial.score(X_test, y_test)
scores = cross_val_score(clf_multinomial, X_train, y_train, cv=5)
print(scores)
print(score)

And these are the scores:
[0.75       0.875      0.66666667 0.95833333 0.86956522]
0.8637666498061035

This is the second classifier:
from sklearn.naive_bayes import BernoulliNB
clf_multivariate = BernoulliNB()
clf_multivariate.fit(X_train, y_train)
y_predicted = clf_multivariate.predict(X_test)
score = clf_multivariate.score(X_test, y_test)
scores = cross_val_score(clf_multivariate, X_train, y_train, cv=5)
print(scores)
print(score)

And these are the scores:
[0.5        0.5        0.54166667 0.54166667 0.52173913]
0.5

From what I understood from the answer posted here, the first classifier works better because my dataset has lots of features (11k) instead of just 1. However, it's pretty strange that I got 0.5 in the second classifier which is an high value considering the number of features. What are the differences between the classifiers?

The difference is explained in the documentation:

Like MultinomialNB, this classifier is suitable for discrete data. The difference is that while MultinomialNB works with occurrence counts, BernoulliNB is designed for binary/boolean features.

So it's not a matter of number of features, it's about how the values of the features are interpreted: Multinomial can deal with multiple discrete values, whereas Bernouilli deals only with binary variables.
The doc also mentions this option:

binarize: float or None, default=0.0
Threshold for binarizing (mapping to booleans) of sample features. If None, input is presumed to already consist of binary vectors.

Since you didn't provide a value for this option in your code, the default 0.0 applies. This means that when using Bernouilli all the variables are converted to
binary variables: anything lower or equal to 0 is 0, anything higher is 1. This explains why Bernouilli works with your data, albeit not as well as Multinomial: probably for many features in your data the fact that the value is zero or not is a good indication for the label.


How to accomodate different database / schema names when writing M queries in Power BI? I am working with Power BI and I need to setup a template with some input parameters, one of them being the name of the database to get data from.

To do this, I have setup a parameter (called project_database) so that users can specify the database to be used in an Query via the Parameter Manager in Power Query Editor.

I wrote a M Query that collects data from a database like so (with project_database being the input parameter):

let
    Source = MySQL.Database("127.0.0.1", project_database, [ReturnSingleDatabase=true]),
    working = Source{[Schema=project_database,Item="example_table"]}[Data],
in
    working

The problem is, this query will generate a result table with all the generated column names being based off of the actual database name (what project_database resolves to.)

For example, if the database name was xyz, the generated column names would be xyz.id, xyz.fielda, xyz.fieldb etc...)

How do I properly reference columns throughout the M query if I cannot rely on the schema portion of their names remaining consistent, since Power BI generates them based off of the schema name?

I attempted referencing the column names by

project_database & ".field"

But this is insanely tedious to do throughout the every M query. There must be a better approach?

Thanks!

There is no public documentation on generating column names... So hopefully I can help shed some light here.

When creating the data source with MySQL.Database, you can actually pass in a function that will be used to generate the column names, NavigationPropertyNameGenerator

First, we need to create the function that will generate these names. Basically, I want the exact same behaviour as the default except to not change the name depending on the schema / selected db.

Create a function to generate names, I created a blank query and used the below code (which will replace all instances of "abc." to "xyz.", where abc is the actual name of the schema.)

let
    ListAccumulateFunction = (state, current) =>
    let
        effectiveName = Text.Replace(current, project_database & ".", "xyz."),
        occurances = List.Count(List.FindText(state{1}, effectiveName)),
        newName = effectiveName & (if occurances > 0 then " " & Text.From(occurances + 1) else ""),
        newNamedList = List.Combine({ {effectiveName}, state{1} }),
        newColumnsList = List.Combine({state{0}, {newName} })
    in
        {newColumnsList, newNamedList},

    ColumnNameGenerator = (p, a) =>
    let
        working = List.Transform(a, each _[TargetTableName] & " (" & Text.Combine(List.Distinct(_[SourceKeys])) & ") "),
        renamedDuplicates = List.Accumulate(working, {{}, {}}, ListAccumulateFunction)
    in
        renamedDuplicates{0}

in
    ColumnNameGenerator

Now we can use this function in throughout the project in data sources as the name generator

Source = MySQL.Database("127.0.0.1", project_database, [ReturnSingleDatabase=true, NavigationPropertyNameGenerator=ColumnNameGenerator]),


What should be the main concerns when generating extra data? I've been participating in a project where I had to cut most of the dataset due to mismeasurements/invalid values and we ended up with a small dataset.

The dataset is regarding  the following:

Tool Deflection : Metrology - Sensor Measured
Tool Wear : Metrology - Sensor Measured
Piece - Form Accuracy : Metrology - Sensor Measured
Piece - Roughness : Metrology - Sensor Measured
Machine Input - Parameters : System Inputs

I thought of generating more data for each metrology dataset, using the Machine Inputs as reference, however, I feel like it could potentially bias my model.

As you can see, I'm quite confused with respect to this subject.

What should I expect if I move towards this solution?
Is there any restriction I should be aware of?

It's as you said there is a chance of bias in the dataset. To avoid this you'll have to conform to an algorithm to generate data. Right off the bat, the solution I would propose is parametric methods. Find the statistical distribution of data and depending on that distribution fill in data accordingly.

If you want to know a bit more about parametric methods you can look at my answer here.

What does it mean for the training data to be generated by a probability distribution over datasets

For some machine learning methods look at this answer I posted here.

What are the best way to handle missing values

There is never one single correct way to do what you're asking, but there are good ways and it depends on the nuances of your data.


How to Improve Low Accuracy Keras Model Design? I am trying to train a system that looks at some data points and predicts the quantity of surfers on a surf break. I have labeled the pattern for the past 2 months and I have 1500+ training examples with observations every 15 minutes during the day (excluding night time)

My data is shown below (*kooks = surfers):

I am using Keras and here's the code:

*I am removing the month feature from the input matrix before processing it. I also did the preprocessing.MinMaxScaler() drill.

model = Sequential()
model.add(Dense(64, input_dim=6, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='relu'))

model.compile(optimizer='adam',loss='mse', metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=10)
history = model.fit(X, y, validation_split=0.33, epochs=200, batch_size=15, verbose=0, callbacks=[early_stopping_monitor])

The results I am getting are extremely poor:

Test score: 0.015
Test accuracy: 0.12

I have tried multiple optimizers and multiple activation functions, but haven't landed at a satisfactory model yet.

I have a couple of suspicions:

The data is not really predictable, as the system is getting confused as some times many of the features are the same (see lines 0 and 1), but the expected output is completely different.
The design of the model is not well suited (I don't really know how to design the hidden layer dimensions)
The loss function, optimizer and/or activation functions on each layer (including the output layer) are not well suited.

Am I doing something really wrong or this is just the nature of the beast? Any thoughts / advices?

Looking at the training epochs, it seems to me you set a patience parameter that is too short. Please consider removing early stopping at all, for a model trained on 1500 observations only. Early stopping comes useful for particularly heavy models, but in this you shouldn't need it.

I think the size of each mini-batch is very small. That would make gradient descend very noisy, please consider increasing its size, or using full batch training as well.

Additionally, I think you have implemented a Network that is too big. Your input is very small, therefore you don't need to expand its signal on layers of size 64. There are too many nodes that are trying to "learn" not many things, IMHO. A good architecture could be:

model = Sequential()
model.add(Dense(6, input_dim=6, activation='relu'))
model.add(Dense(6, activation='relu'))
model.add(Dense(1, activation=None))

That would make your model faster to train, and ensure that each node is learning relevant features of your data.

I would also change the output layer. Since you want to predict an outcome, you need an output node with no activation (i.e. linear activation). That is mandatory for regression tasks with unbounded output.

Additional things you can try are:

change dropout levels (but for such a small network it might not be needed at all),
try regularization techniques, such as Batchnorm, L1 - L2 regularization, different weights initialization... you name it,
try alternative activation functions.

Also, for medium-to-small size datasets, it is possible for other ML algorithms such as Random Forests or SVMs to beat Neural Networks in performance.


Cross-Validation in Anomaly Detection with Labelled Data I am working on a project where I train anomaly detection algorithms Isolation Forest and Auto-Encoder. My data is labelled so I have the ground truth but the nature of the problem requires unsupervised/semi-supervised anomaly detection approach rather than simple classification. Thus I will use the labels for validation only.
Since I will not train the model with the labels, unlike supervised learning where I would have X_train, X_test, y_train and y_test, what is the right approach for model validation here?
If this were supervised learning, I would split data into 3 parts: train, CV and test, doing K-Fold CV. But now I feel like I can simply divide my data into 2: train and test and simply fit all of the train data, predict and tune the models according to. Then finally, predict the test data.
So my question is, should I include some kind of CV in this model? What is the right way here?

After thinking thoroughly and having worked on the problem I mentioned, I will respond to my own question.
In supervised learning, we divide the data into three parts, namely train, dev and test sets. We use dev/validation test to see how our fitted model does in unseen data. What is unseen in this case, is the target variable, y_test. During training, y_train is seen by the model while y_test is kept unseen. In order to use every part of train and dev data, and eliminate the effect of chance, we do it K-Fold.
In case of unsupervised learning, the model does not see any y_train because only X_train is fitted. Since the model is not exposed to the labels, we can simply go with train and test set, rather than train, dev and test. In this scheme no K-Fold CV is required either because we can simply predict for all of the train + dev data at once.
Hence, it makes sense to use the labels of X_train, y_train, for validation during model development, and it is not required to use K-Fold CV.


Association Rule Learning for Home Electricity or Water Data? My university has a building on campus where electrical usage and water usage is monitored down to the second.  Many electrical loads and water usageis tracked and stored in a database.  In the water dataset, the column headers are the location, and the rows contain the gallons used (per second).  In the electrical dataset, the power consumption (in Watts) is tracked in the rows and the load type is the column header.

My general problem is as follows: - how could I use the data from this highly monitored building, to benefit the university and give them insights into buildings that aren't monitored like this one?

My data is of the time-series, so I am trying to think of certain clustering or classification algorithms that could help me extract actionable data from it, and perhaps even association rule learning to find relations between the water and electricity data?

Thanks for any insights!

In order to draw conclusions from this one building's data on other buildings on the campus, you would need a set of parameters shared between the building you know about and the buildings you want to predict for. For example, if the monitored building is of a male dormitory type, perhaps you can infer something for male dormitories; if the building is at location X, perhaps you can infer something for buildings at or close to location X. You'd need as many as possible of such additional columns to extrapolate knowledge to other buildings.

As for visualizing/predicting resource usage of that single building, I suggest you try data mining tool Orange, which I find phenomenal for prototyping machine learning workflows. It also has a time series add-on, which I can't vouch for, but it does seem to have some forecast capabilities.


What is the possible range of SVR parameters range? I'm working on a regression problem. While tunning the Parameters of SVR I got the following values c=100, gamma= 10 and epsilon =100. For which I got 95 percent r-square. My question is what is the theoretical range of these parameters values.?

I support vector regression the inverse regularization parameter $C$ can be selected from the interval $[0,\infty)$. In which $C=0$ means that we are very heavily regularizing and $C\to \infty$ no regularization.

The parameter $\varepsilon$ is also from the interval $[0,\infty)$. In which $\varepsilon=0$ forces the regression to penalize every point that is not exactly on the regression line. Whereas $\varepsilon > 0$ allows an indifference margin around the regression in which a deviation will not be counted as an error.

Additionally, there are slack variables of $\xi\geq 0$ and $\hat{\xi}\geq 0$. These are zero if a point is inside the indifference margin. If a data point lies above and outside the indifference margin we will have $\xi>0$ and if a data point lies below and outside the indifference margin we will have $\hat{\xi}<0$.

I think you mean the form parameter of your radial basis function when you talk about $\gamma$. If we have

$$\varphi(\boldsymbol{x}_i,\boldsymbol{x}_j|\gamma)=\exp\left[-\gamma||\boldsymbol{x}_i-\boldsymbol{x}_j||^2\right]$$

then $\gamma \in (0,\infty)$ (Note the minus sign in front of $\gamma$). For $\gamma \to 0$ we make the kernel flatter as $\varphi \approx 1$. If $\gamma \to \infty$ we will get a very peaked kernel. Which will be 1 when $\boldsymbol{x}_i\approx \boldsymbol{x}_i$ and almost zero everywhere else.

You should also have a look at the documentation for the implementation of these parameters. It might happen that these parameters are not implemented as you think (see note on $\gamma$).


Different R Markdown in same document for each output type In R Markdown, I would like to be able to conditionally include certain chunks depending on what I am outputting to. For example if I am knitting a PDF I would like to include a small bit of LaTeX to do a page break, but if I am knitting HTML from the same document, I would like to skip that, or perhaps include something else. So how can a document "know" where it is going, and selectively include things?

I was able to get different inclusions and rendering with the following document:

---
output: html_document
---

```{r, results='asis', echo=FALSE}
library(knitr)
pandocto <- opts_knit$get("rmarkdown.pandoc.to")
if (pandocto == "docx") {
  cat("## Word Heading\n\nHello world")
} else if (pandocto == "html") {
  cat("## HTML Heading\n\nHello world-wide-web")
} else if (pandocto == "latex") {
  cat("## LaTeX/PDF Heading\n\nHeLlO wOrLd")
}
```

```{r}
str(opts_knit$get()
```

If you vary the head YAML with what you need, perhaps one of:

output: html_document
output: pdf_document
output: word_document

you can see the difference. The tricks are knowing that rmarkdown tells you the format to which pandoc is rendering, and the use of results='asis' in the block header to capture the output as raw-markdown (or latex, as appropriate).

I included the second block (str(...)) so you can see what else is "known" during document compilation/rendering.


How to change plot size in nltk.plot() I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot:

import nltk
from nltk.corpus import inaugural
cfd = nltk.ConditionalFreqDist(
            (target, fileid[:4]) # "[:4]" slices only the years of the speeches
            for fileid in inaugural.fileids()
            for word in inaugural.words(fileid)
            for target in ["liberty", "equality", "brotherhood"]
            if word.lower().startswith(target))

Because it gets very crowded otherwise:

cfd.plot(title="French ideals in US-American speeches through time")

the __doc__ doesn't seem to mention it:

print(cfd.plot.__doc__)

        Plot the given samples from the conditional frequency distribution.
        For a cumulative plot, specify cumulative=True.
        (Requires Matplotlib to be installed.)

        :param samples: The samples to plot
        :type samples: list
        :param title: The title for the graph
        :type title: str
        :param conditions: The conditions to plot (default is all)
        :type conditions: list

And I think there is nothing on it in the NLTK documentation, but also not in the matplotlib documentation (where I figured the plot functionality comes from):

http://www.nltk.org/py-modindex.html#cap-p
http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.plot

(sorry for the code-block links, I'm new here and can't post more than 2 links but still felt like making it easier for someone reading this :)

I'd be glad if someone could point me into the right direction! Thanks!

Try something like this:

import matplotlib.pyplot as plt
plt.figure(figsize=(30, 20))  # the size you want

# your code goes here


Is there a dataset with news articles and their headlines? I need a set of news headlines and articles to help me in a project on automatic summarization. Is there such a dataset or something similar?

The most widely used ones in text summarization research is the DUC dataset. If you see a paper using dataset "DUC 2015" or "DUC 2016" that's from here.

I have also personally used the Reuters arcihve. You just need to download each article with wget or something similar. See also here.

The CNN / DailyMail dataset is also widely used in summarization especially in recent years, although it labels itself as a Q&A dataset.


Analyzing Sentiments of Financial News related to a Company I'm trying to build a model which gives me the sentiments of the Financial News related to a company and I want to predict the stock price accordingly. But the major problem that I'm facing is understanding the news for the counterpart.

Let's say I have a news headline as "Total is under pressure and CEO has confirmed that they are planning to sell stakes soon". My model will always give negative sentiment which is correct, but this might actually be a good news for Shell or Exxon, lets say. But how do I tell my model that it is actually a good news for Shell.

Also is there any good process to understand which news relate to which companies and how I can calculate the sentiment accordingly. Maybe a good labelled data-set or pre-trained architecture which might help me out?

P.S. Most important, is there any labelled dataset or any other pre-trained architecture which I can use to calculate the sentiments of financial news?

But how do I tell my model that it is actually a good news for Shell.

You would probably need a complex semantic analysis from which you could build a knowledge graph, from which you could extract the logical implications of one news for other entities not mentioned in the news.

  P.S. Most important, is there any labelled dataset or any other pre-trained architecture which I can use to calculate the sentiments of financial news?

You could ask on https://opendata.stackexchange.com/.
As far as I know, really good resources (data and algorithms) for financial applications are very expensive and usually kept secret by the biggest financial institutions.


Strange out of memory while loading lots of pictures before input CNN for deep learning I’m using the TFlearn, and want to classify pictures to two category. But the strange out of memory while loading lots of pictures before input CNN for deep learning. The RAM is 64 G in my deep learning box. I using 482426 pictures(resize to 224 x 224) for training (241213 are A category, the other 241213 are B category ). The process shows the '%MEM: 98.4' whenever the I load these images, so that I can't complete the process. I would like to know how can I edit my code for this situation?

imgs = []

for filename in glob.glob(Learning_Data_Path+"/Training/A/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img)
    imgs.append(img_arr)

for filename in glob.glob(Learning_Data_Path+"/Training/B/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img)
    imgs.append(img_arr)

imgs = np.array(imgs)

y_data = np.r_[np.c_[np.ones(Training_A_num), np.zeros(Training_A_num)],np.c_[np.zeros(Training_B_num), np.ones(Training_B_num)]

You should load images in batches. 480K of 224*224 sized images is too large for any machine let alone a 64GB one.

You can use image_generators from keras or tf.keras

https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing

https://keras.io/preprocessing/image/