How should the bias be initialized and regularized? I've read a couple of papers about kernel initialization and many papers mention that they use L2 regularization of the kernel (often with $\lambda = 0.0001$).

Does anybody do something different than initializing the bias with constant zero and not regularizing it?

Kernel initialization papers

Mishkin and Matas: All you need is a good init
Xavier Glorot and Yoshua Bengio: Understanding the difficulty of training deep feedforward neural networks
He et al: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

From the Stanford CS231N Notes:

Initializing the biases. It is possible and common to initialize the
biases to be zero, since the asymmetry breaking is provided by the
small random numbers in the weights. For ReLU non-linearities, some
people like to use small constant value such as 0.01 for all biases
because this ensures that all ReLU units fire in the beginning and
therefore obtain and propagate some gradient. However, it is not clear
if this provides a consistent improvement (in fact some results seem
to indicate that this performs worse) and it is more common to simply
use 0 bias initialization.

In LSTMs it's common to initialize the biases to 1 - see for example.


Purpose of validation data NN Aside from using validation data to tune the hyperparameters is there any other benefit to including validation data to the model?

All I ever read about is it being used to tune hyperparameters and check for overfitting. Is the checking for overfitting separate from tuning the hyperparameters?

Training: Tune the parameters (weights and biases)
Validation: Tune hyperparameters
Test: Evaluate the model

So, if we are NOT tuning the hyperparameters, the validation set is pointless?

The whole idea of a validation set is that the model does not know about this data, so you can get an unbiased estimation of the model's performance. Then based on this unbiased estimation you find the best parameters for the model. The problem is, that finding the hyperparameters is in itself a way of training your model. So with optimized hyperparameters the model starts to overfit on the validation set. That is why to check the real accuracy of the model you actually need a different part of data, that your model never saw. This is the testing part of data.

Therefore in your case, when you do not have any hyperparameters, you can just use the division on train and test.

If you want a higher accuracy of estimating your model's accuracy than it is better to use Cross Validation instead of just dividing on train and test. Neural networks usually do not do full cross validation because it means increasing the computation time several times (5 for 5-fold cross validation).

With hyperparameters the ideal way is to do a double cross validation. One for validation set and one for test set. This is too expensive computationally so it is only used on very simple models, like a ridge regression.

Also very few models, that I know of, do not have hyperparameters. And usually those that do not have hyperparameters perform poorly compared to the ones that have. Ridge regression is often better than linear regression. Neural networks with variable number of layers perform better than fully automated neural networks.


Cluster elements that appear in the same lists Suppose I have a multitude of sets with (unordered) combinations of elements and I want to determine which elements tend to appear together.

For example

Given the following sets:

{a,e,g}
{a,e,h}
{a,e,i}
{b,f,j}
{b,f,k}
{b,f,l}
{d,c,m}
{d,c,n}
{d,c,o}

Element-pairs where both elements tend to occur in the same sets will have lower distance:

# Low-distance pairs:
{a,e}, {b,f}, {d,c}

# Medium-distance pairs
{a,g}, {b,j}, {d,m}, ...

# High-distance pairs:
{g,h}, {j,n}, {b,f}, ...

Currently

I'm implementing DBSCAN with a custom distance metric. I use the following distance metric between two elements:

d(a,b) = 1 - numsets(a, b) / (numsets(a,!b) + numsets(b,!a))

Where d(a,b) denote the distance between elements a and b. While numsets denotes how many sets fulfill some conditions:

numsets(a, b) - the number of sets that contain a and b
numsets(a,!b) - the number of sets that contain a but not b

This solution should achieve the goal however it's not a pretty solution and I couldn't find this problem on SE. In terms of solving the problem, is there a more sensible distance metric? In terms of implementation, is there a nicer way to do this?

That is a data mining problem, specifically affinity analysis.
One common method to solve it is the Apriori algorithm.


Is there any proven disadvantage of transfer learning for CNNs? Suppose I know that I want to use a ResNet-101 architecture for my specific problem. There are ReseNet-101 models trained on ImageNet.

Is there any disadvantage of using those pre-trained models and just resetting the last (few) layers to match the specific number of classes or should that be the default option?

Please don't simply post your gut feeling... I have a gut feeling as well, but I want to know.

Based on my experience, not just for ImageNet, if you have enough data it's better to train your network from scratch. There are numerous reasons that I can explain why.

First of all, I don't know whether you have had this experience or not but I've trained complicated CNNs neworks with over 25 million parameters. After reaching 95% accuracy, after convergence, I changed the learning rate to a bit bigger number to find another probable local minimum. I've not found any answer till today but whenever I did so, my accuracy decreased signifincantly all the time and it never imporoved even after more than thosands of epochs.
The other problem is that whenver you use transfer learning, your training data should have two options. First of all, the distribution of the training data which your pre-trianed model has used should be like the data that you are going to face during test time or at least don't vary too much. Second, the number of training data for transfer learning should be in a way that you don't overfit the model. If you have a few number of labeled training data, and your typical pre-trained model, like ResNet has millions of parameters, you have to be aware of overfitting by choosing appropriate metrics for evaluation and good testing data which represent the real distribution of the real population.
Next problem is that you can not remove layers with confidence to reduce the number of parameters. Basically the number of layers, as you yourself know, is a hyper-parameter that there is no consensus on how to be chosen. If you remove the convolutional layers from the first layers, again based on experience, you won't have good learning because of the nature of the architecture which finds low level features. Furthermore, if you remove first layers you will have problem for your dense layers, because the number of trainable parameters changes in this way. Densely connected layers and deep convolutional layers can be good points for reduction but it may take time to find how many layers and neurons to be diminished inorder not to overfit.

If you don't have enough data and there is already a pre-trained model, you can do something that can help you. I divide my answer to two parts for this situation:

The pre-trained model does not have common labels for most of its classes

If this is the case, you can forget all the fully connected layers and replace new  ones. Basically what they do is just classifying the features that the network has found to reduce the error rate. About the convolutional layers you have to consider two major points:

Pooling layers try to sum up the information in a local neighbourhood and make a higher representation of the inputs. Suppose that the inputs of a pooling layer have nose, eyes, eybrows and so on. Your pooling layers somehow attempt to check whether they exist in a neighbourhood or not. Consequently, convolutional layers after pooling layers usually keep information which may be irrelevant to your task. There is a downside for this interpretation. The information may be distributed among different activation maps as Jason Yosinski et al. have investigated this nature at Understanding neural networks through deep visualization where you can read One of the most interesting conclusions so far has been that representations on some layers seem to be surprisingly local. Instead of finding distributed representations on all layers, we see, for example, detectors for text, flowers, fruit, and faces on conv4 and conv5. These conclusions can be drawn either from the live visualization or the optimized images (or, best, by using both in concert) and suggest several directions for
future research and These visualizations suggest that further study into the exact nature of learned representations—whether they are local to a single channel or distributed across several .... A partial solution can be keeping the first layers which find low level features that are usually shared among different data distributions and removing deeper convolutional layers which find higher abstractions.
As stated, the main problem of convolutional layers is that the information they find may be distributed among different activation maps. Consequently, you can not be sure by removing a layer you can have better performance or not.

The pre-trained model has some common classes with your labels

If this is the case you can visualize the activations using the techniques described here. They have shown that although human face is not a label in ImageNet, some of internal activation maps get activated by face. This observation can be seen for other labels too. For instance, netwroks for deciding a scene contains a car or not are usually sensetive to roads and trees. The image below shows that which parts of ouputs get activated for which parts of images. This can help you when you don't have enough data and you have to use transfer learning.

Based on the answer here The standard classification setting is an input distribution $p(X)$ and a label distribution $p(Y|X)$. Domain adaptation: when $p(X)$ changes between training and test. Transfer learning: when $p(Y|X)$ changes between training and test. In other words, in DA the input distribution changes but the labels remain the same; in TL, the input distributions stay the same, but the labels change. Consequently, domain adaption problems also can be considered for the mentioned solutions.


Effect of removing duplicates on Random Forest Regression I have a dataset with several million samples that have 5 features and 1 target, which I am using for a regression model.  With very large sample counts some models (like Random Forests) become very large (several GB when pickled).
These data often have duplicates or near duplicates - these are real observations - but the measured values are just coincidentally identical (a consequence of the limited input range and precision of the instruments).
What is the effect (in theory) or removing duplicates on model accuracy?

I agree with the previous answer about your specific question: your model is likely to be less accurate if you just remove training data.
Moreover:

With very large sample counts some models (like Random Forests) become very large (several GB when pickled).

It sounds like the size of your final model is the reason why you're considering pruning your dataset. Assuming you are using scikit-learn, I'd recommend looking at the documentation and the parameters that you can tweak that impact your final model size.
Specifically, parameters like n_estimators, max_depth, min_samples_split, and min_samples_leaf all have default values that tend to generate large models. For example, max_depth is set to None and therefore (as per the docs) "nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples." If you experiment with these you are likely to be able to generate much smaller models, in terms of size-on-disk, without needing to prune your dataset.


How are boosted decision stumps different from a decision tree? For example, taking the image from sebastian raschkas post "Machine Learning FAQ":

I would expect a very similar (if not exactly the same) result for a decision tree: Given only two features, it finds the optimal feature (and value for that feature) to split the classes. Then, the decision tree does the same for each child considering only the data which arrives in the child. Of course, boosting considers all the data again, but at least in the given sample it leads to exactly the same decision boundary. Could you make an example where a decision tree would have a different decision boundary on the same training set than boosted decision stumps?

I have the intuition that boosted decision stumps are less likely to overfit because the base classifier is so simple, but I couldn't exactly pin point why.

The decision boundary in (4) from your example is already different from a decision tree because a decision tree would not have the orange piece in the top right corner.

After step (1), a decision tree would only operate on the bottom orange part since the top blue part is already perfectly separated. The top blue part would be left unchanged.

The boosted stumps, however,  operate (as you mentioned) on the full dataset again, which can lead to different results.


Practical limitations of machine learning Working on some applied machine learning problems, I've started to encouter some practical difficulties. Those difficulties relate to - but are not limited to - convergence of the learning process, stability trough recalibration, explainability, stability of the explainability trough recalibration. Those problems are a bit difficult to handle, notably because:

They are not really dealt with in introductory theoretical books. (However, there are some advanced resources to deal with them individually.)

They are mostly specific to advanced ML models, compared to more standard approaches.

You only get aware of them once you stumble upon them by doing some very specific tests.

I would be interested in some serious sources on the main practical problems related to machine learning (not necessarily how to deal with them). Both as a checklist of pitfalls to avoid and as an authoritative argument if I have to formulate some concerns about any future ML solution. Can you help me with any sources?

How does one even start describing these practical limitations?

Most authoritative source on the main practical (and theoretical) problems related to machine learning is formal framework describing it. Mathematics.

Right here we have a problem, even tough you look at all of these ML algorithms and you conclude its math, we do not explain the whole process of a ML problem. I cant find a citation but Bengio or someone said that computer science is hard science because you have mathematical underpinning for everything, and ML is more of a soft science where you learn by trying, but only after you tried. (Not always the case ofcourse, thats why everyone is researching ML now to give it some structure)

Take a simple neural network. You know its matrix multiplication, backpropagation bla bla. Great. But what are the topological properties of certain architecture, convergence criteria, what functions can it approximate. Some of these questions are known and/or are being researched. Lets look at this one for example: What functions can we guarantee (in a formal sense) that a NN can approximate and under what conditions?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources

But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

Conclusion there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.

Concluding your question There are best practices that constantly evolve and you can get a check-list that just isnt relevant (Take computer vision problems, check list 2 years ago isnt same as today). BUT what is constant and remains the best authority is formal under pinnings such as mathematics. It can tell you directly the "best practice" of when it would be futile to even try to approximate a function.


LSTM multivariate time series prediction is performancing very poor I have been working on Multivariate time series in Python. The method I have used is LSTM. But the prediction is looking very bad. even my RMSE and MAE value is so high. RMSE=18.089, MAE=14.907. It is no good for prediction. I want to predict such a way that the error should remain in 0 to 1. The model I have applied to predict is-
from keras.models import Sequential
from keras.layers import Dense,LSTM,Lambda
from keras.layers import Dropout
#create LSTM model
model=keras.models.Sequential([

    keras.layers.LSTM(75,return_sequences=True,input_shape=(xtrain.shape[1],xtrain.shape[2])),
    keras.layers.LSTM(30,return_sequences=True),
    keras.layers.LSTM(10,return_sequences=True),
    keras.layers.LSTM(10),
    keras.layers.Dropout(0.25),
    keras.layers.Dense(1)

])

model.compile(loss='mae',optimizer='adam')
history=model.fit(xtrain, ytrain,epochs=100,validation_data=(xtest, ytest),batch_size=64)

But the overall prediction is not good. I have split the train and test data for 60% and 40% respectively.

How I can improve my prediction?

Several things you can do here:

try normalizing the time series
try using another loss: loss='mse'
maybe do a 80-20 split (make sure that your test data comes chronologically after the training data to avoid forward looking bias)


How can I determine best performing groups in a series Not sure if this is right place for this question, but taking an example of the following trend, how can I determine that groups A and B are the areas where the result has been best?

I have already tried by doing an average and finding all those above the average. But finding out a group and driving a conclusion is tricky part. How can I do this?

Not sure to understand what you are asking. Your purpose is to find the only one best group ?
From the picture you provided, two things are unclear to me:

A and B areas don't have the same width, how are you defining your width ?
A and B are covering areas where the signal looks exactly the same. Is it that for the sake of the example you copy-pasted some fake data or does it have to do with what you are finally trying to do ?

If your width is fixed then you should be able to compute a series of local averages and pick out the best one.


Pandas Query Optimization On Multiple Columns My training dataset is around 5 MB and test dataset is of the same size. I have run some simulations over the whole dataset couple of times. However, in a latter solution, I ran queries on two columns (say A and B). Now, in the calculation, for each row in the test dataset, I have to get the result of the following query. This time, the program takes forever. I can optimize this on my own writing some extra code (sorting will help in my case and in fact I can skip pandas altogether if I want to), but I wanted to check if there is a better built-in solution should I ever require that (also, I don't want to if I don't have to). The query is like this:

def f(A, B, x, y):
    data = df.loc[df[A].isin([x]) & df.B.isin([y])]
    return len(data)

Here, df is the DataFrame (training dataset). For each row in test dataset, data find all rows where column A has value x and B has value y. As far as I know, isin is slightly faster, so I used it. However, I think this is why it does not run within a second like it did before because each time I run this query, it needs a lot of time. What I have to do next is something like this this.

for i in range(0,l):
    ret = f(A, B, test[A].values[i],test[B].values[i])

Clearly, in every loop, it will calculate the result of the query. Other than this, rest of the functions in my code are mathematical that do not need much calculation.

Let me know if the problem is not clear since I did not use direct codes (replaced it with dummy variables and other lines do not have anything that has high complexity).

Short answer
There are far faster ways vectorized ways to do this than brute-force loops. One particular option while remaining Pandas-level would be
(tra_df.groupby(tra_df.columns.tolist())
       .size()
       .reindex(tst_df.values.T.tolist(), fill_value=0)

This should offer you an enormous performance boost, which could be further improved with a NumPy vectorized solution, depending on what you're satisfied with.

Long answer
Suppose that your training and test data looked something like this.
import pandas as pd; import numpy as np; from numpy.random import RandomState

rnd_st = RandomState(8675309)

tra_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3),
                           B=rnd_st.randint(0, 10, 10**3)))
tst_df = pd.DataFrame(dict(A=rnd_st.randint(0, 10, 10**3),
                           B=rnd_st.randint(0, 10, 10**3)))

First of all, you aren't really using Pandas correctly with
def f(A, B, x, y):
    data = df.loc[df[A].isin([x]) & df.B.isin([y])]
    return len(data)

isin isn't intended to match against single values like this, but rather a list or set of values when you don't have another option. In your case, you can achieve the same boolean index directly with
(df.A == x) & (df.B == y)

Furthermore, there's no point indexing your DataFrame again with df.loc only to take the length of it... you might as well just use sum directly on the boolean vector to count the matches. Putting this together and using apply over the test DataFrame, you can achieve your same output with
tst_df.apply(lambda x: ((tra_df.A == x.A) & (tra_df.B == x.B)).sum(), axis=1)

But this is still very slow - wrapping the above approach in a function and comparing it to your original approach on the sample data
%timeit your_way(tra_df, test_df)
1 loops, best of 3: 819 ms per loop
%timeit direct_er_way(tra_df, test_df)
1 loops, best of 3: 673 ms per loop

Much faster would be to use groupby and then reindex, as instead of brute-force looping this offers a vectorized solution where we are effectively hashing the counts.
(tra_df.groupby(tra_df.columns.tolist())
       .size()
       .reindex(tst_df.values.T.tolist(), fill_value=0)

Benchmarking,
 %timeit groupby_way()
100 loops, best of 3: 3.04 ms per loop
I'm not that well-versed in NumPy, but I can safely assume that were this function still not fast enough to meet your needs then a NumPy vectorized solution avoiding some of the overhead would be the next step.


Why is n-grams language independent? I don't understand how n-grams are language independent. I've read that by using character n-grams of a word than the word itself as dimensions of a vector space model, we can skip the language-dependent pre-processing such as stemming and stop word removal.

Can someone please provide reasoning for this?

The way I read this, you are actually asking two questions:

How do character n-grams help to encode knowledge that is often encoded with the help of techniques such as stemming ?
Why are n-grams language independent? I'm not totally sure on what you mean by this one, but I'll take a stab at it

Character N-grams

Some languages (most languages that I know of, but some more than others) have grammatic rules that change the morphology of a word: one house, two houses. In a vanilla Vector Space Model house and houses are not identical and form two dimensions of your model. As if they were as different as house and apple.

We know that English applies these morphological operations on the words and we can counter that by bending the words back into their 'stem'.

For instance, the Snowball stemmer would bend them back to a token that does collide:

  House  -> Hous
  Houses -> Hous

Note: Hous is not what we think of as a stem, hence my use of the quotes around 'stem'

N-gramming is basically splitting text into all subsequences sequences of length N. If we apply that (N=3, forgetting about start and stop for simplicity) on our example strings we would get something like:
>

  House -> Hou, ous, use
  Houses -> Hou, ous, use, ses

Note that we end up with 4 new dimensions, 3 of which collide. This reduces our previous found need for stemming. We could argue that it still doesn't collide fully, but then again, we might not want it to.

Language Independence

I don't think n-grams are language independent. Certainly not on the word level: some languages are more regular and some languages are more context-free. Some languages follow a pattern that looks more like adding words to the end as you go, and those languages are probably more suited for modelling with n-grams. And this probably holds too for character-based n-grams.

However, as previously argued, they add flexibility to the V.S.M. when it comes to handling morphology. That morphology would normally require 'manually' encoding knowledge about the language at hand. With this more or less out of the way your system needs less configuration to the language at hand, which makes it less dependent on the language.


Some confusions on Model selection using cross-validation approach https://stats.stackexchange.com/questions/11602/training-with-the-full-dataset-after-cross-validation explains the procedure and the importance of doing cross-validation to assess the performance of the method/ classifier. I have few concerns which I could not clearly understand from that answer. It will be immensely helpful if these are clarified.

Consider that I am using the Matlab's fisheriris dataset. The variable meas contains 150 examples and 4 features. The varaible species contains the labels. I have put the data and labels into a variable: Data = [meas species]According to the procedure outlined above,

I have split the data set Data using cvpartition into 60/40 where 60% is the Xtrain and 40% is a separate Xtest data subsets.
Using Xtrain I perform k fold cross-valiation and inside each fold I validate the model using the indices from Xtrain. This loop is used to tune the hyperparameters of the model. I never use Xtest in selecting the hyperparameters. Is my understanding correct?

Confusion 1) The answer in the link says

You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.

"use the full dataset to produce your final model as the more data you use the more likely it is to generalise well"

I am a bit confused on what  dataset  and  whole set  are we referring to and how is building the final model by using cross-validation on the whole set using the selected hyper-parameters different from building the classifier on the whole dataset using the hyper-parameters?

I wanted to verify if my understanding of this part is correct or not. Does this statement mean that using  the cross-validated hyper-parameters obtained using the Xtrain, should the classifier be build by re-training on the Xtrain subset or on  Data ?

Should my final model be the one from Data?

Confusion 2) What is the role of the unseen Xtest data set? In papers is the performance reported on the Data or on the untouched Xtest ?

I'm not exactly sure of cvpartition's routine, so I'll try to provide a more generalised answer.

"Whole set" v "full data set"

These are the same in this instance. Model hyperparameter tuning can be done by feeding the full / whole / complete data set into a cross validation process. Inside this cross validation, the data is split into k folds, where (k - 1) folds are used to build a model, and the remaining fold is used to test the model (since the remaining data is essentially unseen to the model). This is repeated k times; results can be averaged and standard deviation calculated.

Seen v unseen data (a.k.a. modelling and hold out samples)

This is essentially a variation on the cross validation approach above, except the split is done only once: the full data is split into two subsets -  Xtrain and Xtest in your case. In this approach, you would build a model using Xtrain only and test it on Xtest (there is no repetition).

So, what's the difference?

Both approaches attempt to do the same thing - somehow validate a model's performance and ability to generalise on unseen data. Cross validation is arguably the stronger technique, especially on smaller data sets.

Once you've successfully parameterised your model (using either approach), building your model on the whole data set is advisable since more data > less data.


How can I create a VM instance with GPUs on Google Cloud Platform? How can I create a VM instance with GPUs on Google Cloud Platform?
When I go to https://console.cloud.google.com/compute -> CREATE INSTANCE, I only see CPUs and no GPUs, as shown in the video below. I did select a region+zone that is supposed to have GPUs according to https://cloud.google.com/compute/docs/gpus

https://cloud.google.com/compute/docs/gpus (mirror):

I see that some VMs from the marketplace comes with GPUs but I'd prefer to configure the VM myself.


I am trying to make a Classifier using Machine learning to detect malwares,am I headed in the right direction? I am trying to make a malicious software detector using machine learning.I have obtained a dataset from internet which containes md5 hash and other features of pe files.I fed this databse to weka to find out the most important features,which came out to be only one(static_prio).I tried running the K-NN algorithm on the data using only one feature(static_prio) and I got 99.82% accuracy but I am a little confused about my approach here.I am naive with machine learning so please help by putting me on the right track.Thank you.

It is not clear in your post whether or not you are using training/test sets so I will answer in this way:

The first thing to do if you want to validate your results is to cut your set into a training set and a validation set. This way you train the K-NN method on your training set, and you use the trained classifier on the validation set. Then you can monitor the validation error, and the training error.

In the best case scenario, it should look like this :

The aim is to obtain a model that generalizes well, which means being able to accurately predict the class a sample belongs to, when it has never seen the sample before.


How to scale an array of signed integers to range from 0 to 1? I'm using Brain to train a neural network on a feature set that includes both positive and negative values. But Brain requires input values between 0 and 1. What's the best way to normalize my data?

This is called unity-based normalization. If you have a vector $X$, you can obtain a normalized version of it, say $Z$, by doing:

$$Z = \frac{X - \min(X)}{\max(X) - \min(X)}$$


Applying machine learning algorithms to subset of attributes in dataframe I have this huge mixed data set consisting of both numerical and categorical attributes which upon OneHotEncoding results into a data set with very high dimensionality.

Is it wise to apply machine learning algorithms like K-means clustering, dimensionality reduction and regression on subsets of data set? For example applying K-means clustering to numerical columns first and join the result with categorical data set later.

Applying a machine learning algorithm on only a subset of the data and including other subsets later does not allow the algorithm to assess the importance of each attribute equally.

For example, say you have a data set called A, which has subsets B and C. Without loss of genearality, if you fit a model ('apply an algorithm') on subset B, and then include subset C later, then you're saying 'given subset B is already in the model, assess the impact of including subset C'. Instead, if you apply the entire algorithm on the entire data set (A), then you're allowing the algorithm to discover which features are most important for the desired outcome.

That being said, it may be wise to process the different elements of your data set differently. That is, categorical covariates may be modelled differently from continuous covariates. If you're using something like a feed-forward neural network, then it's not a big deal, but if you're using a more traditional statistical model you may need to take that into account. For example, in R, you need to specify that a categorical covariate is in fact a 'factor' variable.


When exactly should I use weighted loss? Do I have to use it in any case when the class distribution is imbalance
(Train: class A:10%, B:90% and Test: class A:10%, B:90%)

or when it is different
(Train: class A:10%, B:90% and Test: class A:50%, B:50%)

Thanks!

As with almost every problem in real life, regarding how to handle class imbalance there is no golden rule to follow for any case and systematically obtain good results, only some recipes that you can try.

When your class distribution is imbalanced (Train: class A:10%, B:90% and Test: class A:10%, B:90%)), one recipe that can improve your results is using reweighting to give more importance to the minority class. There are, however, other approaches, like oversampling the minority class, or subsampling the majority class. Another option would be to use a loss function that is tolerant to class imbalances.

Having a different distribution between the training and test sets (Train: class A:10%, B:90% and Test: class A:50%, B:50%) can also be handled with the approaches above. However, you should think about why you have such a difference: while the existence of class imbalance in the training and validation data can be caused by the distribution of the data in real life, the differences in class imbalance between training and test data is a consequence of the creation process of those datasets, either caused by a conscious decision or by chance. Here are two examples that could lead to this situation:

In this case, the 50%/50% ratio reflects the reality of the data that will be received in the production environment, so we decided to define a test set with that ratio and using the rest of the data for training. It happened that the rest of the data had this imbalance. In this case, you should mitigate the problem of the data imbalance in the training data, maybe with some techniques like reweighting or resampling.
In this case, the 90%/10% ratio reflects the reality of the data that will be received in the production environment. However, we created the test set and the data set separately, maybe with less quality control for the training data. This led the team in charge of the training data to make some decisions about the data collection and the team in charge of the test data to take different decisions. The differences in the class imbalance between training and test data and casual, not driven by any conscious decision. In this case, this is a problem in how you have created your datasets. What you have to do in those cases, is to re-split your data to have the same distribution in both sets, trying to have a test set that is as close as possible to the real data and, if possible, having the same distribution in the training and test sets.

In all these cases, nevertheless, we should take into account the function we use to measure how good our model is. If the real data has a strong class imbalance, we should not choose the accuracy but something like the AUC.


Understanding the "Wide" part of Google's wide and deep Google's wide and deep recommender model sounds really cool, but I'm struggling to believe I'm grasping the wide section right so wanted to check my understanding.

Their paper says the following:

  The wide component consists
  of  the  cross-product  transformation  of  user  installed  apps
  and impression apps

  Each example
  corresponds to one impression

Let's say we have 5 apps, A through E. My understanding is that the cross-product transformation would represent that as 20 columns, representing each possible combination of installed and impressed app (making 25, but then presumably the 5 "matching" cross-products like and(installed=App_A, impressed=App_A) would be removed because presumably Google is smart enough not to impress Apps the user already has). Let's also say we have 3 Users, called X - Z. X has installed apps A and C, and is shown app B and D. Y has installed App B and is shown A and E. Z has installed apps A, C and D and is shown apps B and E. With that dataset, the cross-product transformation should look (I think) like this:

My question is; is my understanding of the transformation there correct? If so that's going to be one gigantic matrix in fairly short order, particularly given they have over a billion users and a million different apps.

Indeed you will have a large and very sparse matrix. The two important concepts in this are:

feature cross: you have two categorical features (here impressed and installed) and you do a carthesian product of both to create a new feature (as you did here). The issue is that crosses generate sparse matrices (as you can see in your example)

Neural netowrks perform well with dense, correlated features while linear models work better with sparse and low correlated feartures. Your cross features are sparse with low correlation, which leads to the second concept:

wide and deep network: since your dataset is a mix between dense features (continuous features) and sparse (cross) you can separate your network in two: the sparse will be directly sent to your output and thus treated as they would be in a linear model while your dense features will go through several hidden layers before going to the output.

The wide part behaves just like a linear model (you can also use sparse matrix calculus to speed it up) and the deep part like a traditional neural network and you get the best of both worlds. Given the size of the dataset (billion users x million apps), treating part of the data as a sparse input will speed up training and inference.

I went a bit further than what you asked but I figured it could be of help.


Does increasing kernel size in a CNN result in higher accuracy on the training set? In a convolutional neural network, does increasing the size of kernel always result in better training set accuracy? For example, if I use 5x5 kernels in a CNN instead of 3x3 ones, will it always generate better training accuracy?

Increasing kernel size means effectively increasing the total number of parameters. So, it is expected that the model has a higher complexity to address a given problem. So it should perform better at least for a particular training set. Or will it be harder to learn a bigger kernel?

I am not concerned about the validation set accuracy here.

I'd say there is no direct relation between the kernel size and the accuracy.

If you start using larger kernel you may start loosing details in some smaller features (where 3x3 would detect them better) and in other cases, where your dataset has larger features the 5x5 may start detect features that 3x3 misses.

So, I'd say "no".

Anyway, if you add a second convolutional layer on top of your first you'd start having something that's close to larger features. E.g. your small kernels in the first layer would detect small features but the second layer would detect features which are composed of several features from the previous layer.


Understanding data normalisation So I know that when we have different parameters with different value ranges we have to standardise these values. Also, I read that when a parameter does in fact require higher values then we should not normalise it. However, isn't this always the case? For example, if we have age and salary, isn't salary always going to be greater than age? Wouldn't normalising these values lose possible significant information?

Also, should you always normalise the entire dataset, or can one simply normalise a number of specific columns?

Normalisation is a very blurry concept. It is quite misunderstood most of the times. I will take the specific case of Neural Nets and explain it:

Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes it will lead to more oscillations (intuitionally since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (for error being propagated from a single next layer node). But the weight updates may have a optimal direction in the 4th quadrant, so the weight updates will follow a zig zag pattern which will result in loss of efficiency.
Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. So you fail to train from the start and it continues inf is generally followed by NaN which will continue to eternity. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.


How to best accomplish high speed comparison of like data? I attack this problem frequently with inefficiency because it's always pretty low on the priority list and my clients are resistant to change until things break.  I would like some input on how to speed things up.

I have multiple datasets of information in a SQL database.  The database is vendor-designed, so I have little control over the structure.  It's a sql representation of a class-based structure.  It looks a little bit like this:

Main-class table
 -sub-class table 1
 -sub-class table 2
  -sub-sub-class table
 ...
 -sub-class table n

Each table contains fields for each attribute of the class.  A join exists which contains all of the fields for each of the sub-classes which contains all of the fields in the class table and all of the fields in each parent class' table, joined by a unique identifier.

There are hundreds of classes. which means thousands of views and tens of thousands of columns.

Beyond that, there are multiple datasets, indicated by a field value in the Main-class table.  There is the production dataset, visible to all end users, and there are several other datasets comprised of the most current version of the same data from various integration sources.

Daily, we run jobs that compare the production dataset to the live datasets and based on a set of rules we merge the data, purge the live datasets, then start all over again.  The rules are in place because we might trust one source of data more than another for a particular value of a particular class.

The jobs are essentially a series of SQL statements that go row-by-row through each dataset, and field by field within each row.  The common changes are limited to a handful of fields in each row, but since anything can change we compare each value.

There are 10s of millions of rows of data and in some environments the merge jobs can take longer than 24 hours.  We resolve that problem generally, by throwing more hardware at it, but this isn't a hadoop environment currently so there's a pretty finite limit to what can be done in that regard.

How would you go about scaling a solution to this problem such that there were no limitations?  And how would you go about accomplishing the most efficient data-merge?  (currently it is field by field comparisons... painfully slow).

Can't you create a hash for each classes, and then merge rows by rows, field by field only the classes where the hash changed ? It should be faster if most of the classes don't change..

Or a hash of each rows or maybe columns.. depending on how the data normally change..


What is best practice to feature engineer from prior event counts? Say for example I am building a model to predict a customer churn event from Spotify, with my target being whether a customer churns in the next 90 days.
One feature I might expect could be predictive of this event is customers checking their billing statements online - so I might engineer features for each customer on each training date to encode the information of how many times they have checked their billing statements.
For example, I might create a feature CHECKBILL_CNT_0_10 which is a count of how many times this customer has checked their online bill in the last 10 days, with many of these such features across different time ranges.
I have seen two different styles of how data scientists do this:

CHECKBILL_0_10, CHECKBILL_0_30, CHECKBILL_0_90 ...
CHECKBILL_0_10, CHECKBILL_10_30, CHECKBILL_30_90 ...

Both technically encode the same information; however, I'm wondering if one of these options offers advantages over the other? I'm inclined to think that option 2 would be preferable since the features would be less correlated, & therefore the model might learn more easily, but this is speculative.

You may want to try both options out and see which is better. Feature engineering I think is more like a trial and error (iterative) process.


How to make a linear model with a constant value in R? I'm working on an unassessed homework problem from unpublished course notes of a statistics module from a second year university mathematics course.
I'm trying to plot a 2-parameter full linear model and a 1-parameter reduced linear model for the same data set. I can't figure out how to plot the 1-parameter model; all attempts so far have either given errors or a non-constant slope.
xs <- c(0,1,2)
ys <- c(0,1,1)
Data <- data.frame(x = xs, y = ys)
mf <- lm(y ~ x, data = Data) # model_full
mr <- lm(y = mean(y), data = Data) # model_reduced; this is the bit I can't get to work
attach(Data)
plot(x, y, xlab="x", ylab="y")
abline(mf$coefficients[1], mf$coefficients[2])
abline(mr$coefficients[1], mr$coefficients[2])

To use only a single parameter, which will be a constant, you have to specify the lm formula as follows: y ~ 1. R will then fit the model, where the coefficient for this constant will be equal to the mean of y, see also this stats stackexchange answer.


Uniform detection: a starting point? Given an image of a worker, I need to verify if he/she is wearing the company's uniform.
I tried to Google, but either the search results are about uniform sampling or some object detection tutorials.
Might you help me to point out some starting points, ideally on some similar projects published somewhere?
Many thanks,

You should take out the image embeddings of the uniform (under different conditions) and the employee and match the embeddings. You can also remove the face of employee after detecting it using MTCNN.


Don't understand classification equation for hard margin SVM I am trying to get a grasp of hard margin SVMs. In the lecture I am watching the professor talks about a classification equation which when a positive sample is input, returns a value of $1$ or more; and when a negative sample is input, returns a value of $-1$ or less. The graph below shows the vector $\overrightarrow{w}$ which is perpendicular to the separating hyperplane and an arbitrary point with unknown class $\overrightarrow{u}$.

In the lecture it says that:

$$\overrightarrow{w} \cdot \overrightarrow{u_+} + b \geq 1 \textrm{ , for positive class samples}$$

$$\overrightarrow{w} \cdot \overrightarrow{u_-} + b \leq -1 \textrm{ , for negative class samples} $$

To me it seems that we should take the projection of $\overrightarrow{u}$ on $\frac{\overrightarrow{w}}{|\overrightarrow{w}|}$, since this would give the component of $\overrightarrow{u}$ in the direction of $\overrightarrow{w}$. If this component is greater than the distance to the decision boundary/hyperplane, $b$, then $\overrightarrow{u}$ is a positive sample, if it is less then it is a negative sample. In math terms

$$\frac{\overrightarrow{w}}{|\overrightarrow{w}|} \cdot \overrightarrow{u_+} - b > 0$$

$$\frac{\overrightarrow{w}}{|\overrightarrow{w}|} \cdot \overrightarrow{u_-} - b < 0$$

If $\overrightarrow{u}$ lies on the decision boundary then the above expressions will be equal to 0.

If $\overrightarrow{u}$ lies on a support vector hyperplane, then the above expressions will be equal to $m$ or $-m$ for positive and negative samples respectively, where $m$ is the margin from the decision boundary to any support vector.

I don't understand the lecturer's equations. Why is the value for positive sample classification $\geq 1$? Why is the value for negative sample classification $\leq -1$? Furthermore, what does $b$ represent in the lecturer's equations?

Basically, I have trouble understanding the proposed equations, but they should be doing the same thing as mine.

Very good question. Its worth understanding following concepts about SVM to understand about your point

SVM get trained from the training dataset in such a way that for every instance Xi in dataset
W.Xi + b <= -1 if Xi belongs to -ve class
W.Xi + b >=1 if Xi belongs to +ve class
This also means that for edge instances of +ve and -ve classes the value will be 1 and -1 respectively. This is what it means by making the street as wide as possible.
Above condition is not necessarily true with new unseen instance 'U' because it hasn't been seen by SVM algorithm yet so it can also be inside the street separating the classes
Above equation also implies that the edge instances are separated by 2 units (whatever the unit is). To make the street gap 2 unit, W not give projection of Xi perpendicular to the street but it also scales it to make the gap 2 units. This means W is not necessarily a unit vector.
In other word we are not comparing the component of U in direction perpendicular to the street but scaled version of it so that the street is always 2 units wide

This is intuition based explanation of SVM but mathematically also above equation is the constrain of the optimization problem needed to solved for SVM


How to calculate the output shape of conv2d_transpose? Currently I code a GAN to generate MNIST numbers but the generator doesnt want to work. First I choose z with shape 100 per Batch, put into a layer to get into the shape (7,7, 256). Then conv2d_transpose layer
to into 28, 28, 1. (which is basically a mnist pic)

I have two questions
1.) This code doesn't work for obvious. Do you have any clue, why?
2.) I am very aware how transpose convolution works but I can't find any resource to calculate the output size given input, strides and kernel size specific to Tensorflow. The useful information I found is https://arxiv.org/pdf/1603.07285v1.pdf but e.g. padding in Tensorflow works very different. Can you help me?

mb_size = 32 #Size of image batch to apply at each iteration.
X_dim = 784
z_dim = 100
h_dim = 7*7*256
dropoutRate = 0.7
alplr = 0.2 #leaky Relu

def generator(z, G_W1, G_b1, keepProb, first_shape):

    G_W1 = tf.Variable(xavier_init([z_dim, h_dim]))
    G_b1 = tf.Variable(tf.zeros(shape=[h_dim]))

    G_h1 = lrelu(tf.matmul(z, G_W1) + G_b1, alplr)
    G_h1Drop = tf.nn.dropout(G_h1, keepProb)  # drop out

    X = tf.reshape(G_h1Drop, shape=first_shape)
    out = create_new_trans_conv_layer(X, 256, INPUT_CHANNEL, [3, 3], [2,2], "transconv1", [-1, 28, 28, 1])
    return out

# new transposed cnn
def create_new_trans_conv_layer(input_data, num_input_channels, num_output_channels, filter_shape, stripe, name, output_shape):
    # setup the filter input shape for tf.nn.conv_2d
    conv_filt_shape = [filter_shape[0], filter_shape[1], num_output_channels, num_input_channels]

    # initialise weights and bias for the filter
    weights = tf.Variable(tf.truncated_normal(conv_filt_shape, stddev=0.03),
                          name=name + '_W')
    bias = tf.Variable(tf.truncated_normal([num_input_channels]), name=name + '_b')

    # setup the convolutional layer operation
    conv1 = tf.nn.conv2d_transpose(input_data, weights, output_shape, [1, stripe[0], stripe[1], 1], padding='SAME')

    # add the bias
    conv1 += bias

    # apply a ReLU non-linear activation

    conv1 = lrelu(conv1, alplr)

    return conv1

...

    _, G_loss_curr = sess.run(
        [G_solver, G_loss],
        feed_dict={z: sample_z(mb_size, z_dim), keepProb: 1.0} #training generator

Here is the correct formula for computing the size of the output with tf.layers.conv2d_transpose():

# Padding==Same:
H = H1 * stride

# Padding==Valid
H = (H1-1) * stride + HF

where, H = output size, H1 = input size, HF = height of filter

e.g., if `H1` = 7, Stride = 3, and Kernel size = 4,

With padding=="same", output size = 21,
with padding=="valid", output size = 22

To test this out (verified in tf 1.4.0):

import tensorflow as tf
import numpy as np

x = tf.placeholder(dtype=tf.float32, shape=(None, 7, 7, 32))
dcout = tf.layers.conv2d_transpose(x, 64, 4, 3, padding="valid")

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    xin = np.random.rand(1,7,7,32)
    out = sess.run(dcout, feed_dict={x:xin})
    print(out.shape)


In which cases shouldn't we drop the first level of categorical variables? Beginner in machine learning, I'm looking into the one-hot encoding concept.

Unlike in statistics when you always want to drop the first level to have k-1 dummies (as discussed here on SE), it seems that some models needs to keep it and have k dummies.

I know that having k levels could lead to collinearity problems, but I'm not aware of any problem caused by having k-1 levels.

But since pandas.get_dummies() has its drop_first argument to false by default, this definitely has to be useful sometimes.

In which cases (algorithms, parameters...) would I want to keep the 1st level and fit with k levels for each categorical variable?

EDIT: @EliasStrehle's comment on above-mentioned link states that this is only true if the model has an intercept. Is this rule generalizable? What about algorithms like KNN or trees which are not exactly models in the statistic definition?

First, if your data has missing values, get_dummies by default will produce all zeros, so that perfect multicollinearity doesn't actually hold.  Also, from a data manipulation standpoint (without regard for modeling), it makes some sense to keep the symmetry of having a dummy for every value of the categorical variable.
In a decision tree (and various ensembles thereof), keeping all the dummies is beneficial: if you remove the first dummy, then the model can only select on that level by selecting (through several steps in the tree, rather unlikely!) "not this other dummy."
Then again, it's probably better not to one-hot encode at all for decision trees, but for now some packages don't deal innately with categorical variables.
K-nearest neighbors seems like it would also benefit from keeping all levels.  The taxicab distance, limited to the dummies of one feature, between two points with different values is 1 if one of their values was the removed dummy, otherwise 2.
But again, it seems like KNN would be better off without one-hot encoding, but instead some more informed measure of distances between the category's values if you can come up with them.
See also https://stats.stackexchange.com/questions/231285/dropping-one-of-the-columns-when-using-one-hot-encoding
(In particular, when using regularization in a linear model, it may be worth keeping all dummies.)


One hot encoding alternatives for large categorical values I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?
I found this interesting link.
But they are converting to class/object which I don't want. I want my final output as a data frame so that I can test with different machine learning models? Or, is there any way I can use the generated matrix to train the other machine learning models other than Logistic regression or XGBoost?
Is there anyway I can implement it?

One option is to map rare values to 'other'.  This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.

I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian.  See e.g. the paper Deep Knowledge Tracing, which says this approach is motivated by the idea of compressed sensing:

BARANIUK, R. Compressive sensing. IEEE signal processing magazine 24, 4 (2007).

Specifically, they map each vector of length N to a shorter vector of length log2(N).  I have not done this myself but I think it would be worth trying.


Supervised learning: classifying/learning a single discrete value Assume that I have a single feature dataset X comprising of say a 1000 samples.  The feature is an integer that may take up any value upto 2^32. Among these, one particular value alone is considered malicious. The target/lable vector flags out the malicious value as 1 and others as 0.  What would be the right approach to train a model that could learn this single value and classify it as 1 and all else as 0?

I don't really see this as a learning problem really, if it's the same malicious value you have to watch out for..

Only one "positive" sample (corresponding to the malicious value) would be enough to find the malicious integer.

Have I misunderstood something in your question?


Thesis on avalanche prediction using One Class SVM I'm doing my thesis on avalanche prediction using machine learning.

For my input features I'm using avalanche accidents with features like slope, altitude, facing direction of the slope, combined with according weather data from the day the avalanche occurred.

I want to predict an avalanche when certain variables combine and create a deadly avalanche situation. So 1: the avalanche occurs. 0: The avalanche does not occur.

The only data in my database are occurred avalanches, I got around 200 samples. So I don't have any data of a non deadly avalanche situation, which is mostly the case.

My question is if a One Class SVM is a good approach to take on this classification?

Your problem seems to belong to novelty detection in the general area of OCC problems.

So, the short answer is: yes. You can apply SVDD (Support Vector Data Description) method to get the smallest hypersphere containing samples in the dataset and then assess whether a new observation is an outlier or not.

Of course, the less representative your dataset is, the less accurate your classifier will be.


Linear Model for Linear Regression I'm new in Machine Learning and of the first concept I would like to learn is linear regression. I read that to apply linear regression I need to use a linear model. Starting from this assumption I know that this is a simple model for linear regression :

y = w0 + w1x

The definition of linear regression says that the dependent variable y should be a linear combination of the parameters w (but it is not necessary the same for the independent variable x )

So we can say that also this is a linear regression model :

y = w0 + w1x1 + w2(x2)^2

Also in this case, I should say that this is a linear regression model because for the definition, w0 , w1 and w2 are still linear in the expression. Even if there is a quadratic term for the independent variable x2.

Now , I have this question.
A model like the following :

y = w1 x1 + w2 x2 + w3 x3 + w4 x1x2 + w5 (x2^3)

Is it still a linear model ? My first answer is yes , because for the definition the parameter terms are linear , but I'm not sure of it.
Does anyone got any hint ?

Theoretically, you do indeed have a linear model, yes.

You have a linear relationship between the dependent variable and your parameters.
You would still be going through the process of performing linear regression, fitting a line through points via a linear combination of some regressors. However you have a non-linear equation due to the higher-order regressors you have manually inserted (x1x2 and x2^3).

This is enough to still call it a linear model generally speaking - see some useful answers in this thread.

In general I wouldn't say you have a linear model in the strictest sense at the end, as you are modelling linear relationship between your dependent variable y and non-linear combinations of your regressors: x, x2, and x3, but perhaps there could be an underlying feature to be observed which is exactly equal to x1x2, call it x4, and then you would have removed one of the non-linear covariates. Tyhe same goes for the final non-linear term.


What measures can I use to find correlation between categorical features and binary label? For analyzing numerical features, we have correlation. What measures do we have to analyse the relevance of a categorical feature to the target value? If there isn't a direct measure, how can we achieve this?

Chi-squared test is known, but I can't find any implementation of it for categorical values.
One other way is to label encode into numerical values, but that assigns certain priority to higher valued labels.

Checking if two categorical variables are independent can be done with the Chi-Squared test of independence where we perform a hypothesis test for that. Let's say A & B are two categorical variables then our hypotheses are:

H0: A and B are independent

HA: A and B are not independent

We create a Contingency table that counts for the combination of outcomes from two variables, if the null hypothesis(H0) is correct then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the actual values are.

For Example
Suppose we have two variables in the dataset

Obesity: Obese, Not obese
Marital Status: Married, Cohabiting, Dating

We observe the following data:
Oij for a cell (i,j) is the observed count in given data

             |    dating   |   married    |  cohabiting  |  Total       |
  -----------|------------:|:------------:|:------------:|:------------:|
  Obese      |      81     |     147      |   103        |   331        |
  Not obese  |      359    |     277      |   326        |   962        |
  Total      |      440    |     424      |   429        |   1293       |

Expected Counts Calculation i.e. Expected counts if H0 was true.

Eij for a cell (i,j) as
Eij=row j total * column i total / table total

             |    dating   |   married    |  cohabiting  |
  -----------|------------:|:------------:|:------------:|
  Obese      |      113    |     109      |   110        |
  Not obese  |      327    |     316      |   319        |

X2-statistics Calculation
Statistics

Assuming independence, we would expect that the values in the cells are distributed uniformly with small deviations because of sampling variability
so we calculate the expected values under H0 and check how far the observed values are from them.

We use the standardized squared difference for that and calculate Chi-square statistics that under H0 follows χ2 distribution with df=(n−1)⋅(m−1) where n & m are the number of categories in the first & second variable respectively.

\begin{equation}
\chi^2 = \sum_ i \sum_ j \frac{ ({O_{ij} - E_{ij}})^2}{ E_{ij}}
\end{equation}

χ2 value comes out to be 30.829

We can use R to find the p-value

tbl = matrix(data=c(81, 147, 103, 359, 277, 326), nrow=2, ncol=3, byrow=T)
dimnames(tbl) = list(Obesity=c('Obese', 'Not obese'), Marital_status=c('Dating', 'Married','Cohabiting'))
chi_res = chisq.test(tbl)
chi_res
        Pearson's Chi-squared test
data:  tbl
X-squared = 30.829, df = 2, p-value = 2.021e-07

Since p-value < 0.05 we reject the null hypothesis, we can conclude that obesity and marital status are dependent.

There also exists a Crammer's V that is a measure of correlation that follows from this test.

Putting values in the formula, R code

sqrt(chisq.test(tbl)$statistic / (sum(tbl) * min(dim(tbl) - 1 )))

0.1544

So we can say there is a weak positive correlation between obesity and marital status.

I hope I am clear with the explanation.


Will historical data lead to target leakage? I'm bulding a employee churn model. I've employee data from 2016 to 2019 (of people who stayed/left the company), my goal is to train using data from 2016 to 2018 and predict on 2019.

Since there's people who did not leave the company between 2016-2019 there's alot of repetead employees, so my training set is: data from 2018 about employees who did not leave the company injected with data of employees who left the company in 2016/2017/2018 in order to only have one person once in the training dataset.

My question

Does having only the persons who left in 2016/2017 lead to target leakage?
I'm not using time dummies but can my model overfit thinking employees are more likely to leave the company in 2016/2017 because it's what it sees more often?
And if so how can I avoid this problem?

Thanks :)!

I'm going to kind of ignore your question. To me it sounds like you have a dataset containing essentially employee starting and 'end dates' or if the employee is still working for you no end date. This sounds like the standard case for a class of regression techniques called 'survival regression' or survival analysis. I suggest you look into this.

Now for your question, Im not too sure about target leakage as I cannot quite see how, if you have a structural change in employees year by year then theres definitely some leakage yeah.


Does scikit-learn use regularization by default? I just fitted a logistic curve to some fake data. I made the data essentially a step function.

data = -------------++++++++++++++

But when I look at the fitted curve, the slope is very small. The function that best minimizes the cost function, assuming cross entropy, is the step function. Why does it not look like a step function? Is there some regularization, L1 or L2, done by default?

Please take a look at the documentation. The first line shows the default parameters, which include penalty='l2' and C=1.0.

You actually cannot disable regularization completely, you can only regularize less... try setting C=1e10 for example.


Orange 3.15 GUI window size issue (Windows 8.1) I'd just like to report an annoying GUI bug I am experimenting under Windows 8.1 using Orange 3.15.
The work space available doesn't size up with the window making the use of the software quite frustrating.
Here is a video of the bug.
Does anyone have a fix for that by any chance?

This is a known issue: https://github.com/biolab/orange3/issues/3193

It is caused by a bug in Qt 5.9.6 (QTBUG-16252). You should downgrade Qt to 5.9.5.

To do that open the "Orange Command Prompt" from the Window 8.1 'Start Screen' (start menu) and type in conda install qt=5.9.5


How to make R or Python as fast as SAS for ODBC Oracle queries? I want to use R or Python to query big structured SQL-type data, but they are very slow compared to SAS.

I tried using R and Python to return a 1.3 million record Oracle ODBC passthrough query. The query took 8-15 seconds in SAS, 20-30 seconds in Python, and 50-70 seconds in R. Does anyone know why?

R Packages Used:

First I used the RODBC package in R to query to the Oracle database. Then I tried the ROracle package, but both packages were much slower than SAS.

Python Packages Used:

For Python, I used Oracle's cx_Oracle package for the query.

Thanks a lot,
Sean

This answer some of your questions from a Python perspective:

  Is Python any faster?

This question is a bit tricky to answer, it will depend on your usage of Python, but Python is not a fast language per se. However, the pandas library in Python have been reported to handle tables of 33M-100M rows, see this. I myself have used to handle around 10M rows from a Postgres table. For a detailed experimentation using pandas, see this. In the link they apply some operations on datasets of 88M rows and 74 columns.

  Do we need Hadoop or parallel processing or something else to make R/Python as fast as SAS?

Before trying using Hadoop or Spark, I recommend you to follow some optimization tricks (tips):

[1] A Beginner’s Guide to Optimizing Pandas Code for Speed

[2] Using pandas with large data

This link (Don't use Hadoop - your data isn't that big) can be useful too.

Also for a comparison of SAS and pandas you can read the pandas documentation on such comparison or this.

  Is it because R and Python are matrix-based languages and SAS is more
  traditional-database-oriented?

Python is not a matrix-based language, as far as I know the language does not offer by default any capabilities for handling matrices. I think your referring to the numpy/scipy stack, this is a separated library.


Why is PyTorch's DataLoader not deterministic? I've set the seeds like this (hoping to cover all bases):

random.seed(666)
np.random.seed(666)
torch.manual_seed(666)
torch.cuda.manual_seed_all(666)
torch.backends.cudnn.deterministic = True

The below code will still output DIFFERENT batches for both namesTrainLoader1 and namesTrainLoader2 but they should really be the same. How come creating the model is affecting the deterministic values?

namesDataset = NamesDataset()
namesTrainLoader1 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader1:
    print(each)

model = TorchRNN(inputSize, hiddenSize, outputSize)

namesTrainLoader2 = DataLoader(namesDataset, batch_size=5, shuffle=True)
for each in namesTrainLoader2:
    print(each)

Output for namesTrainLoader1:

('saiki', 'close', 'sloan', 'horos', 'roman')
...

Output for namesTrainLoader2:

('david', 'abeln', 'hatit', 'holan', 'protz')
...

I also tried using worker_init_fn (e.g. with lambda x: 0) in the DataLoader, but that made no difference.

Why is this not deterministic? How can I make it deterministic? i.e. reset the internal seed of the DataLoader?

If you want to shuffle the data in a deterministic way, how about shuffling the dataset beforehand e.g. in a simple list of filenames, then simply reading that list deterministically in a single-processed loop, with shuffle = False in the DataLoader??

Another things that may cause non-deterministic behaviour is using multiple processes - then there are operations that are passed out and managed by the operating system, which doesn't pay attention to any of the random seeds you set. Performance is dependent on available resources i.e. affected by other activities running on your host machine.

In addition to that, any interaction between CPU and GPU could be causing non-deterministic behaviour, as data transfer is non-deterministic (related Nvidia thread). Data packets can be split differently every time, but there are apparent CUDA-level solutions in the pipeline.


What is the general approach I can use to predict whether a domain is related to a brand using a supervised learning algorithm? I have a number of domain names that may or may not be related to a particular brand.  For instance, if the brand is UPS, www.upssucks.com, www.upspackagesupplier.com, and www.ihateups.com might all be labeled as "related" because the website content is talking about UPS.  www.ilovepups.com and www.pushupssuck.com aren't related to the website UPS.  I want to use my trained dataset to create a prediction of whether a given domain is related to a brand using only the registered domain name as the input.  It seems like some off-the-shelf classifiers should work, but I am very new to this type of ML project.  What would be the first approach one would take to start making predictions?  I am planning on doing this in Python with scikit learn if that makes any difference.

It will be almost impossible to do this your way as you arw trying to derive context based solely on the domain names, and domain names more than often don't carry a specific context. Since the word ups occurs in each domain and the words coming before or after hardly carry much specific information. If you could extract some information about the website by hitting the url, you will not need any machine learning at all. I am not saying this can't be done through ML, all I am saying it won't be that effective. If you use scikit learn you will need to find a way to tokenize the domain, in your case a character level tokenization will be useful, throw in a BOW (Bag of words, characters in your case) create one-hot vectors out of it and use them as features to predict the label. If you have a lot and I mean lots you can train a character level convolutional neural network and it might perform better. Cheers.


Feature selection and classification accuracy relation One of the methodology to select a subset of your available features for your classifier is to rank them according to a criterion (such as information gain) and then calculate the accuracy using your classifier and a subset of the ranked features.

For example, if your features are A, B, C, D, E, and if they are ranked as follow D,B,C,E,A, then you calculate the accuracy using D, then D, B then D, B, C, then D, B, C, E... until your accuracy starts decreasing. Once it starts decreasing, you stop adding features.

In example1 (above), you would pick features F, C, D, A and drop the other features as they decrease your accuracy.

That methodology assumes that adding more features to your model increases the accuracy of your classifier until a certain point after which adding additional features decreases the accuracy (as seen in example 1)

However, my situation is different. I have applied the methodology described above and I found that adding more features decreased the accuracy up until a point after which it increases.

In a scenario such as this one, how do you pick your features? Do you only pick F and drop the rest? Do you have any idea why the accuracy would decrease and then increase?

feature selection involves several approaches just like methods for machine learning. Idea is to keep most relevant but not redundant feature for predictive model that can yield optimal accuracy.

In your case, I can not see which method you are using for feature selection but assuming that you are not taking account of multivariate nature of feature dependency. Say you have N features, likely reason that your model accuracy drops after n top feature(s) but improves by adding n+k (where n < k < N when features are in descending order based on information gain) is due to inter-dependency (more relevance and less redundancy) of top n and k features. Univariate feature selection does not necessarily get optimal model accuracy when features are inter-dependent and not mutually exclusive. From philosophical point of view, set of optimal features is analogous to a quote by Aristotle: "The whole is greater than the sum of its parts"!

For optimal feature selection, I often is Caret package in R language where one may do feature selection using recursive feature elimination (RFE) among several other approaches. There is also a package called mRMRe to do feature selection based on maximum relevance, minimal redundancy.

Best,
Samir


How Hyper-linked library vs traditional library differs from each other as ML problem? Traditional library can be understood as a system, that archives the collective information from the mediums produced by our society, by indexing them to shelves. It is assumed that libraries have single ontology by which the books can be categorised to a single shelf. Many machine learning research papers exist for this problem.

Hyper-linked library could be understood in such way, that instead of a medium having one place in the ontology, there could be multiple. Imagine a book as a symbolic link to the physical book.

This aspect changes the machine learning problem slightly as instead of forming a clear hierarchy and simple tree structure, the hyper-linked library is a multi-label classification problem instead of hierarchical multi-class classification problem.

What are the important aspects that one has to consider when moving from traditional library problem towards a hyper-linked library problem? Links to research papers more than welcome!

To get discussion started, here are some of my intuitions.

Instead of hierarchy, it seems that the hyper-linked problem is more about optimization of information entropy in a tree structure: giving all labels to all documents would be a bad solution and also giving shortest path possible is far from optimal (due to being too vague). It seems that the multi-class classification problem of hyper-linked libraries would benefit from some cost factor in relation to how many labels a document is allowed to have and how (non-)generic they should be.

Scale-free networks might provide such cost factor as they can normalize network through-put even when the amount of documents grows unboundedly, e.g. the concept space can be frozen with a guarantee of keeping the navigation depth as constant, but still being able to distribute the documents "evenly" by preferential attachment based on some optimization goal that would give privilege for some documents over others (a scale-free network, where new links are also made between existing nodes).

This is actually the very same problem Google solved with PageRank. Internet is a scale-free network by topology and search engine provides us a symbolic link to the original based on keyword mapping.

However, while Google is entirely free from ontologies, hyper-linked libraries need to provide some baseline structure from existing ontologies. Also, if we would consider the reality of News Agencies classifying their own content, the life-cycle of some content is not as "eternal" as with content stored by libraries. Discovery of new media products would be important part of the problem. Again, seems like Google has solved these kind of problems.


Plagiarism detection with Python Background
Using Python, I need to score the existence of a quote, containing around 2-7 words, a longer text. The quote doesn't have to match the text precisely, but similar words should have the same order.
For example, given the following long text:

The most beautiful things in the world cannot be seen or touched, they are felt with the heart

The following quotes should be scored high (say, above 80 / 100):

The beautiful thing in our world
World cannot see
They feel with the heart

Since they are not precise, but they preserve the order.
While, on the other hand, these quotes should be scored lower (say, below 50 / 100):

The beautiful heart cannot be felt or seen
They are the most seen in the world
These words don't even appear on this text

Because (the first 2) appear entirely in the text, but do not preserve the order.
The problem
This task cannot be accomplished by simply checking the existence of each word in the text. I don't know which algorithm fits best for this task.
What I have tried
Most of the functions in fuzzywuzzy (partial_token_sort_ratio, token_sort_ratio and etc) scored the later terms higher.
partial_ratio did score the earlier terms higher, but the quote

These words don't even appear on this text

Got 52 / 100 which is unreasonably high.
My question
How can I use python to score the existence of short quotes in longer texts as mentioned above?

Python's fuzzywuzzy uses Levenshtein Distance which looks at character level differences.

You have to explore other approaches to text similarity. Find algorithms that nonlinearly weight n-grams differences, such as Q-gram.

python-string-similarity repo has implementations of many text similarity algorithms.


Is there a way to Label/Annotate My Bubble Plot (Scatter plot with a z-axis) on matplotlib? I would like to label my bubble plot, with the labels being INSIDE the bubbles. Something like this.

Is There any way to do that?

What I've tried:

plt.annotate(temp.index, (temp.gre_total, temp.ugrad_gpa))
plt.show()

But that throws an error. I think that would only work if I looped through every value? I'm not very sure.

You can use the seaborn package, using the scatterplot marker size to generate your bubbles. Then you need to loop over the datapoints and add a text labet to each point in your scatterplot.

# Load libraries
import pandas as pd
import matplotlib.pylab as plt
import seaborn as sns

# Create example dataframe
df = pd.DataFrame({
'x': [1, 1.1, 1.2, 2, 5],
'y': [5, 15, 7, 10, 2],
's': [10000,20000,30000,40000,50000],
'group': ['Stamford','Yale','Harvard','MIT','Cambridge']
})

#Create figure
plt.figure(figsize = (15,10))

# Create scatterplot. alpha controls the opacity and s controls the size.
ax = sns.scatterplot(df.x, df.y, alpha = 0.5,s = df.s)

ax.set_xlim(0,6)
ax.set_ylim(-2, 18)

#For each point, we add a text inside the bubble
for line in range(0,df.shape[0]):
     ax.text(df.x[line], df.y[line], df.group[line], horizontalalignment='center', size='medium', color='black', weight='semibold')

Which outputs:


Looking for a model architecture that can work with an unordered list of arbitrary length I'm looking for a machine learning model architecture that takes in an arbitrary number of inputs and generates one output. Pretty much like GRU or LSTM, it's just that the order of the items in the input is irrelevant. So f([x1, x2, ..., xn])=y, where each x is of shape [i] while y is of shape [j] (not considering the batch dimension). And f([x1, x2, ..., xn])=f([xn, xn-1, ..., x1]) or any other order of the input. In other words, f treats its input as a set, unlike RNNs that treat their inputs as a list.
Is there such an architecture?

You can pass all of the input to a MLP layer, and take the aggregation of the results. It would be independent of the order


Does shuffling data for time series forecasting help? So I am trying time series forecasting using LSTM's. The aim is to predict $Y$ given $X$ using regression.

I had already converted the input data into a sliding window format such that if my input data was of the form:

X = [x0, x1, x2,.....]
Y = [y0, y1, y2,.....]

Then I converted it into:

Xnew = [(x0, x1, x2), (x1, x2, x3), (x2, x3, x4),...]
Y    = [         y2,           y3,           y4,...]

Still, upon training my data I find a very high validation_loss.

Since validation_split takes only the fraction of the data from the end, I thought maybe I should try and randomize the data before training it. However, in that way, will time series have any meaning?

I found a similar question, but I had apparently already tried what was suggested:  Is it valid to shuffle time-series data for a prediction task?

Shuffling data would not seem to make sense here, since your model has "memory". You're not predicting $y_i$ from only $x_i$, but also $x_{i-1}$ and $x_{i-2}$. If you shuffle the data and perform prediction, you are implying that $x_1, x_2, x_3$ should give the same value as $x_2, x_1, x_3$ or $x_9, x_5, x_3$, or any series of values that merely ends in $x_3$ (since the target value is always $y_3$, regardless of the other $x$). If your target value actually does depend on preceding variables, shuffling the data breaks that relationship. If it does not depend on preceding values, it's arguably not a time-series model, since the ordering of observations is irrelevant.


How do you get an RNN to learn in real time? In order to train a recurrent neural network, you have to unfold it say 50 times and treat it like a chain of RNN cells.

Now if you are using this RNN to train in realtime, then it seems you may as well just have a chain of 50 cells. Maybe fix their weights to all be the same as in a convolutional neural network.

So are RNN's any use in learning in realtime for temporal data?

I think you misunderstood the fundamental of RNN.

  fix their weights to all be the same".

Yes that's exact what RNN is suppose to be. In any type of RNN, you always only have one real RNN cell. The unrolled RNN cells are merely representations of that RNN cell overtime. In a single training step, the weights of those unrolled RNN cells are always the same.

  Sp are RNN's any use in learning in real-time for temporal data?

Yes, you can train an RNN in real-time with a catch. Although in theory the sequence is arbitrary long, but in practice we can only backprop a certain step back due to


doc2vec - How does the inference step work in PV-DBOW I am quite confused about how we generate new paragraph vectors in PV-DBOW?

If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph?

In the original paper the authors wrote:

  At prediction time, one needs to perform an inference step to compute
  the paragraph vector for a new paragraph. This is also obtained by
  gradient descent.

It is unclear to me how they would compute the paragraph vector for a new paragraph using gradient descent.

The original paper does a lot of hand-waving on the implementation of inference step and is not clear. So your confusion is justified. I'll explain at high level below. I'm assuming only PV-DBOW model.

Training Phase

In this model, we forget word ordering information and setup a very simple neural network.
Represent all input document tags in a vocabulary to get unique ID for each document tag. Do the same for words.
For a given document ID, the input is one-hot encoded representation of document tag IDs. Output is one hot encoded representation of a randomly selected word.
We want to setup training such that for a given tag ID, a randomly selected word from that document will be predicted with high probability.
So the neural network getting trained transforms one hot encoded representation to document/tag vector. The tag vector is passed through another layer with softmax at the output. Both set of weights are adjusted during training phase.
What training achieves: It represents all document tags in a new space such that probabilities for randomly selected words in each document are maximized starting from that vector space representation to softmax output.
Important to note that there are two set of weights. Input to hidden layer and hidden to softmax output.

Inference Stage

Words are unique across document. So a word 'India' in training and inference stage gets mapped to same ID in vocabulary.
But there's not such concept for document ID since documents are assumed to be unique and there's no shared ID between training and inference.
Inference stage runs a (sort of) reverse calculation. What vector space representation is most appropriate for this document, if I use the same set of weights from hidden space to output layer? The weights from hidden layer to softmax output are kept constant!
Select a word from new document at random. Start with a random representation for the document vector (input to hidden layer). Pass it through neural network from hidden to softmax output with constant weights (learnt during training). Adjust the randomly initialized weights such that softmax probability is maximized for the selected word. Repeat this process many times.
This is the reason why you need stochastic gradient descent and need to specify number of steps.

Summary

Document IDs are not common from training and inference stage (like word IDs in w2v).
There are two set of weights, keep one constants during inference stage based on learning from training phase. Adjust the other using SGD.
You mapping new document to document vector space such that documents which had similar words during the training phrase are close in the document vector space.

This is a very good tutorial on the implementation by Richard Berendsen. I have ignored additional complexities like negative sampling, context window from the explanation.


What are some good error metrics for multi-label (not mutli-class) problem in industry? What are some good error metrics for multi-label (not mutli-class) problem in industry?

http://scikit-learn.org/dev/modules/multiclass.html

A common example is the Jaccard similarity coefficient:

$J(Y, P) = \frac{|Y~\cap~P|}{|Y~\cup~P|}$

where $P$ is the set of predicted labels for an instance and $Y$ is the true set of labels. This gives a value between $0$ and $1$ for each instance, which you can average over the whole test set to give a score. If $P = Y$, then $J(Y, P) = 1$.

This is implemented in scikit-learn as sklearn.metrics.jaccard_similarity_score.


Pandas: how can I create multi-level columns I have a pandas DataFrame which has the following columns:

n_0
n_1
p_0
p_1
e_0
e_1

I want to transform it to have columns and sub-columns:

0
    n
    p
    e
1
    n
    p
    e

I've searched in the documentation, and I'm completely lost on how to implement this. Does anyone have any suggestions?

Finally, I found a solution.

You can find the example script below.

#!/usr/bin/env python3
import pickle
import pandas as pd
import itertools
import numpy as np

data = pd.DataFrame(np.random.randn(10, 5), columns=('0_n', '1_n', '0_p', '1_p', 'x'))

indices = set()
groups = set()
others = set()
for c in data.columns:
    if '_' in c:
        (i, g) = c.split('_')
        c2 = pd.MultiIndex.from_tuples((i, g),)
        indices.add(int(i))
        groups.add(g)
    else:
        others.add(c)
columns = list(itertools.product(groups, indices))
columns = pd.MultiIndex.from_tuples(columns)
ret = pd.DataFrame(columns=columns)
for c in columns:
    ret[c] = data['%d_%s' % (int(c[1]), c[0])]
for c in others:
    ret[c] = data['%s' % c]
ret.rename(columns={'total': 'total_indices'}, inplace=True)

print("Before:")
print(data)
print("")
print("After:")
print(ret)

Sorry for this...


Are DBSCAN and dbscan from the sklearn.cluster package different? I'm new to DBSCAN. I was looking at a few examples online and came across a few instances where the following lines were used while importing the dbscan module:

from sklearn.cluster.dbscan_ import DBSCAN
from sklearn.cluster.dbscan_ import dbscan

I would like to know if there is anything different between them? Or is it necessary for me to import both DBSCAN and dbscan?

Here's one link where both lines are used in the import:
https://gemfury.com/stream/python:scikit-learn/-/content/cluster/tests/test_dbscan.py

Such questions are easily answered if you check the source code yourself.

https://github.com/scikit-learn/scikit-learn/blob/1495f69242646d239d89a5713982946b8ffcf9d9/sklearn/cluster/dbscan_.py#L350

One is an object-oriented wrapper with a more common API, the other the underlying function that does the actual work. Since the wrapper just calls the function, the results are not substantially different, only the presentation.

Hence, usually you won't need both.


How to use a multiple linear regression model built from normalized data I built a linear multivariable regression model from normalized data (for the interval [0; 1]). Initially, the data was not normalized, I normalized the data by myself (independent and dependent variables).
I want to use this model to make predictions from newly received data (I get the values of independent variables and I want to predict the value of the dependent variable). The problem is that the data comes in a raw, unnormalized form.

How can I normalize newly arriving data if only one "observation" is received?
What if I want to get the real values of the dependent variable using my model, and not the normalized ones?

So, the question asks:

How to normalise incoming (individual observations)
How to get the real value predictions and not the normalised values.

When we do normalisation using the Sci-kit learn module, instead of using the very handy fit_transform() method in the scaler, you could instead perform a .fit() over your original observations and then apply the .transform() to the newly-observed values. Obviously, in this case you need separate scalers foreach feature, as they are distributed differently from one another.

Again in Sci-kit learn, there is an inverse_transform() method which reverts the normalised value back to the original scale.


What's the fastest way to do a text analysis over user reviews on a website for a beginner? I want to analyse user reviews for certain products as part of a research project without having to learn analytics from scratch, as my requirement is temporary. I need to do the following:

The user reviews are formatted in separate columns per user and indexed over multiple pages (almost 2000 reviews per product) - so first extract all the text intelligibly.
Analyse frequency of certain words and phrases(once per user), the overall connotation of the comment, most frequent words of all the reviews combined, etc.

Questions:

Where do I begin doing this?
How should a beginner approach this?
How long is it going to take?
Could there be an easy to use software or a bot where I can just give in my requirements and would just give me the results?

Welcome to DS.SE, @shubham.

Since the question is a bit general, I can just provide you some link to start with. Once you have more specific requirements, the community can help further.

An easy-to-use tool that I knew for sentiment analysis was IBM Alchemy which seems retired now in favor of IBM Watson NLU.

Despite IBM, other analytical platforms such also provide sentiment analysis APIs/tools. For example, see Google AI & Machine Learning Products
 and Deep AI. This article on Medium reviews some this tools in brief.


Time series on syslogs Is time series model suitable for network syslogs considering the fact the messages are sequential and the messages are outputted as a result of dependency between themselves which can range from normal notifications to alarms if there are.

Can the state of the art models like ARIMA be applied ?

What kind of questions can be answered with applying time series models on syslogs ?

No, according to your description the syslog is not a time series data.

Time series data means to have a numerical value at each time stamps. For instance, if your log records the CPU usage (a number between 0 to 100%) then it is a time series data.

If you log is a sequence of text message, like the /var/log/syslog in Linux, then it is not a time series data.


Backpropagation chain rule example My question is in regards to an MIT course example.

The instructor delves into the backpropagation of this simple NN.

I have two questions.

Why do we seem to disregard the weights of the second layer (in blue)?
The red circle, should this not be $\partial y_2$ (followed by $\frac{\partial y_2}{\partial P_3}$)?

Why do we seem to disregard the weights of the second layer (in blue)?
-----because activation function is after that block, so differential with the o/p y is as good as with the weight.
The red circle, should this not be ∂y2 (followed by ∂y2∂P3)?
-----y2 and y1 is the o/p of the same block i.e. both must be same


What does it mean when the coefficient of the squared term is negative in regression? I am reading a research paper which models a regression model where the returns are regressed on the number of ad exposures.

the equation looks something like this:

$Returns = beta_1*nExp + beta_2*nExp^2$

nExp: The number of times the user looks at the ad.

Returns: The revenue from the user

The result is that: $beta_1$ is positive (and statistically significant), which means that the ad exposures has a $+ve$ relation with the revenue.

However, $beta_2$ is negative. And the research paper says:

  term for the square of the number of exposures, which has a negative
  coefficient suggesting diminishing returns to ad exposure.

What does it really mean? What does it mean when the term's coefficient is $+ve$ but the coefficient of it's squared term in the equation is $-ve$?

In this context probably the plain English way to put it is that 'returns increase with additional exposure to the ad, but there is a tapering effect at the upper end of exposures. Looking at this picture of a parabola here (i.e a graph of y= -ax^2+bx+c):

Most likely your returns data are between 0 and the peak so you don't actually see  a decline, just a reduction in the rate of increase.