Sliding window approach using SVR & LightGBM I'm working on a multivariate time series forecast using a couple of ML algorithms (Neural Networks, Support Vector Machines & Gradient boosting algorithms). I need to measure the performance of each model.
I've implemented the 1st model using Tensorflow 2.0. Training & testing data was created using tf.Dataset API.
The data format is (window_data, forecast), where window_data represents a set of 24 timesteps and forecast represents  the next timestep.

Now I need to train 2nd & 3rd model using SVR (LinearSVR to be more precise) and LightGBM.
Is it possible to feed the model with a windowed dataset like in my 1st model?

Tensorflow is designed to really make your life easier (especially with the fancy new additions with the 2.0 version). I think you should use pandas to generate sliding windows: I would think that for a N sized time series with K lookback window you will have N-K+1 examples by sliding lookback.

And while you're at it have you treated your lookback window as a hyperparameter? i.e optimise your model performance on a validation test by searching the best lookback?


Unit Step Activation Function Using the unit step activation function for the artificial neuron, determine a set of weights (and threshold value) that will produce the following classification:

x1  x2  output
-0.2 0.5    0
0.2 -0.5    0
0.8 -0.8    1
0.8 0.8     1

I understand the theory of unit step activation function (maybe beginner) but i am stumped as to how to calculate the weights and threshold values. I thought that it is specific to the problem and is provided as input. Please help

Here is what i did. Not sure if this is correct. Will have update next week.

I plotted these 4 points on a graph with x1 and x2 on x and y axis respectively. since its a unit step function, which essentially has 1 or 0 output - i tried to draw a linear regression line such that all "0" result points are on 1 side and "1"  result points are on other. This gave me the possible weights w1 and w2 and i constructed an equation

w1= 1.5 and w2 = 0.5

Then i made a table to check if the equation provided the actual results

i started adjusting the w1 and w2 to try to "fit" to training data but one data was just not working!!

i could bring the error down at w1=1 and w2 =0.39

Then - the threshold value, T
it should be in such a way that the z = x1w1+x2w2 + T and y=0 if z<0 and y=1 if z<=0. The value of T should change the results of (0.2,-0.5 ) to fit but not change the result of other points. So i started at -0.4 and worked my way up to -0.44.

Final equation came out to be.
y = x1 + 0.39x2 -0.44

This fits the current training data.


GANs to augment training data I have been reading about Generative Adversarial Networks (GANs) and was wondering if it would make sense to train a generator function only to use it for creating more training data.

In a scenario where I don't have enough training data to build a robust classifier, can I use this limited data to train a generator that'll produce samples good enough to improve the accuracy of my discriminator (classifier)?

Yes and no depending on how you define "good enough samples".
You will likely end up with a chicken and egg problem: you want to use the GAN to generate training data, but the GAN doesn't have enough training data itself to generate convincing enough samples.

Other techniques exist for data synthesis of training images. For example: adding noise, flipping axis, change luminosity, change color, random cropping, random distorsion.


When is Gradient Descent invoked on the objective function while running XGboost? is it at the end of every tree? or only after all trees are build? I tried to think in both ways but didn't get a clear picture.

Can we focus more the part "the loss function is applied between training models" I am trying to understand how is Gradient Descent is applied to minimize the objective function function.

slide no 36 in http://www.saedsayad.com/docs/xgboost.pdf

  Data points reach to a leaf will be assigned a weight. The weight is
  the prediction

how is the weight assigned ? how is the weight calculated ? why predict the weight again ? Can someone shed some light on this?

I thought of throwing some historical perspective into this...

The initial algorithm is called AdaBoost. Unlike most of machine learning, this algorithm has its genesis in hard theory. The authors were trying to answer the following theoretical question:

  Is it possible somehow to combine weak models and create a very accurate predictor?

Weak models are models that are hardly better than chance. Think about this question. It is quite a philosophical question... And, as it happens, it can be answered by hard mathematics.

They actually demonstrated that yes. Their paper is actually worth reading. They have since published a lot of interesting papers from different perspectives, for instance using game theory.

The AdaBoost algorithm is also very simple and worth checking because it is the basis of xgboost.

AdaBoost works with any model. The only restriction is that the model supports giving weights to each observation. This is because it trains models sequentially by increasing (or decreasing) the weight associated to each observation to make that observation more important (or less important).

The cool thing is that the algorithm is very unlikely to overfit. But notice that this is an empirical claim, not a theoretical one. The algorithm can overfit, but it has been found (empirically) to be very resilient.

This algorithm has been expanded into Gradient Boosting, which has more flexible loss functions. This algorithm is an AdaBoost adaption by the same guy that invented random forests. They are not concerned about theoretical questions like the authors of AdaBoost, they just want to make the algorithm more flexible and efficient. For instance, they also update the weights using all previous weak models, not just the last trained model.

xgboost is just an intelligent implementation of Gradient Boosting. You can see the basic algorithm in wikipedia. Each decision tree is trained sequentially and weights are computed based on the errors from the current ensemble.

I can hear you say:

  What? xgboost trains trees in sequence? How is xgboost so fast then?

This website explains it very well. While trees are trained sequentially, each individual decision tree is trained in parallel by using highly creative techniques. Most importantly, nodes in the same depth do not depend in each other, so you can paralelize them.

Long story short: the loss function is applied between training models.

When it comes to making predictions, then the entire ensemble can be used in parallel.


Dropout in other machine learning models Dropout is a widely used technique in deep learning. Dropout was built for neural networks, but I wonder if other prediction models can use this idea as well as a regularizer. Do you know of any similar technique in linear regression, SVMs or tree-based methods?

Random forests could be thought of as using a kind of dropout-esque technique as each split node only considers a random subset of the features, effectively 'dropping out' the other ones. Also, sometimes in large tree ensembles, each tree is only given a random subset of features to begin with, akin to dropout on the input layer of a neural network.


How to make LightGBM to suppress output? I'm trying for a while to figure out how to "shut up" LightGBM. Especially, I want to suppress the output of LightGBM during training (the feedback on the boosting steps).

My model:

params = {
            'objective': 'regression',
            'learning_rate' :0.9,
            'max_depth' : 1,
            'metric': 'mean_squared_error',
            'seed': 7,
            'boosting_type' : 'gbdt'
        }

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=100000,
                valid_sets=lgb_eval,
                early_stopping_rounds=100)

I tried to add verbose=0 as suggested in the docs. But this does not work.
https://github.com/microsoft/LightGBM/blob/master/docs/Parameters.rst

Does anyone know how to suppress LightGBM output during training?

As @Peter has suggested, setting verbose_eval = -1 suppresses most of LightGBM output (link: here).

However, LightGBM may still return other warnings - e.g. No further splits with positive gain. This can be suppressed as follows (source: here ):

lgb_train = lgb.Dataset(X_train, y_train, params={'verbose': -1}, free_raw_data=False)
lgb_eval = lgb.Dataset(X_test, y_test, params={'verbose': -1},free_raw_data=False)
gbm = lgb.train({'verbose': -1}, lgb_train, valid_sets=lgb_eval, verbose_eval=False)


What is the best way to normalize a set of datasets I have a data set that contains the same Time series "Sensor readings" for different days and I want to make a deep learning model to predict these values. What I did was I splatted the data into Time series data according to the day, then I normalized it separately (min-max) (the readings have different ranges, for example, the max value for the first day is 100 but the max for second is 48) but I'm really confused now do I need to normalize it using the max/min of the all days or what I did was right?

If you know strict bounds on the sensor output, that would be better than normalizing by the min/max of the dataset. Even if the bounds are not necessarily strict, but simply reasonable, that would suffice. For example, if there are no theoretical bounds on a temperature sensor, you might reasonably impose strict bounds given prior knowledge about its environment (e.g. if a temp sensor was placed in NY, you might assume strict bounds as -30C to 50C)
If you were to normalize by the min/max of the training data, what do you expect to happen if the deployed model encounters a value outside this range? If, for example, your training set had min 5 and max 30, how would you normalize an input of -10? It would be much more intuitive and reliable to shift up to a nonnegative domain, (i.e. subtract your strict minima), then scale to a value in [0,1] via the strict range.
Also, depending on your neuronal activation functions, consider centering your data in addition to scaling it.


Is it possible to plot decision boundaries for only a subset of features? I have a sklearn Random Forest classifier with 59 features as input. I'd like to plot the decision boundaries of only two features at indices i1 i2.

If I use the average/median values for the remaining features, the classifier ends up in a path that ignores the features i1/i2.

I'd like to plot the decision boundary of i1/i2 while ignoring the remaining features.
Is this possible? or useful at all?

For decision trees, this can be both possible and useful. For the random forest (RF), each decision tree has their own boundaries for these features. And, since RF trains each tree with a subset of original features, some trees don't have any these features and so boundaries. This might also occur due to pruning. So, you have a set of boundaries, which can be plotted at the same time to get a feeling of how the trees in the forest learnt your dataset. However, these boundaries also depend on other features. If some more explanatory feature is selected in a higher node in some trees, the low level thresholds might mislead you.


Should I rescale tfidf features? I have a dataset which contains both text and numeric features.

I have encoded the text ones using the TfidfVectorizer from sklearn.

I would now like to apply logistic regression to the resulting dataframe.

My issue is that the numeric features aren't on the same scale as the ones resulting from tfidf.

I'm unsure about whether to:

scale the whole dataframe with StandardScaler prior to passing to a classifier;
only scale the numeric features, and leave the ones resulting from tfidf as they are.

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is.

According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange:

  (it's) (...) usually is a two-fold normalization.

  First, each document is normalized to length 1, so there is no bias
  for longer or shorter documents. This equals taking the relative
  frequencies instead of the absolute term counts. This is "TF".

  Second, IDF then is a cross-document normalization, that puts less
  weight on common terms, and more weight on rare terms, by normalizing
  (weighting) each word with the inverse in-corpus frequency.

Tf-Idf is meant to be used in its raw form in an algorithm. Other numerical values are the ones that could be normalized if the algorithm needs normalization or the data is just too small. Other options can be using algorithms resistant to different ranges and distributions like tree based models or simply using regularization, it's up to the cross-validation results really.

But categorical features like bag-of-words, tf-idf or other nlp transformations should be left alone for better results.

However, there is also the idea of normalizing one-hot coded variables as something that can be done as a standard step same as in other datasets. And it's presented by a prominent figure in the field of statistics.

https://stats.stackexchange.com/a/120600/90513


PyTorch: How to use pytorch pretrained for single channel image If I have to create a model in pytorch for images having only single channel. How can I transform my model to adopt to this new architecture without having the need to compromise the pre-trained weights on which it has been trained upon.

I came across a code where the user had this very innovative method to tackle this problem. Here is the small trick to convert any pre-trained network to accept 1 channel images without loosing pertained weights.

arch = models.resnet50(num_classes=1000, pretrained=True)
arch = list(arch.children())
w = arch[0].weight
arch[0] = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=2, bias=False)
arch[0].weight = nn.Parameter(torch.mean(w, dim=1, keepdim=True))
arch = nn.Sequential(*arch)

Basically here, weights of the first Conv2d is taken and stored in w. Afterwards, we are creating new Conv2d which has 1 channel and replacing its parameter with w for which we take mean.
I am pretty sure this can be achieved in keras or tenserflow too.

Let me know if you happen to try it.


Algorithm for detecting parity of integers I'm still a beginner in machine learning and I wanted to do a simple test today with scikit-learn and I couldn't achieved what I expected. I created a dataset in which one column are natural numbers and the other column is whether the number is even or odd, I mean, only 0 or 1. Then I tried several different algorithms and I couldn't predict correctly the output for numbers different than those in the dataset. My question would be this: which algorithms could be useful for this purpose??? I thought for instance Logistic Regression could do the trick or KNN classifier, I'm starting to learn Machine Learning and felt very disappointed that this simple thing didn't work...

Best regards

A fundamentally linear model like logistic regression will never work well, because its assumptions are not at all true for your data set. It presumes that probability (OK, really, log odds) of being positive or negative changes linearly in each input, but, it alternates with each integer value in your input. KNN's assumption likewise does not match. For each integer, its neighbors have an opposite classification. These will never work well as applied to your input, because your input doesn't match their usage.

If however you include the parity of the input (X mod 2) as a feature, they should all trivially learn this model.

This is a pretty good simple lesson in feature engineering and understanding the assumptions you take on when applying an algorithm.


Why do pre-trained CNNs use low image resolution? I want to use a pre-trained convolutional network for image classification. My base data has resolutions of 500x500px up to 1000x1000px. Pre-trained architectures often expect less (between 255 and 299px in case of Googles Inception network).

Firstly: Would it potentially have a big impact to use higher resolution images? I.e. is it worthwhile investigating it?
Secondly: Does it make sense and is it possible to use a pre-trained network on low resolution and re-training the last layer/classifier with higher resolutions?

Would it potentially have a big impact to use higher resolution images?

Yes, if you increase the input size to your convolutional neural network, the size of each activation map for each layer increases, so you will have more computation. Also if you use same architecture, the number of neurons and consequently the number of parameters, in dense layers increases.

  Does it make sense and is it possible to use a pre-trained network on low resolution and re-training the last layer/classifier with higher resolutions?

The answer is no. When you train a network with a special size of input, you reserve variables to hold the weights and middle variables. If you increase the size of input, dense layers will have different size, so their number of wights should vary too.

To wrap up, for networks with classification tasks, it is appropriate to pass the network small size of images. For other tasks like edge detection where the information of edges can be destroyed by resizing, you have to be careful. In those cases you have to find an appropriate size of the image in order to keep the important information. The small size of the inputs is for reducing number of operations and number of parameters.


How does "linear algebraic" weight training function work? This answer shows that linear and polynomial function weights can be trained using this matrix operation:

$w = (X^TX)^{-1}X^Ty$

Therefore, algorithms such as gradient descent are not necessary for these functions. By my understanding, gradient descent for linear regression model finds perfect derivative for each weight so that cost function is minimal.

Before asking for connections between gradient descent and equation above, let's separate the equation in smaller steps:

$X = [1,2,3,4,5,6,7,8,9,10,11,12]$

$y=[2.3,2.33,2.29,2.3,2.36,2.4,2.46,2.5,2.48,2.43,2.38,2.35]$

Let's turn $X$ vector to matrix by adding column of 1's which will be used to train the bias value:

\begin{bmatrix}
    1 & 1 \\
    2 & 1 \\
    3 & 1 \\
    4 & 1 \\
    5 & 1 \\
    6 & 1 \\
    7 & 1 \\
    8 & 1 \\
    9 & 1 \\
    10 & 1 \\
    11 & 1 \\
    12 & 1
  \end{bmatrix}

Transpose: $X^T$:

\begin{bmatrix}
     1 & 2 & 3 & 4 & 5 & 6 & 7 & 9 & 10 & 11 & 12 \\
     1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1
  \end{bmatrix}
Matrix multiplication $X^TX$:

\begin{bmatrix}
     650 & 78 \\
     78 & 12
  \end{bmatrix}
Inverse matrix operation $(X^TX)^{-1}$:

\begin{bmatrix}
     0.00699301 & -0.04545455 \\
     -0.04545455 & 0.37878788
  \end{bmatrix}
Matrix multiplication $(X^TX)^{-1}X^T$:

array([[-0.03846154, -0.03146853, -0.02447552, -0.01748252, -0.01048951,
        -0.0034965 ,  0.0034965 ,  0.01048951,  0.01748252,  0.02447552,
         0.03146853,  0.03846154],
       [ 0.33333333,  0.28787879,  0.24242424,  0.1969697 ,  0.15151515,
         0.10606061,  0.06060606,  0.01515152, -0.03030303, -0.07575758,
        -0.12121212, -0.16666667]])

Matrix vector multiplication $(X^TX)^{-1}X^Ty$:

\begin{bmatrix}
     0.01174825 & 2.30530303
  \end{bmatrix}

This seems to be the perfect slope for minimizing cost. But I'm unable to understand how does it exactly work.

It seems like that the algorithm such as gradient descent would've took extremely large amount of iterations to estimate perfect slope and bias value.

How/why does this equation exactly work? Is it somehow related to differentiation? Could it be compared to algorithms such as gradient descent? If so, how? Is it possible to use this equation with sigmoid functions?

Your goal is to find a $w$ such that

$$Xw \approx y$$

and way to model this problem is to minimize the objective function:

$$\min_w\|Xw-y\|^2.$$

Differentiating with respect to $w$ and equate it to zero gives us

$$2X^T(Xw-y)=0$$

$$X^TXw-X^Ty=0$$

$$X^TXw=X^Ty$$

if $X^TX$ is invertible, then we have

$$w=(X^TX)^{-1}(X^Ty)$$

Remark:

We tend to avoid computing inverse and prefer gradient-based method. Complexity of the normal equation method is cubic.


What Machine Learning Algorithm could I use to determine some measure in a date? I am getting stuck with this problem. Let's say that we have the next information.

CustomerID: 1, Date: 3/2/2018, Quantity: 3, Total: 390.78, Min: 130.26, Max: 130.26

We want to determine given a day, month and year what will be the total summation of sales until the last day of that month, I'm using Microsoft Azure Machine Learning Studio and I modified the data a little bit (I know a little bit of Python and R), for a day we could have the next data.

Date: 3/2/2018, TotalSalesToday: 4023.45, FirstToToday: 1322.92 TargetValue: 42611.27

where TargetValue is the total summation of sales until the last day of the month (or the value that we want to predict), FirstToToday the Summation of the sales from the first day up to that day, TotalSalesToday the total summation of the sales of that day. There are some columns that we can find or generate given the day like RemainingWorkDays, RemainingHolidays, RemainingNonWorkDays, etc. And maybe 31 columns that we can make telling the ML what is the summation of all the days before -1, all the days before -2, and so on.

I did an experiment on Microsoft Azure Machine Learning Studio and it's giving a 100% coefficient of determination (I was using Boosted Decision Tree Regresion and Tunel Model Hyperparameters), I think that's because the ML knows that in a given month the TargetValue doesn't change it's value, so it does something like if(month == 2) PredictValue = 42611.27, what I can do? When testing this ML let's suppose that in the first day we got a TotalSalesToday: 1000000 so my ML returns 50000, and obviously this is not a logical and coherent answer regarding this value (1000000).

Is there something I need to change in the data? What do we need in order to make the ML gives at least a coherent answer? Is there something I forget?

Thanks in advance!! :)

I would calculate the daily deltas of your sales data and implement a time series forecast model by using an Azure ML R/Python model module that predicts the daily deltas until the end of the month and returns the resulting monthly sum.

If you are not familiar with time series analysis, here is a link to a good description of some basic methods and how to apply them using R.

I think, would predict the daily deltas using an ARIMA model. I found this website containing a description of a time series analysis work flow using R with which you could find a good ARIMA model.

I hope, the above is useful for you.


conversion of dictionary to json to be sent to requests.post I have a data frame like this.
df=pd.DataFrame({'sub1':[10,20,30,40],'sub2':[5,10,15,20],'sub3':[1,2,3,4]})

and a dictionary like this.
stud={}
stud['name']="abc"
stud['id'] ="AB10"
stud['address']="some_address"

I convert that data frame to dictionary and combine with the stud dictionary.I wrote the following code.Then this stud dictionary should be made in a json format and sent as a web service.
stud={}
stud['name']="abc"
stud['id'] ="AB10"
stud['address']="some_address"
for i in range(0,len(df)):
   data_sample={}
   data_sample_row=df.iloc[i]
   a=data_sample_row.to_dict()
   stud['data_sample']=a
   str_details=str(stud)

I would like to get the result as shown below.
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':10,'sub2':5,'sub3':1}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':20,'sub2':10,'sub3':2}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1':30,'sub2':15,'sub3': 3}}"
"{'name':'abc','id':'AB10','address':'some_address','data_sample':{'sub1': 40,'sub2':20,'sub3': 4}}"

I would like to send this as a parameter in requests.post as following.
response=requests.post(url="some_url",data=str_details)

If I don't convert to string, gives me an error,
JSONDecodeError: Expecting value: line 2 column 1 (char 1)
If I pass json=str_details, as argument gives me the same error, How to resolve this and get the desired results.

To build your data use sth like:
import copy, json

def make_objects(ref_obj, df):
    objects = []
    for i in range(len(df[df.columns[0]].values)):
        cobj = copy.deepcopy(ref_obj)
        cobj['data_sample'] = {}
        for col in df:
            cobj['data_sample'][col] = int(df[col].values[i]) if df[col].dtype == np.int64 else (float(df[col].values[i]) if df[col].dtype == np.float64 else df[col].values[i])
        objects.append(cobj)
    return objects

df = pd.DataFrame({'sub1':[10,20,30,40],'sub2':[5,10,15,20],'sub3':[1,2,3,4]})
objects = make_objects({'name':"abc",'id':"AB10",'address':"some_address"}, df)
# in case you want to json-encode the returned objects
json_output = json.dumps(objects)


tensorboard showing the epoch loss and accuracy for validation data but not training data I am doing binary classification using CNN and viewing the results using tensorboard. The problem is that the tensorboard is not showing the loss and accuracy for training data.

As seen curves for training data are not shown

It is hard to know what is happening from just that screenshot and no code.

The training and validation plots are usually separated on the page, not lines on the same graph.

If you are using Tensorflow 2.0, there is a known issue, regarding the syncing of TB and the tfevent file (where logs are stored). A couple of things to try:

Try adding the TensorBoard callback with the argument: profile_batch=0
Try restarting the tensorboard a few times... reading can fail or be very slow to load

I am referring to the tensorflow.keras API:

tf.keras.callbacks.TensorBoard(
    log_dir='logs', update_freq='epoch',
    profile_batch=0,                      # <-- default value is 2
)


What exactly is a class in kNN search? I am trying to create a kNN search from scratch for a project. I think I have the concept of it and how it works, but I can't understand what exactly is a class.

I have a matrix of $X*Y$ where $X$ is the number of elements and $Y$ their vector. So if the values are completely random, is every element/row a different class, or do I specify the classes some other way?

Given the context of your question I assume your are referring to the k-NN classifier. The idea of the classification method is you have a set of feature vectors $f_i$, in general, $f_i \in \mathbb{R}^d$  where $d$ is the dimension of the vectors. Additionally you have a class $c_i$ for each $f_i$. To classify an unseen feature vector $f_j$ you select the $k$ nearest neighbors (under some distance, usually the euclidean distance) and the most common class among the neighbors is your prediction.

So, summarizing, the input of the algorithm are the feature vectors $f_i$ and the classes $c_i$. For more on the algorithm see this wikipedia link.

The image (from wikipedia) below illustrates a typical example, in this example the classes are: red triangle,blue square and the point to predict is the green circle. Each feature vector is a two dimensional point, i.e. $f_i, f_j \in \mathbb{R}^2$. In this situation the predicted class will be red triangle.

For an explanation using both Python and R see this.


Does PCA change the values of the data? Principal Component Analysis is a means to reduce the dimensionality of data, if I understand correctly.

So if I have a 1000 sample point 12 dimensional matrix and reduce it to a 1000 sample point 2 dimensional one, then are the values of the sample points themselves changed in some way? or are simply 10 dimensions/columns thrown out and 2 of the original remain?

Yes, the new 2 dimensional values will be a projection of original 12 dimensional points onto the two principle components (vectors).

please refer to the first figure in this clear tutorial:
http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/


Is it better to use F1 score or AUC metric for imbalanced data classification? I have a text classification problem, where the "positive" examples are the minority. What metric is better to use for binary classification for this case - F1-score or AUC?

F1-score and AUC are two evaluation measures for binary classification but they are not comparable:

F1-score measures the performance of a hard classifier, i.e. a system which predicts a class for every instance. Through precision and recall it compares for every instance the predicted class vs. the gold-standard class.
AUC measures the performance of a soft classifier, i.e. a system which predicts a probability (or a score) for every instance. The difference is that the system doesn't decide which class the instance belongs to, so informally it can be seen as an "unfinished classifier". If one decides a threshold on the probability to separate the classes then it becomes a hard classifier.

Both are fine to be used with imbalanced data, that's not a reason to pick one or the other.
AUC is useful to study the general behaviour of a method without deciding a particular threshold. Sometimes the choice of a particular threshold can have a strong impact on performance, so using AUC avoids the issue completely. However strictly speaking AUC doesn't give the performance of a classifier.


How to verify A/B test Is there a way to check the result of A/B test to determine the better one.

Suppose,

Group A has 100 views and 10 clicks.

Group B has 10 views and 2 clicks.

How to chose which one produces more clicks ?

There are ways of understanding which of these is the right one to go with. Unforunately 10 views means no calculation is going to be great for you, but i'll try to explain both.

T-tests and Z-tests are used by a lot of AB Testing tools, as described here, to calculate winners. You'll be able to input views and clicks, and receieve back stats that describe if Group B's treatment is to win. You get back a yes/no answer, based on tolerances provided.

Bayesian calculations, on the other hand, rely on a percentage likelihood of being a winner, after which you can make a judgement.

If possible, you should run this for longer and try to drive more views to Group B. If the interaction rate of 20% persists, you'll see an eventual winner there (100% lift over the 10% rate for Group A).


Should we have only binary labels for LSTM? Is it possible to have non-binary labels for LSTM? I mean an array like

[ 100 120 140 20 50 70]

Instead of

[1 0 1 0 0 1 1]

for example! Isn't this opposite of LSTM's essence and doesn't reduce it's performance?

Welcome to the site. Generally speaking, you can have any labels you need/want and I don't think that the "essence" of LSTM will be affected. Recall that LSTM is special because it has the ability to "forget" and throw away data (mostly data that leads to factor fading). So, your choice of labels will not prevent you from leveraging the wonderful benefits of LSTM.

However, it's possible that you are confusing your question with one-hot-encoding. You should definitely take your labels and convert them to one-hot-encoding during both your training and prediction cycles. You're not changing your labels per-se, you're merely creating them in a way that allows you to better use them in neural networks (all neural networks; one-hot-encoding is not unique to LSTM).


Channels and neurons in a CNN I am in the process of learning what a convolutional layer is, and its functions. I have a couple of questions. One is what in_channels and out_channels are. In my code, I have the following:
nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(4,4), padding=1)

I understand that the in_channels for this layer is 3 because RGB, but I'm confused on why out_channels is 64, and why I can change it to seemingly any other number without the program crashing.
Another question is what exactly a "neuron" is. For convolutional layers, is a neuron each filter? Also, if it is, what do the activation functions judge when evaluating each filter?

out_channels can be thought of as the number of features/filters the layer can learn. So, for eg., if a layer of a CNN has out_channels set to 64, it can "learn" 64 features. These features could be edges/patterns. This is a hyper-parameters and can set to an arbitrary number.
The activation of a neuron in a layer will be high if the input to that layer contains the feature learned by the layer, that's how a convolution (or correlation) function works.
In a CNN, a neuron is the result of computing the dot product of the input and the filter/kernel at every window of the convolution.


Open cv and computer vision I'm new to computer vision, and I'm looking for a good place to start from, what's better between open cv in python or open cv in c++

Depends on your programming skills. Here is the summary:

OpenCV is a great tool originally developed in C++ and after a while a Python interface was added. In industry (and also many academic research group) C++ is the popular language for Computer Vision as the nature of data needs efficiency in computation. So starting with python is an easy way to learn Computer Vision and OpenCV capabilities but when you are applying for a job in industry or academia, they most probably need you to get your hands dirty with C++.


Which policy gradient method is used for continuous action spaces? Which policy gradient method is used that deals with continuous action spaces?

You can use  Soft Actor Critic or PPO from the same website. Most PG methods can be used in discrete and/or continuous action spaces.


Combining multiple neural networks with different activation functions I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values.

P = (P1 + P2 + P3)/3

Where 0 < P1 < 1, -1 < P2 < 1, 0 < P3 < 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?

As you are trying to average out the values, and given the three have different domains, it makes sense to bring all in same domain before averaging out. You can normalize P2 (tanh) to 0-1 and then average the values.

If you want to try another way, you can combine these three networks and take the three outputs p1, p2, and p3 and input it to some more dense layers and finally predict the single output. This way instead of averaging out, you are bringing in some nonlinearity which will be helpful to learn the task.


Getting some data from a csv file into a list of tuples using Python I have a csv file of almost 2 million rows and about 20 columns. I am interested in about 40000 of these rows, and in 5 of the columns 'squat', 'bench', 'deadlift', 'bodyweight', 'total'. I'd like to get tuples of these 5 values, but only of the rows that have 'female' in column 'sex' and 'ipf' in column 'parent federation'. Also sometimes one of the first 5 columns has no value specified, I'd like to ignore those.

I can do this by going row by row, checking the condition and then adding a tuple to a list, but this is way too slow and takes up too much memory. Is there a way to expand on this how to take CSV file input in list of tuples using only a couple of columns and filtering?

you can use Pandas library to solve this problem. pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language. further details can be found here:
Pandas Dataframe Documentations


One hot encoding for multiple label(trainy) in .fit() method? I have a mobile price classification dataset in which I have 20 features and one target variable called price_range. I need to classify mobile prices as low, medium, high, very high.

I have applied a one-hot encoding to my target variable. After that, I split the data into trainX, testX, trainy, testy. So my shape for trainX and trainy is (1600,20) and (1600,4) respectively.

Now when I try to fit trainX and trainy to logisticRegresion, i.e -> lr.fit(trainX,trainy) I am getting an

  error and it says: bad input (1600,4)

So, I understood that I have to give trainy value in shape (1600,1) but by one hot encoding I have got array of 4 columns for each individual price_range as per the concept of one hot encoding.

So now I am totally confused how people use one hot encoding for target variable in practice? please help me out.

So now I am totally confused how people use one hot encoding for
  target variable in practice? please help me out.

This mostly comes down to the tool you are using. Sklearn, which I assume you are using, does not use one hot encoded target variables. So your y should be of dimension (1600, 1) where the classes are 0, 1, 2 and 3. Instead of applying one hot encoding you can use a LabelEncoder to get it on the correct format.

I suspect the reason for your confusion comes from having seen deep learning frameworks such as Tensorflow and Keras. With them you always one hot encode your target variable.

Short answer:

Using sklearn: Label encode
Using deep learning: one hot encode


Relatively high regularization parameters for XGBoost model only way to prevent overfitting I am modeling a continuous regression/forecasting problem for very right-skewed data. I've been using ElasticNet and Huber regression with quite a bit of success, and have recently moved into using XGBoost to see if it'll provide any additional value. The dimensions of my training matrix is 60,000 rows by 500 columns.
What I've found is that the much simpler, more interpretable ElasticNet/Huber regression models very often outperform any XGBoost model I've built. The only way I can get XGBoost to compete is by using a ton of different forms of regularization. In particular: the most performant XGBoost models have had reg_alpha/reg_lambda parameters in the [10-150] range; gamma in the [25, 100] range, subsample of 0.5, colsample_by_tree of 0.5, and shallow max_depths, e.g. 3/4/5, with around 150 n_estimators.
From what I've gathered in various tutorials online, gamma values over 10 or 20 seem to be very high, although I completely acknowledge that statement could be very dependent on the characteristics of the dataset being used.
For this super-regularized model, the predictions and feature importances make sense from an intuitive perspective.
I guess I'm just looking for some input – is it insane that I have such high regularization parameters, or am I more justified than once thought in these high values, since the proof seems to be in the pudding with the model's predictive power/generalizability and important features?

I support your "proof is in the pudding" sentiment.
Some of those hyperparameters are not that extreme, in my experience.  Boosted trees very often perform best with weak individual learners; your max_depth is right in line with what I'm used to seeing as best.  The score regularization penalties (alpha, lambda) don't play as important a role in my experience, but I'm used to seeing optimal parameters chosen in the high double-digits.  Your subsampling and column subsetting rates also seem reasonable, if on the lower end of what I've generally seen as being optimal.  Your gamma is quite high, but that doesn't mean something is wrong; perhaps if you shrink the max depth a bit you could relax the gamma regularization, but I don't think that's in any way necessary.
One possible explanation for this situation: your data is relatively linear and without interactions, so that xgboost doesn't get its main benefits.  And your data is noisy enough that, lacking those nonlinear trends, xgboost ends up fitting to noise readily unless you strongly regularize it.


Eigenvectors and eigenvalues for natural language processing How are eigenvectors and eigenvalues can be applied/applicable to natural language processing problems ? Any examples ?

Latent Semantic Analysis (LSA) relies on linear-algebraic decompositions (e.g. SVD), which in turn involve eigenvectors/values (see here).

Not sure if this is quite what your question is driving at, but in general, eigenvectors are useful in data analysis because they define some "natural" direction in the data. For example, the eigenvectors of the covariance matrix of some data are all perpendicular to one another, and the i'th one points in the direction of i'th greatest variance in the data. In other words, the first eigenvector points in the direction where the data's variance is greatest, the second one in (perpendicular) direction of second-greatest variance, and so on.
This is why eigenvector-based decompositions can be so useful in feature selection and dimensionality reduction. In informal terms, they transform the data into perpendicular features that are aligned with some "natural" axes in the data.


When using an unsupervised alogirthm, what is the "learning" part since it belong to machine learning field? I had a brief experience with machine learning by using a clustering algorithm, i also read the basic ideas and calculations of a simple classification algorithm.
Now, i would read more about "machine learning" and I found many similar definitions like the following:

"Machine learning is the science of getting computers to act without being explicitly programmed..."
"Machine learning is a type of artificial intelligence (AI) that allows software applications to become more accurate in predicting outcomes without being explicitly programmed... "

My understanding is that the ability to learn and adapting deduction (output) without reprogrammation is the main idea, and based on my personal understanding, this "adaptation" is only possible with "supervised algorithm" with a new training which can permit a change/adaptation/improvement on the output model with the same program and source code.
So based on my understanding again, this "adaptation" and "learning" definition doesn't fit unsupervised machine learning algorithms, since the model with all calculations is fixed and implemented! Any change will need an update on source code!
Therefore, I would have corrections to my misinterpretation, and more clarifications to have a better understanding of "machine learning" and unsupervised/supervised learning relation.

You said:

  and based on my personal understanding, this "adaptation" is only possible with "supervised algorithm"

However, it is not true. Let's consider a clustering method as an unsupervised algorithm like K-means. Each iteration in K-means algorithm is a basis for the next iteration of the K-means up to the convergence of the algorithm. Hence, it means we are learning the structure of the data in each iteration up to reaching a specific target value. Although it is unsupervised, we are learning the structure of the data in each iteration. Indeed, it is a kind of (unsupervised) learning at all.


Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax? I'm trying to figure out why when using decision trees for multi class classification it is common to calculate a score and apply softmax, instead of just taking the averages of the terminal nodes probabilities?

Let's say our model is two trees. A terminal node of tree 1 has example 14 in a node with 20% class 1, 60% class 2, and 20% class 3. A terminal node of tree 2 has example 14 in a node with 100% class 2. Then our prediction for training example 14 is [10%, 80%, 10%].

Why use Softmax instead of this averaging approach?

Note: I am looking to apply this knowledge to understanding xgboost better, as well as a simple 1 tree classification model.

Depending on the parameters you used for your model, it may not be calibrated in probabilities. That is, your model output a score, that is helpfull to give a relative order between your instance, but the score may not reflect the real % chance of the output happening.

Softmax, will at least garanty that your output are between 0 and 1 and sum to one. This will give you an output that is nearer to a probability (but that may not be enough to be calibrated to 'historical' probabilities).

PS: I don't think there is a link with xgboost, unless you have very specific options.


How will ADA Boost be used for solving regression problems? I have an idea of how ADABOOST will be used for classification but I want to get the idea of how to re-weight and thus use ADABOOST in case of regression problems.

Here are link to some famous boost of regressor.

Adaboost.R2: Improving Regressors using Boosting Techniques
Adaboost.RT: Experiments with AdaBoost.RT, an Improved Boosting
Scheme for Regression

Scikit-Learn have many implementations:

AdaBoostRegressor
Decision Tree Regression with AdaBoost


Understanding ConfusionMatrix for Google Prediction API I'm trying to analyze my training model, Google Prediction API provides analyze method to get insights for the model.
Currently I want to improve confidence levels, for my predictions, I haven't found any documentation in how to read this ConfusionMatrix, any insights will be great:

{
 "kind": "prediction#analyze",
 "id": "modelX",
 "selfLink": "https://www.googleapis.com/prediction/v1.6/projects/projectX/trainedmodels/modelX/analyze",
 "dataDescription": {
  "outputFeature": {
   "text": [
    {
     "value": "labelA",
     "count": "681"
    },
    {
     "value": "labelB",
     "count": "127"
    },
    {
     "value": "labelC",
     "count": "814"
    },
    {
     "value": "labelD",
     "count": "427"
    }
   ]
  },
  "features": [
   {
    "index": "0",
    "text": {
     "count": "2049"
    }
   }
  ]
 },
 "modelDescription": {
  "modelinfo": {
   "kind": "prediction#training"
  },
  "confusionMatrix": {
   "labelA": {
    "labelA": "14.17",
    "labelB": "0.17",
    "labelC": "3.83",
    "labelD": "0.67"
   },
   "labelB": {
    "labelA": "0.50",
    "labelB": "2.00",
    "labelC": "1.33",
    "labelD": "0.00"
   },
   "labelC": {
    "labelA": "1.17",
    "labelB": "0.00",
    "labelC": "70.00",
    "labelD": "3.50"
   },
   "labelD": {
    "labelA": "1.17",
    "labelB": "0.17",
    "labelC": "4.17",
    "labelD": "12.17"
   }
  },
  "confusionMatrixRowTotals": {
   "labelA": "18.83",
   "labelB": "3.83",
   "labelC": "74.67",
   "labelD": "17.67"
  }
 }
}

As explained in the documentation:

  This shows an estimate for how this model will do in predictions. This is first indexed by the true class label. For each true class label, this provides a pair {predicted_label, count}, where count is the estimated number of times the model will predict the predicted label given the true label.

If you are not sure what a confusion matrix, see Wikipedia, where the "actual class" refers to the same thing as the "true class" in the Google documentation.


Python metaheuristic packages I need to use a metaheuristic algorithm to solve an optimization problem on a Python codebase.

Metaheuristics usually need to be written in C++ or Java as they involve a lot of iterations, while Python is weak from this point of view.

Questions:

do any Python metaheuristic packages which wrap faster languages as
C++/Java exist?
do any Python metaheuristic packages based on maybe
cython on numba exist?
other solutions?

As a long shot, if you have a Python version (or implement it yourself), running it with PyPy instead of Python might make things much faster, as it is well suited to code that uses Python built-ins and also many loops. It optimises these cases very well, through tricks such as garbage collection. The latest version also supports Python3.5 and 3.6 as well as NumPy.

From their website the main benefits:

  Speed: thanks to its Just-in-Time compiler, Python programs often run
  faster on PyPy. (What is a JIT compiler?)

  “If you want your code to run faster, you should probably just use
  PyPy.” — Guido van Rossum (creator of Python)

  Memory usage: memory-hungry Python programs (several hundreds of MBs
  or more) might end up taking less space than they do in CPython.

  Compatibility: PyPy is highly compatible with existing python code. It
  supports cffi and can run popular python libraries like twisted and
  django.

  Stackless: PyPy comes by default with support for stackless mode,
  providing micro-threads for massive concurrency.


Can the same CNN architecture be used for different data sets? I have a CNN architecture that works well on 32x32x3 images. Can I use that same architecture for a data set made up of 28x28x1 images? (Both data sets have 10 classes). If this is possible, what changes would I need to make to the architecture I have?

One way is to redesign your CNN architecture to fit your new data input. However, if you want to use the current model you have, you should:

(1) Retraining: Re-train the model with new data set images

(2) Reshape new images: You should add a pre-processing layer to convert 28x28x1 images to 32x32x3 images. This could be done by replicating a single channel in other two channels and adding boundaries to your images. This deformation process should be the same on all new input images.


Real world example of Generative model & Discriminative model After going over the math/stats behind the Generative model &  Discriminative model, I still have no intuition about it. Does any one have a good real-world example or use
 case for Generative model and  Discriminative model ? Thanks!

A generative model is able to generate instances from a given distribution. So let's try to get our model to generate instance from the distribution of all hand written digits. This includes numbers 0 to 9 and for each of these numbers there are a very large set of possible way of writing the same number. Take the number 4 from the MNIST dataset, which are comprised of handwritten numbers.

You can quickly see how many different ways each of these numbers can be written yet still represent the same value. This is the distribution we want to capture using our generative model. The model works by training a generator to effectively try and cheat a discriminator that has been trained to distinguish between real and artificial instances from the desired distribution. When the discriminator can no longer distinguish between the artificial and real instances, we can say the generative model can successfully produce examples which match this distribution.

In other words, once the generator starts producing realistic digits, we can say that the generator is producing instances which match the instances captured in the MNIST dataset.

This idea of matching a distribution can be applied to  wide range of applications where you want to draw from a distribution. Some for demonstration purposes of the techniques without any real useful application and some very useful applications which are currently being used for a number of tasks. Here is a short lists of some examples:

Generating artificial faces
Text to image generation
Producing fake voices or noises
Image denoising
Changing geometries within an image
Coloring images
MRI image reconstruction

Most importantly, to generate new Pokemon.

In theory, given sufficient data, any distribution (anything in the world, including the world itself) can be modeled by a generative model from which we can draw instances. So any time you think of a simulation in the far future some generative model will be able to observe multiple real examples and then produce a large number of simulated examples. For example, simulating car crashes, a generative model can visualize multiple real crashes, then we can use the generative model to produce millions of similar instances for testing purposes without having to buy more cars. Of course GANs are not that powerful, they require a lot of data to learn very constraint distributions. However, through innovation anything is possible, until then we can continue to generate images of handwritten numbers and bedroom scenes.


Clusters with bounded diameter In my application, I want to have clusters whose diameters are bounded by some fixed number. Also, the number of clusters in the data is unknown and, therefore, the clusters must be discovered without a 'k' parameter.
What method should I use?

Hierarchical clustering, with complete linkage will find clusters with a maximum pairwise distance i.e. diameter.

You need two parameters:

Distance function
Maximum distance = height where to cut the tree


How do you make an NN for image classification invariant to translation **and** rotation? CNN are (I think) invariant to small translations of the input image, i.e., they will classify to the same class an image $X$ and an image $X'$ such that all pixels have been translated along a vector $\mathbf{v}$ by some "small" distance $d$. They're not, however, invariant to rotation. I.e., if $X'$ is obtained by $X$ applying a rotation of arbitrary degree $\theta$ around an axis $z$ orthogonal to the plane of the image, they won't necessarily classify it to the same class as $X$. Practitioners usually solve this by data augmentation, but this is unnecessarily wasteful and anyway not an option in my case. Is there a way to make the NN architecture invariant to both isometries? I don't have to use a CNN, but since CNN already enjoy some sort of translation invariance, I figured it would be easier to modify them to get rotational invariance , than to use a completely different architecture.

It is not possible to have general rotationally-invariant neural network architecture for a CNN*. In fact CNNs are not strongly translation invariant, except due to pooling - instead they combine a little bit of translation invariance with translation equivariance. There is no equivalent to pooling layers that would reduce the effect of rotation this way (although for very small rotations the translation invariance will still help).

You can however construct features and create a pipeline that reduces incoming rotation differences in your inputs. For example, this answer on Signal Processing Stack Exchange suggests calculating dominant gradient of an image, then rotating the image so that this is always oriented the same way before further processing. If your image has strong straight edges, you could do similar by detecting those (e.g. by using a Hough transform) and rotating the input so that these are always oriented the same way. These approaches work only within certain image tasks, but can save on processing time and potentially increase accuracy if they are possible. Effectively they are a form of input normalisation.

A more radical idea might be to perform a map to polar co-ordinates in your image before processing. This would effectively convert rotational (and radial) variance into translation variance in your image. A CNN processing this mapped image would effectively convert its translation invariance into rotational invariance on the original unmapped image. But the cost would be losing all translation invariance, so only worth considering if your inputs have high variance in rotation but low variance in translation.

* Never say never. This is caveated by special cases, for example the paper Group Equivariant Convolutional Networks explains an architecture that adds support for multiples of 90 degree rotation (taking advantage of grid structure in computer images and the weight matrices that construct neural network layers). However, if you want to support free rotation of values other than 90, 180, 270 degrees, then as far as I know, there is no way to do that architecturally within the network.


What is the correct file extension for Orange image embedding? I am using Orange data mining in order to create a facial recognition workflow with some prototype widgets. To do this, I am trying to import some images (test data) using the import widget.
My question is: which file format do I need to use in order for the image data to be compatible with the import widget? In other words, which file format should I use to upload the image?
I have tried using JPEG but when I click "browse for images" to search for images on my desktop, it doesn't allow me to select anything to import.
Does this imply that it is not compatible? Is there a specific file format that I should be using in order to do the image detection/embedding process?
Thanks.

As r.kfr said: Import Images works with folders, not individual images. Place all images in a folder and select it for import.
Note that if you have images in different folders, Orange will consider each folder name as a class label for images.

Alternatively, you can create an Excel/csv file and define the path to the image. Mark it with meta type=image in the third header row. Here is how to define the header.


Preprocessing of Sudoku Dataset from Kaggle Dataset: https://www.kaggle.com/bryanpark/sudoku

I would like to create a neural network for this Dataset.

Feature:

X:  [ '004300209005009001070060043006002087190007400050083000600000105003508690042910300'
 '040100050107003960520008000000000017000906800803050620090060543600080700250097100']

Output:

Y:  [ '864371259325849761971265843436192587198657432257483916689734125713528694542916378'
 '346179258187523964529648371965832417472916835813754629798261543631485792254397186']

The zeros represent a blank box, and the data is a flattened grid of 9x9.

I tried using the following code, but I found out that the data needs a significant amount of preprocessing.

def preprocess():

data = pd.read_csv('Sudoku/sudoku.csv')
print('Data: ', data.head())

x = data[data.columns[0]].values
y = data[data.columns[1]].values

x = x.reshape(-1, 1)
y = y.reshape(-1, 1)

print('\nx: ', x[0])
print('\ny: ', y[0])

return x, y

x, y = preprocess()
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.2)

scl = StandardScaler()
train_x = scl.fit_transform(train_x)
train_y = scl.fit_transform(train_y)
test_x = scl.fit_transform(test_x)
test_y = scl.fit_transform(test_y)

model = Sequential()
model.add(Dense(100, input_dim=1, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(train_x, train_y, epochs=3, validation_split=0.1, verbose=2)

I would like to know how to work with this data and of course, also the structure of my neural network.

You need to convert this large list of strings to a numpy array containing lists of integers. Here is how you can get the data into the correct format for a neural network:

import pandas as pd
from numpy import array

def preprocess():

    data = pd.read_csv('sudoku.csv')

    x = data[data.columns[0]].values
    y = data[data.columns[1]].values

    x = x.reshape(-1, 1)
    y = y.reshape(-1, 1)

    x = array([list(map(int,set[0])) for set in x])
    y = array([list(map(int,set[0])) for set in y])

    return x, y

x, y = preprocess()

print('\nx: ', x[0])
print('\ny: ', y[0])

I haven't thought enough about the problem to advise on how best to go about solving it, but at least this should get you started on playing around with different methods. Be aware that it will take a minute or so to run, as the script is now converting a large amount of strings to integers.


Is it possible to use word2Vec to derive hyperonymy (hyponymy or ISA relation)? It's easy to have hyperonymy in WordNet, e.g. to know that "tea" is a case of "beverage". Is it possible to use word2Vec in this way?

A paper entitled "Dependency-Based Word Embeddings" found that word embeddings that defined context using dependency-based parsing generated cohyponyms. For example, "florida" was embedding near  "texas", "louisiana", "georgia", "california", and "carolina".


Sklearn: applying cost complexity pruning along with pipeline I have a data set with categorical variables. I have defined a decision tree algorithm and transformed these columns to numerical equivalent using one hot encoding functionality in sklearn:
Create Decision Tree classifer object:
clf2 = DecisionTreeClassifier(criterion = 'entropy')
pipe = make_pipeline(column_trans, clf2)            # (1)
pipe.fit(X_train2,y_train2)

where:
column_trans = make_column_transformer(
            (OneHotEncoder(),['ShelveLoc','Urban','US']),
             remainder = 'passthrough')

Now when I built the decision tree without using sklearn but using pandas directly for categorical feature encoding, I was able to find the suitable candidates for alpha to prune the decision tree by
path = clf.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas = path.ccp_alphas
ccp_alphas = ccp_alphas[:-1] #remove max value of alpha

where as now given that my model is baked into pipe argument in (1) when I try to find candidate alphas
path = pipe.cost_complexity_pruning_path(X_train2, y_train2)

I get an error message saying pipe does not have the attribute called cost complexity pruning. and looking at all the attributes available to pipe, I can't find cost complexity pruning as well.
Is it only possible to do cost complexity pruning if you are building the model without using the pipe functionality in Sklearn?

Pipelines themselves don't generally carry the methods and attributes of the final estimator, aside from basics like predict, predict_proba, transform.  If you need to access a method of a step, you should access the step itself using one of:
pipe[-1]
pipe['decisiontreeclassifier']
pipe.named_steps['decisiontreeclassifier']

However, in this case it's a little trickier, because cost_complexity_pruning_path needs the dataset X, y, but you need your pipeline's transformer to apply to it first.  It's a little cumbersome, but I think this should work and is relatively straightforward:
pipe[-1].cost_complexity_pruning_path(
    pipe[:-1].transform(X),
    y,
)

(Note that pipe[-1] is the final estimator in the pipeline, and pipe[:-1] is every step except the last.)


bias and variance trade off related question I am having difficulty to understand the expected squared errors formula in this website:

$y=f(x)+e$  true regression line

$\hat{y}=\hat{f}(x)$   your estimated regression line

$error(x)=\bigg(\mathbb{E}[\hat{f}(x)]-f(x)\bigg)^2+\mathbb{E}\bigg[(\hat{f}(x)-\mathbb{E}[\hat{f}(x)]\bigg]^2+var(e)
        =bias^2+variance+\text{irreducible error}$

my question is what is the best way to understand bias and variance from this mathematical equation?

my understanding of variance here is:

assume you collect a sample with 1000 records and you generate regression line based on this sample. You get a model: $\hat{f}(x)=1+2x$

so at a given x value, for example: $x=1$ then $\hat{f}(x)=1+2*1=3$

then you re-sample another 1000 records and generate regression line:

$\hat{f}(x)=2+3x$

when $x=1$, $\hat{f}(x)=5$

you repeat this process for N times (assume N is a large number and close to infinity)

then your $E[\hat{f}(x=1)]$ is $(3+5+\dots+N)/N$

assume your $E[\hat{f}(x=1)]$ is $5.5$

once you have calculated $\mathbb{E}[\hat{f}(x)]$ for a given value of x=1, you should be able to compute $\mathbb{E}\bigg[\hat{f}(x)-\mathbb{E}[\hat{f}(x)])^2\bigg]$

$((3-5.5)^2+(5-5.5)^2+...)/(n-1)$

my interpretation above is focusing on variance of $\hat{Y}(x)$ at a each given $x$-point(re sampling to get $\hat{y}(x)$)

but my instinct tells me I should calculate based on only one set of data (1000 records) and calculate $\mathbb{E}[\hat{Y}(x)]$ at different x point.

please let me know which one of interpretation is correct or if both are wrong, please explain using this formula how I should interpret the terms in this formula.

The bias of an estimator $\hat{f}(x)$ of $f(x)$ is by definition:
$$Bias(\hat{f},f)(x)=E[\hat{f}(x)]-f(x)$$
which is some definite (not-random) function $\phi(x)$.
Why this is so? The intuitive way to understand it is that estimation itself is a random process. Given random data will produce a random estimation. However the expected estimation is $E[\hat{f}(x)]$, so it is only natural to define bias as the difference between expected estimation and true value.
In case the bias of the estimator $\hat{f}(x)$ (for $f(x)$) is zero, in other words:
$$E\left[\hat{f}(x)\right] = f(x)$$
Then the estimator is called unbiased. It has no (systematic) bias in the estimation process and the expected estimation coincides with the true value. If furthermore, this estimator has minimum possible variance is called minimum variance unbiased estimator. All these are part of the basic theory of statistics.
The linked tutorial you mention simply computes the total Mean Square Error (MSE) by the various components it can be split into (a general theorem for any type of estimation process):
MSE = (irreducible noise error) + (estimator variance) + (bias squared)
or in other words:
$$MSE(x) = Var[\epsilon] + Var[\hat{f}(x)] + Bias(\hat{f},f)(x)^2$$
The estimator variance is the same formula as for any other random variable and is certainly part of the total MSE.
Derivation:

Suppose $y = f(x) + \epsilon$, $E[\epsilon]  = 0$, $Var[\epsilon] = \sigma^2$
For any random variable $r$ we have $E[r^2] = Var[r] + E[r]^2$ (by the definition of variance).
For any non-random value $c$ we have $E[c] = c$, $Var[c] = 0$
$E[y] = E[f(x) + \epsilon] = E[f(x)] + E[\epsilon] = E[f(x)] = f(x)$
Similarly $Var[y] = Var[\epsilon]$ (expand the definition and use previous facts)
Since $\hat{f}$ and $\epsilon$ can be taken as independent, we can write the total error as:

${\displaystyle {\begin{aligned}
\operatorname {E} {\big [}(y-{\hat {f}})^{2}{\big ]}
&=\operatorname {E} {\big [}(f+\varepsilon -{\hat {f}})^{2}{\big ]}\\[5pt]
&=\operatorname {E} {\big [}(f+\varepsilon -{\hat {f}}+\operatorname {E} [{\hat {f}}]-\operatorname {E} [{\hat {f}}])^{2}{\big ]}\\[5pt]
&=\operatorname {E} {\big [}(f-\operatorname {E} [{\hat {f}}])^{2}{\big ]}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}+2\operatorname {E} {\big [}(f-\operatorname {E} [{\hat {f}}])\varepsilon {\big ]}+2\operatorname {E} {\big [}\varepsilon (\operatorname {E} [{\hat {f}}]-{\hat {f}}){\big ]}+2\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})(f-\operatorname {E} [{\hat {f}}]){\big ]}\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}+2(f-\operatorname {E} [{\hat {f}}])\operatorname {E} [\varepsilon ]+2\operatorname {E} [\varepsilon ]\operatorname {E} {\big [}\operatorname {E} [{\hat {f}}]-{\hat {f}}{\big ]}+2\operatorname {E} {\big [}\operatorname {E} [{\hat {f}}]-{\hat {f}}{\big ]}(f-\operatorname {E} [{\hat {f}}])\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {E} [\varepsilon ^{2}]+\operatorname {E} {\big [}(\operatorname {E} [{\hat {f}}]-{\hat {f}})^{2}{\big ]}\\[5pt]
&=(f-\operatorname {E} [{\hat {f}}])^{2}+\operatorname {Var} [\varepsilon ]+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\\[5pt]&=\operatorname {Bias} [{\hat {f}}]^{2}+\operatorname {Var} [\varepsilon ]+\operatorname {Var} {\big [}{\hat {f}}{\big ]}\\[5pt].
\end{aligned}}}$
Why MSE? It is a smooth suitable metric for the accuracy which has nice analytic properties. Other types of error are also possible.
Further reading (and derivations and references therein):

Bias of an estimator
Bias–variance tradeoff
Derivation of the Bias-Variance Tradeoff


Large amount of Sigmoid outputs are ones and zeros I have Keras neural network for binary classification with final layer having one output with Sigmoid activation. I have noticed that large amount of output numbers are strictly one or zero (rather than between 0 and 1 as expected). What could be the reason for this?

At first I thought maybe network is 100% sure about accuracy of these numbers, but noticed that some of these predictions are actually incorrect.

Edit:

Model:

model = tf.keras.models.Sequential()

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu, input_shape=(X_train.shape[1],)))

model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(1, activation = tf.nn.sigmoid))

model.compile(optimizer = tf.keras.optimizers.Adam(), loss = 'binary_crossentropy', metrics = ['accuracy', roc_auc])

model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test))

y_pred = model.predict(X_test)

yhat = []

for pred in y_pred:
    if pred >= 0.7:
        yhat.append(1)
    else:
        yhat.append(0)

Data consists of 8 columns (features) and 270 000 rows (9th column is 'y' column). Out of these 270,000 rows only 9% contained labels of class 1 (rest were zero), so I downsampled the data (just removed bunch of data with label 0), trained the model and then did the prediction on full data. I modified how ones and zeros were determined by Sigmoid though, I changed threshold from 0.5 to 0.7 (which was the ROC score I got on downsampled data)

Apparently the problem was that I forgot to normalize data. As I found out, sigmoid turns any input greater than 10 into 1 and inputs less than -10 into 0. Normalization solves this problem (as normalized values are between 0 and 1)


Why do I not get 100% Accuracy with KNN with $K=1$ I am playing with KNN on the Iris Dataset

I expected to get 100% accuracy with $K=1$ since every point should predict itself based on the Voronoi volume around it created by the KNN algorithm.

However using Scikit Learn I do not find this result. Here is my code.

import pandas as pd
import numpy as np

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix
from sklearn.datasets import load_iris

iris = load_iris()

X = pd.DataFrame(iris['data'])

X.columns = ['sepalLength','sepalWidth','petalLength','petalWidth']

featureX = 'sepalLength'
featureY = 'sepalWidth'

X_2 = X[[featureX,featureY]]

y = iris['target']

y = (np.array(y)==2).astype(np.int)

knn = KNeighborsClassifier(n_neighbors=1, p=2)
knn.fit(X_2, y)
y_pred = knn.predict(X_2)
cm = confusion_matrix(y, y_pred)

print(cm)

I get the following output for the confusion matrix. I see 11 out of 150 samples are incorrectly classified.

[[95  5]

 [ 6 44]]

Why is it not perfect ?

In your training set (X_2,y), there are some samples with the same input features X_2 but different labels y. For example, the 73rd and 147th samples, which are labelled into class 0 and 1, respectively, have the same input values [6.3, 2.5]. There are more samples like this in the dataset. Therefore, you could not construct a perfect classifier for such data.


Cannot interpret feed_dict key as Tensor: Tensor Tensor(“Placeholder:0”, shape=(3, 3, 3, 32), dtype=float32) is not an element of this graph I have a working Keras model that makes predictions great in the repl but fails to load in a Flask app on accessing multiple times..

K.clear_session() and graph = tf.get_default_graph() did not work for me.

model1 = Sequential()

model1.add(Conv2D(32, 3, padding='same', input_shape=(256,256,3), activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("First layer...")
model1.add(Conv2D(32, 3, padding='same', activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("Second layer...")
model1.add(Conv2D(64, 3, padding='same', activation='relu'))
model1.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

model1.add(Flatten())
model1.add(Dense(64))
model1.add(Activation('relu'))
model1.add(Dropout(0.5))
model1.add(Dense(1))
model1.add(Activation('sigmoid'))

model1.load_weights("Weights_model1.h5")

model2 = Sequential()

model2.add(Conv2D(32, 3, padding='same', input_shape=(256,256,3), activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("First layer...")
model2.add(Conv2D(32, 3, padding='same', activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

#print("Second layer...")
model2.add(Conv2D(64, 3, padding='same', activation='relu'))
model2.add(MaxPooling2D(pool_size=(2, 2), data_format="channels_first"))

model2.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors
model2.add(Dense(64))
model2.add(Activation('relu'))
model2.add(Dropout(0.5))
model2.add(Dense(1))
model2.add(Activation('sigmoid'))

model2.load_weights("Weights_model2.h5")

Training_water_data = 'C:\\Users\\ysmehta\\'+folder+'\\'
training_water_data = [x for x in sorted(os.listdir(Training_water_data))]
x_train_water_data = np.empty((len(training_water_data),256,256,3),dtype = 'float32')
for i,name  in enumerate(training_water_data):
    im = cv2.imread(Training_water_data + name).astype('int16').astype('float32')/255.
    im = im[:,:,:]
    im = cv2.resize(im,dsize = (256,256),interpolation = cv2.INTER_NEAREST)
    x_train_water_data[i] = im

x_train_water_data = x_train_water_data.reshape(x_train_water_data.shape[0],256,256,3)

for i in range(0,x_train_water_data.shape[0]):
    A = x_train_water_data[i]
    B = A.reshape(1,256,256,3)
    prediction1 = model1.predict(B,verbose=0)
    if (prediction1 >= 0.7):
    prediction2 = model2.predict(B, verbose=0)
else:
    returnstring="No manhole found"
    prediction2 = -1
if(prediction2 >= 0.6):
    returnstring="Water Leakage"

else:
    if(prediction1 >= 0.7):
        returnstring="Not a Leakage"
return returnstring;

if __name__ == "__main__":
    app.run(host='131.163.144.196',port=5000)`

I had the same issue in Flask. The following code resolved it for me:

from keras import backend as K
K.clear_session()

Also read this discussion.


Machine learning approach for predicting set members Say I have a large training dataset containing sets of 40 items each, and each item in the set is unique (so every training input is a set $S=\{i_1, i_2, ..., i_{40}\}$), and there are more than 40 unique items that can be part of a set.

I would like to be able to predict which items are a probably member of a set, given some incomplete set. So let's take the following example:

Training data:
$S_1 = \{1,2,3\}$,
$S_2 = \{3,4,5\}$,
$S_3 = \{6,7,8\}$
Say I then have an input $S_4 = \{3\}$, I would want the approach to give back that 1, 2, 4 and 5 are more probable set members than 7, 8. Ideally with some probability value.

I've considered the following:
Using the apriori algorithm to learn some association rules. I wasn't sure how to interpret the support or lift as a probability of set membership.

Training a Multilayer perceptron on the input (probably one-hot encoded) to learn weights corresponding to the various input items. However, if I were to simply give the 40-item sets as input and output then the network would just learn to copy the input, giving no information about possible other set members. I've thought about giving all variations of the 40 item set as input, with the 40 item set as output, but this would result in $2^{40}$ possibilities per input which would be massive.

Is there some machine learning approach or data structure that could help in this situation?

You could train an embedding model. Each element would be projected onto a location a vector space based on its co-occurrence with other elements. Then finding similar elements could be done with a nearest neighbor search.


how to create new columns in pandas using some rows of existing columns? i have a dataset like this

my desire format is like this

I tried using index slicing eg
dll.loc[:4,'category'] = "CAPITAL FUND"
dll.loc[5:10,'category'] = "BORROWING"
but this idea is risky so is there any idea to solve this?

Instead of perhaps iterating of each row and filling the gaps as required, I would suggest trying to do it via indexing. The solution is:

df['category'] = df.where(~df.id.isnull())['item'].ffill()

Here I break down my solution to help you understand why it works.

Imagine your dataframe is called df. I created a small version of yours as follows:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame.from_dict(
            {'id':   [1, None, None, 2, None, None, 3, None, None],
             'item': ['CAPITAL FUND', 'A', 'B', 'BORROWINGS', 'A', 'B', 'DEPOSITS', 'A', 'B']})

In [3]: df       # see what it looks like

Out[3]:

 id          item
0  1.0  CAPITAL FUND
1  NaN             A
2  NaN             B
3  2.0    BORROWINGS
4  NaN             A
5  NaN             B
6  3.0      DEPOSITS
7  NaN             A
8  NaN             B

I get the dataframe back where the id column is not null (~ reverses the isnull()).  On the resulting dataframe, I take only the item column (using [item]) and then fill the missing gaps, using the previous valid value in that column.

In [4]: df['category'] = df.where(~df.id.isnull())['item'].ffill()

In [5]: df
Out[5]:
    id          item      category
0  1.0  CAPITAL FUND  CAPITAL FUND
1  NaN             A  CAPITAL FUND
2  NaN             B  CAPITAL FUND
3  2.0    BORROWINGS    BORROWINGS
4  NaN             A    BORROWINGS
5  NaN             B    BORROWINGS
6  3.0      DEPOSITS      DEPOSITS
7  NaN             A      DEPOSITS
8  NaN             B      DEPOSITS

The trick is to understand this part: df.where(~df.id.isnull())['item']

It returns really the whole dataframe, with the values where ~df.id.isnull() is True. Then only the item dataframe. The result is this:

In [6]: df.where(~df.id.isnull())['item']
Out[6]:
0    CAPITAL FUND
1             NaN
2             NaN
3      BORROWINGS
4             NaN
5             NaN
6        DEPOSITS
7             NaN
8             NaN

Now it should be clear why the final .ffill() works as we would like. It forward fills the missing values, using the last known valid value.


Dealing with data with no classes I am new for data science subject and I am trying to learn on my own. Please bear with me if this is a stupid question. I have a dataset that has no classes. The data set comprises people's activity on Office 365 and my goal is to predict whether the person is experiencing pressure or not using machine learning algorithms, but the dataset does not have classes as you all would expect. Can you please help me how I can solve this problem? Is there any possible way to create classes prior to applying ML models on the data?

Thanks in advance.

In absence of class labels, your best choices are either an unsupervised machine learning approach (e.g., k-means clustering), or leverage descriptive statistics to see whether you can find any pattern in the data have. For example, looking at the distribution of your features may reveal outliers that could be associated with your outcome of interest.


If you look at the source code, the PCA is calculated through the SVD. I believe it iterates until "good enough."
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/decomposition/pca.py


Multicollinearity and impact of individual features Assume the following scenario:

I have four features: $x_1$, $x_2$, $x_3$, and $x_4$
There are non-negligible multi-collinearity among the features.
I want to predict $y$ (response variable) with those 4 features.
I use simple multiple linear regression model: $y = a_1x_1 + a_2x_2 + a_3x_3 + a_4x_4$

Let's say that I want to understand the impact that different components of a chair have on the chair's retail price. For example:

$y\,\,\,$ = chair's retail price

$x_1$ = color of cushion used

$x_2$ = overall design of a chair

$x_3$ = strength of a chair

$x_4$ = softness of a chair

$x_1$ is completely independent, but other features are all somewhat impacted by the other features due to multicollinearity. For example, changing the color of cushion changes the design of a chair. Changing the design (structure) of a chair changes the strength of a chair.

I've heard that the analysis of regression coefficients are unreliable under severe multi-collinearity.

Assuming that the multiple regression model fits the chair price well, can I naively use each feature's regression coefficient to understand the impact of each feature on the response variable? If not, what technique should I use?

Ex 1: If I use a red cushion ($x_1$), I can increased the retail price by 3 dollars

Ex 2: If I use conference room style chair ($x_2$), I can increase the retail price by 12 dollars

When you face multicollinearity, your regression coefficients will likely be biased, because under multicollinearity, the regression can not tell apart the different effects: https://datascience.stackexchange.com/a/57118/71442.

When you use only one variable at a time, you will face the omitted variable bias, because there are no other confounders, to which the regression could attribute relevant effects. https://en.wikipedia.org/wiki/Omitted-variable_bias

To my best knowledge, there is no easy-to-go way to mitigate one or both of the described effects. You should carefully check the correlation between your $x$ and decide if multicollinearity is a problem. If this is the case, and you believe (for theoretical reasons), that all the highly correlated $x$ are important, you could try to find other representations of (some) $x$ to mitigate multicollinearity, e.g. dummy/indicator representations.

I guess your chair example is generic, so I don‘t speculate on this. Maybe you can give some more background regarding the actual problem.

The discussion above refers to causal modeling. If you are only interested in making predictions (not statistical inference), you could look into shrinking coefficients using lasso.


How do I stack copies of a 1D tensor in keras? I have a tensor x of shape (5, 1). I would like to stack 10 copies of it so that I get a tensor of shape (5, 10). In a sort, it would look like [x|..|x]. How to do it the keras way? I have tried the Concatenate layer, but I got an error.
Help is welcome!

After many troubles with Concatenate, I simply did:
x = Reshape((5, 1))(x)
t = ones((1, 10))
stacked_x = x @ t

Maths help a lot. Should have thought about it earlier.


is there a way to normalize [-3,1] to ${\begin{bmatrix} \dfrac{-3}{\sqrt{10}}\\ \dfrac{1}{\sqrt{10}}\\ \end{bmatrix}}$ with python? I am learning SVD by following this MIT course

The lecturer is trying to normalize a vector

$${\begin{bmatrix}
-3\\
1\\
\end{bmatrix}}$$

to

$${\begin{bmatrix}
\dfrac{-3}{\sqrt{10}}\\
\dfrac{1}{\sqrt{10}}\\
\end{bmatrix}}$$

I tried this with Python NumPy

np.linalg.norm(v1,ord=2,axis=1,keepdims=True)

and got

array([[3.],
       [1.]])

I would like to get something like this

[[-0.9486833 ],
[ 0.31622777]]

is there a way with Python (for instance, NumPy) to do the job? any other 3rd party library is also appreciated.

You have already computed that, but you've not bound the output to a variable, also called name in python. Try the following snippet:

result = np.linalg.norm(v1,ord=2,axis=1,keepdims=True)
print(result)

Based on the edit, I update the answer. As you may find answers to your question, a typical way to find what you need is something like the following function:

def normalize(v):
    norm = np.linalg.norm(v)
    if norm == 0:
       return v
    return v / norm

Equivalently, there is a function called normalize in sklearn.preprocessing which can be employed for your task.


Downloading a large dataset on the web directly into AWS S3 Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?

Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I just want to supply the download URL to S3 and wait for them to download it to their filesystem. It seems like an easy thing to do, but I just can't find the documentation on it.

Since you obviously posses an AWS account I'd recommend the following:

Create an EC2 instance (any size)
Use wget(or curl) to fetch the file(s) to that EC2 instance. For example: wget http://example.com/my_large_file.csv.
Install s3cmd
Use s3cmd to upload the file to S3. For example: s3cmd cp my_large_file.csv s3://my.bucket/my_large_file.csv

Since connections made between various AWS services leverage AWS's internal network, uploading from an EC2 instance to S3 is pretty fast. Much faster than uploading it from your own computer. This way allows you to avoid downloading the file to your computer and saving potentially significant time uploading it through the web interface.


How to deal with a biased feature in Machine Learning (date) I have a model that predicts the lifespan of a horse. The dataset has samples from 1980 to 2019 and among the features there is one called birth_date, labeled with the lifespan in years for each horse. The problem is knowing that a horse usually lives between 20 and 30 years, if we look from 1980 to 1990 we have the full list of horses and their lifespans, but from 2000 to 2019 we see only samples of horses that were born and died within this time span but not the ones that are currently living, therefore the birth_date is a biased feature.

Is there a way to use the birth_date feature without having to worry about the biased data, or some technique to minimize his effects on the final predictions?

What you are describing are censored observations in survival analysis. Indeed, those who are still alive will obviously die but just at a later, observable date in a time frame past our study date. By counting them as survived on a logistic regression we will clearly bias our model to favor survivals for those particular horses, unrealistically.

Hence, you have survival data and should approach it like so. The Cox proportional hazards rate model allows you to relate hazard rate functions to predictors, and while many use linear models for the "predictor" part there are other ML methods like xgboost and random forests that have the ability to return Cox PH models. These models allow for censored observations as you describe.


find value which occurred more times in a group from data frame column R I have a data frame with latitude and longitude of a particular place. one place can have multiple lat, longs and those lat, long values can be same or different. I need to find the correct lat, long based on no.of occurances of lat, long for particular place.
example input data frame
df:

place    lat      long
hsy      7.343    32.849
hsy      7.343    32.122
hsy      7.567    31.567
hsy      6.934    32.122
bls      2.67     6.2
bls      3.345    5.9
bls      2.987    6.321
bls      2.987    4.56
bls      1.876    6.321
bls      2.987    6.321

expected output data frame
    place    lat      long
    hsy      7.343    32.122
    bls      2.987    6.321

expected output for lat is which ever value is occurred more times among its values for particular place and the output for 'long' column value also which ever value is occurred more times among its values for particular place.
tried below code using across, but across is not working. getting Error: 'across' is not an exported object from 'namespace:dplyr' or could not find function "across" error. any alternative for across function?
df %>%
  group_by(place) %>%
  summarise(across(c(lat, long),~names(sort(table(.x), decreasing = TRUE))[1]))

I think below code may help you.
df <-data.frame(place=c('hsy','hsy','hsy','hsy','bls','bls','bls','bls','bls','bls'),
                lat = c(7.343,7.343,7.567,6.934,2.67,3.345,2.987,2.987,1.876,2.987),
                long=c(32.849,32.122,31.567,32.122,6.2,5.9,6.321,4.56,6.321,6.321) )

library(dplyr)

Mode function which will fetch number which has highest frequency
mode_fun<-function(x){
  tbl=table(x)
  tbl=sort(tbl,decreasing = T)
  return(as.numeric(names(tbl[1])))
}

df1  = df %>% group_by(place) %>% summarise(lat1=mode_fun(lat),long1=mode_fun(long))


Why does my loss value start at approximately -10,000 and my accuracy not improve? I'm developing a multi-label classifier using the Keras library, but I am stuck with a relatively low accuracy of about 2% and my loss value per epoch is around -10,000 with little change between epoch. I'm relatively new to Keras but need to use it for my university work, and I am wondering what is going wrong.

My model currently looks as follows:

model = Sequential()
model.add(Dense(1024, activation='relu', input_shape=X.shape[1:]))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(3, activation='sigmoid'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])

model.fit(X, y, batch_size=20, epochs=100)

In this case, the input_shape[1:] is 4096, a flattened feature array and the multi-label I'm trying to classify has a first value that ranges from 1 - 163, the second value ranges from 1 - 2000, and the third value ranges from 2007 - 2014.

The issue that I am having is while running my model I get a result that does not change during each epoch as is shown below:

I don't know what I'm doing wrong, I'll include how the features and the labels look below in respective order:

[['0.0' ' 0.0' ' 0.0' ..., ' 5.7333' ' 0.0' ' 2.58643']
 ['0.0' ' 3.12623' ' 0.0' ..., ' 0.0' ' 0.0' ' 2.93147']
 ['0.0' ' 0.0' ' 0.0' ..., ' 7.49419' ' 0.0' ' 1.55746']
 ['0.0' ' 0.0' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.3666']
 ['0.0' ' 4.67996' ' 0.0' ..., ' 0.0' ' 0.0' ' 0.0']]

[['78' '1' '2010']
 ['78' '1' '2010']
 ['78' '1' '2014']
 ['78' '1' '2013']
 ['78' '1' '2012']]

I'm at a loss of what to try and as I have mentioned, I'm new to Keras and deep learning so any help will be greatly appreciated.

You are using the sigmoid activation function on the output layer that squashes the values between 0 and 1, but you are setting the targets outside of that range. I suggest scaling your outputs between 0 and 1 for each output unit. By the way, using a library like conx that is built on Keras would give you a warning about this very issue. For more, see: http://conx.readthedocs.io/en/latest/Getting%20Started%20with%20conx.html#A-Simple-Network


How to get correlation between two categorical variable and a categorical variable and continuous variable? I am building a regression model and I need to calculate the below to check for correlations

Correlation between 2 Multi level categorical variables
Correlation between a Multi level categorical variable and
continuous variable
VIF(variance inflation factor) for a Multi
level categorical variables

I believe its wrong to use Pearson correlation coefficient for the above scenarios because Pearson only works for 2 continuous variables.

Please answer the below questions

Which correlation coefficient works best for the above cases ?
VIF calculation only works for continuous data so what is the
alternative?
What are the assumptions I need to check before I use the correlation coefficient you suggest?
How to implement them in SAS & R?

Two Categorical Variables

Checking if two categorical variables are independent can be done with Chi-Squared test of independence.

This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the actual values are.

There also exists a Crammer's V that is a measure of correlation that follows from this test

Example

Suppose we have two variables

gender: male and female
city: Blois and Tours

We observed the following data:

Are gender and city independent? Let's perform a Chi-Squred test. Null hypothesis: they are independent, Alternative hypothesis is that they are correlated in some way.

Under the Null hypothesis, we assume uniform distribution. So our expected values are the following

So we run the chi-squared test and the resulting p-value here can be seen as a measure of correlation between these two variables.

To compute Crammer's V we first find the normalizing factor chi-squared-max which is typically the size of the sample, divide the chi-square by it and take a square root

R

tbl = matrix(data=c(55, 45, 20, 30), nrow=2, ncol=2, byrow=T)
dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))

chi2 = chisq.test(tbl, correct=F)
c(chi2$statistic, chi2$p.value)

Here the p value is 0.08 - quite small, but still not enough to reject the hypothesis of independence. So we can say that the "correlation" here is 0.08

We also compute V:

sqrt(chi2$statistic / sum(tbl))

And get 0.14 (the smaller v, the lower the correlation)

Consider another dataset

    Gender
City  M  F
   B 51 49
   T 24 26

For this, it would give the following

tbl = matrix(data=c(51, 49, 24, 26), nrow=2, ncol=2, byrow=T)
dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))

chi2 = chisq.test(tbl, correct=F)
c(chi2$statistic, chi2$p.value)

sqrt(chi2$statistic / sum(tbl))

The p-value is 0.72 which is far closer to 1, and v is 0.03 - very close to 0

Categorical vs Numerical Variables

For this type we typically perform One-way ANOVA test: we calculate in-group variance and intra-group variance and then compare them.

Example

We want to study the relationship between absorbed fat from donuts vs the type of fat used to produce donuts (example is taken from here)

Is there any dependence between the variables?
For that we conduct ANOVA test and see that the p-value is just 0.007 - there's no correlation between these variables.

R

t1 = c(164, 172, 168, 177, 156, 195)
t2 = c(178, 191, 197, 182, 185, 177)
t3 = c(175, 193, 178, 171, 163, 176)
t4 = c(155, 166, 149, 164, 170, 168)

val = c(t1, t2, t3, t4)
fac = gl(n=4, k=6, labels=c('type1', 'type2', 'type3', 'type4'))

aov1 = aov(val ~ fac)
summary(aov1)

Output is

            Df Sum Sq Mean Sq F value  Pr(>F)
fac          3   1636   545.5   5.406 0.00688 **
Residuals   20   2018   100.9
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So we can take the p-value as the measure of correlation here as well.

References

https://en.wikipedia.org/wiki/Chi-square_test
http://mlwiki.org/index.php/Chi-square_Test_of_Independence
http://courses.statistics.com/software/R/R1way.htm
http://mlwiki.org/index.php/One-Way_ANOVA_F-Test
http://mlwiki.org/index.php/Cramer%27s_Coefficient


Learning curve of CNN model I have a graph for a model on train accuracy and validation accuracy:

But I'm having trouble interpreting it. By the way i interpreted, I would say it is of poor performance. But I would like to know how this graph is interpreted in a more broad manner.

As in if we were to look at the point where the validation acc begins to bisect the train acc line, and went all the way down, what does this indicate? Would appreciate some help on this as I'm horrible at interpreting graphs.

https://en.wikipedia.org/wiki/Overfitting

This model is over-fitting. Better train accuracy (and validation accuracy that gets worse with successive iterations) indicates over-fit.

For CNN Next steps should be to reduce complexity of the model and adding droputs / batch normalization.

Few articles on this :

https://towardsdatascience.com/handling-overfitting-in-deep-learning-models-c760ee047c6e

https://machinelearningmastery.com/how-to-reduce-overfitting-with-dropout-regularization-in-keras/


Is there an established methodology for human expert evaluation of machine learning predictions? Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$.

In this case, however, it is known that $Y$ is not the best possible values for this variable. That is, $Y$ is actually the sum of $Y^{true}$ and $\epsilon$.  $Y^{true}$ is the hidden true value of the variable that cannot be known to anyone. $\epsilon$ does not follow a known distribution or pattern.

In this case, any error measure $f(Y, \hat{Y}$) is not trusted. In principle, it is believed that human experts can judge the quality of predictions $\hat{Y}$ by examining the values of $X$. However, there are many ways where bias can sneak into this evaluation and it might be difficult to quantify the prediction quality.

Is there an established methodology (i.e., in terms of number of experts, the followed procedures, on which scale the prediction quality is judged, how conflicting expert opinions are accounted for, etc.) for evaluating the quality of such predictions?

If it makes it easy to understand or answer, the data and problem can be assumed to be in a business domain.

Measuring the inter-annotator agreement is one of the ways to measure the reliability of the annotations.

(I assume that there are also other methods but I'm not that knowledgeable about the topic)


How big does a Neural Net need to be to learn anything Is there literature on how big a net needs to be to learn an arbitrary classification problem where the input is a n-bit integer and the output is one of k categories? I'm interested in both a theoretical bound and the size needed in practice with current training and initialization techniques.

To be specific, the neural net is allowed to see all possible inputs and their respective outputs, so I will not hide anything from the training set, as the function is not assumed to have any structure whatsoever.

The size of the Neural Network is usually treated as a hyper-parameter. You need to estimate it from a separate cross-validation set.

For Neural nets there are no theoretical proven bounds (there are some papers here and there, but they are not applicable in practice).

You need to use the good old "cross-validation" method to search for the best fit value of the size of NN.

P.S. : There is a paper from Yoshua Bengio with all trick and traits for training such machines.

https://arxiv.org/abs/1206.5533


Link prediction benchmark datasets I'd like to test a new algorithm for link prediction on graphs. I'm interested in  both weighted/unweighted and directed/undirected graphs.

What are some common benchmark datasets that researchers often use to test their algorithms? I know that within Computer Vision people often use MNIST or CIFAR, but I haven't found similar datasets for link prediction.

A cursory Google search gave me:

http://noesis.ikor.org/datasets/link-prediction
http://ial.eecs.ucf.edu/travian.php
http://konect.uni-koblenz.de/networks/

Another way would be to create your own dataset by downloading say Github user's graph, defining a network by adding edges  between user A and user B if A follows B and so on. Then you can take snapshots of this graph at different time instances and see if you can use last t snapshots to predict the edges in the t+1th snapshot. Of course that would be a lot more extra effort and the links I posted may suffice. Please let me know if it helps.