Outlier detection on categorical network log data I am working with a completely categorical network log data that consists of source ip address, destination ip address, source port, destination port, protocol.

Data Preprocessing performed : Converted IP addresses to integers. Removed rows with NA's. Dis not perform scaling as it doesn't make any sense with categorical data and also since I'm computing Hamming distance.

Modelling and Implementation : I have implemented 'Hamming Distance' to compute the distance matrix for the set. I have tried converting the categorical variables to numerical by converting them to factors and perform Kmeans on the data, however the accuracy I'm getting is very low as expected. I have used KModes as well and I have got cluster labels to the data, I'm having challenge proceeding further from here.

Question : Once the distance matrix is computed, how can I go about detecting the outliers with such data type? I really appreciate any recommendations. Thanks in advance

Data Snippet :  2887562076 2344654028        6   41940      80
                2344654028 2887562076        6      80   41940
                2344654028 2887562076        6      80   41940

Foremost, in my opinion categorical data should not be converted to continuous format at all. This is statistically incorrect.

Second, if a categorical data say Sex with two levels Male, Female exists then it can be written as 1,2, but, it should be specified that these numbers are to be treated as nominal and not continuous.

Continuing further, one method of detecting outliers in categorical data is to calculate the frequency of the data points and then average each such obtained frequency to the total number of data points. The data points with the minimum frequencies can be considered as outliers.

You might find this paper and this SO post implementation in R interesting.


When should you use learning rate scheduling over adaptive learning rate optimization algorithm? In order to converge to the optimum properly, there have been invented different algorithms that use adaptive learning rate, such as AdaGrad, Adam, and RMSProp. On the other hand, there is a learning rate scheduler such as power scheduling and exponential scheduling.

However, I don't understand at what kind of situations you should use one over the other. I feel that using adaptive learning rate optimization algorithm such as Adam is simpler and easier to implement than using learning rate scheduler.

So how can you use it apart properly, depending on what kind of problems?

I'm not sure about other fields but recently in the field of deep neural network training there is this arXiv submission, The Marginal Value of Adaptive Gradient Methods in Machine Learning.

  Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.


Anomaly detection on a time-series data in a CSV format using python I have a time versus current data for a days work which is as follows:

11/15/2016  5:12:27 10:42:27    26.61
11/15/2016  5:12:28 10:42:28    42.27
11/15/2016  5:12:29 10:42:29    25.48
11/15/2016  5:12:30 10:42:30    24.24
11/15/2016  5:12:31 10:42:31    25.91

The first column being the date
The second column being the time in GMT
The third column being time in IST
The fourth column being the value of the current used by the machine

Can someone suggest me an algorithm for finding the anomalies in the pattern plotted by current versus time in IST?
Any help regarding my approach will be appreciated too.

With so little data, you aren't going to get specific algorithms suggested. In addition most folk would like you to give it a try first. My approach would be to first plot the time vs. signal data. Visually inspect the data for outliers. People are pretty good at this. I would then try to describe a mathematical model to fit the data with the outliers removed. Finally you can identify the outliers as those points greater than some statistical measure away from the difference of the model and the data.


How do feature selection on a sparse matrix? Say I want to do features selection on a sparse matrix, i.e., 10,000 rows x 1500 features, but the matrix is mostly sparse. Let's say the features are all numeric and the target is binary and discrete.
What's the correct and efficient way to apply feature selection? Moreover, I'm interested in applying mutual information on it.

You can do a dimentionality Reduction as your matrix is Sparse. I would suggest to use PCA. PCA will reduce your 1500 input into k dimensional input of your choice with as much information retained as possible . Here k is a hyperparameter that you need to tune and fine the best one.
Another Approach is LASSO classfier which is a linear model with L1 regularization. This model will perform automatic feature selection and zero out the weight of feature that is not needed. But your input columns must be independent.


Logistic regression program does not give statistically correct result I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also random, I should expect the result to follow a standard distribution. I have ten classes and 10 initial examples, so I am expecting my accuracy to be close to 10, most of the time, however I am often (too often) getting results that are quite far from 10 (like 0 or 49). I should expect to get those results sometimes, but not so often. Is there something wrong in my code?

import theano
import theano.tensor as T
import numpy as np
rng = np.random
#rng = np.random.RandomState(0)

ninputs = 100
nfeatures = 784
noutputs = 10
nhiddens = 50

#training rate
tr_rate = 0.1
training_steps = 1

D = (rng.randn(ninputs, nfeatures), rng.randint(size=ninputs, low=0, high=noutputs))

# Feed-forward
x = T.dmatrix('x')
t = T.ivector('t')

w = theano.shared(rng.randn(nfeatures, noutputs), name="w")
b = theano.shared(np.full(noutputs, 0.01), name="b")
sigma = T.nnet.softmax(T.dot(x,w) + b)
prediction = T.argmax(sigma, axis=1)

parameters = [w, b]

xent = -T.mean(T.log(sigma)[T.arange(t.shape[0]), t])          # Cost function

cost = xent.mean() + 0.01 * (w ** 2).sum()                     # Regularisation

gparameters = [T.grad(cost, param) for param in parameters]    # Compute the gradient of the cost

pars = zip(parameters, gparameters)

train = theano.function(
          inputs=[x,t],
          outputs=[xent],
          updates = [(param, param - tr_rate * gparam) for param, gparam in zip(parameters, gparameters)],
          allow_input_downcast = True )

predict = theano.function(inputs=[x], outputs=prediction)

# Train
for i in range(training_steps):
    train(D[0], D[1])

result = predict(D[0]) - D[1]

N = ninputs

error = 0
for index in result:
    if result[index] != 0:
        error += 1

correct_guesses = N - error
accuracy = (N - error)*100/N

print
print "correct predictions = %f over %i examples" % (correct_guesses, N)
print "accuracy = %i%%" % accuracy

If you initialized the weights randomly and predicted immediately, you should probably expect 1/num_classes accuracy.

But you are training first. The data may be random, but it is still possible to overfit it. With a sample that small and wide, you could imagine some degree of separability of the classes by chance. In that scenario, your network can learn weights that are better than random (far from perfect in only one iteration).

The effect would likely be lessened by using much more data or holding out/using new random data to test accuracy.


How to do feature selection for clustering and implement it in python? I am trying to implement k-means clustering on 60-70 features and I came across a post for feature selection technique on quora  by Julian Ramos, but I fail to understand few steps mentioned. I am also wondering if its the right method to select the best features for clustering?

These are the steps mentioned in the post :

  Sf={∅} #Set of features selected, initially empty

  Perform k-means on each of the features individually for some k.
  For each cluster measure some clustering performance metric like the Dunn's index or silhouette.
  Take the feature which gives you the best performance and add it to Sf
  Perform k-means on Sf and each of the remaining features individually
  Take the feature which gives you the best performance and add it to Sf
  If you have reached the desired number of features stop, else go back to 4

Also, how do we implement the same in python. I wish to write function for the same that selects best k and implement all the other steps.

Often people confuse between unsupervised feature selection (UFS) and dimensionality reduction (DR) algorithms as the same. For instance, a famous DR algorithm is Principal Component Analysis (PCA) which is often confused as a UFS method! Researchers have suggested that PCA is a feature extraction algorithm and not feature selection because it transforms the original feature set into a subset of interrelated transformed features, which are difficult to emulate (Abdi & Williams, 2010).
An UFS approach present in literature is Principal Feature Analysis PFA. The way it works is given as;
Steps:

Compute the sample covariance matrix or correlation matrix,
Compute the Principal components and eigenvalues of the Covariance or Correlation matrix A.
Choose the subspace dimension n, we get new matrix A_n, the vectors Vi are the rows of A_n.
Cluster the vectors |Vi|, using K-Means
For each cluster, find the corresponding vector Vi which is closest to the mean of the cluster.

A possible python implementation of PFA is given below;
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from collections import defaultdict
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.preprocessing import StandardScaler
import pandas as pd
# create some dummy data
df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])
print(df)
class PFA(object):
    def __init__(self, n_features, q=None):
        self.q = q
        self.n_features = n_features

    def fit(self, X):
        if not self.q:
            self.q = X.shape[1]

        sc = StandardScaler()
        X = sc.fit_transform(X)

        pca = PCA(n_components=self.q).fit(X) # calculation Cov matrix is embeded in PCA
        A_q = pca.components_.T

        kmeans = KMeans(n_clusters=self.n_features).fit(A_q)
        clusters = kmeans.predict(A_q)
        cluster_centers = kmeans.cluster_centers_

        dists = defaultdict(list)
        for i, c in enumerate(clusters):
            dist = euclidean_distances([A_q[i, :]], [cluster_centers[c, :]])[0][0]
            dists[c].append((i, dist))

        self.indices_ = [sorted(f, key=lambda x: x[1])[0][0] for f in dists.values()]
        self.features_ = X[:, self.indices_]

# Usage
pfa = PFA(n_features=3)
pfa.fit(df)
# To get the transformed matrix
x = pfa.features_
print(x)
# To get the column indices of the kept features
column_indices = pfa.indices_

Results
        num_legs  num_wings  num_specimen_seen
falcon         2          2                 10
dog            4          0                  2
spider         8          0                  1
fish           0          0                  8
[[-0.50709255  1.73205081  1.23942334]
 [ 0.16903085 -0.57735027 -0.84802649]
 [ 1.52127766 -0.57735027 -1.10895772]
 [-1.18321596 -0.57735027  0.71756088]]

Reference
Abdi, H., & Williams, L. J. (2010). Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4), 433-459.


How to detect different brands of milk I'm trying to create a object detection model to detect different type of milk. What is the best approach to achieve the result in the picture below? As you can see in the picture, this model did pretty well on locating and detecting different brand of milks. I've tried Faster R-CNN but they mix up products with similar shape really bad(not only on the classifier but also on the religion proposal net). I labeled my training images in the same way as in the picture below. Any help will be appreciated!

Have you tried regression here is a nice project to get you started could you give some feedback on the output?


SKLearn DT regressor - good enough score? What constitutes as a "good enough" score for a Decision Tree Regressor? The .score() function gives us a general score about our model. This can be 1 if the model predicts all data with a 100% accuracy, and can be arbitrarily worse. If I understand correctly, a score of 0 means the prediction is quasi constant. But starting with what value can we say that the prediction is "usable" (i know this is ambiguous, but still). Is a score of 0.4-0.5 good enough to use the predictions in practice as opposed to not using any machine learning techniques?

The quality of the score depends on the context of the project.

One approach is a model comparison approach. Define a baseline model. Then see if Decision Tree Regressor is better than the baseline.


How to deal with string labels in multi-class classification with keras? I am newbie on machine learning and keras and now working a multi-class image classification problem using keras. The input is tagged image. After some pre-processing, the training data is represented in Python list as:
[["dog", "path/to/dog/imageX.jpg"],["cat", "path/to/cat/imageX.jpg"],
 ["bird", "path/to/cat/imageX.jpg"]]

the "dog", "cat", and "bird" are the class labels. I think one-hot encoding should be used for this problem but I am not very clear on how to deal it with these string labels. I've tried sklearn's LabelEncoder() in this way:
encoder = LabelEncoder()
trafomed_label = encoder.fit_transform(["dog", "cat", "bird"])
print(trafomed_label)

And the output is [2 1 0], which is different that my expectation output of somthing like [[1,0,0],[0,1,0],[0,0,1]]. It can be done with some coding, but I'd like to know if there is some "standard" or "traditional" way to deal with it?

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly generate one-hot-encodings in other code.

If you want one-hot encoding, you can use LabelBinarizer instead. This works very similarly:

 from sklearn.preprocessing import LabelBinarizer
 encoder = LabelBinarizer()
 transfomed_label = encoder.fit_transform(["dog", "cat", "bird"])
 print(transfomed_label)

Output:

[[0 0 1]
 [0 1 0]
 [1 0 0]]


Standard deviation as outlier detection I have a quite basic question: A standard deviation is defined such that around ~66 % of the data lies within it. And around ~99 % within three standard deviations.

When I wanna' use the standard deviation as an outlier detection, I struggle with this definition as there will always be outlier. But I probably have a misunderstanding somewhere, somehow..

In other words: 3 standard deviations contain always about 99 % of the data, hence there will always be $ outlier > 3 std $, no matter if they are outlier.

What is the issue I don't see?

Standard deviation isn't an outlier detector. It can't tell you if you have outliers or not.

What it will do is effectively remove outliers that do exist, with the risk of deleting a small amount of inlying data if it turns out there weren't any outliers after all. As such, I think it's useful as a "quick-and-dirty don't want to spend too much time on this problem" method of ensuring your dataset only contains inlying data points. As long as you understand the risk of potentially binning 1% of inlying datapoints, I think it's fine.


LightGBM vs Sklearn LightGBM- Mistake in Implementation- Exact same parameters giving different results While passing the exact same parameters to LightGBM and sklearn's implementation of LightGBM, I am getting different results. Initially, I was getting the exact same results on doing this, however, I made some changes to my code and now I can't find out why they're not coming the same. This means that the performance metrics and feature importance are coming differently. Please help me figure it out, I can't figure out the mistake I am making. It could either be a mistake in the way I am implementing LightGBM using the original library or in sklearn's implementation. Link for explanation on why we should get identical results.
x_train, x_test, y_train, y_test = train_test_split(df_dummy[df_merge.columns], labels, test_size=0.25,random_state=42)

n_folds = 5

lgb_train = lgb.Dataset(x_train, y_train)

def objective(params, n_folds = n_folds):
    """Objective function for Gradient Boosting Machine Hyperparameter Tuning"""

    print(params)

    params['max_depth'] = int(params['max_depth'])
    params['num_leaves'] = int(params['num_leaves'])

    params['min_child_samples'] = int(params['min_child_samples'])
    params['subsample_freq'] = int(params['subsample_freq'])

    # Perform n_fold cross validation with hyperparameters

    # Use early stopping and evalute based on ROC AUC
    cv_results = lgb.cv(params, lgb_train, nfold=n_folds, num_boost_round=10000,
                        early_stopping_rounds=100, metrics='auc')

    # Extract the best score
    best_score = max(cv_results['auc-mean'])

    # Loss must be minimized
    loss = 1 - best_score
    num_iteration = int(np.argmax(cv_results['auc-mean']) + 1)

    of_connection = open(out_file, 'a')
    writer = csv.writer(of_connection)
    writer.writerow([loss, params, num_iteration])

    # Dictionary with information for evaluation
    return {'loss': loss, 'params': params, 'status': STATUS_OK, 'estimators': num_iteration}

space = {
    'min_child_samples': hp.quniform('min_child_samples', 5, 100, 5),
    'reg_alpha': hp.uniform('reg_alpha', 0.0, 1.0),
    'reg_lambda': hp.uniform('reg_lambda', 0.0, 1.0),
    'colsample_bytree': hp.uniform('colsample_bytree', 0.5, 1.0),
    'max_depth' : hp.quniform('max_depth', 3, 10, 1),
    'subsample' : hp.quniform('subsample', 0.6, 1, 0.05),
    'num_leaves': hp.quniform('num_leaves', 20, 150, 1),
    'subsample_freq': hp.quniform('subsample_freq',0,10,1),
    'min_gain_to_split': hp.quniform('min_gain_to_split', 0.01, 0.1, 0.01),

    'learning_rate' : 0.05,
    'objective' : 'binary',

}

out_file = 'results/gbm_trials.csv'
of_connection = open(out_file, 'w')
writer = csv.writer(of_connection)

writer.writerow(['loss', 'params', 'estimators'])
of_connection.close()

trials = Trials()
best = fmin(objective, space, algo=tpe.suggest, trials=trials, max_evals=10)
bayes_trials_results = sorted(trials.results, key = lambda x: x['loss'])

results = pd.read_csv('results/gbm_trials.csv')

# Sort with best scores on top and reset index for slicing
results.sort_values('loss', ascending = True, inplace = True)
results.reset_index(inplace = True, drop = True)
results.head()
best_bayes_estimators = int(results.loc[0, 'estimators'])

best['max_depth'] = int(best['max_depth'])
best['num_leaves'] = int(best['num_leaves'])

best['min_child_samples'] = int(best['min_child_samples'])

num_boost_round=int(best_bayes_estimators * 1.1)
best['objective'] = 'binary'
best['boosting_type'] = 'gbdt'

best['subsample_freq'] = int(best['subsample_freq'])

#Actual LightGBM

best_gbm = lgb.train(params=best, train_set=lgb_train, num_boost_round=num_boost_round)

#Sklearn's Implementation of LightGBM

best_sk = dict(best)
del best_sk['min_gain_to_split']
sk_best_gbm = lgb.LGBMClassifier(**best_sk, n_estimators=num_boost_round, learning_rate=0.05, min_split_gain=best['min_gain_to_split'])
sk_best_gbm.fit(x_train, y_train)

sk_best_gbm.get_params()

A kind soul from Github was able to answer this.
The answer is this:
for the parameters consistency problem, you can try to use a new lgb_train before lgb.train, like
lgb_train = lgb.Dataset(x_train, y_train)
best_gbm = lgb.train(params=best, train_set=lgb_train, num_boost_round=num_boost_round)

lgb_train is lazy-inited, and only inited one-time, so it will be constructed in the cv part. And some parameters (like min_child_samples) in that part may change the lgb_train. Therefore, lgb_train may is inited by different parameters. (So it is better to use a new lgb.train in cv part as well.)


How to find the ranges in Equal frequency/depth binning? I have been looking into the site http://www.saedsayad.com/unsupervised_binning.htm and there it shows range values to the right under equal frequency binning ....

I have so much looked to find how the ranges are formed but has been in vain. So , can somebody explain how those ranges are formed for the given dataset or explain the freq binning procedure in a good manner.

Great thanx for any help on this!

As far as I can see the choice of the bin size /frequency is arbitrary in those examples. Frequency binning is simple choosing you bin boundaries in a way that the bin content size is the same. For the frequency approach it looks like the order the elements by size and calculate the bin edges in the middle between the highest element of bin A and the lowest of bin B.

If you want to be really fancy you can use methods like bayesian blocks that actually choose bin size dynamically with respect of information content.


Normalized Euclidean Distance versus cross correlation? Normalized Euclidean Distance and Normalized Cross - Correlation can both be used as a metric of distance between vectors.

What is the difference between these metrics?  It seems to me that they are the same, although I have not seen this explicitly stated in any textbook or literature.

thank you.

These two metrics are not the same.

The normalized Euclidean distance is the distance between two normalized vectors that have been normalized to length one.  If the vectors are identical then the distance is 0, if the vectors point in opposite directions the distance is 2, and if the vectors are orthogonal (perpendicular) the distance is sqrt(2).  It is a positive definite scalar value between 0 and 2.

The normalized cross-correlation is the dot product between the two normalized vectors.  If the vectors are identical, then the correlation is 1, if the vectors point in opposite directions the correlation is -1, and if the vectors are orthogonal (perpendicular) the correlation is 0.  It is a scalar value between -1 and 1.

This all comes with the understanding that in time-series analysis the cross-correlation is a measure of similarity of two series as a function of the lag of one relative to the other.


local minima vs saddle points in deep learning I heard Andrew Ng (in a video I unfortunately can't find anymore) talk about how the understanding of local minima in deep learning problems has changed in the sense that they are now regarded as less problematic because in high-dimensional spaces (encountered in deep learning) critical points are more likely to be saddle points or plateaus rather than local minima.

I've seen papers (e.g. this one) that discuss assumptions under which "every local minimum is a global minimum". These assumptions are all rather technical, but from what I understand they tend to impose a structure on the neural network that make it somewhat linear.

Is it a valid claim that, in deep learning (incl. nonlinear architectures), plateaus are more likely than local minima? And if so, is there a (possibly mathematical) intuition behind it?

Is there anything particular about deep learning and saddle points?

Let me give an explanation based on multivariate calculus. If you have taken a multivariate course, you will have heard that, given a critical point (point where the gradient is zero), the condition for this critical point to be a minimum is that the Hessian matrix is positive definite. As the Hessian is a symmetric matrix, we can diagonalize it. If we write the diagonal matrix corresponding to the Hessian as:
$$
  D =
  \begin{bmatrix}
    d_{1} & & \\
    & \ddots & \\
    & & d_{n}
  \end{bmatrix}
$$
the Hessian being positive definite is equivalent to $d_1 > 0, \dots, d_n>0$.

Now let's think about deep learning cost functions. Deep learning cost functions depend on lots of parameters in a very complicated manner, so the Hessian will have a complicated expression itself. For this reason, we can think that the values of $d_1,\dots,d_n$ are not biased towards negative or positive values. For this reason, given any critical point, the probability of every value $d_i$ to be positive can be assumed to be $1/2$. Moreover, it is reasonable to assume that the values of $d_i$ do not depend easily on the values of $d_j$, due to the high non-linearity of the Hessian matrix, so we will take the probabilities of them being positive as independent events.

For this reason, given a critical point, the probability of it being a minimum is:
$$
P(d_1 > 0, \dots, d_n > 0) = P(d_1 > 0)\cdot \cdots \cdot P(d_n > 0) = \frac{1}{2^n}
$$

The probability of any critical point being a minimum decreases exponentially with the dimension of the input space. In deep learning,
this space can range from 1000 to $10^8$, and in both cases $1/2^n$ is ridiculously small. Now we are convinced that, given any critical point that we come across, it is very unlikely that it is a minimum.

But what about maxima?

The maxima of a function are the minima of minus the function. For this reason, all the arguments used previously can be used to minus the cost function and we conclude that every critical point has probability of $1/2  ^n$ to be a maximum.

For this reason, given a critical point, the probability of it being a saddle point is
$$P(saddle) = 1 - P(maximum) - P(minimum) = 1 - \frac{1}{2^n} - \frac{1}{2^n} = 1 - \frac{1}{2^{n-1}}$$

Which is very close to 1 if $n$ is large enough (which typically is in deep learning).


Join the List and the Data frame for matching list elements in Python I have a list =[a,b]  and

Dataframe as below

If the text in the 'Text column' matches  with List, then need to add additional column matchedItem

So output should be

Please suggest on this..How to get this output after the join (with exact match and partial match)..

also alternatively output in another dataframe(partial match)

Code

To resolve

I am trying something like this one

df["kpiName"] = getKPIName(df1.text.str)
kp1=['a','b']

def getKPIName (str):
    return [kp1.index(x) for x in kp1 if str in x]

But this results in
error
TypeError: 'in ' requires string as left operand, not StringMethods

The question is presented in a somewhat confusing fashion. However, I'm assuming OP has a dataframe setup something like this:

import pandas
data = { 'No': [ 1, 2, 3, 4 ], 'Text': [ 'a', 'ab', 'b', 'bd' ] }
df = pandas.DataFrame( data )

In addition to the df file, OP has a kpi list consisting of 'a' and 'b'. There is also a mask variable, which I am assuming was meant to catch all cases where the df.Text is equal to either 'a' or 'b'. This is where the first mistake is. The way OP wrote that line would match cases where the character a is in any part of the element. In other words 'a' in 'ab' is True. I suspect, OP wanted that statement to be False, which meant it needs to be written as 'a' == 'ab' which would be false. I fixed it in the lines below:

kpi = [ 'a', 'b' ]
mask = df['Text'].apply( lambda x: any( item for item in kpi if item.lower() == x ) )

Basically I changed the item.lower() in x to item.lower() == x. Next, OP indicated that there should be a new column called matchedItem, which duplicates the df.Text when there was a match and '----' when there was not a match. Given the mask variable that OP had written, this can be done simply as such:

df[ 'matchedItem' ] = '----'
df.matchedItem[ mask ] = df.Text[ mask ]

In other words, first I set '----' as the default value for all the elements in column matchedItem. Next, I copy over the elements from column Text only for the rows identified in the mask variable. This results in the following output for print( df ):

   No Text matchedItem
0   1    a           a
1   2   ab        ----
2   3    b           b
3   4   bd        ----

There's a number of things that are unclear on the asking of this question (e.g. did OP really want to set mismatches as '----'? Is No supposed to be an index or a separate column? etc.), but hopefully this answer will guide OP towards the desired result.


Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy? My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima".
In my opinion this does not work at all, and should lead to picking the model that best fits the test_set, but may not be generalizable enough for it to work with the actual distribution behind it.
Is there any material or research on this? I really think this method is archaic and makes no sense, but i can't argue with my professor without actual scientific material.

You are absolutely correct that this is an problematic approach. Your testing set should only be used at the last possible stage before deploying a model.
By using your testing set to make modeling decisions you will introduce bias which will favor the observations found in your test set and may not generalize. In an ideal world, your test set would represent the real distribution perfectly, however in practice this is never the case. Thus the suggested approach would result in a model that matches the test set's distribution more closely, but does not generalize to the real distribution.
The correct approach is to separate your data into a

training set,
validation set and,
testing set.

You then set the testing set aside until you have chosen a final model. With the validation set you are very much permitted to do what your professor suggested. You can fit multiple models and then pick the best one, or use all the models together in some bagging structure. Once you are satisfied you can then test against your testing set to see if the selected model generalized well.


How does the construction of a decision tree differ for different optimization metrics? I understand how a decision tree is constructed (in the ID3 algorithm) using criterion like entropy, gini index, variance reduction. But the formulae for these criteria do not care about optimization metrics like accuracy, recall, AUC, kappa, f1-score and others.

Packages on R and Python allow me to optimize for such metrics when I construct a decision tree. What do they do differently for each of these metrics? Where does the change happen?

Is there a pattern to how these changes are done for different classification/regression algorithms?

Splitting is the same your are optimising for the parameters.

In other words, gini, entropy etc logic will stay the same but the breadth, width, number of features etc will be different when you optimise it with different loss functions.

Accuracy, precision, F-scores etc. are evaluation metrics computed from binary outcomes and binary predictions. They are NOT (but there are modification) for them to be a loss function.
For model training, you need a function that compares a continuous score (your model output) with a binary outcome - like cross-entropy. Ideally, this is calibrated such that it is minimised if the predicted mean matches the population mean (given covariates)


How to define confusion matrix for classification? Below is the dataset where the response variable is play with two labels (yes, and no):

No. outlook temperature humidity    windy   play
1   sunny       hot     high        FALSE   no
2   sunny       hot     high        TRUE    no
3   overcast    hot     high        FALSE   yes
4   rainy       mild    high        FALSE   yes
5   rainy       cool    normal      FALSE   yes
6   rainy       cool    normal      TRUE    no
7   overcast    cool    normal      TRUE    yes
8   sunny       mild    high        FALSE   no
9   sunny       cool    normal      FALSE   yes
10  rainy       mild    normal      FALSE   yes
11  sunny       mild    normal      TRUE    yes
12  overcast    mild    high        TRUE    yes
13  overcast    hot     normal      FALSE   yes
14  rainy       mild    high        TRUE    no

Here are the decisions with their respective classifications:

1: (outlook,overcast) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 3, 7, 12, 13]

2: (humidity,normal), (windy,FALSE) -> (play,yes)
[Support=0.29 , Confidence=1.00 , Correctly Classify= 5, 9, 10]

3: (outlook,sunny), (humidity,high) -> (play,no)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 1, 2, 8]

4: (outlook,rainy), (windy,FALSE) -> (play,yes)
[Support=0.21 , Confidence=1.00 , Correctly Classify= 4]

5: (outlook,sunny), (humidity,normal) -> (play,yes)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 11]

6: (outlook,rainy), (windy,TRUE) -> (play,no)
[Support=0.14 , Confidence=1.00 , Correctly Classify= 6, 14]

You are just predicting if Play = Yes or Play = No.

The confusion matrix would look like this:

             Predicted
          +------+------+
          |  Yes |  No  |
    +-------------------+
A   |     |      |      |
c   | Yes |  TP  |  FP  |
t   |     |      |      |
u   +-------------------+
a   |     |      |      |
l   | No  |  FN  |  TN  |
    |     |      |      |
    +-----+------+------+

TP: True positives
FP: False positives
FN: False negatives
TN: True negatives

The accuracy can then be calculated as (TP + TN)/(TP + FP + TN + FN).


What are the assumptions of linear regression Can anyone explain the assumptions of linear regressions?

If possible with an example?

Is this really important to check these assumptions before proceeding?

Here are the links I have found:

https://en.wikipedia.org/wiki/Linear_regression#Assumptions

https://www.statisticssolutions.com/assumptions-of-linear-regression/

http://r-statistics.co/Assumptions-of-Linear-Regression.html


How to automatically verify official documents? I am new to machine learning and data science. I apologise if the question seems very basic. I have a requirement where I need to verify information submitted via a form with the corresponding official document. My approach till now has been to use google vision to extract text and use regex to extract necessary fields and compare with the form information. This is not always reliable because of the image quality and vision captures the noise as well.

Previously, I was thinking of just comparing each data file and search it in the text extracted and provide a metric of certainty.

I talked to one of my colleagues and he suggested using some for of supervised learning algorithm to process the documents, so that they automatically extract the key fields.

I want to ask would this method be simpler than my current approach. I am also worried about the scalability of my approach in event of minor changes in document formats.

I am looking for links for some articles or books related to this and answer to why using my own model will be better than just searching each word in a text.

Edit: The data can be imagined as a visiting card which contains the name of person, his office address, his contact number and the company's name. The problem here is different visiting cards can have different formats. Moreover the information is repeated twice: Once in English and the second in another language. For example , NAME : JANE DOE is followed by नाम : जेन डोए in next line. I only need the English name. The number of formats are constant but are high. Moreover, the address is not read together by Google Vision in most cases and using regex can become too complicated and case specific. I need to verify employee identity cards as proof that they are working for the company they claim to.

Please refer the below link which is related to Machine learning in document analysis and recognition.

https://www.researchgate.net/publication/242506468_Machine_Learning_in_Document_Analysis_and_Recognition


How does one decide when to use boosting over bagging algorithm? What kind of problem, circumstances and data makes it more suitable to apply boosting instead of bagging methods?

Bagging and boosting are two methods of implementing ensemble models.

Bagging: each model is given the same inputs as every other and they all produce a model

Boosting: the first model trains on the training data and then checks which observations it struggled most with, it passes this info to the next algorithm which assigns greater weight to the misclassified data

Because of this both bagging and boosting reduce variance. However boosting is better at improving accuracy vs a single model whilst bagging is better at reducing overfitting.

I would advise training a single version of the ensemble aka decision tree for random forest and then seeing where improvements can be made.
Good article explaining boosting vs bagging


Is it possible to update data and retrain just one of several data series in bigquery model I am building something very similar to this BigQuery ML example project.
My system is different in two ways:
Firstly it will need several thousand time-series so I would prefer to use the multiple-series feature rather than having thousands of individual models.
Secondly is the data is more unpredictable in the long run (rather than periodic or seasonal) so needs retraining quite often, with only local trends being detected.
The data is actually monitoring voltages in battery-operated devices, which usually drops at a linear rate, but sometimes can drop much faster depending on usage, then get recharged randomly. I am forecasting the future voltage and predicting when a critical level will happen. I've tested one model and the predictions are impressively good, ARIMA seems to balance both long-term typical behaviour with the local recent changes even though it's aperiodic.
Concerning the data-pipeline, I receive new data from each device individually at random intervals. So I need to push this new data into the model, retrain it and refresh the forecast for that device's time-series.
I can limit the number of times this refresh happens, e.g. once per hour, rather than doing it in event-based in realtime. Near realtime would be great, but it's not essential.
If I had thousands of single models, this would be trivial to design, I'd update the data (BiqQuery table) and then re-create that specific model, which takes some 30 seconds.
However having all the series in one model, it seems (from the docs) that I'd have to throw away the entire model of all thousands of series, retrain the entire thing. I don't currently know how long this would take, how it scales, but I assume it will be significantly longer and presumably more expensive. All but one of of the time series are unchanged so this seems a very wasteful operation.
So is there a way to retrain just one data-series in a BigQuery ARIMA model?
I have read the interesting notes under the limitations section for the time-series specific CREATE MODEL syntax which discusses processing thousands of series.
I have read this question but I wasn't convinced by the assertion of using warm start as he leaves no doc references, and it doesn't seem available in the ARIMA model.
I have also read this question which is more about the general necessity of retraining.

After implementing and training this model and understanding the details better, it seems the short answer is NO, I have to remake and train the entire model.
It takes about 7 minutes to train about 3,500 series with about 100 datapoints each from a seed dataset. This may grow as the incremental ETL adds data indefinitely.


time series prediction with several independent variables using RNNs and Keras I came across this excellent question and answer by Adam Sypniewski. Here method of using RNNs (e.g. LSTMs and GRUs) for predicting time series where the dependent variable depends on several independent variables. Just wondering, is anyone aware of some Keras code for this?

Please go through this tutorial:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

The author focuses on LSTM-RNNs with explicit instructions on how to re-shape your dataset into the format that Keras requires, which can be VERY tricky if not properly understood.


What happens if GBM parameters (e.g., learning rate) vary as the training progresses? In neural networks there is an idea of a "learning rate schedule" which changes the learning rate as training progresses.

This made me ask the question, what would be the impact of varying parameters in a GBM as a function of the number of trees?

Take the learning rate for example.  For GBMs using the MART algorithm, the contribution of each tree is weighted by a function of the error and the learning rate.  Trees fit early on have a higher impact; trees fit later on have less impact.  What if the learning rate was a function of $N$ such as $\exp(-a N)$ where $a$ would be the decay parameter of the learning rate?

Other parameters could vary as well.  For example the max depth of each tree could start out high and then decrease as training progresses.  Going beyond just the tree parameters, other examples are the subsample percentage if using bagging or parameters of a loss function (e.g., Huber loss parameter $\delta$).

Learning rate decay is implemented in, e.g., XGBoost and LightGBM as callbacks.  (XGBoost used to allow the learning_rate parameter to be a list, but that was deprecated in favor of callbacks.)  Similar functionality for other hyperparameters should be possible in the same way.
xgboost.callback.reset_learning_rate
LightGBM callback discussion
I played around with these ideas (for learning rate and tree depth) a while back, but didn't get improved performance.  But you should try it out; if you do see significant gains, it'd be great to add it as an answer here.


What is the difference between these two methods of batching with stateful LSTM With stateful LSTM, are these two methods effectively the same:

for _ in range(3):
    batch_input_shape = [1,10,2]
    x_batch, y_batch = batcher()
    model.train_on_batch(x_batch, y_batch)
model.reset_states()

and

batch_input_shape = [3,10,2]
x_batch, y_batch = batcher()
model.train_on_batch(x_batch, y_batch)
model.reset_states()

i.e. the first method 30 rows of data spread across 3 separate batches of 10 sequences and reset_states() called after all 3 train_on_batch() calls. The second method 30 rows of data submitted in a batch of 3 with 10 sequences, and reset_states() called immediately after.

Assuming it all the same data, and putting aside training times (first method will take longer) is the end result the same?

Answering my own question: they are the same.


Sentiment Analysis of News Headlines I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it.

Initially, I used TextBlob library for sentiment analysis to generate a polarity score. But the polarity score being generated for news headlines are not accurate. It is classifying negative news as positive news.

For example: Goldman Sachs CEO apologizes to people of Malaysia. This news is being classified as positive.

Post that, I've tried to build a custom model using spaCy library. I trained the custom model on 500 manually labelled news headlines. After training the model, I ran it on new set of headlines. The accuracy has improved but there is still scope for improvement. The accuracy is particularly poor with regard to financial news and editorial headlines.

My expectations is to create a model that can accurately predict the sentiment of news headline. So that I can plot sentiment trend about the subject over a time period.

Try to find a pattern in the sentences which are failing.
Use negspacy, find out words which are negative, replace them with opposite words. eg -
'not good' -> 'bad'
try using bert transformers pretrained sentiment classification models.


Counting the transition in a dataframe overtime I am stuck at a problem and am thinking how to come out of it. I want to write a code in python with dataframe as below:

data = {'Id':['a', 'a', 'b', 'b', 'c', 'c'],
        'value':['Active', 'Notactive', 'Active', 'Superactive', 'Notactive', 'Superactive'],
        'date':['8-09-2019','15-09-2019','8-09-2019','15-09-2019','8-09-2019','15-09-2019']}
df = pd.DataFrame(data)
I want to reach to the result where i count the number of users who did transition in the 7 day difference:
Active to notactive: 1
Active to superactive: 1
Not active to superactive: 1
Active to Active: 0

Any help on how to proceed would be appreciated on python 3.7

Thanks

For a small dataset try this:
# change type of date column so that it is possible to find min and max correctly
df['date'] = pd.to_datetime(df['date'])

# put value of the oldest resord and newest record side by side
df_min_max_grouped = pd.merge(
    df.loc[df.groupby('Id')['date'].idxmin()],
    df.loc[df.groupby('Id')['date'].idxmax()],
    on='Id',
    suffixes=('_old', '_new')
)

# group by old value and new value and find counts
results = df_min_max_grouped.groupby(['value_old', 'value_new']).size()

results

Output:

value_old  value_new
Active     Notactive      1
           Superactive    1
Notactive  Superactive    1
dtype: int64

The approach I chose for this answer:

Convert date values to be of type datetime, Correct ordering will not work with string dates.
First find the oldest and the newest records for any Id,
This happens in : df.loc[df.groupby("Id")['date'].idxmin()] for oldest and df.loc[df.groupby("Id")['date'].idxmax()] and newest records.
Join the oldest and newest records on the basis of Id, and then
Group by old value and new value combination to get a count in each group.

I found that resetting index of the resulting Series produces nice output:
results.reset_index()

value_old
value_new
0

0
Active
Notactive
1

1
Active
Superactive
1

2
Notactive
Superactive
1

The combinations not listed here have 0 records against them. I hope you find this helpful!


how to do incremental deep learning on data stream that can adapt to constantly generated data points? I am currently trying to learn a deep learning model on a data stream, which constantly generate new data points over time.  The goal is to generate a real-time DL model that can well adapt to newly generated data while does not affect the prediction for old data too much.  I am wondering if anyone know what might be the best way to do this "incremental deep learning on data stream"?  Any information would be much appreciated, thank you!

What you are describing does not require a change in the deep learning models you would use but a change in how to estimate them: you need an online estimation procedure rather than an off-line one. You can have a look at the seminal work of Botou [1998], or Chapter 8 of the book by Goodfellow et al. (2016).


SMOTE oversampling for class imbalanced dataset introduces bias in final distribution I have a problem statement where percentage of goods (denoted by 0) is 95%, and for bads (denoted by 1) it is 5% only. One way is to do under sampling of goods so that model understands the patterns properly for both the segment. But going with under sampling is leading to high loss of data which will directly lower down my model performance. Hence I have opted for over sampling of bads, but over sampling has its own problem too:
Check this code snippet:
from imblearn.over_sampling import SMOTE

sm = SMOTE(random_state = 33)

x = train_data.drop(['target'], axis = 1)
y = train_data[['target']]

x_new, y_new = sm.fit_sample(x, y)

y.target.value_counts()     # 0 ->  26454  1-> 2499
y_new.target.value_counts() # 0 ->  26454  1-> 26454

after oversampling, I get my equal no. of goods and bads, but the problem is that variable distribution is getting affected.
For e.g. I have 'age' variable, in case of good, the bucket wise variable distribution is
 1 - 25 years - 20%
26 - 50 years - 35%
50+     years - 45%

and distribution of bad is (Before OverSampling):
 1 - 25 years - 50%
26 - 50 years - 30%
50+     years - 20%

But post oversampling the distribution of bads is changing:
 1 - 25 years - 40%
26 - 50 years - 35%
50+     years - 25%

So now the distribution of this variable for Good Vs Bad is not that much effective as it was earlier (before oversampling).

Is there any way that doing oversampling does not affect my variable distribution?

Class imbalance is a frequent problem in machine learning and techniques to balance the data usualy are of two flavors: undersampling the majority, oversampling the minority or both.
One can always partition the data according to some variables and separately oversample each partition so as to maintain some measure (eg given data distribution). In the same way that separate oversampling can be achieved for only $1$ variable, in the same way separate oversampling can be achieved for $n$ variables. Of course more complex but certainly doable. For example one takes all distinct combinations of variables (or ranges of variables for continous variables) and separately oversamples each such cluster in order to maintain the given data distribution.
The above is a straightforward technique, although one should note that if minority class does not have enough samples there is no guaranty that the given data distribution reflects the (true) underlying data distribution (in other words it may not constitute a representative sample in statistical sense). So for these cases oversampling the whole data, without extra assumptions about underlying distribution, is a maximally unbiased method in the statistical sense.
There is some research lately on hybrid and intelligent methods for (oversampling) class imbalance problems without introducing bias during the process. The following references will provide the relevant background:
Cross-Validation for Imbalanced Datasets: Avoiding Overoptimistic and Overfitting Approaches, October 2018

Although cross-validation is a standard procedure for performance
evaluation, its joint application with oversampling remains an open
question for researchers farther from the imbalanced data topic. A
frequent experimental flaw is the application of oversampling
algorithms to the entire dataset, resulting in biased models and
overly-optimistic estimates. We emphasize and distinguish overoptimism
from overfitting, showing that the former is associated with the
cross-validation procedure, while the latter is influenced by the
chosen oversampling algorithm. Furthermore, we perform a thorough
empirical comparison of well-established oversampling algorithms,
supported by a data complexity analysis. The best oversampling
techniques seem to possess three key characteristics: use of cleaning
procedures, cluster-based example synthetization and adaptive
weighting of minority examples, where Synthetic Minority Oversampling
Technique coupled with Tomek Links and Majority Weighted Minority
Oversampling Technique stand out, being capable of increasing the
discriminative power of data

Learning from Imbalanced Data, 9,  SEPTEMBER 2009

With the continuous expansion of data availability in many
large-scale, complex, and networked systems, such as surveillance,
security, Internet, and finance, it becomes critical to advance the
fundamental understanding of knowledge discovery and analysis from raw
data to support decision-making processes. Although existing knowledge
discovery and data engineering techniques have shown great success in
many real-world applications, the problem of learning from imbalanced
data (the imbalanced learning problem) is a relatively new challenge
that has attracted growing attention from both academia and industry.
The imbalanced learning problem is concerned with the performance of
learning algorithms in the presence of under represented data and
severe class distribution skews. Due to the inherent complex
characteristics of imbalanced data sets, learning from such data
requires new understandings, principles, algorithms, and tools to
transform vast amounts of raw data efficiently into information and
knowledge representation. In this paper, we provide a comprehensive
review of the development of research in learning from imbalanced
data. Our focus is to provide a critical review of the nature of the
problem, the state-of-the-art technologies, and the current
assessment metrics used to evaluate learning performance under the
imbalanced learning scenario. Furthermore, in order to stimulate
future research in this field, we also highlight the major
opportunities and challenges, as well as potential important research
directions for learning from imbalanced data.

Data Sampling Methods to Deal With the Big Data Multi-Class Imbalance Problem, 14 February 2020

The class imbalance problem has been a hot topic in the machine
learning community in recent years. Nowadays, in the time of big data
and deep learning, this problem remains in force. Much work has been
performed to deal to the class imbalance problem, the random sampling
methods (over and under sampling) being the most widely employed
approaches. Moreover, sophisticated sampling methods have been
developed, including the Synthetic Minority Over-sampling Technique
(SMOTE), and also they have been combined with cleaning techniques
such as Editing Nearest Neighbor or Tomek’s Links (SMOTE+ENN and
SMOTE+TL, respectively). In the big data context, it is noticeable
that the class imbalance problem has been addressed by adaptation of
traditional techniques, relatively ignoring intelligent approaches.
Thus, the capabilities and possibilities of heuristic sampling methods
on deep learning neural networks in big data domain are analyzed in
this work, and the cleaning strategies are particularly analyzed. This
study is developed on big data, multi-class imbalanced datasets
obtained from hyper-spectral remote sensing images. The effectiveness
of a hybrid approach on these datasets is analyzed, in which the
dataset is cleaned by SMOTE followed by the training of an Artificial
Neural Network (ANN) with those data, while the neural network output
noise is processed with ENN to eliminate output noise; after that, the
ANN is trained again with the resultant dataset. Obtained results
suggest that best classification outcome is achieved when the cleaning
strategies are applied on an ANN output instead of input feature space
only. Consequently, the need to consider the classifier’s nature when
the classical class imbalance approaches are adapted in deep learning
and big data scenarios is clear.

Hope these notes help.


k-means clustering unclear result I am new in datamining. I decided to play with k-means clustering in r with simple аrtificial data.

set.seed(101)
x1 <- runif(100,36.6,37.5)
x2 <- runif(100,37.6,38.4)
x3 <- runif(100,38.5,40)
x<-c(50,150,250)
y<-c(37,38,39.25)

centers<-c(37.05,38,39.25)
all <- c(x1,x2,x3)
c1<-kmeans(all,centers=centers,iter.max = 1000,nstart=1,algorithm="Lloyd")
plot(all,col = c1$cluster)
points(x,y,col="red",pch=19)

I generated three random data sets x1,x2,x3. Then i taked center of each cluster like mean of each set and used k-means algorith of r. In result i get what you see in picture.

What i am making wrong?

Why second cluster contains the part of third set? How to improve result?

Why result is so bad instead sets is lineary separate?

The main question is how to improve clustreing result?

A 1 dimensional data set is "linearly separable" at any set of points between the data points. Kmeans isn't a nearest-neighbour type of clustering algorithm, it just divides the space so that each point belongs to its nearest cluster centre, so your data could have huge gaps between the source clusters, but the algorithm would be "wrong" if the clusters had different ranges.

For example, if you generate points from two clusters on the intervals (0,1) and (2,20), even though there's a gap between 1 and 2, the clustering algorithm will still assign a lot of the points from the higher source cluster to the lower cluster because they are nearer that cluster centre. Its distance to cluster centre for each points that counts, not distance to neighbours. Use another clustering algorithm (like nearest-neighbours) if you are expecting gaps between clusters.


How to color a tif file in R? I have a tif file that shows a map. On white background the border of ditricts are drawn with black, nothing else is on the picture. I want to color the districts according to certain values. Is it possible to do that in R? (since the picture doesn't "know" the name of districts in advance I suppose I have to do some manual work but it's ok for me)

UPDATE: the picture is similar to this one: http://www.geoindex.hu/wp-content/uploads/evk_106_20140101.png

As I said, maybe there is already a public domain vector file that fits your needs, e.g.

library(raster)
map <- getData("GADM", country="HUN", level=2)
plot(map, col=rainbow(length(map)), border=NA)

Otherwise, try a dedicated tool to polygonize your raster image and save it as a shapefile. E.g. using QGIS (German):

After saving the shapefile, you can load and process it in R:

library(raster)
map <- shapefile("evk_106_20140101.shp")
plot(map, col=rainbow(length(map)), border=NA)

(Side note: I assume it's ok for any copyright owner oto modify and use the image for demonstrational purposes in the above way. If not, drop a comment and I'll delete it.)


Can we learn f(x)=1/x using a neural network exactly? Is there a way to train a neural network as $f(x) = {1 \over x}$ precisely?

Since the activation function itself can be an inverse function, the answer is yes.


Python TimeSeries Error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value I'm creating a time series model in Python. I converted a int field that counted the days (1,2,3,4) to a datetime field using the following code:
from datetime import datetime, timedelta

start = datetime.strptime('2016-01-01', '%Y-%m-%d')

df['Day'] = [start + timedelta(x) for x in range(730)]

I checked and it worked. The first date starts with 01/01/2016. I checked the dtypes of my two columns. The Day column is a datetime64[ns], while the revenue columns is float64.
When I try to plot the following code, I get an error that says I'm receiving this error: ValueError: view limit minimum -36834.64916714945 is less than 1 and is an invalid Matplotlib date value. This often happens if you pass a non-datetime value to an axis that has datetime units. I'm a little confused as to what's causing this since my datatype on the day columns is a datetime.
rolling_mean = df.rolling(window = 12).mean()
rolling_std = df.rolling(window = 12).std()
plt.plot(df, color = 'blue', label = 'Original')
plt.plot(rolling_mean, color = 'red', label = 'Rolling Mean')
plt.plot(rolling_std, color = 'black', label = 'Rolling Std')
plt.legend(loc = 'best')
plt.title('Rolling Mean & Rolling Standard Deviation')
plt.show()
plt.close()

I assume an arbitrary df like:
df = pd.DataFrame(np.random.rand(720, 3), columns=list('abc'))

pyplot.plot([...]) doc expects a x and y, but you didn't specify what the x should be. If we include x as datetime values and y as the desired mean or std values, then the code looks like:
start = datetime.strptime('2016-01-01', '%Y-%m-%d')

df['day'] = [start + timedelta(x) for x in range(720)]

rolling_mean = df.rolling(window = 12, on="day").mean()
rolling_std = df.rolling(window = 12, on="day").std()

colors=['olive', 'gold', 'green', 'red', 'black', 'blue', 'purple', 'coral', 'teal']
for idx in df.columns.drop('day'):
    plt.plot(df['day'], df[idx], color = np.random.choice(colors), label = 'Original')
    plt.plot(rolling_mean['day'], rolling_mean[idx], color = np.random.choice(colors), label = 'Rolling Mean ' + str(idx))
    plt.plot(rolling_std['day'], rolling_std[idx], color = np.random.choice(colors), label = 'Rolling Std ' + str(idx))
plt.legend(loc = 'best')
plt.title('Rolling Mean & Rolling Standard Deviation')
plt.show()
plt.close()

This will give you this rainbow-coloured plot (if you wish other colors, check here)


How can I get students to stick with data science? I am a chief data scientist at one of the largest corporations in the world. I am constantly approached by employees to teach them more about data science and help them on their learning path. The problem is that no one really seems to stick with it over the long term, everyone just wants the "shortcut" to the sexy data scientist job without realizing the work needed.

So, in effect, how can I get more of my "students" to stick with data science? What other techniques have you all used (if any) to teach people successfully? I am posting this because I'm tossing around the idea of starting a small study group where multiple people will help each other learn data science and I will guide them along. I am hoping that this accountability to their peers (instead of just one-on-one with me) will help them.

I have not tried this yet, but I have decided to try it on a friend who is a software engineer. He is so used to picking up skills on the fly and I am concerned he will lose interest since data science takes a bit longer to learn.

I suspect a top-down-approach, like Jeremy Howard's fast.ai, will have an easier time to keep hobby students interested. I think people that express interest often come for the fancy deep stuff and getting them hands on with that will probably make it feel more worthwhile for them. Then when they already have toys to play with you can slowly progress into the fundamentals.

I don't know if it's the "best" way to learn data science, but learning the wrong way is better than losing interest and quitting.


What do the parameters used in crop mean? When we have an image to be used as an input to a CNN and we want to classify only part of the image, we usually feed the classifier with a crop of the image.
Lets say my image is called frame and x, y, w and h are xmin, ymin, xmax and ymax, respectively:
frame = frame[y:y + h, x:x + w] #Crop a part of the image

What does y:y or x:x mean and why do we sum them to h and w, respectively?
I've been seeing some people performing the crop in the following way:
frame = frame[y:h, x:w] #Crop a part of the image without adding to `w` and `h`

I saw the second approach being used in some places like in the following line: https://github.com/balajisrinivas/Face-Mask-Detection/blob/master/detect_mask_video.py#L51
What's the difference?

Lets say my image is called frame and x, y, w and h are xmin, ymin, xmax and ymax

You're confusing $w$ with $xmax$ and $h$ with $ymax$: Usually $w$ is the width of the crop whereas $xmax$ is the horizontal position of the end of the crop. Similarly $h$ is the height and $ymax$ is the vertical position of the end of the crop.
Logically since $x$ is the (horizontal) start of the crop and $w$ is the width, we can obtain $xmax$ like this: $xmax=x+w$.
Example: in a 100x100 image, let's say we want to crop a 20x20 square in the centre: $x=40, y=40, w=20, h=20, xmax=60, ymax=60$.
In the following code:

frame = frame[y:y + h, x:x + w]

the operator : is used to represent a sequence (for instance 3:7 means 3,4,5,6) so y:y + h represents the sequence from y to y+h, i.e. from $y$ to $ymax$. Same for x+w, so this line would select the part of the array corresponding to the crop.
Your second example is wrong due to the same confusion, the actual code is:
face = frame[startY:endY, startX:endX]

In this case the author is directly using the end coordinate endY (same as $ymax$) instead of calculating it as startY+h.


How to use binary text classifier(built using SVM with TF-IDF) to classify new text document? I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports).

But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the occurrence of a word in all other documents.

Do I have merge test and train data every time I receive a new document for classification? It will change the model as well every time.

Am I missing something? I think, although SVM on TF-IDF giving good results it can not be used in production.

Is there any other way to tackle this issue?

Lets take an example

Training Set:
Doc_1: Chelsea won the match. {Sports}
Doc_2: India won the third test match against Austrailia {Sports}
Doc_3: I want to sleep {Non-Sport}
Doc_4: 13 palace to see in Auckland {Non-Sport}

New Testing Set:
Doc_5: Climate change impacts in Austrailia

Now how can I find IDF score of "Austrailia" in Doc_5 without merging this document with training set?

Since Doc_5 contains the word "Austrailia", it will change the IDF score of "Australia" in Doc_1 will also change, thus model needs retraining

What is your model built in?

Most popular libraries have a score function separate from the training part. You should be able to just pass the new document to the score function of the trained model and get back the predicted class.


Interpretation of PCA visualisation I am trying to build a classifier to predict the ratings of a show during a specific time.

I have extracted around 109 features, some relating to the time field namely,

Day of Year
Month of year
Is on a weekend?
During business hours?
Public Holiday?

I also included some categorical features and used a label binariser for which channel it appeared on, and the broadcaster.

I wanted to check the linearity of the dataset, which would inform me as to whether a linear regressor could be used or something non-linear like a neural network. I decided to do dimensionality reduction using PCA in order to visualise if the dataset was linearly separable in 2D.

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
data_scaled = pca.fit_transform(df[cols])
plt.plot(data_scaled[:,0], data_scaled[:,1], 'ro')
plt.xlabel('first component')
plt.ylabel('second component')
plt.show()

I am very confused by the result and am not able to interpret this.

The plot of the first component:

The plot of the second component:

What could the PCA results tell? What would cause these graphs?

Do not apply PCA to categorical data

PCA attempts to find the dimensions which contains the most variance in a dataset. When you have categorical variables, the distance between points and the variance captured by a variable are ill-defined. First of all, you have no proper distance measure to tell how far apart two categories are, and secondly, the ordering of your categories will have a big impact. Ordering months categorically will result in a small distance between January and February, but a large one between December and January. If you're categorizing things like broadcaster or channel, there is no natural order to put things in, so your distance measure will be essentially meaningless. Your PCA plot looks strange because you have applied it to categorical data - this is not a meaningful representation of your data.


Classification training using probabilites and not raw classes (factors) I have a problem where instead of having classes, i.e. a vector of 0s and 1s, I have the probability of an observation belonging to a class.

A vector with 0.1, 0.95, 0.2, 0.3, etc.

The obvious approach is using regression and it works relatively well, but I'm interested in an approach that trains on these probabilities that an observation belongs to a class and classifies them.

A Multinomial Classification approach has also been tried. The problem with this approach is that it doesn't take into consideration the order of the classes  (factors) which causes in some cases that the highest predicted class for an observation is at one end (let's say 0.2) and the second one is at the other end (0.8). Further, the more classes there are the less scalable this approach it becomes. Conversely, if there are too few classes gaps in predictions start to appear.

So my question is: Is there a classification algorithm that supports training probabilities instead of just factors (0s and 1s)?
Alternatively, is there another approach that is not regression nor classification that can solve this problem?

Beta Regression

You could use beta-regression. I have no practical experience with this type of regression. However, it might be the right method for your task. As far as I understand, the link function is chosen so to restrict $\hat{y} \in [0,1]$.

Here is an R implementation, where the docs say:

  Fit beta regression models for rates and proportions via maximum
  likelihood using a parametrization with mean (depending through a link
  function on the covariates) and precision parameter (called phi).

Example:

library("betareg")
data("GasolineYield", package = "betareg")
summary(GasolineYield$yield)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.0280  0.1165  0.1780  0.1966  0.2705  0.4570

br = betareg(yield ~ batch + temp, data = GasolineYield)
preds = predict(br, newdata=GasolineYield)
summary(preds)

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04571 0.10309 0.16364 0.19655 0.26429 0.50792

Regression Models For Ordinal Data

Ordinal Logistic Regression could be used for this problem since classes are ordered and multinomial classification does not take the order of classes into consideration. In practice, this algorithm doesn't scale to many classes or many observations because its computationally expensive.

Here is an example of fitting a cumulative link model (CLM) such as the proportional odds model to data using the ordinal package in R.

require("ordinal")
fm1 <- clm(rating ~ contact + temp, data=wine)
summary(fm1)

formula: rating ~ contact + temp
data:    wine

link  threshold nobs logLik AIC    niter max.grad cond.H
logit flexible  72   -86.49 184.98 6(0)  4.01e-12 2.7e+01

Coefficients:
       Estimate Std. Error z value Pr(>|z|)
contactyes   1.5278     0.4766   3.205  0.00135 **
tempwarm     2.5031     0.5287   4.735 2.19e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Threshold coefficients:
    Estimate Std. Error z value
1|2  -1.3444     0.5171  -2.600
2|3   1.2508     0.4379   2.857
3|4   3.4669     0.5978   5.800
4|5   5.0064     0.7309   6.850

Regression with a Logistic Link Function

As suggested by Ben Reiniger in the comments of the question, another alternative is simply to use a Logistic Link function in a regression model.

An example would be using xgboost with reg:logistic as the objective function. However, many libraries may not support this behavior as they need the target be either one or zero.


Predicting Missing Features I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to have a probability of the "0" features being "1".

I am thinking this can be a Neural network with all features as input and same as output. But then I don't know how the training would work. I don't have ground truth.

I would like some help expressing my problem and hopefully not reinvent the wheel. Is this is a classical problem in ML, and what approach can be applied?

A simple approach could be the following: suppose $i \in \{0,1\}^d$ is the vector you want to predict which of the $0$ entries could be $1$ and $j \in J$ the rest of the feature vectors. Take the $k$ nearest neighbors, under some suitable distance (Jaccard, Hamming, Manhattan distance). For each $0$ entry the probabilities could be the percentage of the $k$ nearest neighbors that have $1$ in the corresponding entry.

This problem has been extensively study in the collaborative filtering community. The best known example being the Netflix Prize. This blog post provides a nice explanation of this approarch for binary data.

Another, more involved, approach is matrix completion, in particular check this reference. If you are into deep learning check this.


From a mix of real-world and back-calculated data how to remove the part that was back-calculated? I have a geostatistical dataset and I've been building linear regression model, but when I plotted the data I've noticed that part of the data shows an absolute straight line trend, i.e. it is most likely not a real-world data, but it was back-calculated.

All the data is in pandas dataframe and I guess I need to find subset of the dataset that shows a perfect linear relationship, but I'm not quite sure how to approach it.

Try to fit several linear regression models, for very small number of observations from the data set (for instance, $n$ or $n+1$ randomly selected points, where $n$ is the number of dimensions). If there are enough perfectly linearly aligned points and enough models, it is likely that one of the models will be built from points that were only drawn from the back-calculus.

Then compare the models predictions to the actual data. For the models which perform very well on a large number of test cases (i.e. more than $m$ points are correctly predicted with an error of $\epsilon$ or less), these observations have probably been computed.

You will have to define $m$ and $\epsilon$ based on your knowledge of the problem / by adjustment.

If I may suggest, you can then keep these observations in your dataset, but give them a lower weight for the final model training.


How to expand abbrevations in text during preprocessing? Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has abbreviations in plural forms with and without contractions like:

Mgr's = manager
mgrs = manager
mgr = manager

All 3 points to a manager. Im able to capture the plural form with contractions using a regex(r"'s") and removing the 's' but, in case of no contractions i'm creating one more entry in the dictionary with plural form of the abbreviations.
Im somehow feel this is duplication and not a clean approach. Is there any better solution to address this problem?
Any immediate help on this is much appreciated. Thank you

Do it in two steps: replace the ones with 's' first, then do the rest of them.


Classifying on imbalanced dataset I have incidents VS normal operation of my working environment. It is a skew dataset. My prediction accuracy is 95%.

Question:
1. Is it common practice among data scientist to accept this prediction?
2. Do I have to rework by resample and balance the skewness and train & test again?

I am sorry to ask for partial opinion based question.

The first question is very broad, 95% accuracy doesn't say a lot, there are tasks where 99% is standard and there are tasks where an accuracy of 60% is amazing.
It is hard to answer this question without more information about your model, dataset and the affects of its outcome. Basically you should ask yourself if it is good enough for your task.

Regarding the second question, I would suggest to use a weighted loss function, where mistakes on the rarer classes have a higher price (it encourages the model to classify them correctly). For optimal performance, you should retrain your model from scratch.

Note for future reference, it is better to present the results in a confusion matrix, than just the final classification (gives more data, especially when we talk about a skewed dataset).


you have 99 leaf species and 594 test records. Also the output is (594,). Here each value represents corresponding leaf species columns index.
 check the species data you would have received after One-HotEncoder/get_dummies.
each value of "computed_predictions" corresponds to their respective column number.
you can export species column_name to a list(of 99 labels) and get the labels using list index(i.e. computed_predictions respective value)


Resources for text classification algorithms I'm mining raw Facebook comments (irrelevant) and i am looking for an algorithm that can classify their context as negative/positive/neutral. So you can think of the output in the form of two columns. First column the comment (exists), and second column its classification. If possible i'd like for the algorithm to be able to process the sentence as is (e.g. not having to remove stop words etc). Can someone recommend any resources that i can refer to? I have no problem doing this in either Python or R.
Thanks

So, I believe what this question is asking for literature around sentiment analysis of Facebook comments (positive/neutral/negative) sentiment.
There is quite a good paper on this using an encoder-decoder architecture here: https://www.aclweb.org/anthology/Q18-1002/


Help with approaching this problem correctly newbie here trying to figure this out:

I have a dataset which looks like:

facility_id   area_zipcode  staff_count   population
FacilityA       98007           21          24889
FacilityB       98290           52          32714
FacilityC       98065           43          12699
FacilityD       98801           9           40977
FacilityE       98104           64          13095

Here is my problem statement:

There are five healthcare facilities in five different areas, each with their respective staff count. Population in a certain area can only go to the facility which is nearest to them. So for people living in area A, facility A is the closest, but sometimes the staff count may not be enough to handle everybody.

We're assuming at least half of the population will require healthcare, and that one staff member can only tend to about 2000 patients per year. How would we optimize the staff's distribution in all these facilities such that all five facilities can tend to the majority of their populations, thereby minimizing everyone's travel times. Basically allocate staff count to where they are most needed, increasing or decreasing capacity in facilities.

My speculation so far:

This is certainly not a classification problem, I'm thinking it's a regression problem since we're interested in some real value outputs i.e optimal staff count for each facility. Because we have multiple outputs, I'm leaning towards multi-output regression. I'm also supposed to find travel times using bing maps API, which I have successfully figured out how to do, but I'm confused with what to do with all these travel times. (like a to b, a to c, a to d etc) or how to add them to my dataset. There seem to be two priorities in the problem statement:

1) Optimize staff counts

2) Minimize travel times

And currently I'm a bit confused as to how I should tackle this, I was hoping someone could point me in the right direction, with how to model this problem and how to achieve both those goals. I'm working in python, and I can make any assumptions to simplify the problem. Any help is appreciated!

Also if you know of research papers, or others who have worked on similar problems, please guide me on how to look for them, what type of problem is this etc?

This problem sounds like an optimization problem , where the goal to find the best solution under constraints. In this case, the solution space would minimize travel times while meeting staff constraints.
Often the most difficult aspect is encoding the raw data into a format that works for the optimization solver. Generally, try a simplest encoding that could possible work. For example, use distance as a proxy for travel times.
PuLP is a linear programming package for Python that could solve the problem once it is properly encoded.


What is syntax V and S standing for nominal subject? I was reading the recent paper https://www.aclweb.org/anthology/P19-1580.pdf and noticed that in section 5.2, the syntactic relation is studied in terms of the "direction between two tokens". In table 1, the result is further shown with direction like $$v \to s, s \to v$$ for nsubj(nominal subject).

However, what does V and S refer to here and where I can find more materials about this ?

This is explained in the Methodology section of the paper:

  We calculate for each head how often it assigns its maximum attention weight (excluding EOS) to a token with which it is in one of the aforementioned dependency relations. We count each relation separately and allow the relation to hold in either direction between the two tokens.

The nsubj dependency label is used in arcs between the verb ($v$) and a subject ($s$). In the paper, they define this "dependency score" taking the maximum attention weight and checking if it points to the expected word according to the dependency parse arc, in either of the directions. This is why they count it from verb to subject and from subject to verb.

If you want to know more about the conventions usually followed in dependency parsing, including these nsubj, amod, etc, you can take a look at the Stanford typed dependencies manual.


Overfitting in Huggingface's TFBertForSequenceClassification I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve the overfitting using some dropout but the performance is still poor. The model is as follows:
# Get and configure the BERT model
config = BertConfig.from_pretrained("bert-base-uncased", hidden_dropout_prob=0.5, num_labels=13)
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-uncased", config=config)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=0.00015, clipnorm=0.01)
loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.CategoricalAccuracy('accuracy')

bert_model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
bert_model.summary()

The summary is as follows:

When I fit the model, the outcome is:
history = bert_model.fit(train_ds, epochs=30, validation_data = test_ds)

From my experience, it is better to build your own classifier using a BERT model and adding 2-3 layers to the model for classification purpose. As the builtin sentiment classifier use only a single layer. But for better generalization your model should be deeper with proper regularization. As you have around 13 class you should use deeper model with a good number of training examples for each class.


Why are bigger embedding vectors not necessarily better? I'm wondering why increasing the dimension of a word dimension vector in NLP doesn't necessarily lead to a better result. For instance, on examples I run, I see sometimes that using a pre-trained 100d GloVe vector performs better than a 300d one. Why is this the case? Intuitively a larger dimension should become almost like a one-hot encoding and be in some more "accurate," no?

You can think about phenomenons close to the curse of dimensionality.

Embedding words in a high dimension space requires more data to enforce density and significance of the representation.
A good embedding space (when aiming unsupervised semantic learning) is characterized by orthogonal projections of unrelated words and near directions of related ones. For neural models like word2vec, the optimization problem (maximizing the log-likelihood of conditional probabilities of words) might become hard to compute and converge in high dimensional spaces.

You’ll often have to find the right balance between data amount/variety and representation space size.


1x1 convolutions, equivalence with fully connected layer I'm confused by the concept of equating a 1x1 convolution with a fully connected layer. Take the following simple example of a 1x1 convolution of 2 input channels each of size 2x2, and a single output channel.

The only way I can relate this to fully connected layers is to say that there are 4 fully connected layers, one for each location in the input feature map (inputs and outputs colour coded).

From what I can understand my interpretation is consistent with the Network in Network paper[Lin et al. 2013] which describe the 1x1 as being equivalent as cross channel parametric pooling

  The cross channel parametric pooling layer is also equivalent to a
  convolution layer with 1x1 con- volution kernel.

I have seen
 this one from Yann LeCunn equating 1x1 convolutions to a fully connected layer. And I have read this answer and I'm just not seeing the equivalence between a 1x1 convolution over an input volume and a single fully connected layer...

Any insight would be appreciated, if you can please relate back to the example above. Thanks!

The interpretation that the 1d convolution given in the OP can be duplicated with four separate fully-connected layers is correct (see diagram).  Also, in at least some implementations, kernel weights used during a 1x1 convolution can be made trainable the same way weights in a fully-connected layer can be made trainable.  These points made, every fully-connected layer can not be mathematically duplicated by an equivalent 1x1 convolution. This is based on the definition that 1x1 convolution performs a "column-wise dot product" such that every pixel column in a multi-layer feature map is reduced to a single number (pixel).  A fully-connected layer intermixes weights differently from the way weights are intermixed when performing a 1x1 convolution.  In summary, fully connected layers and 1x1 convolutions each have their own use cases -- some overlap among these use cases exists; however, the two are not intended to be mathematically equivalent in a general sense.


One hot encoding at character level with Keras I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says:

Note that Keras has built-in utilities for doing one-hot encoding of text at the word level or character

I have looked into Keras metods and I cannot find which function he is referring to. keras.utils.to_categorical does not seem to be applicable directly here, as it requires int inputs.

I would like to efficiently encode some text at character level for a small RNN project: what can I use there?

I think that you are looking for the keras Tokenizer with the char_level=True flag:

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True)
tokenizer.fit_on_texts(your_dataset_train)
sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test)

Now that you have sequences of Integer, you can use keras.utils.to_categorical =)


Which one first: algorithm benchmarking, feature selection, parameter tuning? When trying to do e.g. a classification, my approach currently is to

try out various algorithm first and benchmark them
perform feature selection on the best algorithm from 1 above
tune the parameters using the selected features and algorithm

However, I often cannot convince myself that there may be a better algorithm then the selected one, if the other algorithms have been optimized with the best parameter / most suitable features. At the same time, doing a search across all algorithms * parameters * features are just too time-consuming.

Any suggestion on the right approach / sequence?

I assume you mean feature selection as feature engineering. The process I usually follow and I see some people do is

Feature engineering
Try a few algorithms, usually highly performant ones such as RandomForest, Gradient Boosted Trees, Neutral Networks, or SVM on the features.

2.1 Do simple parameter tuning such as grid search on a small range of parameters

If the result of step 2 is not satisfactory, go back to step 1 to generate more features, or remove redundant features and keep the best ones, people usually call this feature selection. If running out of ideas for new features, try more algorithms.

If the result is alright or close to what you want, then move to step 3

Extensive parameter tuning

The reason for doing this is that classification is all about feature engineering, and unless you know some incredible powerful classifier such as deep learning customized for a particular problem, such as Computer Vision. Generating good features is the key. Choosing a classifier is important but not crucial. All the classifiers mentioned above are quite comparable in terms of performance, and most of the time, best classifier turns out to be one of them.

Parameter tuning can boost performance, in some cases, quite a lot. But without good features, tuning doesn't help much. Keep in mind, you always have time for parameter tuning. Also, there's no point of tuning parameter extensively then you discover a new feature and redo the whole thing.


Validate via predict() or via fit()? There are several possibilites to evaluate a model:

hist = model.fit(x_train, y_train,
          (...)
          validation_data=(x_test, y_test))

or to use .evaluate() resp. .predict(). As far as I know I would use X_test and y_test for validation and prediction.

My question: How are they different? When I .fit() and .evaluate() afterwards, which information do I gain by .fit(validation_data(...))?

.fit --> You can give Train data and Validation data as well so Train data is used for Training and Validation data is used for validating the Trained model at the end of each epoch.
.predict --> It takes the only X and gives the output of the model.
.evaluate --> It will take X, Y and will evaluate similar to an above evaluation in .fit but after training total model we will use this.
You can write
.evaluate = loss_function(.predict, y_true)
score(.predict, y_true)


Aggregate several attributes into one using PCA I am working on a soccer dataset. As we all know, soccer has a lot of different metrics related to the game. We have Key passes, Accurate Passes, Shots on Target, Saves, Clearances and so on...
I have more than 200 columns which makes it infeasible to analyze every single player based on every stat.

There are some stats that are more important to one position that the others. For instance, I want to know the Shots on Target of a forward but not for goalkeepers. So, I'm thinking in a way to group important stats per position.

My first thought was using PCA and I would like to know what you think about it. Is it possible to get the principal component for the stats that I'm interested for every position?

Example:
Goalkeepers have the following interesting stats:
- Diving Save
- Saves Inside Box
- Saves Outside Box
- Sweeper Keeper

Strikers:
- Inbox Attempts
- Key Passes
- Shots on Target

I want to reduce the dimensionality of all these important stats and have a singular stat resumes how the player of that position is performing.

Is PCA the best choice or is there a better option?

Kind Regards

I see. Yes PCA is a good tool that you can use on your choice of attributes. The tricky part is that PCA is based on variance. Hence, if you have 2 attributes which are more important and have low variance and then 5 more with a lot of variances then the latter will be predominant on the first PC (I guess some standardisation would help). Hence, maybe it makes more sense to incorporate some corresponding weights on each attribute, a weighted average for instance (dummy example, but may fit the cause).


How should I collect data for an "identifier" rather than classifier? I'm trying to build a CNN to identify whether or not a picture contains a lion. One of my classes would obviously be filled with pictures of lions, and pictures that have lions.

What about the data I put in my not_lion class? Should I fill it with images of random things (for example, take a few images from each class in ImageNet)? Should I put in pictures of things similar to lions, like tigers, so the network better learns what a lion actually is?

My idea is to make 50% of the not_lion class pictures of tigers, pumas, etc, and 50% random stuff - I think that this would make the network learn what a lion actually is, and also ensure the network doesn't classify a laptop as a lion.

It depends on the use you are giving. The $not.lion$ class should contain sufficient data of different things, but also should contain tigers, cats, african landscapes without lions, etc.

Your idea of 50-50 is good.

Add this: The weights $w$ could help you define where your model should focus: Lesser weights when there is something easily distinguishable from a lion like a computer, but larger weights when there is an african landscape.


How to interpolate and check correlation of two time series with differing cardinality I want to check how correlated two time series are, but they don't have the same cardinality. They have different number of data points because the timestamps the data are collected are different. Available libraries I have found require that the cardinality be the same. Therefore, I would like to ask if there is a library, an algorithm I can implement myself, or some advice you can give me to approach the problem. Thank you

I understand that your time series are unevenly spaced. In this case, why not simply use a library like traces and transform them to evenly spaced time series.


Overall classification is generally better when the decision rules of each component classifier differ and provide complimentary information.

So the question becomes: Can you set up your component classifiers so that their decision rules are different and compliment one another based on the feature space? e.g. Does Post 1 have a significantly different feature space than Post 2?  etc. If so, the ensemble approach should be beneficial.

Which technique?  If you can highly train each classifier and make it an expert in different regions of the feature space, try models:

mixture model
mixture distribution
gating subsystem
winner take all.


Creating a composite score from dataset with no target variable I have a dataset that includes 6 variables about prospective sales opportunities (probability of closing, days until expected close, age of opportunity, etc.). 2 of the columns are categorical and 4 are continuous. I am looking to create a composite score of each opportunity based on these variables.

I have tried manually assigning scores to each column based on arbitrary ranges, but the composite scores ended up being too similar. I want to optimize the scoring for each variable. Is there a way to do this through machine learning? I want to find out which variables are the most important, or "stand out" the most from the rest of the pack. I also would like to see if any variables are redundant.

If you dont have the outcome of the observation, it is not possible to find which variables are more important (important to predict what ? the « what » is the target value).

I encourage you to gather some records (20 times the number of features is a good starting point) with targets so the algorithm can understand what features are relevant to the specific outcome you want.

However it is already possible to calculate correlations between variables and do some data analysis to understand how much variance each feature contains.


Recording changing values over time I am somewhat confident this is the correct approach but I need a little sanity check before setting out on this.

I want to record a numerical value for a category over time. There will be many categories and may appear as follows:

1  cat_1
4  cat_2
10 cat_3
4  cat_4
10 cat_5
8  cat_6
99 cat_7
4  cat_8

Say, the count (on the left) on the left will be different each time I poll for the data.

I will want to run analysis on this later on.

My plan is to store the data each time I poll for it in a JSON encoded object in S3 (along with the timestamp). This means it will be in raw form.

Later on I will want to query that information to see the values per category over time.

Will this be sufficient and easy to analyse later?

It sounds like you're planning to store all the input data. That has to be sufficient if your problem can be solved. There is nothing else...

Whether it's "easy", that's a relative term and it depends on many things. It depends on the kind of analysis that you want to do and the amount of data, among other things. If you have less then a few MB of data and you just want to calculate basic statistics like the mean or the sum, then Excel is much easier than S3. But S3 is very scalable, and it can easily be read from systems that allow to train complex models, so if you have a lot of data and that's what you want to do, then it could be a good choice.


Why am I getting very different results between SVC, LinearSVC and Naive Bayes? I am doing classification by using bag-of-words model. The goal is to locate users based on their tweets. Splitted the data as 80% training and 20% test.

I did experiments with sklearn's SVC and Naives Bayes. The results 35% and 42% accuracy respectively. However, when I try the sklearn's LinearSVC algorithm, it gives me 80% which is shocking.

This is the part of the code:

text_clf = Pipeline([
    ('vect', CountVectorizer(stop_words='english')),
    ('tfidf', TfidfTransformer()),
    ('clf', LinearSVC()),
])
text_clf.fit(train_data, train_target)

What might be the reason for that? Why LinearSVC performs really good?

These models are used for convex optimization. which means that there is only one solution for the problem. LinearSVC and SVC are used for same purpose but the optimization techniques used are different. E.g. LinearSVC intercept is penalized while in SVC it isn't. there might be differences scaling or default loss function. Hence they produce different results. LinearSVC tends to optimize faster. in multiclass classification, liblinear does one-vs-rest by default whereas libsvm does one-vs-one.

if you want to know how they can produce similar results. You can check out this question.
https://stackoverflow.com/q/33843981/5947203

Now the reason why SVN produced better results than Naive Bayes is that in this problem features are important. Naive Bayes treat them as independent while SVN looks at the interactions between them to a certain degree. Mathematically, One is probabilistic while other is geometrical. The dependencies are not captured by Naive Bayes hence it does not produce good results.