Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split I'm trying to use GridSearchCV from scikit-learn and look at the difference between train/test metrics.

When I do a normal test/train split with RandomForestRegressor, the metrics are comparable.  Something like:
Train R2: 0.97
Test R2: 0.85

However, when I try to use the same data with GridSearchCV, the testing and training metrics seem to be completely different, the Test accuracy is
a large negative number instead of being something between 0 and 1.

from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import GridSearchCV

...

rf_reg = RandomForestRegressor(max_depth=10,
                              random_state=RANDOM,
                              n_estimators=100,
                              n_jobs=N_JOBS)

param_grid = {
              "min_samples_leaf": [1, 2, 4, 10]
             }
grid_cv = GridSearchCV(rf_reg, param_grid, cv=5, return_train_score=True)
grid_cv.fit(X, y)

I'm surprised that when I examine the scores for testing and training, they appear to be two different metrics.  The default score for RandomForestRegressor is R2, but the results for the test sets look like they're another metric entirely.

results = pd.DataFrame(grid_cv.cv_results_)
print('Train scores:\n', results['mean_train_score'])
print('Test scores:\n', results['mean_test_score'])

Train scores:
0    0.974572
1    0.963771
2    0.936328
3    0.877382
Name: mean_train_score, dtype: float64

Test scores:
0   -5.948434
1   -5.798446
2   -6.034835
3   -6.655515
Name: mean_test_score, dtype: float64

The train scores make sense to me, they should be between 0- 1 because I'm expecting R2 error metrics, the default for a RandomForestRegressor.  But why are the test scores a different metric? They should also be between 0 and 1, how is it possible to get negative numbers?  This does not make sense to me.

The same thing also happens with cross_val_score, I'm expecting an R2 metric, but it returns negative numbers. Even explicitly setting the scoring method to 'r2' returns negative numbers.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(rf_reg, X, y, cv=5, scoring='r2')
print(scores)
[ -5.15970579  -4.67536964 -13.17643335  -2.11630272  -4.51688508]

R2 can be negative if the model is arbitrarily worse according to the sklearn documentation
So the very negative train scores were indicative of an extremely bad performance.
Why was the test performance so colossally bad in GridSearchCV when it was decent in a simple test/train split?
The main problem is that train_test_split chooses observations randomly while GridSearchCV does not!
My problem was that the dataframe was sorted by the target variable!
The GridSearchCV and cross_val_score do not make random folds.  They literally take the first 20% of observations in the dataframe as fold 1, the next 20% as fold 2, etc.
Let's say my target is a range between 1-50.  If I sort my dataframe by target, then all observations are in order from 1 to 50.  The first fold of the cross-validation will take (for example) only observations with a target between 1-10, save this for the test, then train the model only on targets of 20-50.  This is why it was performing so badly!  The model was trained on a certain range, the test set only included a target range the model had never seen before!
The solution is simple. Shuffle the original dataframe before splitting into X, y for cross-validation.
df = df.sample(frac=1, random_state=0)

This solved my problem, now the test and train scores from GridSearchCV are both between 0-1, comparable to a simple train_test_split.
Lesson learned: Always shuffle a dataframe before a cross-validation - otherwise the folds will be subject to any biases in the order of how data was collected.


Creating a training dataset from analytical solution I am currently redesigning an inverse problem on an experimental technique, but I am having doubts about how to create a training dataset. Here is the problem I am trying to solve:
I have already created a model in order to solve the forward problem, such that if I input $x_1,x_2,...,x_n$, the model solves a generalized eigenvalue problem and returns the expected output $y_1,y_2,..,y_m$.
Thus, what I am trying to do is to generate random $(\vec{x},\vec{y})$ pairs in order to create a training data set that will allow me to solve the inverse problem. Considering this, I am confused as if the random $(\vec{x},\vec{y})$ pairs should be considered as synthetic data (since they come from the actual analytical solution). If so, should I generate the training data by following a probability distribution and adding noise?
Thanks in advance for the help!

I think your question depends almost entirely on the actual problem that you're trying to represent, i.e. it depends on (your) expert knowledge.
Yes, data generated this way should be considered synthetic. However there's no general requirement to add noise: adding noise (as well as how much and in which way) is normally meant to make synthetic data more realistic, i.e. closer to real data.
In general with synthetic data it's important to show that the data is representative enough of the real phenomenon it represents. Again this depends on expert knowledge but most of the time it's essential to make the distribution of the data realistic as well. So yes, if you know the true probability distribution it's certainly a good idea to make the synthetic data follow it.


Suggestions on what patterns/analysis to derive from Airlines Big Data I recently started learning Hadoop,
I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data),

I want some suggestions as what type of patterns or analysis can I do in Hadoop MapReduce, i just need something to get started with, If anyone has a better data set link which I can use for learning, help me here.

The attributes are as:

1   Year    1987-2008
2   Month   1-12
3   DayofMonth  1-31
4   DayOfWeek   1 (Monday) - 7 (Sunday)
5   DepTime actual departure time (local, hhmm)
6   CRSDepTime  scheduled departure time (local, hhmm)
7   ArrTime actual arrival time (local, hhmm)
8   CRSArrTime  scheduled arrival time (local, hhmm)
9   UniqueCarrier   unique carrier code
10  FlightNum   flight number
11  TailNum plane tail number
12  ActualElapsedTime   in minutes
13  CRSElapsedTime  in minutes
14  AirTime in minutes
15  ArrDelay    arrival delay, in minutes
16  DepDelay    departure delay, in minutes
17  Origin  origin IATA airport code
18  Dest    destination IATA airport code
19  Distance    in miles
20  TaxiIn  taxi in time, in minutes
21  TaxiOut taxi out time in minutes
22  Cancelled   was the flight cancelled?
23  CancellationCode    reason for cancellation (A = carrier, B = weather, C     = NAS, D = security)
24  Diverted    1 = yes, 0 = no
25  CarrierDelay    in minutes
26  WeatherDelay    in minutes
27  NASDelay    in minutes
28  SecurityDelay   in minutes
29  LateAircraftDelay   in minutes

Thanks

There really is no wrong answer here, but I recommend predicting flight cancellations (#22) and/or delays (25-29), since this is how I often see this data set being used. It could also have practical significance to you if you should ever find yourself flying to or departing from one of the worst offending airports/airlines.

I'm not sure if you have a choice (perhaps your employer requires it), but don't use Map Reduce -- it's incredibly difficult to learn/maintain, it's slow, and on top of that it has become obsolete. Use something like Spark's ML lib (http://spark.apache.org/docs/latest/mllib-guide.html). It's much easier to use and is much more current.


LSTM text generation Most of the examples I have found online for LSTMs refer to "random" text generation.
One of the problems that I'm trying to solve is to generate a "summary" of many docs into 1 doc. For example:

News Article: Barack Obama life history (2K words)
Wikipedia: Barack Obama  (10K words)
Barack Obama biography (10K words)

Will generate an article about Obama in X words.

Is LSTM the right approach?

Although you can generate text in this way - sampling from a RNN trained to predict next character or word - it will not be meaningful. At best it will be grammatically accurate (in terms of nouns, verbs, adjectives etc) but semantic nonsense. It would not be a summary, except by lucky accident.

To generate summaries using LSTM directly, you would need to train a network with real examples of summary inputs and outputs. This would be similar task to machine translation, but much harder due to variability in size of the inputs. It is unlikely that you will find enough training data to test the idea fully, and it is not clear that such a direct approach can yield acceptable results even with large amounts of training data.

In fact text summarisation is not a solved problem by any means. There are deep learning approaches, such as Google Brain team's effort using TensorFlow, which you could study to get some sample code and a sense of state-of-the-art. This approach uses an attention model to extract apparently informational content (i.e. content that would have low probability of appearing in some assumed generic document, thus is assumed to be interesting due to standing out). It is possible to use a trained LSTM to build such an attention-based model - the intuition is that the parts of the document that the already-trained LSTM is least able to predict are likely to contain noteworthy information.


What are the basic approaches for balancing a dataset for machine learning? Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?

Well, you just train it on unbalanced dataset, it is not a problem. I don't think you need to apply some special techniques.

The only case where you may want to do something special is if your classes are skewed by their nature (if the skewness is the property of class itself, not of just your specific dataset). For example, if you build a classifier which will tell if someone has a malignant tumor in X-ray image or not. By nature of the task absolute majority of patients will not have a malignant tumor. In such case you may want to tune your classifier algorithm a bit, for example introduce a weighting into your SVM or smth like it.

A special case is if you have EXTREMELY unbalanced classes. For example 100000 of positive examples and 20 negatives. In such case you will want to go away from classification task to the approach called 'anomaly detection'.


Is there any python framework which take a dataframe and give all important relation? Is there any python framework which takes a dataframe and gives all important relations?

For example, feature 1 and feature 2 have strong correlation when feature 3 (nominal) is equal to a special value.

There is a one-liner solution using a Python package called pandas-profiling that gives you a quick way into most crucial statistical explanatory analysis including various correlations and many more. The documentation provides a demo that is worth checking.


ValueError in CNN+RNN model in keras I am trying to build a CNN+RNN model for a computer vision problem.
below is my code

def cnn_with_rnn(shape):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), strides=(2, 2), activation="relu",kernel_initializer='truncated_normal',bias_initializer='truncated_normal', input_shape=shape))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(64, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(MaxPool2D(pool_size=(2, 2)))
    model.add(Conv2D(128, (3, 3), strides=(2, 2),kernel_initializer='truncated_normal',bias_initializer='truncated_normal',  activation="relu"))
    model.add(LSTM(50))
    model.add(Dense(1))
    model.compile(optimizer=Adam(lr=1e-5), loss="mse", metrics=[custom_metric])

My image is a RGB image with the following shape - [66,200,3], where 3 is the number of color channels.

When i am trying to run the above code , i am getting the following error

ValueError: Input 0 is incompatible with layer lstm_3: expected ndim=3, found ndim=4

How can i combine CNN+RNN for colored images and how to solve my above problem?

It might not be directly possible to shoehorn the output of your CNN directly into an LSTM (at least without being a lot more thorough with your dimensions).

Another approach is to have a look at the Keras wrapper layer: TimeDistributed. You essentially extract features using your Conv layers as usual, but do that over time-steps, not just a random mini-batch. The features the come out then are e.g. over 5 consecutive timeframes -> this would be one single sample for the LSTM. If you perform a minibatch of say 10 samples, this means 10 * 5 = 50 input frames.

Straight from the linked documentation:

  This wrapper applies a layer to every temporal slice of an input.

  The input should be at least 3D, and the dimension of index one will
  be considered to be the temporal dimension.

  Consider a batch of 32 samples, where each sample is a sequence of 10
  vectors of 16 dimensions. The batch input shape of the layer is then
  (32, 10, 16), and the input_shape, not including the samples
  dimension, is  (10, 16).

  You can then use TimeDistributed to apply a Dense layer to each of the
  10 timesteps, independently:

# as the first layer in a model
model = Sequential()
model.add(TimeDistributed(Dense(8), input_shape=(10, 16)))
# now model.output_shape == (None, 10, 8)

As a hint for dimensions, make use of model.summary() on a compiled model in order to inspect the dimensions of each layer through the model. You will likely have to get a pen and paper out to work through the dimensions, as things start to become a little complicated when using such a wrapper layer.

Unfortunately I don't have any links/tutorials to show how this is exactly done in Keras, but hopefully the description above gives you enough info for some useful Bing! searches. Just kiddin... Google searches  ;-)


Nice real data sets for testing DBSCAN? I'm looking for real datasets on which I could test my DBSCAN algorithm implementation, that is, a dataset of points in (ideally 2 dimmensional) space, or a set of nodes and info about the distances between them.

I have looked on SNAP and CRAWDAD for such datasets, like datasets of road networks with distances, or cities with GPS coordinates, etc, but I haven't found any!

I know that the DBSCAN is said to be one of the best algorithims of it's kind on real data, but can't seem to find the real data sets people use...

Suggestions?

If you want to test whether your algorithm works as expected, I'd use sklearn datasets. They allow you to create simple synthetic 2D data with certain properties: circles, half moons, etc.

If you want "real" datasets, here is an interesting resource found after a brief search:

https://uni.hi.is/helmut/2019/06/20/datasets-for-dbscan-evaluation/

It seems to be a collection of datasets used in the literature.

Otherwise, I'd recommend you to look for image segmentation datasets, for instance, maps, as they make good candidates for DBSCAN. Kaggle is good place to search, so is the Google Dataset Search tool


Reward is converging but actions taken by trained agent are illogical in reinforcement learning I am training a reinforcement learning agent using DQN. My state space has 6 variables and the agent can one action which is discretized into 500 actions

My reward structure looks like

thermal_coefficient = -0.1

        zone_temperature = output[6]

        if zone_temperature < self.temp_sp_min:
            temp_penalty = self.temp_sp_min - zone_temperature
        elif zone_temperature > self.temp_sp_max:
            temp_penalty = zone_temperature - self.temp_sp_max
        else :
            temp_penalty = 0

        reward = thermal_coefficient * temp_penalty

my temp_sp_min is 23.7 and temp_sp_max is 24.5. When i train the agent based on epsilon greedy action selection strategy, after around 10000 episodes my rewards are converging.  When I test the trained agent now, the actions taken by the agent doesn't make sense, meaning when zone_temperature is less than temp_sp_min it is taking an action, which further reduces zone_temperature.

I don't understand where am I going wrong. Can someone help me with this?

Thanks

Without seeing the rest of your code, this is a bit tricky to answer, but you have to make sure that reward = -temp_penalty, i.e. the negtive penalty, otherwise you would learn exactly the behaviour you described. Here I'm assuming you excluded all other potential sources of error.

Further, I think it might be helpful to issue a reward if the agent stays within the limits you defined, i.e. in the else-clause set temp_penalty = -1. or something like that. I personally found this type of tweaking to be very helpful for DQN.


How can I improve the results of my clustering I am working on a project with the idea to cluster the sound waves of key strokes on a computer. So far what I have done was recorded about 50 keystrokes per key (only have done 1 - 10 so far), found their peaks and isolated the waveforms, then I created a chroma vector as the Feature Vector and run k-means on the resulting feature matrix.

After some very minor filtering I was able to get an accuracy of about 90% (taking accuracy as the mode cluster label for a group of keys as the true value, how many are the true value / total values). However, when I try to essentially classify new data it is far from accurate.

My initial thoughts as to why this probably wont work:

The difference in any audio feature between keystrokes is minimal
Many many more samples are required
Clustering is not the best approach here (I don't have any formal reasoning here but I would guess a supervised alg would work better).
The pre-processing I'm doing is not right. The only things I have done were normalize the audio files and use a smoothing function on the waveforms (no frequency analysis here!).

Any suggestions are appreciated.

Always use a supervised algorithm when you have labeled data for your problem. Why would you ignore the labels, your most valuable bit of information?

To improve quality, you most likely need to improve your features.


Keras BatchNormalization axis I use spectrogram as input to a Convolutional Neural Network I have created with tensorflow.keras in Python.

Its shape is (time, frequency, 1).

The input's shape of the CNN is (None, time, frequency, n_channels) where n_channels=1 and the first layer is a Conv2D. In between every Convolutional layer I use a BatchNormalization layer before an Activation layer.
The default value for BatchNormalization is "axis=-1".
Should I leave it as it is or should I make it with "axis=2" which corresponds to the "frequency" axis?

The thought behind this is that the features of a spectrogram are represented in the frequency axis.

Interesting question  :)

Using spectrograms means you are essentially using images (of frequencies varying over time). I understand the content is in understood like the graph i.e. with axes time and frequency, but as far as the network knows, you are giving it black and white images; assuming your last dimension (=1) is the channels dimension.

You normally want to take the batch-norm over the features, so it could depend on what you see as a feature. Do you care about the shape of the peaks/troughs of the spectra, or the values encoded in the final channel? The meaning of axis when used in terms of BatchNormalization might be a little confusing. Have a look at these explanations and some of the points here.

So as far as pure images go, I would recommend keeping the default axis=-1.

Remember that a 2d convolution operation is looking for spatial correlations over the image itself - the kernel slides from left to right, top to bottom. So mixing that idea with then taking batch norm over your second axis (frequency) is literally an orthogonal idea.

I don't know if it will work out, but I think it might make most sense if your spectrograms are aligned in time, that is they cover the same nominal time period and so for two of them with the same size, the x-axis has the same scale. Otherwise you would be taking batch-normalization over different time-frames and therefore stirring up temporal information between the samples.


extract document topic vectors from lda model how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation

I've done this before in Gensim, hopefully it will help:

train_vecs = []
for i in range(len(your_training_examples)):
    top_topics = lda_train.get_document_topics(train_corpus[i], minimum_probability=0.0)
    topic_vec = [top_topics[i][1] for i in range(20)]
    train_vecs.append(topic_vec)

The above would give the top 20 topics for every document. 'train_corpus' is the result of doing something like this in Gensim once you have a bigram object from the 'Phrases' Gensim model class:

train_corpus = [id2word.doc2bow(text) for text in bigram]


Estimating and/or determining the amount of data sufficient enough to train a model To train a nice supervised algorithm (for instance, a dependency parser, a parts-of-speech tagger or NER) data is essential, but how many samples are necessary or enough? From what kind of perspectives can we try to estimate or/and determine how much data we should annotate? Or what kind of techniques can we use to predict the amount before we get the results of the first attempt?
I know that words can be separated into two types: closed class type and open class type. Should we consider that?
If such methods do not exist, how can we know if our model (mostly high variance) suffers from data shortage (knowing that data is never enough) or other reasons (like poor structure or bad training process or bad hyperparameters, etc.) after we get the results of our first version of the model and think the results are not satisfactory?

There are a few different perspectives from which to determine the amount of data you need. Those include:

Project complexity: Each parameter that your model has to consider in order to perform its task increases the amount of data that it will need for training.
Training method: As your models is forced to understand a greater number of interlinking parameters, the resulting complexity forces a change in the way that it is trained.
Labeling needs: Depending on the task you’re doing, data points can be annotated in different ways.
Diversity of input: In cases where your model’s input won’t be highly controlled, more data will be necessary to help your model function in that unpredictable environment.

You could begin working on your model with the data you have, then add more data when you feel it becomes necessary. Ideally, this will give you a very clear picture of your data needs, and they'll likely fit into one of the above categories.

This article covers training data needs in-depth, so it might be helpful. It also includes examples of dataset sizes for a variety of language-based and NLP projects. Unfortunately, there's not a lot of consistency among projects; some of the projects (text language identification) used 3000 training and 1000 test samples, while others (sentiment analysis) used datasets with over 10,000 samples. The differences here can likely be attributed to availability of data (tweet data is much easier to collect than South African language texts, for example).


TensorFlow with Phonegap I'm new into the ML Scene and I want to create a phonegap app involving Tensorflow but I'm unsure where to start or if this is even possible. Can anyone give me a hand (Probably by linking me to some resources)? My app will just use tensor flow image recognition (probably pre-trained).

Thanks, Felix.

TensorFlow is a tool to write computation using data flow graphs; this being said, if you want your app to use a pre-trained model only, there is no requirements to use TensorFlow specifically. You could even use one of the ML library written in Javascript to import and run the pre-trained model.

Javascript DL libs: http://cs.stanford.edu/people/karpathy/convnetjs/, https://github.com/dmlc/mxnet.js/
TensorFlow for mobile: https://www.tensorflow.org/mobile/
Caffe Android: https://github.com/sh1r0/caffe-android-lib
DL4J Android: https://deeplearning4j.org/android


find the difference between two columns in specific rows I have the following columns from one data frame

 0         1585742136995
 1         1585742137014
 2         1585742137035
 3         1585742137058
 4         1585742137080
          ...
 177809    1585570661653
 177810    1585570661675
 177811    1585570661686
 177812    1585570661709
 177813    1585570661731

Name: acctimestamp, Length: 177814, dtype: int64>

and

0         1585742136982
1         1585742136996
2         1585742137015
3         1585742137036
4         1585742137058
          ...
177809    1585570661632
177810    1585570661653
177811    1585570661676
177812    1585570661698
177813    1585570661710
Name: gyrtimestamp, Length: 177814, dtype: int64>

I want to start from 299 row and by 200 rows to find and print the difference between two columns (acctimestamp, gyrtimestamp)

so let's say at 299+200=499 row what is the difference between the 2 columns. After that what is the difference in 699 row And I want to do this for all the len of the data frame.

I am facing difficulties with the for loop can you help me a little bit?
how a loop like this would be?

i write this df['diff'] = df['acctimestamp'] - df['gyrtimestamp'] but it only show me the difference line by line

how to start from 499 row print the difference and print every 200 row?

i try this code:

for i in range (len(df)):
   i=200
   df1=df.loc[299+i, "acctimestamp" ]-df.loc[299+i, "gyrtimestamp"]

but the counter print the difference in line 499 and do not continue all the way to the end of the data frame

for i in range (99,len(df),200):
    try:
       df1=df.loc[i+200,'acctimestamp'] - df.loc[i+200,'gyrtimestamp']
       print(df1)
    except:
       print('End')

Is what you're looking for? Step is 200. So i will take 299 -> 499 -> 699 etc.., and you're taking only the rows between i ( 299 ) and i + 200 ( 499 ) and so on.


Cost-sensitive Logloss for XGBoost I want to use the following asymmetric cost-sensitive custom logloss objective function, which has an aversion for false negatives simply by penalizing them more, with XGBoost.

$$
\begin{array}
\\
p &= \frac{1}{1+e^{-x}}
\\
\hat{y} &= min(max(p, 10^{-7}, 1-10^{-7})
\\
FN &= y \times log(\hat{y})
\\
FP &= (1-y) \times log(1-\hat{y})
\\
Loss &= \frac{-1}{N}\sum_i 5 \times FN + FP
\end{array}
$$
I have calculated the gradient and hessian for this loss function:
$$
\begin{array}
\\
\frac{dLoss}{dx} &= 4py + p - 5y
\\
\frac{d^2Loss}{dx^2} &= (4y + 1) * p (1.0 - p)
\end{array}
$$

And my code:

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = 1.0 / (1.0 + np.exp(-y_hat))
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y)

xgb_pars = {'eta': 0.2, 'objective': 'binary:logistic',
      'max_depth': 6, 'tree_method': 'hist', 'seed': 42}

model_trn = xgb.train(xgb_pars, d_trn, 10, evals=[(d_trn, 'trn'),
            (d_val, 'vld')], obj=logistic_obj, feval=err_rate)

Running the code in verbose mode prints out the following. The two columns on the right hand side gives the error calculated by my own error calculation function passed as feval. I'm not sure why XGBoost still shows the error calculated by its own objective, but the problem is that it apparently hasn not used my updating rules, as its error decreases but my custom error starts increasing after five iterations. If I comment out the objective directive, it apparently defaults to RMSE which makes matters worse.

[0] trn-error:0.065108  vld-error:0.056749  trn-error:0.782048  vld-error:0.755389
[1] trn-error:0.064876  vld-error:0.056645  trn-error:0.727871  vld-error:0.695685
[2] trn-error:0.064487  vld-error:0.05651   trn-error:0.699920  vld-error:0.662203
[3] trn-error:0.064573  vld-error:0.056553  trn-error:0.691798  vld-error:0.64864
[4] trn-error:0.064484  vld-error:0.056514  trn-error:0.698498  vld-error:0.649974
[5] trn-error:0.064483  vld-error:0.056514  trn-error:0.716450  vld-error:0.662659
[6] trn-error:0.064470  vld-error:0.056507  trn-error:0.742848  vld-error:0.683847
[7] trn-error:0.064466  vld-error:0.056506  trn-error:0.775665  vld-error:0.71153
[8] trn-error:0.064435  vld-error:0.056497  trn-error:0.813440  vld-error:0.744165
[9] trn-error:0.064164  vld-error:0.056393  trn-error:0.854973  vld-error:0.780628

I'm answering my question. Well, I've ditched XGBoost for LightGBM (the only Microsoft product I seem to enjoy by far) but since the interface is quite similar, the same should apply to XGBoost. Apparently I don't need to apply Sigmoid to predictions. I don't know why the examples suggest otherwise.

def logistic_obj(y_hat, dtrain):
    y = dtrain.get_label()
    p = y_hat # p = 1. / (1. + np.exp(-y_hat))
    grad = p - y
    hess = p * (1. - p)
    grad = 4 * p * y + p - 5 * y
    hess = (4 * y + 1) * (p * (1.0 - p))
    return grad, hess

def err_rate(y_hat, dtrain):
    y = dtrain.get_label()
    # y_hat = 1.0 / (1.0 + np.exp(-y_hat))
    y_hat = np.clip(y_hat, 10e-7, 1-10e-7)
    loss_fn = y*np.log(y_hat)
    loss_fp = (1.0 - y)*np.log(1.0 - y_hat)
    return 'error', np.sum(-(5*loss_fn+loss_fp))/len(y), False


Correct output labelling for RNNs/ANNs with sequential data and single label I have a labelled time-series dataset; for example, a sequential input length nn from kk sensors corresponds to class '1' (see below for n=50,k=2n=50,k=2). How can I correctly classify it?

If I use RNN with infinite input length, the input layer will have 2 units (2 channels), but what about the output? Label '1' applies to the full sequence, not just a single input.

Therefore, there are two related questions: 1) how do I design an output unit? It has to reflect the input of the full sequence, 2) how do I compute an error? It also needs to account for the error of the whole sequence.

If I use ANN with a sliding window, the problem remains virtually the same - different classes can have comparable values in the window, so the output must correspond to the full sequence.

In other words, how do I match a time series to just 1 number?

EDIT:

For example, there is an $n \times k$ array labelled '1' (this can be for example readings off $n$ sensors, which match 1 class, like a gesture: 'snap'), and another $m \times j$ array labelled '0', where $n$ is the number of features/input neurons and $k \neq j$ is the length/size of the input data. If I use an RNN to feedforward the first array I do it $k$ times, obviously. But how do I label it? Do I use a unity vector length $k$, get an error each time and them average out to get the sample error and backprop? I don't think so, but not too sure.

Recurrent Neural Networks can learn very complex mappings between sequences (unlike Feed Forward Nets which are capable of transforming only fixed-sized vectors). You really only need a RNN which takes the sequence of the form $(a_t, b_t)$ as the input (assuming $t \in \{1...T\}$ you have seq. $a$ and $b$ both with $T$ samples and you have to classify these signals as one of $k$ classes). Then you can run the following procedure (pseudocode) for the sampling (given already trained model):

for t in (1..T)           //loop over an input sequence
    rnn.input[0] = a[t]   //assign t-th point of the blue seq.
    rnn.input[1] = b[t]   //assign t-th point of the green seq.
    rnn.forward()         //make forward pass
rnn.output.normalize()    //renormalize output probabilities
class_prob = rnn.output.max()
class      = rnn.output.argmax()

For example we have the following sequence with length 4:
signal A: {1, 3, 4, 6}
signal B: {4, 3, 0, 1}
Then in the 1st timestep we assign vector $(1, 4)$ to the input layer, then $(3, 3)$ and so on. At each timestep we propagate the input forward the network updating its hidden state which is responsible for remembering past datapoints. After every forward pass our neural network generates a prediction at the output layer which is temporarily uninmportant. This fluctuating prediction reflects the rnn's belief about the part of our dataset- it have seen only $t$ first input vectors. After we propagated entire signals we can do the classification.
Your RNN's output layer should be $k$-dimensional and its $i$-th component is the pseudo-probability of the signal beeing in class $i$. For example you can do argmax (take the index with greatest prediction) or sample with a softmax distribution. For example if you have class $a$ and class $b$ and your output vector is $(5, 15)$ then your signal is classified as $b$. Or if you do softmax then $softmax(5, 15) = (0.000045, 0.999955)$ so you assign the class $b$ with nearly 100% probability.
Training RNN is much more triky. I would suggest you taking some existing toolkit which have RNN such as LSTM already implemented. To train Recurrent NN we can apply the above procedure but at the end (after iterating over the sequence) we calculate Gradient of the loss function and do a backward pass. However this technique is called Backpropagation through time (it is really similar to ordinary FFN backprop, but we must "unfold" out RNN in time to adjust recurrent connections). Also notice that if you choose to have softmax output units your loss function should be a Cross-Entropy loss function.
But what is the expected output vector to compute gradient loss from? If your dataset contains $N$ (green-blue) sequences each of them is in $k_n$ class then their expected output vector is simply one-hot vector $(0...,1,...0)$ where $1$ is at $k_n$-th component only. For example if you have a sequence classified as $b$, then its output vector target is $(0,1)$. @EDIT: Suppose that as in a previous example your sequence have class $b$ so you expect to see the vector $(0,1)$. You take now the prediction of the net (say $(0.34, 0.57)$) and calculate the error function as normal f.e. squared error or cross-entropy.

More information you can learn from karpathy and hinton


What is the best way to organize the datasets for my task? I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city.

In the dataset, number of samples (or users) for each city depends on the city size. (i.e. If the city A is more populous than the city B, the city A has more users in the dataset.) This seems fair, but it creates an unbalanced dataset.

Right now, I am planing to collect another dataset; a dataset that is more balanced (i.e. there will be the almost the same amount of users for each city although there will still be more users in big cities). Doing this makes sense or should I continue with the unbalanced dataset? What approach would it be good for that task?

When you uniformly take samples from a society, definitely  chance of selecting from cities is directly related to their population. Therefore, more users will be selected from more populous cities and it's one of the most important characters of the problem you trying to solve. I think if you want to balance the data-set, you ignore one this important character of your data and also your problem.

I strongly recommend to continue with the unbalanced data-set and handle it by choosing an appropriate loss function and evaluation method.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Using a trained Model from Pickle I trained and saved a model that should predict a sons hight based on his fathers height.
I then saved the model to Pickle.
I can now load the model and want to use it but unfortunately a second variable is demanded from me (besides the height of the father) I think I did something wrong when training the model?

I will post the part of the code wher I think the error is in, please ask if you need more.

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

#testing the model with an example value
TestValue = 65.0
filename = 'Father_Son_Height_Model.pckl'
loaded_model = pickle.load(open(filename, 'rb'))
result = loaded_model.predict(TestValue)
print(result)

The error message says:

ValueError: Expected 2D array, got scalar array instead:

array=65.0.

Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Thanks alot in advance.

You need to use loaded_model.predict(TestValue), not loaded_model.score(TestValue). The latter is for evaluating the models accuracy, and you would also need to pass the true height of the son, which is the y value it's asking for.


SVM vs RVM, when to use what? I'm currently working on a project where I'm supposed to compare the efficiency of SVM vs RVM, there seems to be a lot of information to be gathered about RVM whereas I find rather old documents about RVM. It's the first time I use this forum so I hope my question isn't off-topic but my questions are:

1) Which datasets are more suited for the two models and what are the upsides/downsides for each model?
2) As mentioned above, the articles I find about SVM are rather old, I'm currently curious about recent applications of SVM and to what and where they have been implemented.

Sources or just standard responses are highly appreciated. Thanks in advance

RVM is identical to SVM, but provides a probability distribution of scores.

RVM:

Pros:

RVM is better than SVM in terms of accuracy.(depends on size of training data), more training data implies more accuracy in RVM. (argued and experimentally shown in the paper [1])
Since it provides the probability distribution of score, not a point estimate. It opens up a variety of options to get point estimate out of Bayesian inference, like percentile, mean, median, average.

Cons:

Since it uses EM algorithm for learning, it might get stuck at local minima.
Large data is required for learning

SVM:

Pros:

Standard SVM is a convex optimization problem(both hard and soft), therefore they always produce unique solution i.e. converge to global optima.

There is one very good paper about this. You can get more details here:

[1] (PDF) A comparative study of relevant vector machine and support vector machine in uncertainty analysis., accessed Jan 08 2019.


XGBOOST (sklearn interface) REGRESSION error I am trying to run a GRIDSEARCHCV (sklearn) on XGBRegressor. Documentation on the parameter says that if regression, then objective = reg:squarederror.(see https://github.com/dmlc/xgboost/tree/master/demo/regression) However, whenever I am trying to run the search, I am getting an error saying XGBoostError: b'[13:39:54] src/objective/objective.cc:23: Unknown objective function reg:squarederror.

I am not sure how to get around this problem. For the sake of completeness, below is the piece of code I am using for this purpose.

cv_params = {
             'n_estimators' : np.arange(100, 1201, 100),
             'max_depth' : np.arange(2, 10)
             }

xgbr_params = {'objective':'reg:squarederror','n_jobs':-1,'random_state':4444,'min_child_weight':1,
              'eta':0.3,'subsample':0.8,'gamma':0.5,'colsample_bytree':0.8}

opt_xgbr = GridSearchCV(xgb.XGBRegressor(**xgbr_params)
                              ,param_grid=cv_params,scoring='r2',cv=5,n_jobs=-1,return_train_score=True, verbose=3)

Any help would be greatly appreciated.

Thanks

Upgrade your xgboost version. reg:squarederror was added in 0.83 release

(In version 0.82 or below, use reg:linear)

In general, XGBoost support information is at https://discuss.xgboost.ai/


Is data partitioning necessary for an explanatory model and why? I've come accross the following paragraph in the To Explain or To Predict? paper by Galit Shmueli.

  In explanatory modeling, data partitioning is less common [than in predictive modeling] because of
  the reduction in statistical power. When used, it is usually done for
  the retrospective purpose of assessing the robustness of ˆf. A rarer
  yet important use of data partitioning in explanatory modeling is for
  strengthening model validity, by demonstrating some predictive power.
  Although one would not expect an explanatory model to be optimal in
  terms of predictive power, it should show some degree of accuracy.

I understand why data partitioning is useful in the case of a predictive model, which is to assess the generalization capacity of a model. However, in the case of an explanatory model, I don't understand why it should show some degree of accuracy in terms of predictive power since it's not the objective of the model.

Here comes my question: is data partitioning necessary for an explanatory model and why?

An explanatory model is used to identify and explain what causes some particular outcome (i.e. identify/quantify the drivers of effect).

Even though the model will not be used for prediction, it needs to be accurate enough to adequately predict the response so that you can be sure the conclusions you draw from the model are valid. As an extreme example, if your explanatory model does not predict the outcome at all, then all the inferences you make from this model are useless.

So then, how does one validate model accuracy? In most cases, data partitioning and validation on the hold-out sample is used; in your case the validation is more of a sense-check for accuracy.


General approach to extract key text from sentence (nlp) Given a sentence like:

Complimentary gym access for two for the length of stay ($12 value per person per day)

What general approach can I take to identify the word gym or gym access?

Shallow Natural Language Processing technique can be used to extract concepts from sentence.
-------------------------------------------
Shallow NLP technique steps:

Convert the sentence to lowercase

Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words)

Extract n-gram i.e., a contiguous sequence of n items from a given sequence of text (simply increasing n, model can be used to store more context)

Assign a syntactic label (noun, verb etc.)

Knowledge extraction from text through semantic/syntactic analysis approach i.e., try to retain words that hold higher weight in a sentence like Noun/Verb

-------------------------------------------
Lets examine the results of applying the above steps to your given sentence Complimentary gym access for two for the length of stay ($12 value per person per day).
1-gram Results: gym, access, length, stay, value, person, day
Summary of step 1 through 4 of shallow NLP:

1-gram          PoS_Tag   Stopword (Yes/No)?    PoS Tag Description
-------------------------------------------------------------------
Complimentary   NNP                             Proper noun, singular
gym             NN                              Noun, singular or mass
access          NN                              Noun, singular or mass
for             IN         Yes                  Preposition or subordinating conjunction
two             CD                              Cardinal number
for             IN         Yes                  Preposition or subordinating conjunction
the             DT         Yes                  Determiner
length          NN                              Noun, singular or mass
of              IN         Yes                  Preposition or subordinating conjunction
stay            NN                              Noun, singular or mass
($12            CD                              Cardinal number
value           NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
person          NN                              Noun, singular or mass
per             IN                              Preposition or subordinating conjunction
day)            NN                              Noun, singular or mass

Step 4: Retaining only the Noun/Verbs we end up with gym, access, length, stay, value, person, day

Lets increase n to store more context and remove stopwords.
2-gram Results: complimentary gym, gym access, length stay, stay value
Summary of step 1 through 4 of shallow NLP:

2-gram              Pos Tag
---------------------------
access two          NN CD
complimentary gym   NNP NN
gym access          NN NN
length stay         NN NN
per day             IN NN
per person          IN NN
person per          NN IN
stay value          NN NN
two length          CD NN
value per           NN IN

Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym, gym access, length stay, stay value

3-gram Results: complimentary gym access, length stay value, person per day
Summary of step 1 through 4 of shallow NLP:

3-gram                      Pos Tag
-------------------------------------
access two length           NN CD NN
complimentary gym access    NNP NN NN
gym access two              NN NN CD
length stay value           NN NN NN
per person per              IN NN IN
person per day              NN IN NN
stay value per              NN NN IN
two length stay             CD NN NN
value per person            NN IN NN

Step 5: Retaining only the Noun/Verb combination we end up with complimentary gym access, length stay value, person per day

Things to remember:

Refer the Penn tree bank to understand PoS tag description
Depending on your data and the business context you can decide the n value to extract n-grams from sentence
Adding domain specific stop words would increase the quality of concept/theme extraction
Deep NLP technique will give better results i.e., rather than n-gram, detect relationships within the sentences and represent/express as complex construction to retain the context. For additional info, see this

Tools:
You can consider using OpenNLP / StanfordNLP for Part of Speech tagging. Most of the programming language have supporting library for OpenNLP/StanfordNLP. You can choose the language based on your comfort. Below is the sample R code I used for PoS tagging.
Sample R code:
Sys.setenv(JAVA_HOME='C:\\Program Files\\Java\\jre7') # for 32-bit version
library(rJava)
require("openNLP")
require("NLP")

s <- paste("Complimentary gym access for two for the length of stay $12 value per person per day")

tagPOS <-  function(x, ...) {
  s <- as.String(x)
    word_token_annotator <- Maxent_Word_Token_Annotator()
    a2 <- Annotation(1L, "sentence", 1L, nchar(s))
    a2 <- annotate(s, word_token_annotator, a2)
    a3 <- annotate(s, Maxent_POS_Tag_Annotator(), a2)
    a3w <- a3[a3$type == "word"]
    POStags <- unlist(lapply(a3w$features, `[[`, "POS"))
    POStagged <- paste(sprintf("%s/%s", s[a3w], POStags), collapse = " ")
    list(POStagged = POStagged, POStags = POStags)
  }

  tagged_str <-  tagPOS(s)
  tagged_str

#$POStagged
#[1] "Complimentary/NNP gym/NN access/NN for/IN two/CD for/IN the/DT length/NN of/IN stay/NN $/$ 12/CD value/NN per/IN     person/NN per/IN day/NN"
#
#$POStags
#[1] "NNP" "NN"  "NN"  "IN"  "CD"  "IN"  "DT"  "NN"  "IN"  "NN"  "$"   "CD"
#[13] "NN"  "IN"  "NN"  "IN"  "NN"

Additional readings on Shallow & Deep NLP:

Integrating Shallow and Deep NLP for Information Extraction


IsolationForest Decision Function vs. Anomaly Prediction Question I'm currently working on an unsupervised anomaly detection project, and for it I'm using IsolationForest through scikit-learn. My question is, why/how is it possible for the model to predict something to be an anomaly when it is within the decision function space for inliers?

I've attached my results here:

Could the size of the decision function space be due to my input dimension vs this 2 dimensional projection?

I also made a quick plot of anomaly score vs. prediction (0 = inlier, 1 = anomaly):

As seen, there exists outliers above the threshold score, which doesn't make sense to me. Can someone explain?

It looks that your PCA is not mapping well the data. I recommend you to look at other dimensionality reductions techniques, such as, UMAP, TSNE, in order to see if you can achieve better representation. If you do not have too much data, you could use also MDS since it will preserve local distances (however, computationally very expensive):

https://scikit-learn.org/stable/modules/generated/sklearn.manifold.MDS.html

If only the visualization is what your are after, then you could just simply run Isolation Forest on 2 Principal components, then the boundaries should be more obvious in the first picture. Of course, the actual results may be misleading.


SVM hyperplane equations for linearly seperable data I'm going through Wikipedia article on SVM and came across these equations:

It says:

  With a normalized or standardized dataset, the parallel hyperplanes to
  our required hyperplane $wx+b=0$, can be described by the equations
  $wx+b=1$ and $wx+b=-1$

What is meant by normalized and standardized data here? If the data is not standardized, what equation should be used?

Normalized / standardized data are necessary to train an SVM classifier. Before running an SVM you should always scale your variables: it can be done either through min-max scaling or with standardization (in statistics are called Z-scores).

It's a fundamental dataprep operation, without it a classifier such as SVM wouldn't be useful. That is because if variables are not on the same scale the ones with the larger variance will dominate all the others in the training.

The equations that you quoted are the support vectors. In the image next to them: the dashed lines whose distance must be maximixed by the SVM classifier. The values of 1 and -1 are the labels that the model is attributing to the target classes (in a binary classification problem).

Hope this helps, otherwise let me know.


Visualizing decision tree with feature names from scipy.sparse import hstack
X_tr1 = hstack((X_train_cc_ohe, X_train_csc_ohe, X_train_grade_ohe,X_train_price_norm, X_train_tnppp_norm, X_train_essay_bow, X_train_pt_bow)).tocsr()
X_te1 = hstack((X_test_cc_ohe, X_test_csc_ohe, X_test_grade_ohe, X_test_price_norm, X_test_tnppp_norm, X_test_essay_bow, X_test_pt_bow)).tocsr()

X_train_cc_ohe and all are vectorized categorical data, and X_train_pt_bow is bag of words vectorized text data.
Now i applied decision tree classifier on this model, i got this.

i took max_depth as 3 just for visualization purpose.

my question is i want to get feature names in my output instead of index as X2599, X4 etc.
i know i can do it by vect.get_feature_names() as input to export_graphviz, vect is object of CountVectorizer(), since i have already merged this vectorized data using hstack, now how to get feature names in this decision tree.

hstack preserves the order of the columns, so you can piece together the feature names for each of your component arrays.  OneHotEncoder (if that's what you used) and CountVectorizer both support get_feature_names, so concatenating the lists of feature names should be possible.  To give full details would require more details about how each of the arrays was generated.  You might consider using ColumnTransformer in the future, which handles all that concatenation for you and also provides its own get_feature_names.


Partial autocorrelation and Autocorrelation difference I have been trying to understand better the PACF and ACF, but I'm literally struggling.

Have been using a series of articles like:
https://machinelearningmastery.com/gentle-introduction-autocorrelation-partial-autocorrelation/

And even looking at the method signature of statmodels for PACF and ACF I am still not able to get it.

Can somebody be so kind to make a "for dummies" explanation with some example of what is happening when data a considered?

Is it correct saying, among the other things, that

ACF is the plot of the AutoRegressive (AR) modelling whereas
PACF does the same but for the Moving Average (MA)

Or is it the other way round?

ACF is the plot of the Moving Average (MA) modelling whereas
PACF does the same but for the AutoRegressive (AR).

Good evening,

Let's begin with a simple example: you have a time series process, for example some process with correlation of up to lag 4.

An order (p) process is time series process where current value depends on previous p values.
You can think for examples of weekly sales in a supermarket, and your AR(4) model in this case shows that your current week sales depend on the previous 4 weeks of sales (on week 1, week 2, week 3, and week 4).

Now, let's get into ACF and PACF.

You have your AR(4) model, which basically tells you that points are correlated up to lag 4, meaning that your correlation between wk 1 and wk 5 is the same as between wk2 and wk 6 (note that the "distance" between them is 4 lags). Similarly, for any number of lags <4 this rule works the same way: corr wk 1 and wk 2 is same as wk 2 and wk 3 and so on. Assume that correlation at lag 1 (wk 1 and wk 2) is stronger than lag 2 (wk1 and wk 3), this means that as number of lags increases, the autocorrelation will also decrease. At each further lag the information from previous lags carries over.
Here is a visualization.

PACF is a completely different concept. What it primarily focuses on is finding out the correlation between two points at a particular lag. In your case, say you want to find the "independent" correlation between wk4 and wk3, this is exactly what PACF will show you.
Here is a visualization.

All together:
ACF data for wk4 will include all the information up to wk3 (wk1, wk2, wk3).
PACF data for wk4 will include "independent" (partial) correlations between wk3 and wk4 only (meaning it will by correlation between these points which wasn't explained by their mutual correlations.

To your point about AR and MA processes, it's the other way around. The lag at which ACF becomes very small is order of MA process. And the lag at which PACF becomes very small is order of AR process.

Hope this helps.


When do you use FunctionTransformer instead of .apply()? I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing

from sklearn.preprocessing import FunctionTransformer
logger = FunctionTransformer(np.log1p)
X_log = logger.transform(X)

In other words, she's applying a function over the rows of a column.  I assumed this could be done more simply using .apply().  I feel that there must be something more to the reason why a data analyst would import FunctionTransformer.  Could someone help me understand what differentiates the .apply() method from FunctionTransformer?

FunctionTransformer is useful because it allows you to apply a custom function in  a pipeline.  Because Pipeline() from sklearn.pipeline only works with objects that implement the .transform() and .fit() methods, you use FunctionTransformer to change your custom function to allow .transform() and/or .fit() to be used on it.

You could transform a DataFrame or Series by using .apply() (or something similar like a list comprehension), but you wouldn't be able to use that function in Pipeline() without first using Function Transformer.

(answer adapted from a DataCamp module "Multiple types of processing: FunctionTransformer" from the class "Machine Learning with the Experts: School Budgets")

Example:

# Import FunctionTransformer
from sklearn.preprocessing import FunctionTransformer

# Obtain the text data: get_text_data
get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)

# Obtain the numeric data: get_numeric_data
get_numeric_data = FunctionTransformer(lambda x: x[['numeric', 'with_missing']], validate=False)

# Fit and transform the text data: just_text_data
just_text_data = get_text_data.fit_transform(sample_df)

# Fit and transform the numeric data: just_numeric_data
just_numeric_data = get_numeric_data.fit_transform(sample_df)

# Print head to check results
print('Text Data')
print(just_text_data.head())
print('\nNumeric Data')
print(just_numeric_data.head())

<script.py> output:
    Text Data
    0
    1        foo
    2    foo bar
    3
    4    foo bar
    Name: text, dtype: object

    Numeric Data
         numeric  with_missing
    0 -10.856306      4.433240
    1   9.973454      4.310229
    2   2.829785      2.469828
    3 -15.062947      2.852981
    4  -5.786003      1.826475


Tensorflow training with batch size of (1, None, features), but model expects extra dimension I've made an autoencoder like below, to accept variable-length inputs. It works for a single sample if I do model.fit(np.expand_dims(x, axis = 0) but this won't work when passing in an entire dataset. What's the simplest approach in this case?

import numpy as np
import tensorflow.python.keras.backend as K
from tensorflow.python.keras.layers import Input, LSTM, Lambda
from tensorflow.python.keras.models import Model

def repeat(x):
    step_matrix = K.ones_like(x[0][:, :, :1])
    latent_matrix = K.expand_dims(x[1], axis = 1)
    return K.batch_dot(step_matrix, latent_matrix)

timesteps = None
features = 2
latent_dim = 10

inputs = Input(shape = (timesteps, features))
encoded = LSTM(latent_dim, name = "encoded")(inputs)
decoded = Lambda(repeat)([inputs, encoded])
outputs = LSTM(features, return_sequences = True)(decoded)
autoenc = Model(inputs = inputs, outputs = outputs)
autoenc.compile(optimizer = "adam", loss = "mse")
encoder = Model(
    inputs = autoenc.input, outputs = autoenc.get_layer("encoded").output
)

x1 = np.ones((20, 2))
x2 = np.ones((30, 2))
x3 = np.ones((40, 2))
X_train = np.array((x1, x2, x3))

autoenc.fit(x = X_train, y = X_train, epochs = 10, batch_size = 1)

I managed to solve my problem with a generator, which expands the dimensions for each single batch to return shape (1, None, 2).

class SingleBatchGenerator:
    def __init__(self, X):
        self.X = X

    def __call__(self):
        for i in range(len(self.X)):
            xi = np.expand_dims(self.X[i], axis=0)
            yield xi, xi

X = [np.ones((np.random.randint(1, 100), 2)) for _ in range(100)]
gen = SingleBatchGenerator(X)

ds = tf.data.Dataset.from_generator(
    generator = gen,
    output_types=(tf.float64, tf.float64),
    output_shapes=((1, None, 2), (1, None, 2)),
)

autoenc.fit(ds.repeat(), steps_per_epoch=len(X), epochs=500)


Behaviour of Learning Algorithms on Random Data Suppose we collect data for 100,000 tosses of a fair coin and record "Heads" or "Tails" as the value for the attribute outcome and also record the time, temprature and other irrelevant attributes.

We know that the outcome of each toss is random so there should be no way of predicting future unlabeled data instances.

My question is how do learning algorithms (support vector machines, for example) behave when we apply them on random data such as this?

They will of course still learn some best decision boundary. We know it will be meaningless, but there will still be better and best coefficients for the algorithm to learn when fitting to this particular instance of data from this random process. It may produce better than 50% accuracy on the data set, but of course this is purely due to overfitting whatever the data happens to be. It will not predict future outcomes with more than 50% accuracy.


Binary (Unary) Recommendation System with Biased Views I would like to create a content recommendation system based on binary click data that also takes views into account.

What content a user has been exposed to, and therefore has the chance to click on, is currently biased by a rule based system that is not always documented. I do have view data (if a user saw the content on their screen, regardless of whether it was clicked.), and am wondering how to take this into account with a traditional matrix factorization recommendation system such as this item-item approach, or if there are other other better options.

Any suggestions for implementation in Python are a bonus!

The simplest answer with the cleanest solution may be one that combines your binary click data and view data into another model feature that you can optimize against. What this looks like depends on your knowledge of your end-to-end system.

For instance, is it "good" that a user clicks on something with a high amount of views, or better if a user clicks on something with a low amount of views? You might have very different formulas depending on this question alone:

normalization_function((1/views) * mean(clicks)) vs perhaps normalization_function((view) * mean(clicks))

Be sure to check the assumptions of your matrix factorization implementation, but this new features may just be able to be swapped with your click data.

As a side note, I don't know exactly what you are implementing, but clicks and views generally represent different things -- as a rough guide (clicks = engagement, view = eyeballs), so it might not mean much to combine these.


Time Series Plot for floating values I have a Dataframe which looks as shown below

I am trying to make a line plot for looking at the peaks for both columns (a,b), I have gotten as far as
sns.set_style("darkgrid")
plt.plot(wr['a'][:100])
plt.show()

but the plot looks shabby,

wr.set_index(['Date_x'],inplace=True)
wr['a'][:100].plot()
wr['b'][:100].plot()

I am looking to have something like this

Any Help is Appreciated.

You can try this
wr.reset_index().plot(x='index', y=['a', 'b'], figsize=(10,5), grid=True)

The reset_index() will change the index back to column
The x = 'index' will use the date index as the x axis
The y=['a','b'] will plot both columns as two time series
figsize just tell the plot how big to be
grid=True will add a grid on the plot

You will need to give some extra 'love' to your styling to make it look exactly as your example.


What is the preferred approach for this problem? I have the Data of 10,000 users Time Session in a website/App, The Login time, logout time, the person activity,

The Data is available for 60 days ( per user )

Using this 60 days data for 10k users, can I predict the active time of the respective user on 61st day? if yes what is the best approach and please suggest which type of problems I can refer to solve this

Thank you

Convert your training data to the following format -

a = login_time
b = logout_time
c = person_activity
train_data_X = [
               [[a, b, c], [a, b, c], ......, [a, b, c]]
               [[a, b, c], [a, b, c], ......, [a, b, c]]
               .
               .
               .
               [[a, b, c], [a, b, c], ......, [a, b, c]]
             ]

train_data_Y = [
                 [[time_day_1], [time_day_2], ...... [time_day_60]]
                  .
                  .
                  .
                 [[time_day_1], [time_day_2], ...... [time_day_60]]
                ]

Explaination -

You Input(training_data_X) data has shape 10000 * 60 * 3 because you have 10K users, data for 60 days and each data point depends on 3 variables namely a, b and c.
Your Target(training_data_Y) data has shape 10000 * 60 * 1 because you have just one number to output i.e active time.

Now, this is a Regression Problem because you are trying to predict a continuous value i.e active time.

You can model this type of problem with a number of time-series models. But since you got a multivariate input, i.e [a, b, c](refer training_data_X). You would be benefitted a lot using a Seq2Seq model to generate your time-series. If just want to generate 61st time-step you could use it just for that as well.

Specific details and implementation


Why is having high p-value makes dependent variable significant in backward elimination? In backward elimination, I heard the steps of fitting the model by keep removing the highest p-value(a.k.a. insignificant independent variable) each time like below

  Select a significance level to stay in the model(e.g. SL = 0.05)
  Fit the full model with all possible predictors
  Consider the predictor with the highest P-Value(P > SL)
  Remove the predictor
  Fit model without this variable (Repeat step 3-5 until P <= SL)

But the part which I don't get is why is having higher p-value makes the corresponding independent variable insignificant. Doesn't having high p-value mean it's more close to the null hypothesis so that that variable is more significant?

Actually, the Null Hypothesis is that the predictor is not significant. Taken from the book Introduction to Statistical Learning:

Null Hypothesis: There is no relationship between X and Y
Alternative Hypothesis: There is some relationship between X and Y

If we have a high p-value, we have an expressive result showing that the null hypothesis is correct and therefore the estimated coefficient come from a normal distribution around zero and can be discarded. I hope this answer your question, please let any comment if you need more help.


XGboost - Choice made by model i am using XGboost to predict a 2 classes target variable on insurance claims. I have a model ( training with cross validation, hyper parameters tuning etc...) i run on another dataset.

My question is :

is there a way to know why a given claim has been affected to one class i.e. the features that explain the choice made by the model?

The purpose is to be able to justify the choice made by the machine to third party human.

Thanks for your answer.

You can use the ELI5 library to explain the feature contributions to individual predictions for XGBoost models.

See Explaining Predictions in the docs, copied below:

  To get a better idea of how our classifier works, let’s examine
  individual predictions with eli5.show_prediction():

from eli5 import show_prediction
show_prediction(clf, valid_xs[1], vec=vec, show_feature_values=True)


Non-categorical loss in Keras I am training a neural network (arbitrary architecture) and I have a label space that is not one-hot encoded, but continuous. The reason is that for the given problem, it is not possible to assign a single class only, it is more of a probability mapping. So in the end, my targets sum up to 1 again, but they are not 1-hot.

I wonder if I am misunderstanding Keras documentation, but for what I read, there is no Crossentropy implementation for this. There is categorical and sparse_categorical (which seem to do exactly the same, but only expect a different label format). My idea was to wrap every single target index into a binary crossentropy, but this feels wrong and I think there is a better solution. Can you please help me find an appropriate CE loss for my task?

It sounds like you want your model to output a probability distribution which matches a "ground truth" probability distribution.  Instead of a cross-entropy loss, you should try Kullback-Leiber divergence (keras.losses.kullback_leibler_divergence).
KL-divergence measures the difference between two probability distributions.  Minimizing KL-divergence should cause your predicted distribution to match the actual distribution.
By the way, KL-divergence is not just a workaround for Keras's cross-entropy limitations.  It's actually the better loss function for this task.  From the wikipedia page on KL-divergence (emphasis added by me):

$$D_{KL}(P \| Q) = H(P, Q) - H(P)$$
where $H(P, Q)$ is the cross entropy of P and Q, and $H(P)$ is the entropy of P (which is the same as the cross-entropy of P with itself).
The KL divergence $D_{KL}(P \| Q)$ can be thought of as something like a measurement of how far the distribution Q is from the distribution P.  The cross-entropy $H(P, Q)$ is itself such a measurement, but it has the defect that $H(P, P) = H(P)$ is not zero, so we subtract $H(P)$ to make $D_{KL}(P \| Q)$ agree more closely with our notion of distance.


Salesman problem with additional conditions and features I'd like to specify the kind of a problem I encountered. I need to compose the best route for a car driver who goes to different cities.
His aim to check in the most proper car park in a city taking into account arrival and departure dates and time, distance to a parking lot, traffic, a car type and cars density at a parking lot. The driver can travel through cities in random order.
I know it's an optimization problem and it also resembles the salesman and the transportation problems. But also it looks like a recommendation task. I'm confused about it. I want to use the gradient boosting or NN's to solve it, but I'm stumped by the strict mathematical statement of the problem in the mathematical language. Any clarifications, references to similar works or blogs are desired.

I couldn't tell for sure if it is applicable without seeing the exact program, but you could look into constraint programming for a solution.

Constraint programming aims to find the best solution to an optimization problem with a series of constraints, which can be hard or soft. Hard constraints rule out a possible solution (for example, no more than 10 hours drive on the same day), while soft constraints would dictate which of the remaining solution is the best, for instance awarding points for staying the night in cities with good acommodation, or having roughly the same amount of miles every day.

Constraint programming is based on backtracking, so it doesn't scale well at all for problems with many possible solutions, so it might not be for you. Worth taking a look at at least. Hope it helps!


Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization I'm new to programming with Python, and so far it's been a headache to create a build environment- and need your support and expertise in this area.

Background I'm running a MacBook, and using Sublime Text 2, and needing to learn python. I'd like to finish this tutorial on youtube on Data Science (manipulating excel files really "aggregating data on excel"- i'll post the link below and need to import pandas- but apparently I can't.

Also, worth mentioning; I'll also be using Numpy, Xlrd, Matplotlib in the future, and not sure if these modules are available on Sublime Text 2.

The Challenge:

When I run this line, I get an error.

import pandas as pd

I have researched the problem, and it seems that I don't have package control on my sublime text and found this site with the package control addition- so here's the website that I found with a long code to download the package.

https://packagecontrol.io/installation#st2

1) Is this a legit site to download this software?

2) Can I even add these modules to Sublime Text 2 to do Data Science & view excel documents

3) If I add this installation "package control", will this package allow me to download pandas, and other modules into Sublime Text 2

4) If Yes, where can I find instructions for installing the Pandas/Numpy/Xlrd modules?

https://www.youtube.com/watch?v=4_BPNnKEMn8

Sublime Text is a text editor - it is independent of your Python setup. I would suggest following a tutorial on setting up Anaconda on your machine, then follow a second tutorial on making sublime text find the Anaconda Python executable.

Here is a tutorial for setting up Python with Anaconda on a Mac.
Here is something I found with a simple search (disclaimer: I don't know anything about sublime text!)

Once you have anaconda installed, you would need to run this command in terminal to install the packages you need:

conda install numpy pandas xlrd matplotlib


How to present Market Basket Analysis Results? I am working on a Retail Company's in-store transactions for 3 months. I have performed the Market Basket Analysis on the same and I'm getting hundreds if not thousands of association rules. I am using the apriori algorithm from mlxtend.frequent_patterns import apriori in Python and I have used different support values in apriori(basket_sets, min_support=0.01, use_colnames=True), all the way from 0.01 to 0.4.

If I use a support value too high, (for some stores there are no rules found), there are very few association rules, and if I use a support value too low, its very difficult to make sense of the association rules generated since there are too many of such rules.

Since I've chosen to go with the low support value, I wanted to understand ways in which I could present the rules which would make business sense to the data owners. If there is any literature (I've tried googling at least 10 different queries but all of them return "How to do Market Basket Analysis and its applications!") on how to present the Association results, that would be really good.

Thanks

The best way to understand multiple association rules is to visualize them. This makes it even easier to present. This paper covers multiple approaches for visualizing association rules. Go through its references. They also suggest their tool, but it is in R. If you want resources for python try searching for "association rules visualization python" and you'll find some resources.


How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation? How to construct the document-topic matrix using the word-topic and topic-word matrix calculated using Latent Dirichlet Allocation?
I can not seem to find it anywhere, even not from the author of LDA, M.Blei.

Gensim and sklearn just work, but I want to know how to use the two matrices to construct the document topic-matrix (Spark MLLIB LDA only gives me the 2 matrices and not the document-topic matrix).

using the word-topic and topic-word matrix

This would be the same, wouldn't it? The model generates a words x topics and a topics x documents matrix, there is not much left to calculate, the model practically spits it out.

Using the Gibbs method gives you the counts Ntw[1..T][1..W] and Ndt[1..D][1..T], where for example Ndt[1][5] is the amount of words assigned to topic number 5 in document number 1.


Anomoly detection method selection I need to decide between SVM (One-Class Support Vector Machine) and PCA (PCA-Based Anomaly Detection) as anomaly detection methods. Azure ML is used and provides SVM and PCA as methods - hence the choice of 2 possible methods.

Does anyone have suggestions or a defined process for method selection? (Similar to cheat sheets you get for selecting a regression method).

The use case is to detect anomalies in high frequency network traffic data (from firewalls, routers & switches)?

Without putting in the time to look through Azure's documentation, my guess is that their PCA method is really just a way to do a feature reduction, then use some algorithm they have to classify. Best thing to do is try both methods and then CV and compare performances. https://gallery.azure.ai/Experiment/1219e87f8fb84e88a2e1b54256808bb3


Classify sensor data (multivariate time series) with Python's scikit-learn decision tree i'm trying to apply scikit learns decision tree on the following dataset with the goal of classifying the data:

sensordata:

multiple .csv files
every .csv file has multiple sensors (see here)
each .csv file has one label (0 or 1)

So far I've tried to train my model with Pandas Series. It worked, but the decision tree couldn't differate the features/sensors. Is pandas series the right approach for analyse data like this? Or does anyone have another solution for this problem?

For usage you need to flatten the 2D raw sensor data into 1D features. Below code demonstrates the basics.

What kind of feature engineering to apply for best predictive effect depends entirely on the nature of your sensors and problem. There are no details about this in the question or data provided.

Feature Engineering

The overall process is:

Look for patterns in the data (Exploratory Data Analysis)
Attempt to create a new feature which describes this pattern
Evaluate the new set of features using cross-validation
Analyze the samples that your classifier got wrong (Error Analysis)
Repeat from 1) until performance is good enough

Here are some things you should try:

Plot the raw sensor data from a few samples of the positive and negative class.
Plot the distributions (histogram) for each class of each raw sensor values across the entire dataset.
Try to standardize the data. For each time-series of sensor data, remove the mean and divide by the standard deviation for each sample.
Try some standard statistical summarizations on each time-series. Max, min, mean, std, skew, kurtosis. Unlikely to be better than something tailored to the patterns you see, but sometimes performs OK.

Focus first on uni-variate features per sensor. The decision-tree will be good at combining these together.

Fitting to classifier

import numpy
import pandas
from sklearn.ensemble import RandomForestClassifier

def get_sensor_data():

    timesteps = 10
    times = numpy.linspace(0.1, 1.0, timesteps)
    df = pandas.DataFrame({
        'time': times,
        'sensor1': numpy.random.random(timesteps),
        'sensor2': numpy.random.random(timesteps),
        'sensor3': numpy.random.random(timesteps),
        'sensor4': numpy.random.random(timesteps),
    })

    return df

samples = [ get_sensor_data() for _ in range(100) ]
labels = [ int(numpy.random.random() > 0.5) for _ in range(100) ]
assert len(samples) == len(labels)

print('sample from CSV file:\n', samples[0], '\nlabel', labels[0], '\n')

def to_features(data):
    # remove time column
    feature_columns = list(set(data.columns) - set(['time']))
    # TODO: do smarter feature engineering here
    sensor_values = data[feature_columns].values
    # Note: the features must be 1D for scikit-learn classifiers
    features = sensor_values.flatten()
    assert len(features.shape) == 1, features.shape
    return features

features = numpy.stack([ to_features(d) for d in samples ])

assert features.shape[0] == len(samples)
print('Features:', features.shape, '\n', features[0])

# XXX: do train/test splits etc
est = RandomForestClassifier(n_estimators=10, min_samples_leaf=0.01)
est.fit(features, labels)

Example output

sample from CSV file:
    time   sensor1   sensor2   sensor3   sensor4
0   0.1  0.820667  0.346542  0.625512  0.774050
1   0.2  0.821934  0.241652  0.485608  0.188131
2   0.3  0.264697  0.780841  0.137018  0.117096
3   0.4  0.464143  0.457126  0.972894  0.600710
4   0.5  0.530302  0.027401  0.876191  0.563788
5   0.6  0.598231  0.291814  0.588032  0.143753
6   0.7  0.627435  0.036549  0.276131  0.311099
7   0.8  0.527908  0.197046  0.580293  0.123796
8   0.9  0.068682  0.880533  0.956394  0.787993
9   1.0  0.244478  0.306716  0.586049  0.373013
label 1

Features: (100, 40)
 [0.82066682 0.62551234 0.77405    0.34654243 0.82193414 0.48560828
 0.18813108 0.24165186 0.26469686 0.1370181  0.11709553 0.78084136
 0.46414318 0.97289382 0.60070974 0.45712632 0.53030219 0.8761905
 0.5637877  0.02740072 0.59823073 0.58803188 0.14375282 0.29181434
 0.62743516 0.27613083 0.31109894 0.03654882 0.52790773 0.58029298
 0.1237963  0.19704597 0.06868206 0.95639405 0.78799333 0.88053276
 0.24447754 0.5860489  0.37301339 0.30671624]

```


Merging two different csv with python So I have one csv with 'films' in a database that have a unique ID

movie_title      | ID       | popularity
spider-man        unique ID   4.2
captain marvel    unique ID   9.9

Then about every week, I would like to update that popularity with a new one from another list of movies.

New movie list:

movie_title      | ID       | popularity
spider-man        unique ID   5.5
captain marvel    unique ID   7.4

How can I merge these two csv and output a new csv with only the value popularity updating? I have other values that also change over the week, but I just did popularity for simplicity. The unique ID must be in the new csv as well not only the changes; because, when I resubmit back to the database that is the only way that it will update with the ID.

The second list does not contain the ID's.

What you need is a pandas.DataFrame.update.

Use pandas.read_csv to read the csv file to a Pandas DataFrame, then do the update. Once you have made the update you can use pandas.DataFrame.to_csv to convert it back to a csv.


Random state in machine learning models I am confused about random_state parameter in some algorithms like AdaboostClasifier, DecisionTree and so on

Here is an example

from sklearn.model_selection import *
from sklearn.ensemble import AdaBoostClassifier
param_grid = {
    'learning_rate':[0.001, 0.10, 0.1, 1],

             'n_estimators':range(50, 400, 50)
             }

abc = AdaBoostClassifier(random_state=123)

# run grid search
grid_abc=GridSearchCV(abc, param_grid, scoring = 'accuracy')
grid_abc.fit(X_train, y_train)
#The best hyper parameters set
print("Best Hyper Parameters:\n",grid_abc.best_params_)
print("training accuracy:\n",grid_abc.best_score_)
prediction=grid_abc.best_estimator_.predict(X_test)
#importing the metrics module
from sklearn import metrics
#evaluation(Accuracy)
print("Accuracy:",metrics.accuracy_score(prediction,y_test))
#evaluation(Confusion Matrix)
print("Confusion Matrix:\n",metrics.confusion_matrix(prediction,y_test))
print("\t\tclassification report")
print("-" * 52)
print(metrics.classification_report(prediction,y_test))

The accuracy results is 0.9420289855072463

But when i change the random_state value to 0
I got another accuracy results 0.8584070796460177

How can i fix the result and be sure of the final results.It is ricky by the way Do i have to set random_state value of the train_test split as the classifier or no?

If i understand correctly you want to make sure your results will remain constant and won't change?
Or in other words, are you trying to make sure your results are reproducible?

If so, this FAQ in the scikit-learn site has exactly what you are looking for.

I'll also repeat what it says here:
An algorithm might have multiple points that introduce randomness to the process and thus introduce randomness to the result.

One method to make sure your result are constant is to set every possible random_state available in the functions that you use.
The risk in this method is that you might miss some of the places that uses a seed.

The second method, which they also describe in the FAQ, is to set a global seed once.
scikit-learn will use it for all it's random processes.
So setting a global seed like this

import numpy as np
np.random.seed(42)

will make your results constant and reproducible.


How to balance class weights correct for a CNN in Keras, given an unbalanced data set? I want to use class weights for training a CNN with a imbalanced data set.
The question arise if the sum of the weights of all examples have to stays the same?

My previous plan was to use the function compute_class_weight('balanced,np.unique(y_train),y_train) function from scikit-learn.

But I'm totally unsure if this is even suitable for the class weights of a CNN?

Thank you in advance for each tip

If the "cost" for experimenting is not really that big I suggest you take the time to experiment and take this as a learning opportunity and just try if it could actually work.

There are many approaches to address class imbalance and setting class weight is one of them and the easiest to implement.

Change loss function (for example to focal loss for binary classification with extreme imbalance)
Oversampling and Undersampling
Setting class weights
Use specific algorithm that are build to address this problem e.g. siamese network which is very useful when you only say have very few training sample of object of interest.
etc.

Specifically for your case, I can tell you the specific case that it could fail base on my experience. So basically this very likely fail when you have extreme class imbalance say like 1% positive and 99% negative. How this could fail is simply because using class weighting in this case will put very high value on the positive sample and if your model fails to detect this, the penalty is very high and hence lead to unstable training. To top it off consider a hypothetical situation your model predict the positive class correctly on epoch 10 and then it fails on epoch 11. For this case you might get a loss for example 1.3 for epoch 10 but then on epoch 11 your loss could go to say like 37.7 simply because it fail to detect said sample. This could also affect any callbacks that utilize this loss.

In summary if the situation could be as I described then don't use this otherwise just play around and find out what's best for you.


PELU activation: how does it work, and how to implement? I have encountered PELU (Parametric Exponential Linear Unit) in the literature, but I can't find practical applications of it.

Moreover, I have some questions about how it works:

Are its parameters learned, or are they hyperparameters?
Is a Keras Custom layer required for a TensorFlow implementation?
In the original paper, it was applied to convolutional models only. Could you share links to applications outside Computer Vision?
Computational burden aside, is its performance proven (outside original paper) to be superior to more classical activations from the ReLU family?

I have not used PELU activation function by myself, so dont know much about its performance benefits but can say

a, b, and c looks to be hyperparameters only.
and it can be directly implemented in tensorflow 2.0 as

tf.cond(h, lambda:c*h, lambda:a(tf.exp(h/b)-1))


Adding and subtract inbetween row inputs and value equal to the first column next row using pandas Assume I have a dataset with three inputs:

   x1   x2  x3
0  a    b   c
1  d    e   f
2  g    h   i
3  j    k   l
4  m    n   o
5  p    q   r
6  s    t   u
      :
      :

0,1,2,3 are times, x1, x2, x3 are inputs that are measured. So here x1 inputs are measured at every one hour. x2 and x3 will be measured at different time. What I need to do , I want write that what ever the measured in x1, x2, x3 it will add and subtract the values are equal to the x1 input next time value
So here what I want to do is:

    x1   x2   x3   y
 0  a    b    c   a+b-c==d
 1  d    e    f   d+e+f==g
 2  g    h    i   g+h-i==j
 3  j    k    l   j+k-l==m
 4  m    n    o   m+n-o==p
 5  p    q    r   p+q-r==s
 6  s    t    u   s+t-u==v
         :
         :

I tried shift method and it didn't work for me what I want exactly. I tried another method and it worked, but didn't came as I want. Here I upload the code.

Code :
 data = pd.read_csv('data6.csv')
 i=0
 j=1
 while j < len(data):
   j=data['x1'][i] - data['x2'][i] + data['x3'][i]
   i+=1
   j!=i
  print(j)

This is works , but it is just showing only one data

  63

In my csv file this is second input value of x1 input.
I want to write this code contonously happened and read the value as I shown above.
Can anyone help me to solve this problem?

My csv file

So are you evaluating the equality of a+b-c and d for each sample? Meaning, you aim to save the boolean value? If so:

import pandas as pd
#a sample dictionary
data = {'x1':[1,0,4,5,8,1],
     'x2':[3,4,5,6,8,9],
     'x3':[4,5,1,-2,4,5]}
df = pd.DataFrame(data) #converting the dictionary to dataframe
df['y'] = 0 #initializing the y column with 0, also you can initialize with FALSE boolean value
for i in range(len(df)-1): #iterating between all the rows of dataframe
    df['y'].iloc[i] = df['x1'].iloc[i] + df['x2'].iloc[i] - df['x3'].iloc[i] == df['x1'].iloc[i+1]

Note: for the last row, since the content of column y should be calculated based on the next row, the value cannot be calculated, that is why we have set (len(df)-1).

Result:

   x1  x2  x3      y
0   1   3   4   True
1   0   4   5  False
2   4   5   1  False
3   5   6  -2  False
4   8   8   4  False
5   1   9   5      0

The problem with your code:
First, you are not considering that subsequent row to calculate the accuracy of the expression $a+b-c ==d$
Second, if that print(j) is outside the loop, is it? then you are looping in the rows of data frame while j is less than data frame's length, So when the flow is out of this while, a single value j would be returned.


When to use k-medoids over k-means and vice versa? I had someone ask me about k-medoids at work and don't know about the performance of this algorithm over other clustering algorithms (namely k-means as it is most similar to it). In this case, it was recommended for use on taxonomic data (ie bacterial/viral species/strains), but I do not know why this is better.

time complexity of k medoids is  $O(k * (n-k)^2)$.

is time complexity of a comparable k-means algorithm the same?
When would you use one or the other?
What qualities does one look for to use k-medoids?
What are the differences in the output?

1) Time complexity of KMEANS

As explained in this post :

  KMeans is an NP-hard problem. However, running a fixed number $t$ of iterations of the standard algorithm takes only $O(t*k*n*d)$, for $n$ (d-dimensional) points, where $k$ is the number of centroids (or clusters). This what practical implementations do (often with random restarts between the iterations).

2) When would you use one over the other?

As mentioned in this Wikipedia article, K-medoids is less sensitive to outliers and noise because of the function it minimizes.

  It is more robust to noise and outliers as compared to k-means because it minimizes a sum of pairwise dissimilarities instead of a sum of squared Euclidean distances.

Also, K-medoids can use a variety of similarity measures where K-means is limited to euclidian (pairwise) distance. Excellent explanation [here].(https://stats.stackexchange.com/a/81496/279276)

3) What qualities does one look for to use k-medoids?

I would recommend using it whenever Euclidean Distance does not make sense in your data. If Euclidean Distance does not make sense (i.e. unrelated categorical variables : "has wings", "# of legs"), minimizing the sum of squared Euclidian distances probably won't either.

4) What are the differences in the output

The main difference is that medoids (equivalent to centroïds in K-Means) belong to the data points. You will never have a medoid that is somewhere between points. Instead, it will be superimposed on an existing point. This post shows it clearly.

It makes sense, especially for a categorical feature (# of legs), not to have a cluster center at 3.347 legs.

Hope this helps.


How to (better) discretize continuous data in decision trees? Standard decision tree algorithms, such as ID3 and C4.5, have a brute force approach for choosing the cut point in a continuous feature. Every single value is tested as a possible cut point. (By tested I mean that e.g. the Information gain is calculated at every possible value.)

With many continuous features and a lot of data (hence many values for each feature) this apporach seems very inefficient!

I'm assuming finding a better way to do this is a hot topic in Machine Learning. In fact my Google Scholar search revealed some alternative approaches. Such as discretizing with k-means. Then there seem to be a lot of papers that tackle specific problems in specific domains.

But is there a recent review paper, blog post or book that gives an overview on common apporaches for discretization? I couldn't find one...

Or else, maybe one of you is an expert on the topic and willing to write up a small overview. That would be tremendously helpful!

No, you probably don't want to try all possible cut points in a serious implementation.  That's how we describe it in simple introductions to ID3, because it's easier to understand, but it's typically not how it is actually implemented, because it is slow.  In particular, if there are $n$ data points, then you'll need to test $n-1$ candidate thresholds; using the naive algorithm to calculate the information gain of each of those candidate thresholds takes $O(n)$ time per candidate, for a total of $O(n^2)$ time.

In practice, there are optimizations that speed this up significantly:

Don't try all possible thresholds.  Instead, pick a random sample of 1000 candidate thresholds (chosen uniformly at random out of the set of $n-1$ candidate thresholds), calculate the information gain for each, and choose the best one.
Use dynamic programming to efficiently compute the information gain of all $n-1$ splits, in total of $O(n)$ time, by reusing computation.  The algorithm is pretty straightforward to derive.


Statistical analysis vs Data mining Why the A/B Testing, the Correlation and the Regression are considered methods of Statistical Analysis, while the method of "Association Rules" belong to the Data Mining field ?

Which are the difference between Statistical analysis and Data mining

The key difference is that statistical analysis reports on different aspects on your data, while data mining mines for a new knowledge about it, discovering some information sometimes not expected in advance.


Learning signal encoding I have a large number of samples which represent Manchester encoded bit streams as audio signals. The frequency at which they are encoded is the primary frequency component when it is high, and there is a consistent amount of white noise in the background.

I have manually decoded these streams, but I was wondering if I could use some sort of machine learning technique to learn the encoding schemes. This would save a great deal of time manually recognizing these schemes. The difficulty is that different signals are encoded differently.

Is it possible to build a model which can learn to decode more than one encoding scheme? How robust would such a model be, and what sort of techniques would I want to employ? Independent Component Analysis (ICA) seems like could be useful for isolating the frequency I care about, but how would I learn the encoding scheme?

I suggest the use of Hidden Markov Models, with two possible states: (1) high levels and (0) low levels.

This technique might be helpful to decode your signal. Probably you would need a specific HMM for each codification.

If noise is an issue an FIR filter with a Blackman-Harris window function would allow you to isolate the frequency you're concerned with.


Binary classification to predict various targets I am using keras neural networks for a binary classification task. I have a large dataset with 19 columns. Each of these columns is of binary type, and therefore the entire dataset is just 1s and 0s, like so (obviously this is not the entire dataset, which has 19 columns and many more rows):

I want to make 19 predictions, one for each column, where the other 18 are used as independent variables. So essentially, first I'll make col 1 the target, using the other 18 columns to predict it. Then I'll make col 2 the target and use the other 18 to predict it etc etc...until column 19. Is there any way to do this effectively without making 19 separate models for each column? Thanks!

If you create 19 models, each model will learn weights corresponding to predictors and target variable values.

For example- When you create one model by taking col1 as target, the model will learn weights for giving output col1. When you take another column, lets say col2 as target, if you use the previous model, it won't identify the difference between col1 and col2 (i.e., target) as your data is binary. The model will continue to learn weights for each column, but it won't be the correct one as it will be result of learning all target columns.

You could have used multi-label classification if you wanted to predict different targets at same time using same predictors for each target. Since this is not the case, the only way for getting 19 predictions is to create 19 different models, one for each column as target.


Cross between an edit distance algorithm and a phonetic algorithm My aim is to find an edit distance algorithm which penalises transformations differently depending on the phonetic context.
Take the Levenshtein algorithm, for example; it penalises the same operation - like a character substitution - equally, regardless of character context.
However, it seems to me that all substitutions/insertions/deletions are not made equal. For example, in some contexts, the transformation C->K should not be as harshly penalised as A->T.
A phonetic algorithm, like Double Metaphone, can help here. It maps both kosmetik and cosmetic to the same phonetic key, but it's aggressive and discards information. There's a clear difference between these two strings even though the phonetic key is the same.
Simply taking the Metaphone key and applying an edit distance algorithm is a poor choice for my goal.
Does a suitable method exist here? Or is the best option to manually fiddle with the internals of an edit distance algorithm?

The operations in the edit distance can have different weights that you can try to set manually. In Python, you can use e.g., strsympy or weighted-levenshtein package.
There is also Learnable Edit Distance algorithm that allows learning the operation weights from data. I think it might be a good idea first to estimate the operation cost first manually and then fine-tune it with the learning algorithm.  Unfortunately, I am not aware of any reasonable implementation.
There is also a CRF-based classifier that is based on the learnable edit distance. Based on training data, it estimates the probability of one string being a transcription of another one. This approach requires positive and negative training examples.