Twitter Sentiment Analysis: Detecting neutral tweets despite training on only Positive and Negative Classes I am a newbie when it comes to machine learning. I am trying to get hands on experience by analyzing different supervised learning algorithms using scikit-learn library of python. I am using the sentiment140 dataset of 1.6 million tweets for sentiment analysis using various of these algorithms.

I don't know if it is a stupid question, but I was wondering whether if it'd be possible to classify into three classes (positive, negative and neutral) when you've only trained over two classes (positive and negative). The sentiment140 training set consists of only two classes (positive and negative) of 1.6M tweets in total but their test set consists of 500 tweets over three classes (positive, negative and neutral), so it got me thinking.

So is this possible? If yes, how do I proceed to identify the neutral tweets?
Intuitively, I can use to find the conditional probability of each classified tweet in the test set over each class (using predict_proba) and tell if it's neutral if it is below a certain threshold (say less than 0.7) for both the positive and negative classes. Is this the right way to go?

The quick (and not very satisfying) answer is "it depends" -- specifically it depends upon what your underlying conceptual model of human emotion is and how it manifests in verbal/written behavior.

What is your characterization of neutrality in relation to positive and negative valence? Can documents be put on some sort of quantitative scale with neutral sandwiched between positive and negative? This position has linguistic support at least with simple phrases which express single valence states (the solution is {great > good > acceptable > poor > horrible}).

As @dmb poster suggested, if this is your conceptual model, then you might reasonablly argue that neutral falls between pos/neg and all you have to do is determine the optimal boundaries/cutoff. Thus, you can assign neutral to a test case even if the classifier wasn't trained on neutral cases (though you do need some way of determining these cutoffs).

But what about more complex cases? What happens when you move from the level of the phrasal unit to larger sentential and discourse level units? How would you rate the sentence "I love apples but hate bananas"? Do the positive and negative elements cancel each other out to create a neutral? You can easily see how muddied this gets when we start to talk about real human texts. In my opinion, I don't think it is particularly meaningful to talk about a document-level emotion score. Rather, I believe people express emotion/sentiment that is directed at an individual objects "I hate bananas" -> hate(subj, obj) that get combined into discourse-level constructions.

So, yes, I think you could reasonable defend the use of outputting a neutral category if only pos/neg are used to train... BUT you will need to justify your use of particular cutoffs as well as consider how to deal with longer documents that express multiple (conflicting) emotions. My preferred course of action would be to have people read and label tweets as pos, neg, neutral, and mixed (then ensure inter-coder reliability) to create my training data... let the algorithm do the hard work of finding the cutoff values.


alternatives to pyspark to query a csv file locally (no distributed computation needed) I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as matplotlib to plot data. I know there are some workarounds to do it with pyspark but I found it very complicated.

My question is: is there any alternative to pyspark, possibly python-native, that allows to query a csv file allowing to use python libraries?

pandas can do it. It can read in a csv file with a single line and put the data in a dataframe that you can manipulate.


Can TF turn a given graph into a recursive one? I have a complex graph that takes an input $x_t$, it does a bunch of convolutions, tensor shape manipulations etc... until it produces $y_t$.
I want now to try a different approach, where I feed $y_t$ at some point into the graph (I concatenate it to some features after a convolution), while the net is processing $x_{t+1}$.

So the net should do the following ($y_0$ is a fixed initial input):

$(x_0,y_0)\rightarrow y_1$

$(x_1,y_1)\rightarrow y_2$

etc...

I have been looking at the standard RNN implementations in TensorFlow (GRU, LSTM, etc), but I don't quite understand how I can do this, because it seems that LSTM and the other RNN units have a fixed graph (like, a fully connected layer) and I cannot use the one that I need instead.

Do I need to subclass tf.nn.rnn_cell.RNNCell? That seems problematic too because it seems to admit only 2D tensors as input.

EDIT: I just discovered this awesome article and I'm going to document myself about tf.scan.

I finally managed to find a solution by using tf.scan (see this question on stackexchange)

The trick is to create a model object that calls tf.scan in one of its methods using a step function that operates on the elements of a list of inputs.

The function tf.scan takes care of feeding the output of the step function back into itself for the next iteration, and it is quite flexible so you might be able to do what you need without the need to implement a full RNN from scratch (say, with tf.nn.raw_rnn).

Below, the minimal working example that I had posted over there:

import tensorflow as tf
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2' # to avoid TF suggesting SSE4.2, AVX etc...

class Model():
    def __init__(self):
        self._inputs = tf.placeholder(shape=[None], dtype=tf.float32)
        self._predictions = self._compute_predictions()

    def _step(self, old_state, new_input):
        # ---- In here I will write a much more complex graph ----
        return old_state + new_input

    def _compute_predictions(self):
        return tf.scan(self._step, self._inputs, initializer = tf.Variable(0.0))

    @property
    def predictions(self):
        return self._predictions

    @property
    def inputs(self):
        return self._inputs

def test(sess, model):
    sess.run(tf.global_variables_initializer())
    print(sess.run(model.predictions, {model.inputs: [1.0, 2.0, 3.0, 4.0]}))

test(tf.Session(), Model())


Ensemble Model vs Normal model If I get 95+ % accuracy in normal models, should I still consider Ensemble models? Why should I choose Ensemble models over normal models?

It is a problem specific and data specific problem. For one problem, 95% accuracy may be good, for some other it might not be very good and there might still be room for improvements. Likewise, it might happen that the problem is well defined, your architecture is good but due to a lot of noise in the data, you can't do better than 95%. In that case, we will say the results are good.

Ensembles: Ensembling is a general term for combining many classifiers by averaging or voting. It is a form of meta-learning in that it focuses on how to merge results of arbitrary underlying classifiers. Generally, ensembling improves the final results but again, not necessarily. If you are ensembling two highly correlated models, your ensembled model will give almost the same result as your stand-alone models.


What are some easy to learn machine-learning applications? Being new to machine-learning in general, I'd like to start playing around and see what the possibilities are.

I'm curious as to what applications you might recommend that would offer the fastest time from installation to producing a meaningful result.

Also, any recommendations for good getting-started materials on the subject of machine-learning in general would be appreciated.

I would recommend to start with some MOOC on machine learning. For example Andrew Ng's course at coursera.

You should also take a look at Orange application. It has a graphical interface and probably it is easier to understand some ML techniques using it.


Keras weird loss and metrics during train I am doing some testing with tensorflow, and I bumbed into a very weird behaviour.
Here is my code
fashion_mnist = tf.keras.datasets.fashion_mnist
(train_images1, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images1[:32] / 255.0
train_labels = train_labels[:32]
test_images = test_images / 255.0

batch_size = 32
epochs     = 1

train_data = tf.data.Dataset.from_tensor_slices((train_images, train_labels)).batch(batch_size)

adam = tf.keras.optimizers.Adam(lr=0.002, beta_1=0.9, beta_2=0.999)

input_layer   = tf.keras.layers.Input(shape=(28,28,))
flatter       = tf.keras.layers.Flatten()(input_layer)
dense1        = tf.keras.layers.Dense(128,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01),
                                     activation='relu')(flatter)
dense2        = tf.keras.layers.Dense(64,
                                     kernel_regularizer=tf.keras.regularizers.l2(0.01),
                                     activation='relu')(dense1)
output_layer = tf.keras.layers.Dense(10,
                                     activation='softmax',name='output')(dense2)
model_naive = tf.keras.models.Model(inputs=input_layer,outputs=output_layer)

model_naive.compile(optimizer=adam,
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
model_naive.summary()
history = model_naive.fit(x=train_data, validation_data=train_data, epochs=epochs)
model_naive.evaluate(train_data)

I simply import fashon_mnist (only one batch) and pass it as both train and validation data, to make comparison. No dropout in the network... so I would expect to find same loss and metric but this is the output
1/1 [==============================] - 1s 815ms/step - loss: 5.4532 - accuracy: 0.0312 - val_loss: 5.0106 - val_accuracy: 0.3125

To be sure I even did a model.evaluate() and this is what I find
1/1 [==============================] - 0s 12ms/step - loss: 5.0106 - accuracy: 0.3125

exactly the same found during training.
So, provided the evaluation is correct...what are these numbers "loss: 5.4532 - accuracy: 0.0312" ? I am using only one batch, so I would expect no averages over batches are involved.
Please help me to understand, this is driving me crazy. Thank you!
EDIT:
With only 1 batch keras weirdly seems to print loss and score before applying gradients. The same does not happen with more than one batch where probably some strange average is performed.
Still not solved the issue, any comment is still very welcomed!

This is because the calculation of the loss and accuracy are done before the first weight update (i.e. the model with the initialized parameters). After the loss is calculated the first time the loss is used to backpropagate the error throughout the network and to update the parameters. The loss is not calculated again during training (since this would just be an added training cost). The adjusted parameters are then used to calculate the loss for the next batch (training or validation). It would therefore make more sense to compare the training statistics of the second batch with the validation statistics of the first batch.


How to do this complicated data extrapolation, prediction modeling? I have some very complicated data about some movie sales online, first for each data entry, I have a key which is a combination of five keys, which are territory, day, etc, and then, for each key I have the sales for a period of time, and other information, like the movie's box office and genre.
For each day, there is a delay for the data loading to the database, around ten hours, I try to fill the gap, do some data extrapolations.
For each movie we sell, there is some decay of selling since the new release of the movie, i.e. usually for each movie, it follows some sales decay pattern.
For a recent day, I pulled some data, and I found that some decay pattern:

And for that day, the sales for each key can range from around $150000 to $0. The pic is as follow:

In the picture, the 15000 means there are around 15000 keys for each day.
Found this article.
I am trying to predict for each key, the sales amount, like for a movie, territory, day etc combination, the sales amount, how much dollars, means for that movie, that territory, that day, how much money we get from selling online. I tried ARIMA time series model, but there is some concerns for that model, seen from the pics, there is some seasonal thing, and decay thing for the movie, so the sales prediction can not be always flat, there may be a pump after a going down, it may happens on a weekend, since there is seasonal thing, and the decay trend, etc, how to capture these things. Thank you for your reply!
I am not sure whether can be applied, and how to be applied here.
Thanks a lot in advance.

I am not sure I understood the problem,however if you are trying to predict sales amount my guess is ARIMA might not be the right choice as it will not consider external variables.My suggestion is to gather related features such as how did the other movie from same genre did at that time of the year in that region,weather,star presence,presence of other popular matches or tournament at that time and etc.Let me know your thoughts.


acf function shows error while fiiting time series I am fitting a time series model in R. I plotted the time series to check for the pattern of it. It has 30625 observations. The observations are half hourly

plot.ts(ts)

I was checking acf plot to see for auto correlation.

acf(ts)

If ACF plot decreases slowly i.e. it has auto correlation out to higher number of lags, then it probably needs higher order of differencing.

I took the logarithm of the series and then checked its acf plot.

acf(log(ts))

I am getting an error

Error in plot.window(...) : need finite 'ylim' values
In addition: Warning messages:
1: In min(x) : no non-missing arguments to min; returning Inf
2: In max(x) : no non-missing arguments to max; returning -Inf

What does this error means? What should I do remove this error.
I also think the significance bounds given by dotted line are not correctly calculated. What should I do in these two cases?

Logarithm cannot handle numbers less than and equal to zero. You can exclude them from your time-series and then add them to the graph as points with different colors (if they are not many that would make the graph ambiguous and interpretation of it hard) or just simply exclude them but have the original graph side by side as well.

EDIT:

As proposed in the comments this would be a better option to avoid zeros:

log(x+1)

or for negative numbers as well:

log(|x|+1)


Is there a python version of Mitchell's face pose recognition code? The original code for chapter 4 in Tom Mitchell's book for recognizing face pose from images is here:  http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/faces.html

But it's in C, not that there is anything wrong with that and I guess that's a really good project to try to port it to Python but I am looking for something I can git clone quickly and use for a personal project.

I have created a Notebook (ipynb) file with an implementation of this algorithm.  It is available on Github


way to add target delay on TimeseriesGenerator from keras.preprocessing.sequence from deep learning with python book, it created function for data generator.

I thought I can do the same using TimeseiresGenerator from keras package but I was not able to add target delay. Is there any way to do it?

EDIT: I found a way by using np.roll on target data. But if there is a better way please let me know.

You can use

  pandas shift

By default the TimeseriesGenerator performs a lag of 1 timestep. Then, a workaraound is to create a pandas Series, and use the shift function.
Let train be your sequence, and let forecast be an int representing the "size" of your lag.

Eg:

label_train = pd.Series(train)
train_generator = TimeseriesGenerator(train, label_train.shift(-forecast+1).values)

Obs: Be careful to pad your sequence after performing the lag.


Balanced Linear SVM wins every class except One vs All I am training a normal and a balanced Linear SVM using imbalance data, and testing both using F1-score.

(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it "oversamples" from the minority class and "undersamples" from the majority class.)

Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:

                   normal vs balanced
target 01 - scores: 0.272 vs 0.608
target 02 - scores: 0.391 vs 0.587
target 03 - scores: 0.433 vs 0.546
target 04 - scores: 0.659 vs 0.655
target 05 - scores: 0.000 vs 0.257
target 06 - scores: 0.431 vs 0.475
target 07 - scores: 0.000 vs 0.249
target 08 - scores: 0.000 vs 0.053
target 09 - scores: 0.576 vs 0.155
target 10 - scores: 0.000 vs 0.550
OneVsRest - scores: 0.565 vs 0.540

But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.

Here is the code I have used:

# -*- coding: utf-8 -*-

from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.metrics import f1_score
import numpy as np

d = np.loadtxt('yeast.csv', delimiter=',')
X = d[:, 0:-1]
y = d[:, -1]

print '                     none vs balanced'
for target in np.unique(y):
    scores = [0] * 2
    k = 6
    for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
        for i, w in enumerate([None, 'balanced']):
            _y = (y == target).astype(int)
            yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts])
            scores[i] += f1_score(_y[ts], yp) / k
    print 'target %02d - scores: %.3f vs %.3f' % (target, scores[0], scores[1])

scores = [0] * 2
for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
    for i, w in enumerate([None, 'balanced']):
        m = OneVsRestClassifier(LinearSVC(class_weight=w))
        yp = m.fit(X[tr], y[tr]).predict(X[ts])
        scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / k
print 'OneVsRest - scores: %.3f vs %.3f' % (scores[0], scores[1])

As the dataset, I have used here Yeast (UCI), for which a sklearn-ready version can be found here.

Results:

                     none vs balanced
target 01 - scores: 0.241 vs 0.612
target 02 - scores: 0.402 vs 0.604
target 03 - scores: 0.444 vs 0.552
target 04 - scores: 0.666 vs 0.650
target 05 - scores: 0.000 vs 0.286
target 06 - scores: 0.370 vs 0.500
target 07 - scores: 0.000 vs 0.280
target 08 - scores: 0.000 vs 0.082
target 09 - scores: 0.563 vs 0.147
target 10 - scores: 0.000 vs 0.511
OneVsRest - scores: 0.567 vs 0.543

Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?

EDIT: I think the fact that one model wins pretty much every "battle", but not the "war" is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] > score[k2] we can assume it prefers k1 > k2. See wikipedia.

I think the following synthetic example explains what is going on:

from sklearn.base import clone
from sklearn.svm import SVC
import numpy as np
from sklearn.datasets import make_blobs

X, y = make_blobs(1000, centers=[[1, 1], [-1, -1], [1, -1]], cluster_std=0.8)

models = [
    ('normal', SVC()),
    ('balanced', SVC(class_weight='balanced')),
]

import matplotlib.pyplot as plot
plot.ioff()

for i, (name, m) in enumerate(models):
    for k in np.unique(y):
        m = clone(m).fit(X, (y == k).astype(int))

        xx, yy = np.meshgrid(np.arange(-3, 3, 0.1), np.arange(-3, 3, 0.1))
        z = m.predict(np.c_[xx.ravel(), yy.ravel()])
        z = z.reshape(xx.shape)
        colors = ['blue', 'green', 'red']
        plot.contour(xx, yy, z, colors=colors[k])
    plot.scatter(X[:, 0], X[:, 1], c=y)
    plot.title(name)
    plot.show()

The balanced SVM is clearly superior when considering each model individually, at least as measured by the F1 score, which punishes false positives and false negatives equally.

But One-vs-Rest uses maximum score, which in this case is given by the distance to the decision hyperplane. It is not yet clear why this is worse in some cases, but clearly it is possible for one model to beat another for individual classes, but to lose for One-vs-Rest: how well the relative distances are modeled is what counts in One-vs-Rest.


Does the choice of normalization change dramatically the result of a KMeans I'm using a KMeans to get the profile of several users according to several columns (I'm working with RStudio).

To analyze my clusters, I decided to realize a radar chart, so I decided to use feature scaling : x-min(x)/diff(range(x)), to have my values in [0,1] (to get a quite good idea of my data per cluster).
However, since there are multiple choice for normalization, I was wondering if doing my analysis with another choice for normalization - for instance : x-mean(x)/sd(x) - would give me the same results (in a general way at least)

Or am I completly wrong for considering my scaled data and should I use my unscaled data in my radar chart ?

Yes. Feature scaling can completely change the clustering result.

People usually scale data to [0:1] or to have a standard deviation of 1.

However, that is nothing but a heuristic.

In many cases, the need for scaling is nothing but a symptom, caused by inappropriate data for the method. You can't just fix this by some naive scaling, but it's just a hack that often works.

For statistically meaningful results, all axes should be scaled to reflect attribute relevance, such that a difference of 1 unit is of the same importance in each attribute.


Data augmentation based on the class type in the CNN model I would like to use CNN model to classify images but some classes in my dataset have low amount of data.

Can I apply data augmentation based on the number of the images in the class?

For example, the classes that contain 10 images will be 50 after augmentation that means the number of images is increased 500%, and the classes that contain 20 images will be increased 250% to be 50 images and the classes that contains 30 images will be increased 166% to be 50 images.

So, I would like to increase the data based on the number of images in the class.

Try under/ or oversampling and aditionally use data augmentation for your input data.


How to get a count the number of observations for each year with a Pandas datetime column? I'm working on a column that I converted from a object to a datetime datatype in Pandas. I'm trying to get a count of how many of observations are there for each year.

This is the column: df_raw['filed_date'] and the output is:

198894   2017-12-05
198895   2017-12-05
198896   2017-12-05
198897   2017-12-06
198898   2017-12-06
198899   2017-12-07
Name: filed_date, Length: 198900, dtype: datetime64[ns]

I couldn't find a method to just access the date and then daisy chain it to the value_counts() method. Any advice?

from collecitons import Counter
Counter([date.year for date in df_raw['filed_date'].values])


Playing cards object detection problem I'm trying to train an object detection model to detect some of playing cards for example it will detect 1,2,3, king, queen and jack.
And I'm making a class of not and I have put examples of other cards such as 4,5,6,7,8,9 but for me it doesn't make sense to make this additional class.
So how to solve the problem of detecting other kind of cards as an objects should I make this class or remove it and keep only the required classes?

In your case you have 3 options for training your model:

Make a class for each card number, so 13 classes.
Make a class for 1, 2, 3, king, queen, jack and one bucket class for "other", so 7 classes.
Make a class for 1, 2, 3, king, queen, and jack, so 6 classes. And whenever the model is not confident of any, assume it is "other".

Best would be option 1.
Option 1 would be the more difficult to get data for because you need labelled data for more classes, but that should not be a problem because it seems you already have it.
Option 1 allows your model to classify all classes, and let's you achieve the same as option 2 with post processing (i.e. if predicted_class in [4,5,6,7,8,9] then "other").
Option 2 could work, but the numbers in the "other" class are just as different from each other as to 1, 2 and 3. To be able to make this distinction, the first intuition is that a "larger" model would be needed in comparison to option 1. This approach would be handy if the "other" class is not similar to the other classes. For example, a model that classifies sports cars, family cars, clown cars and "other vehicles" (i.e. tractors, bikes and trucks).
Option 3 is best left for tasks that the "other" class is not defined, or too large to model. For example, a model that classifies if the playing cards are Uno, or Poker, or Blitz, or "any other game" card. In this case, there are too many "other" types of game cards for you to label and model.


How to save dynamic files from colab to Google Drive? I have processed some png images on Colab and saved them in the Colab Directory. Now I want to save these images from Colab to Google Drive. How can I do this?

Just mount your drive into Colab and create the checkpoint in the mount point location. Let me try to explain this with an example:

Mount the drive as follows:

from google.colab import drive
drive.mount('/content/gdrive')

This will mount your drive and your google drive files should be all under /content/gdrive/My Drive/ location

Now use your location as you would use any other location in bash or create the check-point directory as follows:
You can use the ! to run bash commands in notebooks as follows:

!cp -r <PATH_TO_PNG_FOLDER> /content/gdrive/My Drive/


How to calculate LFW accuracy of face recognition model? In my research I have observed many of the face recognition algorithms propose their model accuracy interms of LFW dataset accuracy. I understood that LFW is a open source database and I did download that.

I have developed my own DNN model implemented for face recognition which is similar to facenet architecture. Can anyone help with the procedure to calculate LFW accuracy of my model?

FaceNet should embed pictures in a high dimensional space in such a way that similar pictures are close (wrt to a given distance metric) to each other and different images are farther apart.

That means, to calculate the accuracy of your model you need to find your threshold distance d which delivers the highest accuracy. Given such a distance, you can calculate the accuracy like you would do for a binary classifier. That is, the accuracy is the "Number of correctly classified pairs/Number of pairs"

For example, consider the three images A1,A2 of person A and B1 of person B. And say your DNN embedds the images such that the pair-wise L2 distance (i.e., the euclidean distance) is 1,3, and 7. Then you can see that if use a distance threshold of 2 you get 100% accuracy, and otherwise only 66%.

+----------+--------------+-------------+------+------+
|   Pair   | Ground Truth | L2-distance | d=2  | d=4  |
+----------+--------------+-------------+------+------+
| A1 A2    |            1 |           1 |    1 |    1 |
| A1 B1    |            0 |           3 |    0 |    1 |
| A2 B1    |            0 |           7 |    0 |    0 |
+----------+--------------+-------------+------+------+
| Accuracy |              |             | 1.00 | 0.66 |
+----------+--------------+-------------+------+------+

If you are using Tensorflow, you can follow David Sandberg's instructions to caculate evaluation metrics (including Accuracy) of your model here.

IMPORTANT: To compare the performance of your approach to other approaches, you should follow one of the protocols of the LFW dataset. The protocols are described in detail here (see Section 4). Basically, they have a special view on the dataset, which asks you to report the mean accuracy + error rates on 10 folds of that dataset.


What are the drawbacks of traditional clustering algorithms? As a newbie, I am interested what the major drawbacks of traditional clustering algorithms are. The purpose is to understand for which class of data or in which scenarios I shouldn't use traditional clustering methods (e.g. K-means)

Check out this great answer for K-means in particular:
How to understand the drawbacks of K-means
My favorite resource for comparing the strengths and weaknesses of clustering algos is from sklearn's docs:
Comparing different clustering algorithms on toy datasets


Sort dataframe by date column stored as string I have dataframe named df1.

I want to sort data frame month column according to month (Jan, Feb, March..).

For that I used code:

sorted_df = df1.sort_values(by='month')
print(sorted_df)

but the output is sorted by alphabetic order of month column.

I think the reason for that in this month column data type is object so month column is sorted by alphabetic order.

The question is: how sort values in column month in correct order (according to order of months in year)?

My dataframe:

Your month column is really a partial date (i.e. the year is missing), but you could still just convert that column to a proper date format and then sort.
You will have to specify a format, so the parser knows roughly what to do.

df["date"] = pd.to_datetime(["20-Apr"], format="%d-%b")
df.sort_values("date")   # ascending by default

Without any year information, it will add the year as 1900.

If you want to add e.g. this year, you could do the following, adding 12*20 months to the new date column:

df["fixed_date"] = df.date.apply(lambda x: pd.tseries.offsets.shift_month(x, 12*20))


Training a CNN with limited weight sharing I am currently working with speech recognition, in which i would like to try to use CNN instead of the normal feature extraction step.

I been reading this paper which proposes method using cnn. The input is a  visual representation of mel-log filter bank energies of audio files.

And the output is phoneme recognised for each third frame section a portion (a frames, b frequency_bands) of the image.

The network is a CNN, and they propose a different weight sharing - limited weight sharing, since the patterns being seeked for doesn't occur  equally anywhere on the image, but localised to certain frequency areas.

  Using separate sets of weights for different frequency bands may be more suitable since it allows for detection of distinct feature patterns in different filter bands along the frequency axis. Fig. 5 shows an example of the limited weight sharing (LWS) scheme for CNNs, where only the convolution units that are attached to the same pooling unit share the same convolution weights. These convolution units need to share their weights so that they compute comparable features, which may then be pooled together.

I am not sure I understand the concept of this weight sharing..

Should the weights be shared for each frame but limited in frequency range?

or should it both be limited for each frame and frequency range?

They made a illustration of this weight sharing:

From what i can decipher from the image - is Limited weight sharing option 2.

Each frame do not have the same weight, multiple convolution is applied on to the same frame, and the convolution for the next frame starts at a lower frequency than than the previous frame and has stride = 2. So somehow is the convolution only performed in the diagonal of the image... sounds weird?

Sounds like i've misinterpreted something incorrectly here?
Any ideas on how to implement?

It's important to note that before doing this, they split the frames into individual feature maps.  Note Fig. 1:

The "45 rows" in the upper left of your diagram (Fig 4.) corresponds to 15 frames times 3 feature maps.

The convolution is a 1D convolution, so they are not actually convolving across multiple frames, only across frequencies.  The authors then sum across feature maps in the normal way, meaning summing across frames and features.

Another way of looking at it is in Eq. 9

The variable $O_i$ is a vector across frequency bands only.  The sum is over feature maps which includes frames.


Why are eigenvectors produced by np.linalg.eig different than the PCA components stored in the instance of the PCA object? I am trying to understand why eVec (produced by np.linalg.eig) is different than pca.components_.T from the instance of the PCA class.  It was my understanding that the eigenvecters of the covariance matrix are the principal components after descending sort by eigenvalues.
An explanation in simple terms would be appreciated.
import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler

df = pd.read_csv(
'https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv'
)

df = df.drop(['model', 'vs', 'am'], axis = 1)

df = df.apply(lambda x: pd.to_numeric(x))

M = df.to_numpy()

Mnorm = M-np.mean(M, axis=0)

Mnorm = Mnorm/np.std(M, axis=0)
#  This is the normalized source data.

C = (Mnorm.T @ Mnorm) / (Mnorm.shape[0] - 1)
#  This is the Covariance Matrix without bias.

eVal1, eVec1 = np.linalg.eig(C)

eVal = eVal1[np.flip(np.argsort(eVal1))]
#  eVal is sorted according to the order of the eigenvalues.

eVec = eVec1[np.flip(np.argsort(eVal1))]
#  The same sort order as above is applied to the eigenvectors.

### From sklearn:

scaler = StandardScaler()

scaler = scaler.fit(df.to_numpy())

Anorm = scaler.transform(df.to_numpy())

pca = PCA(n_components=9)

pca_transform = pca.fit_transform(Anorm)

assert (Mnorm == Anorm).all().all()
#  This tests that Mnorm was probably constructed correctly.

assert (C.round(10) == pca.get_covariance().round(10)).all().all()
#  This indicates that the Covariance Matrix (C) was constructed correctly - the rounding is arbitrary.

assert (eVec.round(5) == pca.components_.T.round(5)).all().all()
#  However, eVec and pca.components_.T are not equal.

Two problems:

Your sorting is incorrect:

eVec = eVec1[np.flip(np.argsort(eVal1))]

sorts the rows of the matrix, but you want to sort the columns.  Replacing this with
eVec = eVec1[:, np.flip(np.argsort(eVal1))]

fixes this issue.

The sign of the eigenvectors are sometimes opposite.  (That's fine, being an eigenvector is scale invariant, and while both np.linalg and sklearn use unit eigenvectors, that still leaves a choice of two.  I'm not sure how the packages end up picking one.)


k-means clustering or classification? Why is choosing the k in the k-means clustering method based on a feature (take a dead or alive patients scenario as an example, k will be 2) considered clustering rather than classification?

Choosing k=2 because you want to find classes "dead" and "alive" does usually not work. That is a common toy approach that barely works on exercise data such as the Iris data set.

Because k-means is unsupervised, there is no guarantee that for k=2 it will find dead/alive. It may as well find male/female (k=2) or Chuck Norris/everybody else (k=2).


Decreasing n_estimators is increasing accuracy in AdaBoost? I was exploring the AdaBoost classifier in sklearn. This is the plot of the dataset.
(X,Y are the predictor columns and the color is the label)

As you can see there are exactly 16 points in either side that can be easily miss-classified. TO check how the performance increases with increasing n_estimators I used this code

for i in range(1,21):
    clf = AdaBoostClassifier(DecisionTreeClassifier(max_depth=2),
            n_estimators=i,algorithm='SAMME')
    clf.fit(X_df,y)
    y_pred = clf.predict(X_df)

    from sklearn.metrics import confusion_matrix as CM
    #CM(y_pred,y_pred1)
    #CM(y,y_pred1)
    print(i,CM(y,y_pred))

Upto n_estimators = 13 all the 32 points were miss classified. The confusion matrix is

[[84 16]
 [16 84]]

(
with the exception of n_estimators=8. Here all red points were properly classified)

[[100 0]
 [16 84]]

From 13 onwards it began flipping in a weird way. The confusion matrices are
 given in order.

[[84 16] | [[ 84  16] | [[84 16] | [[ 84  16] | [[ 84  16] | [[100   0] | [[100   0] | [[84 16]
[16 84]] | [  0 100]] | [16 84]] | [  0 100]] | [  0 100]] | [ 16  84]] | [  0 100]] | [16 84]]

Where apparently n_estimators=19 is giving better performance than n_estimators = 20.

Can someone please explain what is happening and what is causing this behavior?

In short, AdaBoost works in that way that it trains in subsequent iterations and then measures the error of all available weak classifiers. In each subsequent iteration, the "validity" of incorrectly qualified observations is increased, so that classifiers pay more attention to them. So confusion matrix could be shown after each iteration(after 13).
In case n_estimators=19 its look like you have the perfect fit so for bigger values of n_estimators model starts to overfit, which gives worse performance. In your case please read about early stopping. That technique help you find you best value of n_estimators.

https://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_early_stopping.html


Why does the listed order of features specified in the data set matter to the random forest classifier >>> from sklearn.ensemble import RandomForestClassifier

>>> clf = RandomForestClassifier(n_estimators=10, random_state=1)
>>> Y=[0,1]

>>> X = [[3,2,1,0], [7,6,5,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[0, 1, 2,3], [4,5,6,7]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[3,2,1,0], [7,6,5,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

>>> X = [[3,1,2,0], [7,5,6,4]]
>>> clf = clf.fit(X, Y)
>>> print clf.feature_importances_
[ 0.2  0.1  0.1  0. ]

Assume the features have names.
When I shuffle/change the listed order of features specified in the training data set, the importance for each feature changes.
That means the resulted random forest classifier also changes.
Note that I have rule out the effect of randomness, by fixing the random seed.

Why does the listed order of features specified in the data set matter to the random forest classifier, given that the random seed is fixed? Thanks.

Since your fixed the seed, with default option bootstrap=True each tree is build on a fixed subsample of your data. With the default option max_features='auto' each tree uses 2 features. Since your fixed the seed, it is always the same, let us say, "1st and 4th" features for the first tree. And they are different in each of your shuffles. So, the tree is different. The same applies to each tree.

By the way, with 4 features in total, only 6 (4*3/2=6) various trees are possible. With n_estimators=10 some of these trees are necessarily repeated.
And in each shuffle those are different trees.


Keras: X and Y are the same, yet validation accuracy is 50%, what is wrong? I am trying to understand what is going on so I built a simpler version of my project. I set the X and the Y to be identical and I'm trying to predict Y using X, this should be very simple, but my setup isn't working. Here is my code :

import numpy
import keras
import pandas

# I want to evaluate the model when X and Y are the same
# This should be very easy for the model to evaluate
X = numpy.random.randint(2, size=100000)
Y = X

# Setup the model
model = keras.models.Sequential()

model.add(keras.layers.Dense(1, input_dim=1, init='uniform', activation='relu'   ))
model.add(keras.layers.Dense(1,              init='uniform', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

hist = model.fit(X, Y, nb_epoch=10, validation_split=.3, batch_size=10, verbose=1)

df             = pandas.DataFrame()
df['loss']     = hist.history['loss']
df['acc']      = hist.history['acc']
df['val_loss'] = hist.history['val_loss']
df['val_acc']  = hist.history['val_acc']
df.index       = df.index + 1
#
print(df)

And this is my output:

        loss       acc  val_loss   val_acc
1   0.693162  0.504357  0.693300  0.496233
2   0.693150  0.503100  0.693250  0.496233
3   0.693157  0.502357  0.693132  0.503767
4   0.693171  0.502214  0.693119  0.503767
5   0.693167  0.502043  0.693121  0.503767
6   0.693129  0.504014  0.693133  0.503767
7   0.693167  0.503243  0.693129  0.503767
8   0.693157  0.502357  0.693181  0.496233
9   0.693180  0.502614  0.693141  0.503767
10  0.693170  0.502300  0.693119  0.503767

I expected that accuracy to go to 100%, but that is not the case. What am I doing wrong?

This is the example that I was following.

You should think about how the initial values impact the ReL Units. If, for example, you use init='one' for the activation='relu' layer you'll get the desired result (in this simple setup).


(Feature Selection) different results from L2-based and Tree-based I am doing feature selection using Sklearn:

Tree-based feature selection : RandomForestClassifier.feature_importances_
L2-based feature selection: LogisticRegression.coef_

Target variable is binary classes. The training set is standardized.

How should I interpret when a certain feature shows significant importance in Random Forest estimator, but negative coefficient in Logistic Regression?

Negative coefficient in logistic regression means negative relationship between predictor variable and the response variable.

Example- in a model price can be a predictor and will have negative relationship with binary response variable product purchased or not.

And neagtive coefficient  in logistic regression does not mean relationship has low strength, it only means changes in predictor has a reverse effect on response variable and if coefficient is highly negative it means feature is very important and small changes in it impacts response but in reverse direction.

Feature importance does not tell you nature or direction of relationship but only tells you strength of relationship so they are never negative. Coefficients in logistic regression tells you both strength and direction or nature (positive or negative) of relationship. Also high importance in random forest means strong relationship between predictor and response but importance column derived from tree based models is silent on nature or direction.

Hope this helps.


LSTM training/prediction with no starting sequence ML newbie here. As an exercise, I'm trying to build a character based language model based on a simple 1 layer LSTM. Based on what I've learned about LSTMs, a common usage is to take in a sequence of characters and then predict the next character. What I don't fully understand is how I would go about predicting the very first character when there's no preceding sequence yet (or, by extension, predicting a character when there's not a long enough preceding sequence as input to the LSTM units).

The best solution I can thing of is to reserve a special character in the vocabulary to represent the abscence of any characters. A toy example:

Full training corpus: "foo"
LSTM unit count: 2
"Absent character" symbol: ABSENT
"End of file" symbol: EOF
Training inputs:
    sample: [ABSENT, ABSENT] label: 'f'
    sample: [ABSENT, 'f'] label: 'o'
    sample: ['f', 'o'] label: 'o'
    sample: ['o', 'o'] label: EOF

My question is: what's the best practice for doing this type of thing? Am I on the right track?

You might choose to demand predictions only after N steps of your sequence have elapsed. Then, predictions are trust-worthy.  You've got to give your LSTM something to begin from, some context so to speak.

Usually you sum the errors your network produced across all timesteps, but in such a case you ignore its outputs until the N'th timestep onwards.
As follows, any prediction it makes shouldn't contribute to the sum of your errors: TotalErrorOfTheSequence

Notice, in such a case, when backpropagating, your gradient won't flow into the network before N'th timestep.

You can also learn the initial state directly, by backpropagating with respect to the actual cell_t0. Traditionally you would initialize cell_t0 with zeros, but because you can compute a gradient towards it, you can pull its starting value closer and closer to what's usually needed.

It's possible to improve the best starting-state of LSTM with noise, which does actually improve the quality of predictions. It mitigates network overfitting on early states of your sequence where the error is usually the largest due to small context available, and where corrections are strong. "Forecasting with Recurrent Neural Networks: 12 Tricks". - read the 3rd trick

Here is my understanding, please correct it if it's wrong:

1) train your Cell state as a variable (as described above), to get a good value.

2) Construct a random variable, which is centered around your cell_t0.

3) Collect a buffer of several hundred G-values, then start sampling randomly and use each one as a maximum possible deviation from your learned 'cell_t0'. So you randomly deviate away from the learned 'cell_t0' by some sampled G-Value

Each of such G values is the value of the gradient towards the actual already learned cell_0

I think it will make sense to keep track of which such G values are the oldest, and slowly remove them from the buffer when they become obsolete

The noise is to be disabled during testing.

Once again, not sure if I understood that approach correctly

Also have a look at these posts:

https://datascience.stackexchange.com/a/33994/43077
https://stats.stackexchange.com/a/319854/187816


Logistic Regression - Coefficients have p-value more than alpha(0.05) I am very new to the machine learning field and have been practicing logistic regression on few sample data sets.
I have built a Model using the logistic regression algorithm. Few of the coefficients have a p-value of more than 0.05 (which is the alpha I am  considering).

R-Code for building the model and a summary of the model is given below.
model.bank.1 <- glm(y~., data=bankfull, family ="binomial")
summary(model.bank.1)

Now, before considering AIC, Residual/Null Deviance, confusion matrix and ROC for evaluating. I have observed that the p-values for some of the independent variables is more than 0.05(age has very high p-value, what does this mean?). In such a case, what should be done? Should i straight away remove all such predictors from my model ? Is there any way by which i can make p-value of these predictors less than 0.05 ?
What are the things needed to be checked before going on to evaluate the model using the AIC,Deviance, Confusion Matrix and ROC measures ?
Edit 1: I've tried standardizing the numerical columns, but there is no change in the model at all.

The p-value summarises a statistical test for a coefficient not to be statistically different from zero. So basically, when the p-value is > 5%, the estimated coefficient can be positive or negative (the confidence intervall includes positive and negative values).

Often this is interpreted in a way that some variable does not make a reliable contribution to a model. This usually is relevant in causal modelling. But this does not mean that the variable does not contribute to a model and should be excluded. Omitted variable bias can be a massive problem!!! (see: Econometric Theory and Methods, Davidson/Mackinnon, Chapter 3.2 for the OLS case - very interesting stuff). Also in joint significance tests, a non-significant variable can still be relevant.

Especially if you are interested in predictions, the p-value is not so much of a problem. In tendency you can say that overspecification of a model is less harmful than underspecification. In doubt, keep the non-significant variable in the model (it may only be a weak predictor).

However, if you have weak predictors you may also look towards Lasso or Ridge-Regression. In this regression types, variables with little or no contribution to prediction are "shrunken". This is a really cool thing to do and might be the first next thing for you to look at: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html#log

I like Lasso/Ridge, because model selection is (at least to some extent) data driven in this case.

Also have a look at "An Introduction to Statistical Learning
with Applications in R" (Chapter 6.2). You may find a copy of the book online. The R code examples in the book are really instructive.


Best approach for image recognition/classification with few training data After research, I've decided to take advice on the approach to use for our machine learning problem as I'm still not sure which to take.
Our problem:
We have our labels as a set of website names (Google, Twitter and Facebook for example). We need to train our model on a set of images that belong to each website, and later be able to give the model any image and predict which website it might be. Also note that the image could belong to none of the websites trained so I need some sort of None output.
One of the problems we're facing is the small number of data for each set (around 40 images for Facebook login forms that could be pushed to 200 with color transformations if that makes a difference while training.)
What we need:
Some sort of image classifier/predictor/recognizer (still not sure which) that will detect the class of the input image (Facebook, Google or Twitter for example) or tell us that it's none.
Problems we might face:

I'm not sure if our model is subject to over-fitting since our set of input images are limited and we should already have trained the model on. Take the facebook login page for example, they all look the same with small variety in the location of the fields. So our training and out-of-sample data are almost the same.

What I've done so far:

I've already started working on a Convolutional Deep Network using keras, but I'm worried that deep networks would be too complex for our data, or our data is too simple(alike) for the model to be trained correctly.
One Shot Learning and Siamese Networks, but this needs to have an output that must belong to the data we've trained on (which disregards the None possibility)
Sklearn's different classifiers, but a classifier must have an either or output for one of the labels, which again disregards the None possibility.
Finally, and what sounds most logical personally, is to use sklearn's OneClassSVM on each label (so if I have Google, Facebook and Twitter then I would have 3 models, one that corresponds to each label) and then pass my input image to each model to determine to which it belongs. The problem with this is if I have a lot of classes (which could be the case) then I'd have to run each one.

One final approach could be generating probabilities for each class after inputting an image to our model, assuming that these probabilities should not sum to 1 as to keep the None possibility.
Any suggestions or advice would be helpful. Thank you!

Your data seem to be very small to be used for training a CNN. I do not fully understand what does it mean that an image "belongs" to "Twitter" and not to "Google".

Nevertheless, my best advice to you is to use simple sklearn classifiers and gather some data that does not belong to any of your options to be used as the None class. The None class should include a similar amount of data as the other classes and then you can train any classier and check the scores.

Regarding the over-fitting situation, you should consider doing cross-validation (because your data is too small for hold out set). Try doing a 5-fold cross-validation and assess how good your model performs on average.

Good Luck!


Is it possible to achieve activity recognition on a single image? If so, can you give me resources?

The short answer, yes and even with a good performance, although not with same accuracy as with multiple frames.
There are actually many different architectures that rely on classifying each single frame as part of their process (you can treat the overall model as an ensemble-model of multiple frames).

There is a great article by "Schindler et al."  about exactly this question:

Action Snippets: How many frames does human action recognition require?


Why Liblinear performs drastically better than libsvm linear kernel? l have a dataset of dim=(200,2000) 200 examples and 2000 features. l have 10 classes.

l used sklearn  for both cases :

svm.svc(kernel=linear)
LinearSVC()

However LinearSVC() performs drastically better than svm with linear kernel. 60% against 23%.
l'm supposed to get the same or comparable results since they are fed with same parameters and data.

What's wrong ?

Thank you

Here is just a guess, but according to me, the linearSVC might perfoms better than SVM with linear kernel because of regularization.

Because linearSVC is based on liblinear rather than libsvm, it has more flexibility and it gives you the possibility to use regularization with your SVM (default is L2-Ridge regularization).

Because you have more features than observations, it exists multiple solutions to your classification model. Some of them are more "robust" than other ones. L2 regularization will help you reduce the ampltitude of your model coefficients and maybe lead to a more stable solution.

More on LinearSVC in the documentation :

http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html

More on Ridge regression benefits when working with a lot of features (and so multicollinearity) :

https://stats.stackexchange.com/questions/118712/why-does-ridge-estimate-become-better-than-ols-by-adding-a-constant-to-the-diago


Master of Statistics or Master of Data Science Many Universities now offer Masters Degrees in data science; I'm curious as to people's opinions on whether a traditional master of statistics is a better to fit for working in the field of in data science.

As everyone here knows, data science as a field is broad, and similarly the experience required for a particular job is broad.

Depending on where you want to end up, either degree could suit you better. If you would like more of a traditional statistician's job which has been rebranded to data scientist in order to gain more attention, then perhaps the MSc Stats would be more applicable. However, if your field requires more complex computer programming or database management, you may want to look into the masters in data science or masters in computer science, depending on your background and interest. Going with a "hot" degree like data science means that you will likely learn a lot of common analyses in superficial detail in order to be employable quickly, but with a more traditionally academic MSc Stats or MSc Com sci, you will learn the theoretical basis with the potential loss of applicability. My personal bias is towards to academic side (MSc/PhD Stats or applied math), but I think this is a decision you should think hard about. Regardless of your decision, in general data science / analysis / programming / whatever this field is requires quite a lot of self directed learning, so whatever program you choose must provide you with the skills and tools to be able to teach yourself whatever you need to know.

I would personally take a hard look at what specific kinds of jobs appeal to you, and which programs may prepare you better. Talk to the admissions officers and look up the career paths of past students. Where did they end up? Is that where you want to be? Keep in mind the kind of life you want to live and the kind of work you want to do. I personally find anything financially or profit driven absolutely soul-sucking, but enjoying doing almost the same work in the medical field. Different skill sets, different programming languages, and different vocabulary are required, and the more you know before you enter the field the better you can prepare yourself for the the journey to come.

Regardless, have fun, and don't worry too much about making mistakes. No one knows how things will work out in the long run. You can prepare yourself with as much knowledge as possible, but you'll never know which was the truly right choice, just which one was your choice.


What is better opensource alternative for identifying small face other than yolo? I was trying to identify small face meaning that I want to know who that face belong to according to training dataset. I have previously use yolov4 to detect small object before and I know the accuracy is not very satisfying and I have read in internet that yolo is very terrible at detecting faces. Do anyone know what is the alternative opensource that is better than yolo v4 for identifying small face? Please share your knowledge. Thanks.
Edit 1: I mean like for identifying this face is for example angelina jolie face,etc(face recognition/ identification) not face detection(knowing existence of face)

Here is a model tailored for your problem. And here is the research paper for the model.


When can it be called transfer learning? A common definition of transfer learning is:

"Transfer learning is the improvement of learning in a new task
through the transfer of knowledge from a related task that has already
been learned."

— Chapter 11: Transfer Learning, Handbook of Research on Machine Learning Applications, 2009.
This raises the question, when a task can be termed "related".
Let's assume a neural networks is trained to estimate house prices for american houses.
Could it be called transfer learning, if I retrain/finetune the model to estimate european house prices?
Is it still the same task or can it be considered a separate task?

Good question but rather besides the point.
What I mean is that even though estimating european house prices can be considered the same task (as estimating american house prices), nevertheless in this example the NN was re-purposed using transfer learning methods from another task.
Whether one would like to call the european house prices as same or related task, is not so much the point here (which can become pedantic), but rather if transfer learning methods were used and how succesfull they were.
IMHO


Is reseating passengers a reinforcement learning problem? Requirement is to optimally move passengers from one seat map to another which has a different configuration.

Move should be based on many rules like -
1) Families should be sitting together
2) Those who were at windows seat should be moved to window seat if available.
3) Kids should be seated at middle seat.
4) and many more..

There are many seat map configurations available and at a time we need to move passengers from one to another as per the requirement.

Is it a reinforcement learning problem ? If yes, how it can be approached.

Reinforcement learning is more about interacting with an environment, and while this could be posed as an RL problem, I think using Global Optimization would be a more direct approach.

Essentially you want to design a cost function that describes how good a particular seating is and then use it to search the space of possible seatings.

For example to solve the problem with Simulated Annealing:

Design a cost function $e(s)$ that measures how good a seating arrangement is. Lower cost means better seating.
Design an acceptance probability function $P(e, e', T)$ that takes the costs of seating arrangements $s$ and $s'$, and a temperature $T$, and returns a probability $P$ with the properties (a) $P>0$ even if $s'$ is worse than $s$ (b) the better $s'$ is relative to $s$, the higher $P$ is and (b) the lower the temperature, $T$, the close $P$ is to $0$ when $s'$ is worse than $s$.
Set some large initial $T$ and design an annealing schedule for decreasing $T$ with the number of iterations.
Start with some seating arrangement $S$. This can be random or chosen according to a greedy strategy that tries to satisfy as many constraints as possible.
Repeat for $i$ in $k$ steps (for some sufficiently large $k$):

Consider a neighboring state $S'$ that can be reached by randomly swapping two seated people or moving someone to an empty seat. Set $S$ to $S'$ with probability $P(S, S', T)$
Decrease $T$ according to the annealing schedule at step $i$

The simulation starting at (4) can be run once or many times saving only the best result. At the end you should arrive at close to or exactly the best possible seating arrangement.


What metrics determine the quality of the model? Working on this Kaggle competition, and have some questions. Using this code:

def r2_keras(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true - y_pred))
    SS_tot = K.sum(K.square(y_true - K.mean(y_true)))
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

The output of my training looks like:

0s 138us/step - loss: 0.1340 - mean_squared_error: 0.1340 - r2_keras: 0.7565 - val_loss: 0.4112 - val_mean_squared_error: 0.4112 - val_r2_keras: 0.4064
Scaled Validation r2: 0.5182
Unscaled Validation r2: -152.1261

I am using 20% of the training data for validation.

I am tracking these metrics during training:

Training loss, mse and r2
Validation loss, mse and r2

I get these metrics on the model after training:

R2 for Validation on scaled data
R2 for Validation on unscaled data

scaler = StandardScaler()
scaled_train = scaler.fit_transform(train_df)
scaled_test  = scaler.transform(test_df)
...
m.fit( X_train, Y_train, epochs=epochs, validation_data=(X_test,Y_test))
....
from sklearn.metrics import r2_score
scaled_r2 = r2_score(prediction, scaled_test_df[[target]].values)
unscaled_r2 = r2_score(descaled_prediction, test_df[target].values)

So, my questions are:

Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?
Why isn't the unscaled r2 the same as the scaled r2?
The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?

Unscaled and scaled r2's are not highly correlated (0.31 AAMOF). Which one would best describe the accuracy of the model on unseen data?

I don't think this is a matter of which will describe the generalization error better, because both of them are describing the same thing, just on different scales. So, the advice would be to use the accuracy metric consistent with the metric that will be used for predictions on unseen data.

  Why isn't the unscaled r2 the same as the scaled r2?

This is because MSE is scale dependent.

  The model r2 is not the same as any of the validation r2's during training (val_r2_keras). Shouldn't the trained model r2 be the same as the one reported during the training?

Why do you think so?

They are different because the datasets for training and for validation are different.


Square-law based RBF kernel What is the Square-law based RBF kernel (SQ-RBF)? The definition in the table at the Wikipedia article Activation Function looks wrong, since it says

y =
     1 - x^2/2     for |x| <= 1
     2 - (2-x^2)/2 for 1 < |x| <= 2
     0             for |x| > 2

but this makes it discontinuous at x = 1

That is indeed quite odd. I can't really find many other sources on SQ-RBF, except here, where the definition differs from wikipedia, but when I plot this, it does not recover the image they show. However, if you define

y =
    1 - x^2/2        for |x| <= 1
    0.5*(|x|-2)^2    for 1 < |x| <= 2
    0                for |x| > 2

you can recover a sensible shape.


Is there any text similarity databse available for phrases? I want to train my application for phrase similarity. I want my model to predict similarity score for phrases as shown in below examples.
ex-

International Business Machines = I.B.M
Synergy Telecom = SynTel
Beam inc = Beam Incorporate
Sir J J Smith = Johnson Smith
Alex, Julia = J Alex
James B. D. Joshi = James Joshi
James Beaty, Jr. = Beaty

Is there any dataset available to train this type of model?

This is a difficult problem, but definitely worth exploring.
An interesting resource to look into is DBpedia. It aims to extract structured information from the Wikipedia project. It is available under a free license (CC-BY-SA).
You can conveniently explore the project online, e.g.:

IBM
Beam Suntory

Note that you are restricted to the extensive but ending knowledge on Wikipedia, for example Synergy Telecom/SynTel seems not to have an entry. Your creativity would be required to overcome this limitation.


Creating parallel keras layers I am new to Keras and ML and I want to create a NN that can seperate a bitmap-like image into its visual components.
My approach is to feed a two dimensional image (lets say 8 by 8 pixels) into a NN, which then outputs a three dimensional matrix (e.g 3 by 8 by 8). Every element of the first dimension represents an image of a component (see this illustration).
As a first version I created a Keras sequential model with 1 flatten layer and 2 dense layers, 192 units each (3x8x8=192). After around 5000 training images, the model performance was still mediocre.
Question: Is there a way of splitting the output of the last dense layer into three seperate images that will then be processed in a parallel manner by further dense layers? Is there a better approach to the problem I am facing?

So, yes and no.
First, there's not a layer that does this in the standard Keras API. It might be possible to write a custom layer to do it, but I'm not comfortable enough doing that to guide you.
What you COULD do is create three different layers, each of which accepts the whole thing as input, and let each of them figure out what parts they're responsible for. More on that in a minute.
Second, to do it requires you to use the Functional API. The Sequential API is just that - sequential. But functional is pretty easy once you understand what's going on.
Ok. Now for the above approach.
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Flatten, Concatenate
from tobias_code import get_my_data, compiler_args

data = get_my_data() # obviously, this is a stand-in for however you get your data.

input_layer = Input(data.shape[1:])
hidden = Flatten()(input_layer)

This is where the Functional API differs from the Sequential API. Because the Sequential API assumes each layer attaches to the previous layer, you don't have to manually do so. The Functional API allows you to connect layers however you'd like! But to allow for that, it requires you to manually connect each layer to the previous. I typically reuse 'hidden' to chain my layers together, deviating only for the input, output, and any unusually connected layers.
hidden = Dense(192, activation='relu')(hidden)
main_output = Dense(192, activation='relu')(hidden)

Here's where yours would end. You'd call model = tf.keras.Model(input_layer, main_output), do whatever compiling you had to do, and begin training. But we're going to continue on. (You probably wouldn't use relu for your output layer, but eh...)
# I'm going to build each individual parallel set of layers separately
branch_a = Dense(96, activation='relu')(main_output)
branch_a = Dense(48, activation='relu')(branch_a)
output_a = Dense(24)(branch_a) # This will be one of my outputs, so I want a linear activation

branch_b = Dense(96, activation='relu')(main_output) # note that it is main_output again
branch_b = Dense(48, activation='relu')(branch_b)
output_b = Dense(24)(branch_b)

branch_c = Dense(96, activation='relu')(main_output) # and again. 3 layers are all sharing that output.
branch_c = Dense(48, activation='relu')(branch_c)
output_c = Dense(24)(branch_c)

Now, the Functional API can handle multiple outputs. You can define separate loss functions for each output and separate metrics for each output. But I haven't done a lot with that, so rather than tell you incorrectly how to do it, we're going to concatenate your outputs into one, which is probably what your output looked like in the sequential model.
final_output = Concatenate()([output_a, output_b, output_c])
model = tf.keras.Model(input_layer, final_output)
model.compile(**compiler_args)
model.summary()

Now, I haven't run it, and I make no claims as to its efficacy; I pretty much typed it here and am hoping for the best. But it should get you where you want to be, and hopefully if it's not exactly what you want it will set you on the right path.


border_mode for convolutional layers in keras Keras has two border_mode for convolution2D, same and valid.
Could anyone explain what "same" does or point out some documentation? I could not find any document on the net (except people asking that it be implemented in theano as well).

With border mode "valid" you get an output that is smaller than the input because the convolution is only computed where the input and the filter fully overlap.

With border mode "same" you get an output that is the "same" size as the input. That means that the filter has to go outside the bounds of the input by "filter size / 2" - the area outside of the input is normally padded with zeros.

Note that some libraries also support the border mode "full" where the filter goes even further outside the bounds of the input - up to "filter size - 1". This results in an output shape larger than the input.

There's a short explanation in.. numpy's convolve documentation:

http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.convolve.html


Word2Vec - document similarity Lets say I have text data for different documents from 2005 - 2015. I want to compare the similarity between $t$ and $t-1$ documents. So I take the document at 2006 and compare it with the document at 2005, take the document at 2007 and compare with the document at 2006 … all the way to 2015, compared with 2014.

I have computed for each year a Word2Vec model independently of other year and obtained high dimensional arrays for each Word. So I have 10 Word2Vec models from 2005 - 2015.

Whats the best way for me to compare the documents similarity from here.

Previously I used TF-IDF where I was able to for each document have a large matrix with words in the rows and documents in the columns. Then I could combine he TermDocumentMatrix at $t$ and $t-1$ and compute the cosine similarity.

However Word2Vec gives much high dimensional arrays and I cannot think how to compare W2V models from $t$ to $t-1$.

Any help would be great!

you might want to look into 'approximate nearest neighbors' analysis, and particularly the annoy package from Spotify:

https://github.com/spotify/annoy

It's not as precise as an exact comparison, but it may be close enough to get you what you need.


Random Forest application with 40+ Predictor Variables I am using R package randomForest to build a Random Forest model for classification. Ultimately, I need to choose one of five programs for a group of individuals based on historical data. The final variable that is being predicted is a categorical "1, 2, 3, 4, or 5" variable.

I need to incorporate 30-40 variables--their importance will also be analyzed-- to reach a decision for each individual.

I have no problem training a model, initially. Below is a sample data set being trained with R's randomForest package. I am running up-to-date versions of both the package and RStudio.

set.seed(101)
train <- sample(1:nrow(Boston), 300)
Boston.rf <- randomForest(medv~., data = Boston, subset = train)
Boston.rf

My question is: Once I have a model trained, tested, and cross-validated, how do I actually apply new data to this model? Is it viable to use that many variables--all categorical, some binary--for this?

The randomForest package supports various tasks using an existing randomForest object.

For instance, it offers the predict method which will perform prediction using a given trained forest and a given dataset. As an example, if you have a trained forest called mod and a dataset called my_data, predictions can be obtained by running predict(object=mod,newdata=my_data).

It's worth reading through the package documentation which gives good examples of using the functionality.

I would note that any new data you supply to a trained forest object should have the same encodings and transformations as the data used to train the forest with.


PySpark: How do I specify dropna axis in PySpark transformation? I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument axis = 'columns' in dropna(). Here an example in a GitHub post.

How do I do this in PySpark ? dropna() is available as a transformation in PySpark, however axis is not an available keyword.

Note: I do not want to transpose my dataframe for this to work.

How would I drop the furniture column from this dataframe ?

data_2 = { 'furniture': [np.NaN ,np.NaN ,np.NaN], 'myid': ['1-12', '0-11', '2-12'], 'clothing': ["pants", "shoes", "socks"]}

df_1 = pd.DataFrame(data_2)
ddf_1 = spark.createDataFrame(df_1)
ddf_1.show()

I  know this is a bit late, but I struggled with this also. This is my attempt at removing null columns from a Spark Dataframe.
from pyspark.sql.functions import when, isnull

colsthatarenull = df.select([(when(isnull(c), c)).alias(c) for c in df.columns]).first().asDict()
namesofnullcols = {key:val for key, val in colsthatarenull.items() if val != None}.values()
df = df.drop(*namesofnullcols)


How to improve my f1 score in stories analyze I got an assignment to build a model that identify the gender of the text writer.
The assignment score will determine by my model f1_score, to get the maximum points, T need it will be at least 0.7.
I'm given a 360 stories and their writer gender, and another 150 to check on (without their gender).
I tried a few different ML modle, a few normalization method and a few stop words corpus. and yet to reach my destination.
if you have any tips/information I would appreciate it!!
** the stories languages is NOT English, but my native language (Hebrew)**

Author gender prediction belongs to the category of stylometry tasks, i.e. related the the style of the text rather than the content. In general this distinction requires a quite different approach than standard content-based tasks, here are a few of the most common differences:

Content words (nouns, adjectives, adverbs) are potentially less important, so TFIDF and similar methods make little sense. It depends on the data though.
Usually it's counter-productive to remove stop words, because they can be actually very useful and distinctive of an author's style (at least their proportion in a text).
For reasons which are not really clear, it's been found experimentally that bigrams and trigrams of characters are often better features than words. This has been observed at least in English, I don't know if this is applicable to other languages.

In general it's not recommended to target a specific level of performance unless it's a baseline known to be reachable, e.g. when another model can do it on similar data. That being said, in the case of this task I would say that 0.7 F1-score is reasonably realistic, but it really depends a lot on the data.
In case you are interested in researching the literature in order to apply state of the art methods, there have been some shared tasks about this question, sometimes mixed with other predictions such as age. search for "author profiling" and "gender profiling" in the PAN Shared Tasks. You might also find other relevant methods in the general category of "author classification/verification".


Confusion matrix, when mistakes below diagonal are better than above the diagonal I have a classification problem and I am producing a confusion matrix. Ideally one wants to get all results in the diagonal. I get quite many points around diagonal for different algorithms. Still for my use-case I want to favor algorithms that underpredict the class (I have ordinal data) and not overpredict.
Is there a metric that can measure under and overprediction and rate those errors with a different weight? The typical accuracy, precision terms assume that all mistakes are the same.
Of course I can try to implement my own metric but I am quite sure that I am not the first that is having this issue.
Any metric available that you know already?
Thanks
Alex

You are incorrect when you say accuracy and precision assume the same mistakes. They are quite different in nature as to when they are applied for separate use-cases.
For more details on how they are different - https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9
As per your situation, you could opt for precision as you are looking only for the right answers, in case of a spam email detection.
There is no metric as such which calculates how much your model underfits or overfits but a genera idea can be made when you compare the validation curve and the training curve.


reduction of sample from videos sample Well, I post the same question in the main stack before finding the right place, sorry.

A friend of mine is working with more than a 100 videos as sample for his neural network. Each video last more than a couple of minutes with around 24 frames per second. The objective, using deep learning, is to detect movement through all the samples.

The problem for him is the quantity of data he is dealing with. The training part require/consumes too much time. I'm no expert with data preparation, but I thought maybe he could turn all frame into dataframe, clean them from mono color image (full black/white), turn them into gray instead of full rgb and compress them but, I'm not sure if it will be enough.

Do you think of better method to reduce the training sample?

Reduce the size e.g. using cv2.resize()
Compress the image (it is not lossless) e.g. cv2.imencode()
Lower the frame rate
Use lower precision - images are uint8 when loaded, but the deep learning frameworks use float32 by default. You could try float16 or mixed precision.

Using JPEG compression has been shown to be fairly good in terms of the reduction in memory and minimal loss of performance. Have a look a this research.

You could also drop the frame rate, so say 10 FPS. The actually values could be computed based on the expected velocity of the moving objects -> do you really require 24 FPS for the task?

Otherwise, the hardware you are using will determine which steps to take afterwards. Memory, number of operations, inference speed etc. will change how you optimise the process.

You mentioned "dataframe", so I will just point out that using Pandas Dataframes to hold raw image data, whilst looking easy, it generally very inefficient due to the number of data-points involved (pixels), and the fact that Pandas DataFrames are essentially annotated NumPy arrays - the annotations take a lot of space. Better to load into pure numpy arrays and use OpenCV for things suchs as making gray-scale (black and white) images from RGB, resizing them, normalising pixel values, and so on.


One Hot Encoding for any kind of dataset How can I make a one hot encoding for a unknown dataset which can iterate and check the dytype of the dataset and do one hot encoding by checking the number of unique values of the columns, also how to keep track of the new one hot encoded data with the original dataset?

I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes.
import numpy as np
import pandas as pd
from category_encoders.one_hot import OneHotEncoder

pd.options.display.float_format = '{:.2f}'.format # to make legible

# make some data
df = pd.DataFrame({'a': ['aa','bb','cc']*2,
                   'b': [True, False] * 3,
                   'c': [1.0, 2.0] * 3})

cols_encoding = df.select_dtypes(include='object').columns
ohe = OneHotEncoder(cols=cols_encoding)
encoded = ohe.fit_transform(df)

Note that you can change the way you handle unseen data with

handle_unknown: str

options are ‘error’, ‘return_nan’, ‘value’, and ‘indicator’. The default is ‘value’. Warning: if indicator is used, an extra column will be added in if the transform matrix has unknown categories. This can cause unexpected changes in dimension in some cases.


How the "def match(self, example)" method is automatically calling and how the example arguments is working here? trainung_data = [
    ['Green', 3, 'Apple'],
    ['Yellow', 3, 'Apple'],
    ['Red', 1, 'Grape'],
    ['Red', 1, 'Grape'],
    ['Yellow', 3, 'Lemon'],
]
header = ["color", "diameter", "label"]
def is_numeric(value):
    return isinstance(value, int) or isinstance(value, float)
class Question:
    def __init__(self, column, value):
        self.column = column
        self.value = value

    def match(self, example):
        val = example[self.column]
        if is_numeric(val):
            return val>=self.value
        else:
             return val>=self.value
    def __repr__(self):
        condition = '=='
        if is_numeric(self.value):
            condition = '>='
        return "Is %s %s %s?" % (header[self.column], condition, str(self.value))

Now if i call the class like:

Question(0, 'Green')

It give me output like this

Is color == Green?

Typing

Question(0,'Green')

should not print out anything. However if you type

print(Question(0,'Green'))

then, the $\_\_$repr$\_\_$ method is called.


Speed decay proof for L2 regularization and non-normalizied weight initiation I'm working through http://neuralnetworksanddeeplearning.com/chap3.html,
textbook by Michael Nielsen.

While most problems are rather straightforward, I'm very stuck on these two questions:

Sketch a heuristic argument that: (1) supposing $\lambda$ is not too small, the first epochs of training will be dominated almost entirely by weight decay; (2) provided $\eta\lambda \ll n$ the weights will decay by a factor of $ \exp(−\eta\lambda/m)$ per epoch;

To give a self-sufficient context, the weight decay function is:

$w \rightarrow \left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}$,

and cost function is

$C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln
(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2$

(Classic logloss)

Weights in the network are initiated as $N(0,1)$.

My intuition for (1) goes as follows: As weight initialisation is non-normalized by size of previous layer, activation values are likely to be either $1$ or $0$ with equal probability (which is pointed out by the author). $y_j$ has $0.1$ probability of being 1. Given that $\frac{\partial C_x}{\partial w} = a^{L-1}_k(a^L_j - y_j)$ for logloss and sigmoid neurons, expected value of $\frac{\partial C_x}{\partial w}$ on the first iteration is $0.2$.
Given that $\lambda$ is arbitrary, we can be sure that
$|\frac{\eta \lambda}{n} w|$ can be larger than $|\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}|$

Despite questions being rather informal, I don't think this solution is a good one.

As for the (2), I lack any ideas whatsoever, except driving $\frac{\eta \lambda}{n}$ to zero, but that seems to have zero chance of yielding the required result.

I hope someone can give me a hand in this.
Thank you!

P.S. There is a chance I'm going to the wrong overflow, but I used
https://meta.stackexchange.com/questions/130524/which-stack-exchange-website-for-machine-learning-and-computational-algorithms
question as a guidance for overflow choice.

A note: I had asked similiar question here, but that might not be the best overflow for this question. I'll put a link to whichever get's answered first :)

Ok, I have found the source of confusion, thanks to @Emre for the insight.
The task talks about epochs of training, while I was trying to solve one batch update. I think this confusion is statistically possible enough to validate existence of this question :)


What would be the best machine learning approach for sets of varying sizes? I have the following problem: I have two differents sets of labels (extracted using N.E.R) and given a combination of labels of the first set (a,b,c or d) I have a supervised set of best combination of the second (x,y,z) as an "answer".
The problem is, both can vary in size.
A hypothetical training data would be something like:
{a1,b2,c4,d1} -> {x2,y4,z5}
{a1,b1,c1} -> {x2,y2,z1}
{a4,b2,c4,d1} -> {x1,y3,z5}
...
{a4,b2,c4,d3} -> {x1,y3,z5,w2}

Of course new types of combination of the first set would appear and, using ML, I'd expect to like to give the best prediction.
So, what would be the best machine learning approach for that situation?

This could be modeled as multi-label classification.  The features are nominal values, and the targets are the presence or absence of nominal values.
There are wide variety of algorithms that can learn multi-label classification. The "best" one is empirical question that depends on the specific dataset. One popular option is random forest classifier.
There is also a "strict" version of problem where each combination is considered a unique label. Then it would be multi-class classification. But the targets might be too sparse to learn.


Recommended Tutorial Videos or Books on Feature Engineering Using Python I will appreciate it if you guys can recommend for me a good hands-on tutorial videos or books on feature engineering using Python. I do not want videos or books that teach only the theory behind feature engineering but one that will show me some codes and how to do it.
Thanks.

There are several articles and tutorials online. For example,
PythonDataScienceHandbook or towardsdatascience.
Depend on your particular problem, you can also apply the Automated tools for feature engineering.
Most of these materials have python code and links for your learning.


How does KNN handle categorical features For a K nearest neighbors algorithm using a Euclidean distance metric, how does the algorithm compute euclidean distances when one(or all) of the features are categorical? Or does it just go by the most commonly occurring value among the neighbors?

So e.g. if the 2 features of 3 neighbors are age and gender with values:
age,gender=[ [20,M], [31,F], [23,M] ], and we need to pick the 2 nearest neighbors for a new observation [20,F], how do we convert the gender to a numeric scale to compute euclidean distances?

It doesn't handle categorical features. This is a fundamental weakness of kNN.  kNN doesn't work great in general when features are on different scales. This is especially true when one of the 'scales' is a category label.  You have to decide how to convert categorical features to a numeric scale, and somehow assign inter-category distances in a way that makes sense with other features (like, age-age distances...but what is an age-category distance?).

If all features are categorical, and inter-category distances are all treated as somehow equal, your job actually gets a little easier since you aren't converting between categorical and scalar scales.


String values Using python with pandas Unique ID        Student Name      BMI       LOW BMI        HIGH BMI
HYD_61601_1200  Akshay Kumar Kale   17.0    NON-UNDER BMI   CANCER
YDD_62329_1008  MAMATHA  ERRA      12.9     NON-UNDER BMI   CANCER
HYD_61601_1000  SURESH BATTU       21.49    NON-UNDER BMI   CANCER
HYD_61601_1021  Vishal  Kavali     13.69    NON-UNDER BMI   CANCER
HYD_61601_1212  DIVYATEJA  M       15.5     NON-UNDER BMI   CANCER
MBNR_61413_2003 Aishaarya P        13.39    NON-UNDER BMI   CANCER
MBNR_61413_1230 SUGUNA             21.5     NON-UNDER BMI   CANCER

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Importing the dataset
dataset = pd.read_csv('HB.csv')
dataset.head()
dataset['HB']<12.5
dataset['LOW HB'] = dataset['HB'] < 12.5
dataset['HB']>12.5
dataset['HIGH HB'] = dataset['HB'] > 12.5
temp = {True:'ANEMIA', False:'NON-ANEMIA'}
dataset['LOW HB'] = dataset['LOW HB'].map(temp)
temp = {True:'CANCER', False:'NON-CANCAER'}
dataset['HIGH HB'] = dataset['HIGH HB'].map(temp)

above data and code i just want to know whether i can assign multiple values r not(for example, i want assign to cancer new values like stage 1, stage 2, final stage) if it possible can any suggest me code r condition to be applied

After you answer in comments:

You can add a new column with either new information (coming from another file or other source), or you can do it based on the existing columns. In fact you are already adding new columns in your code.

I will add a new column called STAGE both ways to give you an example of each.

From static data

I assume you have the stages in a list, like this:

stage_data = ['stage 1', 'final stage', 'stage 1', 'final stage',
              'stage 2','stage 1','stage 2',]

If you have the stages of each of the existing rows, you can add it simply as follows:

dataset['STAGE'] = stage_data

NOTE: The length of the list should match the length of your dataset i.e. the length of the pandas DataFrame. Check it with this:

assert len(dataset) == len(stage_data), "The lengths don't match!!"

From a condition on existing columns

# Make new column simply filled with zeros - we will fill it after
dataset['STAGE'] = np.zeros(len(dataset))

Now you can use conditions on existing columns and fill the rows that match with your corresponding stage:

# assume people with BMI above 25 must be `stage 1`
data[dataset.BMI > 25]['STAGE'] = 'stage 1'

# assume people with BMI below 15 must be `final stage`
dataset[dataset.BMI < 15]['STAGE'] = 'final stage`

# Stupid example to fill the gaps and show how you can use two conditions:
dataset[(dataset.BMI <= 25) & (dataset.BMI >= 15)]['STAGE'] = 'stage 2'


ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, I am new to python and programming for that matter. I am trying to solve a special type of wave equation using finite element method and carry out simulations in python. I am getting some errors which I do not understand and I need help. Below is the error and subsequently a part of the code.

ValueError                                Traceback (most recent call last)
<ipython-input-138-b33b0055e38e> in <module>
      6     # Finite Element Method
      7     #unew = (dt**2) * Minv @ (f*src[it]  -  K @ u) + 2*u - uold
----> 8     vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v
      9     v = v, vnew
     10     unew = (dt) * vnew

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 1000)

Time extrapolation

for it in range(nt):
    # --------------------------------------
    # Finite Element Method
    #unew = (dt**2) * Minv @ (f*src[it]  -  K @ u) + 2*u - uold
    vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v
    v = v, vnew
    unew = (dt) * vnew
    #uold, u = u, unew
    u = u, unew

# --------------------------------------
# Finite Difference Method
pnew = (dt**2) * Mfd @ ( f/dx*src[it]+ D @ p) + 2*p #- pold
p = p, pnew

# --------------------------------------
# Animation plot. Display both solutions
if not it % iplot:
    for l in line1:
        l.remove()
        del l
    for l in line2:
        l.remove()
        del l
    line1 = plt.plot(x, v, 'k', lw=1.5, label='FEM')
    line2 = plt.plot(x, p, 'r', lw=1.5, label='FDM')
    plt.legend()
    plt.gcf().canvas.draw()

It has to do with your coding style. Your code is too similar to mathematical notion. Issues include doing too much on single line and operator precedence in Python

Your code:

vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v

Breaking into simpler statements would allow for easier debugging.

term_1 = (f*src[it])  -  (Kv @ v) -  (Ku @ u)
term_2 = term_1 + v
term_3 =  Minv @ term_2
vnew = dt * term_3


What is the difference between all the different types of learning within machine learning? This is a question that is really hard to google, and the differences are confusing. Does anyone have good examples of the differences between them all?

Supervised Learning
Semi-Supervised Learning
Distant Supervision
Active Learning
Lightly Supervised Learning
Weakly Supervised Learning
Competitive Learning
Reinforcement Learning
Unsupervised Learning

Supervised Learning

In general, supervised learning refers to a situation in which you have some $X$ that is related to $y$, so that you can model how both are related (on average). In this case you can build a statistical model:

$$y = \beta X + u.$$

The model learns $\beta$, so that you are able to predict $y$ by: $\hat{y}=\hat{\beta}X$.

Examples are regression or classification tasks (e.g. using linear models, neural nets, boosting etc).

Unsupervised Learning

When you have only $X$ and no response $y$, you face an unsupervised problem. In this case, the focus is to identify patterns in $X$, e.g. by clustering. Examples for unsupervised learning are k-nearest neighbors (KNN) or principle components (PCA).

The book Introduction to Statistical Learning provides a very good overview. Alternatively, the book Elements of Statistical Learning provides a more comprehensive overview.

Reinforcement Learning

Reinforcement learning is a little different. Here you try to find an optimal response to some problem, based on past experiance. Examples are self driving cars or a game of tic tac toe. So for every state of the environment (e.g. the state of the tic tac toe game), RF aims at finding an optimal response (best next move). Googles DeepMind and AlphaGo are RF-based. The book by Sutton and Barto is the standard literature here.


Why use bootstrapping? The wiki page for bootstrapping says that you use it in the case where the underlying distribution is unknown. Why is bootstrapping, or sampling with replacement, better than just calculating the variance and other properties from the data directly?

In general when you are evaluating statistics there are two main approaches: either you use some distributional assumptions (a generating model for your data) or not. Whenever you can you would probably go with a statistical model. But often the distributional assumptions can be wrong or they are not available to you.

If you are doing inferences about the mean of the population, than you can rely on central limit theorem and build a statistic for that. The same for variance. However what about estimating something which does not rely on general assumptions? Bootstrap works for any kind of statistic, this is where it lies its power. It's simple, and does require only minimal assumptions.

And there is another problem which appears in practice. Estimating mean rely on central limit theorem. It is true that your set of assumptions required only independent and identical distributed data. But it's something more. CLT works on large samples, it works in the limit. For small samples it can go wrong more often. And what about if your distribution is skewed. While it's true that CLT relies only on i.i.d. the needed sample size for estimating a mean on a skewed distribution is in general greater that for a symmetric distribution.

I found bootstrapping very useful in two main situations: when the sample is fairly small (but not tiny) and when the distribution is not clean (suppose it's a mixture of two distributions). And is very useful because you can build not only estimates of the population parameters, but also estimations for confidence intervals of p-values. And because it's easy to apply I also found sometimes useful even when you are assuming some model. Doing in parallel can help you to see if there is something really spooky in your sample or assumptions if inferences produces too different results, it makes you think or re-evaluate your setup.


Train-Test split for Time Series Data to be used for LSTM values = df.values
train, test = train_test_split(values)

#Split into train and test
X_train, y_train = train[:, :-1], train[:, -1]
X_test, y_test = test[:, :-1], test[:, -1]

Executing the above code splits the time series dataset into training- 70% and testing 30%. I want to control the train-test split as 80-20 or 90-10. Can someone please help me understand what train[:, :-1] does in this context?
It is borrowed from https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/.
Note : I cannot split the dataset randomly for train and  test and the most recent values have to be for testing. I have included a screenshot of my dataset.
If anyone can interpret the code, please do help me understand the above. Thanks.


Hidden Markov Model on R Studio Is there any detailed materials that can help explain how to set up HMM on R Studio?

You can take a look at here. As you can read from there:

  Download and unpack hiddenDomains

  If you haven't already done this, download the latest version of hiddenDomains from the Sourceforge website

  Now unpack it

shell$ tar -xzvf hiddenDomains.VERSION.NUMBER.tar.gz

  Where VERSION and NUMBER represent the release that you downloaded.

  Install R and required R packages (if necessary)
  Download and install R if you don't already have it.

  If you don't already have them, you will need to install two hidden Markov model libraries:

  depmixS4
  HiddenMarkov
  You can do this by starting R on the command line:

shell$ R

  Or, if you prefer RStudio, you can use that.

  Regardless of how you started R, type the following commands to install the packages.

> install.packages("depmixS4")
> install.packages("HiddenMarkov")

Here and here may also be helpful for knowing how to use it.


Correlation between time to event data and continuous data I want to measure the correlation between the survival time which is a time to event data and the patient's activity count which is measured on continuous scale. What type of correlation coefficient is available to measure the strength of these two variables?

There is nothing wrong with using Pearson's correlation coefficient (frequently called THE correlation coefficient) with these variables. I would also have a look at Spearman's rank correlation coefficient. It is less sensitive to extreme values.

http://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient

However, both coefficients have the drawback that they only measure linear relationships. Hence it is worthwile to have a look at the scatterplot of the variables. Adding a smoother to the graph can be worthwile too, especially  if the plot is clutterd and you have trouble to spot a relationship with the naked eye.