How do I assess which sentiment classifier is best for my project? I wanted to perform Twitter sentiment analysis on Twitter tweets. When I googled it, I found a stanfordNLP code of sentiment class. I used it for finding Twitter sentiment later I came to know TwitterGate model is made for Twitter tweets only.

Then I replaced the stanfordNLP model with TwitterGate model. I did not find any difference in the sentiment results. Both models give the same result.

However, when I tested the same tweets with Monkeylearn API and Datumbox API sentiment, I got confused with results. In NLP and twitterGate most tweets sentiment is given as negative, where the same tweets sentiment is given as neutral in Monkeylearn & Datumbox APIs code.

How can I know which model is giving the correct sentiment?

A couple of important points:

Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared.
The ground truth for sentiment data sets is established by people. The best you can hope for from any trained ML classifier is that it closely matches the opinion of those people when it predicts on new data.
Your different sentiment analysis tools could well vary in quality, because they will have been built with different technologies at different times, and with different training data.

There are a couple of ways I can think of that you could assess the classifiers for your purpose:

1. Test against your own labelled test data

It is important to source this yourself to ensure it does not overlap with training data used by any of the models you want to compare (if it did overlap then it gives a big advantage to any model that trained using it). You will need to collect ground truth data on sentiment for all the texts.

Before you test, it is a good idea to pick a metric that you care about for your intended use. Accuracy is not the only metric, but it is probably fine for your purpose. You might choose other metrics if there are different costs associated with incorrect classifications - e.g. if assigning negative sentiment needs to be done cautiously.

Then run each classifier across your test set and calculate the metric. The classifier with the best metric is your best guess at the one you should use.

If you have a large enough test set, you can split it into parts and get a measure of the estimate error in your metric. This will help you see whether differences in the test metrics are significant.

This can be the best approach, especially if you can source data that you know matches your project goals. However, it would take a long time and a lot of effort.

If you are in a hurry, you could just search for and use someone else's dataset e.g I found these ones on a quick web search: a Kaggle competition and a free sample hosted by Niek Sanders (I have no idea of the quality of these - you should sample them and see if the data is useful for you). There is a risk though that these data sets were used to train one of your classifiers, so it would give you a false high rating.

2. Read up on associated reports and papers

Each of the sentiment analysers should explain the model class and data used to train it in documentation. Quite often you can find papers comparing the different approaches and quoting accuracy scores on a standard data set. If you are lucky, you will find enough comparisons that you get some sense of which algorithms are considered "cutting edge" and which are out-dated.

3. Collect text that is classified differently by algorithms you are considering, and get feedback

A variation of (1), you can do this at any stage, and it might work well in the context of a live product. For all the data that is classified, log how different classifiers respond to it, and where there is discrepancy, save it for later assessment. You will need some people to read and help classify the text (ideally without seeing what the classifiers thought), and you can collect metric data over time and use the classifier that does best once you have collected enough examples to make a clear decision.

Ultimately what matters is the results of using your classifier in a project. Just labelling data has no inherent purpose. It is the consequences of those labels that matter. So you need to be driven by results. If you use one classifier and get feedback that the product is not performing well, you may be able to trial another one. If your product is used by many people, and enough are giving feedback of quality, you could even A/B test a couple of classifiers in production.


Soccer Field Segmentation I would like to develop a soccer field segmentation method. For this purpose, I prepared a training image data set and annotated field and non-field pixels. Following is a gr-chromacity plot of all training samples, colored with respect to their labels.

I want to train a classifier for inferring the label of a new sample. The first approach comes to my mind is using Gaussian mixture models to model both distributions. Would you recommend another method for this purpose?

I would not suggest GMM at this point as the distribution of points in the space is not well-shaped enough. Even if you want to use it it's better to look at your data in PC space (i.e. using PCA). My suggestion would be:

1) Think of your features. What are they? Are you going to use these gr-chromacity as features? If yes you should know that kernel methods work better on this as the features are highly nonlinear. The image show that you need a feature mapping anyway.

2) It seems you have already thought of kernel methods as you put SVM as a tag. you can use it for classification. Might work better than GMM here. Also think of probabilistic graphical models as they have been used intensively for image segmentation and your images are structured enough (a football field has its fixed position in the image anyway).

3) If you have raw labeled dataset, I'd recommend to think of smarter features for segmentation. in gr-chromacity you already loose some information about colors which is the most important thing for you here. I would recommend taking the position of pixels into account as well. Then a PCA on the new data may reveal some more linearly separated classes.


How to measure accuracy of a route prediction I developed a new route prediction algorithm and I am trying to find a metric that informs on how well a prediction was.

This metric is meant to be used offline, meaning that the goal is not to measure the quality of the prediction when it is need in real time. Instead, We are given a set $R=\{r_1,r_2,...r_{|R|}\}$ of routes that occurred in the past and for each $r_i\in R$ we take a small prefix of $r_i$ and provide it as input to the algorithm which in turn outputs a prediction route $p_i$.

Therefore, given the set $R=\{r_1,r_2,...r_{|R|}\}$ and the corresponding predictions set $P=\{p_1,p_2,...p_{|P|}\}$, I want to compare each pair of routes $(p_i,r_i)$ to determine how different $p_i$ is from $r_i$.

Can anyone point me in the right direction?

My first idea was to compute the area comprised between the two routes $p_i$ and $r_i$, but I really do not know how to interpret the output in terms of good or bad result.

Thanks for the attention

You need to define what a "quality route" is in the context of your problem and score based on that. This will likely be something along the lines of the total distance of the route, the expected time to traverse the route, the expected fuel/resource consumption required to complete the route, or some combination of these. You presumably want to predict routes that are as good or better than the routes given in your set R: you need to define what it means for a route to be "better."


Where can I find a crowdsourced dataset for multi-label classification with individual participant labels? I am looking for a dataset that has been crowdsourced, and corresponds to a multiple answer correct per question setting. I need the individual labels given by the participants from the crowd.

I have searched online (such as here), but all I get is datasets where the answers have been aggregated to create a ground truth. I also found sources like this, which provide such a dataset but apply to the single label case.

I would appreciate any help. Thank you.

I'll think you'll find what you're looking for in this question on Open Data. I've checked the Pew Research Center data, and found this poll about cyber-security that seems to be something like what you look for.


How to visualize image segmentation results I am using u-net to do semantic segmentation for N>1 classes. The input size is (128,128,3), the output size will be (128,128,N). what is the correct way see the prediction as an image ot size n1 x n2 x 1 ?

My try: I took the maximum if every (i,j) vector (of size N) in the output tensor to get an scalar and plot a 128 x 128 image.

If you are saying your model produces a vector of length N for each pixel (holding the probability of each class for that pixel), then you are doing the right thing to select the maximum value, of course.

For the plotting, if you leave the image in 128x128, your output will simply be gray-scale and if you are plotting the raw maximum values you took from each vector, I imagine the image will be very noisy.

You need to map the class of each pixel to a predefined colour mapping. For example, if you have the classes: tree, car, building, sky, you should decide on a colour for each of those. For example, specify a dictionary mapping each class to a colour:

colour_mappings = {
    'tree': (255, 255, 255)   # white
    'car': (255, 0, 0)        # red
    'building': (0, 255, 0)   # green
    'sky': (0, 0, 255)        # blue
    }

No when you have the output matrix of size 128x128, you can use filtering to replace each of your classified pixels to the colour you put in the mapping above.

Example

I use the colour mapping above along with a simulated matrix of pixel classification results (for simplicity just 5x5, not 128x128), which is randomly filled with values from [0, 1, 2, 3]:

classes = array([[0, 3, 3, 1, 0],
                 [2, 2, 0, 3, 2],
                 [3, 1, 0, 0, 1],
                 [1, 3, 2, 2, 2],
                 [1, 3, 1, 2, 1]])

Now create an array with three channels, which we will fill based on our results. I just initialise it with zeros, which would be black in colour, which is ok because non of my classes in the mapping above are black (just to avoid confusion):

# initialise an empty (black) matrix to hold the image
segmented_image = np.ones((5, 5, 3))          # we require three channel dims for RGB colour

Let's say the classes above (in the same order) have the numbers 0, 1, 2, 3 - you could then fill the empty image matrix with the colours as follows.

# fill the segmented image based on the class, using your mappings
segmented_image[classes == 0] = colour_mappings['tree']

This will produce the following image, where we see all pixels from the class 'tree' now show the mapped colour, white:

Now we can do the same for the remaining classes:

segmented_image[classes == 1] = colour_mappings['car']
segmented_image[classes == 2] = colour_mappings['building']
segmented_image[classes == 3] = colour_mappings['sky']

This produces the following final image:

Which can be checked against the classes matrix shown above.


What is the purpose of 'oversampling' when the test set is still unbalanced? I understand that both training and testing sets should have the same distribution and also understand that we should not touch the test set (in terms of oversampling). But we know that oversampling the training set (specifically in multiclass classification) totally changes the distribution of the training set. For example:

The distribution of my training set before oversampling is: 90%, 5%, 3%, 2% [for classes A, B, C, and D]
The distribution of my training set after oversampling is: 25%, 25%, 25%, 25% [for classes A, B, C, and D]
The distribution of my training set using stratified cross-validation is: 90%, 5%, 3%, 1% [for classes A, B, C, and D] -->as stratifying keeps the distribution of the original data.

Could someone please explain why do we use oversampling when both training and testing sets need to have the same distribution?

Could someone please explain why do we use oversampling when both
  training and testing sets need to have the same distribution?

We use it because train and test set does not have to have the same distribution of labels. What is important is that the test set comes from the same distribution as your "real" data so that it provides a reliable measure. Post split the training set can be sampled as you see fit, as long as the results are good on your test set.


anomaly detection in vehicle sensor data I am currently diving deeper into understanding more about anomaly detection in regards to vehicle's data generated by sensors.

It seems like there is no proper book or article that goes deeper into this topic.

Hoping to get help on where and how I could find anything regarding this topic. Maybe some of you had to deal with something like this and have acquired some knowledge on where to find some sources.

would really appreciate some help

Anomaly Detection in sensor data can be approached in many ways. A thorough introduction is available in these books:

Anomaly Detection Principles and Algorithms by Mehrotra et.al (2017)
Outlier Analysis by Aggarwal et.al (2017)

These cover the different approaches like distance-based, clustering-based et.c, along with time-series models as well as ensemble models.


Keras CNN image input and output I'm trying to build a CNN similar to this:

For this purpose I chose to use Keras since I worked with it before (simple RNN and FFNN only).
My training data consists of the input data (25 x 25px images) and the output, being 25 x 25px images as well.
The problem is, that I don't know how to build the Model using Keras. How do I specify the output to be of the same shape as the input?

Here is what I have so far:

batch_size = 5
input_shape = (batch_size, 28, 28,  32)

model = Sequential()
model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(1000, activation='relu'))
model.add(Conv2D(64, (5, 5), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1),
                 activation='relu',
                 input_shape=input_shape))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.SGD(lr=0.01),
              metrics=['accuracy'])

I know, that the code will not work like that. I just want to clarify what I'm trying to do. The code above was taken from: Keras tutorial – build a convolutional neural network in 11 lines

As of now I just tried to reverse the code from the point, where the tutorial would output the predicted label for the given image to transform the result back to the wanted output image.

I know this question was asked sometime ago. I took a look at the diagram and the example code. First, the example code is way off. Here's a description to help those curious how to understand the diagram.

The diagram (architecture) shows only the frontend of a CNN. That's the convolutional layers. On the leftside are a series of VGG convolution blocks. The links to the right side are identity links between the VGG convolution blocks, which pass through their own convolution before being matrix added into the output of the convolution block. The addition of the convolution on the identity link makes it a variant of a ResNet architecture.

The UV 224x224x2 would be the output from the convolutional frontend. It would then be flattened and passed into the DNN layers.


Charts with or without grids? I am writing a thesis and don't know if the charts below should have grid lines or not. Do they need the grid lines?

An example without the grid lines:

I would say that it is really up to you here. I don't think that the grid lines are distracting in any way, but can be helpful if you are trying to show more exact results. For example, it if it important that the scatter plot shows exact points then keep them. However, if you are demonstrating more of a general trend, then you could go without them. I'm not sure if these are your finished graphs or not, but make sure to label the color differences in the lines and provide a title. Also, is there a reason why your x-axis is on the top of the graph? If you want to keep it there and use the grid lines, try to have the numbers be above the graph instead of being slightly blocked by them.


What are good sources to learn about Bootstrap? I think that Bootstrap can be useful in my work, where we have a lot a variables that we don't know the distribution of it. So, simulations could help.
What are good sources to learn about Bootstrap/other useful simulation methods?

A classic book is by B. Efron who created the technique:

Bradley Efron; Robert Tibshirani (1994). An Introduction to the Bootstrap. Chapman & Hall/CRC. ISBN 978-0-412-04231-7.


Python - Get FP/TP from Confusion Matrix using a List I using two different classifiers to predict a binary target (Random Forests and Decision Trees). Now I want to evaluate my model creating a confusion matrix. For example, for predicting the binary value using random forests I've:

training_features, test_features, training_target, test_target, = train_test_split(df.drop(['score_goal'], axis=1),
                                              df['score_goal'],
                                              test_size = .3,
                                              random_state=12)

clf_rf = RandomForestClassifier(n_estimators=25, random_state=12)
clf_rf.fit(training_features, training_target)
print("Accuracy using Random Forest Classifier is ", clf_rf.score(test_features, test_target)*100)

I'm confusing because I don't know how I can compare the predicted values to identify how many False Positives, etc. I have.

Anyone knows how can I build that function?

Thanks!

Looks like you're using scikit-learn. So why not explore a bit more? Scikit has a metrics module, that can be of use for your problem. Essentially, what you need is to have two separate arrays - one with real labels and another with predicted labels. And then you're good to go, you could call metrics.classification_report or metrics.confusion_matrix or metrics.accuracy_score, all of them use the real labels and the predicted labels.

There's nothing wrong with using clf_rf.score(test_features, test_target), but it will only give you a single value. If you look at the source code, what happens is that the score method calls a predict method with test_feature for prediction of labels, which occurs behind the scenes.

It's better to actually capture those predicted labels, so that you can reuse them.

clf_rf.fit(training_features, training_target)
predicted_target = clf_rf.predict(test_features)
accuracy = sklearn.metrics.accuracy_score(test_target, predicted_target)
cnf_matrix = sklearn.metrics.confusion_matrix(test_target, predicted_target)
class_report = sklearn.metrics.classification_report(test_target, predicted_target)

And then you could do whichever you like with the calculated metrics, print, plot, etc.
Have a look at the examples that are included for each model/metrics you're using 1, 2, 3, 4, etc.


ValueError:invalid literal for int() with base 10: 'No'' Trying to do a binary classification and have the class names as string but when it gives me the error mentioned in the title. I tried to give integer labels as 0 and 1, the code works perfectly fine. I tried all the suggestions from stackoverflow regarding the same error but nothing is working.Here is the code snippet:

labels = ["No", "priorlocaltx"]
#labels = [0, 1]
keyword_identifiers_no = ['surgery', 'radiotherapy', 'brachytherapy', 'Newadjuvant', 'neo', 'Adjuvant', 'Mets at first diagnosis', 'M1HSPC']
keyword_identifiers_yes = ['no prior tx', 'BCR', 'Neoadjuvant', 'neo', 'Adjuvant', 'Hormone refractory', 'Mets', 'No mets']

cancer_df.insert(2, 'Result', 0, allow_duplicates = False)
#print(cancer_df.head())
for i in range(1, len(cancer_df)):

    for word in keyword_identifiers_no:
        if word in cancer_df.iloc[i]['Patient Segment(s)']:

            cancer_df.at[i,'Result'] = labels[0]
            #print(cancer_df['Patient Segment(s)'].dtype)

    for word in keyword_identifiers_yes:
        if word in cancer_df.iloc[i]['Patient Segment(s)']:
            cancer_df.at[i,'Result']= labels[0]

    cancer_df.to_csv('Result1.csv')

It seems that your 'Result' column is of type int and you are trying to set some rows to str. Stick to int or category (docs).


One hot vector output in classification task I'm working on CNN model and I used one hot vector type of labels. The number of classes is 3: [1,0,0], [0,1,0], [0,0,1].
net(x)

I'm getting such an output: [0.8439, 0.1355, 0.0757], which is obviously 1st class. The question: why a sum of values in this vector exceeds 1? Also, I got earlier even one negative value of those 3. On what it is depending and how to know what these "outputs" could be.

Here it's a mistake to one-hot-encode the class, because it turns the task into multi-label classification instead of regular multi-class classification. In your task an instance can only have a single class, so the class should be encoded as an int (for example with LabelEncoder).
This is why the predicted probabilities don't sum to 1, because in multi-label classification the classes are independent of each other. For example the output [0.9,0.4,0.7] means that class 1 label is predicted true at 90%  and false at 10%, class 2 label is predicted true at 40% and false at 60%, class 3 label is predicted true at 70% and false at 30%. It wouldn't make sense to pick the maximum probability of the 3 classes in this scenario.
See also this answer.


Why can it be that my neural network is predicting the contrary? I did the coursera deep learning course where as an assignment you have to complete a few functions of a neural net. Everything worked great so I tried to implement it from scratch.

Did it all and when I tested it it behaved really strange. I removed the hidden layers and left only one neuron on the output one so its basically logistic regression.

I'm training it with the following:

X = [[-3], [-2], [3], [2]]

Y = [[0], [0], [1.], [1.]]

but when I use that same X to check, it predicts aprox. [[1.], [1.], [0], [0]]

the bias is close to 0 and the single weight is -27. After each iteration the cost increases from 0.7 to 29.9 where it stays for the last 7 epochs.

If I change Y to [1, 1, [0], [0]] the prediction is 0.5 for all instances, and both bias and weight is close to 0. With this Y configuration the cost stays at 0.69 through all iterations.

If I change the update line from w -= alpha * grad to += the situation is the same but all the way round.

This is my gradrient function

    def gradients(self, X, Y):
    dws = []
    dbs = []
    h, activations, zs = self.forward(X)
    da = - (np.divide(Y, h) - np.divide(1 - Y, 1 - h)) #dJ/dOutput
    for d in np.arange(1, len(self.weights))[::-1]:
        a = activations[d] #output for layer d
        z = zs[d] #linear for layer d
        w = self.weights[d] #weights for layer w

        dz = da * a * (1-a)
        m = w.shape[1]
        dw = 1./m * dz.dot(activations[d].T)
        db = 1./m * np.sum(dz, axis=1, keepdims=True)
        da = w.T.dot(dz)
        dws.insert(0, dw)
        dbs.insert(0, db)
    dws.insert(0, None)
    dbs.insert(0, None)
    return dws, dbs

I'm using an empty weight at 0 so its easier to match the layer's activations, zs and weights.

For all layers the activation function is sigmoid. Once it works I'll see if I change the hidden to ReLu.

Here is the whole code:
GitHub

After being all day trying to figure this out I had to post this question to find clarity.
The problem is here

dw = 1./m * dz.dot(activations[d].T)

it should be

dw = 1./m * dz.dot(activations[d-1].T)


Caffe net.predict() , predict same probability I trained pre-trained ResNet-101 following caffe model(By Kaiming He github) with 800000 data for training and 200000 data for validation. After I train this model, I got 59% accuracy for 1st and accuracy-top5 is 82% with 30 epoch as seen below picture.

But when I tried predict some images with this model(net.forward()), the results always produce same probability like below even though I tried with other images.

First thing I thought was image preprocessing problem in predict step like subtracting mean values or adequating batch size corresponding with training step. But all of these step was correctly set up.
I checked all other questions having a same problem with me but couldn't find a solution. As the above picture(1) is showing, I assume the training process wasn't something wrong.

I followed "Image Classification and Filter Visualization(00-classification.ipynb)" file provided from Caffe editing model_def and model_weights with my model_def and model_weights having a 30 epoch.

I solved this problem. The problem was use_global_stats setting in deploy.prototxt. In training step, use_global_stats has to be set as false because mean/var need's to be update. But when I predict using deploy.prototxt use_global_stats has to be set as true.


Looking for example infrastructure stacks/workflows/pipelines I'm trying to understand how all the "big data" components play together in a real world use case, e.g. hadoop, monogodb/nosql, storm, kafka, ... I know that this is quite a wide range of tools used for different types, but I'd like to get to know more about their interaction in applications, e.g. thinking machine learning for an app, webapp, online shop.

I have vistors/session, transaction data etc and store that; but if I want to make recommendations on the fly, I can't run slow map/reduce jobs for that on some big database of logs I have. Where can I learn more about the infrastructure aspects? I think I can use most of the tools on their own, but plugging them into each other seems to be an art of its own.

Are there any public examples/use cases etc available? I understand that the individual pipelines strongly depend on the use case and the user, but just examples will probably be very useful to me.

In order to understand the variety of ways machine learning can be integrated into production applications, I think it is useful to look at open source projects and papers/blog posts from companies describing their infrastructure.

The common theme that these systems have is the separation of model training from model application. In production systems, model application needs to be fast, on the order of 100s of ms, but there is more freedom in how frequently fitted model parameters (or equivalent) need to be updated.

People use a wide range of solutions for model training and deployment:

Build a model, then export and deploy it with PMML

  AirBnB describes their model training in R/Python and deployment of PMML models via OpenScoring.
  Pattern is project related to Cascading that can consume PMML and deploy predictive models.

Build a model in MapReduce and access values in a custom system

  Conjecture is an open source project from Etsy that allows for model training with Scalding, an easier to use scala wrapper around MapReduce, and  deployment via Php.
  Kiji is an open source project from WibiData that allows for real-time model scoring (application) as well as functioanlity for persisting user data and training models on that data via Scalding.

Use an online system that allows for continuously updating model parameters.

  Google released a great paper about an online collaborative filtering they implemented to deal with recommendations in Google News.


Association between features Given the anonymized dataset of features below, where:

"code" is a categorical variable.
"x1" and "x2" are continuous variables.
"x3" and "x4" are extracted features. They are the mean values of "x1" and "x2" respectively for each individual code.

       code  x1  x2  x3  x4
    0   100   1   2   2   4
    1   100   2   4   2   4
    2   100   3   6   2   4
    3   200   4   8   5  10
    4   200   5  10   5  10
    5   200   6  12   5  10
    6   300   7  14   8  16
    7   300   8  16   8  16
    8   300   9  18   8  16

Looking at the columns, for each code, x3 and x4 features have similar values - when x3 is 2 or 5 or 8, the code would be 100 or 200 or 300 respectively, and when x4 is 4 or 10 or 16, the code would be 100 or 200 or 300 respectively.

Intuitively, leaving these columns as they are without dropping any would lead to redundant features while training a model. My question is how true is this my hypothesis? I'm not so confident about it. Does it really matter when training a model? Does it depend on the model type (tree based or otherwise)?

1) Those extracted features are "combination" of original features. Yes, it's additional information. And yes, it could be redundant, but I'd check that after creating those features by doing standard feature selection operations. Here's a great guide from Kaggle. At the same time, it might be a very good feature engineering decision - completely depends on a dataset.

2) Yes, it matters - redundant information can negatively influence model performance. But you find out to understand whether it's redundant or not. The minimum bad thing you can face - is how much time does training takes (more features - more computation resources). The more serious thing is overfitting.

3) Yes, and it depends on how your model is able to process categorical features. If it cannot do anything specific with them, you should remove the original categorical features. XGboost or CatBoost (those are tree-based both) are able to process categorical features from the box. Generally, tree-based models work good with a lot of features, from my experience. And linear models require right feature selection.

Also, here's a link to a great example on Target Feature encoding, it might be also useful, however, you must be very attentive to train/test splits and overfitting using it.


Confidence score for all observations is between 0.50 - 0.55 Hello Data Science Stack Exchange Community,
This question will appear to be open-ended, however any answers or thought will be much appreciated. I am trying to go-through a pre-trained Random Model Classifier with minimum documentation like what was the confusion matrix, ROC-AUC curve of the classification problem when the model was developed. I only have the pickle file and the data set on which it needs to run. When I ran the model I observed for most of the cases the prediction score or confidence score is between 0.50 to 0.55
I would like to know is it right to say that "model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55"
Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.
Again any thoughts and answers will be much appreciated.

Random Model Classifier

It feels like there's a word missing here, you probably mean "Random Forest [model] classifier" or "Random Fields classifier"?

the confusion matrix

This is an important piece of information, because if you have a new (annotated) sample this allows you to compare a few things:

compare the distribution of the classes in the original test data and the new dataset
calculate the expected performance by class, analyze the differences if you evaluate the model against the new sample
...

I would like to know is it right to say that "model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55"

It would be technically correct to say that indeed, but maybe not for the reasons that you mention (see below). While this predicted value is often called a confidence score or probability, it doesn't necessarily represents the confidence of the model in a normalized range [0,1] like one would expect, even though ideally it's supposed to. The details depend on the exact type of model, but in general in ML the model is only as good as the data it's trained with so most of the time the predicted score is just the "best guess" of the model. So it shouldn't be interpreted as "probability of class C", and anyway in the case of classification the only use of the value is to compare two instances x and y relatively to each other: if x has a higher predicted score than y, then the model considers x more likely than y to be in class C. The actual value doesn't matter and it's better not to rely on it.

Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.

This conclusion is incorrect. To be clear, it might be true in some particular case, but one cannot conclude this from the information given in the question. Why? Because the ROC curve (and the AUC) don't depend at all on the actual values of the score, they depend only on the order of the instances according to the score. The ROC curve shows the performance for different theresholds separating the predicted positive/negative instances. Whether the scores range in [0.51,0.52], [0,1] or even [-1234,+5678] doesn't have any influence, it's the proportion of negative instances under the threshold and positive ones above it which determine the performance at any given point.
In case you're interested, I gave some more detailed explanations about ROC curves in this answer.


Extract company names/job titles from free text I have a complete Hadoop platform with HDFS, MR, Hive, PIG, Hbase, etc., Python, R, Java. All data sets have a large size.

The data set A, describing the jobs of people working in a company, is composed of the following fields:

Id Person: a unique alphanumeric identifier per person.
Start Date: a date format iso entry in the post
End Date: iso size release date of the position. If the date is not given, it is the current position
Job Title: a text field containing the title and the name of the company. The text is free, non-standardized, French and / or English and can contain typos. Ex: Director Big Data Analytics with Google, Commercial Manager at [missing text] , Manager at googole ...

My question is; how can I create a feature to easily process the name of company of the job (jobtitle)?

Thank you in advance

I think what you want is to extract company names from "Job Title". In natural language process, we call this kind of research as "Name Entity Recognition(NER)". You can try to use  Stanford Named Entity Recognizer (NER)[http://nlp.stanford.edu/software/CRF-NER.shtml]. Stanford NER performs very well on English contents and there are lots packages for many programming language:

  UIMA: Florian Laws made a Stanford NER UIMA annotator using a modified version of Stanford NER, which is available on his homepage. [Old version.]

  Perl: Kieren Diment has written Text-NLP-Stanford-EntityExtract, a Perl module that provides an interface to Stanford NER running as a server.

  Ruby: tiendung has written a Ruby Binding for the Stanford POS tagger and Named Entity Recognizer.

  Python: Dat Hoang wrote pyner, a Python interface to Stanford NER. [Old version.] NLTK (2.0+) contains an interface to Stanford NER written by Nitin Madnani: documentation (note: set the character encoding or you get ASCII by default!), code, on Github.

  F#/C#/.NET: Sergey Tihon has ported Stanford NER to F# (and other .NET languages, such as C#), using IKVM. See also pages on: GitHub and NuGet.

  PHP: PHP-Stanford-NLP. Supports POS Tagger, NER, Parser. By Anthony Gentile (agentile).

If you are not satisfied with the performance of Stanford NER, you can also train you own models to extract company names by crawl company names from several popular sites with company names, such as Linkedin/Facebook/Glassdoor...etc


Normal equation for linear regression is illogical Currently I'm taking Andrew Ng's course. He gives a following formula to find solution for linear regression analytically:
$θ = (X^T * X)^{-1} * X^T * у$
He doesn't explain it so I searched for it and found that $(X^T * X)^{-1} * X^T$ is actually a formula of pseudoinverse in a case where our columns are linear independent. And this actually makes a lot of sense. Basically, we want to find such $θ$ that $X * θ = y$, thus $θ = y * X^{-1}$, so if we replace $X^{-1}$ with our pseudoinverse formula we get exactly the $θ = (X^T * X)^{-1} * X^T * y$.
What I don't understand is why nobody mentions that this verbose formula is just $θ = y * X^{-1}$ with pseudoinverse. Okay, Andrew Ng's course is for beginners and he didn't want to throw a bunch of math at students. But octave, where the assignments are done, has function pinv() to find a pseudoinverse. Even more, Andrew Ng actually mentions pseudoinverse in his videos on normal equation, in the context of $(X^T * X)$ being singular so that we can't find its inverse. As I mentioned above,  $(X^T * X)^{-1} * X^T$ is a formula for pseudoinverse only in case of columns being linear independent. If they are dependent (e.g. some features are redundant), there are other formulae to consider, but anyway octave handles all these cases under the hood of pinv() function, which is more than just a macro for  $(X^T * X)^{-1} * X^T$. And Andrew Ng instead of saying to use pinv(X) * y gives this: pinv(X' * X) * X' * y, basically we use a pseudoinverse to find a pseudoinverse. Why?

Hello Oleksii and welcome to DSSE.
The formula you are asking about is not for a pseudoinverse.
$\theta = (X^TX)^{-1}X^Ty$
Where

$\theta$ is your regressor
$X$ is a matriz containing stacked vectors (as rows) of your features/independent variables
$y$ is a matriz containing stacked vectors (or scalars) of your predictions/dependent variables

This equation is a solution to a linear set of equations:
$Ax = B$ that occurs in trying to minimize the least squares loss.
The reason why you see this pinv() on the code is because if X has not enough linearly independent rows, $X^TX$ (also known as $R$, the autocorrelation matriz of the data, and it's inverse is called the precision matriz) will result in a singular (or near singular) matriz, which inversion might not be possible. Even if this singularity only happens because of the working precision of your computer/programming language.
Using pinv() is usually not recommended, because even if it allows to compute a regressor it will overfit the training data. Alternative solutions to working with singular matriz is adding $\delta~I$ to $R$
where $\delta$ is a small constant (usually 1 to 10 eps) and $I$ is the identity.


Advantages of stacking LSTMs? I'm wondering in what situations it is advantageous to stack LSTMs?

From What are the advantages of stacking multiple LSTMs? (I'll only update the answer there):

  From {1}:

    While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that
    deep RNNs work better than shallower ones on some tasks. In
    particular, Sutskever et al (2014) report that a 4-layers deep
    architecture was crucial in achieving good machine-translation
    performance in an encoder-decoder framework. Irsoy and Cardie (2014)
    also report improved results from moving from a one-layer BI-RNN to an
    architecture with several layers. Many other works report result using
    layered RNN architectures, but do not explicitly compare to 1-layer
    RNNs.

  References:

  {1} Goldberg, Yoav. "A Primer on Neural Network Models for Natural Language Processing." J. Artif. Intell. Res.(JAIR) 57 (2016): 345-420.
  https://scholar.google.com/scholar?cluster=3704132192758179278&hl=en&as_sdt=0,5
  ; http://u.cs.biu.ac.il/~yogo/nnlp.pdf


How to save prediction values for the whole data in Keras I am using pre-trained VGG16 model to classify images located in the folder. Currently, I am able to classify only one single image.

How can I modify the code to classify all the images in the folder
How can I save the prediction values for each image ?

Below is my code :

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.applications.imagenet_utils import decode_predictions
import matplotlib.pyplot as plt

filename = 'cat.jpg'
# load an image in PIL format
original = load_img(filename, target_size=(224, 224))
print('PIL image size',original.size)
plt.imshow(original)
plt.show()

# convert the PIL image to a numpy array
# IN PIL - image is in (width, height, channel)
# In Numpy - image is in (height, width, channel)
numpy_image = img_to_array(original)
plt.imshow(np.uint8(numpy_image))
plt.show()
print('numpy array size',numpy_image.shape)

# Convert the image / images into batch format
# expand_dims will add an extra dimension to the data at a particular axis
# We want the input matrix to the network to be of the form (batchsize, height, width, channels)
# Thus we add the extra dimension to the axis 0.
image_batch = np.expand_dims(numpy_image, axis=0)
print('image batch size', image_batch.shape)
plt.imshow(np.uint8(image_batch[0]))

# prepare the image for the VGG model
processed_image = vgg16.preprocess_input(image_batch.copy())

# get the predicted probabilities for each class
predictions = vgg_model.predict(processed_image)
print (predictions)

# convert the probabilities to class labels
# We will get top 5 predictions which is the default
#label = decode_predictions(predictions)

Thank you

First, you could always just wrap you code with a loop.

files = [file1, file2, ...]
predictions = []
for file in files:
    original = load_img(file, target_size=(224, 224))
    numpy_image = img_to_array(original)
    image_batch = np.expand_dims(numpy_image, axis=0)
    processed_image = vgg16.preprocess_input(image_batch)  # you dont need to copy :)      predictions_cur = vgg_model.predict(processed_image)
    print(predictions_cur)
    predictions.append(predictions_cur)
    #label = decode_predictions(predictions_cur)

Secondly, you could improve the efficiency of this code by using the tf.data api.
For example:

from tensorflow.data import Dataset

batch_size = 32
num_threads = 4
target_size=(224, 224)

def preprocess_image(file):
    # file is not a tensor, so we have to use functions that work on tensors
    img_string = tf.read_file(file)
    img = tf.image.decode_png(img_string)
    img = tf.image.resize_images(target_size)
    img = vgg16.preprocess_input(img)  # this function can handle tensors as well.
    return img

files = [file1, file2, ...]
dataset = Dataset.from_tensor_slices(tf.constant(files))
dataset = dataset.map(preprocess_image, num_threads)
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(1)
predictions = vgg_model.predict(dataset, steps=len(files) // batch_size)
# You could decode the predictions from here :)

Explanation

Let me elaborate on each step :)

dataset.from_tensor_slices(files) Will just build a dataset object, where the samples it will use are your files.

dataset.map(preprocess_image, num_threads)
This indicates to preprocess each sample before we use it.
This will also be made in num_threads parallel calls, which will haste the process in case we batch samples together for prediction.

dataset = dataset.batch(batch_size)
This will batch samples together for prediction. This is useful to make predictions in parallel and haste the process.

dataset.prefetch(1)
This will prefatch the next batch while the previous one is being predicted upon. So this will also make the process faster because you save time this way.

and finally predictions = vgg_model.predict(dataset, steps=len(files) // batch_size)
This will predict over the all the files we've provided to the dataset above.
Do notice that we indicate that the number of steps is len(files) // batch_size so if len(files) isn't a multiply of batch_size, some images will not be predicted upon. If you want all the images to be predicted upon, you can use a batch_size that divides len(files), or just use batch_size = 1.


How to apply k means in my dataset? I have to detect anomalies from my data-set. The anomaly is about in which area and in which time of network usage(total_activity in my data) drastically improved. Help me to know how to apply k-means for this data-set.

Kmeans only works on numerical data, so

throw you categorical data out,
standardize your numerical data,
calculate a distance matrix for a selected metric (for ex. dist() with euclidean),
run kmeans(distance_matrix,k=3) (try k=3 to say k=7)
look at model$cluster to get the data point classes


Does CRNN use sparse tensor value for its label? I just read paper about cnn + rnn for text recognition. The labels of dataset is tensor of char index (e.g [0, 1, 2 ] for image with label "abc").  Since the label of each input has different length do I need to convert the label to sparse tensor value ? since the paper does not mention about it.

My assumption here is that if the input vectors are of variable length, you should use something like keras.preprocessing.sequence.pad_sequences  (as well as a masking layer to ignore these padded values) as a means of reformatting all input samples to the same size so that the input to your recurrent model meets the requirement of having a fixed num_timesteps length.
Function Documentation:

Pads sequences to the same length.
This function transforms a list of num_samples sequences (lists of
integers) into a 2D Numpy array of shape (num_samples, num_timesteps).
num_timesteps is either the maxlen argument if provided, or the length
of the longest sequence otherwise.
Sequences that are shorter than num_timesteps are padded with value at
the end.
Sequences longer than num_timesteps are truncated so that they fit the
desired length. The position where padding or truncation happens is
determined by the arguments padding and truncating, respectively.


Financial Time Series data normalization I'm using Keras in R to predict financial time series. It's easy to normalize price, simply compute returns or log returns, usually it's enough. I want to use Goldman Sachs Financial Conditions Index and MSCI World Index to predict other securitites and I want to use levels and their returns or first differences.  I think that using minmax or z-score normalization is not appropriate because series distribution will change. So, the question is how to normalize nonstationary time series data?

Z-score normalization, as you have already guessed, cannot deal well with non-stationary time series since the mean and standard deviation of the time series vary over time.

Min-max and another commonly used normalization in stationary data, the decimal scaling normalization depend on knowing the maximum values of a time series.

The most commonly used method for data normalization of non-stationary time series  is the sliding window approach (J. Lin and E. Keogh, 2004, Finding or not finding rules in time series). In short:

  The basic idea of this approach is that, instead of considering the
  complete time series for normalization, it divides the data
  into sliding windows of length ω, extracts statistical
  properties from it considering only a fraction of ω
  consecutive time series values (H. Li and S. Lee, 2009, Mining frequent itemsets over data streams using efficient window sliding techniques, Expert Syst. Appl., v. 36, n.
  2, p. 1466-1477. J.C. Hull, 2005), and normalizes each
  window considering only these statistical properties. The
  rationale behind this approach is that decisions are usually
  based on recent data.
  (...)
  The sliding window technique has the
  advantage of always normalizing data in the desired range.
  However, it has a drawback of assuming that the time series
  volatility is uniform, which is not true in many phenomena

Another more advanced and less used (so far) is Adaptive Normalization

  can be divided into three stages: (i)
  transforming the non-stationary time series into a stationary
  sequence, which creates a sequence of disjoint sliding
  windows (that do not overlap); (ii) outlier removal; (iii) data
  normalization itself.

Check the link on Adaptive Normalization and all its references, there is relevant information in there.


Does ensemble (bagging, boosting, stacking, etc) always at least increase performance? Ensembling is getting more and more popular. I understand that there are in general three big fields of ensembling, bagging, boosting and stacking.

My question is that does the ensembling always at least increase the performance in practice? I guess mathematically, it is not true. I am jsut asking a real life situation.

For example, I could train 10 base learner, and then stack them with another learner, which is at 2nd level. Does this 2nd-level learner always outperform the best of the base learners in practice?

The short answer is no.

I have worked on several projects that evaluated an ensemble of several classifiers versus the classifies themselves.  In some cases the precision and recall was better with the ensemble, but more often, it was not.  That's not to say it's not worth investigating.  But sometimes, there is one model that does a reasonable job of classifying data, but it can get drowned out in an ensemble.  Perhaps a weighted ensemble might improve the results, but its not a clear cut approach to improving the performance.

In practice, I would try several models and then try an ensemble of the models.  If the ensemble is the best, however you define best, go with it.  But sometimes it is easier to just pick the best base model and then figure out how to tune that model.


binning high cardinality categorical features one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently occurring values (with x depending on the frequency distribution), and then create a binary flag feature like 'is_in_top_x_us_cities'. Or creating one-hot features for each of the top x cities.

Can someone explain the relative disadvantages to this approach as opposed to using something like Weight of Evidence binning?

The obvious disadvantage is that you lose information.

Probably the best approach is to generate new features based on the city, using domain knowledge: size of the cities, but also relevant geographical/societal/economic features of those cities.  Without domain knowledge and additional data, maybe clustering on your data can help to group together similar cities (though now your model will see that same information in the city-groups as well as in the data you used to create the clusters).

Weight of evidence is sort of like clustering similar cities, but according only to your target variable.  If you have cities with very few samples, then the WoE scores will suffer from high variability, so you'll probably want to lump them together anyway.


What does IBA mean in imblearn classification report? imblearn is a python library for handling imbalanced data. A code for generating classification report is given below.
import numpy as np
from imblearn.metrics import classification_report_imbalanced
y_true = [0, 1, 2, 2, 2]
y_pred = [0, 0, 2, 2, 1]
target_names = ['class 0', 'class 1', 'class 2']
print(classification_report_imbalanced(y_true, y_pred,target_names=target_names))

The output for this is as follow
                pre       rec       spe        f1       geo       iba       sup

 class 0       0.50      1.00      0.75      0.67      0.87      0.77         1
 class 1       0.00      0.00      0.75      0.00      0.00      0.00         1
 class 2       1.00      0.67      1.00      0.80      0.82      0.64         3

avg/total      0.70      0.60      0.90      0.61      0.66      0.54         5

What is the meaning of iba in this classification report. Here pre stands for precision, rec stands for recall, spe stands for specificity, f1 stands for f1 measure, and geo stands for geometric mean. All these are metrics for measuring performance of imbalanced classes.

If you look at the imblearn documentation for classification_report_imbalanced, you can see that iba stands for "index balanced accuracy". For more information on what the index balanced accuracy is and it's value in cases on imbalanced datasets, have a look at the original paper.


TFIDF for very short sentences I'm trying to build a regression model, in which one of the features contains text data. I was thinking in using scikit-learn's  sklearn.feature_extraction.text.TfidfVectorizer. The issue however, is that the actual strings contain very few words. Exactly 1.8 in average. Here's a sample:

 print(df.keyword)

0 fre lifeproof
1 car stereo
2 analog clock
3 refrigerator

So my question is,

is the TfidfVectorizer also suited for such case? Or will such sparse resulting matrix not be benefiting the model?
Is there a more suited approach for such case?

It depends what it's going to be used for, but in general it can make sense to use TF-IDF with short sentence. The main difference with the more standard case of long sentences is that TF (Term Frequency) won't play any role since the frequency will almost always be 1. IDF can still be useful though, assuming it's relevant to assign more weight to rare words than to frequent words.

However the problem is that comparisons such as cosine are often going to be zero, since there will be little chance of words in common between two sentences.


Difference between Gensim word2vec and keras Embedding layer I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the same thing, they all try to convert a word into a feature vector.

Am I understanding this properly? What exactly is the difference between these two methods?

Thanks!

Yep, you're right!  As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation.  This process is called word embedding, and finding word embeddings is the task of the keras Embedding layer.

Ideally, word embeddings will be semantically meaningful, so that relationships between words are preserved in the embedding space.  Word2Vec is a particular "brand" of word embedding algorithm that seeks to embed words such that words often found in similar context are located near one another in the embedding space.  The technical details are described in this paper.

The generic keras Embedding layer also creates word embeddings, but the mechanism is a bit different than Word2Vec.  Like any other layer, it is parameterized by a set of weights.  The weights are randomly-initialized, then updated during training using the back-propagation algorithm.  So, the resultant word embeddings are guided by your loss function.

To summarize, both Word2Vec and keras Embedding convert words (or word indices) to a hopefully meaningful numeric representation.  Word2Vec is an unsupervised method that seeks to place words with similar context close together in the embedding space.  Keras Embedding is a supervised method that finds custom embeddings while training your model.


How to label "other" while labeling image for object detection/classification? I want to train a model to recognize the different categories of food e.g. rice, burger, apple, pizza, orange and other things.

After the first training, I realized that the model is detecting other objects as food. e.g. hand as fish, phone as Chocolate, person as candies.

I get a very low loss because the testing dataset and validation must have at least a pictures of food. But when it comes to a picture of an object other than food, the model fails. How do label the dataset in a way that the model will not do any detection if there is no food on the picture?

This highly depends on your test data. Suppose you have trained your data which all contains some kind of food. If you give it a hand at test time, it will try to find the most similarity it has with the current labels and outputs the one which is more similar than the others. The point is that your test environment is really important. If you are going to test your app near the sea, your training data set should contain data on that condition. If you are going to use it during the night, your dataset should be taken during the night. In your case, you have a simple task, for the current dataset add an entry with a zero which means you are trying to add a new class. After that, try to add new pictures that are not images of food and add one in the new entry for these images. But consider this important point. The number of images in each class should have the same distribution as the test data.


keras' ModelCheckpoint not working I'm trying to train a model in keras and I'm using ModelCheckpoint to save the best model according to a monitored validation metric (in my case the Jaccard index).

While I can see the model improving in tensorboard, when I try to load the weights and evaluate the model it isn't working at all. Furthermore, by the timestamp on the file where the weights are supposed to be stored, I can tell that they are not being saved at all. The timestamp corresponds roughly to the time I started training.

Has anyone encountered such a problem before?

Do you run ModelCheckpoint on its default parameters (besides monitor)?

ModelCheckpoint has a parameter called mode which specifies the type of metric to be used. mode can take 3 values 'min' 'max' and 'auto' (which is the default):

min: means that you want to minimize the metric (e.g. the loss function).
max: means you want to maximize the metric (e.g. accuracy).
auto: attempts to figure what to do on its own. If you look at the code, it checks if the metric's name contains 'acc' or if it starts with 'fmeasure'. If yes it sets the mode to max, if not it sets it to min.

In your case, you monitor the jaccard index, which is a metric you want maximized, so you want the mode set to max. Normally because "jaccard" contains the string "acc", even if the mode is set to auto it should work fine.

If however you named your metric something arbitrary (e.g. my_metric), the default mode will be set to min, which means that it will store the weights that achieve the least performance on your metric, which should be the weights of the first epoch.

Suggestion: next time try with mode='max' to be sure.


Is the magnitude of the gradient a weakness of Gradient Descent? The formula for Gradient Descent is as follows:
$$
\mathbf{w} := \mathbf{w} - \alpha\; \triangledown C
$$

The gradient itself points in the direction of steepest ascent, therefore it is logical to go in the opposite direction by subtracting it. But besides the direction, the gradient also has a magnitude which actually doesn't say anything about the path to the optimum.

My question is, is this considered a weakness of gradient descent and the reason for the learning rate?

The magnitude doesn't say anything about the path to the optimum, in fact nothing in gradient decent knows anything about the optimum and the algorithm relays on local information only. However if you assume that the space of available solutions is relatively smooth and doesn't contain sharp drops, you can treat the magnitude of the gradient as a score of your confidence in your current surrounding.

So if the gradient magnitude is high, you are fairly certain that you are far from a good solution and you can make big steps and if the magnitude is small, than you are closer to a good solution and should make smaller changes.

The learning rate is just a hyper-parameter that controls how much we are adjusting the weights of our network with respect that said loss gradient. It can also be viewed in terms of your confidence in the surrounding solutions space. It also helps the model to converge deeper into a specific minimum point (as we reduce the learning rate during training).


A Derivation in Combinatory Categorial Grammer I am reading about CCG on page 23 of Speech and Language processing. There is a derivation as follows:

(VP/PP)/NP , VP\((VP/PP)/NP) => VP?

Can anyone example this please? This make sense if

VP\((VP/PP)/NP) is equivalent to (VP\(VP/PP))/NP

and

(VP/PP)/NP is equivalent to VP/(PP/NP).

But they seem at least non-trivial from the text!

Any help would be greatly appreciated.

CS

I guess I got it. It is simply:

(X/Y)(Z\(X/Y)) which Z.

In the example,

X = VP/PP, Y = NP, and Z = VP.


What exactly is the linear layer in the transformer model? Please see this image:

There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:

Are these linear layers simply dense or fully connected layers? Let's consider the weight matrix Wi Q. Does this represent a dense layer with "Q" nodes? As they are using matrices as input rather than 1D vectors, I am getting a little confused.

In the original Transformer article, these linear layers are just matrix multiplications.
As described in the paragraph you referred to in your question $W^Q$ is a matrix of dimensions $d_{model} \times d_k$, that is, it is a fully connected layer with $d_k$ units.
In practical implementations, these have the optional addition of a bias vector. You can see their actual definition in the fairseq code.


Time Series Clustering The thing that I am trying to do is the time series shapes classification.

Basically the problem is as following:

Let's say I have some time series and my goal is to have an algorithm that "finds" the similar shapes in the data, that occur throughout the dataset and labels it accordingly (a, b, c...etc.)...The image might be worth than 1000 words, so I have attached the workflow that I have in mind

It would be great if anyone directs me to the algorithms/papers/codes that deal with this kind of problems

I was also looking for something similar and found a couple of libraries (https://matrixprofile.org/libraries/) that might work but I haven’t tried them out much yet.

I hope that helps and if anyone has any other suggestions I’d be interested as well.


How does ML Clustering put to a practical real-world use? Newbie alert to data science and ML.
I'm learning Supervised and Unsupervised learning at the moment and Supervised learning is easy to digest and I can relate to a lot of practical use cases. Unsupervised learning is where I just couldn't correlate to a the real-world use cases (although I found numerous of quotes where people say they use it for Customer Segmentation, Fraud Detection etc).

For argument sake, I'll just quote a sample taken from one of the MS Azure Studio examples.

The dataset contains Countries and their average protein intakes in various forms of food).

When this dataset is run through a KMeans algorithm, it creates 3 clusters and fits the country names in these clusters.

So in this specific example what is the problem I'm trying to solve?

Am I looking to find similar countries based on protein intake habits?

Am I creating groups based on the given dataset and then there is a human intelligence to qualify these groups (or clusters) to say "Vegetarian Rich Countries", "Red Meat Rich Countries" etc. Then when a new country comes we predict whether this country falls in which cluster?

In this case, there is an intermediate human intelligence is needed in the workflow which requires labelling the cluster (as opposed to labelling each datapoint in classification). Is this a correct understanding?

You want to understand the data.

So you run a clustering, then study how the points in a cluster differ from the points not in a cluster. Then based on these observations, you form a hypothesis. For example, you may notice that a cluster contains countries who eat a lot of fast-food and who are overweight. Then you can formulate the hypothesis that fast-food causes overweightedness, and then test that hypothesis.

This is a form of explorative data analysis. There is not a mathematical function to maximize, but it is a tool for humans to understand their data and then be able to formulate new hypotheses that would not have sprung to your mind otherwise.


Preparing for a Machine Learning Design Interview I am not sure if this is a relevant post here but:

I made it to the final round for the Machine Learing Engineer position at Facebook. The final round interview is virtual (thanks to Corona) and will consist of:

2 - General Algorithmic Coding questions
1 - Machine Learning Design
1 - General Systems Design
1 - Career Background

Most of these I know what I need to do to prepare but the content given to me for the Machine Learning Design is scarce to say the least but typical example questions for these are:

Design a newsfeed ranking algorithm
Design local search ranking
Design if an old post would be good for "On this Day" memories feature

This is the key highlights of what is expected for the Machine Learning Design interview:

You should be able to describe the components of an end-to-end ML system, including but not
limited to; model development, evaluation, and deployment.

You should be able to use existing toolsets to model the problem and break down its
components.

Be ready to analyze your approach while understanding the tradeoffs between your design
decisions.

We expect you to have a good understanding of common ML tools/techniques, but we do not
expect you to know and memorize every ML algorithm out there.

Now, although I know Machine learning decently well (I would like to think so) and I do mainly do Machine/Deep learning at my curreny job. I have never designed a Machine learning system. Thus I am not sure how to prepare for this particular portion of the interview and I have not really found much material online in terms of designing machine learning system.

Any advice would be greatly appreciated. My interview is set in the beginning week of April.

To wrap up you can have the following to prepare for ML interviews:

Machine Learning Engineering Book
Machine Learning Systems
Towards Data Science (Medium): They have a lot of interesting posts there.

And don't forget Data Science Stack Exchange. On this site, you can have a lot of interesting questions and answers. If you don't know the answer for your interview question, feel free to drop it here.


Is there a good German Stemmer? What I tried:

# -*- coding: utf-8 -*-

from nltk.stem.snowball import GermanStemmer
st = GermanStemmer()

token_groups = [(["experte", "Experte", "Experten", "Expertin", "Expertinnen"], []),
                (["geh", "gehe", "gehst", "geht", "gehen", "gehend"], []),
                (["gebäude", "Gebäude", "Gebäudes"], []),
                (["schön", "schöner", "schönsten"], ["schon"])]
header = "{:<15} [best expected: n/n| best variants: 1/n | overlap: m]: ...".format("name")
print(header)
print('-' * len(header))
for token_group, different_tokens in token_groups:
    stemmed_tokens = [st.stem(token) for token in token_group]
    different_tokens = [st.stem(token) for token in different_tokens]
    nb_expected = sum(1 for token in stemmed_tokens if token == token_group[0])
    nb_variants = len(set(stemmed_tokens))
    overlap = set(stemmed_tokens).intersection(set(different_tokens))
    print("{:<15} [as expected: {}/{}| variants: {}/{} | overlap: {}]: {}".format(token_group[0], nb_expected, len(token_group), nb_variants, len(token_group), len(overlap), stemmed_tokens))

what I get:

experte  [as expected: 0/5| variants: 3/5 | overlap: 0]: ['expert', 'expert', 'expert', 'expertin', 'expertinn']
geh      [as expected: 3/6| variants: 4/6 | overlap: 0]: ['geh', 'geh', 'gehst', 'geht', 'geh', 'gehend']
gebäude  [as expected: 0/3| variants: 1/3 | overlap: 0]: ['gebaud', 'gebaud', 'gebaud']
schön    [as expected: 0/3| variants: 1/3 | overlap: 1]: ['schon', 'schon', 'schon']

The two main problems are:

Overlaps: schön != schon
Non-working stemming, e.g. [experte, expertin, expertinnen], [ich gehe, du gehst, er geht]

A not so serious side problem is matching my expectations. So if the stemmer could actually bring the word in a basic form (not simply the stem), then it would be easier to analyze.

More Examples

Clashes

Input -> Output != clash
mittels -> mittel != "Das Mittel"

Unmatched expectations

Input -> Output / expected
Mädchen -> madch / Mädchen
Behaarung -> behaar / Behaarung

Big problem and very good question!

I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization.

Looking at the output below, I don't think that spacy will solve your problem to be honest. However, I just wanted to let you know about this option.

Spacy Lemmatization:

#pip install spacy
#python -m spacy download de

import spacy
nlp = spacy.load('de_core_news_sm')

mywords = "Das ist schon sehr schön mit den Expertinnen und Experten"

for t in nlp.tokenizer(mywords):
    print("Tokenized: %s | Lemma: %s" %(t, t.lemma_))

Result:

Tokenized: Das | Lemma: der
Tokenized: ist | Lemma: sein
Tokenized: schon | Lemma: schon
Tokenized: sehr | Lemma: sehr
Tokenized: schön | Lemma: schön
Tokenized: mit | Lemma: mit
Tokenized: den | Lemma: der
Tokenized: Expertinnen | Lemma: Expertinnen
Tokenized: und | Lemma: und
Tokenized: Experten | Lemma: Experte


Learning to Rank Application If there's a website/app that sells products and my job is to determine the order/ranking in which the products should be displayed.

For example : I click on restaurants and a list of restaurants pops up, I have to determine in what order the restaurants should be displayed.

All the data like ratings, distance to the customer, profit to us, prices, CTR, total number of views etc is available.

But how should I approaching this problem of rankings them in an efficient order ?
I have read about Learning to Rank and Collaborative Filtering but I am not sure if those are right for my problem, please help me out.

Thanks

Collaborative Filtering would definitely be a good start. This would work as follows. For each user, you can fill in rankings for restaurants they have not yet ranked by averaging the ranks (or some other aggregation) of similar users who have rated that restaurant.

Alternatively, you could take a content based approach. Here, each restaurant has a collection of keywords, and using or rating a given restaurant updates a user's profile for those keywords. Then, the user's profile can be compared (via cosine similarity or something) to all other restaurants profiles. Then you order accordingly.

It may be more useful to map into a latent space first, but up to you.


How can I get the max value and corresponding index of a tensor simultaneously in Tensorflow? The only way I came out is using tf.argmax with tf.reduce_max. But I'm afraid of that it may be inefficient to find the max value twice.

Before you worry about performance, I'd recommend profiling it. Finding the maximum is an inexpensive operation. Also, can you index into the element using the return value of argmax? That way you'd only take the max once.


Matrix factorization how to initialize weights and biases? I have a matrix factorization and I'm wondering how I should initialize its weights and biases.
When getting prediction (recommendation), after computing a dot product and adding bias I want to use sigmoid function on that to get value from 0 to 1.
But when introducing a sigmoid here I also introduce a possibile vanishing/exploding gradient problem. For that I think that weights can be initialized with xavier function. But what aboud biases? Should I just use uniform distribution from (-0.01, 0.01) for example?

For matrix factorization I usually see it being initialized by a uniform distribution from [0, 1) like in this (pytorch) or a truncated normal with mean=0.0 and std=1.0 as in this (tensorflow).


Training the parameters of a Restricted Boltzman machine Why are the parameters of a Restricted Boltzmann machine trained for a fixed number of iterations (epochs) in many papers instead of choosing the ones corresponding to a stationary point of the likelihood?

Denote the observable data by $x$, hidden data by $h$, the energy function by $E$ and the normalizing constant by $Z$.  The probability of $x$ is:
\begin{equation}
P(x) = \sum_h P(x,h) = \sum_h \frac{e^{-E(x,h)}}{Z}.
\end{equation}
The goal is to maximize the probability of $x$ conditional on the parameters of the model $\theta$. Suppose one has access to a sample of $N$ observations of $x$ with typical element  $x_i$. As estimator, one could find the roots of the derivative of the average sample log-likelihood:
\begin{equation}
\left\lbrace \hat{\theta} \in \hat{\Theta} :  N^{-1} \sum_{x_i} \frac{\partial  \log p(x_i)} {\partial \theta} = 0 \right\rbrace
 \end{equation}
and chose the one $\theta^\star \in \hat{\Theta} $ maximizing the empirical likelihood.  There exists many different ways to approximate the derivative of the log-likelihood to facilitate (maybe even permit) its computation. For example, Contrastive Divergence and Persistent Contrastive Divergence are used often. I wonder whether it makes sense to estimate the parameters $\theta$ recursively until convergence while continuing to approximate the derivative of the log-likelihood. One could update the parameters after seeing each data point $x_i$ as:
\begin{equation}
 \theta_{i+1} = \theta_{i} - \eta_i \frac{\partial  \log p(x_i)}{\partial \theta_i}
 \end{equation}
  The practice I learned in Hinton et al. (2006) and in Tieleman (2008)   is different though: Both papers define a number of fixed iterations a priori. Could somebody kindly educate me why recursively updating the parameters until convergence is not a good idea? In particular, I'm interested whether there's a theoretical flaw in my reasoning or whether computational capacities dictate sticking to a fixed number of iterations. I am grateful for any help!

I believe the problem is that log-likelihood is not directly computable, because of exponential in number of units complexity. There exist different proxies to the true log-likelihood, for instance pseudo log-likelihood (more here), and in principle you can train RBM until PLL is not changed too much.

Hovewer, there is a lot of randomness involved in the training process, so PLL most likely will be quite noisy (even true log-likelihood would be I believe).


Understanding the Gini/AUC metric as out-of-development performance metric Assume we develop a model for a binary classification task that reaches a certain Gini/AUROC estimate on the validation ( or training ) sample, among others. This is an overall good metric, often used for evaluating the ability of the model to separate the samples into, say, goods vs bads.

Further, assume this model is adequate and  will be used for further collection of new samples with a certain cutoff value. What should be expected Gini/AUC estimates on the newly collected sample?

From what I'm noticing, on the training sample there were clear cases where the model was able to distinguish and separate it with large probabilities. On the other hand, with applied cuttoff, say, <50%, the new sample with collect only those cases where no such clear separation is possible (because if it would, the case might not get collected). With such approach, for me it seems logical that the overall separation in the new sample will be lower, resulting in lower out-of-development-period Gini/AUC.

Is this the expected behaviour in normal production environments? Am I understanding things correctly?

Note: I understand that there are other simple metrics, such as sensitivity/specificity, hoslem.test and others, allowing for measuring and visualising True/False Positives. However, I have found that Gini/AUC is often a key metric when discussing and comparing classification models.

The advantage which train/test/validation dataset separation has is that you separate your dataset into:

The individuals which you know the exogenous variables and the output: Training
The individuals which you know the exogenous variables and the output (but you suppose you don't know which the output is): Test
The individuals you know the exogenous variables but not the output: Validation

Every DS or ML model is made so it is prepared to receive a validation dataset in the future and try to get every metric just almost as good as if it was the train dataset.

The test dataset has the objective of simulating the situation of having data but not output, and then you have the output to measure the behaviour and comparing the modelled vs real output.

So, for a concrete answer:
The behaviour you should expect from the validation (or newly collected sample) is the same as the test dataset.

Given that the underlying phenomenon and sampling technique remains the same.

For more information:
https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7


How to change the input of conv2d from gray scale to coloured image I have CNN with Tensorflow for gray images with weights

  w1=[9,9,1,64], w2 = [1,1,64, 32] and w3 = [5,5,32,1]

Now I want to use RGB 3 channel data instead of gray scales. Can anybody help me how to change

  w1, w2, and w3

for RGB images?

Is it possible to use conv2d function for RGB images in Tensorflow?

Change $w1=[9,9,1,64]$ to $w1=[9,9,3,64]$. The first two numbers specify the size of the filter, the third specifies the number of input channels and the last one specifies the number of output filters.

  Is it possible to use conv2d function for RGB images in Tensorflow?

Yes, conv2d is used for one channel, grey level, images and three channel images, coloured images.


To create a pre-trained model for C# I created a model with machine learning. Using the library Accord.NET.
I would like to keep the pre-trained model in a file (not in the RAM memory) how this is possible?
Note: not necessary to create the model with Accord.NET could be with Python (TensorFlow, PyTorch, etc) but to use the pre-trained model from my desktop application in C#.

Welcome to the site - great first question!

In your scenario (assuming I understand it correctly) you have a saved model in python right? So, what I would do is create a new API layer (written in python, using something like the Flask framework) and I would create API endpoints for your algorithmic functions (could be as simple as just 1 for your predict function).

From there, you could run that python script on a server and expose that API to the outside world and it will run/respond just like any other REST API service. At that point, it would be fairly trivial to write C# code to access the API (or any other language you want).


Missing Values in Data I have experienced that most of the datasets contain missing values, which make our task bit challenging.

Please let me know how to fill up those missing values in an efficient way? and is there any specific techniques to handle missing values?

Yes there are so many approaches to handle missing data or missing values depending on the task and the property of the data itself. For example in time series you can think about forward filling or even backward filling, max, mean or median over a time lapse. There are also 'smarter' ways like training a model over the available data and try to predict the missing ones. The latest requires much data so can give efficient results. You can refer to these two links for more information.

How to Handle Missing Data with Python
Machine learning algorithms to handle missing data


CNN for checking existance of single label i am just thinking about training a neural network which uses data of only one single label.

For example: Assuming i have many images which contain a dog. Now i want to teach the network how a dog looks like, without images containing no dogs.

More formal: I want to check for availability of one label, so in theory, i need 2 datasets: Dog and NoDog

The question i am raising to myself now is: How would i define the NoDog-class? I can throw in anything, some random noise or just any unlabelled data which does not contain a dog? But how do i make sure that i
sample the "no-dog"- space correctly? Which features does a "no-dog" image have? The "no-dog" space is huge, it is extremely difficult to describe it, somehow.

I had different ideas:

-Use the same dog-images, but cut out the dogs and fill it with noise

-Train a gan and use the discriminator for the binary classification

-Use tons of unlabelled data

Does anyone know how to tackle this problem in a systematic manner? Thank you

Convolutional models can be used either for classification or not. You cannot train a classifier on a single label, because the loss function of such models is meant to punish the model every time an image is missclassified. If you have a single label, accuracy will be 100% all the time, by definintion. And such a model would be useless.

I think you have two options:

Get a bunch of images that are not dogs, then train a CNN classifier.
Train an Autoencoder for dimensionality reduction on your "dog data". These models can be used for anomaly detection tasks. Once you have trained such a model, when you feed a non-dog image it should be able to recognize it as an anomaly and flag it.

Both methods could work equally good. The first is the simplest one, but if you like playing and experimenting with DL the second is much more fun.


Normalized metric for comparing regression models performance I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It was also hard for me to explain whether performance loss by 225 (in terms of MAE), when switching from one model to another, is significant or not.

To me it was clear that both are little because I knew the context: we we're talking about predicting house prices ranging from \$34,900 (min) to \$755,000 (max), so that

MAE=17,000 is just 2.5% of the difference between max and min
Change in MAE by \$225 is just 0.03% of the difference between max and min.

Are there some normalized metrics for comparing performance of regression models without the need to know the context?

Which of those metrics are available in scikit-learn? For instance, it provides mean_absolute_error and mean_squared_error but they are not normalized.

Edit:

After Joe B suggestions, I've plotted a graph to see deviations between predicted and expected price. In fact, that's gives more insight than a single-number metric:

A couple suggestions

MAE represents your mean error. This is essential to recognize as having an error of even half of your MAE on the low end of your spectrum ($34,900) is huge. However, to your point, at the high end of your spectrum it is quite small.
To solve the above, you should plot a graph to identify the patterns in your error. You'll plot your predicted y value vs. your actual y value and see where your predictions deviate from the y=x line.

Normalized metrics:

you may use an r-squared value. This might be ideal because its looking at the percent of variance explained by your regression and thus is a relative measure of fit. Your r-squared value will always be between 0 and 1 (assuming your model is not worse than always guessing your mean value) where 1 is perfect. Implement as follows:

from sklearn.metrics import r2_score
y_pred = model.predict(X_train)
print(r2_score(y_true, y_pred))


Why is Reconstruction in Autoencoders Using the Same Activation Function as Forward Activation, and not the Inverse? Suppose you have an input layer with n neurons and the first hidden layer has $m$ neurons, with typically $m < n$. Then you compute the actication $a_j$ of the $j$-th neuron in the hidden layer by

$a_j =  f\left(\sum\limits_{i=1..n} w_{i,j} x_i+b_j\right)$, where $f$ is an activation function like $\tanh$ or $\text{sigmoid}$.

To train the network, you compute the reconstruction of the input, denoted $z$, and minimize the error between $z$ and $x$. Now, the $i$-th element in $z$ is typically computed as:

$
z_i = f\left ( \sum\limits_{j=1..m}  w_{j,i}' a_j+b'_i \right)
$

I am wondering why are the reconstructed $z$ are usually computed with the same activation function instead of using the inverse function, and why separate $w'$ and $b'$ are useful instead of using tied weights and biases? It seems much more intuitive to me to compute the reconstructed with the inverse activation function $f^{-1}$, e.g., $\text{arctanh}$, as follows:

$$
z_i' = \sum\limits_{j=1..m} \frac{f^{-1}(a_j)-b_j}{w_{j,i}^T}
$$

Note, that here tied weights are used, i.e., $w' = w^T$, and the biases $b_j$ of the hidden layer are used, instead of introducing an additional set of biases for the input layer.

And a very related question: To visualize features, instead of computing the reconstruction, one would usually create an identity matrix with the dimension of the hidden layer. Then, one would use each column of the matrix as input to a reactivation function, which induces an output in the input neurons. For the reactivation function, would it be better to use the same activation function (resp. the $z_i$) or the inverse function (resp. the $z'_i$)?

I don't think that your assumption $w' = w^T$ holds. Or rather is not necessary, and if it is done, it is not in order to somehow automatically reverse the calculation to create the hidden layer features. It is not possible to reverse the compression in general, going from n to smaller m, directly in this way. If that was the goal, then you would want a form of matrix inversion, not simple transpose.

Instead we just want $w_{ij}$ for the compressed higher-level feature representation, and will discard $w'_{ij}$ after auto-encoder is finished.

You can set $w' = w^T$ and tie the weights. This can help with regularisation - helping the autoencoder generalise. But it is not necessary.

For the autoencoder to function it doesn't actually matter what activation function you use after the layer that you are pre-training, provided the last layer of the autoencoder can express the range of possible inputs. However, you may get varying quality of results depending on what you use, as normal for a neural network.

It is quite reasonable to use the same activation function that you are building the pre-trained layer for, as it is the simplest choice.

Using an inverse function is possible too, but not advisable for sigmoid or tanh, because e.g. arctanh is not defined < -1 or > 1, so likely would not be numerically stable.


Looking for algebras designed to transform time series I am looking for information on (formal) algebraic systems that can be used to transform time-series - in either a practical or academic context.

I hope that there exists (at least one) small, expressive, set of operators - ranging over (finite) time-series.  I want to compare and contrast different systems  with respect to algebraic completeness, and brevity of representation, of common time-series transformations in various domains.

I realise this question is broad - but hope it is not too vague for datascience.stackexchange.  I welcome any pointers to relevant literature for specific scenarios, or the general subject.

Edit... (Attempt to better explain what I meant by an algebraic system...)

I was thinking about "abstract algebras" as discussed in Wikipedia:

http://en.wikipedia.org/wiki/Algebra#Abstract_algebra
http://en.wikipedia.org/wiki/Abstract_algebra#Basic_concepts

Boolean Algebras are (very simple) algebras that range over Boolean values.  A simple example of such an algebra would consist the values True and False and the operations AND, OR and NOT. One might argue this algebra is 'complete' as, from these two constants (free-variables) and three basic operations, arbitrary boolean functions can be constructed/described.

I am interested to discover algebras where the values are (time-domain) time-series.  I'd like it to be possible to construct "arbitrary" functions, that map time-series to time-series, from a few operations which, individually, map time-series to time-series.  I am open to liberal interpretations of "arbitrary". I would be especially interested in examples of these algebras where the operations consist 'higher-order functions' - where such operations have been developed for a specific domain.

It seems like you are overthinking this and getting caught up in your own head a little.

We can define a restricted set of time series as those which are well behaved (non-asymptotic) and single valued.  These time series then form a time series algebra which have additive, multiplicative, commutative, and associative properties.  There is an identity element (straight line with value 1) and a zero element (straight line with value 0).  We even have a well defined inverse member for all time series not crossing zero.  I believe this even forms an Abelian group.

Further, previous posters are correct that this group can be perfectly represented by various orthogonal vector spaces such as polynomials and Fourier space.  This is true whether we are talking about Lagrange polynomials, Fourier series, or Gauss-Lobatto-Legendre polynomials.

The point then becomes... where does this get you? You got caught up in the suggestion to employ Fourier series in that you didn't want to remain in frequency space, but this is entirely unnecessary.  You can, for instance, Fourier transform into frequency space, apply a high frequency filter, and then transform back into the time domain.  This is incredibly effective in removing noise (although band pass filters tend to work better).

Stepping back from the question a little, it seems like you might really be asking:  Given typical time series and typical time series operations (smoothing, de-seasoning, de-trending, etc) do the time series and their typical operations constitute a group with a well defined algebra?  The answer is, yes.  All of these operations just involve finding other time series which can be added, subtracted, multiplied, or divided by the original time series in order to yield a more physically tractable (interpretable) representation.


Topics to learn in Neural Network I have recently started learning Deep Neural Networks and was going through the tutorials online. Everywhere I saw that the topics post Image classification using CNN is a little hazy. No one seems to follow a guide as to what topics should one learn after learning image classification.
I would really appreciate if someone could guide me so as to learn advanced topics and become good to get started with research in this field.
The topics i have covered so far are:

Basic Machine Learning

Linear Regression
Logistic Regression
SVM
Decision Trees
Optimization Algorithms
Regularization Methods
Different Loss Functions
Different Metrics during Training
Feature Extraction
HyperParameter Tuning

Basic Neural Network

ANN
CNN

Image Classification
Image Augmentation before training
Visualizing Filters

I have started learning Object detection using CNN but I have this feeling that I am missing some topics which can strengthen my foundation.
Thanks in advance.
Any guidance is appreciated.

If you want to start with research, as you say, I would make sure not to skip the theory and underlying mathematics of machine learning. Knowing what model and hyperparameters to choose for a certain problem and being able to implement this quickly is very useful in corporate settings, but research mostly requires a deep mathematical understanding of what is going on.
In my opinion the book 'deep learning' by Goodfellow and Bengio has a good structure that allows you to get familiar with the mathematical foundations of deep learning. When you understand this, you will be able to read and understand state-of-the art papers if you want and do your own research. You can find the book for free online I think.


I think the number of elements in each lists are different. (26 , 22 , 26 , 18)

Zip() function uniformly maps all elements of lists of same sizes. (And that's why the 1's are missing and would round off to 18 elements)

Try maintaining the same number of elements in each List and try to find the mean again.


What is difference between text classification and topic models? I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic modeling over documents to identify a topic? Can I use classification methods to classify the text inside these documents?

Text Classification

I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their words. Then I give you new documents and ask what you think the label for each one should be. The labels have meaning to me, not to you necessarily.

Topic Modeling

I give you a bunch of documents, without labels. I ask you to explain why the documents have the words they do by identifying some topics that each is "about". You tell me the topics, by telling me how much of each is in each document, and I decide what the topics "mean" if anything.

You'd have to clarify what you me by "identify one topic" or "classify the text".


Multivariate Time series analysis: When is a CNN vs. LSTM appropriate? I have multiple features in a time series and want to predict the values of the same features for the next time step. I have already trained an LSTM which is working okay, but takes a bit long to train.
So now my question: is it reasonable to use a CNN instead of an LSTM, even though it is a time series? Are there any indicators for when you should never switch to a CNN?

Is it reasonable to use a CNN instead of an LSTM, even though it is a time series?
Yes, it is. Convolutional Neural Networks are applied to any kind of data in which neighboring information is supposedly relevant for the analysis of the data.
CNN are very popular with images, where data is correlated in space, and in video, where correlation happens both in space and time.
Are there any indicators for when you should never switch to a CNN?
CNNs are limited in a sense: they have a rigid, forward structure.
If you are trying to perform:

classification in sequences that have varying length ($N$ to $1$);
trying to output another sequence which has no fixed proportion between their length and the length of the input ($N$ to $M$).

Simple feed-forward neural networks will fail (due to dimension inconsistency). So, in that case you should use a recursive neural network such as a LSTM.


Example of Logistic Regression using a proportion as the dependent variable? I am trying to run a logistic regression on a data set where my dependent variable is a proportion of a binary variable, rather than the binary variable itself.  I have seen a ton of documentation that says this is possible, but I am having trouble finding an example of how to actually do it.  I am open to using scikit learn, statsmodels, or any other library that will do it.

I have added a photo showing a simplified version of my data.

successes here is just a count of a binary (1/0) outcome.  instead of having the individual observations, I only have them rolled up, but my understanding is that it is still a logistic regression problem.  I want to predict the dependent variable 'proportion' based on the features.  I understand this conceptually, but am just trying to find an example of this in python.  all of the examples I have seen assume a binary dependent variable.

your help is appreciated!

As the link by @Spacedman shows, binomial regression works well. If you don't have the attempts and successes but, instead, have just the proportion, then you'll want to use beta-regression. After a little digging, it doesn't seem like this is available in Python. Here's a blog post demoing it in R.


Is the way to combine weak learners in AdaBoost for regression arbitrary? I'm reading about how variants of boosting combine weak learners into final predication. The case I'm consider is regression.
In paper Improving Regressors using Boosting Techniques, the final prediction is the weighted median.

For a particular input $x_{i},$ each of the $\mathrm{T}$ machines makes a prediction $h_{t}, t=1, \ldots, T .$ Obtain the cumulative prediction $h_{f}$ using the T predictors: $$h_{f}=\inf\left\{y \in Y: \sum_{t: h_{t} \leq y} \log \left(1 / \beta_{t}\right) \geq \frac{1}{2} \sum_{t} \log \left(1 / \beta_{t}\right)\right\}$$ This is the weighted median. Equivalently, each machine $h_{t}$ has a prediction $y_{i}^{(t)}$ on the $i$'th pattern and an relabeled such that for pattern $i$ we have: $$
y_{i}^{(1)}<y_{i}^{(2)}<, \ldots,<y_{i}^{(T)} $$ (retain the association of the $\beta_{t}$ with its $y_{i}^{(t)}$). Then sum the $\log \left(1 / \beta_{t}\right)$ until we reach the smallest $t$ so that the inequality is satisfied. The prediction from that machine $\mathrm{T}$ we take as the ensemble prediction. If the $\beta_{t}$ were all equal, this would be the median.

An Introduction to Statistical Learning: with Applications in R: The final prediction is the weighted average.

As such, I would like to ask of the way of aggregation is mathematics-based, or because the researcher feels it's reasonable.
Thank you so much!

The ISL description is of gradient boosting (regression, with mse as the loss function), not of AdaBoost.  There, $\lambda$ is constant, not weights for each tree.  Since each tree is fitted to the residuals, we need to add the results to better approximate the true values, not average.
However, the title question is still an interesting one.  It does seem probably mostly arbitrary, but at least some testing has been done, see e.g. "Experiments with AdaBoost.RT, an Improved Boosting Scheme for Regression" by Shrestha and Solomatine.