Convolution and Pooling as Infinitely strong priors I am currently reading about Convolutional Neural Networks in Deep Learning Book. I am stuck on section 9.4 titled "Convolution and Pooling as Infinitely String Priors".

Could someone please intuitively explain to me what prior probability distribution is and what is its association to Convolution and Pooling operations in context of CNNs. Thank you!

A prior distribution expresses your assumptions about the model without observing any data. E.g. when doing linear regression, you a priori assume that the slope is close to zero. Now you start measureing data points and it turns out that the slope should probably be close to one, so you compromise and pick a value somewhat inbetween. If your belief in your prior assumption of zero slope is weak, it does not take much data to convince you to pick a slope closer to one. If your belief is strong, you will pick a slope close to zero anyway and demand to see many data points until you move it slowly towards one.

In case of regression the strength of your belief would be parameterized by the weight of the regularization term in the optimization objective. The connection between regularization and priors is the same for neural networks (see here for a little more detail).

So you can think of the strength of the prior as a measure for how much evidence you would need to see until you deviate from your prior assumptions about the model.

When you train a CNN, you assume a priori that this network structure is the best for your problem, i.e. that the model should internally compute convolutions. Since the CNN structure is build into the model, no amount of data will convince you to abandon this structure in favour of e.g. a fully-connected NN. Hence the prior is infintely strong.

The argument for pooling etc. would be similar. I would even claim that most if not all non-trainable parameters can be seen as an infinitely strong prior.


What's the best classification model for this recommendation engine? I'm not a data scientist but I'm trying to implement a recommendation engine on my company. My application runs on PHP but I'll use Python to process this data.

My company is an online school, with 40 online courses as of now. I have a CSV file with around 30k users preferences, and it looks like this:

0 means that user is not subscribed (I consider here that he has no interest), while 1 means subscribed (interested).

My idea is to compare one single user array such as [0,1,0,0,0,1,1...] with all this data and return a grade for each course with the probability of interest for this user.

I was thinking of using a Multinomial Logistic Regression, but as far as I know (and I don't know much) it would return me a binary result, right?

What classification model would you recommend me to use? Ideally, my result should be something like:

[0.95, 0.1, 0.54, 0.3, 0.87...]

Cheers!

Without more information about your dataset, it's impossible to recommend one particular classifier over another.

If you want your classifier to return a vector of probabilities, then if you're using the sklearn library, you could use the predict_proba method.

Here's an example:

from sklearn.datasets import load_digits
digits = load_digits(2)
from sklearn.linear_model import LogisticRegression
preds = LogisticRegression().fit(digits.data, digits.target).predict_proba(digits.data)
print([i[1] for i in preds])


How many people can use a single Hadoop cluster at one time? I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a class of 12 to 24 students.

Also, I am wondering if anyone could recommend whether the specifications each for the nodes on a high-end educational-level cluster should be the same as for any other production-level cluster ( ie. 64G-128G RAM, 24TB hard drive space, 8 cores, etc.). I believe that the dataset sizes that students would be using will range between 20MB minimum to about 0.500TB maximum, I imagine that we will ultimately be working on real problems and datasets, even if they are not exactly considered to be big data.

For initial learning you could very easily do proof of concept work against individual VMs (a 4GB VM with a pseudo cluster is enough to do basic mapreduce examples in). If you're going to use Spark I would lean towards higher memory counts if they're within budget. I'd also keep an eye towards more, lower cost nodes. A stack of Intel i7 NUCs (or similar) with 2TB consumer SSDs and 32GB of RAM will cost less than $1k per node and 10-15 of them could easily handle a class of 24 students.


Any clear winner for Data Science in Scala? I have found a number of libraries and tools for data science in Scala, I would like to know about which one has more adoption and which one is gaining adoption at a faster pace and to what extent this is the case. Basically, which one should I bet for (if any at this point).

Some of the tools I've found are (in no particular order):

Scalding
Breeze
Spark
Saddle
H2O
Spire
Mahout
Hadoop
MongoDB

If I need to be more specific to make the question answerable: I'm not particularly interested in clusters and Big Data at this moment, but I'm interested in sizable data (up to 100 GB) for information integration and predictive analytics.

Not sure anybody have worked with all these tools, so I'm going to share my experience with some of them and let others share their experience with the others.

MongoDB addresses problems that involve heterogeneous and nested objects, while data mining mostly works with simple tabular data. MongoDB is neither fast with this type of data, nor provide any advanced tools for analysis (correct me if you know any). So I can think of a very few applications for Mongo in data mining.

Hadoop is a large ecosystem, containing dozens of different tools. I will assume that you mean core Hadoop features - HDFS and MapReduce. HDFS provides flexible way to store large amounts of data, while MapReduce gives basis for processing them. It has its clear advantages for processing multi-terabyte datasets, but it also has significant drawbacks. In particular, because of intensive disk IO during MapReduce tasks (that slows down computations a lot) it is terrible for interactive development, iterative algorithms and working with not-that-big datasets. For more details see my earlier answer.

Many algorithms in Hadoop require multiple MapReduce jobs with complicated data flow. This is where Scalding gets shiny. Scalding (and underlying Java's Cascading) provides much simpler API, but at the moment uses same MapReduce as its runtime and thus holds all the same issues.

Spark addresses exactly these issues. It drops Hadoop's MapReduce and offers completely new computational framework based on distributed in-memory collections and delayed evaluations. Its API is somewhat similar to Scalding's with all MR complexity removed, so it's really easy to get started with it. Spark is also the first in this list that comes with data mining library - MLlib.

But Spark doesn't reinvent things like basic linear algebra. For this purpose it uses Breeze. To my opinion, Breeze is far in quality from scientific packages like SciPy, Octave or Julia, but it is still good enough for most practical use cases.

Mahout relies on Hadoop's MapReduce and thus is terrible for iterative algorithms. Spire and Saddle look cute and probably have their niche, but seem to be much less well-known than Breeze. I couldn't find much information about H2O, so it doesn't look like a big player here (comments from people who used it are welcome).

Some quick summary. Spark seems to be the most simple, flexible and fast-growing project for large-scale data processing. It facilitates a number of new projects (e.g. Shark or Spark SQL) and penetrates into existing (including Cascading and Mahout). Spark knows how to utilize HDFS API, and thus scales to terabytes of data easily. And for data mining downshifters who don't want to bother with a cluster setup there's always pure Breeze.


Max Pooling in first Layer of CNN I am seeing, in all the notebooks that I found, that Max Pooling is never used in the first layer of a CNN. Why this? Is it a convention among data scientist to do not use max pooling in the first layer? Or is it an error to use in the first layer?

The purpose of max pooling operation is to decrease the spatial dimensions of the input while also being robust by only considering the maximum values.
Generally, as you might have noticed, most CNNs aim at decreasing the spatial dimension of the input while increasing its depth. Very broadly speaking, you can think of this as trying to encode the information enclosed in the spatial image into channels representing different aspects of the input.
To answer your question, It would not have been an error to max pool the input right away, however, in that case, the input would not have been through enough layers to extract its enclosing information. Having said that, max pooling that early might result in losing spatial information while not gaining anything valuable depthwise.


illumination problem for face landmark detection I trained dlib face landmark detection model using ibug and yaleB datasets. Accuracy of the model is dropping when a bright light or a shadow appears on the side face. Used default parameters and similar images were included while training. Is there any solution to overcome the problem.

Maybe you might try out the brightness data augmentation; in Keras you have something like:

ImageDataGenerator(brightness_range=[0.2,1.0])

in case you want to generate different brightness levels between 20% and 100% based on your train images. For example (taken from Deep learning for computer vision book by Jason Brownlee):


Random Forest but keep only leaves with impurities below a threshold Is there an algorithm out there that creates a random forest but then prunes all the leaves that have an impurity measure above a certain threshold that I would determine?
In other words, if I set min samples per leaf to be 500 and leaves have to have at least a 90% purity for example, the algorithm would only keep leaves that respect these parameters.
My dataset is extremely noisy so most leaves have a gini impurity around 0.5 but some leaves are almost around 0. I care only for the latter in my use case. Is there an algorithm that does something like what I described?

If some leaves are pruned then it means that the model cannot predict the instances which would normally have fallen into these leaves. The logic of a decision tree is to represent every possible instance: like any supervised learning method, it must be able to predict for every possible instance as input. In some cases where all the leaves have low purity, this condition would even make the model unable to predict anything at all. This is why there is no such algorithm, at least not as a regular supervised learning algorithm.
However you can rely on the probability predicted for the instance instead, this gives you exactly the same information in a way consistent with standard ML.


Linear regression: variables with high P-value Often, after training a linear regression model on data, some variables/features would have high P-value, which means they are not statistically significant. Although there are automated methods like variable selection, such as Step-wise, LASSO, etc. I would like to understand the best practice for manually selecting variables.

One thing I can think of is to check for collinearity and perhaps discard or combine variables that shows high correlation. One can also compare models using the F test, or perhaps use domain knowledge. Aside from that, what other best practices are there?

I feel simply discarding variables that shows high P-value is too simplistic. And there is no guarantee that the remaining variables will all have high P-value after you have discarded some variables.

In general, there is no clear and easy way for deciding which features to include in a model.

That being said, there are different strategies you can use to process features in an efficient way:

Domain knowledge

One of the most important aspects when determining important features is the knowledge of the specific domain related to your dataset. This might mean reading past research papers that have explored similar topics or asking key stakeholders to determine what they believe the most important factors are for predicting the target variable.

So, these methods depend on your knowledge from the domain.

Wrapper methods

These methods determine the optimal subset of features using different combinations of features to train models and then calculating performance. Every subset is used to train models and then evaluated on a test set. Sp, these methods could end up being very computationally intensive, though they are highly effective in determining the optimal subset. It is not suggested to use these methods with large feature sets because they are computationally expensive.

For this case you can use Recursive Feature elimination (RFE)

Filter methods

In these methods, feature selection methods carried out as a pre-processing step before even running a model.

They work by observing characteristics of how features are related to one another. Different metrics can be used to determine which features will get eliminated and which will remain. They also return a feature ranking that tells you how independent variables are ordered in relation to one another. They will remove the variables that are considered redundant.

One example of this method is "VarianceThreshold" from sklearn

Embedded methods

These methods are included within the actual formulation of the machine learning algorithm. The most common type of embedded method is regularization such as lasso.


Discarding rare words when comparing texts - per text, per comparison, or per codex? I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency.

first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear  <10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).

Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare?
I can think of three options:

Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.

What option would serve me better in finding meaningful N-gram representations of texts?

Many thanks.

Tf-idf (text frequency - Inverness document frequency) will highlight your interests.

https://en.m.wikipedia.org/wiki/Tf–idf


Prevent model from over-focusing on strong features I have a classification model (DNN/Linear layers with some transformers and other things later). The input to the model are several different modalities of different lengths and different amounts of information. I am trying to mitigate the dimensionality difference by projecting different modalities into the same dimensional space and then combining them into the same space. However, the model seems to be focusing completely on the strongest input features and almost completely ignores the abundance of weaker ones. All of the input features are normalized (bools to 0/1 and continuous to mean=0, stddev=1), so the issue is not the scale of feature values but the predictive power of those few features which end up choking others.
Are there any methods out there for addressing this?

Use dropout.
Dropout is a form of regularization where you zero out the input components with a certain probability.
If you want your model to learn representations that rely less on some features, just place a dropout layer before those features enter the rest of the network. This will force the network to learn more robust representations that depend less on the strongest features.


Audio classification data balance I'm trying to make a "car sound detector" I have the data from https://urbansounddataset.weebly.com/urbansound.html site. Which has 1000 labeled sounds for 10 different classes(Car sound, dog bark, drill..etc). So using a binary classifier I want to detect if a given sound sample is a car -> label 1 or any other sound from that data set -> label 0.

So data is highly unbalanced. 1000 1-labeled car sounds against 9000 0-labeled other sounds.

To make a more balanced set, for example in both test and training sets I can create %50 / %50  split using oversampling(not sure its a good idea). But in a real life scenario (a microphone listening environment sounds) the ratio of a car sounds is not gonna be %50. Maybe for every 100 sounds the mic picked only one percent will be a car sound.

So what would be a better/realistic to way to prepare and split the data for test and training set?

I think you might have confused the concept of training set, validation set, and actual input. The training set helps you build a model; the validation test let you validate that performance of your model. (sometimes people also use real life testing test to ensure the model is not overfitting) Once you have a model created, your model will be like a black box that takes your input (i.e., could be a single sample) and give you a classification result. So it really doesn't matter if your only 1% of your input data is a car sound; if the model is built properly it will be able to give you a good prediction.


Convolution Neural Networks on microcontrollers I am wondering if an already trained convolutional neural network can be represented as a formula just like a perceptron can ($x_1w_1 + x_2w_2 + \ldots =$ PREDICTION). I know that the formula could be a lot more complicated like this, but in general would it be possible to train a CNN in Python or Matlab, get the weights and create an arithmetic, exponential, whatever formula made with those acquired weights?

I want to do this because I am trying classify in a PIC32 which I think does not have the capacity to perform a CNN within itself.

Or do you know if there is a "c library" that can do CNNs?

What you are looking for is often called an inference engine. Tensorflow Lite for microcontrollers was just announced. I have not seen anyone use it on PIC32, but it should be possible to port it to that platform.

For ARM Cortex-M microcontrollers there are a couple more options, for instance uTensor and CMSIS-NN. The latter is low level implementation using integer math only, which could be a good inspiration if your microcontroller does not have hardware support for floating point.


What is Monte Carlo dropout? I understand how to use MC dropout from this answer, but I don't understand how MC dropout works, what its purpose is, and how it differs from normal dropout.

Let's start with normal dropout, i.e. dropout only at training time. Here dropout serves as a regularization to avoid overfitting. During test time, dropout is not applied; instead, all nodes/connections are present, but the weights are adjusted accordingly (e.g. multiplied by the keep ratio, which is 1 - dropout_ratio). Such a model during test time can be understood as a average of an ensemble of neural networks.
Notice that for normal dropout, at test time the prediction is deterministic. Without other source of randomness, given one test data point, the model will always predict the same label or value.
For Monte Carlo dropout, the dropout is applied at both training and test time. At test time, the prediction is no longer deterministic, but depending on which nodes/links you randomly choose to keep. Therefore, given a same datapoint, your model could predict different values each time.
So the primary goal of Monte Carlo dropout is to generate random predictions and interpret them as samples from a probabilistic distribution. In the authors' words, they call it Bayesian interpretation.
Example: suppose you trained an dog/cat image classifier with Monte Carlo dropout. If you feed a same image to the classifier again and again, the classifier may be predicting dog 70% of the times while predicting cat 30% of the time. Therefore you can interpret the result in a probabilistic way: with 70% probability, this image shows a dog.


Data scraping & NLP? I'm scraping data from Bing search results for (non-commercial purposes, of course) on Python using BeautifulSoup. I've entered an Indian dessert name, called 'rasmalai' as the word that I am focusing on. The code I'm using returns the title and a description of the web page. I've also extracted the links for the results. Here is the code I used:

from bs4 import BeautifulSoup
import urllib, urllib2

def bing_search(query):
    address = "http://www.bing.com/search?q=%s" % (urllib.quote_plus(query))

    getRequest = urllib2.Request(address, None, {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 Chrome/65.0.3325.162 Safari/537.36'})

    urlfile = urllib2.urlopen(getRequest)
    htmlResult = urlfile.read(200000)
    urlfile.close()

    soup = BeautifulSoup(htmlResult)

    [s.extract() for s in soup('span')]
    #unwantedTags = ['a', 'strong', 'cite']
    #for tag in unwatedTags:
        #for match in soup.findAll(tag):
           # match.replaceWithChildren()

    results = soup.findAll('li', {"class" : "b_algo" })
    for result in results:
        print "# TITLE: " + str(result.find('h2')).replace(" ", " ") + "\n#"
        print "# DESCRIPTION: " + str(result.find('p')).replace(" ", " ")
        print "# ___________________________________________________________\n#"

    return results

if __name__ == '__main__':
    links = bing_search('rasmalai')

Now that I have the links, web page title, and a short description, I want to extract keywords using NLP. In the end, I'd like to produce a CSV file with the dish name and associated keywords. Could someone guide me to some resources on how to do this part?

Thank you so much in advance.

NLTK

A great starting point for keyword extraction is the NLTK (natural language toolkit) library.  To extract keywords, you probably need to tokenize your data, breaking each word out into a token, and ignore the most common or unimportant words known as "stopwords".  Assuming you're searching for keywords across a large number of query results, identify the most important terms in each document using TF-IDF (term frequency–inverse document frequency).  There are tools and tutorials for this in the NLTK documentation.  Sort the resulting token-scores, choose the highest scoring tokens, and these are a good start at your keywords.


how to get the Polysemes of a word in wordnet or any other api? how to get the Polysemes of a word in wordnet or any other api. I am looking for any api. with java any idea is appreciated?

There are several third-party Java APIs for WordNet listed here: http://wordnet.princeton.edu/wordnet/related-projects/#Java

In the past, I've used JWNL the most: http://sourceforge.net/projects/jwordnet/

The documentation for JWNL isn't great, but it should provide the functionality you need.


Best way to store large data set using R from Twitter? I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R.  have difficulty storing tweets because my machine has only 8 GB of memory. It ran out of memory even before I set it to retrieve for one day. Is there a way where I can store the tweets straight to my disk without storing into RAM? I am not using the streaming API as I need to get old tweets.

Find a way to make your program write to disk periodically. Keep count of the number of tweets you grab and save after that number is high. I don't write R but psuedocode might look like:

$tweets = get_tweets();
$count = 0;
$tweet_array = array();
for each ($tweets as $tweet) {
  $tweet_array += $tweet;
  $count++;
  if ($count > 10000) {
    append_to_file($tweet_array, 'file_name.txt');
    clear_array($tweet_array);
  }
}


Does an RL agent still learn if its actions are "blocked"? Say we have a game that is a maze environment where there is a character to be controlled through the maze. When the agent (the character) approaches a wall, it may try to execute an action that would lead it into the wall, which is not permitted. Instead of executing this action, we place an if statement in the code that checks if the action is permitted, and if not, does not execute the action, and simply proceeds to the next state.

Another similar example is if an RL agent is being trained in a stock trading environment, where say it tries to sell more stock than it actually owns, or buy less than the minimum amount. Similarly as before, we place an if statement that checks for these conditions and either allows the action (allows the trade) or moves on to the next state.

Does the agent still learn, even if we "override" it and block certain actions? How else would we go about restricting the agent to certain actions in certain states e.g. if a wall is to the left of the agent in the game environment, how would we restrict the agent to only move forward, backward or right, but not left?

By overriding the Agent's action, Agent can theoretically take this action over and over and do nothing else, there is nothing to stop the agent from doing this, or teach the agent that this is not the intended action here.

What I usually do in these situations is, I punish the agent if the action is not desirable. So if agent take the action to go to the left, but there is a wall on the left side, I would not change the state of the environment (So the agent won't move), but I would also send a minus value (punishment) as a reward. This way after some training, the agent would learn that this action is not desirable.

The same can be applied to you Stock example. So if the agent tries to sell more stock that it actually has, you just don't sell the stock but also punish it with a big minus reward. This way it makes it easier for the agent to actually understand the environment.

Hope this helps.


Make the CNN to say "I don't know" I am currently working on an image classification problem. To ease the implementation I used transfer learning in Keras with Resnet50 as base model.
I have 2 classes, however there are many possibilities that the input image may not belong to both the classes. In such cases the CNN should be able to output as Unknown.
Note : I can not create a separate class for unknown as it can come from any distribution.

I read that the Bayesian neural network would help in such cases.

Could you please help me in understanding how can I implement Bayesian CNN with transfer learning. Any material or reference link would be really helpful.

Thank you

You could put a one-class classification model before your CNN.

This would mean that you treat both your classes as one and then frame it as an anomaly detection problem. There are some different ways of achieving this. One way could be to do dimensionality reduction on the images and then use that image encoding to train an outlier detector like an one-class SVM. You can get some ideas here:

Learning Deep Features for One-ClassClassification

Anomaly Detection using One-Class Neural Networks

You can also look at cbeleites answer to this related questions. This was actually where I learned about one-class classification on images and it goes into a bit more detail and explains some caveats with using it.


How to train with cross validation? and which f1 score to choose? I got similar results in 2 models which consists of similar algorithms.
Model 1 with cv=10 has a f1'micro' of 0.941. See code below.
Model 2 only train test split (no cv) has f1'micro' 0.953.
Now here is my understanding problem. Before I did a Grid-Search to find best hyperparameters. Now I would like to do just a cross validation to train the dataset. Like the red marked in the picture. In the code there is still the Grid Search inside.
Question 1:
Is this code doing that what I want? (is that a cross validation to train the dataset?)
Question 2:
when I have 2 models like in the picture, model 1 with cross validation (red marked) and model 2 with train validation test data -> what are the reasons to choose model 1 with cross validation and why?

X = df.drop('columnA', axis=1)
y = df.columnA

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.20, random_state=42)

xgb_params = {
    'max_depth': [6],
    'learning_rate': [0.3]}

xgb_clf = GridSearchCV(XGBClassifier(random_state=42),
                      xgb_params,
                      cv=10,
                      n_jobs=-1,
                      verbose=True)

xgb_clf.fit(X_train, y_train)

xgb_pred = xgb_clf.predict(X_test)
print(accuracy_score(xgb_pred, y_test))
print(f1_score(xgb_pred, y_test, average='micro'))

Im sorry if my point of view is strange, but I have a lack of knowledge and Im confused about Cross Validation and Kfold and how to use it.

I think you confused some technical names. Cross-Validation is the name of the procedure, and it has some techniques or approaches such as k-fold cross-validation, train test split, etc. All are techniques to measure the performance of a model.
In your case, you have the first model that is assessed using 10-fold cross-validation and has an f1-score of 0.941, and the second model is assessed using the train test split approach and has an f1-score of 0.953. In this case, choosing the better model depends on what you want to give the privilege. In other words, whether you focus more on False Predictions or True Predictions, or False Negatives of False Positives. For this purpose, check the confusion matrix below.

There are several scenarios which mainly are:

Using Recall metric

You can use it if False Negatives are more important for you. In the medical analysis, for example, False Negatives are tried to be minimized and scientist measures model's performance essentially by using Recall. Because, for example, it is a more acceptable case to predict a person cancer but actually that person is healthy while compared to the case where a person is labeled as healthy but in actual that person has cancer.

Using Precision metric

It is preferred when False Positives are more important than False Negatives.

Using F1-score

It helps to identify the state of incorrectly classified samples. In other words, False Negative and False Positives are attached more importance.

Using Accuracy score

It is mostly used when True Positive and True Negatives are prioritized.
So back to your question, you should not choose the model that best 'fits' (or performs) in a test, but you should choose a model that fits your demands most and the test must be chosen depending on the demands. If you want to have balanced False Negatives and False positives use the f-score to choose a model, which is the model2 in the above-mentioned case. However, if you focus on a different target, make your choice according to a performance metric.


Generate series of values using Keras GAN architecture I'm trying to generate something like that:
Which is a random sample from my real data function (that i'm trying to mimic).
numbers = [ 54.,  87., 103., 209., 356., 371., 383., 448., 452.,  38.,  37.,
          30.,  22.,  24.,  48.,  52.,  61.,  66.,  87., 150., 163., 406.,
         545., 557., 566., 508., 413.,  26.,  22.,  22.,  12.,  19.,  22.,
          26.,  28.,  90.,  93., 126., 155., 362., 476., 470., 455., 410.,
         345., 252., 233.,  62.,  45.,  42.,  40.,  38.,  37.,  26.,  27.,
          16.,  18.,  21.,  21.,  22.,  63.,  67.,  96., 177., 228., 331.,
         382., 183.,  38.,  31.,  14.,  13.,  13.,  18.,  19.,  21.,  23.,
          61.,  68.,  93., 104., 179., 273., 428., 446., 388.,  96.,  77.,
          19.,  18.,  13.,   4.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
           0.]

How would your Generator & Discriminator would look like???
Here's my attempt to create G & D that will be able to do so:
def create_G(num=100):
  G_in = Input(shape=num)
  x = Dense(num/2,activation=LeakyReLU())(G_in)
  x = Dropout(0.5)(x)
  x = Dense(num/4,activation=LeakyReLU())(x)
  x = Dropout(0.5)(x)
  x = Dense(num)(x)
  G = Model(G_in,x)
  G.compile(loss='binary_crossentropy',optimizer=Adam(learning_rate=0.001))
  return G

def create_D(num=100):
  D_in = Input(shape=num)
  x = Reshape((-1,1))(D_in)
  x = Conv1D(num,3,activation='relu')(x)
  x = Dropout(0.5)(x)
  x = Flatten()(x)
  x = Dense(2,activation='sigmoid')(x)
  D = Model(D_in,x)
  D.compile(loss='binary_crossentropy',optimizer=Adam(lr=0.003))
  return D

I'm training the right way, with fake and real examples.
But my G is not able to fool the D.
Here is an example of the values of G.predict(noise) in blue, compare to the real values in orange .

What am I missing?
Would appreciate any suggestions for different architecture.

In case it will help someone in the future.
This is what solved it...
def create_G(num=100):
    G_in = Input(shape=noise_size)
    x = Reshape((-1,1))(G_in)
    x = Conv1D(2,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(4,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(8,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(16,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(32,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(64,3,activation=LeakyReLU())(x)
    x = BatchNormalization()(x)
    x = Dropout(0.5)(x)
    x = Conv1D(100,3,activation='tanh')(x)
    x = Flatten()(x)
    G = Model(G_in,x,name='Generator')
    G.compile(loss=wasserstein_loss,optimizer=G_rmsprop_optimizer)
    return G

def create_D(num=100):
    D_in = Input(shape=num)
    x = Reshape((-1,1))(D_in)
    x = Conv1D(num/2,3,activation=LeakyReLU())(x)
    x = Conv1D(num/10,3,activation=LeakyReLU())(x)
    x = Conv1D(num/50,3,activation=LeakyReLU())(x)
    x = Flatten()(x)
    x = Dense(128,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(64,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(32,activation=LeakyReLU())(x)
    x = Dropout(0.5)(x)
    x = Dense(8,activation=LeakyReLU())(x)
    x = Dense(2,activation='sigmoid')(x)
    D = Model(D_in,x,name='Discriminator')
    D.compile(loss=wasserstein_loss,optimizer=D_rmsprop_optimizer)
    return D

Cheers :)


What is the order of elements in an image in python? I have a set of images which are loaded from an h5 file. I checked their dimensions and I get (209, 64, 64, 3).

read = h5py.File('datasets/train_catvnoncat.h5', 'r')
read['train_set_x'].shape

  (209, 64, 64, 3)

it means that there are 209 images but the point that I cannot understand is that, what is (64, 64, 3)?
I have used the following code for plotting:

import matplotlib.pyplot as plt

plt.imshow(read['train_set_x'][1])
plt.show()

and I get a colored image which is 64 by 64. before this, I thought for (., ., .) shapes, the second number specifies the number of lines and the third one specifies the number of rows. also the first one specifies the number of the mentioned (row and column) arrays.
My question is that in numpy if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator; Why this is different in images and rows and columns are arranged differently in images. Shouldn't it be (3, 64, 64)?

The [64, 64, 3] shape you have found is a common convention to represent a colour image in (x, y, colour_channel) dimensions.

The key word here is convention - there is no inherently preferred way to represent a colour image in terms of fundamental maths or computing needs, and even within Python you will find multiple conventions, varying in the ordering within the dimension - e.g. OpenCV uses (x, y, channel) convention for the shape, but has channels in order BGR - so channel 0 is blue - whilst most other libraries will use RGB ordering (ignoring for now the alternative colour spaces).

  My question is that in numpy if you have a three dimensional array, for accessing rows and columns you have to change the second and third entries in the indexing operator

When you have a 3-dimensional array, what you decide to call "rows" and "columns" is also a convention. It depends partly on what that array represents, and there is no single way to visualise the contents.


Preparing training data for NLP machine learning task I have the natural language sentences as follows:
This is a black chair. It is next to the table.
Each phrase that represents an object is annotated with an object Id. For example, in the above sentence, we have:
This: 15, black chair: 15, It: 15, table: 14
(where, 14 and 15 are object Ids)
I would like to train a model to predict the object Id of each phrase representing an object for a new sentence. From what I understand, each training example will consist of the following structure:
Inputs: sentence + object phrase
Output: object id (from 18 available ids)
I would need to repeat the above for each object phrase in a sentence
My question: How do I prepare the training data for this task? How do I represent each object phrase (eg: 'black chair') and each sentence for training the neural network?

Nlp like spacy remove stop works and identify proper nouns and nouns.  It does fairly well at identifying nouns but is not perfect.  Try a sample of your data and see how accurate your percentages become.  Use the medium model


Tensorflow uses more memory, the more epochs it completes I created a genetic algorithm "optimizer" for Tensorflow but it is written in python. I know TensorFlow was not designed like this and I need to rather create the optimizer in C++ using their API's but I found out about it only after I already programmed the optimizer and I don't really have the time to do research on how to create a tf.train.Optimizer(). My GA "optimizer" is not really an optimizer as it is more a tf.Variable.assign() call. See code below

The "optimizer" updated a series of weight matrix variables after the evolve takes place. Each variable represents a weight matrix between two layers. If it is a network with inputs, one hidden layer and outputs then there will be two variables. Weights between input to hidden then hidden to output. Although this is a 2d matrix but my weights act as an individual in my population so I have a 3d matrix. 1 dimension represents the population amount and the 2d matrix as the weigths. Weight updates are then done by crossover or mutation which consists of a lot of multiplication, matmul and summation operations. My guess is the variables I create for the new layers is not garbage collected. I am guessing tensorflow keeps the variables alive as I don't ever call a train step provided by the tf.train.Optimzer classes since I created my own.

Here is my code to run my GA optimizer

for i in range(epochs):
    sess.run(fitness_per_individual, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)
    evolved_layers = evolve_layer(nn_layers, fitness_per_individual, population)
    for tensor_idx in range(len(nn_layers)):
        layer = nn_layers[tensor_idx]
        evolved_layer = evolved_layers[tensor_idx]
        layer = layer.assign(evolved_layer)

        # I don't have to feed my training data again I just do it here to keep tensorflow happy.
        sess.run(layer, feed_dict={x: train_data, y: train_labels},
                 options=run_options,
                 run_metadata=run_metadata)

Is there a way I can dispose of variable matrices created by tf.multiply or is there no way around it without creating a tf.train.Optimizer. Also is there a good practice guideline I can read up for TensorFlow.

It seems like you are adding nodes to your computational graph in this line: layer = layer.assign(evolved_layer). The assign-operation is just as multiplication or addition a node in the graph and you construct a new one in every step. Call tf.Graph.finalize() after defining your model. This will raise an error whenever a node is added to the graph and help you debugging your code. Ideally you want the graph to be fixed after construction.  I suspect you will get an error in the optimizer.

You want all the operations to be defined before you run the training loop and then just run the respective nodes. In your case you'd define layer_assign = layer.assign(evolved_layer) before the loop and in the loop only use sess.run(layer_assign). Of course you would have to treat all operations in evolve_layers in the same way.


train test data split up in datasets In a dataset consisting of 1,000 samples, it has been shown that a 70-30 split
(i.e. 70% of the samples used for training, 30% for validation) will provide a good estimation
of the test accuracy of the trained models. If the dataset size increases to 10,000 samples, what
split would you suggest?

The current approach use 70/30 or 80/20, the most used is 80/20 (train/test). However there is other things you should check, for example if you data is balanced. If your data is not balanced you might want to use undersample or oversample.


Machine Learning - Same impurity values Imagine this example:

We see that the attributes color and points have the same value. What attribute should we choose for the construction of the tree?

A decision tree will automatically select the attributes that are predictive of the targets. The automatic learning of meaningful features is one the primary advantages of machine learning in general and decision tree in particular.


Deep network not able to learn imbalanced data beyond the dominant class I have data with 5 output classes. The training data has the following no of samples for these 5 classes:
[706326,  32211,   2856,   3050,    901]

I am using the following keras (tf.keras) code:

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(50, input_shape=(dataX.shape[1],)),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(50, activation=tf.nn.relu),
    tf.keras.layers.Dropout(rate = 0.5),
    tf.keras.layers.Dense(5, activation=tf.nn.softmax) ])
     adam = tf.keras.optimizers.Adam(lr=0.5)

model.compile(optimizer=adam,
              loss='sparse_categorical_crossentropy',
              metrics=[metrics.sparse_categorical_accuracy])
     model.fit(X_train,y_train, epochs=5, batch_size=32, class_weight=class_weights)

y_pred = np.argmax(model.predict(X_test), axis=1)

The first line on class_weight is taken from one of the answers in to this question: How to set class weights for imbalanced classes in Keras?

I know about this answer: Multi-class neural net always predicting 1 class after optimization . The difference is that in that case, the class weights wasn't used whereas I am using it.

I am using sparse_categorical_crossentropy which accepts categories as integers (don't need to convert them to one-hot encoding), but I also tried categorical_crossentropy and still the same problem.

I have of course tried different learning rate, batch_size, no of epochs, optimizer, and depth/length of the network. But it always is stuck at ~0.94 accuracy which is essentially I would get if I predict the first class all the time.

Not sure what is missing here. Any error? Or should I use some other specialized deep network?

1) A five-layer neural network is one heck of a complex model for a data set with less than 1 million points. (I’m trying to find a good link for this, but the intuition is that your choice of model should be driven by the complexity of the available data, and not by what you think the real target function is like.) If this is for a real-world project, a tool like XGBoost might work better on this data set — out of the box, you’ll spend less time dealing with problems related to imbalanced classes, poorly scaled data, or outliers. Of course if this is specifically for learning about neural networks, that advice isn’t much help!

2) For a class distribution that’s as skewed as your data, you might get more mileage from re-sampling the training data rather than re-weighting the classes during training. Down-sample the majority class first (just throw away majority samples at random); if that’s not satisfactory then try something more complicated like using SMOTE to up-sample the minority classes. Try taking this to the extreme; build a (collection of) new training sets by randomly sampling only 1,000 points from each class.

The intuition here is that, for neural networks, as far as I know, re-weighting classes basically means re-scaling the gradient steps during training based on the class weights. If the classes are skewed 10:1, that makes sense: we take a step that’s 10 times as far for a minority sample. If the classes are skewed 1000:1, as in your case, it makes less sense — we’ll take 1,000 small steps as we optimize on the majority class, and then a single gigantic step in an essentially random direction when we happen to see a minority sample, followed by 1,000 tiny steps trying to un-do this work, etc. We don’t see enough minority samples to allow information about their class to average out.


How to consider categorical variables in distance based algorithms like KNN or SVM? For example lets say I have a dataset with independent features age, gender, name, and income. While my dependent variable is load approval status.

If I want to use KNN or SVM, do I need to convert gender into a numerical variable? and if yes, how is it used in algorithms such as KNN or SVM which are distance based algorithms?

The traditional way of dealing with categorical variables is to one-hot encode these. One hot encoding will transform one categorical variable with n values into n different features which values can be only 1 or 0, 1 indicating the presence of the corresponding category and 0 the absence.

As an example, instead of having:

---------------
|id  |Category|
---------------
| 0  |    A   |
---------------
| 1  |    B   |
---------------
| 2  |    A   |
---------------

You would have:

--------------------------------
|id  | Category_A | Category_B |
--------------------------------
| 0  |     1      |     0      |
--------------------------------
| 1  |     0      |     1      |
--------------------------------
| 2  |     1      |     0      |
--------------------------------

Distance based algorithms then use the vectors representing the features to compute distances, whether it is the euclidean, cosine, hamming distance... After one hot encoding, you can see that these vectors are exclusively numeric, the 'categorical' aspect being translated into the addition of new dimensions.

With one hot encoding, there are a few common things you must watch out for:

It is recommended to drop the last column created by one hot encoding, because it can be inferred from the others. As an example, if you have a feature F taking three possible values A, B and C, one hot encoding would produce three columns F_A, F_B and F_C. You have F_C = not(F_A + F_B), i.e. 'if the value is neither A nor B then it is C'. In this case you should keep only columns F_A and F_B (or F_A and F_C , F_B and F_C whatever) since the third column is redundant and adds therefore no information. Keep in mind that if you F allows the presence of missing values, this has to be taken in account by either imputing the values, or considering the 'missing value' as a fourth possible category.
One hot encoding augments the dimension and the sparsity of the dataset. This can also harm the performance of the algorithm, especially when you have a lot of categorical variables with a lot of possible values.In order to avoid that, you can apply dimensionality reduction (as an example with PCA) to the one hot encoded dataset, and/or perform feature selection.


Random Parameters to fix in ML to perform controlled experiments Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy results. For example, running Random Forest can produce accuracy of 0.78, then when run again with no change in data, setup, code, it can result in 0.79. This brings the challenge of inability to perform controlled experiment, when I am testing some changes in input and their effect on output.

So in order to be able to perform perfectly controlled experiments to achieve the best model output, what are the extensive set of random parameters I should fix? I want whole process to be completely deterministic.

PS: I am using Sci-kit Learn environment with additional algorithms such as XGBoost, CatBoost, LightGBM.

I assume there are some parameters, random_state(s) I should fix in NumPy, too.

Scikit uses numpy for pseudo-random number generation. So to fix random state in various scikit calls, you use numpy.random.seed(12345) and then use scikit. You would want to record the random seed when you log the model so you could reproduce the same run later.

If your code (or something you call) also uses Python's random number generator, you would set random.seed too.

How to set the seed depends on the library. For example I believe most xgboost APIs expose a seed parameter instead. Not sure about catboost.

You're also depending on the library even exposing a way to seed any pseudo-random choices it makes. It's possible some library doesn't quite do so. (Or in the case of Spark, sometimes the result can even conceivably depend on order of distributed execution, which is hard to control.)


Pandas throwing "Error tokenizing data. C error" while loading data sets from URL I am trying to work on the Titanic competition to get hands on experience with data science & machine learning.  I tried to load up the datasets from GitHub but pandas threw the following error:

ParserError: Error tokenizing data. C error: Expected 1 fields in line 32, saw 2

I tried to follow the advice of other SO users so I added skiprows=1 parameter in my pd.csv() call to skip the first row but it didn't work.

import pandas as pd

train_dataset = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/train.csv", skiprows=1)
test_dataset = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/test.csv", skiprows=1)
ground_truths = pd.read_csv("https://github.com/oo92/titanic-files/blob/master/gender_submission.csv", skiprows=1)

train_dataset.head()

The path that you are accessing from is a Github repository page which is a webpage, it does not return CSV. You have to click on 'raw' option in Github and then pass the URL which in your case is:

test = pd.read_csv('https://raw.githubusercontent.com/oo92/Titanic-Kaggle/master/test.csv')


Scheduler for activation layer parameter using Keras callback I want to train a keras model with a custom activation layer.
The custom activation layer has one fixed non trainable parameter.
I want to change/set this non trainable parameter of all custom activation layers in model during training after few epochs.
How to achieve this using keras callback?

You will need to write a custom callback for this, that implements the on_epoch_end method. Roughly it should look something like this
class CustomCallback(keras.callbacks.Callback):

    def __init__(self, freq):
        super().__init__()
        self.freq = freq   # how often to change the parameter

    def on_epoch_end(self, epoch):

        if epoch % freq == 0 and epoch > 0:

            weights = self.model.get_weights()

            # here you change the weight you want, e.g. it is the 5th layer
            weights[4] = weights[4] / 10

            self.model.set_weights(weights)


Why does Gradient Boosting regression predict negative values when there are no negative y-values in my training set? As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my training or testing set. I have about 10 features, most of which are binary.

Some of the parameters that I was tuning were:

the number of trees/iterations;
learning depth;
and learning rate.

The percentage of negative values seemed to max at ~2%. The learning depth of 1(stumps) seemed to have the largest % of negative values. This percentage also seemed to increase with more trees and a smaller learning rate. The dataset is from one of the kaggle playground competitions.

My code is something like:

from sklearn.ensemble import GradientBoostingRegressor

X_train, X_test, y_train, y_test = train_test_split(X, y)

reg = GradientBoostingRegressor(n_estimators=8000, max_depth=1, loss = 'ls', learning_rate = .01)

reg.fit(X_train, y_train)

ypred = reg.predict(X_test)

Remember that the GradientBoostingRegressor (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now if the tree in stage i predicts a value larger than the target variable for a particular training example, the residual of stage i for that example is going to be negative, and so the regression tree at stage i+1 will face negative target values (which are the residuals from stage i).
As the boosting algorithm adds up all these trees to make the final prediction, I believe this can explain why you may end up with negative predictions, even though all the target values in the training set were positive, especially as you mentioned that this happens more often when you increase the number of trees.


RF and DT overfitting I am new with Machine Learning and I started with some lessons in Kaggle. There, I learnt how to use DecisionTreeRegressor() and RandomForestRegressor() from sklearn.

However, I cannot really understand how I can verify that my explanatory variables do not overfit the model. For example, the lessons included evaluation with the use of Mean Absolute Error. MAE and MRSE can evaluate whether my Decision Tree depth is optimal or not, but not if my explanatory data are even relevant.

I come from Economics, so I am used to deal with such problems using diagnostics or $R^2$. Is there any equivalent to $R^2$ benchmark to determine whether my explanatory variables are overfitting my model or not?

Yes, and that benchmark is called validation data. The idea is to split your data in some training data, which is what you are going to fit to your models in order to estimate the parameters in them (and make them learn), and some validation data, which is the data you are going to use in order to evaluate your model.

What you can do is compute the error on your validation data (which the models haven't been trained on) for the different models you want to use, and you would like to keep the model with the lowest validation error.


Set seed for a Class that calls Keras Models I have class that I use to optimize parameters of a Keras LSTM model. It is known that to set seed for keras, one must input the follow on its code. But what I'm not understanding is where to put it in the case of a class that will build and modify the models.
from numpy.random import seed
seed(1)
from tensorflow.random import set_seed
set_seed(2)

Should it be in the __init__ like bellow?
from numpy.random import seed
from tensorflow.random import set_seed

class OptimizeLSTM:
    def __init__(self, X_train, y_train, X_test, y_test, verbose=False):
        self._X_train = X_train
        self._y_train = y_train
        self._X_test = X_test
        self._y_test = y_test
        self._verbose = verbose
        seed(1)
        set_seed(2)

No you should declare it ahead of the class, just after importing the packages. You have declare the seed for numpy and tensorflow separately. For further details read this blog -
https://machinelearningmastery.com/reproducible-results-neural-networks-keras/


How can I perform weighted addition of three layers in keras? I would like to perform the weighted addition of three outputs from different Keras layers such that the weights are trainable. How can I achieve this? I am using tensorflow 2.0 as backend for Keras.

I solved the problem using subclassing in keras. The code is shown below:
class Wt_Add(keras.layers.Layer):
def __init__(self, units=1, input_dim=1):
    super(Wt_Add, self).__init__()
    w_init = tf.random_normal_initializer()
    self.w1 = tf.Variable(
        initial_value=w_init(shape=(input_dim, units), dtype="float32"),
        trainable=True,
    )
    self.w2 = tf.Variable(
        initial_value=w_init(shape=(input_dim, units), dtype="float32"),
        trainable=True,
    )
    self.w3 = tf.Variable(
        initial_value=w_init(shape=(input_dim, units), dtype="float32"),
        trainable=True,
    )

def call(self, input1, input2, input3):
    return tf.multiply(input1,self.w1) + tf.multiply(input2, self.w2) + tf.multiply(input3, self.w3)

Usage:
wt_add = Wt_Add(1,1)
sum_layer = wt_add(input1, input2, input3)


Partitioning Weighted Undirected Graph I have an array of edges and weights:

[['a', 'b', 4],
 ['a', 'c', 3],
 ['c', 'a', 2],
 ...]

I have about 100,000 edges and weights are between 1 and 700, most around 100.

I am thinking of using Markov Cluster Algorithm however wanted to reach out to see if this is the best to use. What about Affinity Propagation? In either case, what is the workflow? Do you typically have a way to measure how well clustered the results. Is there an equivalent to a silhouette score? Is there a way to visualize the clusters?

Even a simple Internet search reveals numerous papers on graph clustering approaches and algorithms. This paper is most likely the best starting point, as it presents a rather comprehensive overview of the topic in terms of the problem as well as approaches, methods and algorithms for solutions. The rest you can find easily via online search. In regard to graph clustering visualization, I recommend you to check my relevant answer - I'm pretty sure that the tools I reference there are able to visualize graph clusters as well.


Having Trouble Using Orange Data Mining in Linear Progression in Continuous Data I'm having trouble with a simple task. I just imported an Excel sheet (CSV) into Orange.

It recognized the features as continuous, but it couldn't run Linear Progression or Discretized on it.

How can I debug/fix this?

Orange needs either a one-line or three-line header. Yours is two-line, hence it will cause confusion. Either add an extra blank line below the second or remove the second line altogether (Orange should recognize these values correctly).


Validation curve unlike SKLearn sample I'm trying to implement the validation curve based on this SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs X and y are shaped (266531, 23) and (266531,).

The curve looks like this:

And my code is:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil

np.random.seed(0)

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.show()

UPDATE

A comment suggested I made X and y identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.

This is exactly your code just with digits data:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil
from sklearn.tree import DecisionTreeClassifier

np.random.seed(0)

# X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()

digits = load_digits()
X, y = digits.data, digits.target

indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.figure(figsize=(10,10))
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
#plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color="r")
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
#plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color="g")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.savefig('DSSE.jpg')
plt.show()

And this is the result:

So whatever the problem is, it's in your data that you didn't explain how you created it. Either double check it or post it here so your question can be answered. I assume somewhere in data preparation you made X and Y identical by mistake.


How to generate training instances from multivariate normal distribution? I would like to generate a training and test set for the binary classification problem. I want the instances to be generated from a multivariate normal distribution and have 5 features. Can someone explain how to do this in python?

This can be done by using inverse transformation theorem
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.preprocessing import QuantileTransformer

def make_gausssians_binary_classification(n_samples= 1000,
                                          n_features= 5,
                                          n_informative= 5,
                                          n_redundant= 0,
                                          random_state= 42
                                          ):

    X, y = make_classification(n_samples= n_samples,
                           n_features= n_features,
                           n_informative= n_informative,
                           n_redundant= n_redundant,
                           random_state= random_state)

    transformation = QuantileTransformer(output_distribution= "normal").fit(X)

    X_normal = transformation.transform(X)

    return X_normal, y

X, y = make_gausssians_binary_classification(n_samples=10000)


Using machine learning technique to predict commodity prices Has anyone here tried to predict a commodity's price by using other commodities prices as features in a machine learning algorithm? What techniques have been successful?

Based on your question there are couple of things which I would assume to answer your question:

As you need to predict the commodity price the data which is collected is time series data.
Since you want to use other commodity to predict, it means that you don't have any past data of the product which you want to predict.

The answer could be derived by performing some exploratory analysis on the existing data. i.e., based on your business understanding you need to decide which product is similar to the new product. This kind of techniques is used to understand the sale of the new product/how is it going to perform after the launch.

Techniques which can be used over here are Time Series Analysis like ARIMA, if seasonality is present then SARIMA, if no trend then Exponential Smoothing(too many spikes),  there are other models like Auto Regression, Moving Average, Croston if there are 0's etc.

This is one way of looking at your problem.


Doubt on retrieving tweets for sentiment analysis? I wanted to do my first project on sentiment analysis.
My goal is trying to do sentiment analysis on tweets that mention
'Uber' and see if there is correlation with the stock price: 'UBER'.

Problem is it went public this year, so I would need tweets from this year, my question is:

Is there any way of avoiding paying for the api:
 https://developer.twitter.com/en/pricing   ? How can I get let's say 10,000 tweets of this year mentioning uber?

Or maybe, given I don't have any experience shall I instead try doing it on an already created dataset? ie https://www.kaggle.com/davidwallach/financial-tweets ? (problem here is that this data set only has 28k tweets, and if I filter for the ones only naming stock 'xyz', I will likely get only a hundred tweets...)

First you need to setup the twitter api and get the credentials and add those to the congig.cfg file linked here-
https://filebin.net/5a6lfserhqzjg93a/config.cfg?t=dcqh998i

Afterwards you can use this link https://filebin.net/nfeju0bjvh8tocl3/tweetering.py?t=dy6bk7x5 to download the tweets which contain Uber.
Just run it on command line using - python tweetering.py Uber

Hope it works for you. You can also find many links on medium.com where you can find the complete twitter sentiment analysis steps, reading which you will understand a lot of things.


What is Image Annotation? Why do we need to use Labelimg tool for object detection?

After labeling the bunch of training images using labelimg tool which will give CSV file How that CSV file works with TensorFlow object detection API and Keras?

Can we detect image localization without an image annotation tool? like auto annotation.

As per my understanding, ML works on the concept of the type of Input+Output you feed during training, will expect(Input) and predict the same output.

So labeling is must for Supervised learning.

For Image localization use case, you have to trained your model with similar type of data i.e. label should have full annotation/details with respect to dimensions(x,y,height,width) and object class. Only then you can expect the Object localization(bounding box) during prediction.

Few of my posts possibly be helpful for you:

How YOLO training and prediction works for an object fall in multiple grid?

YOLO: How many bounding boxes?

Please go through these videos for more details. Object Detetction and Localization


How to get the number of syllables in a word? I have already gone through this post which uses nltk's cmudict for counting the number of syllables in a word:

from nltk.corpus import cmudict
d = cmudict.dict()
def nsyl(word):
  return [len(list(y for y in x if y[-1].isdigit())) for x in d[word.lower()]]

However, for words outside the cmu's dictionary like names for example: Rohit, it doesn't give a result.

So, is there any other/better way to count syllables for a word?

You can try another Python library called Pyphen. It's easy to use and supports a lot of languages.

import pyphen
dic = pyphen.Pyphen(lang='en')
print dic.inserted('Rohit')
>>'Ro-hit'


How to transform a folder of images in a csv file I have a folder with a lot of images that I want to use to bild a classificator using a SVM model in python with sklearn.
I've always used csv file as train/test set with sklearn, how can I make it? (a csv file with records corrisponding to images and a variable for every pixel)

You are describing a one-time pre-processing step that will crawl through your folder and turn each image into a line of data and then save the entire collection in a CSV file. In turn, that file becomes your gold standard dataset.

If I was in your position, I would look into the Keras pre-processing tools that already provide python libraries to quickly do this task. It's a common need for image processing, the Keras library is very mature and can do this for you.


Alternative methods of population-level classification In a recent study, I am generating a series of logistic regression and random forest models to predict some binary outcome. For the sake of understanding, suppose this outcome is the occurrence of diabetes in 1,000 patients at a local hospital. I know the occurrence of diabetes in this population, and can thus generate a number of classification models of this outcome using this population.

Suppose I then want to use these predicted models to project the probability of diabetes in a second population where I don't observe diabetes classification (say, N = 10,000). I can use my models generated above to predict diabetes on a person-by-person basis in this second population.

For this particular study, I'm most interested in predicting the level of diabetes at the population-level of this second population. For example, I may want to know the % of this second population that potentially has diabetes.

In the literature, it appears that most studies use some threshold (say, 50%) and sum up the # of observations with a predicted probability greater than the threshold. However, I've found other studies that have simply summed the probabilities to generate the # of, say, diabetes patients in this second population.

I'm not overly familiar with the machine learning literature, but do both of these approaches make sense? Both approaches make sense and likely lead to similar estimates with well-balanced data. However, unbalanced data may lead to distributions where the choice matters (see attached). Thoughts?

I think the second approach is much superior to the first approach for two main reasons.

Reason #1: The first approach forces you to set a parameter (the threshold) somewhat arbitrarily.  When reporting your results, I think you would need to justify your choice of threshold.  Sure, 50% seems reasonable prima facie, but what if your models happen to underestimate the probability of diabetes?  Then a lower threshold might be appropriate.
This is particularly concerning if you are doing academic work, since it would allow you to "fudge" results by nudging the parameter in one direction or another.  It's best to avoid arbitrary parameters when possible.

Reason #2: You've already noted that, for some distributions, the choice of method will yield different estimates.  The second method will give you a more robust estimate of the target statistic.

To illustrate, suppose you have a very accurate model. Consider a case where your "second population" has five members, and your model outputs the following probabilities of diabetes for each member:

[0.49, 0.49, 0.49, 0.49, 0.49]

If you use the first method to estimate the percentage of the population that has diabetes (with a threshold of 50%), then your estimate will be 0%.  This is clearly unrealistic - the probability of none of the patients having diabetes is only $0.51^5 = .0345 = 3.45\%$.

If you use the second method, then you will compute the expected number of diabetes cases as follows:

$0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 + 0.49*1 = 2.45$ cases

If you express this estimate as a percentage of the population, then your estimate will (obviously) be 49%.

I suppose you could also consider this estimate flawed, since it is impossible to actually observe. This will become less of a problem for large populations (like your case with n=10,000), but if it bothers you then you could round the expected number of diabetes cases to the nearest whole number.  In the contrived example above, this would yield an estimate of 40%.  This is still a much better estimate than that obtained using the first method.

Hopefully that helps!


Evaluating likelihood of egg breaking when falling in random container on concrete I am working on a project where I would like to predict whether an egg will break if it is put in a container that is then dropped on concrete.

I am looking at the different factors that play a role in whether the egg will break. So far, I have come up with the following:

mass
friction
padding
shape of container

What other variables could help predict the outcome?

This question might be better answered on the physics stackexchange, but I'll take a crack at it.

Although the factors you mentioned probably affect the likelihood the egg will break, some of them are not specific enough to be descriptive features. For example, how do you quantify "shape"? I will try to provide some more features that can be quantified, but some of them might take effort to retrieve.

Egg Features

Eggshell thickness

the thickness will probably not be constant, so you can take the mean, standard deviation, and even isolate some key points on the egg to sample such as the top/bottom points.
some studies indicate that "eggs with thin but more uniform eggshell were stronger than those with thick but less uniform eggshell" from this paper
you might be able to measure thickness using optical techniques such as imaging or lasers.

Temperature

both of the egg and the surroundings

Time since egg was laid

this might be approximated by the expiration date, particularly when only using a single brand

Density

you can calculate this by putting the egg in water (fresh eggs sink) to first determine the volume, then divide the mass by this volume.

Calcium deposits

the bumps you can see and sometimes feel on chicken eggs are calcium deposits that range in frequency and size. I am not sure how you would quantify these, but they could be another factor.

Wikipedia indicates some abnormalities with eggs that you might want to account for or remove from your samples. I copied the abnormalities here:

double-yolk eggs
yolkless eggs
Double-shelled eggs: where an egg may have two or more outer shells, is caused by a counter-peristalsis contraction and occurs when a second oocyte is released by the ovary before the first egg has completely traveled through the oviduct and been laid.
Shell-less or thin-shell eggs possibly caused by egg drop syndrome

Container Features

ideally the impact energy would not be absorbed by the egg, so somehow you need to measure this using estimates of the material metrics as well as shape parameters. One such measurement might be:

Average surface area that contacts the ground

You might find looking at the container's convex hull useful for visualization purposes

Many other factors can contribute such as:

Hardness
Resilience
Elasticity
Stiffness
Ductility

Lastly, there are probably many features that can be calculated by combining the egg and container, but those seem to be much more complicated. For example, measuring the amount of movement, on average, an egg can endure within a container by applying pressure to certain areas. I am not sure how you would easily extract these types of data, but they certainly exist.


Is it possible the model be better on a few epochs rather than hundreds of epochs? I have very interesting experience in my CNN binary image classification. Do you think the result is by chance or there is a logic behind it?
I used InceptionV3 transfer with softmax (I know you will say why not ReLU) but it is what I did.
I trained on 100 epochs. But the result was terrible. from the training process I noticed in the 12-th epochs the result is excellent (both train accuracy and validation accuracy ).  So I trained the model on 12-th epochs. and suprisely the result on the test data was also excellent.
Does it mean on a few epoch the result is better?

Yes, this is the reason you should use 'early stopping' in your models which will stop training when the model is not improving or you can keep the history of the training to pick the epoch that had the best performance.
The reason you get excellent results in the 12th epoch, but terrible performance in the 100th epoch is simply you are overtraining. By overtraining, you are causing overfitting, and the model is not able to generalize, instead, it imitates your data. Thus, the model will have high accuracy in in-sample data but comparably bad in out-of-sample data when you train a lot.
Moreover, take into account that unnecessarily complex and poorly regularized models are likely to overfit also. Especially, when the input data size is small. But in any case, if you lose performance as you train the model, this is probably because of overtraining.
For this reason, try to always have a graph of training accuracy vs test accuracy (or validation accuracy) by epoch. Thus, you can observe where (in which epochs) train and test accuracy move together. Where your train accuracy is >> test (or validation) accuracy then there happens overfitting, and where your test (or validation) accuracy is >> train accuracy then you are underfitting there.


trade offs between number of features with its score I am running k-mean clustering on ~200000 samples. The dataset has in total 14 features. One feature is id and the rest are categorical.

I have been playing with which features to include in the clustering and the metric Im using is Silhouette.

I would like advice on how to decide which analysis is better. A cluster with fewer features and a higher score (i.e., .8) or a cluster with more features and a lower score (i.e., 30)

My assumption is that the one with more features and a lower score is better because the algorithm has more information that describes the sample. However, those extra features may be making it harder for the algorithm to put the samples into groups.

Any advice/tips?

First of all, the Silhouette score comes always with visual inspection so be careful using it. The idea behind this score is having either Normally distributed clusters or having pretty compact clusters. If the intrinsic clusters within your data are not one of those, then this score is somehow meaningless.

Secondly, the question also needs a re-thinking. "Which features to include in the clustering?" ... well, all!

The characteristic of a dataset is more than just the combination of characteristics of each individual features. In these cases, usually Dimensionality Reduction algorithms are used, which reduce the dimensionality of data and extract/select informative features according to the entire feature set. Note that a feature that you might exclude, makes a meaningful partitioning in combination with some other features.

at the end I recommend having a look at K-Modes which is designed for clustering datasets with nominal attributes. Most probably helps you more than k-mans.

If you like to get some insight about Graph Clustering (which is also related to your problem and is one of typical solutions) you can comment here.

Good Luck!


What is bad, good and excellent metric score for time series model? I have created a couple of models for my master project and I used several metrics for evaluation. I used MSE, MAE, MAPE, RMSE not because I really learned about them a lot, because I saw in many other projects these metrics being used.
Now I have a problem, I need to interpret results. I search for some articles or some studies that classify metrics performance as good or bad or excellent. The only material I found now is this one.
With this interpretation

MAPE <     10% is Excellent, MAPE <     20% is Good

It would be nice to provide this formal or informal interpretation for each of the metrics MSE, MAE, MAPE, RMSE specifically for time series prediction problems.

I do not think there is a silver bullet for metric value, because it is highly problem-specific, one way or another you need to compare your result to baseline in order to understand how viable it is.
For example, in time series common baseline for the next point predicted is a previous point or some moving average. You can get an error metric like MAPE (but be careful with it, since it has its downsides) and compare your models MAPE to Baseline MAPE (say moving average) and with this comparison, you can argue about how your approach is better.
Also, check this article regarding MAPE and other metrics.


Classification of Orientation I am using inertial sensors to capture motion from the wrist/ ankles of subjects and synthesising this data into a feature set to be able to classify the activity performed by the subject (i.e. standing, walking, sitting).

On top of the metrics output by the sensors I would also like to be able to use the orientation of the limb segments for classification. I have been using an existing algorithm to determine the quaternions and can also convert these into Euler angles if need be.

My question is how best process these quaternions in order to feed them into a classifier to provide information about limb segment orientation to the classifier. The options for feeding the data into the classifier as far as I'm aware are as follows:

1) Feed each element of the quaterion as a feature: My concern is that each element of the quaternion is clearly non-linear and the classifier I use may not be sophisticated enough to be able to determine useful orientation information from this metric

2) Convert to Euler angles: This again suffers from a non-linear relationship with orientation relative to a world space depending on order of successive rotations (may be solved using extrinsic euler angles?)

3) Calculate Quaternion modulus from the non-unit quaternion: I have seen this method suggested in a paper on classification of EEG signals but I'm not sure of it's usefulness for this application. To be honest I'm not entirely sure on the what Quaternion modulus actually represents in real term other than for it's application in normalising quaternions.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4813911/#B40-sensors-16-00336

Many thanks in advance for any help!

I don’t know about your method 3, but I do know about quaternions and Euler angles. I was discussing them over a beer last night, believe it or not.

The orientation (attitude) at each point has 3 degrees of freedom so 3 Euler angles describe it completely. A quaternion has one extra element which is superfluous. The point of quaternions is that some geometrical calculations are easier than with Euler angles.

The fact that the relation between input data and output data is nonlinear is not a problem, if you have sufficient data and you have a nonlinear model (basically, unless you insist on logistic regression).

But you can make it far easier for your model to learn the relation if you transform your input, or compute a different input from your raw data, that has a simpler relation.

For example, as input for your model you could provide the Euler angles of the orientation of each foot, the lower leg, the upper leg. But you can also compute the bend at the knee and the ankle, and the model will probably learn from that more easily.


Correlation and Naive Bayes I would like to ask if the Pearson correlation between fields (but not the class field) of a dataset affects somehow the performance of Naive Bayes when applying it to the dataset in order to predict the class field.

As you probably know, naive here implies that the "fields" are independent. So your question boils down to does correlation imply dependence. Yes, it does. See here.

https://stats.stackexchange.com/questions/113417/does-non-zero-correlation-imply-dependence

So, if your features show correlation then this will have an adverse effect on the naive assumption. Despite this fact, Naive Bayes has been shown to be robust against this assumption. If your model still suffers from this, however, you could consider transforming the space to be independent with methods such as PCA.


Active Mask - image segmentation I'm trying to implement an active mask code derived from Active mask segmentation for the cell-volume computation and golgi-body segmentation of hela cell images, Srinivasa et al 2008.

I have found a code that will allow me to do this, however, there is one section of the code that is giving me errors and I don't know why. Here is the section of the code:

def _converge(P, R, b, max_iters_converge=1000):
    Pm = numpy.empty(P.shape, numpy.float32)  # Pre-allocate: saves time
    argmax = numpy.zeros_like(P)
    maxval = numpy.zeros(P.shape, numpy.float32)
    for i in range(max_iters_converge):
        maxval *= 0
        maxval -= 1e8
        for m in range(int(P.max()) + 1):
            Pm[:, :] = (P == m)
            Pm = ndimage.gaussian_filter(Pm, b) + R[m]
 # Pm=ndimage.convolve(Pm,numpy.ones((2*b+1,2*b+1)))+R[m]
            argmax[maxval < Pm] = m
            maxval = maxval * (argmax != m) + (argmax == m) * Pm
        if (P == argmax).all(): break
        P = argmax.copy()
    mis = defaultdict(range(1, P.max() + 1).__iter__().__next__())
    mis[0] = 0  # set 0 to 0, because it is special
    for i in range(P.size):
        P.flat[i] = mis[P.flat[i]]
    return P

The area that is giving me the most problem is the:

mis = defaultdict(range(1, P.max() + 1).__iter__().__next__())

Originally the code was written for Python 2.4 but I made some changes so that it will work on Python 3.6, but I don't really understand what the defaultdict is used for.

Here is a link to the full original source file: active mask

The defaultdict in this case is meant to return the next increasing integer each time a new element is encountered in P.flat().  But to also return the same integer each time that same element is found again. You should have encountered the error:

  TypeError: first argument must be callable or None

defaultdict expects a callable. It will call the passed in function each time there is a lookup miss, and set the dict to that value. To fix that error you need to remove the () from the __next__() to leave:

mis = defaultdict(range(1, P.max() + 1).__iter__().__next__)

However this is overly complicated for what it is doing, and can be simplified a bit by recognizing that there is no reason to have the upper limit in the range, it is just an increasing integer which can be done with itertools.count() liike:

import itertools as it
mis = defaultdict(it.count(1).__next__)


Decision Tree Classifier to classify values based on values of other columns I have data with multiple labels, for example

My X set is fromt second to third column, and I want to classify either first column or the last column, so I made my Y the last column.

The goal is so that if I would classify Vios it would return me Car or 0 in other words it can find its way to the first row.

Classification use case:

classify("poodle") #just pretend this is a working function

returns: Pets

How I did it in attempt to train my model:

from sklearn.feature_extraction.text import TfidfVectorizer
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 72)
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(X_train)
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
clf3 = RandomForestClassifier().fit(X_train_tfidf, y_train)

I'm using a guide from somewhere on the net that works a bit the same with it, but at the end im getting returned:

ValueError: Found input variables with inconsistent numbers of samples: [5, 4156]

I knew immediately i was doing it wrong. How do I train a model so that it achieves my goal? Any relevant guides or techniques I should be following intead of this? I dont even know the correct way to use vectors on this case.

Couple of things:
Cant replicate the problem exactly but if you follow these steps you are not exposing yourself

TfidfVectorizer is CountVectorizer + TfidfTransformer, you are exposing yourself to unnecessary complexity and potential errors

Use pipelines, cant stress this enough, its a compact way to pack all the sklearn transformers together AND THEN use fit, predict methods...

I would advise you to follow something like that, or find similiar problem here


Bug in sentiment analysis and classification for unlabeled text I'm working on the transcript of Trump and Biden's debate and want to analyze the sentences and classify negative, positive, or neutral comments, but I ran into one problem. I used both TextBlob and the transformers pipeline to analyze the sentiment but unfortunately in both ways, there are some very disastrous flaws!
For example,I found that TextBlob recognizes -0.70 polarity in "fewer people are dying every day" (negative comment)
or the transformers pipeline recognizes "The audience here in the hall has promised to remain silent." as a negative comment with 0.99 percent certainty!
Why do you think it's happening?
Is there any way we can prevent this?
Is there any way better than this for analyzing the sentiment of unlabeled text?
Also, I'm not comfortable with sentences like "Oh, Really?!" being classified as neutral. It's more of a sarcastic or negative comment I think.
Here's my colab notebook, I've added one "Problem" markdown where I've observed these examples.

First, congratulations for thinking to do a qualitative analysis of the results :) I know it should be obvious, but so many people just assume that the system works and don't bother checking their output.
Now, strictly speaking what you're seeing is not a bug. These are errors made by a statistical system. A statistical system is not meant to get everything right, it's only meant to label the input "to the best of its knowledge", and its knowledge is limited primarily by (1) the data it was trained with and (2) the assumptions made in the design of the model itself.
I don't know the exact characteristics of the systems that you used, but I can make an educated guess about the errors that you mention:

"fewer people are dying every day" is likely to be predicted as negative because it contains the word "dying". Probably there were no (or very few) examples in the training data which contain the word "die" and are labelled positive. As a consequence the system assumes that any sentence containing "die" is likely negative. One may notice that the positive semantics of "fewer people dying" is completely lost on the system, because it focuses on simple clues (individual words), it's not able to parse more complex phrases.
"The audience here in the hall has promised to remain silent." would be a similar case: the word "silent" or perhaps the two words "remain silent" likely were found only in negative examples during training, so the system just generalizes wrongly that a sentence containing these words is negative.
The sarcastic "Oh really?!" is an even more complex concept for the system to properly identify. The task of sarcasm detection is studied on its own because it's such a difficult task for a machine. I don't follow this field closely so I could be wrong, but I don't think the task has reached any satisfying level of maturity yet, let alone been integrated with standard sentiment analysis systems.

Nonetheless these errors don't mean that the results are useless. If you annotate manually a random sample and evaluate the performance of the system on this sample, hopefully you'll see that overall the system performs decently. That's what is expected of a statistical system: it's not reliable on an individual basis, but normally it's doing a good job in average.
More generally, all these errors show that the problem of Natural Language Understanding is far from being solved yet... and it might never be. The good news is that there's still a lot of interesting problems to solve for NLP scientists ;)


Multi-label text classification. 3-4 labels per text, 100 labels overall My first pet ML project, so please pardon if I phrase something incorrectly. Recently I had IMDB sentiment analysis binary classification practice on Tensorflow site. Now I am keen to do multiple label classification of the abstracts in the newspapers.
I have prepared a Pandas dataframe with 2 key columns [sectors_array] and [text]. It is 10 0000 rows. There are 100 sectors overall. [sectors_array] column has 1 to 4 sectors (Same article can fall into multiple categories). Text is a string up to 500 chars. Text contains a piece of article and sectors categorize the text, i.e whether it is food, sport, cinema, politics etc.
So far I cleansed text column to remove single characters, urls, punctuation and did removal of stop words and Lemmatisation for nltk tokenized text.
For multiple sectors I added a hundred of columns with 1/0 flags for each sector.
Now what could be my direction from there in order to classify the new text to existing sector(s)?
Would this big number of sectors present an issue for categorisation?
Which library should be best for the task? Tensorflow/Spacy?
Ideally I would want to present up to most probable 4 sectors for the piece of text.

I think it's always a good idea to start with a basic option. Once you have a decent basic model, you can try to improve performance with something more advanced and you would have a baseline so you know if the performance of the advanced model is really good or just ok.
In this case the basic option would be to train 100 binary classifiers, one for every label. The text could be represented as a TFIDF vector. I would suggest removing the rare words, it can significantly improve performance. Then a traditional model like Naive Bayes, Decision Trees or SVM can be used.
The number of labels by itself is not an issue, however their distribution matters, especially if some of them have very few instances (expect bad performance on these).


Dropping attributes leads to better classifier accuracy? (Titanic Set) I am currently tackling the Titanic Dataset on Kaggle. The goal is to find a classifier that can predict whether a passenger will survive or die.
The dataset has features that I believe are strongly linked to the likelihood of survival, i.e. the passenger class. Clearly, the survival probability is higher for passengers travelling in class 1 than in class 3. Additionally, there is a "Fare" attribute, which describes the exact price a passenger has paid.
I have tried different classifiers (e.g. Stochastic Gradient Descent and Random Forest) and both have a better accuracy when I drop the passenger class attribute.
Another thing I have noticed is, that if I condense the attributes "Siblings" and "Parents/Children" into a single attribute "TravelledAlone" (which can be 1 or 0), the classifier becomes more accurate.
In total these steps increase the accuracy from about 70% to almost 90%.
Can this be explained? After all, my results have become better by throwing away information...

I can't say for sure why this happens for this particular data set, but, presumably "Fare" and "Class" are highly correlated. The classifier may do better reasoning about one of them rather than splitting its reasoning across both, as they have mostly the same information.
It's also possible indeed that just knowing whether the person had any family members is most of the useful info, and trying to reason about how many is unhelpful as it is mostly noise that the classifier will overfit.
It's also possible that, given the tiny data set size, you can find lots of ideas that happen to fit the data better, but wouldn't necessarily generalize.