ML that learns to predict and play a simple wagering game I have a simple game I'm building for fun, just to see how well ML can work with simple data sets.

Basically it's just a game where it has turns that go like this:

Computer generates a random number $x$, and does not show the player.
Player wagers that they can guess a number lower than $x$. Call the wager amount $w$
Player tries to guess a low number $g$.
If $g \lt x$, then player gains $wg$ points.
If $g \ge x$, then player loses their wager, $w$ points.

If player has funds $f_t$ at start of turn $t$ then another way to put this is:

$$f_{t+1} = \begin{cases}
    f_t + wg,& \text{if } g\le x\\
    f_t - w,              & \text{otherwise}
\end{cases}$$

Here's an example of play:

Start with $f=50$
Turn 1, $x=11$, player wagers $w=8$ and guesses $g=9$. Player gains $+72$ points, so $f=122$ at the end of the turn.
Turn 2, $x=10.5, w=4, g=7$, points change by $+28$, $f=150$.
Turn 3, $x=20, w=1, g=6$, points change by $+6$, $f=156$.
Turn 4, $x=2, w=10, g=15$, points change by $-10$, $f=146$.

I wanted to use ML to try and predict this by feeding it pre-generated turns to see if it can find any patterns. There are only a few variables involved, so I figured it shouldn't be overly complicated. Ideally I would like for the ML to learn how to play the game.

I am wondering what type of ML is applicable to this kind of problem? It is not clear to me where to start, even though I have made the game simple. I have more complex games that I made in the past that I would like to try to apply ML to as well.

There are a few different data science and ML techniques you could throw at the numbers output by this game. You could try to analyse human players' style. You could generate a table of expected gains/losses by ignoring actual amount gained or lost and estimating the probability distribution of $x$ even though the player did not see it, just from know whether $x \gt w$ or $x \le w$ from enough examples.

If your end goal is to find out how a computer might play such a game, then one clear machine learning choice would be Reinforcement Learning (RL). It is not the only way for a computer to play a game - there are very many optimisation techniques for this. However, it is a machine learning approach - it learns from data observations - and in many variations it includes numerical analysis that you may find interesting, such as chance of winning from a certain position, or predicted future rewards.

RL is not a single algorithm, or even a single approach. Instead it is a way of framing a problem, and all ways of solving that kind of problem are considered to be RL. For RL to work, a problem needs to be framed as a Markov Decision Process (MDP). The good news is that your example game, and similar ones, are good examples of MDPs already.

In RL, the way that an agent plays a game is called its policy. The usual goals of RL include measuring the performance of a specific policy, or discovering the best policy.

So, where to start? First, if you want to try an RL process on your game you should probably do a few things:

Understand what in the game is the State, what are the Actions and what are the Rewards.

The state is anything that the agent knows about and can affect the outcome. Probably the only thing to worry about here is the current funds $f$, although this may not have a huge influence unless $w$ can always go up to $f$. Note each value is treated as a different state.
The action is the agent's choice of both $w$ and $g$ - note each specific combination is considered a different action. If there are a large number of actions, this can make learning harder, requiring more advanced RL algorithms, so I recommend initially you try variations of your game with limited number of choices.
The most obvious choice of reward is the change to $f$ at the end of each turn. It doesn't have to be though, it depends on what you consider a "winning condition" to be for the game. See below.

Simplify the game options a little. Set a small limit to possible ranges of $x, w, g$. This will make it easier to try things out initially.
Set a clear goal that you want to achieve, and make sure that the game structure works for that. You may need to adjust the game representation for even simple change such as "get the most reward in 10 turns" - because that might encourage large wagers at certain steps, and the agent will need to know how many turns it has left (so your state becomes combination of $(f, t)$. Another variation that may work is to set the goal of getting to a certain amount of funds, such as 1000. In which case, the reward would not be the increase in $f$, but actually be +1 for getting to that target, and 0 for any other result. That might radically change the behaviour of the agent, which could be interesting to experiment with.
Look up simple Reinforcement Learning algorithms. I suggest you start with tabular methods - perhaps Monte Carlo Control, or Q-Learning. They can typically be implemented in a few tens of lines in Python/Numpy.

A useful resource for RL is Sutton & Barto's Reinforcement Learning: An Introduction. The draft of the second version is free to download.


Redundancy - is it a big problem? I am trying to create a sentiment analysis program which will classify some of the tweets which i have collected under a hashtag. There are 7750 tweets in the dataset and I am labeling them into the two classes now. Then I will use a neural network to classify them into positive and negative classes.

The problem with the dataset is that it posses a lot of redundant data (Retweets basically). Manually deleting them is not possible and I have tried to find a programmable solution via the Tweepy API but couldn't find any.

So my question is do I need to get rid of these redundant records or should i leave them as is?

Assuming you want to learn sentiment this is a problem. What happens when you feed this to a Machine Learning algorithm is that it will give more weight to the tweets that are in there multiple times, while not learning more information. It's unlikely that a tweet that has been retweeted 10 times carries more significant information about sentiment than one that hasn't been retweeted. What you could do is add the number of times the tweet was in the set as a feature to your sentiment model, it's possible something can be learned from that fact (maybe positive tweets are retweeted more often), but you should keep it at 1 row for every distinct tweet.

Getting rid of these redundant records should not be difficult programmatically though. I don't know what language you are using but if you only consider the body of the tweet (the content) you could iterate over your tweets, keep a list of all the unique bodies, combined with other meta information (like user, labeled sentiment) and if the content of the next tweet is already in there, just do not add it. Look for 'distinct' functionality as opposed to redundant, enough information out there.


Force selecting samples in majority class with random forest Context: I have some data to fit a random forest classifier (binary output) with 1 being a very rare event. In particular, in my training set, there are only 614 1's out of 29400 points. I am using sklearn RandomForestClassifier.

I am setting class_weight = balanced to prevent the model from simply predicting 0 to every case. And it is working great!

However, there is also a small group within the 0 class (maybe only 20 - 30 cases) (Edit: actually about 300) that I want my model to capture. I believe because of the sampling natures when building trees, these class are not chosen very often. Is there a known way to solve this problem?

My thought:

Add an additional filter after the RF. Trouble is, it's hard to find some easy categorization methods for these 20 -30 negative cases.
Force the RF to include these samples when building the trees. Hence this post...

Thanks.

I would suggest that you drop those 30 events and all 1's from your dataset (i presume you know which 30 they are). Then randomly select 584 samples from the remaining dataset and the stick it back together. This would give you a full dataset equally weighted that contains all 30 for definite.

See the following pseudo code (it won't run because i don't know how your identifying these 30 cases so i'm generalising)

data = import()

ones = data.select('1')

interesting = data.select('something to select the 30 data points')

data.drop('1' and 'something to select the 30 data points')

randomList = randomListGen()

selected = data.select(randomList)

joined = join(selected, ones, interesting)

As i said before this won't run on anything, if you give the language you're working in and a minimum working example then i might be able to produce something more solid.


Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image I need to fine-tune a CNN to classify two classes: dogs and cats, for example. However, I want the CNN to be able to tell if there are no dogs nor cats in a given image. Hence, I'm thinking of using a third class called background.

The goal is to fine-tune the network with lots of images: images of cats go to the cat class, images of dogs go to the dog class, and every other image goes to the background class.

This way, the fine-tuned network would be able to classify a dog as a dog, a cat as a cat, and (hopefully) everything else as background.

Is this the right way to do it? Would it work? I can't seem to find reliable information about this online.

My problem is a little bit more complicated, but knowing this would be a very good start.

Actually you are in the right path but in the question you are wrong in the second paragraph.

What you should do is as follows:

change the output layer to 4 classes, dogs, cats, dogs and cats, none.
If you have so many data, don't freeze the convolutional layers and train them all but if not, freeze them and fine tune the last layers, maybe just the dense layers.


Grid Search and High Variance I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which parameter is the best. This appears to happen based on how the random number generator splits the data into train/test sets.

I have tried :

CV 5-fold
Split of 75%

Fixing the random seed does help (or using the same test set), but it concerns me that I get such variations based on what samples are in the test set. It seems alarming that the 'best parameter' is so dependent on a particular shuffle of the data and I worry how it translates to real world use.

What are people's approach to situations like this? I was thinking I could just repeat each test multiple times and take the average, but that has very large computation costs and seems very inefficient.

I'm afraid you have to repeat the K-fold CV a few times (with different seeds each time) and average the results. I'd guess that the high variance comes from the small size of the dataset.


Metrics show badly performing model for multiclass So I have a model that I am training on a multiclass (30-40 classes) imbalanced data set (smallest class 4000 samples, largest 14 million). The data consists of strings and I extract unigram and bigram counts as features, plus string length and entropy.

I first remove any 0 variance features (~1M -> 2k feats.), then using sklearn.decomposition.TruncatedSVD I reduce it further to 500 feats. The data is then split to 10k samples batches (min 500 samples/class), following the distribution of each class in the raw data. Splits continue this way until all samples from the largest class has been added to a split.

Then I run imblearn.under_sampling.RandomUnderSampler (100k samples for majority classes) and imblearn.over_sampling.SMOTE (50k) for minority classes) for under- and oversampling on each of the files to fix the imbalance.

After this I train several models and then compare different metrics to get a better idea of which is the best choice:

sklearn.linear_model.SGDClassifier(loss='log', alpha=1e-4,tol=1e-4,penalty='elasticnet',max_iter=1000,shuffle=True)
sklearn.linear_model.Perceptron(alpha=1e-3, penalty=None,tol=1e-4,max_iter=1000,shuffle=True)
sklearn.linear_model.PassiveAggressiveClassifier(C=1.0,tol=1e-4,max_iter=1000,shuffle=True)
sklearn.neural_network.MLPClassifier(solver='adam',learning_rate='adaptive',activation='tanh',validation_fraction=0.05,alpha=1e-5,learning_rate_init=1e-4,max_iter=20,hidden_layer_sizes=(100,))
sklearn.naive_bayes.BernoulliNB(alpha=1.0e-5, fit_prior=True)

As you can see in the picture below the metrics show that the classifier is not working optimally. As far as I have understood from searching around, AUC and cohen_kappa are the more realiable metrics for multiclass and imbalanced data sets.

I am concerned that the AUC metric show that all models are doing well, but the others, in particular cohen_kappa which is significantly lower (I know though that it is hard to directly compare the metrics).

Am I doing any obvious mistakes in this, or any obvious ways I can improve performance/metric scores?

I think you need to consider these metrics also i.e., Precision and Recall when you decide on some model.

As you can in the above outcome that there are Precision and Recall values, you din't explain anything about the problem's business perspective.

For example: Consider a data set with 100,000 observations. A data set consist of candidates who applied for Internship in Harvard. Apparently, harvard is well-known for its extremely low acceptance rate. The dependent variable represents if a candidate has been shortlisted (1) or not shortlisted (0). It was found ~ 98% did not get shortlisted and only ~ 2% got lucky.
In this scenario you need to understand that precision and recall plays a vital role at the time of selecting model.

In the same way if your business problem is inline with the above scenario then you need to select a model with high Precision and Recall.

In the above screenshot which you have shared, it think you have good performance scores too(Assuming that your model is not over fitting). So, you can select which ever model suits your business problem.

Hope my answer is helpful!


Math Behind GOSS (Gradient-Based One Side Sampling)? As per my understanding through books & Google Search,

GOSS (Gradient-Based One Side Sampling) is a novel sampling method that downsamples the instances on the basis of gradients. As we
know instances with small gradients are well trained (small training
error) and those with large gradients are undertrained. A naive
approach to downsample is to discard instances with small gradients by
solely focussing on instances with large gradients but this would
alter the data distribution. In a nutshell, GOSS retains instances
with large gradients while performing random sampling on instances
with small gradients. Source

LightGBM uses a novel technique of Gradient-based One-Side Sampling (GOSS) to filter out the data instances for finding a split value while XGBoost uses pre-sorted algorithm & Histogram-based algorithm for computing the best split.

Can someone please explain the math behind GOSS?

Wang et al., (2019) have provided a nice and clear explanation. Please, check out their paper to find the answer you are looking for:
Part II. BAYESIAN OPTIMIZED LIGHTGBM
Section: A. The Principle of the LightGBM
Wang, R., Liu, Y., Ye, X., Tang, Q., Gou, J., Huang, M., & Wen, Y. (2019). Power System Transient Stability Assessment Based on Bayesian Optimized LightGBM. 2019 IEEE 3Rd Conference On Energy Internet And Energy System Integration (EI2). doi: 10.1109/ei247390.2019.9062027


Use all available data to build Logistic Regression model Using K-Fold, I chose to use Logistic Regression for a project of mine.

I made it learn on my X_train (80% of data), and tested it on my X_test, with good results.

My question is : now that I need the model to be used, can I rebuilt the model with the same parameters but on all my data ? Will there be overfitting, or maybe as I saw that those parameters doesn't, will it be safe ?

Yes, you can(Atleast in most cases). Infact before going to production, you must do this, it will give you better accuracy provided you keep the cross validated parameters same.

When you participate in competitions this is a well followed strategy to increase the score.

However, sometime domain constraints prevent you to use the test/validation data.

Eg: When you have split your data based on time periods, you might not want to include the latest data just because those observations in real time might not have matured enough to be considered into a model.

A good use case is fraud identification model in Insurance. It takes time for a claim to be identified as fraud. Hence without accounting for this, if you put all your data into model, the most recent claims will have been marked as no fraud, but in near future they might actually turn out to be identified as fraud. Your model will have learnt from wrong samples.


Can continuous variables decrease classification model accuracy? I have been playing around with titanic dataset. Here Fare column is a continous variable. I've read people stating that in a classification model it's best to have categorical variables than continous features. So I was wondering if I convert the age into categorical bins will it improve my model accuracy?

There's no concrete property of classifiers in general that lend themselves to using categorical variables rather than continuous variables.

You can definitely try binning the variables, but many classifiers (particularly, tree-based classifiers) will implicitly bin the variables optimally within the algorithm itself.

If you add the continuous variable, and the performance of your classifier doesn't improve, it's very possible that the variable is not predictive of the target or that all of the information (in the information-theory sense) that that variable carries is already expressed by other variables already in the model.


onehotencoder random forest In a Random Forest context, do I need to setup dummies/OnehotEncoder in a dataset where features/varibles are numerical but refer to some kind of category?
Let's say I have the following variables:

Where Y is the variable I want to predict. X's are features.
I will focus on X1. Its numerical but refers to a specific category (i.e. 1 refers to math, 2 refers to literature and 3 for history).
Do I need to apply OnehotEncoder (or dummy approach) for a Random Forest algoritm?
I guess I don't need to do it, but I 'm not sure.

In theory, categories themselves can be handled in decision trees. However, most python implementations will only work with numbers, so you need to convert them. It looks like you already have. OneHot Encoding is one way to do that, but now that you've done that, it isn't necessary. You may try other schemes to turn them into 1 2 3, if there isn't a logical reason to how you have them this way. This is called Categorical Encoding. One popular way is to order them alphabetically and number them. Another way, in order of their frequency. At the end of the day, it won't matter a whole lot because the random forest will bin the number according to its algorithm. It might make minor differences though, so maybe try different ways. But no, you won't need to OneHot encode your categories because they are already numbers.


How to measure model success in production I have a model running on a productive system. The model predicts if some lead will become a sale. How would you develop a check, which checks the success and the accuracy of the model? There is a time component because it took months where the lead could become a sale.

Short-term targets and more out-of-time samples.

Work with the business and your data to develop relevant short-term targets. For example, if a sale takes 6 months after first-contact and the sale is the modeled target, set first-contact as short-term target for measurement. Set other steps in the funnel as a short-term target. Then run your analysis with these. Similar to predicting a charge-off of a loan. If a model is used to give a loan and the loan is 2 years, we do not wait 2 years from model deployment to get the first day of measurements. That could be a lot of bad loans. Perhaps half of the people who typically charge off do so within 12 months. And then 75% charge off within 20 months. You need to examine what is relevant and how predictive those short-term targets are of the modeled target. Use your training data and SMEs.

Also, if your process takes time, then you did not use the most up-to-date data in modeling. For example, if a loan has a 24 month modeled target, the model could not have used the latest 24 months, since that does not have a mature target (assuming this is not a survivor model - then this paragraph does not hold). But now the model can be evaluated on that data as each month progresses. The model did not score that data for action, but the model can be evaluated. Perhaps some loans would not have been given with the new model, but this is better than waiting 2 years to determine that the model is bad.


Neural network for linear regression? In this example on tensorflow website, I was surprised to see a neural network's predicted values graph as a straight line. Isn't the purpose of neural networks to approximate non-linearities of data into it?
Please let me know what I am missing here.

Neural networks are very good function approximators. Hence, they can approximate a wide range of nonlinear functions. Remember that linear functions are easier to represent than nonlinear functions. Hence, the neural network will clearly be able to approximate a linear function. This can be easiest seen if we only use linear activation functions. But we can also use a nonlinear ReLU and still be able to arbitrarily approximate the function on a compact set.

The question that we should have is rather: Is it overkill to approximate a linear function with a neural network when a linear regression would do the job? The answer should be clear that you should rather use a linear regression instead of a neural network.

The given example just wants to demonstrate that even without knowing the relationship between our predictors and criterion (this is sometimes called domain knowledge) the neural network will still be able to approximate the function without the need for additional domain knowledge.


For which real world data sets does DBSCAN surpass K-means.? For clustering, DBSCAN surpass k-means in terms of handling arbitrary  shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data sets that have special features (for example, the moon-shaped data sets). Can someone point me to some real world datasets where DBSCAN outperforms K-means.

I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit.

If you've ever gone hiking or mountain climbing on a day with fog or a low cloud cover, there are times where you get to the top of the peak and can only see the surrounding peaks poking up through the clouds.  I like to use this analogy when I think of DBSCAN.  The density filtering allows one to select a threshold where data is kept, and all of the remaining data is filtered out.

Take a look at this Seattle crime incident data.  Suppose I want to cluster the data by location to form pseudo neighborhoods i.e. neighborhoods that are roughly defined the the geography of where criminal events tend to occur. This is an example of k-means working just fine in location analytics:

Seattle crime data superimposing a primitive map with k-means clustering

If you know Seattle at all, you can see that the clusters tend to pick out neighborhoods and this works quite well in breaking up neighborhoods.  Now suppose I want to pick out the high crime areas of Seattle in order to identify the hot spots. No matter how I adjust k, the k-means clustering doesn't really provide any additional insight.  But the density filtering in DBSCAN does a wonderful job of identifying high crime areas:

Seattle crime data superimposing a primitive map with DBSCAN clustering

I think this gives the jest of some strengths that can be exploited using the two algorithms.  There's nothing special about the crime data.  For what its worth, I have a tutorial that goes through the analytics of identifying a user's home and work locations from their cell phone GPS pings. This is another case where DBSCAN was very useful, but the DBSCAN specific parts a buried a ways down in the tutorial.

Hope this helps!


Bigdata cluster compatible distributed predictive model I might be asking a dumb question but my question is can I write a python program (lets say a classifier) using some library that scales in hadoop (not only using a simple parallel processing).The reason I am confused is
1)The scikit learn python codes don't scale in big data.
2)The spark mlib is written in scala and not in python.
3)Although h2o is open sourced I don't think it is written in python.

Spark does have a pretty good Python API, check out this tutorial.

For traditional Hadoop stack, take a look at mrjob, it lets you write MapReduce jobs in Python and run them on several platforms.


I am doing classification using SVM for two classes , is this result good and what does these values refer to? precision    recall  f1-score   support

           0       0.92      1.00      0.96        12
           1       1.00      0.98      0.99        52

    accuracy                           0.98        64
   macro avg       0.96      0.99      0.98        64
weighted avg       0.99      0.98      0.98        64

what does these values refer to?

Well, if it is a good results or bad results is decided by the task you are performing. But so far, it looks like good results. Coming to the metrics, precision, recall and F1 can be learnt from wikipedia here.
However, the interpretation of your results goes like this:
The left most extreme 0 or 1 indicates the class (in your binary classification task).  And the values 0.92, 1.00, 0.96 corresponds to precision, recall, and F1 of the class 0 respectively.
Likewise, the next line indicates precision, recall, and F1  for class 1.
support indicates support count, which in simple terms is the number of samples for a given class in the test set.
Macro average precision/recall/f1-score is the average precision/recall/f1-score for all classes.
Weighted average precision/recall/f1-score is the weighted average precision/recall/f1-score for all classes, here the support count is used as weights.


Map similar like clusters in two different cluster sets I have two sets of clusters (Cluster_set_1 and Cluster_set_2) obtained by analysing two different sets of news paper articles.

A cluster includes word/word phrases as given in the examples below.

C1 in Cluster_set_1: Energy, Fuel, Oil, Mining
C2 in Cluster_set_1: school, education, students, schools, million, read
...
...
etc.

C1 in Cluster_set_2: Gas, oil, pipeline
C2 in Cluster_set_2: program, business, management,information, reports
...
...
etc.

Now I want to find clusters in the two cluster sets that are similar/related by considering the word/word phrases in the cluster as given in the example below.

Example:
Cluster of 'Energy, Fuel, Oil, Mining' in Cluster_set_1 is mostly similar/related to
Cluster of 'Gas, oil, pipeline' in Cluster_set_2

Reason: Because they both includes word/word phrases related to Energy

Since, I am dealing with two separate sets of clusters, what would be a suitable approach that I can use to connect the clusters in the two different sets?

I would investigate a soft clustering approach using HDBSCAN clustering.
Some reading on text clustering here, which includes discussion on clustering with DBSCAN, HDBSCAN's non-hierarchical precursor


Syntax error but nothing appears to be wrong? I'm trying to create a decision tree, and everything appeared to be going smoothly until I encountered this syntax error. I don't see what the issue is?

print ("start fitting the data")
import warnings
warnings.filterwarnings("ignore")

for cv in tqdm(range(3, 6)):
    create_grid = GridSearchCV(pipeline, param_grid=check_params, cv=cv)
    create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean()))
    print("score for %d fold CV := %3.2f" %(cv, create_grid.score(X_test, y_test)))
    print ("!!!! best fit parameters from GridSearchCV !!!!")
    print (create_grid.best_params_)

print ("loop has exited")

jupyter throws the error:

  File "<ipython-input-38-6b9dca710e77>", line 8
    print("score for %d fold CV := %3.2f" %(cv, create_grid.score(X_test, y_test)))
        ^
SyntaxError: invalid syntax

In the second line of the for loop

create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean()))

You need an extra closing parenthesis like so

create_grid.fit(X_train.fillna(X_train.mean(), y_train.fillna(y_train.mean())))


GridSearch returns worse results than default configuration I'm using the following code for accuracy score calculation. Why is it so that the default configuration gives better result than GridSearch?

Default configuration

clf = svm.SVC(kernel='rbf', gamma='auto')
clf.fit(x_train, y_train.values.ravel())

y_train_pred = clf.predict(x_train)
y_test_pred = clf.predict(x_test)

print('Train set accuracy: '+'{}'.format(metrics.accuracy_score(y_train, y_train_pred)))
print('Test set accuracy: '+'{}'.format(metrics.accuracy_score(y_test, y_test_pred)))

Train set accuracy: 0.861101243339254
Test set accuracy: 0.8480113636363636

GridSearch configuration

param_grid = {'C': (0.001, 0.01, 0.1, 1, 10),
              'kernel': ('linear', 'poly', 'rbf', 'sigmoid'),
              'class_weight': ('balanced', None),
              'gamma' : ('scale', 'auto'),
              'shrinking': (True, False)}

grid_search = GridSearchCV(svm.SVC(gamma='scale'), param_grid, cv=5)
grid_results = grid_search.fit(x_train, y_train.values.ravel())

print(grid_results.best_score_)
print(grid_results.best_estimator_)
print(grid_results.best_params_)

0.8373001776198934
SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False)
{'C': 1, 'class_weight': None, 'gamma': 'auto', 'kernel': 'rbf', 'shrinking': True}

Technically: Because grid search creates subsamples of the data repeatedly. That means the SVC is trained on 80% of x_train in each iteration and the results are the mean of predictions on the other 20%.

Theoretically: Because you conflate the questions of hyperparameter tuning (selection) and model performance estimation. GridSearch is used for selecting a combination of hyperparameters, performance estimation has not yet happened. The only comparison you should be making is between the parameter combinations within the CV itself (grid_results.cv_results). In my opinion, the reported CV train accuracy is within acceptable boundaries from non-CV training (meaning your SVC is able to extract a lot of generalization from subsamples). See e.g Cawley 2010

It would be interesting to see the reported performance on x_test on the retrained estimator grid_results.best_estimator_.predict(x_test), if it was kept separate from x_train, and how it is different from the first results.


Measuring the success of text summarization I am trying to make a text summarization program that will take a text article and reduce it to a para or 2.

Since I am a newbie with no idea of NLP, it is hard to approach and break down the problem. So I was wondering if there was a measure that is used to check for effectiveness and correctness of text summarization. I tried googling this, but nothing that suits my purpose.

Does something like this even exist? or am I going in the wrong direction?

You will first need a human written summary as your correct output. You can then compare the original and generated ones using Rouge scores. They compare the similarity between two giving paras by comparing different combinations of sub-phrases. Search for Rouge and you'll get different types of rouge scores, you'll probably need to use more than one. Also, there are libraries available that calculate rouge scores so you don't have to implement it yourself!


pandas series match multiple keywords Is there a direct python pandas method to match values of series and update different series with some string ? I couldn’t find any direct method of doing it.
Here the match is to find a value in a series that is made up of given set of keywords and one mandatory keyword.
Example :
Here’s the keywords table :
KW:
case,kws,mand_kw,Friendly name
1,full|name|nm|txt|dsc[0-9],full,Full Name
2,first|name|nm|txt|dsc|[0-9],first,First Name
3,last|name|nm|txt|dsc|[0-9],last,Last Name
4,legal|name|nm|txt|dsc|[0-9],legal,Legal Name
5,nick|name|nm|txt|dsc|[0-9],nick,Nick Name

DF
_id,field_name,field_friendly_name,purpose_of_use,is_included_in_report
1,FullName,,,
2,xyz,,,
3,custname,,,
4,FullNm,,,
5,FirstName,,,
6,abc,,,
7,FirstNm,,,
8,LastName,,,
9,LastNm,,,
10,LegalName,,,
11,LegalNm,,,
12,NickName,,,
13,pqr,,,
14,NickNm,,,

For each of kws from KW find match in field_name in DF that it is made up of only keywords specified in kws and not any other words and a mandatorily should have a mand_kw keyword.
For example :
Case1:
Find all field_name that is only made up of these set of keywords “Full|name|nm|txt|dsc|[0-9]” and mandatory contains keyword "full" if so, assign field_friendly_name of KW - “Full Name” to field_friendly_name in DF and update is_included_in_report as true.
If the value is not made of these keywords, then skip.
_id,field_name,field_friendly_name,purpose_of_use,is_included_in_report
1,fullname,Full Name,,TRUE
4,fullnm,Full Name,,TRUE
8,fullname74,Full Name,,TRUE
9,fulltxt,Full Name,,TRUE

here's what I've tried:
import pandas
import re

KW = pd.read_csv("regex.csv")
DF = pd.read_csv("Default-Profile.csv")

for i in range (0,KW.kws.size):
    for j in range (0,DF.field_name.size):
            if (DF.field_name.match(r"KW.kws[i]") == True && (DF.field_name.match(r"mand_kw[i]" == True)):
                    DF.field_friendly_name[j] = KW.field_friendly_name[i]
                    DF.is_included_in_report[j] = "True"


Cannot clone object '' My model-

#define model
model = Sequential()
model.add(Dense(128, activation='relu', input_dim=n_input_1))
model.add(Dense(64, activation='relu'))
#model.add(Dense(32, activation='relu'))
#model.add(Dense(16, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse',metrics=['mse'])

Now I am trying to tune the hyper parameters by this code-

from sklearn.model_selection import GridSearchCV
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# define the grid search parameters
batch_size = [10, 20, 40, 60, 80, 100]
epochs = [10, 50, 100]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1,
cv=3,scoring='neg_mean_absolute_error')
grid_result = grid.fit(scaled_train,y_train_c)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

I am getting this error-

TypeError                                 Traceback (most recent call last)
<ipython-input-10-c6e1e39d878e> in <module>
      9 param_grid = dict(batch_size=batch_size, epochs=epochs)
     10 grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3,scoring='neg_mean_absolute_error')
---> 11 grid_result = grid.fit(scaled_train,y_train_c)
     12 # summarize results
     13 print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\model_selection\_search.py in fit(self, X, y, groups, **fit_params)
    631         n_splits = cv.get_n_splits(X, y, groups)
    632
--> 633         base_estimator = clone(self.estimator)
    634
    635         parallel = Parallel(n_jobs=self.n_jobs, verbose=self.verbose,

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\base.py in clone(estimator, safe)
     58                             "it does not seem to be a scikit-learn estimator "
     59                             "as it does not implement a 'get_params' methods."
---> 60                             % (repr(estimator), type(estimator)))
     61     klass = estimator.__class__
     62     new_object_params = estimator.get_params(deep=False)

TypeError: Cannot clone object '<keras.engine.sequential.Sequential object at 0x0000023DD4D5F488>' (type <class 'keras.engine.sequential.Sequential'>): it does not seem to be a scikit-learn estimator as it does not implement a 'get_params' methods.

I have no idea whats causing this issue. I am implementing a MLP for time series forecasting.

The xyzSearchCV classes in sklearn require the estimator to be compatible with the sklearn API; in particular, the estimator('s clones)'s hyperparameters get set using set_params, which needs to be defined as a class method for the estimator.  So a Keras Sequential model will not work.  Keras does provide an sklearn wrapper: https://keras.io/scikit-learn-api/


What Clustering Method Should I Use? My data is a group of 10 thousand points (each having an node location (x,y)) that are spread across a plane. They are also chromatically-colored based on their weight.
I need to finalize a bayesian nonparametric clustering method that groups points on mainly weight, but also on distance: that is, clusters are by defintion have some relevance to distance, but there are clear topological distinguishing factors between the first quarter and the last quarter of data (I say quarter as an arbitrary amount; in reality, the exact number and topology of the cluster changes through iterations).

As you can see in this picture above, I’ve tried to use notability to create a crude chromatically-colored image of data with varying cluster topology types; over each iteration of my algorithm, the clusters, as mentioned, change location (based on their weight) and their shape, and some overlap (and the possiblility of new clusters (or a possible decrease in the total number) is very high per iteration, where this image represents one iteration of x points)
Additionally, since I am doing this analysis with data via python, i was thinking of using the T-SNE machine learning package as a substitute for a generic clustering method, but I only have a limited knowledge on its functionality. Also, since my data is based on the same weighted scale, it may be overkill.
EDIT: I changed the picture to show overlapping clusters, so it is clearer what I mean. However, keep in mind even these visible clusters are not homogenous in weight (they still vary but in a small threshold). Sure there is noise, but I really want to treat each cluster independently to see each cluster’s behavior over time (as well as clusters that are newly formed, hence the nonparametric method)

One option is spectral clustering. Spectral clustering can find the "connectedness" in data.


Which decision tree model is used in "standard" random forest? Is that CART? Why not using C5.0 tree?

A perhaps more general question:

Since C5.0 tree frequently have better performance than CART, why people still use CART to build random forest (or people already are doing this)?

My experience is with sklearn and python.

A Random Forest is a bagging of decision trees, the sklearn package uses a Decision Tree Classifier which is a CART.

You can see in this post that I made a bit ago of how to build a Random Forest tree the exact same as a decision tree.


Watch list of Tweets with unknown model I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True).
It has 46 000 words in 25 dimension vector space. I don't know anything about the model used to obtain such vectors.
I also have a 75-word watch list that all but one belong to model 's vocabulary.
I want to monitor tweets from ABC and XYZ for "matches" associated with my watch list.
How can I monitor scraped tweets for terms in my word list ? I do not know what to make of their vector representation in model, nor what to do with my word that's not in the vocabulary

There are several steps:

Setup a system that collects tweets from certain accounts.
Preprocess the tweets to create tokens that similar to the tokens in the pretrained model (e.g., lowercase and/or stem).
Create two sets of watch list tokens and tweet tokens. Take the intersection of the two sets.

There are two options for tokens that in tweets but not in pretrained model:

Ignore them.
Explicitly model them as "UNK" / unknown


How to use regularizer in AllenNLP? Apology if this sounds a bit lame.

I am trying to use Allennlp for my NLP tasks and would like to use regularization to reduce overfitting. However from all the online tutorials, all the regularizers are set as None, and I still couldn't find out how to use the regularizer after many many attempts.

If I use the example in the official tutorial (https://github.com/titipata/allennlp-tutorial) , what if I want to add in regularizer for LSTM and feedforward layer?

class AcademicPaperClassifier(Model):
    """
    Model to classify venue based on input title and abstract
    """
    def __init__(self,
                 vocab: Vocabulary,
                 text_field_embedder: TextFieldEmbedder,
                 title_encoder: Seq2VecEncoder,
                 abstract_encoder: Seq2VecEncoder,
                 classifier_feedforward: FeedForward,
                 initializer: InitializerApplicator = InitializerApplicator(),
                 regularizer: Optional[RegularizerApplicator] = None) -> None:
        super(AcademicPaperClassifier, self).__init__(vocab, regularizer)
        self.text_field_embedder = text_field_embedder
        self.num_classes = self.vocab.get_vocab_size("labels")
        self.title_encoder = title_encoder
        self.abstract_encoder = abstract_encoder
        self.classifier_feedforward = classifier_feedforward
        self.metrics = {
                "accuracy": CategoricalAccuracy(),
                "accuracy3": CategoricalAccuracy(top_k=3)
        }
        self.loss = torch.nn.CrossEntropyLoss()
        initializer(self)

    def forward(self,
                title: Dict[str, torch.LongTensor],
                abstract: Dict[str, torch.LongTensor],
                label: torch.LongTensor = None) -> Dict[str, torch.Tensor]:

        embedded_title = self.text_field_embedder(title)
        title_mask = get_text_field_mask(title)
        encoded_title = self.title_encoder(embedded_title, title_mask)

        embedded_abstract = self.text_field_embedder(abstract)
        abstract_mask = get_text_field_mask(abstract)
        encoded_abstract = self.abstract_encoder(embedded_abstract, abstract_mask)

        logits = self.classifier_feedforward(torch.cat([encoded_title, encoded_abstract], dim=-1))
        class_probabilities = F.softmax(logits, dim=-1)
        argmax_indices = np.argmax(class_probabilities.cpu().data.numpy(), axis=-1)
        labels = [self.vocab.get_token_from_index(x, namespace="labels") for x in argmax_indices]
        output_dict = {
            'logits': logits,
            'class_probabilities': class_probabilities,
            'predicted_label': labels
        }
        if label is not None:
            loss = self.loss(logits, label)
            for metric in self.metrics.values():
                metric(logits, label)
            output_dict["loss"] = loss

        return output_dict

The API is a bit confusing. You pass a RegularizerApplicator instance to the model, which takes a list of tuples of the form (regex, regularizer). The regex matches against your model's parameter's name. For example, if you had layer called linear_relu_stack.0.bias, linear_relu_stack.0.weight, you could apply a single regularizer to both with the regex "^linear_relu_stack.0.(bias|weight)$".
The regularizer itself is an just instance of L1Regularizer or L2Regularizer, where you can specify the alpha.
In all, your model would look something like this after adding regularization:
from allennlp.nn.regularizers.regularizer_applicator import RegularizerApplicator
from allennlp.nn.regularizers.regularizers import L2Regularizer

AcademicPaperClassifier(*model_args, regularizer=RegularizerApplicator([
    (".*LSTM.*", L2Regularizer(alpha=0.01)),
    (".*FFN.*",  L2Regularizer(alpha=0.01)),
]))

The API documentation is the best resource for this: http://docs.allennlp.org/v0.9.0/api/allennlp.nn.regularizers.html#allennlp.nn.regularizers.regularizer_applicator.RegularizerApplicator.from_params


Information Gain in R I found packages being used to calculating "Information Gain" for selecting main attributes in C4.5 Decision Tree and I tried using them to calculating "Information Gain".

But the results of calculation of each packages are different like the code below.

> IG.CORElearn <- attrEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1, estimator = "InfGain")
> IG.RWeka     <- InfoGainAttributeEval(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi, dataUSE1)
> IG.FSelector <- information.gain(In_Occu ~ In_Temp+In_Humi+In_CO2+In_Illu+In_LP+Out_Temp+Out_Humi,dataUSE1)

> IG.CORElearn
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi
0.04472928 0.02705100 0.09305418 0.35064927 0.44299167 0.01832216 0.05551973
> IG.RWeka
   In_Temp    In_Humi     In_CO2    In_Illu      In_LP   Out_Temp   Out_Humi
0.11964771 0.04340197 0.12266724 0.38963327 0.44299167 0.03831816 0.07705798
> IG.FSelector
         attr_importance
In_Temp       0.08293347
In_Humi       0.02919697
In_CO2        0.08411316
In_Illu       0.27007321
In_LP         0.30705843
Out_Temp      0.02656012
Out_Humi      0.05341252

Why do the results of calculation of each packages be different?
And Which one is right?

This is not a complete answer to your question, but I can explain at least part of the problem. Since you do not provide your data, I cannot reproduce your results. However, it is easy to demonstrate the same problem with other data.  I will use the well-known iris data set that is provided with R and Weka and is easy to access.

The same problem is apparent with the iris data.

library(CORElearn)
library(RWeka)
library(FSelector)
IG.CORElearn <- attrEval(Species ~ ., data=iris,  estimator = "InfGain")
IG.RWeka     <- InfoGainAttributeEval(Species ~ ., data=iris,)
IG.FSelector <- information.gain(Species ~ ., data=iris,)

IG.CORElearn
Sepal.Length  Sepal.Width Petal.Length  Petal.Width
   0.5572327    0.2831260    0.9182958    0.9182958

IG.RWeka
Sepal.Length  Sepal.Width Petal.Length  Petal.Width
   0.6982615    0.3855963    1.4180030    1.3784027

IG.FSelector
             attr_importance
Sepal.Length       0.4521286
Sepal.Width        0.2672750
Petal.Length       0.9402853
Petal.Width        0.9554360

As with your example,  all three packages give completely different results.

One Issue: Units (Base of Logarithm)

If you look at the documentation for information.gain in FSelector,  you will see this parameter description:

  unit
  Unit for computing entropy (passed to entropy). Default is "log".

Following that trail,  we look at the description of the entropy function and see:

  unit
  the unit in which entropy is measured. The default is "nats"
  (natural units). For computing entropy in "bits" set unit="log2".

If we override the default and calculate IG using unit="log2" we get

IG.FSelector2 <- information.gain(Species ~ ., data=iris, unit="log2")
IG.FSelector2
             attr_importance
Sepal.Length       0.6522837
Sepal.Width        0.3855963
Petal.Length       1.3565450
Petal.Width        1.3784027

Notice that now the values for Information Gain agree with RWeka for Sepal.Width and Petal.Width. Part of the difference was simply using a different base for the logarithm. RWeka uses base 2 (entropy measured in bits). By default, FSelector uses base e,  but allows you to change the base and get some of the same results. It appears from the documentation that neither RWeka nor CORElearn let you select the base for the logarithm.

But it is almost stranger that once we get RWeka and FSelector in the same units, they agree on two variables, but not the other two. There is still something else going on.


Performing a linear regression with Perceptron I was wondering about the link between the linear regression and the perceptron!
Perceptrons were used as binary classifiers i.e to classify binary labels ( 0 or 1 ).
My question is How can you perform a linear regression with perceptron and what is the link?
I think the point is hidden in the activation function which we are applying to the perceptron!
I would be appreciated if you can help me with the beginner's perspective on Neural Networks.

If you use a linear activation function on your perceptron you are essentially creating a linear regression where the weights connecting to your perceptron are analogous to the the coefficients on your linear regression model. The only difference in this case is you would normally fit your perceptron using backpropagation, but for the linear regression model you would likely use OLS. In either case - you will end up with a series of coefficients multiplying on your input variables in a linear fashion with some bias term.
This of course is the most basic use of a perceptron. The advantage of the perceptron is you can use non-linear activation functions to fit non-linearities in your data, and your ability to stack them together to form a larger neural network.


Why are NLP and Machine Learning communities interested in deep learning? I hope you can help me, as I have some questions on this topic. I'm new in the field of deep learning, and while I did some tutorials, I can't relate or distinguish concepts from one another.

Why to use deep networks?

Let's first try to solve very simple classification task. Say, you moderate a web forum which is sometimes flooded with spam messages. These messages are easily identifiable - most often they contain specific words like "buy", "porn", etc. and a URL to outer resources. You want to create filter that will alert you about such suspecious messages. It turns to be pretty easy - you get list of features (e.g. list of suspicious words and presence of a URL) and train simple logistic regression (a.k.a. perceptron), i.e. model like:

g(w0 + w1*x1 + w2*x2 + ... + wnxn)

where x1..xn are your features (either presence of specific word or a URL), w0..wn - learned coefficients and g() is a logistic function to make result be between 0 and 1. It's very simple classifier, but for this simple task it may give very good results, creating linear decision boundary. Assuming you used only 2 features, this boundary may look something like this:

Here 2 axes represent features (e.g. number of occurrences of specific word in a message, normalized around zero), red points stay for spam and blue points - for normal messages, while black line shows separation line.

But soon you notice that some good messages contain a lot of occurrences of word "buy", but no URLs, or extended discussion of porn detection, not actually refferring to porn movies. Linear decision boundary simply cannot handle such situations. Instead you need something like this:

This new non-linear decision boundary is much more flexible, i.e. it can fit the data much closer. There are many ways to achieve this non-linearity - you can use polynomial features (e.g. x1^2) or their combination (e.g. x1*x2) or project them out to a higher dimension like in kernel methods. But in neural networks it's common to solve it by combining perceptrons or, in other words, by building multilayer perceptron. Non-linearity here comes from logistic function between layers. The more layers, the more sophisticated patterns may be covered by MLP. Single layer (perceptron) can handle simple spam detection, network with 2-3 layers can catch tricky combinations of features, and networks of 5-9 layers, used by large research labs and companies like Google, may model the whole language or detect cats on images.

This is essential reason to have deep architectures - they can model more sophisticated patterns.

Why deep networks are hard to train?

With only one feature and linear decision boundary it's in fact enough to have only 2 training examples - one positive and one negative. With several features and/or non-linear decision boundary you need several orders more examples to cover all possible cases (e.g. you need not only find examples with word1, word2 and word3, but also with all possible their combinations). And in real life you need to deal with hundreds and thousands of features (e.g. words in a language or pixels in an image) and at least several layers to have enough non-linearity. Size of a data set, needed to fully train such networks, easily exceeds 10^30 examples, making it totally impossible to get enough data. In other words, with many features and many layers our decision function becomes too flexible to be able to learn it precisely.

There are, however, ways to learn it approximately. For example, if we were working in probabilistic settings, then instead of learning frequencies of all combinations of all features we could assume that they are independent and learn only individual frequencies, reducing full and unconstrained Bayes classifier to a Naive Bayes and thus requiring much, much less data to learn.

In neural networks there were several attempts to (meaningfully) reduce complexity (flexibility) of decision function. For example, convolutional networks, extensively used in image classification, assume only local connections between nearby pixels and thus try only learn combinations of pixels inside small "windows" (say, 16x16 pixels = 256 input neurons) as opposed to full images (say, 100x100 pixels = 10000 input neurons). Other approaches include feature engineering, i.e. searching for specific, human-discovered descriptors of input data.

Manually discovered features are very promising actually. In natural language processing, for example, it's sometimes helpful to use special dictionaries (like those containing spam-specific words) or catch negation (e.g. "not good"). And in computer vision things like SURF descriptors or Haar-like features are almost irreplaceable.

But the problem with manual feature engineering is that it takes literally years to come up with good descriptors. Moreover, these features are often specific

Unsupervised pretraining

But it turns out that we can obtain good features automatically right from the data using such algorithms as autoencoders and restricted Boltzmann machines. I described them in detail in my other answer, but in short they allow to find repeated patterns in the input data and transform it into higher-level features. For example, given only row pixel values as an input, these algorithms may identify and pass higher whole edges, then from these edges construct figures and so on, until you get really high-level descriptors like variations in faces.

After such (unsupervised) pretraining network is usually converted into MLP and used for normal supervised training. Note, that pretraining is done layer-wise. This significantly reduces solution space for learning algorithm (and thus number of training examples needed) as it only needs to learn parameters inside each layer without taking into account other layers.

And beyond...

Unsupervised pretraining have been here for some time now, but recently other algorithms were found to improve learning both - together with pretraining and without it. One notable example of such algorithms is dropout - simple technique, that randomly "drops out" some neurons during training, creatig some distortion and preventing networks of following data too closely. This is still a hot research topic, so I leave this to a reader.


What is the difference between Dilated Convolution and Deconvolution? These two convolution operations are very common in deep learning right now.

I read about dilated convolutional layer in this paper : WAVENET: A GENERATIVE MODEL FOR RAW AUDIO

and De-convolution is in this paper : Fully Convolutional Networks for Semantic Segmentation

Both seem to up-sample the image but what is the difference?

In sort of mechanistic/pictorial/image-based terms:
Dilation: ### SEE COMMENTS, WORKING ON CORRECTING THIS SECTION
Dilation is largely the same as run-of-the-mill convolution (frankly so is deconvolution), except that it introduces gaps into it's kernels, i.e. whereas a standard kernel would typically slide over contiguous sections of the input, it's dilated counterpart may, for instance, "encircle" a larger section of the image --while still only have as many weights/inputs as the standard form.

(Note well, whereas dilation injects zeros into it's kernel in order to more quickly decrease the facial dimensions/resolution of it's output, transpose convolution injects zeroes into it's input in order to increase the resolution of it's output.)
To make this more concrete, let's take a very simple example:  Say you have a 9x9 image, x with no padding. If you take a standard 3x3 kernel, with stride 2, the first subset of concern from the input will be x[0:2, 0:2], and all nine points within these bounds will be considered by the kernel. You would then sweep over x[0:2, 2:4] and so on.
Clearly, the output will have smaller facial dimensions, specifically 4x4. Thus, the next layer's neurons have receptive fields in the exact size of these kernels passes. But if you need or desire neurons with more global spatial knowledge (e.g if an important feature is only definable in regions larger than this) then you will need to convolve this layer a second time to create a third layer in which the effective receptive field is some union of the previous layers r.f..
But if you don't want to add more layers and/or you feel that the information being passed on is overly redundant (i.e. your 3x3 receptive fields in the second layer only actually carry "2x2" amount of distinct information), you can use a dilated filter. Let's be extreme about this for clarity and say we'll use a 9x9 3-dialated filter. Now, our filter will "encircle" the entire input, so we won't have to slide it at all. We will still however, only be taking 3x3=9 data points from the input, x, typically:
x[0,0] U x[0,4] U x[0,8] U x[4,0] U x[4,4] U x[4,8] U x[8,0] U x[8,4] U x[8,8]
Now, the neuron in our next layer (we'll only have one) will have data "representing" a much larger portion of our image, and again, if the image's data is highly redundant for adjacent data, we may well have preserved the same information and learned an equivalent transformation, but with fewer layers and fewer parameters. I think within the confines of this description it's clear that while definable as resampling, we are here downsampling for each kernel.

Fractionally-strided or transpose or "deconvolution":
This sort is very much still convolution at heart. The difference is, again, that we will be moving from smaller input volume to a larger output volume. OP posed no questions about what upsampling is, so I'll save a bit of breadth, this time 'round and go straight to the relevant example.
In our 9x9 case from before, say we want to now upsample to 11x11. In this case, we have two common options: we can take a 3x3 kernel and with stride 1 and sweep it over our 3x3 input with 2-padding so that our first pass will be over the region [left-pad-2:1, above-pad-2: 1] then [left-pad-1:2, above-pad-2: 1] and so on and so forth.
Alternatively, we can additionally insert padding in between the input data, and sweep the kernel over it without as much padding. Clearly we will sometimes be concerning ourselves with the exact same input points more than once for a single kernel; this is where the term "fractionally-strided" seems more well-reasoned. I think the following animation (borrowed from here and based (I believe) off of this work will help to clear things up despite being of different dimensions. The input is blue, the white injected zeros and padding, and the output green:

Of course, we're concerning ourselves with all of the input data as opposed to dilation which may or may not ignore some regions entirely. And since we're clearly winding up with more data than we began, "upsampling".
I encourage you to read the excellent document I linked to for a more sound, abstract definition and explanation of transpose convolution, as well as for learning why the examples shared are illustrative but largely inappropriate forms for actually computing the represented transformation.


MinHashing vs SimHashing Suppose I have five sets I'd like to cluster. I understand that the SimHashing technique described here:

https://moultano.wordpress.com/2010/01/21/simple-simhashing-3kbzhsxyg4467-6/

could yield three clusters ({A}, {B,C,D} and {E}), for instance, if its results were:

A -> h01
B -> h02
C -> h02
D -> h02
E -> h03

Similarly, the MinHashing technique described in the Chapter 3 of the MMDS book:

http://infolab.stanford.edu/~ullman/mmds/ch3.pdf

could also yield the same three clusters if its results were:

A -> h01 - h02 - h03

B -> h04 - h05 - h06
      |
C -> h04 - h07 - h08
                  |
D -> h09 - h10 - h08

E -> h11 - h12 - h13

(Each set corresponds to a MH signature composed of three "bands", and two sets are grouped if at least one of their signature bands is matching. More bands would mean more matching chances.)

However I have several questions related to these:

(1) Can SH be understood as a single band version of MH?

(2) Does MH necessarily imply the use of a data structure like Union-Find to build the clusters?

(3) Am I right in thinking that the clusters, in both techniques, are actually "pre-clusters", in the sense that they are just sets of "candidate pairs"?

(4) If (3) is true, does it imply that I still have to do an $O(n^2)$ search inside each "pre-cluster", to partition them further into "real" clusters? (which might be reasonable if I have a lot of small and fairly balanced pre-clusters, not so much otherwise)

As correctly pointed out above MinHash and SimHash both belong to Locality Sensitive Hashing.
Reference: https://en.wikipedia.org/wiki/Locality-sensitive_hashing

The main difference between the two is the way collision is handled,

SimHash, uses cosine similarity
MinHash, uses Jaccard Index.

Answers to your questions:

No. They uses different collision handling techniques to validate similarity. Also there is a variant on single Hash Function for Min Hash but it works differently. For more details, check the following reference out, https://en.wikipedia.org/wiki/MinHash (Variant with a single hash function)
Yes, https://github.com/chrisjmccormick/MinHash/blob/master/runMinHashExample.py
I think the complexity can be reduced to $O(n \log n)$ with modified form of binary search while clustering.


Validation of a model generated by automated deep learning system How would one go about selecting the validation set used to evaluate trained models by an automated system, in order to ensure that each new model is at least as good as, or better than the previous model?

Lets say that in this case we have a regression problem, and we can depend on a single performance metric such as RMSE.

I'm working on a program that automatically trains, tests and if the tests passed, deploys a new deep learning model every x new data samples/time.
Each training is done on the original dataset (labeled data) and the new dataset which is labeled by predictions of the previous model.

These were the options I came up with and some pros/cons.

Option A:
Create a validation set once, always apply this set
+ Always have a consistent benchmark
- Validation set might get 'out of date' in cases where the relevancy of data might change with time

Option B:
Create an initial validation set, add X percentage of new training samples to the original set
+ Consistent time spread of samples

Option C:
Create a random validation set every training
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
- Test results can vary each training session

Option D:
Cross-validation
+ Benchmark is most likely to contain a relevant set of samples (That are in data)
+ Likely least variance between each training session
- Computationally expensive, even more so for deep learning models

I would suggest Option E: Efron-Gong "optimism" bootstrap. It is similar to Cross-validation in spirit, but requires orders of magnitude less repetitions as CV and it exploits the whole dataset. The procedure is described in section 6 of this reference.


Build a model to classify given string/text input I need to build ML/NN model to classify/predict a given string pattern. Sample training data looks as shown in the image. Input will be the string in the column "Id Number", i need to tell to which class it belongs to in column "Id Type".

How do i move forward in building a model for text classification?
How to convert string to digit for using embedding in keras?

From the sample you have here it is not obvious that you need to apply some fancy ML technics. A simple rule based approach might also give you great results.
For instance:

if the string contains only numbers, then return B
If the string starts with 5 letters then return A
If the string starts with 2 letters and and then numbers return D
Etc.

That being said, to transform your strings into numbers a simple approach -looking at your data- could consist in assigning to each character a value in the set {0,1,...,9,10,..35,36}, 0 being assigned to the value 0, 9 to the value 9, A to the value 10, Z to the value 35, and NULL to the value 36 (as your strings don't have the same size, it might come in handy to introduce some placeholder for the blank values so that all your final vectors get to be of the same size)

Hope this helps!


asymmetric cost function for deep neural network binary classifier I am building a deep neural network based binary classifier, with single output. The loss function I actually want to minimize is
$$ \mathcal L(\hat y,y) =
\begin{cases}
0,  & \text{if $\hat y$ = 0} \\
1,  & \text{if $\hat y$ = 1 & $y$ = 0} \\
\gamma \approx -0.7 ,  & \text{if $\hat y$ = 1 & $y$ = 1}
\end{cases}
$$
where $y \in \{0;1\}$ is sample's label, $\hat y \in \{0;1\}$ - classifier's output and $\gamma$ - a hyperparameter. I think this is a case of assymmetric loss. (One can see it as betting: no reward when not betting, stake 1\$, payout 1.7\$ if bet wins)

From what I know by now, this loss function is probably not suited well for backpropagation and gradient descent. Question: Is there a better-suited formulation?

The often used cross-entropy loss doesn't allow for trade-off tuning between precision and recall. LINEX and LINLIN are assymetric by design purpose, but I coundn't find an example of a deep NN trained with them. An alternative could be leaving the loss function as it is and resorting to SPSA, but I'd like to keep it simple if possible.

Edit: I came up with
$$ \mathcal L(\hat y,y) = - (\gamma \hat y)^{y}(-\hat y)^{1-y}$$
At the moment, I have no clue if it is going to work for NN learning. I'm concerned (maybe unnecessarily) that, without logarithm, it is not convex regarding NN weights. (Last layer has sigmoid activation.) Here is the log loss shown for comparison.
$$ \mathcal L(\hat y,y) = - (y\log \hat y + (1-y)\log(1-\hat y)) = -\log[\hat y^y (1-\hat y)^{1-y}]$$

Rather than 0 use 0.1 or some other small non-zero value. Use the true y as an indicator function, multiply your class probability by your reward component, and sum terms for all possibilities to get a usable loss, like this:

$Loss(y, y_p) = (1-y)(1)(y_p) + (y)(-0.7)(y_p) + (0.1)(1-y_p)$

Where $y_p =P(\hat{y}=1|X)$


Interpreting Gradients and Partial Derivatives when training Neural Networks I am trying to understand of purpose of partial differentiation in NN training by knowing how to interpret gradients and their partial derivatives. Below is my way of interpreting them so I would like to know if I am correct, and if not, could someone please point me in the right direction.

If we are working with functions that depend on a single variable, then derivative of that function with respect to that particular variable is a slope (i.e. constant) which tells us how the changes in the dependent variable will effect the changes in the function value.

If we are working with functions that depend on several (N) variables, then derivative of that function with respect to all of these dependent variables is a gradient (i.e. vector of partial derivatives) which points to the direction of function extreme. Each partial derivative corresponds to a specific dimension in the N dimensional space that we are trying to optimize (e.g. quadratic cost function C(W,b)).

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct? If not, please correct my understanding of partial differentiation in NN training procedure.

Also, what is the role of Jacobian matrix in NN training?

Thank you so much!

My question is, when we calculate partial derivative with respect to one parameter (e.g. weight between input x1 and 1st hidden layer neuron) then we are treating all other weights and biases as constants and we are evaluating how will cost function change if we were to take a step in the direction that is represented by that particular weight. Is this correct?

Yes.

Long answer: that is exactly the meaning of partial derivative. It's the impact of a specific variable, while keeping all the others constant. It's an all things equal condition (or ceteris paribus, if you like Latin).

You need it in order to understand how much that specific parameter, at that current value, is contributing to the final Loss. You have to keep everything else equal in order to understand what is that parameter's responsibility of the final model's error.

  Also, what is the role of Jacobian matrix in NN training?

More generally, the Jacobian matrix of a Neural Network is a matrix of the partial derivatives of the y. It's size is:

( Number of observations, Number of parameters )

It's a way to store/represent your gradient information. The application of the chain rule to execute backprop is executed on those in practice.


Breaking a DataFrame based on Specific Column Values Assume I have a following dataframe which has years of experience and salary.

I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I would like it to include both columns.

One way to separate these values out of your pd dataframe is to use Pandas indexing. There are many different routes that you could go, for this, I would recommend:
new_data = pd.data[data['salary'] >= 45000]

Should separate your data out nicely.
You can read further about this here


R: Revalue multiple special characters in a data.frame R noob here..

I have the following data frame

>data
 Value Multiplier
1  15    H
2   0    h
3   2    +
4   2    ?
5   2    k

where the multiplier is of class factor. The values of K & k is 3, + is 5 and ? is 2.

I have used

> data$Multiplier <-  revalue(data$Multiplier, c("+"="5"))
> data$Multiplier <-  revalue(data$Multiplier, c("?"="2"))
> data$Multiplier <-  revalue(data$Multiplier, c("K"="3"))
> data$Multiplier <-  revalue(data$Multiplier, c("k"="3"))

Is there a better way of doing it?

That seems pretty straight forward to me. I'm pretty new too but in general I'm not sure if you can get better than one command. Though you could have combined all that:

> newValueVector <- c("+"="5", "?"="2", "K"="3", "k"="3")
> data$Multiplier <-  revalue(data$Multiplier, newValueVector)


what metrics to evaluate rank order results? I have searched on stackexchange and found a couple of topics like this and this but they are not quite relevant to my problem (or at least I don't know how to make them relevant to my problem).

Anyway, say I have two sets of prediction results, as show by df1 and df2.

y_truth = [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
y_predicted_rank1 = [6, 1, 7, 2, 8, 3, 9, 4, 10, 5]
y_predicted_rank2 = [4, 1, 7, 2, 8, 3, 9, 6, 10, 5]
df1 = pd.DataFrame({'tag': yy_truth, 'predicted_rank': y_predicted_rank1}).sort_values('predicted_rank')
df2 = pd.DataFrame({'tag': yy_truth, 'predicted_rank': y_predicted_rank2}).sort_values('predicted_rank')

print(df1)

#   tag predicted_rank
#1  1   1
#3  1   2
#5  1   3
#7  1   4
#9  1   5
#0  0   6
#2  0   7
#4  0   8
#6  0   9
#8  0   10

print(df2)
#   tag predicted_rank
#1  1   1
#3  1   2
#5  1   3
#0  0   4
#9  1   5
#7  1   6
#2  0   7
#4  0   8
#6  0   9
#8  0   10

By looking at them, I know df1 is better than df2, since in df2, a negative sample (zero) was predicted to have rank #4. So my question is,  what metric can be used here so that I can (mathematically) tell df1 is better than df2?

For comparing two rankings Spearman's rank correlation is a good measure. It's probably worth a try, but since your gold truth appears to be binary I would think that top-N accuracy (or some variant of it) would be more appropriate (advantage: easy to interpret). You could also consider using the Area Under the Curve (AUC), using the predicted rank as variable threshold (less intuitive to but doesn't require choosing any top N).


Loss function to maximize sum of targets I have a dataset {$X_i$, $Y_i$}. $Y_i$ - real value targets, they can be negative. The task is to teach classifier $f: X_i \rightarrow \{0, 1\}$ to maximize the sum below
$$
L(f) = \sum_i f(X_i) * Y_i
$$
So $f(X_i) = I[Y_i > 0]$ would be the ideal classifier.
I'm looking for appropriate smooth loss function to teach NN. NN output is probability $p(X_i)$, so $f(X_i) = I[p(X_i) > 0.5]$. I have tried the next loss function:
$$
BadLoss(p) = \sum_i p(X_i) * Y_i
$$
but because $Y_i$ can be negative it is bad-defined loss and NN converges to nothing good. The best results I got were with usual logloss:
$$
Loss(p) = \sum_i logloss(p(X_i), \; I[Y_i > 0])
$$

Can be there better loss functions to maximize the sum $L(f)$?

Define $Z_i = 1[Y_i>0]$, i.e., $Z_i = 1$ if $Y_i > 0$, else $Z_i = -1$.  Now  the data set $(X_i,Z_i)$ defines a two-class boolean classification problem.  The solution to that boolean classification problem with highest accuracy solves your original problem (minimizes your loss function).

So, you are basically asking how to train a neural network to solve a two-class boolean classification problem.  A reasonable approach is to put a softmax layer at the output of the neural network and train it using the cross-entropy loss, as usual.  During test time, to pick a class, you pick whichever of the two classes has the higher likelihood output from the softmax stage.  In other words, at training time, you look at the actual outputs (continuous-valued) from the softmax layer and use the cross-entropy; at test time, you compare the two outputs and pick whichever is larger to choose the classification (and apart from that ignore their exact values).

The shortcoming of this approach is that it doesn't take into account the value of the $Y_i$'s, only their sign.  To improve this, use weights on the samples.  We're again going to train a boolean classifier on the training set $(X_i,Z_i)$, but this time we'll weight each sample in the training set differently.  Put the weight $Y_i$ on the sample $(X_i,Z_i)$ in your training set.  In other words, when training the boolean classifier, the loss function will be a weighted sum of the loss for each sample:

$$\text{Loss}(\theta) = \sum_i Y_i \cdot \ell(f_\theta(X_i),Z_i)$$

where $\theta$ are the model parameters, $f_\theta(X_i)$ is the output of the classifier on input $X_i$, and $\ell(\cdot,\cdot)$ is the cross-entropy loss.  Notice that errors in samples where $Y_i$ is large are penalized more, and errors in samples where $Y_i$ is small are penalized less; this is appropriate, as that's exactly what will happen when you use the classifier $f_\theta$ to compute $L(f_\theta)$.  This should improve on the approach you're taking.


How to choose between discounted reward and average reward? How to select between average reward and discounted reward?

And when average reward is more effective in comparison with discounter reward and when vice versa is correct?
Is is possible to use both of them  in a problem? Because as I understand the RL reward  is based on average reward or discounted future reward, but I think this paper use the discounted and average together. Is it correct: we use discounted future reward in order to training and average reward in test and evaluation? What is wrong in my understanding?

In this picture, figure 2 of the paper "Playing Atari with Deep Reinforcement Learning":

The authors report the "average reward". However, in the same paper, the authors also mention "discounted reward". So, I'm confused. What is the difference between discounted reward and average reward?

The Average reward in that figure is used as a measure of performance. In other words, the score of the agent playing the game. You do not track reward per episode as this doesnt indicate a general improvement in the learning process. Instead you track the average reward over training epochs. If it steadily increases this means that your agent indeed is learning.

The discounted reward is used to create some kind of future-reward dependencies and is used in the learning equations. So, instead of evaluating how good is a particular state according to the immediate reward you received, you also take into account the future reward from your next state. In RL you attempt to max your expected return and some methods estimate the expected reward from every state.

Please note that my answer gives a high level description and is not referring to a specific RL algorithm (as there are many variations). I would suggest you to understand very well the simple tabular form of Q-learning before moving to RL and function approximators combinations.


Estimating average daily consumption with samples randomly scattered in time I want to estimate my daily water consumption. I have taken pictures of the water meters (total m3 used since last reset) every now and then, but without any regularity. There can be a difference of a few days to several weeks between samples.
What would be the best way to estimate this? I have thought of the following approaches:

Create a double-entry table with the sample dates in the column and in the row headers. Each cell is the average consumption per day between the corresponding two dates. Finally, average all the cells in the table.
Calculate the daily consumption between every two consecutive samples. Finally, average all of them.

It seems to me that the first one would give a better estimation given the higher number of samples compared, but I am not sure if this is valid.

If you only want the average daily consumption over the whole period of time, you can simply calculate the difference between the last and first reading and divide by the total number of days.
As far as I understand your explanation:

your method 1 would not give you the true average over the period, unless you multiply each individual average by a weight corresponding to the number of days in the period.
your method 2 gives the true average, since it would represent every day individually (if I understand correctly)

However if you want to be able to observe the variations across time while smoothing some of the irregularities, you could:

calculate the daily average for every day like in method 2.
calculate a rolling average over a fixed number of days for every day. This is not perfect if there are long periods with no reading, but it should show the general trend better.
Plot a graph based on rolling average.


Understanding Layers in Recurrent Neural Networks for NLP In convolution neural networks, we have a concept that inner layers learn fine features like lines and edges, while outer layers learn more complex shapes.

Do we have any such understanding for layers in RNNs (like LSTMs), something like inner layers understand grammar while outer layers understand more complete meanings of sentences assuming that we are using the LSTM for some natural language task like text summarization?

Its not like it just understands grammar.

In LSTMs the network tries to preserve the hidden states over time. By doing this they try to learn long-term dependencies in the language and relationships between words at variable distances.

LSTM does this by using its three famous gates.

Forget gate -  Tries to remember only the important features and relationships overtime.
Input gate - Adds new information to old cell state at each time time step.
Output gate - Produces new output by taking into account old cell state and output at each time step.


Is the following a good method for determining learning rate in a neural network? Start with a default Learning rate of 0.1

Run the gradient descents normally.

If the cost function ever gives a higher value than the previous iteration, divide the learning rate by 10.

Repeat steps 1-3 until the learning rate becomes $10^{-6}$.

Starting with a high value is not a good idea. Since we are not aware of where the training starts on the Loss surface, so it can easily diverge.
Also, in between we need a higher LR to quickly move on saddle points (plateaus).
 Both start and end should be at a small value
A well-known approach is called Cyclical-Learning-Rate suggested by Leslie N. Smith in this paper Link
It suggests following a Cyclic path e.g. Triangle, Sinusoidal, etc.
It says to first increase from a minimum bound and starts decreasing when reached a maximum bound.
$\hspace{3cm}$
Paper has clearly explained -
- How to find the bounds
- How to decide the stepsize


How to check the randomness or the distribution of an attribute in a dataset? I have a dataset. I want to make its randomness nature match with another dataset (2nd one). Basically I want to modify the 2nd dataset values and make them similiar in distribution or nature in accordance with the 1st dataset.
I'm working on Spyder using Python 3.
I want to do it for some experimental purposes

One option is to check and compare feature statistics across the datasets: mean, variance (or standard deviation), kurtosis, and skewness.

If you prefer a more visual approach (I do!), you can plot density or histograms of each feature, creating overlays for each dataset.

pandas can be useful in calculating the statistics; for visualisation I like using seaborn.


Tree based method are robust against low probability feature space zones when using ML general interpretability methods? I have this intuition but I'm not able to verify it.

There are a lot of techniques to understand the effect of single features in ML models. Some take inspiration from counterfactual frameworks, like ceteris paribus, and evaluate the unconditional contribution of feature $X$ by observing the change in prediction when varying $X$ values leaving all other variables fixed.
The most common of such techniques are PDPs https://christophm.github.io/interpretable-ml-book/pdp.html.

The problem is that this methodology are not robust for impossible combinations of the predictors. For example in a model to predict bike-sharing count given weather conditions and period of the year, it's possible to make predictions for a temperature of 40°C in wintertime, even if there's no such data point in the training set.

There are various techniques to accommodate for this bias, like accumulated local estimation plots (ALE). I was wondering though if tree-based methods (simple or ensemble) are naturally more robust than regression-based ones to such bias; I expect this, because tree-based predictions vary only among partitions of the feature space that are present in the data, while regressions allow prediction variation for never observed predictors combinations.

For example, this is the output of a conditional tree trained on the bikes   problem:

[1] root
|   [2] temp <= 12.2
|   |   [3] season in SPRING, SUMMER
|   |   |   [4] temp <= 4: 1663 (n = 64, err = 30258081)
|   |   |   [5] temp > 4: 2852 (n = 133, err = 216353574)
|   |   [6] season in WINTER
|   |   |   [7] hum <= 82.3: 4315 (n = 90, err = 117371810)
|   |   |   [8] hum > 82.3: 2781 (n = 9, err = 26537744)
|   [9] temp > 12.2
|   |   [10] hum <= 84.8
|   |   |   [11] windspeed <= 13.2: 5877 (n = 256, err = 454812206)
|   |   |   [12] windspeed > 13.2: 5285 (n = 149, err = 326330122)
|   |   [13] hum > 84.8: 3382 (n = 30, err = 47251364)

as expected, the temperature and the seasons are correlated, therefore we won't find rules regarding winter for higher (>12.2) temperatures.

So I expect that forcing Winter with a temperature of 14 won't produce a different prediction than Summer. I expect also that this robustness would replicate also for more complex blackbox models like random forests and boosted trees.

Instead, regression-based methods will allow impossible predictions as shown by the following linear model, where the effect of temperature is unbounded.

   (Intercept)           temp   seasonWINTER            hum      windspeed   seasonSUMMER holidayHOLIDAY
        4888.4          152.1         1307.1          -37.6          -64.0          673.2         -621.4

Can someone confirm/dispute this, preferably with a theory-based explanation?

Well, are you content with a system that doesn't work on that one day in winter when the temperature actually reaches 40 degrees? And would you care, since you probably now got other things to worry about

An assumption of independence is usually made to deal with sparse situations. Naive Bayes for instance works pretty well for document classification tasks: Each word (token) in a document is taken as an individual observation governed by a probability distribution belonging to the document class

Tree-based classifiers, on the other hand, generate composited rules and are thus geared toward exploiting conditional probabilities. Eg: If it is 12.2 degrees or below, and it is Winter, then the humidity is the discerning factor in bike use.

NOTE: That even though your rule sample is small, it is still quite complex Suppose we eyeball it and make a naive rule: If humidity < 83 roughly add around 55% That would emulate your rules quite well, and make it less complex, unless it happens to be a cold spring or summer day. Is that really a rule? Would that cause faulty predictions? Or did we just see little or no cold days that varied in humidity in that sample (200 data points)? I wouldn't start betting my bottom dollar that people react differently to humidity given the date on the calendar

You pose that the tree rules protect against probability mass leaking to impossible situations. Conversely you could pose that naive probability would help you make predictions about situations that were sparse in the dataset. Whether you'd want that is up to your reasonable judgement. It is machine learning, not machine wisdom


What level of math is required for machine learning research There are several levels of math understanding:

Know the math
Know the intuitions behind math concepts
Know the intuitions and proofs of math concepts
Know the intuitions, proofs of math concepts and be able to apply them to deduce new results

My question is what level of understanding is required for machine learning research? Especially for publishing papers in conferences like CVPR or NeurIPS.

There can be no objective answer to this question. Obviously the more one understands the better, but the field of ML is vast, quite specialized and ranges from very theoretical to very applied research, so it's perfectly reasonable to publish in ML without a strong background in maths.

A better way to estimate your own ability to publish papers in a particular area or journal is to study recent papers published in this area/journal. You should be able not only to understand them but to redo the reasoning:

Understand the problem and the solution proposed by the authors
starting from the initial problem, how would you solve it? Can you think of alternatives to the authors' solution?
Can you find limitations to their approach and improve on it?

If you reach step 3 then congratulations: you are ready to publish your own research!


Can reducing the number of classes in multi-label classification increase performance? This is more of an open question with people which have experience in this.  I'm working on a multi-class multi-label classification for chest x-rays. I would like to know how much can reducing the number of classes increase performance for remaining classes before moving forward and making big changes to the training pipeline.
For the moment, there's about 50 classes with AUC performance varying between 0.7 and <~ .99. Final outputs used are some tweaked sigmoids and loss function is based on cross entropy as it's multi class multi label. However, not all 50 classes are equally important. One option would be to give class weights. But I'm also thinking about just dropping or merging some classes instead. Hence my question:
Has anyone experience a significant increase in performance for the remaining classes by merging / dropping-out some classes ?
Thanks !

Yes, merging / dropping classes will tend to increase performance. If you merge classes, there will be more examples per class which will tend to decrease the variance of a model. Reducing the total of number classes has the potential to allow the model better fit the data, reducing bias. Many models are constrained by learning capacity.


Inferring Relational Hierarchies of Words I am new to natural language processing and I have not heard of a problem similar to mine yet. I was wondering if anyone could refer me to a method for solving my problem, or tell me how this problem is referred to in the academic literature, so that I can look for resources online.

Here is the problem :
From some text (wikipedia articles, for example), I would like to extract the hierarchy of different concepts that can be found in it. By hierarchy I mean a tree wherein A is a descendant of B if A or one of A's parents (transitive) is defined by B. For instance, normal distribution would be a descendant of probability (since normal distribution is defined using probabilities) and probability would be a descendant (or child) of mathematics. Since it is transitive, normal distribution would also be a child of mathematics.

One way I thought about solving this is by looking at the number of times a word A is used alone (called A), the words A and B are used together (called A AND B, 'together' could be, for instance, in the same article or in the same paragraph, or in the same sentence), and the number of times the word B is used alone (called B). Let A be mathematics and B be probability. Then, if the ratios (A AND B)/A and (A AND B)/B are low, then it could imply that there is no direct link between A and B (but a link could exist through transitivity). Conversely, if A is bigger than B, A is a bigger concept than B. If A and B are almost the same then they are probably siblings (children of the same parent).

Let's take 3 examples:

Mathematics (A) and carrot (B). A AND B is really low compared to A and B, so there is no direct link between them (or only an indirect link by transitivity).
Mathematics (A) and probabilities (B). A AND B is quite high compared to B, and A is much bigger than B, so B should be a child of A (probabilities is a child of mathematics).
Topology (A) and Probabilities (B). A AND B is relativaly high (the texts that present the different areas of mathematics will likely speak about the 2), A and B are about the same order of magnitude, so A and B should be the children of a same parent. Indeed, Topology and Probabilities are the children of Mathematics.

This way of solving the problem is far from perfect, for instance 'the' (A) and 'probability' (B) would probably end up saying probability is a child of the (because A AND B is huge and A is much bigger than B).

If anyone knows some papers on this or has any ideas on how I might solve this problem, I would appreciate some direction. Also, does my solution seem viable? How could it be improved?

Look up taxonomy/ontology construction/induction. Relevant papers:

Automatic Taxonomy Construction from Keywords via Scalable Bayesian Rose Trees
Topic Models for Taxonomies
OntoLearn Reloaded. A Graph-Based Algorithm for Taxonomy Induction
Ontology Population and Enrichment: State of the Art
Probabilistic Topic Models for Learning Terminological Ontologies


How can i build a machine learning model to identify a specific word? I am trying to build a model that can be used to identify the word sales whenever a group of text are passed through the model. Should I use Azure or Python for this? I have done the text preprocessing and extracted the n-gram features in Azure, what should I do next and which model should I use?

It is always best to approach a problem using the simplest possible tools. In your case there is no need to use a machine learning (AI) model to detect the word sale.

To find the word sale within a long datastream, I would do the following: parse through the entire String while looking at segments of size 4 at a time, convert the substring to lowercase. Determine if this substring is "sale". If it is save the index of the "s" into a list. Return the list which will include the index of each start of the word sale within your text.

public ArrayList<Integer> getIndexSale(String inputString){
   ArrayList<Integer> indexes = new ArrayList<>();
   String[] queue = new String[4];

   queue[0] = 0;
   queue[1] = 0;
   queue[2] = 0;
   queue[3] = 0;

   for(int i = 0; i<inputString.length(); i++){
      queue[0] = queue[1];
      queue[1] = queue[2];
      queue[2] = queue[3];
      queue[3] = inputString.charAt(i);

      String temp = java.util.Arrays.toString(queue).toLowerCase();

      if(temp.equals("sale"))  indexes.add(i);
   }

   return indexes;
}

As mentioned by Dan Carter: if you do not care about the indices of the word "sale" within your string then it is simpler to use:

public boolean getIndexSale(String inputString){

   if(inputString.toLowerCase().contains("sale")) return true;
   return false;
}


Reading strategy for ML books I am trying to read the following list of books on statistical learning. I have a BSCS and about 4 yrs of experience working in image processing and parallel programming. I won’t be an expert in the field by any means, however my aim is:

not to be a script kiddie, using tools and algorithms without understanding the hows and whys.
be able to read and digest the latest research in statistical learning, specially w.r.t computer vision.

Prerequisites I have studied in preparation:

Matrix Algebra by James E. Gentle
Statistical and Mathematical Methods lectures by Carlos Fernandez-Granda

Books to read:

An Introduction to Statistical Learning with Applications in R by Robert Tibshirani et al.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Robert Tibshirani et al.
Understanding Machine Learning From Theory To Algorithms by Shai Shalev-Shwartz et al.
Pattern Recognition and Machine Learning by Christopher M. Bishops
Information Theory, Inference, and Learning Algorithms by David J.C. MacKay
Deep Learning by Ian Goodfellow et al.
Convex Optimization by Stephen Boyd et al.

I am looking for a reading strategy. I'd specially appreciate input from users who’ve read the majority of the books.

I've read all but the last one. If you are serious about computer vision you won't get much out of this reading list. However, if you are set on these books I would start with Goodfellow's Deep Learning which is fairly self-contained and the only book that covers computer vision enough to make it rewarding for that purpose. You can branch out from here and read additional material referenced in the book where you lack background.

If you are less concerned with computer vision you can start with the other books, they are often read in the following order:

An Introduction to Statistical Learning with Applications in R
The Elements of Statistical Learning
Bishop's Pattern Recognition and Machine Learning

If you finished with these you could tackle McKay which is quite dry and boring. Shai Shalev-Shwartz et al. is a nice book that covers a lot of the material in the other books but from the perspective of Empirical Risk Minimization and covers Vapnik–Chervonenkis theory. It is more theoretical and is more structured in the classical theorem-proof-corollary scheme. The perspective taken is less practical but adds to the other books.

I don't know the book on Convex Optimization but you can probably read it in any order. If you want to read all of the books and don't care how long it takes, you can probably read that first as it sounds more foundational.