(Not a programming question) Is there a common word to indicate predict or forecast The general understanding is:

Predict: past value - could be data the model has seen (data from train set) or past data model has not seen (data from test set) (together in-sample data)
Forecast: future value - data which neither model or user has seen (out-of-sample data)

Related question: https://stats.stackexchange.com/questions/65287/difference-between-forecast-and-prediction
Question: Is there a common word that can mean either? This will help in talking or writing or explaining to non-technical people.

I am not sure if I agree with the first one, since predict is widely used to replace forecast as well. But other related verbs we often hear are:

To estimate
To infer / make an inference


Addition and subtraction of values of two columns when using Orange GUI I am a new user to the Orange GUI.

I would like to know if there is anyway I can add or subtract two columns and then return the result as a new column?

Use Feature Constructor like so:

and X1 will be the name of your new feature (you can obviously change it)


How to download dynamic files created during work on Google Colab? I have two different files and on the first, I tried to save data to file as:

np.save(open(Q1_TRAINING_DATA_FILE, 'wb'), q1_data)

On second file, i'm trying to load it the same way using:

q1_data = np.load(open(Q1_TRAINING_DATA_FILE, 'rb'))

I then get the error:

FileNotFoundError: [Errno 2] No such file or directory: 'q1_train.npy'

I searched my google drive but couldn't find this file.

Platform: https://research.google.com

Edit:
I'm trying to run below Kaggle problem on Colab platform. The author has two files (Jupyter and nbs) - one to prepare and 2nd to train. The step on nb1 where it's creating some files - which later to be consumed by file 2 is where I'm struck.

https://github.com/bradleypallen/keras-quora-question-pairs/blob/master/quora-question-pairs-training.ipynb

Based on what I've seen and experienced, the best way is to store and retrieve your data from your drive account. Actually your question is a bit unclear but first I say, try to use the following command to see the current files in your directory, although I guess each 12 hours they all would be deleted automatically.

!ls

Anyway I recommend the following instructions:

Use the following code for having permission to access to your drive account:

!pip install -U -q PyDrive

import tensorflow as tf
import timeit

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

Use the following code to get the id of contents in your drive:

file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file1 in file_list:
  print('title: %s, id: %s' % (file1['title'], file1['id']))

Put the id of the desired file, e.g. a typical text file, in the content of the following dictionary with id key:

downloaded = drive.CreateFile({'id': 'the id of typical text file'})
file = downloaded.GetContentString()
print('Downloaded content "{}"'.format(len(file)))

Till now you have copied the text file, then you have to write it in your Colab disk using the following code:

text_file = open("your desired name.txt", "w")
text_file.write(file)
text_file.close()

Create & upload a file.

uploaded = drive.CreateFile({'title': 'filename.csv'})
uploaded.SetContentFile('filename.csv')
uploaded.Upload()
print('Uploaded file with ID {}'.format(uploaded.get('id')))

Downloading from Colab without Uploading to drive

from google.colab import files
files.download('your typical h5 file or what ever.h5')

For more information about transferring different data formats there are more explanations in the notebook provided with Colab.


Feature selection - SelectKBest sklearn I would like to ask how to set paramater k in function SelectKBest for feature selection. I have now around 2300 features, so I think that default value 10 is not enough. Is there any approach, how many features choose or it is just on testing to find some compromise between accuracy and number of features?

This is a hyperparameter. And so this is usually approached with some sort of hyperparameters search with cross-validation, like GridSearchCV or RandomizedSearchCV.
https://scikit-learn.org/stable/modules/grid_search.html


Is there an oriented clustering algorithm? I'm looking for a clustering algorithm that will make cluster depending on a orientation. The DBSCAN algorithm cluster points based on a constant radius :
https://upload.wikimedia.org/wikipedia/commons/thumb/a/af/DBSCAN-Illustration.svg/800px-DBSCAN-Illustration.svg.png
Is there a implementation of DBSCAN that is based on "ellipse instead of circle" ?
EDIT: MY SOLUTION
Ok so my solution was to work on my data set.
I had a set of 2D points and I wanted to favor the definition of clusters depending of a given orientation.
My solution was to center my set of point on the origin of the coordinate system, rotate them by the orientation you want and apply this vector field on the set of point : X(x, y) = (x-x*a, y) , where a is the factor that determine if the orientation should matter a lot or not (a ∈ [0, 1]) .
Then apply the DBSCAN of this modified dataset.
I hope I was clear enough, don't hesitate to ask me if it's not the case.

If I remember correctly, non-negative matrix factorization (NMF) can be used as a clustering approach that can recover clusters that are along vectors, for example. It may work for your dataset. It factors a data matrix $D \in \mathbb{R}^{m * n}$ into two matrices $W \in \mathbb{R}^{m*k}$ and $H \in \mathbb{R}^{k * n}$. Effectively, $W$ contains the weights that are applied to each vector in $H$ to reconstruct the original data; one way of using this method is to interpret the $n$-dimensional vectors in $H$ as clusters (these vectors would be the 'directions' that your data is along) and the $k$-dimensional vectors in $W$ as the data-example-wise affinities for the different clusters. One method to cluster with this process is to simply place each of the $m$ data examples into the cluster with the index of the highest value in the $W$ vector.

There are implementations in several standard libraries, including sklearn, so it should be relatively easy to try it out.  Good luck, and welcome to the site!


Modules on Python which are useful for missing Word/Letter prediction in text paragraphs from a coprpus Can someone please recommend me some Python packages which I can use for missing word prediction in a paragraph of text. The missing words are a part of existing semantic list of words.( List of missing words are a common set of words like was ,were etc). If there are better packages in R which can help me with this  it would be very useful if you post it here.

Thank you.

This problem seems to be exactly (or at least very similar to) that in the Billion Word Imputation challenge at Kaggle. Some of the approaches discussed in the forums might help you out. My initial thoughts for useful tools/methods would be word2vec, the stanford NLP tools and sklearn.


NLP Transformers: How to get a fixed sentences embedding vectors size? I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)

def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   embeddings = all_layers[0]
   return embeddings

# Here we see that the shape of the embedding vector depends on the number of tokens in the sentence

u = embed("Bonjour, ça va ?")
u.shape # torch.Size([1, 7, 768])
v = embed("Salut, comment vas-tu ?")
v.shape # torch.Size([1, 9, 768])

Imagine now, I want to calculate the cosine distance between the vectors (tensors in our case) u and v :

cos = torch.nn.CosineSimilarity(dim=0)
cos(u, v) #will throw an error since the shape of `u` is different from the shape of `v``

I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of tokens?

=> The first solution I'm thinking of is calculating the mean on axis=1 (mean embedding of tokens in the sentence) since axis=0 and axis=2 have always the same size :

cos = torch.nn.CosineSimilarity(dim=1) #dim becomes 1 now

u = u.mean(axis=1)
v = v.mean(axis=1)

cos(u, v).detach().numpy().item() # works now and gives 0.7269

But, I'm afraid that I'm hurting the embedding when calculating the mean!

=> The second solution is to pad shorter sentences out, that means:

giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
look up for the sentence with the longest tokens and embed it, get its shape S
for the rest of sentences embed then pad zero to get the same shape S (the sentence has 0 in the rest of dimensions)

What are your thoughts? What technique would you use and why?

I don't think there will be a definitive answer, but I suspect that you'll get better results using the averaging method rather than the padding method.

One big problem with the padding method is that it's sensitive to word order. For example, the sentences "Gibbons are one type of ape" and "One type of ape is the Gibbon" look very different if we do a word-by-word comparison.  "Gibbons" is very different than "One", "are" is very different than "type", etc.  This problem will be offset slightly because RoBERTa embeddings are context-sensitive, but you get the idea.

Another big problem is that the padding can dominate the similarity comparison for sentences with different lengths.  Suppose we want to compare the following two sentences which have very similar meaning:

Sentence1: "Very furry" would be an apt description for most mammalian species.

Sentence2: Mammals have hair.

We would add 8 padding tokens to sentence2 in order to give it the same length as sentence1.  But then the similarity computation is dominated by the padding tokens and not by the actual content of the sentence.

To me, the averaging approach seems superior because it avoids these two problems.  Of course it also has drawbacks.  The "sentence embeddings" obtained by averaging the word embeddings will be noisy because each word is given equal weight.

You could try running a keyword detector over the sentences you wish to compare.  Then compare the means of the top-k keywords instead of the whole sentence.


key parameter in max function in Pyspark In the example given for the max function for PySpark:

Pyspark

>>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
>>> rdd.max()
43.0
>>> rdd.max(key=str)
5.0

Q1. How does it get evaluated to 5.0 when key=str ? Is it based on conversion to character/string type ?

Q2. Is there any value the parameter "key" can take ?

I also found the function definition of "max" at this location

https://github.com/adobe-research/spark-gpu/blob/master/src/rdd.py

def max(self, key=None):
        """
        Find the maximum item in this RDD.
        :param key: A function used to generate key for comparing
        >>> rdd = sc.parallelize([1.0, 5.0, 43.0, 10.0])
        >>> rdd.max()
        43.0
        >>> rdd.max(key=str)
        5.0
        """
        if key is None:
            return self.reduce(max)
        return self.reduce(lambda a, b: max(a, b, key=key))

You pass a function to the key parameter that it will virtually map your rows on to check for the maximum value. In this case you pass the str function which converts your floats to strings. Since '5.0' > '14.0' due to the nature of string comparisons, this is returned. What is usually a more likely use is using the key parameter as follows:

test = sc.parallelize([(1, 2), (4,3), (2,4)])
test.max(key = lambda x: -x[1])
(1,2)

Because of the - we sort descending and the x[1] means we use the second entry in our tuples as the key.


Memory efficient encoding logic for group categories I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this:

     GroupID           System        State       TimeStamp        etc...
0    [1, 2, 3, 4]         A           REC           ...
1    [1, 2, 3, 4]         A           SNT           ...
2    [2, 4]               B           REC
3    [2, 4]               B           PND
4    [2, 4]               B           COM
5    [2, 4]               B           SNT
6    [2]                  C           RCV
7    [2]                  C           ACC
...

There are more than 100000 different group IDs in over 3 mil alerts.

Creating a column with a single Group ID value (not a list) means some alerts will appear more than once, which is not good given the already huge dataset.
Creating a separate column for each group (binary encoding) would expand my data too much horizontally.

What is a memory efficient way of encoding Groups?

There is several techniques that could work for you:

Target Encoder: Works well when there is a high cardinality of a categorical feature.
Ordinal/Label Encoding: Tradition label encoding
Weight Of Evidence:  tells the predictive power of an independent variable in relation to the dependent variable


Select more or less features if results are almost the same I am having a dataset of 3500 observations with 70 features each with binary labels/targets for classifications purposes.

My aim is to score more than 90% precision and the highest recall possible for this precision.

I have tested many algorithms and thus far Random Forest performs the best.

Specifically, with 70 features I am getting:

Auroc: 0.71
Precision: 0.94
Recall: 0.18

However, I had a look at the importance of my features by calling the built-in function feature_importances_ of the RandomForestClassifier of SkLearn and by also using the Boruta algorithm with the boruta_py module.

I removed the 20 least important features. The number 20 was quite arbitrary because even the next 10 least important features were relatively close in terms of importance to these 20 least important ones. Only the top 10 most important features had a significant difference in importance in comparison with the others.

Then the results with 50 features were the following:

Auroc: 0.7
Precision: 0.91
Recall: 0.17

Since I can get almost the same results with less features should I remove these features?

Please also into take into account that in the future my dataset will be expanded to 5000 observations.

It is very contextual. There's a rule that feeding all the data will yield better (unbiased) results but not at all times. If you know anything about the data, then it is always advised to remove the most obvious ones that won't matter. Additionally, you can use other metrics like 'P-Value' to check the variable significance.

Since there's more data that will be added in the future, I'd rather wait for it and then remove the insignificant variables.


Transposing Every nth row to column in a large dataset I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a "long" .txt file as follows:

'cat1/: Topic1_Variable1'
'cat2/: Topic1_Variable2'
'cat3/: Topic1_Variable3'
'cat4/: Topic1_Variable4'

'cat1/: Topic2_Variable1'
'cat2/: Topic2_Variable2'
'cat3/: Topic2_Variable3'
'cat4/: Topic2_Variable4'

'cat1/: Topic3_Variable1'
'cat2/: Topic3_Variable2'
'cat3/: Topic3_Variable3'
'cat4/: Topic3_Variable4'
...

Fro analysis and sharing with others, I really would like to see it formatted as follows:

cat1              cat2              cat3              cat4
Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4
Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4
Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4

I think that this may be easier in R, but I honestly am drawing a complete blank in SAS. I've even played with MS Access to try to get it to look the way I want but the program crashes every time (due to the size?). At any rate, I have looked into some of the statements in PROC TRANSPOSE and PROC SQL but it seems that most functions within those procedures are utilized to combine duplicate 'Topics'. In the data I have been provided, each "group" represents an individual response to a question with several thousand individuals repeated, I want to retain the independence of each occurrence and not perform a UNION as defined in PROC SQL. At this point, I feel like I am over-thinking this but I just can't get around the mental block and actually do what I am working toward.
Any help or guidance is much appreciated. I'm open to trying all suggestions or ideas, I think I have access to most statistical computing programs.

Hey is Python or other tools an option here? Since you mentioned it is a large dataset, you might want to iterate over it instead of loading all of them at once.

Here is a solution in Python:

import pandas as pd
from collections import defaultdict

inputs = [
'cat1/: Topic1_Variable1',
'cat2/: Topic1_Variable2',
'cat3/: Topic1_Variable3',
'cat4/: Topic1_Variable4',
'cat1/: Topic2_Variable1',
'cat2/: Topic2_Variable2',
'cat3/: Topic2_Variable3',
'cat4/: Topic2_Variable4',
'cat1/: Topic3_Variable1',
'cat2/: Topic3_Variable2',
'cat3/: Topic3_Variable3',
'cat4/: Topic3_Variable4',]

outputs = defaultdict(list)

for item in inputs:
    cat, topic = item.split('/: ')
    outputs[cat].append(topic)

print pd.DataFrame(outputs)

Output:

               cat1              cat2              cat3              cat4
0  Topic1_Variable1  Topic1_Variable2  Topic1_Variable3  Topic1_Variable4
1  Topic2_Variable1  Topic2_Variable2  Topic2_Variable3  Topic2_Variable4
2  Topic3_Variable1  Topic3_Variable2  Topic3_Variable3  Topic3_Variable4


Algorithm for deriving multiple clusters Suppose I have a set of data (with  2-dimensional feature space), and I want to obtain clusters from them. But I do not know how many clusters will be formed. Yet, I want separate clusters (The number of clusters is more than 2). I figured that k means of k medoid cannot be used in this case. Nor can I use hierarchical clustering. Also since there is no training set hence cannot use KNN classifier to any others (supervised learning cannot be used as no training set). I cannot use OPTICS algorithm as I do not want to specify the radius (I don't know the radius) Is there any machine learning technique that would give me multiple clusters (distance based clustering) that deals well with outlier points too?
This should be the output:

The fact is that you could use any of the algorithms you mentioned, and in general any algorithm that requires to set the number of clusters as a parameter (or any other parameter that indirectly sets the final number of clusters, like the threshold in a hierarchical clustering algorithm.)

The solution to your problem is model selection. Model selection methods evaluate different clustering solutions, and select the one that optimizes a given criterion.

For instance, in the case of K-means, you could find a solution for a range of k values, and keep the one that maximizes any cluster validation measure (see the Wikipedia entry for cluster analysis to read about some examples of cluster validation measures).

There are automatic and more complex approaches (one specific example is "Automatic Cluster Number Selection Using a Split and Merge K-Means Approach" by Muhr, M. and Granitzer, M., but this is just an example). These methods use cluster validation measures to automatically split or merge clusters, but the idea is basically the same.


SVM, which range to use when normalizing I am using the SVM classifier from Scikit Learn.
I was wondering is there is a know-best-practice when it comes to normalization. I'm using different normalization tecniques, but all my normalized data are now between 0 and 1.

Using GridSearchCV, I always end up picking the linear kernel for the SVC. Would it be better if I were to normalize having my values between -1 and 1?

I feel like the answer is yes, thinking about situations like this one:

But I wanted to know if there is an official answer. Thanks a lot

Typically one decides between a range based normalization technique and a mean and standard deviation based normalization technique, which is known as standardization.
Standardization is often preferred to some types of normalization as normalization can put too much weight on anomalous data at the extrema of the cohort. This is typically not a problem, but certain discretization techniques can then hide the interesting structure in the data in a very narrow region. In your case, the structure might be being missed by your grid search, which is why you always arrive at a linear kernel. I suggest first standardizing your data and retrying your grid search. Does this arrive at a non-linear kernel?
If not, then also just try choosing some different kernel types and see if they outperform the linear kernels after you have used a grid search to optimize the other parameters.
The 3D structure in your example that is not apparent when projected into 2D should not be particularly applicable to this case since dimensionality and linearity are very different concepts.
Hope this helps!


Why is sum a succinct constraint? I'm new to data mining and have been going through constraint-based query mining lately. I came across the concept of succinctness which basically details a constraint as succinct, if we can generate all the candidate item-sets precisely, based on an itemset satisfying the constraint.

A more formal definition is :

  Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1

Example, min(S.Price) <= v  is succinct
But, sum(S.Price) >= v  is not succinct

I understand why the former is a succinct constraint => as all the candidates can be generated by ensuring that one of the subsets satisfies that constraint.
But I fail to understand why the latter is not a succinct constraint. Any pointers on this would be helpful !

We can show that the "sum above threshold" is not succinct by providing a counter example.

As you wrote the definition is

  Given A1, the set of items satisfying a succinctness constraint C, then any set S satisfying C is based on A1 , i.e., S contains a subset belonging to A1

Hence, we can provide a counter example by providing a set A1 satisfying the constraint while none of its subsets satisfying it.

Consider three items, a,b and c such that each one on them costs 1.
Let the constraint C be sum(S.Price) >= 3

For the set {a, b, c} the sum of prices is 3 and therefore the constraint C is satisfied.
For each of the subsets of {a, b, c} the sum of prices is lower than 3 and therefore C is not satisfied.
We found a counter example in which a set satisfies a "sum above threshold" while none of its subsets satisfies it. Hence, "sum above threshold" is not succinct.


Bechmark for Movielens I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.

In this instance, I'm interested in results on the MovieLens10M dataset.  It seems to be referenced fairly frequently in literature, often using RMSE, but I have had trouble determining what might be considered state-of-the-art.

I did find this site, but it is only for the 100K dataset and is far from inclusive:

http://www.recsyswiki.com/wiki/MovieLens_100k_benchmark_results

As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve You should benchmark Your solution with suitable method. The best source for this can be publication, where as You noticed, many times results for ML dataset are given. What's more about benchmarking recommender systems, sometimes the best prediction results are not so important as other aspect e.g. novelty or diversity.

For most common methods try this benchmark results for librec project:
http://www.librec.net/example.html
There are no results for ML-10M, only ML-1M for rating and ML-100K for ranking. But You can always run chosen algorithm by yourself. There are other projects of this kind: http://www.mymedialite.net/, http://lenskit.org/.

Saying about evaluation of recommender systems this project should be mention:
https://github.com/recommenders/rival
It's dedicated for data preparation (splitting) and evaluation for recommender systems. It was presented at last RecSys'2014 [1]. And also look at this [2]. There are some benchmark results too.

[1] A. Said and A. Bellogín, “RiVal – A Toolkit to Foster Reproducibility in Recommender System Evaluation,” pp. 371–372, 2014.

[2] A. Said and A. Bellogín, “Comparative Recommender System Evaluation : Benchmarking Recommendation Frameworks,” pp. 129–136.


Weighing each label in multi-label classification If, in addition to predicting labels using a multi-label classifier, I'm interested in predicting the weight of each label, what approach should be taken? To give an example, let's say I'm trying to predict movie genres from their plot and for a movie like Terminator the classifier predicts ['Sci-Fi', 'Action'], then is it also possible to estimate the proportion of those genres in that movie, like it's 70% Action and 30% Sci-Fi?

Multi-label classifier does give the probability for each class; is it a good idea to just normalize those probabilities and use them as the weights?

Yes, of course, this technique exists. In XGBoost, for instance, you can change the objective function to the multi:softprob which in specifics does:

  multi:softprob: same as softmax, but output a vector of ndata *
  nclass, which can be   further reshaped to ndata * nclass matrix. The
  result contains predicted probability of each data point belonging to
  each class.

From my memory, if it answers your question I think it rounds up to 1.


How to iterate and modify rows in a dataframe( convert numerical to categorical) I have a pandas dataframe like this

0    15.55
1    15.55
2    15.55
3    15.55
4    20.84
Name: Y1, dtype: float64

I want to convert the values of Y1 to categorical (i.e) if its greater than 18.25, I want it 1 else 0

Can someone please help me on how to do it

This is what i tried so far

for temp in TRAIN_ID1:
    train_ID1.loc[(train_ID1['Y1'] > 18.250000), 'Y1'] = 1
    train_ID1.loc[(train_ID1['Y1'] < 18.250000), 'Y1'] = 0

But im getting an error

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

TypeError: an integer is required

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-118-2cccb791d834> in <module>()
      1 for temp in train_ID1:
----> 2     train_ID1.loc[(train_ID1['Y1'] > 18.250000), 'Y1'] = 1
      3     train_ID1.loc[(train_ID1['Y1'] < 18.250000), 'Y1'] = 0

~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\series.py in __getitem__(self, key)
    621         key = com._apply_if_callable(key, self)
    622         try:
--> 623             result = self.index.get_value(self, key)
    624
    625             if not is_scalar(result):

~\Anaconda3\envs\deeplearning\lib\site-packages\pandas\core\indexes\base.py in get_value(self, series, key)
   2558         try:
   2559             return self._engine.get_value(s, k,
-> 2560                                           tz=getattr(series.dtype, 'tz', None))
   2561         except KeyError as e1:
   2562             if len(self) > 0 and self.inferred_type in ['integer', 'boolean']:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

KeyError: 'Y1'

As the error suggests, you don't have a column called Y1. Hence the error. Here is my suggestion to fix this. Assuming you input data looks like this -

15.55
15.55
15.55
15.55
20.84

Read it in pandas this way -

import pandas as pd

df = pd.read_csv('path/to/file.csv', header=None)

Provide a column name for this -

df.columns = ['Y1']

If you have more columns, just fill the df.columns list accordingly.

Finally, use the pandas best practices as per their latest documentation to assign a new column -

df = df.assign(Y2= (df['Y1'] > 18.250000).astype(int))

Output

print(df)

 Y1  Y2
0  15.55   0
1  15.55   0
2  15.55   0
3  15.55   0
4  20.84   1

Note: Since I don't have full visibility on what you are working on, I have assumed what might be the problems you are facing. If this doesn't work let me know.


Data preparation and machine learning algorithm for click prediction I am new to machine learning. I have a task at hand of predicting click probability given user information like city, state, OS version, OS family, device, browser family, browser version, etc. I have been advised to try logit since logit seems to be what MS and Google are using. I have some questions regarding logistic regression:

Click and non click is a very very unbalanced class and the simple GLM predictions do not look good. How can I make the data work better with the GLM?

All the variables I have are categorical and things like device and city can be numerous. Also the frequency of occurrence of some devices or some cities can be very very low. How can I deal with this distribution of categorical variables?

One of the variables that we get is device ID. This is a very unique feature that can be translated to a user's identity. How can I make use of it in logit, or should it be used in a completely different model based on user identity?

Some suggestions

Remove items appearing too infrequently in the data. That will reduce the dimensionality by several orders of magnitude. If a feature occurs less than say 10 times, it's likely that it's not adding any predictive value, and it may lead to overfitting due to low frequency
Try a Linear SVM instead. They handle large dimensional data very well in terms of not overfitting. They also often have the option to assign relative weights to different classes, which may help address your unbalanced problem above. The sklearn svm (which simply wraps some other packages such as libsvm) has this option.
Don't use the ID column. Producing a model per user will most probably lead to overfitting. Instead, feed in attributes that describe the user that allows the model to generalize over similar users. You could try fitting a separate model per user, but you need a lot of data per user to do this well.
It sounds like you really need to try some feature selection here, to reduce the dimensionality of the problem. But try 1 and 2 first, as they may give you good results sooner (although the end solution may still work better with some good feature selection). Sklearn again has a number of options for feature selection.


Procedure for selecting optimal number of features with Python's Scikit-Learn I have a dataset with 130 features (1000 rows) . I want to select the best features for my classifier. I started with RFE but Its taking too long, i done this:

number_of_columns = 130

for i in range(1, number_of_columns):
    rfe = RFE(model, i)
    fit = rfe.fit(x_train, y_train)
    acc = fit.score(x_test, y_test

Because this took to long, I changed my approach, and I want to see what you think about it, is it good / correct approach.

First I did PCA, and I found out that each column participates with around 1-0.4%, except last 9 columns. Last 9 columns participate with less than 0.00001% so I removed them. Now I have 121 features.

pca = PCA()
fit = pca.fit(x)

Then I split my data into train and test (with 121 features).

Then I used SelectFromModel, and I tested it with 4 different classifiers. Each classifier in SelectFromModel reduced the number of columns. I chosed the number of column that was determined by classifier that gave me the best accuracy:

model = SelectFromModel(clf, prefit=True)
#train_score = clf.score(x_train, y_train)
test_score = clf.score(x_test, y_test)
column_res = model.transform(x_train).shape

End finally I used 'RFE'. I have used number of columns that i get with 'SelectFromModel'.

rfe = RFE(model, number_of_columns)
fit = rfe.fit(x_train, y_train)
acc = fit.score(x_test, y_test)

Is this a good approach, or I did something wrong?

Also, If I got the biggest accuracy in SelectFromModel with one classifier, do I need to use the same classifier in RFE?

You may have a try on Lasso (l1 penalty) which does automatic feature selection by „shrinking“ parameters. This is one of the standard approaches to data with many columns and „not so many“ rows.
sklearn.linear_model.LogisticRegression(penalty=’l1‘,...

See also this post.
Edit:
The book „Introduction to Statistical Learning“ gives a really good overview. Here are the Python code examples from the book. Section 6.6.2 covers the Lasso.


Create new variable in python for each iteration I have two different dataframes like below.

And I need to make a dataframe like the following one.

Can you please help on how to that in python?

Thanks

The operation above is called cross-join. So one way to do it is like this. Let your first dataframe called df1 and the second one called df2.

df1['key'] = 0
df2['key'] = 0
df_joined = df1.merge(df2, how='outer', key='key')
df_joined.drop('key', inplace=True, axis=1)

This should work.


What is the correct way to compute lift in lift charts How is "lift" computed? i was reading about "Gain and lift charts" in data science.

I picked the following example  from https://www.listendata.com/2014/08/excel-template-gain-and-lift-charts.html

I am clear on how the gain values are computed. Not clear about lift values are computed? (last column in table)

Lift is computed by comparing performance with a random selection model. I'll explain with your example below,

assume that we didn't have any statistical/ML model for ranking/scoring the respondents.
In that case assume we did a random ordering of respondents.
A decile (10% of total population) is expected to have 10% of the respondents. In your case, there should've been (approximately) 488 respondents in 2500 cases.
But after ordering the cases by score, you are seeing 44.71% of the cases in first decile against expected 10% (in random/no model case). This gives the gain of 44.71/10 = 4.471.
For next decile, cumulatively you have covered 20% of the cases. You'd expect a random/no model scenario covers 20% of the respondents. But using scores, we covered 80% of them. That gives a cumulative lift of 80/20 = 4.


Deriving new continuous variable out of logistic regression coefficients I have a set of independent variables X and set of values of dependent variable Y. The task at hand is binary classification, i.e. predict whether debtor will default on his debt (1) or not (0). After filtering out statistically insignificant variables and variables that bring about multicollinearity I am left with following summary of logistic regression model:

Accuracy ~0.87
Confusion matrix [[1038 254]
                 [72 1182]]
Parameters Coefficients
intercept  -4.210
A          5.119
B          0.873
C          -1.414
D          3.757

Now, I convert these coefficients into new continuous variable "default_probability" via log odds_ratio, i.e.

import math
e = math.e
power = (-4.210*1) + (A*5.119) + (B*0.873) + (C*-1.414) + (D*3.757)
default_probability = (e**power)/(1+(e**power))enter code here

When I divide my original dataset into quartiles according to this new continuos variable "default_probability", then:

1st quartile contains 65% of defaulted debts (577 out of 884 incidents
2nd quartile contains 23% of defaulted debts (206 out of 884 incidents)
3rd quartile contains 9% of defaulted debts (77 out of 884 incidents)
4th quartile contains 3% of defaulted debts (24 out of 884 incidents)

At the same time:

overall quantity of debtors in 1st quartile - 1145
overall quantity of debtors in 1st quartile - 516
overall quantity of debtors in 1st quartile - 255
overall quantity of debtors in 1st quartile - 3043

I wanted to use "default probability" to surgically remove the most problematic credits by imposing the business-rule "no credit to the 1st quartile", but now I wonder whether it is "surgical" at all (by this rule I will lose (1145 - 577 = 568 "good" clients) and overall is it mathematically/logically correct to derive new continuous variables for the dataset out of the coefficients of logistic regression by the line of reasoning described above?

Deriving a probability value (the continuous variable you are talking about) from the logistic model is a perfectly sound thing to do. The probability value is actually the main output from the model.

Getting from the probability value to a decision rule (e.g. from default probability to credit granting decision) is an another step that will also need to incorporate a number of business decisions concerning risk appetite - i.e. how risky clients are you willing to approve not to miss a potentially large amount of business, what the interest is going to be for different risk scores etc. The tradeoff this refers to is the well known sensitivity-specificity tradeoff which is, apart from the confusion matrix you are using, probably best visualised by the ROC curve.

From the confusion matrix it is also apparent that you were training the model on a balanced sample (default rate around 50%). This is quite unusual in credit risk modeling, usually the default rate is well below that. If that is the case, you will probably need to calibrate the probabilities, for example by fitting a yet another logistic regression model on your probabilities as a single variable (this works best when the probabilities are normally distributed). The calibrated probabilities that this second model will predict will accurately reflect the actual real-world probabilities of default.


Decision tree, how to understand or calculate the probability/confidence of prediction result For example, a drug prediction problem using a decision tree. I trained the decision tree model and would like to predict using new data.

For example:

patient, Attr1, Attr2, Attr3, .., Label
002      90.0   8.0    98.0 ...   ? ===> predict drug A

How can I calculate the confidence or probability of the prediction result of drug A?

What data mining package do you use?

In sklearn, the DecisionTreeClassifier can give you probabilities, but you have to use things like max_depth in order to truncate the tree. The probabilities that it returns is $P=n_A/(n_A+n_B)$, that is, the number of observations of class A that have been "captured" by that leaf over the entire number of observations captured by that leaf (during training). But again, you must prune or truncate your decision tree, because otherwise the decision tree grows until $n=1$ in each leaf and so $P=1$.

That being said, I think you want to use something like a random forest. In a random forest, multiple decision trees are trained, by using different resamples of your data. In the end, probabilities can be calculated by the proportion of decision trees which vote for each class. This I think is a much more robust approach to estimate probabilities than using individual decision trees.

But random forests are not interpretable, so if interpertability is a requirement, use the decision tree like I mentioned. You can use grid search to maximize the ROC AUC score by changing hyperparameters such as maximum depth to find whatever decision tree gives the most reliable probabilities.

EDIT: In case I wasn't clear enough, I think it's awful to use a single decision tree to predict probabilities. I have extended my answer in a blog post.


Warning when plotting confusion matrix with all sample of one class I have two arrays: the first one with all the correct labels (they are all set to zero since each sample belong to the same class) and another one with all the labels predicted by my neural network. What I want do to is plot a simple confusion matrix to show the results. This is my code:
test_labels = [0, 0, 0, ...., 0] #all set to zero
predict_labels = [0, 0, 1, ...., 1]
matrix = confusion_matrix(test_labels, predict_labels)
fig = plt.figure()
colors = ['orange', 'green']
labels = ["Normal", "Anomaly"]
sns.heatmap(matrix, xticklabels=labels, yticklabels=labels, cmap=colors, annot=True, fmt="d")
plt.title("Confusion matrix")
plt.ylabel("Actual")
plt.xlabel("Predicted")
plt.close()

The problem is that when I run this code I get the following warnings:

UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use zero_division parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
RuntimeWarning: invalid value encountered in true_divide   recall =
tps / tps[-1]
UndefinedMetricWarning: No positive samples in y_true, true positive
value should be meaningless   UndefinedMetricWarning)

How can I solve it?

If you insist on giving no positive examples it will have the same problem. As the error message states, the tool wants to compute the recall. The recall metric is given by:
$$recall = \frac{tp}{tp+fn}$$
$tp$ (true positive) is equal with the number of positive cases predicted as positive, which is $0$ in your case.
$fn$ (false negatives) is the number of positive cases predicted as negative, which again is $0$.
What you ask is to compute $\frac{0}{0}$ which is undefined. You can add positive cases also or you should study the tool (I'm not too familiar with Python stack)and perhaps you find an option to not compute recall metrics for you.


What is Compatible Function Approximation theorem in reinforcement learning? I am following David Silver's RL course. In the policy gradient section, I found this slide that I would like have an explanation of.

What are these two conditions?

What is the logic behind the first derivative equality? Is it just that we assume these two derivatives should be equal since there should be some kind of connection with direction of value function approximation gradient and our policy like-hood?

Then what is the epsilon value? What is this mean square value? Q value with w parameters means the Q values function new approximation. And the other Q value is the value got by following our policy right? So is this OFF policy value approximation or ON policy? I think this is on policy since normally in the Q value update we take the max.

There are lots of questions but I will try to answer in a way that might clear things up for you and also give you some guidance. Please note that the proofs for your questions involve lots of math operations so instead I will provide you with references.

Your main reference is the paper from Sutton PG Methods with Function Approximation. I highly recommend you to read the paper a couple of times (or even more!) and do some search in the relevant literature when you will be familiar with the main objectives, notation and math around the general approach of the methods. PG Methods are not easy to get a grasp of them mainly because of their sampling nature, notation and discrete/continuous math involved.

PG Methods satisfy (or at least should) the PG theorem (eq. 2 from the paper). An interesting approach would be to substitute the true $Q^\pi (s,a)$ by some approximate function ($f_w$ in the paper, $Q_w$ in your question). Now, we are wondering what conditions should be satisfied by that proposed approximation in order to satisfy the PG Theorem.

The first thing you notice is that a natural choice for updating the parameters $w$ is to update them towards the direction that minimizes the mean squared error of the exact $Q^\pi (s,a)$ with the function approximation. In your question this is the $\epsilon$. In such a scenario the exact $Q^\pi (s,a)$ is estimated using unbiased samples such as $r_t$.
This is explained in detail in Part 2 of the paper.

For the PG theorem to hold (proof consists of the 3 lines before Part 3) the grad of your approximate function should satisfy the compatibility condition. To sum up we started from PG theorem and we found a suitable family of function approximators for our action-value function that the PG theorem holds. In Part 3 you can see an example of a compatible function. From this of course you can use even non-linear approximators such as NNs.

A clarification on on/off-policy: The David Silver's slide that you posted here has to do with theoretical guarantees and has nothing to do with an actual RL algorithm. By the way Q-learning algorithm in which you use the $max_{a'}{Q(s',a')}$ is OFF-policy as you you don't actually use for updates the ongoing policy.

Hope this helps!


Calculate the similarity between pairs of time series data I have 5 pieces of time series data. It is the weekly sales of 5 different stores (A,B,C,D,E). There are no missing values. A quick visual inspection shows that these 5 pieces of time series data have similar trend & seasonality.
I would like to calculate/quantify how similar Store B,C,D,E is to Store A respectively.
I know how to calculate the simple cosine distance and Euclidean distance, and I have experience dealing with time series data (e.g. ARIMA, Prophet), but never had to deal with finding (i.e. quantifying) the similarity / distance between time series data.
Some google search suggests that "dynamic time warping" could be the method to go. I have not yet heard of this algorithm before, so not sure if this is indeed the right direction.
Additionally, can I still perform cosine distance and Euclidean distance on time series data? Is that doable?
How do I approach this issue? Any insights are appreciated.

Euclidean distance metric is unsuitable for time series...In short, it is invariant to time shifts, ignoring the time dimension of the data. If two time series are highly correlated, but one is shifted by even one time step, Euclidean distance would erroneously measure them as further apart. You might use it to compare your time series if you are extracting their features. Don't need to go and use directly PCA. You can use simple descriptive stats such as mean, min, max, kurtosis, etc...Then you can compare it with Euclidean.
I advise you to give it a try with DTW. ;)
How does it work?
http://alexminnaar.com/2014/04/16/Time-Series-Classification-and-Clustering-with-Python.html   (you can check that blog post, it explains it nicely)
Good luck to you!


Matplotlib -- TypeError: 'module' object is not callable I am new into ML. As per video tutorials when i try to execute the following lines i get an error.
However the instructor get successful execution.

Error is :

i have google about it but didnt get any clue. May be there is change in library ?
How can i solve this Error.

Instead of
import matplotlib as plt

use
import matplotlib.pyplot as plt


How to classify support call texts? I have a spreadsheet with thousands of records regarding support requests.
Case number, Issue description, etc.

Our goal is to classify these records in many categories in order to assign them the right priority.

Example:

Customer can't use pickup feature.
Customer can't dial 911 or Long Distance numbers.

For item number 1, I have decided to use a category called Best Effort and for item 2, an Urgent category.

Customer can't use pickup feature, BEST_EFFORT
Customer can't dial 911 or Long Distance numbers, URGENT

I'm planning to setup a dictionary of words.

best_effort = ['pickup','record','conference']
urgent = ['system is down','911', 'can't dial emergency','call center is down']

My goal is to use TFIDF and then cosine similarity to find best match and category. Does it makes sense? Any better recommendation to classify this type of information?

Rather than use an external dictionary of keywords that are indicative of target class, you may want to take your raw data (or rather a random subset of it), then hand-label your instances (assign each row a label, BEST_EFFORT or URGENT). This becomes your training data - each row of data can be transformed into a bag-of-words vector indicating the presence/absence of the word in that particular text. You can train a classifier on this data, for example a naive bayes classifier, which can then be tested on the held out unseen test data. The advantages of the  proposed approach are: (1) automated computation of features vs. hand created dictionary; (2) probabilistic/weighted indicators of class vs. binary dictionary indicators.


Convert a list of lists into a Pandas Dataframe I am trying to convert a list of lists which looks like the following into a Pandas Dataframe

[['New York Yankees ', '"Acevedo Juan"  ', 900000, ' Pitcher\n'],
['New York Yankees ', '"Anderson Jason"', 300000, ' Pitcher\n'],
['New York Yankees ', '"Clemens Roger" ', 10100000, ' Pitcher\n'],
['New York Yankees ', '"Contreras Jose"', 5500000, ' Pitcher\n']]

I am basically trying to convert each item in the array into a pandas data frame which has four columns. What would be the best approach to this as pd.Dataframe does not quite give me what I am looking for.

import pandas as pd

data = [['New York Yankees', 'Acevedo Juan', 900000, 'Pitcher'],
        ['New York Yankees', 'Anderson Jason', 300000, 'Pitcher'],
        ['New York Yankees', 'Clemens Roger', 10100000, 'Pitcher'],
        ['New York Yankees', 'Contreras Jose', 5500000, 'Pitcher']]

df = pd.DataFrame.from_records(data)


semi supervised learning doubt only classify points with confidence above threshold I currently have a dataset with approximately 5% labelled points and 95% unlabelled. I would like to label some of the unlabelled points only if I am very confident and leave the rest NaN. Personally I would like to use a random forest but I am not sure if that is possible - I assume I am going to have to use some generative model?

One of the reasons I would like to do this is because the known points do not contain all the labels therefore I would like to classify as many of the unknown points as possible before using unsupervised learning on the rest.

Is there a library I could use?

Most sklearn categorizers have probability outputs.

CLF.predict_proba

from that you can decide the threshold.


Is this a correct way improving a statistical model? I am building a fair amount of statistical models - text classifiers and sequence taggers. The statistical models are linear in features - Logistic Regression and linear chain first order CRF.

The models undergo several stages starting from a rough, initial version all the way to a mature production model. Given a specification and some dataset (often with incomplete annotation, noisy labels and features), I build the first model from a small, hand labeled dataset. This dataset is then grown iteratively by adding more labeled examples (e.g. using crowdsourcing).

Then comes the test phase. The statistical model is actively evaluated within a larger system by testers. I, as a developer, start getting the failure reports on their input. I am supposed to fix the model, update it in the system and deploy it.

Every individual error report is fairly small to manually go through and fix every misclassification. Obviously it is incorrect just to simply add the failure cases with correct labels to the training set. You never know if the model learned that new pattern.
I do this by comparing the failure cases with a vast unlabeled dataset of examples (real usage data but no labels) and extract similar examples. I label those similar examples, add to the training set and re-train the model. Then I check if the new model fixes the originally reported issues.

My question is: Is this a correct approach to build and continuously improve the statistical models? May be there are some hidden caveats that will bias my model in some sense?

No offense, but let me guess, you work for a large company where there is no person with a full understanding of the process overseeing this? I say this because it sounds as if the testers want you to manually and intentionally over-fit your models. There is no such thing as a perfectly accurate model.

Yes, I would say that there is bias in your model building. New model input should be a random selection not by finding and labeling similar instances to ones that were misclassified.

Your production model will degrade over time (the distribution in the population has changed). You should have a sense of a threshold, an error rate at which is no longer exceptable and you will need to retrain. To retrain, you could add a random selection of new labeled data to your existing data, then train, validate, select model, deploy, and monitor.

You may "fix" an error report with a new retraining, but it is an unrealistic expectation that this will also cover all previous error reports. Your model needs to generalize. See here:

https://en.wikipedia.org/wiki/Test_set

Look at the image at the bottom. In essence you are being asked to create the orange model were you want to create the green model, which generalizes better to unseen data.


What does the term "proportional to" mean in Bayes Equation? I don't have a background in maths so I sometimes get confused by basic definitions. Let's for instance consider Bayes Theorem in Bayes Data Analysis:

$P(\theta|\textbf{y}) \propto P(\theta) P(\textbf{y}|\theta)$

How should this equation be understood? What does it mean that the RHS is proportional to LHS?

It means that $P(\theta | y ) = kP(\theta) P(y | \theta)$, where $k$ is a constant that does not depend on $\theta$.

In fact, the Bayes Theorem states $P(\theta | y ) = \frac{P(\theta) P(y | \theta)}{P(y)}$. This proportion has to be added in order that $\sum_{\theta}P(\theta | y) = 1$ is satisfied (that is, $P(\theta | y)$ has to be a probability).


Python library to implement Hidden Markov Models What stable Python library can I use to implement Hidden Markov Models? I need it to be reasonably well documented, because I've never really used this model before.

Alternatively, is there a more direct approach to performing a time-series analysis on a data-set using HMM?

For another alternative approach, you can take a look at the PyMC library.
There is a good gist created by Fonnesbeck which walks you through the HMM creation.
And if you become really eager about the PyMC, there is an awesome open-source book about Bayesian Modeling. It does not explicitly describe Hidden Markov Processes, but it gives a very good tutorial on the library itself with plenty of examples.


What are non-machine-learning methods called? For my masters, I'm comparing different approaches to solve a problem (like image segmentation).

I'm comparing machine learning and deep learning approaches to «classical» algorithms (like simple or adaptative threshold).

How would you call these «classical» or «standard » approaches? Like algorithms that don't rely on data but are «hand-designed».
I can't seem to find a correct term.

Machine learning models work on principle of probabilistic approach where you try to fit the function to map input with output. There are no hard bound rules that classify the output based on input.

Deterministic approach like if else are categorized as  non machine learning algorithms where every time it follows the same rules to evaluate the results.


In a neural network, is it possible to gradient descent with more than one input? I went through a few tutorials, examples recently, and all (not sure if just for demonstration purposes) done gradient descent for one input. To get a deep understanding of backpropagation, I wrote a program to do backpropagation just do understand it more deeply. In linear/logistic regression, it makes sense to do gradient descent on the average of the costs through multiple inputs, and outputs, because the there's just one layer of weights, and the inputs directly affect the outputs.
In case of neural networks, we get back a layer of activations (outputs), and we have the expected outputs with matching shape, so we're getting the costs, by subtracting the expected output, with our actual output, and we're propagating this back with the chain rule. But this way we have to compare our costs with the activations of the neurons, which are unique, and dependent on the inputs. So even if we would take a bunch of inputs, get their costs, and average the costs, how could we decide which neuron activation layer should it be compared to?

As to what I understand from your question, yes you can have multiple features (variables/input) and use gradient descent to minimize the loss function which has the input variables. Here is an article that gives you a basic vector calculus idea as to what exactly does a gradient is and what it does in the algorithm!
https://towardsdatascience.com/wondering-why-do-you-subtract-gradient-in-a-gradient-descent-algorithm-9b5aabdf8150


How to train NLTK Sequence labeling algorithm for using custom labels/Train set? I am trying to recognize and classify the entity types based on the IOB/Sequence labeling.

I was able to use nltk.ne_chunk() which is already trained to recognize named entities using their train set.

I was wondering if there is any way to train the algorithm using my on training set and labels using training data like the following,

(
send        O
sms         B-TASK
to
8714349616  B-MOB
how         B-MSG
are         I-MSG
you         I-MSG
)

(
sms        B-TASK
how        B-MSG
are        I-MSG
you        I-MSG
to         O
8714349616 B-MOB
)
etc..

is it possible using NLTK ? Any examples or Tutorials ?

Have a look on conditional random field. Conditional random fields are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data.
CRF++ is a simple, customizable, and open source implementation of Conditional Random Fields (CRFs)


Build neural network to calculate points for a board game I want to be able to calculate how much points each player has for a board game by taking a picture on the board game.
I do this as hobby, not for university or professional purposes. I will use it while playing with my friends and also I hope this will help me build some knowledge about ML.
I plan to do that with supervised learning. Take many pictures of the game and tell the machine what is the correct output.
The game scoring is not something straight-forward. Players may control different territories. A territory is controller if a player has units there even if there is enemy buildings. The game is Scythe. From my previous knowledge about ML I know that small changes in the image (for example the angle) from the images you gave to the machine to learn may bug it to recognise them.
Questions (I do not want someone to tell me how to do it but rather which approach is better so I can read more about it):

Is supervised learning the best option in this case?
I plan to build many micro neural-networks for the different cases - recognise who controls a territory and which territory (which territory seems very hard); separate network to recognise if there are resources on some players territory (this seems very hard, as the combination of resources might be from 0 to many; shall I just put different combinations of resources (+numbers of them) and take many pictures?). There are lots of combinations. I cannot take a pic on every of them.

Theoretically, you could take many pictures and map these pictures to the score of each player. However, I would advise against it. First, you would need plenty of pictures and it might be infeasible to cover all possible game scenarios. Second, game scoring is discrete whereas a traditional neural net would approach it as a regression. This means that your outputs will likely be close but rarely perfectly on the money (and you will need to round up). You could however try to predict who is currently winning and turn this into a classification problem.
Now back to predicting points. Since there already exists a complex point-scoring system based on the board, it would be rather silly to try to let a neural network figure it out on its own. Instead, I would suggest that you implement the point-scoring system and have the neural network translate a picture of the board into a data representation of that board. Your system would do something like this:
def scoring_system(board):
    // Insert logic here
    return points

def your_neural_network(image):
    // Your code
    return board

points = scoring_system(your_neural_network(image))

So, instead of having this be a regression problem, you have multiple problems to solve:

object detection (where are the pieces)
object classification (what type of piece is it?)
image segmentation (where is the board, what territories are there)
Etc.

Now some of these bits could be solved with traditional computer vision, but neural networks can be well-equipped as well.


Feature extraction from resume using Python without rule based logic I am working on a resume parser project. Currently, I am using rule-based regex to extract features like University, Experience, Large Companies, etc.

So basically I have a set of universities' names in a CSV, and if the resume contains one of them then I am extracting that as University Name.  In the same way I have a list of Large Companies in CSV and if the resume contains any of them then I flag it as Yes.

So these are rule-based logic and can never be fool-proof considering different countries have different resume formats.
Is there any other way of doing it to improve the accuracy and make it a global solution?

Yes there is. Look at this paper of general information extraction

Where you can construct features and generalise around extracting, without any hard coded rules.


Machine Learning - Precision and Recall - differences in interpretation and preferring one over other I have summarising this from lot of blogs about Precision and Recall.
Precision is:

Proportion of actual positives that classifier has predicted as positive.

meaning out of the sample identified as positives by classifier as positive, how many are actually positive?
and Recall is:

Proportion of actual positives were predicted as positive correctly.

meaning out of the ground truth positives, how many were identified correctly by the classifier as positive?

That sounded very confusing to me. I couldn't interpret difference between both of them and relate each to real examples. some very small questions about interpretation I have are:

if avoiding false-positives matter the most to me, i should be measuring precision; And if avoiding false-negatives matters the most to me, i should be measuring recall. Is my understanding correct?
Suppose, I am predicting if a patient should be given a vaccine, that when given to healthy person is catastrophic and hence should only be given to an affected person; and I can't afford giving vaccine to healthy people. assuming positive stands for should-give-vaccine and negative is should-not-give-vaccine, should I be measuring Precision? or Recall of my classifier?
Suppose, I am predicting if an email is spam(+ve) or non-spam(-ve). and I can't afford a spam email being classified as non-spam, meaning can't afford false-negatives, should I be measuring Precision? or Recall of my classifier?
What does it mean to have high precision(> 0.95) and low recall(< 0.05)? And what does it mean to have low precision(> 0.95) and high recall(< 0.05)?

Put simply, in what kind of cases is to preferable or good choice to use Precision over Recall as metric and vice versa. I get the definition and I can't relate it to real examples to answer when one is preferable over other, so I would really like some clarification.

To make sure everything is clear let me quickly summarize what we are talking about. precision and recall are evaluation measures for binary classification, in which every instance has a ground truth class (also called gold standard class, I'll call it 'gold') and a predicted class, both being either positive or negative (note that it's important to clearly define which one is the positive one). Therefore there are four possibilities for every instance:

gold positive and predicted positive -> TP
gold positive and predicted negative -> FN (also called type II errors)
gold negative and predicted positive -> FP (also called type I errors)
gold negative and predicted negative -> TN

$$Precision=\frac{TP}{TP+FP}\ \ \ Recall=\frac{TP}{TP+FN}$$
In case it helps, I think a figure such as the one on the Wikipedia Precision and Recall page summarizes these concepts quite well.
About your questions:

if avoiding false-positives matter the most to me, i should be measuring precision; And if avoiding false-negatives matters the most to me, i should be measuring recall. Is my understanding correct?

Correct.

Suppose, I am predicting if a patient should be given a vaccine, that when given to healthy person is catastrophic and hence should only be given to an affected person; and I can't afford giving vaccine to healthy people. assuming positive stands for should-give-vaccine and negative is should-not-give-vaccine, should I be measuring Precision? or Recall of my classifier?

Here one wants to avoid giving the vaccine to somebody who doesn't need it, i.e. we need to avoid predicting a positive for a gold negative instance. Since we want to avoid FP errors at all cost, we must have a very high precision -> precision should be used.

Suppose, I am predicting if an email is spam(+ve) or non-spam(-ve). and I can't afford a spam email being classified as non-spam, meaning can't afford false-negatives, should I be measuring Precision? or Recall of my classifier?

We want to avoid false negative ->  recall should be used.
Note: the choice of the positive class is important, here spam = positive. This is the standard way, but sometimes people confuse "positive" with a positive outcome, i.e. mentally associate positive with non-spam.

What does it mean to have high precision(> 0.95) and low recall(< 0.05)? And what does it mean to have low precision(> 0.95) and high recall(< 0.05)?

Let's say you're a classifier in charge of labeling a set of pictures based on whether they contain a dog (positive) or not (negative). You see that some pictures clearly contain a dog so you label them as positive, and some clearly don't so you label them as negative. Now let's assume that for a large majority of pictures you are not sure: maybe the picture is too dark, blurry, there's an animal but it is masked by another object, etc. For these uncertain cases you have two possible strategies:

Label them as negative, in other words favor precision. Best case scenario, most of them turn out to be negative so you will get both high precision and high recall. But if most of these uncertain cases turn out to be actually positive, then you have a lot of FN errors: your recall will be very low, but your precision will still be very high since you are sure that all/most of the ones you labeled as positive are actually positive.
Label them as positive, in other words favor recall. Now in the best case scenario most of them turn out to be positive, so high precision and high recall. But if most of the uncertain cases turn out to be actually negative, then you have a lot of FP errors: your precision will be very low, but your recall will still be very high since you're sure that all/most the true positive are labeled as positive.

Side note: it's not really relevant to your question but the example of spam is not very realistic for a case where high recall is important. Typically high recall is important in tasks where the goal is to find all the potential positive cases: for instance a police investigation to find everybody susceptible of being at a certain place at a certain time. Here FP errors don't matter since detectives are going to check afterwards but FN errors could cause missing a potential suspect.


Generate text using user-supplied keywords I've got a use case where I need to generate sentences based on a set of user supplied keywords. Here is an example of what I need:

User input:

End-User: Data Scientists

Region: Middle East

Country: UAE

Solution: BigPanda

Application: machine learning

Benefits: lower costs and runtime

Output (Curly-Brackets are just there to highlight):

Learn how {data scientists} in the {Middle East} such as in the {UAE} are using {BigPanda} to streamline their {machine learning} processes to {lower costs and runtime}.

So the model needs to use the keywords given by the user and generate similar sentences. I also have a dataset of about 2000 of such sentences, which may come in handy.

One way I thought this could be possible is by using the GPT-2 model and perhaps finetuning it with the dataset, but I haven't been able to figure out how I would actually go about using it for something like this.

Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.


LSTM Multivariate time series forecasting with multiple inputs for each time step I want to predict an output variable for the next day, for each of the users in my dataset. I was thinking of using LSTMs for achieving this.

The dataset

The dataset I am using has multiple inputs for each time step and it is dependent on the value from one of the inputs.

Each user_id has multiple features for each day and gives an output in the range of [1, 10]. Not all users have data for each day.

-----------------------------------------------------------------
|     day    |   user_id   | feature_1 |   feature_n  |  output |
-----------------------------------------------------------------
| 2020-02-11 |      1      |     20    |       9      |    8    |
| 2020-02-11 |      2      |     15    |       8      |    4    |
| 2020-02-12 |      1      |     10    |       2      |    6    |
| 2020-02-13 |      2      |     16    |       9      |    5    |
| 2020-02-13 |      3      |     19    |       1      |    7    |
| 2020-02-13 |      1      |     14    |       4      |    9    |

I want to forecast the output for each of the users for the next day.

I was thinking to split the dataset in multiple datasets for each of the users and use only the data for the user to predict the output, the problem is that my dataset isn't big.

Is there any way to use the whole dataset and after training the model to get as output the output value for each of the users for the next day?

The only possible way is to create a dataset separately for each user; in the end, if you have 10 users, then you would have 10 different unrelated time series in the same .csv, since each user can exhibit specific characteristics. Evidently we cannot expect to throw 10 different unrelated time series into an LSTM and expect decent results.

The solution to this is to extract the users with the most entries (you could start with the user with the most entries) and apply in the first instance a simpler algorithm, not necessarily an ml-based one but a statistical one, such as VAR.


Reinforcement learning: easily learnable state representation I have created a simple OpenAI Gym environment, which consists of:

A continuous 2D world with x and y in range [0.0, 1.0]
A rabbit which slowly moves randomly in the world with a constant speed
A 'wolf' controlled by the agent. The wolf moves at a constant speed
The actions are [turn left by a constant angle, turn right by a constant angle, do nothing (continue straight)]
The state is [agent_x, agent_y, agent_bearing, rabbit_x, rabbit_y, rabbit_bearning]. Bearings are in radians [0.0, 2*pi]. All values are floating point numbers.
The reward is 30 for catching the rabbit (catching means the agent getting sufficiently close to the rabbit). -0.1 for each timestep without catching the rabbit.
Maximum timesteps 260

I am having trouble solving this environment. Agents trained in it get scores only slightly better than a random agent even after long training sessions. I have tried Deep Q-learning (with experience replay, target network) REINFORCE (with and without baseline) and PPO.

Conceptually, the problem is quite simple. The agent just needs to learn to turn towards the rabbit. However, it occurs to me that the state representation might make the problem more difficult, since only one of the six variables is directly under the agent's control, and three of them (the rabbit state) are completely random.

Does state representation generally affect how difficult a problem is? Is this a poor state representation? Are there rules of thumb for how to design states? Would it help to reformulating the state, eg. to [distance_from_agent_to_rabbit, angle_between_agent_and_rabbit]?

Environment source code here.

Does state representation generally affect how difficult a problem is?

Yes, due to it being easier or harder for a neural network to learn the relationships between input features and the target policy or value function.

  Is this a poor state representation?

No, it should be fine, provided you are interested to see a RL learn to convert from cartesian coordinate space to solving this problem, you should be able to use the state directly as ML features.

It could be made into better features for a NN in two ways:

Scaling to fit in ranges with zero mean and limited max absolute values.
It could be engineered so that features include domain knowledge of the problem to be solved.

The first part is optional for you, but I think that the bearing values benefit a little from being scaled and centred. In a more general case, even if the state description was complete, you may still want a step that re-scales it.

The second part is tricky - yes you could find better features, but part of the challenge is to create an agent that learns to do this itself. In this case, you may want to gain experience training agents where it is less easy to engineer "golden" features, and focus on RL methods.

  Are there rules of thumb for how to design states?

Conceptually, the problem of state representation can break down into three parts:

Observations. Raw observations are not always direct candidates for a state representation. In a toy problem, often this is ignored and you feed data from a simulation that looks useful as a state. In the real world you can be limited by what is detectable.
State description. Unless you want to explore POMDPs then you usually want the state description to possess the Markov property. This might already mean processing observations into something else - e.g. using history of last 3 observations, keeping running totals or calculating differences.
Feature vector. Once you have decided to use function approximation for calculating action values or a policy function, then your inputs need to conform to how the function approximation works. For most function approximators, this means numerical values. For neural networks, it means scaling inputs to fit relatively small ranges. There is also the question of feature engineering using domain knowledge of the problem.

There is some overlap between these design steps, the distinctions are somewhat artificial. When you are designing a test problem like yours, you may find it simple to combine all three steps into a single observation = state = feature. However, in real-world problems, each of the steps may require some consideration.

You should also consider the likely nature/shape of the function that you will be approximating. Action-value methods like Q learning need to approximate their value functions, whilst policy gradient methods like REINFORCE and PPO need to approximate policies. Sometimes the map between input features and the target function is simple, and if you are lucky some intuition can lead you to figuring that out. This is also a big driver when choosing between DQN or PPO for example - what seems easier to figure out, the correct action, or the value of a state/action pair?

  Would it help to reformulating the state, eg. to [distance_from_agent_to_rabbit, angle_between_agent_and_rabbit]?

Maybe. The angle feature that would seem to most help is the difference in angle between the wolf's bearing and the vector between the wolf and rabbit. Then the correct action would map very clearly from that to steer the wolf towards the rabbit - in most cases a negative value means steer right and a positive value means steer left.

However, if you do this, you will in some ways have changed the nature of the problem to be solved. You have to ask, are you interested in applying your domain knowledge of the problem to help the agent, or are you interested to see if the agent can figure out an internal representation that discovers this relationship? For a toy problem, you may want to deliberately make something harder to learn.

  I am having trouble solving this environment. Agents trained in it get scores only slightly better than a random agent even after long training sessions. I have tried Deep Q-learning (with experience replay, target network) REINFORCE (with and without baseline) and PPO.

As discussed in comments, the biggest factor towards this actually turned out to be a bug in your environment code, where the agent could end up with a bearing value out of range. The environment still worked, because you are using trig functions to calculate movement, but this causes even larger range of bearing values plus makes states that are identical appear different to the agent making things even harder to learn.

An experiment

I managed to solve this environment using a simple single-step DQN-based agent, and had time to experiment with some different input features to the NN to demonstrate my points above. In each case, I used exactly the same hyperparameters for RL and NN (expect in the last case I had to change the size of the NN's input layer).

I counted the number of training episodes required (including ~80 episodes of purely random behaviour to start experience replay) for 100 completed training runs, and tried some different state feature representations. I counted the environment as "solved" when 100 test runs using the agent acting greedily scored an average return of 20 or more. I did not count the test runs towards the number of training episodes.

I got these results:

Unaltered state 629.99 +-23.66 episodes, but failed in 28 out of 100
Scaled state 577.78 +-19.41 episodes, no failures
Engineered state 153.84 +-3.26 episodes, no failures

There is not quite a significant difference in 100 trials between number of episodes for unscaled and scaled state. However, the high number of failures (gave up after 1500 episodes of training) for unscaled features is a significant difference.

For the scaled state variation, The scaling for the cartesian coordinates was $f_i = 2*(s_i-0.5)$ and the scaling for the bearings was $f_i = 0.5 * (s_i-\pi)$

The engineered state used [distance between wolf and rabbit, angle between vector to rabbit and wolf's bearing, rabbit's bearing] scaled as above, and the performance was radically better.

Another consideration

As an aside, it is worth mentioning episode timeout and "done" flag.

You need to carefully consider what it means for the episode to time out. There is a difference between this being part of the challenge (to succeed within a time limit) and being for training convenience (to avoid wasting time learning from overlong or stuck episodes). Sometimes it is a big difference:

For training convenience. The "done" flag is an annoyance here, you want to avoid claiming the episode is really over to the agent, as it will falsely learn that some states, sometimes, end the episode - these states may then even be desirable to the agent as they seem like they stop the flow of negative reward. You don't want to store that white lie in the experience replay table - a simple work-around is to have your agent stop prematurely at least 1 step before the environment times out.
Part of the challenge. If the environment can really time out, and not randomly stop at any point, then in order to preserve the Markov property, you must include a representation of the time in the state - it can be time so far or remaining time. Otherwise you have turned the problem into a POMDP, and added complications for calculating value functions.


How to predict numerical value "y"? if I have datasets having different number of parameters I have three excel sheets for training (all numerical values).

1st excel sheets have parameters a, b, c and output as y
2nd excel sheet have parameters a, b, c, d, e and output as y
3rd excel sheet have parameters a, b, c, d and output as y

I have to predict value y if values of only a, b, c are given.

Should I apply linear regression?

If you have to predict $y$ only given $a,b,c$, then as a first look just discard $d,e$ etc from other data sets*. Any kind of regression might be useful, you will have to explore that - you might as well start with linear regression as it is simplest.

In general, if you are not sure about which algorithm to apply for best results, you should set up an experiment to try multiple models:

Decide on what "best" means by choosing a test metric (mean square error is a common choice for regression problems).
Split your data into training, cross-validation and test sets.
Train different types of model on the training set:

Tune any hyper-parameters (of more complex models) by taking the best results from the cross-validation set.
Pick your best cross-validation result as your guess at the best model.

Finally get an unbiased estimate of its performance by using your test set.

* There might be ways to use $d,e$ values to refine some models, but if you are at the stage of asking whether or not to use linear regression, this is not worth exploring initially.


Extracting amount from free text I want to extract various amounts and tenure of contracts from different contract documents that we have.
For example: Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as agreement fee.
Expected output : 3 Months, $3000
Please note that this is just an example but the sequence, format, currency and tenure is not fixed in the actual problem.

Named Entity Recognition (NER) models should be able to identify money amounts.

Other NLP techniques such as Dependency Parsing or Constituency Parsing can be used to identify the subject of the sentence - or the person the amount is referred to.

For the months amount, I think once you have other informations that's something you could extract with a "hard coded" script.


RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed I am referring this previously asked question in stack-overflow which remains unsolved till now.

I am facing same problem with pytorch when I am solving multiclass classification.

Here is my exception

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at c:\programdata\miniconda3\conda-bld\pytorch_1532501501514\work\aten\src\thnn\generic/ClassNLLCriterion.c:93

I have added whole code of mine with the dataset I am trying with but still get stuck on that

# -*- coding: utf-8 -*-
"""
Created on Thu Jul 11 22:41:07 2019

@author: Ananda Mohon Ghosh
"""

import pandas as pd
import numpy as np
import os
import torch
from torch import nn
from torch.autograd import Variable
import torch.utils.data as utils
from sklearn import preprocessing
from sklearn.metrics import mean_squared_error, mean_absolute_error
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torchvision import datasets, transforms, models
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.classificaion =nn.Sequential(
            nn.Linear(16, 12),
            nn.LogSoftmax(dim=1)
        )

    def forward(self, x):
        clsf = self.classificaion(x)
        return clsf

def oneHotCode(y):
    y = y-min(y)
    nOfClasses = len(set(y))
    labels = y.reshape(len(y), 1)
    return (labels == torch.arange(nOfClasses).reshape(1, nOfClasses)).long()

def batchData(xBatch, yBatch):
    return np.concat((xBatch, yBatch),axis=1)

def findMinMax(df):
    #find max-min
    minimum = 999999
    maximum =  -999999

    for di in  df:
        tMin = min(di)
        tMax = max(di)
        if(tMin<minimum):
            minimum = tMin
        if(tMax>maximum):
            maximum = tMax
    return maximum, minimum

dataFrame = pd.read_csv('data/features_test_encoded16_ds1_batch_sig.csv')
dataFrameValues = dataFrame.values

availabelCuda = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
print('Availabel CUDA ', availabelCuda)

#variables
num_epochs = 10
batch_size = 128
learning_rate = 1e-3
weight_decay = 1e-5
momentum = 0.9

clsf = Net()
model = clsf.to(availabelCuda)

lossFunc= nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate,  weight_decay=weight_decay)

observations =  dataFrameValues.shape[0]
featureSetLength = dataFrameValues.shape[1]
featureSet = dataFrameValues[0:len(dataFrameValues), 0:featureSetLength-1 ]
maximum, minimum =findMinMax(featureSet)

featureSet = (featureSet-minimum)/(maximum-minimum)

labelSet = dataFrameValues[0:len(dataFrameValues), featureSetLength-1 ]

Xtrain = featureSet[0:int(observations*0.7), :]
Xtest = featureSet[int(observations*0.7):observations, :]
Ytrain = labelSet[0:int(observations*0.7)]
Ytest = labelSet[int(observations*0.7):observations]

X = torch.Tensor(Xtrain)
Y = torch.Tensor(Ytrain.reshape(-1,1))
trainDataset = utils.TensorDataset(X, Y)

dataloaderTrain = utils.DataLoader(trainDataset, batch_size=batch_size, shuffle=False)

print('Model running on ', availabelCuda)

for epoch in range(num_epochs):
    for xX, yY in dataloaderTrain:
        optimizer.zero_grad()
        dataX = Variable(xX).to(availabelCuda)
        dataY = Variable(yY.long()).to(availabelCuda)

        outclass = clsf(dataX)
        print(outclass.type(), dataY.view(-1).type())

        loss = lossFunc(outclass, dataY.view(-1))
        optimizer.zero_grad()               # clear gradients for this training step
        loss.backward()                     # backpropagation, compute gradients
        optimizer.step()
    print('Epoch: ', epoch, '| train loss: %.4f' % loss.data.numpy()) #CPU

and here is the link of my dataset. Actually, I was hoping that I have problem with data normalization but eventually I couldn't solve it and still have this issue. Any kind of help is appreciated. Thanks.

Okay after 4 days I got my solution eventually. I figured it out that my classes are started from 1 to 12, so, in total I had 12 classes.

The problem was appearing when I was about to calculate the predicted class vs actual class loss with the loss function. Instead of having start from 1 that function always from class 0 instead of 1 and continuously look for 0 to 12 which indicated it do have 13 classes in total which generates an exception.

So how do I solved that? I just docked 1 form each labels and shifted it from 0 to 11 and it went fine. That's how I solved the issue. Mention, I got couple of solution at PyTorch forum regarding the same issue and those wasn't helpful at all, literally!


How can I fixed the filter and Kernel Size of a CNN? I have created 4 x 4 2d images from a signal. Now, I want to feed this data to a Convolutional neural Network. How I can choose the nubmber of filter, Kernel Size for this small, shape and size of dataset (4 x 4, 320 images). Should I use maxpooling layer ?
Note:
I have already implemented a model for this dataset and accuracy is 78.125%.  The model is,

Since the images are just 4x4 in size, you can do the following :

You can resize the image to a much larger dimension like 28x28 and then use sharpen or histogram equalization to bring out the contrast. Then use a 3x3x16, 3x3x 32 kernel arrays in 2 convolutional layers. The rest is fully connected.

The images can be input as 4x4 images but use a shorter neural net. One convolutional layer with padding and 3x3x 32 shape would do.

Use something similar to google nets, inception module, in which output from multiple convolutions are combined.


Help with DDP Mining algorithm for Effective Classification of data sets from 2 groups I'm trying to implement the DDPmine algorithm from this article as part of some project, and I do not understand where in the algorithm we use the Class Label of each transaction?
We have transactions from 2 different groups spouse group has a class label "0" and group b has the class label "1" and we want to find the Discriminative Patterns that are frequent in each group but not on the 2 groups combined but in which part of the algorithm we consider this?
what I'm missing here?

I didn't read the paper in detail but I can see that the class label is used in Algorithm 2 "branch and bound" p. 3: it's not very clear because it's only used through Information Gain (IG). I assume that the IG is calculated between the feature and the class, in order to find the "most discriminative feature", i.e. the one which gives the most information about the class.


Maximum Likelihood with Gradient Descent or Coordinate Descent blows up Context
The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are
$$
\widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \mu)^2
$$
These can be found (for example) by taking derivatives of the average log-likelihood
$$
\frac{1}{n}\sum_{i=1}^n \log p(x_i) = -\frac{1}{2}\log(2\pi) - \frac{1}{2n\sigma^2}\sum^n_{i=1} (x^{(i)} - \mu)^2 - \log \sigma
$$
Question: What if I want to use a gradient-based method?
Yes, I know I can just use the estimators found above. However, I want to find such estimators using a gradient-based method such as coordinate descent or gradient descent. These are the gradients with respect to $\mu$ and with respect to $\sigma$ (which you can set equal to zero to find the estimators above)
$$
\begin{align}
\frac{\partial}{\partial \mu} \frac{1}{n} \sum^n_{i=1} \log p(x^{(i)})
&= \frac{\overline{x}}{\sigma^2} - \frac{\mu}{\sigma^2} \\
\frac{\partial}{\partial \sigma} \frac{1}{n}\sum^n_{i=1} \log p(x^{(i)}) &= \frac{1}{n\sigma^3}\sum^n_{i=1}(x^{(i)} - \mu)^2 - \frac{1}{\sigma}
\end{align}
$$
I tried using them in gradient descent
$$
\begin{align}
\mu_{t+1} &\longleftarrow \mu_t + \gamma \left(\frac{\overline{x}}{\sigma^2_t} - \frac{\mu_t}{\sigma^2_t}\right) \\
\sigma_{t+1} &\longleftarrow \sigma_t + \gamma\left(\frac{1}{n\sigma^3_t}\sum^n_{i=1}(x^{(i)} - \mu_{t+1})^2 - \frac{1}{\sigma_t}\right)
\end{align}
$$
or in coordinate ascent (where I would keep, say $\sigma_t$ fixed and optimize $\mu_t$ for $n_{\text{inner}}$ times and then switch: keep $\mu_t$ fixed and optimize $\sigma_t$ for $n_{\text{inner}}$ times. All this for $n_{\text{outer}}$ times. However it seems to blow up for some reason and not give me the obvious answer. You can run the code here.

What am I doing wrong?

You correctly computed the gradients, but the optimization step is incorrect. Your gradient is computed at point $(\mu,\sigma)$, and, therefore, it should be used to update $\mu$ and $\sigma$ simultaneously. Correct update rule is
$$
\begin{align}
\mu_{t+1} &\longleftarrow \mu_t + \gamma \left(\frac{\overline{x}}{\sigma^2_t} - \frac{\mu_t}{\sigma^2_t}\right) \\
\sigma_{t+1} &\longleftarrow \sigma_t + \gamma\left(\frac{1}{n\sigma^3_t}\sum^n_{i=1}(x^{(i)} - \mu_{t})^2 - \frac{1}{\sigma_t}\right)
\end{align}
$$
The formula that you have in the question implements the alternating optimization which may or may not converge (it is guaranteed to converge if the gradient norm along the optimization path is at most one).
Therefore, I corrected your code as follows:
# Loop through and update mu and sigma
mus = [mu]
sigmas = [sigma]
for i in range(n_iter):
    # Compute gradients
    mu_grad = (np.mean(x) - mu) / (sigma**2)
    sigma_grad = (np.mean((x - mu)**2)/sigma**3 - 1 / sigma)
    # Update mu and sigma
    mu, sigma = mu + gamma_mu*mu_grad, sigma + gamma_sigma*sigma_grad
    # Store mu and sigma
    mus.append(mu)
    sigmas.append(sigma)

The result converges quickly.


Match users based on the content of their articles I have users in my database that I would like to match up or group togetter based on the content of there articles. I cant seem to find how this kind of problem is being solved today. Any advice will help.

Available Data:

1) Each user's posts (anything written by them, like a blog).

2) Tags for each post (tags that the user gave to their post when they created it)

Goal:

1) Match/Group users based on available data.

2) Produce match percentage.

Attempt:

I matched people based on the number of exact matches of their tags.

Example: user1 has [car,honda,sports], user2 has [car,food].

This will give a 33% match.

As you can image this does not work very well. Most users have 20 tags but typically get a match percentage of 0% even if they are talking about similar things.

Problem:

Tags that have a clear relationship like CAR and HONDA are NOT matched.

Question:

How can I match/Group users based on tags or the content of there articles?

One way could be to apply word-embeddings for semantic similarity checking. word2vec model generates feature vectors which could capture semantic similarity. For example, the closest vector to car will be honda, ferrari, vehicle, bike. Train a model using large amount of data from wikipedia dumps or the one released by google. It has fine quality vectors. Gensim has a nice implementation of word2vec

For each blog articles, pre-process the data by removing the stop-words and stemming them. From the resulting words, collect the more frequent words. Do this for all the articles and check for the similarity among the frequent words in other articles as well. So that one article with frequent words car, race, tournament, ferrari, F1 will have more closer vectors in article with frequent words bike, honda, racer.

Or other way is to look for similar vectors in tags itself. It is good to play with it for sometime, so that you get to know which features work better for the dataset you have.


Clustering high dimensional data TL;DR: Given a big image dataset (around 36 GiB of raw pixels) of unlabeled data, how can I cluster the images (based on the pixel values) without knowing the number of clusters K to begin with?

I am currently working on an unsupervised learning project to cluster images; think of it as clustering MNIST with 16x16x3 RGB pixel values, only that I have about 48 million examples that I need to cluster.
Without knowing their identities, I do know that some of the images are definitely related because they come from the same source, but - say - I also don't know an appropriate K in order to "just" run K-means on the set yet.

I was thinking of doing some manual 2D embedding using t-SNE and then clustering manually in the embedded space (a simpler task than doing it manually in 16x16x3-d), but all t-SNE implementations I could find required loading the data into memory. I also thought about first running t-SNE, then K-means on the t-SNE embedded data, but if you look at the results of t-SNE from MNIST, it's very obvious that these clusters might and probably will be distorted and skewed in nonlinear ways. So even if I were to know a K, the clusters would probably be split up. Using Mahalanobis distances for K-means might be an interesting thing, but since I don't know covariances to begin with, that appears to be a dead end as well.

Currently I'm trying if I can run PCA compression on the examples to at least gain some memory back for t-SNE, but that might or might not work ... can't say for now.

Can somebody give me a pointer in the right direction to do this (ideally, but definitely not necessary in a Python, TensorFlow or Apache Beam/Dataflow context)? I was working on porting a Streaming/Ball K-means a while ago which does have the nice property of creating new clusters "on demand", but before I start implementing that again in Python/TensorFlow/Dataflow, I was hoping that somebody could give me some ideas where to start or what to avoid.

I don't think any of the clustering techniques "just" work at such scale. The most scalable supposedly is k-means (just do not use Spark/Mahout, they are really bad) and DBSCAN (there are some good distributed versions available).

But you will be facing many other challenges besides scale because clustering is difficult. It's not as if it's just enough to run the algorithm and then you have clusters. Clustering is an explorative technique. There is no "correct" clustering. But rather you will need to run clustering again and again, and look at every cluster. Because there will not be a single parameter setting that gets everything right. Instead, different clusters may appear only at different parameters.

But the main challenge in your case will likely be the distance function. Except for idealized settings like MNIST, Euclidean distance will not work at all. Nor will anything working on the raw pixels. So first you need to do feature extraction, then define a similarity function.

When it comes to clustering, work with a sample. Cluster the sample, identify interesting clusters, then think of a way to generalize the label to your entire data set. For example by classification (your labeled data points are your training set, predict the labels of unlabeled points).