Unsupervised Text Classification with Python: Kmeans I am working on a project to build a text classifier of questions being asked. There are no labels provided in my data so I have chosen to go with an unsupervised approach. This solution needs to read a new question, and determine which category of classification it falls into
I have been working on a KMeans model, but despite everything I try I just cannot get any decent results. I have even hand selected a data set, cleaned it further by hand to make it as straightforward as possible, and my clusters are still overlapping, and I am getting a low silhouette score of 0.26015371238537227 (this is the highest I've been able to get). My raw data isn't very great to work with, but I still don't understand how it performs so poorly when I hand select/clean the best examples to use as a test. I have included my code for kmeans section below.
Does anyone have any suggestions? I'm a kind of at a loss here. Are there any other good ways to classify unlabeled text? I'm also relatively new to this type of work and this is my first ML based project too so I'm sure there is some knowledge gap to account for as well. I'm about ready to give up on the unsupervised approach and do some manual work to label the data myself and build a supervised model. Figured I'd check here first to see if all hope is lost or not. Any help is greatly appreciated!
    data = pd.DataFrame(text)
    data.columns = ['KM']
    tfidf = TfidfVectorizer(max_df=0.80, min_df=5, max_features=10000)
    text = tfidf.fit_transform(data['KM'].values.astype('U'))

    kmeans = KMeans(n_clusters=8, init='k-means++', max_iter=300, n_init=10,
    random_state=20)
    kmeans.fit(text)

    clusters = kmeans.predict(text)

    pca = PCA(n_components=2)
    two_dim = pca.fit_transform(text.todense())

    scatter_x = two_dim[:, 0]
    scatter_y = two_dim[:, 1]

    plt.style.use('ggplot')

    fig, ax = plt.subplots()
    fig.set_size_inches(20,10)

    cmap = {0: 'green', 1: 'blue', 2: 'red', 3: 'orange', 4: 'black', 5: 'purple', 6:
    'yellow', 7: 'pink'}

    for group in np.unique(clusters):
        ix = np.where(clusters == group)
        ax.scatter(scatter_x[ix], scatter_y[ix], c=cmap[group], label=group)

    ax.legend()
    plt.xlabel('PCA 0')
    plt.ylabel('PCA 1')
    plt.show()

    order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]

    terms = tfidf.get_feature_names()
    for i in range(8):
        print("Cluster %d:" % i, end='')
        for ind in order_centroids[i, :10]:
            print(' %s' % terms[ind], end='')
        print()

    score = silhouette_score(text, labels=kmeans.predict(text))
    print(score)

If you're questions are relatively short and the number of labels are not excessive, I would strongly suggest you take a look at zero-shot classification which can be used to classify data to categories without training a model.
Both multi-class and multi-label classification are possible.
Specifically I recommend using a pre-trained language model such as BART MNLI
You can try it out of the box with some of your questions and labels and see how it works: TRY!
From my experience I think that unsupervised approaches won't give the desired results. So if zero-shot classification is no option for you for whatever reason, I would suggest you try something like snorkel to programmatically build some training data.


Identify Bad Products from given parameters using neural networks I have a problem at hand to identify Good/Bad Products using given parameters. The number of parameters are in the order of 5000s and there are multiple values for the parameters. However I do not have a labelled set of data which says these are the products that are good or bad.

For Example, Say the parameters are AX, AY, AZ, B, C, DX, DY, etc.
Each of them has a different range. Is decision trees the right approach?

Can classification be applied to this problem?

No.  Classification requires labelled data.  Without labelled data there is no way to solve this.  How would you do anything at all, if you don't know which of the products in the training set are good and which are bad?  There's no basis for making a decision of any sort.


xgboost feature selection and feature importance when you create the new feature for data analysis
for linear regression, it is clear that the feature has to be linear with other features is better

but for xgboost what is the guideline to make a new feature.
1. what is the guideline to new features for xgboost
2. when xgboost split out the feature importances. is it better just drop the non-important features?
3. if I want to know about this related field what should I study please let me know the link and books or other resorucees

1) There are no guidelines for new features for xgboost, or even linear regression. It is not necessarily true that features have to be linearly related to the outcome in a linear regression, one can use transformations and interactions to capture non-linear relationships. The problem? In a linear regression we have to manually model the non-linearity, which is a pain to find if you have more than just a handful of features and not much prior information. Xgboost (hopefully) finds these non-linear relationships automatically.

The "guidelines" for new features, regardless of your model, is really driven by the actual problem at hand. That is, what knowledge do you have about the possible relationship between a variable X and Y, and furthermore, are there better ways to express the relationship using the current information that you do know? Here is an example; suppose I have a problem where I wish to predict the probability of some customer defaulting on a loan I provide to them. I am given their past payment history, provided as a series of payments (if made). I can use external knowledge from my background in finance to know that there is an explicit formula to calculate the given interest rate in the past, given a series of payments at known intervals. I can then use this calculated interest rate as a variable to get a better representation for the relative risk of a client (as opposed to just aggregating the payments themselves) since I know that riskier people get higher interest rates due to a higher implicit probability of defaulting. Another example: in insurance, people in my country typically get into fewer accidents in the winter than the summer. Perhaps, if I were trying to predict accident occurrence over time I would strive to include some seasonality indicator variables.

I have read some guidelines in this book here by Kuhn and Johnson regarding interaction features, namely that more important features are generally more likely to be better candidates to include as interactions (product of two or more predictors are interactions, in case you were wondering).

This process is commonly referred to as feature engineering, where we essentially manipulate our current data such that the model can learn easier.

2) Maybe. For some problems yes, for other problems, no. The curse of dimensionality is real and definitely can lead to overfitting. One can use the feature importances from xgboost to drop variables and improve their model but often you can overfit to the training set when doing this and you may actually make your model worse than if you had done nothing at all. Hence, it is incredibly important that you wrap this process in a cross validation scheme to see if the feature selection process actually improves your model. Two popular algorithms that make use of feature importance scores for variable selection are recursive feature elimination and permutation methods (so called "null importances").

3) This topic is unfortunately not as well covered in literature though feature selection, extraction, and engineering are very fruitful topics being researched right now. That being said, this aforementioned book might be of use to you (current a work in progress, so it might not be free forever):

http://www.feat.engineering/


When to use model.train() vs model.eval() in Pytoch? I have a model that is used in a reinforcement learning algorithm for checkers, a la AlphaZero. Similar to that network, mine features batch normalization after each convolution layer. I am aware that this will cause different behavior/output when using .eval() vs .train()

However, I am unsure of when to use eval() vs train(). The model is used at two different points in the algorithm: First, the network is used to generate many games of self-play. Secondly, the network is trained using the positions of theses games, with the evaluation labels taken from the terminal value of the game (-1, 0, +1) and the 'improved policy' labels are taken to be the visit counts after the UCB-tree-search.

It seems to me that when the network is fully trained, I will use .eval(), as that should be 'what the network really thinks'. Therefore, for the games of self-play, I should also use .eval(). Ostensibly, this should result in stronger games of self-play and thus higher quality data. Finally, if I used .eval() in the self-play step I must also use it in the learning phase, otherwise if the network outputs are different the loss won't even be calculated using the actual outputs! I know that the network learns valuable information from .train(), as the batch norm layers learn about the mean/variance of the data universe.

As I type this, I am starting to suspect that I should be using .train() for both the self-play and learning phases. Still that seems wrong as the discrepancy in output between train() and eval() for a given position can be quite large.

I can see arguments to have the self-play phase use both .train() and .eval(), so I had a look at the implementation of facebook's ELF OpenGo and saw that they have the model in eval mode during self-play (see selfplay.py). I would do as they do because their software seems to work.


PCA Regression Problem I have a regression problem whereby my data has 21 features and I wish to apply dimensionality reduction using PCA. As far as I know, all the tutorials I have seen so far use PCA for classification problems. I did do PCA for regression but I am un-able to display the nice scatterplots that show the PC1 on the x-axis, the PC2 on the y-axis, and the targets in the middle.

I wrote the following code

        X = self.X

        pca = PCA(n_components=NUM_FEATURES_PCA)
        principal_components = pca.fit_transform(X)

        principalDf = pd.DataFrame(data=principal_components,
                                   columns=['PC1', 'PC2'])
        finalDf = pd.concat([principalDf, self.df[[self.target_variable]]], axis=1)
        plt.scatter(finalDf.loc[self.df[self.target_variable], 'PC1']
                   , finalDf.loc[self.df[self.target_variable], 'PC2'], s=50)
        plt.xlabel('PC1', fontsize=15)
        plt.ylabel('PC2', fontsize=15)
        plt.title('2 component PCA', fontsize=20)
        plt.show()

So in other terms, could we display such plots for PCA in regression ? Or should we transform the continuous target variable to a categorical (labeled one) through binning or like-wise ?

references: these plots

First of all, you can project your explanatory variable (continuous) in your first plane (PC1 + PC2). The direction of the arrow (projection) and how far goes from the axis origin will tell you how the points are distributed according to this representation of variables in your factorial plane.

On the other hand, the quick answer is to group your continuous variables into chunks (descretizise your variable into an ordinal one), then you'll have the same plot as the reference.

Furthermore, you could try to color your scatter plot using a color scale (from white to black, red to blue...), and you'll see if there's some kind of progression of your data in that factorial plane according to the continuous variable.

These three "strategies" actually are showing the same, although the second one is more sensitive to the cuts.

Summarizing:

Project the continuous variable in the factorial plane (what PCA usually does).
Group your continuous variable in groups and plot it (easier way to see although it is sensitive).
Color your scatter plot using an scale (using min and max value of your continuous variable).


Use distribution probability as a feature in ML model I built an LSMT model to predict sick cows. I also have risk factors like cow size and height (static risk factor) that I want to combine into the ML model. I found that size is geometrically distributed. My question is how I insert it as a feature to the model? I know that $P(x=K)= p*q^(k-1)$ but I don't know how to combine it as a feature.
Thank you.

As a general approach I would say you need to generate new features, that use your prior knowledge. For example, if you have a known size distribution, then for each specific size you can calculate its probability and use it as a new feature.
As I side-note, the geometric distribution of cow sizes seems very surprising to me, I would expect to see some gamma distribution or just normal (if size is measured in cm/inches).


What is a good way to store processed CSV data to train model in Python? I have about 100MB of CSV data that is cleaned and used for training in Keras stored as Panda DataFrame. What is a good (simple) way of saving it for fast reads? I don't need to query or load part of it.

Some options appear to be:

HDFS
HDF5
HDFS3
PyArrow

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.


Parse documents to obtain subjective sentiment I'm working on a project which deals with MRC (Machine Reading Comprehension).
I would like a machine to read an article and give me the sentiments based on a provided token.

For Instance given the input:  "Jackson Heights should see an increase in real estate, While other areas of brookln should see a steep decline".

Given the token: "Jackson Heights" Should return a score indicating positive sentiment: e.g .72

Given the token: "Brooklyn" Should return a score indicating negative sentiment: e.g -.72

I've tried to solve this problem in several different ways.  I've been following along with a tutorial for sentiment analysis.  I've linked to the corresponding jupyter notebook.
The issue is that this "Sentiment Analysis" doesn't really do much for multiple reasons.

Sentiment analysis removes stop words such as weren't and were.   So the statement "I would buy google stock" and the statement "I wouldn't buy google stock" would evaluate to the same sentiment
Sentiment analysis is only per text so understanding what subject applies to sentiment is important.  e.g "I love chicken and I hate fish" may return a negative sentiment because neural nets have a tendency to put a stronger weight on statements closer to the end.   However the sentiments to be tokenized based on the subject.

I've investigate a few open source technologies for this such as AllenNLP However, this didn't seem promising because the model assumes the question can be answered.
Given the text: "The price of GOOG should increase by 3% next week"
and a question "How much will Yahoo increase"
the output still highlights "3%" instead of returning nothing.
After I investigated  a simple Question answering bot using Kaggle.
This seems a bit more robust but it seems a bit overkill, I don't want to investigate a complicated AI model if there is a simpler solution.
To put the question in its simplest form.
I would like to parse a document into a list of tokens with their sentiment score.  However the solutions i've tried, remove important information, and full NLP seems to be a bit overkill.
could someone outline an simple efficient process for obtaining tokenized sentiment analysis?

As far as I understand, the task you try to do would require a system to correctly analyze the sentence at a semantic level. In general this is Natural Language Understanding and there's no solution to this problem.
Given your examples it looks like one problem is separating the different parts of the sentence, so that you can analyze the correct part of the sentence for each entity independently. This is doable: you can use shallow or deep parsing methods to extract the chunks or subtrees, and then you could apply sentiment analysis to just to the part which contains the target entity.
In general don't expect perfect results with sentiment analysis or any semantic task. Even though DL has brought great progress in the recent years, these problems are active research questions (and might still be researched for a long time).


Group-by correlation vs All observations correlation Let's suppose that I have a dataset with datapoints about footballers.

The data are about footballers' performance and information (e.g goals, assists, injuries, age, weight etc) on a monthly basis for the last 2 years.

My goal is to see how the performance and status of a footballer at a particular month is related to his performance and status of the next month.

At a first stage, I just want to run some correlation to detect some of these relationships.

In this case, does it make sense to run a separate correlation at each footballer's data of the last 2 years and then average the correlation results across players or directly run a correlation across all footballer's data at any month?

The name of the procedure you are looking for is

  Autocorrelation

This procedure looks a lot like the first you describe.

Answer fot this question in ResearchGate


How to recognise when to stop training based on Overfitting/Underfitting? I am trying to train a LSTM network, over a total of 200 epochs, with hidden layer size of 100 and 1 dense layer after the LSTM layer. I have used a batch size of 10 for the same. Basically, I am confused as to why the loss curve which I get (with MAE as loss criteria and Adam Optimiser) is looking very different from what a good model generally gives. I believe that the likely reason may be that the training is occurring over more number of epochs than should be ideal, and it is underfitting/overfitting, but I am not sure that how to recognise the same.

The loss curve for the model is

I would like to be sure of whether the model is overfitting or undercutting, and if I need to reduce the training epochs (say from 200 to 20?).Being new to this, is there any specific point to identify when to stop the training process (such as based on this loss curve). Any help in this regard is appreciated.

Overfitting :

  The model tries to memorize what it has learned. Hence, it could not
  classify unseen samples.

In case of overfitting, the validation accuracy stops increasing and the validation loss also does not decrease.

It means that the model can no more generalise itself to get a validation accuracy above a certain threshold.

Hence, you can stop the training, when the val_acc does not change for a specific number of epochs.

Underfitting :

  Underfitting means that model is not able to classify any of the
  samples even after learning them.

The model should stop its training when the accuracy and loss seem to be constant or they only revolve around a certain value.

In your case :

The loss for the train as well as test  seem to decreasing simultaneously. The test curve flattens a bit earlier. It could be treated if the learning rate is decreased.

Tip: If you are using Keras, try the EarlyStopping callback.


Fixing data inconsistencies I'm trying to analyze some data I have but there is a lot of inconsistencies in my data.

I have a SQL table that I'm trying to analyze.

The table is a table of universities with the following structure: name:string, city:string, state:string, country:string
Name is always present however city, state, country can be missing.
My main issue is that there are a ton of typos and different declination of a university name.
For example here are the declination of Standford Unversity I find when I do  SELECT "universities".* FROM "perm_universities" WHERE (name like '%stanford%'):

stanford university - stanford - ca - united states of america
the leland stanford junior university - stanford - ca - united states of america
leland stanford jr. university - stanford - ca - united states of america
stanford university graduate school of business - stanford - ca - united states of america
the leland stanford junior university (stanford university) - stanford - ca - united states of america
leland stanford junior university - stanford - ca - united states of america
stanford university - stanford -  -
leland stanford jr. university, graduate school of business - stanford - ca - united states of america
stanford law school - stanford - ca - united states of america
stanford - stanford - ca - united states of america
stanford university, graduate school of business - stanford - ca - united states of america
stanford graduate school of business - stanford - ca - united states of america
stanford univerity - stanford - ca - united states of america
stanford university (the leland stanford junior university) - stanford - ca - united states of america
the leland stanford jr. university - palo alto - ca - united states of america
leland stanford junior university, school of law - stanford - ca / n/a - united states of america
stanford universit - stanford - ca - united states of america
the leland stanford university - stanford - ca - united states of america
leland standford stanford junior university - stanford - ca - united states of america
stanford university - cambridge - ma - united states of america
the leland stanford junior university 'stanford university' - stanford - ca - united states of america
stanford university school of law - stanford - ca - united states of america
stanford univresity - stanford - ca - united states of america
the leland stanford jr. university (stanford university) - stanford - ca - united states of america
leeland stanford junior university - stanford - ca - united states of america
leland stanford junion university -  - ca - united states of america
leland stanford junior university (stanford university) - stanford - ca - united states of america
the leland stanford junior university - stanford -  -
stanford university - graduate school of business - stanford - ca - united states of america
graduate school of business, stanford university - stanford - ca - united states of america
stanford universoty - stanford - ca - united states of america
leland stanford junior university - stanford -  -
stanford univeristy - palo alto - ca - united states of america
leland stanford university - palo alto - ca - united states of america
stanford university - stanford - ca / n/a - united states of america
the leland stanford junior university, stanford university - stanford - ca - united states of america
the leland stanford junior university graduate school of business - stanford - ca - united states of america
stanford universtiy - stanford - ca - united states of america
stanford univerisity - stanford - ca - united states of america
stanford university - stanford - ct - united states of america
stanford law scool - stanford - ca - united states of america
mba: stanford university - stanford - ca - united states of america

They are all the same university, but some have typos, some have different names, some have no cities, some have the wrong cities, ... The data isn't great.

So I'm trying to fix it. How can I consolidate this data?

Since this dataset is already organised in a table, you can leverage standard SQL functions to perform a large part of the cleanup.
A record seems to be composed of 4 fields, for example:

university name, city, state, country
stanford law school - stanford - ca - united states of america

You could follow these steps to get a cleaner representation of this dataset:

Starting with the highest level (country), find unique values use it to identify all similar sounding words by listing soundex matches with itself (build a join query of the table with itself).
Use these suggested similar matches to fix all mistakes by updating the names.
Continue in this manner till you've fixed all four fields.
Identify missing states by using the city name to query the correct state from the rest of the table; if state and country are missing for "leland stanford junior university", then use the city name "stanford" to join it with itself and get the state/country name from the other records in the table.
For the university name, identify all abbreviations using grep to search for words ending in a dot character. Replace them with full expansions.
Next, breakup the university name into individual words and dump these into a temporary table in a single column. De-duplicate the values in the column so it only contains unique values.
Run the same soundex matching join query as in step 1 to identify similar sounding names, append these suggested similar names in a second column of the temp table.
Manually do a quick sanity check of what you've obtained as suggestions and delete any invalid matches.
Write a procedure to replace the words in each university name with suggested replacements, and you will have a much cleaner dataset.


Identfying spikes in data I am a little new at this - I am used to just querying data and not so much analysis of the data so any help would be greatly appreciated.

I have some data that is trending month over month. Some months with an increase in volume and others with a decrease. The goal is to identify a "spike" in the volume. Basically, is the increase abnormally large. I was trying to figure out the best way to do this. I thought the best thing to do was take the standard deviation and anything outside of two deviations - BAM!, that's a spike. Then I thought about it some and I got concerned about what happens next month. That spike is still in that data, it happened, but doesn't it shift the standard deviation so that the next month may not show a "spike" when without the prior "spike" it could have? For continuous data maybe this is less an issue but my data is rolling, I only have the last 6 months of data. If there is a "spike" this month and every one is put on notice, and the level never changes, then in six months wouldn't it seem like there is no spike?

I guess my question is, how would you quantify a measure for a "spike" given this limitation?

Your intuition about the algorithm is about right and is a known technique from six-sigma.

As you may expect from six-sigma, take 3 times deviation. You could also take the running average and deviation to compare the new value to if you expect trends.

You need to inspect the spike but can consider removing it, which will reduce variations again.


Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies I'm very new to neural networks and have recently learnt about the loss functions used with neural networks.
This question is in regards to the mean square error metric, defined as (from the textbook I'm using):
$(\frac{1}{n})\sum_{i=1}^{n}(h_{\theta}(x^{i}) - y^{i})^{2}$
Where $h_{\theta}(x^{i})$ gives the predicted value for $x^{i}$ with model’s weights $\theta$ and $y^{i}$ represents the actual prediction for the data point at index $i$.
Looking up about this function online, I've seen different sources say different things. I can't seem to work out what n actually represents.
I understood it as representing the number of neurons in the output layer and hence you'd just be finding the difference between the actual neuron value and the predicted value of the network given the weights.
Some of the sources say it represents the number of training samples. If this is the case however, what does $h_{\theta}(x^{i})$ represent? Is it a sum of the output neuron values itself? Also if n is this, wouldn't that mean you'd have to run the function many times over all the training samples to minimize it? Whereas with the previous understanding of n, you could run it on certain samples and not all of them.

First, try to understand a few points -

Output Neuron value and the prediction both are the same things. In the case of Classification, we convert the output probability to Class based on a Threshold.

MSE is used in Regression and In a regression problem, you mostly have one output Neuron e.g. Price. You may have more if you want to club multiple targets e.g. Bounding Box problem

The "N" in the denominator is the number of different errors calculated which is basically the number of samples in context.
With "In context", I meant if it's at the time of backpropagation it is the batch_size. if it's at the end of epoch/training it's the whole training dataset.

what does $h_θ(x^i)$ represent?

It is the prediction(value of output Neuron) for $i_{th}$ sample.


Does building a corpus make sense on a documentation project? I have zero to experience in data science or machine learning. Because of this I am not able to determine if building a corpus does apply to the problem I am trying to solve.

I am trying to build a reference site for cloud technologies such as AWS Google Cloud.

I was able to build structured data and identify primary entities with in a single ecosystem using standard web scraping and sql.queries.

But I wanted to have the ability to have a mechanism that can autonomously identify entities and related information that is relevant to that entity and other entities it has relationships with.

Given that a specific ecosystem documentation follows a certain style can I use few entities as training docs and then have it classify the information I mentioned above.

Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Anyways is this problem I am trying to solve fit into the general NLP ML space?

Anyways is this problem I am trying to solve fit into the general NLP ML space?

Generally speaking, feeding the source data in bulk to a ML system is unlikely to give the kind of structured output you expect. It's likely that you would have to somehow guide the process in the direction of what you want to obtain, and this might take a lot of time and effort (depending on the requirements).

That being said, there are indeed NLP methods which are meant to extract specific pieces of information from text, and it usually works quite well with domain-specific data (provided it's done correctly, caveats apply). I'm just going to list a few typical tasks which can be done:

Named Entity Recognition would be the most common and probably the most simple, since there are many existing libraries. Most libraries use a pre-trained model, but it's likely to give much better results when it's trained on the kind of data it's applied to (of course that usually means manually annotating your own training set).
Text classification can be used to automatically assign documents to a category (class) among a set of predefined categories. This is supervised so you would also need a training set containing labelled documents. Again there is a number of algorithms and libraries available.
Simple information retrieval methods based on measuring semantic similarity (see e.g here) between terms and documents can be used to search e.g. documents relevant to a term.
Topic modeling is an unsupervised approach which groups similar documents together (clustering). Since it's unsupervised it doesn't require any training data, but on the other hand what the algorithm finds as "topics" (groups) is often different than what a human would expect.
Extracting relations between concepts (typically between named entities) is a more advanced task which usually requires more work in order to capture the specifics of the job. I'm not aware of any general library for that.

Overall there are many things possible, but the first step would be to try to design the system precisely, typically using some of the existing tasks as building blocks.

  Is it fine to include a specific document in multiple categories? For example an instance in AWS can be in category ec2 and a general category Computing unit

Yes it's fine, but if you want a ML classifier to be able to predict several classes you will need to use multi-label classification (the "standard" is single label).

  Is the starting point to this is to build a corpus? I tried it out nltk categorized corpus builder.

I'd recommend building a corpus only once you have a clear idea of how you're going to use it. Also it's usually an experimental process with lots of back and forth, so try to progress iteratively rather than starting with strong assumptions/decisions which might later turn out not relevant.


Is converting a categorical value into numerical needed to find a correlation? I have a small dataset of 1300 observations x 20 features. They are all numerical but one, which is categorical; this was calculated independently and relates to each observation in any case.

I'm now attempting to find the correlation of each features in my dataset, but a simple dataframe.corr() would omit the categorical from the calculation.

I have two choices as far as I can see:

Do not consider the categorical value, but this means not being able to suggest whether the adoption of internal processes from which that feature infers the value are optimal or not
Convert into a numerical

The categorical value looks like the school grading system:

 A: higher
 B: ...
 ...
 E: low

I don't think that converting into a numerical would result in a loss of magnitude so long I how what that conversion has been made.
But here's my crux.

Should I do something like: A = 1 & E & 5 .. or A = 5 & E = 1?
Would the two different values eventually affect the correlation process in the end?

I've been seeing minimal differences.
For instance, on the same dataset with the A starting at 1 I got Rating correlated to my Y variable at -0.33; when A starts at 5 it returns -0.32.
What I noticed is the correlation varies, and goes positive the more I refine the dataset.

Also, do consider I am also after using this dataset to later do some linear regression, and calculate the RMSE.

Any advice is welcome.

UPDATE:

I was able to further play around with the dataset, and forked it in two way, replacing the Rating score in two ways:

With {'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5}
With {'A': 5, 'B': 4, 'C': 3, 'D': 2, 'E': 1}

The results are NOT what I would have expected (opposed values), which means I am now more confused than before.

Dataset below for you to test:

    Index Ranking   Rating Correlation  # Results   Label
    0   1   0.064138    840 PKW_A1
    1   3   0.087673    245 PKW_A1
    2   5   -0.028258   111 PKW_A1
    3   7   0.017542    117 PKW_A1
    4   9   -0.249403   77  PKW_A1
    5   11  -0.138552   51  PKW_A1
    6   13  -0.090198   41  PKW_A1
    7   15  -0.333333   18  PKW_A1
    8   17  -0.076830   17  PKW_A1
    9   19  -0.113594   24  PKW_A1
    10  1   0.027015    840 PKW_A5
    11  3   0.116202    245 PKW_A5
    12  5   0.134111    111 PKW_A5
    13  7   0.094221    117 PKW_A5
    14  9   -0.070592   77  PKW_A5
    15  11  -0.127137   51  PKW_A5
    16  13  -0.275387   41  PKW_A5
    17  15  0.092450    18  PKW_A5
    18  17  0.055994    17  PKW_A5
    19  19  0.081427    24  PKW_A5

Your categorical feature is an ordinal one meaning it conveys an order between the levels that represent it. So you can convert it to a numerical feature, however the difference between 2 consecutive levels is unknown and cannot be translated into numbers.

Example : High - Medium - Low --> 3 - 2 - 1 or 4 - 2 - 1 or 5 - 2 - 1.
See? The order is preserved but the difference between the levels is arbitrary.

Also you oughta now that picking values for your feature affect correlation due to the fact that you're free to pick any values you want for your categorical feature.


How can I find synonyms and antonyms for a word? I found some code online where I can feed in a word, and find both synonyms and antonyms for this word.  The code below does just that.

import nltk
from nltk.corpus import wordnet   #Import wordnet from the NLTK
syn = list()
ant = list()
for synset in wordnet.synsets("fake"):
   for lemma in synset.lemmas():
      syn.append(lemma.name())    #add the synonyms
      if lemma.antonyms():    #When antonyms are available, add them into the list
          ant.append(lemma.antonyms()[0].name())
print('Synonyms: ' + str(syn))
print('Antonyms: ' + str(ant))

My question is, how can I choose a word, like 'and', and find all synonyms and antonyms based on comments that exist in a field in a dataframe?  Here is a sample of the first 10 lines from my dataframe.

feels weird; may be a fake!
package came end missing box. since itâ€™s gift i update actual fit.
birkenstock amazing shoe!!!! i wish i ten pairs!
delivered advertised.... shoe looks fake.
second pair i had. nothing beats them.
they totally fake ðŸ˜¡. they felt weird i finally noticed â€œmade germanyâ€ logo above. they also smell like glue leather. infuriating happen!
i've birkenstock wearer 35 years. i wear 10 women's size ones i ordered wear big. i need size down.
great brand, packaging good, dad likes them. the size clearly needs fixed others don't order bigger sizes wanted. some people don't want bigger size actually wear.
false advertising.
a bit loose compared birkenstocks size. still like though. very comfy

The general approach is:

Split the text in tokens.
For each token, find synonyms and antonyms and store them with the token

from collections import defaultdict

from nltk.corpus import wordnet
from nltk.tokenize import word_tokenize

text = """
feels weird; may be a fake!
package came end missing box. since itâ€™s gift i update actual fit.
birkenstock amazing shoe!!!! i wish i ten pairs!
delivered advertised.... shoe looks fake.
second pair i had. nothing beats them.
they totally fake ðŸ˜¡. they felt weird i finally noticed â€œmade germanyâ€ logo above. they also smell like glue leather. infuriating happen!
i've birkenstock wearer 35 years. i wear 10 women's size ones i ordered wear big. i need size down.
great brand, packaging good, dad likes them. the size clearly needs fixed others don't order bigger sizes wanted. some people don't want bigger size actually wear.
false advertising.
a bit loose compared birkenstocks size. still like though. very comfy"""

tokens = word_tokenize(text)

# Store results in a dict where key is token and value is a set of synonyms and antonyms
tokens_syns, tokens_ant = defaultdict(set), defaultdict(set)

for token in tokens:
    for synset in wordnet.synsets(token):
        for lemma in synset.lemmas():
            tokens_syns[token].add(lemma.name()) # Add the synonyms
            if lemma.antonyms():    # Check if antonyms are available
                tokens_ant[token].add(lemma.antonyms()[0].name()) # Add antonyms

# Check for 'and'
token = 'and'
print(f'For the word "{token}":')
print('The synonyms are:', tokens_syns[token])
print('The antonyms are:', tokens_ant[token])

# Conclusion 'and' has no synonyms or antonyms according to WordNet


how can i fix this problem ? I tried to train a simple linear VSM classifier library(kernlab)
 letter_classifier <- ksvm(letter ~ ., data = letters_train,
                       kernel = "vanilladot")

>  Setting default kernel parameters   Error in .local(x, ...) :    No
> Support Vectors found. You may want to change your parameters In
> addition: Warning message: In .local(x, ...) : NAs introduced by
> coercion

Check the structure of letters_train.
SVM is for continuous and categorical variable analysis, and surely one of your variables (letter) is likely still a character variable. Change it to factor and try again.


K-Fold and Random Subsampling (RSS) Dataset generation? Let say if I have a large dataset of 300k instances with 200 features, I want to reduce its size. Can I apply K-Fold technique to the 200 features then the trimmed dataset are applied with RSS to trim the instances? Its like K-Fold to reduce Features then RSS to reduce instances then I get a small-scaled dataset with the less number of features and randomized smaller instances. Can this be done?

K-fold cross-validation is a technique you use when you have too little training data. You use it to expand your training dataset, leaving only a tiny bit for testing (for example, 5%). Then you use k-fold cross-validation to select a different 5% for testing each time, and a different 95% of the full dataset for training each time.  That way, you can get a good idea of the performance of your model, even if you only kept 5% of the data apart for training.

This is just an example, it could be that 5% is fine, or even too much, to hold out for testing. It depends mainly on the size of your dataset, and in case of classification, for example, it matter if one class has very few examples, so that perhaps in your 5% for testing there may not be a single example of that class.

What you're trying to do is called dimensionality reduction. The go-to technique for that is PCA. Make sure you normalize your features first (subtract the mean and divide by the variance), and then drop features until you still have, for example, 95% of the variance of your data explained. If in the end you find that you cannot get good performance from your model, it could be that you have lost too much information. In that case, you may have to increase this to 99%, adding features back in.

RSS is ok to reduce the number of training examples you have in your dataset, but it is only necessary if the order of examples is not random already. If it's random already, then you can just train on the first 10%, or whatever you need, and save yourself the trouble.


Why do we only care about convex functions when doing Gradient Descent/SGD? I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of the function.
However, there are also functions which are not convex, but for which local minima are also global minima, for example, a function which looks like this:
Isn't there a way to characterize every function which "works well" with gradient descent? Something like "if f has a local minima, then it's also a global minima", which would be a weaker hypothesis than convex?
PS: When writing this, I also realize that we have a problem with functions like $x \rightarrow{} x^3$, for which gradient descent could stop on a "plateau". Anyway, my question remains the same: Isn't it possible to define a more general class of functions for which gradient descent works better than the convex functions?

Probably in the 1-dimensional case this more precise class of functions you aim at would be something like:

Piece-wise monotonous (thus approachable with gradient descent locally) with up to one flip (thus you know that one optimum would be where the pieces meet, the other would be at the extremes)

However, even if one is able to come up with such a definition for the simpler space of 1-dimensional functions, it becomes much more complicated to come up with a generalisation that would be valid in the more generic multidimensional case.
Despite this difficulty, the effort still might be worthwhile if there's going to be a distinctive benefit of doing it. In this particular moment, I personally wonder if this might be the case. Surely, it depends on the context of your problem. Wouldn't a possible approach be to break down your problem into steps:

Try to break down your surfaces/functions in piecewise convex
Apply gradient descent on the pieces
Compare the optima that you have identified


Problem with Linear Regression and Gradient Descent import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import numpy as np

x = np.linspace(0, 10, 1000)
points = np.genfromtxt("Dokumente/Salary.csv", delimiter=",")
points2 = points[1:, :]

def gradient_descent(current_b, current_m, learning_rate, points):
    b_gradient = 0
    m_gradient = 0
    N = float(len(points))
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        b_gradient += -(2/N) * (y - ((x * current_m) + b_gradient))
        m_gradient += -(2/N) * x * (y - ((x * current_m) + current_b))
    b_new = current_b - (learning_rate * b_gradient)
    m_new = current_m - (learning_rate * m_gradient)
    return [b_new, m_new]

def compute_error(points, b, m):
    total_error = 0
    N = len(points)
    for i in range(0, len(points)):
        x = points[i, 0]
        y = points[i, 1] / (10 ** 4)
        total_error += (1/N) * ((y - ((m * x) + b)) ** 2)
    return total_error

def run(points, staring_b, starting_m, learning_rate, num_iterations):
    b = staring_b
    m = starting_m
    for i in range(num_iterations):
        learning_rate = learning_rate
        b, m = gradient_descent(b, m, learning_rate, np.array(points))
        plt.plot(i, compute_error(points, b, m), "ro")
    return [b, m]

learning_rate = 0.000001
init_b = 0
init_m = 0
number_of_iterations = 10000
print(compute_error(points2, init_b, init_m))
[b, m] = run(points2, init_b, init_m, learning_rate, number_of_iterations)
print(compute_error(points2, b, m))

plt.plot(points2[:,1] / (10 ** 4), "ro")
x = np.linspace(0, 35, 1000)
plt.plot(x, ((x * m) + b))
plt.show()

I played with learning rate and the number of iterations, but this does not fit the data intuively.
Can i get some tips on my code please?
Thank you in advance.

Gradient descent probably didn't converge. I'd sugest scaling your feature on a [-1,1] or [0,1] scales, try increasing your learning rate and maybe add some code which checks the gradient's slope and if there is perhaps sequence of <0,>0,<0 it means that the learning rate is locally too large and cost function can't converge. In that case, learning rate should be decreased.


Should we also include negative instance in cross-validation process of one-class classifiers? For a one-class classifier to do text classification, only positive instances are used for training.
However, in the cross-validation process to select the best hyperparameters, should we also include negative instances to evaluate the classifier? or use only positive instances? What's the tradition?

The evaluation design depends on the goal, i.e. how the classifier is intended to be used in production. The goal of a one-class classifier is to find instances of this class among other instances, so it has to be evaluated with data which includes negative instances. The choice of the proportion and type of negative instances should be driven by the final goal.
Note that in terms of evaluation and cross-validation the setting is the same as regular supervised classification, the difference is only that negative instances are not used during training.


Is it possible to share and merge classification models? I'm starting my machine learning study and I'm trying to figure out a simple question:

Let's say that I have two models, one that recognizes cats, and another one that recognize dogs.

Now I have a camera and I want to recognize both cats and dogs using my models.

Obviously, I don't want to create a third model to recognize both, having all the work to label each animal, so, is there a method to "merge" both models into one?

I'm asking this question because I want to understand one thing: why ML engineers don't share their models so then we can create aggregated models? For example, if a person A has a model that classifies people and a person B has a model for animals. Why they can't just share their models so then each one will have a more powerful model without needing to re-train everything?

I'm sorry if this is a too basic question, but I didn't see on Google any clear explanation. Thanks.

I believe that I've found an answer here and here.


Statistical comparison of 2 small data sets for 2X increase in the population mean I am trying to determine whether or not we are 90% confident that the mean of a proposed population is at least 2 times that of the mean of the incumbant population based on samples from each population which is all the data I have right now. Here are the data.

incumbantvalues = (7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)

proposedvalues =  (17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

I have no idea if either population is or will be normal.

The ratio of the sample means does exceed 2.0 but how does that translate to confidence that the proposed population mean will be at least twice that of the mean of the incumbant population with 90% confidence ?

Can re-sampling (bootstrapping with replacement) help answer this question ?

Yes, in principle, resampling can help answer this question.

incumbent <- c(7.3, 8.4, 8.4, 8.5, 8.7, 9.1, 9.8, 11.0, 11.1, 11.9)
proposed  <- c(17.3, 17.9, 19.2, 20.3, 20.5, 20.6, 21.1, 21.2, 21.3, 21.7)

set.seed(42)

M  <- 2000
rs <- double(M)

for (i in 1:M) {
    rs[i] <- mean(sample(proposed, replace=T)) - 2 * mean(sample(incumbent, replace=T))
}

To make the assessment, you should choose one (not both) of the following:

A. The (two-tailed) 90% confidence interval for the difference in the (weighted) means using Hall's method is:

ci.hall <- 2 * (mean(proposed)-2*mean(incumbent)) - rev(quantile(rs,prob=c(0.05, 0.95)))
names(ci.hall) <- rev(names(ci.hall))
ci.hall

   5%   95%
-0.29  2.95

This is appropriate if you have any concern about missing the possibility that mean(proposed) might actually be less than 2 * mean(incumbent).

B. The proportion of resample means >= 0 provides the (one-tailed) estimate that mean(proposed) is at least twice mean(incumbent):

sum(rs>=0)/M

[1] 0.8915

The problem is that the samples are really rather small and resampling estimates
can be unstable for small n. The same issue applies if you want to assess normality and go with parametric comparisons.

If you can get to, say, n >= 30, the approach described here should be fine.


Improving the pix2pix Architecture for Sketch to Image Translation on a Dataset of Sketches of People to Photos of People For a university project I need to create a neural network which translates sketches of people into images.
In order to implement such a neural network, I decided to implement a pix2pix GAN architecture.
The neural network is trained an evaluated on a modified version of the CUFS dataset provided by my professors.
While the neural network is able to perform image transalations, as shown in the image pair below, I wonder if there are ways to improve the quality of the results?
In particular I was wondering if there are ways make the image look sharper?

I am already augmenting the relatively small training dataset of 70 sketch-photo pairs to 770 ones.
Furthermore, I tried to decrease the value of the $\lambda$ parameter.
Regarding the epoch, I figured out that the validation loss does not drop significantly after 160 iterations anymore.
I implemented the network in a Jupyter notebook hosted on Google Colab.
As you can see in the notebook's last cell, the currently achieved L1 loss on the testing dataset is about 0.191 on average.
Here is a link to the dataset used for training and testing.
Please feel free to take a look at the code and tell me ways on how to improve the code.
In addition, I would be most grateful if you could provide additional ways on how to improve the generated image quality.

I managed to obtain better results with less artifacts by implementing the code of the paper "High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks".
I found this code to be quite nice working for my purposes.
If you need, I can also provide a Colab notebook with my ported code.


Types of Machine Learning/Data Analysis I am looking for a framework or chart that classifies all the different types of data analysis/machine learning. I want to use this classification to organize my knowledge/areas to learn.

Examples of what I mean by types of data analysis/ML include:
 - Geographical Maps
 - Image Recognition (CV, OCR...)
 - Signal Processing
 - Text/Speech Comprehension

Is there a chart/framework classifying all those domains ?

You can use ML/DA in almost every sector, but to generalise there are few domains which i would like to list

Broadly it is classified based on data

Text data
Image data
Signal/Periodic Data
Financial data/Numbers

In here we can further classify or give names to the domains. Text data broadly falls under Natural Language Processing (NLP). Here you can work with text generation, text segmentation, document classification, chatbots, topic modelling, Summary extraction & much more
NLP Research,
NLP Application I have added links to 2 articles listing current research & applications by current industry standards.

Image data is broadly falls under Computer vision. Here you work with videos, live streams, images. They can be satellite images, road accidents, machine parts etc. You can do analysis of football match, or even do analysis on plastic pollution in oceans. Sub topics can be classified as

Image tagging
Image based information gathering
Object prediction
Vision based health prediction
Biological researches
Image rectification using NNs

And much more. If you want to have hands-on experience for image based data, please visit this site, here you will get basic idea of what topics one can achieve with image data.

Periodic data consists of measurements from machines, observations of how they work, it also includes biological data. Basically our normal signal processing or risk analysis. It's application is much needed for machinery makers, health sectors & financial brokers, vehicle positioning etc. Anything which has some time based data into it can be used to simplify routes, analyse risks & avoid unnecessary wear & tear.

But generally this domains are used together with each other unless you go for research. Working for a solution firm i have encountered many projects where i had to gather text data from image & use topic modelling for classification. Had a project where i had to analyse data generated from video cam observing human to predict their HR for next few seconds.

Hope this helps you in identifying what domain interests you more.


RStudio/RMD files to create a student version assignment from the instruction version assignment solution Does anyone know plugin or package in Rstudio to create a student version assignment from the instruction version assignment solution?

I know nbgrader for Jupyter Notebook that can do this, but I am working on RMD files in RStudio.

How about learnr?

Here is the link to the package information:
https://cran.rstudio.com/web/packages/learnr/index.html


Large no of categorical variables with large no of categories I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1).

Most of my features are categorical with large number of categories. For example: one of them has 310 categories but the top 10 most frequently occurring variables account for ~50% of the training and test data.
I was thinking of keeping these top k frequently occurring values and encoding all the other values as another category "others".

Would it be okay to do so?

Also, what kind of classifier would be optimal for such a task? I was looking at random forests. However, due to noisy data (too many categories, too many features) my model doesn't generalize too well (low recall and precision).

Thank you for your time.

PS: Data not time series

I used to work a lot with such data in the past.
When you have many categorical features with high cardinality, you must avoid one-hot encoding them because it consumes too much memory and above all trees built from them will be too deep and worse than trees built without one-hot encoding.
You need a Tree based model library implemented with native categorical feature support (i.e. not needing the categorical features to be one-hot encoded).
I suggest you to use one of those 4 implementations :
- CatBoost
- LightGBM
- H2O GBM
- H2O Random Forest
Scikit learn and XGBoost implementations still need one-hot encoded categorical variables so I don't recommend using one of these libraries if your dataset has high cardinality categorical variables (i.e. with more than about 10 levels/categories).
See this :

  LightGBM offers good accuracy with integer-encoded categorical
  features. LightGBM applies Fisher (1958) to find the optimal split
  over categories as described here. This often performs better than
  one-hot encoding.


How to compute score and predict for outcome after N days Let's say I have a medical dataset/EHR dataset that is retrospective and longitudinal in nature. Meaning one person has multiple measurements across multiple time points (in the past). I did post here but couldn't get any response. So, posting it here
This dataset contains information about patients' diagnosis, mortality flag, labs, admissions, and drugs consumed, etc.
Now, if I would like to find out predictors that can influence mortality, I can use logistic regression (whether the patient will die or not).
But my objective is to find out what are the predictors that can help me predict whether a person will die in the next 30 days or the next 240 days, how can I do this using ML/Data Analysis techniques?
In addition, I would also like to compute a score that can indicate the likelihood that this person will die in the next 30 days? How can I compute the scores? Any tutorials links on how is this score derived?, please?
Can you please let me know what are the different analytic techniques that I can use to address this problem and different approaches to calculate score?
I would like to read and try solving problems like this

This could be seen as a "simple" binary classification problem. I mean the type of problem is "simple", the task itself certainly isn't... And I'm not even going to mention the serious ethical issues about its potential applications!
First, obviously you need to have an entry in your data for a patient's death. It's not totally clear to me if you have this information? It's important that whenever a patient has died this is reported in the data, otherwise you cannot distinguish the two classes.
So the design could be like this:

An instance represents a single patient history at time $t$, and it is labelled as either alive or dead at $t+N$ days.
This requires refactoring the data. Assuming data spans a period from 0 to $T$,  you can take multiple points in time $t$ with $t<T-N$ (for instance every month from 0 to $T-N$). Note that in theory I think that different times $t$ for the same patient can be used in the data, as long as all the instances consistently represent the same duration and their features and labels are calculated accordingly.
Designing the features is certainly the tricky part: of course the features must have values for all the instances, so you cannot rely on specific tests which were done only on some of the patients (well you can, but there is a bias for these features).

To be honest I doubt this part can be done reliably: either the features are made of standard homogeneous indicators, but then these indicators are probably poor predictors of death in general; or they contain specialized diagnosis tests
for some patients but then they are not homogeneous across patients, so the model is going to be biased and likely to overfit.

Ideally I would recommend splitting between training and test data before even preparing the data in this way, typically by picking a period of time for training data and another for test data.
Once the data is prepared, in theory any binary classification method can be applied. Of course a probabilistic classifier can be used to predict a probability, but this can be misleading so be very careful: the probability itself is a prediction, it cannot be interpreted as the true chances of the patient to die or not. For example Naive Bayes is known to empirically always give extreme probabilities, i.e. close to 0 or close to 1, and quite often it's completely wrong in its prediction. This means that in general the predicted probability is only a guess, it cannot be used to represent confidence.

[edit: example]
Let's say we have:

data for years 2000 to 2005
N=1, i.e. we look at whether a patient dies in the next year.
a single indicator, for instance say cholesterol level. Of course in reality you would have many other features.
for every time $t$ in the features we represent the "test value" for the past 2 years to the current year $t$. This means that we can iterate $t$ from 2002 (2000+2) to 2004 (2005-N)

Let's imagine the following data (to simplify I assume the time unit is year):
patientId birthYear year     indicator
1         1987      2000     26
1         1987      2001     34
1         1987      2002     18
1         1987      2003     43
1         1987      2004     31
1         1987      2005     36
2         1953      2000     47
2         1953      2001     67
2         1953      2002     56
2         1953      2003     69
2         1953      2004     -    DEATH
3         1969      2000     37
3         1969      2001     31
3         1969      2002     25
3         1969      2003     27
3         1969      2004     15
3         1969      2005     -    DEATH
4         1936      2000     41
4         1936      2001     39
4         1936      2002     43
4         1936      2003     43
4         1936      2004     40
4         1936      2005     38

That would be transformed into this:
patientId yearT age indicatorT-2 indicatorT-1 indicatorT-0   label
1         2002  15  26           34           18             0
1         2003  16  34           18           43             0
1         2004  17  18           43           31             0
2         2002  49  47           67           56             0
2         2003  50  67           56           69             1
3         2002  33  37           31           25             0
3         2003  34  31           25           27             0
3         2004  35  25           27           15             1
4         2002  66  41           39           43             0
4         2003  67  39           43           43             0
4         2004  68  43           43           40             0

Note that I wrote the first two columns only to show how the data is calculated, these two are not part of the features.


Using SVM as a binary classifier, is the label for a data point chosen by consensus? I'm learning Support Vector Machines, and I'm unable to understand how a class label is chosen for a data point in a binary classifier. Is it chosen by consensus with respect to the classification in each dimension of the separating hyperplane?

The term consensus, as far as I'm concerned, is used rather for cases when you have more a than one source of metric/measure/choice from which to make a decision. And, in order to choose a possible result, you perform some average evaluation/consensus over the values available.

This is not the case for SVM. The algorithm is based on a quadratic optimization, that maximizes the distance from the closest documents of two different classes, using a hyperplane to make the split.

So, the only consensus here is the resulting hyperplane, computed from the closest documents of each class. In other words, the classes are attributed to each point by calculating the distance from the point to the hyperplane derived. If the distance is positive, it belongs to a certain class, otherwise, it belongs to the other one.


Not able to interpret decision tree when using class_weights I'm working with an imbalanced dataset. I'm using a decision tree (scikit-learn)  to build a model. For explaining my problem I've taken iris dataset.
When I'm setting class_weight=None, I understood how the tree is assigning the probability scores when I use predict_proba.
When I'm setting class_weight='balanced', I know its using target value to calculate class weights but I'm not able to understand how the tree is assigning the probability scores.
import sklearn.datasets as datasets
import pandas as pd
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

iris=datasets.load_iris()
df=pd.DataFrame(iris.data, columns=iris.feature_names)
y=iris.target

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.33, random_state=1)

# class_weight=None
dtree=DecisionTreeClassifier(max_depth=2)
dtree.fit(X_train,y_train)

dot_data = StringIO()
export_graphviz(dtree, out_file=dot_data, filled=True, rounded=True, special_characters=True, feature_names=X_train.columns)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png()) # I use jupyter-notebook for visualizing the image

# printing unique probabilities in each class
probas = dtree.predict_proba(X_train)
print(np.unique(probas[:,0]))
print(np.unique(probas[:,1]))
print(np.unique(probas[:,2]))

# ratio for calculating probabilities
print(0/33, 0/34, 33/33)
print(0/33, 1/34, 30/33)
print(0/33, 3/33, 33/34)

The probabilities assigned by the tree and my ratios (determined by looking at tree image) are matching.
When I use the option class_weights='balanced'. I get the below tree.
# class_weight='balanced'
dtree_balanced=DecisionTreeClassifier(max_depth=2, class_weight='balanced')
dtree_balanced.fit(X_train,y_train)

dot_data = StringIO()
export_graphviz(dtree_balanced, out_file=dot_data,filled=True, rounded=True, special_characters=True, feature_names=X_train.columns)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

I'm printing unique probabilities using below code
probas = dtree_balanced.predict_proba(X_train)
print(np.unique(probas[:,0]))
print(np.unique(probas[:,1]))
print(np.unique(probas[:,2]))

I'm not able to understand (come-up with a formula) how the tree is assigning these probabilities.

We should consider two points. First, class_weight='balanced' does not change the actual number of samples in a class, only the weight of class $w_{c_i}$ is changed. Second, the [un-normalized] probability of class $c_i$ in each node is calculated as

$w_{c_i}$ x (number of samples from $c_i$ in that node / size of $c_i$)

For example, in balanced mode, the [un-normalized] probability of $c_3$ in the green leaf is calculated as

$33.\bar{3}\% \times (3 / 36) ≈ 2.778\%$

compared to $36\% \times (3 / 36) = 3\%$ in unbalanced mode.

The probability (normalized) in balanced mode would be:

$100 \times 2.778/(2.778+32.258) \% = 7.9289\%$

Remark. The word "probability" is not applicable to each isolated node  except for the root node. This is the un-normalized version of the probability used to classify a data point inside a leaf, though the normalization is not required for comparison. However, the notion is applicable to the aggregate of nodes at the same level and the leaves from upper levels (i.e. set of all samples).


Can I use Tensorboard to manage-fjobs and visualize learning on non-Tensorflow algoithrms? (e.g. Scikit?) I am still searching for a great tool that manages jobs and visualizes learning from my models. Tensorboard is obviously one option given it's massive support.

But is it possible to organize jobs and visualize learning on tensorboard for algorithms not from the TF library?

For logging values with TensorBoard (without running TensorFlow), you can use the tensorboard logger library.


How would you categorise email subjects to find similar emails? I have a list of email subjects like

<XYZ> commented on <ABC>
Weekly review for <Company>
Your account is ready

And I want to find patterns in them so I can group them.

Is there a well known algorithm I can use?

You'll probably have to experiment a bit with different approaches.  Let me outline two different kinds of approaches you could try.

Topic modelling

You could try applying unsupervised topic modelling to your subject lines.  LDA is probably the most widely used method.

Topic modelling tries to find a limited number of "topics", and assigns each subject line to one or more "topics" based on the words in the subject line.

Clustering

You could try using clustering.  Broadly, you would find some way map each subject line to a feature vector, and then apply some unsupervised clustering method.  There are many options for each of those two steps.  To get feature vectors, you might try any of the existing word embeddings; e.g., you might try word2vec.  For clustering, there are many, many clustering algorithms; e.g., you might try k-means.  I recommend you do a little reading on these topics and then experiment with them a bit.

Caveats

A warning.  Don't set your expectations too high. Subject lines are typically very short, and that will make it hard for these techniques to find clusterings.  In other words, you are operating in a regime that's known to be hard for existing NLP and ML techniques.

If you can find any other features that might assist with clustering similar emails (e.g., the identify of the sender? the mailing list it was sent on, if any?), including that information might improve the quality of the clusters.


Clustering analysis for observations with lists as data So I have several samples analyzed for their chemical composition. After data analysis, for each sample, I have a list of compounds found and their corresponding relative abundance. Some compounds are unique but most are actually found in most samples.

I want to do clustering analysis based on these list of compounds. How do I go about this? Specifically how to vectorize my dataset since each observation is actually an array with both numerical (abundance) and categorical (compound label) variables.

K-means would be a fine clustering method for you to start with, though you will have to provide the number of clusters you wish for it to return (not sure if you know that/can figure it out). Otherwise check out DBSCAN.

As for the mix of numerical vs categorical data types, all you will need to do is one-hot encoding on your categorical variables. What that does is that it will take all of the known possibilities for a category and it creates new features out of them. A 1 is assigned if the sample is part of that category, and a 0 if it is not. In this way you can use numerical and categorical at the same time, just make sure to normalize your numerical data!


Does it make sense to expand word embeddings so that each array index is a feature input or should the embedding itself be a model input? If you are building a DNN, say, with two layers, and you want to use embeddings as one of your feature inputs, what's the best way to input the embedding?

I'm trying to understand if I should break the embeddings up so that every array value in the embedding becomes its own input feature to the model or whether the embedding should be kept in array form.

I've been following AirBnB's model for inspiration.

I'm trying to predict a binary classification in the final layer.

Breaking the array you lose distributed representation-point of word embedding.

There is information in other dimensions (words) that are distributed in the word-embedding representation, breaking them up makes no sense.


Pretrained features return worse results for sklearn classifier/pipeline So I have a following scenario:

Pipeline, that transform text/dict/numerical data and classifies the result with linear regresion. It looks something like this:

args = {
    'y': labels,
    'cv': 5,
    'scoring': 'accuracy',
    'n_jobs': -1
}

feats = ('features', FeatureUnion([
    fulltext_pipeline,
    numerical_pipeline,
    nela_desc_pipeline,
    v_tags_pipeline,
    os_pipe
]))

clf = ('clf', LogisticRegression(multi_class='auto',
    solver='liblinear',
    tol=0.02,
    C=500,
    penalty='l1',
    random_state=0))

os_pred = Pipeline([
    feats,
    clf
])

And for evaluation I am using cross_val_score. So far so good, everything works as suppoused, but if I try to extract "features" and run them separately, I get worse results. The example looks as follows:

features = feats[1].transform(data)
np.average(cross_val_score(clf[1], features, **args))
np.average(cross_val_score(os_pred, data, **args))

I expected to get an exact same results here (as in theory they should do the same - maybe I am missing something?)

I've tried with .transform(data), .fit_transform(data), .fit_transform(data, y=labels). The odd thing for me was that all three methods generate the same output (sparse matrix) which is far worse compared to results from pipeline.

Data is in form of sparse matrix

<421x862 sparse matrix of type '<class 'numpy.float64'>'
    with 361371 stored elements in Compressed Sparse Row format>

While labels are normal strings (3 classes).

Results are as follows:

0.672436 from np.average(cross_val_score(clf[1], features, **args))

0.772509 from np.average(cross_val_score(os_pred, data, **args))

I double checked for any random seed or something of this sort, but the results are consistend and pipeline/classifier return always the same result. And for me it looks like that this is quite a big difference (given the fact that the range is [0.00; 1.00]).

Question at the end is: Why classifier seems to be biased in a pipeline scenario compared to just prefeeling it with features directly?

I do aknowledge that pipeline does some kind of additional steps of transforming/fit-transforming/etc., but with my current knowledge of sklearn library I cannot reproduce them "outside" the pipeline.

Normalization

Long story short normalization was the problem in my case. Because I haven't shared the transformer pipelines code in FeatureUnion, I've missed to tell that each one of them ends with normalization. So that's what the biggest difference.

In case with pretrained -> normalization is done over the whole dataset and nothing more
In case of pipeline -> normalization is done 10 times (twice per each fold -> once for train data, once for test data)

And that's how I do get different values => such difference in results.


What exactly is a skewed dataset ? Like above, I'd like to know, what exactly a skewed dataset is.

The explanation from stats.se.com sounds to me more like what I call an imbalanced dataset?

What is the distinction?

In the context of the link, a skewed data set is referring to a dataset with a class imbalance problem. They are trying to build a classifier, but they have many more negative examples than positive examples. It's not a very precise term, but I've heard to used in this context a few times.


How to compute Hessian matrix for log-likelihood function for Logistic Regression I am currently studying the Elements of Statistical Learning book. The following equation is in page 120.

It calculates the Hessian matrix for the log-likelihood function as follows

\begin{equation}
\dfrac{\partial^2 \ell(\beta)}{\partial\beta\partial\beta^T} = -\sum_{i=1}^{N}{x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))}
\end{equation}

But is the following calculation it is only calculating $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i^2}$ terms. But Hessian matrix should also contain $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i\partial\beta_j}$ where $i\neq j$.

Please explain the reason for missing out these terms.

Beta is a vector of parameters, therefore:

$
\frac{\delta l(\beta)}{\delta\beta}=
[\frac{\delta l(\beta)}{\delta\beta_1}\quad\frac{\delta l(\beta)}{\delta\beta_2}\quad\frac{\delta l(\beta)}{\delta\beta_3}\quad...\quad\frac{\delta l(\beta)}{\delta\beta_n}]$ and so

$
\frac{\delta(\frac{\delta l(\beta)}{\delta\beta})}{\delta\beta^{T}}=
\begin{bmatrix}
\frac{\delta l^2(\beta)}{\delta\beta_1^2} & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_1\delta\beta_n} \\
\frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_2^2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_2\delta\beta_n} \\
\vdots & \vdots & \ddots & \vdots \\
\frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_1} & \frac{\delta l^2(\beta)}{\delta\beta_n\delta\beta_2} & ... & \frac{\delta l^2(\beta)}{\delta\beta_n^2}
\end{bmatrix}$, which is your Hessian.

The term on the right side of your equation is also a matrix, because there is a multiplication of vectors in it: $x_i \cdot x_i^T$, which gives a $n \times n$ matrix.


Spark 1.6.1 - Determining the number of clusters in a data set There existis any way to find the number of cluster needed to use in K-Means?

I'm having some difficulties to find a method to determine k...

Many thanks!

You can use the Elbow method. https://en.wikipedia.org/wiki/Elbow_method_(clustering)


Why is my keras cnn model giving me the same result for every prediction? I have a simple cat or dog cnn model.
My model has an 80% validation accuracy and an 84% training accuracy

import keras
import tensorflow as tf
from keras.models import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPooling2D
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
import numpy as np
from keras.models import load_model

classifier = tf.keras.Sequential()

classifier.add(Conv2D(32,(3,3),input_shape = (64,64,3),activation ="relu"  ))

classifier.add(MaxPooling2D(pool_size = (2,2)))

classifier.add(Conv2D(32,(3,3),activation ="relu"  ))
classifier.add(MaxPooling2D(pool_size = (2,2)))
classifier.add(Flatten())
classifier.add(Dense(units = 128, activation = "relu" ))
classifier.add(Dense(units = 1, activation = "sigmoid" ))
classifier.compile(optimizer = "adam", loss = "binary_crossentropy", metrics = ["accuracy"])
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set= train_datagen.flow_from_directory('dataset/training_set',
                                                 target_size=(64, 64),
                                                 batch_size=32,
                                                 class_mode='binary')

test_set = test_datagen.flow_from_directory('dataset/test_set',
                                             target_size=(64, 64),
                                             batch_size=32,
                                             class_mode='binary')
classifier.fit(training_set,
               steps_per_epoch=int(8000/32),
               epochs=25,
               validation_data=test_set,
               validation_steps=int(2000/32))
print(training_set.class_indices)
from keras.preprocessing import image
test_image = image.load_img("dataset/training_set/cats/cat.1.jpg",target_size= (64,64))
test_image = image.img_to_array(test_image)
test_image = np.expand_dims(test_image, axis = 0)
result = classifier.predict(test_image)

You are applying a preprocessing step to both your train and test images: rescale=1./255. This normalises the values of the image pixels to be in the range of (0,1) instead of (0,255).

However, when you are doing your predictions you are not applying this rescaling to your test image. This translates into testing an image with pixel values 255 higher than your model is expecting, thus always predicting the same (always a 1 I gues...) You should rescale your test samples as well. Just try:

print(classifier.predict(test_image/255))


Why is spam detection a classification problem and not a class modelling problem Trying to get my feet wet with machine learning on text.

The most common dataset I've seen in this space is the sms dataset with classes ham and spam.

And the most common and successful approach seems to be to model this as a binary classification problem and to use a multinomial naïve Bayes to solve it.

However I'm trying to understand why this is a binary classification problem.

I understand that the spam category has some common features associated with it across the class - such as ads, offers , free discounts and so on.

But there's no definition for what is a ham class is there? The definition of ham is - everything other than spam.

So why is this a binary classification task?

For more context - I'm trying to solve the problem of whether a news article belongs to the politics class or to the non-political class.

Suppose I have a labelled dataset of around 3000 samples in each class.

The non political class is a mix of classes like sports , religion , science and technology and miscellaneous.

Will a binary classifier work better than an algorithm such as oneclassSVM where anything other than political news is an outlier ?

What are some of the other algorithms that I can use to solve this problem? I have heard about PU learning but I haven't seen any implementations of algorithms in any machine learning libraries ( I'm working with python)

If any of you have experience doing class modelling on text. Please share your comments and insights

Thank you!

Regarding the case of spam vs ham, you are right that the spam category has common features (words), whereas the ham category could have multiple sub-categories, each with distinct feature sets. However, these distinct features can also be used to label an instance as "ham". Eg: If spam messages don't usually talk about sports scores, then the occurrence of the word "score", could be used to classify a message as "ham", even when it co-occurs with typical spam words such as "ad", "offer", etc. However, the one-class classifier cannot make use of such a mechanism.

In a similar manner, specific features will help to identify a news article as belonging to "sports", "religion", etc, thereby making them easier to label as "non-political". The miscellaneous class would be more diverse, however.

One-class SVMs are useful in cases where you want to detect novel instances, the kind of which you have not seen before, and hence cannot characterize in advance. Binary classification can be used when the understanding is that the available labeled dataset covers the typical kind of examples you would see while using the model for prediction.


Leave one out Cross validation using sklearn (Multiple CSV) I have 52 CSV files in a folder. I want to build a model based on this data. That's why I want to Leave one out cross-validation on these data. How can I do this using sci-kit learn in python?

I tried from sci kit document and also search many resources.

import glob
import numpy as np
import pandas as pd
from sklearn.cross_validation import LeaveOneOut
path=r'...................\Data\New
design process data'
filelist=glob.glob(path + "/*.csv")
loo=LeaveOneOut()
for train,test in loo.split(filelist):
   print("%s %s" % (train, test))

But it showed errors.

init() missing 1 required positional argument: 'n'

I am new in python as well as sci-kit learn. If anyone can help me, It would be a great convenience.

Leave One Out Cross Validation is just a special case of K- Fold Cross Validation where the number of folds =  the number of samples in the dataset you want to run cross validation on.

For Python , you can do as follows:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(classifier , X = input data , y = target values , cv = X.shape[0])

Here , cv = the number of folds . As cv = number of samples here , we will get Leave One Out Cross Validation.
The length of list(scores) will be equal to number of samples in input data.


how to augument speech sentiment dataset? I am building an LSTM to recognize if the person is sad, happy, angry or neutral. This is done by feeding-in his the wave of his voice into the network, as a sequence of bytes (each byte is 0-to-255).

The problem is, my dataset is not large enough, are there efficient ways I could augument my dataset? I am training on short 1.5 second clips and I have 800 of those, which is not enough.

My current augumentation is:

to add variations in volume
to add a bit of white noise, which makes it worse :(

Reversing the sequences doesn't seem to be applicable, after all, my network will be predicting non-reversed speech when it's fully trained.

Your problem is to identify whether a person is sad, happy, angry or neutral from his voice which is emotion classification problem. For speech, we use frames of short duration like 10-20 ms, and extract features from the same like MFCC or other frequency domain features. We extract short duration frames because speech is non-stationary over time, and frequency domain features make the features shift-invariant.

I suggest you read some latest research papers on emotion classification to get latest research in emotion classification.

Emotion in speech is captured by variation in pitch and amplitude across time. So capturing short time energy and instantaneous pitch frequency in speech across time  are basic features for emotion classification.

Modifying speech by adding variation in volume and white noise will not help at all.

You need to build a emotion classification system with 4 classes: sad, happy, angry or neutral, using the below steps:

Extract frames of 10-30 ms duration
Convert these frames to frequency domain features like STFT or extract instantaneous pitch
Use these STFT features along with pitch to train a LSTM


Number of parameters in an RNN I'm using a basic RNN as in the figure below (say for translation).
The model has the following structure:

\begin{aligned} s_t &= \tanh(Ux_t + Ws_{t-1}) \\ o_t &= \mathrm{softmax}(Vs_t) \end{aligned}

Assume m is the vocabulary size and n that of the hidden layer.
If $x_{t}=\{0,1\}^{m}$ and U is a $ n \times m$  matrix then W is a $ n \times n $ matrix.
If $o_{t}$ is $\mathbb{R}^{k}$ and $s_{t}$ is $\mathbb{R}^{n}$then V is a $ k \times n $ matrix.

What's the # parameters for this RNN model?

The entities W , U and V are shared by all steps of the RNN and these are the only parameters in the model described in the figure. Hence number of parameters to be learnt while training = $dim(W)+ dim(V)+ dim(U)$.

Based on data in the question this = $ n^{2}+ kn + nm$.

where,

n - dimension of hidden layer
k - dimension of output layer
m - dimension of input layer


When can embeddings be useful for small input spaces? I've noticed that some authors tend to use embeddings for almost everything. E.g. in the AlphaStar paper, or below an example for the OpenAI5 paper (link):

The Hero Embedding here encodes information about which hero out of 17 was picked. With 5 pickable heroes per game, that would yield a one-hot encoded vector of size 17 * 5 = 85. Instead, they chose to encode this information using an embedding. Does that not seem a bit overkill?

My understanding was that embeddings are very useful for large scale inputs, such as word-corpora, as they reduce the dimensionality nicely. What's the benefit of doing it on smaller inputs such as above? Is it just convenience? Or maybe I am misunderstanding what embedding means here?

EDIT: I wanted to add that EVEN if the Hero Embedding contains more unique information, such as health, mana, etc, then it would still be very small, likely under 200 values. So when does it make sense to start using an embedding, as opposed to using the values directly? What's the underlying thought?

The advantages of embedding is converting categorical variables to numeric variables. Those numerical variables have semantic meaning. Thus any valid mathematical operation for numerical values has interpretable meaning.
One-hot encoding just tells the model that two entities are different. Embedding tells how different the entities are.


Intuition for the regularization parameter in SVM How does varying the regularization parameter in an SVM change the decision boundary for a non-separable dataset? A visual answer and/or some commentary on the limiting behaviors (for large and small regularization) would be very helpful.

The regularization parameter (lambda) serves as a degree of importance that is given to misclassifications. SVM pose a quadratic optimization problem that looks for maximizing the margin between both classes and minimizing the amount of misclassifications. However, for non-separable problems, in order to find a solution, the miclassification constraint must be relaxed, and this is done by setting the mentioned "regularization".
So, intuitively, as lambda grows larger the less the wrongly classified examples are allowed (or the highest the price the pay in the loss function). Then when lambda tends to infinite the solution tends to the hard-margin (allow no miss-classification). When lambda tends to 0 (without being 0) the more the miss-classifications are allowed.
There is definitely a tradeoff between these two and normally smaller lambdas, but not too small, generalize well. Below are three examples for linear SVM classification (binary).

For non-linear-kernel SVM the idea is the similar.
Given this, for higher values of lambda there is a higher possibility of overfitting, while for lower values of lambda there is higher possibilities of underfitting.
The images below show the behavior for RBF Kernel, letting the sigma parameter fixed on 1 and trying lambda = 0.01 and lambda = 10

You can say the first figure where lambda is lower is more "relaxed" than the second figure where data is intended to be fitted more precisely.
(Slides from Prof. Oriol Pujol. Universitat de Barcelona)


Finding orthogonal input patterns associated with logistic function output I've been given this problem but cannot seem to get an analytical solution. I've tried satisfying the logistic function with several vectors but have difficulty finding ones which are also orthogonal. The problem begins with me being given a neural unit weight vector,
$$w =\begin{bmatrix}
           1 \\
           \frac{1}{4} \\
           \frac{1}{9} \\
         \end{bmatrix}$$
The neuron output u is related to its input pattern v∈R3 by
$$u = f(w^Tv + 1)$$
where
$$f(x) = (1 + e^{-x})^{-1}$$
To find a pair of orthogonal vectors, v1 and v2, for which the unit output $u_{i} = 0.5, i=1,2.$

So far I know that $w^TV = 0$ for the function output to be 0.5.

All you have to do is complete $\omega$ to an orthogonal $\mathbb{R}^3$ basis, as $\omega^T V = 0$ and $v_1$, $v_2$ are orthogonal. You can do this using Gram Schmidt orthogonalization. The two other vectors of the basis will be $v_1$ and $v_2$.

Edit:

The initial vectors to do the Gram Schmidt process can be any pair of independent vectors, for instance $(1, 0, 0)$ and $(0, 1, 0)$.


Free parameters in logistic regression When applying logistic regression, one is essentially applying the following function $1/(1 + e^{\beta x})$ to provide a decision boundary, where $\beta$ are a set of parameters that are learned by the algorithm, and $x$ is an input feature vector. This appears to be the general framework provided by widely available packages such as Python's sklearn.

This is a very basic question, and can be manually implemented by normalization of the features, but shouldn't a more accurate decision boundary be given by:  $1/(1 + e^{\beta (x - \alpha)})$, where $\alpha$ is an offset? Of course an individual can manually subtract a pre-specified $\alpha$ from the features ahead of time and achieve the same result, but wouldn't it be best for the logistic regression algorithm to simply let $\alpha$ be a free parameter that is trained, like $\beta$? Is there a reason this is not routinely done?

You get the same effect from including a bias term, i.e. $\frac{1}{1+\exp(-\beta x + bias)}$, and the defaults for most software is to include such bias.

To see why, you can do the math, expanding $\beta(x-a)$ and seeing that you can express the sum of those differences as a single variable.


Decision tree ordering I am interested in finding out how decision trees chose the order in which they split. I understand that splitting is based in information gain. The attribute with the lowest information gain is chosen as the root node.

If I had a data set with columns:

credit standing
age
income,
marriage status

and I was interested in finding out what determines a person to have a good or bad credit status, Am I correct in saying, I calculate entropy and information gain for each of these categorical attributes against the independent attribute i.e what I am investigating (credit standing), and that the calculation with the lowest information gain is chosen as the root node.

For example, if this root node (first split) was Age, Is the entropy and information gain of Age (the new independent attribute) against the remaining attributes (marriage status and income) calculated, and the calculation with the lowest information gain is then chosen as the second split node, and so on?

ie.

information gain:

credit standing vs age    = 0.01
credit standing vs status = 0.1
credit standing vs income = 0.2

Age is chosen as root (first split) node.

then, information gain:

age vs status = 0.2
age vs income = 0.1

income is chosen as second split node.

Am I understanding this correctly?

Here in your Scenario you need to select the one with more Information Gain rather than the least one and the process goes on till you reach the last feature/last node.

Go through this Link. I think your doing it vice versa, I agree with Emre.

In the above link it was explained with an example to decide whether to play tennis or not.

Do let me know if you have any issues.


Sort numbers using only 2 hidden layers I'm reading the cornerstone paper Sequence to Sequence Learning
with Neural Networks by Ilya Sutskever and Quoc Le. On the first page, it briefly mentions that:

A surprising example of the power of DNNs is their ability to sort
N N-bit numbers using only 2 hidden layers of quadratic size

Can anyone briefly outline how to sort numbers using only 2 hidden layers?

Doing some research, I found a paper which proves that sorting can be done with at most 3 layers, and that their solution is optimal if you restrict the size of the network to be polynomial w.r.t. to the number of input numbers:

Depth Efficient Neural Networks for Division and Related Problems, see Theorem 7 on page 955 (page 10 in the PDF).


Should the different layers of deep learning models have same size or they should be changed based on a rule I see a lot of people varying the width of each layer in a deep neural network. ie. Input->50->100->150->Output. I'm curious what, if any, are the advantages of this structure over static layer widths ie. Input->50->50->50->Output.

Given the following three model structures what would be the real-world implications and differences between them and how they interpret data?

Example 1:

  Input -> 100 -> 100 -> 100 -> Output

Example 2:

  Input -> 50 -> 100 -> 150 -> Output

Example 3:

  Input -> 150 -> 100 -> 50 -> Output

Actually it is not completely clear which deep neural nets you are referring to but I guess you are referring to Dense, aka fully connected, networks. It depends on your data but for simplification, based on the answer here, neurons of the first hidden layer try to find lines for separating the data, and the neurons of the subsequent layers try to consider them simultaneously, I suggest reading the hole answer. As you increase the number of neurons in the first layer, you try to increase the number of lines for separation, The last shape of the link, and as you try to increase the number of e.g. the second layer, the number of convex shapes will increase. If you increase them too much you will over-fit your data.

Increasing the number of filters, neurons or kernels, in convolutional nets has another meaning. If you increase the number of filters, it means that you try to find more features that can represent your data better. In conv layers you actually try to find features of data. Although you are somehow doing same procedure in a dense net but because the neurons are less connected and they get updated similarly in conv nets, the features learned by conv nets are completely different from dense layers. for more information I recommend you taking a look at here.