Topic modeling for short length sentences I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like org.java.somepackage.validateLogin). What I want to do is to give a representative label for each cluster.

For the sake of simplicity, let's assume I'm able to clean the data (i.e. break 'validateLogin' to 'validate' and 'login')

I've done a little research on the subjects of topics modeling and cluster labeling and encountered a few algorithms, such as: LDA, NMF and TF-IDF (not quite an algorithm on itself)

Basically, many algorithms are documents oriented which are reach of words and not short-text/labels oriented.

It is worth mentioning:

We can use the fact that different clusters might have different labels, so maybe a proper label for a cluster could be unique words in the overall bag of words (that can be done with TF-IDF I guess)
Labels can be one word but can also imply a hierarchy (i.e. packageA.packageB.packageC.funcName)

I'd be glad if you could give me your insights on this problem and what approach could fit here.

A possible approach might be to use the most predictable and predictive word(s) in each cluster as its name(s). The following is inspired by Fisher's category utility used by the COBWEB algorithm.

An attribute is highly predictable in a cluster $C_l$ if most elements in that cluster have the same value $a$ for it, thus a word represented by attribute $A_i$ has a predictability of $P(A_i=a|C_l)$

An attribute is highly predictive for a cluster $C_l$ if knowing it's value $a$ implies you can say with high certainty to which cluster it belongs, expressed as $P(C_l|A_i=a)$

Assume now you process each label org.java.somepackage.validateLogin as a sentence: "org java somepackage validate Login" and apply one-hot encoding to all your sentences in the dataset. The occurrence of a word is now represented by a value of 1 for it's corresponding attribute.

The task of representing each cluster by a word can now be formulated as finding for each cluster $C_l$ the word represented by attribute $A_i$ that equals 1 and has the highest predictability and predictiveness, weighted by the total probability of this word being in a sentence.

$$ P(A_i=1|C_l)\times P(C_l|A_i=1) \times P(A_i=1)$$

Which can be calculated by counting occurrence of a word per cluster and cluster per word.


Feature Selection - Conditional Entropy I've developed an algorithm to define conditional entropy for feature selection in text classification. I'm following the formula at Machine Learning from Text by Charu C. Aggarwal (5.2.2). The author mentions that Conditional Entropy values are between (0, log(number of classes)) in which my case is (0, 0.6931472).
The author also mentions that features with the largest values can be removed, but he don't give further information about the criteria to define 'largest' (is it only the max value of entropy or a set of the largest entropy values?)
Have you ever guys applied Conditional Entropy for feature selection? If so, based on results, what criteria was used to define features to be removed.
Here a summary of my Conditional Entropy results:
        E.tj.
 Min.   :0.5701
 1st Qu.:0.6562
 Median :0.6563
 Mean   :0.6558
 3rd Qu.:0.6564
 Max.   :0.6564

Individual feature selection methods assign a numerical value to every feature so that features can be ranked according to this value. The calculated value is chosen to represent how much the feature contributes to knowing the label/response variable: common choices are conditional entropy, but also information gain or correlation.
The actual values assigned to the features are not really useful on their own, what matters is the ordering of the features according to this value. Thus the standard method for selecting the features is not to choose a particular threshold on the value, but simply to choose the number of features one wants to obtain.
Example: in a text classification task, there are 1000 documents and a vocabulary of 20000 unique words as candidate features. Using all the words would certainly cause overfitting, so we decide to use only 100 words as features. We can calculate the conditional entropy of every word with respect to the label, and then select the bottom 100 words according to the corresponding ranking as features (the 19900 other words are ignored).
Since individual feature selection is very efficient, it's often possible (and a good idea) to try a range of values as the number of features, and train/test the model for each of these values. This way one can experimentally determine the optimal number of features (the one which maximizes performance on the data). Note that this a form of hyper-parameter tuning, and therefore one has to use a validation set for the tuning stage, and then the final model (with the selected optimal number of features) should be applied on a fresh test set.


Binning which variables? I will try to implement a k-means algorithm over this dataset:

Team_Categorical  CreditAmount_Numeric   Retired?_Binary
       A                 15.3                   1
       B                 12                     0
       C                 6.2                    1

In order to apply k-means algorithm I need to transform the categorical variables into numeric. So I will apply binning transformaton over the Team_Categorical field. So I will have this dataset:

A    B   C  CreditAmount_Numeric   Retired?_Binary
1    0   0          15.3                 1
0    1   0          12                   0
0    0   1          6.2                  1

My question is: Should I transform CreditAmount_Numeric to binary too?

To answer your question, you need not transform the numeric to binary variable(you meant binning right?).

I will try to explain it with the above example:

Let's start with row-1 where A = 1 , B =0 , C=0 so it means that you are talking about A. So the value of CreditAmount_Numeric belongs to A and the retired binary value also belongs to A. Similarly, you can deduce for the rest records.

Conclusion is that you need not do any transformation on your numeric variable.


How to learn noSQL databases and how to know when SQL or noSQL is better I want learn about NoSQL and when is better to use SQL or NoSQL. I know that this question depends on the case, but I'm asking for a good documentation on NoSQL, and some explanation of when is better to use SQL or NoSQL (use cases, etc). Also, your opinions on NoSQL databases, and any recommendations for learning about this topic are welcome.

Check Martin Fowler's Personal website. He writes good and specially answers one of your questions in his book: "NoSQL Distilled"


Is this matrix correctly built? I was reading an article called "Ensemble learning in recommender systems: combining multiple user interactions for ranking personalization" where they explain a method they use called "BPR Optimization Criterion" to develop their own algorithm, and also they explain that BPR's authors apply a technique to calculate the order of relative item-item relevance for the same user to obtain a personalized ranking.

Basically, the first matrix contains only positive interaction of the user and the item, '?' is used for unknown interaction that may be positive, negative or not interaction at all. Then they compare each pair of items for each user, so if the user has interaction with the item i and has not interacted with the item j, then the user prefer i over j. But when they express that on the secondary matrix, it's all the opposite, it appears a minus sign for interaction i1,j2 and i1,j3.

So I wonder if the image is wrong or I'm missing something. Thanks.

Looks simply wrong to me. The second 'matrix' should be transposed.


EM clustering with missing and misspelling data I am currently working on a project that requires me to cluster the unlabeled input. The records contain personal information such as name, DOB, height, sex, etc. We need to cluster the same person in one group, here is the sample data:

+------------------------------------+
|        Record1     Record2         |
+------------------------------------+
| First Name   'Harry'     'Harry'   |
| Middle Name  'Jay'       'J'       |
| Last Name    'Potter'    'Potter'  |
| DOB Month    1           1         |
| DOB Day      1           1         |
| DOB Year     1993        1993      |
| Dr License   'A1234567'  Null      |
| Sex          1           1         |
| Address      Hogwarts'  'Hagwarts' |
+------------------------------------+

Those two records should be clustered in the same group.
I am still in the beginning phase of the project. I want to try with EM clustering using Gaussian Mixture model. But I do not know how I should preprocess the string, or should I do it and how to deal with the Null value.

Convert your key-Value Table to a table in wide format (1 row per person) , then calculate the distance matrix using the Jaccard Distance (which can convert categorical values into numeric values)

library(dummies)
dummy_dat <- dummy.data.frame(my_data)

jaccard_dist <- dist(dummy_dat, method = "binary")

#  Distance Matrix
jaccard_dist


Chi-square as evaluation metrics for nonlinear machine learning regression models I am using machine learning models to predict an ordinal variable (values: 1,2,3,4, and 5) using 7 different features. I posed this as a regression problem, so the final outputs of a model are continuous variables. So an evaluation box plot looks like this:

I experiment with both linear (linear regression, linear SVMs) and nonlinear models (SVMs with RBF, Random forest, Gradient boosting machines ). The models are trained using cross-validation (~1600 samples), and 25% of dataset is used for testing (~540 samples). I am using R-squared and Root Mean Square Error (RSME) to evaluate the models on test samples.  I am interested in finding an evaluation measure to compare linear models to nonlinear.

This is done for a scientific research. I was pointed out that R-square might not be appropriate measure for nonlinear models, and that Chi-Square test would be a better measure for goodness of fit.

The problem is, I am not sure what is the best way to do it. When I browse Chi-square as goodness of fit, I only get examples where Chi-square test is used to see whether some categorical samples fit a theoretical expectation, such as here. So here are my considerations/questions:

One way I could think of is to categorize predicted (continuous) values into bins, and compare predicted distribution to the ground truth distribution using Chi-Square test. But that doesn't make much sense, i.e. we have a machine learning model that perfectly predicts ground truth values 2,3, and 4, and values 5 predicts as 1, and values 1 as 5 - Chi-Square test that I propose here would reject null hypothesis, although the model is mispredicting 2 out of 5 values.
As referred in a tutorial from USC I could use formula (1) to compute Chi-Square value, where experimentally measured quantities (xi) are my ground truth values, and hypothesized values (mui) are my predicted values. My question is, what is the variance? If we observe each value 1,2,3,4, and 5 as a distinct category, than the variance of ground truth within each category is equals to zero. Also, how one computes degree of freedom (N-r)?
Related to the statement I am interested in finding an evaluation measure to compare linear models to nonlinear is Chi-Square test the best (or even good) choice? What I've seen so far in machine learning competitions for regression tasks, either MSE or RSME are used for evaluation.

Use your test data to compare the predictive performance of each model.

In R you could do this like:

linear.predictions <- predict(linear.model, newdata = test.data)
nonlinear.predictions <- predict(nonlinear.model, newdata = test.data)

linear.percent.difference <- (test.data$TARGET_VARIABLE -
                              linear.predictions)  /
                              test.data$TARGET_VARIABLE

nonlinear.percent.difference <- (test.$TARGET_VARIABLE -
                                 nonlinear.predictions) /
                                 test.dtat$TARGET_VARIABLE

linear.grade <- mean(linear.percent.difference)
nonlinear.grade <- mean(nonlinear.percent.difference)

This is a pretty simple way to do it, but it is one that works for me and is easy to understand, especially if your audience is going to eye-glaze as soon as you say "Chi-square..." Get creative!


Is One Hot Encoding Vectorized? Sorry for such a weird question - I do not even know if this makes sense, however I thought of this during my intro to Python course at uni and have wondered about it since.

So I have some experience with Python for data science, but have never taken a structured course. I am in an intro class, and on the first day we were asked to count how many times a certain name appeared in a list. Obviously the answer I came up with was to initialize a variable to zero then iterate through the list and add one to itself whenever the i'th name in the list was equal to the target name.

However, in Andrew Ng's deep learning courses, there is a heavy emphasis on vectorizing our calculations whenever possible. This got me wondering how I could vectorize this task.

What I came up with, in an ideal scenario, is to create a vector where each the target name is replaced with 1, and the other names are replaced with 0. Then I could just take the sum of the vector and I would have my answer. The problem with this is the only way I know how to create said vector is to iterate through the original list, defeating the purpose this way of solving the problem.

Anyways, while I understand that creating this vector is not exactly one hot encoding, is there any way to vectorize this process? Is one hot encoding done iteratively. If not, are there other examples where basic iterative tasks can be turned into tasks able to be processed in parallel?

Sorry if this is a dumb question or does not make any sense - just something I was curious about as a noob to base Python.

You can read the sklearn documentation here. If you click in source you can check the code.
If you want to do one hot encoding you can do
    >>> from sklearn.preprocessing import OneHotEncoder
    >>> enc = OneHotEncoder(handle_unknown='ignore')
    >>> X = [['Male', 1], ['Female', 3], ['Female', 2]]
    >>> enc.fit(X)
    OneHotEncoder(handle_unknown='ignore')
    >>> enc.categories_
    [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)]
    >>> enc.transform([['Female', 1], ['Male', 4]]).toarray()

You should only try to implement this if you are an experimented developer. A couple lines of code that implement OHE in a non optimal way is:
for col in columns_encoding:
    variables = df[col].unique()
    for v in variables:
        df[col + str(v)] = [1 if row == v else 0 for row in df[col].values]

where columns_encoding is a list of the columns you want to encode and df your dataframe


How can the labels of AgglomerativeClustering be re-computed? I'm using scikit learn's AgglomerativeClustering on a large data set.
I want to modify the distance_threshold after the model has already been computed. Computing the model is slow (quadratic time) but it should easily be possible to re-compute the labels for a new distance_threshold in linear time because the model stores the children_ and distances_ arrays permanently. But how can the labels be re-computed for a different distance_threshold?
It can be assumed that distance_threshold was originally set to 0, i.e. the entire tree was computed.

One option to speedup computation with different thresholds is caching the results using the memory option.
Something like this:
from sklearn.cluster import AgglomerativeClustering

clustering = AgglomerativeClustering(compute_full_tree=True,
                                     distance_threshold=1.0,
                                     memory='mycachedir',
                                     n_clusters=None,)


Rolling window features for multiclass classification I'm doing a multiclass classification and data is considered as not being a time-series. Working on a feature engineering and trying to solve the problem with classic KNN, RF, boosting etc. I'm creating new features based on rolling window and found out that people usually do mean aggregation.

Does the mean characteristic of rolling distribution is the most informative?
Are there any other characteristics (like std, quantilies, IQR, etc.) that may be used?
Is it worth to perform some kind of data transformation (like scaling, quantile transfomation or Box-Cox)?

Usually the point of a rolling window is to calculate and use a moving average:

A moving average is commonly used with time series data to smooth out short-term fluctuations and highlight longer-term trends or cycles.

By definition the only meaningful aggregation for this kind of purpose is the mean: the idea is to represent the same type of value as the original series while correcting noisy variations.
Anything else is not the standard use case for a rolling window, so the answer to questions 2 and 3 would completely depend on the application.


How to avoid covariate shift in python and distribute classes in each train and test phase? We all know that with the use of sklearn package from python, we can create X_train, X_test, y_train and y_test via this code:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

I want to make sure that each training and testing phase of a multi-class data set, have 66.33/33.33 percent of each class values so the prediction and accuracy would get better. All i want is 66.33 percent of class A in training set and 33.33 percent of Class A in test set. And, so on for other classes, like B, C, D and etc. in a given multi-class data set.

Is the code provided enough to achieve this or should i write extra code?

Thanks

You need to call train_test_split(X, y, test_size=0.33, stratify=y) assuming that y contains your class labels. Alternatively, you can call StratifiedShuffleSplit directly.


use .apply() function to change values to a column of the dataframe I have a dataframe which is the following:

and I would like to consider only the column of instructions and keep just the values push, test, mov, test ,....., so just the first word of each string inside each list. What I am doing is the following:

dataFrame['opcodes'] = dataFrame['instructions'].apply(instruction.split()[0] for instruction in dataFrame['instructions'])

but I get the following message:

TypeError: 'generator' object is not callable

so, my objective is to change the values only of the column instructions.

Can somebody please help me? Thank's in advance.

use lambda function like as follows

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()[0] for i in x])


How to chose a Machine Learning algorithm? I was wondering, are their any guidelines or any rules of the thumb as to which algorithms perform best for each task?

What I'm looking for is something along the lines of:

  NLP tasks are usually accompanied highly-dimensional and sparse data. Algorithm X performs well under these circumstances.

etc.

I would totally start here. You're going to want to tweak things when they don't perform as well as you want, but it gives you a great starting point.


Why can distributed deep learning provide higher accuracy (lower error) than non-distributed one with the following cases? Based on some papers which I read, distributed deep learning can provide faster training time. In addition, it also provides better accuracy or lower prediction error. What are the reasons?

Question edited:

I am using Tensorflow to run distributed deep learning (DL) and compare the performance with non-distributed DL. I use the number of dataset 1000 samples and step size 10000. The distributed DL uses 2 workers and 1 parameter server. Then, the following cases are considered  when running the code:

Each worker and non-distributed DL use 1000 samples for training sets, same mini-batch size 200
Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2), non-distributed DL use 1000 samples for training sets, same mini-batch size 200
Each worker uses 500 samples for training sets (first 500 samples for worker 1 and the rest 500 samples for worker 2) with mini-batch size 100, non-distributed DL use 1000 samples for training sets with mini-batch size 200

Based on the simulation, for all cases, distributed DL has lower RMSE than non-distributed DL. In this case, the RMSEs of distributed DL are as follows: Distributed DL in Case 2 < Distributed DL in Case 1 < Distributed DL in Case 3 < Non-distributed.

In addition, I also add the training time (i.e., the number of steps is 2 x 10000) for non-distributed DL, the results are still not as good as distributed DL.

One reason can be the mini-batch size, however, I wonder the other reasons why the distributed DL has better performance using the aforementioned cases?

About the accuracy: Going with the strongest reason; memory problems will diminish due to the distribution of the computation. That will allow you to increase your training batch size which will reduce the gradient noise due to small mini-batch sizes. The steeper gradient moves will be towards the minima, with less noise.

You can refer to this video for deeper understanding: https://www.youtube.com/watch?v=-_4Zi8fCZO4&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&index=16

About the speed: It is more obvious I think. You distribute your gradient descent computations to multiple machines or CPUs/GPUs/TPUs, so a faster training speed you acquire as a result.


How to add non-image features along side images as the input of CNNs I'm training a convolutional neural network to classify images on fog conditions (3 classes). However, for each of about 150.000 images I also have four meteorological variables available that might help in predicting the classes of the images. I was wondering how I could add the meteorological variables (e.g. temperature, wind speed) to the existing CNN structure so that it can help in the classification.

One way I can already think of is creating another (small) feedforward neural net alongside the CNN and then concatenating the outputs of the CNN layers and the hidden layers of the non-image neural net to each other at the dense layer.

The second way I could think of is just contacting these features to the dense layer. However, in this case, the non-image variables will (I think) only be able to make linear predictions.

Are there any other (better) ways that the non-image features could be included in the model? And what would be the advisable method considering the amount of data I have?

Another question I have is whether or not I should unfreeze the convolutional layers while training with these non-image features? These layers of a Resnet-18 (which were initialized as pre-trained on ImageNet) have already been fine-tuned using the images. My guess is that I should keep them frozen and only unfreeze the dense layer since it is only here that the non-image features come into 'contact' with the image features (not earlier in the CNN). If I'm wrong on this, please say so!

My solution is like your first recommendation, but with slight changes.

Construct your convolutional layers and stack them till the flatten-layer. This network should be fed with the image data.
Flat your activation maps
Construct a fully connected network with the desired number of neurons and layers.
Concatenate the outputs of the flattened layer of the convolutional net and the fully connected net.
add some dense layers and connect them to the last layer which represents your classes.

You can use customary cost functions for this architecture.


Error Analysis for misclassification of text documents I am working on a text classification work. The purpose of this work is to classify whether a particular document belong to class A or Class B.

I used KNN algorithm and i am able to get some decent results. However I want to know two things.

Why a particular document has been classified as Class A or Class B? What keyword or information that made a document to be classified as such?
How to perform mis-classification analysis?

Kindly help.

It seems to me that both of your questions could be answered by storing the retrieved neighbours on your test set and giving them a thorough analysis. Assuming you are using a unigram + tf-idf text representation and a cosine similarity distance metric for your K-NN retrieval, it would be trivial once you have a classified document to display the K neighbours and analyze their common unigrams and their respective tf-idf weights in order to see what influenced the classification. Moreover, doing it on your misclassified documents could help you understand which features caused the error.

I'd be interested to know if there is a more systematized approach to those issues.


If a categorical feature only occurs a few times in a data set, should I drop it? I have a data set of mostly categorical variables. When I one-hot encoded them some of the features occur less than 3% of the time.

For instance the Tech-support feature only occurs 928 times in a data set with 32561 samples ie. it only occurs 2.9% of the time.

Is there a general cutoff point for when I should scrap these variables? I'm cleaning up this data set for binary logistic regression and an SVM.

Thank you!

Indeed it's often a good idea to remove boolean features which are very rare, but the problem is that choosing a threshold by intuition is not necessarily optimal. Whenever possible the optimal value should be determined experimentally, and typically that should be possible for efficient methods such as log regression or SVM. The idea is simply to consider a range of values as threshold and run a grid search using a subset of the training set. The threshold is exactly like an hyper-parameter for the learning method.


Comparison between cost functions to determine the "best" model? I'm building an LSTM neural net for time series prediction (regression) and I am incorporating custom loss functions into training. I'm trying to determine which cost function (of 3 cost functions) gives the "best" model, in other words, trying to define what "best" means.
The 3 cost functions yield results that are all on different scales, and in addition 2 produce positive numbers while the last produces negative numbers.
I have 5 datasets and I am training one model per dataset. With 3 losses, this gives me 3 * 5 = 15 losses in total. On validation data, the results look something like the below.
         | loss1 | loss2 | loss3
==================================
data1    | 1.106 | 5.074 | -1.872
==================================
data2    | 1.067 | 2.390 | -1.903
==================================
data3    | 0.823 | 4.724 | -1.892
==================================
data4    | 1.157 | 4.809 | -2.233
==================================
data5    | 0.583 | 2.854 | -2.120
==================================
Average  |   x   |   x   |   x
==================================

My goal is to somehow compare how effective a model trained with loss1 is compared to a model trained with loss2 on predicting out of sample data.
One procedure I have tried is to standardise each loss: (loss - mean(loss)) / std(loss) across datasets, then take an average across datasets for each loss and check which is smallest. However with a small sample size of 5, I don't know if this is valid. The average, after standardizing, could be a simple average, or a geometric average, or a harmonic mean. If I apply this method to the table above, I get these results
                   |  loss1   |  loss2   |  loss3
========================================================
Simple Average     | -3.9e-16 | -3.9e-16 | -9.7e-16
========================================================
Geometric Average  | 0.82     | 0.93     | 0.90
========================================================
Harmonic Average   | 2.69     | 2.36     | 2.55
========================================================

We can see that depending on the type of average chosen, a different loss is considered best. For simple average loss3 is best, for geometric loss1 is best, and for harmonic loss2 is best... sort of confusing.
Is this a valid way to compare model performance across different loss functions, or is there another method that would be more suitable to determine the "best" model?
An alternative would be to look at an accuracy metric across models (the same accuracy metric for all losses and datasets) and check which loss yields the highest accuracy.

tl;dr since you are referring to accuracy, I'm guessing you have a classification task.
There is no way to evaluate a model's actual performance, by its loss.
The goal of the loss function is to train the model and not to show how well this model classifies.
What you should do to see the best model is to evaluate it by a classification metric (e.g. accuracy). You should note that it is best to measure these on the validation set to avoid overfitting.

Some things to consider...

Different loss functions, don't measure the same thing. For example take into consideration MAE and MSE. MSE penalizes larger errors disproportionately, compared to MAE. Which is the best? Depends on the problem and what you want!

Classification depends on your model's predictions, which are by their nature continuous. However, when we are referring to the "best model" we want the one that classifies better. This can cause misleading effects, because a model can reduce its loss by becoming more "confident" in the samples it classifies as correct (usually due to overfitting).

You can look at the following as an example of this:
label | model1 | model2
  0   | 0.995  |  0.5
  0   | 0.400  |  0.5
  1   | 0.600  |  0.5
  1   | 0.600  |  0.5

model1 is a better classifier ($3/4$ correct predictions), but model2 ($2/4$) would have the smallest loss*.
*technically this depends on the loss function, but this stands on MAE, MSE and cross entropy losses.


AlexNet CNN how can it be applied to my case? I'm working on my last year project where I'm given digitized WSI (Whole Slide Images), though they're fairly small around 1390x1040 size (which is unusual). These images are of cases of Glioblastoma Multiforme (brain cancer) which is stained with Ki-64 index, which results in what I assume malignant parts being marked as brown. Here's a small example of what I'm looking at.

My objective in simple terms is to count the blue and brown cells (estimation of proliferation indices) which based on what I've researched is a segmentation problem. I've also came to conclusion that AlexNet is a successful architecture for this purpose. However, the trouble I'm having is that this data is unlabeled and as a Comp Sci student, I don't think I have enough expertise to annotate the ground truth values. My question the boils down to this, are there any alternative methods should I explore, or should I drop this dataset altogether?

Potentially what you could do is use a GMM with 3 modes to cluster the image into 3 "partitions". The first would contain all the pixels with blue, the second all the pixels with brown, and the final should contain the background pixels. You would cluster based on the raw RGB values (unless you could calculate some more semantic deep learning based feature for each pixel to cluster). You would then do some form of colour sampling on the pixels assigned to each of the 3 modes to determine which are the brown blobs and which are the blue blobs. You can count easily by computing the binary image from the GMM blobs.


Master degree in Data Science I am looking to change careers and would appreciate some advice.

I have an undergraduate degree in English Literature and a JD. Needless to say, these were not the best decisions and I would like to change my career. I have always enjoyed math and science, and after months of research and self study, I have decided that I would like to pursue statistics.

My question is basically this : would it be better to get another Bach. Degree in statistics or should I take calculus 1-3, linear algebra, probably and statistics and some computer science courses at a community college then try to get into a grad program instead?

I am afraid that without the bachelors degree I will not have the required knowledge for a masters even after taking the courses at my local c.c. However, another bachelors degree may be a worthless waste of time.

Please share your thoughts. Also, sorry if this is not the right place for this question.

Data science is a multi discipline subject.
In addition to math you need some programming skills and ideally - understanding of the domain where you are going to solve problems.

You can start from online-courses and some introduction literature.
Look at the answers at current site:

Data Science Sertifications
Books about Data Science
Data Science vs Data Mining
Data Science is a trend or a long term concept
Ph.D. in Data Science?
Statistics + Computer Science = Data Science?
Starting career as Data Scientist, is Software Engineering experience required?
Career switch to Big Data Analytics
And more about career...

I can't resist from placing this illustration for the millionth time:


What are the 'hottest' future areas of Machine Learning and Data Science? What would you guys identify as the most important areas of machine learning and data science in the future? What kind of business do you see companies focused on ML doing in the future?

To give some background to the question, I'm asking this as someone hugely interested in ML at a conceptual level but with no real experience on it save for some online courses. My intention is to learn the necessary math, coding, etc. and as I'm very entrepreneurial person, eventually found my own ML company. I've found that the field is vast and thus I feel one of the biggest challenges is to select a specific area (I guess method would be a better word here) to focus on and build expertise.

There is a brilliant answer by Yann LeCunn recently on a Quora session.
What are some recent and potentially upcoming breakthroughs in deep learning? As he quotes, adversarial training proposed by Ian Goodfellow is one of the hottest area. Apart from that, I think Memory Networks, RNNs are widely used right now and can solve various problems up ahead.

Coming to what companies might do, I see a lot of text analytic companies. Considering the amount of consumer data we have now, there is a great need to be able to understand and analyze it. Computer Vision using ML or Deep learning is fast catching up. Using cameras for face detection, object tracking, fraud detection, tracking number plates is a huge field for companies to handle.

Also, I think this might help you to prepare. To catch up with the overlooked developments, refer this which might give you an insight about what can develop in the future.


Removing Categorial Features in Linear Regression This is more of a design question regarding linear regression. Here is some info on our dataset:

Our dataset has 8 features; 3 of them being categorical. We are willing to perform linear regression to fit our target data.
We have tried including all of our 8 features (categorical ones being encoded in integer) and doing the linear regression.
We have also tried taking out the categorical features and running the linear regression algorithm for each possible combination of our 3 categorical features. That of course yields in a lot of regression runs (each category has 4 possible values; so 64 to be exact).

The latter approach gave us better results. So, fixing the categorical features and creating a new dataset for each combination turned out to give better estimations.

What does this tell us about our data?

Categorical features make the data non-linear, so they should indeed be taken out? For example, there is a complex relationship between categorical feature A and categorical feature B that can't be captured in linear regression?
If I am forced to step out of linear regression, which algorithm should I try to apply? I would prefer to have one dataset with 8 features instead of having 64 datasets with 5 features. There should be an algorithm that can capture this model.

I think using Linear Regression is not a good option as,

This performs very well on numeric variables(categorical -> binary).
Cannot handle Missing Data(suggestible to ignore those records).
When you are trying to predict, By chance a category is not trained then it cannot predict.
For example: A variable has 4 categories and in the random sample it
could pick only 3 categories, if the test has 4th category then it
will throw an error. So, we should be careful when the data is
divided between Test and Train

Now, what are the other algorithms which are available:

Random Forest(you cannot use this when any category variable has more than 52 categories, in your case it shouldn't be an issue)
XGBoost

Do let me know if you need any additional explanations.


How to combine different models in Keras? I have a pre-trained network, consist of two parts, the feature extraction, and the similarity learning.

The network takes two inputs and predicts the images are same or not.

The feature extraction part was VGGNet 16 with all layers freezed.

I only extracted the feature vectors and learned the similarity network which consists of the two convolutional layers followed by four dense layers.

Note: Removed last layers from the image due to large size.

Now, I want to fine-tune the last convolutional block of VGGNet and want to use two different VGGNet feature extractors for each type of image.

I have loaded the trained model and created a new model which starts from the Merged_feature_map layer:

model = Sequential()
for layer in ft_model.layers[3:]:
    model.add(layer)

Now, the new model will only contain similarity network without the feature extraction part.

I have loaded two VGGNets for each type of image and unfreeze their last convolutional block as:

vgg_left = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=(img_width, img_height, channels)))
vgg_left.name = "vgg_left"

vgg_right = VGG16(weights="imagenet", include_top=False, input_tensor=Input(shape=(img_width, img_height, channels)))
vgg_right.name = "vgg_right"

At that moment, I have 3 models and I want to combine them. The output from both VGG networks should be the input of Merged feature map. How to combine them and make them a single model.

bottleneck_features_r = vgg_left(left_input)
bottleneck_features_s = vgg_right(right_input)

It should be like:

You can get the output of your models with model.output or get_layer
and combine them with tf.keras.layers.concatenate


Confusion Matrix of LSTM using Sklear getting error I want to calculate the confusion Matrix of my LSTM model. Shape of y_test= (17799,1) y_Pred= (17799,1) I used thefollowing code:from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)getting following error: raise ValueError("{0} is not supported".format(y_type))

ValueError: continuous is not supported
both y_test and y_predict are normalized using minmaxscaler(0,1)
first few rows of both varaibles looks like:
y_test={0,0,0,0.0216212}
y_predict={0.000433061,0.000433061,0.000433061,0.000450924}

Please suggest how to calculated confusion matrix, tp,fp,tn,fn .

A confusion matrix can be drawn for a classification problem, where the Machine Learning Model (in your case LSTM) predicts the target variable into one of the N classes.

Can you confirm that your problem is a Classification problem and not a Regression one?

If in case you have a regression problem, then you might want to use a different evaluation metric.


How to model predict on Tensorflow model that has feature columns? So far all the coding examples using feature columns do not have examples of how to format their model.predict(...).  I tried using raw string and also putting them into np.arrays.  I am very new so please be gentle :).
Here is the official page, as you can see, no predict example:
https://www.tensorflow.org/tutorials/structured_data/feature_columns

You can find prediction examples in the next tutorial: Classify structured data using Keras Preprocessing Layers.
The list of tutorials in the series is in the bottom left part:

Specifically, the prediction examples are in the "Inference on new data" section (the last one in the tutorial).


Feature Importance based on a Logistic Regression Model I was training a Logistic Regression model over a fairly large dataset with ~1000 columns.

I did apply scaling of features using MinMaxScaler.

I was wondering how to interpret the coefficients generated by the model and find something like feature importance in a Tree based model.

Should I re-scale the coefficients back to original scale to interpret the model properly?

It will be great if someone can shed some light on how to interpret the Logistic Regression coefficients correctly.

No, you do not need to re-scale the coefficients. To the contrary - if they are scaled, you can use them as a way to compare feature importance.

Let's assume that our logistic regression model has coefficients {$ a_i$}, relating to the different (scaled) variables {$x_i$}.
A change of $\Delta x_i $ in the variable $ x_i $  will result in an increase (or decrease, if $a_i$ is negative) of $ a_i \Delta x_i $ in $ log({\hat p_i \over {1-\hat p_i}}) $, i.e. the logit function of $ \hat p_i $, where $ \hat p_i $ is the predicted probability that the i-th example is in the positive class.

So, if the variables are scaled, you can say that if $ a_i$ is larger, then $x_i$ is more important in the model.


How to better represent three sets of categorical data? Given three set of data with categorical integer x-axis with the same range (0-10):

from itertools import chain
from collections import Counter, defaultdict

from IPython.display import Image
import pandas as pd
import numpy as np

import seaborn as sns
import colorlover as cl
import matplotlib.pyplot as plt

data1 = Counter({8: 10576, 9: 10114, 7: 9504, 6: 7331, 10: 6845, 5: 5007, 4: 3037, 3: 1792, 2: 908, 1: 368, 0: 158})
data2 = Counter({5: 9030, 6: 8347, 4: 8149, 7: 6822, 3: 6153, 8: 5075, 2: 3973, 9: 3205, 1: 2247, 10: 1580, 0: 1059})
data3 = Counter({4: 9884, 5: 9058, 3: 9048, 2: 6942, 6: 6630, 1: 4596, 7: 3784, 0: 2524, 8: 1912, 9: 917, 10: 345})

df_data1 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data1.items()])))
df_data2 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data2.items()])))
df_data3 = pd.DataFrame(list(chain(*[(int(k),)*v for k,v in data3.items()])))

I can plot three different barcharts for each set of data as such:

sns.set(style="white")
g = sns.factorplot(x=0, data=df_data1, kind="count", color='#4882D5', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()
g = sns.factorplot(x=0, data=df_data2, kind="count", color='#9CD2D5', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()
g = sns.factorplot(x=0, data=df_data3, kind="count", color='#EF988D', size=6, aspect=1.5, order=None)
g.set_xticklabels(step=1)
plt.show()

[out]:

What I want to highlight is the fact that:

for data1, we see that the center of mass is inclined to the right where the x-axis is higher at 8
for data2, it's normally shaped where the center is at 5
for data3, the center of mass is at 4 on the lower side.

Imagine:

data1 x-axis represents the no. of blueberry a person buys at the supermarket
data2 x-axis represents the no. of watermelon a person buys at the supermarket
data3 x-axis represents the no. of oranges a person buys at the supermarket

And

The y-axis for all 3 datasets represent the number of people who buys the respective no. of fruits as per the dataset.

It's a little hard to convey the message with the barcharts.

Is there a better way to represent the dataset and highlight what I want to say?

Is there a way to present the 3 graphs as one? How would it look like?

You shoud put the barcharts next to each other as described in this example.
import numpy as np
import matplotlib.pyplot as plt

N = 5
men_means = (20, 35, 30, 35, 27)
men_std = (2, 3, 4, 1, 2)

ind = np.arange(N)  # the x locations for the groups
width = 0.35       # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(ind, men_means, width, color='r', yerr=men_std)

women_means = (25, 32, 34, 20, 25)
women_std = (3, 5, 2, 3, 3)
rects2 = ax.bar(ind + width, women_means, width, color='y', yerr=women_std)

# add some text for labels, title and axes ticks
ax.set_ylabel('Scores')
ax.set_title('Scores by group and gender')
ax.set_xticks(ind + width / 2)
ax.set_xticklabels(('G1', 'G2', 'G3', 'G4', 'G5'))

ax.legend((rects1[0], rects2[0]), ('Men', 'Women'))

def autolabel(rects):
    """
    Attach a text label above each bar displaying its height
    """
    for rect in rects:
        height = rect.get_height()
        ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,
                '%d' % int(height),
                ha='center', va='bottom')

autolabel(rects1)
autolabel(rects2)

plt.show()


Why does removal of some features improve the performance of random forests on some occasions? I completed feature importance of a random forest model. I removed the bottom 4 features out of 17 features. The model performance actually improved. Shouldn't the performance degrade after removal of some features, given that some data has been lost? What are some reasons to explain the performance improvement?

A basic decision tree is pruned to reduce the likelihood of over-fitting to the data and so help to generalise. Random forests don't usually require pruning because each individual tree is trained on a random subset of the features and when trees are combined, there is little correlation between them, reducing the risk of over-fitting and building dependencies between trees.

There could be a few reasons why you get this unexpected improved performance, mostly depending on how you trained the random forest. If you did any of the following, you potentially allowed overfitting to creep in:

a small number of random trees was used
trees with high strength were used; meaning very deep, learning idiosyncrasies of the training set
correlation between your features

and so removing features, you have allowed your model to generalise slightly more and so improve its performance.

It might be a good idea to remove any features that are highly correlated e.g. if two features have a pairwise correlation of >0.5, simply remove one of them. This would essentially be what you did (removing 3 features), but in a more selective manner.

Overfitting in Random Forests

You can read a bit more about the reasons above on Wikipedia or in some papers about random forests that discuss issues:

Random forest paper by Leo Breiman - states in the conclusion section:

  Because of the Law of Large Numbers, they do not overfit.

but also mentions the requirement of appropriate levels of randomness.
Elements of Statistical Learning by Hastie et. al (specifically section 15.3.4 Random Forests and Overfitting) gives more insight, referring to the increase of the number of data samples taken from your training set:

  at the limit, the average of fully grown trees can result in too rich a model, and incur unnecessary variance

So there is a trade-off perhaps, between the number of features you have, the number of trees used, and their depths. Work has been done to control the depth of trees, with some success - I refer you to Hastie et. al for more details and references.

Here is an image from the book, which shows results of a regression experiment controlling the depth of trees via minimum Node Size. So requiring larger nodes effectively restricts your decision trees from being grown too far, and therefore reducing overfitting.

As a side note, section 15.3.2 addresses variable importance, which might interest you.

I assume that you trained ("grew") your random forest on some training data and tested the performance on some hold-out data, so the performance you speak of is valid.


Multiclassification problem I was wondering what happens when an image not in the training set is provided to the model in a multiclassification problem? Does it just classify something which is close to this image?

As the model is not trained to recognize an image from this new specific class, the only thing it will do, is to give a probability-or similarity measure  for each of the classes for which the model has been trained on. Hence in a Classification problem, the class with the highest probability for this testing image will be the classification output.


Transfer Learning for CNNs and Batch Norm Layers In some transfer learning models, we set the training argument to False to maintain the pre-trained values of Batch Normalization for example, but the trainable attribute to False to freeze the weights. Then the new "top-layer" is added and we re-train the model. Afterwards for fine-tuning, we can re-train the weights and set the trainable attribute to True. However, what does the argument training=True do for a layer? The stackoveflow answer here does not make sense to me. When the argument training is True, that, to me, implies we are doing some type of learning: i.e. the BN mean and variance are being updated, Dropout is being applied, and the weights are from the backwards pass. What is the difference between training=True and training=False? The Keras FAQ states it just means inference is being performed, but what was being trained when training=True?
Lastly, this is being nit-picky, but in this notebook Google does transfer learning with the MobileNet V2 model. In same from above they use Xception model. Both models have BN, but in the second tutorial, they pass the training=False argument in the base_model implying do not update BN. Whereas in the first they make no mention of training=False. Why might that be? I see the first one is copyrighted in 2019 and the second one in 2020, which might imply the discrepancy.

but what was being trained when training=True?

Let's try to understand BatchNormalization(BN) layer first as it has more elements.
TL;DR -
γ, β are learned. These are initialized just like normal weights and learned in Backpropagation.
May read this crisp and spot-on answer on these parm Stat.SE

Formally, BN transforms the activations at a given layer  x  according to the following expression:
 BN(x)= γ⊙(x−μ)/σ + β
coordinate-wise scaling coefficients  γ  and offsets  β.
[Quoted - http://d2l.ai/]

each BN layer adds four parameters per input: γ, β, μ, and σ (for example, the first BN layer adds 3,136 parameters, which is 4 × 784). The last two parameters, μ and σ, are the moving averages; they are not affected by backpropagation, so Keras calls them “non-trainable”. However, they are estimated during training, based on the training data, so arguably they are trainable. In Keras, “non-trainable” really means “untouched by backpropagation.”
[Quoted - Hands-on machine learning with scikit-learn keras and tensorflow, Aurélien Géron

training=True: The layer will normalize its inputs using the mean and variance of the current batch of inputs.
training=False: The layer will normalize its inputs using the mean and variance of its moving statistics, learned during training. [Quoted - Keras doc for BN]

So, if you will not set it False it will continue updating μ and σ with every batch of test data example and normalize the output accordingly. We want it to use the values from the Training phase.
By default, it is False and fit method set it to True.

Dropout
Dropout is simpler of the two. We need this Flag here so that we can compensate(during testing) the loss to the Output value(on an average basis) due to the switched-off(during Training) Neurons.

Suppose p = 50%, in which case during testing a neuron would be connected to twice as many input neurons as it would be (on average) during training. To compensate for this fact, we need to multiply each neuron’s input connection weights by 0.5 after training. If we don’t, each neuron will get a total input signal roughly twice as large as what the network was trained on and will be unlikely to perform well. More generally, we need to multiply each input connection weight by the keep probability (1 – p) after training. Alternatively, we can divide each neuron’s output by the keep probability during training (these alternatives are not perfectly equivalent, but they work equally well)
[Quoted - Hands-on machine learning with scikit-learn keras and tensorflow, Aurélien Géron

On the example of Models difference
Though, these are subjects to try and check.
 But I believe generally we start to fine-tune when we believe that the upper layer is smoothened to match with the initial layers, to avoid a large flow in forward and backdrop. So the logic stated to keep it Flase in 2019 example might not hold too strong every time.


What is the name of this similarity distance metric? def distance_metric(seed, base):
    num = 0.0
    den = 0.0
    num = sum(numpy.minimum(seed,base))
    den = sum(numpy.maximum(seed,base))
    dist = round(1.0 - 1.0*num/den,4)
    return dist

The metric is used to gauge similarity in the context of locality sensitive hashing.

Items within a bucket are kept if their distance is < 0.16.

This is the weighted Jaccard Index.

https://en.wikipedia.org/wiki/Jaccard_index#Weighted_Jaccard_similarity_and_distance

This is different from the regular Jaccard Index (Similarity).


Multiclass classification with high number of classes, high number of features and small sample size I am working on a biology related dataset with over 300K features, and I only have about 5K samples. I want my model to classify many classes. For this problem in particular the class is age. Each age such as 10 or 35 will be individual classes. So roughly 80 classes (range from 10 to 90) are needed for this problem.

I immediately know that regularization is needed to shrink the number of features to prevent overfitting. I just don't know whether such a dataset can be treated as a multiclass classification problem with many number of classes. If I need more data, how many data will be enough for the model to learn? Or are there any clever ways I can do for this problem?

You can try dimensionality reduction techniques such as PCA, which will reduce your numbers of features and maximizing the information describing your objects.

Moreover, if you want to classify the age which is a continuous variable, you may want to try regression instead of classification


Difference Between Performance Scores I need some help to understand the meaning between these different scores.
Currently, I am doing the classification problems using machine learning, and I have obtained the results for the classification as shown in the image below.

To obtain the results like in the image I use the code:

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
print(confusion_matrix(y_test,y_pred))
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test, y_pred))

Then I also try to use the code below to get the recall, precision and f1-score

print(precision_score(y_test,y_pred))
print(recall_score(y_test,y_pred))
print(f1_score(y_test,y_pred))

Results :
Precision : 0.19601
Recall : 0.44360
F1-score : 0.27188

Then I also try this code for weighted using this code:

print(precision_score(y_test,y_pred, average='weighted'))
print(recall_score(y_test,y_pred, average='weighted'))
print(f1_score(y_test,y_pred, average='weighted'))

Results :
Weighted Precision : 0.8588
Weighted Recall : 0.7684
Weighted F1-score : 0.8048

The problem right now is I already confuse with all these values. What does the meaning of avg/total value in the image, the values of the second code I try, and the value of the weighted metrics as my third code. Which value that I should use to know that the performance of the classifier is good or not ? Hope someone can help me.

What you are obtaining are different metrics on the predictions you made with a given model (so far I haven't given you any new information).

What you printed are several results from different metrics which measure different things in a model.

As you should know, sometimes the data scientist (you in this case) need to know how the model performs well in positive cases, or how does it perform in negative cases, etc.

The first thing you printed is called the confusion matrix, it crosses the information on Positives/Negatives in the real-world vs your model.

In the first square: When you said positive and it's positive in your data. The last square is negative and you said negative and (depending on how your output is made) the 2nd and 3rd squares are false positives and false negatives.

The precision measures how well your model detects Positives when your model says they are positive, meaning: How your model says the truth about positives $Precision = \frac{True Positives}{True Positives+False Positives}$

The recall measures how well your model detects Negatives when your model says they are Negatives, meaning: How your model says the truth about negatives $Recall = \frac{True Negatives}{True Negatives+False Negatives}$

The f1-score gives you the harmonic mean of precision and recall. The scores corresponding to every class will tell you the accuracy of the classifier in classifying the data points in that particular class compared to all other classes.

The support is the number of samples of the true response that lie in that class.

You can find documentation on both measures in the sklearn documentation.


Convolution backpropagation I'm in the progress to learn, and understand different neural networks. I pretty much understand now feed-forward neural networks, and back-propagation of them, and now learning convolutional neural networks. I understand the forward-propagation of them, but having issues understanding their back-propagation. There is a very good resource explaining the convolutional layer, however, can't understand the back-propagation.
In my understanding, according the back-propagation algorithm of feed-forward neural networks/multi-layer perception, if I have the following input (its items as $i$), and filter (its items as $w$), giving the output (its items as $o$).
$$\begin{pmatrix}i_{1}^1 & i_{2}^1 &  i_{3}^1\\\ i_{4}^1 & i_{5}^1 &  i_{6}^1\\\ i_{7}^1 & i_{8}^1 &  i_{9}^1\end{pmatrix} * \begin{pmatrix}w_1^1 & w_2^1\\\ w_3^1 & w_4^1\end{pmatrix} = \begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix}$$
So if we want to calculate for example how much $w_1^1$ affected the cost $C$, we need to know how much $w_1^1$ affected its corresponding output item $o_1^1$, and how much $o_1^1$ affected the cost $C$ which gives the following equation:
$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$
Where in my thinking we have to think back how we get the output regarding to $w_1^1$ to calculate $\frac{\partial o^1}{\partial w_1^1}$.
To get $o_1^1$, we multiplied $w_1^1$ with $i_1^1$, to get $o_2^1$, multiplied $w_1^1$ with $i_2^1$, to get $o_3^1$, multiplied $w_1^1$ with $i_4^1$, to get $o_4^1$, multiplied $w_1^1$ with $i_5^1$.
To calculate $\frac{\partial C}{\partial o^1}$, it depends on how the output is connected with the next layer. If it is an another convolutional layer, then we have to calculate how each output item is connected to the next layers outputs, which will be their connecting weights.
So if we see an example, where we put a 2x2 filter on $o^1$, to get the final output $o^2$ (which will give a single output with 1x1 size):
$$\begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix} * \begin{pmatrix}w_1^2 & w_2^2\\\ w_3^2 & w_4^2\end{pmatrix} = \begin{pmatrix}o_1^2\end{pmatrix}$$
Where in my thinking the back-propagation for $w_1^2$ is:
$$\frac{\partial C}{\partial w_1^2} = \frac{\partial o^2}{\partial w_1^2}\frac{\partial C}{\partial o^2} = o_1^1 * 2(o^2_1 - y_1)$$,
and the back-propagation for $w_1^1$ is:
$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$
Where: $$\frac{\partial o^1}{\partial w_1^1} = (i_1^1 + i_2^1 + i_4^1 + i_5^1)$$
And: $$\frac{\partial C}{\partial o^1} = \frac{\partial o_1^2}{\partial o_1^1}\frac{\partial C}{\partial o_1^2} + \frac{\partial o_1^2}{\partial o_2^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_3^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_4^1}\frac{\partial C}{\partial o_1^2}$$
So: $$\frac{\partial C}{\partial o^1} = w_1^2 * 2(o_1^2 - y_1) + w_2^2 * 2(o_1^2 - y_1) + w_3^2 * 2(o_1^2 - y_1) + w_4^2 * 2(o_1^2 - y_1)$$
Am I right? Because as I'm reading through the article above, it seems completely different.

Note that a CNN is a feed-forward neural network. Thus, if you understand how to perform backpropagation in feed-forward neural networks, you have it for CNNs.
A convolution layer can be understood as a fully connected layer, with the constraints that several edge weights are identical and many edge weights are set to 0.
You can also build a pooling layer in this way. For example an average pooling layer is nothing but a specific convolution layer, with fixed weights.
For max-pooling, use the fact that $\max\{x,y\} = \frac{x+y+|x-y|}{2}$.


Regressing Elbow-like Plot I have data consisting of light wavelength and some coefficient. The wavelength in the data I have is in steps of 10s and I need the coefficient for a wavelength that is between the values I have (eg. 432.5 nm). Data:
wavelen    coeff
380         0.982
390         0.564
400         0.378
...         ...

I was thinking of regressing the data I have to find a function that represent the data, but not quite sure what type of function I should choose. I tried exponential and power functions, but the results not accurate for the first few wavelengths due to the coefficient drops extremely fast for earlier wavelengths.

What would be the best type of regression for this data? I was also thinking of splitting the regression for the first few data points, like a piecewise function. Is this a valid method?

Answer
Thought I'd elaborate a little on my comments with a demo of different curves.
The main thing is that short of fitting a curve backed by theory (which we've discussed and I know isn't an option), I don't think there's any one method which is more 'valid' than the others.  With just a single value at each wavelength, using a notion such as error doesn't help, as we can just choose splines so that they pass through the original data points.  You could try to use theory to suggest the curve should be smooth (if that is appropriate) but each of the interpolation methods I demonstrate below provide smooth curves, so this is unlikely to be fruitful as you would need to argue some degree of smoothness and it does not sound like that argument can sensibly be made.
Illustration of Interpolation Curves
To illustrate some of the options for interpolating this data, I've taken the first few data points provided in the question and added a few estimated from the provided plot.
Using that data, I fitted a smoothing spline (smooth.spline), loess regression (loess), cubic spline (spline) and polynomial (lm with poly).
The result can be seen in this plot:

I haven't tried to tune any of the methods but you can still see the main differences.  The splines are all quite similar.  Obviously, they have the advantage of passing through the given data points.  You could argue that the 'anchoring' of the spline curves would make interpolating at values close to the original wavelengths more likely to be closer to the 'true' value (but there is no guarantee that would be the case unless some theory can be applied to explain why the curve should be smooth like that).  On the other hand, you could consider the polynomial to give some degree of 'regularisation' and so may postulate that, were many data points to be taken, the true values may 'average out' to a smoother curve of this shape.  Without more data, it really is a case of choosing the assumptions that you wish to make rather than choosing an objectively better way to interpolate this data.
It sounds like gathering more data (repeat readings to average and/or intermediate values) would be the only way to really hone in on a method but I realise that may well not be practicable.
Below is the R code I used to generate the plot, in case it's of use.
library(ggplot2)
library(magrittr)
library(tidyr)
theme_set(theme_light())

wavelen <- seq(380, 440, 10)
coeff <- c(0.982, 0.564, 0.378, 0.220, 0.175, 0.160, 0.150)

dat <- data.frame(wavelen = wavelen, coeff = coeff)

# Create a new vector of wavelengths with a finer resolution to view the results
new_wavelen <- seq(380, 440, 1)

# Smoothing spline
smooth_spline <- smooth.spline(dat$wavelen, dat$coeff)
spline_pred <- predict(smooth_spline, new_wavelen)
spline_pred <- as.data.frame(spline_pred)

# Loess Regression
loess_reg <- loess(coeff ~ wavelen, dat)
loess_pred <- predict(loess_reg, new_wavelen)

# Third degree polynomial
poly_mod <- lm(coeff ~ poly(wavelen, 3, raw=TRUE), dat)
poly_pred <- predict(poly_mod, data.frame(wavelen = new_wavelen))

# Cubic spline
cubic_spline <- spline(x = dat$wavelen, y = dat$coeff, method = "fmm", xout = new_wavelen)

# Plot results
plot_dat <- cbind(new_wavelen, spline_pred$y, loess_pred, poly_pred, cubic_spline$y)
colnames(plot_dat) <- c("wavelen", "Smooth", "Loess", "Poly", "Cubic")

plot_dat <- plot_dat %>%
  as.data.frame() %>%
  pivot_longer(-wavelen, names_to = "Method", values_to = "Coeff")

dat$Method <- "Original_Data" # hacky way to get the original data in the plot and legend.
fig <- ggplot(data = plot_dat, aes(x = wavelen, y = Coeff, colour = Method, linetype = Method)) +
  geom_line()
fig <- fig + geom_point(data = dat, mapping = aes(x = wavelen, y = coeff))
fig <- fig +
  labs(title = "Demo of Different Curve Fitting Methods",
       x = "Wavelength")
fig


Increase Accuracy on Keras Multiclass Classifier I have a dataset with the shape (430, 17). My output is a single column consisting of possible options "Best", "Medium", "Worst". I split my data into X and y, and then into training and testing sets after using the StandardScaler to scale X.
I then using the LabelEncoder and get_dummies to prepare my output values.
>>>X_train[:1]

   array([[-0.62115527,  0.73646208,  1.27931666, -1.13977066, -0.03124594,
        -0.06595102, -0.35812162,  2.47947661,  0.01705523, -0.75215293,
        -0.43173137, -0.15181261]])

>>>y_train[:1]

   array([[0, 1, 0]], dtype=uint8)

I then prepared a keras Model based on an example from the iris dataset that also worked with a multiclass classifier using Keras.
model = Sequential()
model.add(Dense(20, input_dim = 17 , activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(64, activation = 'tanh'))
model.add(Dense(3, activation = 'softmax'))
model.compile(Adam(lr=0.04),'categorical_crossentropy', ['accuracy'])

model.fit(X_train, y_train, epochs = 100, verbose=1)

y_pred = model.predict(X_test)

scores = model.evaluate(X_test, y_test)
print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))

I have been changing the inputs and general architecture of the model for the last few hours, but I cannot seem to be able to get the accuracy above 63.64%.
Sample Output:
Epoch 95/100
11/11 [==============================] - 0s 2ms/step - loss: 0.6250 - accuracy: 0.7037
Epoch 96/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7214 - accuracy: 0.6980
Epoch 97/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7595 - accuracy: 0.7123
Epoch 98/100
11/11 [==============================] - 0s 2ms/step - loss: 0.7197 - accuracy: 0.6838
Epoch 99/100
11/11 [==============================] - 0s 2ms/step - loss: 0.6658 - accuracy: 0.6895
Epoch 100/100
11/11 [==============================] - 0s 1ms/step - loss: 0.7103 - accuracy: 0.6838
3/3 [==============================] - 0s 997us/step - loss: 0.8703 - accuracy: 0.6364

My question is:
Can you advise on what I can do to increase the accuracy of the validation data?
Edit 1:
Changed the hidden layer nodes to 12, and changed to activate to relu. Here is a picture of the training and validation so far:

Edit 2:
Changed the focus of the posting from two questions to one. I also added the most recent model, and results:
model = Sequential()
model.add(Dense(20, input_dim = 17 , activation = 'tanh'))
model.add(Dense(24, activation = 'tanh'))
model.add(Dense(24, activation = 'tanh'))
model.add(Dropout(0.11))
model.add(Dense(24, activation = 'relu'))
model.add(Dense(3, activation = 'softmax'))
model.compile(Adam(lr=0.002),loss = 'categorical_crossentropy', metrics = ['accuracy', "mse", "mae"])
history = model.fit(X_train, y_train, epochs = 800, validation_split = 0.2, verbose=0)

You should understand if the model you built is able to learn from the data. Therefore, plot the accuracy and the loss for both training and validation set versus the number of epochs.
Then, if the model needs more epochs, give it more epochs. Otherwise, try different optimizers, activation functions, number of neurons, number of layers, and batch size.

From the (flawed) plots you provided (please, plot accuracy for training and validation on one graph, and loss for training and validation on another graph), you can see two clear phenomena:

accuracy stops to increase after a very few epochs;
loss curve is unstable.

which suggest that the model does not learn from the data (I would start off with a shallow architecture, tuning number of neurons, trying different optimizers, activation functions, and then number of layers) and the dataset could be unrepresentative.


How can I improve a machine learning model? I am a Machine learning newbie and i am trying my hands with a dataset which has 9 features and my aim is to figure out the optimal multi class classification model which fits my dataset.

I applied PCA on my dataset and reduced the dimensionality to 2 and now i can visualize my dataset and this is how it looks like on my training dataset.

I am not sure how to choose a best model when data is distributed in such a way and i am looking for suggestions on what techniques are out there.

Is Getting more data a viable next step?

It is questionable whether doing the PCA and reducing the dimension to 2D was a good idea. You can try experimenting with the number of components in PCA or even trying some non-linear dimensionality reduction like LLE:

  Locally linear embedding (LLE) seeks a lower-dimensional projection of the data which preserves distances within local neighborhoods. It can be thought of as a series of local Principal Component Analyses which are globally compared to find the best non-linear embedding.


Can a convention convolution neural network train correctly with different training image size and ratio? For example the task  of transformation, the model consist convolutional layer and pooling layer only, take input of image, and output a feature map (loss MSE, trying to produce feature map that exactly match feature map in label)

The forward pass can take any image size and output a feature with size = [w/(n_max_pooling*2), h/(n_max_pooling*2)]

Thus, with batch size of 1, there are no "dimension mismatch error" when we feed different image size, different ratio. The network just output a feature map, due to the nature of convolutional layer and max pooling layer.

For example: (batch_size, channel, width, height)

Sample 1:  1x3x128x128 => target 64x64
Sample 2: 1x3x768x1024 =>
target 368x512

....

I would like to know in that case of inconsist input shape, will the network be able to train and update it weight correctly ?

Do not do this.

You are providing the algorithm with examples your training data should conform with the same dimensions otherwise you will skew the weights with alternate examples that do not reflect a true positive as like the original batch.

Your data preparation is important, if you have the ability to standardise the size then you should do this at the start before training.

If you have a library of images that you want to train on that are all different sizes; I suggest that you do the following:

Use opencv for the following
1) Write a script that cycles through the images
2) use the script to clip the images on the x:y axis making a square
     2b)you can automate this by using the click point for centre which then clips and save the new sized image

obviously save all the files in another directory but they should all be standardised now.


Does ridge regression always reduce coefficients by equal proportions? Below is an excerpt from the book Introduction to statistical learning in R, (chapter-linear model selection and regularization)

"In ridge regression, each least squares coefficient estimate is shrunken by the same proportion"

On a simple dataset, I obtained 2 non-intercept coefficients b1=-0.03036156 and b2=-0.02481822 using OLS. On l2 shrinkage with lambda=1, the new coefficients were b1=-0.01227141 and b2=-0.01887098. Both haven't reduced by equal proportions. What am I missing here?

Note:

the assumption made in an Introduction to Statistical Learning book for the quoted statement is n=p
the scale of both variables in my dataset is same

As I know we have below equation for Ridge Regression:

\begin{equation}
RSS_{Ridge} = \Sigma_{i=1}^{n} (\hat{y}_{i} - y_{i})^2 - \lambda \Sigma_{j=1}^{p}(\beta^2)
\end{equation}

First of all, it seems to me, if lambda goes higher does not mean that coefficients go down with inverse relation to lambda. Because the power of beta is two and the lambda is one.

I think you refer to page 226 of "An Introduction to Statistical Learning" book. In the footnote of the figure, the writer says:

"The ridge regression and lasso coefficient estimates for a simple setting with n = p and X

a diagonal matrix with 1’s on the diagonal. Left: The ridge regression coefficient estimates

are shrunken proportionally towards zero, relative to the least-squares estimates.

Right: The lasso coefficient estimates are soft-thresholded towards zero."

In that figure shows that if we got +2 and -2 coefficient in the OLS model and if Ridge shrunk +2 to +1.8 then we are sure Ridge will shrink -2 to -1.8. So, in both cases with the same proportion of 0.2 coefficients go to near zero.


treeExplainer algorithm intuition I'm reading the paper about the treeExplainer; the pseudo-code of Algorithm 1 is a bit cryptical as most of the variables are not even defined (same with sampling and all details involved).
Is there a more intuitive explanation of how the algorithm works, at a high-level?

The paper you refer to actually states the following intuition:

  Algorithm 1 estimates $E[f(X)|do(X_S = x_S )]$ by recursively following the decision path for $x$ if the split feature is in $S$, and taking the weighted average of both branches if the split feature is not in $S$.

It seems to be a slight modification of the original description in arxiv 1706.06060:

  For a tree model $E[f(x) | x_S]$ can be estimated  recursively using
  Algorithm 1, where $v$ is a vector of node values, which takes the value
  $internal$ for internal nodes. The vectors $a$ and $b$ represent the left
  and right node indexes for each internal node. The vector $t$ contains
  the thresholds for each internal node, and $d$ is a vector of indexes of
  the features used for splitting in internal nodes. The vector $r$
  represents the cover of each node (how many data samples fall in that
  subtree).

I presume $S$ is an item in the set of all subsets of N features.


Difference between sklearn’s “log_loss” and “LogisticRegression”? I am a newbie currently learning data science from scratch and I have a rather stupid question to ask. I’m currently learning about binary classification, and I understand that the logistic function is a useful tool for this. I looked up the documentation and noticed that there are two logistic related functions I can import, i.e. sklearn.metric.log_loss and sklearn.linear_model.LogisticRegression. When and where should I use them, and what’s the difference?

On a broader note, what’s the difference between a metric and a model, and why is the log loss function a metric? Apologies if this question sounds completely nonsensical, but this is a genuine source of confusion for me!

The metrics module by Scikit-learn implements functions assessing prediction error for specific purposes (Regression, classification, etc.). Model is the algorithm that actually does the classification/regression/clustering (as per need) for you.

Log-loss measures the accuracy of, say a classifier. It is used when the model outputs a probability for each class, rather than just the most likely class.

EDIT: Thanks to @mapto for suggesting documentation reference:

sklearn.metrics.log_loss,
and
sklearn.linear_model.LogisticRegression


How to Extract Information from the Image(PNG) I'm trying to extract some particular information from the image(png).

I tried to extract the text using the below code

import cv2
import pytesseract
import os
from PIL import Image
import sys

def get_string(img_path):
    # Read image with opencv
    img = cv2.imread(img_path)

    # Convert to gray
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    # Apply dilation and erosion to remove some noise
    kernel = np.ones((1, 1), np.uint8)
    img = cv2.dilate(img, kernel, iterations=1)
    img = cv2.erode(img, kernel, iterations=1)

    # Write the image after apply opencv to do some ...
    cv2.imwrite("thres.png", img)
    # Recognize text with tesseract for python
    result = pytesseract.image_to_string(Image.open("invoice.png"))
    os.remove("invoice.png")

    return result

if __name__ == '__main__':
    from sys import argv

    if len(argv)<2:
        print("Usage: python image-to-text.py relative-filepath")
    else:
        print('--- Start recognize text from image ---')
        for i in range(1,len(argv)):
            print(argv[i])
            print(get_string(argv[i]))
            print()
            print()

        print('------ Done -------')

But I want to extract data from particular fields.

Such as

 a) INVOICE NO.
 b) CUSTOMER NO.
 c) SUBTOTAL
 d) TOTAL
 e) DATE

How can I extract the required information from the below image "invoice"?

PFB

If all of your images are similar to this one(or have a small set of possible designs), you can simply reference the location (pixel-wise) on the image where this fields are and slice it.

After slicing you can use any OCR algorithm to extract that data.

If your data has more variation than that, you can use OCR on the entire image, which is usually a slow algorithm.

If you have less than 50 or so images to do that, it is more efficient to do this by hand (not worth writing code)


Main options on how to deal with imbalanced data As far as I can tell, broadly speaking, there are three ways of dealing with binary imbalanced datasets:

Option 1:

Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold: https://scikit-learn.org/0.16/modules/generated/sklearn.cross_validation.StratifiedKFold.html ).
For each fold apply a resampling technique (upsampling, downsampling or a combination of both) separately on the "training" and "test" sets.
Use a "traditional" metric for evaluation: for instance the AUC of the ROC curve (TP Rate vs FP Rate).

Option 2:

Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
Do not apply any resampling technique.
Use an "alternative" metric for evaluation: for instance the AUC of the Precision-Recall curve or something like the F-score (the harmonic mean of Precision and Recall).

Option 3:

Use something like XGBoost and tune the scale_pos_weight ( https://xgboost.readthedocs.io/en/latest/tutorials/param_tuning.html ).
Create k-fold Cross-Validation samples randomly (or even better create k-fold samples using Stratified k-fold).
Use a "traditional" metric for evaluation: for instance the AUC of the ROC curve (TP Rate vs FP Rate).

My main question is if I correctly interpret what the options are. Is there any conceptual mistake in what I'm saying? Is it appropriate to use Stratified k-fold in the three cases when dealing with imbalance? Is it not necessary to apply any resampling when using XGBoost and tuning scale_pos_weight? When some resampling is applied (Options 1 and 3) does it make sense to use a "traditional" metric and does not make sense to use an "alternative" metric? In general, the resampling has to be applied separately on training and test sets? Etc.

Also, it would be nice if you have any good reference to SMOTE and ROSE, regarding how they work, how to apply them and how to use them with python.

your resume is quite good, but I'm not comfortable in dividing the broad discussion to those three more or less sharply separated roads. But indeed, often a technique similar to one of those is chosen. Just let me underline something about them:

the Area Under the Curve (AUC) of Precision and Recall has been shown as being slightly better than AUC ROC, but don't expect miracles. They are quite similar in practical situations. In facts, the performance of the metric you choose (AUC ROC, AUC Precision-Recall or similar), intended as it's effect on the test set, depends strongly on the issue you are trying to solve.
The option of XGBoost scale_pos_weight helps, but does not miracles. If you already choose to use XGBoost, then activate it. But don't choose XGBoost just because have this option available.

Also, consider that:

in real-world application your samples reflect a set of biases of the System which collect them. This means that if you have small imbalance (up to 1 to 20 or even 100) is better to avoid any balancing technique and keep the imbalance as it is. This will probably guarantee a better model in production. For the same reason, sometimes using non-stratified k-fold gives better models than the stratified version.
high-thought synthetic samples creation techniques as SMOTE often miss their target. This because the complexity of the issue you are trying to solve can be so high that SMOTE (or similar techniques) is anyway unable to create decent synthetic samples. In those cases, is better to just use undersampling or (better) oversampling. Indeed, just duplicating the samples of the lower samples class is an astonishing good technique, in terms of performance.


Do I need a next for loop in order to get all values? When i print this in the module without printing to file, it prints all values. But when I print it to the designated file, it only prints the first value.

#Create the model base
from sklearn.feature_selection import chi2
import numpy as np
N = 2
for UNSPSC, category_id in sorted(category_to_id.items()):
features_chi2 = chi2(features, labels == category_id)
indices = np.argsort(features_chi2[0])
feature_names = np.array(tfidf.get_feature_names())[indices]
unigrams = [v for v in feature_names if len(v.split(' ')) == 1]
bigrams = [v for v in feature_names if len(v.split(' ')) == 2]

with open('results.csv', 'w') as f:
 print("# '{}':".format(UNSPSC), file=f)
 print("  . Most correlated unigrams:\n. {}".format('\n. '.join(unigrams[-N:])), file=f)
 print("  . Most correlated bigrams:\n. {}".format('\n. '.join(bigrams[-N:])), file=f)

How can i adjust this so that all values get printed onto the file?

Don't forget to open the file first..

Py 2.6

For item in list:
    the_opened_file.write("%s\n" % item)

Py 3.x

with open(filepath,'w') as fileptr:
    for item in list:
        fileptr.write("{}\n".format(item))

If you want to save your list itself,
Convert it to numpy format using np.assarray(list) and then use np.save(file,array,....(other opts) and reload it whenever required using np.load()


Keras Fit Function (R): Train Regression Model with multiple Labels I am trying to implement a deep learning model in R using Keras. Let's say I had a dataset of people's faces and a CSV with information about the person's age, gender, and ethnicity. I want to train the model to predict a person's age from their photo.

For this, I have to use convolutional neural networks. Here is my pseudocode:

library(keras)

model <- keras_model_sequential()

model %>% ## define CNN model’s architecture.
## I will figure this out later

# configure model
model %>% compile(
    loss='mean_squared_error',
    optimizer='adam',
    metrics='mae'
)

history <- model %>% fit(
# train_array is a 300x100x100x1 array
# train_array stores the pixel values of 300 grayscale images
# of resolution 100x100
    x = train_array, y = ???,
    epochs = 10, batch_size = 30,
 )

My question is regarding the ??? marked above. I have three pieces of useful information to train the model: age, gender, and ethnicity. How do I add this information to the model? Again, I want the model to predict a single prediction of age.

I looked at the Keras fit function documentation but couldn't figure it out. It says that y is a "Vector, matrix, or array of target (label) data (or list if the model has multiple outputs)..."

EDIT:
That makes sense. Now the question is how do I set up the x list? Here's what I have so far:

trainFeatures <- list(pixels = train_array, gender = as.factor(trainGenders), ethnicity = as.factor(trainEthnicity))

history <- model %>% fit(
    x = trainFeatures, y = trainAge,
    epochs = 10, batch_size = 30,
)

I downloaded a toy dataset and set up a quick CNN model. I got the following error message:

Error in py_get_attr_impl(x, name, silent) :
  AttributeError: 'list' object has no attribute 'dtype'

x is an array/matrix of the inputs/features/independent variables.

y is an array/matrix of the output(s)/target(s)/dependent variable(s).

Your x array should contain the features extracted from the dataset of people's faces (however you may choose to do that). You can even include the gender and ethnicity alongside the features extracted from the dataset, if this information will be available to you while you deploy the model.

Your y array should contain your target variable, i.e., the age associated with each face.

Response to EDIT:

Unfortunately, I actually haven't used Keras with R. But judging by the error you are seeing (and the prefix of the function), I would say it is most likely compatibility issues between R lists and whatever Keras expects underneath. Keras is implemented in Python, so it is probably expecting a Numpy array or just a Python list. I suspect you will have better luck creating an R data.frame for your input features.

This tutorial I found might be helpful as it uses Keras and R to build a CNN.


Determining usage/context of terms/acronyms/abbreviations in python Background: I work for a major airline in the United States and have been given a pretty significant text mining and machine learning task. Since there are several unique terms, acronyms, and abbreviations in aviation, I have written a program that reads through the documents. The program is using 4 groups of terms; WORDS, STOPS, AIRPORTS, and SYNONYMS. WORDS, STOPS, and AIRPORTS are all sets where the first two were initialized from NLTK and the last was initialized with a manually built list of airports that my employer services. SYNONYMS is a nested dictionary 'trie' style structure where the branches are misspellings and the leaves are the terms that they represent. it was initialized as an empty dictionary. I have also created a command prompt UI that informs me when it finds a word that it doesn't know and allows me to correct the word or add it to one of the aforementioned groups. The documents are being tokenized by spaCy and fed into scikit-learn machine learning algorithms.

Problem: I've started noticing that some of the words that I previously accepted in WORDS or SYNONYMS are showing up with alternate meanings and could have a drastic impact on how well the algorithm operates. For example the word act can be the verb act; or it can be an abbreviation for actuator or active; or it can be an acronym for auxiliary central tank or a couple other things. At present, I'm using the UI to manually correct or ignore these occurrences, which is time consuming, cumbersome, and prone to errors.

Desired result: What I'm looking for is a method of determining whether 'lt' means left or light depending on context (lt wing vs. warning lt) with minimal manual entry. At a minimum I would like some high level ideas/discussion on how this may be accomplished in python. Optimally, I would like some example code, preferably using the same tools I already have; spaCy, NLTK, and sklearn but I am flexible and willing to learn.

Notes: I've considered a couple methods of doing this but I haven't been able to come up with any ideas strong enough to try. For example, I've considered marking the terms with the pos inside the sets by concatenating it with the word before adding it like 'rat:noun'. The issue with this is that rat:noun could be an animal or it could be an abbreviation for ram air turbine, which is also a noun.

After rubber ducking with a friend of mine, I decided to take a two stage approach.

First, whenever I see a word I've recognized as a multiple use word, I flag it in the system and give it the appropriate correction. Once it evaluates that part, it stores the corrected word and enough words such that there is sufficient context in a tuple... inside a list... inside a dictionary. It all looks something like use_case = {'abbr': ('five word abbr context string', 'correction').

Secondly, I've created a dictionary of scikit-learn LinearSVCs that looks like svm_dict = {'abbr': LinearSVC}.

Once these things are established and the program is run again, when it finds one of the multiple usage words it first checks use_case to see if that exact set of 5 words surround this case. If it does it swaps it out for the original correction, otherwise it tries to predict the usage with the appropriate SVC and asks the user if the prediction is correct.

I know it's not fully automated, but neither is the program in which it runs. I figured the occasional yes/no question to the user was easier than leaving it off the lists and rewriting it every... single... time...

Either way, I have it all in place but haven't gotten to the point where it's being used yet. So, I'll either be editing, deleting, or accepting this answer sometime next week when I've had some time to test'r out.


How to continue incremental learning when a categorical variable has been assigned additional category labels? Please help answer this question or point me to any resource.

There is a model in an environment where training happens with new data and the data is discarded after training is completed.
This keeps on happening in cycles. Hence we dealing with models allowing "partial fit"

In this model, there is a training variable X , which is a categorical variable.
X has some previously known categories C and its encoded as "one hot vector" of length D .

Now suppose we are now observing a new category of X, so we need to extend encoding of C to length D+1 .

The question is, how to do this without losing all the previously learned knowledge by model.

Training again is not an option since we have discarded all the previously held data and only new data can be used for training.

There is already a question Updating One-Hot Encoding to account for new categories
However the answer is either to use entire data to create the "one hot encoding" (which is not possible in this environment since new data is in future) , or to omit the usage entirely (thus loosing some information).

Looks like a special case for incremental learning / life-long learning. There are many papers in this line of work:

https://arxiv.org/abs/1611.07725

https://openreview.net/forum?id=BkloRs0qK7

Most try to preserve the model parameters in a way that reduces the loss of previous knowledge ("catastrophic forgetting"), but some also try to find (or keep) some samples and preserve the original model's response on those samples.


Unhashable type 'list' while looping through dataframe in Python I have the following dataframe comments. I have segregated a list of users based on certain conditions. I want to get the count of words based on those users which are named as gold_users. But I am getting an error in my code TypeError: unhashable type: 'list'. Please help me fix this.
DataFrame(comments)
       Id|                    Text                             |    UserId
        6|  [2006, course, allen, knutsons, 2001, course, ...  |    3
        8|  [also, theo, johnsonfreyd, note, mark, haimans...  |    1

Code
for index,rows in comments.iterrows():
  gold_comments = rows[comments.Text.loc[comments.UserId.isin(gold_users)]]
  Counter(gold_comments)

Expected Output
#Top 10 Words that appear the most in the comments made by gold users with their count.
 [['scholar',20],['school',18],['bus',15],['class',14],['teacher',14],['bell',13],['time',12],['books',11],['bag',9],'student',7]]

TypeError: unhashable type: 'list'.

You will get this error when you are trying to put list as key in dictionary or set because list is unhashable object.
Example you trying to input code such as
dict1 ={ 1:'one', [2]:'two'}
print(dict1)

O/p:
TypeError Traceback (most recent call last)
<ipython-input-73-c4e2fd1e6bf0> in <module>
----> 1 dict1 ={ 1:'one', [2]:'two'}
      2 print(dict1)

TypeError: unhashable type: 'list'

solution :
Try to convert the list into tuple
Hashable object in Python:

int
float
decimal
bool
string
tuple
complex
range
frozenset
bytes

UnHashable object in Python

list
set
dict
bytearray
customclasses


Object Recognition for classification, is it being used in industry? I'm wondering if e-commerce companies where products are offered by users, such as EBay, are using Object Recognition to ensure that an uploaded image corresponds to an specific type of object (clothing, shoes, glasses, etc) either to classify automatically or more importantly to filter undesired images (such as non related or even illegal types).

If so, which algorithms and/or open platforms could be use for doing so? From what I've looked it seems that HOG+Exemplar SVM might be one of the most accurate methods developed so far (http://www.cs.cmu.edu/~efros/exemplarsvm-iccv11.pdf), even having couple of public repo's with Matlab implementations (https://github.com/quantombone/exemplarsvm), but I'm still wondering if this is being used in industry.

As far as I am aware, object recognition is not extensively used in industry yet. Google's image search, for instance, is based on exploiting the text on the web pages rather than the images themselves. I have seen several start-up companies that market prototypes based on object recognition, but these products are either prototypes or not widely used.

Regarding state-of-the-art techniques, you should take a look to recent literature on deep learning and deep neural networks applied to object recognition, along with more specific techniques like convolutional networks.


Add Cell Numbers to Jupyter? Does anyone know a way to add cell numbers (not line numbers within cells)?  I have been using nbextensions for a while but it does not seem to have the ability to label cell numbers.

If you need it for a presentation, open the Notebook with a text editor, find the brackets where cell numbers fall and change them manually. If it's something you have to do once.


What are some ways to train, test and render an RL autonomous drone in a simulator? I completed the Udacity nanodegree in Deep Learning but found the final project to be extremely difficult since the only visual feedback were from plots via matplotlib (rotor speeds, x/y/z coordinates, x/y/z velocity, etc).  Is there any way to render the learning per episode as done in openai gym?

A quick search for Python game engines only confused me more in terms of which one to use.

Here is a link to the final project, ie. fly a quadcopter using DDPG algorithm:  https://github.com/udacity/RL-Quadcopter-2

It has a physics_sim.py, task.py, agent.py and a main ipynb notebook.

I tried contacting a few drone simulator makers including DRL (Drone Racing League) and Velocidrone but the latter does not have API for Python and DRL has not responded.

Therefore, I looked at Unity ML Agents and saw that it's possible to create a simulation environment using free drone assets:

https://unity3d.com/how-to/unity-machine-learning-agents

https://assetstore.unity.com/packages/tools/physics/free-pack-117641

And they have a Python API:

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Python-API.md