Convolutional neural network giving high confidence on wrong classification I am doing a project on sign language recognition through CNN.

My Dataset

300 images for each alphabet and one special symbol for space. Images are taken from three individuals.
Each image has hand centered in white background.
Resolution is 640*480.
So in total 27 labels and 8100 images.

Samples Images

Code

I am using pretrained AlexNet in PyTorch and have changed fully connected layers according the need of the project. I am training for 10 epochs. Batch size is 32. Loss function is Negative Log Likelihood Loss and optimizer is Adam.

AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Linear(in_features=9216, out_features=1024, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=1024, out_features=512, bias=True)
    (4): ReLU()
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=512, out_features=27, bias=True)
    (7): LogSoftmax()
  )
)

Problem

The problem I am facing is that the model is giving very high confidence about 90-100 % on images that have no hand gesture in them or contain wrong hand gesture. I have tried adding dropout layers, training for more epochs but nothing seems to solve this problem. Is there something wrong with the model or the dataset needs to be changed/updated?

Since your model seems to be doing fine on the correct hand signs, I don't think there is anything wrong. It's a common misconception that you can interpret the model output as a confidence measure. Consider the following example.

Let's say you have a binary classifer and you learn to distinguish numbers between -1 and 0 in one class labelled as 0 and numbers between 0 and 1 in the other labelled as 1. Your training data is in these intervals and is labeled correctly. Your classifier in the end will be something like

$$ y = \sigma(wx)$$

with $\sigma$ as the sigmoid function. When you train, you will quickly learn that $w > 0$. The more you train, the larger $|w|$ will become untill you get something close to 0 for all $x \in [-1,0)$ and something close to 1 for $x \in (0,1]$. All perfect and normal so far.

Now you pass the number $x=10$ to the model. The query does not make sense, as this $x$ does not belong to any of the classes the model was trained on and has nothing to do with the training data. There is no reason to assume that the model would give any useful output for $x=10$. But still, the model will the super confident that $10 \in (0,1]$ since for $x=10$ the model will output a value closer to 1 than for any of the training examples. This is clearly not correct.

I hope this clarifies that the interpretation of the classifier output as confidence can be very misleading. If the classes are close together in feature space and you query with an example that is somewhat between the training data, then you could take it as such, but for input that is not represented in the training data, the numbers you will get don't have any meaning at all.

In your application you could introduce the counterexamples to the training data and label them as "other", but that would be risky as well. If you really need a confidence measure, look into Bayesian Neural Networks (e.g. Bayes-By-Backprop, Dropout or Batch norm), other Bayesian methods or bootstrapping.


Is it possible to implement a classifier according to quarters? What about missing data? I have a dataset with several individuals and features. I'm studying behavior over the year (for instance, averages or iterations of money gained, jobs, etc.).

My ultimate goal is to implement a classifier since I have a specific feature for every person (which is equal to 0, 1, or 2). When I first tried to implement a SVM, I ended up with bad results because I did not have enough data / features: I have too many number 1 individuals and not enough 0 and 2's, so my classifier almost always put people into category 1. Therefore, I tried to increase my number of lines by separating my data into quarters (i.e JAN, FEV, MAR, then APR, MAY, JUN, then JUL, AUG, SEPT, and finally OCT, NOV, DEC)

I was wondering two things:

Would that be a good idea? Do I have to be cautious of a particular hypothesis that could impact my results?
In case it is a good idea, I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have "Age" available for my last quarter) ; do I have to drop the feature ? Or would it be wiser to abandon the last quarter ? Or is it possible to make the classifier work despite that lack of information without actually deleting anything ?

Would that be a good idea?

That is hard to tell from your description. It is not an immediately bad idea. If it results in a better classifier (according to cross-validation), then it has probably worked.

The main things that would concern me about splitting behaviour data by quarter and treating as independent are:

Your data samples will very likely be correlated when they share a person. You can work around this by careful splitting between training and cross-validation / test sets. Do not make a fully random split, but split by person - any individuals records should appear only in one of the training, cross-validation or test sets (assuming your goal is to take similar data in production from users who are not in your current database, and predict their class).
There could be seasonal variation in the records that reduce the effectiveness of the split. So a "type 1" person's records in APR-JUN might look like a "type 0" person from JAN-MAR.
How will you receive data in production - when you want to classify new users? If you only want to work on single-quarter data, then your new classifier is fine. If you have more data, you have to deal with your classifier maybe predicting different target variable for the same person depending on the quarter. You could combine these in some way - but if you do so, you should also do this in test to see what the impact of doing this is, which may be counter-productive (you end up with the same number of test examples as if you had not done the split). It might also be OK, perhaps it will add some regularisation.

  Do I have to be cautious of a particular hypothesis that could impact my results?

You have to be very cautious about testing your classifier, because you could get data leakage from the cross-validation and test sets to the training set, which would make you think the classifier is generalising well when in fact it is not. The fix for this described above - split by person when deciding train/cv/test split.

  I have some data available for some quarters of the year but sometimes it is missing (for instance let's imagine I don't have "Age" available for my last quarter) ; do I have to drop the feature ?

Handling missing data is a complicated topic in its own right, there are lots of options. You can start with:

If data is missing at random (i.e. there is no reason to suspect it is related to the target variable, or only impacts certain types of record), you can substitute the mean value of that feature from the training set, or impute it based on a statistical model from the other features.
If data is missing for reasons that might impact the target variable, then you should give that information to the classifier, because it might be an important feature in its own right. You can take the mean or more complex imputed value as before for the original feature, but also you should add a new boolean feature "feature X was missing".

Whether or not you should use the partial data or drop it is not possible to say in general. If you are not sure, then try both and pick the version with the best cross-validation result.


How to compute accuracy for CNN when outputs are one-hot encoded I want to compute accuracy of my model on test data. I have three classes so actual output might look like this :

[0 1 0]

and the predicted output might look like this:

[0.1 0.8 0.1]

How do I compute accuracy with this?
Shall I compute cross entropy and subtract 1 from that?

Predicted class is the one with highest probability in output vector (class B in your case) & accuracy is correct predictions %, unless I'm missing your point.

The problem that you have mentioned is representative of multi-class classification which is solved using Softmax output layer in neutral net. NOTE that Softmax uses Cross Entropy as loss function.

I feel that calculating Cross Entropy on output may not make sense.


How do ML model retain/store the learning(s)? In other words, if the model after training and testing is ready for making future predictions, it must be storing the learning(s) somewhere in memory or disk or cache (or I really don't know).

So, where (how) is it exactly stored?

Hope I asked my doubt clearly for experts to advise.

After generating a predictive model by fitting the data with model.fit(X,y) , "the learnings" are simply coefficients (weights ) that are multiplied by the features ( X ) to give you a certain y. In other words, in the end, what you get is a function that for an input X returns a y, you can store the model on the disk using a joblib or pickle and load it in another environment. machine learning? more like coefficients learning.

Note : This could be edited in case my answer is lacking more details


Layers' function in a convolutional neural networks From Wang et al (2015) "Visual Tracking with Fully Convolutional Networks":

  A top layer encodes more semantic features and serves as a category
  detector, while a lower layer carries more discriminative information
  and can better separate the target from distracters with similar
  features.

I don't quite understand this statement: does this mean that the algorithm first identifies the semantic group to which the object belongs and then tries to distinguish among them? Also, are there any other publications where the difference in the roles of layers been demonstrated?

This is a basic property of convolutional networks. First layers identify simple features and as you go deeper in the CNN each layer uses the features of the previous layer to build more complex ones.

I would say the state-of-the-art article about this is Visualizing and understanding Convolutional Networks by Matthew D. Zeiler and Rob Fergus.
You can download it here


How is a coincidence matrix constructed for computing Krippendorff's alpha? I am looking at two documents to help me learn about constructing coincidence matrices in order to gain a better understanding of Krippendorff's alpha. I am using these two:

https://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&context=asc_papers
https://en.wikipedia.org/wiki/Krippendorff%27s_alpha

There seems to me to be a discrepancy between the two. There probably isn't, but I'm looking for some help in figuring out whether my understanding is wrong, or if there is indeed a discrepancy.

In link 1, I am looking at section B ("Nominal data, 2 observers, no missing data"), where the coincidence matrix is presented. In link 2, I am looking at the section "Coincidence matrices".

Consider the reliability matrix presented in link 1:

In order to calculate the elements of the coincidence matrix, we have the following definition in link 2:

$$o_{vv'}=\sum_{u=1}^{N}\frac{\sum_{i\neq i'}^{m}I(v_{iu}=v)I(v_{i'u}=v')}{m_u-1}=o_{v'v},$$

where $u$ is the horizontal element of the reliability matrix (the columns), $m_u$ is the number of labels actually present in column $u$.

This seems simple enough. For element $o_{aa}$ (or $o_{11}$) we should have:

$$
o_{aa}=\frac{I(a=a)I(b=a)}{2-1}+\frac{I(a=a)I(a=a)}{2-1}+\frac{I(b=a)I(b=a)}{2-1}+\frac{I(b=a)I(b=a)}{2-1}
$$

$$
+\frac{I(d=a)I(b=a)}{2-1}+\frac{I(c=a)I(c=a)}{2-1}+...
$$

and so on. Clearly, only one of the summation elements is non-zero, namely the second element. Hence $$o_{aa}=1.$$

Using the same formula/logic, we arrive at $$o_{bb}=2.$$

But if we look at link 1, it is getting double the value for the coincidence matrix elements, and I don't understand why. I don't even understand the link's explanation as to why it is getting those values for the coincidence matrix.

Can somebody help?

The formula in the Wikipedia article seems to be limited to two coders. Look at the sum of values in u for m coders, i I like j, so each value in u is to be matched with one other value in u.  Actually, this sum would pair values of i with values of j and values in j with values i. So, o sub aa would be 2 not 1.  And o sub bb would be 4, not two.

However, I don’t see how Wikipedia’s formulation could yield coincidence matrices for more than two coders as this would require two sums which pair each of m valors with each of m-1 other values.


Pivoting a two-column feature table in Pandas How can I transform the following DataFrame into one with cities as rows and each cuisine as a column, and 1 or 0 as values (1 if the city has that kind of cuisine)?

I think this turns out to be a very common problem in transforming data into features for machine learning.

I am aware of the Pandas pivot_table functionality, but it asks for a value column, and in this case we don't have any.

import pandas as pd

data = {
    'city': ['NY','NY', 'SF','SF','SF'],
    'cuisine': ['Japanese', 'Chinese', 'French', 'Japanse', 'German']
}

df = pd.DataFrame(data)

If there is no value column - introduce it yourself!

df["value"]=1
pd.pivot_table(df, values="value", index=["city"], columns="cuisine", fill_value=0)

For your example I got (after fixing the misprint in 'Japanse' to 'Japanese')

cuisine  Chinese  French  German  Japanese
city
NY             1       0       0         1
SF             0       1       1         1


Multi-class classification with discrete output: Which loss function and activation to choose? I'm working with a multi-class classification problem, using Keras Sequential models. In my dataset, the output class has one of the following values: (1, 2, 3, 4, 5).

Yes, I have already tried one-hot-encoding the output and using categorical_crossentropy as a loss function and softmax in the output layer.

However, I don't think that softmax and categorical cross entropy is the right choice in my case. In my dataset, the output classes have a certain "discreteness" (or scale). Class 1 is the "worst" and class 5 is the "best".

Let's say if on a specific input the model predicts 2 and the true class is 1, it's a much better prediction as when the model predicts class 5 and true class is 1.

I would like, that the loss function would take these "minor errors" into consideration and not treat all errors the same way.

To sum up, I'm wondering what my options are in terms of loss function and activation of the output layer for my given problem.

UPDATE: I have now tried to use mse as a loss function and replace my output layer with this:

model.add(Dense(1, activation='linear'))

The new method produces much better results, but I'm still wondering if there are any other options for my problem. Maybe a custom loss function?

You could try a Quadratic Kappa Metric loss function. This is a summary of what it actually does.

  A weighted Kappa is a metric which is used to calculate the amount of
  similarity between predictions and actuals. A perfect score of 1.0 is
  granted when both the predictions and actuals are the same. Whereas,
  the least possible score is -1 which is given when the predictions are
  furthest away from actuals. In our case, consider all actuals were 0's
  and all predictions were 4's. This would lead to a QWKP score of -1.
  The aim is to get as close to 1 as possible. Generally a score of 0.6+
  is considered to be a really good score.

The detailed explanation can be found here: https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps


Identifying common keyphrase frequency in large dataset I have a dataset of profiles that contain freeform text describing the work history of a number of individuals.
I would like to attempt to identify frequently used words or groups of words across the set of profiles in order that I can build a taxonomy (of skills) related to the profiles.
For example, if the words 'conversion rate optimisation' appear together 300 times across all profiles, I would see this on my list as a high-frequency keyphrase. I would expect to be able to filter the list based on single keywords, 2 words, and 3-word strings.
I would then be able to manually pick out frequently used keyphrases relating to skills, that could be added to a master taxonomy list.
I would also need some way of filtering out invalid words like ('I', 'and' etc)
What is the best way to get something like this done?

I would like to attempt to identify frequently used words or groups of words

The difficulty here would be to capture multiword terms, as opposed to single words. This implies using n-grams for various values of $n$, and that can cause a bias when comparing the frequency of two terms of different length (number of words).

  I would also need some way of filtering out invalid words like ('I', 'and' etc)

These are called stop words (sometimes function words or grammatical words). They are characterized by the fact that they appear very frequently even though they consist in a quite small subset of the vocabulary (fyi this is related to Zipf's law for natural language). These two properties make them easy enough to list in a predefined list so that they can be excluded, there are many lists available (e.g. here or there).

Since you don't have any predefined list of terms, a baseline approach could go along these lines:

For every value of $n$ to consider, collect all the $n$-grams
Remove any $n$-gram which contains only stop words (or which contains mostly stop-words) (note: it might be better to do this step first, but only if it's safe to assume that multiword terms don't contain stop words)
Calculate the document frequency for every candidate term (the same DF as in TF-IDF weights)
Filter out the terms which have a very low document frequency (experiment with different values for the threshold). This step should eliminate a lot of noise, but probably not all of it.
You will probably still need a bit of manual filtering here if your goal is to obtain a clean list of actual terms. Normally there should be few long n-grams left and the ones left should be good for the most part, however there might still be a lot of false positive unigrams and bigrams.

This approach is very basic but it's easily adjustable, you can adapt it to your data, possibly add steps etc. Otherwise there are probably specialized tools for terminology extraction, but I'm not familiar with any.


Should your learning rate get smaller as your neural net gets larger? Is it correct to say that as you add more layers and more neurons, your learning rate should then decrease?
So, generally speaking, the larger the net, the smaller the learning rate?

The learning rate affects how much your model weights are changed with each update. So the question is what is a good learning rate? This depends on your problem space, dataset and network parameters (ie. Activation functions, complexity, regularisation, optimizers, etc).
Too high a learning rate, your model may not reach convergence due to weights updated to “overshoot” the optimum.
Too low a learning rate, your model may also not reach convergence due to taking too many epochs, as the updates to weights are tiny.
It is best to experiment with the defaults ~0.01, and adjust accordingly based on visualizing the validation loss and training loss change over epochs. Also try out different optimizers (Adam, RMSprop, etc) as these adapt your learning rate over time.
Another approach is to implement a learning rate scheduler, that decays your learning rate over time - the idea being as a model nears the global minima, you want to fine tune its descent with smaller weight updates.


parsing a simple sentence using syntaxnet (on Ubuntu) could someone help me in with the following. I would appreciate it.

I would like to use syntaxnet to analyze a sentence. I installed Ubuntu on windows following this link.

What do I have to do next to make syntaxnet to produce result. The sentence is for example, "David put a book on shelf".

Thank you in advance.

Have a look at the documentation from the Authors of SyntaxNet themselves (Google). That link shows all the main steps that you might want to do.

Looking at the general setup required, it seems like quite an involved process! I can't actually find any nice walkthroughs on how to really run and understand the models... only that first link above!

If you would like to simply parse sentences and are not tied to SyntaxNet, it might be worth looking into similar and (in my opinion) easier to use tools, such as spaCy. You can install it easily using one of the commands from their website:


Extracting Part of Speech (Source and Destinations) using text mining/NLP? I need to extract the source and destination terms from the text
 documents using text mining / NLP / Information Retrieval ?

Example input:

  I am travelling from New York to London.
  I am heading towards playground from home.
  I will be going to Sweden from Boston.
  I was flying from School to Home.

The output can be as follows :

S. No. |  source    | Destination
------ |  ----------|------------
      1| New York   | London
      2| playground | home
      3| Sweden     | Boston
      4| School     | Home

Named Entity Recognition is technique which can be used here. Location is one of the 3 most studied classes (with Person and Organization). Stanford NLP has an open source Java implementation that is extremely powerful.

For Example  let say sentence is "i will be going to Sweden from Boston."

Now here you can use regular expression to detect these LOCATION tags.


How to replace NA Data using a regression? My dataset contains some stocks and some bitcoin titles.
I would like to replace NA data (normal stocks are not traded on sat. and sun.) using a regression on the stock itself.
How can i do this?

Set the column with missing values as your target.

Remove rows where your target is null.

Train a regression model that predicts your target using the rest of your features.

Replace missing values with this regression model's predictions.


what is the difference between Bias and overfilling and under fitting? I am confused about Bias and overfitting as I see it sounds similar but not able to explain differene?

Well, I can't express enough that there are many many many articles and questions on bias-variance, over-fitting and under-fitting.

But let me also try to explain in simple words for you.

"Bias" as a word means- favoring a thing more than another. Like we heard Amazon's alexa doesn't respond to female voices/commands. so it was biased towards men.

Now if want to make a model which is able to predict on unseen data i simply don't want it to be biased like i did't want this system to favor male's voice over female'voice. If its biased it is essentially giving me an error. Its making more assumptions about my target function(which is taking voice commands here).

Now since its assuming a lot and giving me an error, its rather under-fitting.However it actually is not only high bias but+low variance leads to under-fitting but i am keeping variance aside for now. Now, there can be many reason for high bias and low variance.I am not going into details on reasons w.r.t. training/test size.

Now, if my model is able to predict the data correctly on given test data which is a good news,it means there are no errors ,it mean there is less bias. however may be its giving me no error because it learned the "training" data so well, captured every scenario/noise. Now again comes the variance, that there is high variance here, which together with low bias shall lead to over-fitting. it means it is well able to predict on the data i gave it to train but if i feed it new data it won't be able to predict it right i.e. generalization.

Now you might wanna understand the variance also. take again the simple dictionary meaning of variance - which is "the fact or quality of being different, divergent, or inconsistent." now apply this definition in relation with target function.

Please note this is a very very vague explanation for you to follow and there are standard definitions for the concept which i suggest you to follow.

Now there are way better explanation-
https://machinelearningmastery.com/gentle-introduction-to-the-bias-variance-trade-off-in-machine-learning/
https://stats.stackexchange.com/questions/4284/intuitive-explanation-of-the-bias-variance-tradeoff

Here is a good paper-

http://www.cems.uwe.ac.uk/~irjohnso/coursenotes/uqc832/tr-bias.pdf

Most precise visualization is below-


mAP scores on tensorboard (Tensorflow Object Detection API) are all 0 even though the loss value is low I trained a faster-rcnn model on the tensorflow object detection API on a custom dataset. I found that the loss is ~2 after 3.5k steps. However, when I ran eval.py, the mAP scores are all almost 0 as shown below.

I do not understand why this is the case. However, when I look at the images at 3.5k steps, the model has captured some of the boxes as shown below

Can someone please explain why the mAP scores are close to zero, even though the model has learned to output quite a few boxes?

I recommend several checks to make sure you get reasonable mAP@IoU scores for object detection API:

Try varying the Intersection over Union (IoU) threshold, e.g 0.2-0.5 and see if you get an increase in average precision. You would have to modify matching_iou_threshold parameter in object_detection/utils/object_detection_evaluation.py
Try different evaluator classes (the default one is EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'). If you are training on Open Image Dataset it makes sense to use open_images_V2_detection_metrics
Check your eval config file and increase the number of examples used in the evaluation set, e.g.

eval_config: {
  num_examples: 20000
  num_visualizations: 16
  min_score_threshold: 0.2
  # Note: The below line limits the evaluation process to 10 evaluations.
  # Remove the below line to evaluate indefinitely.
  max_evals: 1
}
Train the object detector for more iterations
Check current mAP against reported metrics (e.g. COCO mAP@IoU=0.5):


Kmeans quality vs data Are there any ways to check if more data can help the quality of kmeans clustering?

The closest research I could find was for EM algorithms.

Any thing similar for evaluating kmeans ?

The estimation quality of the mean improves with sqrt(n).

So with more and more data, your mean will get more precise. But the improvements get slower and slower.

It won't help with all the other issues of k-means, such as problems with different cluster diameters and outliers. And more data probably means more outliers...

You can try a similar approach to that paper with k-means with some slight modifications. Try plotting the sum of squared deviations between the holdout set, and the nearest centroid each. As long as this improves, the clusters fit your holdout set better.


Is my normalization off? So I have a DataFrame which consists of order data from customer on an exchange.

I have a column Dollars which is the dollar value of all trades conducted by customers in different currency pairs. I want to train a classifier on this data, so I'm normalizing the Dollar column to get the values into the same range, but I think the normalized values seem quite.. odd. This is a snippet of my DataFrame:

Dollars   | Normalized dollars
181447.50 | 9.10975e-06
281885.00 | 1.41523e-05
290786.00 | 1.45992e-05
70923.00  | 3.56076e-06
1121169.54| 5.62894e-05

These values seem tiny.

I've used sklearn.preprocessing.normalize for this.

It's worth mentioning that the lowest Dollars value is 0.06 while the largest is 5,605,847,772.52, which I assume is the explanation, but I was expecting a 0-1 range, but my largest normalized value is 0.157. Should I do more filtering on the dataset to remove extreme outliers as a general rule?

UPDATE:
Applying the base 10 logarithm to dollars yields some more manageable results, though the normalized values are still fairly small. ScaledDollars is the result of using sklearn.StandardScaler on Log10Dollars:

Dollars   | Normalized dollars| Log10Dollars | LogNorm   | ScaledDollars
181447.50 | 9.10975e-06       | 5.258751     | 0.00384193| -0.761916
281885.00 | 1.41523e-05       | 5.450072     | 0.00398171| -0.573336
290786.00 | 1.45992e-05       | 5.463573     | 0.00399157| -0.560028
70923.00  | 3.56076e-06       | 4.850787     | 0.00354388| -1.16404

You should use scaling instead of normalizing. By normalizing the column you make the norm 1, while you are interested in making the expected value 0 with a standard deviation of 1. With the norm, the more samples you have the smaller your samples will become. Use the scikit-learn StandardScaler instead. Despite this, with these huge numbers you will still get weird results (although most of them will be negative with this skew). What could be an alternative is using a log transformation on your dollars first, and then scaling them. Conceptually you usually look at relative values between dollars, the difference between 10 and 20 dollars is similar to that of 10,000 and 20,000 which the log transformation will capture.


Does L1 regularization always generate a sparse solution? Following is a plot in Bishop's Pattern Recognition and Machine Learning explaining why $L_1$ regularization would lead to sparse solution, and I understand that in this case $w_1$ would be automatically constrained to be 0.  However, it seems to me that it is not always true that the corner point is closest to the optimal unconstrained solution.  For instance, if the optimal unconstrained solution is at the red cross, then there exists some point on the side that is closer to the red cross than corner points.  In this case, do we still have sparse solutions?

I think with your red cross you have in mind the case where $w_1 = w_2$. You are right that in this scenario the solution will not be sparse, since you will also get $w^*_1 = w^*_2$.

However, whenever the coordinates of the red cross are off the line $y=x$ ($w_1\neq w_2$), you can shrink the regularization region so much that eventually you will end up with a similar picture as Figure 3.4 above.

Say your red cross has the coordinates $w_1=r_1$ and $w_2=r_2$ and say your regularization strength is such that $||w||_1 = |w_1|+|w_2| \leq C$ where $C$ defines the size of the shaded region.

Then, for the circular contours, the optimal solution will be where the line $w^*_2 = C- w^*_1$ (for the boundary region) and the line through the red cross at $(r_1,r_2)$ running orthogonal to the boundary region intersect. You can see that the latter has the form $w_2 = f(w_1) = 1\cdot w_1 + r_2 - r_1$. (Orthogonality gives you slope +1 because the boundary region has slope -1.) Setting them equal gives you the intersection at coordinates $w^*_1 = \frac{C+r_1-r_2}{2}$ and $w^*_2 = \frac{C-r_1+r_2}{2}$.

Now, if $r_2=r_1$ you have $w^*_1=w^*_2$ independent of $C$ and the solution won't be sparse.

If $r_1>r_2$ (i.e. you are below the first diagonal) shrinking $C$ to $r_1-r_2>0$ yields $w^*_2=0$ but $w^*_1>0$. You end up in the right corner of the regularization region. (Similar to my image.)

If $r_2>r_1$ (i.e. you are above the first diagonal) shrinking $C$ to $r_2-r_1>0$ yields $w^*_1=0$ but $w^*_2>0$. You end up in the top corner of the regularization region. (Similar to your image.)

As Emre mentioned in the comments, this holds in general and you can "see" this from the Kuhn-Tucker conditions of the optimization problem.


Is it ideally correct to benchmark neo4j as graph processing platform? I would like to know if neo4j can be considered a graph processing platform, even though i know that:

neo4j: is a graph database management system developed by Neo Technology, Inc. Described by its developers as an ACID-compliant transactional database with native graph storage and processing.

graph processing platform: is a platform used for processing graphs, by applying global algorithms towards large graphs (used more in OLAP scenario).

Based on this paper http://www.ds.ewi.tudelft.nl/~iosup/perf-eval-graph-proc14ipdps.pdf, neo4j is benchmarked as a processing platform.

So my question is, if its correct to put neo4j in the group of processing platform like pregel and giraph.

No

As Emre has rightly pointed out, the Chief Scientist of the company himself has written a blog post claiming the same.

However, Neo4j does a lot of computations which graph processing tools can do. In fact, it does graph traversals much faster than giraph, due to the Hadoop overhead and also as it stores the adjacent nodes in a doubly linked list.

So, it's not rare to confuse Neo4j with a graph processing platform (thus the claim in the paper) due to it's overlap with the features of a processing platform like Pregel and Giraph.


Issue with using Sparse Data Frame in Mlxtend Apriori function I am running python 2.7 in anaconda and have installed mlxtend. Based on the latest version of mlxtend, the aprioir class supports sparse dataframe as its input. I have over 500k products that I want to run a market basket analysis on.

I have created a onehot encoded sparse dataframe using a small dataset to test but I am running into df.to_coo() issue on the sparse data frame inside the mlextend apriori function.

Please find the code, the input data file and the errors I get here -
https://github.com/nshahHome/pycode

Click on the view code to see the files.

code = code2.py  ,  input data file= mbatest.txt   ,   errors =    code2-error.html (pdf version)  ,   condalist.txt

I expect the code to not throw errors and try to create frequent_itemsets. The set could be empy if there are no sets > min_support.

This question is now closed as it has been accepted as an enhancement needed by the developer.

https://github.com/rasbt/mlxtend/issues/501


Python writing to Excel file: writerow() takes no keyword arguments I have this script:
    import requests
    from requests import get
    from bs4 import BeautifulSoup
    import csv
    import pandas as pd

    f = open('olanda.csv', 'wb')
    writer = csv.writer(f)

    url = ('https://www......')
    response = get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    type(soup)

    table = soup.find('table', id='tablepress-94').text.strip()

    print(table)

    writer.writerow(table.split(), delimiter = ',')

    f.close()

When it writes to a CSV file it writes everything in a single cell like that:
Sno.,CompanyLocation,1KarifyNetherlands,2Umenz,Benelux,BVNetherlands,3TovertafelNetherlands,4Behandeling,BegrepenNetherlands,5MEXTRANetherlands,6Sleep.aiNetherlands,7OWiseNetherlands,8Healthy,WorkersNetherlands,9&thijs,|,thuis,in,jouw,situatieNetherlands,10HerculesNetherlands, etc.
I wanted to have the output in a single column and each value (separated by comma) in a single row.
I tried to use delimiter = ',' but I got:
TypeError: a bytes-like object is required, not 'str'.
Screesnhot of table:

How can I do this? Thanks!

Please see doc of csv.writer
delimiter param is used when instantiating the writer, not on writerow method.
Furthermore from what I understand, your approach is wrong.
For example you want to scrape the html table and store it as csv file.
But what you do is get the text from table
table = soup.find('table', id='tablepress-94').text.strip()

which is a concatenated string of all columns with no structure and try to split by nothing, which is wrong
while you should be getting a dict of table values, eg using:
table = soup.find('table', id='tablepress-94')
# assuming table has a
#

Column 1	..Column n

# section with column headers, else adjust accordingly
column_headers = [td.get_text() for td in table.find('thead').find('tr').find_all('th')]
print(column_headers)
data = {}
row_index = 0
for row in table.find('tbody').find_all('tr'):
    row_index += 1
    column_index = 0
    columns = row.find_all('td')
    for td in columns:
        if column_index >= len(column_headers):
            column_headers.append('Column_'+str(column_index+1))
        if column_headers[column_index] not in data:
            data[column_headers[column_index]] = []
        data[column_headers[column_index]].append(td.get_text())
        column_index += 1

print(data)
# now output your scarped table data, data into csv properly
writer.writerow(column_headers) # print headers

# print each row of data
for row in range(row_index):
    writer.writerow([data[column][row] for column in column_headers])


Learning rate and memory of Least Mean Square algorithm I am studying neural networks.

  The smaller we make the learning rate, the longer the memory span over which the LMS algorithm remembers past data will be. (Section 3.5(page 103) of Simon Hykin's "Neural Network and Learning Machines":)

I don't understand why this is the case. Could someone explain?

This should be the weight update formula your model uses.

$W_{n+1} = W_n - \mu\nabla\epsilon[n]$,

$\mu$ is the learning rate, and we multiply it with the change we want to do. Obviously, the smaller the learning rate you have, the less update you make, and thus the "longer" the model converges and also the "longer" the model remembers its previous states.

Think like this, if you set the learning rate to 0, you learn nothing and your parameters are never updated.


Keras input dimensions for a MLP I have been training a multilayer perceptron using Keras to make a prediction on a function similar to that of a normal distribution. I have  input variables , and I have one output value .
When I set my input layer to have  neurons as such

model.add(Dense(4, input_dim=4, activation= 'relu'))

the model learns with a  accuracy.
When I tried to use  neurons in my input layer as such

model.add(Dense(35, input_dim=4, activation= 'relu'))

my model learns it with an  accuracy.
I'm not understanding the logic behind this. Surely you have to have only  neurons for the input layer; what is happening with the other  neurons

If I understand your question properly. In first example you have 4 input neurons and they are connected to 4 neurons. In second, 4 inputs are connected to 35 neurons. That's it, you simply add more neurons in hidden layer. Btw, what do you mean saying "learns it with an accuracy"?


Feature extraction; similarity and classification of accelerometer data I have several expert persons performing the same specific action (for example, squat or leap forward) multiple times. Say 5 persons do 100 squats each. They have an accelerometer attached to the same body parts. I record the accelerometer readings and get 100*5 = 500 data samples. They do it for multiple different actions (squat, push up, leap forward, etc). The way they record the action is as follows:

Start recording (push button)
Do the action
Stop recording (push button)

Now I need to see that another person is doing the actions in the correct order. For example, squat, leap forward, stand up, drop down, push up. I take his accelerometer data and continuously feed it to the classifier that needs to tell me if he now has done exactly a squat action, not a leap or a push-up. So, when the first action, namely squat, was identified, I check against leap forward and so on.
There are several problems with this:

These data samples have different amounts of values, since somebody is squatting a bit slower, others do it a bit faster. So, some data
samples have 250 XYZ values, others have 220 or 270, etc. (in range of
+-50). What I do for now is make stricter rules. I discard all the data samples that exceed 250 readings and for ones that have fewer values than 250, I append the values from the beginning to the end so that it gives 250 in total. Works fine, since there is a windup for every action where the person is standing still for a brief moment before he performs the action. This is not optimal, because the experts need to redo the action if they were too slow (the windup was too long) + I append fake data. What would be a better solution to handle this?
For now I am using Random Forest, AdaBoost classifiers with low/high
pass filtered accelerometer data that I map to 750 columns (250 X, 250 Y, 250 Z) with 1
class column. So the prediction tells me something like 70% leap, 25% squat, 5% push up. The classification is sometimes wrong or not precise enough. Thus, I was thinking of extracting some features from my signal series and feed them to the algorithm instead. My problem is that I do not know what features to extract.

The majority of papers that I found focus on human activity recognition to differentiate between walking, running, ascending, and descending stairs. They were not very helpful in the regard that they have continuous data flow of a person walking/running for hours and they use much more sample data. In contrast, with my task, I have data set instances that are separate from each other.
I am not asking to solve those whole tasks for me, just guide me into some direction with a good explanation of why it might be useful.

Regarding your first problem:
I would suggest that you do not discard the the amount of values even if it goes more than or less than 250. Instead what you can do is ; aggregate the values of accelerometer over a time intervals and then tie it up to a single action.

Regarding your second problem:
You have around 750 columns of the data. It would be very difficult to use random forest algorithm on it and get higher accuracy on it. You have to apply dimension reduction and then feature extraction techniques. You can go ahead with PCA (principal component analysis) f 750 independent variables. Reduce it down to 2 or 3 variable and check how much  variance these reduced variables can explain. if it less than 60%. You can apply T-SNE algorithm to extract features more on it.

P.S. to check if your reduced variable can explain your dependent variables (like squat, sitting, pushup), you can plot the scatter plot of reduce variable values and then color the values based on your dependent variable. you can click below link to understand what I am saying

https://blog.bioturing.com/2018/06/18/how-to-read-pca-biplots-and-scree-plots/


GridSearchCV using Random Forest Reg Pipeline rfr = RandomForestRegressor(n_estimators=1, min_samples_leaf=7)
preprocess = make_column_transformer(
                 (StandardScaler(), ~categorical),
                 (OneHotEncoder(handle_unknown='ignore'), categorical))

param_grid = {"RandomForestRegressor__n_estimators": [1,10,20,50,75,200],
              "RandomForestRegressor__min_samples_leaf": range(1,10)}
pipe = make_pipeline(preprocess, RandomForestRegressor())

grid = GridSearchCV(pipe, param_grid, cv=5)

When I do grid.fit(X,y) I am getting following error. Why?

ValueError: Invalid parameter RandomForestRegressor for estimator Pipeline(memory=None,
         steps=[('columntransformer',
                 ColumnTransformer(n_jobs=None, remainder='drop',
                                   sparse_threshold=0.3,
                                   transformer_weights=None,
                                   transformers=[('standardscaler',
                                                  StandardScaler(copy=True,
                                                                 with_mean=True,
                                                                 with_std=True),
                                                  VendorID                 False
store_and_fwd_flag       False
RatecodeID               False
PULocationID             False
DOLocationID             False
passenger_count           True
trip_distance             True...
                 RandomForestRegressor(bootstrap=True, ccp_alpha=0.0,
                                       criterion='mse', max_depth=None,
                                       max_features='auto', max_leaf_nodes=None,
                                       max_samples=None,
                                       min_impurity_decrease=0.0,
                                       min_impurity_split=None,
                                       min_samples_leaf=1, min_samples_split=2,
                                       min_weight_fraction_leaf=0.0,
                                       n_estimators=100, n_jobs=None,
                                       oob_score=False, random_state=None,
                                       verbose=0, warm_start=False))],
         verbose=False). Check the list of available parameters with `estimator.get_params().keys()`.

The problem seems to be that your pipeline uses a fresh instance of RandomForestRegressor, so your param_grid is using nonexistent variables of the pipeline.  There are two choices (I tend to prefer the second):

Use rfr in the pipeline instead of a fresh RandomForestRegressor,  and change your parameter_grid accordingly (rfr__n_estimators).

Change param_grid to use the lowercased name randomforestregressor__n_estimators; see the docs on make_pipeline:

it ... does not permit naming the estimators. Instead, their names will be set to the lowercase of their types automatically.


How to perform feature selection on dataset with categorical and numerical features? I am working on a dataset with 30 columns (29 numerical, 1 non-ordinal categorical). I hot-encoded the categorical feature and reached at 35 columns. To improve training efficiency, I want to perform feature selection on my dataset. However , I am confused with how to handle a dataset with categorical and numerical features combined.

I read that it is not reasonable to apply PCA on dummies given they are discrete. Is it reasonable to apply PCA first on numerical features then concatenate them with dummies?
I tried to implement recursive feature elemination with cross-validation (RFECV) to the entire feature space. But I don't think it is reasonable to remove some but not all dummy features given they are generated out of one category.

Any suggestions? Any help is appreciated.
python pandas scikit-learn feature-selection

It is fine to apply feature selection technique on one hot encoded variables. Because if one particular segment of that variable is correlated with your target, then it is a good news. Your model will understand the scenario better.
Or, You can label encode your categorical variable first so that you still have 30 variables (29 numerical + 1 label-encoded categorical variable). Now try to find the importance value of each variable, and take the relevant ones (Use any method for it: be it RFE, random forest feature selection, pearson's correlation etc). Once you have the final list of variables, and the label-encoded variable is also coming as relevant, it is ok to put it into the model.


Relational Data Mining without ILP I have a huge dataset from a relational database which I need to create a classification model for. Normally for this situation I would use Inductive Logic Programming (ILP), but due to special circumstances I can't do that.

The other way to tackle this would be just to try to aggregate the values when I have a foreign relation. However, I have thousands of important and distinct rows for some nominal attributes (e.g.: A patient with a relation to several distinct drug prescriptions). So, I just can't do that without creating a new attribute for each distinct row of that nominal attribute, and furthermore most of the new columns would have NULL values if I do that.

Is there any non-ILP algorithm that allows me to data mine relational databases without resorting to techniques like pivoting, which would create thousands of new columns?

First, some caveats

I'm not sure why you can't use your preferred programming (sub-)paradigm*, Inductive Logic Programming (ILP), or what it is that you're trying to classify. Giving more detail would probably lead to a much better answer; especially as it's a little unusual to approach selection of classification algorithms on the basis of the programming paradigm with which they're associated. If your real world example is confidential, then simply make up a fictional-but-analogous example.

Big Data Classification without ILP

Having said that, after ruling out ILP we have 4 other logic programming paradigms in our consideration set:

Abductive
Answer Set
Constraint
Functional

in addition to the dozens of paradigms and sub-paradigms outside of logic programming.

Within Functional Logic Programming for instance,  there exists extensions of ILP called Inductive Functional Logic Programming, which is based on inversion narrowing (i.e. inversion of the narrowing mechanism). This approach overcomes several limitations of ILP and (according to some scholars, at least) is as suitable for application in terms of representation and has the benefit of allowing problems to be expressed in a more natural way.

Without knowing more about the specifics of your database and the barriers you face to using ILP, I can't know if this solves your problem or suffers from the same problems. As such, I'll throw out a completely different approach as well.

ILP is contrasted with "classical" or "propositional" approaches to data mining. Those approaches include the meat and bones of Machine Learning like decision trees, neural networks, regression, bagging and other statistical methods. Rather than give up on these approaches due to the size of your data, you can join the ranks of many Data Scientists, Big Data engineers and statisticians who utilize High Performance Computing (HPC) to employ these methods on with massive data sets (there are also sampling and other statistical techniques you may choose to utilize to reduce the computational resources and time required to analyze the Big Data in your relational database).

HPC includes things like utilizing multiple CPU cores, scaling up your analysis with elastic use of servers with high memory and large numbers of fast CPU cores, using high-performance data warehouse appliances, employing clusters or other forms of parallel computing, etc. I'm not sure what language or statistical suite you're analyzing your data with, but as an example this CRAN Task View lists many HPC resources for the R language which would allow you to scale up a propositional algorithm.


How do I calculate the accuracy for graph mining in terms of (top 1%)? I have 3600 samples in my dataset. I split the dataset into the train (2700) and test (900).
My problem is related to new link prediction. I am using the Common Neighbor (CN) method. Using CN, we can predict new links based on the score. The score comes from the CN formula. CN gives a score for every possible edge/path/link. A higher score means a higher possibility to be a new link.
Now, I applied my train dataset to the CN methods. From CN, I have received 4200 new links with a score. From this 4200, I took only the top 1% (based on the score given by CN). So, the top 1% is 42.
Now, I tried to match the 42 with my test dataset. Means, actually, how many edges/links/path I can recover which were missing in the training dataset.
Among these 42 (CN output), 38 samples are perfectly matched with the test dataset (means same edge/link/path of the train and test). But, I did not give or these 38 edges/link/paths were missing in the training dataset.
True Positive: 38
False Positive: 42-38 = 04
False Negative: 900-38 = 862
True Negative: 900 - (38 + 04 + 862) = -04

From this, I am trying to calculate Precision, Recall, Accuracy. But, what is happening, I am getting very little Accuracy and Recall.
Can you give me an idea to improve the Accuracy and Recall?

You have a mining problem which is basically a search problem. In search problem you have TP, FP, FN but not TN (you have it but it is trivial, imagine search in google. Basically bilions of webpages are TN in every search. right?)
I also assume that you are not interested in ranking results as you did not mention it in the question.
In this case you can easily go with Precision and recall:

TP: Graphs which are right and you could find.
FP: Graphs which are not right but your model found.
FN: Graphs which are right but your model did not find.

Then you simply calculate:
Precision: $$\frac{TP}{TP+FP}$$
Recall: $$\frac{TP}{TP+FN}$$
And as a combined score, you may calculate F1:
$$\frac{2TP}{2TP+FP+FN}$$
About Accuracy
When the volume of search space is huge, as I said above, accuray is not meaning much. According to the formulation:
$$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$
when $TN$ is very large then the whole fraction approaches zero and does not inform much.
But if your search space is relatively small, you can calculate accuracy with the formulation above. However it was just a theoretical justification and still your best metrics are precision and recall and their related metrics.
Update For Your Specific Question
Your set of top 1% results may have different sizes due to different size of your model output (If it is always 4200 please write in the comments but in "mining" questions usually the size of output is variant).
In your case TP and FP are pretty straight-forward and meaningful. About FN, if your test set is also ranked, then you can evaluate your results according to top 1% of your ground-truth which includes 90 samples and go on as you already did, or you can fix the size of ground-truth set equal to your 1% (i.e. take 42 top test results) and again calculate same scores. The first one is more stable and significant for comparing different models.
A more precise metric which is related to the ranked results is called NDCG.
If the test set is not ranked then go this way:

$|correctly\_found\_graphs|$: Number of graphs that you correctly found.
$|wrongly\_found\_graphs|$: Number of graphs you found but they are not in ground-truth
$|all\_found\_graphs|$: here 42
$|all\_correct\_graphs|$: here 900

Then calculate Precision@k (here precision@42) metric which calculates the precision only on the top k results of a search output:
$Precision@42 = \frac{|correctly\_found\_graphs|}{|all\_found\_graphs|}=\frac{38}{42}$
This is a well-established information retrieval metric.
$Recall = \frac{|correctly\_found\_graphs|}{|all\_correct\_graphs|}=\frac{38}{900}$
How can I improve the recall?
As long as size of your top 1% results is less than ground-truth, your recall is limited to an upper-bound i.e. even if all 42 all relevant, still your recall is 42/900. Knowing this fact you can use recall but I do not see the point. Recall makes more sense if you take all 4200 outputs into account.
Your calculation of True Negative in the comments is also wrong. TN in your case is the number of all graphs that your model should not have returned and it didn't i.e. 900-42
Moreover, all samples are positive in my test dataset. Not single sample is negatives
See the problem from an Information Retrieval point of view. In this approach, any graph which is not in 900 set is a negative sample.
You can comment in case it is still unclear.


how to build a lift chart using scikitplot I am trying to build a lift chart using scikitplot library.
I am getting the following can anyone guide with the error.
please find the code with the error
CODE:
import scikitplot as skplt
actual = df['Actual']
predicted = df['Pred']

skplt.metrics.plot_lift_curve(act,pdt)

Error:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-38-f14ae9c16143> in <module>
      6 pdt = np.array(predicted)
      7 # plotLiftChart(actual,predicted)
----> 8 skplt.metrics.plot_lift_curve(act,pdt)
      9 # plt.show()

~\Anaconda3\lib\site-packages\scikitplot\metrics.py in plot_lift_curve(y_true, y_probas, title, ax, figsize, title_fontsize, text_fontsize)
   1192
   1193     # Compute Cumulative Gain Curves
-> 1194     percentages, gains1 = cumulative_gain_curve(y_true, y_probas[:, 0],
   1195                                                 classes[0])
   1196     percentages, gains2 = cumulative_gain_curve(y_true, y_probas[:, 1],

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

please guide where I have done the mistake

The mistake is not to have read the documentation ;)

scikitplot.metrics.plot_lift_curve(y_true, y_probas,...)
Parameters:

y_true (array-like, shape (n_samples)) – Ground truth (correct) target values.
y_probas (array-like, shape (n_samples, n_classes)) – Prediction probabilities for each class returned by a classifier.

The second parameter y_probas should be a double-dimension array of the predicted probabilities for every class, not a single dimension array of the predicted class.
You can also check the example provided in the documentation which contains:
y_probas = lr.predict_proba(X_test)


Clustering Data to learned cluster I am new to data science, I have clustered some data using Scipy agglomerative clustering. how can I fit new data into the learned clusters?

dm = pdist ( dataset ,lambda u,v: mlpy.dtw_std ( pd.Series(u).dropna().values.tolist(),pd.Series(v).dropna().values.tolist(),dist_only=True ))
z = hac.linkage(dm, method='average')
cluster = hac.fcluster(z, t=100, criterion='distance')
leader = scipy.cluster.hierarchy.fcluster(z, t=100, criterion='distance')

I would like to cluster new data into the same clusters, How can I do it?

The word "prediction" does not belong to any specific type of machine learning. There is nothing wrong with "predicting" new data to the cluster it belongs to; (e.g. there are many applications that place new customers into pre-discovered market segments). A conditional probability, like that used in classification, is not "stronger" than an unsupervised approach, as it rests its assumption on properly labelled classes; something that is not guaranteed.

This is why there are packages that provide a predict function to clustering algorithms. Here is an example using the flexclust package with the kcaa function. That being said, the prediction step is usually handled by a supervised classifier, so the approach would be to sit a classifier on top of your learned clusters (treating cluster assignments as "labels").

You just have to reason about your weaknesses. As stated above, the weakness in classification is the assumption that labelled data is tagged correctly, whereas the weakness in clustering is that your discovered clusters are assumed to be valid. Unsupervised approached cannot be validated the same way it is done with classification. Clustering requires a variety of cluster validity techniques along with domain experience (e.g. show campaign managers your market segments to validate customer types).

Ultimately, you are just matching an incoming vector (new data) to the cluster most similar. For example, in k-means this could be accomplished by finding the smallest distance between the incoming vector and all the centroids of your clusters. This kind of pattern matching depends on the data you are using.

This works best for clustering techniques that have well-defined cluster objects with exemplars in the center, like k-means. Using hierarchical techniques means you would need to cut the tree to obtain flat clusters, then use the "label" assignment to run a classifier on top. This comes baked with a lot of assumptions, so you need to make sure you understand your data very well, and validate any clusters with non-technical users that have deep domain experience.

POSSIBLE APPROACH
If you're bent on using hierarchical clustering, then here is the general approach. Note I am not suggesting this is the best way. Every approach comes baked with a number of assumptions. You will need to work to understand your data, attempt many models, validate with stakeholders, etc.

Readers can use the tutorial by Jörn Hees to get started in hierarchical clustering if needed:

Create some example data:

from matplotlib import pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage
import numpy as np

np.random.seed(42)
a = np.random.multivariate_normal([10, 0], [[3, 1], [1, 4]], size=[100,])
b = np.random.multivariate_normal([0, 20], [[3, 1], [1, 4]], size=[50,])
X = np.concatenate((a, b),)

Confirm clusters exist in synthetic data:

plt.scatter(X[:,0], X[:,1])
plt.show()

Generate the linkage matrix using the Ward variance minimization algorithm:
(This assumes your data should be be clustered to minimize the overall intra-cluster variance in euclidean space. If not, try Manhattan, cosine or hamming. You can also try different linking options).

Z = linkage(X, 'ward')

Check the Cophenetic Correlation Coefficient to assess quality of clusters:

from scipy.cluster.hierarchy import cophenet
from scipy.spatial.distance import pdist

c, coph_dists = cophenet(Z, pdist(X))

0.98001483875742679

Calculate full dendrogram:

plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z,
    leaf_rotation=90.,  # rotates the x axis labels
    leaf_font_size=8.,  # font size for the x axis labels
)
plt.show()

Determine the number of clusters (e.g. can be done manually by looking for any large jumps in the dendrogram...see Jörn's blog for plotting function):

Retrieve clusters: (using our max distance determined from reading dendrogram)

from scipy.cluster.hierarchy import fcluster

max_d = 50
clusters = fcluster(Z, max_d, criterion='distance')

Map cluster assignments back to original frame:

import pandas as pd

def add_clusters_to_frame(or_data, clusters):
    or_frame = pd.DataFrame(data=or_data)
    or_frame_labelled = pd.concat([or_frame, pd.DataFrame(clusters)], axis=1)
    return(or_frame_labelled)

df = add_clusters_to_frame(X, clusters)
df.columns = ['A', 'B', 'cluster']

df.head()

Build a classifier using this "labelled" data:

Here, I'll just use the original data and the assigned clusters along with a knn classifier:

np.random.seed(42)
indices = np.random.permutation(len(X))
X_train = X[indices[:-10]]
y_train = clusters[indices[:-10]]
X_test  = X[indices[-10:]]
y_test  = clusters[indices[-10:]]

# Create and fit a nearest-neighbor classifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(X_train, y_train)

res = knn.predict(X_test)
print(res)
print(y_test)

predicted labels: [2 2 1 1 2 2 1 2 2 1]

test labels: [2 2 1 1 2 2 1 2 2 1]

As with any classifier, your incoming data needs to be in the same representation as your training data. As new data arrives you run it against the predict function provided by your classifier (here we use sci-kit learn's knn.predict). This effectively assign new data to the cluster it belongs.

Ongoing cluster validation would be required in the model monitoring step of the machine learning workflow. New data can change the distribution and results of your approach. BUT, this isn't unique to unsupervised as all machine learning approaches will suffer from this (all models eventually go stale). As argued by Jörn in the reference above, manual inspection typically trumps automated approaches, so regular visual/manual inspection of the flat clusters is recommended.


What is the difference between outlier detection and anomaly detection? I would like to know the difference in terms of applications (e.g. which one is credit card fraud detection?) and in terms of used  techniques.

Example papers which define the task would be welcome.

(I actually wanted to write this as an answer to the Cross Validated question: Difference between Anomaly and Outlier, but the question is protected - I think answering it here should be fine, despite the lower visibility)

People occasionally argue that there is no difference between an outlier and an anomaly by citing Charu Aggarwal, author of the Book "Outlier Analysis" - particularly, this statement:

  Outliers are also referred to as abnormalities, discordants, deviants, or anomalies in the data mining and statistics literature.

(Source: "Outlier Analysis" (Springer), Charu Aggarwal, 2017, http://charuaggarwal.net/outlierbook.pdf )

However, this statement  does not imply that outliers and anomalies are the same thing - analogously to saying that "Dogs are sometimes referred to as animals" does not mean that they are the same thing.

It's hard to give a formal definition of the terms. The Wikipedia page about outliers refers to the Wikipedia page about anomaly detection and vice versa, and they both contain lots of  possible definitions and interpretations of the terms. Things are becoming worse due to the domain-specific definitions and colloquialities, where it seems to be sufficient when two people of the same field roughly know what the other one is talking about...

However, Varun Chandola tries to give a more precise meaning to the term "anomaly" in his anomaly detection survey. Particularly, he classifies anomalies into three categories:

Point anomalies: An individual data instance can be considered as
anomalous with respect to the rest of data
Contextual Anomalies: If a data instance is anomalous in a specific context (but not otherwise)
Collective Anomalies: If a collection of related data instances is anomalous with respect to the entire data set

(Summarized from "Anomaly Detection - A Survey", Varun Chandola et al, ACM Computing Surveys 2009, http://cucis.ece.northwestern.edu/projects/DMS/publications/AnomalyDetection.pdf )

Here, the term "point anomaly" seems to be closest to what I'd consider as a possible definition of the word "outlier". And this is in line with the statement by Aggarwal: An outlier is an anomaly. But not every anomaly is an outlier.

(The latter may depend on the definition of the word outlier. Of course, one can define it on a meta-level, and say that an outlier is whatever a certain outlier detection algorithm (or model) detects as such. But most definitions that I encountered so far are based on some sort of "distance", "dissimilarity", or "difference" from a "majority" of other data elements. That sounds reasonable...)

An example: There may be several data points:

14.5, 14.2, 14.4, 14.4, 14.4, 14.4, 14.4, 14.4, 14.4, 14.3, 14.2, 14.6

One can compute the mean and standard deviation and will have a hard time arguing why one of these points should be an "outlier".

For a sequence of data points like this

14.5, 14.2, 14.4, 14.4, -64564.4, 14.4, 14.4, 14.4, 14.4, 14.3, 14.2, 14.6

spotting "the outlier" should be easy.

However, assuming that the first sequence describes, for example, average daily outside temperatures, the fact that the exact same average temperature of 14.4 degrees was measured for a whole week could certainly be considered as an "anomaly".

(Probably a "collective anomaly" according to the definitions above, but I won't argue about that...)

Although I'm on thin ice when arguing about the precise or intuitive meaning of certain terms (because I'm neither a data science expert nor a native English speaker), this would mean that "anomaly" is a much broader term than "outlier". But maybe the data science community is just in the process of sorting out proper definitions of these terms.

Update:

Maybe my gut feeling about the literal meaning of certain words is wrong. But for me, the word "outlier" seems to say "lying somewhere out of (or far away from) something (based on some distance measure)". In that sense, the 14.4s in the first example are not "outliers" per se. But of course, things become tricky very quickly here: One could imagine a model for the data that contains the number of consecutive days with equal temperatures (as in a run length encoding). Computing this model for the given data would yield

1 * 14.5
1 * 14.2
7 * 14.4
1 * 14.3
1 * 14.2
1 * 14.6

where the value 7 does have large distance (difference) to the other values in the model. So the "collective anomaly" of 7 consecutive days with equal temperatures has been turned into a "point anomaly" by this transformation.


Encoding correlation I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods.

Let's assume we have data set with highly correlated attributes. Usually encoders are trained to learn representation in lesser number of dimensions. What I'm wondering about is quite the opposite. Would it be possible to learn encoding to higher number of dimensions but less correlated (wishfully non-correlated)? The idea is to turn less-dimensional, very tough problem to high-dimensional but easier one. Kinda unwrap those intricate correlations using NN and decode solutions later.

Edit 1
Of course we assume we know correlation mapping really good. How exactly could I use correlation mapping to unwrap it? Is it fundamentally possible to unmap attribute dependencies?

You could do that with a neural autoencoder using a custom loss function.
Use a hidden layer, let's call it $l_{encoded}$,  with more nodes than the features of the input data.

You have to code the custom loss:
$loss = corr(o(l_{encoded}))+MSE(o(l_{output}),\  input)$,

where $corr(o(l_{encoded}))$ is the correlation of the output of the encoding layer and $MSE(o(l_{output}), input)$ is the mean squared error of the last layers output and the input instance.

Using this loss your model will try to reduce the correlation of the hidden layer while still making sure that it is able to decode the training instance.

I highly doubt this will be of any use though.


How does a Bayes regularization works? I'm trying to get grasp of Bayes regularization algorithm.
List of symbols 1st:
$F$ - objective function
$\gamma$ - regularization parameter
$M$ - number od neural network weights
$N$ - number of data tuples
$e$ - modeling error
$w$ - net weight
$D$ - data set input-output pair
$H$ - Hess matrix of objective function computed based on Jacoby's matrix

I get it that minimalising an objective function:
$$
F = \gamma\sum\limits _{j=1} ^Mw_j^2 + (1-\gamma)\sum\limits_{i=1}^Ne_i^2
$$
is equivalent to maximising likelihood $P(w|D,\gamma)$, which can be computed according to Bayes' theorem:
$$
P(w|D,\gamma) = \frac{P(D|w,\gamma)P(w|\gamma)}{P(D|\gamma)}
$$
Likelihood $P(w|\gamma)$ is assumed to be a gaussian one and can be computed as:
$$
P(w|\gamma)=\left(\frac{\gamma}{2N}\right)^{\frac{M}{2}}\cdot e^{-\frac{\gamma}{2}w^Tw}
$$
Likelihood $P(D|\gamma)$ can be computed too as:
$$
P(D|\gamma)=\left(\frac{\pi}{\gamma}\right)^{-\frac{N}{2}}\left(\frac{\pi}{1-\gamma}\right)^{-\frac{M}{2}}\frac{(2\pi)^\frac{M}{2}\cdot -e^{-F(w)}}{\sqrt{|H|}}
$$
And now my questions are: how is $P(D|w,\gamma)$ computed? Or maybe is there any special assumptions about it? And should I minimalise or maximise $P(w|D,\gamma)$ to minimalise objective function $F$.
I base on those articles:
Article1
Article2

If anyone is able to explain those thing to me, I'd be most grateful.
Thank you,
Max

I have found answer for my question here.
If anyone needs it in future: all above likelihoods are assumed to be a gaussian distributions. Likelihood $P(\gamma)$ is assumed to be uniform. In the article it is shown how exactly are they defined and how to compute further is described in detail in article linked above.


What model to fit pseudo-logarithmic trend? I am dealing with a scatterplot where I am trying to figure out the relationship between two variables, but I have so far failed to identify what the best fit could be. The presence of zero values prevents me from using exponential and power law regression lines.

Zero values are not shown as the Y axis starts from 1, but they are present. My bad.

This is the scatterplot. To me it looks like this is some sort of logarithmic trend but I am not sure about it.

What model will best interpret this trend?

One option is to take a curve fitting approach. For example, scipy.optimize.curve_fit.
Something like this:
import numpy as np
from scipy.optimize import curve_fit

def f(x, a, b, c):
    "Rough functional form of the data"
    return a + b * (1 - np.exp(-c/x))

solution = curve_fit(f, c, y, p0=(3_500, 3_500, 200))


How to pass common inputs to a merged model in Keras I'm attempting to merge a VGG-16 and ResNet-50 through concatenation. I was successful in training and saving the merged model. Here is the code snippet:

from keras import applications
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D
from keras.layers import Concatenate
from keras.models import load_model
# Loading the training data
img_rows=300
img_cols=300
channel = 3
num_classes = 3
batch_size = 10
nb_epoch = 10
#load the first model #########################################################
base_model1 = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))
#get the model summary
base_model1.summary()
#addind the top layers
x1 = base_model1.output
x1 = GlobalAveragePooling2D()(x1) #analogous to flatten()
model1 = Model(inputs=base_model1.input, outputs=x1)
model1.summary()
#load the second model ########################################################
base_model2 = applications.ResNet50(weights='imagenet', include_top=False, input_shape=(img_rows,img_cols,3))
#get the model summary
base_model2.summary()
#addind the top layers
x2 = base_model2.output
x2 = GlobalAveragePooling2D()(x2) #analogous to flatten()
model2 = Model(inputs=base_model2.input, outputs=x2)
model2.summary()

'''merge the models'''
mergedOut = Concatenate()([model1.output,model2.output])
#add a new dense layer and softmax
out=Dense(2048, activation='relu')(mergedOut)
out = Dense(num_classes, activation='softmax', name='predictions')(out)
#create the new model with three branches and one dense layer
model = Model(inputs=[model1.input,model2.input], outputs=out)
model.summary()
###############################################################################
#training the model
hist=model.fit([X_train, X_train], Y_train, batch_size=batch_size, epochs=nb_epoch,
                     shuffle=True, verbose=1, validation_data=([X_valid, X_valid], Y_valid))

However, the inputs to both the models are the same. I would like to use this as a common input to the merged model to avoid passing X_train and X_valid twice during the model.fit. How can I do this?

I found the answer to my question. I need to pass the common input shape to the individual models and concatenate the outputs like this:

input_shape=(img_rows,img_cols,3)
commonInput = Input(input_shape)
out1 = model1(commonInput)
out2 = model2(commonInput)
mergedOut = Concatenate()([out1,out1])


Cluster doc2vec using Affinity Propagation I want to cluster my document vectors (doc2vec) using affinity propagation.

However, I am just confused if I should use cosine similarity or cosine distance to cluster my document vectors. Currently, I am using cosine similarity for my affinity propagation clustering. Thus, my first question is;

Is it correct to use cosine similarity to cluster my doc2vec document vectors?

Moreover, I would like to visualize my cluster results using t-sne. However, I saw that t-sne requires distance matrix as the input. Hence, my second question is;

Is it correct to use distance matrix (cosine distance) for t-sne, while I use cosine similarity for clustering?

If my code is required I can post it too.

Please help me.

Both of them convert the distances back to similarities, albeit using different methods. They will, if I recall correctly, also square the distances.

This may be problematic with the most common variant of cosine distance, which already is a squared distance. So it may be a good idea to modify the methods to be able to directly work with the similarities. But you probably need to modify the source code for this (and understand the methods!)


Can gradient boosted trees fit any function? For neural networks we have the universal approximation theorem which states that neural networks can approximate any continuous function on a compact subset of $R^n$.

Is there a similar result for gradient boosted trees? It seems reasonable since you can keep adding more branches, but I cannot find any formal discussion of the subject.

EDIT: My question seems very similar to
Can regression trees predict continuously?, though maybe not asking exactly the same thing. But see that question for relevant discussion.

Yes - create a region for each data point (i.e., memorize the training data).

Thus it is possible for gradient boosted trees fit any training data, but it would have limited generalization to new data.


Random seed in Machine learning model comparison I would like to ask a question about the random population generation gin splitting the dataset in machine learning classification models.
For example, I used seed = 1 and got accuracy of 0.7 and seed = 5 and got accuracy of 0.8 and seed= 2000 and got accuracy of 0.89 using Adaboost.
I found a research paper using the same dataset i used and accuracy achieved is 0.94 using xgboost model without specifying the seed used in developing the model.
Same exists for other research papers as well.
Which results I need to pick to compare my model with other models proposed in the literature? Meanwhile I implemented all the models proposed in literature with the different seed and found that results are different when compared to the paper. And sometimes without using my result with adaboost is better.
I need help to compare my proposal with other works.

I agree that sometimes the authors are not that clear about some details. If a dataset is already splitted into train/test/validation, then there is not much to do (supposing the dataset is well made). Given a dataset not already splitted then you have (at least) two ways to test your model:

fix a random seed, check that the split is not biased and run several trainings. Even with the same split, the same model can converge to different local minima, because of the stochastic descent;
don't fix a random seed and run several trainings (but store the seed where you keep your experiment data!).

If your findings are not coherent with the literature, and you are sure there aren't bugs in the code, then you should ask specific questions or write to the authors. Depending on the model, you might try some standard validation techniques.


Not Access to Confusion Matrix in SVM.SVC.score Scikit-learn Python I used SVM.SVC function to classify. But when I wanted to calculate the weighted and unweighted average accuracy I couldn't access the confusion matrix. Because of svm.SVC.score only provides a classifier accuracy percentage. How can I calculate WAR and UAR?

you can find part of my script below :

'''

    scaler = StandardScaler()
    scaler.fit(trainX)
    trainXsc = scaler.transform(trainX)
    testXsc = scaler.transform(testX)

    pca = KernelPCA(n_components=j, kernel="sigmoid", random_state=1)

    pca.fit(trainXsc)     # fit pca kernel with train data

    trainXtr    = pca.transform(trainXsc) # transform FV with PCA and dimension reduction
    testXtr     = pca.transform(testXsc)

    svmObject   = svm.SVC(C=2.0, kernel='rbf', degree=3, gamma='scale', coef0=0.0, shrinking=True,
                  probability=False, tol=0.001, cache_size=200, class_weight=None,
                  verbose=False, max_iter=-1, decision_function_shape='ovo', random_state=None)
                                        # SVM Kernel Function

    svmObject.fit(trainXtr, trainY)      # train SVM kernel with train FV

    result = svmObject.score(testXtr, testY)

'''

Instead of using the score method on your trained model, you should use the predict method.

You can then pass the results into the confusion matrix function from sklearn:

from sklearn.metrics import confusion_matrix
y_pred = svmObject.predict(X)
cm = confusion_matrix(y_true, y_pred, sample_weight=sample_weight,
                      labels=labels, normalize=normalize)

There is also a nice function called plot_confusion_matrix:

from sklearn.metrics import plot_confusion_matrix

plot_confusion_matrix(svmObject, testXtr, testY)
plt.show()


Handling NA Values in the Chicago Crime Rate data set I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields.
I feel that even though there are about  6 million rows (data from 2001 - present) that is a lot of data to drop (especially since the rows contain all other data like crime type, ward, date, location description, etc.)

Here are the columns and the number of NA's found in each column:

COMMUNITY_AREA ID CASE_NUMBER DATE BLOCK IUCR PRIMARY_TYPE DESCRIPTION
        616029  0           0    0     0    0            0           0
LOCATION_DESCRIPTION ARREST DOMESTIC BEAT DISTRICT   WARD FBI_CODE X_COORDINATE
                   0      0        0    0       47 614854        0        60921
Y_COORDINATE YEAR UPDATED_ON LATITUDE LONGITUDE LOCATION CRIME_TYPE COMMUNITY_NAME
       60921    0          0    60921     60921        0          0         616120

When I look up RPubs for this project a lot of people are either dropping all rows with NA's or not even bothering to talk about the missing data, both solutions in my opinion are Not good solutions.

Part of why I don't want to drop all those rows is because each row is a valid crime, when I drop them I am now missing crimes, this contributes to my counts, and categories. And since the data does say what type of crime was committed I can include it in these counts.

Has anyone worked with this data before? Do you have a suggestion for handling the missing data? Or can I leave it in there, is there an issue with that?

I plan on doing a time series analysis on Crime Rate, however, there is no missing data in the crime field so I do not think it will effect it.

Assuming Im reading the format of you table correctly, the NA's are exclusively for geo-location attributes. With "District" being the most complete field with only 47 missing values. Are the locations of certain crimes censored for data privacy reasons?

Of the 6 M crimes in your data set:

10 % or ~600,000 are missing or censored for Ward, Community area or name
1 % or ~60,000 are missing spatial coordinates.

You could determine most of the missing Ward and Community data by analysing the coordinate pairs in a GIS.

How you handle missing data will largely be determined by the hypothesis and analysis you intend to do. You mention Time Series, it should be fine to include all crimes for straight time series at the Chicago level. For higher spatial resolutions, you can do district (dropping 47 Missing points), and so on. How much effort you devote to handling the missing data, such as by using steps outlined above, will depend on how useful that data may be for your analysis. I suspect that most people working with this data tend to drop the NA's as they are mapping the crime rates in different areas.


What is the reasonable max number of features for LSTM? I've been trying to find information on how many features I can use in LSTM.
I found that LSTM can handle up to 500-1000 timesteps, but what about features (sequences)?

Can I use 1000 features per timestep?

What is the working maximum and what affects that number? Thank you.

In fact, there is no maximum or reasonable amount of features for LSTM layers in general. Everything is extremely task specific.

Before training an LSTM model with that amount of variables you'd encounter a number of practical problems, for example:

Prohibitive computational costs. I don't think any machine could handle that without crashing.
Its need of massive amounts of data. Such a model would require a dataset of unimaginable size, in order to be trained properly.

But that said, practical problems apart, there is no theoretical threshold to the number of features you can feed into RNNs.


Predictive clustering I have an hypothesis but i don't know if it's true.

If the cluster is dense and we apply a supervised learning on this data, the model generated by this cluster will be more efficient for new data falling into this cluster than other.
Thus we have multiple cluster with associate models

How choose the right associate model to optimize our prediction ?
We can compare the value of the point to cluster's mean and apply the right model !

Any suggestions ?

You could also try tSNE, also available in scikit-learn. It is a probabilistic dimensionality reduction technique, specifically designed for plotting data in two or three dimensions, while preserving the original distances between the samples as much as possible.

You should be able to see how well your data is clustered.


Neural network got a lucky guess. Can it be trusted? Say you come across a loss curve as shown below. At which loss should you trust the model? The initial lucky guess or after it has stabilized?

And more importantly, why?

The loss in the curve suggests that the training can be improved by tuning hyperparameters, especially the learning rate and/or the batch size. Therefore the optimal decision would be to keep refining the model instead of using the already trained model.

If tuning hyperparameters is not an option, at least you should re-split training and validation data or at very least repeat with a different random seed.

If none of those is an option, you should take the best validation loss. The reason is that we are assuming that there is no leak of the validation data to the training data and that the validation data is representative of the data the model is going to be tested with. Without any other evidence, we should assume that the "sweet spot" you found would also lead to better results in a piece of test data the model has not seen before.


Can I apply Feature Selection even though the number of features is smaller than the number of observations? I was reading An Introduction to Statistical Learning when I came across the High Dimensional section, they argue that high-dimensional data causes a lot of problems. My question is, it is necesary to have more features than observations to apply any feature selection model?
If so, can you provide me with any article that talks about it?

No matter if the number of features is bigger or smaller of the number of samples, applying feature selection to identify the features with the biggest influence before proceeding with the modeling improves the model performance and helps prevent overfitting and eliminate noise bias (of course, in cases where the sample size is smaller than the number of features the need for dimensionality reduction is even bigger!).
Keep in mind that feature selection is not the only way to achieve dimensionality reduction, it can also be achieved with the extraction of new, more compact features from the initial ones (the only negative thing with that is that you cannot have clear physical interpretation of the selected features).
It is not clear from you question if you have in mind some regression or classification task (or just generally asking), but in both cases the general logic is the same. Nevertheless, I have provided examples for both:

Here is an example applying most common techniques for dimensionality reduction for classification (including feature selection ones) on the Iris dataset.

Here is an example applying feature selection for linear regression on a sample of size 1000 and 100 features.

Also, here is some additional reading in case you want to undertand better the whole dimensionality reduction subject:

An article discussing dimensionality reduction and suggesting alternative methods.

In this link, dimensionality reduction (pluses and minuses) is again discussed and the most popular linear (feature extraction, no feature selection!) method - PCA - is explained.

A review paper discussing state of the art methods for Feature Selection for Classification can be found here.

I hope this helped! :)


How to use PCA in CNN for image recognition using Keras? I created a CNN model for image classification and I want to use Principal Component Analysis (PCA) but when I run pca.fit() code, the code still running for hours and the RAM become full. So, I want to know how to use PCA in CNN for image recognition using Keras?

My code:

#Data files
train_iris_data = 'Iris_Database_01/Training'
valid_iris_data = 'Iris_Database_01/Validation'
test_iris_data = 'Iris_Database_01/Testing'

#Image data generator
train_iris_datagen = ImageDataGenerator(
rotation_range=10,
shear_range=0.2,
zoom_range=0.1,
width_shift_range=0.1,
height_shift_range=0.1
)

test_iris_datagen = ImageDataGenerator()

#Image batches
image_size = (224, 224)
batch = 32

# Training
train_iris_generator = train_iris_datagen.flow_from_directory(
train_iris_data,
target_size=image_size,
batch_size=batch,
class_mode='categorical')

# Validation
validation_iris_generator = test_iris_datagen.flow_from_directory(
valid_iris_data,
target_size=image_size,
batch_size=batch,
class_mode='categorical',
shuffle = False)

# Testing
test_iris_generator = test_iris_datagen.flow_from_directory(
test_iris_data,
target_size=image_size,
batch_size=1,
class_mode='categorical',
shuffle = False)

pca = PCA(n_components=2)
pca.fit(train_iris_generator)

#pca = PCA(n_components=0.8)
#pca.fit(train_iris_generator)

Standard implementations of PCA calculates statistics across the entire dataset in order to find the projection that has the greatest variance. The entire dataset needs to be loaded into memory for that calculation.
You are using ImageDataGenerator to generate synthetic variations of the data, greatly increasing the size of the training set that has to be loaded into memory.
Here a couple of options to run PCA:

Reduce the number of variations in ImageDataGenerator
Use approximate PCA
Get a larger machine or distribute across a cluster


Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error We want to use cosine similarity with hierarchical clustering and we have cosine similarities already calculated.
In the sklearn.cluster.AgglomerativeClustering documentation it says:

  A distance matrix (instead of a similarity matrix) is needed as input
  for the fit method.

So, we converted cosine similarities to distances as

distance = 1 - similarity

Our python code produces error at the fit() method at the end. (I am not writing the real value of X in the code, since it is very big.) X is just a cosine similarity matrix with values converted to distance as written above. Notice the diagonal, it is all 0.) Here is the code:

import pandas as pd
import numpy as np
from sklearn.cluster import AgglomerativeClustering

X = np.array([[0,0.3,0.4],[0.3,0,0.7],[0.4,0.7,0]])

cluster = AgglomerativeClustering(affinity='precomputed')
cluster.fit(X)

The error is:

runfile('/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py', wdir='/Users/stackoverflowuser/Desktop/4.2/Pr')
Traceback (most recent call last):

  File "<ipython-input-1-b8b98765b168>", line 1, in <module>
    runfile('/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py', wdir='/Users/stackoverflowuser/Desktop/4.2/Pr')

  File "/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 704, in runfile
    execfile(filename, namespace)

  File "/anaconda2/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 100, in execfile
    builtins.execfile(filename, *where)

  File "/Users/stackoverflowuser/Desktop/4.2/Pr/untitled0.py", line 84, in <module>
    cluster.fit(X)

  File "/anaconda2/lib/python2.7/site-packages/sklearn/cluster/hierarchical.py", line 795, in fit
    (self.affinity, ))

ValueError: precomputed was provided as affinity. Ward can only work with euclidean distances.

Is there anything that I can provide? Thanks already.

According to sklearn's documentation:

  If linkage is “ward”, only “euclidean” is accepted. If
  “precomputed”, a distance matrix (instead of a similarity matrix) is
  needed as input for the fit method.

So you need to change the linkage to one of complete, average or single. If you try this it works:

import numpy as np
from sklearn.cluster import AgglomerativeClustering

X = np.array([[0,0.3,0.4],[0.3,0,0.7],[0.4,0.7,0]])

#cluster = AgglomerativeClustering(affinity='precomputed', linkage='complete')
#cluster = AgglomerativeClustering(affinity='precomputed', linkage='average')
cluster = AgglomerativeClustering(affinity='precomputed', linkage='single')
cluster.fit(X)


Selecting the right algorithm for match probability prediction Looking for assistance kick-starting a new machine learning scenario. In this case I need to pair one entity (ex. person) with a group of entities (ex. other people) given a history of matching patterns over time.

For example: Assume we have people who like other people with a combination of traits such as height, weight, education level, etc... This notation of 'like' is bi-directional when calculating probabilities. When a new person arrives I'd like to isolate a group of people (ranked by probability) to recommend them too.

In this case the input would be a single person with all their traits and the output would be a list of people with ranked probabilities of matching. There seems to be two main sets of data

Known people
Traits known people have liked or not with the traits of people they have liked or not

I am relatively new so not sure which types of training model to use. For example, is this something that can be accomplished most reasonable with a linear regression model, binary classification, boost-tree's, matchbox, etc, etc, etc..? Possibly I need a combination.

I am using the Azure Machine Learning studio which has lots of options but I am just not sure where to start in this scenario. Any suggestions or pointers would be appreciated in helping

It seems a variant of Movie Recommendation problem. Take Netflix for example, they have  user ratings for different type of movies; in your case users are telling what attributes they like. When a new user joins and rates the things he/she likes, you will find the people with similar interests, again with Netflix analogy it means recommending movies based on people with same tastes as the new user.

Read on Collaborative filtering, which is used for building recommendation systems. The reason I drew the Netflix/Movie Recommendation analogy is because you can find a lot of literature on it.


differentiating vector with different operation on each elements I have some idea how backpropagation would work for loss function like:
loss=summation(predicted-true)^2
Where predicted and true are vector of same length and same operation throughout the elements.
Now in object localization problem my neural network's output vector's 0th element would denote probability of particular object being in image and rest 4 would tell about bounding box. Now my loss function would be roughly something like this:
loss=CategoricalCrossentropy(pred[0],true[0])+MSE(pred[1:4]-true[1:4])
This loss function worked fine in tensorflow and my NN localized the object as I expected.
My problem is that I am not able to understand  how mathematically differentiation would work when different operations are applied on different elements.

Different operations on different elements don't prevent differentiation in any way.
Lets, say we call your above Loss function:
$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,
where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that mathematically, I don't care which output you used to find $L_1$ or $L_2$, just the fact they both somehow depend on the weights of the model.
Thus, if our aim is to differentiate, we simply get (assuming everything in our model is differentiable):
$$\frac{\partial\mathcal L}{\partial\mathbf w} = \frac{\partial L_1}{\partial \mathbf w}+\frac{\partial L_2}{\partial \mathbf w}$$


Scaling of ordinal data before both hierarchical and KMeans clustering I am new to data analytics. As part of my assignment I have to perform both hierarchical and Kmeans clustering on a data set wherein all applicable variables are ordinal (1-5 rating scale). Do I need to scale the data before performing clustering?

If all applicable variables are ordinal and they're all in the same scale ie: 1-5, scaling would be totally unnecessary.

The whole point of scaling is to bring attributes with values in differing magnitudes to the same scale. Bottomline, here you don't need to scale the data.

Hope this helps!


Using scikit Learn - Neural network to produce ROC Curves I want to verify that the logic of the way I am producing ROC curves is correct. (irrelevant of the technical understanding of the actual code). I have a data set which I want to classify. I am using a neural network specifically MLPClassifier function form python's scikit Learn module.

I am passing a training data set to the fit function and then using the predict function with the testing data set. I am then outputting a confusion matrix with a false positive value and a true positive value.

what I would like to do is calculate a ROC curve where I need a set of true positive and false positive values. would it make sense to run the neural network (MLPClassifier) multiple times with different targets each time and record the different true positive and false positive values?

It seems simple enough for me. Is it the right way to produce a ROC curve with a neural network?

You could use sklearn.metrics.roc_curve.

Besides, Here is an example of what you want to do.

from sklearn.metrics import roc_curve, auc
fpr2, tpr2, threshold = roc_curve(y_test, clf.predict_proba(X_test)[:,1])


How high rmse value can be? While solving the questions for machine learning, I got two values for R square from 2 different regressors, i.e, 0.9999 and 0.9769. So, which should go for as both could lead to overfitting?
Thanks in advance

Are you asking about rmse or r-squared?

The size of rmse depends on the general range of your response feature(predicting target). The lower the better.

As to r-squared, both 0.9999 and 0.9769 are very likely to indicate overfitting. If you have to choose between these two models, my suggestion is to evaluate them using generalization error. You can also do cross-validation if no out-of-sample data are available.