Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent? I want to reduce the dimensionality of a dataset using a stacked Autoencoder. The size of the dataset and the computing power at my disposal make it very difficult to train the Network using simple, full-batch Gradient Descent.

Does it make sense to use Mini-Batch Gradient Descent for this kind fo task? I guess that would generate a noisy reduction of dimensionality?

In general, using an autoencoder for dimensionality reduction using mini-batch gradient descent is perfectly fine.

You should pay attention to the neural architecture you use in order to have the appropriate inductive bias, e.g. image autoencoders tend to use convolutional layers to profit from pattern locality.

You should also assess how well the input data is reconstructed, to ensure that the information is properly retained in the compressed representation.

However, as you did not specify the final destination of the data (i.e. what the reduced-dimensionality data is going to be used for), we can't tell whether the dimensionality reduction would be appropriate in for your specific purposes.


Is my iPython Installation Valid? I just want to know if my ipython is properly installed. I have both Python and Anaconda installed on my Windows. I entered the "pip3 install ipython" command on the Command Prompt and it downloaded and installed the ipython package. However, whenever I try to access it, I see this:

Notice the icon on the top left hand corner:

And whenever I execute a code and will like to continue working in that file, it will take forever for iPython to take me to the next line of code (or in some cases, may not even do so). For example, in the case below, I have not been able to enter any code after the "In [18]: plt.show()" since "In [19]:" would not just come up. Please see this:

I believe this is because the package is not properly installed.

I will be grateful if you guys can just show me what I can do about this?
Thanks.

It looks fine to me :)  the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you have to find it. Once you close that window, your iPython prompt woill return to In [19]. You could alternatively press Ctrl-C in the iPython session, but this will end the session.

If the problem persists (and you cannot even find/close the plot from plt.show()) - maybe have a look at this post, which discusses the various backends used for matplotlib.

I assume you do this to begin with:

import matplotlib.pyplot as plt

plt is a module (as the error message says), so you cannot use it directly like that, but rather the functions that are contained in that module, like scatter() (which worked for you) and then others like plot(), hist() and so on.

Have a look at the relevant documentation with lots of exmaples.

Here is a full example:

import matplotlib.pyplot as plt
import numpy as np

# evenly sampled time at 200ms intervals
t = np.arange(0., 5., 0.2)

# red dashes, blue squares and green triangles
plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^')
plt.show()


Classification followed by regression to handle response variable that is usually zero I have a data set consisting of a bunch of predictors (mostly unbounded or positive real numbers) and a single response variable that I wish to predict. The response is typically exactly zero -- around 90% of the time. I have tried modelling this using standard Gaussian process methods as well as random forests. However, in both cases (although moreso when using random forests) the model seems to handle the data poorly, usually predicting a non-zero response. Now, if the predicted responses were in fact very close to zero I could just set a cut-off below which the values would be rounded to zero, but they are significantly non-zero in many cases.

My idea for a solution is to train two models: a classification model trained on the entire training-set that predicts whether a variable is zero or non-zero, and a regression model trained only on the rows in the training set with a non-zero response. I would then first use the classification model to predict which observations have a response that is exactly zero, and subsequently use the regression model to predict the value of the non-zero responses.

Is this a sound way to solve the described problem? Does this sort of model have a name? Are there better ways to do this?

This sounds entirely reasonable, and the usual name for this structure I have heard for this is just "pipeline" which also applies to other system-feeds-next-system structures - it might also be "machine learning pipeline" or "data processing pipeline".

There are ways to assess performance of a ML pipeline:

You can of course compare the final accuracy or loss value, with the simpler model. Has turning the model into a more complex multi-stage one actually improved things? Sadly nothing is guaranteed, although I would be hopeful in your case initially - in part because you could apply adjustments available to classifier models used to deal with class imbalance issues.
You can decide which part of the pipeline will gain you the most benefit by switching between pipeline-so-far input to each unit and perfect input from the training data. Then you can see how much incremental difference is possible by perfecting that unit in the pipeline.

In your case you have a two stage pipeline, so you can check whether it is worth focusing more effort on the classifier or regression parts by comparing the incremental improvements between:

The unadjusted output of the whole pipeline run end-to-end.
The output of the regression (or zero) assuming that the classifier was perfect.
A perfect score.

Whichever of the two differences gives you the largest difference (2) - (1), or (3) - (2) points at work being most rewarded for working on the classifier or regression stage respectively.

You can see a worked example of this per-stage analysis in Advice for Applying Machine Learning (slides 21, 22), amongst other places.


Machine learning algorithm which gives multiple outputs from single input I need some help, i am working on a problem where i have the OCR of an image of an invoice and i want to extract certain data from it like invoice number, amount, date etc which is all present within the OCR. I tried with the classification model where i was individually passing each sentence from the OCR to the model and to predict it the invoice number or date or anything else, but this approach takes a lot of time and i don't think this is the right approach.

So, i was thinking whether there is an algorithm where i can have an input string and have outputs mapped from that string like, invoice number, date and amount are present within the string.

E.g:
Inp string: the invoice #1234 is due on 12 oct 2018 with amount of 287
Output: Invoice Number: #1234, Date: 12 oct 2018, Amount 287

So, my question is, is there an algorithm which i can train on several invoices and then make predictions?

Keras functional API's are a way your can solve you problem. Using keras functional API, we can build models that resembles more like graphs such as this:

In order to build a model like this, you can use keras as follows:

from keras.models import Model
from keras import layers
from keras import Input

input_layer = Input(shape=(100,), dtype='float32', name="Input")
split_layer = layers.Dense(32, activation='relu', name='split_layer')(input_layer)
first_layer = layers.Dense(32, activation='relu', name='first_layer')(split_layer)
second_layer = layers.Dense(32, activation='relu', name='second_layer')(split_layer)
model = Model(input_layer,[first_layer, second_layer])
model.summary()

In order to compile this model, we can define different loss functions for different layers

model.comile(optimizer=optimizer,
            loss={'first_layer':'mse', 'second_layer':'binary_crossentropy'},
            metrics=['accuracy'])

Once you are done with building the network, you could simply fit you data as follows:

model.fit(X,
        {'first_layer': first_layer_Y,
         'second_layer': second_layer_targets},
          epochs=10
        )


In reference to Gamma Gamma spend model, is this in any way related to Pareto/NBD, BG/NBD or BG/BB model? The reason I ask this question is as follows:

I am currently in a remote intern position.
I've been asked to calculate the Customer lifetime value (past + future) for each customer based on monthly cohorts.
I can choose from P/NBD, BG/NBD or BG/BB model whatever is appropriate.

I came across the Gamma-Gamma spend model. Is this in any way associated with the above models? How?

According to this (https://segment.com/docs/guides/warehouses/how-do-i-forecast-ltv-with-sql-and-excel-for-e-commerce/), they just use average transaction value for each customer and multiply by expected conditional transactions. Is this a sufficient approach by changing individual averages to monthly cohort averages?

The Gamma-Gamma spend model assumes that the expenditure of a client is independent of the transaction behavior.

Therefore, we can use this in conjunction with any CLV model. Models like Pareto/NBD and BG/NBD only predict the transaction behavior and client death (stops being a customer) independent of the spend value.


in which case can I say that the data are bad and I ll achieve nothing using machine learning on it general Infos about my dataset: I have 40k data points and 5 features.
I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS is making an error of 10 meters and you want to correct it. So I brought another super GPS which is very accurate and measured 40k data while driving so in my dataset I have some vehicle informations which are speed, Acceleration, yaw rate, timestamp and wheel angle and I have position Informations which are the ground truth longitudes latitudes and the false longitudes and latitudes from my normal GPS. I'm transofrming those latitudes and longitudes to an x and y just to know how much should I shift my false longitudes and latitudes so that my position can be more accurate and similar to the Ground truth values. Can my data be bad in this case? I'm trying to predict the error in longs and lats that the GPS make so that I can later correct it so it's a regression problem and I'm using those features above to do that which I think they are informative since speed, acceleration, yaw rate and wheel angle are related somehow to a position(am I wrong?)

I'm asking this generally, I read some articles in the internet, that say that data is sometimes bad or the quality of the data is bad but I don't know what the mysterious sentence really mean.

I also had the problem when training neural networks that my loss start to decrease for the first 10-20 epochs and then it stuck on some high value and the network stops learning like if it were struggling to go out of that loss value but it can't. I tried to use only 100 data points instead of all the 40k and I noticed that it worked good, the NN achieved to fit those but as I increase the number of data points the performance become worse(do you have any ideas on this?)

some people suggest that I don't have many data and many features and in this case it would be better to use some machine learning approach since it outperforms NNs in case of small datasets or if I have few features like in my case so I also tried using random forest and I noticed that it gives better results than neural networks but it also doesn't generalize well, even if it gave me good results on train and validation sets, when I try it on test data(data that the random forest have never seen), it perform really bad.

so I was reading in the internet what can cause those problems and I noticed that I sometimes saw people or articles that claim that maybe the quality of the data is bad! but what does this really mean? I thought neural networks can map any kind of data, if I have one feature and one target then neural networks can map those two together, at least it can overfit the data right?

so can someone please tell me what is bad data or better how do I know if my data are bad? well if there is a way to know that then I would probably save time and not start working on a project that will take me a month to complete and then figure out my data is bad. Also can you tell me whether my case make sense ? I mean I find it weird that NNs gives very bad performance way worse than random forest. at least my NN should overfit the data or am I wrong?

NN is not a magic bullet

Not every model works well on every dataset, is the ethos.

Even tough NN is very powefull for certain tasks you could find tasks where linear regression would be a better choice.

Quality of data could mean 2 things:

a) data is unstructured and not processed. Think in terms of some super ugly text logs

b) data is uninformative. There are no patterns that can be extracted no matter what you try.

To conclude, experiment with different approaches and always have a holdout dataset to do sanity checks


How to start learning Python for Machine Learning without spending money? I want to start learning Python for Data Science without spending money.
Where to start learning?
Which resources such as books and websites do you recommend?

Thank in advance for answer!

There is plenty of material on the internet in the form of blog posts. This, together with the philosophy of learning by doing, leads me to the following advice: google "python XXX tutorial" where XXX stands for a basic machine learning algorithm and, for a few of the first results, use Google Colab to mimic what the tutorial explains.

Whatever you don't understand, you look up in google (e.g. what is Google Colab, loops in python, python named arguments, etc).

The main python libraries used in ML probably are scikit-learn and pandas. The online documentation of these and any other library used in the tutorials will probably be very useful to understand what each piece of code does.

Some ML algorithms you may want to explore are:

Linear regression
Logistic regression
k-nearest neighbors
k-means

When going through the tutorials, you will probably find many important transversal concepts, e.g. difference between classification, regression and clustering, data split in train/validation/test sets, performance measures like accuracy, the ROC curve, the area under it; look up in google any term you don't understand, the first results or their wikipedia pages will probably give you a basic understanding of them.


Reverse of Yeo Johnson variable transformation done by preprocess function in caret package to get original Target and predicted values I have done a Yeo Johnson transformation by using preprocess from caret package. I have predicted the target variable using linear regression. Now, I would like to reverse the transformation for both the target and the predicted values.
I know this question has been answered before, but I still could not solve my problem. This question was answered here and here.
I could not modify it for Yeo Johnson.

I am not sure there is an elegant caret solution, but VGAM offers what you are looking for, an inverse of the yeo-jhonson transformation : https://www.rdocumentation.org/packages/VGAM/versions/1.0-3/topics/yeo.johnson.


Aggregating multiple encoded categorical values I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables.

I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,000. One-hot encoding does not make sense has it would increase the feature space by too much.

Each observation in my dataset can take multiple values for the CATEGORY feature, for instance, row 1 could have the value a but row 2 could have the values a, b, c, d

I have managed to encode each individual value in the feature but am unsure how to aggregate these values for each row.

How should these encoded values be combined?

If individual categories are important in your analysis, you could split the category column into multiple columns based on the amount of different category values then pivot your data set to have multiple row entries per category.

Visually

recordid | category | value
-------- | -------- | -----
   1     |   a      |  5
   2     |   a,b,c  |  10

would become

recordid | category |  value
-------- | -------- | ------
   1     |   a      |   5
   2     |   a      |   10
   2     |   b      |   10
   2     |   c      |   10

You could then make further aggregations or transformations as necessary.

Otherwise you could consider "a" and "abc" and "abcd" a new category, i.e. an item with categories "abc" is different from an item with just the category "a"


How to model the probability of detecting an image, given it is seen multiple times Are there any existing methods/models describing the probability of an object being detected by a computer vision algorithm given it is seen $n$ times at similar angles and orientations? I know that an autonomous car may, for example, have trouble recognizing a stop sign and as a result the image recognition system being used may have a bounded box around the stop sign continuously appearing and disappearing signaling that the object recognition algorithm is only detecting the stop sign a certain amount of the time it sees it. I would like to understand this phenomenon in a more general sense.
More precisely put, I would like to model that: given an object is detected with probability $p$ if it is seen 'once' (AKA during a moment in time), then what is the probability of it being detected if it is seen for a second time, a third time, etc... Each 'moment in time' may be characterized by a single video frame/image or some other unit of time. Each time the image is seen it may be in a similar but not identical orientation.

An approach that is not specific to the image domain is to use a probabilistic data structure like a Count-Min Sketch. A Count-Min Sketch data structure can accumulate information to estimate the observed frequency of an input value based on the past set of input values by using multiple hashing functions over the input.


Kernel trick explanation In support vector machines, I understand it would be computationally prohibitive to calculate a basis function at every point in the data set. However, it is possible to find this optimal solution due to the so-called kernel trick.

Other answers to this question use advanced math and statistics jargon to answer the question  (I assume) properly, causing it to be inaccessible to general a data science audience.  Could someone post a "big-picture" description  (i.e., not necessarily thorough or technically complete) illustrating what the kernel trick is and how it works?

The kernel trick is based on some concepts: you have a dataset, e.g. two classes of 2D data, represented on a cartesian plane. It is not linearly separable, so for example a SVM could not find a line that separates the two classes. Now, what you can do it project this data into an higher dimension space, for example 3D, where it could be divided linearly by a plane.

Now, a basic concept in ML is the dot product. You often do dot products of the features of a data sample with some weights w, the parameters of your model. Instead of doing explicitly this projection of the data in 3D and then evaluating the dot product, you can find a kernel function that simplifies this job by simply doing the dot product in the projected space for you, without the need to actually compute projections and then the dot product.
This allows you to find a complex non linear boundary that is able to separate the classes in the dataset. This is a very intuitive explaination.


Why vSGD-fd optimization algorithm isn't popular? As the paper by Schaul & LeCun states

The findings are clear: in contrast to the other algorithms tested,
vSGD-fd does not require any hyper-parameter tuning to give reliably
good performance on the broad range of tests: the learning rates adapt
automatically to different curvatures and noise levels.

Auto-adapting learning rate sounds like a huge deal,
but why is everyone seem to be using Adam and other optimizers?

This is my answer at Cross Validated for the same question asked by the same person:

It's already 2019, still nobody answered this question. I don't understand why vSGD isn't popular either. But I do have some reasons from my own:

not real auto-adapting. Like v-SGD, ADAM is also a so-called auto-adapting algorithm, but that is not true. The learning rate, and window size in v-SGD, the \beta terms in ADAM all need tuning. The new-variants like AMSGrad and NosAdam seem to be more robust though.
too "complex". v-SGD uses a "bprop" term to estimate the Hessian diagonal,
and later there is also a finite-difference version. These are somehow "complex" methods to use in computer science and engineering. The numerical instability and  the inherent inaccuracy can cause a lot of trouble. That may be the reason why Tensorflow and Pytorch developers didn't include v-SGD in their package. And that also caused very few comparisons with v-SGD in the subsequent optimization papers.
speed. If the estimated Hessian is a good approximate, then it surely speeds up. However, first, the local landscape may not be the "noisy quadratic loss" as stated in the paper; second, the estimate is also very rough. Actually, there can be no exact estimate of Hessian diagonal. These factors all make the speed of v-SGD algorithm not so competitive.

That is only some understanding with not a lot of practice. Hope someone could point out probably more accurate points.


What are the activation functions in Convolutional Layers for? I read a lot about CNNs but I didn't quite understand some things:

What are the activation function in CLayers for? If I understood it right, the only weights in these layers are the ones in Filters, and for the activation function a weighted sum is needed?

The computational effort should increase, shouldn't it? When there are many Filters many Feature Maps (Map with the dot-products) are produced. All of them are given to the next Layer, so if they are as big as the input Image and there are 10 Filters in the first CLayer the second CLayer would have to apply 10x the computational effort(per Filter in the second Layer) that the first layer had or, since they all take all the last layers outputs (the feature maps).

If there are more Layers than one, how does the backpropagation know what they have to change on each one of them, especially since the backpropagation occurs after all layers are applied.

(CLayer = Convolutional Layer)

1 - Activation functions are non-linear functions. These are added in between layers which are simply Linear transformations.
Example without activation function:
ConvLayer1(Input) -> ConvMaps1
ConvLayer2(ConvMaps2) -> ConvMaps2

Mathematically, this would be $I_{nput} \circledast K_{ernel_1} \circledast K_{ernel_2} $,
which is equivalent to $I_{nput} \circledast K_{ernel_3} $,
where $K_{ernel_3} = K_{ernel_1} \circledast K_{ernel_2} $
That means that creating two layers would simply waste computational effort and give no gain whatsoever.
2 - Yes, if you keep the same image Width and Heigh and just stack more channels, there convolution maps will grow out of control, this is why you compensate this with Pooling Operations and use of large strides. Also, increasing the number of parameters without proper care and architecture design usually leads to overfitting.
3 - Backpropagation is simply a computationally efficient and convenient way of expressing the Partial Derivatives of the function and acts exactly as it would if you just took the derivative of the whole function at once. There is plenty of content online on it, you probably didn't understand the process of weight updating.


Trouble performing feature selection using boruta and support vector regression I was trying to select the most important features of a data set using Boruta in python. I have split the data into training and test set. Then I used SVM regressor to fit the data. Then I used Boruta to measure feature importance.The code is as follows:
from sklearn.svm import SVR

svclassifier = SVR(kernel='rbf',C=1e4, gamma=0.1)
svm_model= svclassifier.fit(x_train, y_train)

from boruta import BorutaPy

feat_selector = BorutaPy(svclassifier, n_estimators='auto', verbose=2, random_state=1)
feat_selector.fit(x_train, y_train)
feat_selector.support_
feat_selector.ranking_
X_filtered = feat_selector.transform(x_train)

But I get this error KeyError: 'max_depth'.
What might be causing this error?
Does Boruta work with any kind of models? i.e linear models, tree-based models, neural nets, etc.?

Without the full code, there are few things to check but I would try as follows:
Note that you are passing svm_model to BorutaPy and according to your code you should pass the fitted object that is svm_model
from sklearn.svm import SVR

svclassifier = SVR(kernel='rbf',C=1e4, gamma=0.1)
svm_model= svclassifier.fit(x_train, y_train)

from boruta import BorutaPy

feat_selector = BorutaPy(svm_model, n_estimators='auto', verbose=2, random_state=1)
feat_selector.fit(x_train, y_train)
feat_selector.support_
feat_selector.ranking_
X_filtered = feat_selector.transform(x_train)

EDIT:
After reading the implementation of Boruta and since this is based on tree models.
 Parameters
    ----------
    estimator : object
        A supervised learning estimator, with a 'fit' method that returns the
        feature_importances_ attribute. Important features must correspond to
        high absolute values in the feature_importances_.

SVM does not have the attribute feature_importances_, so Boruta can only receive tree models like DecisionTrees, RandomForest, XGB, etc.
If you want to use SVM anyway I would recommend to change the feature selection algorithm to PermutationImportance, which is quite similar way of computing importance base on random repeated permutation, but in this case you will have to provide a metric to measure the decrease on performance when a feature is shuffled.


How to interpret training and testing accuracy which are almost the same? Note - I have read this post but still don't understand
I have a Naive Bayes classifier, when I input my training data to test the accuracy, I get 63.05%. When I input my test data, the accuracy is 65.00%.
Why are the training and test accuracy almost identical? For information, my data is split in 70/30. Does this mean that there is no overfitting?

Why are the training and test accuracy almost identical?

Nearly identical performance on the training set and test set is a good outcome, it means the model is doing what it's supposed to do. To give an intuitive comparison:

The performance on the training set is equivalent to how well a student can redo the exercises which have been solved by the teacher during class. The student might just have memorized the answers by heart, so it's not a proof that they understand.
The performance on the test is equivalent to how well the student can solve some similar exercises that they haven't seen before in a test. This is a much better indication that the student truly understands the topic.

Does this mean that there is no overfitting?

Yes, it proves that there's no overfitting. To keep with my comparison, overfitting is equivalent to memorizing the answers.
However there can be other problems which bias the result:

The performance on the test set is 2 points higher than the performance on the test set. This probably means that the test is very small, because if it was a large enough sample the performance wouldn't be higher. If the test set is too small, the performance is less reliable (any statistics obtained on a small sample is less reliable).
Accuracy can be a misleading evaluation measure. It only counts the proportion of correct predictions, so if a large proportion of instances belong to the same class then the classifier can just predict any instance as this class and obtain high accuracy. For example here if the majority class is around 63-65%, then it's possible that the classifier didn't learn anything at all. Looking at precision/recall/F1-score gives a more accurate picture of what happens.

[edit] Important note: as Nikos explained in a comment below, my answer assumes that you have a proper test set, i.e. that the train and test sets are sufficiently distinct from each other (otherwise there could be data leakage and the test set performance would be meaningless).


How to properly rotate image and labels for semantic segmentation data augmentation in Tensorflow? What's a proper procedure for doing the image and label rotation for semantic segmentation in dataset augmentation using Tensorflow?

Images

I have seen the function tf.contib.image.rotate(), but this function fills empty space with zeros (from docs):

  Empty space due to the rotation will be filled with zeros.

I would like to fill that empty space with a different value (maybe some constant, like the dataset mean pixel). How this can be done in Tensorflow (I know that there are options in Keras Image Preprocessing, but I need TF)?

Labels

Also, what about the labels? If I just use the same function (tf.contrib.image.rotate()), it will fill the empty space with zeros suggesting that pixel in those places belong to class with id 0 (since I have class labeled with 0). The one solution could be to put ignore label on those pixels (e.g. 255), but, again, the current function doesn't support default fill value ...

I have found the following solution.

Labels

Here is a snippet filling an empty space in label with ignore label.

IGNORE_LABEL = 255
label = tf.subtract(label, IGNORE_LABEL)
label = tf.contib.image.rotate(label, angle)
label = tf.add(label, IGNORE_LABEL)

This way, by subtracting IGNORE_LABEL, you shift the interval to [-255, 0]. When label (segmentation labels) is rotated, empty spaces are filled with zeros. By adding back IGNORE_LABEL, you shift interval back to [0, 255] meaning that empty space is set to IGNORE_LABEL.

Images

Similar trick can be applied to image rotation to fill empty space with preferred value.


Word vectors to Sentence Vectors How can I use the vectors of words in a sentence to get the vector of that sentence . I have used strategies like - Averaging the individual word vectors or a tf-idf weighted combination of the words . While these hacks work , there are obvious problems with these . Wanted to know what would be some other way to do this

In this paper a state of the art method (unsupervised smoothed inverse frequency) is described, you can find an implementation of this method here.


Data cleaning and data transformation before EDA? What is the most common order of data cleaning, data transformation and exploratory data analysis?

For me it seems most logical to do data cleaning, then EDA and finally data transformation (encoding of categorical variables, and feature scaling).

Doing data transformation before the EDA, seems to make the EDA not that useful, as you cant ex. check for stuff like:

  Passengers in the age interval 0-18 has a higher chance of survival

(if feature scaling has been applied to age feature).

But then again, doing data transformation after the EDA, also miss out on chance of encoding categorical variables and thereby visualize correlations of those with the target variable.

What is the order of the mentioned processes? And is there even an order?

Although not very helpful, the answer is probably "it depends".

I like to do data cleaning and some EDA together since EDA can highlight appropriate treatments to clean the data - e.g. influencing how to handle missing values.

I think data transformation should be done just prior to modelling; whether or not you need to do any transformation at all depends on the techniques you plan to use.


diseases risk prediction using social media comments I am trying to build one risk calculator using diabetes patients comments on social media platform.

I have list of comments given by diabetes patients for their experience on diabetes.

When any new user come on my platform, I collect what symptoms he is facing.

I match user symtoms with comment data symtoms and try to calculate the risk score.

approach 1.

(Numbers of comments in database having user entered symptoms) / (Total comments entered by diabetes patients)

approach 2.

(Numbers of comments in database having diabetes as a symptom and also user entered symptoms) / (Total number of comments in database having diabetes as a symptom)

Is there any better approach to calculate the risk score using user entered symtoms and diabetes patient comment data?

You should also have comments/symptoms of a reference group of non-diabetes patients to be able to make a distinction yes/no diabetes.

Do you have users of which you know they don't have diabetes?
If not, "Is often thirsty?" will always map to diabetes, because you have no other data.

Non-realistic example:

Suppose 99% of diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.
Then suppose Suppose 99% of non-diabetes patients eat 10 candy bars a day, and 1% does not eat candy bars.
There is a strong correlation between eating candy bars and having diabetes.
Your data of only diabetes patients shows that both people who eat 10 candy bars and people who eat no candy bars develop diabetes.
Data that includes non-diabetes patients will reveal that people that eat 0 candy bars are much more likely to not have diabetes.

Or you should have comments and diabetes type data to be able to make a distinction diabetes type 1 or type 2.
"How often do you use medication?" is a strong indication between the two, but you'd need the data.

Furthermore, expect a risk score with at least weighted numbers that can be either positive or negative, for weak or strong (inverse) correlation between an answer and indication for diabetes.

There are a lot more advanced techniques (e.g. with neural networks) that will have better results, including finding correlations between answers.

If you are doing anything more serious than 'Go see the doctor' on >50% of positive indicators, you should really do more research on data science models, but also on how to formulate the questions etc.


What are some methodologies for performing feature selection for simple feed-forward neural networks? In multiple linear regression there is an F-test which can be used to evaluate whether or not a covariate has a meaningful impact on a model. This is typically done through either a forward selection or backwards selection algorithm. Does such a meaningful process exist for neural networks as well? The only reason I ask is because working with neural nets is an inherently stochastic process, so I do not know how I should try and get accurate bounds for the F-statistic.

I usually try to do the following process, I don't know whether it has name or not:

Try to find features which an expert can say what the label is without hesitating.
Plot correlation matrix for your sample to investigate whether each feature has correlation with others or not. I do this process whenever the input features are so much. Then try to reduce the correlated features or try to apply PCA. Consider the point that PCA alone does not care about the labels, so it may find principal components that your data would be so difficult to be separated but not necessarily.


Auto-Encoder customized layer training My question is related with model-weights optimization during back propagation.
In this image I'm trying to represent an auto-encoder having 7 layers where 4th one is center layer.

If my understanding is correct, during back propagation outcome of jth+1 layer, will help in optimizing the weights of jth layer. If it is correct, I want to restrict the weights training of Encoder layer and want to perform my customized training.

Customized training should be like this:

Till layer 4 normal back propagation.
Layer 3 weights will get trained(optimized) using the output of 4th layer as mentioned in PINK arrow.
Layer 2 weights will get trained with the outcome of layer 5 as mentioned in YELLOW color arrow.
Layer 1 weights will get trained with the outcome of layer 6 as mentioned in BLUE color.

I want to know how I can  implement such customized trainable architecture?I'm familiar with Tensorflow, but I'm not able to figure out any solution and kind of stuck.

NOTE: Bear with my drawing.

You have to create multiple training sessions, in which you selectively freeze the layers you don't want to train. Each Sequential() model can be seen as a list of layer objects. Each of these layers have the argument trainable = True/False. You Freeze the layers you don't want to be trained classically, and proceed to

Then, using Keras Models, you connect layers that are not adjacent as you depicted in the picture. Once all layers have been trained, you transfer the weights into a news Sequential() shell to compose a final model.

It's going to be time consuming, but it's doable.


error when submitting machine learning project I am trying to make a submission of a machine learning classification problem. I have a test dataset where to try my model. To submit I have to build a csv file. The prblem is that when I go building this csv file by doing:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')
sub.head()

I get the following error message:

AttributeError                            Traceback (most recent call last)
<ipython-input-37-f223adaf5068> in <module>
  1 sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')
----> 2 sub.head()

AttributeError: 'NoneType' object has no attribute 'head'

Can somebody please help me? Thank's in advance.

The return type of pandas.DataFrame.to_csv is None.

So your first line assigns sub = None:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')

This is probably what you wanted:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class})
sub.to_csv('sub.csv')
sub.head()


CNN for unsupervised anomaly detection I'm wondering if the following strategy has been already used and could work

Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training data.
Is there any way to use it to detect image of horse as anomaly?

For example, with a ruled based system we could says

if P(cat) and P(dog) ~0.5 then it's an anomaly

another way could be to take feature vector at last fully connected layer and compare vector very different from other could be considerated as anomaly

Do you have any related paper? is it a totaly dumb idea?

I'm not really sure if if P(cat) and P(dog) ~0.5 then it's an anomaly would be sound.

But what you are referring to is not a strange idea: A CNN forms a conceptual description of inputs based on patterns in previously seen data. New data that is less like the train data doens't compress and reconstruct well. The approach usually used for this is called autoencoding.

Roughly: If you would build an autoencoder by reconstructing images of cats and dogs, the in and out images would eventually (after enough data) approach each other (the distance you'd define would decrease). A new image from another domain would reconstruct less well (The distance between reconstruction and input would be larger than average).


Is there a known convolutional net architecture to calculate object masks for images? I would like to train a convnet to do the following:

Input is a set of single channel (from black to tones of grey to white) pictures with a given object, let's say cars.
Target is, for every picture in the set, the same picture, however pixels are either black or white. The pixels corresponding to the car object are in white (i.e. intensity 255) and the pixels corresponding to the background are black (i.e. intensity 0).

After training I would like to feed the net with pictures of cars and I would like the prediction - the ideal prediction at least - to be a picture with pixels either black or white, where white corresponds to the object and black to the background.

I assume that the input layer is a 2D convolutional layer and the output layer is also a 2D convolutional layer, each one with as many neurons as pixels in the pictures.

Can anyone please explain what kind of network architecture would accomplish just that?

It could be either the architecture (in theory) or implemented in code.

I expect to tweak it, but it would be nice not to start from scratch.

I'm surprised nobody mention fully convolutional neural networks (FCNs) for semantic segmentation.

They are inspired by the original AlexNet style convnets that ended with one or two densely connected layers and softmax classifier. But FCNs dispense with the dense layers and stay fully convolutional all the way to the end.

Here's the basic principle. Take AlexNet, or VGG or something like that. But instead of using the parameters in the classifier to compute a scalar for each category, use them to compute a whole array (i.e. image) using a  1x1xNUM_CATEGORIES convolution. The output will be NUM_CATEGORIES feature maps, each representing a coarse-grained "heat map" for that category. A map of dogness, a map of catness. It can be sharpened by including information from earlier layers with "skip connections".

EDIT: Just one further bit of good news: the authors of that paper provide implementations of their nets in Caffe's Model Zoo. Tweak away!


Is this a case of overfitting? I am training a CNN-LSTM-FC network with around 1 Million parameters network for spatial time series prediction but the validation loss does not even come close to the training loss after 1-2 epochs. Is this a clear case of overfitting? Any inputs would be appreciated!
I am using keras with tensorflow backend.

I would rather say, that in your training, underfitting occurs, because the train values are much better, than the test values. This can occur, when your memory cells cannot store enough information, so maybe try to increase the units or nodes of your LSTM, so the loss can converge better.


Once a Feature is one-Hot encoded, it is as code as an "N" new feature.
You may discard those which has low Feature-importance


how to modify sparse survey dataset with empty data points? I am working on a data set where the categorical variables have lots of empty spaces (not "NA" but ""). For example, one variable has 14587 empty spaces out of 14644 observations. There are many such variables where most of the observations are empty.In fact it is a survey dataset where the participant just chose to ignore a particular question.

I have never handled similar dataset. I am looking for advise as to how best to handle such datasets before any modeling is done. Deleting the rows or the variables with lots of empty spaces doesn't seem feasible.

Thanks a lot.

I would consider approaching this situation from the following two perspectives:

Missing data analysis. Despite formally the values in question are empty and not NA, I think that effectively incomplete data can (and should) be considered as missing. If that is the case, you need to automatically recode those values and then apply standard missing data handling approaches, such as multiple imputation. If you use R, you can use packages Amelia (if the data is multivariate normal), mice (supports non-normal data) or some others. For a nice overview of approaches, methods and software for multiple imputation of data with missing values, see the 2007 excellent article by Nicholas Horton and Ken Kleinman "Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models".
Sparse data analysis, such as sparse regression. I'm not too sure how well this approach would work for variables with high levels of sparsity, but you can find a lot of corresponding information in my relevant answer.


How to implement gradient descent for a tanh() activation function for a single layer perceptron? I am required to implement a simple perceptron based neural network for an image classification task, with a binary output and a single layer, however I am having difficulties. I have a few problems:

I am required to use a tanh() axtivation function, which has the range [-1, 1], however, my training labels are 1 and 0. Should I scale the activation function, or simply change 0 labels to -1?
So the gradient descent rule dictates that I shift the weights according to the rule:

$$
\Delta \omega = -\eta \frac{\delta E}{\delta \omega}
$$

I am using the mean square error for my error:

$$
E = (output - label)^2
$$

Considering my output is $o = tanh(\omega .x)$, x is my input vector and $y_i$ is the corresponding label here.
$$
\frac{\delta E}{\delta \omega} = \frac{\delta (y_i - tanh(wx))^2}{\delta \omega} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) \frac{\delta wx}{\delta w} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) x \\= -2(y_i - o)(1 - o^2)x
$$

I implemented this is python, the dot product of the input vector with the weights turns out to be too large, which makes $tanh(x)=1$ and $1-o^2 = 0$, so I can't learn. How can I circumvent this problem?

Thanks for the replies.

The implementation:

def perc_nnet(X, y, iter = 10000, eta=0.0001):
    a, b, c = X.shape
    W_aug = np.random.normal(0, 0.01, a*b+1)
    errors = []

    for i in range(iter):
        selector = rd.randint(0,c-1)
        x_n = X[:,:,selector].ravel() #.append(1) #has the bias as well
        x_n = np.append(x_n, 1)
        v = x_n.dot(W_aug)
        o = np.tanh(v)
        y_i = y[:,selector] if y[:,selector]==1 else -1
        MSE = 0.5*(o - y_i)**2
        errors.append(MSE)
        delta = - eta * (o - y_i) * (1 - o**2) * x_n
        W_aug = W_aug + delta

    return W_aug, errors

How can I circumvent this problem?

TLDR : Normalize your input data.

Why?

Notice how tanh actually works on the input data:

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]))
array([-1. , -1., -1. , -0.76159416,  0.,0.76159416,  1. ,1.   , 1])

If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.

This would explain

  the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.

Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000)
array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001,  0.00999967,  0.09966799,  0.76159416])

Now, tanh treats all of them differently.

Note
You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax.

As to why should be a whole different answer.
Here is a link for the same.


Is it possible to use an array of graph coordinates as an input variable? Say I have 1000 graphs that shows sales every year for the last 10 years for 1000 different companies. And say each of those graphs belong to either domestic countries or foreign countries.

Is it possible I could input the different graphs into a classifier? That is, could a model predict based on the graphs whether or not the country was domestic or foreign? If so, how would you do that in python or r?

Yes, the graph coordinates can be used as features for training your algorithm( whichever classification algorithm you choose). Features will be X and Y and output will be labelencoded type of country.


KL divergence loss goes to zero while training VAE I'm trying to train a variational autoencoder to perform unsupervised classification of astronomical images (they are of size 63x63 pixels). I'm using an encoder with 2 convolutional layers and a dense layer, and a similar structure for the decoder. I'm performing Xavier initialization of gradients. I'm using the Adam optimizer with a learning rate of 1e-4.
I observe that the KL divergence starts at very small values (roughly of the order of 1e-4) and suddenly vanishes after a few epochs while training, while my reconstruction loss reduces normally (I use MSE as the reconstruction loss). What could be the reason? Should I perform scaling of the losses separately?
This is my model.
initializer = glorot_normal()

def sampling(inputs):
  z_mean, z_log_var = inputs
  epsilon = k.random_normal(shape=(k.shape(z_mean)[0], 2), mean=0., stddev=0.1)
  return z_mean + k.exp(z_log_var) * epsilon

input_images = Input(shape = (63,63,1))
conv1 = Conv2D(16, (3,3), activation = 'relu')(input_images)
conv2 = Conv2D(8, (3,3), activation = 'relu')(conv1)
flattened = Flatten()(conv2)
x = Dense(4, activation = 'relu')(flattened)

z_mean = Dense(2, name = "z_mean")(x)
z_log_var = Dense(2, name = "z_log_var")(x)
z = Lambda(sampling, output_shape = (2,))([z_mean, z_log_var])

encoder = Model(input_images, [z_mean, z_log_var, z], name = "encoder")

latent_inputs = Input(shape = (2,))
x = Dense(59*59*8, activation = 'relu')(latent_inputs)
x = Reshape((59,59,8))(x)
conv4 = Conv2DTranspose(8, (3,3), activation = 'relu')(x)
decoded = Conv2DTranspose(1, (3,3), activation = 'softmax')(conv4)

decoder = Model(latent_inputs, decoded, name = "decoder")

z_mean, z_log_var, z = encoder(input_images)
vae_decoder_output = decoder(z)
vae = Model(input_images, vae_decoder_output, name = "VAE")
vae.summary()

This is the loss function that I'm trying to implement.
recon = MSE(input_images, vae_decoder_output)
recon = k.mean(recon)
kl_loss = 1 + z_log_var - k.square(z_mean) - k.exp(z_log_var)
kl_loss = k.sum(kl_loss, axis = -1)
kl_loss *= -0.5
vae_loss = k.mean(kl_loss*10**3+recon)
vae.add_loss(vae_loss)

take a look at the paper "Generating Sentences from a Continuous Space" by Bowman. In Section 3.1 it is explained why LSTM_VAE tend to this behaviour:
"This problematic tendency in learning is compounded by the lstm decoder’s sensitivity to subtle variation in the hidden states, such as that introduced by the posterior sampling process. This
causes the model to initially learn to ignore ~z and
go after low hanging fruit, explaining the data with
the more easily optimized decoder. Once this has
happened, the decoder ignores the encoder and little to no gradient signal passes between the two,
yielding an undesirable stable equilibrium with the
kl cost term at zero. We propose two techniques
to mitigate this issue."
....


Training my neural network to overfit my training dataset I want to train my neural net to overfit the training data. Should I just keep fitting my model with the same training data by using K-fold validation and setting epochs to infinity? Then, after it reaches nearly 100 percent prediction, I can just stop the training. Is it the right way to do it?

If you want to overfit, then yes you just need to keep fitting the training data through your network until you reach as close to zero training loss as possible (note that zero loss is stronger than 100% prediction, and will result in a greater amount of overfitting). There is no need to use cross validation. If your network is sufficiently complex, then it won't take very long to reach very high accuracy on the training data.


How do I Combine two CNN models (h5 format)? I have two CNN models, both of them are trained on the same dataset.

How do I combine/ensemble both to make predictions on test data?

# Load Keras Models

model1 = tf.keras.models.load_model('/kaggle/input/models/model1.h5')
model2 = tf.keras.models.load_model('/kaggle/input/models/basic_cnn.h5')

How to combine these 2 models to make model3 ?

I have to use predict_generator function to predict from the ensembled model3.

You can put a dense layer combining both outputs.

model1 = tf.keras.models.load_model('/kaggle/input/models/model1.h5')
model2 = tf.keras.models.load_model('/kaggle/input/models/basic_cnn.h5')

inputs = tf.keras.layers.Input(shape=input_shape)
combined = tf.keras.layers.Concatenate()([model1(inputs), model2(inputs)])
outputs = tf.keras.layers.Dense(n_outputs)(combined)

model3 = tf.keras.models.Model(inputs, outputs)

Set input_shape and n_output accordingly to your data and targets.

You should then freeze your pre-trained weights and train the final dense layer to correctly choose which weight to assign to outputs of your models.

model1.trainable = False
model2.trainable = False

model3.fit(your_data)


Given you have images stretched out as columns in a table with ~48,500 rows, I am assuming you have the raw images that are 220x220 in dimension.

You can use a function available via OpenCV called inpaint, which will restore missing pixel values (for example black pixels of degraded photos).

Here is an image example. Top-left shows the image with missing values (in black). Top-right shows just the missing values (the mask). Bottom-left and bottom-right are the final output, comparing two different algorithms for filling the images.

I would suggest trying both methods on your images to see what looks best.

Have a look at the Documentation for more details on the algorithms themselves. Here is the documentation of the actual function.

As for code, it will look something like this:

import opencv as cv    # you will need to install OpenCV

dst = cv.inpaint(img, mask, 3, cv.INPAINT_TELEA)

the first argument is your image with missing values
the second is the mask, with locations of where missing pixels are, i.e. which pixels should be filled/interpolated.
third is the radius around missing pixels to fill
fourth is the flag for the algorithm to use (see link above for two alternatives)

For each image, you can generate the mask with something like this:

mask = image[image == np.nan]


Preprocessing text so that two words without a separating space (or hyphen separated) are detected Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the same thing? i.e. replace all such occurrences by "bigram". I should also mention that I have no prior knowledge of what exact bi-grams  are present in the corpus.

Two possible strategies:

Hard coded rules

Statistical learning

Hard coded rules means creating a mapping from original text to the normalized representations.  The text will have to be manually inspected to find commonly occurring patterns. Hard coded rules are often done with regular expressions.
A statistical learning approach looks for patterns in the data. Bigrams can be defined as the conditional probability of a token given the preceding token. When that probability is above a threshold the two tokens are merged.


How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn I'm trying to get mean test scores from scikit-learn's GridSearchCV with multiple scorers.

grid.cv_results_ displays lots of info. But
grid.cv_results_['mean_test_score'] keeps giving me an error.

I've checked the docs and similar questions with no luck. What am I doing wrong?

Code:

scorers = {
    'r2': 'r2',
    'nmsle': 'neg_mean_squared_log_error',
    'nmse': 'neg_mean_squared_error'
}

params = [
    {
        'regressor': [GradientBoostingRegressor()], 'preprocessing': [None],
    },
]

grid = GridSearchCV(pipe, params, cv=5, scoring=scorers, refit='nmse')
grid.fit(X_t, y_train)

grid.cv_results_['mean_test_score']

Error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-161-a021fe7b05bb> in <module>
     20 print(f'Val score set score rmse: {rm}')
     21
---> 22 grid.cv_results_['mean_test_score']
     23
     24 # for mean in means:

~/anaconda3/lib/python3.6/site-packages/sklearn/utils/deprecation.py in __getitem__(self, key)
    124             warn_args, warn_kwargs = self._deprecations[key]
    125             warnings.warn(*warn_args, **warn_kwargs)
--> 126         return super(DeprecationDict, self).__getitem__(key)
    127
    128     def get(self, key, default=None):

KeyError: 'mean_test_score'
```

For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_scorer_name').
so use

  grid.cv_results_['mean_test_(scorer_name)']

Ex: grid.cv_results_['mean_test_r2']


"Deep Noether's Theorem": Building in Symmetry Constraints If I have a learning problem that should have an inherent symmetry, is there a way to subject my learning problem to a symmetry constraint to enhance learning?

For example, if I am doing image recognition, I might want 2D rotational symmetry. Meaning that the rotated version of an image should get the same result as the original.

Or if I am learning to play tic-tac-toe, then rotating by 90deg should yield the same game play.

Has any research been done on this?

From Emre's comment above, Section 4.4 of Group theoretical methods in machine learning by Risi Kondor has detailed information and proofs about creating kernel methods that inherently have symmetries. I will summarize it in a hopefully intuitive way (I am a physicist not a mathematician!).

Most ML algorithms have a matrix multiplication like,
\begin{align}
s_i &= \sum_j W_{ij}~x_j \\
  &= \sum_j W_{ij}~(\vec{e}_j \cdot \vec{x})
\end{align}
with  $ \vec{x} $ being the input and $ W_{ij} $ being the weights we wish to train.

Kernel Method

Enter the realm of kernel methods and let the algorithm handle input via,
\begin{align}
s_i &= \sum_j W_{ij}~k(e_j,~x)
\end{align}
where now we generalize to $ x, e_j \in \mathcal{X} $.

Consider a group $ G $ that acts on $ \mathcal{X} $ via $ x \rightarrow T_g(x) $ for $ g \in G $. A simple way to make our algorithm invariant under this group is to make a kernel,
\begin{align}
k^G(x, y) &= \frac{1}{|G|} \sum_{g \in G} k(x, T_g(y))
\end{align}
with $ k(x, y) = k(T_g(x), T_g(y)) $.

So,
\begin{align}
k^G(x, T_h(y)) &= \frac{1}{|G|} \sum_{g \in G} k(x, T_{gh}(y)) \\
  &= \frac{1}{|G|} \sum_{g \in G} k(x, T_{g}(y)) \\
  &= \frac{1}{|G|} \sum_{g \in G} k(T_{g}(x), y)
\end{align}

For $ k(x, y) = x \cdot y $ which works for all unitary representations,

\begin{align}
k^G(x, T_h(y)) &= \left[ \frac{1}{|G|} \sum_{g \in G} T_{g}(x) \right] \cdot y
\end{align}

Which offers a transformation matrix that can symmeterize the input into the algorithm.

SO(2) Example

Actually just the group that maps to $ \frac{\pi}{2} $ rotations for simplicity.

Let us run linear regression on data $ (\vec{x}_i, y_i) \in \mathbb{R}^2 \times \mathbb{R} $ where we expect a rotational symmetry.

Our optimization problem becomes,
\begin{align}
\min_{W_{j}} &\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\
\tilde{y}_i &= \sum_j W_{j} k_G(e_j, x_i) + b_i
\end{align}

The kernel $ k(x, y) = \| x - y \|^2 $ satisfies $ k(x, y) = k(T_g(x), T_g(y)) $. You could also use $ k(x, y) = x \cdot y $ and a variety of kernels.

Thus,
\begin{align}
k_G(e_j, x_i) &= \frac{1}{4} \sum_{n=1}^4 \| R(n\pi/2)~\vec{e}_j - \vec{x}_i \|^2 \\
  &= \frac{1}{4} \sum_{n=1}^4 ( \cos(n\pi/2) - \vec{x}_{i1} )^2 + ( \sin(n\pi/2) - \vec{x}_{i2} )^2 \\
  &= \frac{1}{4} \left[ 2 \vec{x}_{i1}^2 + 2 \vec{x}_{i2}^2 + (1 - \vec{x}_{i1} )^2 + (1 - \vec{x}_{i2} )^2 + (1 + \vec{x}_{i1} )^2 + (1 + \vec{x}_{i2} )^2 \right] \\
  &= \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1
\end{align}

Note that we needn't sum over $ j $ because it is the same for both. So our problem becomes,
\begin{align}
\min_{W} &\sum_i \frac{1}{2} (y_i - \tilde{y}_i)^2 \\
\tilde{y}_i &= W \left[ \vec{x}_{i1}^2 + \vec{x}_{i2}^2 + 1 \right] + b_i
\end{align}

Which yields the expected spherical symmetry!

Tic-Tac-Toe

Example code can be seen here. It shows how we can create a matrix that encodes the symmetry and use it. Note that this is really bad when I actually run it! Working with other kernels at the moment.


Convert List to DataFrame Dictonary:

{(8758148.0, 'CI Alpine Growth Equity Fund'): D    0.000016
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Growth Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Small Companies Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI American Value Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Equity Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Investment Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Small Cap Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Canadian Small/Mid Cap Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Global Bond RSP Fund'): D    0.0
 Name: 8758148.0, dtype: float64,
 (8758148.0, 'CI Harbour Fund'): D    0.0
 Name: 8758148.0, dtype: float64 }

I need the dataframe to be:

id          Name                                Value
8758148.0   CI Alpine Growth Equity Fund        0.000016

I am stuck in issue where I need to convert list into such a data frame with certain name of the columns

Let's see. I modified the string a bit, so that it can be saves as a dictionary.

g = {(8758148.0, 'CI Alpine Growth Equity Fund'): 'D    0.000016 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Growth Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Small Companies Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI American Value Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Equity Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Investment Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Small Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Canadian Small/Mid Cap Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Global Bond RSP Fund'): 'D    0.0 Name: 8758148.0, dtype: float64',
 (8758148.0, 'CI Harbour Fund'): 'D    0.0 Name: 8758148.0, dtype: float64' }

Now here is the code:

df = pd.DataFrame([i[0] for i in g.keys()], columns=['id'])
df['Name'] = [i[1] for i in g.keys()]
df['Value'] = [i.split()[1] for i in g.values()]
df

    id          Name                            Value
0   8758148.0   CI Alpine Growth Equity Fund    0.000016
1   8758148.0   CI American Growth Fund 0.0
2   8758148.0   CI American Small Companies Fund    0.0
3   8758148.0   CI American Value Fund  0.0
4   8758148.0   CI Canadian Equity Fund 0.0
5   8758148.0   CI Canadian Investment Fund 0.0
6   8758148.0   CI Canadian Small Cap Fund  0.0
7   8758148.0   CI Canadian Small/Mid Cap Fund  0.0
8   8758148.0   CI Global Bond RSP Fund 0.0
9   8758148.0   CI Harbour Fund 0.0


Instead of binarising discrete variables for NN algorithms, could I attempt to assign them a value by splitting into buckets and sorting? Creating a boolean for each value of each discrete variable explodes the number of features. I have an idea for solving it but I wasn't able to find anything on this on the internet, so perhaps someone could either point me to a resource or let me know whether they think this is possible. And if not, maybe point to another solution (all I could find was binarisation).

So if we have training vectors $X_i$ and similarly labels $y_i$, I propose we partition the training set by some variable $j$, so that for each $X_i$ for which $j_i$ equals some value $q$, $X_i$ will be in $P(q)$.

Then for each $q$ we calculate the mean $y_i$ for $P(q)$ and sort those averages, mapping each $q$ onto its index in the sorted list. This value is used as the feature for $j$. I suspect that this ought to be better than just giving each value a random index, as any remaining non-linearity will be hopefully fixed by the hidden layers. But it seems that these non-linear patterns will be easier to pick up if the function is somewhat quasi-monotonous.

I think this could be improved, potentially instead of just finding the mean, another measure could work better and also for multiple discrete features, I am not sure if this process should be repeated for each variable separately or through a multi-dimensional partitioning.

The main question is of course, would this process be helpful at all, and if not, is there anything similar that could solve this problem?

Just taking the mean seems better, that way the distances between the groups is better represented than just indexing them by order. With the target you should be a bit careful not to include the current row for the mean, since for the test set and the actual predictions you cannot use that target either (for obvious reasons). To get even more information about the category you try to get rid of is to also take the variance or standard deviation of the target within this group and take the mean and variance of important numerical features for these categories. If you have multiple categories you can also take the expected value for features and targets conditioned on multiple categories at once. This is a nice compromise between keeping enough information on high cardinality while keeping the dimensionality low.

Here's an example where the c is the categorical variable, n are numeric and y is your target. We want to get rid of c (because c could have 1000s of values).

xc_1 | xn_1 | xn_2 | y
A    | 2    | 4    | 1
A    | 1    | 2    | 1
A    | 4    | 3    | 0
B    | 3    | 5    | 0
B    | 5    | 4    | 1
B    | 6    | 6    | 0

To keep things a little more simple we will disregard the values of the current row, which is better anyway. For the first row we will look at all other A rows, take the averages (and maybe other statistics) of the numeric features and of y, we will do the same for B, then we get:

xc_1 | xn_1 | xn_2 | xnc_1 | xnc_2 | yc_1 | y
A    | 2    | 4    | 2.5   | 2.5   | 0.5  | 1
A    | 1    | 2    | 3     | 3.5   | 0.5  | 1
A    | 4    | 3    | 1.5   | 3     | 1    | 0
B    | 3    | 5    | 5.5   | 5     | 0.5  | 0
B    | 5    | 4    | 4.5   | 5.5   | 0    | 1
B    | 6    | 6    | 4     | 4.5   | 0.5  | 0

We have captured a lot of information about category 1, which we can now discard.


Whether to replace NaN values in result column I have a training dataset where we have to predict "Result" based on features "A", "B", "C" and "D" using machine learning. For a few rows, the "Result" is empty (7/19612).
While for the other features I have filled the NaN values with their mean, I don't understand whether or not to do so with the result column. Will it better to drop the rows entirely instead?

There is no correct way as dealing with nan values it depends on the dataset.But i would suggest rather than doing single imputation you could do multiple imputation using IterativeImputer in sklearn.
As very less target variables are missing it won't make a significant impact unless the problem statement is sensitive to finding outliers like fraud detection.
Refer Iterative Imputation


Neural network with multiple layer: learning function Here is my code to implement the learning of my neural network using the backpropagation learning. The algorithms is stable but I don't learn correctly the output. Do you see anything wrong in my learning process?

//### Parameter ###
#define Nb_entry 2
#define coeff_app 0.01
#define par_momentum 0.8
#define par_nb_test 0.2
#define para_tolerence 0.05
#define para_stop_learning 1000

Here are my learning functions

void fonction_neuron(double * input, Neuron * neuron_info)
{
    int i=0; // loop variable
    double net=0;

    // Computation of the net value type: net = w[0]*bias + sum(w[i]*imp[i])
    net=(*neuron_info).weight[0]; //bias = 1
    for(i=1;i<(*neuron_info).nb_input;i++) net+=(*neuron_info).weight[i]*input[i];

    (*neuron_info).output=net;
    //print_stat_neuron(neuron_info);
}

void fonction_network(Neuron N_network[][10], int nb_layer, int *nb_neuron_per_layer,double *input)
{
    int i,j; //loop variable
    double previous_layer_output[10]={0};

    //Propagation of the signal into the neural network
    for(i=0;i<nb_layer;i++)
    {
        if(i!=0) for(j=0;j<nb_neuron_per_layer[i-1];j++)
        {
            previous_layer_output[j]=N_network[i-1][j].output;//save previous layer output
            //printf("previous_layer_output[%d]=%f\n",j,previous_layer_output[j]);
        }

        for(j=0;j<nb_neuron_per_layer[i];j++)
        {
            //printf(" i=%d j=%d \n", i,j);
            N_network[i][j].old_output=N_network[i][j].output; //save previous value
            if(i==0)fonction_neuron(input, &(N_network[i][j])); //first layer
            else fonction_neuron(previous_layer_output, &(N_network[i][j])); //other layer using the previous layer output
        }
    }
}

Here are some questions to help with the implementation of the algorithm:

1- You did the sum, but where/what is the activation function (sigmoid, hyperbolic tangent, Gaussian, etc.)?

2- Is multilayer? How did you forward? Did you use the sum along with activation of each neuron past for the other layers?

3- Have you stored the derivatives of activations to the backpropagation?

4- Will you use the traditional backpropagation algorithm or genetic algorithm to adjust the network weights?

5- If you choose the sigmoid and backpropagation, here is an example algorithm (this algorithm the derivative of the sigmoid is already calculated $O_{j}$ $(1-O_{j})$.


Features standardization - Multilayer perceptron I have serious doubts concerning the features standardization done before the learning process of a multilayer perceptron.

I'm using python-3 and the scikit-learn package for the learning process and for the features normalization.

As suggested from the scikit-learn wiki (Tips on pratical use), I'm doing a features standardization with the Preprocessing module, which means that all my features will appear as standard normal specifications.

The problem is that in the showed example:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
# Don't cheat - fit only on training data
scaler.fit(X_train)
X_train = scaler.transform(X_train)
# apply same transformation to test data
X_test = scaler.transform(X_test)

The standardization is done first on the training set, and then, after that, it's applied on the test set.

Which means, that the distributions parameters (mean, st.deviation) will be estimated only from the training set.

Now, because I need to save my multiple datasets in a database, on which I will perform the hold-out sampling for the training-test folds, I have done a preprocessing on all my instances at once, because I don't want to do that at run time execution.

It's clear that in this way the distributions parameters will change and the results will be different.

Now, Someone knows the differences between the two approaches? Are there issues doing the preprocessing at once on all the dataset?

Generally, preprocessing parameters are fit only on the training subset, because otherwise you could overfit your data and overestimate quality of your model on the test subset.

With feature standardization, however, overfitting is not so dangerous, so I assume you can preprocess all your dataset at once safely.

The best practice, however, is to do all the processing in one pipeline and apply cross-validation to the whole pipeline:

from sklearn.pipeline import make_pipeline
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score, GridSearchCV
scaler = StandardScaler()
model = MLPClassifier()
pipeline = make_pipeline(scaler, model)
scores = cross_val_score(pipeline, X, y)

If you use scikit-learn cross-validation, incorporating pipelines would be simple.


Why is this not ordinary convolution? I am currently studying this paper (page 53), in which the suggest convolution to be done in a special manner.

This is the formula:

\begin{equation} \tag{1}\label{1}
q_{j,m} = \sigma \left(\sum_i \sum_{n=1}^{F} o_{i,n+m-1} \cdot  w_{i,j,n} + w_{0,j} \right)
\end{equation}

Here is their explanation:

As shown in Fig. 4.2, all input feature maps (assume I in total), $O_i (i = 1, · · · , I)$ are mapped into a number of feature maps (assume $J$ in total), $Q_j (j = 1, · · · , J)$ in the convolution layers based on a number of local filters ($I × J$ in total), $w_{ij}$ $(i = 1, · · · , I; j = 1, · · · , J)$. The mapping can be represented as the well-known convolution operation in signal processing.
Assuming input feature maps are all one dimensional, each unit of one feature map in the convolution layer can be computed as equation $\eqref{1}$ (equation above).

where $o_{i,m}$ is the $m$-th unit of the $i$-th input feature map $O_i$, $q_{j,m}$ is the $m$-
th unit of the $j$-th feature map $Q_j$ of the convolution layer, $w_{i,j,n}$ is the $n$th element of the weight vector, $w_{i,j}$, connecting the $i$th feature map of the input to the $j$th feature map of the convolution layer, and $F$ is called the filter size which is the number of input bands that each unit of the convolution layer receives.

So far so good:

What i basically understood from this is what I've tried to illustrate in this image.

It seem to me what they are doing is actually processing all data points up to F, and across all feature maps.  Basically moving in both x-y direction, and compute on point from that.

Isn't that basically 2d- convolution on a 2d image of size $(I x F)$ with a filter equal to the image size?.
The weight doesn't seem to differ at all have any importance here..?

It's analogous to an extremely specific convolution step on a 2D image. That is, it's analogous to an $N \times I$ image with one feature map (e.g. a black & white image, if we're talking about the input), and you choose to use $J$ filters of size $F \times I$ which spans the entire width of the image and only strides along the length to create $J$ feature maps of size $(N - F + 1) \times 1.$ The restrictions here are that:

-The "image" (or current layer you're working on) necessarily only has one feature map

-The filters span the entire width of the image, so they don't stride along that direction, and the resulting feature maps have a width of $1.$

You could then re-interpret the $(N-F+1) \times 1$ resulting feature maps as a single $(N-F+1) \times J$ image, which then creates the exact same restrictions for the next convolutional layer (or more restrictions if you're doing pooling).

So yes, the analogy is there, but it's only analogous to a very restricted class of convolution on 2D image arrays, which I don't think is very useful. The kind of application it would be used for is black and white images for which you don't care about translation invariance along one of the dimensions.

The reason CNNs on images are much more flexible than this analogy allows for is because the input objects (and each respective hidden layer) of an image CNN is a 3D array, not a 2D array as it is in this case. The third dimension is the feature maps (the RGB values for the input image, for example). Then you could allow the filter to be of any size in both dimensions, and stride along both directions. I would prefer to keep the text interpretation when it comes to 2D inputs of a CNN.


Finding parameters of image filter using classified pairs I want to solve the problem of finding a parameter vector for an image filter (let us assume we know nothing about how the filter works, but we can feed it an input image and a set of parameters to produce an output image).

Thus, having a set $\{{I_k, J_k:=F_{\alpha}(I_k)}\}_{k\in\overline{1,N}}$ of $I_k$ images together with their filtered counterparts, $J_k$, what solutions would you recommend for finding $\alpha^\ast$ such that given $I^\ast$ the result $F_{\alpha^\ast}(I^\ast)$ is in the same "style" as the one of the $N$ training correspondence pairs.

I suppose one option  is to use a convnet to transform $I_k$ into a feature vector, $v_k$, and then concatenate $\alpha_k$ to obtain $u_k =(v_k,\alpha_k)$. Once this is done, use a regression method to estimate the $\alpha^\ast$ part of $u^\ast$.

I would like to find an alternative solution to what seems like a candidate for the style transfer approach (e.g. https://arxiv.org/pdf/1703.07511.pdf). That approach seems to solve the problem differently, and I envision situations where I need to simply use a filter rather than let a network "guess the style of that filter".

Additional details and possible assumptions

Given the invoked no free lunch prospects, let us assume, for a paeticular problem from this class, that $F$ is a non-linear kernel-based filter that maps $I$ to $J$ as a result of an iterative and convergent process. More specifically, let $F$ be a mean shift filter with the $\alpha=(\rho, \sigma_s,\sigma_r)$ using a concatenated Gaussian kernel and a Parzen window of size $\rho$. Intuitively, I would be tempted to guess that this filter is not smooth w.r.t. $\alpha$, but a formal investigation is required (I suspect it is not smooth given that infinitesimal changes in the size of the window could shift the output towards another mode, indicating a step function behaviour).

In general, it is correct to assume that $\alpha \in \mathbb{R}^d$, with $d \ll N$.

Given the goal of finding $\alpha$ when both the filter action is known (either via numerical computation in general, or, in closed-form if the filter is a gaussian blur, for example), we can be confident that the $N$ input samples have non-constant $\alpha_k$ vector values to start with.

But for sake of generalizability, it would be more elegant to pursue a solution that does not need to know how the filter operates without actually applying it to an input. The first approach suggested in the comments and based on convnets seems to fit this scenario and the optimization problem is taking into account the filter error. However, it would be interesting to hear more opinions, perhaps involving shallow approaches, even at the expense of designing the solution to address the concrete mean shift filter example from above.

Your parameter $\alpha$ has fairly low dimension.  Therefore, I recommend that you apply optimization methods directly to try to find the best $\alpha$ (without trying to use convolutional neural networks and regression for this purpose).

Define a distance measure on images, $\|I-J\|$, to represent how dissimilar images $I,J$ are.  You might use the squared $L_2$ norm for this, for instance.

Now, the loss for a particular parameter choice $\alpha$ is

$$L(\alpha) = \sum_{k=1}^N \|F_\alpha(I_k)-J_k\|.$$

We can now formulate your problem as follows: given a training set of images $(I_k,J_k)$, find the parameter $\alpha$ that minimizes the loss $L(\alpha)$.

A reasonable approach is to use some optimization procedure to solve this problem.  You might use stochastic gradient descent, for instance.  Because there might be multiple local minima, I would suggest that you start multiple instances of gradient descent from different starting points: use a grid search over the starting point.  Since your $\alpha$ is only three-dimensional, it's not difficult to do a grid search over this three-dimensional space and then start gradient descent from each point in the grid.  Stochastic gradient descent will allow you to deal with fairly large values of $N$.

This does require you to be able to compute gradients for $L(\alpha)$.  Depending on the filter $\alpha$, it might be possible to symbolically calculate the gradients (perhaps with the help of a framework for this, such as Tensorflow); if that's too hard, you can use black-box methods to estimate the gradient by evaluating $L(\cdot)$ at multiple points.

If $L_2$ distance doesn't capture similarity in your domain, you could consider other distance measures as well.

I expect this is likely to be a more promising approach than what you sketched in the question, using convolutional networks and a regression model.  (For one thing, there's no reason to expect the mapping from "features of $I_k$" to "features of $J_k$" to be linear, so there's no reason to expect linear regression to be effective here.)


Is there any NLP library or package which can help in adding comma, punctuation, newlines appropriately to text? I have a movie transcript without commas, punctuation, or newlines. Is there any NLP technique that can help to implement this?

This can be solved with "text segmentation". NLP libraries have code for breaking given text into :

Sentences
Phrases
Words

With this, you can break text into sentences and insert . or ? for each sentence. Similarly, dependency tree will help with inserting some punctuation marks (not all).

Example (breaking text into sentences):

import spacy
nlp = spacy.load('en_core_web_sm')
text = "I was expecting a surplus of cute close-ups but Burton does surprisingly little to win us over He's never been big on treacle but a bit more warmth in this chilly movie which barely follows the outline of the 1941 original would have gone a long way"
text_sentences = nlp(text)
for sentence in text_sentences.sents:
    print(sentence.text)

Output is :

  I was expecting a surplus of cute close-ups but Burton does
  surprisingly little to win us over

and

  He's never been big on treacle but a bit more warmth in this chilly
  movie which barely follows the outline of the 1941 original would have
  gone a long way

More details : https://spacy.io/usage/linguistic-features


Clustering 2D curves I have a set of curves in 2D space each expressed as a set of (sampled) data points.
Each set has more or less the same number of items - eventually I guess I’ll use binning to make sure the number of points is the same (say 50) if that can help.
I would like to cluster the curves in N groups. Computing N should be part of the solution.
Possible translations on the first dimension are irrelevant.
I have a k-means implementation available.
I was thinking to transform the problem into a 100-dimensional space (50x2) where the samples of each curve become features.
Could this approach work?
Is there a better one, either using k-means or a different algorithm?

Yes, you could start with k-means on the 100-dim data. Make sure you have normalized your data to perhaps zero mean and unit variance before running k-means.
Use the elbow method (google it) to determine the optimal number of clusters for your data.
By the way you also want to check if the two dimensions are correlated in any way. If it is then you can use PCA to convert the 2 dimensions into a single dimension that explains most of the variance in your data and then cluster using 50-dim per observation.


Clustering with imbalanced data and groups I have a problem that is about identifying clusters of highly correlated items. I initially focused on building a model and features that put similar data items close to each other. The main challenge is that I have a case of imbalanced data, as follows:

Tens of Millions of items are random and not necessarily correlated.
Hundreds of clusters of items (composed of 10-1000s of elements) exist* or may emerge. *I do have partial ground truth for the existing ones.
Clusters are very different, in size and properties.

I'd like to return the identified clusters, and the elements within each cluster. F1 should be a good measure.
To move forward, I can think of threshold-based hierarchical clustering. Are there other techniques to consider?

As you have the partial ground-truth (assuming for ALL clusters) I would suggest following a creative idea derived from Region Growing in image segmentation.
As your clusters are imbalanced in number of the points thus density, they are probably captured by localy using DBSCAN. Run DBSCAN with different parameters and evaluate on capturing your ground-truth into right clusters. The partitioning which gives the best result on your ground-truth evaluation will be your final clustering.


Finding a Data pattern I am new to this data science field.

I have data of points in 3D space and each point "helps" a metric. I have the sets of points and corresponding metrics.

Data might look like:

[Pt_set1_1], [Pt_set1_2], [Pt_set1_3], ..., [Pt_set1_20], Metric1, Metric2, Metric3
[Pt_set2_1], [Pt_set2_2], [Pt_set2_3], ..., [Pt_set2_20], Metric1, Metric2, Metric3

I want to identify which points help which metrics. Is this problem solvable with the data science algorithms. Which algorithm would help me? Please let me know. Any help would be appreciated.

If I'm understanding you correctly, you have a bunch of sets of 20 3D points, and for each set of points you have three metrics associated with it.

Yes, it's definitely possible to learn which points are most predictive for each metric. I would recommend that you split your data such that the point sets are in one numpy array which looks like

x1_1, y1_1, z1_1, x1_2, y1_2, z1_2, ..., x1_20, y1_20, z1_20
x2_1, y2_1, z2_1, x2_2, y2_2, z2_2, ..., x2_20, y2_20, z2_20
...

and then create one 1D numpy array for each of your metrics which contain the value of each metric for every point set. Then, you can use something like a RandomForestRegressor to try to predict the metrics individually from the points. Once you have the random forest trained with the fit(X, y) method, you can extract the importances of each coordinate in the points by getting the feature_importances_. For example, something like this:

rf1 = RandomForestRegressor(n_estimators=1000)
rf1.fit(points, metric1)
print(rf1.feature_importances_)

rf2 = RandomForestRegressor(n_estimators=1000)
rf2.fit(points, metric2)
print(rf2.feature_importances_)

rf3 = RandomForestRegressor(n_estimators=1000)
rf3.fit(points, metric3)
print(rf3.feature_importances_)

Hope this points you in the right direction!


Dropout dividing by compensation term = overshoots the result? When applying dropout mask, why is it acceptable to divide the resulting state by the percentage of survived neurons?

I understand that it's to prevent signal from dying out. But I've done the test, and found that it disproportionally magnifies the resulting state.

Assume the original state is $(0.1, 0.1, 0.2, 5.0)$
and our mask is $(0, 0, 1, 1)$ (with 50% of neurons that survive).

So, the original length is $$
\begin{align*}\sqrt{(0.1\times 0.1 + 0.1\times 0.1  + 0.2\times 0.2 + 5\times 5)} &= \sqrt{25.06}
\approx 5.006.\end{align*}$$

As for the masked vector, its length is $$\sqrt{0.2\times 0.2 + 5\times 5} = \sqrt{25.04} \approx 5.004.$$

Applying the compensation gives $5.004 / 0.5 = 10.008.$

This seems incorrect: my compensation just blew up the state vector. Perhaps we should be compensating differently - more carefully? I think it would even get worse if we mask the individual weights (like DropConnect does).

In my actual test, the state-vector of $192$ elements has length of  $0.885$ and the masked vector, with compensation has length of $1.305$

Your example is cherry-picked: You mask out small numbers and keep a large one.
But dropout is applied randomly. Each of the following six masks, and of the corresponding values for the vector length, is equally likely to appear:
$$
\begin{align*}
&(1, 1, 0, 0): &\sqrt{0.1^2 + 0.1^2} &\approx 0.1414,\\
&(1, 0, 1, 0): &\sqrt{0.1^2+ 0.2^2} &\approx 0.2236,\\
&(1, 0, 0, 1): &\sqrt{0.1^2+ 5^2} &\approx 5.0010,\\
&(0, 1, 1, 0): &\sqrt{0.1^2 + 0.2^2} &\approx 0.2236,\\
&(0, 1, 0, 1): &\sqrt{0.1^2 + 5^2} &\approx 5.0010,\\
&(0, 0, 1, 1): &\sqrt{0.2^2 + 5^2} &\approx 5.0040.\\
\end{align*}
$$
The average vector length is
$$
\frac16 (0.1414+0.2236+5.0010+0.2236+5.0010+5.0040) = 2.5991,
$$
which is roughly half of the original vector length $5.006$. So it makes sense to divide it by the dropout rate of $50\%$.


Grouping similar data/images I'm using CNN, RNN and OpenCV to identify people and cars within images, once I identify several images I'm cropping them and dividing them in cars and people.

I would like to group all same-looking cars and all same-looking people. What would be the best approach to do so?

I'm recollecting these images from an IP camera, so I cannot base my code in colors, because the light is not the same when is day and night. Plus the cars aren't always looking at the same direction. I guess SVM would work nice, but I would need classify some data by hand first. I'm looking for something to make clusters of similar images without a supervising method.

Would something like k-means work?

Try t-SNE (http://lvdmaaten.github.io/tsne/), it can help you group pictures that have the same properties via dimensionality reduction. You can see an example here http://cs.stanford.edu/people/karpathy/cnnembed/ where t-sne is applied on various imagenet samples.

This will group images with similar content close to each other (dogs near dogs, black dogs near black dogs...). However, if your black dog is yellowish because of the lighting, i suspect it will not be close to the same black dog under normal lighting condition. For that, maybe you can try black and white pictures? or add a black & white layer on top of your RGB input.

I did not experiment it myself, but you may also want to have a look at DeepBit http://www.iis.sinica.edu.tw/~kevinlin311.tw/cvpr16-deepbit.pdf "an unsupervised deep learning approach to learn compact binary descriptor for efficient visual object matching".


Why is Regularization after PCA or Factor Analysis a bad idea? I have done Factor Analysis on my data and applied various machine learning models on it. I particularly find it giving high MSE value for Ridge and Lasso Regression compared to other models. I want to know the reason why this happens.

In principle, PCA is unsupervised and therefore label agnostic. That means the down projections forced into the PCs may as well not be related to what the model is trying to predict. That may be able to measure with the amount of variance your PCs are capturing.
In essence, PCA shall never be used as a means for regularisation but rather for dimensionality reduction. You could alternatively try a VIF approach, although not sure what your exact goal is.


Predicting change of shapes/coordinates I'm trying to find a way to predict/calculate how a shape (e.g. outline of a glacier) will change in the future—based on its history (previous shape) and additional factors (e.g. Δtemperature).

In my example: I have the shape/coordinates of a glacier and an average temperature at 1970, 1985, 2000, 2015. How can I give an estimate on how that shape will look like in 2030, based on the previous shapes and a predicted temperature?

The shapes would ideally come in a format similar to this:

[
  [
    [-113.74926783355818, 48.32440832757374],
    [-113.74926767330584, 48.32440813255405],
    [-113.74926748239692, 48.32440458296232],
    [-113.74926717549286, 48.32439887665663],
    [-113.74926686858888, 48.32439317035087],
    ...
  ],
  [
    [-113.75763099733634, 48.32877681033601],
    [-113.75763955455557, 48.32877660502744],
    [-113.75764546550784, 48.32878027200765],
    [-113.75765402272751, 48.32878006669797],
    [-113.75766003497893, 48.32878563609638],
    ...
  ]
]

But I imagine it could be necessary to convert this to coordinate offsets (Δx, Δy) similar to what sketch_rnn is doing.

Any pointers would help. Even that this is too ambitious. Thanks!

Disclaimer: I'm a beginner with this and hope the question isn't too naïve :)

Given that you only four data points (i.e., 970, 1985, 2000, 2015), there is no way to apply machine learning to make an accurate prediction.

If you get more data points, you can start with a sign prediction of surface area. You predict if the shape is increasing or decreasing in size.


Similarity matching between two distinct datasets (marketing case study) I am working for a company that sells different products to customers. My objective is to find customers that are likely to purchase product X based on the profiles of customers that already purchased product X.
My first idea was:

to collect relevant variables for customers that already purchased product X (dataset A)
perform a cluster analysis of this dataset to generate customer personas for dataset A
collect the same variables for customers that have not purchased product X (dataset B)
and finally measure the distance between customers in dataset B to the centroids of medoids of the generated clusters of dataset A

Unfortunately, this is less straightforward than I thought:

For one, I would have to cluster categorical and numerical data. Therefore, I would compute the gower-distance to get a dissimilarity matrix between data points of dataset A that I would then cluster by means of PAM (partitioning around medoids) clustering. I do not know how to apply data points of dataset B to infer a distance to the PAM medoids because those medoids relate to the dissimilarity matrix of dataset A rather than the actual data points.
Secondly, the generated clusters of dataset A are less descriptive than expected.

In conclusion, I would like to have a second opinion. Is the way I described really a good way to tackle the problem at hand? Or do you have other ideas?
Would be happy for your input - best wishes.

If I understand your question correctly, you have two groups of people: Group A, each of whom has purchased the product, say yogurt; and Group B, each of whom has not purchased yogurt. Your problem at hand is to find all people in Group B who will be likely to purchase your yogurt, if they have a similar profile as people in Group A.
This seems to be a very common problem in causal inference, where you need to match the treated people with the control group, but since one person could not be both treated and untreated, we need to find "similar" people on both sides such that they are comparable (in terms of their characteristics, or variables) so that we can make a causal inference from there.
Now, returning to your problem, I don't think it is necessary to do the clustering for matching. Instead, you could consider the "matching" approach commonly used for causal inference. Here is an r packages that comes to my mind: MatchIt.
In essence, what you need to do is to regard Group A as treatment and Group B as control. I believe they provide many different ways of matching algorithm and you can certainly play around to see which one works for you the best.


Errors generated in Rstudio I'am using Rstudio to create some codes. the cods works pretty well, without any error and gives the output desired. But when I turn on the PC after period and re-execute the same code, it generes errors especially in reading external files like

(Error in names(out) <- named :    'names' attribute [10] must be the
same length as the vector [3]).
Error during wrapup: 'names' attribute [10] must be the same length as
the vector [3]

(these 2 different errors are generated from executing the same part of code 2 times. the code want to tokenize a corpus of 10 files and save it in  tables of 3 columns)
I'am using the same dataset from the same location and I'am sure of calling the correct libraries
library("tm")

#seting the corpus
corp <- Corpus(DirSource(path),
               readerControl=list(reader=readPlain,
                                  language='en_CA',
                                  load=TRUE));

myTokenizer <- function(x, n, n_min) {
  library(tokenizers)
  tok <- unlist(tokenize_ngrams(x, n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3,
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
  corp<-tm_map( myTokenizer(x=corp, n=6, n_min=1) )
writecorpus(corp)

You need to convert the elements of corpus to character:

myTokenizer <- function(x, n, n_min) {
  tok <- unlist(tokenize_ngrams(as.character(x), n = n, n_min = n_min))
  M <- matrix(nrow=length(tok), ncol=3,
              dimnames=list(NULL, c( "gram" , "num.words", "words")))
  M[, 3] <- tok
  M[, 2] <- lengths(strsplit(M[, 3], "\\W+"))  # counts the words
  M[, 1] <- 1:length(tok)
  return(M)
}
res <- tm_map(corp, function (x) myTokenizer(x, n=3, n_min=1))

or apply your function directly without using Corpus:

res <- myTokenizer(x = list("A Sample Text ...", "Lorup ipsum x 12"), n=3, n_min=1))