How to update bias and bias's weight using backpropagation algorithm I'm writing my own training algorithm, but I don't know how to set the bias weight.

Have I to set bias in any layer?

Must the bias weight, be updated in every layer?

There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer.

There are usually two different approaches taken when implementing bias. You can do one or the other:

As a separate vector of bias weights for each layer, with different (slightly cut down) logic for calculating gradients.
As an additional column in the weights matrix, with a matching column of 1's added to input data (or previous layer outputs), so that the exact same code calculates bias weight gradients and updates as for connection weights.

In both cases, you only do backpropagation calculation from neuron activation deltas to the bias weight deltas, you don't need to calculate the "activation" delta for bias, because it is not something that can change, it is always 1.0. Also the bias does not contribute deltas back further to anything else.


Clustering mixed data types - numeric, categorical, arrays, and text I have a dataset with 4 types of data columns:

              numeric  categorical            tags                     text
id
1               51585           27  [A, B, C, ...]  "Some text bla bla bla"
2               53596           27  [B, D, E]               "Other text..."
3             1176345           27  [D, A, F, ...]                    "..."
4                 168           24             NaN                    "..."
5               88564           22             NaN                    "..."

numeric - continuous numeric values.
categorical - discrete categories, either numbers or strings (the type doesn't really matter because I can convert it to whatever works)
tags - array containing discrete values. Each row can have a different array length.
text - a string of text.

I am new to data science so maybe this is a "beginner" question.

How can I use all of these different data types in a clustering algorithm?

Here is what I learned so far:

K-means is good for numerical data. I successfully applied it to a subset of my data with only numerical columns. I also used some evaluation metrics (such as silhouette coefficient) to help me choose the number of clusters. So this works in principal but since it's not using most of my data the results are not good.
Then I read about clustering categorical data. I found the Gower Distance which is a distance between categorical data. So far I've used it with K-means (I passed the distance matrix generated by Gower into K-means). From here it should be easy to join the Gower distance matrix with the numeric columns from my original dataset and pass all of them to K-means.

I am aware there are other clustering algorithms besides K-means, and I plan to check some others as well. But before I do, I want to find some way to utilize all of my data in a single algorithm.

The tags and text columns stump me. I can't find a way to use them for clustering. I found some articles about clustering words from a text document - this is not what I want to do. I want to use a text column as one (or more) "feature" among others for clustering.
I am aware of the "bag of words" method for converting text into a vector of numbers. I can also easily imagine how to use this same method for converting the tags into a vector. However that seems like a bit of an overkill because it will increase the dimentionality of my data by a lot. Are there other ways to tackle this?

Bottom line - I am looking for a way to use all these data types together for clustering. I summarized what I know so far, but I am open to any solution, even if it's completely different from what I've listed above.

Thanks!

For the tags: Do you know how they are generated? How many unique tags do you have? If they are self-generated (ie lots of tags that can be subsets of other tags). You might need to do tag consolidation which will also help with dimensionality reduction of the word vectors. If you can provide a bit more information about what your data looks like and where it comes from, I can perhaps provide a more in depth answer.

For the text: You might want to try using word embeddings. You can use a pre-trained word2vec model.

I am not sure if it makes sense to use two different distance metrics. Your categorical data looks like its integers, is it ordinal, or are those indexes?


What are the audio features to best describe a music? I'm working on the content-based filtering part of a recommender system for an audio streaming project.

I firstly used the k-mean algorithm with music genres and one-hot encoding to  classify musics into different groups.
But, in order to get more precise results I want to change it and use audio features to feed the model instead.

So my questions are:
 - is my approach correct.
 - what are the most relevant audio features I can extract from an audio file.

Thx for your answers.

edit:
right now, I'm extracting those features:

music tempo
zero-crossing rate
duration
spectral centroids
spectral roll-off
MFCC
spectral bandwidth
spectral contrast

I want to know at which degree those audio features are relevant to 'describe' an audio extract.

Every cepstral coefficients can be considered as one of the best features for defining a musical piece.

Most famous being the Mel Scale, as I can see you are already extracting MFCCs, you are good to go. Although you should have mentioned, which MFCC are you extracting, from experience (a little bit) first 15 are usually the most useful cause they have a positive value. You can also work on more intrinsically robust GFCC.

They can then be used to obtain Spectrograms and so on.

I would suggest not using kNN, prefer Random Forrest, moreover audio signals require a lot of preprocessing. DCT and STFT are a must.


How to interpret agglomerative coefficient agnes() function? how to interpret agglomerative coefficient in agnes() function of cluster package?

Example:

res.agnes <- agnes(dataset,method = 'ward')
res.agnes$ac
0.934

Not sure if you are facing a concrete problem. Anyway, here it goes a definition of the agglomerative coefficient, from Finding Groups in Data: An Introduction to Cluster Analysis:

  Generally speaking, the AC describes the strength of the clustering
  structure that has been obtained by group average linkage. However,
  the AC tends to become larger when n increases, so it should not be
  used to compare data sets of very different sizes.

Also, if you are familiar with the silhouette,

  The AC can also be compared to the silhouette coefficient (SC)

First sight, the coefficient you get points to a pretty reasonable cluster structure in your data, since it is closed to 1: the coefficient takes values from 0 to 1, and it is actually the mean of the normalised lengths at which the clusters are formed. That is, the lengths you see when you look at your dendogram.

EDIT after comments: indeed it can be misleading that the text in the book I referenced ("Finding Groups in Data...") refers in my copy-paste to the group average linkage. However, the coefficient refers to the banner (which is like a kind of simpler or binary dendogram) and just validates the structure that the clustering produces, regardless of the method.

This is another reference to the agglomerative coefficient with no reference to the method used (although of course always in the context of hierarchical clustering).

EDIT 2 after comments: I explain below my understanding on how the coefficient is calculated, always using the same reference.

The formula for the coefficient is

$$AC = \frac{1}{n} \sum_{i} l(i)$$

i.e. the mean of the l(i), calculated as follows:

  For each object i, we look at the line containing its label and
  measure its length I(i) on the 0-1 scale above or below the banner.

Although I could not make the plot with 0-1 scale, it is just a normalization of the lengths.

The banner is a binary representation of the dendogram, where:

in the left side, when all objects are separated in the initial stage of the agglomerative algorithm, the banner is just blank
as soon as 2 objects get together in the same cluster, they appear in the banner
the l(i) are marked by the points in the x axis in which the objects get together, and are the lengths of the horizontal bars of the plots

Always clearer with an example, actually taken to a similar one in the same book.
Let's imagine 5 objects in a space of 4 dimensions; 4 of the objects very closed together, and one of them far away.

df <- data.frame("c1"=c(2, 2, 2, 2, 10), "c2"=c(2, 2, 2, 2, 10),
                 "c3"=c(2, 2, 2, 2, 10), "c4"=c(2, 2, 2, 2, 10))

Let's do the clustering,

library(cluster)
r <- agnes(df, method="ward")

The bannerplot actually looks a bit ugly:

but anyway, in there the 4 objects that are at the same position are joined at the beginning in a cluster, and the 5th is at distance 20. Personally, it is much easier to look at the dendogram:

however the information is pretty much the same.

Now the AC calculation. The lengths of the bars are 1 (normalized) for the first 4 objects, and 0 for the last one, thus the AC is 0.8:

> r$ac
[1] 0.8

In the dendogram, one should imagine the bars rotated 90 degrees, and they go from the root of the tree to the point in which the groups are formed.


Alternating Least Squares can anyone explain difference between Alternating Least Squares(ALS) and the recommendation systems?
it will be helpful if you give me an example.

Recommendation systems eg. Netflix movie recommendations, I assume its pretty straight forward to get. Recommendations systems can be in either of content based filtering, collaborative filtering and combination of these two.
Now, we can use matrix factorization for solve this learning problem (problem here is the recommending movies). cut short, it turns out to be an optimization problem and there are method available such as gradient decent (slow and costly in this case) and another is ALS algorithm.
So, ALS is an algorithm which you can use to solve the learning problem a.k.a build recommendation systems.
Please read here more-
http://stanford.edu/~rezab/classes/cme323/S15/notes/lec14.pdf


Idenitity between TD(0) algorithm and Policy Evaluation in Dynamic Programming when alpha is equal to 1 TD(0) algorithm is defined as the iterative update of the following:

$$ V(s) \leftarrow V(s) + \alpha({r + \gamma V(s')} - V(s) ) $$

Now, if we assume alpha to be equal to 1, we get the traditional Policy Evaluation formula in Dynamic programming. Is it correct?

No - Dynamic programming estimates the value of the next state by first looking at all possible next states. Temporal difference 0 estimates the value of the next state by only looking at a single next state.


Dimensionality of the target for DQN agent training From what I understand, a DQN agent has as many outputs as there are actions (for each state). If we consider a scalar state with 4 actions, that would mean that the DQN would have a 4 dimensional output.
However, when it comes to the target value for training the agent, it is usually described as a scalar value = reward + discount*best_future_Q.
How could a scalar value be used to train a Neural Network having a vector output?
For example see image in
https://towardsdatascience.com/deep-q-learning-tutorial-mindqn-2a4c855abffc

I am of the opinion that this architecture is only one among others that can solve the same problem (for example one may have only 2 outputs one for the chosen action and one for the $Q$ value of that action, but I will not elaborate further on this).
What this architecture does is to output the whole function $Q(a)$, that is the $Q$ value as a function of action $a$. So each output node represents the $Q$ value for a certain action $a$ corresponding to that node (so node 1 corresponds to the $Q$ value for action $a_1$, node 2 to $Q$ value for action $a_2$ and so on..)
BUT this is NOT a vector output, in the usual sense of the term. This is a functional output that represents the whole function $Q(a)$ for each action $a$.
As far as the learning/decision rule is concerned things are as usual. Hope this is clear.


Convolutional neural networks for non-image applications? If I remember correctly, Convolutional neural networks (CNN) have first been developed for image classification purposes (see work from LeCun et al.).

The convolution process that "slides" over the input image is helpful indeed to detect some features.

Question: do you have examples of successful application of CNN for non-image data?

They can be employed wherever you can find meaningful adjacent patterns in the input. As an example, you can take a look at here for its use in NLP:

You can also read the well-known paper Convolutional Neural Networks for Sentence Classification which is about sentence classification.


Sentiment Analysis Tutorial I am trying to understand sentiment analysis and how to apply it using any language (R, Python etc). I would like to know if there is a good place on internet for tutorial that I can follow. I googled, but I wasn't very much satisfied because they were not tutorials but more of theory. I want theory and practical examples.

The NLTK book is by far the best tutorial on basic NLP I have seen(in Python).

The Coursera course on NLP is also fairly good. It takes off from the basics and takes the student to a novice level.


Building a text classification model from scratch I do not know if this is the right place to post my question.
I am a beginner in data science and machine learning techniques. I would need to build a modem that can allow me to classify texts and run some sentiment analysis.
Right now I have only data collected and they miss any information about sentiment analysis.

Data collected (texts) are approximetely 50000 and they where already cleaned of punctuation and stop words.
I heard about the possibility to build some neural networks or use a logistic regression, but I do not know anything about specific models to use for that.

Furthermore, I might consider to build a new model from scratch (I know that can take ages and a lot of efforts) but to do it ai would need to know what I would need (for example, already existing model/dataset to train with dictionaries and sentiment analysis).
The problem is that what inwould like to do is to classify a text as positive or negative (sentiment); also, classify it as fake or not fake.

Do you have any suggestion or advice?
If you need further information, I would be happy to provide them.

To train a model on your collected data, you need to label your text first. For example, given a sentence

  Cats can fly.

It need to be labelled as 'Faked'. Otherwise your model wouldn't know how to classifier between 'Faked' and 'Not Faked'.

However, labelling needs huge effort. So if your data doesn't have any label, I suggest you use a trained model to predict or train a model on an existing sentiment classification dataset. You can search on the Google for the appropriate model of sentiment analysis.


Constructing a Maximum Entropy Classifier for Sentence Extraction So I'm reading this paper which uses a max ent classifier for sentence extraction. The parametric form for a conditional max ent model is :
$$ P(c|s) = \frac{1}{Z(s)} \exp \sum_i \lambda_if_i(c,s)$$
where $f_i(c,s)$ is a feature which has a weight $\lambda_i$ associated with it.
Now, the paper states that conjugate descent was used to find the optimal set of weights (page 3) - and this is what I'm unable to comprehend. How to use/apply conjugate gradient descent to calculate the optimal set of weights?

Conjugate gradient descent is a variation on gradient descent. Gradient descent is a method of finding the minimum of a function by taking steps that reduce the error between the data and the model parameters. Conjugate gradient descent extends gradient descent by searching a plane, instead of a line. The plane is defined as a linear combination of the gradient vector and the previous descent step vector. Conjugate gradient descent is very good at finding the solutions to a set of sparse linear equations.

Gradient descent and variations are general methods for finding the best parameters. The best practice is to define your specific model and then call on a separate gradient descent package to search for optimal values (i.e., hyperparameters) for the model.


Unexpected results from scikit learn regression decision tree Apologies for this newbie question. I have a scikit learn DecisionTreeRegressor with muti-variable output. If the output is in the format  [ output_var1, output_var2 ], where each variable is a continuous number not an integer, why the result is [1, 1] instead of [1.5, 1.5] ? What needs to be changed in this model to get [1.5, 1.5] ?

from sklearn.tree import DecisionTreeRegressor

X = [ [1,1], [2,2], [3,3] ]
y = [ [1,1], [2,2], [3,3] ]

print('X:' , X)
print('-----------------------------')
print('y:' , y)
print('-----------------------------')

regr = DecisionTreeRegressor()
regr.fit(X, y)

X_test = [ [1.5, 1.5] ]
print('X_test:' , X_test)
print('-----------------------------')

y_result = regr.predict(X_test)
print('y_result:' , y_result )

Result:

X: [[1, 1], [2, 2], [3, 3]]
-----------------------------
y: [[1, 1], [2, 2], [3, 3]]
-----------------------------
X_test: [[1.5, 1.5]]
-----------------------------
y_result: [[1. 1.]]

This is the visualization for the inducted tree from your data:

As you can see, it can never predict [[1.5, 1.5]]. Your data is just too pure, the inducted tree will fit it perfectly, having all leaf nodes with only one sample. If a leaf node has more than one sample in it, then the predicted value is the mean of the y values for those samples.

Furthermore, trees are not the best model to induct the identity function ($y = f(x) = x$). Trees work by creating rules and dividing the feature space orthogonally (i.e. only vertical or horizontal lines in the feature space).

In other words, your model didn't learn that f(1.5, 1.5) = (1.5, 1.5) because a) there's not enough data, b) trees are not good at doing that anyway.


Importing Excel format data into R/R Studio and using glmnet package? I have no problem importing Excel formatted data into R/R Studio and use all other R packages that I use.  But, when I want to use the glmnet package to develop a regularization model, I invariably run into the following error (after specifying my regularization model and attempting to run it):

Error in storage.mode(y) <- "double":
  (list) object cannot be coerced to type 'double'

Here is what I have already tried to resolve this:

De-format the numbers in Excel (no scientific notation, no %, etc.)
Did copy-paste-special values several times
At time of importing the spreadsheet, converted every column from "include" to "numeric" type
After importing the data, converting it to a matrix.

But, none of the above have eliminated the error.

After soliciting assistance from many colleagues, I found the answer to my problem.  And, I thought it be worth sharing.  glmnet package requires that you format your data as matrix.

What I had already done is import and transform the data as matrix as follows:

data.matrix(data)

But, that is not enough.  You still have to convert the X and Y arrays as matrix as follows:

a) First do the x matrix:
x = as.matrix(data,...);
b) Second do the y matrix: y = as.matrix(data,...).

Once I did the above change the glmnet package seems to work just fine.

I am unclear why the glmnet package is so much more sensitive to how much you matricize your data.  But, that is the way it is.


K-Fold Crossvalidation in Tensorflow when using flow_from_directory for image recognition Disclaimer: I have very little experience with Tensorflow.

I have a custom dataset with 20 categories with 100+ images in each. I am doing 5-fold cross validation using InceptionV3 for transfer learning. The easiest way to load this dataset into Tensorflow that I was able to find was flow_from_directory. The method works for one fold, but not for 5 folds since you can't set the folds. How would I go about dividing up the generators into 5 folds? Should I use an alternative method of importing data instead of flow_from_directory? There was a similar question where the answer was seemingly just importing it in a different way.

from tensorflow.keras.preprocessing.image import ImageDataGenerator
datagen=ImageDataGenerator(preprocessing_function=preprocess_input,
                                 validation_split=0.2)
train_generator=datagen.flow_from_directory('/content/dataset',
                                                 target_size=(299,299),
                                                 color_mode='rgb',
                                                 batch_size=32,
                                                 class_mode='categorical',
                                                 shuffle=True,
                                                  subset='training')
val_generator = datagen.flow_from_directory('/content/dataset',
                                                 target_size=(299,299),
                                                 color_mode='rgb',
                                                 batch_size=32,
                                                 class_mode='categorical',
                                                 shuffle=True,
                                                subset='validation')

The easiest way I found was replacing flow_from_directory command to flow_from_dataframe (for more information on this command see).

That way you can split the dataframe. You just have to make a dataframe with images paths and labels.

 i = 1
 df_metrics = pd.DataFrame()
    kf = KFold(n_splits = 10, shuffle = True, random_state = None)
    for train_index, test_index in kf.split(dataframe):
        trainData = dataframe.iloc[train_index]
        testData = dataframe.iloc[test_index]
        print('Initializing Kfold %s'%str(i))
        print('Train shape:',trainData.shape)
        print('Test shape:',testData.shape)
        epochs = 30

    train_datagen = ImageDataGenerator(rescale=1./255,validation_split=0.2)
    test_datagen = ImageDataGenerator(rescale=1. / 255)

    train_generator=train_datagen.flow_from_dataframe(
    dataframe=trainData,
    directory="./train/",
    x_col="id",
    y_col="label",
    subset="training",
    batch_size=batch_size,
    shuffle=True,
    class_mode="categorical",
    target_size=(img_width, img_height))

    validation_generator=train_datagen.flow_from_dataframe(
        dataframe=trainData,
        directory="./train/",
        x_col="id",
        y_col="label",
        subset="validation",
        batch_size=batch_size,
        shuffle=True,
        class_mode="categorical",
        target_size=(img_width, img_height))

    test_generator=test_datagen.flow_from_dataframe(
        dataframe=testData,
        directory="./test/",
        x_col="id",
        y_col="label",
        batch_size=1,
        shuffle=False,
        class_mode="categorical",
        target_size=(img_width, img_height)

       .
       .
       .

       i +=1


Random-Forest-based Similarity Matrix for clustering: how does it behave? I am in the following context:

Data: static, baseline health data at the patient level, 40 features, sparse (~ 25 binary features with many 0 or many 1 + other categorical features)

Objective : to cluster instances into clinically meaningful sub-population or clinical context, to get a sense of at risk sub-populations (on a Follow-Up outcome)

Considered approach  (see this short blog article (2 minute's read) for its rationale):

Fit a random forest on the Follow-Up outcome, using all features (no denoising or removing correlation)
Use co-occurence in trees leaves to get a similarity matrix of the patients
Turn in into a distance matrix
Cluster patients with this distance matrix

My questions are as follow:

I have only found literature using this methodology for unsupervised clustering (i.e. RF is learned on random target variable) : Shi and Horvath 2016 and Dalleau 2018 (unpublished). Does anyone have insights on those methods in general (other references, personnal experience...)?
Do you know of any article written about the supervised use of RF to create similarity matrices (with or without mention of clustering)?

I recently presented a poster at a conference where we used the same approach you describe for clustering. Generally, I think it's a great approach for clustering, as you can get clusters which have a distance that is relative to an outcome or variable that you are interested in.

For some insight, I have a few pointers:
1) When getting the co-occurence in trees, depending on how many subjects you have, this can end up being a very sparse matrix. To make the matrix less sparse you can increase the minimum number of samples required in terminal nodes.
2) After you get the similarity matrix, do a PCA on this, ending with individuals in rows and PCs in columns. Get the distance between individuals in this PCA space, restricting to some top number of components that you find acceptable. I recommend this because the similarity matrices can be huge if you have a lot of cases.


multiple digit detection This question is about detection of a number having multiple digits in a single image.
I have a trained model in tensorflow using Deep CNN for image recognition. the training was done on cropped images which only had a digit and its label.
Now, the task is to identify any number from the image using this trained model.
I would like this model to work on arbitrary sized number. so how do i extract certain features from the image to be predicted so that individual digit can be detected.
For reference this is the link to the google research paper though it does not elaborate on the issue
here is the data
this question is similar but it does not answer fully as digits nay be too small and close to each other.

For this problem, I could provide you with 2 approaches, 1. Naive approach 2. Actually used for object detection.

Talking about the Naive approach, in this approach we can build an end to end pipeline for digits extraction using the following pipeline:

Take images and clean image (maybe take threshold or tune brightness and contrast to remove noise and artefacts)
Take the cleaned image and find the contours of the text inside the image over the text and then finally detect the numbers.

This is certainly one solution but has a lot of complications involved alongwith.
To solve this problem, here is other approach that is used in actual OCR. For this, we follow this pipeline:

Take a cleaned image
From left to right (In the order of actual human reading pattern) run a sliding window and after Non-max suppression, find the text.
For each text found using the sliding window, run the prediction and store the prediction in the required format of your choice.

Although this too contains a number of complications and problems related to a number of other problems, this is still the best method and can be tuned as per the requirement to extract the exact required text.


Differences between applying KMeans over PCA and applying PCA over KMeans Short question:

As stated in the title, I'm interested in the differences between applying KMeans over PCA-ed vectors and applying PCA over KMean-ed vectors.

Long question:

Let's suppose we have a word embeddings dataset. Each word in the dataset is embeded in R300.

We want to perform an exploratory analysis of the dataset and for that we decide to apply KMeans, in order to group the words in 10 clusters (number of clusters arbitrarily chosen).

After doing the process, we want to visualize the results in R3. We could tackle this problem with two strategies;

Strategy 1 - Perform KMeans over R300 vectors and PCA until R3:

Apply KMeans to the R300 embeddings.
Perform PCA to the R300 embeddings and get R3 vectors.
Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/KMeans_PCA_R3.html

Strategy 2 - Perform PCA over R300 until R3 and then KMeans:

Perform PCA to the R300 embeddings and get R3 vectors.
Apply KMeans to the R3 embeddings.
Plot the R3 vectors according to the clusters obtained via KMeans

Result: http://kmeanspca.000webhostapp.com/PCA_KMeans_R3.html

Are there any differences in the obtained results? Any interpretation?

In case both strategies are in fact the same. Why is that?

There is a difference. In your first strategy, the projection to the 3-dimensional space does not ensure that the clusters are not overlapping (whereas it does if you perform the projection first).

This is because some clusters are separate, but their separation surface is somehow orthogonal (or close to be) to the PCA.

If you increase the number of PCA, or decrease the number of clusters, the differences between both approaches should probably become negligible.


Dimension of weight matrix in neural network Why would the dimension of $w^{[2]}$ be $(n^{[2]}, n^{[1]})$ ?

This is a simple linear equation, $z^{[n]}= W^{[n]}a^{[n-1]} + b^{[n]}$

There seems to be an error in the screenshot.
the weight, $W$ should be transposed, please correct me if I am wrong.

$W^{[2]}$ are the weights assigned to the neurons in the layer 2

$n^{[1]}$ is the number of neurons in layer 1

Screenshot from Andrew Ng deeplearning coursera course video:

There seems to be an error in the screenshot. The weight, $W$ should be transposed, please correct me if I am wrong.

You are wrong.

Matrix multiplication works so that if you multiply two matrices together, $C = AB$, where $A$ is an $i \times j$ matrix and $B$ is a $j \times k$ matrix, then C will be a $i \times k$ matrix. Note that $A$'s column count must equal $B$'s row count ($j$).

In the neural network, $a^{[1]}$ is a $n^{[1]} \times 1$ matrix (column vector), and $z^{[2]}$ needs to be a $n^{[2]} \times 1$ matrix, to match number of neurons.

Therefore $W^{[2]}$ has to have dimensions $n^{[2]} \times n^{[1]}$ in order to generate an $n^{[2]} \times 1$ matrix from $W^{[2]}a^{[1]}$


Clustering of numerical data I am trying to do clustering in my dataset which has 4 numerical fields. Please find the file attached : http://www.filedropper.com/example_3

I tried with this code:

from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2, random_state=0, max_iter = 300).fit(dffinal)

I know that there are 2 classes in this example, that was the reason i tried with 2 clusters. Out of 4200 rows, first 3196 rows belong to a class and the remaining rows belong to an another class.

But when i do clustering, cluster labels are randomly assigned, and accuracy is less than 10%. Just wondering whether my features are not good enough for clustering or should I try with some other clustering algorithm.

Any help would be appreciated. Thanks.

You forgot to preprocess your data.

K-means is really sensitive to scale and outliers.

Also: clustering is not classification.

It may well be that e.g. one of your classes has a dense subtype. Just because the clustering found something else than your labels (and it's not supposed to find the same, is it?) does not mean it failed. It just didn't classify the data the same way as your labels (but if that is what you wsnt, then you should have been using a classification method instead.)


Handling data imbalance and class number for classification 1 Is there a way to handle data imbalance? ie if data in each class for training is not balanced, say some classes have 50 documents some other have 200 documents. How to handle this?

2 How to handle the classification problem with a large number of classes? I have around 50+ classes (may increase once more data is available) to learn. I am trying out different algorithms and features. Is there any way to handle classification problem with a large number of classes?

People talk a lot about data imbalance, but in general I think you don't need to worry about it unless your data is really imbalanced (like <1% of one label).  50/200 is fine. If you build a logistic regression model on that dataset, the model will be biased towards the majority class - but if you gave me no information about an input to classify, the prior probability is that the new input is a member of the majority class anyway.

The question you want to be able to answer is whether you are differentiating classes fine - so if you do have a minority class, do NOT use 'accuracy' as a metric. Use something like area under the ROC curve (commonly called AUC) instead.

If your data is really super imbalanced, you can either over-sample the minority class or use something called 'SMOTE', for "Synthetic Minority Over-Sampling Technique", which is a more advanced version of the same thing.  Some algorithms also let you set higher weights on minority classes, which essentially incentivizes the model to pay attention to the minority class by making minority-class errors cost more.

To learn to differentiate between lots of classes, I think (a) you will need to have a ton of examples to learn from and (b) a model that's expressive enough to capture class differences (like deep neural network, or boosted decision tree), and (c) use softmax output. If those still don't work, you might try a 'model-free' approach like K-nearest-neighbors, which matches each input to the most similar labeled data.  For kNN to work however, you need to have a very reasonable distance metric.


Mimic a Mahout like system I have a data set, in excel format, with account names, reported symptoms, a determined root cause and a date in month year format for each row. I am trying to implement a mahout like system with a purpose of determining the likelihood symptoms an account can report by doing a user based similarity kind of a thing. Technically, I am just hoping to tweak the recommendation system into a deterministic system to spot out the probable symptoms an account can report on. Instead of ratings, I can get the frequency of symptoms by accounts. Is it possible to use a programming language or any other software to build such system?

Here is an example:

Account : X Symptoms : AB, AD, AB, AB

Account : Y  Symptoms : AE, AE, AB, AB, EA

For the sake of this example, let's assume that all the dates are this month.

O/P: Account : X Symptom: AE

Here both of them have reported AB 2 or more times. I could fix such number as a threshold to look for probable symptoms.

This seems to me as the plain old recommendation problem. The Accounts are the USERS and the Symptoms are the ITEMS. Each time an Account shows a particular Symptom your system will increment a count value.

Creating the following dataset:

ACCOUNT, SYMPTOM, COUNT

Now you can use/implement any sort of recommender system (Mahout is only an option, have you seen MyMediaLite) or you can even implement yours.

Let's reuse your ideas:
* You'd like to use a user-based similarity
* If an Account has shown 2 or more times the same symptom it seems to be important

So you can filter out the Account, Symptom pairs with less than 2 counts, and with the rest you create the following datasets:

User, Item dataset:

ACCOUNT, SYMPTOM

Table with a unique column containing all Users:

ACCOUNT

Table with a unique column containing all Items:

SYMPTOM

Now you can use directly the User-KNN algorithm from MyMediaLite.

With the recommender model already trained you can pass any ACCOUNT as input and it will give you a ranked list of the most probable SYMPTOMS that might appear.

Obs.: Initially ignore the time, then you could use it to partition your data in past/future and evaluate the recommendation in a more realistic way. ;-)


How to utilize user survey answers and the actual usage in forecasting power usage using LSTM? I have the pre-trial survey and post-trial survey conducted of around 5000 users for Smart Meter installation.

With this I have power usage reading recorded every 30 min for around one and a half years.

survey csv is like this

User Question 1 Question 2 .......
1000    a           a      .......
1001    b           a      .......
.       .           .      .......
.       .           .      .......
.       .           .      .......
5000    b           a      .......

power usage csv is like this

User date usage
1000 20001 0.003
1000 20002 0.025
.... ..... .....
.... ..... .....
.... ..... .....
.... ..... .....
.... ..... .....
1000 65047 0.52
1000 65048 0.14

I want to forecast power usage of the user based on the past power usage and the survey using LSTM.
How to start with this?

Check out the example "Multi-input and multi-output models" here in the documentation. It gives an example how to combine an LSTM time series model with additional input. You would use separate input layers for the power usage and for the survey data. After running the power usage data through the LSTM you combine the results using "concatenate".

(You can ignore the Embedding layer in the example since your power usage data isn't text data.)


Making Prediction on logistic regression using SAS proc logistic data= train descending;
   class Emp_status Gender Marital_status;
   model Default = Checking_amount  Term  Credit_score  Gender  Marital_status
   Car_loan  Personal_loan  Home_loan  Education_loan  Emp_status
   Amount  Saving_amount  Emp_duration  Age  No_of_credit_acc;
run;

As you can see, I already made a logistic regression on train dataset. However, how can I make the prediction on the test dataset? I checked online, but there are not many sources to implement this.

proc surveyselect data=work.data method=srs seed=2 outall
samprate=0.7 out=work.data_subset;

data training;
set work.data_subset; *dataset in your work directory;
if selected = 1;
run;

data testing;
set work.data_subset; *dataset in your working directory;
if selected = 0;
run;

ods graphics on;
proc logistic data=work.training descending plots=roc;
class Gender /param = effect ref = first; *categorical variable;
model default = x1 x2 x3 x4 x5
/ link=logit;
score data=work.testing out=work.logisticOoutput;
run;
ods graphics off;

Using proc surveyselect to split the dataset 70% 30%, we can split our dataset into train and test. Then, we can run logistic regression on train data. see the performance on the test dataset.

score data=work.testing

This command is running the regression on the test set. see the result in the output.


Giving higher priority to certain inputs in SKLearn Random Forest Here is the sample data I have:

Tag 1(Val: X), Tag 2(Val: Y), Tag 3(Val: Z), Label (Val: P)

Tag 1(Val: A), Tag 2(Val: B), Tag 3(Val: C), Label (Val: Q)

Tag 1(Val: D), Tag 2(Val: E), Tag 3(Val: F), Label (Val: R)

Tag 1(Val: G), Tag 2(Val: H), Tag 3(Val: I), Label (Val: S)

I am using Scikit Learn Random Forest Classifier.
I want the Classifier to give higher priority to certain inputs.
For Example: 100x priority to Tag 1: 10x priority to Tag 2: 1x priority to Tag 3.

Does Scikit-Learn have any parameters where I can specify this?

[Since the purpose of the Random Forest is to learn the priority of the inputs, is my question nonsensical?]

Appreciate your insight and support.

You've kind of answered your own question there!

One of the strengths of machine learning with algorithms like Random Forests is that they are able to determine the feature (input) importance themselves. My advice would not be to weight them for now (not sure you actually can with SKLearn's RF) and let the model determine it itself.

What you can do is give additional weight to the rows (observations) of your data set, or give additional weight to different classes of the label (for example, if there aren't many 'P's in your label but they're important, you can bump up their weight relative to the other classes).


Change rate of cross validation data, after training Say we have N of labeled data, and we need to take some part for the cross validation (we will skip test part for this case). We chose, 0.6 part for the training and 0.4 for validation.

After training neural Network with early stop, we have found 8 epochs, as optimal to stop, and have received good enough results.

Q. In case, we have very limited N training samples. May we use all samples in new model training, and just stop it's training after discovered epochs? Without separating it to train and cross validation, and testing it, at all (or even, change rate of separating, to 0.9 train, 0.1 cross validation).

Maybe there is known technologies for such cases? Thanks.

Due to stochastic nature of NN training, the best epoch may vary upon each restart. In other words, at epoch 8, each of (the best, under-fitted, over-fitted) cases may happen. However, if you train multiple times and the best model is consistently found at (or around) 8th epoch, it is safe to say 8th epoch gives the best model away from under- or over-fitting, thus definitely validation set can be added to training set to improve the performance.

A more solid approach would be to plot the effect of training size (10% up to 90%) on the best epoch and the validation error. This means producing two plots (training size, the best epoch) and (training size, validation error), where each point is an average over multiple restarts. This way you can better find (1) the best epoch, and (2) the degree to which the added validation set will going to boost the performance on unseen test data, i.e. extrapolating the validation error for training size 100%.

It is possible that performance goes flat after for example 70% of training set, implying that adding the validation set has no gain.


How to cluster texts by most relevant words I have a huge amount of documents and every document has its own portrait, where a portrait has this structure (document_id, word, weight). TFIDF, basically.
I want to cluster these documents into different clusters, say, 10.
I'm trying to implement the K-Means algorithm with sklearn, but I have almost zero experience with data science whatsoever. All tutorials that I found get texts as input from Wikipedia or somewhere else, but I don't have access to the texts themselves. I have only their portraits. Hope that makes sense.
Is this something that can be achievable with sklearn and if so, can you guide me where to dig or what to look at

You can use these words with their weight as a vector representation of the document. The important point is to make all the documents vectors over the full vocabulary, so that any position $i$ in any vector always represents the same word $w_i$. This means that a vector should contain zeros in all the positions corresponding to a word which is not in the document.
Using these vectors you can indeed use k-means to cluster the documents. Of course the quality of the results depends on the data: if there are very few words in common, it cannot work very well.


Which are the latest Optimization techniques in artificial intelligence? My project work is optimization in power system using artificial intelligence (like fault location and classification,load forecasting and context awareness and IoT etc ) and I have used PSO (particle swarm optimization, supervised fine tuning strategy etc). Please can anyone suggest a new technique which will give best result in comparison to other techniques?

You are asking for suggestions of a technique which will give the best results in comparison to other methods, that's difficult to say a priori as it will depend on your particular features and dataset and will be guided by trial and error and comparison of different algorithms (this is also related to the No Free Lunch Theorem, see some discussion here). However, there are certainly many optimization techniques advances in Artificial Intelligence. In this answer I'll cover the ones I'm aware of.

Recently there have been several nice advances where new perspectives in optimization models have been developed classic problems Machine Learning. These works have been published by renowned Operations Research academic Dimitris Bertsimas. See for example this link. For example, I find the following publications from the list to be very interesting and promising (you can find free PDF versions at the last link):

Bertsimas, D., King, A., & Mazumder, R. (2016). Best subset selection via a modern optimization lens. The annals of statistics, 44(2), 813-852.

In this work, the authors present a Mixed Integer Optimization approach for solving the best subset selection problem (i.e. to choose a subset of features out of a larger set in linear regression given a collection of observations). Typically the Lasso technique is used, but in several computational experiments it was outperformed by this method. Moreover, this new theory develops criteria for proving that the adquired solution is indeed optimal (or near-optimal), which was not possible using heuristic methods.

Bertsimas, D., & Dunn, J. (2017). Optimal classification trees. Machine Learning, 106(7), 1039-1082.

In this case I cite the abstract:

  State-of-the-art decision tree methods apply heuristics recursively to create each
  split in isolation, which may not capture well the underlying characteristics of the dataset. The optimal decision tree problem attempts to resolve this by creating the entire decision tree at once to achieve global optimality. In the last 25 years, algorithmic advances in integer optimization coupled with hardware improvements have resulted in an astonishing 800 billion factor speedup in mixed-integer optimization (MIO). Motivated by this speedup, we present optimal classification trees, a novel formulation of the decision tree problem using modern MIO techniques that yields the optimal decision tree for axes-aligned splits. We also show the richness of this MIO formulation by adapting it to give optimal classification trees with hyperplanes
  that generates optimal decision trees with multivariate splits. (...) We comprehensively benchmark these methods on a sample of 53 datasets from the UCI machine learning repository. We establish that these MIO methods are practically solvable on real-world datasets with sizes in the 1000s, and give average absolute improvements in out-of-sample accuracy over CART of 1–2 and 3–5% for the univariate and multivariate cases, respectively. Furthermore, we identify that optimal classification trees are likely to outperform CART by 1.2–1.3% in situations where the CART accuracy is high and we have sufficient training data, while the multivariate version outperforms CART by 4–7% when the CART accuracy or dimension of the dataset is low.

In their computational experiments the model has even outperformed Random Forest and XGBoost, as can be seen on slide 29/41 of this presentation Bertsimas gave earlier this year. This is quite impressive, as these algorithms are among the most frequently used by winning solutions in Kaggle (see for example this link and this one).

Bertsimas, D., & King, A. (2017). Logistic regression: From art to science. Statistical Science, 32(3), 367-384.

This case is similar to the first one, but for logistic regression instead of decision trees. Again they achieved new algorithms for training a classic Machine Learning model, getting high quality solutions as well as near-optimality guarantees.

Besides this line of work, large scale convex optimization is another area where lots of attention have been drawn. Big data and machine learning/deep learning applications motivate the development of new algorithms and theory to achieve scalable and efficient solutions in this new contetxt. See for example Bottou, L., Curtis, F. E., & Nocedal, J. (2018). Optimization methods for large-scale machine learning. SIAM Review, 60(2), 223-311.

All the advances I mentioned are very recent, so don't expect to have access to available and scalable libraries to use them out of the box. Therefore, the usefulness of them for your particular project will depend on (1) the scope of your project (research or more short term objective?), (2) the size of your datasets, and (3) the available time you have for working on it and delivering a solution.


what is label shift? I'm studying a paper about Named Entity Recognition. The following is a part of the abstract:

To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific
sources of errors: unknown words and label shift or ambiguity.

I don't know what "label shift" definition is. The paper doesn't explain it and I can not find anything that I could understand by Googling it.

Label shift is the opposite of a covariate shift.
 In this case, the assumption is that even though the feature distribution remains the same, the Label distribution might changes.
 e.g. Symptoms --> Diseases
 It can be different for different country (based on medical education of the Country/Doctor)
 It can change with time also based on advancement in Medical knowledge
  Similar logic can be built for "Words --> Slang". It can change with time due to the acceptance of these words.
Read the references for a formal explanation.
References-
Dive into Deep Learning
Detecting and Correcting for Label Shift with Black Box Predictors


How to choose the features for an algorithm from the given attached screenshot? How to choose the features from the given attached heat map & correlation factor for the classification algorithm?

I have 6 different features i.e., ac233fc01403, ac233fc02eaa, ac233fc015f6, ac233fc044aa, ac233fc0455d, ac233fc0451f. From the 6 different features, I should classify in which segment(i.e.,class) the test data is?

How to choose from the given plot, which are the feature has to be considered for the algorithm to achieve the better efficiency?

Why start with feature selection/engineering if you have so few features to begin with?

Best practice should be to include all features/variables in your first draft model and look at the accuracy, quality of resulting segments, etc.

If there is need to improve and to leave out a feature you could then use the metrics of variable importance of your chosen model (e.g. variable importance from random forest) to decide which feature was least relevant and could be discarded for a slimmer model.

In general model performance shouldn't suffer to much from including irrelevant features and therefore selection isn't the same issue as it would be for less robust models.


DBSCAN clustering on document [updated]? I am new in topic modeling and text clustering domain and I am trying to learn more. I would like to use the DBSCAN to cluster the text data. There are many posts and sources on how to implement the DBSCAN on python such as 1, 2, 3 but either they are too difficult for me to understand or not in python.
I have a CSV data that has userID and message that they wrote as follows:

user.csv (number of csv rows:400 (#message))

userID         messages
112   The car was broken and Kevin fixed it
.
.
.

I know some steps to apply DBSCAN such as:

Remove stop words
Find similarity distance ( I have a code that does the cosine similarity)

I am also aware that sci-kit learn has the demo at 4 but I prefer the manual implementation that I can see what's going on in the code.

It would be great if you can provide your help with code that I can run in my side to learn it.

Bilgin!

Anony-Mousse puts right questions and gives good suggestions. Before you use the self-implemented DBSCAN code - write it on paper. Perhaps it is not the best algorithm at all for your database so try sci-kit learn implementation first to see the results.

Here are the Python implementation https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py and here is the theory https://github.com/chrisjmccormick/dbscan/blob/master/dbscan.py

Good luck!


Why did sampling boost the performance of my model? I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.

So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.

I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?

As you mentioned in a comment, you are upsampling before splitting the test set, which leads to data leakage; your scores are not to be trusted.  The problem is that a given positive sample may be duplicated and then put into both the training and the test set.  Especially with tree models, this is very likely to correctly predict that sample in the test set.  The story with SMOTE is similar, but as you pointed out, not quite so severe.  In SMOTE you're interpolating between positive samples (see image from imb-learn docs), so if some of those points are in the training set and some in the testing set you're still more likely to correctly identify those points.

Instead, you should split first, upsample the training set second.  Alternatively, set class weights (this has the benefit of being independent of the split).  Now your test set has a different distribution that the training set, so you'll need to adjust the class prediction threshold, or adjust the probability predictions.  See e.g. "Convert predicted probabilities after downsampling to actual probabilities in classification?".  Part of the question here is whether you want actual estimates of the probabilities, or just care about the class predictions.

There's a serious question about whether resampling techniques are helpful at all.  See e.g.
"What is the root cause of the class imbalance problem?"
"When is unbalanced data really a problem in Machine Learning?"
As a first attempt, I would stick with the original data, fit the random forest, and have a look at different thresholds.

In your case, I would worry that 88 positive samples may just not be enough to see a meaningful pattern.  (It might be; it depends on how separated the classes are.)


Grouping factors in more general ones I have to work on this database on R. I have tried to make a tree model, but all the variables are factors with too many levels. I tried changing the column "operator" in three levels (keeping private, having military for all "military - state" and air company for all the others) using something like getData$operator[grep("Military", getData$operator)] <- "Military" but it changes into NAs since it's not a valid factor.

I could change it to string and make it work, but it doesn't solve the "if not rivate or military, change to air company" part.
Is there any other way?

Make a new column equal to „company“. Use the grep to replace military by „military“, use the grep to replace private by „private“. Change the new column to a factor.


How do I convert a summation equation to a vector equation (backpropagation)? $$a_j^l=\sigma(\sum_{k} {w_j}_k^l {a}_k^{l-1}+b_j^l)$$
$$a^l=\sigma( w^l {a}^{l-1}+b^l)$$
In a resource I have been reading, the above equations describe the activation of a neurone. They have the same function (I agree) but the first is a summation representation, whereas the second is this equation but in a vector form. My questions are, What did they do to remove the subscripts, and what is the process for converting summation equations to vector equations?

$a_j^l$ is the $j$-th component of $a^l$. Hence, the first equation is $1$ dimensional, the second equation is a vector equation.

Similarly for notaiton $b_j^l$ and $w_{jk}$.

$\sigma$ means sigmoid function. If the input is a vector, then it evaluated the vector componentwise.

To answer your main quesiton, recall that from definition of  matrix vector product, where $A$ is a matrix and $x$ is a vector, $Ax$ produces a vector $y$ where the $j$-th component is $y_j= \sum_k A_{jk}x_k$.

That is element of $$y=Ax$$

can be written as $$y_j = \sum_k A_{jk}x_k$$

elementwise.

Hence that is how we can "remove" the summation sign.


LSTM Long Term Dependencies Keras I am familiar with the LSTM unit (memory cell, forget gate, output gate etc) however I am struggling to see how this links to the LSTM implementation in Keras.

In Keras the input data structure for X is: (nb_samples, timesteps, input_dim).

Suppose that the shape of X is: (1000, timesteps = 10, 40).

1) Does this mean that the LSTM cells will only consider ‘batches’ of 10 previous time steps ?

2) Or is the output from LSTM cells passed between these sets of 10 timesteps I.e could you capture long term dependencies 50 timesteps out?

The last number 40 has nothing to do with the sequence length, its a hyper-parameter setting that is basically the length of the vector 'representation' of each token in the sequence. In this case , its 40 length. If you set it to 40 and use input embeddings of input 300-dimension (common if Glove), then the 300-dimensional word gets mapped to a 40-dimensional word that goes through the LSTM permutations.
The idea is very similar to the number of 'kernel maps' in a CNN, if you're familiar with those. Its just a way to tell your network how many internal features you want your model to generate as your LSTM does its thing. More features means stronger representational power and better information flow, but at the possible cost of overfitting.


How to implement LSTM using Doc2Vec vectors? I would like to build a ANN for text classification, which has an LSTM layer, and using weights obtained via a Doc2Vec model trained before:

model_doc2vec = Sequential()
model_doc2vec.add(Embedding(voacabulary_dim, 100, input_length=longest_document, weights=[training_weights], trainable=False))
model_doc2vec.add(LSTM(units=10, dropout=0.25, recurrent_dropout=0.25, return_sequences=True))
model_doc2vec.add(Flatten())
model_doc2vec.add(Dense(3, activation='softmax'))
model_doc2vec.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

At the moment I am not able to get the weights in the Embedding() layer mentioned above. I would like to know which is the easiest way to get these weights.

Weights are nothing but the pretrained word vectors . You can use any of word2vec or Glove embedding and create an embedding matrix to get that . Essentially each row of the matrix will be the vector for a word in the word2vec/Glove vocabulary. Please have a look here


How do I predict new/unknown data in Bayesian linear regression? This is my first question on this forum. I just got started with Bayesian statistics. While I do understand the motivations behind Bayesian methods, I am a little unclear on what the predictions even mean. Consider a standard regression problem of predicting the price of a house given its area in square feet. Assume the optimal parameters (slope and intercept) of the model have been found. The price of any new house (x_new) is just the number - {intercept +  slope * x_new}.

In Bayesian linear regression, we work with the so-called posterior predictive distribution (abbreviated PPD). But what is the PPD anyway?

1) Is it a probability density function (pdf) with some parameters like mean and (co)variance? If so, how do I obtain a single value of the house price from this density function? Should I just take the mean of this distribution or are there sophisticated techniques available?

2) Is it a real number, given by the equation - intercept +  slope * x_new? If so,
a. Are intercept and slope sampled from the posterior distribution of slope and intercept?
b. Or are the posterior mean values of slope and intercept used for computing the price of the new house?

Let me help you get started -

you don't get a single value here like you get in linear regression,rather posterior distribution of the parameters. yes, The output here is generated from a normal (Gaussian) Distribution characterized by a mean and variance , nothing special I am afraid. But depending on the mean and variance you can determine which price is more likely to be.
yes its not a real valued number like price here but the probability ,output is the likelihood multiplied by the prior probability of the data.

Here is a very simple explanation, I couldn't put it in better words-

https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7

And here is the more mathematical background with this chapter from stanford-

http://cs229.stanford.edu/section/cs229-gaussian_processes.pdf

and you are not alone as these Author's agree-

"Gaussian process regression models, though perhaps somewhat tricky to understand
conceptually, nonetheless lead to simple and straightforward linear algebra implementations"


what is filter and kernel_size? For below line of code

model.add(Conv2D(filters = 32, kernel_size = (5,5),padding = 'Same',
                 activation ='relu', input_shape = (28,28,1)))

Here, what does 'filers' and 'kernel_size' mean? or what is filter and kernel_size ?

Filters are used to extract features from images in the process of convolution.

  filters: Integer, the dimensionality of the output space (i.e. the
  number of output filters in the convolution).

  kernel_size: An integer or tuple/list of 2 integers, specifying the
  height and width of the 2D convolution window. Can be a single integer
  to specify the same value for all spatial dimensions.

For detailed undestadning of filter and kernel_size, refer:
https://www.saama.com/blog/different-kinds-convolutional-filters/
https://keras.io/layers/convolutional/


Do I need to encode the target variable for sklearn logistic regression I'm trying to get familiar with the sklearn library, and now I'm trying to implement logistic regression for a dataframe containing numerical and categorical values to predict a binary target variable.
While reading some documentation I found the logistic regression should be used to predict binary variables presented by 0 and 1.
My target variable is "YES" and "NO", should I code it to 0 and 1 for the algorithm to work properly, or there is no difference?
Maybe I just didn't get the idea but can someone confirm this to me.

The string labels work just fine, here is an example:

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
import numpy
X, y = load_iris(return_X_y=True)
y_string = numpy.array(['YES' if label == 1 else 'NO' for label in y])
clf = LogisticRegression(random_state=0, solver='lbfgs', multi_class='multinomial').fit(X, y_string)
y_pred = clf.predict(X[50:100, :])
print(y_pred)

Output:

['NO' 'NO' 'NO' 'YES' 'NO' 'YES' 'NO' 'YES' 'NO' 'NO' 'YES' 'NO' 'YES'
 'NO' 'NO' 'NO' 'NO' 'YES' 'YES' 'YES' 'NO' 'NO' 'YES' 'YES' 'NO' 'NO'
 'YES' 'NO' 'NO' 'YES' 'YES' 'YES' 'YES' 'YES' 'NO' 'NO' 'NO' 'YES' 'NO'
 'YES' 'YES' 'NO' 'YES' 'YES' 'YES' 'NO' 'NO' 'NO' 'YES' 'NO']

Yo can replace y_string to y for the numerical example.


How regression algorithm works on categorical features I have a dataset with features that most of them are nominal categorical features, I have converted my model to indicator values,

Original

F1,F2,L
1 ,1 ,50
2 ,3 ,30

After indicator values

F1-1,F1-2,F2-1,F2-3,L
1   ,0   ,1   ,0   ,50
0   ,1   ,0   ,1   ,30

I used different regression algorithm (Poisson, Bayesian, Decision Tree reg, Decision Forest reg, Boosted decision tree reg, linear regression, neural network), but all of then have low performance (r2 ~ 20-30)

Then I was thinking how regression can find values, then I found something interesting : relation of data with label
They are like below picture

But in most of the books and examples and samples suitable data for regression are like below

And this is the point I got confused!

So my question is how regression (or which algorithms) are suitable for predicting values in high categorical data

TL;DR: It is a convention to convert categorical features to numeric features before you can use them in regression, or any other machine learning algorithm for that matter. Technically, there is nothing that is stopping ML algorithms from working with categorical features but their software implementation would be prohibitively expensive, hence this convention in ML practice.

Turning to your problem, if you are trying to convert categorical label values into numerical values, there are various methods but it appears the best one for your data would be One-Hot encoder. Both Scikit-Learn and PySpark, and most other libraries provide handy functions for it, as it is very common operation. For example:

from sklearn.preprocessing import OneHotEncoder
one_hot = OneHotEncoder()
one_hot.fit(...your_columns...)

Once you have all data in numeric form, you can use pretty much any algorithm.

As for the last figure, it is scatter plot between x and y, and it is not obvious what x and y are. Weirdly enough, it is not one-on-one relationship because a given value of x seems to produce multiple value of y?!!! And I am not sure what you are plotting in earlier figures that is giving you straight lines!


Analysis of Split (A/B) tests using Poisson and/or Binomial Distribution Cross posting this from Cross Validated:

I've seen this question asked before, but I have yet to come across a definitive source answering the specific questions:

What's the most appropriate statistical test to apply to a small A/B test?
What's the R code and interpretation to analyze a small A/B test?

I'm running a small test to figure out which ads perform better. I have the following results:

Position 1:

variation,impressions,clicks
row-1,753,26
row-3,767 7

Position 2:

variation,impressions,clicks
row-1,753,16
row-3,767 13

Position 3:

variation,impressions,clicks
row-1,753,2
row-3,767 7

I think it's safe to say these numbers are small and likely to be not normally distributed. Also, it's click data so there's a binary outcome of clicked or not and the trials are independent.

Appropriate test

In analyzing each position for significance, I think comparison with a binomial or Poisson distribution makes the most sense.

According to the OpenIntro Stats (and other sources) book, a variable follows a Poisson distribution "... if the event being considered is rare, the population is large, and the events occur independently of each other."

The same source classifies a binomial variable approximately the same way adding that the probability of success is the same and the number of trials is fixed.

I appreciate this is not an either/or decision and analysis can be done using both distributions.

Given A/B (split) testing is a science that has been practiced for several years, I imagine that there is a canonical test. However, looking around the internet, I mostly come across analysis that uses the standard normal distribution. That just seems wrong :)

Is there a canonical test to use for A/B tests with small #'s of clicks?

Interpretation and R code

I've used the following R code to test significance for each position:

Position 1:

binom.test(7, 767, p=(26/753))

Exact binomial test

data:  7 and 767
number of successes = 7, number of trials = 767, p-value = 1.077e-05
alternative hypothesis: true probability of success is not equal to 0.03452855
95 percent confidence interval:
 0.003676962 0.018713125
sample estimates:
probability of success
           0.009126467

I interpret this result to mean: The probability of success in the test group is indeed different than the control group with a 95% confidence interval that the success probability is between .368% and 1.87%

ppois(((26-1)/753), lambda=(7/767), lower.tail = F)
[1] 0.009084947

I interpret this result to mean: Given a Poisson distribution with a click rate of 7 per 767 trials, there is a 0.9% chance of having a click rate of 26 or more per 753 trials in the same distribution. Contextualized in the ad example,
there is a .1% chance that the control ad actually performs the same as the test ad.

Is the above interpretation correct? Does the test and interpretation change with the different positions (i.e. are the results of the Poisson test more appropriate for Position 3 given the small numbers)?

OK, so here's your data.

dd <- data.frame(position=rep(1:3, each=2),
                 variation=rep(c(1,3), 3),
                 impressions=rep(c(753, 767), 3),
                 clicks=c(26,7,16,13,2,7))

which is

  position variation impressions clicks
1        1         1         753     26
2        1         3         767      7
3        2         1         753     16
4        2         3         767     13
5        3         1         753      2
6        3         3         767      7

The two model assumptions you're thinking about are Binomial

mod.bin <- glm(cbind(clicks, impressions-clicks) ~ variation + position,
               family=binomial, data=dd)

where the dependent variable is constructed to have the count of the event of interest in the first column, and the Poisson

md.pois <- glm(clicks ~ variation + position + offset(log(impressions)),
               family=poisson, data=dd)

where the log(impressions) offset is necessary whenever the number of trials differs across observations.  This means coefficients are interpretable in terms of change in rate not change in count, which is what you want.

The first model generalises the binom.test to a setting with covariates, which is what you have.  That gets you a more direct answer to your question, and better (if not perfect) measurement of the relevant uncertainty.

Notes

Both models assume no interaction between variation and position ('independent effects').  This may or may not be reasonable.  You'd want more replications to investigate that properly.  Swap the + for a * to do so.

In this data summary confirms that the two models give rather similar results, so concerns about Poisson vs Binomial don't seem to matter much.

In the wild, count data is usually overdispersed, that is: more variable than you'd expect from a Poisson with a constant rate or a Binomial with constant click probability, often due to unmodeled determinants of click rate / probability.  If that's the case then prediction intervals from these models will be too narrow.


What's The Difference Between The Terms Predictor And Feature For the term 'predictor', I found the following definition:

  Predictor Variable: One or more variables that are used to
  determine or predict the target variable.

Whereas Wikipedia contains the following definition of the word 'feature':

  Feature is an individual measurable property or characteristic of a
  phenomenon being observed.

What is the difference between 'predictor' and 'feature' in machine learning?

Feature and predictor are used interchangeably in machine learning today though I must admit that it seems that feature is being used more than predictor. The definition is the one on Wikipedia which you have already mentioned. The term predictor comes from statistics, here one definition:

An independent variable, sometimes called an experimental or predictor variable, is a variable that is being manipulated in an experiment in order to observe the effect on a dependent variable, sometimes called an outcome variable.

and my favorite definition:
A predictor variable explains changes in the response.

In a nutshell:
X columns: features, predictors, independent variables, experimental variables.
y column(s): target, dependent variable, outcome, outcome variable.


Can 2 dimensional input be applied to SVM? When considering Support Vector Machine, in an take in multiple inputs. Can each of these inputs be a vector??
What i am trying to say is, can the input be a 2 dimensional vector??

If I understand your question correctly. Yes, SVM can take multiple inputs. My suggestion for handling a vector as a feature would be to expand it out. For example,

x0 = (1,2)       x0 = 1
x1 = .4  ----->  x1 = 2
x2 = 0           x2 = .4
                 x3 = 0

If this does not capture all of the characteristics of the vector that are important, then you may want to add other features (like magnitude of the vector) as well.


Document similarity matching between Doc2Vec documents I am creating a Doc2Vec model out of hundreds of PDF documents.

I have 17 documents that are part of this Doc2Vec that I want to use to check similarity with other documents in the Doc2Vec model.

For instance I want to something like: model.similarity(tag5, tag30)

Can this be done?

Based on what you have shared, it seems like Doc2Vec should be suited to your objective.

That said, I think the Doc in the package name can lead people astray.  It gives the impression that you can feed in any documents and it will find the similarities among them. And while you certainly can feed whole documents, the reality is that the similarity / dissimilarity may not be aligned in a useful way to your task.  I have found that training the model on smaller chunks of the document, like sentences or paragraphs, allows for the model to identify subtle differences that can then be aggregated to the whole document.

In your case, are the 17 documents you mentioned, exemplars of 17 different types of documents whose label you want to assign to the entire corpus? If so, Doc2Vec may be overkill.  Have you looked into using Tf-Idf? Sometimes this approach works well if you don't necessarily need the word embeddings.  Just some food for thought.


How is the LSTM RNN forget gate calculated? I understand that the LSTM recurrent neural network has a forget gate,  input gate, and output gate.

However, I do not understand how the equation below calculates 'forget' information (from Chris Olah's LSTM post)

How does this forget gate decide which information is discarded from the previous cell state? In other words, what is forgotten or not.

It appears to me that this equation just takes $x_t$ from the current state and $h_{t-1}$ from the previous state.

Can someone explain how this equation works as a forget gate to retain or discard cell state?

The equation and value of $f_t$ by itself does not fully explain the gate. You need to look at first term of the next step:

$C_t = f_t \odot C_{t-1} + i_t \odot \bar{C}_{t}$

The vector $f_t$ that is the output from the forget gate, is used as element-wise multiply against the previous cell state $C_{t-1}$. It is this stage where individual elements of $C$ are "remembered" or "forgotten".

Due to the sigmoid function, the vector $f_t$ behaves like a binary classifier for each element, with saturated values tending to settle on either not modifying $C$ at all (a value of $1$) or "forgetting" what the previous value was (a value of $0$). Of course intermediate values are also possible, and the analogy there between simply remembering and forgetting values is less direct.

The analogy with forgetting or remembering helps towards understanding the improvements regarding preserving gradients over multiple timesteps. For any step where an element of $f_t$ is close to $1$ (and thus the effect of previous timesteps is being "remembered"), then the corresponding gradient elements backpropagating from $\nabla_{\theta} C_{t}$ to $\nabla_{\theta} C_{t-1}$ remain the same, avoiding the gradient loss seen in more simple RNN architectures (especially related to saturated values). Once a previous element in $C$ is "forgotten", then the error gradient connection is cut off between time steps.


Dropout on which layers of LSTM? Using a multi-layer LSTM with dropout, is it advisable to put dropout on all hidden layers as well as the output Dense layers?  In Hinton's paper (which proposed Dropout) he only put Dropout on the Dense layers, but that was because the hidden inner layers were convolutional.

Obviously, I can test for my specific model, but I wondered if there was a consensus on this?

I prefer not to add drop out in LSTM cells for one specific and clear reason. LSTMs are good for long terms but an important thing about them is that they are not very well at memorising multiple things simultaneously. The logic of drop out is for adding noise to the neurons in order not to be dependent on any specific neuron. By adding drop out for LSTM cells, there is a chance for forgetting something that should not be forgotten. Consequently, like CNNs I always prefer to use drop out in dense layers after the LSTM layers.


What is the dimension of the filters if the input image has only one channel? I have a grayscale image with dimension HxWx1 (one channel). To build a CNN using the grayscale image as an input image, what is the dimension of the filters?

I read from some websites, it says that if the input image has only one channel, then the filter must be NxNx1. However, some other websites mentioned differently. It says even if the input image has one channel, it is fine to use filters with dimension NxNx3 by simply converting the grayscale image (HxWx1) to a RGB image (HxWx3).

Therefore, my question is: with input image that has only 1 channel, must the dimension of the filter be NxNx1 or NxNx3?

(I self-study this, so I might get the idea and understanding wrong. Help me, please.)

Note;
H: height, W: width, N: height and width of the filter

It is better, in most cases, to convert Grayscale images to RGB images with equal corresponding values in the 3 channels.

In this case, your filter size will be NxNx3.

The reason this approach is preferred is, among other things, if some time in future, you decide to use a pre-trained CNN you would most definitely need an RGB image to use its weights.


When it is a good idea to train a model also on the test data? I have a small restricted dataset, it is not very small but accuracy will be much better if I will have more data. I have split it to train and test datasets:  85%/15%. I have chosen NN model and trained it, measured loss, it is ok. Now I should use it in production. Should I train it additionally on the test data, because in such way I will get better accuracy?

The test set is there to show you how the model can perform to unknown cases and basically reassures that it is not overfitted. The selection of the "optimal" model needs to be made based on its performance on the test set and NOT on the training set, because what you need is the model with the best behavior in unknown cases.

In my humble opinion, just splitting the file in training and test set is not the best approach, as there might be bias in the accuracy calculation just because of the random way data were splitted. I would go with cross-validation in order to ensure that the model has been extensively tested. In cases with small amount of data, leave-one-out-cross-validation is often used in order to maximize the amount of available data for the training. You can read more on Cross validation and Leave-one-out-cross-validation (theory and python implementation) here.

When you have found the model with the best performance using cross-validation, I believe you can feel free to train it on the whole set of data before giving it to production.


K-Means Clustering too crowded I have written a simple python code that opens a csv files and then clusters the values of one column. There around 10k rows

This is my code

import pandas as pd
import numpy as np
import random as rd
import matplotlib.pyplot as plt

data = pd.read_csv('file.csv', encoding='unicode_escape')

data.head()

feature_names = ['numbers_col']
X = np.asarray(data[feature_names])

from sklearn.cluster import KMeans

labels = KMeans(5, random_state=10).fit_predict(X)
plt.scatter(X[:, 0], X[:, 0], c=labels,
            s=50, cmap='rainbow');

This is how the result looks like

The result doesn't look that good it looks all liner and the clusters are shown just like dots.

I am asking for some advice so my output would look more  towards what a normal K-Means output looks like: Example

The figure looks exactly how it should.

plt.scatter(X[:, 0], X[:, 0], c=labels, s=50, cmap='rainbow');

You are plotting the a value again itself, in two dimensions it will give you a line of the form y = x
Check what you want to plot, if you have only one feature it doesn't mean that the K-Means wouldn't work, you will have a clustering, but the plot its correct.


Document classification using convolutional neural network I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text or document.
My problem is that there are too many features from a document. In my dataset, each document has more than 1000 tokens/words. To feed each example to a CNN, I convert each document into a matrix by using word2vec or glove resulting a big matrix. For each matrix, the height is the length of the document, and the width is the size of word embedding vector. My dataset has more than 9000 examples and it takes a lot of time to train the network (a whole week) which makes it difficult to fine-tune parameters.

Another feature extracting method is to use one-hot vector for each word, but this will create very sparse matrices. And of course, this method even takes more time to train than the previous method.

So is there a better method for extracting features without creating large input matrices?
 And how should we handle variable length of documents? Currently, I add special strings to make document have the same length, but I don't think it's a good solution.

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence would be a "document").

If you don't want to use Doc2vec, one way to create the sentence vectors would be to average the word vectors for each sentence, giving you a single vector of the same width for each sentence. This may not be as precise as some methods available through Doc2Vec but I have used it with considerable success for topic modeling.

Either way once you have your sentence vectors, line them up in sequence for each document like you're already doing for your word vectors and run then through your model. Because the sequence length for each document is shorter, your model should train more quickly than with word vectors.

By the way, this method could work when scaled up or down to meet your accuracy and speed needs. (e.g. if your CNN still trains too slowly with sentence vectors, you can create paragraph vectors instead).

One way to handle documents of different length is through padding. Your document sequences should all be equal in length to your longest document. So if your longest document is 400 sentences then all document sequences will be 400 vectors in length. Documents shorter than the max length would be padded with vectors filled with zeros.


Creating bar plot with frequency distribution based I have a dataframe with this kind of structure:

ID | Size | Value
1  | 1    | 10
2  | 2    | 20
3  | 1    | 10
4  | 1    | 10
5  | 1    | 15
6  | 1    | 12
7  | 1    | 20

I want to create bar plot of the distribuition of column "Values" when the "Size" column is equal to 1.

With the current example:

the x-axis should have: 10, 12, 15 and 20
the y-axis should have 1 (for the 12, 15 and 20) and 3 for the 10

I don't have a lot of code. Basically I just created a new dataframe with only the rows where Size=1 and then performed value_counts():

data1 = data.loc[(data['Size'] == 1)]
count = data1['Value'].value_counts()

The variable count now has a Series object with a Value and a counter.

How can I split the value of count in order to have two lists (values and counters) to send to the plotting code?

What you are looking for is a histogram. It plots the distribution of a given series of values. So, you can just extract the values that you want to plot from your dataframe and then use matplotlib's histogram function. Here is the code:

import matplotlib.pyplot as plt
to_plot = data.loc[df['Size']==1]['Value']
plt.hist(to_plot)
plt.show()

This is the output I get, note that I haven't done any formatting, this is just the raw output.

The above option is more flexible. But in your case you can also use the following:

import matplotlib.pyplot as plt
to_plot = data.loc[df['Size']==1]['Value']
to_plot.value_counts().plot.bar()


Detect Missing Records in Dataset I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear and another to appear in the data as a whole.  Occasionally, a widget will just disappear and so the dataset is incomplete, invalidating the whole dataset for that day.

What I am looking for is a method of comparing the current set of widgets with another set of widgets to detect if any widgets are missing.  I am not trying to create the values, just identify that they are missing.  I could do time-series, but that feels like overkill on so many widgets and there are multiple attributes on which data might be missing.  I was hoping for something more set based that might account for the regular changes in widgets but detecting the unusual dropouts.  I am sure I just need to adjust the way I am thinking about the problem.

Any ideas would be much appreciated?

One option is hashing, assigning a numerical value to each widget. The best hashing option is if each widget had a unique id, something like a serial number. If a widget does not have inherent unique, then a hash value can be created by applying a hash function to the features of widget.
After creating hash values for each widget, set comparison can be used to see if any hashed values are different between the two sets of widgets.


Library/package/tool for geographical data visualizations I am looking for a good free existing tool which visualizes geographical data (let's say in the form of coordinates) by plotting them on a map.

It can be a library (see this question on StackOverflow, which suggests a Python library called basemap, which is interesting but not dynamic enough, namely it does not allow for interactivity) or a complete toolkit.

Existing things I found are oriented towards realizing web pages, which are not my ultimate goal (see Exhibit or Modest Maps). I'd like something to feed with data which spits out an interactive map where you can click on places and it displays the related data.

Assuming you want your visualizations to be available on the internet, there are many options for this:

Google Maps (and Google Fusion Tables)
Mapbox (and the related LeafletJS)
CartoDB
D3

It really comes down to exactly what you want to do and how much control you want to have over the map.  Additionally you should consider what sort of interactivity you need.

Google Maps is probably the most user friendly for very basic map functions but is somewhat limited in what you can do stylistically.

Mapbox and CartoDB are both user friendly and offer good options for styling and displaying different varieties of data.  However, they also both tend to require subscription fees for anything other than small maps. Also, the last time I checked, CartoDB explicitly handles animation and time-series data where Mapbox does not.

D3 will probably give you the most control over display, animation, and interactivity but also has a long learning curve. Even if you don't want the map to be available on the internet, this is still a very good tool for making interactive visualizations that run in the browser.

If you don't care as much about the visualizations being online, you can get a lot of work done in open GIS software like QGIS or GrassGIS, though I don't know if user interactivity is really an option there.

As I said though, it really comes down to the specifics of exactly what you're trying to do and how comfortable you are with various aspects of mapping and coding.


Want to Add More Sample in Majority Class I want to add more sample in majority class (not in minority class using oversampling techniques), how to do this?

let suppose I have a car data set having 100 total samples, with class distribution A= 40 sample, B= 25 sample, C= 20 sample, D= 15 Sample, now my question is I want to add sample in A class, so that A= 200, ............. and left the remaining class unchanged

Any oversampling technique should work just fine.

For example, SMOTE is a technique for generating samples for a specific class. It works by selecting a class, finding the closest samples and generating one at a random position between them. The algorithm doesn't explicitly require those samples be from the minority class. Theoretically, you can apply this principle to the majority class as well.

You can see more details about oversampling techniques (including SMOTE) from this jupyter notebook, which implements logic behind manually.


Mxnet deepdog (hot dog not hot dog) example - how does the network know it is classifying a hotdog Looking at the mxnet documentation.
It takes the pretrained squeenext1_1 weights, and sets imagenet_hotdog_index variable to 713.
    net = models.squeezenet1_1(pretrained=True, prefix='deep_dog_', ctx=contexts)
# hot dog happens to be a class in imagenet.
# we can reuse the weight for that class for better performance
# here's the index for that class for later use
imagenet_hotdog_index = 713

Then they set a 2 class output layer on top of it
    deep_dog_net = models.squeezenet1_1(prefix='deep_dog_', classes=2)
deep_dog_net.collect_params().initialize(ctx=contexts)
deep_dog_net.features = net.features
print(deep_dog_net)

Where I get confused is in the classify_hotog function - it is applying softmax to the output layer, and then returning the highest result index. This would make perfect sense if we had somehow told the network to compare against index 713. But it is called for prediction before the index variable is being reused? How does the network know the class to compare against is index 713/hot dog? We've basically taken squeezenet and reduced it down to 2 class output. But how does the network know what class to compare it to? Why would it give a high probability to the second argument/class when showing a hotdog - seems to me it shouldn't know what class it is comparing it to?
out = mx.nd.SoftmaxActivation(net(image.as_in_context(contexts[0])))
print('Probabilities are: '+str(out[0].asnumpy()))
result = np.argmax(out.asnumpy())

I would have expected it to maybe use the full squeezenet output layer, and set something like
if np.argmax(out.asnumpy()) == 713:
     "Hot Dog!'
Else:
     "Not hot dog!"

Never understood this and would appreciate if anyone could help me get this detail.

You are on the right track. But you would have to realize that 713 is only, effectively, being used as an index lookup to get the weights just for that class and "seed" your new model. From there, model becomes a binary classification model where 0 is nothotdog and 1 is hotdog. Once the weights are picked up, 713 shouldn't be found anywhere else in your model.

So the output, initially, is going to be something [0.493, 0.507], each corresponding to the probability that the image belongs to one of the two classes. From there, the call with argmax() is looking for which probability is highest, according to the binary selection. So it would pick 0.507 and call it a hotdog since that's corresponds to the binary value of 1.

If you want to verify me on all this, go to your output code and do something like print(out.asnumpy()) and you'll see what I mean.


How to setup and run Conda on Google Colab I am interested in using Google Colab for data modeling. How do I install conda, create an environment and run python in a notebook? I did some searching and found some helpful hints, but had several issues with this. I can only get a partially functional environment so far.
I get stuck in running another cell in the same environment. Seems that switching cells resets the environment back to default.

Found a way to get Miniconda working in Google colab. For now, use source activate, not conda activate in the 2nd cell. Still working out the bugs with using conda to activate.
Full Notebook demo here:
https://donaldsrepo.github.io/Notebooks/GoogleColabCondaCreateEnv.html
github with demo notebook:
https://github.com/donaldsrepo/SampleNotebooks/blob/master/CondaCreateEnv/GoogleColabCondaCreateEnv.ipynb
Google uses Python 3.6.9 by default. I created an environment using Python 3.6.10.
Steps to follow:

Install miniconda
Create a new environment
Run some python code to test the environment is being used

Some sample code:
# try to get the bare minimum to get a new conda env working
conda_path = ''
try:
    conda_path = !which conda
finally:
    print('')

if (len(conda_path) == 0):
    print('installing miniconda')
    !wget https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh && bash Miniconda3-4.5.4-Linux-x86_64.sh -bfp /usr/local
    !conda update conda -y -q
    !source /usr/local/etc/profile.d/conda.sh
    !conda init
    !conda install -n root _license -y -q
else:
    print('found miniconda')

conda_envs = !conda env list
res = [i for i in conda_envs if 'test36' in i]
if (len(res) == 0):
    print('not found test36 env', len(res))
    !conda create -y -q --name test36 python=3.6
else:
    print('found test36 env', len(res))

Next cell:
%%bash
source activate test36

python
import sys
# maybe only need this the first time we run this notebook
sys.path.append('/usr/local/lib/python3.6/site-packages')

print("Python version")
print(sys.version)

Output:
Python version
3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0]

Note that version 3.6.10 is the one installed in my personal conda environment. You can also see your new environment here:

A few things to point out:

If you open a new notebook, your new environment does not exist. Seems to be stored in a transient location, similar to docker.


The meaning of multi-class classification rules The meaning of multi-class classification rules

Example: I have two classification rules (Refund is a predictor and Cheat is a binary response):
(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57
(Refund, No) → (Cheat, Yes) Support = 0.3, Confidence = 0.43

=> multi-class classification rules:

(Refund, No) → (Cheat, No) v (Cheat, Yes)

When predicted classification for test data, (Cheat, No) will be selected priority so why we need to have (Cheat, Yes) in multi-class classification rules here?

(Cheat, No) will be selected (cheat will be classified as No) for the rule (Refund, No).

In a binary response variable like cheat, all the information can be inferred from just the first rule:

(Refund, No) → (Cheat, No) Support = 0.4, Confidence = 0.57

The other rule is redundant.

However, in the case of a multi-class response variable, we would like to have all the rules written out so we exactly know the likelihood of the rule implying each of the different classes in the response variable. To keep things consistent, this is also done for the case when the response variable is binary.


Keras: How to connect a CNN model with a decision tree I want to train a model to predict one's emotion from the physical signals. I have a physical signal and using it as input feature;

  ecg(Electrocardiography)

I want to use the CNN architecture to extract features from the data, and then use these extracted features to feed a classical "Decision Tree Classifier". Below, you can see my CNN aproach without the decision tree;

model = Sequential()
model.add(Conv1D(15,60,padding='valid', activation='relu',input_shape=(18000,1), strides = 1,  kernel_regularizer=regularizers.l1_l2(l1=0.1, l2=0.1)))
model.add(MaxPooling1D(2,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Conv1D(30, 60, padding='valid', activation='relu',kernel_regularizer = regularizers.l1_l2(l1=0.1, l2=0.1), strides=1))
model.add(MaxPooling1D(4,data_format='channels_last'))
model.add(Dropout(0.6))
model.add(BatchNormalization())
model.add(Flatten())
model.add(Dense(3, activation = 'softmax'))

I want to edit this code so that, in the output layer there will be working decision tree instead of model.add(Dense(3, activation = 'softmax')). I have tried to save the outputs of the last convolutional layer like this;

output = model.layers[-6].output

And when I printed out the output variable, result was this;

  THE OUTPUT:  Tensor("conv1d_56/Relu:0", shape=(?, 8971, 30),
  dtype=float32)

I guess, the output variable holds the extracted features. Now, how can I feed my decision tree classifier model  with this data which is stored in the output variable? Here is the decision tree from scikit learn;

from sklearn.tree import DecisionTreeClassifier

dtc = DecisionTreeClassifier(criterion = 'entropy')
dtc.fit()

How should I feed the fit() method? Thanks in advance.

You can make the feature extraction to an intermediate model in keras. It could look something like this:

# Build a model for only the feature extraction layers
feature_extractor = Sequential()
feature_extractor.add(Conv1D(15,60,padding='valid', activation='relu',input_shape=(18000,1), strides = 1,  kernel_regularizer=regularizers.l1_l2(l1=0.1, l2=0.1)))
feature_extractor.add(MaxPooling1D(2,data_format='channels_last'))
feature_extractor.add(Dropout(0.6))
feature_extractor.add(BatchNormalization())
feature_extractor.add(Conv1D(30, 60, padding='valid', activation='relu',kernel_regularizer = regularizers.l1_l2(l1=0.1, l2=0.1), strides=1))

# Keep adding new layers for prediciton outside of feature extraction model
x = feature_extractor.output
x = MaxPooling1D(4,data_format='channels_last')(x)
x = Dropout(0.6)(x)
x = BatchNormalization()(x)
x = Flatten()(x)
prediction_layer = Dense(3, activation = 'softmax')(x)

# Make a new model combining both
cnn_model=Model(inputs=feature_extractor.input, outputs=prediction_layer)
cnn_model.compile(optimizer=opt,loss=loss)

Then you train using the full cnn_model but you call predict only on your feature extraction model with feature_extractor.predict(X). Then you can use the output of the prediction to train your decision tree like this:

# Train full network, both feature extractor and softmax part
cnn_model.fit(X, y_one_hot) # y needs to be one hot for keras

# Predict only the output of the feature extraction model
X_ext = feature_extractor.predict(X)

dtc = DecisionTreeClassifier(criterion = 'entropy')

# Train the decision tree on the extracted features
dtc.fit(X_ext, y) # y should be one-dimensional for sklearn