How to compare and find common values from different columns in same dataframe? I would like to compare two columns and find common value sets in each column, then output the rows with the common values.

Let's say I have a dataframe with:

no.(col1) | Username (col2) | Referral(col3) | email(col4)

0 | john | mike | email0@email.com

1 | peter | paul | email1@email.com

2 | joan | patricia | email2@email.com

3 | mike | john | email3@email.com

The output would be "0 | john | mike | email0@email.com" and "3 | mike | john | email3@email.com" because they have the same values in col2 and col3 respectively.

let's consider we have a data frame named df, then one approach might be saving these two columns in different dataframes and then trying to compare them and find out their similarities, hence we would have:

column1 = df.iloc[:,1].values
column2 = df.iloc[:,2].values

Then, saving all the indices in column1 where the set exists in column2

equal_indices = []
for i in range(len(column1)):
    for j in range(len(column2[or column1, since they are equal])):
        if column1[i] == column2[j] and column2[i]==column1[j]:
            equal_indices.append(i)
            print(i,j)
            print(column1[i], column2[i])

Now equal_indices contains all the indices you want. Then you can delete the rows with similar columns from column1 or column2. or just return the found indices of column1 from the dataframe df:

df.iloc[[equal_indices]]


Get data from intermediate layers in a Pytorch model I was trying to implement SRGAN in PyTorch and I have to write a Content loss function that required me to fetch activations from intermediate layers for both the Generated Image & Original Image.
I'm using pretrained VGG-19 and according to the paper I need the ReLU activations
Can anybody guide me on how can I achieve this?

You can save the activations for intermediate layers into a variable within your forward method and then return them in addition to the output value.


Keras autoencoder not converging Could someone please explain to me why the autoencoder is not converging? To me the results of the two networks below should be the same. However, the autoencoder below is not converging, whereas, the network beneath it is.

autoencoder implementation, does not converge

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')])
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder,
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)

autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)

non-autoencoder implementation, converges

model = Sequential()
model.add(Dense(32,16,activation='tanh'))
model.add(Dense(16,32))
model.compile(loss='mean_squared_error', optimizer=rms)

model.fit(trainData,trainData, nb_epoch=numEpochs, batch_size=batch_size,
            validation_data=(testData, testData), show_accuracy=False)

The new version (0.3.0) of Keras no longer has tied weights in AutoEncoder, and it still shows different convergence. This is because weights are initialized differently.

In the non-AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32). In the AE example, Dense(32,16) weights are initialized first, followed by Dense(16,32), and then when you create the AutoEncoder instance, Dense(32,16) weights are initialized again (self.encoder.set_previous(node) will call build() to initialize weights).

Now the following two NNs converge exactly the same:

autoencoder = Sequential()
encoder = containers.Sequential([Dense(32,16,activation='tanh')])
decoder = containers.Sequential([Dense(16,32)])
autoencoder.add(AutoEncoder(encoder=encoder, decoder=decoder,
                        output_reconstruction=True))
rms = RMSprop()
autoencoder.compile(loss='mean_squared_error', optimizer=rms)
np.random.seed(0)
autoencoder.fit(trainData,trainData, nb_epoch=20, batch_size=64,
            validation_data=(testData, testData), show_accuracy=False)


Does a post-doc in Mathematics help getting a Data Science job? I am a graduate student in Mathematics, and I have been recently offered a post-doc at a good university. While it is not a top school in the most strict sense of the word, it is one of the best positions in my field of research.

However, I have recently started wondering whether academia is really the right choice for me, and I am considering learning data science with the intention of transitioning towards industry.

The question is, assuming I was to take that step and decide to start studying towards leaving academia, would it be better for me to take the post-doc, or perhaps stay one more year in graduate school?

The obvious advantage of taking the post-doc is that it would give me ample time of financial stability to learn new things. However, I am mostly interested in an answer from the point of view of starting a career in data science: is a post-doc in Mathematics something that would be considered a worthwhile addition to one's resume, or does that perhaps make one look a little bit too "academic" and actually impede one's ability to get an industry job?

The advantage of staying for one more year in graduate school would be perhaps be that since it is a top American program, I would have access to plenty of resources that I am not sure would be available at a good European university. Moreover, it seems to me that there are more good opporunities in data science in the US rather than abroad. Thus, if my goal was to get a job here, I wonder if it wouldn't be easier to just not leave the US in the first place..? I am not a US citizen, and so work authorization is something I have to consider.

There was perhaps a time where a math Phd would be an immediate ticket to a data science position but I think that phase has come and gone. The reason? It has a simple answer: Tensorflow.

Tensorflow is essentially all of Google's Phds getting together, writing out the formulas for a large number of data science models and, in the process, that takes a lot of the math out of data science. And this makes sense since most math in data science is "write one code" anyways.

In a similar vein, DataRobot is effectively the best Kaggle competitors in the world getting together and also taking a lot of the math out of data science.

So, I hate to be the bearer of bad news, but the math Phd just isn't attractive to most data science teams.


Record Linkage problem I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip.

We create pairs between data sets  and calculate cosine similarity and then pass cosine values for all features pairs to Gaussian Mixture model.We started with 2 cluster, with expectation of one match cluster and one no match cluster.But ML does not build one match cluster and  matches are in both the clusters.

Before passing to ML, i use Standard scaler and minmax scaler , but still don't get a clear nomatch and match cluster.If we increase the cluster same thing happens.

Match could be High cosine similarity  in Name,Address,State,City & zip or Name ,address ,zip or any other combinations.We are dealing with huge volume , so we are using Spark ML.

How can we achieve optimal clustering?

Clustering is the wrong tool for this. And so is cosine similarity, bag of words model etc.

Obviously, standard scaling on house numbers and zip codes is a svery bad idea, don't you think?

There are special techniques developed for this kind of data with a reason. For example, it's unlikely that the customer entered house number 100 but lives in 5. That single difference is a very clear indication that it probably is not the same customer. On the other hand, "Park Rd." And "Parkrd." almost certainly is the same, but in cosine similarity that is three tokens difference, pretty big.

Clustering is an explorative technique, not one to achieve automatic matching. Choose other tools instead.


Correcting log-bias in the output of an XGB I have previously worked with GAMs, where I was trying to do regression on a log-transformed variable. The log-transformation introduced a negative bias in the average of the predicted variable, and I corrected for this by multiplying each of the predictions $\exp(\hat y)$ by the factor

$$\langle \exp( \delta \hat y) \rangle$$

where $\delta \hat y$ was the residuals from the GAM.

Now I am using XGB, and trying to do regression on a log-transformed variable $y$ once again. The predictions $\hat y$ satisfy

$$
\frac{\sum_i \hat y_i}{\sum_i y_i} = 0.999
$$

so overall it looks good. However, when I exp-transform the variables I get

$$
\frac{\sum_i \exp(\hat y_i)}{\sum_i \exp(y_i)} = 0.861
$$

which is considerably worse. I suspect that this is due to the negative bias. Is there a way to correct for the bias in the XGB as in a GAM/GLM?

From this perspective, modern ML regression techniques are not different to classic ones. A mean unbiased model on the log scale will automatically be mean biased after retransformation (by Jensen's inequality).

Depending on the aim of your analysis, applying some retransformation bias correction factor might be useful as well for algorithms like XGBoost. (Be careful with models that impose L1/L2 penalties on the intercept though - a rare occasion.)

The specific correction factor can be calculated directly from the residuals in the same way as for OLS.


"Understanding Machine Learning: From Theory to Algorithms" the universal approximation theorem I'm readying on "Understanding Machine Learning: From Theory to Algorithms" the Universal approximation theorem:

  ..."Networks are universal approximators. That is, for every fixed precision parameter, $\epsilon >0 $, and every Lipschitz function $f : [−1; 1]^{n} \rightarrow [−1; 1]$, it is possible to
  construct a network such that for every input $\textbf{x} \in [−1; 1]^{n}$, the network outputs
  a number between $f(x) − \epsilon$ and $f(x) + \epsilon$".

It seems to me that the function $f : [−1; 1]^{n} \rightarrow [−1; 1]$ is Boolean.

I think that the n-dimensional unit hypercube ${\displaystyle [-1,1]^{n}}$ can be replaced with a compact set of $ \mathbb{R} ^ {n} $ but the codomain makes me puzzled.

I was expecting a function:
$f : \mathbb { R } ^ { n } \rightarrow \mathbb { R }$

Yes, it is applicable to a Boolean function and you can modify the result to compact set.

A result where the codomian is $[-1, 1]$ would cover regression and not restricting ourself to boolean.

You might or might not have a result for $f: \mathbb{R}^n \to \mathbb{R}$. A theorem states what conditions it requires in order to hold, for conditions that are not promised by a theorem, you have to check carefully whether the same conclusion can still apply. In particular, if the proof uses compactness, then you can't replace $[-1,1]$ with $\mathbb{R}$.


User based recommendation factoring in user data The question is:
what algorithms (and libraries) should i use if i want to build a recommender system with the following data in mind representation:

USER_ID  ZIP  Movie1  Movie2  Movie3
1  2483  5  0  3
2  2483  4  1  5
3  2345  3  1  5

Basically i want to factor in user data into a recommendation of a movie.

(zips can be transformed to long/lat but thats another question and out of scope now)

i was searching the internet for hours with no success. So i will be grateful if someone can point me in the right direction.

Collaborative Filtering:

Match users to people with similar tastes –recommend what they like
Commonly used in e-retail
Avoids the issue of users only being recommended more of what they already like (allows serendipity)

Example:
Your Scenario.
Methods: Euclidean Distance, Cosine Distance, Pearson Correlation Coefficient (most common).
Content-Based Recommendation:

Match users directly to products and content
Recommend based on what you have bought or viewed in the past
Commonly used for document recommendation: webpages, news articles,
blogs etc.

Examples:
Which Movie(s) might our user like?
Option1: The user selects preferences for the various features using pull-down menus etc. We match against the movies using Vector-Space methods (described next)
Methods: Vector-Space Method, K Nearest Neighbour(kNN)
Option2: The user rates a sample of the movies (explicitly or implicitly) as like/dislike. we then build a user profile model for that user using machine learning.
Method: Decision Trees


How to interpret Keras predict output? I am new in Keras and would want to apply a neural network on this dataset:
https://www.drivendata.org/competitions/57/nepal-earthquake/

I have proprocessed the dataset transforming categorical variables into numerical with pd.get_dummies pandas method. Also, target (that is 1, 2 or 3 - depending on the damage grade) is tranformed into three columns indicating the probability of being one of those values.

The NN I wrote is simple, but I don't understand what are the values that predict method returns.

# first neural network with keras tutorial
from numpy import loadtxt
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
import keras.backend as K

# define the keras model
model = Sequential()
# 12 8 3 Accuracy: 57.08
# 25 12 3 Accuracy: 56.89
model.add(Dense(25, input_dim=68, activation='relu'))
model.add(Dense(12, activation='relu'))
model.add(Dense(3, activation='softmax'))
# compile the keras model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[get_f1])
# fit the keras model on the dataset
model.fit(X, y, epochs=5, batch_size=10)

predicted = model.predict(test)

This is the output for the first individual:

array([0.09522187, 0.57914054, 0.32563758], dtype=float32)

Any idea?

PD: If you request some additional information or code, please let me know :)

Thanks a lot!!!

The output of softmax is a probability distribution, It gives the probability of each class of being correct. So, you have to find out
the array index of the max value in array.

predicted_class = np.argmax(predicted)


Can pooling ever increase accuracy in convolutional neural networks? In ConvNets, pooling is used to downsize the input volume, leading to fewer parameters, leading to computational efficiency and possibly helping with overfitting.

But can pooling ever increase the accuracy of your CNN model? I suppose this could happen if your CNN was overfitting when pooling was not used, and introducing pooling prevented overfitting, which could lead to better accuracy performance during testing. But if overfitting was not a problem to begin with, and you applied pooling, is it possible for accuracy to increase as a result of the pooling?
I don't see how this would happen since pooling is essentially making a lower-order computation, so your accuracy should always be $\leq$ un-pooled CNN?

I think this is a very interesting question. It tells me that you have a deep understanding of the basic concepts in machine learning, overfitting - underfitting, ... etc.

Let us start with making sure that we all agree that max pooling does not add any additional parameters to the network, max pooling is a well defined operation and there is no need to do any training to max pooling layers.

The main challenge in answering your question is that it is really difficult to address the effect of having max pooling as part of the network without considering other factors: the curse of dimensionality (the size of the output of the feature extraction part), the size of the network (especially the fully connected layers) and the overfitting issue.

In one of the experiment I did, I tried to answer a similar question. But first, I want to isolate the effect of using max pooling from changing the size of the network. But how to do this: I replaced the max pooling function with a custom version: instead of pooling 1-value from (4 x 4), I used a custom function that pooling (4 x 4) from (4 x 4) where all the values in the output are equal the max value in the input (4 x 4). I increased also the size of the convolution kernels  in later layers (to make sure that I am looking at a larger receptive field in later layers)

You can see that there will be many redundant information. I was able to isolate the advantage of changing representation from sparse to rich representations. Which can be beneficial (think about max pooling in sparse coding to understand how this works).

The answer from my own perspective:

generating rich representations can improve the classification accuracy.
if you do not have max pooling, the fully connected layers can be really huge.
if you do not have enough data, you will have overfitting problem.
in my case, the database I have contains ~ 12M images, since data is the best regularization technique, it was difficult for me to understand if there is any overfitting problems (which I believe it will be the case if you do not have enough images).

I totally understand that from information theory point of view, we are loosing information in max pooling, but it looks like max pooling keeps the most important information that can be used to build a strong multi-layer full connected network at the end.

please do not take my answer as an ultimate truth, I just wanted to share my concerns and thoughts about this based on one of the experiments I did before


Facial recognition architecture Image recognition uses deep learning, and in particular CNNs to train on and recognise faces. Usually, this entails training on lots of data.
However, recently, we have seen face recognition being deployed everywhere, and being used for passport control, for example, and some airlines have deployed in lieu of boarding pass scanners. How is this accomplished? How can they achieve such accuracy without having hundreds of pictures of everyone? How can, for example, governments, with a quick scan of my face, recognise me and identify my name and identity without having any picture of me besides the ones available from my official ids? There is a link here which, however, does not really explain it technically. Is it done using neural nets? What is their architecture. Are there papers describing it (technically) anywhere?

Have a look at a few networks:
-Siamese Network
-One-Shot Learning Model

Once going through these models you will understand how they can very well with very limited amounts of data.


prepare email text for nlp (sentiment analysis) I have text of emails, which also contains disclaimers, phone numbers, email addresses, file attachment names, addresses, greetings etc.

At the moment I blindly pass this text through an OOTB sentiment analyser called Vader with poor results (i.e. if I open an email marked as negative, my human understanding does not confirm the sentiment - looking at the core English text).

I could use regex etc. to strip out email addresses, file names etc. but other text components (e.g. addresses, disclaimers) are harder to remove. Btw, disclaimers are often negative ...

Anyway, I wonder if anyone is aware of text prep methods for my scenario - to extract core human text. Google searches were moderately successful. Thanks!

I have done something similar in the past.  I'll sketch an outline for you.

First you break the text into paragraphs and tokenize.  Then write some regexp rules to capture the data you want to remove.  For instance, if an email signature commonly contains a phone number, paragraph, and a website, you can count those features and flag it based on some threshold you decide.

Next, do likewise with the other features you mentioned.  My experience is it's highly domain dependent so you really need to look at the data and use your best judgement.

The result of this process should be a data consisting of tokenized paragraphs where the paragraph has been labeled 'noise' or 'clean' based on the feature count.

From there, convert your token representation using tf-idf or another type of embedding.  You should be able to use this as input to your favorite classifier, and I have had success using SVMs to that end.

The result is going to be biased towards your rules but you are also leveraging features that are in the labeled examples that are not explicitly in the rules.  Particularly so for longer paragraphs.

It might seem a bit a bit janky but believe it or not it works.


How to find similar meaning log messages in the source code? Currently we have 10 thoundands log messages in the source codes which is maintained by different groups.
Some messages may be similar, but the sentence is not the same. My purpose is to create a centralized file which contains the unified message for reference.
For example:

printf("Failed to open the file %s", file)
printf("%s: failed to open file %s", TIMESTAMP, file)
printf("Failed to create the file %s")

I can extract all the message body which is in bold.They are not strict English sentence. but manually categorizing them takes too much time.
Is there a way to automatically categorizing the logs?
I just want to reduce the manual work.

A simple idea, non-supervision:
 1.Stemming, reduceing punctuations. Then you can use Bag of word with TF-IDF to represent the log sentences.

Run K-means to cluster your sentences. If your sentences are belong to
5 type. You can set K=5.

Of course, you can use Regular Express to classify those sentences.


Problem encountered while updating a particular cell of a pandas data frame in Python I am trying to read in an excel file into a pandas data frame using pandas.read_excel() function. The first column is 'Date'. The first entry into that column has asterisk mark which I want to remove. Here are the first 2 rows of the data frame DF.
0   1/4/1996*   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN
1   1996-01-11 00:00:00 13881   11735   2532    4188    32064   21989   3819    2586    0   NaN NaN`

In order to remove that *, I am doing this:DF['Date'][0] = '1/4/1996'. However, once I do that, I am getting a warning message
/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  app.launch_new_instance()

and it changes the cell value, but, it also attaches another 'Date' column at the end of the df like this:
0   1/4/1996   11540   10841   3592    1827    26476   14510   6900    0   0   NaN NaN NaN NaN NaN NaN NaN NaN NaN 1/4/1996

I don't understand why this is happening.
I would greatly appreciate it if someone can explain this or point out the correct way of changing the cell value.

Use a combination of string processing and type casting to process the whole column in one swoop:

  excelDF[datecol] = pandas.to_datetime(excelDF[datecol].str.replace('*',''))

I tried using str.rstrip('*') but I could not get it to work.


Is it possible to detect a person using facial recognition on the fly? I am working on a small app for face detection in Python using face_recognition and opencv libraries.

I have trained the data on a number of people and used face recognition to login into the app.

If it detects the person from the picture, it allows him to log in, otherwise they are sent to the registration screen where the person has to upload his photos and other information.

So if a new registration comes in, I have to run the model again with additional data.

As the number of entries increases the model will take more time, affecting the time to identify a newly registered person.

How can I prepare for this slow-down, down the road?

Can I reduce the time taken to train the model again or train the model with incremental data?

If you need to re-train the model to classify new faces, this will not scale well to registering new people routinely. You may also suffer from glitches in accuracy during new registrations unless the training routines are carefully monitored.

Instead, recognition systems that need to register new items typically don't re-train. They are trained on the general task of separating objects - identifying whether they are different or what is different about them.

One common way to achieve this separation is to use the NN to map images of faces to a descriptive vector, and match each new image according to distance to stored vectors of profile pictures. The distance, even to the closest stored vector, should be small in order to consider it a successful match. Registering a new user is then a matter of saving a new vector embedding calculated from the neural network. Matches can be done with database lookups - they are still limited by how fast you can do the distance calculations looking for a match, but you only need run the NN forward once. The distance calculations can be fast and batched - it should be easy to use e.g. tensorflow or torch to calculate 1000 potential matches in a fraction of a second.

To ensure that faces are well categorised according to their differences, ignoring lighting, pose, hairstyle etc, the network needs to be trained with this objective in mind. One way to do this is to train with triplet loss using two pairs of images, one that should match, another that should not.

Andrew Ng's course on CNNs covers this approach very well, starting from this lecture.


Learning a simple sequence with RNN (Keras) I am trying to learn a very simple sequence using an RNN (implemented in Keras)

The input sequence is randomly generated integers between 0 to 100:

x=np.random.randint(0,100, size=2000)

while the expected output value for time t is the (t-2)th input term i.e:

yt=xt-2

such that an example dataset looks like this:

+------+------+
|  X   |  Y   |
+------+------+
|    0 |   NA |
|   24 |   NA |
|   33 |    0 |
|    6 |   24 |
|   78 |   33 |
|   11 |    6 |
|    . |    . |
|    . |    . |
+------+------+

Note: I drop the NA rows before training.

I am trying to train a simple RNN to learn this sequence as below:

xtrain=np.reshape(df['X'], (df.shape[0], 1, 1))
#to match dimension of input shape for SimpleRNN layer.

model=Sequential()
model.SimpleRNN(2, input_shape=(None,1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(x=xtrain, y=df['Y'], epochs=200, batch_size=5)

however, I find that this implementation results in a local minima which predicts constant value(~50) for all test observations.

Could anyone help me with the right way of implementing a basic RNN in Keras to learn this sequence?

Using the raw integers as inputs and targets will make this a very difficult task. A better approach would be to come up with a vector for each number. You can simply encode each number directly as a vector, use a "onehot" representation, or use an Embedding layer. You can see an example of embeddings in conx that is built on Keras here:

http://conx.readthedocs.io/en/latest/Predicting%20and%20Generating%20Texts.html?highlight=embeddinglayer

and and example without embeddings using onehot representations here:

http://conx.readthedocs.io/en/latest/AliceInWonderland.html


Is web analytics similar to data science? I just finished PhD and initially wished to work on data science and deep learning. However, after some rounds of interviews, I have been offered a job of web analytics and business intelligence at a medium size company. Is there any similarity with data science, and is there a future in it?

Because of some precarious situation, I have to accept this job, but should I keep looking for another job meanwhile, or will the experience be helpful to rise up the ladder or join a big brand?

We can see Web Analytics like an application of Data Science concepts. Almost a field of it. Usually it's very close to Business Intelligence for marketing aims and the main goal is to analyse users information.
So it's about marketing.


Is there an alternative to Google Maps to find GPS coordinates of a place in Germany? For probably any place on Earth, you can search it in Google Maps, make right-click, pick the option "What's here" and you get GPS coordinates.

If I don't want to use Google Maps API, is there an alternative open API to get GPS coordinates by address? To be less broad, say in Germany?

There are a number of APIs for OpenStreetmap, e.g. via html request:

https://nominatim.openstreetmap.org/search?q=pleikartsförsterstrasse,%20heidelberg&format=json&polygon=1&addressdetails=1

A json is returned. You can easily access the type of information you need.


R Combine Multiple Rows of DataFrame by creating new columns and union values I have a dataframe in R that looks like this

 ID    APPROVAL_STEP   APPROVAL_STATUS   APPROVAL_DATE     APPROVER
1234      STEP_A          APPROVED        23-Jan-2019     John Smith
1234      STEP_B          APPROVED        21-Jan-2019      Jane Doe

I need it to look like this

 ID    STEP_A_STATUS  STEP_A_APPROVAL_DATE  STEP_A_APPROVER  STEP_B_STATUS  STEP_B_APPROVAL_DATE  STEP_B_APPROVER
1234     APPROVED         23-Jan-2019         John Smith       APPROVED        21-Jan-2019            Jane Doe

And of course, with the original dataframe, any of APPROVAL_STATUS, APPROVAL_DATE, or APPROVER can be NA.

What is the most elegant way to do this? I know how to do it by looping through the unique IDs, grabbing each row, creating new columns, etc.; but is there any way to do this in a more elegant way (e.g., using tidyverse)?

What you are asking is how to convert data from "long" form to "wide" form.

You can just use the reshape() function which is in the stats package.

#' I constructed your data frame here
x<-data.frame(ID=c(1234,1234),
          APPROVAL_STEP=c("STEP_A","STEP_B"),
          APPROVAL_STATUS=c("APPROVED","APPROVED"),
          APPROVAL_DATE=c("23-Jan-2019","21-Jan-2019"),
          APPROVER=c("John Smith","Jane Doe"))

#' Now to reshape the data

library(stats)
reshape(x,
    timevar="APPROVAL_STEP",
    idvar="ID",
    sep="_", direction = "wide")

Your output should look like.

  ID APPROVAL_STATUS_STEP_A APPROVAL_DATE_STEP_A APPROVER_STEP_A APPROVAL_STATUS_STEP_B APPROVAL_DATE_STEP_B
1 1234               APPROVED          23-Jan-2019      John Smith               APPROVED          21-Jan-2019
  APPROVER_STEP_B
1        Jane Doe

The column names is a cosmetic issue, but that's the gist of it.

Hope this is helpful!


Speech Dataset for Spanish ASR We are building ASR system for native spanish language. We are looking for the spanish speech datasets to train our model. If anyone can provide the links/sources of spanish speech datasets(paid/free) it would be very useful.

Have a look at the Common Voice project from Mozilla. They have an open source dataset of multiple languages, Spanish currently has 324 hours of validated speech data (out of 579 total). You can download the dataset for Spanish and other languages at https://commonvoice.mozilla.org/en/datasets.


what is the loss function in char recognition using Tensorflow? I have code in Tensorflow using convolution neural network to recognize the characters in street view Text (SVT) data.

Since the label type is string, what should I use instead of tf.nn.sparse_softmax_cross_entropy_with_logits() in the loss function?

I cannot use tf.nn.sparse_softmax_cross_entropy_with_logits() because the labels her must be an int dtype??

The loss function is correct, you just need to convert categorical variables into numerical representations using one-hot vector encoding. Please take a look at this.


Will unnecessary features harm the tree based model? Is it necessary to drop noisy features (eg column of random numbers) from tree features? I think it's not. sometimes it may benefit but will never cause any harm to the model. Because at each split model is checking which feature will reduce the impurity. Sometimes by chance random numbers may be the one.

This is not a direct answer to your question, but more like an experiment. I created a simple script in Python where I ran multiple times the Iris dataset with the regular columns and also with 4 extra columns with random numbers. Then I store the difference of the accuracy between the two models and plot the distribution of it. If you try it yourself as well, you will see that there are cases that the "clean" dataset has better accuracy, even if the majority of the results are exactly the same.

from sklearn.datasets import load_iris
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import numpy as np
import warnings
import seaborn as sns
import random as random

warnings.filterwarnings("ignore")

iris = load_iris()

df = pd.DataFrame(data= np.c_[iris['data'], iris['target']], columns= iris['feature_names'] + ['target'])

df['rand1'] = np.random.randint(0, 2, df.shape[0])
df['rand2'] = np.random.randint(0, 2, df.shape[0])
df['rand3'] = np.random.randint(0, 2, df.shape[0])
df['rand4'] = np.random.randint(0, 2, df.shape[0])

all_inputs = df[iris['feature_names']].values
all_inputs_with_random = df[iris['feature_names']+['rand1', 'rand2', 'rand3','rand4']].values
all_classes = df['target'].values

dif = []

for i in range(100):
    a = random.randint(0,1000)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs, all_classes, train_size=0.7, random_state = a)

    dtc1 = DecisionTreeClassifier()
    dtc1.fit(train_inputs, train_classes)

    a1 = dtc1.score(test_inputs, test_classes)

    (train_inputs, test_inputs, train_classes, test_classes) = train_test_split(all_inputs_with_random, all_classes, train_size=0.7, random_state = a)

    dtc2 = DecisionTreeClassifier()
    dtc2.fit(train_inputs, train_classes)

    a2 = dtc2.score(test_inputs, test_classes)

    dif.append(a1-a2)

sns.distplot(dif)


Include time as a variable in regression model I am currently working on a regression problem which requires me to predict the costs of a fixed asset. I have used several variables to do so and derived a predicted cost. However, my superior has wanted me to incorporate time as a variable in the regression model which I am at a loss on how to do so. My data set looks something like this which does not resemble a time series dataset.

Name    Capacity    OEM      Country   Date of valuation  MONTH   YEAR   Cost
A1       220        JAPAN    JAPAN     1/1/2012            1      2012   300,000,000
A2       220        JAPAN    JAPAN     1/1/2012            1      2012   300000000
B1       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B2       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B3       400        CHINA    CHINA     1/3/2013            3      2013   475000000
B4       400        CHINA    CHINA     1/3/2013            3      2013   475000000
C1       750        INDIA    USA       1/5/2016            5      2016   268000000
C2       750        INDIA    USA       1/5/2016            5      2016   268000000

The variables that I have used are capacity, OEM and country. Any help on how to incorporate time to my regression problem is welcomed.

Theres an approach I'd take which consists of two steps. The second is optional buy highly recommended.

create variables for day, month, year separately.
process those numbers as cyclic features (day 31 is as close to day 1 as to day 30, for example. If you don't do anything about that, and store days from 1 to 31, there's no place considering the cyclic nature of the feature). This post exaplins this in detail.

Hope this helps.

EDIT1:
Also, not asked, but depending on the algorithm you're using, you may want to normalize the numerical features you have.


How to use ontologies for text classification? I am new to machine learning and want to classify sentences using ontologies (taxonomies/ knowledge graphs) and supervised learning methods (I have an annotated training dataset).

My question is how to use the ontology for this task? Is the following method correct?

I will first perform the tokenization, stemming and stop word removal (pre-processing). Then, I will search for each term in the ontology and after finding them, I will add their related hierarchy to an array or vector for each document. Then I will train the classifiers (supervised) on those vectors.

Please let me know if this method is correct or if there are steps that I am missing here.

Thanks! :)

Since no one answered my question, I'll describe what I found here for others.

Similar to what I described above in the question, you go over pre-processing, then look for the terms in the ontology and replace them with the whole branches from the ontology. After that, you train your model.

Notes:

Since you have the ontology with all the relevant terms, you can leave the stopwords in while pre-processing. Stopword removal can be performed via lists or using sklearn or Spacy. In addtion, Spacy performs lemmatization, which can be done via ontologies too (more accurate, but probably slower).
Use SPARQL for working with ontologies and pattern matching. Spacy does patter matching too; also, Lucene and several other tools. However, I haven't used them with ontologies. Similarity algorithms can also be used.
I found Random Forests to work best among traditional supervised methods. However, probably using neural networks might produce better results. Please keep in mind you can use alternative methods for text classification without ontologies, such as word embedding and topic models. Spacy seems to be a good tool for text classification too. It has nice documentation.


How to read file from user in Shiny and assign it to a variable in global.r? I want to read a csv file as input from user in Shiny and assign it to a variable in global.r file.The code I have in ui.R is

fileInput('file1', 'Choose CSV File',
                     accept=c('text/csv', 'text/comma-separated-values,text/plain', .csv'))

The code I have in main Panel of server.R is

textOutput('contents')

The code I have currently in server.R is

output$contents <- renderText({
      if(is.null(input$file1))return()
      inFile <- input$file1
      data<-read.csv(inFile$datapath)
  print(summary(data))

 })

I want to assign this input$file1 to a variable called 'data' in global.r. Please let me know if this is possible.
Thanks

I have added the following code to global.R file

data <- reactiveValues()

I used assign function in server.R to assign values to data in global.r

output$contents <- renderText({
      if(is.null(input$file1))return()
      inFile <- input$file1
      data2<-read.csv(inFile$datapath)
  assign('data',data2,envir=.GlobalEnv)
  print(summary(data))
 })


Should reinforcement learning always assume (PO)MDP? I recently just started learning reinforcement learning and learned that reinforcement learning algorithms work under the assumption of MDP or POMDP.
However as I read A3C and recent vision based deep RL papers, it seems some of them aren't assuming MDPs but used RNNs or LSTM to make it seem as if it is MDP.

So my question is: how does reinforcement learning algorithms work without the assumption of (PO)MDPs?

How does reinforcement learning algorithms work without the assumption of (PO)MDPs?

It doesn't. The theory of reinforcement learning is tied very strongly to an underlying MDP framework. The RNN-based solutions that you are referring to are fully compatible with such an MDP model, and don't even require a POMPDP to be useful.

Without the core guarantees of a (PO)MDP model, or something closely equivalent, it is not clear that any learning could occur, with any kind of agent. The MDP model of an environment is about describing consistent behaviour, that is predictable to some degree, and random/stochastic otherwise, and the predictable parts make it amenable to at least some optimisation. The split into states, actions, time steps and rewards help organise the thinking around this. They are not necessary for other kinds of policy-search approaches, such as genetic algorithms. However, if you try to break away from something that would fit to a (PO)MDP, it would break any other kind of meaningful policy too:

If actions had no consequence, then you could learn the value of being in a particular state, but you could not optimise an agent. This could be modelled as a Markov Reward Process, provided state transitions were not completely random, otherwise just learning associations of state to reward using a supervised learning approach would be the best you could do.
If rewards were not consistently based on any data available to the agent, not even history, but not random, then there is no way to learn how to predict or optimise rewards.
Similarly for state transitions, if they bear no relation to any information known about the environment, current state or history, but are not random, then there is no way to learn about the non-randomness, and no kind of agent could generate a meaningful policy to take advantage of knowledge about the system, because the knowledge available is not relevant. However, if the current state still influenced what rewards were available to which actions, then a contextual bandit approach might work (plus a supervised learning approach could predict currently available rewards).

When the information about rewards or state transitions is not directly available, but can be inferred or guessed at least partially from history or context, then you can model this as a POMDP.

One common scenario you can face is that you have available some observations about the environment, but are not sure how to construct a state description that has the Markov property. The velocity of an object might be such a detail, when your observations only give you positions. Technically a POMDP and this observation/state mismatch are the same basic issue, if you decided arbitrarily that your observation was the state.

When faced with this mismatch between easily-available observations and a state description based on history that would be more useful, you can either try to engineer useful features, or you can turn to learning models that will infer them. That is where using RNNs can come in useful as part of RL, and they can help with both observation to state mapping and also inferring more complex hidden state variables in POMDPs. Use of hidden markov models to model a "belief state" that augments the observed state is similar to the latter use of RNNs.


Spectral clustering result interpretation I cluster a toy data set into three groups with spectral clustering. Please see the code below. affinity='rbf' by default.

from sklearn.cluster import SpectralClustering
import numpy as np

points = np.array([[5,6], [3,3], [2,2], [7,3], [8,3], [5,4], [5,5], [9,3], [7,1]])

spectral = SpectralClustering(n_clusters=3)
labels = spectral.fit(points).labels_

The output is three clusters with labels [1 2 2 0 0 1 1 0 2] shown on the image. Why point 8 happens to be in the same cluster with 1 and 2, not with points 3,4, and 7?

UPDATE:

Affinity matrix:

Spectral embedding result:

Supposedly because of the weighting / normalization.

Point 8 is fairly far from all it's neighbors, so it's edges have a similar weight; and the normalization increases the edge weight to give every point the same influence. It's two other cluster members are (except they have one neighbor) similar. So I wouldn't be surprised if the two strongest edges after normalization of 8 are to 1 and 2.

To understand this better, I suggest that you compute the affinity matrix and visualize the affinities as edge strengths. The spectral embedding may also be worth looking at.

But this is an interesting example about the drawbacks of spectral clustering: normalization may give too much weight on "outliers".


Visualizing Decision Tree of K-Nearest-Neighbours classifier I'm using Sklearn's KNN to build a classifier and was wondering if there is any way to visualize the decision tree that the algorithm builds. Maybe something of this fashion

$k$-NN does not build a decision tree to classify a new instance, it looks at the class of the most similar examples (the nearest neighbours) in the training set. So, in short, no you cannot get a decision tree from $k$-NN. You can build a decision tree for your dataset directly by using scikit-learn's DecisionTreeClassifier instead if you need a decision tree.


Logistic Regression can't fit my data I'm trying fit my data , but I couldn't fit it.Data set

0.50,0
0.75,0
1.00,0
1.25,0
1.50,0
1.75,0
1.75,1
2.00,0
2.25,1
2.50,0
2.75,1
3.00,0
3.25,1
3.50,0
4.00,1
4.25,1
4.50,1
4.75,1
5.00,1
5.50,1

and my code

data = np.loadtxt('dat', delimiter=',',dtype=None);

x=data[:,0:1];
y=data[:,1].reshape(x.size/x[0].size,1);
a=np.ones(shape=(y.size,x[0].size+1));
a[:,1:2]=x;
q=np.ones(shape=(a.shape[1],1));
alpha=0.003

for i in range(500000):
    h=1/(1+np.exp(-np.dot(a,q)))
    for j in range(q.size):
        q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);
plt.axis((-1,10,-1,5))
plt.plot(x,y,'x',x,h);
plt.show();

So I tried different learning rates(alpha),tried different number of iterations but my fitting data is looking like this
but it's should looks like this enter link description here

What am I missing? Is there any logical error or something like that? Thanks for your deal.

The problem is in the following line:

q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);

(h-y) has shape (20, 1), a[:,j] has shape (20), multiplying them results in a shape (20, 20), which is wrong.

Try a[:,j:j+1] instead of a[:,j] and it'll start working:

q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j:j+1]);

This gives the following plot:

You could adjust the shapes of your variables a bit to make it easier:

x=data[:,:1]
y=data[:,1]
a=np.hstack([np.ones((len(x), 1)), x])
q=np.ones(a.shape[1])
alpha=0.003

for i in range(10000):
    h=1/(1+np.exp(-np.dot(a,q)))
    for j in range(len(q)):
        q[j]=q[j]-alpha*np.sum((h-y)*a[:,j])

plt.plot(x,y,'x',x,h)
plt.show()

Also note that in Python you don't need ; at the end of each line.


Probabilistic Outlier Detection (edited + clarified) Measured data $D \in \mathbb{R}^3$, every $d^i \in D$ is $d^i_{(x)}$, where the  $x=[x_1, x_2]$. Simply said, the measured data are function of $x$. It is known, the dependency is linear, such as:

$$d^i_1 = a_1^Tx+b_1$$
$$d^i_2 = a_2^Tx+b_2$$
$$d^i_3 = a_3^Tx+b_3$$

where $a_{1-3} \in \mathbb{R}^2$ and $b_{1-3} \in \mathbb{R}$, so that $d^i_{1-3}$ is a set of 3 simple 2D planes, over $x=[x_1, x_2]$.

While it is known, that $d^i_{(x)}$ is function of $x$, the $x$ is not measured. Therefore the parameters $a_{1-3}$ and $b_{1-3}$ are difficult to find. The data interpretation is simplified to 3 dimensions for visual clarity, real problem is of higher dimension.

Data Example

The figures below show $d^i_{1-3}$, as a function of $x=[x_1, x_2]$, assuming the $x$ is measurable. This is only to visualize the data set.

dimension 1 of $d^i$

dimension 2 of $d^i$

dimension 3 of $d^i$

Problem formulation

How to assign probability $p(d^i)$ to measured $d^i$, such that it reflects the fact that it lies in the feasibility region (the 3 green hyper-planes) ?

1) Simply said, how to train the model using $d^i = [d^i_1, d^i_2, d^i_3]$ to spot the outlier $d' = [d^q_1, d^q_2, d^k_3]$. The outlier is marked by red dot and deviates only in the 3rd dimension. (please, suggest ways that work in higher dimensions as well).

In practice, $x=[x_1, x_2]$ is not measured !

2) Does the knowledge of the linearity of $d^i_{(x)}$, help ? What if the dependency is not linear ?

Idea

I think I can have 3 simple Gaussian PDFs, where each PDF gives the probability for one {$d^i_1$, $d^i_2$, $d^i_3$}, and the parameters (mean and variance) are function of the other 2 $d^i_{1-3}$. For example:

$$p(d^i_1) = N(d^i_1|\mathrm{E}_{(d^i_2, d^i_3)},\mathrm{Var}_{(d^i_2, d^i_3)})$$

then it follows for the other 2 dimensions:

$$p(d^i_2) = N(d^i_2|\mathrm{E}_{(d^i_4, d^i_3)},\mathrm{Var}_{(d^i_4, d^i_3)})$$
$$p(d^i_3) = N(d^i_3|\mathrm{E}_{(d^i_1, d^i_2)},\mathrm{Var}_{(d^i_1, d^i_2)})$$

P.S.

The outlier $d'$ is placed at the border (y=0), just to enhance the visibility. The outlier can be measured with any value of $x=[x_1, x_2]$.

In essence anomaly detection is about finding a metric with which to measure the similarity between instances and then determining a threshold when crossed constitutes an anomaly. There are parametric anomaly detection algorithms which require some hyper-parameters to be set or information about the distribution to be known. There are also non-parametric techniques which do not have these stringent requirements.

Let's consider the simple case of identifying anomalies in a population based on height. We can assume this to be a 1D Gaussian distribution. Now we need a similarity metric, let's use Euclidean distance for simplicity (the difference in individual's heights). Next we need a threshold.

Classical statistics

Classical methods would parametrize the Gaussian distribution of your population. Let us assume that after polling our population we identify that the mean height is 170cm with a variance of 15cm. Now we can set the threshold to be any multiple of the variance $\sigma$. Typically we choose $3\sigma$ as a thresholds since the ingroup would include 99.7% of the population, all those outside of the $3\sigma$ range can thus confidently be said to be anomalous.

You can also look at the application of a generalized likelihood ratio test (GLRT) to find anomalies.

Let's visualize this

import numpy as np

data = np.random.normal(170, 15, 10000)

plt.hist(data)
plt.plot([215,215],[0,3000])
plt.plot([125,125],[0,3000])
plt.show()

Parametric techniques

The examples will require that we have prior information regarding the distribution of the data. This technique will fit an ellipse around the Gaussian distribution of our data in order to contain the elements with the highest in-group correlation.

from sklearn.covariance import EllipticEnvelope
estimator = EllipticEnvelope()
estimator.fit(data.reshape(-1,1))

plt.scatter(data, np.random.randint(0,1,10000), c=estimator.predict(data.reshape(-1,1)))
plt.show()

We see that the yellow dots are the 'normal' instances and the dark blue ones are the anomalies. You can also notice that this algorithm selects an anomaly threshold very similar to the one we obtained using classical statistics methods.

Non-parametric algorithms

This is a play on the Random Forests algorithm based on an ensemble of decision trees. The decision trees are trained to identify anomalies through information loss decisions and isolating instances of your data.

from sklearn.ensemble import IsolationForest
estimator = IsolationForest()
estimator.fit(data.reshape(-1,1))
plt.scatter(data, np.random.randint(0,1,10000), c=estimator.predict(data.reshape(-1,1)))
plt.show()

Non-parametric $p$-value estimation algorithms

These are personal favorite and have been very successful in a number of tasks.

Learning Minimum Volume Sets
http://www.stat.rice.edu/~cscott/pubs/minvol06jmlr.pdf

Anomaly Detection with Score functions based on Nearest Neighbor Graphs
https://arxiv.org/abs/0910.5461

New statistic in P-value estimation for anomaly detection
http://ieeexplore.ieee.org/document/6319713/

I don't have time tonight but tomorrow evening I will elaborate on the GLRT, and the $p$-value estimation algorithms.

Extending the above to more dimensions

Of course all the described algorithm can be extended to $n$-dimensions. This is hard to plot so we will go through a 2D example and a 3D example. However, determining the probability of these being anomalies is not so simple. For some of these algorithms you will need to devise some sort of distance metric.

2D example

Let's consider this artificial data. Where tall people above 170cm have eye colors with a blue content of mean 1 and variance 0.5. And short people have lower blue content at a mean 0 and variance 0.5 (let's just assume eyes can have negative pigmentation whatever that could possibly mean).

import numpy as np

n = 10000
X = np.zeros((n,2))
X[:,0] = np.random.normal(170, 15, n)
for i in range(n):
    if X[i,0] < 170:
        X[i,1] = np.random.normal(0, 0.5)
    else:
        X[i,1] = np.random.normal(1, 0.5)

plt.scatter(X[:,0], X[:,1])
plt.show()

Classical statistics

Any p-value test used in classical statistics can be extended to multiple dimensions. This is the easiest when the features are uncorrelated. When this is the case the probability of each feature can be multiplied together. If this is not the case then you need to compute the p-value for a 2D Gaussian distribution.

For our case this is even worse! It's bimodal (there are two means for a feature). You can see this here. Our data would look something like this image from the Wiki.

I hope you can appreciate how difficult this problem is to solve. Luckily this is no longer the classical era of statistics.

Parametric techniques

These algorithms can easily be moved onto $n$-dimensions.

from sklearn.covariance import EllipticEnvelope
estimator = EllipticEnvelope()
estimator.fit(X)
plt.scatter(X[:,0], X[:,1], c=estimator.predict(X))
plt.show()

In the 1D example we discussed how this algorithm assumes a unimodal Gaussian distribution. This is not the case for our current data and you can see the failures of this model. It seems to classify many extreme examples as being part of the nominal set when they should have likely been identified as anomalies.

Non-parametric techniques

This technique can also be applied to $n$-dimensional data. Moreover, it can be used for multimodal Gaussian distributions as you can see we get better results.

from sklearn.ensemble import IsolationForest
estimator = IsolationForest()
estimator.fit(X)
plt.scatter(X[:,0], X[:,1], c=estimator.predict(X))
plt.show()


Predict the average temperature for next 30 years Objective: I want to predict the average temperature for next 30 years.

Q1: What type of dataset is suitable for this (what columns should it contain)

Q2: What are the independent variables for average temperature?

Q3: Let's say I trained the model with 30% data. Does that mean predicted 30% is for next 30 years? Or is it present data? If not how do I predict the average temperature for next 30 years with the trained model?

Do you have context where this problem originates from? You seem to be mixing up a few things.

Q1: depends on what dataset do you have available, or can you find? It can be just year and average temperature of the year, and then you'd apply a regression or ARIMA model.
But maybe you have average temperatures for various countries and regions, and then you could predict average temperatures per region.

Q2: again, depends on what dataset and columns you have. Typical independent variables are variables whose variation does not impact the temperature. Those typically have near 0 correlation with the temperature.

Q3: depending on the dataset, you probably should not use machine learning at all. Maybe regression/ARIMA is enough. And regression to predict values you should not do on partial data.

When you train a model with 30%, you use 70% for validation or scoring the model.

Your prediction could be just the extrapolation on the fitted curves on your data set.
You also have some information on the variations (R2), that can help give probability scores.
A typical outcome could be a 95% confidence interval that the temperature will be Tx +/- dTx.


time series prediction using arima and non linear trend and too much residuals I am working on forecasting a financial index, i tried decomposing the time series using :

from matplotlib import pyplot
from statsmodels.tsa.seasonal import seasonal_decompose
result = seasonal_decompose(dataset, model='multiplicative', freq=12)
result.plot()
pyplot.show()

And i got the following result:

The results show that the time series is not stationary and it has a unit root (I used ADF and KPSS tests) and that the mean and std are constant in time!

I am wondering if i should use ARIMA or SARIMA since they are adapted to linear trend (my trend is not linear as shown in the image) or move to using LSTM, NN ... ?
Or even ARIMA or SARIMA are not adapted to this type of time series?

Long Short Term Memory (LSTM) is one option given that you have about 9 years of historical data.
You can take a model comparison approach where you split the data and see which algorithm is best at predicting the hold-out data.


ValueError: Found input variables with inconsistent numbers of samples [43,19] So, I've been trying to split my dataset into a 70-30 ratio using train_test_split in order to work things out with sklearn's PLS. However, I encountered an error that says:

# Read data
df = pd.read_csv(
    filepath_or_buffer='colontumor.csv',
    header=None,
    sep=',')

df.tail()

X = df.iloc[:, :-1].values

# Split data to train and test on 70-30 ratio
X_train, X_test = train_test_split(
    X, test_size=0.30, random_state=None)

pls = PLSRegression(n_components=2)
pls.fit(X_train, X_test)
Y_pred = pls2.predict(X_train)

and somehow encountered this error:

Line (17): pls.fit(X_train, X_test)
ValueError: Found input variables with inconsistent numbers of samples: [43, 19]

Is there any solution for this? I've been circling around for like hours now.

The line that gives the error is:

pls.fit(X_train, X_test)

The second argument in fit should be your labels, the value you want your model to be able to predict. Instead you are inputting X_test, which contains the same features as X_train but with another length, causing your error. You want an y_train with your target value and make it look like this:

pls.fit(X_train, y_train)

But to get this value you will need to get your labels from you dataframe. It is hard to know how without knowing the data, but this might give you an idea:

X = df.drop(['name of label column'], axis=1).iloc[:, :-1].values
y = df['name of label column'].iloc[:, :-1].values

# Split data to train and test on 70-30 ratio
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

pls = PLSRegression(n_components=2)
pls.fit(X_train, y_train)


Regression Trees - Splitting and decision rules I understand that a regression tree is built by splitting a node, such that the MSE for the label/output variable is minimized in each of the two resulting nodes.
I have two questions about this:

1.)
Is the search for the optimal split even dependent from the input variables?  The MSE could be minimized through exhaustive search for two subsets which  minimize the MSEs of the label in each of the subsets. No knowledge about input variables is needed for this.
If this is the case, how are the decision rules for upcoming instances, for which an output should be predicted, set?
How is it decided which feature to split at what point in order to obtain the split into the 2 subsets?

2.)
Or does the algorithm run through all possible splits (split at each value of each feature once) and then chose the one with the minimal MSEs?
This way the decision rule would be clear.

Thanks in advance

Short Answer

1.Yes feature variables are needed for split. No the best 2 subsets for mse reduction are not created.
2.Yes

Long answer
Decision trees are greedy algorithms that choose a feature and split at each node and use that feature and cut to divide the data.

So as you mentioned in point 2 the tree starts building with all y values in the first node and iterates over all combinations for each feature and selects the best feature and value split to divide into 2 groups which are further split.

The termination conditions would be number of observations in the leaf node and/or the threshold for decrease in mse from in the split.


Is this clear overfitting? The orange curve is train accuracy and blue is validation accuracy. Is this clear overfitting or should I let it run for more epochs?

With custom dataset (1D data with 70 features) I trained a 2 layer MLP.
Network Architecture: [70-200-200-4].
I'm only able to reach ~50+% accuracy. Any suggestions on what steps I can take to improve accuracy? (Obtaining more data isn't an option)

Thanks in advance!

When training accuracy increases while validation accuracy remains constant or decreases, then the model is most likely overfitting, or it may be saturated.

You can try the following methods to increase accuracy:

Momentum
Regularization techniques: Early Stopping, Dropout, Weight Decay, etc
Synthetic Data and related techniques such as SMOTE
Ensembles


Running SQL-like queries over large schemaless JSON dataset in the cloud? I've got about 5 million JSON files, about 50GB in total. They do not have a consistent schema (they're broadly the same format, but some have extra extension fields, some have missing fields, etc - the schema is quite complexly nested).

I would like to run SQL-like queries across these files - e.g. finding the count of files with a certain property, finding the count of files where property is in a numeric or time range, etc.

I have the files locally, and in an S3 bucket. I would ideally like to store the data in the cloud, so that others can run queries against them without having to have the files locally.

What are my options? Here are some solutions I've evaluated so far:

Apache Drill.

Pros: Great for SQL-like queries and 'on-discovery' schemas.
Cons: Ruled out because it struggles with a dataset this size both locally (runs out of memory) and hooked up to S3 (simply doesn't return results, I think because it has to get a local copy of the data).

Google BigQuery.

Pros: Great for SQL-like queries, stores in the cloud, large datasets, plus the friendly user interface is a bonus. JSON can be loaded from command line.
Cons: I'd need to define a schema, and remove or fill any data that doesn't conform to it. This is a hassle, and a problem because it means I can't query some data that I would like to (e.g. extension fields).

Amazon Athena.

Pros: Cloud support (runs against S3), and (I think) has SQL-like queries.
Cons: Seems to require a defined schema (one you define at runtime, rather than in advance)? Also seems to need the files in the query to match the schema, which wouldn't work for my data. However, I'm not sure this is correct, because the documentation is quite confusing.

Google Cloud Datastore.

Pros: SQL-like queries with GQL, stores in the cloud. NoSQL, so schemaless.
Cons: Seems to require pre-defined indexes in order to run queries (?), limited query support (e.g. no aggregation or joins). No documentation on how to load JSON files into it (possibly needs Cloud Dataflow setup).

I'm leaning towards BigQuery, mainly because it's much easier to get up and running than any other solution. However, the requirement for a predefined schema is a major hassle.

Are there any solutions I've missed? Have I misunderstood Athena's requirements about schemas?

You want to write structured queries (have your cake) but have unstructured files (eat it too).

Is there any way you can play with the files a bit and have a common schema between them, with some fields just empty? That would make it easier.

That being said, BigQuery is probably your least bad solution as you found. Amazon's new Redshift Spectrum is kind of similar but will also require a schema to be defined.


How to stop a neural network from regressing to the mean I'm very familiar with neural networks for classification, but I'm trying a regression task for the first time. I'm finding that the network tends to go towards guessing a mean for the whole dataset rather than making case specific predictions.
What are the potential reasons for this behaviour and how can I stop it?
Hint
In my specific case this behaviour is present in both the training and validation set. The training set is very small, and it's questionable as to whether the inputs have a strong correlation with the target.
Also might be worth noting that I'm training on images. So I'm using a CNN.
Update
I did some inspection of my model and found that it drives the output of my CNN backbone to 0 and simply uses the single bias on my fully connected layer to produce the output. I also found that if I freeze the bias during training, I can increase the learning rate indefinitely without the loss blowing up...

The regression tasks are not very different from classification and the behavior you faced with is probably due to a bug in the code.
If your training set is small and the network is comparably large, it should overfit on it (correlation with inputs doesn't matter, there was a paper https://arxiv.org/pdf/1611.03530.pdf that shows you can randomly shuffle imagenet labels and train one of the version of resnet on it and get perfect accuracy on that training set). Sample agnostic predictions mean underfitting. You can also try to overfit a small batch and get a zero loss. If you won't be unable to do this, there's a bug, definitely. Possibly your gradients affect the latter layers much more than the first layers, which means after some epochs of training you still have random outputs from the first layer and it's no wonder the net is trying to predict the mean of the dataset. This can happen if your net is very deep and you're not using residual connections. But it's not the case for a net of 10 (for instance) or less layers. Maybe your learning rate is too high and your first layers can't be tuned correctly. To check whether your first layers extract some meaningful information, try to look at the outputs of some of your first layers. If those outputs look like a complete noise, double check the code for optimization. You can find more interesting cases of bugs and the behavior of a buggy code in Andrej Karpathy's blog http://karpathy.github.io/2019/04/25/recipe/


Confusion for considering accuracy or standard deviation in selecting the best parameters I have a model with a various parameters to test.

The size of the dataset I have is not really large (~500 documents).

My issue is that when I test the parameters using 10 CV, some of them produce high accuracy value but the Standard deviation value of the folds (accuracy values of the folds) is high.

ex.

Model setup 1: acc: 0.81, STD: 0.23
Model setup 2: acc: 0.76, STD: 0.05

Setup 1 has higher accuracy but the std is high, where setup 2 has lower accuracy but with more stable results.

Thus, how can I pick the best model?

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the std dev, so you have to arbitrate a tradeoff between performance and stability:

The safe option is to choose the model with lower accuracy and low variance. It might not always perform optimally but at least it won't perform too bad.
The risky option is the high accuracy, high variance model: in average it will perform best, but you have a higher risk that it actually performs poorly.

This choice depends on the context, i.e. what the model is intended for.


Decision Tree : how to determine target in a model with no labels? I am studying classification algorithms using decision tree approach in Python.
I would have some questions on this topic, specifically regarding the target (y) in my dataset.
I have a dateset made by 20000 observations and a few fields:

Customer
recorded date
amount
status (if married or not)
children (if any children in the family)
nationality (if American or not)

And so on.
Most of these fields are binary (yes/no).
Based on this I would like to determine if this customer is trustworthy or not. As you can see, I have no label about trusting, but I have some initial information: for example the amount. If the amount is 0 or < 0, the customer has no money so he/she can be considered not trusted.
Then, I could consider status: if he/she is married, then it could be considered trustworthy, as there could be another salary to take into account.
And so on.
My doubt are in splitting my dataset, as it asks about y variable.
What would it be in this case? I have no explicit target..

Use Clustering under unsupervised learning. That will categorize the customer based on similar parameters. You can define the number of cluster you need to form, in you case it is two(trustworthy and not). If there are more features it will be more helpful for the algorithm.
This might help.
https://towardsdatascience.com/an-introduction-to-clustering-algorithms-in-python-123438574097


Time horizon T in policy gradients (actor-critic) I am currently going through the Berkeley lectures on Reinforcement Learning. Specifically, I am at slide 5 of this lecture.

At the bottom of that slide, the gradient of the expected sum of rewards function is given by
$$
\nabla J(\theta) = \frac{1}{N} \sum_{i=1}^N \sum_{t=1}^T \nabla_\theta \log{\pi_\theta(a_{i,t} \vert s_{i,t}) (Q(s_{i,t},a_{i,t}) - V(s_{i,t}))}
$$
The q-value function is defined as
$$Q(s_t,a_t) = \sum_{t'=t}^T \mathbb{E}_{\pi_\theta}[r(s_{t'},a_{t'})\vert s_t,a_t]$$
At first glance, this makes sense, because I compare the value of taking the chosen action $a_{i,t}$ to the average value in time step $t$ and can evaluate how good my action was.

My question is: a specific state $s_{spec}$ can occur in any timestep, for example, $s_1 = s_{spec} = s_{10}$. But isn't there a difference in value depending on whether I hit $s_{spec}$ at timestep 1 or 10 when $T$ is fixed? Does this mean that for every state there is a different q value for each possible $t \in \{0,\ldots,T\}$? I somehow doubt that this is the case, but I don't quite understand how the time horizon $T$ fits in.

Or is $T$ not fixed (perhaps it's defined as the time step in which the trajectory ends in a terminal state - but that'd mean that during trajectory sampling, each simulation would take a different number of timesteps)?

In this case, I think it doesn't matter when you reach $s_{spec}$, but how the q-value gets updated because of taking an action at that state.
Therefore there shouldn't be different q-values for each possible $t\in \{0, ..., T\}$, only q-values for each possible actions.
I'm sure it does make a difference for being at a state at a specific timestep, but it's the agents job to learn this by using the RL algorithms (like policy gradient method in the lecture).

In regards to $T$ being fixed or not, horizon $T$ can be infinite or fixed to a finite number.
For example, if $T$ is fixed to $10$, the agent should learn a policy that maximizes the total discounted rewards in the finite amount of time, but it may not be the most optimal policy. When $T$ is infinite, there is more time to explore and figure out the most optimal policy.

The closest method I know that takes notice to when the state-action pair was encountered is Experience Replay that is used in DQN.

I'm also learning Reinforcement Learning right now! I recommend Deep RL Bootcamp since they give you labs in Python which are really intuitive.


Do repeated sentences impact Word2Vec? I'm working with domain-oriented documents in order to obtain synonyms using Word2Vec.
These documents are usually templates, so sentences are repeated a lot.

1k of the unique sentences represent 83% of the text corpus; while 41k of the unique sentences represent the remaining 17% of the corpus.

Can this unbalance in sentence frequency impact my results? Should I sub-sample the most frequent sentences?

Are the sentences exactly the same, word to word? If that is the case I would suggest removing the repeated sentences because that might create a bias for the word2vec model, ie. repeating the same sentence would overweigh those examples single would end with higher frequency of these words in the model. But it might be the case that this works in your favor for find synonyms. Subsample all the unique sentences, not just the most frequent ones to have a balanced model.

I would also suggest looking at the FastText model which is built on top of the word2vec model, builds n grams at a character level. It is easy to train using gensim and has some prebuilt functions like model.most_similar(positive=[word],topn=number of matching words) to find the nearest neighbors based on the word embeddings, you can also use model.similarity(word1, word2) to easily get a similarity score between 0 and 1. These functions might be helpful to find synonyms.


Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn? There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos.

My question is, what's the main benefit of using these libraries compared to, for example, sklearn.model_selection.GridSearchCV() or sklearn.model_selection.RandomizedSearchCV?

In my case with hyperas I noticed one of the distinct advantage over gridsearch, that is, gridsearch function takes only one array as input. My requirement was two be able to send two array as input as I am working with siamese network. I could do it with hyperas out of the box. So hyperas is more flexible than gridsearchcv.  Check this example


Handling near duplicate observations in a regression / Bayesian model I am working on a model where the underlying data is inherently correlated by groups.  So some of my observations are almost duplicates but not quite.

The problem is pretty simple,  I have a y variable to predict from a discrete x variable and several other potential predictor variables which may or may not be significant.   The observations are not quite independent, they're taken from groups of underlying events but I want to handle this better.

I could approach the problem by only selecting one observation from each underlying group but I think this would lose a lot of data and statistical power.  I would like some method where I can keep all my datapoints but possibly impose some artificial weight scheme or something to weight data less if there are near-duplicates elsewhere or if they are from the same underlying event.

In particular, I'm using a Bayesian regression approach with pymc3 so could do some fudging of variances in my likelihood to fit less for values which have lots of close duplicates from the same underlying group.

Does anyone have experience with similar problems and solutions to deal with it?
I think my model is currently very much overfitting due to high correlations between some predictors and predictions.

Some models are fine with duplicate data. For example, this is how oversampling works in unbalanced datasets, Also Naive Bayes should handle this.
However there are many other models will end up giving high prediction rates because of this. Duplicate rows can cause problems due to the same data being in the train and test datasets. Those rows will give a high accuracy to the model, since the training dataset already knows the answer.
Exact duplicates can be found and removed using

df.duplicated()
df.drop_duplicates(inplace=True)

Finding similar data will be a bit more difficult. You can use these same 2 functions on a partial set of columns if you think that will find more matches. Or you can run the data through a cluster algorithm (k-means) and sample from the different clusters, or add a weight column based on the clustering output, maybe divide by the number of points in each cluster. You can then pass in this weight vector into the model.


Histogram is extremely skewed to the left everyone.
Hope you are all okay.

I am pretty bad at visualizations in Python.
I am working on a movie dataset and now I want to understand how values of variable 'vote_count' for every movie in the dataset are distributed.
So I am using very simple code:

plt.hist(df_movies.vote_count,bins=7)
plt.show()

And the output looks the following way:

As it is easy to notice the distribution is skewed very much to left.
I realize that I should  focus on plt.xticks(), but I tried to play around and the histogram still looked awful.

Could you please give me some advices how to make it more understandable and useful to get insights.

Thanks a lot in advance!

I think there are a few "easy wins" here.

You might add more bins - you are already using the bin setting. Just add something high like 100 or even 1000 to get a first feeling for the data
You can define the range of your bins. For example you could set the range with a list of two entries range = [0, 5000] as an additional parameter
You can consider not showing all data- maybe movies with a few ratings or movies with a lot are not what you focus on. You can use the range argument from point two to limit the plot to the region interesting for you.
You can try a cumulative and normed plot cumulative = True, normed = True this can be very useful to get a better estimate on statements like "80% of movies have less than 100 ratings" - for cumulative plots I recommend a very large number of bins.

I hope this helps and let me know if something is unclear


Is there any advanced collocation tool based on NLP+DL? I am not native English speaker and often times I use Ozdic to find the correct word choice. It's good overall but its database is quite limited. I think a tool based on Word2Vec should be able to do a great job for finding collocations/cooccurance.

  With skip-grams, given a window size of n words around a word w, word2vec predicts contextual words c; i.e. in the notation of probability p(c|w). Conversely, CBOW predicts the current word, given the context in the window, p(w|c).

I am looking for a tool that takes a sentence with a blank spot as input and gives alternative options for the blank spot as an output.

You are describing the "Language Modeling" problem in NLP. Language modeling finds the probability distribution over sequences of words - given the words I have seen so far, what are the most likely words that will come next or are missing.

word2vec can find reasonable collocations/cooccurrence for individual words. However, word2vec will do a poor job with sentence completion because it does not model grammatical dependencies.

You can find a word2vec nearest individual word demo here.


How to use ML to forecast sales of a brand new product I am working on a forecasting problem and came across this issue. How do I forecast sales of a brand new product? For example, a product has been introduced in the store and the store would like to know how the sales will look like.

I figured this would be similar to cold start problem in recommender systems. But still I would like to know if there are any standard approaches for such problems.

You could use Bass or Gamma/Shifted Gompertz method to do that. You could use  diffusion package in R.

Here an example (it is form documentation of the package) :

library(diffusion)

fitbass <- diffusion(tsChicken[, 2], type = "bass")
fitgomp <- diffusion(tsChicken[, 2], type = "gompertz")
fitgsg <- diffusion(tsChicken[, 2], type = "gsgompertz")
# Produce some plots
plot(fitbass)

plot(fitgomp)

plot(fitgsg)

fc_bass <- predict(fitbass,h=20)

fc_gomp <- predict(fitgomp,h=20)

fc_gsg <- predict(fitgsg,h=20)

plot(fc_bass)

plot(fc_gomp)

plot(fc_gsg)

The only disadvantage of the methods is that they do not take care of seasonality of the sales of the products.


Types of maps in Interpretable Machine Learning I have worked on Interpretable Machine Learning (IML) for over 1 year. However, there are some terminologies that always make me confused.
For example, saliency maps/heat maps. Are they same?
Are activation maps visualizations of activation values of a unit(s) in Neural networks? Do you know any other types of maps?
Thank you.

Difference between saliency and heat maps - https://www.reddit.com/r/EyeTracking/comments/hhy2u5/what_is_the_difference_between_saliency_maps_and/

Activation maps - https://computersciencewiki.org/index.php/Feature_maps_(Activation_maps)

These two links should answer your question!
Regarding types of maps, they could be many but activation maps are the most used. People can create various maps so i will not be stressing on the question. I am no expert but I have written one paper on iml. Smoothed Score-CAM. Do look into it :)


How do you measure performance for word prediction tasks? Say I have to predict the next word in a sentence, given the initial few words.

Suppose the prefix is "I went to _____". This prefix is common enough that it might appear 10 times in the training data with a few different variations:

I went to college: 5

I went to California: 3

I went to London: 2

In such a case, suppose my model predicts "college" as the right answer. It would still get only a 50% score if I used accuracy as a metric, the data naturally has multiple (correct) answers for the same input. How do I solve this?

Your accuracy should be this: Did the phrase existed before? It means either storing all combinations of 4-grams or searching the data on the fly.

But then the prediction algorithm that uses any combination from the previous data will be 100% accurate. I do not really understand the reason for such accuracy estimation.


Add ID information from one dataframe to every row in another dataframe without a common key I am trying to combine two dataframes into the second dataframe, but duplicating the first dataframe into every row of dataframe two.

The first dataframe (df1) contains the identifing information, and the other dataframe (df2) contains data about that identifying information.  They are in different dataframes since they come from different files.  I am sure there is an easy way to do this, but cant seem to find it.  The basic problem is there are no unique identifiers on df2 yet, and I need them from df1.

For example, I get df1 from a file containing the identifing information (this dataframe will always only have 1 row, some chance that the column names change as well):

df1:

              UID    Name       ID2
0           21YX4    Alan  21458200

while df2 comes from another file, but contains no base identifying information, like this (this dataframe can have very many rows and columns, and columns vary by file).  df2's keys in this case are c and d (the index is reset), but that is not the problem, the real ID is in df1:

df2
Out[1184]:
           a        b       c   d    e
0   1.427962       NaN  shark   0  0.0
1  -0.977258  7.000000  shark   1  NaN
2  -0.301779       NaN  python  0  NaN
3   0.393833       NaN  panda   1  1.0
4   0.494462       7.0  panda   2  0.0
5  -0.526740  5.000000  shark   2  NaN

Desired result

      UID    Name       ID2           a         b       c   d    e
0   21YX4    Alan  21458200    1.427962        NaN  shark   0  0.0
1   21YX4    Alan  21458200   -0.977258   7.000000  shark   0  NaN
2   21YX4    Alan  21458200   -0.301779        NaN  python  1  NaN
3   21YX4    Alan  21458200    0.393833        NaN  panda   1  1.0
4   21YX4    Alan  21458200    0.494462        7.0  panda   2  0.0
5   21YX4    Alan  21458200   -0.526740   5.000000  shark   2  NaN

I have tried concat, join as well as looping through the keys, but no luck so far.
A join like:

df2=df2.join(df, how='outer')

puts df1 into df2, but as a new row with no data, and no ID values for the data rows.  Is reshape or repeat necessary somehow or is there an easy way to do this?

Any suggestions?

Well, it turned out I accidentally answered my own question in the title.  My answer to this is to insert a new common key into both dataframes, then delete the new key after the 2 are merged.  Maybe there is a more elegant way, but this works.

df.insert(0,'mykey','abcdef')
df2.insert(0,'mykey','abcdef')
df2=pd.merge(df,df2,how='outer',on='mykey')
df2.drop(columns=['mykey'])

Where mykey is the new common key, but make sure it will not match any existing columns, while 'abcdef' is random, but needs to be the same added to df1 and df2.


What are the advantages of the rectified linear (relu) over the sigmoid activation function in networks with many layers? The state of the art of nonlinearity is to use rectified linear units (ReLU) instead of a sigmoid function in deep neural networks. What are the advantages?

The sigmoid function becomes asymptotically either zero or one which means that the gradients are near zero for inputs with a large absolute value.
This makes the sigmoid function prone to vanishing gradient issues which the ReLU does not suffer as much.

In addition, ReLU has an attribute which can be seen both as positive and negative depending on which angle you are approaching it. The fact that ReLU is effectively a function that is zero for negative inputs and identity for positive inputs means that it is easy to have zeros as outputs and this leads to dead neurons. However, dead neurons might sound bad but in many cases, it is not because it allows for sparsity. In a way, the ReLU does a similar job of what an L1 regularization would do which would bring some weights to zero which in turn means a sparse solution.
Sparsity is something that, lots of times, leads to a better generalization of the model but there are times which has a negative impact on performance so it depends.
A good practice when using ReLU is to initialize the bias to a small number rather than zero so that you avoid dead neurons at the beginning of the training of the neural network which might prevent training in general.


How does one fine-tune parameters and weights at the same time? I have been having my hands full with training a model to classify web pages. This is the first time ever that I am doing this, so I know very little about ML. I'm here to learn. :-)

Currently, I am using a Pipeline with FeatureUnion to run a a GridSearchCV to tune my parameters. However, I am slightly confused about how this process ought to go. For the whole process of creating a predictive model I read this answer on "Machine Learning Steps" question, which was helpful, but I still have some questions.

First of all: are parameters independent of your model? The answer linked above suggests first testing different models, and then do parameter tuning. However, I started with tuning my parameters on a Naive Bayes multinomial model, with the idea to test the models (e.g. svm, kNN, NBmn) afterwards with the 'optimal' parameters. But the answer above seems to suggest otherwise. Does this mean that the parameter tuning depends on the best model you choose, which makes it useless to first tune the parameters?

Apart from that, I am also not sure how to correctly tune my parameters per feature. I have 13 features, and put them all in the Pipeline. For things like TfidfVectorizer for eight features, I'd like to tune max_features, ngram_range, and max_df. However, because this takes quite a long time to run I thought I'd start with two features, make sure I get the optimal value, set them in the Pipeline (not in the parameter tuning), and add a feature. However, I feel that this is not the way at all. Is it required to the complete tuning as a single step, or is it possible to do it in steps?

Lastly, I noticed that you can also pass the feature weights of FeatureUnion as a parameter, and tune it. But as far as I have found you can only add specific values to test (example below) and not a range (e.g. 0-1 with steps of 0.1), which makes it hard to set the parameter values as you've got to guess which could be a good value. Is there a better way?

parameters = {
    'union__transformer_weights': [
        {'f1':1, 'f2':0.4, 'f3': 0.2,'f4': 1,'f5': 0.4,'f6': 0.1},
        {'f1':0.4, 'f2':0.9, 'f3': 0.2,'f4': 0.4,'f5': 0.4,'f6': 0.2},
        {'f1':0.8, 'f2':0.7, 'f3': 0.2,'f4': 0.7,'f5': 0.7,'f6': 0.4}
    ]
}

You can test each model with default parameters, and keep the model that seems best.

Or, for each model, you can tune the parameters for that model, to see what is the best that's achievable for each model (by separately tuning its parameters) -- then keep the best model.  This takes more computation power but might be slightly better.

Yes, parameter tuning depends on the model.  Each model may have a very different set of parameters, with entirely different meaning. You can't take parameters for a random forest and then try to use them as parameters for a SVM (for instance); that doesn't even make sense, as they have different parameters.

Yes, there are many possibilities on what order you do feature selection vs choosing a model vs parameter tuning.

I don't know what the "feature weights of FeatureUnion" are, so I can't answer that part.


Use of decision trees for classifying images I am new at Machine Learning and reading about it I wonder if it is possible (and convenient) to use decision trees to classify images.

For instance, to classify faces

Yes, you can use decision trees or random forests for this task and this was even the state-of-the-art approach until the early/mid 2010's. For example, here is a paper from 2010 in which exactly this was done. Nowadays, we use convolutional neural networks (CNN's) as they usually yield a higher accuracy and are more convenient to train, as long as you have a lot of images and decent computational ressources.

The most convenient thing about CNN's is that they implicitly create the features by themselves during the training process, while other machine learning techniques, e.g. decision trees, require you to create a set of features beforehand.

For example, in the referenced paper, the authors had to generate the local binary patterns of the images and then used them to train their decision tree. With a CNN they would have simply used the original images and would most probably got a higher accuracy. However, note that they would have needed more images to train a CNN than to train a decision tree.


How to generate higher quality jpg plots on R I have tried generating higher quality of data visualization plots from RStats. I tried increasing frame dimensions but it's still max at about ~350kb. How do I generate higher quality images from RStats?

I think if you save the plot as SVG (Scalable Vector Graphics) you can scale the image to any size. See this answer for an example.
Second, if you want to print larger, you might need to tweak the DPI (Dots per inch), as done here.


How to create vectors from text for address matching using binary classification? I want to compare two addresses for their similarity (purely similarity in terms of textual occurrences rather than semantic similarity). I have a labelled dataset having "Address1", "Address2" and "Similar", where "Similar" takes a value of 0 (not similar) or 1 (similar).

Now, how would I convert the text in "Address1" and "Address2" into numerical vectors, so as to treat this as a binary classification problem?

I would rather recommend to consider each address as a separate instance. (Separate row in feature matrix)

Each address can be further broken into attributes like: House no., Street, City, State, Country.

Above attributes can be vectorised based on distance between them. Like vectorise each country from west to east(incremental numeric value). Then vectorise each state in country from west to east. Similarly for street. In case two entities fall on same longitude, assign consecutive numerical values.

Once each attribute is vectorised, similarity between two addresses can be Euclidean distance between vectors of two addresses.

Anyways leaving aside data science approach, this problem can be solved further simply by getting distance between two addresses using Google Maps API.


Help interpreting GAN output, and how to fix it? After a few tries, I had trained a GAN to produce semi-sensible output. In this model, it almost instantly found a solution and got stuck there. The loss for both the discriminator and generator were 0.68 (I have used a BCE loss), and the accuracies for both went to around 50%. The output of the generator looked at first glance good enough to be real data, but after analysing it I could see it was still not very good.
My solution here was to increase the power of the discriminator (increased the size of it) and re-train. I hoped by making it larger it would force the generator to create better samples. I got the following output.

It seems that as the GAN loss increases, and is producing worse samples, the discriminator can pick it out more easily.
When I check my output from the trained generator I see it follows some basic rules the real data is following, but again under closer scrutiny, they fail more complex tests the real data would pass.
My questions are:

Is my above interpretation of the plots correct?
For this run, have I made the discriminator to powerful? Should I increase the power of the generator?
Is there another technique I should investigate to stop this form of mode collapse?

Thank you
EDIT:
The architecture I am using is a form of Graph GAN. The generator is just a series of linear layers. The discriminator is 3 Graph Conv Layers, then some linear layers. Slightly similar to this paper.
Two potentially unconventional things I am doing:

There is no batch normalisation, I have found this has a very negative effect on the training. Though I could try and persevere with it.
I am using StandardScaler to scale my data. This choice was made as it easily allows you to unscale data. This is useful as I can take the output of the generator and easily transform it into an original scale. However, StandardScaler does not scale things between 1 and -1, so I cannot use tanh as the final activation function of my generator, instead the final layer of the generator is just Linear.

The outputs of the GAN (once rescaled and the shape has been changed) are similar to:
[[ 46.09169   -25.462175   20.705683  -31.696495 ]
 [ 35.10637   -18.956036   15.20579   -24.803787 ]
 [ 10.253135   -5.759581    5.9068713  -6.3003526]]

An example of the truth is:
[[ 45.6         30.294546   -17.218746   -29.41284   ]
 [  1.8186008    1.7064333    0.5984112    0.19312467]
 [ 44.31433     28.234058   -17.615921   -29.262213  ]]

Notably, the top-left value in the matrix will always be 45.6. My Generator does not even consistently produce this.

In theory, we should first train the discriminator to optimal. However, if it becomes too good, then it will reject fake data every time, thus making the generator unable to learn. Unlike other ML task, lower GAN losses does not mean the training is converging. Newer GANs have better techniques to make training GAN easier such as Wasserstein loss, spectral normalization and progressive growing.


When to remove outlier in preparing features for machine learning algorithm I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, then the histogram of this variable in train and test data set looks pretty much the same. See the following figure.

My question is: I still need to use the training data (with both features and labels) to make predictions on the test data (with features only). In this case, how should I deal with this feature variable? I was thinking about removing the top 1% data in both training and test data set, but as I still need to make predictions on that 1% test data, so this is not a good idea I guess. In this example, as the empirical distribution of this variable in both training and test data sets look the same before and after removing the "outlier", should we just leave this variable unchanged? Also, in general, how should we handle the outlier before we put the feature into the machine learning algorithm?

Dealing with outliers requires knowledge about the outlier, the dataset and possibly domain knowledge. Given this, there are many options to handle outliers.

Without taking a look at your specific data, it could be that this outlier represents a total? Perhaps the data source you have included totals, which should be removed.

Generally, figuring out what to do with outliers requires investigating the outlier.

If the outlier is a data processing or entry error, it can generally be removed, or replaced with say, the mean (without the outlier).

If errors have been ruled out think about whether the outlier could be a legitimate value? There are no Right or Wrong answer.

Instead, documentation of your decisions is really important. I'd suggest running your models with the outlier, and without, comparing the results and document any decision to remove or transform the outlier.

Depending on your purpose, it should become relatively clear as to whether the outlier should be removed, eg if you're visualising the data, removing the outlier will likely have more a meaningful impact than retaining it.