Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided.

I first created a solution with Hashing Vector and SGD classifier with actually gave around ~84% accuracy.

After going through many online content I found that Doc2Vec is the current cutting-edge representation of Document or paragraph in numerical format.

So I changed my solution to use Doc2Vec method for Feature Engineering but the accuracy got from this is only ~54%.

Code:

Reading and Cleansing Data

import logging
import datetime
import re
import string
import codecs
import pandas as pd
import numpy as np

from gensim.models.doc2vec import Doc2Vec,TaggedDocument
from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.utils.class_weight import compute_class_weight
from sklearn.metrics import accuracy_score
from sklearn.linear_model import SGDClassifier

import warnings
warnings.filterwarnings("ignore")

#Reading the input/ dataset
data_file = "Consolidated_input_dataset.txt"
data = pd.read_csv(data_file, header = 0, delimiter= "\t", quoting = 3, encoding = "utf8")
data = data.dropna()

#Cleansing the input dataset removing non alphabets
data['cleansed_desc'] = data.COMMODITY_DESC.str.lower().str.replace('[^a-z]',' ').str.replace('\s+',' ')

#Spliting to list for traing Doc2Vec
data['cleansed_desc_split'] = data.cleansed_desc.str.split()
train_data, test_data, train_label,  test_label = train_test_split(data[["cleansed_desc", "cleansed_desc_split"]],
                                                                   data[["Label"]],
                                                                   test_size=0.3, random_state=100, stratify=data.Label)

Hashing Vector:

sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

vectorizer = HashingVectorizer(n_features=90000, ngram_range=(1,3))

vectorizer.fit(train_data.cleansed_desc)
data_features = vectorizer.transform(train_data.cleansed_desc)
sgd_model.fit(data_features, train_label.Label)

#Predict Output
test_features = vectorizer.transform(test_data.cleansed_desc)
output_node1_predict = sgd_model.predict(test_features)

print(accuracy_score(test_label.Label, output_node1_predict))

Output:

84%

Doc2Vec:

#Creating Doc2Vec

data_tagged = train_data.apply(
    lambda r: TaggedDocument(words=r['cleansed_desc_split'], tags=[train_label.loc[r.name].Label]), axis=1)

doc2vec_test = Doc2Vec(dm=0, vector_size=100, negative=5, hs=0, min_count=2, sample=0, epochs=5, workers=8)
doc2vec_test.build_vocab(data_tagged)
doc2vec_test.train(data_tagged, total_examples=doc2vec_test.corpus_count, epochs=doc2vec_test.iter)

train_data["Doc2Vec"] = train_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))
test_data["Doc2Vec"] = test_data.cleansed_desc_split.apply(lambda x : doc2vec_test.infer_vector(x))

#Train Model
sgd_model_full = SGDClassifier(loss='modified_huber',  n_jobs=-1, n_iter=8,
        random_state=42, alpha=1e-06, class_weight="balanced", verbose= 2)

sgd_model.fit(train_data.Doc2Vec.tolist(), train_label.Label)

#Predict Output
output_node1_predict = sgd_model.predict(test_data.Doc2Vec.tolist())

print(accuracy_score(test_label.Label, output_node1_predict))

output:

54%

Parameters for Doc2 Vec:

Vector Size =100
Window = 10
Epoch=5
min_count=2
Negative=5

Total number of Documents = 3450000+
Vocabulary Size = 46000+

If I read your model correctly, you only performed 5 epochs with the Doc2Vec model.  This is probably not be enough for the network to learn the word embeddings.  Has your loss leveled out after 5 epochs?  Try running it for 50 epochs and see if it makes any difference.  Conceivably, you could need thousands of epochs to achieve a reasonable model.


LSTM for capturing multiple patterns I am trying to use an LSTM to predict daily usage for users. I have data for (say) 90 days of usage for a large number of users. Based on business knowledge (and initial analysis) we know users fall roughly into different categories. E.g. daily users would have a non-zero usage almost every day, weekly users would have one or two days of non-zero usage every 7 days and monthly users would have a couple of days with non-zero usage per 30 days.

Sample data where each column is one day starting from October 1st and each row is data for one user. (The usage 'cycle' of each user might start on any day).

User 1: 10,  8, 10,  9, 0, 0, 11, ...
User 2:  0,  0,  0, 20, 0, 0,  0,  0,  0, 18,  0,  0,  0, ...
User 3: 40,  0,  0,  0, .....

where User 1 might be a "daily" user, User 2 is a "weekly" user and User 3 is a monthly user.

My first question is that can a single LSTM/deep learning model capture these different types of patterns?
The goal is to predict the daily usage (next couple of days based on past 90 days) for individual users.

Currently I am using a really simple LSTM (in Keras):

model = Sequential()
model.add(LSTM(1, input_shape=(90, 1), unroll=True))
model.compile(optimizer='rmsprop', loss='mse')

To help the model 'capture' the fact that different users might have different levels of usage I added the average usage (of non-zero values) for each user as the first data point for each row. The remaining 90 data points for each user remain as shown in the table above.

My second question is if I really need to add the average of the values to 'help' the model?

The problem is that even after 100 epochs the error remains unchanged.
And finally what can I change to make this to work?

Thanks in advance.

Modeling is a process where you invent hypotheses and then try to reject them. Without seeing your data, there is no guarantee any of these would go far. But here are some thoughts.

90 days of daily observations seem too few, which won't lead to much confidence no matter what model you use.

1.1 If you treat each user as one sequential observation, yes your dataset is sufficient for models like RNN. But ask yourself the question, are there users out there who are unique and different and you don't know about how they differ from the rest.

You seem to start with a belief that there are three types of users. I would suggest don't. Let the data tells you how many types are there. And for the purpose of predicting future usage, you don't care about this type information.
Recurrent neural net is a good choice if you believe the user behaviors are temporally dependent individually and have cross-sectional dependence. Otherwise, a time series model like ARIMA may do a better job and require less resources to fit.
Lastly, if you want to normalize the data, you should do so by subtracting each row by the row mean and divide by the row standard deviation. By adding the mean as the first observation, you're going to confuse the neural net and it's not smart enough to treat this data point differently. Unless there is a huge variance across users, I would start without normalization, fit a model, figure out some sort of out-of-sample MSE, then repeat on normalized data.

EDIT : There is no reason to train multiple RNNs on this set of data if you do not have a good segmentation argument.


Classification/Prediction based on Multivariate Time Series So, I have a time series with many independent variables (X's) and an outcome variable Y (that I want to predict, think a 2 class logistic regression where output would either be 1 or a 0). Kindly see a sample below:

Timestamp       X1      X2      X3          X4          Y
1:00            1       0.5     23.5        0           0
1:01            1       0.8     18.7        0           0
1:02            0       0.9     4.5         1           0
….
1:30            1       1.9     5.5         1           1
1:31            0       1.7     4.3         0           1
…
…

Now I want to predict or rather classify Y as 0 (stable) or 1 (unstable) (Note that when Y becomes 1 it remains 1 for certain interval of time, same when it is 0)

So Y will be dependent on sequence variables (Please note that it is a time series, and not a standard regression problem where every row can be fed to an Algorithm for classification, the output here is dependent on a sequence of inputs/rows), for instance Y may become 1 when X2 starts increasing and X3 starts decreasing and so on (there are many independent variables X1…XN).

The way I was thinking in order to solve this problem was to extract say m hours of data before Y becomes 1 and do some descriptive statistics on X in order to derive new features (like mean of X1, std of X2, last change point of X4 and so on for the set of this extracted data) to convert the X’s to a single row feature vector. The outcome ‘Y’ of this single row feature vector is 1 as we have just extracted the data before Y became 1. So this way I am able to convert a time series into a standard classification/prediction problem. Similarly I can take the other class i.e. Y=0 and follow the same process.

The other approach that I thought about was to incorporate a sequence model, something like Hidden Markov Model where the hidden states might be stable (say for Y=0) and unstable (for Y=1) and then I go about emission and transition probabilities. But this HMM will be multivariate considering there are many X’s on which Y is dependent. This seems a bit complex?

Any ideas on modeling the above problem will be appreciated.

Train an LSTM-RNN to perform direct sequence classification. This essentially means that it will have multiple inputs and 1 output, i.e. the label (0 or 1). In Keras/Python this is very easy to implement, just make sure that you have a Dense layer in the end with sigmoid activation so that the output is between 0 and 1. You train the network based on your labeled data and then it outputs the label by itself. A useful tutorial on how to do this can be found here. The most important thing is that it inherently deals with linear/nonlinear cross-correlation between inputs so you don't have to explore them yourself. It is also capable of learning the dynamics of the input signals, because of its inherent memory.

Keep in mind that in overall it is a very convenient solution because it works like a black box that accepts time series and "spits" out their labels.

This approach has worked successfully for me for time series classification :)


Can I sum up feature vectors of a user‘s collection? I want to find items that are similar to items users already have in their collection. Every item has attributes, so I created feature vectors where every element of the vector represents an attribute and is either $0$ or $1$ (if an item has that attribute).

For the user collection I summed up all vectors, creating one vector which I then used to calculate similarities with other items.

Is this a correct approach or should I make this "user vector", binary like the other ones? Or is it easier to just calculate $n \times m$ (I.e. user items and new items) similarities?

The set of new items will consist of $\sim1000$ items, while the user collections tend to be $<1000$. As similarity function I used cosine distance, but wanted to try Pearson coefficient as well.

You can use total sum of boolean values. That will be fast and give a general notion of similarity.
A more useful metric might be Hamming distance, the sum of matching booleans between two vectors.


What is the level of measurement / name of the scale of list-features? If you look at publications, you can have a dataset

title of publication
list of authors
number of pages
year of publication

The Level of measurement of "number of pages" is  interval scale, the year of publication is interval scale as well, the title is nominal. But what is the list of authors? Simply saying that it is nominal seems not to capture a major part of this attribute.

With a list such as this list of authors, the only representation which truly capture the semantics is a (long) list of boolean features, one for each possible author in the set. Ideally one would also pre-process the whole set of authors in order to match the different ways a name can be written, e.g. "J. Doe" = "John Doe" = "Doe, John".


Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system? (Note: Pulled this question from the list of questions in Area51, but believe the question is self explanatory. That said, believe I get the general intent of the question, and as a result likely able to field any questions on the question that might pop-up.)

Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm


How to programmatically suggest changes to a text to match some other text? I'm currently thinking of the system that would compare 2 texts (source, target) and suggest changes to the source text to match more the style of the target. It should not convert it to the target but more suggest to rephrase some sentences, changes words, add some key words and so on.
Any ideas how this could be done?
Thanks a lot for any suggestions!

Some time ago, I stumbled upon the Author Obfuscation task of PAN16 workshop. The task at hand is given a document, paraphrase it so that its writing style does not match that of its original author.

Although, it's not exactly what you want, you can take a look at their proceedings and find useful tips regarding authors style and how to "change" it (i.e. in your case, an "author" is a collection of documents from the same source).

For example, one approach
denotes specific metrics that define an author's style (average sentence word count, use of stopwords, pos-type frequencies). You could use the target's scores in these metrics as your gold standard and try to modify the source text, in ways discussed in the proceedings of the workshop, so as to mask your source text to match the target one (in terms of these metrics).


What is the best model for a recommendation system using implicit ratings? I have a similariy matrix that looks like this:

I have a bunch of user vectors with 1s and 0s, with a 1 indicating that someone has clicked on an email (as part of a campaign) and zero to indicate they haven't. Implicit ratings as I have come to understand them.
In terms of an approach, after researching different options, it would appear that the best algorithm to start with a is a matrix-factorisation approach. My question is: what is the best algorithm to start as a baseline model for this type of problem? Any advice on potential pitfalls. Any help would be appreciated. Thanks!

There are different approaches for your case, but I believe you can employ approaches that are based on the self-attention mechanism. There numerous studies that show they perform better than the latent outcomes that can be achieved with SVD. For instance, you can see the paper of SAIN which its code is also open source. In the paper, you can see that they have shown their approach has better performance than SVD and usual self-attention mechanisim. By the way, it is based on self-attention.
If I want to summarize the method, it uses the contents of the products and the items and finds the interactions between the words, and it outputs the score that a user may assign to a product. It has good performance.
Due to the fact that you have the contents of mails, this approach may help you.


Binary classification toy problem I'm trying to build a toy model which can identify a constant difference between two variables: (if variable1- variable2>10 then 1 else 0).

This should be a quite simple task for any regression model, but I want to solve it with NN. However all simple NN I built can not give me more than 51% accuracy.

Is this something that I do not understand?

My code:

seed = 7
np.random.seed(seed)

x1 = np.arange(50000)
x2 = x1+10+(0.5-np.random.rand(len(x1)))
X = np.column_stack((x1,x2))

Y = (x2-x1)>10

encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

train_X, test_X, train_y, test_y = train_test_split(
    X, Y, train_size=0.9, random_state=0)

model = Sequential()
model.add(Dense(2, input_dim=2,activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1,  activation='sigmoid'))

model.compile(optimizer=RMSprop(lr=0.001),
  loss=binary_crossentropy, metrics=[binary_accuracy])

history = model.fit(train_X, train_y, epochs=1000, batch_size=10,
  validation_data=(test_X, test_y), verbose=1)

Your problem is that neural networks work poorly when the input is not scaled to a simple range. A usual choice is to scale and offset each column so that it has mean 0 and standard deviation 1.

In your case, x1 and x2 vary from 0 to 49999 and roughly 10 to 50009. This range for inputs will causes lots of numeric issues.

With a balanced dataset as you have, 51% accuracy is basically just guessing (within experimental error), so the network has learned nothing. Try again with scaling - e.g.

x1 = (x1 - 25000) / 14433
x2 = (x2 - 25000) / 14433

I have tested your code with this addition, and it gains 100% validation accuracy within the first epoch.

If you want to assess other values in testing later you will need to scale them in the same way.

Note your predictions may be off in testing when x1 and x2 are not close to 10 apart, because you have only trained with examples which are close to exactly 10 apart. How the network behaves when this is not the case - e.g. for inputs of x1 = 100 and x2 = 1000, or x1 = 90 and x2 = 15, may not generalise well compared to the original comparison function.


Python - AttributeError: type object 'DataFrame' has no attribute 'read_csv' I'm trying to create some charts using Python. I've this dataset in a CSV file:

Banana  Water   Rice
Rice    Water
Bread   Banana  Juice

And I've this code:

 import numpy as np
    from pandas import DataFrame
    import matplotlib
    matplotlib.use('agg') # Write figure to disk instead of displaying (for Windows Subsystem for Linux)
    import matplotlib.pyplot as plt

    data = DataFrame.read_csv("test.csv", index_col=1, skiprows=1).T.to_dict()

But when I execute the code I'm getting the following error:

AttributeError: type object 'DataFrame' has no attribute 'read_csv'

How can I solve this problem?

Many thanks!

read_csv() is not available on DataFrame. to read csvs using pandas -

import pandas as pd
data = pd.read_csv("file_name")

if you check type(data), it will be pandas DataFrame.


Automatic labelling of text data based on predefined entities I'm new to NLP.
I have a folder containing .txt files which are legal and specific documents. I want to label all these files based on four predefined labels. How can I do that automatically?

The task you have is called named-entity recognition. From wiki:

  Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a subtask of information extraction that seeks to locate and classify named entity mentions in unstructured text into pre-defined categories such as the person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.

Since this is a common NLP task there are libraries that are made to do NER out of the box. One such library is spaCy which can do NER as well as many other NLP tasks using Python.

You will not be able to perform NER without first training a model on your custom labels/entities. You need to have some labelled data to train on, maybe you already have this or you can label it manually. SpaCy wants yo have the data labelled with location of each entity on the format:

[("legal text here", {"entities": [(Start index, End index, "Money"),
                                   (Start index, End index, "Judge"),
                                   (Start index, End index, "Tribunal"),
                                   (Start index, End index, "State")]}),
("legal text here", {"entities": [(Start index, End index, "Money"),
                                  (Start index, End index, "Judge"),
                                  (Start index, End index, "Tribunal"),
                                  (Start index, End index, "State")]})
...]

Example on how to training a spaCy model for NER (taken from docs):

from __future__ import unicode_literals, print_function

import plac
import random
from pathlib import Path
import spacy
from spacy.util import minibatch, compounding

# training data
TRAIN_DATA =   Insert you labelled training data here

@plac.annotations(
    model=("Model name. Defaults to blank 'en' model.", "option", "m", str),
    output_dir=("Optional output directory", "option", "o", Path),
    n_iter=("Number of training iterations", "option", "n", int),
)
def main(model=None, output_dir=None, n_iter=100):
    """Load the model, set up the pipeline and train the entity recognizer."""
    if model is not None:
        nlp = spacy.load(model)  # load existing spaCy model
        print("Loaded model '%s'" % model)
    else:
        nlp = spacy.blank("en")  # create blank Language class
        print("Created blank 'en' model")

    # create the built-in pipeline components and add them to the pipeline
    # nlp.create_pipe works for built-ins that are registered with spaCy
    if "ner" not in nlp.pipe_names:
        ner = nlp.create_pipe("ner")
        nlp.add_pipe(ner, last=True)
    # otherwise, get it so we can add labels
    else:
        ner = nlp.get_pipe("ner")

    # add labels
    for _, annotations in TRAIN_DATA:
        for ent in annotations.get("entities"):
            ner.add_label(ent[2])

    # get names of other pipes to disable them during training
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
    with nlp.disable_pipes(*other_pipes):  # only train NER
        # reset and initialize the weights randomly – but only if we're
        # training a new model
        if model is None:
            nlp.begin_training()
        for itn in range(n_iter):
            random.shuffle(TRAIN_DATA)
            losses = {}
            # batch up the examples using spaCy's minibatch
            batches = minibatch(TRAIN_DATA, size=compounding(4.0, 32.0, 1.001))
            for batch in batches:
                texts, annotations = zip(*batch)
                nlp.update(
                    texts,  # batch of texts
                    annotations,  # batch of annotations
                    drop=0.5,  # dropout - make it harder to memorise data
                    losses=losses,
                )
            print("Losses", losses)

    # test the trained model
    for text, _ in TRAIN_DATA:
        doc = nlp(text)
        print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
        print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc])

    # save model to output directory
    if output_dir is not None:
        output_dir = Path(output_dir)
        if not output_dir.exists():
            output_dir.mkdir()
        nlp.to_disk(output_dir)
        print("Saved model to", output_dir)

        # test the saved model
        print("Loading from", output_dir)
        nlp2 = spacy.load(output_dir)
        for text, _ in TRAIN_DATA:
            doc = nlp2(text)
            print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
            print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc])

if __name__ == "__main__":
    plac.call(main)

Then when you have a trained model you can use it to get your entities:

doc = nlp('put legal text to test your model here')

for ent in doc.ents:
    print(ent.text, ent.start_char, ent.end_char, ent.label_)


KNN efficient implementation The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production.
As a comparative example, when I use a neural network, I already have at my disposal high level tools allowing me to apply the neural network to examples (either libraries allowing me to smartly exploit the hardware of my devices when I want to do embedded, or infrastructures allowing me to use my neural networks at a lower cost if I want to run them on the cloud), and these tools are generally black boxes and are very well optimized, especially when you pass them batches of examples.
It seems that there are no such tools to use a KNN on a large scale for example on a cloud; and I guess one has to implement it oneself.
A naive implementation of KNN consisting in calculating the distances from the inferred example to all known and recorded examples makes the computational cost and thus the financial cost of using it explode (which strangely enough becomes much higher than the one I pay when I use a neural network, even though the algorithm seems simpler).
Do you know people who have shared efficient and cheap implementations of KNN? Or do you have any ideas on the subject ?
I started to imagine heuristics to quickly disqualify examples that are too far away without having to go through them one by one.
Concretely, this would consist in a set of n-dimensional nested grids such that searching in each sequence of nested grids is like searching in a tree or in a hash table and allows to locate the example in as many bins as sequences of nested grids. One then expects that the closest examples are either in the same bin or in a contiguous bin (because the example may be close to the bin boundary).
I suppose one could also imagine dividing the space into decision zones (so that a 1-KNN would generate a Voronoi diagram for example).
However, I am not a researcher and my ideas will certainly not be able to challenge the state of the art or already existing and explored ideas.

I suggest you use facebook's faiss. It is a library for similarity search, which can be used to compute kNN on large vector collections.
From facebook's own numbers:

With approximate indexing, a brute-force k-nearest-neighbor graph (k = 10) on 128D CNN descriptors of 95 million images of the YFCC100M data set with 10-intersection of 0.8 can be constructed in 35 minutes on four Maxwell Titan X GPUs, including index construction time.
Billion-vector k-nearest-neighbor graphs are now easily within reach. One can make a brute-force k-NN graph (k = 10) of the Deep1B data set with 10-intersection of 0.65 in under 12 hours on four Maxwell Titan X GPUs, or 0.8 in under 12 hours on eight Pascal P100-PCIe GPUs. Lower-quality graphs can be produced in under 5 hours on the Titan X configuration.


Averaging on skewed distribution I have to compose a feature which summarizes the blocks area of different sections of cities. (A block is defined as the space contained by streets).

I could compute the arithmetic average of areas, but the data is often very skewed in the sections. I could compute the arithmetic average of log(areas), which transform the data to a more normal distribution, but the problem is that the data is skewed differently section by section.
How would you create this feature?

You could use the median. That protects against the "Bill Gates walks into a bar, making everyone there a millionaire, on average" effect. It's used with wage distributions for exactly the reason that they tend to be very skewed.


How to calculate time difference in between rows using loop in panda python I have a CSV file with columns date, time. I want to calculate row-by-row the time difference time_diff in the time column. I wrote the following code but it's incorrect.
Here is my code and at bottom, my CSV file:

#def time_diff(x):

date_array = []
date_array.append(pd.to_datetime(data['date'][0]).date())
start = []
end = []
temp_date  = pd.to_datetime(data['date'][0]).date()
start.append(pd.to_datetime(data['time'][0], format='%H:%M:%S').time())
for i in range(len(data['date'])):
    cur_date = pd.to_datetime(data['date'][i]).date()
    if( cur_date > temp_date):
        end.append(pd.to_datetime(data['time'][i-1], format='%H:%M:%S').time())
        start.append(pd.to_datetime(data['time'][i], format='%H:%M:%S').time())
        date_array.append(cur_date)
        temp_date = cur_date
end.append(pd.to_datetime(data['time'][len(data['date'])-1], format='%H:%M:%S').time())
if start <= end:
    return end - start
else:
    end += timedelta(1) # +day
    assert end > start
    return end - start

for i in range(len(date_array)):
    s_time = datetime.datetime.combine(date_array[i],start_time[i])
    e_time = datetime.datetime.combine(date_array[i],  end_time[i])
    timediff = (e_time - s_time)

My CSV file:

First column and second column are date and time. Third column shows expected time_diff in hours.

I realize that this has already been answered, but I thought I would propose another solution that takes advantage of vectorization.  It should perform better if there are a lot of records by avoiding iteration.

First, I'll create a test data set:

import pandas as pd
import datetime
from tabulate import tabulate
import numpy as np

start_date = datetime.datetime(2018, 1, 1, 00, 0, 0)
end_date = datetime.datetime(2018, 12, 31, 00, 0, 0)

duration = (end_date - start_date).total_seconds()
custom_index = range(0, 20)

duration_df = pd.DataFrame(columns=['Random Duration'], index=custom_index)
duration_df['Random Duration'] = duration

duration_df['Random Duration'] = duration_df['Random Duration'].apply(lambda x: np.random.randint(0, x))
duration_df['Random Date'] = duration_df['Random Duration'].apply(lambda x: start_date + datetime.timedelta(seconds=x))
duration_df = duration_df.drop(columns='Random Duration')
duration_df = duration_df.sort_values('Random Date')
duration_df = duration_df.reset_index(drop=True)

Now we can create the new column with this one liner:

duration_df['Hours Since Previous'] = (duration_df['Random Date'] -
                                       duration_df['Random Date'].shift(1)).astype('timedelta64[h]')

This will give you the following results:

    Random Date            Hours Since Previous
--  -------------------  ----------------------
 0  2018-01-16 21:56:47                     nan
 1  2018-01-24 17:54:12                     187
 2  2018-02-13 19:30:24                     481
 3  2018-02-21 04:18:23                     176
 4  2018-03-05 02:07:50                     285
 5  2018-03-13 20:18:30                     210
 6  2018-04-16 09:05:13                     804
 7  2018-04-17 20:30:46                      35
 8  2018-05-17 02:39:36                     702
 9  2018-05-20 16:43:20                      86

How this works

duration_df['Random Date'].shift(1) creates a new series of dates that are offset by one row.

(duration_df['Random Date'] - duration_df['Random Date'].shift(1)) creates a pandas.Timedelta() object and .astype('timedelta64[h]') converts the resulting Timedelta to hours.


Is Keras a fancy tool or is it actually helpful in research or both? I’m in second year of my PhD in a environmental-science laboratory and there aren’t many experts on machine learning here.

In one part of my research I should use a multilayer perceptron to build a model from a series of inputs that I determined before. So for this part I suggested to use a deep learning framework, specifically Keras with Tensorflow or CNTK, but in the lab my supervisors told me that for a real application they don’t want to depend of a framework from Google or Microsoft and they prefer to develop they own tools that won’t change in the time if there’s for example an upgrade or something like that. One researcher created a neutral net and he obtained good results, but when we are interested in make some fine tuning, his code is more complex to update or to test new things like an dropout for example.

I started to work with those frameworks having in mind the reproducibility because I only have to use the functions from Keras and define the parameters. I think other researchers who want to test/use my model should use the configuration I suggest and don’t try to understand the code I provide. So I obtained some good results and am now in the phase of fine tuning, but the work of the other researcher has a slightly better results but after a regularisation step that I didn’t try yet.

So this researcher told me that I should abandon my work with Keras and start using his code. He also told me that Keras is just some fancy tool and in research is not required to have a very precise model, just good results and use the code that works.

So my question is this: Is it right what this researcher told me? Is Keras actually used in research? If there are some frameworks that are developed, why not use them instead of rediscovering the wheel, am I right?

In general any tool that many researchers use is by definition useful/helpful.

For the particular case of Keras and other neural network frameworks (like PyTorch, TensorFlow, etc), a lot of people use them. You can see this by reading papers in the topic, as it is usually mentioned which framework the implementation is made in. You can also check "popularity" by counting references, for example for keras and tensorflow, both have thousands of citations.

In general you should prefer well known tools that are used in the field. These are validated and you can at least have a degree of trust that the results they produce are correct (minus user mistakes, of course). Implementing a neural network framework from scratch is not easy and any reviewer can point out that there could be programming errors and/or mistakes. Also as you mention there is the advantage of reproducibility, as it is much more likely that people can run your code.

Also note that many frameworks were actually made by researchers, so they are specifically targeted for other researchers, and their design is made for easy experimentation. In keras for example, it is easy to implement some custom functionality as a loss or layer, so you can integrate it with another network design. A framework can have another target userbase such as deployment for low power platforms or computers, which Keras doesn't.

Finally, you have to consider that since a lot of people use Keras, there is a large community around it from where you can get support, such as Stack Overflow or the keras-users google group.  Using custom code will make this more difficult or impossible.

If you want not to depend on a specific framework, you can always export networks in ONNX format so you can use it with another framework or your own code.


Data available from industry operations I'm going to start my degree thesis and I want to do a fault detector system using machine learning techniques. I need datasets for my thesis but I don't know where I can get that data. I'm looking for historical operation/maintenance/fault datasets of any kind of machine in the oil & gas industry (drills, steam injectors etc) or electrical companies (transformators, generators etc).

A huge list of open data sets is listed here:

Publicly Available Datasets

Including Amazon, KDnuggets, Stanford, Twitter, Freebase, Google Public and more.


Is it possible to know the output vectors of MLP Classifier of scikit learn? I'm a beginner with scikiti-learn library.
I have an ANN with 3 input, 2 hidden layers and 3 output.

mlp = MLPClassifier(hidden_layer_sizes= hidden_layers,max_iter=iterations, activation=activation_fun)

I read on the documentation that the classifier uses softmax for the output activation function and cross-entropy loss function.
I have a multi-class problem where the three outputs will predict the classes 0,1,2.
My question is that. How can I retrieve the vectors that enconds the classes 0,1,2?
example:
[1,0,0] -> 0
[0,1,0] -> 1
[0,0,1] -> 2

If you are interested on the probability output of your model, simply call mlp.predict_proba(X)


Nearest Neighbors on mixed data types in high dimensions I would like to be able to use nearest neighbors to attempt to find the most similar samples to a subclass of samples (think treated vs untreated) in a dataset with continuous, categorical, and text features.

Toy data set:

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, QuantileTransformer
from sklearn.neighbors import NearestNeighbors
from sklearn.compose import ColumnTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
from sklearn.decomposition import TruncatedSVD

np.set_printoptions(suppress=True)

a = [20, 100, 10000, 500]
b = [1, 2, 3, 2]
c = ['dog', 'cat', 'foo', 'cat']
d = ['apple apple fruit',
     'mercedes bmw chevrolet',
     'monster dragon snake',
     'mercedes chevrolet bmw buick']

z = [a,b,c,d]

names = {0: 'col1', 1:'col2', 2:'col3', 3:'col4'}

X = pd.DataFrame(z).T
X = X.rename(names, axis='columns')
X

Will create:

   col1    col2 col3    col4
0   20      1   dog     apple apple fruit
1   100     2   cat     mercedes bmw chevrolet
2   10000   3   foo     monster dragon snake
3   500     2   cat     mercedes chevrolet bmw buick

As we can see, samples 1 and 3 are the most related. They have many of the same vocabulary, share two labels (col2 + col3) and considering the distribution of col1 they are fairly close together. We transform them into a feature array and ask for nearest neighbors like so:

numeric = ['col1']
numeric_transformer = Pipeline(steps=[('scaler', QuantileTransformer())])

cat = ['col2', 'col3']
cat_transformer = Pipeline(steps=[('onehot', OneHotEncoder())])

text = ['col4']
text_transformer = Pipeline(steps=[('tfidf', TfidfVectorizer()),
                                   ('svd', TruncatedSVD(n_components=2))])

prep = ColumnTransformer(transformers=[('num', numeric_transformer, numeric),
                                       ('cat', cat_transformer, cat),
                                       ('text', text_transformer, 'col4')], sparse_threshold=0)

X_transformed = prep.fit_transform(X)

nn = NearestNeighbors(n_neighbors=2)
nn.fit(X_transformed)

d, i = nn.kneighbors(X_transformed[1].reshape(1,-1))
i

Correctly returns array([[1, 3]], dtype=int64) with the 1 indicating the self match and the 3 being the nearest neighbor.

But on a real world dataset, across 100+ dimensions, should I be using a custom distance function? for nearest neighbors with sklearn, if the column was one of the ones that was text, we could use cosine distance, another distance across the high dimensional one hot encoded variables, and another distance calculation with real continuous transformed variables (col1).

But is this....hacky? Is there some way to deal with the heterogenous data for nearest neighbors search without this? I worry that my decisions as to how to weight each of the three types of variables make the end results very subjective and open to criticism.

Conceptually, there's no reason why this 'could' not work, but practically speaking, there are probably better approaches than using KNN.

Suitability of KNN

One immediate problem with KNN is that each new instance must be compared to each existing reference instance before a class can be determined. As the number of reference instances increases, the time to classify a new instance increases.  With highly dimensional data, you're going to need a lot of reference instances.

Dimensionality of the Data

If you are set on using KNN, you will want to do everything you can to reduce the number of dimensions, particularly on the categorical columns.  In your example, you may be better off reducing the animals down to some larger groups, for example: [Pet, Domesticated, Wild] or biological family.  If not, once you encode the categorical values, you will maybe have thousands (to millions) of animal columns alone.  If you don't have many examples of each, you're going to match on everything but the animal because
$$Distance\ between:\ dog\ and\ cat = 1$$
$$Distance\ between:\ dog\ and\ giraffe = 1$$
So, this will mean other attributes in the data will likely be more similar, yet, they may be uninformative to the classification process.  For example, number of legs.

On the positive side though, KNN may deal with the introduction of new attributes a little better than others because the classification is based on the currently available data, so adding a new column is not as big a deal as say adding a new column to a neural network, which will need to be retrained to be able to accept a new data set.

Conclusion

While there are other pros and cons to KNN and ways to optimize it for performance; based on your description of the data, and perhaps making a few assumptions, KNN does not sound like a good fit for your classification problem.

Most models are going to perform better if you can reduce the number of dimensions, but if the number and nature of attributes is relatively static, I would start with something like a Decision Tree.  The benefit of starting with a decision tree is that you can get a feel for the importance of the columns to the class assigned.


What is the definition of imbalanced data set I have thousands of data sources generating data from similar type of hardware. The different sources create different dynamics in the datasets though!

Even though the features are the same the data sets have very diverse characteristics.

I am working on a multiclass classification problem trying to see how much specific models can be used to tackle that domain.

The number of classes differ on different data sources so different models need to be built. That means that in the end I have many many different models to evaluate. Similar input but the number of classes to be predicted at the output is differrent.

Since this is a multiclass classification problem things like confusion matrices are used and multiple ROC curves.

Now I am trying to see in more details what might be causing poor performance in the poorest performing models. Typically the reasons are:
1.not enough measurements
2.heavily imbalanced datasets
3. a combination of 1 and 2

The problem is that I do not have a definition on a multiclass problem what is an imbalanced dataset. Ideally if I could use a specific "rule" to label my datasets, I would be able to see things like correlation of imbalanced set and precision.

When it comes to imbalanced dataset for multiple class a threshold is not enough, since is the distribution of the available measurements between the classes that is important. For that I have no idea on how to handle that.

How would you handle this case ?

Thanks a lot for reading this and contributing to this community.

Regards
Alex

The problem of an imbalanced dataset is the problem of generative classifiers that use the prior probability for calculating the predicted label. As the labels have a lower prior they get a lower probability.
There are several ways to cope with imbalanced datasets:

Oversampling the minority classes ,randomly add observations from the minority classes, so the prior probability of every class will be the same.
Undersampling - if you have a dataset with a lot of observations but the majority class is few times larger than the minority, choose randomly a subset of the whole dataset that includes the same number of observations for each label.
Use Data Augmentation to generate synthetic data that tries to simulate the same distribution of the features in a label.
Weighted classifiers - there are classifiers support weights for the labels.
If you use Neural Network model, you can do Transfer Learning. Copy the weights of the model from a model with balanced data (you told that you have models with a similar feature vector), copy the network (with the weights) and replace the last layer to randomly initialised (better to use Xavier initialiser) . Then freeze all of the layer weights except for the last one and train it. It's better to keep the same proportion between the classes using (1) or (2) and it's also recommended combine with (3).


Scraping Mouse Over Generated Data I am trying to scrape some data from a website with very little success. Basically there is a route overlaid on google maps and whenever you mouse over specific sections of the map (about 200 in all) it fetches 7 fields from a database and displays them on screen. Doing a single map manually would take about 30 minutes and be quite imprecise. There are about 10,000 map routes I want to scrape so this is not realistic to do it manually. Is there a relatively straightforward way of automating this process?

I've been building web scrapers for over 5 years, and I have to say that a web scraper is rarely "relatively straightforward" simply because of how idiosyncratic each website is. It usually takes a minimum of 10 hours per site to code your scraper if you know what you're doing.

Whenever you have to interact with the page that you're scraping, I recommend Selenium. It's open source and works with most major languages, including Python, Java, and Scala. Moving a mouse around is possible, but I think in your case it might be easier to directly call the javascript that is triggered by the mouse movements. Your web scraper would iterate over all of the hoverable html elements and call the on-hover javascript on each element. However, the devil is in the details, and you're going to need to post a lot of questions on Stackoverflow and do a lot of Googling before you get to a final solution.

I've heard that Google has an API for their maps capability. That might be a lot easier.


Samples in a PCA form lines, what are the reasons? I have come across some PCA plots in which the samples are forming "lines", such as this one:

Or this one:

What kind of data can generate such PCA plots? Are there anything wrong with these PCA?

In general (or common cases) you would find those lines when applying PCA, which are the result of finding those Principal Components Analysis, meaning the directions of maximum variance of your original datatset, i.e., the components which give you more information out of your data.
As explained with examples in scikit-learn docu: PCA finds orthogonal directions in the raw feature space that correspond to directions accounting for maximum variance.


Simulation - identify the input parameters that impact the most the output I am studying (via simulation) a system that has several input parameters. The output of the system is influenced by the input parameters. My goal is to identify the parameters that have the most impact on the output of the system. I ran a large amount of simulations and for each one of them I changed the value of each parameters using a random algorithm. To identify the parameters that have the most impact on the output I was planning on doing a basic multivariate linear regression including all the parameters, however, it turned out to have a low R2 and low F-stat.

I am working a little bit out of my element here so I am looking for suggestions regarding what I should investigate next to achieve my goal.

The context of the study is building energy efficiency and the main difficulty is that some input parameters have multiple effect on the system. For example: if I lower the total lighting power used in the simulation it will have for effect to decrease the cooling load of my building but if at the same time I was increasing the cooling efficiency of my cooling system the energy saved by increasing the efficiency would be less than if the lighting power was not decreased. In this example I would like to show which has more impact in regards to the energy consumption. However, in what I am trying to do I have a lot more input parameters.

With my background on Data Analysis and Nonlinear Regression, your question seems to be very general and blurry. I guess what you are looking for is called Sensitivity Analysis in some contexts.

You need to understand that parameters may have different orders of impact on different sub-spaces of variables (and Parameters) space.

You may also want to generate multiple sub-models based on this sub-spaces, so in each sub-model only a few of your parameters have the main effect and the rest could be neglected.

Again you need to make your question a little more detailed. for example tell us why you want to do this analysis. because your final goal totally impacts the solution.


What are some possible reasons that your multiclass classifier is classifying alll the classes in a single class? I have unbalanced classes.
Group1 N = 140
Group2 N = 35
Group3 N = 30

I ran the code on this data and all the Groups got classified as Group1.
I thought that since group1 is the majority group this is not a surprise.
Then i ran the same code but this time with SMOTE, now all groups are 140, and i still got the same results, where all the groups were classified in Group1. Then i balanced the class weights (W/O SMOTE), but still got the same results. This was confusing to me. What am i doing wrong? Can someone help me understand this please? or what can i do to improve the model?
I tried 5 different classifiers (KNN, AdaBoost, SVC, RF, DT) and in 4 out of 6 i got the same result!

Here's the code:

#Splitting data to training and testing
X_train, X_test, y_train, y_test = train_test_split (X, y, test_size = 0.1, random_state=42)

#Apply StandardScaler for feature scaling
sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform (X_test)

#SMOTE
sm = SMOTE(random_state=42)
X_balanced, y_balanced = sm.fit_sample(X_train_std, y_train)

#PCA
pca = PCA(random_state=42)

#Classifier regularization (SVC).

svc = SVC(random_state=42, class_weight= 'balanced')
pipe_svc = Pipeline(steps=[('pca', pca), ('svc', svc)])

# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_svc = [{'pca__n_components': [2, 5, 20, 30, 40, 50, 60, 70, 80, 90, 100, 140, 150]},
                   {'svc__C':[1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
                    'svc__kernel':['rbf', 'linear','poly'],
                    'svc__gamma': [0.05, 0.06, 0.07, 0.08, 0.09, 0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007,
                                   0.008,0.009, 0.0001, 0.0002, 0.0003, 0.0004, 0.0005],
                    'svc__degree': [1, 2, 3, 4, 5, 6],
                    'svc__gamma': ['auto', 'scale']}]

clfsvc = GridSearchCV(pipe_svc, param_grid =parameters_svc, iid=False, cv=10,
                      return_train_score=False)
clfsvc.fit(X_balanced, y_balanced)

# Plot the PCA spectrum (SVC)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clfsvc.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_svc = pd.DataFrame(clfsvc.cv_results_) #(Added _svc to all variable def)
components_col_svc = 'param_pca__n_components'
best_clfs_svc = results_svc.groupby(components_col_svc).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_svc.plot(x=components_col_svc, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()

#Predicting the test set results (SVC)
y_pred1 = clfsvc.predict(X_test)

# Model Accuracy, how often is the classifier correct?
Accuracyscore_svc = accuracy_score(y_test, y_pred1)

print("Accuracy for SVC on CV data: ", Accuracyscore_svc)

# Making the confusion matrix to describe the performance of a classifier
from sklearn.metrics import confusion_matrix
cm1 = confusion_matrix (y_test, y_pred1)

#accuracy
# Get accuracy score
accuracy1 = accuracy_score(y_test, y_pred1)
print('Accuracy1: %.2f%%' % (accuracy1 * 100.0))

#Checking shape after confusion matrix
print (X_test)
print (y_pred1)

print (cm1)

I found the reason finally. I was using the unstandardized X_test.
Thanks.

Edit:

previously i defined the y_pred like so;

y_pred = clf.predict (xtest)

and then constructed the confusion matrix like so;

cm = confusion_matrix (y_test, y_pred)

However, i forgot that i previously changed xtest using standard scalar like so;

x_test_std = sc.transform (xtest)

and that the new x_test that should be used is xtest_std not x_test.

When i realized this and used the proper x_test_std, everything worked and made much more sense.


Handling unwanted negative numbers With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a mix of negative percentage in there which is undesirable.
Is there a way to handle  unwanted negative numbers in pandas python before running it through a machine learning model.

How to handle invalid values like this is an extremely common problem in machine learning, since most datasets contain errors of some kind.

There are a few ways to do it. For example, you could set them all to 0:

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = 0

Or you could fill them with the avg(SoilHumidity), and create an extra feature to flag to the model that they were missing:

import numpy as np

df['SoilHumidityInvalid'] = np.where(df.SoilHumidity < 0, 1, 0)

df.loc[df.SoilHumidity < 0, 'SoilHumidity'] = df.SoilHumidity.mean()

Or, you can try to impute them somehow. Either by back or forward filling (I.E. taking the value from the next or the previous row in your dataset) or by creating a model that uses the other features of your dataset to predict what these invalid values should be.

The right method can depend; sometimes domain knowledge guides you (i.e. if you know the sensor can mistakenly read negatives when it should read 0, then you know to fill with 0). Failing that, I would just try a couple of methods and use cross-validation to see which improves your model the most.


R Recommender System for very sparse matrices required I am trying to build a recommender system based on a large and very sparse matrix. Dimensions of that matrix would approximately be 12000 x 37000, possibly even more rows up to 100000. However, this matrix is extremely sparse. With the 12000x37000 version, about 0.053% of the matrix is non-NA. I've tried SVD, but alas, to no avail. To ensure I have not caused any error during my proceedings:

I created a data.table with unique triplets of "User" - "Item" - "Rating". I should mention "Rating" can stretch anywhere from 0 to about 150.
Now, I applied dcast.data.table to the triplet table, corrected the problem with the first value becoming a column, converted to a matrix. Now, I had a matrix with users as rows, items as columns and the rating as cell content.

Split into Test and Validation set, replaced "NA" with 0, subtracted row means for every row, applied propack.svd from the "svd" package to that matrix, multiplied the three matrices delivered by propack and added the row means to it. (user-means).

After that, I compared the values from the validation set to the corresponding values in my prediction matrix.. and no surprise, the Root mean square error was horribly high, around 6-7.(Mean of non-NA values is around 4.5). I've tried multiple variants of normalisation too, but I just could not get the RMSE below 5.8, ever.

Is there any way to build a viable item recommender system for this dataset? Possibly via arules or clustering?

I would look into the Soft Imputation method that has an implementation in R. It uses iterative soft-thresholding to compute missing values. Calculations are done with a matrix class called "Incomplete" to deal large sparse matrices and allows for quick calculation of scaling/centering rows and columns.

I've had good success using this completing a 10,000 by 10,000 very sparse matrix so I'd imagine it should do fairly well with your dataset.


What are some best Text Representation techniques in NLP I've studied about various text representation techniques like : Bag of Words, N-gram data modelling, Tf-idf, word embedding etc.

I would like to know which among all the techniques are most efficient when it comes to data modelling or representation for a supervised text classification across a large number of categories.

I might have around 40 categories and then around a same number of sub-categories upto 4 levels.

There's no simple answer to this question. As far as I know in general the choice depends mostly on the type of classification:

Bag of Words (usually with tf-idf weights) is a simple but quite efficient representation for classification based on the text topic or similar, assuming the classes are reasonably distinct from each other.
Word embeddings are a more advanced option for semantic-based classification. They can handle more subtle semantic relations but require being trained on a large training corpus. Using pre-defined embeddings can be a solution but then there's the risk that the original training data isn't perfectly suitable for the dataset.
N-grams models can be used in many different ways but are often chosen when the classification involves syntax and/or writing style. Note that the higher the value $n$, the larger the training corpus needs to be, this can also be taken into account in the choice.

  I might have around 40 categories and then around a same number of sub-categories upto 4 levels.

It depends on the data but 40 classes is already a very challenging classification task. For the sake of simplicity let's assume a uniform distribution over classes: a random baseline accuracy would be 1/40 =  2.5%. Of course it depends on the data and a good classifier will do better than that, but don't expect too much...

Now 4 levels of 40 sub-categories means 40^4 = 2.5 millions classes! Even assuming you have enough data (say around 10 instances by class in average, that is 25 millions instances!), it's very unlikely that a classifier will be able to predict anything useful from such a huge amount of classes.


How to initialize a new word2vec model with pre-trained model weights? I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset pretrained model). I have been struggling with it for a couple of weeks. Now, I just found out that in gesim there is a function that can help me initialize the weights of my model with pre-trained model weights.
That is mentioned below:
reset_from(other_model)

    Borrow shareable pre-built structures (like vocab) from the other_model. Useful if testing multiple models in parallel on the same corpus.

I do not know this function can do the same thing or not. Please help!

Thank Abhishek. I've figure it out! Here are my experiments.

1). we plot a easy example:

from gensim.models import Word2Vec
from sklearn.decomposition import PCA
from matplotlib import pyplot
# define training data
sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],
            ['this', 'is', 'the', 'second', 'sentence'],
            ['yet', 'another', 'sentence'],
            ['one', 'more', 'sentence'],
            ['and', 'the', 'final', 'sentence']]
# train model
model_1 = Word2Vec(sentences, size=300, min_count=1)

# fit a 2d PCA model to the vectors
X = model_1[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

From the above plots, we can see that easy sentences cannot distinguish different words' meaning by distances.

2). Load pre-trained word embedding:

from gensim.models import KeyedVectors

model_2 = Word2Vec(size=300, min_count=1)
model_2.build_vocab(sentences)
total_examples = model_2.corpus_count
model = KeyedVectors.load_word2vec_format("glove.6B.300d.txt", binary=False)
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format("glove.6B.300d.txt", binary=False, lockf=1.0)
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)

# fit a 2d PCA model to the vectors
X = model_2[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

From the above figure, we can see that word embeddings are more meaningful.
Hope this answer will be helpful.


Why my cost function is so high? I am trying to implement the gradient descent algorithm from scratch and use it on the Boston dataset. Here is what I have so far:

import numpy as np
from sklearn.datasets import load_boston
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

dataset = load_boston()
X = boston['LSTAT']
y = boston['MEDV']

def linearRegression2(b0 = 0, b1 = 1, lr = 0.00001, n_times = 1000):
partial_B0 = partial_B1 = 0
errors = []
m = len(y)
i = 0
for i in range(n_times):
    error = y - (b0 + X * b1)
    #Updating b0 and b1
    partial_B0 += -lr * (error.sum() / m)
    partial_B1 += -lr * (X.T.dot(error) / m)
    b0 = b0 - (lr * partial_B0)
    b1 = b1 - (lr * partial_B1)
    errors.append(mean_squared_error(y, (b0 + X * b1)))
    i += 1

print(b0, b1)
print('Iteration i', i)
return errors

error = linearRegression2()
plt.plot(np.arange(1, 1000+1), error)
plt.ylabel('Error')

However, when I plot my cost function, it is decreasing but still too high. I tried to decrease the learning rate, increase the iteration but not changing much.

One mistake I see is the fact that you are using the learning rate twice, one when calculating the partial derivatives and once when updating b0 and b1. The learning rate should only be used when updating b0 and b1 and is not used in calculating the partial derivates to b0 and b1.


Understanding the ||w|| = 1 constraint for SVMs Is it correct to say that the reason why ||w|| is set to 1 in the formula for the geometric margin is that it then is the same as the functional margin (i. e. gives the same information) while still being scale invariant?
I come to that from this site: Support Vector Machine (SVM) where it says "If ||w||=1, the functional margin is equal to geometric margin. "
If not, then why can ||w|| not be set to 0.7 oder 2.53 or 14?
I have read various blogs and pdfs on the topic. Some don't adress this at all or just say things along the lines of "this is a mathematical trick" and don't really explain it.

The constraint is interesting in itself as it prevent divergences (or other strange behavior) for the optimisation problem that is the calibration process. The value it is set to not that much. $||w|| = 1$ is a practical convention. For $||w||$ to be constrained to 2.53 you would have need to :

Have some practical interest for it to being set to 2.53, which there is none. $||w|| = 1$ might help simplify some theoretical calculations.
Reach a consensus between users that it would be good to use this value. It's not hard to see why the consensus was reached for 1 instead of 2.53.


unimportant features impact on model's performance Using XGBoost and RandomForests, do unimportant features (according to the feature_importances_ attribute) hurt the model's performance?

Do I need to carefully select highly correlated and import features? Or do I throw everything in and hope that it can correctly add some information on the target variable?

Yes, unimportant features can hurt the model's performance. This happens in my experience in a few ways:

Efficiency - they make the fitting process slower. Particularly if you're one-hot encoding categorical features, and end up with a large and useless sparse matrix. If these reductions in efficiency force you into taking steps that might impact on actually informative features, like a PCA transformation, you're then potentially also directly impacting on predictive ability.
A minimum number of samples is needed per feature for models to effectively learn. By including redundant features, this sample:feature ratio is diminished, which makes it more difficult for many models to determine whether or not a feature is a useful predictor.
You make it more likely to overfit to the "noisy" features.

You can recursively reject features that the model's feature_importances_ routine has decided are unimportant using sklearns recursive feature elimination. Or, in the exploratory phase of building your model, you can assess predictive power using visualisation or hypothesis testing.

This nicely highlights the improvement in performance by ditching redundant features:

https://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html#sphx-glr-auto-examples-feature-selection-plot-rfe-with-cross-validation-py


Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss? A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples.
This is difference than NCE Loss, which doesn't use a softmax at all, it uses a logistic binary classifier for the context/labels. In NLP, 'Negative Sampling' basically refers to the NCE-based approach.
More details here
I have tested both and they both give pretty much the same results. But in word embedding literature, they always use NCE loss, and never sampled softmax.
Is there any reason why this is? The sampled softmax seems like the more obvious solution to prevent applying a softmax to all the classes, so I imagine there must be some good reason for the NCE loss.

Both negative sampling (derived from NCE) and sampled SoftMax use a few samples to bypass the calculation of full SoftMax.
The main problem comes from this comment in the linked pdf:

Sampled Softmax
(A faster way to train a softmax classifier)

which is only used for sampled SoftMax, although, negative sampling is as fast for the same reason that is working with few samples. If their performances are at the same level, this could be the reason why researchers are not convinced to switch over to sampled SoftMax. In academia, it is almost always the case that older methods are preferred over new, but equally-competent methods for the sake of credibility.
Negative sampling is NCE minus the logistic classifier. Roughly speaking, it only borrows the term "F(target) + sum of F(negative sample)s". Negative sampling is most prominently introduced in the Word2Vec paper in 2013 (as of now with 11K citations), and is backed by the mathematically rigorous NCE paper (2012). On the other hand, sampled SoftMax is introduced in this paper (2015) for a task-specific (Machine Translation) and biased approximation:

In this paper, we propose an approximate training algorithm based on
(biased) importance sampling that allows us to train an NMT model with
a much larger target vocabulary

Noting that negative sampling also allows us to train "with a much larger target vocabulary".


In Google Colab you just need to specify the use of GPUs in the menu above. Click:

Edit > Notebook settings >

and then select Hardware accelerator to GPU.

At that point, if you type in a cell:

import tensorflow as tf
tf.test.is_gpu_available()

It should return True.


Multi-output regression problem with Keras number of features: 12 , -15 < each feature < 15

number of targets: 6 , 0 < each target < 360

number of examples: 262144

my normalization: I normalized the features so that they are between 0 and 1. I normalized the targets so that they are between 1 and 10.

This is the model that I am using:

model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(6, activation='linear')
])

model.compile(optimizer="rmsprop", loss='mean_squared_error', metrics=['accuracy'])

model.fit(training_x, training_y, epochs=10, batch_size=100)

This is the best result that I have got (training):

235929/235929 [==============================] - 8s 33us/step - loss: 8.9393e-04 - acc: 0.6436

testing:

loss: 0.00427692719418488

acc: 0.033187106618348276

I get almost 0% accuracy on the test set! I need a model to solve this ML problem.

Accuracy is a metric for classification, not regression.

$$Accuracy = \frac{\text{Correct classification}}{\text{Number of classifications}}$$

So when you use accuracy for regression only the values where actual_label == predicted_label are evaluated as true are counted as correct classifications. That will happen quite rarely when you are doing regression, resulting in an accuracy that is close to zero.

Instead you should use something like mean absolute error or mean squared error as validation metrics for regression.


Is the ultimate challenge in ML simply computational power? I am stuck on a theoretical roadblock in learning about machine learning, because I have not seen this explicitly addressed anywhere. In my studies, it seems as if Cross-validation (or some variant thereof, like LOOCV, or potentially another, but similar, validation scheme like bootstrapping) is the be-all-end-all of model selection. Choosing models and their parameters via exhaustive CV to maximize fit but also balance overfitting seems the optimal way to create models, and computational power is only getting cheaper. So what is there left to do for the human analyst?

I apologize in advance for this amateurish question, but could anyone fill in this gap for me, and potentially suggest some sources on model selection?

There is something known as the VC dimension of a hypothesis class. This refers to the maximum number of datapoints with arbitrary binary labels that can be correctly classified by a model from the hypothesis class.

https://en.wikipedia.org/wiki/VC_dimension

If your number of datapoints is larger than the VC dimension for the hypothesis class you have chosen (say set of 2d hyperplanes), then no matter how much you tune your model using cross validation you can never achieve complete accuracy. Hence, the analyst has the important job of choosing the correct hypothesis class while making sure it doesn't overfit. In the case of deep learning this would mean coming up with a specific architecture and is often one of the most difficult tasks.


Python and R good tutorials? I would like to learn both Python and R for usage in data science projects.

I am currently unemployed, fresh out of university, scouting around for jobs and thought it would be good if I get some Kaggle projects under my profile.

However, I have very little knowledge in either language. Have used Matlab and C/C++ in the past. But I haven't produced production quality code or developed an application or software in either language. It has been dirty coding for academic usage all along.

I have used a little bit of Python, in a university project, but I dont know the fundamentals like what is a package , etc etc. ie havent read the intricacies of the language using a standard Python Textbook etc..

Have done some amount of coding in C/C++ way back (3-4 years back then switched over to Matlab/Octave).

I would like to get started in Python Numpy Scipy scikit-learn and pandas etc. but just reading up Wikipedia articles or Python textbooks is going to be infeasible for me.

And same goes with R, except that I have zero knowledge of R.

Does anyone have any suggestions?

There is an online data science "game" that takes you from learning how to use Python for loading a csv and using scikit to machine learning algorithms such as support vector machines. Here is a blog post with a demo video and the actual site is Explore Data Science. Personally, I think its genius.


what should be the order of class names in sklearn tree export function (Beginner question on python sklearn) I am trying a simple example with sklearn decision tree. I am giving "number,is_power2,is_even" as features and the class is "is_even" (of course this is stupid)

Here is the code

from sklearn import tree

features =[[1,0,0] , [2,1,1] , [3,0,0] , [4,1,1] , [5,0,0] , [6,0,1] ,  [900,0,1] , [1001,0,0]] #val,pow2,even
labels =  ['o'     ,  'e'    , 'o'     ,  'e'    ,  'o'    ,  'e'    ,   'e'      ,  'o'] #is even

clf = tree.DecisionTreeClassifier()
clf = clf.fit(features,labels)

print (clf.predict([[203,0,0]]))

import pydot

import pydotplus
from IPython.display import Image
dot_data = tree.export_graphviz(clf, out_file=None,
                     feature_names=['number','pow2','even'],
                     class_names=['o','e'],
                     filled=True, rounded=True,
                     special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data)
# Image(graph.create_png())
graph.write_pdf("1.pdf")

The decision tree correctly identifies even and odd numbers and the predictions are working properly.

The decision tree is basically like this (in pdf)

     is_even<=0.5
        /\
       /  \
   label1  label2

The problem is this. The label1 is marked "o" and not "e". However if I put class_names in export function as

class_names=['e','o']

then, the result is correct. I thought the output should be independent of class_names order.

Am I doing something wrong, or does the class_names order matter. If the latter is true, what is the right order (for an arbitrary problem)

The names should be given in ascending numerical order.

Saw this in the code's docs:

class_names : list of strings, bool or None, optional (default=None)
    Names of each of the target classes in ascending numerical order.
    Only relevant for classification and not supported for multi-output.
    If ``True``, shows a symbolic representation of the class name.


Method for predicting winner of call for tenders Introduction: Lately I've been looking into different machine learning methods to work around different business problems. By now I have a good, basic understanding of most regression and classification methods, and I'm able to use these methods to predict numeric values given other numeric values and/or simple categories (e.g. an employee's salary given age, years of experience and level of education) or a binary classification (e.g. will this employee leave the company based on the same variables).

What I'm looking for: However, I haven't found the right method for the problem I initially wanted to solve, which involves predicting a non-numeric, non-binary value from a mix of numeric and categorical data. I'm not looking for an in-depth explanation of how to solve the exact problem, but merely advise on which techniques/methods to look into. Ideally something that could be done with R.

The business problem: I have historical data on public tenders (i.e. public sector instutions buying goods/services from private contractors through calls for tenders). The data includes variables like:

Orderer - i.e. who announced the tender (1 of ~150 municipalities/state insitutions)
Type of procurement (1 or more of thousands of industrial classification codes)
Estimated value of contract - A numeric value estimating the value of the contract (at a point before the winner is chosen).
Winner - i.e. which contractor won the tender (1 of ~2000 private
companies)

What I want to do is predict the winner of a tender given the three other variables. It's obviously not a regression problem, and the classification methods I know seem inadequate in handling the problem too. The data are clean and streamlined (no alternate spelling of the different orderer/contractor names, etc.). Any ideas about what to look into?

Having the classifier try to predict one of 2000 possible values is going to be tough. A common approach is to either bucket the possible targets/labels or decompose the problem. For example:

Instead of trying to predict the exact winning company, group the winners into similar groups and predict the group. For example, predict if the winner will be a large public company, small/medium business, or an independent consultant.
The nature of procurement probably varies across the buyer and what is being bought. A model that is good at predicting which company will win the business of a Fortune 500 company probably has a different structure than who is good at winning the business of a small city. In a similar fashion, the competition for who is going to build a bridge is going to be different and not include the companies who will bid on implementing a new website. Partition your data into similar competitions and then try to predict the results. This will hopefully have the side effect of reducing the number of potential targets.


Find differences of two protein sequence files I have two separate files A and B of more than 100000 records of protein sequences. Now I need to find the sequences that are in A but not in B. Both files are in text format. Each ID and Sequence is separated by new line character. I tried making pandas data frame and finding the difference but couldn't make it work.
File A:
>Q63544|9
----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY
>Q63544|51
KTKQGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKT
>Q63544|54
QGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEE
>Q63544|67
VMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRK
>Q63544|72
TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP
>Q63544|73
KTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEPP

File B:
>Q63544|51
KTKQGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKT
>Q63544|54
QGVTEAAEKTKEGVMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEE
>Q63544|67
VMYVGTKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRK
>Q63544|73
KTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEPP

What I need: A-B
>Q63544|9
----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY
>Q63544|72
TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP

Any help or suggestion would be appreciated.

You can do this very easily using python sets as follows:
with open("file_a.txt", "r") as f:
    data_a = f.read().splitlines()

with open("file_b.txt", "r") as f:
    data_b = f.read().splitlines()

set(data_a) - set(data_b)

# {'----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY',
#  '>Q63544|72',
#  '>Q63544|9',
#  'TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP'}


Decision tree classifier: possible overfitting I have a dataset with following specifications:

Training dataset with 52968 samples with 8562 positives
Test Dataset with 13242 samples with 2135 positives
There are 137 features

I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:

imp = Imputer(strategy="most_frequent", axis=0)
var_thr = VarianceThreshold(threshold=1.7)
pca = RandomizedPCA(n_components=16)
clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)

return Pipeline(steps=[('imp', imp),
                       ('var_thr', var_thr),
                       ('pca', pca),
                       ('clf', clf)
])

I also tried to increase training data with positive results:

series = y_train[y_train==1]
dupli = x_train.loc[series.index.tolist(), :]
for _ in range(5):
    x_train = x_train.append(dupli)
    y_train = y_train.append(series)

return x_train, y_train

After fitting my model, the score result for my test data is 0.9954, and the cross validation is:

cross_val_score(clf, x_train, y_train, cv=5)
[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]

Classification report for training data is perfect:

             precision    recall  f1-score   support

          0       1.00      1.00      1.00     44406
          1       1.00      1.00      1.00     42810

avg / total       1.00      1.00      1.00     87216

The confusion matrix is:

[[44203   203]
 [  190 42620]]

but the test data is much worse:

             precision    recall  f1-score   support

          0       0.85      0.85      0.85     11107
          1       0.21      0.21      0.21      2135

avg / total       0.75      0.75      0.75     13242

Confusion matrix is:

[[9428 1679]
 [1687  448]]

I used GridSearchCV for the threshold, n_components, max_features and max_depth.
How can I improve my model and obtain better prediction?

EDIT ------->
I changed clf in pipeline. I used RandomForestClassifier.

clf = RandomForestClassifier(
    n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15,
    random_state=1
)

Now cross validation is

[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]

Classification report for training data with confusion matrix:

score train result: 0.8514148780040359
             precision    recall  f1-score   support

          0       0.86      0.85      0.85     44406
          1       0.85      0.85      0.85     42810

avg / total       0.85      0.85      0.85     87216

[[37757  6649]
 [ 6310 36500]]

Classification report for test data with confusion matrix:

score test result: 0.7341791270200876
             precision    recall  f1-score   support

          0       0.89      0.79      0.83     11107
          1       0.30      0.47      0.36      2135

avg / total       0.79      0.73      0.76     13242

[[8719 2388]
 [1132 1003]]

It looks better but I search model with + 0.90 recall for traning and test data sets.

Perfect training AUC is the hallmark of overfitting. When searching parameters, I often use a combination of test set AUC and the difference between training AUC and test AUC. When just using test set AUC in the search loss function, I found the model to still be pretty sensitive to which random set of test rows it sampled.

I also agree with the commenters that you should get better results with much smaller max_depth, probably on the order of 1-10, and ensemble models. Random forests are an excellent place to start.


How to do Named Entity Recognition in Tables? What are approaches to do Named Entity Recognition on Tables? I am referring to tables which have a column header and the corresponding information in the cells below the header, or the respective with line headers, not tables which contain full sentences. Since these NER lack the context of a sentence and are sort of determined by a header, the approaches I found (most depend on context) do not really work.

As far as I know you don't have a lot of options, you're probably stuck with heuristics:

Capital letters
Regular expressions (e.g. for dates)
List of predefined entities (e.g. from Wikipedia) stored in a dictionary


What to do after GridSearchCV()? I happily created my first NN and performed hyperparameter optimization through GridSearchCV. I just don't know what to do next.
Do I have to fit it again with the best parameters GridSearchCV() revealed? is there an elegant way to do so?
Otherwise, how to proceed?

def create_model(...

    model.compile(loss='mean_squared_error', optimizer=optimizer, metrics=['accuracy'])
    return model

model = KerasRegressor(build_fn=create_model, verbose=0)

> hypparas
{'batch_size': [2, 6], 'optimizer': ['Adam', 'sgd'], 'opt_par': [0.5, 0.8]}

GridSearchCV(estimator=model
                    , param_distributions=hypparas
                    , n_jobs=1
                    , n_iter=20
                    , cv=3
                    )

grid_result = grid_obj.fit(X_train1, y_train1, callbacks = [time_callback])

print("Best: %f using %s" %  (grid_result.best_score_, grid_result.best_params_), "\n")

means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']

for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

    Best: -0.941568 using {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}

    -1.725617 (0.620383) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 2}
    -1.595137 (0.224487) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 2}
    -0.941568 (0.149151) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 2}
    -1.338372 (0.523434) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 2}
    -1.094907 (0.121018) with: {'optimizer': 'Adam', 'opt_par': 0.5, 'batch_size': 6}
    -1.588476 (0.569475) with: {'optimizer': 'sgd', 'opt_par': 0.5, 'batch_size': 6}
    -1.443133 (0.342028) with: {'optimizer': 'Adam', 'opt_par': 0.8, 'batch_size': 6}
    -1.275414 (0.331939) with: {'optimizer': 'sgd', 'opt_par': 0.8, 'batch_size': 6}

You can use grid_obj.predict(X) or grid_obj.best_estimator_.predict(X) to use the tuned estimator. However, I suggest you to get this _best_estimator and train it again with the full set of data, because in GridSearchCV, you train with K-1 folds and you lost 1 fold to test. More data, better estimates, right?


Why do internet companies prefer Java/Python for data scientist job? I see a many times in job description for data scientist asking for Python/Java experience and disregard R. Below is a personal email I received from chief data scientist of a company I applied for through linkedin.

  X, Thanks for connecting and expressing interest. You do have good
  Analytics Skills. However,  all our data scientists must have  good
  programming skills in Java/Python as we are a internet/mobile
  organisation and everything we do is online.

While I respect the decision of the chief data scientist, I am unable to get a clear picture as to what are the tasks that Python can do that R cannot do. Can anyone care to elaborate? I am actually keen to learn Python/Java, provided I get a bit more detail.

Edit: I found an interesting discussion on Quora.
Why is Python a language of choice for data scientists?

Edit2: Blog from Udacity on Languages and Libraries for Machine Learning

So you can integrate with the rest of the code base. It seems your company uses a mix of Java and python. What are you going to do if a little corner of the site needs machine learning; pass the data around with a database, or a cache, drop to R, and so on? Why not just do it all in the same language? It's faster, cleaner, and easier to maintain.

Know any online companies that run solely on R? Neither do I...

All that said Java is the last language I'd do data science in.


Scikit Learn's RandomForestRegressor is not giving results on large data set I have a Pandas dataframe X(20346, 4116). All independent columns have binary variable as 0 or 1. Whereas dependent column has continuous variable. When I execute below code using the scikit-learn library, it takes for ever and I am not getting any result.

X_train, X_test, y_train, y_test = train_test_split(X13, y, train_size=0.8)

rf = RandomForestRegressor(n_estimators=100,  max_features='auto', oob_score=True,
                            n_jobs=1, verbose=0)
rf.fit(X_train, y_train)

I am using this for feature selection. In case of less number of rows I can get result. But when number of rows go up then this problem starts happening. Do you have any idea how to make it work?

Some of the settings are likely too aggressive for your machine:

As others have pointed out, n_jobs might help by taking advantage of openMP threading that is likely available on a multicore processor.

But lets also look at some other parameters from the documentation:

RandomForestRegressor(n_estimators=10, criterion='mse', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=1, random_state=None, verbose=0, warm_start=False)

Why don't you set n_estimators to 10 and then slowly ramp it up.
max_depth should probably be set both so you trees aren't too computationally intensive and so you don't overfit your model (are you cross validating?)
You have set max_features to auto which specifies that it use n_features, but you have not set n_features.  You could consider turning this down to some percentage of your full feature set.

In general you need to cross validate your model and tinker with the parameters in order to find something that runs on your box in finite time using the available memory and performs adequately.  You could also consider tuning and cross validating a simpler decision tree and then using that insight to tweak the parameters of the random forest.

Sample the data:

Big datasets often require lots of computing power.  I sometimes drastically reduce the size of my training data while I am zeroing in on the best model and then time the scaling as I increase the dataset.  This will give you a better idea of how much time you should expect for the full problem.

Other models:

Finally, maybe you have a problem for which 'bootstrap aggregation' is a bad fit. Maybe another type of aggregation would work better like 'boosting' in which case a 'random jungle' or 'ada-boost' algo might work better.  Or perhaps you might want to move to a computationally less intense method that doesn't require aggregation like a 'support vector regressor'.  Scikit-learn has a great benefit in that it is very simple to swap out different models and try different pipelines.

Cloud Computing:

I'm not sure what box or laptop you are running this on, but it might just be easier to spin up an AWS (Google/MS/IBM) instance and run this on a well-powered box in the cloud.  The AWS spot market is crazy affordable over the weekends or in the middle of the night, so you can work on some very powerful 8 core machines for $.50 per hour, so a 24 hour job is only $12.

Hope this helps!


Can a single-layer ANN get XOR wrong? I'm still pretty new to artificial neural networks. While I've played around with TensorFlow, I'm now trying to get the basics straight. Since I've stumbled upon a course which explains how to implement an ANN with back propagation in Unity, with C#, I did just that.

While test-running the ANN with one hidden layer containing 2 neurons, I noticed, that it doesn't always get XOR right. No matter how many epochs it runs or how the learning rate was set. With some settings it happens more often than with other setting.

Usually I get something like this:

+---+---+------+
| 0 | 0 | 0.01 |
+---+---+------+
| 0 | 1 | 0.99 |
+---+---+------+
| 1 | 0 | 0.99 |
+---+---+------+
| 1 | 1 | 0.01 |
+---+---+------+

But in other occasions it looks more like this:

+---+---+------+      +---+---+------+      +---+---+------+
| 0 | 0 | 0.33 |      | 0 | 0 | 0.01 |      | 0 | 0 | 0.33 |
+---+---+------+      +---+---+------+      +---+---+------+
| 0 | 1 | 0.99 |      | 0 | 1 | 0.99 |      | 0 | 1 | 0.33 |
+---+---+------+  or  +---+---+------+  or  +---+---+------+
| 1 | 0 | 0.66 |      | 1 | 0 | 0.50 |      | 1 | 0 | 0.99 |
+---+---+------+      +---+---+------+      +---+---+------+
| 1 | 1 | 0.01 |      | 1 | 1 | 0.50 |      | 1 | 1 | 0.33 |
+---+---+------+      +---+---+------+      +---+---+------+

I've noticed that in every case, the sum of the outputs is ~2. It also doesn't happen most of the time but still quite often. Depending on what settings I use it happens every two or three runs, or it happens only after 10 or 20 runs. For me it seems more like a mathematical quirk in the stochastic nature of neural networks. But I'm not good enough with math to actually figure this one out by myself.

The question is: Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation? If it's not an error in the implementation, what is going on here? Is it because of the very symmetrical nature of an XOR? Which is the reason a single neuron can't handle it, as far as I understood.

I know I could post the source code as well, but I already double and triple checked everything, since I had a mistake in it with the bias calculation. Back then the values were completely off all the time. Now I'm just wondering if this sort of thing could actually happen with a correct implemented neural network.

Assuming the implementation is as simple as possible, with no advanced concepts, is it likely for something like this to happen or is it definitely an error in the implementation?

In my experience, using the simplest possible network, and simplest gradient descent algorithm, then yes this happens relatively frequently. It is an accident of the starting weight values, and technically a local minimum of the cost function, which is why it is so stable when it happens. In the basic implementation you have then there are only 6 starting weights. If they are selected randomly, the chances of a "special" pattern (such as the weights to hidden layer being all positive or all negative) are relatively high (1 in 8 for all positive or all negative weights between input and first hidden layer).

This is also why the values sum to 2 - given the the network is stuck on the wrong part of the error surface, it will still minimise the cost function as best it can given that constraint, and this will usually end up with compromise values that still meet statistical means overall in the predictions. If you doubled up some, but not all of the input/output pairs (e.g. a training set of 6 inputs $\{(0,0:0), (0,1:1), (1,0:1), (1,1:0), (0,1:1), (1,0:1)\}$, then the network may converge to different wrong mean value when it failed.

  Is it because of the very symmetrical nature of an XOR, which makes it impossible to handle for a single neuron?

You don't have a single neuron here. Unless you mean in the output layer? In which case, no, this is not to do with having a single neuron in the output layer.

Pretty much any more advanced NN feature, or simply more randomness, will stop this problem happening. E.g. make the middle layer have 4 neurons instead of 2, use momentum terms, a larger dataset with random "mini-batches" sampled from it.

In general this kind problem does not seem to happen on larger, more complex datasets and larger more complex networks. These can have other problems, but getting stuck in a local minimum far away from the global minimum tends not to happen. In addition for those scenarios, you typically don't want to converge fully into a global minimum for your dataset and error function, but are looking for some form of generalised model (that can predict from input values that you have not seen before).

On a practical note, if you want to add an automated test showing your NN implementation can solve XOR, then either use fixed starting weights or a RNG seed that you know works. Then your test will be reliable, even if the NN is not in all cases.


How to read html tables under multiple headers and combine them in a single pandas dataframe? I have an html file like this:

Group 1

    Col1
    Col2
    Col3

    ValA
    ValB
    ValC

Group 2

    Col1
    Col2
    Col3

    ValP
    ValQ
    ValR

I want to read it into Pandas as if it had this structure:

   Caption
   Col1
   Col2
   Col3

   Group 1
   ValA
   ValB
   ValC

   Group 2
   ValP
   ValQ
   ValR

I can do it easily with PowerQuery the language for PowerBI:

let
    Source = Web.Page(File.Contents("multiple_tables.html")),
    #"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Column1", "Column2", "Column3"}, {"Col1", "Col2", "Col3"}),
    #"Filtered Rows" = Table.SelectRows(#"Expanded Data", each ([Caption] <> "Document") and ([Col1] <> "Col1"))
in
    #"Filtered Rows"

Is there a way to achieve this effect in less than 10 lines of code using Python/Pandas plus some html parser opensource library? Or should I resign myself to writing lower level code to handle this?

With a bit of parsing via BeautifulSoup, we can get a pandas.Dataframe using pandas.read_html() like:

Code:

def get_tables(source):
    elems = iter(BeautifulSoup(source, 'lxml').find_all(['table', 'h1']))
    df = pd.DataFrame(
        pd.read_html(str(next(elems)), header=0)[0].iloc[0].rename(h1.text)
        for h1 in elems)
    df.index.names = ['Caption']
    return df

Test Code:

import pandas as pd
from bs4 import BeautifulSoup

with open('test.html', 'r') as f:
    print(get_tables(f))

Results:

         Col1  Col2  Col3
Caption
Group 1  ValA  ValB  ValC
Group 2  ValP  ValQ  ValR