Creating dummy variables I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable.
However, the dates in the dataset are displayed as long descriptors such as "Sunday 21st May 2005" in a column labelled v584 .

How can I create a dummy variable to say 0 = before Sunday 29th May 2005, 1 = after Sunday 29th May 2005 from the dates listed?

And how can I include only specific date (E.g., only including the dates a week before "Sunday 29th May 2005" in 0 and only the dates a week after in 1)?

The lubridate package provides functions to easily convert character strings into dates in a sensible way. For example, the dmy function can parse strings in a variety of formats so long as the string is in the order of day-month-year. The lubridate functions handle all sorts of separators and extraneous characters.
As for your dummy variable, after you convert your v584 column into dates, just create a new column:
new_column <- as.integer(v584 > dmy("Sunday 29th May 2005"))


Sentiment analysis of tweets (Train model on a labelled dataset and use on some other unlabelled data) I have a huge amount of tweets on a particular topic say 'ABC' and the data is not labelled. I want to perform multi-class sentiment analysis of these tweets. I tried many unsupervised clustering techniques like Kmeans, DBScan, Agglomerative clustering from sklearn but the max silhoutte score that I have reached is 0.31 and the kmeans gives large negative score. I have performed cleaning and encoding of tweets using Bert embeddings, Word2Vec but nothing seems to change.
Suppose I used some other labelled multiclass dataset and build a classifier and then use that classifier to identify sentiment in my target data, will it be good enough? Is this approach correct and logical?
I have found these general speech datasets. Will they suffice my purpose of getting correct sentiments for the "ABC" tweets dataset?
I found this another emotion dataset related to tweets.

The natural approach is to use a labelled dataset and a supervised learning technique. You can start with something simple, like using tf-idf for feature generation and train a simple logistic regression model.
I think this is the first thing you should try, I see it more likely to succeed than the unsupervised techniques, and it is simple enough.


Intutitively what advantages are there to use more data than absolutely necessary while fitting a linear regression? If I have p features in a vector, I can get a unique solution for the weights by using just the p independent rows(data points) in the input, why should then I be using more data points than absolutely necessary?
What advantage does it offer?

If your sample(a subset of the population) is able to represent your population, then it isn't required to use all your data. In other words, if your model can extract all the possible patterns from your sample, then it is good enough. It will also save you from time and space complexity while training. But, let's say your model1 is the one that trained with a subset of your all data, and model2 is trained using all the data you have (except validation and test sets). It is possible that while you did a performance evaluation, your test set hasn't perfectly shuffled and some type of the samples (or samples belong to a specific pattern) weren't in that test set, thus your accuracy was high in model1. In other words, the probability of your population's not including a specific pattern is lower than or equal to the probability of your subset's not including a specific pattern. Thus, if we imagine that your test data was poorly shuffled or your population still somehow includes specific types of data (or pattern) then it will perform better in future data. Unless you are 100% sure that your sample can fully represent your population (which is almost impossible most of the time), then it is always better to use more data which will reduce the chance of spurious results. It is also possible that your model2 can perform worse than model1 if you have outliers in it.
To conclude, unless you observe that using more data gives a rise to loss of performance, then always use all the data by considering your test set's not being perfect and subset's always containing less than or equal characteristics about your whole population. The model2 statistically guarantees you higher performance than model1 in unseen data if model2's data is outlier-free. Because it will learn better to generalize your data than model1.


How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning) I have created and analyzed around 16 machine learning models using WEKA. Right now, I have a CSV file which shows the models' metrics (such as percent_correct, F-measure, recall, precision, etc.). I am trying to conduct a (modified) student's t-test on these models. I am able to conduct one (according to THIS link) where I compare only ONE variable common to only TWO models. I want to perform a (or multiple) t-tests with MULTIPLE variables and MULTIPLE models at once.

As mentioned, I can only perform the test with one variable (let's say F-measure) among two models (let's say decision table and neural net).

Here's the code for that. I am performing a Kolmogorov-Smirnov test (modified t):

from matplotlib import pyplot
from pandas import read_csv, DataFrame
from scipy.stats import ks_2samp

results = DataFrame()
results['A'] = read_csv('LMT (f-measure).csv', header=None).values[:, 0]
results['B'] = read_csv('LWL (f-measure).csv', header=None).values[:, 0]
print(results.describe())
results.boxplot()
pyplot.show()
results.hist()
pyplot.show()

value, pvalue = ks_2samp(results['A'], results['B'])
alpha = 0.05
print(value, pvalue)
if pvalue > alpha:
    print('Samples are likely drawn from the same distributions (fail to reject H0)')
else:
    print('Samples are likely drawn from different distributions (reject H0)')

Any ideas?

This is a simple solution to my question. It only deals with two models and two variables, but you could easily have lists with the names of the classifiers and the metrics you want to analyze. For my purposes, I just change the values of COI, ROI_1, and ROI_2 respectively.

NOTE: This solution is also generalizable.
How? Just change the values of COI, ROI_1, and ROI_2 and load any chosen dataset in df = pandas.read_csv("FILENAME.csv, ...). If you want another visualization, just change the pyplot settings near the end.

The key was assigning a new DataFrame to the original DataFrame and implementing the .loc["SOMESTRING"] method. It removes all the rows in the data, EXCEPT for the one specified as a parameter.

Remember, however, to include index_col=0 when you read the file OR use some other method to set the index of the DataFrame. Without doing this, your row values will just be indexes, from 0 to MAX_INDEX.

# Written: April 4, 2019

import pandas                       # for visualizations
from matplotlib import pyplot       # for visualizations
from scipy.stats import ks_2samp    # for 2-sample Kolmogorov-Smirnov test
import os                           # for deleting CSV files

# Functions which isolates DataFrame
def removeColumns(DataFrame, typeArray, stringOfInterest):
    for i in range(0, len(typeArray)):
        if typeArray[i].find(stringOfInterest) != -1:
            continue
        else:
            DataFrame.drop(typeArray[i], axis = 1, inplace = True)

# Get the whole DataFrame
df = pandas.read_csv("ExperimentResultsCondensed.csv", index_col=0)
dfCopy = df

# Specified metrics and models for comparison
COI = "Area_under_PRC"
ROI_1 = "weka.classifiers.meta.AdaBoostM1[DecisionTable]"
ROI_2 = "weka.classifiers.meta.AdaBoostM1[DecisionStump]"

# Lists of header and row in dataFrame
#  `rows` may act strangely
headers = list(df.dtypes.index)
rows = list(df.index)

# remove irrelevant rows
df1 = dfCopy.loc[ROI_1]
df2 = dfCopy.loc[ROI_2]

# remove irrelevant columns
removeColumns(df1, headers, COI)
removeColumns(df2, headers, COI)

# Make CSV files
df1.to_csv(str(ROI_1 + "-" + COI + ".csv"), index=False)
df2.to_csv(str(ROI_2 + "-" + COI) + ".csv", index=False)

results = pandas.DataFrame()
# Read CSV files
# The CSV files can be of any netric/measure, F-measure is used as an example
results[ROI_1] = pandas.read_csv(str(ROI_1 + "-" + COI + ".csv"), header=None).values[:, 0]
results[ROI_2] = pandas.read_csv(str(ROI_2 + "-" + COI + ".csv"), header=None).values[:, 0]

# Kolmogorov-Smirnov test since we have Non-Gaussian, independent, distinctive variance datasets
# Test configurations
value, pvalue = ks_2samp(results[ROI_1], results[ROI_2])
# Corresponding confidence level: 95%
alpha = 0.05

# Output the results
print('\n')
print('\033[1m' + '>>>TEST STATISTIC: ')
print(value)
print(">>>P-VALUE: ")
print(pvalue)
if pvalue > alpha:
    print('\t>>Samples are likely drawn from the same distributions (fail to reject H0 - NOT SIGNIFICANT)')
else:
    print('\t>>Samples are likely drawn from different distributions (reject H0 - SIGNIFICANT)')

# Plot files
df1.plot.density()
pyplot.xlabel(str(COI + " Values"))
pyplot.ylabel(str("Density"))
pyplot.title(str(COI + " Density Distribution of " + ROI_1))
pyplot.show()

df2.plot.density()
pyplot.xlabel(str(COI + " Values"))
pyplot.ylabel(str("Density"))
pyplot.title(str(COI + " Density Distribution of " + ROI_2))
pyplot.show()

# Delete Files
os.remove(str(ROI_1 + "-" + COI + ".csv"))
os.remove(str(ROI_2 + "-" + COI + ".csv"))


Error: An operation has `None` for gradient with categorical_crossentropy I am trying to train my discriminator network using Keras with TensorFlow backend.
The network is meant to classify the input into one of the 9 output labels. I am passing a 2D input (height, width, no channels) and a one-hot vector for the output. I was able to train the network independently using fit(). However, now that I have switched to train_on_batch, it is giving me the error mentioned.

This is my discriminator code:

def build_discriminator(time_steps, feature_size, input_spectrogram=None):
    spectrogram = Input(shape=(time_steps, feature_size))
    # spectrogram = tf.placeholder(tf.float32, shape=(None, time_steps, feature_size))
    layer0 = Reshape((time_steps, feature_size, 1))(spectrogram)
    layer1 = Conv2D(32, kernel_size=(3,3), padding='same')(layer0)
    #model.add(LeakyReLU(alpha=0.01))
    layer2 = MaxPooling2D(pool_size=(4,4))(layer1)

    layer3 = Conv2D(16, kernel_size=(3,3), padding='same')(layer2)
    #model.add(LeakyReLU(alpha=0.05))
    layer4 = MaxPooling2D(pool_size=(4,4))(layer3)

    layer5 = Conv2D(16, kernel_size=(3,3), padding='same')(layer4)
    #model.add(LeakyReLU(alpha=0.05))
    layer6 = MaxPooling2D(pool_size=(4,4))(layer5)

    layer7 = Flatten()(layer6)
    layer8 = Dense(16)(layer7)
    prediction = Dense(9, activation = 'softmax')(layer8)
    # prediction = Dropout(0.1)(layer9)

    model = Model(spectrogram, prediction)

    opt = optimizers.Adam(lr=0.002, beta_1=0.5)
    model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

    return model

This is the code that trains the discriminator:

    x_real = batch_x[:half_batch, :, :]
    labels_real = batch_labels[:half_batch]
    d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))
    # generate 'fake' examples
    x_fake, labels_fake = generate_fake_samples(batch_x[half_batch:, :, :], batch_labels[half_batch:], generator)
    # update discriminator model weights
    d_loss2, _ = discriminator.train_on_batch(x_fake, to_categorical(labels_fake, num_classes=9))
    # update the generator via the discriminator's error
    g_loss, acc = gan.train_on_batch([batch_x, batch_targets], to_categorical(batch_targets, num_classes=9))

It is throwing the error on this line:

d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))

The error traceback:

  File "gan.py", line 126, in train
    d_loss1, _ = discriminator.train_on_batch(x_real, to_categorical(labels_real, num_classes=9))
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1513, in train_on_batch
    self._make_train_function()
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 316, in _make_train_function
    loss=self.total_loss)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 75, in symbolic_fn_wrapper
    return func(*args, **kwargs)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/optimizers.py", line 504, in get_updates
    grads = self.get_gradients(loss, params)
  File "/home/pallavi/anaconda3/lib/python3.7/site-packages/keras/optimizers.py", line 93, in get_gradients
    raise ValueError('An operation has `None` for gradient. '
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.

I am using tensorflow-gpu 2.0.0 with keras 2.3.1.
Could someone please help me understand where I am going wrong? Thanks is advance!

This link might be helpful to you. you might be missing something in your code.

https://github.com/keras-team/keras/issues/12521


Retraining EfficientNet on only 2 classes out of 4 EfficientNet model was trained on ~3500 images for a 4-class classification: A, B, C and Neither – with accuracy of 0.985 – by someone else, not me. I'm quite new to ML.  So we have this model, and it works pretty well.

As more real-world images are coming in from the users, we see more errors. Is it possible to improve the performance a bit by adding more training examples?

Collected and manually sorted 35 images of class A and 425 of class "Neither". Does it even make any sense with so small dataset?

I have put images into a folder with 4 subfolders, one per each class. Two are empty, two have images. Using Keras ImageDataGenerator flow_from_directory() method with validation/training = 0.2/0.8 ratio.

model.fit() training shows low loss and accuracy close to 1, while validation loss is high and accuracy is around 0.21 — 0.25 only.

Model performance after my "training" is totaly ruined.

Is this approach to retraining with a little set of images wrong?

How can I "add", not "replace" the model "skills" with a new added little set of images?

Maybe I should have trained a fresh model initialised with the ImageNet weights as in the initial training, and just add the new images to the set of images used, again, in the initial training?

The problem you describe is called transfer learning, which uses uses a pretrained model (such as EfficientNet) that is trained on an existing dataset and applies it to a different dataset. Since this method uses a pretrained model, the new dataset on which the transfer learning needs to happen doesn't need the same number of training examples compared to when the model would be trained from scratch.

In your specific case, you could look into using an existing EfficientNet model that is pretrained of a different dataset (such as ImageNet) and add one or two layers to make the model specific for your dataset (i.e. number of output neurons equal to the number of classes you are trying to classify). You can then train the pretrained model on your specific dataset, however, make sure to start by fixing the parameters of the pretrained model to make sure the model keeps the knowledge it has learnt.


Time series forecasting with RNN(stateful LSTM) produces constant values I have a time series daily data for about 6 years(1.8k data points). I am trying to forecast the next t+30 values, Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values.
I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.
Is this the correct way to forecast time series with LSTMs?
How can this behaviour be explained?
Will this be the case even for a time series with great seasonality?
Is this behaviour expected even for a very large time series data?
Should I rather try to train the network with Y matrix having 31st day to 60th day values for the same X?
My train data looks something like this:
array([[-0.35811423, -0.22393472, -0.39437897, ..., -0.36718042, -0.37080689, -0.35267452], [-0.22393472, -0.39437897, -0.13327289, ..., -0.37080689, -0.35267452, -0.2030825 ], [-0.39437897, -0.13327289, -0.1532185 , ..., -0.35267452, -0.2030825 , -0.25294651]

Architecture: Input LSTM layer (20 neurons) 1 hidden lstm layer 20 neurons 1 output dense layer, batch size as 1. Stateful lstm, I reset model states after each epoch.

Is this the correct way to forecast time series with LSTMs?

  Train data independent matrix (X)=Sequences of previous 30 day values Train (Y)=The 31st day value for each of previous 30 day values

You method could work in theory. But typically, for vanilla LSTM, the setup looks like $x = [x_0, x_1, ..., x_{n-1}]$ and $y = [x_1, x_2 ..., x_n]$. I suspect you will get much better result if you do it this way.

Highly recommend Colah's excellent explaination if you haven't read it already.

  I followed the following methodology to forecast: Y for t+1 is first forecasted, Then X matrix row is shifted by 1 day and the forecasted Y is appended to the end of this row, then use this row to predict t+2 value and continue. However in each sequence after (usually) t+3 days the forecasted values become constant for the rest of the t+n days.

This is not recommended because any error in prediction is compounded over time. please find my answer in the following post for a detailed explanation
Is time series multi-step ahead forecasting a sequence to sequence problem?

How can this behaviour be explained?

If you are getting constant prediction value, the chance is you are having so-called "gradient explosion problem", try to clip the gradient before backprop might be the solution.

Correction. Had a bit better thinking about it.  it is not a gradient explosion problem here. More likely it is due to saturated neuron. Because LSTM uses sigmoid function, if the input is not properly scaled or weight is initialised inproperly, it could cause neuron saturation very quickly.


Find matches between columns and manipulate data based on match I have a pandas dataframe that contains two columns, components, and sub-components (first table below). I would like to add a product column as shown below (second table). The problem is,  some (not all) components are actually sub-components (B is also contained in the sub-component column in this simplified example). I would like some help/hints to write code in python to create this product column.

I envisage something like this but am finding it difficult to get started:

Iterate through Component column look for matches in the sub-component column, if no match is found write Component name to row in Product column - where a match is found in the sub-component column, look for the name of Component corresponding to sub-component and write it to the next row in the column Product. The line containing the "component in the sub-component column" can then be deleted.

Component           Sub-component
A                   a
A                   **B**
A                   c
B                   d
B                   e
B                   n
B                   a
B                   v
B                   c
B                   i
C                   g
C                   s
C                   g
C                   a
C                   c
C                   i
C                   q
C                   g
C                   v
C                   b
C                   l

Product Component           Sub-component
A       A                   a
*A      A                   **B***
A       A                   c
A       B                   d
A       B                   e
A       B                   n
A       B                   a
A       B                   v
A       B                   c
A       B                   i
C       C                   g
C       C                   s
C       C                   g
C       C                   a
C       C                   c
C       C                   i
C       C                   q
C       C                   g
C       C                   v
C       C                   b
C       C                   l

One possible way to solve this would be to create a dictionary of which components should be changed and then use replace. So in the example above, we want to create a dictionary {'B': 'A'} since all B should be replaced by A in the new Product column.

This can be done as follows:

components = df['Component'].unique()
d = df[df['Sub-component'].isin(components)].set_index('Sub-component')['Component'].to_dict()

and then we can create the new column:

df['Product'] = df['Component'].replace(d)

Resulting dataframe:

    Component   Sub-component   Product
0   A           a               A
1   A           B               A
2   A           c               A
3   B           d               A
4   B           e               A
5   B           n               A
6   B           a               A
7   B           v               A
8   B           c               A
9   B           i               A
10  C           g               C
11  C           s               C
12  C           g               C
13  C           a               C
14  C           c               C
15  C           i               C
16  C           q               C
17  C           g               C
18  C           v               C
19  C           b               C
20  C           l               C


Predict the loss amount for insurance policies using the historical trends and features? Anyone who has worked with insurance policy datasets, please guide me to an appropriate dataset for this problem .

Also, what is this 'loss amount' and what model will be appropriate for it?

Loss amount - "In insurance. The diminution, destruction, or defeat of the value of, or of the charge upon, the insured subject to the assured, by the direct consequence of the operation of the risk insured against, according to its value in the policy, or in contribution, for loss, so far as its value is covered by the insurance" (not exactly my area of expertise).

If you are looking for "insurance policy datasets", you can try the following link:

insurance data on data.world

You would probably need some regression model, but you are not providing enough information for more specific answer.


Unsupervised text classification with R/Python I am relativity new to machine/deep learning and NLP. As a part of my Phd thesis I have scraped vast number of job vacancies (most of them are in Polish, and about 10% are in English ones) and then extracted required skills/competencies. As an output I got a vector of strings contains single skills/competences. I performed initial preparation (removed extra spaces, special symbols, stop words etc.) and built then a frequency distribution table to figure out the most demanded skills. The most commonly required skills were as follow:
prawo jazdy kat b ## driver license
umiejętność pracy w zespole ## team work
wykształcenie wyższe ## higher education
czynne prawo jazdy kat b ## driver license
znajomość pakietu ms office ## MS Office
prawa jazdy kat b ## driver license
prawo jazdy kategorii b ## driver license
fluent english
doświadczenie na podobnym stanowisku ## work experience
dobra znajomość języka angielskiego  ## English
excellent english skills
doświadczenie ## work experience

As you may see some of these competencies are quite similar. The problem is that job offers were provided by a different employers. European classification of skills/competences qualifications and occupations define about 3500 different competencies for instance.
My aim is to create trivial classifier using logistic regression, SVM or random forest that would be able to classify skills in real time using pre trained model. The problem is that I can not label vector of unique skills because of vast number of skills within its synonyms. I spent a while trying to figure out a solution.
So, the idea was to perform cluster analysis and to put similar skills into separate groups (clusters) and then label this clusters. In this case I would have much less group to label. Moreover, I would have a lot of synonyms within each group, which would increase precision of classification (I guess). Using labelled data set I would be able to trained model, and then to classify future job vacancies (skills). However, as I said I am new to machine learning, so I am not sure whether my solution would be good enough.
So, I am looking for advice/tutorials/links that would help me to solve my problem rather than for complete solutions. I can use both R and Python. Would be appreciated for any help.

I can think of two approaches.

You could use term-frequency/inverse-document-frequency (tf-idf) to cluster the vectors. Personally, I would start by first clustering the full text of the original job vacancies and then use this to assign clusters to the vectors. I have the feeling that it will outperform clustering directly the vectors.
There are implementations both in R and in Python. I would use Python for clustering in a language other than English. You can use TfidfVectorizer from sklearn.
I've a post on medium with an example of clustering wikipedia articles in Python if you are interested.

You can start by a list of key/basic skills and then try to match your vectors to the items of this list. For defining a distance between words/phrases you can:
a. use Jaccard index (ex. see this)
b. use SpaCy in Python to find descriptions that are similar. I believe there is a language model for Polish. I haven't tried it but I would use something that is described in "The Beginner’s Guide to Similarity Matching Using spaCy". Since you are trying to find the similarity between phrases (not single words) you might also  want to consider this stackoverflow discussion.


What can weka do that python and sklearn can't? I would like to build a variety of classification and regression decision trees. My use case focuses on extraction and communication of decision rules. Previously weka was used in my organisation for decision tree learning. What can weka do that Python or Sklearn can't?
I currently use pandas, numpy, scipy, and sk-learn and other libraries for the majority of my workflow.

Weka's decision trees are from the Quinlan family, whereas sklearn uses CART.
The most notable difference is that Quinlan trees aren't restricted to binary splits: a categorical column will be split into subtrees for each level.
Another is how missing values are dealt with, but there are some differences in individual implementations, so it's not straightforward to compare the two branches.
https://stackoverflow.com/q/9979461/10495893
TDIDT Decision Trees algorithm
Otherwise, I expect the main real difference is in whether it's easier to deal with python or java.  If you want to extract decision rules, you may be looking to post-process a decision tree; I know of skope-rules to do this in python, but not whether such a thing is easy in weka.


Can we use Stop Words while using multinomial navies theorem? I'm collecting Twitter tweets for sentiment analysis. I chose to use Multinomial Navies theorem for finding the sentiment.

I found some examples of SVM theorem making use of stop words. My question is: can we apply stop words in Multinomial Cavies theorem or not?

The basic task here I get is sentiment analysis of tweets. So for this, we can first extract features and then use a classifier. Stop words are those which are present for grammatical purposes but do not add any value/meaning to the sentence.
So, yes you can remove the stop words from the tweet, extract the features and then pass it to the classifier. You can get already existing stop words from

from nltk.corpus import stopwords
stop_words = stopwords.words('english')

Removing the stop words doesn't always improve the accuracy.
Take a look at this as it is a very similar problem statement as yours. Hope it helps!


How to use "sann" function in R to solve graph problems? I came across a package in R which has a function called sann for simulated annealing.

sann uses parameters fn and gr to optimize and to select new points, respectively.

For something like the max-clique or max-stable set problems, fn would be a summing function, but it's less clear how one would formulate gr to fix these graph computations. In these cases, how would gr "select"?

First, some clarification on terminology.

A package in R is a collection of R functions, data, and compiled code in a well-defined format.

SANN (sann) is not a package. Depending on which package you're using, sann is either a function or, more often, a method used within an optimization function.

Packages containing sann include optim, trustOptim, consPlan, and constrOptim.

In the package optim, the sann method is implemented as:

> func <- function(x){
+   out <- (x[1]-2)^2 + (x[2]-1)^2
+   return <- out
+   }>
> optim(par=c(0,0), fn=func, gr = NULL,
+       method = c("SANN"),
+       lower = -Inf, upper = Inf,
+       control = list(), hessian = T)

As you said, for the "SANN" (sann) method gr is used to generate a new candidate point. If it is NULL a default Gaussian Markov kernel is used.

Now in your use case -- the case of a graph -- what you probably want to do is to use par and value to pass values to fn and gr. This is a nice feature of this implementation of SANN in optim which is covered a little more than half way  through this documentation page.

par is the best set of parameters found and value is the value of fn corresponding to par.


What is the defining Set in NLP I am reading the paper Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings here is the pdf.

On page 6, we read:

Step 1: Identify gender subspace. Inputs: word sets W , defining sets D_1 , ..., D_m.

However, they paper before and after this statement does not mention what these defining sets are? Can anyone give me a definition or description of these sets?

Thank you.

If you read the following sentence at the first line of section 6:

  The debiasing algorithms are defined in terms of sets of words rather than just pairs, for generality, so that we can consider other biases such as racial or religious biases.

$D_1, D_2, \ldots, D_m$ are the set of words, in general, to be considered in the de-biasing algorithm, instead of considering a pair of words such as Computer Programmer and Houseworker for the de-biasing genders of man and woman. Hence, one of $D$s could be {'Computer Programmer', 'Houseworker'}, here, as an example.


How to combine data having similar distribution? I have a collection of time series data with data points of around 2 years of daily data. I am thinking of a way to increase the number of data points in it so that the neural network gets a better understanding of the fluctuations in the data. I am suggesting a hypothesis where I try to cluster similar time-series data following similar distribution, in order to increase the number of data points fed into the neural network. Is this a correct way to approach the problem? If so, on what basis do I combine these similar data together so that I can feed it into the neural network/model?

Grouping values together is commonly called quantization or binning. Binning increases the signal-to-ratio in data.

Generally, time-series data is binned by time (also called rolling-up). For example, rolling-up all values over a range of days into a single month value.


In linear regression, is there anything I can do if the coefficient for one of the features is unrealistic/inappropriate? I'm building a simple linear regression model that predicts Home Price using Square Footage, Number of Bed(s), and Number of Bathroom(s).

After creating the model, I noticed that the coefficients for Square Footage and Number of Bed(s) were positive, which makes sense since Home Prices increases as Square Footage/Number of Bed(s) increases. However, the coefficient for Number of Bathroom(s) was negative, which makes no sense since Home Price does not decrease as Number of Bathroom(s) increases!

My question is this: is the negative coefficient something I can "fix" by incorporating more data into my model, or is this just a quirk in linear regression?

What could be happening here is multicollinearity - where some of your predictors are correlated. In this case, it is not unreasonable to think that square footage, number of beds and number of bathrooms are all positively correlated with each other. This is not a big deal if your main goal is to just predict house prices, but what multicollinearity does in essence is that it makes it 'harder' to estimate the actual coefficient values, so your interpretations of the coefficients can't carry as much weight. I suspect that using more data will indeed reduce the chance of this happening.


How to get two boxplots on same axis? I have two dataframes (both of different lengths, etc) but they have exact same columns and I would like to explore the distribution of one of these columns by plotting a boxplot.

However, how do I get it on the same axis? And how can I set the y-axis for this? The below code puts the boxplots side by side in separate graphs but I would like them to have the same axis so I can compare them easier.

f, axes = plt.subplots(1, 2)
sns.boxplot( x="status",y="assets" ,data=df1, palette="Set3",ax=axes[0])
sns.boxplot( x="status",y="assets" ,data=df2, palette="Set3",ax=axes[1])
fig.tight_layout()
plt.show()

So the problem is how to visualise your box plots so that they appear in the same plot (axes).
To do this, it is simply a minor alteration to your code.
f, axes = plt.subplots(1, 2)
sns.boxplot(x="status",y="assets" ,data=df1, palette="Set3",ax=axes[0])
sns.boxplot(x="status",y="assets" ,data=df2, palette="Set3",ax=axes[0])
fig.tight_layout()
plt.show()

The alteration is on the data=df2 line.


Similarity of words using BERTMODEL I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model.
I do the following:

from transformers import BertModel
hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

where

token_ids with Shape : [1, 4, 47]
attn_mask with Shape : [1, 4, 47]
seg_ids with Shape : [1, 4, 47]

but I have an error :

TypeError                                 Traceback (most recent call last)
<ipython-input-74-5fa632122cc7> in <module>()
      1 from transformers import BertModel
----> 2 hidden_reps, cls_head = BertModel(token_ids , attention_mask = attn_mask , token_type_ids = seg_ids)

TypeError: __init__() got an unexpected keyword argument 'attention_mask'

How can I fix this error??

and how can I find the word embedding and the similarity of a given word using this model?

I could not find any tutorials or similar codes for this task.

First of all, I think you are confused with pretrained and finetuned.

BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text.

BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other datasets : the knowledge does not apply.

You have a custom dataset. So you don't want to use a finetuned model, but you want to still use a pretrained model (to have access to the general knowledge of BERT).

If you use a BERT model from scratch (not pretrained), you're basically creating an empty model, randomly initialized : it's useless.

Now that we have clarified this, let's see your problem.

What you are doing is calling the constructor of BertModel with the arguments of the forward pass. You first need to create the model, and then use the forward pass :

from transformers import BertModel, BertConfig
config = BertConfig(...)    # Here your parameters to initialize your BERT as you want, such as number of heads, etc...
model = BertModel(config)
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

BUT, as I explained in the first part, you should not do this : it's completely useless and your results are going to be random. You want to use a pretrained BERT, to have some meaningful results :

from transformers import BertModel
model = BertModel.from_pretrained('bert-base-uncased')
hidden_reps, cls_head = model(token_ids, attention_mask=attn_mask, token_type_ids=seg_ids)

Note : Your results can be improved further if you finetune your BERT model on your custom dataset.

Final point : Don't reinvent the wheel

People already tried to use BERT for word similarity. Instead of implementing this from scratch, using only a pretrained model, potentially adding bug to your own implementation, just use some already existing code !

I'm not sure about NER, but for word similarity, you can take a look at BERT Score. Their model is already finetuned for word similarity, so it's likely that you will get higher score using this approach, and also much easier path to take.


Terminology Question: Place Coding While reading the paper "Backpropagation Applied to Handwritten Zip Code Recognition", one
of the original papers attributed to the development of convolutional neural networks,
on page 3, section 3.1 the authors write (emphasis mine):
The remainder of the recognition is entirely performed by a multilayer network. All of the connections in the network are adaptive, although heavily constrained, and are trained using backpropagation. This is in contrast with earlier work (Denker et al. 1989) where the first few layers of connections were hand-chosen constants implemented on a neural-network chip. The input of the network is a 16 by 16 normalized image. The output is composed of 10 units (one per class) and uses place coding.
What does place coding mean? A Google search of this term yielded nothing conclusive. Any insights appreciated.

"Place coding" == "one-hot encoding"
This is the conventional way of encoding a category at the output of a classifier: for N categories, use an N-dimensional vector where the k-th component is "turned on" and all other turned off to encode the category k.
In general, the values for "on" and "off" do not need to be 1 and 0 (or +1 and -1). They could be anything.
The alternatives to "place code" are "coarse coding" and "distributed representation" of which "error-correcting codes" are a special case.
The classifier in the paper uses a form of distributed representation in which similar characters (like o, O, and 0) are represented by similar bit patterns.
That way, when the system is not sure if the character is o, O, or 0, it will produce an output bit pattern that still identifies the character as one of those 3 categories, and leave it up to the post-processing (e.g. a language model) to resolve the ambiguity.


Machine learning algorithm for Low dimension input to high dimension output I am plaining on training a network for body generation, i.e. given some specific measurement,(5 features)  the output will be the a set of vertices representing the obj of the bodies. I am wondering what kind of machine learning algorithm is best suited for this scenario.

If I'm understanding correctly you want to model a set of data (in a network form in this case) for which you give 5 input parameters and you get N objects as output of your model, with N>5.

A network basically classifies an object based on the input parametersand outputs the predicted value. Putting a basic network generation, you can classify what car you are looking at based on 3 certain features: "motor size", "number of seats" and "country where the car was designed". From the network you can extract the model and production year (if enough data is provided)and predict what car you have. However, due to the method you are obtaining an output with smaller dimensionality than the input (usually N_output=1).

However, you can associate certain information to that output. In the previous example it could be a technical datasheet of the car model the network outputs. This technical datasheet can contain many features that were not considered in the model (network), which result in N_output>N_input.

The critical step in this case is selecting correctly what output do you want for your network (classification algorithm) and how do you want to associate that output to further insight of that classification.

The algorithm itself to create the network depends on the data and what output do you want. For example, in a basic KNN of the previous case, you select the output as a "compound weighed effect" of all the inputs that result in the network.


Configuring Incremental XGBoost model I have a large dataset which can't be loaded in memory, hence I decided to use incremental learning using Xgboost. What I have done currently is:

Tuned num_boosting_rounds using a chunk of data
Set early stopping rounds to a value (60) << num_boosting_rounds to prevent overfitting

When my training code runs, it runs for a 1000 rounds on the first two chunks optimizing the loss function. It then stops at 60 rounds for each subsequent chunk as the best value for loss function was observed in the 1000th round in the 2nd Chunk. Is this the correct way to configure the Incremental model? Would this result in my model being sub-par owing to early stop for a majority of training chunks.

for idx,df in enumerate(df_pointer):
    num_round = 1000
    early_stopping_rounds = 60
    param = {'max_depth':5, 'eta':0.02, 'silent':1, 'objective':'binary:logistic', 'eval_metric':'logloss', 'max_delta_step':4, 'scale_pos_weight': 4}
    dtrain, deval = getDMatrixSplit(df)
    watchlist  = [(deval,'eval')]
    bst = xgb.train(param, dtrain, num_round, watchlist,
    early_stopping_rounds=early_stopping_rounds, xgb_model=xgb_model)
    xgb_model = self.model_path +'/xgb_%s_%s.model'%(ml_algo, idx)
    bst.save_model(xgb_model)

The "optimal" stopping point for XGBoost really depends on the data you feed into it.

Using Chunk N and Chunk N+1 for instance, consider the two scenarios:

data in Chunk N and Chunk N+1 is very different - it's likely
that the optimal stopping points for each chunk are very different.
data in Chunk N and Chunk N+1 is very similar - it's likely that the optimal stopping points for each chunk are quite similar.

I would suggest using a different approach - so for each chunk:

use the xgb.cv function to first determine the optimal stopping point
use the results from xgb.cv in xgb.train [i.e. set nrounds = xgb.cv$best.iteration]

As a side-note, have you explored any other techniques / tools to handle large data sets? In R, there are packages designed to tackle similar problems - e.g.bigmemory and ff.


Same input size but cannot fit the model in keras I am trying to fit this model in keras but getting this error :

 Namespace(batch_size=32, epoch=10, num_classes=2)
Start
Train loading
<class 'numpy.uint8'>
Test loading
<class 'numpy.ndarray'>
Validation loading
done_loading
Traceback (most recent call last):
  File "main.py", line 67, in <module>
    validation_data=(x_val, Y_valHot)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 1581, in fit
    batch_size=batch_size)
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 1414, in _standardize_user_data
    exception_prefix='input')
  File "/tools/anaconda3/envs/py35/lib/python3.5/site-packages/keras/engine/training.py", line 153, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking input: expected data to have shape (None, 224, 224, 3) but got array with shape (9730, 244, 224, 3)

The function used to fit the model is

model.fit(X_train, Y_train,
          batch_size=batch_size,
          nb_epoch=nb_epoch,
          shuffle=True,
          verbose=1,
          validation_data=(X_valid, Y_valid),
          )

Can someone tell me where I went wrong.

ValueError: Error when checking input: expected data to have shape (None, 224, 224, 3) but got array with shape (9730, 244, 224, 3) it clearly says that while preprocessing that either X_train or X_valid have (9730, **244**, 224, 3) instead of (9730, **224**, 224, 3)


Unsupervised learning to identify most common basketball plays from spatial data The NBA has a system called Sports VU that tracks x-y coordinates of every player and the ball every 1/10th of a second for every game of the 2013-2014 and 2014-2015 seasons. With some fancy web scraping I now have access to this data and -- because I'm such an avid fan of the NBA -- I would like to identify each team's most common plays. Assume I don't have any knowledge about each team's plays beforehand (so I don't think supervised learning would work here). What would be the best unsupervised learning techniques to use?

If I could trace each player's path over the course of the play, I imagine the problem would be similar to what you would see with image recognition/classification. Anyways, should I use PCA, some kind of neural  network? I understand this is a very broad question -- I don't need to know how to code it (I'm a proficient coder and machine learning practitioner); I'm just looking for high level unsupervised machine learning details.

This is a really interesting problem! Like most really interesting problems, you're unlikely to find an out-of-box solution for this, but I think the field of graph/subgraph similarity has some promise here. I'll go into more detail, but, at a high level, I think you can view players' paths during a play as a collection of five traversals through x,y space, with a vertex being any x,y point you have available (presumably there's some level of time granularity here), and an edge describing a player's movement from one point in x,y space to the next over time. It should be possible to cluster your data using a similarity metric (e.g., see Koutra et al., 2011 for a nice overview). Then, using your own domain expertise, you should be able to identify whether the clusters you've derived have some real-world meaning in basketball.


Learning similarity of representations I am interested in a framework for learning the similarity of different input representations based on some common context. I have looked into word2vec, SVD and other recommender systems, which does more or less what I want. I want to know if anyone here has any experience or resources on a more generalized version of this, where I am able to feed in representations on different objects, and learn how similar they are.

For example: Say we have some customers we are sending different advertisements to, and I would like to create a system to map offers to customers. I am thinking in the lines of creating a customer representation, and a representation of the offers, and feeding them in parallel to a neural network that has a label of whether they acted on the advertisement or not. The idea is that I should be able to locate the best offer for any customer given these representations.

I have looked into siamese networks and word2vec, both are close to what I want. The problem differs slightly in that  for the siamese networks, there are identical parallel networks, which I don't want because my inputs are not equivalent. Word2vec type methodology is also close, but I would want a model to process the inputs on "both sides". A combination of the two, is kind of what I am looking for.

If anyone has any resources on a similar problem statement, I would be very interested in it.

Thanks

You could use a collaborative filtering approach, ie. train a network that learns your customer and offer embeddings in a latent space.
You could randomly initialize a customer embedding matrix (no_of_customers, dimensions_c) and an offer embedding matrix (no_of_offers, dimensions_o).

Your training data would be each click, a customer index, offer index, and a binary variable (0,1) which would be used to lookup the embedding vectors. Then you could concat these vectors and use them as input to a fully-connected layer, or any other architecture you wish to use.


Plotting different values in pandas histogram with different colors I am working on a dataset. The dataset consists of 16 different features each feature having values belonging to the set (0, 1, 2). In order to check the distribution of values in each column, I used pandas.DataFrame.hist() method which gave me a plot as shown below:

I want to represent the distribution for each value in a column with different color. For example, in column 1, all the values corresponding to '0' should be in red color while the values corresponding to '1' in green color and so on. How can I do this? Please help!

There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib patches you can achieve the desired result.

Here's some dummy data to play with:

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randint(low=0, high=3, size=(1000,16)))

Now, here's the magic:

import matplotlib.pyplot as plt

# Plot and retrieve the axes
axes = df.hist(figsize=(12,6), sharex=True, sharey=True)

# Define a different color for the first three bars
colors = ["#e74c3c", "#2ecc71", "#3498db"]

for i, ax in enumerate(axes.reshape(-1)):
    # Define a counter to ensure that if we have more than three bars with a value,
    # we don't try to access out-of-range element in colors
    k = 0

    # Optional: remove grid, and top and right spines
    ax.grid(False)
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)

    for rect in ax.patches:
        # If there's a value in the rect and we have defined a color
        if rect.get_height() > 0 and k < len(colors):
            # Set the color
            rect.set_color(colors[k])
            # Increment the counter
            k += 1

plt.show()


Replace method in pandas not giving expected result I have data as below:

123.12.23.2
110.22.21.23

I want to mask this data as below one

1xx.xx.xx.x

So I tried below code :

 readFile = pd.read_csv("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt",header=None)
 readFile.columns = ['IP']
 readFile['IP']=readFile['IP'].str.replace("(?<! )","X").astype('str')
 readFile

but I gives me data as below one which is not correct:

                                                  IP
0  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...
1  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...
2  XXXXXXXXXXXXXXXXXXX 1XXXXXXXXXXXXXXXXXXXXXXXXX...

I am new to pandas. So where am I going wrong ?

Also, I want to do it without pandas because the platform on which I would be deploying this code may be won't have pandas. So need to be ready for the other scenario as well. Below is my code:

readFiles=open("C:/Users/siddhesh.kalgaonkar/Desktop/data01.txt","r")
 finalValues = re.sub("(?<! ).","X",readFiles)

It gives below error:

>>> finalValues = re.sub("(?<! ).","X",readFiles)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\siddhesh.kalgaonkar\AppData\Local\Programs\Python\Python36\lib\re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

I want to split this data on the basis of delimiter (in case I have multiple columns) and then I have to apply regex logic. Please help me out here.

This is pure(almost) python:

list(map(lambda x: x[0] + '.'.join(['X' * len(c) for c in x[1:].split('.')]), my_df['IP']))

Explanation:
 - Use the map to iterate over each row in my_df['IP'] column.
 - Per each IP value, split into first char and others using the x[0], x[1:]
 notation.
 - Get each part in x[1:] using the split method.
 - For each part get its length and accordingly create a string made of X's.
 - Rejoin this X's string into one string with '.' between them.

If using python2 you dont need the list.


How to use keras flow method? I am building a Convolutional Neural Network with Keras. I want to use my own dataset. This dataset consists of matrices of specified size and filled with 0's which contain a submatrix of specified size and filled with 1's. The submatrix is optional and the goal is to train the NN to predict whether a matrix contains the submatrix or whether it doesnt. To make it more difficult to detect, I am adding various types of noise to the matrices.

Here is a picture of what the matrices look like, the black parts are 0's and the white part are 1's. There is a 1:1 correspondance between the pixels of the image and the entries in the matrix.

Now I want to create a CNN which can classify these images. The easiest approach so far was to use

test_set = test_datagen.flow_from_directory('dataset/training_set',
                                            target_size = (imageSize, imageSize),
                                            batch_size = batchSize,
                                            class_mode = 'binary')

And then I fill the directory training_set with 1000's of images. This works well, but has one major problem:

It is really inefficient, as I'm first creating images, saving them on a hard-dsik, then loading them again. These are a lot of unecessary steps, considering that keras transforms the images back into matrices (or tensors, idk), and my images were initially matrices.

Thus I was thinking about improving the efficiency by having the matrices in a .csv or even having my matrixgenerator.py "feed" a list or array directly into keras. I did some research and found the method flow from Keras which specifies as parameter an input matrix.

Unfortunately this is where I am stuck. I do not understand how to get my matrices into the form keras wants. I also don't know the exact shape the matrices and labels need to have to be able to use the flow method.

It would really help if someone could explain me how to use the flow method/tell me where I find more information on it, or give me a better option to feed my dataset into a keras CNN. Thank you for your help. I'm also ok with using a lower-level tensorflow implementation as long as I understand it.

Assuming you already have resized and other preprocessing your image data into a multi-dimensional numpy array and split the data into training and test.

To use the flow() method. You first want to create a generator using ImageDataGenerator(). The example below DOES NOT DO image augmentation. It will feed only the images in your training set. I added the rescale parameter in case one had not normalized the pixel data.

You then pass your training data to the fit() method of the generator (datagen.fit()).

datagen = ImageDataGenerator(rescale=1./255)
datagen.fit(x_train)

Now you can train your model using the fit_generator() method for your model.

model.fit_generator(datagen.flow(x_train, y_train, batch_size=batchSize),
                    steps_per_epoch=len(x_train) / batchSize, epochs=epochs)

You can do it the manual way:

for epoch in range(epochs):
    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=batchSize):
        model.fit(x_batch, y_batch)

Here's the link to the corresponding Keras documentation: https://keras.io/preprocessing/image/


Feature Selection in Linear Regression I have a insurance dataset as given below. For which I need to build a model to calculate the charges.

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

But I am not sure whether the column "region" can be dropped. Is there any test can be performed to consider only significant variables?

First, transform your columns, then apply linear regression, but do you want to know about the influence of your features on your selected dependent variable?

Read this article: http://www.ritchieng.com/machine-learning-evaluate-linear-regression-model/

It provides great insight at how interpret the coefficients given by the algorithm can be interpreted and further discussed or tweaked.
Read it, then come here for further doubts, you're more than welcomed to do so.


Can I use fit() function with images in Keras? I want to use vgg16 to train a dataset that contains images.

Can I use fit() function instead of fit_generator() in Keras? How?

Yes you can. fit() takes the images in a numpy matrix while fit_generator() takes a generator object (processing images in batch)

Guide for using fit().

Guide for using fit_generator.


Do anomalous input features to autoencoder result in high errors on the corresponding output features? An autoencoder is trained by replicating each training instance to both input and output. However, when predicting for anomaly detection, will the output error be local to the same output feature(s) that were anomalous inputs  e.g. if I have 10 features, and I predict with anomalous input on feature #4, will the high output error be limited to output feature #4 as well, or will high error appear on other outputs? The whole detection becomes more useful if localised to specific features.

The output error will be propagated to the subsequent layers, so if your auto-encoder learns a function over all the features, the anomaly will be weaved across all the features you used as input. One trick is to map specific features (or set of features) to different auto-encoders, and train them as an ensemble. Whether you desire to integrate the specific features anomaly in a global outlier score, or maintain outlier scores for subset of features is then a design choice. For a good use case, see this recent paper.


How to evaluate the performance based on rate data I have the following data:

Goal       Achieved

100        90
150        130
200        175

...

The first column "Goal" is the number which should be done that day, and the column "Achieved" is the number that was actually done that day. Each row stands for a day.

I want to use this data to assess the performance of a worker. Here below I have two ideas:

Method 1

Method 2

Which one of the methods is reasonable and mathematically right?

Or if neither is good, what should I do to evaluate the performance of a worker?

It depends on what you want to achieve, and how you'd define performance.

Suppose it is a running exercise program with 3 exercises in a week with increasing targets.

The first formula calculates the average exercise success, the second formula defines the weekly success, where the exercises are weighted with the goals.

There is no right or wrong, just interpretation of the numbers.

Just to confuse you a bit more: suppose:

Goal: 8, Achieved 1, Goal: 2, achieved 1
Goal: 2, Achieved 0, Goal: 8, achieved 3.

Case 1. beats Case  2. on both instances in formula 1, but loses in formula 2.


Can OLS regression be used to predict from a complete sequence of data? Reading online and following this example from scipy I understand OLS can be used to find data between gaps in a sequence (interpolate?) but I already have a complete sequence and want to predict the future of it (extrapolate, i think). Can OLS be used for that?

Ordinary least squares (OLS) is an optimization method to find the best parameter estimates for linear regression, gradient descent is another.

Regardless of the specific optimization method, linear regression is not appropriate for predicting sequence data. Generally, time series methods are used to predict sequence data.


Your Prediction Model works. What to do after? One of the most crucial skills of a data scientist is not only to be able to build an accurate predictive algorithm but to suggest a set of actions based on that to enhance the goal ratios.

I have build a churn rate predictor for a retail APP which accuracy after testing is about 90%. So now that I know with high accuracy which users will churn out of the APP in the next days, what can I do to try to retain them. Thinking in the overall APP environment the only idea that comes to my mind is:

Push notifications. CRM, use a recommender system to send push notifications based on users profile and micro-moments.

You could do the following:

Find out the most important predictors used by your model; the methods used to identify importance depend on the model used.
For each of the major pedictors try to find out how does each predictor segment the customer population of will-churn vs. will-not-churn. That would be the starting point for framing hypothesis of why these set of customers are churning. Generate reports/info-graphics showing how these predictors impact the tendency to churn. Sometimes predictors are not causes but side-effects of the hidden cause showing up elsewhere. So be open to skepticism about the predictors.
Involve the business/users and have brainstorming sessions to go through each of these hypothesis and evolve the most practical and economic actions that could be taken to reduce churn. Before you get into a discussion, plan for a sufficient amount of time, come up with your best thought actions which you feel will address the situation and have all the details gleaned from your analysis available for sharing in the session.

This is an art more than a science and your ability to understand the domain as well as good communication will help you achieve this goal.


Streamlit widgets doesn't display in Jupyter Notebook I'm trying to create streamlit widgets in Jupyter Notebook but when executing the following code nothing happens.
import numpy as np
import streamlit as st

age = st.selectbox("Choose your age: ", np.arange(18, 66, 1))

According to the following article a SelectBox like this should be created.

But in my code I only get to know that print(age) is 18. What am I doing wrong?

You don't run the code from a notebook. You create a python script with your code and then use this command from a shell: streamlit run yourfile.py
the output will show up in the shell

Reason for this is that streamlit runs on the main thread so it would block the jupyter kernal, you won't be able to use the notebook while the streamlit app is running. However, if you really want to stay inside Jupyter to run the app, you can run the command in a cell, like this !streamlit run yourfile.py


POC - Get an idea to create a Predictive Model I'm trying to look for an idea to create a predictive model having the following data:

Customer_ID - Integer
Catalog_ID - Integer
Country_Code - Integet
Year - Integet
Month - Integer
Day - Integer
Quantity_Purchased - Integer
Product_Purchased - Double

I'm trying to look for a use case that creates a predictive model than can give me the ability to propose a product for a customer the next time that he comes to my website.

Is this a collaborative filtering use case? If yes, I only can use the last two fields of my dataset, right?

Thanks!

Recommender systems are usually either collaborative ("other people who bought x also bought y") or content-based ("item y has similar properties as item x"). Given your very few data features, collaborative filtering seems appropriate. In the basic setting it could use only the purchases of your users. Ideally, you would also have some sort of rating of their purchases.

Also, nothing stops you from using a hybrid approach with additional data if available.


Compare Classification Performance in Datasets drawn from Different Populations I've read some classics about comparison of ML Algorithms i.e.

Dietterich, T. G. (1997). Statistical Tests for Comparing Supervised Classication Learning Algorithms 1 Introduction. Science, 10(7), 1–24. Retrieved from http://dx.doi.org/10.1162/089976698300017197

However I feel totally lost about a specific problem.
Backgrond / Status Quo
I have two dataset ($N_1=552$, $N_2=543$) drawn from different populations.
 Both contain the same set of features and the same criterion (7 class labels).To simplify I will spare the details on preprocessing and hyperparameter tuning.
In the end I have two trained algorithms (i.e. two RandomForests:  $RF_1$ & $RF_2$) for both datasets ($df_1$ & $df_2$) respectively
Goal / Aim
I want to know if it is better to train the algorithm using data drawn from population 1 and evaluate it using data drawn from population 2, or if the opposite is true. So which population generalizes better to the respective other. To be more precise if a measure of the classification performance (i.e. Accuracy or Kappa) for the $RF_1$ (Random Forest trained in dataset 1) tested in $df_2$ is significantly higher (not caused by chance) than the performance for the $RF_2$ (Random Forest trained in dataset 2) tested in $df_1$.

$Acc(RF_1->df_2) >  Acc(RF_2->df_1)$

Question
Is there an apropriate test for that? Is it as simple as the $\chi^2$ or a exact binominal test?

Edit
Or am I comparing apples and oranges, and there is no way one could compare this two classification results? I am very thankful for any direction you can give me.

I understood it wrong ,here is the paper which discuss using multiple data set for the same classifier-

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.141.3142&rep=rep1&type=pdf

They conclude-
"
We theoretically and empirically analyzed three families of statistical tests that can be used for
comparing two or more classifiers over multiple data sets: parametric tests (the paired t-test and
ANOVA), non-parametric tests (the Wilcoxon and the Friedman test) and the non-parametric test
that assumes no commensurability of the results (sign test). In the theoretical part, we specifically
discussed the possible violations of the tests’ assumptions by a typical machine learning data. Based
on the well known statistical properties of the tests and our knowledge of the machine learning data,
we concluded that the non-parametric tests should be preferred over the parametric ones."

I recently learned about 5x2cv paired t test procedure to compare the performance of two models.

Please refer below-

http://rasbt.github.io/mlxtend/user_guide/evaluate/paired_ttest_5x2cv/

It Implements the 5x2cv paired t test proposed by Dieterrich (1998) to compare the performance of two models.


Survival Analysis: Pseudo Observation Vs Stratified Cox Regression. Which one is better? I've been looking into the Cox Regression method for Survival Analysis in Churn Prediction. Cox regression will allow us to determine the probability that a subscriber will unsubscribe after a time $t$, defined by the hazard rate:

$$
h(t \lvert X_i ) = h_0(t)exp\big( \boldsymbol{\beta} ^T\boldsymbol{X}_{i} \big)
$$

Where

$h_0(t)$: Baseline Hazard is a prior Probability that any customer churns at time t when all influencing factors are 0.
$\boldsymbol{\beta} \in \mathbb{R}^D$: Exponent of each Coefficient gives us a Hazard ratio. These should be constant w.r.t time (proportionality assumption).
$\boldsymbol{X}\in \mathbb{R}^{N\times D}$: Set of $N$ sample customers

Problem: Proportionality Hazard Assumption: Cox regression makes an assumption that the Hazard Ratios should remain constant through time $t$. For example, for a covariate $X_1$ = "gender", say $\beta_1=1.8$. In english, it means male subscribers tend to leave the service $80\%$ more than females after a time $t$. However, this $80\%$ should hold for any time $t$.

This is usually an unreasonable constrain for many variables. But there are other methods that can incorporate variables that don’t follow the proportional hazards assumption.

stratified cox regression
pseudo-observations
cox regression with time-dependent covariates

I was just reading up on stratified cox regression. The only apparent downside here is:

The variables that are stratified need to be converted into categorical variables
The stratified categorical variables should not have too many degrees of freedom. This will lead to a LARGE number of models whose parameters need to be estimated.

Question: Is pseudo-observations similar? Does it have less/more rigid constraints? Even so, how is it's performance considering I have copious amounts of data?

I suggest using a model with more relaxed assumptions on proportionality of hazards. In my work I use piecewise constant hazard model, which works wonderfully. Its assumption is that the hazards are proportional in a time interval. It allows using numerical covariates with splines, and time-dependent covariates. Moreover in my experience the model is usually very well calibrated and does not overfit much.


Can neural networks be adapted without recreating them completely? If I have, for example, a classification network which can tell if there is a dog or a cat in a picture, is it possible to adapt the network so it can also learn to detect a mouse? Without making a new one from scratch.
In this case it doesn't make sense, but I'm wondering if for example Netflix has to retrain its network completely with every new show they add. And if so, when do they do that? The first few days may be the most crucial ones, but also the ones with the least data to train a network from.

The actual problem I do have is that I'm trying to train a network to predict the location of public transport vehicles. That by itself is hard enough, but what do I do if there is a new transport line, which I don't have an input neuron for. - Here I would need to add an input neuron.

Or another idea I'm thinking about is a neural network which can help me order my documents by detecting the company and date. I would like to classify an unknown document layout, a few times, so the neural network is able to classify it by itself, but without losing everything it learned so far. - Here I would need to add an output neuron.

It seems like there is a way to do this, since there are some machine learning algorithms out there which seem to pull off stuff like that. Or do they somehow work around this? If so, how?

The answer to your question is "Transfer Learning".

Since the datasets "cat and dog" and "mouse" are quite similar as both are images. In DeepNet for recognising "cats and dogs", any deep learning network in its early layers learn to identify low-level features like edges, etc. It learns high-level feature like eyes, ears, etc in few further layers and in very later layers of DeepNet, it starts to recognise the intended object. In DeepNet for recognising "dogs", similar pattern will follow. On comparing these two, one may find that first few layers of DeepNet in both cases produce similar low-level features like edges, etc.

Hence, first few layers of the DeepNet learned from "cats" can be used as base layers for the DeepNet for "mouse" detection. This technique is called transfer learning.

To apply transfer learning, dataset on which DeepNet has been trained on and dataset on which this technique can be applied must be similar.

This transfer learning video by Andrew Ng would also be helpful in understanding the concept


Does training of neural networks follow the same order in each epoch? Each epoch uses the weight from the end of the previous epoch(correct me if I am wrong). Is the updating of parameters after each batch always in the same order? To rephrase, are the batches always in the same order? Could this bias the learning and are there any adaptations that deal with this.

All the parameters are updated after a batch, there is no notion of order of update.
The batch can or cannot be in the same order. It could lead to overfitting in the sense of the network learn the order of the dataset. An easy turnaround is to shuffle the batch between each epochs.


Training machine to identify hot spots I have a set of 1000 images of the dorsal side of the hand. I need to identify the different joints in the fingers and measure the distance between them. I have already cropped the images to same dimensions, centered the hand, and labeled the required points.

Can someone point me to the next steps and a general direction I should be looking at for training the machine I need? Which algorithms would be the closest for my need?

Convolution Neural Network (CNN) is the state of art algorithm for your problem. The standard terminology for this kind of problem is "feature detection".

Here is a good resource for the basics. http://cs231n.stanford.edu/

Your biggest problem probably is going to be the size of your training images. 1,000 is most likely not going to be enough. You should try to augment your data by rotating and translation etc.


C++ return array from function I would like to implement machine learning algorithm in C++ without using any C++ machine learning library. So I'm writing this initializer function for generating zero matrices but can't figure out how can I accomplish this. I'm actually trying to write C++ code for simple logistics regression for now.

float * intializer_zero(int dimension){
    // z = wx + b.

float b = 0;
float w[dimension]= { };
return w,b;
}

It's throwing error "cannot convert 'float' to 'float' in return."
How can I write this initializer function in C++?

You can do something like that for matrix

mat = new float*[rows];

for (ushort i = 0; i < rows; i++) {
    mat[i] = new float[cols](); //init 0

}


xgboost in R have different results compared to boosted decision tree in Azure ML I have a small data set (4000 records with 10 features) and I used XGBOOST in R as well as Boosted Decision Tree model in Azure ML studio. Unfortunately the results are different. I like to optimize recall and I could pick that as a measure in Azure but I can not do so in R.

I used the same parameters in both platforms. I know seeds might be different but I tried many of them. I always have a much better recall on my validation dataset using the Azure model compared to the R one.

I wonder if there is a big difference behind the methodology used in these two platforms causing me the issues.

I also used cross validation which did not help. Any insight is appreciated.

Thanks

It's hard to say, without being able to know exactly what Azure is doing.

From what they do share, they bin continuous features; you could try tree_method='hist' in xgb to be more similar there.
I can't tell how Azure deals with categoricals or missing values.
Be sure to set xgb's max_depth=0 and grow_policy='lossguide', since you want to use max_leaves instead for a direct comparison.

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/two-class-boosted-decision-tree#usage-tips
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/two-class-boosted-decision-tree#module-parameters


How does the given data gets plotted on a graph I come from a programming background and learning the math behind the data science and algorithms now.

I would like to understand the logic behind how a data gets plotted in a graph when using Logistic regression.

Lets consider a Testing data like this,

ID  Age  Grade  Location   Churn
1   24    1        A         1
2   25    1        A1        1
3   28    1        A2        1
4   31    2        A3        0

And this could get plotted like this in Logistic regression like this -

[A mocked image below]

How did these ID's usually gets plotted in the graph. What's the math behind a data like this getting plotted in a graph.

I understand the plane separating the points. But couldn't understand how the points got there in the first place.

This may be basic for experienced people. But it has been a question in my mind for a while. I tried to get any online materials but I couldn't or may be I don't know what to search for.

Also, does this plotting differs in any way for SVM and other models. I guess only the model splitting the points will differ.

Please give me leads in understanding this one.

First let's understand the problem. This is a classification task. The last column represents your target value and rest of the columns are features for a given id. The target value represents which class a given row will belong to. In the plot it is represented by red (Class - 1) and green (Class - 0). We use rest of the column values to plot a given point.

Now we just simply plot one value vs another. The graph that you have shown is for only two variable but in reality we will be using all the variable and the separation will not be a line, but a hyperplane. Since it is difficult to visualize, we usually use only two dimensions for demonstration purposes.

As an example, here is a plot for Age vs Grade marking different classes with different colors:

I hope you get the basic idea. Also, I suggest you to look at tutorials for kNN classifier. It will give you more understanding of the cocept.


Use test data as train: does it make sense? There is a classification problem(two classes). We have train data, for which we know class labels and we have test data.

Imagine, that you have created model that with good accuracy(~95%) make predictions and we know that we are not overfitted.

If we make prediction on test data, extract objects for which we sure in class label(for example, predict_proba higher than 90%) and add this objects to train data.

Does this tactic make any sense?

This idea will most likely increase the bias in the model. Let's assume that the model has non-zero bias in the model. In this case, when it assumes its predictions to be true, without confirmation from an Oracle as in active learning, the bias of the model increases. In common terms, if the model has some amount of bias in its predictions, and it uses its predictions to learn on, the bias in the model can only increase. This issue does not arise when there is 0 bias in the model to begin with, however, in that case, there is no need to learn any further!

Note that this is a highly intuitive answer but I cannot think of an argument against the intuition :-) I will appreciate any feedback on this.


Principle behind seq2seq model's example in keras? I am referring to seq2seq model's example code in keras (https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py). The model is :

model = Sequential()

model.add(RNN(HIDDEN_SIZE, input_shape=(MAXLEN, len(chars))))

model.add(RepeatVector(DIGITS + 1))
for _ in range(LAYERS):
    model.add(RNN(HIDDEN_SIZE, return_sequences=True))

model.add(TimeDistributed(Dense(len(chars))))
model.add(Activation('softmax'))

In this model we are passing the encoded input vector from encoder's last state to each time step in the decoder.

Now we are not passing any other input to the decoder except the encoded input vector, but in all seq2seq models we pass output sequence also (time delayed) with the encoded input.

How is this a valid seq2seq model? To my surprise it works well. How does this works?

The original Seq2Seq paper uses the technique of passing the time delayed output sequence with the encoded input, this technique is termed teacher forcing.

There exists a simplified architecture in which fixed length encoded input vector is passed to each time step in decoder (analogy-wise, we can say, decoder peeks the encoded input at each time step).

The paper "Learning Phrase Representations using RNN Encoder–Decoder
for Statistical Machine Translation" combines these both techniques (so it passes encoded input vector along with time delayed output sequence as inputs to decoder).


Classification engine with multiple Textual independent fields I have made a classification engine with only one independent field (Comments) and classified them in multiple dependent variables. Now I want to have multiple independent variables in training data (comments,state,age,gender etc).

Currently i am using python 3.6.

Example of what I have done:

Comments                                Classified
(The car wash service is good)          Positive

Example of what I want to do:

 Comments                          Gender    City   Age       Classified
 (The car wash service is good)    Male      LA     40        Positive

As  you can see in the second example we have 4 independent variables affecting the outcome(Classified sentiment). I want to implement a classification engine based on this use case. Please suggest how should i move ahead...???

Edit:- I have used SGDClassifier from sklearn.linear_model.

I have first passed list of comments as vectors which i have vectorized using TfidfVectorizer. Below is the code of vectorization:-

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(min_df=5,
                                 max_df = 0.8,
                                 sublinear_tf=True,
                                 use_idf=True)

train_vectors = vectorizer.transform(Train_data_Comments_list)

then I pass these vectors to train along with their classified labels. Below is the code(I have used partial_fit as i want to train again and again):-

classifier = SGDClassifier.partial_fit(train_vectors, Train_Labels)

Then I use the the classifier to classify rest of the vectors.

test_vectors = vectorizer.transform(Test_data_comments_list)
prediction_liblinear = classifier.predict(test_vectors)

If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it?

If I combine all the fields value in my second Example and then form a single vector from that and then pass those vectors for classification, would that be the ideal way of doing it?

In short, yes. You can convert each input variable into a vectorised feature separately, then concatenate those vectors into one long vector of features per example. Most statistical machine learning models don't care (or "understand") that the feature data has been formed in different ways, it is just treated numerically.

You do need to pay attention to each input data type in order to decide how best to convert to a usable feature. There is no single "best" way to do this, and this is partly an art - you add the "science" part of Data Science by testing your ideas and taking measurements, e.g. accuracy.

These are some quick thoughts on how you might prepare the data into features from each column:

Comments. You have already converted this, you are using a variant of "bag of words" to convert text into a fixed length numerical vector. This is a common approach and hard to beat. If you have a lot of data you can look into more sophisticated models that take account of word order, such as Recurrent Neural Networks, but that's a whole new subject area for now.
Gender. You can convert this into a simple vector, using one hot encoding - this will have two or more columns.
City. You could one hot encode this also. It might also be worth grouping cities and having a smaller feature vector. E.g. group by state/country before one hot encoding.
Age. Either group and one hot encode, or scale to a smaller number,
by dividing by e.g. 50 (this helps algorithms like SGD by keeping distance metrics similar between different feature types). I suspect grouping in typical demographic splits (e.g. 0-11, 12-17, 18-24, 25-34 etc) and one hot encoding would work well for sentiment analysis, because that would capture some generalised differences between uses of text expression.


Interpretation of 'recall' as a conditional probability P( X=>+ | X=+ ) In the context of performance measures for classification, I have a question about recall and precision.

Looking at the definition of recall:-

$recall = \frac{T_p}{T_p+F_n}$

When I look at this, it sounds like a $conditional$ probability to me -- probability that a test instance will be classified as positive, given that it is indeed positive:-

$ recall = Pr(X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive | X=positive ) = \frac{T_p}{T_p+F_n}$

Here I am taking liberty to think (as it goes from the definition of $F_n$ ) that $F_n$ in the denominator is actually count of the test instances which are positive but got mis-classified as negative.

In the same light, if I think of precision now, this is how I am thinking of it as a probabilitty that a new test instance actually being positive given that it is predicted as a positive:-

$ precision = \frac{T_p}{T_p+F_p} = Pr(X=positive | X\hspace{1mm}is\hspace{1mm}predicted\hspace{1mm}as\hspace{1mm}positive)$

Is this interpretation of precision and recall correct?

Your interpretation is correct.
$ F_n $ represents the number of false negatives whereas $ F_p $ represents the number of false positives.
Recall
answers the question:
when presented a positive example, how often does the classifier get it right?
aliases: sensitivity, true positive rate (TPR)
Precision
answers the question: out of all the examples the classifier thought were positive, how often were the examples actually positive?
aliases: positive predictive value (PPV)


What loss function to use for imbalanced classes (using PyTorch)? I have a dataset with 3 classes with the following items:

Class 1: 900 elements
Class 2: 15000 elements
Class 3: 800 elements

I need to predict class 1 and class 3, which signal important deviations from the norm. Class 2 is the default “normal” case which I don’t care about.

What kind of loss function would I use here? I was thinking of using CrossEntropyLoss, but since there is a class imbalance, this would need to be weighted I suppose? How does that work in practice? Like this (using PyTorch)?

summed = 900 + 15000 + 800
weight = torch.tensor([900, 15000, 800]) / summed
crit = nn.CrossEntropyLoss(weight=weight)

Or should the weight be inverted? i.e. 1 / weight?

Is this the right approach to begin with or are there other / better methods I could use?

Thanks

What kind of loss function would I use here?

Cross-entropy is the go-to loss function for classification tasks, either balanced or imbalanced. It is the first choice when no preference is built from domain knowledge yet.

  This would need to be weighted I suppose? How does that work in practice?

Yes. Weight of class $c$ is the size of largest class divided by the size of class $c$.

For example, If class 1 has 900, class 2 has 15000, and class 3 has 800 samples, then their weights would be 16.67, 1.0, and 18.75 respectively.

You can also use the smallest class as nominator, which gives 0.889, 0.053, and 1.0 respectively. This is only a re-scaling, the relative weights are the same.

  Is this the right approach to begin with or are there other / better
  methods I could use?

Yes, this is the right approach.

EDIT:

Thanks to @Muppet, we can also use class over-sampling, which is equivalent to using class weights. This is accomplished by WeightedRandomSampler in PyTorch, using the same aforementioned weights.


Extracting embeddings of Categorical Variables I am trying to build a regression model, for which I have a nominal variable with very high cardinality. I am trying to get the categorical embedding of the column.

Input:

df["nominal_column"]

Output:

the embeddings of the column.

I want to use the op of the embedding column alone since I would require that as a input to my traditional regression model. Is there a way to extract that output alone.

P.S I am not asking for code, any suggestion on the approach would be great.

As far as I understand, usually embeddings are initialized with random values. If there are pretrained embeddings, they can be loaded, but of course there are no such things for categorical variables.

So if you simply create an embedding of a categorical feature, you'll get a vector representation, but it will have random variables. So at first you'll need to train the neural net. After it is trained, you could take the output of the embedding layer and use it in regression model.

But are you sure that you need the embedding? You could do mean encoding or frequency encoding.


Alright so I rewrote some parts of your model such that it makes more sense for a classification problem. The first and most obvious reason your network was not working is due to the number of output nodes you selected. For a classification task the number of output nodes should be the same as the number of classes in your data. In this case we have 5 kinds of flowers, thus 5 labels which I reassigned to $y \in \{0, 1, 2, 3, 4\}$, thus we will have 5 output nodes.

So let's go through the code. First we bring the data into the notebook using the code you wrote.

from os import listdir
import cv2

daisy_path = "flowers/daisy/"
dandelion_path = "flowers/dandelion/"
rose_path = "flowers/rose/"
sunflower_path = "flowers/sunflower/"
tulip_path = "flowers/tulip/"

def iter_images(images,directory,size,label):
    try:
        for i in range(len(images)):
            img = cv2.imread(directory + images[i])
            img = cv2.resize(img,size)
            img_data.append(img)
            labels.append(label)
    except:
        pass

img_data = []
labels = []
size = 64,64
iter_images(listdir(daisy_path),daisy_path,size,0)
iter_images(listdir(dandelion_path),dandelion_path,size,1)
iter_images(listdir(rose_path),rose_path,size,2)
iter_images(listdir(sunflower_path),sunflower_path,size,3)
iter_images(listdir(tulip_path),tulip_path,size,4)

We can visualize the data to get a better idea of the distribution of the classes.

import matplotlib.pyplot as plt
%matplotlib inline

n_classes = 5

training_counts = [None] * n_classes
testing_counts = [None] * n_classes
for i in range(n_classes):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1,2,3,4))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

We will now transform the data and the labels to matrices.

import numpy as np
data = np.array(img_data)
data.shape
data = data.astype('float32') / 255.0
labels = np.asarray(labels)

Then we will split the data.. Notice that you do not need to shuffle the data yourself since sklearn can do it for you.

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, shuffle= True)

Let's construct our model. I changed the last layer to use the softmax activation function. This will allow the outputs of the network to sum up to a total probability of 1. This is the usual activation function to use for classification tasks.

from keras.models import Sequential
from keras.layers import Dense,Flatten,Convolution2D,MaxPool2D
from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

model = Sequential()
model.add(Convolution2D(32, (3,3),input_shape=(64, 64, 3),activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(128,activation='relu'))
model.add(Dense(5,activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Then we can train our network. This will result in about 60% accuracy on the test set. This is pretty good considering the baseline for this task is 20%.

batch_size = 128
epochs = 10
model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

After the model is trained you can predict instances using. Don't forget that the network needs to take the same shape in. Thus we must maintain the dimensionality of the matrix, that's why I use the [0:1].

print('Predict the classes: ')
prediction = model.predict_classes(x_test[0:1])
print('Predicted class: ', prediction)
print('Real class:  ', y_test[0:1])

This gives

  Predict the classes:  1/1 [==============================] - 0s
  6ms/step Predicted class:  [4] Real class:   [4]

Some suggestions

The model you are currently using is the one that is most common for MNIST. However, that data only has a single channel thus we don't need as many layers. You can increase the performance by increasing the complexity of your model. Or by reducing the complexity of your data, for example you can train using the grayscale equivalent of the images, thus reducing the problem to a single channel.


Calculating optimal number of topics for topic modeling (LDA) am going to do topic modeling via LDA. I run my commands to see the optimal number of topics. The output was as follows: It is a bit different from any other plots that I have ever seen. Do you think it is okay? or it is better to use other algorithms rather than LDA. It is worth mentioning that when I run my commands to visualize the topics-keywords for 10 topics, the plot shows 2 main topics and the others had almost a strong overlap. Is there any valid range for coherence?
Many thanks to share your comments as I am a beginner in topic modeling.

LDA being a probabilistic model, the results depend on the type of data and problem statement. There is nothing like a valid range for coherence score but having more than 0.4 makes sense. By fixing the number of topics, you can experiment by tuning hyper parameters like alpha and beta which will give you better distribution of topics.

The alpha controls the mixture of topics for any given document. Turn
it down and the documents will likely have less of a mixture of
topics. Turn it up and the documents will likely have more of a
mixture of topics.
The beta controls the distribution of words per topic. Turn it down and
the topics will likely have less words. Turn it up and the topics will
likely have more words.

The main purpose of lda is to find hidden meaning of corpus and find words which best describe that corpus.
To know more about coherence score you can refer this


Guided topic modeling: generating words from topics I need to generate lists of words related to specific topics for a project. I am familiar with clustering methods of topic modeling such as LDA, but I have something else in mind. Are there any methods to generate lists of related words from a root word? For instance, where I could type "medicine" and it would return a list of words or terms related to medicine, such as "doctor", "malpractice", "office", etc.?

I think instead of building something from scratch, you should utilize the domain knowledge of experts.

One example: Turn Wikipedia's topic portals (e.g., the medicine portal) into
a family tree.


Gradient Tree Boosting I have a classification problem.
In gradient tree boosting I read that-
1. Initially a weak learner is fitted on the entire training dataset.
2. Output of each training row is obtained. In my case it will be {0,1}.
3. Now, the second classifier will train on the residual of the prediction i.e {initial prediction - final prediction}.
My doubt is it seems very odd to calculate the residual in case of classification problems while it is ok when I look at residual calculation in case of mse.
So, what is actually happening in case of categorical target variable. How is the residual being calculated and which loss function is that fitted into?

Thanks in advance.

My layman's understanding is that binary classification is usually calculated using the logit transform.

I believe then that the residuals are the difference between the response and the predicted probabilities and the default metric for this application is the Area-Under-Curve (AUC).


Is an artificial neural network a kind of metaheuristic? In a research paper i have written i am solving a problem through the use of an ANN, which learns to model a technical process through training data.

I compare that to metaheuristic approaches, which can also find relevant outputs from inputs by searching, e.g., with a genetic algorithm and an appropriate cost function.

A reviewer gave me the feedback that ANN are also a kind of metaheuristics, so i should not compare ANN vs. metaheuristics as different kinds of approaches in the paper. Is this correct? I have looked up the definition of metaheuristic and while i think that i could use a metaheuristic optimizer instead of a gradient based one, an ANN itself is not a metaheuristic.

I suppose one could possibly argue it either way, but at the end of the day, it doesn't really matter whether we classify an ANN as a metaheuristic.  It sounds like you have one reader who found it off-putting or confusing to have you compare ANN vs metaheuristic because they would have characterized an ANN as a metaheuristic.  So, I suggest asking yourself three questions: (1) How many other readers will have a similar reaction?  (2) What changes can I make to the paper that would avoid that reaction?  (3) Is there any downside to those changes?  Are they worth it?

In this case, it seems easy enough to rephrase your description of the comparison, so you don't trigger this reaction.  You could be more specific about what comparison you are doing: you compare an ANN to a genetic algorithm.  That might avoid the problem, with no downside.  Being specific is often a good thing.