how to make new columns using another column in pandas? i have a dataframe like this

i want to add new column and my desire format is like this

i use indexing eg
 dll.loc[:6,'category'] = "sectowise"
dll.loc[7:11,'category'] = "productwise"
dll.loc[12:16,'category'] = "collateral wise"
but which is risky because data index can change anytime
is there any method to do this?

Here is an example (I just through it together hastily). You can probably do this a little bit smoother.

import pandas as pd
import numpy as np
from random import shuffle

a = np.repeat(['a', 'b', 'c', 'd', 'e'], 6)
x = np.random.randn(30)
y = np.random.randn(30)
z = np.random.randn(30)

shuffle(a)

a = a.reshape(30, 1)
x = x.reshape(30, 1)
y = y.reshape(30, 1)
z = z.reshape(30, 1)

data = np.concatenate((a, x, y, z), axis=1)
df = pd.DataFrame(data, columns=['item', 'x', 'y', 'z'])

mapping = {'1': ['a', 'b'], '2': ['d'], '3': ['c', 'e']}

for k in mapping:
    df.loc[df[df['item'].mask(
        ~df.item.isin(mapping[k])).notnull()].index.tolist(), 'category'] = k


Fine-tuning a model from an existing checkpoint with TensorFlow-Slim I'm trying to retrain the final layer of a pretrained model with a new image dataset using TensorFlow-Slim.

Lets say I want to fine-tuning inception-v3 on flowers dataset. Inception_v3 was trained on ImageNet with 1000 class labels, but the flowers dataset only have 5 classes. Since the dataset is quite small we will only train the new layers.

An example at official tf github page shows how to do it:

$ DATASET_DIR=/tmp/flowers
$ TRAIN_DIR=/tmp/flowers-models/inception_v3
$ CHECKPOINT_PATH=/tmp/my_checkpoints/inception_v3.ckpt
$ python train_image_classifier.py \
    --train_dir=${TRAIN_DIR} \
    --dataset_dir=${DATASET_DIR} \
    --dataset_name=flowers \
    --dataset_split_name=train \
    --model_name=inception_v3 \
    --checkpoint_path=${CHECKPOINT_PATH} \
    --checkpoint_exclude_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits \
    --trainable_scopes=InceptionV3/Logits,InceptionV3/AuxLogits/Logits

I couldn't fully understand all the parameters in the above code:

train_dir = ?

dataset_dir = new dataset directory location

dataset_name = name of the dataset (but why ? )

dataset_split_name = ?

model_name = name of the model on which we want to train

checkpoint_path = path to the model checkpoint

checkpoint_exclude_scopes = ?

trainable_scopes = ?

Help me figure out what these parameter means and correct me if I'm wrong about any of the parameter?

Note: I'm aware that we can retrain inception_v3 with a method mentioned at official tensorflow website but I want to do the same with tensorflow-slim.

You can get all information you need from the source code.

train_dir: Directory where checkpoints and event logs are written to. (default: /tmp/tfmodel/)
dataset_dir: The directory where the dataset files are stored.
dataset_name: The name of the dataset to load. (default: imagenet)
dataset_split_name: The name of the train/test split. (default: train)
model_name: The name of the architecture to train. (default: inception_v3)
checkpoint_path: The path to a checkpoint from which to fine-tune.
checkpoint_exclude_scopes: Comma-separated list of scopes of variables to exclude when restoring from a checkpoint.
trainable_scopes: Comma-separated list of scopes to filter the set of variables to train. By default, None would train all the variables.


Supervised Learning: time estimation of bike repair I would like to train a model that estimate the time a given shop would take to repair for a bike using the data below:

shops.csv

| variable | description              |
+++++++++++++++++++++++++++++++++++++++
| shop_id  | unique shop identifier   |
| country  | country of the shop      |
| city     | city of the shop         |
| nb_empl  | number of employees      |
| review   | google review            |

repairs.csv

| variable  | description                             |
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
| repair_id | unique repair identifier                |
| price     | value of the repair in local currency   |
| type      | what to repair (brakes, wheel...)       |
| bike_model| model of the bike to repair             |
| size      | size of the bike to repair              |
| shop_id   | shop that carries out the repair        |
| duration  | time it took to repair the bike (TARGET)|

My first thought was to combine the datasets and treat the problem as a supervised learning problem. Meaning that I would use regression, XGBoost, random forest to fit my model in order to get the estimated time of repair, all the shops in the training set being mixed.
But my concern is that my shops might be similar, it doesn't mean that they would spend the same amount of time to repair a bike.
My second thought was to treat each shop independently, so basically one model per shop.

My questions are:

How would you approach the problem yourself?
Is there a way to combine my two thoughts together to get a better model?

But my concern is that my shops might be similar, it doesn't mean that they would spend the same amount of time to repair a bike. My second thought was to treat each shop independently, so basically one model per shop.

You need to decide which problem you're trying to solve, e.g.:

average duration of repair for a particular type of bike and repair id
average duration of repair for a particular shop (can be used to rank shops by their performance)
average duration of repair for a particular type of bike in a particular shop (can be used to predict the best shop for a particular bike)
...

After having clearly defined the problem, you need to format the data so that every instance corresponds to the item for which you want to predict something. For instance if the goal is to predict duration by type of bike, then an instance represents a type of bike so the feature should contain information about the bike and the target variable could be the average duration for this bike. In this case you shouldn't have several instances for the same type of bike.


Any research on segmentation of non-text contents out of (mostly) text-documents? Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?

Although I didn't implement it so far, I am pretty sure natural language text vs code snippets is easy:

For each block, you make compare the distribution of characters to ground-truth natural language text vs. code. See my paper The WiLI benchmark dataset for written language identification, page 4 "Single-Character Frequency Analysis".


Splitting a sentence to meaningful parts I have the following sentence

query = u'tell me about people in konoha who have wind style chakra and are above jonin level'

I want to split the above sentence into the following three parts

[
 "tell me about people in konoha",
 "who have wind style chakra",
 "and are above jonin level"
]

I have looked into sent tokenizers in spacy and nltk, but they are not giving me the desired results.

I am just a beginner in nlp and machine learning and have very limited knowledge so far. It would be awesome if you could direct me to some techniques or available packages through which I could achieve the above results.

EDIT

I'll be having question-like queries like the one above where I would be asking for details about the subject based on some constraints

In the above query the subject was people in Konoha and the constraints are have wind style chakra and above jonin level

I want to extract such relationships from similar queries.

You are looking to create a  parse tree to find multi-token clauses.

Here is code to generate a parse tree:

import spacy
from nltk import Tree

nlp = spacy.load('en')

def to_nltk_tree(node):
    if node.n_lefts + node.n_rights > 0:
        return Tree(node.orth_, [to_nltk_tree(child) for child in node.children])
    else:
        return node.orth_

query = u'tell me about people in konoha who have wind style chakra and are above jonin level'
doc = nlp(query)
[to_nltk_tree(sent.root).pretty_print() for sent in doc.sents]

Output:

    tell
  ____|______
 |         about
 |           |
 |         people
 |     ______|_____
 |    |           have
 |    |       _____|____________
 |    |      |     |     |     are
 |    |      |     |     |      |
 |    |      |     |   chakra above
 |    |      |     |     |      |
 |    in     |     |   style  level
 |    |      |     |     |      |
 me konoha  who   and   wind  jonin

From that parse tree, you can then select the phrases.


How to filter Named Entity Recognition results I have a pipeline built which at the end outputs a bunch (thousands to tens of thousands or more) of named entities. I'd like to do aggregates on those named entities (to see, e.g. how many times a specific named entity is mentioned in my corpus). A problem that I am arriving at; however, is that the named entities often don't match up with each other even though they are the same entity. For example, one instance of the named entity might be "Dr. John Smith" while another instance is "John Smith" or one instance might be "Google" while another might be "Google Inc.". This makes aggregating quite hard to do.

In order to deal with this issue and set "Dr. John Smith" to be the same entity as "John Smith", I was thinking of doing word matching between my named entities. I.e. I would check if named entity A has a word in common with named entity B and if they do set them to be the same entity. This approach is obviously seriously flawed. I will be equating "John Nguyen" and "John Smith" as the same entity even though they are obviously not. What's potentially even worse with this method though is I might run into similarity chains where I have "John Smith" linked with "Richard Smith" linked with "Richard Sporting Goods Inc." linked with "Google Inc." etc etc... While I may be willing to allow issues arising from former problem through, the latter problem appears to be catastrophic.

Are there any accepted techniques in the NLP community for dealing with this issue?

The most advanced (and complicated) approach to this is some sort of weakly-supervised system like Holoclean. It seems promising, but not easy.

The other end of the spectrum are heuristics like the one you propose. If you want to use a string distance approach like you describe, I would create some metric that gives points for both string edit distance and having matching words. If you can break the pipeline into smaller steps, maybe you can use some sort of topic analysis, then look for two similar entities and that topic in a knowledge graph, and see if the point to the same area.

The 2018 Alexa Prize winner paper, Gunrock is full of clever heuristics that you might be able appropriate.

Edit: you should also look at Entity Linking, which is more or less the problem you describe.


LSTM Multi-class classification for large number of classes I want to build a model that classifies 473 classes -product categories-, but I'm facing a problem with loss not decreasing.

Data
I have almost 3,000 data points for each class -473 classes- (data size is almost 1.5 million)
The data is a sequence of 5 words [iPhone, Pro, Max, 0, 0]
and of course they're numbered [345, 344, 123, 0, 0]
Examples:
Input: [iPhone, Pro, Max, 0, 0]
Output: iPhone

Input: [Go, Pro, Camera, New, 0]
Output: GoPro

Input: [LG, TV, 50, Inches, Used]
Output: LG_TV

Input: [Apple, Watch, 42, mm, 0]
Output: Apple_Watch

Loss
Epoch: 1, Loss: 5.607430, Val Loss: 5.538741
Epoch: 2, Loss: 5.493465, Val Loss: 5.516405
Epoch: 3, Loss: 5.487641, Val Loss: 5.513667
Epoch: 4, Loss: 5.474956, Val Loss: 5.508683
Epoch: 5, Loss: 5.472722, Val Loss: 5.508304
Epoch: 6, Loss: 5.472691, Val Loss: 5.510557
Epoch: 7, Loss: 5.472782, Val Loss: 5.508627
Epoch: 8, Loss: 5.472320, Val Loss: 5.533378
Epoch: 9, Loss: 5.472340, Val Loss: 5.520573

I've tried to train it for 50 epochs, but the loss still not decreasing.

Model
I'm using PyTorch
LSTMClassifier(
  (embedding): Embedding(15278, 200)
  (lstm): LSTM(200, 256, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.3, inplace=False)
  (dense): Linear(in_features=256, out_features=473, bias=True)
)

The loss function is: CrossEntropyLoss

Hyperparameters
Batch size: 512
Embedding Dim: 200
Vocabualry size: 15,278
LSTM Layers: 2
Hidden Dims: 256
Optimizer: Adam
Learning rate: 0.002

Can you please direct me to the problem? Is the model weak? Or is it the data having a problem?

One way to approach this is to increase model capacity and see if/when your LSTM is able to learn the train data. In this first step it is ok to overfit (also see this recent question and answer for this approach) since you can add regularization/decrease model capacity later.

These are the parameters I would tweak:

find a good learning rate along a log-scale (try at least 0.1, 0.01 and 0.0001)
increase the number of LSTM layers, e.g. to 5
increase the hidden dimension, e.g. to 1024
slightly increase the embedding dimension, e.g. to 300

If training this model takes too long, tune the learning rate based on your current model and then make the other adjustments. However, I suggest to not make the other adjustments in smaller steps or one by one as that can take a lot of time.


Understanding softmax layer in Deep Neural Network In a DNN, once the logits vector is produced say

$[y_0,y_1]$ where the number of neuron in the logits layer is 2, the condition holds where $y_0 >= 0$ and $y_1<= 1$.

This vector is then passed into the softmax layer where it squeezes the value to be within the range of 0 to 1. However, how do i prove that $y_0$ will always be $>=0$ and $y_1$ will always be $<=1$?

From the way i understand, both $y_0 + y_1 = 1$ hence $y_0 >= 0$ and $y_1 <=1$ but i find that it doesn't really prove on why $y_0 >= 0$ and $y_1 <=1$. Could anyone assist me on my understanding.

Look at the definition of softmax: https://en.wikipedia.org/wiki/Softmax_function

The exp() gives a positive value. Since you "standardize" each value exp(e^z) in the numerator by the sum of all exp(e^z), the values sum to one in total.

Is whis what you are looking for?

Maybe you should have a look at Hasties et al. "Elements of Statistical Learning" (Ch. 11.2 "Neural Nets").

Little R example:

# For standardization (denominator)
base = exp(-2) + exp(5)

# Single exp() values are positive
exp(-2)
[1] 0.1353353
exp(5)
[1] 148.4132

# Standatdize
exp(-2)/base
[1] 0.0009110512
exp(5)/base
[1] 0.9990889

# Total sum equals 1
exp(-2)/base + exp(5)/base
[1] 1


how to find probability of one or more events to happen from an incomplete data set I have a dataset that gives information of a population. For instance, I know the fraction of people that are males (M) and that are within a certain age range (A), P(M & A), and then I know the fraction of males that live in a certain area (L), P(M & L).

What I'm interested in computing is P(M & A & L), which is the fraction of people that are males, are within a certain age range and live in a certain area.

Using Baye's formula I can say that

P(M & A & L) = P(M & A | L) P(L)

But my dataset only gives P(L) and not P(M & A | L). However, if I assumed that M & A and L are independent I have

P(M & A | L) = P(M & A) P(L)

How large is the error on P(M & A | L) if I make this assumption. Do you know of any other method I could use to estimate P(M & A | L) without assuming independence?

Bayes Theorem applies to conditional probabilities.
$
     P(A|B) = \frac{P(A).P(B|A)}{P(B)}
    $

The question you have posed is one of multiple events occurring together. If the events can be considered to be independent of each other, then,

P(A & B & C) = P(A).P(B).P(C)

In your question above,

P(M & A & L) = P(M & A).P(L) = P(M).P(A).P(L)

Based on the data provided by you, you should have these probabilities or can derive them by appropriate summations.

Let's say that .1 probability of males are between 1-10 years of age. Further, probability of males in location L is 0.2. So the probability of males in 1-10 years of age in Location L is 0.1 X 0.2 = 0.02 . This is assuming that all locations have the same probability distribution of age ranges.


Data Analytics how to read ECDF graph Hi there, My question is about how to read ECDF graphs. I am still quite unsure what the jumps / zig-zags in the graph mean and what is happening when there is a horizontal line and so on. I would be happy if someone can explain me how I am suppose to read this graph and what information I can get from it. Thank you

The jumps in the CDF tell you that this is a Discrete random variable as opposed to a continuous random variable. The points on the x-axis where the jumps happen are the values that the discrete random variable take. The quantum of the jump represents the probability of the random variable taking on that value.


How can I predict the post popularity of reddit.com with hidden markov model(HMM)? If I get some posts on reddict.com, how can I predict whether this post will (trending/hot/popular) in the future or not? I would like to use the hidden markov model to predict it, but I don`t know how to define the hidden states and observation sequence...can anyone give me any suggestion? Thanks~

I don't think it makes a lot of sense to use HMM's for this problem. What I would suggest is some kind of text-based classifier.
If you want to use a cool technique, you could use a neural network to learn based on the text of successful posts.

On the other hand, If you want to use an easy technique you could make a predictor for the popularity such as a regression model (try to predict upvotes).


How can I discover topics in a social media data-set? I'm working on a project and i need to discover topics existing in a social media data set. For instance, i wanna extract the topics existing on 200K tweets. Any one recommend to me any machine learning algorithms?

You can take a look at Latent Dirichlet Allocation. In my experience this does very well without too much effort. You need to remove words that don't help like stopwords (and in your case Twitter handles and probably URLs) before feeding it to the algorithm. The only really important parameter that you need to give it is the number of topics. This will depend on your population (are these random tweets, or only tweets from a specific subgroup/hashtag?) and you need to compare some settings. What you can do is print the most important words per topic and see if they indeed do belong together.

If there are different languages in your tweets you need to deal with that beforehand, maybe classify them on language and only keep the English ones for example.


Using ggplot2 to create a bar chart So I'm trying to create a simple bar chart of Survive vs Not Survive for the common Titanic data set in R. I keep getting just the number of No's and Yes's, and not the frequencies or counts associated with each no and yes. This is obviously not what is wanted. I am trying to just practice with ggplot2 and make some graphs. What am I doing wrong here?
The Bad Barchart:

#install.packages("tidyverse")
#install.packages("titanic")

library(tidyverse)
library(titanic)

view(Titanic)

titanic <- as.data.frame(Titanic)

titanic$Survived <- as.factor(titanic$Survived)

ggplot(titanic, aes(Survived, fill = Survived))+
  geom_bar()+
  ggtitle("Barplot to represent Passenger Count who Survived vs who Died")

The number of people in each category is not defined by the number of rows but by the number in the Freq column. You therefore have to take the sum of those values and plot the result, this can be done by supplying the column which holds the value you want to use on the y-axis to aes() and use stat="identity" for geom_bar:
library(tidyverse)
library(titanic)

titanic <- as.data.frame(Titanic)
titanic$Survived <- as.factor(titanic$Survived)

ggplot(titanic, aes(Survived, Freq, fill = Survived))+
  geom_bar(stat="identity")+
  ggtitle("Barplot to represent Passenger Count who Survived vs who Died")


Choosing between TensorFlow or Theano as backend for Keras Keras supports both TensorFlow and Theano as backend: what are the pros/cons of choosing one versus the other, besides the fact that currently not all operations are implemented with the TensorFlow backend?

It was announced on  2017-09-28 that Theano will be discontinued:
From here (Yoshua Bengio):

After almost ten years of development, we have the regret to announce that we will put an end to our Theano development after the 1.0 release,  which is due in the next few weeks. We will continue minimal maintenance  to keep it working for one year, but we will stop actively implementing  new features. Theano will continue to be available afterwards, as per  our engagement towards open source software, but MILA does not commit to  spend time on maintenance or support after that time frame.

So TensorFlow is a better option.


Are batch iteration and epochs different in reinforcement learning compared to supervised learning? I'm following the Udacity "AWS DeepRacer" course about self driving cars with reinforcement learning.
In one lesson, they says this:

Batch size - This determines how many images, randomly sampled from the most recent episode, are used for training before updating the model.

Epochs - Determines how many times you will loop through the batched data before updating the training weights. A larger number of epochs is likely needed if your model is still improving but training has otherwise ceased.

Learning rate - Controls the speed at which your algorithm learns (it enlarges or shrinks the weight update after each epoch).

Using Tensorflow and Keras with Adam, I learned that the weights are updated after each batch iteration.
But in this course, they says that the weights are updated only after each epoch, and the batch size determine how much of the training data to loop through before updating the model. So, what's the difference between updating the model and updating the weights?
In tensorflow with keras, the weights are updated after each batch iteration and also after each epoch? Is it different compared to RL?

Your initial statements are correct.

Epoch is one-single pass over the full training set.

Most people use the word Batch as the number of samples used for one update of the weights. (The back-propagation process calculates the gradients for every single sample in the batch, but the weights update is performed a single time for the mean of the gradients over the batch).

However, originally the word batch was used to mean a training process where the batch size is equal to the overall number of samples in the training set (meaning that the weights update is performed only once per epoch). What we now refer to as "batch", was originally named "mini-batch".

Over the years, the word batch became synonymous with mini-batch (batch training is very rare these days). I think that this is the source to your confusion.


SelectKBest and Correlation returns me excatly same feature selection. How? Im working on selecting most effective features from a dataset with over that 2000 features. Im using different algorithms for that (selectKBest with chi-square, Extra Trees, Correlation etc.) But when I look the features ranking I saw that selectKBest with chi-square are generating excatly same results as Correlation. Is it possible or am I doing something wrong?

My all features consist of 64bit float continuous numbers, between [-8,11] and my target column is binary which can be only 0 or 1.

Updated on 05.09.19: I am STILL searching how can it be possible? I mean I can guess that both methods based on same formula and developed by same person but I need a proof for understand that clearly.

Correlation Function:

cor = data.corr()
# Class is my target column
cor_target = abs(cor["Class"])
# Want to get correlation values for every feature without target column
relevant_features = cor_target[cor_target > 0].drop(labels=["Class"])
#Top 1000 features
relevant_features = pd.Series(relevant_features, index=data.columns).nlargest(1000).index.values

SelectKBest function:

bestfeatures = SelectKBest(score_func=chi2, k="all")
fit = bestfeatures.fit(dataValues, dataTargetEncoded)
feat_importances_chi = pd.Series(fit.scores_, index=dataValues.columns).nlargest(1000).index.values

And the result relevant_features and feat_importances_chi have excatly same results.

Likely what you are seeing is that those features that are highly correlated with each other are overweighting the results, so that if one of the features describes the target well, then the other highly correlated (or uncorrelated) features also match the target just as well.


Decay Parameter in Keras Optimizers I'm currently training a CNN with Keras and I'm using the Adam optimizer. My plan is to gradually reduce the learning rate after each epoch. That's what I thought the decay parameter was for.  For me, the documentation does not clearly explain how it works:

  decay: float >= 0. Learning rate decay over each update.

However, when looking at the used learning rate in tensorboard, it stays the same as the initial learning rate. So, how does this decay parameter actually work?

Adam uses mini batches to optimize. During optimization, you may need go down hill, the cost function, so quickly using a high learning rate. When you reach to points which are near to relatively optimal point you have to reduce the learning rate in order not miss the optimal point. In other words you have to decay learning rate to have more accurate steps by reducing the learning rate. Mini-batch optimizers have multiple steps during one epoch, which all of them may not be true but because they try to minimize the cost for each batch of input data, they finally reach to the relative optimal points.

For each epoch, TensorFlow uses same learning rate and after finishing epoch, the next epoch will be started using the current learning rate divided by the decay parameter. It should not be negative because you are using gradient descent which implies moving toward low-level places.

Recently I was looking the code of optimizers in Keras and I found that as the following code:

if self.initial_decay > 0:
       lr *= (1. / (1. + self.decay * K.cast(self.iterations,
                                             K.dtype(self.decay))))

learning changes each iteration and not each epoch.


[EDIT:]
Your problem is definitely in the generators, in that you do not set the target size, and its default it (256, 256) - as seen in the documentation fro flow_from_directory:

flow_from_directory(directory, target_size=(256, 256), color_mode='rgb', ...)

  target_size: Tuple of integers (height, width), default: (256, 256). The dimensions to which all images found will be resized.

Try setting the target size parameter to (150, 150) and I think it will work. That default seems to be overwriting your preprocessing.

It must be in your generators - I ran the following code and a model trained as expected:

from keras import models, layers, metrics
import numpy as np

model = models.Sequential()
   ...: model.add(layers.Conv2D(32, (5, 5), activation='relu', input_shape=(150, 150,
   ...:  3)))
   ...: model.add(layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
   ...: model.add(layers.Conv2D(64, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Conv2D(64, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Conv2D(128, (3, 3), activation='relu'))
   ...: model.add(layers.MaxPooling2D((2, 2)))
   ...: model.add(layers.Flatten())
   ...: model.add(layers.Dense(300, activation='relu'))
   ...: model.add(layers.Dense(10, activation='softmax'))

In [10]: model.summary()
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 146, 146, 32)      2432
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 73, 73, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 71, 71, 64)        18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 35, 35, 64)        0
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 33, 33, 64)        36928
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 16, 16, 64)        0
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 14, 14, 128)       73856
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 7, 7, 128)         0
_________________________________________________________________
flatten_1 (Flatten)          (None, 6272)              0
_________________________________________________________________
dense_1 (Dense)              (None, 300)               1881900
_________________________________________________________________
dense_2 (Dense)              (None, 10)                3010
=================================================================
Total params: 2,016,622
Trainable params: 2,016,622
Non-trainable params: 0
_________________________________________________________________
In [17]: model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics = ['
    ...: acc',metrics.categorical_accuracy])

# Create some fake data to match your inputs. Each label seems to be 10 points: (1, 10)
In [11]: fakes = np.random.randint(0, 255, (100, 150, 150, 3))
In [24]: labels = np.random.randint(0, 2, (100, 10))

In [25]: model.fit(fakes, labels, validation_split=0.2)

Epoch 1/10
80/80 [==============================] - 2s - loss: 62.8913 - acc: 0.1125 - categorical_accuracy: 0.1125 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 2/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 3/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 4/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 5/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 6/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 7/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 8/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 9/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00
Epoch 10/10
80/80 [==============================] - 0s - loss: 67.0916 - acc: 0.0000e+00 - categorical_accuracy: 0.0000e+00 - val_loss: 71.7255 - val_acc: 0.0000e+00 - val_categorical_accuracy: 0.0000e+00


Similar Candidate Recommendations I've just embarked on a project to recommend similar candidates to employers for a website that offers employers the ability to search its CV database.

My background is in engineering research on biomedical images - from simple predefined feature extraction to CNNs.

I'm about to start with a simple bag of words approach standardised by the frequency of appearance and see where that gets me.

Has anyone done anything similar before and has suggestions for which algorithms might give good results?

The data is very well structured, separated into seperate sections for each previous employment etc.

Try content based filtering, you can cluster candidates based on similarity of keywords and by assigning similarity scores. Check tf-idf/cosine similarity measures. Additionally you can customize your algorithm by giving extra weight for section wise similarities since your data is very well structured.

Also check collaborative filtering on Mahout.


Is there any way to calculate a relevance score between a title and the content of a text? My question might sound a little bit stupid but I am trying to come up with a way to measure the relevance of the title of a text, let's say a piece of news headline, to the content. My idea would be to work with word2vec and have a go with cosine similarity. Are there any more efficient and proper ways to work with this task?

Word2Vec and cosine similarity are certainly a plausible solution. Note that the longer the text is, the more function words are in the text, therefore the average Word2Vec vectors are more similar to each other with the growing text length. This can be partially solved by removing the stop-words.

Another method to try might be cosine similarity with representation from contextual embeddings like ELMo or BERT. Also if you have some training examples (small hundreds might be enough) training a classifier based on BERT should work.

Finally, the fanciest solution is using the summarization model. Generating a title from an article content is one of the tasks that is done in summarization literature. If your data is similar to datasets that are used for training summarization models (or you have enough domain-specific data to train your own summarization model), you can use the model to estimate the probability of the title given the content. Even models that perform poorly on generation, perform well when used discriminatively.


is it acceptable if the reward of test of DQN is lower than reward of training of DQN in minimization problem? if we train a DQN over 40000-60000 episodes for 500 time steps. The mean of reward during last 100 training steps is about 1.1 times of reward during the test process.

Environment is stochastic!

The more time steps should be more stable?? What is the effect of time steps?

For instance, the mean of reward during the last 100 episodes of training is 6.1 however the answer during the test is 5.2 (My problem is minimization).
Could we say our DQN is OK? One test random is 120! however another is 400!

Or vice versa, is it acceptable if the reward of test in minimization is more than for instance 2 times of mean of last 100 training set?
Could we say our DQN is OK?

I did not run as a human! but if it is important I will play this game.

There is a lot of information that might be useful for a more precise answer in your case, but is hard to extract from your question description.

I think I can understand the following:

You are training to minimise a cost instead of maximise a reward (this is unusual in RL, but not a problem).
Random agents score variably, anything from 120 to 400 average cost.
After many training steps, your agent is scoring 6.1 average cost, measured over the most recent 100 time steps.
In testing, the agent scores better, an average cost of 5.2.

The usual way of testing a DQN agent (or any off-policy agent, of which Q Learning or DQN are one type) is to stop exploring actions. I am assuming you do this too. During training, your agent was not always taking the actions that it determined were best. In tests, you are interested to see what the agent thinks is "best" and how good that is. So it should be expected that test scores better than recent training. If you were training using $\epsilon$-greedy action selection, then your scores would be consistent with $\epsilon \approx 0.1$

However, your measurements have limited accuracy in any case. It might be worth calculating your standard error on your test data, to get a sense of how accurate your measure really is. The easiest way is to run your test multiple times, create an array of scores and calculate the standard deviation, mean and standard error as usual using the test results as samples of a random variable.

  Could we say our DQN is OK?

That depends entirely on the problem. It seems that it does far better than acting randomly, which shows at least it has learned something.

In order to get a sense of how well it has done, you need to compare it with something meaningful related to the problem:

For simple problems, such as toy problems used to prove DQN works, it is possible to calculate an optimal answer and see how close the DQN can get.
For problems that humans can attempt too, you can compare the performance of an agent to a human. Ideally a human who is expert at the task.
For problems for which there are some publications, you can compare with state-of-the-art. This is true for many Atari 2600 games for instance.
For problems which have a practical purpose, you can compare with a goal, such as would the agent be good enough to make money or reduce loss compared to current practice.
For some problems you could compare to a simple rules-based agent, coded to perform the specific task.

  The more time steps should be more stable?? What is the effect of time steps?

Up to a limit, the more time and samples that a DQN trains on, the better its performance will be. The best thing is to plot this - number of timesteps vs average cost. You may see sudden increases or even "catastrophic forgetting" (where the agent does no better than random, or even worse) at some times. But if the agent has trained successfully there should be a general improvement over time that reaches a limit as the DQN approaches the maximum capability of the agent.


Question on decision tree in the book Programming Collective Intelligence I'm currently studying Chapter 7 ("Modeling with Decision Trees") of the book "Programming Collective intelligence".

I find the output of the function mdclassify() p.157 confusing. The function deals with missing data. The explanation provided is:

  In the basic decision tree, everything has an implied weight of 1,
  meaning that the observations count fully for the probability that an
  item fits into a certain category. If you are following multiple
  branches instead, you can give each branch a weight equal to the
  fraction of all the other rows that are on that side.

From what I understand, an instance is then split between branches.

Hence, I simply don't understand how we can obtain:

{'None': 0.125, 'Premium': 2.25, 'Basic': 0.125}

as 0.125+0.125+2.25 does not sum to 1 nor even an integer. How was the new observation split?

The code is here:

https://github.com/arthur-e/Programming-Collective-Intelligence/blob/master/chapter7/treepredict.py

Using the original dataset, I obtain the tree shown here:

Can anyone please explain me precisely what the numbers precisely mean and how they were exactly obtained?

PS : The 1st example of the book is wrong as described on their errata page but just explaining the second example (mentioned above) would be nice.

There are four features:

referer,
location,
FAQ,
pages.

In your case, you're trying to classify an instance where FAQ and pages are unknown: mdclassify(['google','France',None,None], tree).

Since the first known attribute is google, in your decision tree you're only interested in the edge that comes out of google node on the right-hand side.

There are five instances: three labeled Premium, one labeled Basic and one labeled None.

Instances with labels Basic and None split on the FAQ attribute. There's two of them, so the weight for both of them is 0.5.

Now, we split on the pages attribute. There are 3 instances with pages value larger than 20, and two with pages value no larger than 20.

Here's the trick: we already know that the weights for two of these were altered from 1 to 0.5 each. So, now we have three instances weighted 1 each, and 2 instances weighted 0.5 each. So the total value is 4.

Now, we can count the weights for pages attribute:

pages_larger_than_20 = 3/4
pages_not_larger_than_20 = 1/4 # the 1 is: 0.5 + 0.5

All weights are ascribed. Now we can multiply the weights by the "frequencies" of instances (remembering that for Basic and None the "frequency" is now 0.5):

Premium: 3 * 3/4 = 2.25 # because there are three Premium instances, each weighting 0.75;
Basic: 0.5 * 1/4 = 0.125 # because Basic is now 0.5, and the split on
pages_not_larger_than_20 is 1/4
None: 0.5 * 1/4 = 0.125 # analogously

That's at least where the numbers come from. I share your doubts about the maximum value of this metric, and whether it should sum to 1, but now that you know where these numbers come from you can think how to normalize them.


One-hot encoding with values other than 1 I was thinking if I have an input which has 36 possible values, and I make it as 36 inputs where exactly one of them is non 0, what is optimal value for each of the non 0 inputs?

It may be:

[1, 0, 0,....,0]
[0, 1, 0,....,0]
[0, 0, 1,....,0]

Or:

[36, 0, 0,....,0]
[0, 36, 0,....,0]
[0, 0, 36,....,0]

Or even:

[6, 0, 0,....,0]
[0, 6, 0,....,0]
[0, 0, 6,....,0]

In order this feature to have same impact on the network as any other feature with N(0,1) distribution by keeping in mind I will use L1 or L2 regularization

So for each weight on a normal input I will have 36 weights on those one-hot inputs, so regarding L1 those weights should be 36 times smaller in order to have same impact? Don't they?

But then their total impact in affecting result is small since only one of them is being multiplied by 1 and included in calculation...

So if you would be kind to explain this and convince me to use 1s instead 6s or 36s, please

If you use 0s and 1s, the design matrix is composed of indicators variable. An indicator variable tells the presence (1) or absence (0) of a variable. The model weights are the magnitude. If you choose something other than boolean values, the model weight will the relative weights to the larger values in the design matrix.
Additionally if you also have numeric values and are fitting a linear model, you should standardize the numeric values. Then all variables are scale from 0 to 1 which allows the model to learn better and faster.


How to adjust the Regression with ANN for last part of function The Blue dots represent the required function and the Black ones represent the predicted function

I am using keras with the following code:-

# Set callback functions to early stop training and save the best model so far
from keras.callbacks import EarlyStopping, ModelCheckpoint
callbacks = [EarlyStopping(monitor='val_loss',verbose=1, patience=5,min_delta=0.002,mode='auto',restore_best_weights=False),
             ModelCheckpoint(filepath='best_model.hdf5', monitor='val_loss', save_best_only=True)]

##The Architecture
np.random.seed(0)
model = Sequential()
model.add(Dense(500,kernel_initializer='normal',activation='tanh',input_dim=1))

model.add(Dense(300, kernel_initializer='normal',activation='relu'))

model.add(Dense(500, kernel_initializer='normal',activation='relu'))

model.add(Dense(1, kernel_initializer='normal',activation='linear'))

model.compile(optimizer='adam',loss='mean_absolute_error',  metrics=['mae','accuracy'])

his = model.fit(X_train,y_train,epochs=150,batch_size=10,validation_data=
(X_val,y_val), verbose=1,callbacks=callbacks)

I tried various architecture but still not getting the result please guide

I see you use are sequential model. Could be a task for an LSTM architecture.
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

Maybe you are overengineering the thing. Judging from the dataplot, a simple OLS regression splines model may also fit well.
https://www.analyticsvidhya.com/blog/2018/03/introduction-regression-splines-python-codes/


Comparing one small dataset with a big dataset for similar records I create a varying small dataset (dataset: X) with 500 records in each query. Everytime I need to compare the dataset with a bigger one (dataset: A) (15 milion records) to find similar (or semi-silmilar) values from three different columns. The values are either one word or a sentences. My algorithms is like this:

make a vector of words in each record in both datasets
with a for loop, search for similarities over the big dataset (e.g. with tfidf). That means each record from the small dataset should look for possible similarities over the big dataset.

However, the problem is searching a big data is very slow. Is there any efficeint way to solve this problem?
Thanks

A way to speed up this process is to preprocess the large dataset, the goal being to store the documents from A in a way which avoids a lot of useless comparisons.

Store each document from A in an inverted index $m$, so that for any word $w$ $m[w]$ is the list of all documents in A which contain word $w$ (note that a document can appear several times in this data structure).
When comparing a new query against $A$, instead of iterating through all documents in $A$ just compare against the subsets which have at least one word in common, i.e. $m[w]$ for every word $w$ in the query.

Couple remarks:

Normally stop words would be excluded from the keys since they appear everywhere and they are not relevant for matching.
The key doesn't have to be a single word, it could also be an n-gram (or several n-grams) or even several words in case it fits in memory.

This kind of problem is frequent in the task of record linkage.


Predicting best time to send email I have a dataset of past emails sent, when they were sent and if they were opened. What would be the best model to predict the best time to send email?

I was thinking of modeling it as time series with values 1(email opened), 0(email not sent) and -1(email not opened).

Good question, probably a good handful of ways to look at this as it sounds like a classic classification problem.

Define the label

First step, before you choose a model, would be to define the label. As you suggested, I would go for email opened = 1 and email not opened = 0. I would then think about throwing the email not sent (-1) data away, as I'm not sure you can learn anything about a probability of opening when it wasn't sent.

It would then be important to check the balance of the label: are they mostly opened or mostly ignored? This will influence any decision to add a class weight if it is imbalanced towards one class.

Define features

So time series could work, but I think another good angle might be to bin the send times into half hourly slots (or shorter, depending on your needs) and treat them as a categorical variable. You would then one-hot encode this feature. Anything else you can add to this will probably help, such as the kind of person it was sent to and what country they're in etc.

Choose model

So it's classification, so have a play around with either logistic regression or something tree based (random forests, gradient boosted trees) to get something going quickly. It's usually a case of trial and error to find the best one.

You can then use traditional metrics of precision/recall/accuracy to measure how good it is. Make sure you know which metric is most important to your problem when optimising the model.


Understanding churn prediction model guys!

I have a dataset with a bunch of costumer-behavior features and the output being "Churned"/"Not churned". I applied a simple Random Forest Classifier and got a nice performance.

With this, I can predict whether or not a given user will churn. But I need to understand what are the patterns among churned users and patterns among non-churned users. How could I achieve that? (Where I could present something like "Usually, users that churn do this, that and that")

PS: No need for a full explanation, I'd happy enough if you give me some directions to what to study so I can achieve this knowledge

Many thanks in advance!

A couple good options would be to look at a Feature/Variable importance plot for your RF model.  Alternatively, depending on the model you could try extracting a couple individual trees from the model and examining them.  However, these methods  wouldn't be definitive; i.e. determining what variables are strong predictors for churn does not mean that they have a causal impact on churn, and an individual tree may be biased and not representative of the aggregation output presented by the RF model. To determine causation, you could use these methods as a starting point to design a test.


Which Machine Learning book to choose (APM, MLAP or ISL)? I'm searching a book as a refresher in machine learning (I have taken a lecture in machine learning sometime ago). I will be applying machine learning in a project.

I have searched a lot of books and found the following three the one which best fits me:

Applied Predictive Modeling by Kuhn and Johnson
Machine learning an Algorithmic Perspective (second edition) by Marsland
An Introduction to Statistical Learning by Hastie

Applied Predictive Modeling and An Introduction to Statistical Learning uses R as examples and Machine learning an Algorithmic Perspective uses Python. The used language does not matter that much because I'm not really interested in language examples (I'm using Matlab) and I will probably skip such examples.

The advantage of Applied Predictive Modeling seems to be that it covers the whole machine learning procedure (feature selection etc.) and seems to be very well written. The advantage of Machine learning an Algorithmic Perspective (second edition) is that it covers more topics (ensemble learning, graphical models, gaussian processes) and has perhaps a bit more math.

Which one of these three books would you recommend (and why)?

An Introduction to Statistical Learning by Hastie

It is like a bible for all those who want to get started at statistical analysis and Machine Learning.  It gives an excellent intuition about the theory behind all those models and algorithms, enough to pursue research and/or dive into another advanced ML book like the Applied Predictive Modeling by Kuhn and Johnson


Difference between Validation data and Testing data? I am bit confused about validating data. What is this data mainly for?? Like I am seeing some tutorial and they have some training[I know it] images , they they have some validation images[donot know] and some testing[I know] images? So what is validating images mainly for??

There are two uses for the validation set:

1) Knowing when to stop training

Some models are trained iteratively - like neural nets. Sooner or later the model might start to overfit the training data. That's why you repeatedly measure the model's score on the validation set (like after each epoch) and you stop training once the score on the validation set starts degrading again.

From Wikipedia on Overfitting:

"Training error is shown in blue, validation error in red, both as a function of the number of training cycles. If the validation error increases (positive slope) while the training error steadily decreases (negative slope) then a situation of overfitting may have occurred. The best predictive and fitted model would be where the validation error has its global minimum."

2) Parameter selection

Your model needs some hyper-parameters to be set, like the learning rate, what optimizer to use, number and type of layers / neurons, activation functions, or even different algorithms like neural net vs SVM... you'll have to fiddle with these parameters, trying to find the ones that work best.

To do that you train a model with each set of parameters and then evaluate each model using the validation set. Finally you select the model / the set of parameters that yielded the best score on the validation set.

In both of the above cases the model might have fit the data in the validation set, resulting in a biased (slightly too optimistic) score - which is why you evaluate the final model on the test-set before publishing its score.


Can you interpolate with QLearning or Reinforcement learning in general? I am currently researching the usages of machine learning paradigms for pathfinding problems. I am currently looking into the reinforcement learning paradigm and I used QLearning for pathfinding.

When there are not many states QLearning seems to be working well, but as soon as the environment gets bigger and the amount of states gets bigger it is performing rather bad. Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model? Is it possible with reinforcement in general or does it require to learn all possible states?

Since the convergence of QLearning is so slow I am wondering if it is possible with QLearning to interpolate the QValue of unexplored states since QLearning does not use a model?

When Q learning is described as "model free", it means that the agent does not need access to (or use) a predictive model of the environment. It cannot refer to state transitions and rewards in advance, but has to experience them in order to learn.

This does not mean that you have to avoid using a learning data model (such as a neural network) in order to generalise to new unseen data.

So, yes, Q learning can interpolate from unseen states and predict their Q value. To do this, you replace the state/action table with a supervised learning method based on descriptions of state $s$ and action $a$ as inputs, that you train as a regression model to predict $Q(s,a)$ (as a variant you can also have just state as input and predict $Q(s,a)$ for all possible actions as a vector in one go).

However, Q learning with a neural network suffers from instability. See Deep Mind's DQN paper for example of a system that solves that instability. In short:

Use experience replay - store S, A, R, S' data for each step and run the Q learning update on random mini-batches of the stored data, instead of online.
Keep two copies of the Q estimator neural network. Train one continuously, and copy it to a "frozen" version every now and then (e.g. every 100 mini-batches). Use the "frozen" copy to calculate the new $Q(s,a)$ targets.

This still might not match your learning scenario. If you want to solve mazes, think carefully about what data is truly available to the agent and how you might use it. For instance if you are using Q learning to solve a maze where you have a map, it is very inefficient approach. This is often shown as a toy problem, because it is possible to view the learning data very easily. But in that toy problem, the agent is not given the map, nor any knowledge of what a grid is.

Here are a couple of suggestions that may still help, separate to using a neural network value estimator:

If you do have a model of your environment (but not a map or other data that could be directly analysed for a solution), joining Q learning with a planning algorithm might work better for you than Q learning, as in Dyna-Q. This is relevant where you have an agent exploring in real time that would benefit from "looking" ahead before taking actions.
If your problem is very sparse rewards (due to larger maze, and only getting different reward at the end), then a worthwhile improvement is to look into multi-step TD learning, where the rewards are propagated back to previous steps more efficiently. Maybe look into $Q(\lambda)$


Detecting off state in the magnitude of accelerometer data? I have a univariate time series signal. It's the magnitude of an accelerometer attached to an engine.

I need to create an algorithm to detect off state, please see the black lines in the image below. The rest of signal is comprised of idle and active states. Idle state looks slightly higher and active state tend to have huge spikes and generally higher mean than both idle and off.

While this is simple, several machines have different mean values based on the size of the engine and the proximity of the sensor to the vibration. So the idea of using thresholds will not help.

I have considered K-Means algorithm. It worked pretty well when three states are available. When one or two are absent, the results are degraded significantly since it attempts to find two classes that don't exist in the data.

I have tried Hidden Markov Models. They looked promising, unfortunately they train themselves to identify the distribution of the states which will again change from one machine to another.

I thought of using standardisation. But I'm hesitant since the mean value of off state will change accordingly.

What unsupervised or semi-unsupervised approaches do you recommend on detecting off/idle/on states?

Seems to be pretty straight-forward thanks to your "well-behaving" data.

Simplest

The naive approach is to check histogram! As you see, the OFF state has generally a low magnitude. Knowing that level of magnitude from the labeld data you have above, you can threshold the histogram.

Why is it naive? :  because you see false positives (see the plot between 4 and 6 PM. there is a short state similar to OFF which is not OFF) and false negatives (immediately after 9 PM you see a peak in OFF state whose magnitude goes far beyond OFF)

what improves this naive approach? :

Taking the history into account (learning the sequential behavior)
Extracting more sophisticated features (e.g. in frequency domain using FFT or time-frequency domain using STFT, wavelet, etc.)
Applying an anomaly detection stage which helps you decide correctly about the peak which happened at 9 PM (that is pretty much an anomaly I assume)

A Bit More Complicated

Extract time frequency features (STFT, Wavelet, etc.) and in case you have labeled data, feed them to a classifier and in case you don't use those features to do some EDA to see if a clustering is possible there. According to your data I'm pretty confident you get good results.

Much More Complicated

I suppose Recurrence Analysis would be a wonderful option (and pretty fun for you!). Specially if the dynamic of the signal is non-stationary (many signals are in practice). The methods explained in the wikipedia page or in this page are not sophisticated.

Good Luck!


Why I would use TF-IDF after Bag-of-Words (CountVectorizer)? In my recent studies over Machine Learning NLP tasks I found this very nice tutorial teaching how to build your first text classifier:
https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a
The point is that I always believed that you have to choose between using Bag-of-Words or WordEmbeddings or TF-IDF, but in this tutorial the author uses Bag-of-Words (CountVectorizer) and then uses TF-IDF over the features generated by Bag-of-Words.
text_clf = Pipeline([('vect', CountVectorizer()),
...                      ('tfidf', TfidfTransformer()),
...                      ('clf', MultinomialNB()),
... ])

Is that a valid technique? Why would I do it?

This is the standard TF-IDF feature extraction: you transform the document counts.  It just looks odd to separate the two steps like this.  sklearn provides both TfidfTransformer and TfidfVectorizer; note the documentation of the latter:

Equivalent to CountVectorizer followed by TfidfTransformer.


Regularization for intercept parameter Why is the regularization parameter not applied to the intercept parameter?

From what I have read about the cost functions for Linear and Logistic regression, the regularization parameter (λ) is applied to all terms except the intercept. For example, here are the cost functions for linear and logistic regression respectively (Notice that j starts from 1):

The main aim of a regularization is to reduce the parameters values which in turn will reduce the effect of the feature associated with that parameter on the models prediction. For example, Since the intercept Theta0 basically doesn't have a feature associated with it, We don't penalize it.


Find specific topics with topic modelling I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better?

Topic Modelling with Mallet

I noticed that mallet finds interesting topics, but these topics just match partially with the topic i want to identify. So the idea is to iterate over all interesting k and search for a topic that meets the requirements with the highest percentage.

Find with elbow several (good coherence score) k topics (=list of good number of k)
Have a look at terms with a high frequency and (manually) map these terms to desired topics (= word list for each searched topic)
model topics for all good k and search for a good match with the word list

Example

I would like to classify/identify text that is about the topic "pollution of the environment"

[17, 15, 20, 33, 41, 42, 55, 120] k => with good coherence score
[dirt, pollution, emission, contamination] => high frequency, and these words would match with the topic
found with k=33 a topic which match these keywords.

This could be used to classify a unseen text (and later on for labeling the text for supervised learning)

Topic "pollution of the environment"

[('pollution', 0.09756215849203013),
   ('dirt', 0.09028743250670891),
   ('emission', 0.05491609816030263),
   ('contamination', 0.02589802450774354),
   ('sea', 0.02088654660674448),
   ('ocean', 0.017281515729574187),
   ('climate', 0.014694946490348864),
   ('crisis', 0.013676484852403893),
   ('waste', 0.011380904652591419),
   ('smog', 0.009812797051311068)])

You can use LDA (Latent Dirichlet Allocation), as input data it only needs different collections of text. As output it generates for each document the main topics (Like: Document1:0.3% Topic1, 0.2% Topic5, 0.2% Topic8, 0.2% Topic12, 0.1% Topic3) and for each Topic the different words with the amounts they contribute to that Topic. You can then browse the words of the Topics and manually decide how to name the label of each Topic.


R-GUI How do i aggregate survey data collected for multiple years and see if they contain a variable? I am new to R and usually rely on stata.

I have a considerably large data frame, it contains data from a broad range of surveys and years, each with their own types of classifications and answers. The variables have not been standardised. The data is at an Individual level.

Questions I have are the following:

How can I aggregate the surveys?
How can Identify the number of surveys which have provided information on a particular variable i'm interested in like education?

Thank you very much,

raj

If I understand your question correctly you'd like to obtain the proportion of missing values of a specific column in your dataset.
This can be obtained with the plyr library

For example:

library(plyr)
df <- data.frame(Study=c('Study_1', 'Study_1', 'Study_2', 'Study_3'), col1=c(NA,NA,5,8), col2=c(8, NA, NA, 7))

column <- 'col1'           # is the column your are interested in

ddply(df, .(Study), function(x){
  length(which(is.na(x[column])))/nrow(x)*100
})

Study  V1
Study_1 100
Study_2   0
Study_3   0

Study_1 has 100% missing values and the other studies none, so except Study_1 all studies contains some information


How do I transform a file to .txt file using pandas? I have to submit a machine learning project, and it has to be node in a .txt file. I know that if I am using pandas, and I want to transform a file from another format to .csv format I can use .to_csv().
Is there something similar for transforming the file into .txt() file using pandas?

pandas.to_csv() as you might know is part of pandas owned IO-API (InputOutput API).
Currently panas is providing 18 different formats in this context.
And of course pandas is also able to write txt-files. With other words pandas is a multi data converter:
Pandas IO tools
To do so with pandas use to_csv with the sep='\t' attribute:
df.to_csv('data.txt', sep='\t')
The next option you have to do it with numpy in that way, which is a bit laborious:

Call pd.DataFrame.to_numpy() to convert a pd.DataFrame to a NumPy array
Use np.savetxt(fname, X, fmt = "%s") with the string "%d" as fmt and the NumPy array as X to write the contents of X into the file fname.


LSTM followed by Dense Layer in Keras I am working on LSTMs and LSTM AutoEncoders, trying different types of architectures for multivariate time series data, using Keras.
Since it is not really practical to use relu in LSTM because of exploding gradients, I added a Dense layer following LSTM, so it is like:
model = Sequential()
model.add(LSTM(number_of_features, batch_input_shape = (batch_size, time_steps, number_of_features), return_sequences = True))
model.add(Dense(number_of_features))

What I want to know is:

Is this fully connected Dense layer connected to only the last step in LSTM? Or does it add a fully connected Dense layer for all time steps?

When I checked the number of parameters to be sure about this. Dense layer has number_of_features $\times$ (number_of_features + 1) parameters, which implies this Dense layer is applied to all time steps in LSTM network. This makes sense since I set return_sequences = True, but even when I set it to False, this does not change, which made me doubt my understanding.
So,

How does Dense work with LSTM with Return_Sequences?
What is its different from TimeDistributed layer?
Why changing return_sequences to False did not result in a reduction in number of parameters of Dense layer, from number_of_features $\times$ (number_of_features + 1), to (number_of_features + 1)?

I have been able to find an answer in Tensorflow Warrior's answer here.

In Keras, when an LSTM(return_sequences = True) layer is followed by Dense() layer, this is equivalent to LSTM(return_sequences = True) followed by TimeDistributed(Dense()).
When return_sequences is set to False, Dense is applied to the last time step only.
Number of parameters were same even when I set return_sequences = False because even though applied to all time steps, they shared the same parameters, that is after all what TimeDistributed() does.


similarity measure with two features I have some question concern similarity measure

Suppose that we have a matrix M where M(i,j) is the similarity measure between user i and user j .

Each user is characterised by : id-user | country | id-artist | id-track

For this I choose to use Jaccard similarity metric.

Jaccard is determined to compute similarity between users based on the tracks that they listened.
My question is : is it possible to take account both id-track and id-artist to measure the similarity between users?

Thank you

Yes, multiple different ways.

First, we could consider (id-artist,id-track) items as the elements of our set, and compute the Jaccard similarity by comparing those sets. Note that if the artist's id gives us no additional information beyond the track id, this will give the same result, whereas it will give different results if a particular track id could be associated with multiple artists.

Second, we could compute the Jaccard similarity on tracks, and then the Jaccard similarity on artists, and then add the two (probably with some constant coefficient scaling the two). This way two users who listen to the same artists, but none of the same songs by those artists, will be rated as more similar than users who listen to different songs by different artists.

What coefficient makes sense? Well, you could start with 0.5 for each (i.e. just .5*similarity_artist+.5*similarity_track), see if that's reasonable, and adjust if it's not.


Why class weight is outperforming oversampling? I am applying both class_weight and oversampling (SMOTE) techniques on a multiclass classification problem and getting better results when using the class_weight technique. Could someone please explain what could be the cause of this difference?

You should not expect class_weight parameters and SMOTE to give the exact same results because they are different methods.

Class weights directly modify the loss function by giving more (or less) penalty to the classes with more (or less) weight. In effect, one is basically sacrificing some ability to predict the lower weight class (the majority class for unbalanced datasets) by purposely biasing the model to favor more accurate predictions of the higher weighted class (the minority class).

Oversampling and undersampling methods essentially give more weight to particular classes as well (duplicating observations duplicates the penalty for those particular observations, giving them more influence in the model fit), but due to data splitting that typically takes place in training this will yield slightly different results as well.

SMOTE creates new observations of the minority class by randomly sampling from a set of "similar" minority class observations. Synthesized observations are computed based off adding a random percentage of the difference between two randomly chosen "similar" observations, for each coordinate (i.e. column). Similar observations are defined typically using the k most closest neighbours to a particular observation of the minority class. This means that, depending on the value of k chosen, as well as numerous other factors such as how similar your observations are in general, the distance measure, etc. SMOTE may or may not be useful for your particular problem.

Methods that deal with class imbalance do not all work the same and is a large area of study. It appears you have noticed that the class weight method is more effective, which is why any sort of method chosen in addressing any potential class imbalance problem needs to be wrapped within a model validation scheme to see if the particular method (or really, using any method at all) yields any benefit.


clustering plus linear model versus non linear (tree) model a team has to create models that predict the cost of deploying a machine over time. This is a regression problem.

The team is further divided into two groups, A and B.

Group A puts lots of attention on selecting attributes and finding rules that would segment the training set into clusters that are more or less homogeneous, and then uses a linear model to create predictions within the cluster.

Group B does not do clustering first, but instead includes the same attributes that team A uses for clustering into a non-linear model (let's say an ensemble of random forests or gradient boosting machine).

The results are similar (or slightly better using the non linear model).

How are results measured? mean squared error over a hold-out set.

The explanation appears to be that, by definition, tree models segment the population using attributes, so that the segments are as homogeneous or pure as possible, given the attributes.

So the work team A is doing to cluster the instances, the tree model is is also doing per se - because segmentation is embedded in tree models.

Does this explanation make sense?

Is it correct to infer that the approach of group B is less demanding in terms of time? (i.e. the model finds the attributes to segment the data as opposed to selecting the attributes manually)

With regards to the end of your question:

So the work team A is doing to cluster the instances, the tree model is is also doing per se - because segmentation is embedded in tree models.
Does this explanation make sense?

Yes, I believe this is a reasonable summary. I wouldn't say the segmentation is "embedded" in the models but a necessary step in how these models operate, since they attempt to find points in the variables where we can create "pure clusters" after data follows the tree down to a given split.

Is it correct to infer that the approach of group B is less demanding in terms of time? (i.e. the model finds the attributes to segment the data as opposed to selecting the attributes manually)

I would imagine that relying on the tree implementation to derive your rules would be faster and less error prone than manual testing, yes.


Non-mutal exclusive classification task examples I am reading the excellent Hands-on Machine Learning with Scikit-Learn and TensorFlow and in chapter 10, the author says:

  "For the output layer, the softmax activation function is generally a
  good choice for classification tasks (when the classes are mutually
  exclusive). For regression tasks, you can simply use no activation
  function at all."

All classification problems I can think of (binary classification, image classification, etc) generate mutually exclusive classes.

Can someone give me a few examples of non-mutual exclusive classification problems?

For example, due to the complexity of the images in the ImageNet database. Algorithms will often use hundreds or thousands of output nodes to be capable of classifying a large array of different things. Researchers also relax the cost function and allow the $k$ highest outputs to be considered. If one of these is correct then the example is considered to have been correctly classified. Furthermore with the existence of such complex data, often times a certain object might be a subset of another object.

For example, consider the picture of a human being. If the algorithm detects a human with the highest activation at the output node, but also has high activations for hands, eyes, face. This is not wrong! The image does in fact contain these objects as well and them being detected by the model is encouraging.


Subdivide a numerical vector with a normal distribution I have a numerical array of prices values. I'd like to do classification on this parameter, so I'd like to create a certain number of classes with the same granularity. I'd like to create a generalized function that given the array and the number of classes i'd like to have it automatically return the price intervals for those classes.
Currently this is my price attribute:

df_cleaned.price.describe()
>>>
count    122668.000000
mean      11253.349594
std        7856.513917
min        1010.000000
25%        4995.000000
50%        8995.000000
75%       15965.000000
max       34991.000000
Name: price, dtype: float64

And I created a function manually to create 6 classes and it looks like this:

def normalize_price(df):
    cond = [
        (df['price'] >= 1000) & (df['price'] <= 4999),
        (df['price'] >= 5000) & (df['price'] <= 8999),
        (df['price'] >= 9000) & (df['price'] <= 15999),
        (df['price'] >= 16000) & (df['price'] <= 24999),
        (df['price'] >= 25000) & (df['price'] <= 34000),
        (df['price'] >= 34001) & (df['price'] <= 40000)
    ]

    choice = [
        1000,
        5000,
        9000,
        16000,
        25000,
        35000,
    ]

    df['price'] = np.select(cond, choice, df['price'])
    return df

Can anyone help me improvint this function? pls thanks

You seem to be describing the method cut in pandas (documentation)

This method does exactly what you want, if you want to separate a dataframe into n equal-sized bins or manually specify the ranges.

Example:

df = pd.DataFrame(np.array([10, 50, 99, 140, 250, 300, 450, 499]), columns=['price'])

bins = pd.IntervalIndex.from_tuples([(0, 100), (100, 300), (300, 500)])
df['price'] = pd.cut(df['price'], bins)

Output:

You can simply specify the number of bins you want:

df = pd.DataFrame(np.array([10, 50, 99, 140, 250, 300, 450, 499]), columns=['price'])

df['price'] = pd.cut(df['price'], bins=3)

Output:

I hope this helps improve your code.


Is it appropriate to use polar images as input to CNNs? Or must they be Cartesian transformed first? I have built a convolutional neural net which trains on data originally represented in polar space (measurements are a function of angle and distance). My pipeline begins by converting coordinates to a Cartesian grid and re-projecting the images. This however results in a significant increase in the data dimensions (i.e. pixel size of the input images) and requires interpolation steps which I would prefer to avoid.

Is there an appropriate method (implemented in keras/tensorflow) I can use for performing (2D) convolutions on the raw polar data? I have followed the cs231n course, so I have some background but am not yet an expert, especially on the theory side. Thanks!

At the basic level, CNNs work by finding spatially-linked correlations i.e. places in the input which often appear close together. For this reason, creating cartesian projections of your data from the polar information sounds like the natural way to go about it.

If there is some inherent structure to your raw data, it might be possible to use something like a CNN, but you'd have to think carefully about the architecture.

Here are a couple of other thoughts that cross my mind that might help you brainstorm or find related research papers:

Casting the polar data into images, as you describe your method, not only increases the dimensions of the data, but usually the sparsity of the data. This makes it very difficult to train something like a CNN, which generally works best for dense chunks of information, such as a photograph. If your data is sparse, you might consider some other pre-processing such as adding blur to the projections.
Polar coordinates with a distance metric is essentially the same as a point cloud. If you represent a series as scans, instead of a set of images, but rather as a 3d pointcloud, you could look into models such a PointNet, PointNet++, VoxelNet (see the example projects at the bottom of that linked webpage). There are examples of object detection or segmentation in 3d pointclouds, which might give you other ideas for your case - all available openly in Tensorflow or another DL framework.
How well do you understand your data? Perhaps some further exploration or visualisation might help spark some ideas or at least provide a better feeling for approaches that could work. Try plotting the cartesian coordinates of your data (if you can map many frames to a global origin), using something like PyntCloud (an example).


Spacy Returns Nonidentical Results for Doc. Examples? I am getting started with Spacy, and I was just experimenting the given examples. It is weird though that when I run them over their containers using their experimental console it gives me different results than when I run them locally.

For example, when I run the following example from their console,

import spacy

nlp = spacy.load('en_core_web_md')  # make sure to use larger model!
tokens = nlp(u'dog cat banana')

for token1 in tokens:
    for token2 in tokens:
        print(token1.text, token2.text, token1.similarity(token2))

It gives me:

dog dog 1.0
dog cat 0.80168545
dog banana 0.24327646
cat dog 0.80168545
cat cat 1.0
cat banana 0.2815437
banana dog 0.24327646
banana cat 0.2815437
banana banana 1.0

But when I run the exact same thing locally, it gives me:

dog dog 1.0
dog cat 0.0
dog banana 0.0
cat dog 0.0
cat cat 1.0
cat banana -0.044681177
banana dog -7.828739e+17
banana cat -8.242222e+17
banana banana 1.0

To me the former makes sense, not my local result. And the problem persists for other examples as well. I am so confused! What is happening locally?

Spacy: '2.0.12'
Python 3.6.5 Anaconda, Inc.
Mac OSx 10.13.5

I am not sure if it is relevant, but I was getting this error running it in Jupyter, so I exported the followings, as suggested in the git, in the terminal before launching the Jupyter:

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

As I commented on your post - I think you have a problem with your environment. If you cannot solve that, you should file a bug with the spacy team on their GitHub.

I tested the same code and could only see a difference in results compared to the website when using language models of different sizes. If you installed the English model using the default command: python -m spacy download en, then you get the smallest model by default.  You should be using the medium-sized model to produce their results. I compare the two below

The script

import spacy

# Small setup
nlp_small = spacy.load('en')
tokens_s = nlp_small(u'dog cat banana')

# Medium setup
nlp_medium = spacy.load('en_core_web_md')
tokens_m = nlp_medium(u'dog cat banana')

# large setup would use 'en_core_web_lg' ...

print('Results using small model:\n')
for token1 in tokens_s:
    for token2 in tokens_s:
        print(token1.text, token2.text, token1.similarity(token2))

print('\nResults using medium model:\n')
for token1 in tokens_m:
    for token2 in tokens_m:
        print(token1.text, token2.text, token1.similarity(token2))

The output

Results using small model:

dog dog 1.0
dog cat 0.53906965
dog banana 0.28761008
cat dog 0.53906965
cat cat 1.0
cat banana 0.4875216
banana dog 0.28761008
banana cat 0.4875216
banana banana 1.0

Results using medium model:

dog dog 1.0
dog cat 0.8016855
dog banana 0.24327648
cat dog 0.8016855
cat cat 1.0
cat banana 0.28154367
banana dog 0.24327648
banana cat 0.28154367
banana banana 1.0

The results are clearly better when using the medium sized model - and they match the results on the website.

Here is the spacy version as used within a conda environment to produce these results:

n1k31t4@n1k31t4~$ conda list | grep spacy
spacy                     2.0.12                    <pip>

And the Python version information:

n1k31t4@n1k31t4:~$ ipython
Python 3.6.6 | packaged by conda-forge | (default, Jul 26 2018, 09:53:17)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

Although I ran the script on Linux (OP used MacOSX) and my Python version is slightly newer... I wouldn't expect these things to explain the difference in results.


Dummies Variables and Scaling in Regression Problems I was wondering if having dummies variable and scaling other variables could joke my model. In particular, I have implemented a Random Forest Regressor by using scikit-learn, but my data model is composed by a set of dummies varibles and 2 numerical variables. I approached in this way:

Convert categorical in  dummies variables
Separate the numerical variables
Scale with Standard Scaler from scikit-learn the numerical variables (at point 2)
Join the dummies and numerical
Split train, test
train the model

Would this approach create an inappropriate bias considering the different scale from dummies and the scaler numerical?
Or, at least, is it correct?

Would this approach create an inappropriate bias considering the different scale from dummies and the scaler numerical? Or, at least, is it correct?

"It is fine" because One-Hot-Encoded features don't have a very large scale. Issue of scale comes when features are at a very different scale e.g. 1 vs 105 etc.
Standardizing them will disturb the sparsity(i.e. log of zeroes). Normalizing them will not have any impact.
What is incorrect here is -
 You are standardizing the dataset prior to the split. You must do the other way round.
What is optional here is -
 You don't need scaling and OHE for a Tree-based model. Just label encode, split and move.


Using classification of previous sample in neural networks I am trying to classify the state of a machine using different features coming from a set of sensors.

I am treating the problem like a time series, so I windowed the stream of the sensors each X seconds so that every sample has size ( X * s_f ).

I am obtaining fairly good results using a CNN but I would like to increase performances using some kind of approach that understands the patterns between different states.

For example if going from state A to B is more frequent than A->C I want the algorithm to include this in the classification.

My idea is to classify each sample taking into account the classification of the previous sample. How can I do this?

The approach you describe is predicting base on the features of the current step and the previous states.
If you believe that there is information on the previous states, than i would try to approach this problem in the Time Series approach.

The algorithms in this approach basically do what you request for, they predict based on the features of the current step and based on the previous states.
You can learn about the approach to understand it better in this cs231n video.

You can also look for tutorials on implementing models that fit the time-series problem.
The most popular model i know is caleld LSTM.
Here is a tutorial on implementing LSTM in keras.

I hope you will find it these helpful.
If you have any more questions or you would like more guidance, just let me know :)


Optimal Dimension of Graph(Vertex) Embedding Let's define a embedding of a graph structure G = (V,E) where $\mid V\mid=v, \mid E \mid=e$
Now define an embedding $f: V \to R^d$ where $d\in \Bbb N$, an optimal dimension of embedding which contains every edge information of $G$.
(G is a directed graph and there exist no weight, thus it's not a network.)
I'd like to find an infimum formula of $d$ represented with $v$ and $e$.

[Backgroud of this problem]
I am trying to construct a neural network which can discern whether the given explanation of word is about the word "be" or the word "exist".
For example "having a real existence" is "exist".

To do this, first I need to find the most smallest dimension of each words' corresponding embedding of vertex for training of my network.

I think that finding the absolute dimension of expressivity is a difficult problem. Here are some important facts to consider when you come with a word embedding size.

Make sure you leave enough dimensions for expressivity. You want to
make sure that there is at least enough dimensions to express the
complexity of the structure you are looking to encode
That the number of dimensions is not too large so that you suffer some difficulties in computation.

To conclude, finding the infimum can be done post-hoc, in the sense that you can evaluate the performance on the function and see which value of $d$ does not significantly decrease the performance and choose it this way. However, at a post-hoc level, the only way I can see of doing this is by evaluating the embedding space beforehand, I have not read too much literature about this.

https://arxiv.org/pdf/1711.00331.pdf - Semantic Structure and Interpretability of Word Embeddings

There is also a lot of work done to evaluate the embedding schemes that we have chosen. In the recent NAACL conference there has been work that has been done in this domain particularly.


Impact calculation- Regression I am dealing with the below data:
\begin{array} {|c|c|}\hline \text{user} & \text{A_PDE} & \text{B_PDE} & \text{C_PDE} & \text{D_PDE} & \text{Sales} & y_\text{pred} & \text{SSE} & \hat{y} \\ \hline 1 & 0.50 & 0.25 & 0 & 1.75 & 0 & 8.660412 & 75.00272 & 0 \\ \hline 2 & 0.00 & 0.00 & 1 & 0.00 & 0 & 4.02256 & 16.18099 & 0 \\ \hline 3 & 0.50 & 1.25 & 1 & 0.75 & 44 & 13.99656 & 900.2064 & 615.8487 \\ \hline 4 & 1.25 & 1.00 & 0 & 0.00 & 0 & 11.02223 & 121.4896 & 0 \\ \hline 5 & 0.00 & 0.75 & 0 & 1.50 & 0 & 7.240974 & 52.43171 & 0 \\ \hline  \end{array}

I am trying to calculate the impact of each of the channels A_PDE, B_PDE, C_PDE, D_PDE. However, since Sales = 0 for most of the records, and my model is predicting non zero sales for those records, my $R^2$ value is very low. How can I fix this? I don't think removing the zero sales record from the model would be right.

Dimensionality reduction based answer: You should consider reducing the dimensionality of your data set. There are multiple ways to reduce the dimensionality of your data set

Reduction by variance: remove variables with very low variance
Reduction by correlation: remove variables that are only weakly correlated with your criterion. For this step, it is important that you split your dataset into training and validation data set before you calculate the correlations or your data results will be contaminated with information from the validation data set.
Reduction with principal components analysis (PCA)
Reduction with decision trees by feature selection
$R^2$ based reduction: Exclude one variable of your data set and calculate the $R^2$. Then exclude another variable of your data set and calculate the $R^2$. Repeat this process for all the variables and remove the variable for which the $R^2$ decreased the least or increased the most. Repeat this process for the reduced data set. Stop when the $R^2$ change is similar for all the variables.
$p$-value based reduction: Determine the weights and remove the weights for which the $p$-value is not significant. For this procedure, you have to consider the cumulation of errors and apply Bonferroni correction.

Regression method based answer: An alternative way without reducing the dimensionality would be to use weighted least squares regression and use higher weights on the sales if you really want to predict them more accurately. If your data is highly nonlinear you could also consider using nonlinear regression to capture the nonlinearity with your model.


Does central limit theorem work well for Pareto distribution? I am new to Data Science. Recently I was studying a course about statistics. One of the tasks there was to check the central limit theorem in practice.

The idea was quite simple: take a continuous random variable; generate, say, 1000 samples from it, each of size n; draw a histogram of the samples. Then find the parameters of the normal distribution using the central limit theorem and draw the PDF of the distribution. As a result, the histogram and the PDF should be, roughly speaking, "similar" (and become more "similar" as n grows).

I chose Pareto distribution and, with this Python code,

import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as sts
from math import sqrt

n = 10

b = 3.0
random_value = sts.pareto(b)
mean = random_value.mean()
variance = random_value.var()

plt.hist(samples(random_value, n, 1000), bins=20, normed=True)

x = np.linspace(0, 6, 100)
pdf = sts.norm(mean, sqrt(variance / n)).pdf(x)
plt.plot(x, pdf, color='r', label='theoretical PDF')

where the samples function was like this

def samples(random_value, sample_size, number_of_samples):
    result = np.asarray([])
    for i in range(number_of_samples):
        result = np.append(result, np.mean(random_value.rvs(sample_size)))
    return result

I got the next histogram and PDF as a result

The result looked suspicious to me: there was no expected similarity. I checked the same code with another continuous distribution (with a uniform distribution), and the graphs looked much more similar.

Is it a feature of the Pareto distribution which makes it a bad choice to demonstrate how the central limit theorem works? Or something is wrong with the code itself (e.g., the parameters of the normal distribution are calculated in a wrong way)? Thank you in advance.

One key assumption of the central limit theorem is that the variance $\sigma^2$ of the underlying distribution is finite.  I would guess that you are using a parameterization of the Pareto for which $\sigma^2 = \infty$, in which case the CLT will not hold.  If you tweak your scale parameter so that $\sigma^2 < \infty$ you should see the result you want.


List of used aliases in Tableau I often get the problem when this or that alias name is already used somewhere, and I can't easily find that variable or aggregation to release the name.

Is there some place in Tableau where I can view/edit/reset full list of aliases?

Finally solved the issue.
You should go to:

Data –> your_data_source –> Edit Aliases –> Measure Names

And see full mapping between variables/aggregations and aliases.


Calibration using predict_proba vs class_weight I am making a Random Forest Classifier to determine whether a sentence is "positive" (1), "negative"(-1) or "neutral"(0).
However, I prefer having false negative than false positive, that is, I prefer saying that a sentence is neutral even if it's not than to say that a sentence is positive when it's neutral, a fortiori if it's negative.
So I use predict_proba, with something like:
def my_pred(rfc, X, weight=0.5):
  res = rfc.predict_proba(X)
  if res[0]>weight: return -1
  elif res[2]>weight: return 1
  return 0

But I wonder if I can make such things (give more importance to the neutral class) using class_weight? Would it be better?

There are several options for you:

class_weight should boost the loss function towards the preferred class. This option is supported by various estimators, including sklearn.linear_model.LogisticRegression,

sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc.

You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:
  y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,

                               method="decision_function")

  precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)

If you plot the precisions  and recalls against the thresholds, you should see the picture like this:

After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification.

Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).
As said there are 2 stages to make this Tuning: in the model training stage (like custom weights) and the prediction stage (like lowering the decision threshold).
Another tuning for the model-training stage is using a recall scorer. you can use it in your grid-search cross-validation (GridSearchCV) for tuning your classifier with the best hyper-param towards high recall.
GridSearchCV scoring parameter can either accepts the 'recall' string or the function recall_score.
Since you're using a binary classification, both options should work out of the box, and call recall_score with its default values that suits a binary classification:

average: 'binary' (i.e. one simple recall value)

pos_label: 1 (like numpy's True value)

Should you need to custom it, you can wrap an existing scorer, or a custom one, with make_scorer, and pass it to the scoring parameter.


Python distributed machine learning I occasionally train neural nets for my research, and they usually take quite a long time to run (especially when I'm working on my laptop).

I'm looking for a way to build the model on any computer and send it up to a server for training and have it return the graphs/accuracies/weights etc. I know there are paid solutions for this but I'm looking for a distributed solution I can run myself.

I have a server set up at home which is about to get a CPU and GPU upgrade. I'd like to be able to set it up so that when I'm working on the LAN, or when I'm working remotely on my laptop, I can send code to the server and have it train the model and return to me the results (or save the results if the sender machine is switched off)

Are there any existing solutions to accomplish something like this? I'm not tied to any specific library, but would prefer to stick with Python if possible

This shouldn't be terribly complicated.

Big picture

Assuming you have Linux on your server: SSH to your server from your work laptop, train the network, receive results (the trained network).

Details

Train the network

If you want to stick with Python, there are two basic options that I am familiar with.

The first is PyBrain, a library built specifically for training neural nets. The syntax is fairly straightforward. The dataset structure is a bit unusual (especially if you are used to just using numpy arrays in scikit-learn), but otherwise, it works rather well. However, it doesn't support GPUs AFAIK, and I don't know how optimized/fast it is.

The second is Google's Tensor Flow. A bit of a heavy-weight if you just want to run vanilla neural nets, but the syntax is also pretty easy to get around (although very different from PyBrain). It's also probably faster than anything else you could find in Python. It also supports GPU training.

Get the results back

This depends on the library you choose (or it could be definitely up to you write your own implementation). But both libraries mentioned above will let you return the trained weights, which you can save as csv files and download upon training. Or you could pickle them and get back that way. Also, Tensor Flow has this Tensor Board functionality which should let you visualize the training and the network structure, but I haven't tried it yet, so can't help on that front.


Standardize numbers for ranking ratios I'm trying to rank some percentages. I have numerators and denominators for each ratio. To give a concrete example, consider ratio as total graduates / total students in a school.

But the issue is that total students vary over a long range (1000-20000). Smaller schools seem to have higher percentage of students graduating, but I want to standardize it, and not let the size of the school affect the ranking. Is there a way to do it?

This is relatively simple to do mathematically.  First, fit a regression line to the scatter plot of "total graduates" (y) vs. "total students" (x).  You will probably see a downward sloping line if your assertion is correct (smaller schools graduate a higher %).

You can identify the slope and y-intercept for this line to convert it into an equation y = mx + b, and then do a little algebra to convert the equation  into normalized form: "y / x = m + b / x"

Then, with all the ratios in your data , you should subtract this RHS:

normalized ratio = (total grads / total students) - (m + b / total students)

If the result is postive, then the ratio is above normal for that size (i.e. above the regression line) and if it is negative it is below the regression line.  If you want all positive numbers, you can add a positive constant to move all results above zero.

This is how to do it mathematically, but I suggest that you consider whether it is wise, from a data analysis point of view, to normalize by school size.  This depends on the purpose of your analysis and specifically how this ratio is being analyzed in relation to other data.