Train on batches in Tensorflow I'm currently trying to train a model on a large csv file (>70GB with more than 60 million rows). To do so I'm using tf.contrib.learn.read_batch_examples.
I'm struggling in understanding how this function actually reads the data. If I'm using a batch size of e.g. 50.000, does it read the first 50.000 lines of the file? If I want to loop over the whole file (1 epoch) do I have to use num_rows/batch_size = 1.200 number of steps for the estimator.fit method?

Here is the input function im currently using:

def input_fn(file_names, batch_size):
    # Read csv files and create examples dict
    examples_dict = read_csv_examples(file_names, batch_size)

    # Continuous features
    feature_cols = {k: tf.string_to_number(examples_dict[k],
                                           out_type=tf.float32) for k in CONTINUOUS_COLUMNS}

    # Categorical features
    feature_cols.update({
                            k: tf.SparseTensor(
                                indices=[[i, 0] for i in range(examples_dict[k].get_shape()[0])],
                                values=examples_dict[k],
                                shape=[int(examples_dict[k].get_shape()[0]), 1])
                            for k in CATEGORICAL_COLUMNS})

    label = tf.string_to_number(examples_dict[LABEL_COLUMN], out_type=tf.int32)

    return feature_cols, label

def read_csv_examples(file_names, batch_size):
    def parse_fn(record):
        record_defaults = [tf.constant([''], dtype=tf.string)] * len(COLUMNS)

        return tf.decode_csv(record, record_defaults)

    examples_op = tf.contrib.learn.read_batch_examples(
        file_names,
        batch_size=batch_size,
        queue_capacity=batch_size*2.5,
        reader=tf.TextLineReader,
        parse_fn=parse_fn,
        #read_batch_size= batch_size,
        #randomize_input=True,
        num_threads=8
    )

    # Important: convert examples to dict for ease of use in `input_fn`
    # Map each header to its respective column (COLUMNS order
    # matters!
    examples_dict_op = {}
    for i, header in enumerate(COLUMNS):
        examples_dict_op[header] = examples_op[:, i]

    return examples_dict_op

Here is the code im using to train the model:

def train_and_eval():
"""Train and evaluate the model."""

m = build_estimator(model_dir)
m.fit(input_fn=lambda: input_fn(train_file_name, batch_size), steps=steps)

What would happen if I would call the fit function again with the same input_fn. Does it start at the beginning of the file again, or will it remember the line where it has stopped last time?

As there is no answer yet I want to try to give an at least somehow useful answer.
Including the constants definitions would help a bit to understand the provided code.

Generally speaking a batch uses n times a record or item. How you define an item depends on your problem. In tensorflow the batch is encoded in the first dimension of a tensor. In your case with the csv file it might be line by line (reader=tf.TextLineReader). It could learn by column but I don't think that this is happening in your code. If you want train with your whole dataset (=one epoch) you can do so by using numBatches=numItems/batchSize.


What is the use of additional column of 1s in normal equation? Currently I am going through Normal Equation in Machine Learning.

$$
\hat\theta = (X^T \cdot X)^{-1} \cdot X^T \cdot y
$$

But when I see how they use this equation, I found they always add an additional column of 1s in the starting of matrix X before transposing.

I don't understand why. What's the logic behind this?

The places where I found such things

1) Coursera - Theory

2) Implementation

  Now let’s compute using the Normal Equation. We will use the inv()
  function from NumPy’s Linear Algebra module (np.linalg) to compute the
  inverse of a matrix, and the dot() method for matrix multiplication:

  X_b = np.c_[np.ones(( 100, 1)), X] # add x0 = 1 to each instance

  Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn and
  TensorFlow: Concepts, Tools, and Techniques to Build Intelligent
  Systems (p. 111). O'Reilly Media. Kindle Edition.

The normal equations are designed such that each coefficient in the model has an input of some kind it's being multiplied against. The column of ones is the "input" to the intercept term.


Pruning in Decision trees Following is what I learned about the process followed during building and pruning a decision tree, mathematically (from Introduction to Machine Learning by Gareth James et al.):

Use recursive binary splitting to grow a large tree on the training
data, stopping only when each terminal node has fewer than some
minimum number of observations.
Apply cost complexity pruning to the large tree in order to obtain a
sequence of best subtrees, as a function of α.
Use K-fold cross-validation to choose α. That is, divide the training
observations into K folds. For each k = 1, . . .,K:
(a) Repeat Steps 1 and 2 on all but the kth fold of the training data.
(b) Evaluate the mean squared prediction error on the data in the
left-out kth fold, as a function of α.
Average the results for each value of α, and pick α to minimize the
average error.
Return the subtree from Step 2 that corresponds to the chosen value
of α.

My question: Going through the above algorithm means we automatically choose the most optimum tree based on minimum average error. Why then, do we have to select maximum depth as the pruning parameter. Shouldn't optimum depth be decided by the algorithm itself and not us?

If anything, shouldn't we be choosing 'K' here, as in how many-fold validation would we like to do?

Can you clarify pls, where is the suggestion, that we have to select max depth for pruning? As you said it is supposed to be done automatically due to some criterion.
Here is some example of post-pruning:

https://stackoverflow.com/questions/49428469/pruning-decision-trees

Yes, we should select 'K' fold to leave some data for the test of pruning efficiency. How many? It depends on your data. You can check out this Stanford lecture slides, for example, where K = 10 is suggested:

https://web.stanford.edu/class/stats202/content/lec19.pdf

as an example. Usually, the K will depend on the size of your data, based on the size of the test fold you want to leave out.


Better way to break out a date? This piece of code should be self-explanatory and it's also a choke-point in my process. Any way to do this better?

geoset['MonthInt'] = pd.to_datetime(geoset['Date received']).dt.month # Add a month
geoset['YearInt'] = pd.to_datetime(geoset['Date received']).dt.year # Add a year
geoset['DayInt'] = pd.to_datetime(geoset['Date received']).dt.day # Add a day of month
geoset['DayOfWeekInt'] = pd.to_datetime(geoset['Date received']).dt.dayofweek # Add a day of week

Obvious optimisation would be converting 'Date received' to pandas datetime once, for example (if you need to retain original column):

geoset['pddate_received'] = pd.to_datetime(geoset['Date received'])
geoset['MonthInt'] = geoset['pddate_received'].dt.month # Add a month
geoset['YearInt'] = geoset['pddate_received'].dt.year # Add a year
geoset['DayInt'] = geoset['pddate_received'].dt.day # Add a day of month
geoset['DayOfWeekInt'] = geoset['pddate_received'].dt.dayofweek # Add a day of week


What is max aggregation on a set of word embeddings? In a paper I see:

$\mathcal{Q}$ is a set of words.

$\psi_{G^w}$ are word embeddings.

so, $\{\psi_{G^w}(w_t), \forall w_t \in \mathcal{Q}\}$ gives me a set of embeddings for all words in $\mathcal{Q}$.

For example, if I have:

Q = {'a', 'b', 'c'}
embedding_gw.shape = (1000, 8) # 1000 words in vocab, embedding size is 8

I will get:

{
  [1,2,3,4,5,6,7,8], #embedding of 'a'
  [8,7,6,5,4,3,2,1], #embedding of 'b'
  [4,5,3,6,7,8,1,2]  #embedding of 'c'
}

The result of $\max\{\psi_{G^w}(w_t), \forall w_t \in \mathcal{Q}\}$ is supposed to be a single vector.

My question is, how do I get this single $\max$ vector?

Do I sum all values in each embedding and pick the largest one?
Do I pick the $\max$ value of each $i^{th}$ position, creating a completely new vector?
Do I do something else?

The second option is the right one. From the paper:

  In this case, we perform a component-wise max over the start
  embeddings of all instances of w to obtain the best features across
  all occurrences of w.

The previous quote can be found a few paragraphs before equation 17, second column, top of the page.


Activation Functions What is the purpose of linear activation functions in keras, isn't the entire point of activation functions to introduce non-linearity?

A combination of multiple linear functions can help one model complex decision regions. Two or more linear functions can be combined to form a piece-wise approximation of a non linear function. Computation of non-linear activation function on a very deep network can be expensive (because of the exponential term involved). ReLu activation function which is a linear activation is used almost by every CNN model and is proved to give better results.
$$ f(x) = max(0,x-1) + max(0,x+1)$$
Consider the function above which is combination of two reLu's.  This approximates the sigmoid function(try plotting the same) and is computationally cheap.


input shape of keras Sequential model i am new to neural networks using keras,
i have the following train samples input shape (150528, 1235) and output shape is (154457, 1235) where 1235 is the training examples,
how to put the input shape, i tried below but gave me a
ValueError: Data cardinality is ambiguous:
  x sizes: 150528
  y sizes: 154457
Please provide data which shares the same first dimension.

code:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy

physical_devices = tf.config.experimental.list_physical_devices('GPU')
print("Num GPUs Available: ", len(physical_devices))
tf.config.experimental.set_memory_growth(physical_devices[0], True)

model = Sequential([
    Dense(16, input_shape=(150528,), activation='relu'),
    Dense(32, activation='relu'),
    Dense(154457)
])

model.compile(optimizer='sgd', loss='mse', metrics=[tf.keras.metrics.MeanSquaredError()])
model.fit(im_features, amp_vo_features, batch_size=10, epochs=30, shuffle=True, verbose=2)

You just need to make sure that your x and y values have the same row dimension. This can be done using numpy.transpose or numpy.swapaxes.
import numpy as np

t = np.random.rand(150528, 1235)
np.transpose(t).shape # (1235, 150528)
np.swapaxes(t, 0, 1).shape # (1235, 150528)

This should give the shapes of (1235, 150528) for x and (1235, 154457) for y.


What is wrong with the below code? I have been working on a project which i took from kaggle. I didn't get the result as mentioned in the website. What am I doing wrong here?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

train = pd.read_csv('../data/dont-overfit/train.csv')
test = pd.read_csv('../data/dont-overfit/test.csv')

train[train.columns[2:]].std().plot('hist')

train[train.columns[2:]].mean().plot('hist')
plt.title('Distribution of means/stds of all columns')

plt.show()

print(train.isnull().any().any())

print('Distribution of first 28 columns')
plt.figure(figsize=(26,24))
for i, col in enumerate(list(train)[2:30]):
    plt.subplot(7, 4, i+1)
    plt.hist(train[col])
    plt.title(col)

plt.show()

Here is the post

the code seems ok plotting the distribution. let me know the exact error.


How can I preprocess multi-page image inputs in a theano/lasagne network? I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no  photographs or icons), and different documents may have a different shape (height and width). I'd like my classification approach to use all pages in each document, rather than just the first page.

As far as I know, the input to a CNN (like theano) needs a standardized shape. My first thought was to create a single image array that has all the pages concatenated. But I would then have to resize/zero pad all concatenated pages to match the length of the document with the largest page count, and use that as my lowest height and width for the set of concatenated pages. If I don't use this strategy I risk losing resolution on the words in the image, but that is a giant input vector.

I feel like splitting each document's pages into separate samples would be a better approach to standardize input preprocessing, but I'm at a loss for how to classify the whole document if I'm just training on loose pages of each document. Can anyone advise on me on this?

First point is that convolutional neural networks would be incredibly expensive to train on images that large. The minimum size you could allow would be num_pixels_per_letter x num_letters_per_row_or_col.

It sounds like you want to achieve two tasks: OCR then document classification. Given that you have predominantly text in these documents, you would almost certainly be better served by using a more a traditional method for OCR. If you want or need to use NNs for the OCR piece, you will almost certainly need to use a sliding window. For inspiration on how you might do that, you could look to Recurrent Neural Networks that process images in sequences

Once you have the text there are again much simpler methods to deal with text than NNs, but they can achieve great results. To deal with different document lengths, you can use an RNN architecture like LSTM. Or you can use paragraph vectors, which do most of the heavy lifting for you and give you an N-dimensional representation of your text.


Gaussian Mixture Models as a classifier? I'm learning the GMM clustering algorithm. I don't understand how it can used as a classifier. Here are my thought:

1) GMM is an unsupervised ML algorithm. At least that's how sklearn categorizes it.

2) Unsupervised methods can cluster data, but can't make predictions.

However, sklearn's user guide clearly applid GMM as a classifier to the iris dataset.

If I have to guess, maybe after clustering, each cluster is assigned to a class label based on some kind of majority voting. However, I can't find any documentation. Could someone shed more light on this process from unsupervised to supervised learning?

A related question: when using GMM as a classifier, is it common practice to simply make n_components=n_classes, instead of checking AIC, BIC, etc.?

Some unsupervised models can make predictions, but not ones that necessarily match the original class labels. Once a GaussianMixture model has been fitted, it can predict which of the clusters a new example belongs to. This is exactly what the predict and predict_proba functions do in this case, and given that the number of clusters is set to 3, the number of classes, the predict function will predict a label from $\{0, 1, 2\}$.

However, this still raises the question of how does the GaussianMixture assign particular labels to the clusters? In general, it is arbitrarily chosen, but in the example from sklearn that you linked, they cheat when initialising the cluster centers:

# Since we have class labels for the training data, we can
# initialize the GMM parameters in a supervised manner.
estimator.means_init = np.array([X_train[y_train == i].mean(axis=0)
                                for i in range(n_classes)])

The initial position for each cluster center is at the center of each class, which also has the additional consequence of correctly ordering the cluster labelling to match the original class labels. This means that the GMM predicting which cluster a new instance belongs to is equivalent to predicting which class it might belong to in this case. I believe this has been done for easy visualisation of the different covariance matrix options.


How to test trained PCA used for compression? I am working on an exercise for using PCA for compression of images and I don't quite understand how to use it on the test data:

I have 300 images of hand drawn sixes, represented by 28x28 matrices, in the train data, and I have used PCA to find an appropriate low dimensional representation of these images (26 dimensions yields me the sought after 90% threshold), giving me a 300x26 Matrix, that I can use to project my images into that space

Now I have to test this with my test data of 10 similar images - so I have to project them into the same space.

Because I can't just use the train space (dimensions don't agree), if I understood correctly, I should run another PCA to find the Principle Components for these new images, but project them into the 26 dimensions as identified by my train PCA (PCA on just the test suggests that 6 dimensions would suffice, but I want the more accurate 26 from my wide range of training data)

But here is where I'm struggling: how do I centre the test data? To centralise the data before using PCA or the dual PCA, I deduct the mean from the data. When building my test PCA should I deduct the mean of the train data, or the mean of the test data?

Found the answer in a different stackexchange question here: https://stats.stackexchange.com/questions/142216/zero-centering-the-testing-set-after-pca-on-the-training-set

Answer: Yes, zero center the test data with the mean from the train data


What is the immediate reward in value iteration? Suppose you're given an MDP where rewards are attributed for reaching a state, independently of the action. Then when doing value iteration:

$$ V_{i+1} = \max_a \sum_{s'} P_a(s,s') (R_a(s,s') + \gamma V_i(s'))$$

what is $R_a(s,s')$ ?

The problem I'm having is that terminal states have, by default, $V(s_T) = R(s_T)$ (some terminal reward). Then when I'm trying to implement value iteration, if I set $R_a(s,s')$ to be $R(s')$ (which is wha I thought), I get that states neighboring a terminal state have a higher value than the terminal state itself, since

$$ P_a(s,s_T) ( R_a(s,s_T) + \gamma V_i(s_T) ) $$

can easily be greater than $V_i(s_T)$, which in practice makes no sense. So the only conclusion I seem to be able to get is that in my case, $R_a(s,s') = R(s)$.. is this correct?

what is $R_a(s,s')$ ?

In this case, it appears to represent the expected immediate reward received when taking action $a$ and transitioning from state $s$ to state $s'$. It is written this way so it could be implemented as a series of square matrices, one for each action. Those matrices might of course be very sparse if only a few transitions are possible, but it's a nice generic form for describing MDPs.

Notation varies between different RL tutorials, so do take care when looking at other sources.

  The problem I'm having is that terminal states have, by default, $V(s_T) = R(s_T)$ (some terminal reward).

No, $V(s_T) = 0$ by definition. The value of a state, is the expected discounted sum of future rewards. A terminal state has no future rewards, thus its value is always $0$.

The "terminal reward" in your system occurs on the transition $s \rightarrow s_T$, so its expected value should be represented by $R_a(s,s_T)$

If the terminal state is some goal state, or a bad exit from an episode, and it doesn't matter how you arrive there, then in the formulation you have given, you still represent it as $R_a(s,s_T)$, just that the values of $a$ and $s$ don't matter for your case. In general they might matter, for other MDPs.

  So the only conclusion I seem to be able to get is that in my case, $R_a(s,s') = R(s)$.. is this correct?

Not in general. However, in some problems this might be a reasonable simplification. If you make that simplification, then you should take care when having a value associated with a single state to be consistent about whether the reward is for entering a particular state, or exiting it. There cannot be a reward for "being in a state", the closest you can get to that is granting a consistent reward when entering a state (ignoring the previous state and action that caused the transition).

It appears here that you have suggested here that you want a reward for exiting a particular state - ignoring which action is taken in it, or what the next state is. I don't know the details of your MDP, so cannot say whether this would work for you. Maybe not, given the rest of your question.

If you are asking in general, then you have the answer $R_a(s,s') \ne R(s)$

If you are asking in order to work on a specific MDP, I suggest take another look at your original problem, with the extra information in this answer.


Cloudera QuickStart VM Error I have installed cloudera CDH5 Quick start VM on VM player. When I login through HUE in the first page I am the following error

“Potential misconfiguration detected. Fix and restart Hue.”

How to solve this issue.

Go into the other link from home - to the cloudera manager.

From there, you'll see Hue can be restarted, but there is probably an alert that needs to be resolved in there first.

If I remember right there's some initial configuration that needs to be done on the quickstart VM that's spelled out as soon as you log into the manager application.


Why does BERT embedding increase the number of tokens? I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here.
I am trying to embed the following sentence with BERT - "Twinkle twinkle little star".
BERT tokenizer generates the following tokens - ['twin', '##kle', 'twin', '##kle', 'little', 'star']
But the final embedded tensor is having a dimension of [1,8,1024]
Why is the number of tokens 8 instead of 6? For any text, I am observing that number of tokens in the final embedding is getting increased by 2. Can anyone please help me to understand this?
I am giving the code snippet here -
from transformers import BertTokenizer, BertForSequenceClassification, BertModel

PRE_TRAINED_MODEL_PATH = 'BERT\wwm_cased_L-24_H-1024_A-16'
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_PATH)
model = BertModel.from_pretrained(PRE_TRAINED_MODEL_PATH)

encoded_input = tokenizer(texts, return_tensors='pt', padding=True)
emb = model(**encoded_input)

A [CLS] token is added to the beginning of the sentence, and a [SEP] token is added to the end.
These two special tokens have specific functions:

[CLS] is needed because it was used in the training loss of BERT to keep the first output position for a different purpose than the rest: it was used for the "next sentence prediction" loss, which trained the model to tell if the 2 text segments passed as input were consecutive in the original text or not. When finetuning BERT, the output of this position is normally used for sentence classification tasks. When BERT is used for feature extraction, the output vector at that position is used as sentence embedding.

[SEP] is used to mark the ending of the sentence.


Convert a pandas column of int to timestamp datatype I have a dataframe that among other things, contains a column of the number of milliseconds passed since 1970-1-1. I need to convert this column of ints to timestamp data, so I can then ultimately convert it to a column of datetime data by adding the timestamp column series to a series that consists entirely of datetime values for 1970-1-1.

I know how to convert a series of strings to datetime data (pandas.to_datetime), but I can't find or come up with any solution to convert the entire column of ints to datetime data OR to timestamp data.

You can specify the unit of a pandas to_datetime call.

Stolen from here:

# assuming `df` is your data frame and `date` is your column of timestamps

df['date'] = pandas.to_datetime(df['date'], unit='s')

Should work with integer datatypes, which makes sense if the unit is seconds since the epoch.


Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to get an encoded output and then we also calculate the average of word vectors for the same sentence separately, will the two new vectors (encoded and average) be comparable? Will their euclidean distance be relatively small?

No, assuming your input vectors are one-hot encodings. These input one-hot encodings are in an $n$-dimensional Euclidean vector space. The last hidden layer of an LSTM is not due to the non-linear activation functions across the encoder. Therefore, an average of the inputs will not necessarily align well in a vector space with the model output, nor are you guaranteed any similarity in cosine/Euclidean distance.


how to use word embedding to do document classification etc? I just start learning NLP technology, such as GPT, Bert, XLnet, word2vec, Glove etc. I try my best to read papers and check source code. But I still cannot understand very well.

When we use word2vec or Glove to transfer a word into a vector, it is like:

[0.1,0.1,0.2...]

So, one document should be like:

[0.1,0.1,0.2...]
[0.1,0.05,0.1...]
[0.1,0.1,0.3...]
[0.1,0.15,0.1...]
.......

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data? I was told that Bert or other NLP models can do this. But I am really curious about how the word embedding are applied in the traditional methods?

So, one document is a matrix. If I want to use some traditional method like random forest to classify documents, how to use such data?

You can't, at least not directly because traditional methods require a fixed number of features for every instance. In the case of document classification the instance must represent the document, so unless all the documents have exactly the same length (unrealistic) it's impossible to use a set of vectors as features.

The traditional approach would consist in representing a document with a vector where each cell represents a word in the vocabulary, and the value is for instance the TFIDF weight of the word in the document.


Bayesian combination of multi-dimensional experts? I have what seems to me to be a slightly complex version of a decision tree problem, that can't figure out how to model, and I'm trying to avoid the "just dump it into an NN" solution.

I have a bunch of expert opinions (from either people or algorithms) where the features dimensions overlap. Both features (inputs) and outputs (decision) are categorical, but I also have a certainty in the decision (let's say 0-1).

For example, let's say that the input features are a,b,c,d,e and there is one output (decisions) from among v,w,x,y,z:

Features   Decision  certainty
 a,b,c       w          0.5
 b,c,d,e     z          0.4
 a,c,e,f     v          0.75
 ...

Without the certainty, this would obviously just be a trivial categorical decision tree problem. However, the certainty both simplifies and complexifies the problem. It simplifies it bcs if the features->decision data is inconsistent (which it is!), you'd have no output at all, so the certainty saves me from that failure mode, but on the otherhand, I'm not sure what to correctly do with the certainty.

The certainties suggest Bayesian combination, and if the data was unidimensional this would be trivial as well. So I'm kind of caught half way between a decision tree and a bayesian model. An obvious cop out is just to dump it into an NN (or even just an NB or regression), using the certainties either as outputs on the categories, or doing something dumb like replicating the I/O pairings per the (un)certainty.

Thanks in advance for any suggestions.

Just so you know, in this case, even the neural network can't help you. For NNs to work, you need to have lots of learnable patterns in the data (meaning data without uncertainity/ irreducible error). If the data itself has a lot of uncertainty in the outcome( i.e. probabilistic outcomes) itself, its very hard of NNs to optimize that. (For more on this, consider modeling following problems with NNs: XOR(with probability =1) vs XOR(with probabilities <1)).

In this case, your best bet might be to use regression. Let us know how it folds out, thanks.


how to draw a negative exponential plot of this situation with python I have a dataset with binary output ($Y$) and I have a column (Duration) contains the duration of each task that is stored by "days" and varied from 1day to 350 days.

when I think logically in our situation, I can deduce that the probability of getting a positive output value ($Y = 1$) require to have small duration task.

But I need to justify my opinion with some plots

I have tried the following source code but It doesn't represent correctly my assumption.

#LoadData

min_duration = plot_data['Duration'].min()
max_duration = plot_data['Duration'].max()
xr_ = list(range(min_duration,  max_duration,  5))
y_ = []

for i in range(0,(len(xr_)-1)):
    a_ = np.logical_and(plot_data['Duration'].values >= xr_[i], plot_data['Duration'].values < xr_[i+1])
    b_ = np.logical_and(np.logical_and(plot_data['Duration'].values >= xr_[i], plot_data['Duration'].values < xr_[i+1]), plot_data['output'].values==1)
    y_.append(sum(b_)/sum(a_))

import matplotlib
matplotlib.pyplot.plot(xr_[1:len(xr_)], y_, 'o')

Based on my previous assumption I must get a plot which contains an exponential form like :

But I have got contrary the following plot:

I want to know where I have a mistake and If there is any other method to justify my assumption

The model you are looking for is this:

$Y=A e^{-Bx} + c$, I know the implementation in R, because I don't know of a nonlinear estimation in Python

This code in R might work:

R=data.frame(X=c(1,2,3,4,5,6,7,8,9),Y=c(1,2,3,3,3,3,3,3,3)) # Data in which X, Y are your data
model=nls(formula = Y~A*exp(-B*X)+C,data=R)
summary(model)

There is a limitation to take into account, is explained in this link, is summarized in the impossibility for all possible models to exist, the "most inside" model should be linear.

First steps with Non-Linear Regression in R

Singular Gradient Error in nls with correct starting values


Are dimensionality reduction techniques useful in deep learning I have been working on machine learning and noticed that most of the time, dimensionality reduction techniques like PCA and t-SNE are used in machine learning, but I rarely noticed anyone doing it for deep learning projects. Is there a specific reason for not using dimensionality reduction techniques in deep learning?

Deep learning does not use dimensionality reduction because deep learning itself is a useful dimensionality reduction technique. Deep learning learns a compressed, nonlinear representation of the data through the hidden layers. Since Deep Learning can learn nonlinear mappings, it is a more flexible dimensionality reduction technique than PCA which restricted to  linear mappings.


Finding lookalike customers for Digital Media targeting I want to develop a model for digital marketing team. We have some set of ideal customers to whom we to send our promotions and have been converted.
Now we want to pick past such cases and want to send promotions to the similar (lookalike) customers for digital marketing.
I know its a broad question but any approach, techniques and suggestions would be appreciated.

This seems to be a pretty common scenario in digital marketing, and a few companies have published their approach to lookalike modeling.

Here are a few links:

Lookalike at LinkedIn
Lookalike at Pinterest
Lookalike at Yahoo
Another lookalike from Yahoo
Academic paper on lookalike (not sure where the authors work)
Google's lookalike patent (this one is a lot harder to read)

Maybe you can draw inspiration from what these companies have done.  Most of the lookalike systems that I've read about fall into one of the following categories:
segment-approximation models, regression models, and similarity-based models.

Segment-approximation is a similar idea to collaborative filtering.  The idea is to find which interests are shared among your seed audience (that is, your past converters).  Then you expand the audience by finding new individuals who have a similar interest profile.  Obviously this can only be done if you have some information about your potential customers' interests.

Regression models try to compute a membership function which ranges from 0 (not similar to the seed audience at all) to 1 (a member of the seed audience).  You use this model to score each potential customer, and those which receive a sufficiently high score are included in the lookalike audience.  One challenging aspect of this approach is that you have to use semi-supervised training.  You have a set of positive examples (past converters), and you have a much larger pool of mixed-label examples (people who haven't converted).

Similarity-based models generally attempt to learn a useful representation for customers in an unsupervised way.  You can then run clustering in the representation space, and your lookalike audience will be those clusters which have high overlap with your seed audience.
Alternatively, you can do similarity comparisons in the representation space, and construct a lookalike audience by thresholding the similarity you find between converters and other potential customers.

The specific learning algorithm that you use will probably vary based on the data you have available.  The links above may spark some ideas.

Hope that helps!


visualize a horizontal box plot in R I have a dataset like this. The data has been collected through a questionnaire and I am going to do some exploratory data analysis.

windows <- c("yes", "no","yes","yes","no")
sql     <- c("no","yes","no","no","no")
excel  <- c("yes","yes","yes","no","yes")
salary <- c(100,200,300,400,500 )

test<- as.data.frame (cbind(windows,sql,excel,salary),stringsAsFactors=TRUE)
test[,"salary"] <- as.numeric(as.character(test[,"salary"] ))

I have an outcome variable (salary) in my dataset and a couple of input variables (tools). How can I visualize a horizontal box plot like this:

Let's start by creating some fake dataset.

software = sample(c("Windows","Linux","Mac"), n=100, replace=T)
salary = runif(n=100,min=1,max=100)
test = data.frame(software, salary)

This should create a dataframe test that will look like somewhat like:

    software    salary
1    Windows 96.697217
2      Linux 29.770905
3    Windows 94.249612
4        Mac 71.188701
5      Linux 94.028326
6      Linux  7.482632
7        Mac 98.841689
8        Mac 81.152623
9    Windows 54.073761
10   Windows  1.707829

EDIT based on comment Note that if the data does not already exist in the above format, it can be changed to this format. Let's take a data frame provided in the original question and lets assume the dataframe is called raw_test.

    windows sql excel salary
1     yes  no   yes    100
2      no  yes  yes    200
3     yes  no   yes    300
4     yes  no    no    400
5      no  no   yes    500

Now, using the melt function/ method from the reshape package in R, first create the dataframe test (that will be used for final plotting) as follows:

# use melt to convert from wide to long format
test = melt(raw_test,id.vars=c("salary"))
# subset to only select where value is "yes"
test = subset(test, value == 'yes')
# replace column name from "variable" to "software"
names(test)[2] = "software"

Now, you will get a datframe test that looks like:

  salary software value
1     100  windows   yes
3     300  windows   yes
4     400  windows   yes
7     200      sql   yes
11    100    excel   yes
12    200    excel   yes
13    300    excel   yes
15    500    excel   yes

Having created the dataset. We will now generate the plot.

First, create the bar plot on the left based on the counts of software that represents usage rate.

p1 <- ggplot(test, aes(factor(software))) + geom_bar() + coord_flip()

Next, create the boxplot on the right.

p2 <- ggplot(test, aes(factor(software), salary)) + geom_boxplot() + coord_flip()

Finally, place both these plots next to each other.

require('gridExtra')
grid.arrange(p1,p2,nrow=1)

This should create a plot like:


How to Bootstrap dataset for 10000 AUC scores? I am new to ML and trying to learn the nuances. I work on a binary classification problem with 5K records. Label 1 is 1554 and Label 0 is 3554.

What I currently do is

1) split the data into train(70%) and test(30%)

2) initiate a model -->  logreg=LogisticRegression(random_state=41)

3) run 10 fold cv -->   logreg_cv=GridSearchCV(logreg,op_param_grid,cv=10,scoring='f1')

4) fit the model -->  logreg_cv.fit(X_train_std,y_train)

5) Do prediction  -->  y_pred = logreg_cv.predict(X_test_std)

Now my question is, how to generate 10000 AUC scores.

I read that people usually do this get a confidence interval of their train and test performance AUC scores.

So, I would like to know how to do this?

I know that bootstrap means generate random samples with replacement from same dataset. But do we still have to split them into train and test? But this looks no different than CV. How do we generate 10000 AUC's and get a confidence interval?

Can you help?

In this question of stats exchange you can see an answer to your question of when to use bootstrap over CV.

You can see a simple tutorial of how to do Bootstrap in Python in this link

How to generate 10k AUC Scores?
AUC is a performance metric and what you are going to measure is the performance of your model 10k times. For that, you have to select 10k times the number of samples that you consider and measure AUC

for i in range(0,10_000):
    sample = df.sample(df.shape[0]/10,random_state=i)
    X = df.drop(columns='target')
    y = df.target
    preds = logreg.predict(X)
    print(roc_auc_score(preds,y))


Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation I would like to use a sklearn pipeline doing this :

( - ) scale the data ( StandardScaler )

( - ) reduce dimensionality ( PCA )

( - ) make a prediction with GradientBoostingRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) make a prediction with RandomForestRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) take the mean of both predictions

but I cannot figure out how to proceed.

Do I have to ( scale & predict ) 2 times or can I scale and then predict 2 times ?

Thks.

You might looking for sklearn.ensemble.VotingRegressor which takes the mean of two regression models.
Here is an example to get you started:
from sklearn.datasets        import make_regression
from sklearn.decomposition   import PCA
from sklearn.ensemble        import GradientBoostingRegressor, RandomForestRegressor, VotingRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline        import Pipeline
from sklearn.preprocessing   import StandardScaler

# Make fake data
X, y = make_regression(n_samples=1_000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=42)

pipe = Pipeline([('scl', StandardScaler()),
                 ('pca', PCA()),
                 ('vr', VotingRegressor([('gbr', GradientBoostingRegressor()), ('rfr', RandomForestRegressor())]))
                ])

search_space = [{'vr__gbr__learning_rate':    [.07, .1, .15]}]

gs_cv = GridSearchCV(estimator=pipe,
                     param_grid=search_space,
                     n_jobs=-1)

gs_cv.fit(X_train, y_train)
gs_cv.predict(X_test)


Distribution of training dataset in binary classification problem Is it important to have the same amount of positives and negatives in our training dataset for a binary classification problem? Or for example, it doesn't matter if we have 70% positives and 30% negatives.

In general it is better to have a good mix of positive and negative labels for a model to learn from. That said, many classification problems suffer from significant label imbalance (e.g. fraud detection).
I don't see an obvious problem with a 70/30 mix of labels, but as you start modeling be sure to check the confusion matrix and look for where you model isn't performing well. If it appears that the imbalance is causing issues, you could try techniques like under-sampling or SMOTE to create a better balance (artificially). The linked article below has more details if you are interested:
https://www.jeremyjordan.me/imbalanced-data/


Clustering big data by reducing data accuracy? I have 1 million rows with 20 attributes to do hierarchical clustering. When I want to build a distance matrix on this data by dist() in R, it says that it needs 5 TB memory. I have these approaches:

Reduce the number of rows by sampling
Change the method of clustering
?

Now, do you suggest another approach?
And I have an idea, I think if I reduce the accuracy of the values and then doing "group by", then I can remove duplicated rows and have a new column with the count of duplicates for each row.
Is there any R package that can do hierarchical clustering with these data?

"group by": count number of duplicated rows and add a column that say how many times this row was duplicated in source.

You have n=20000000 data points and d=20 attributes.

Approach 1

The hierarchical agglomerative clustering has a time complexity of O(n^3)
and requires O(n^2) memory. Therefore it is infeasible for large data sets (not to mention big data)

What you can do is first use some nearly-linear-time clustering algorithm to cluster your data to, let as say, 2000 clusters. This algorithm can be, for example, K-Means. See this stackoverflow answer for its time complexity. This is actually an alternative to your discussion of 'reduce accuracy and group'. Then you apply hierarchical clustering to the found 2000 clusters.

Approach 2
Use the algorithm, which is specially designed for large data.
For example, Birch algorithm


How to derive association from a regression model? I understand how to make predictions with a trained neural network model that uses loss=binary_crossentropy and a 1-node activation=sigmoid output layer to make binary classifications.

But how can I determine the strength of association between a feature and a label? I'm trying to make a neural network that competes with generalized linear models that show p values for each feature.

The predictions are not the important part. I need to provide insight about the features so that we can learn about the biology of a disease (DNA variant is feature and disease is label). I know that there is such a thing as feature importance, but isn't that just a rank ordered list of features?

def create_baseline():

    model = Sequential()
    model.add(Dense(60, input_dim=60, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='sigmoid'))

    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

I think NN is a little problematic here, because you run into the bias-variance-tradeoff. NN are really good in producing estimates with low variance. However, if you look for a causal influence of $x$ on some $y$, you would choose an unbiased estimator (with higher variance). Options are OLS or Lasso/Ridge estimators. Especially with DNA information, Lasso is a good choice since you may face a problem with (relatively) high dimensionality.

If you really want to explore feature importance in a NN setting, I guess you need to resort to stepwise forward/backward techniques similar to the method as used for feature selection. See here and here for some background. To my best knowledge Keras does not come with a predefined function for the problem.


Can I add features that are parts of another feature? I am building a model (implementing both logistic regression and Xgboost) to understand the importance/significance of each feature in whether a customer is going to repurchase to understand what matters for customers to repurchase (More interested in inferences than the predictions)

My feature set looks like this: [income, age, price_product, discount, product_category, delivery_charge, gender, lifestyle, delivery_time_main (P_main), P1, P2, P3, P4 and so on]

P_main is something that the customer sees and may influence their decision to repurchase or not. P1 + P2 + P3 + P4 = P_main. We want to understand how many important are P1, P2, P3, P4 as well as they are stages in the process so we can make inferences about which stage is to be focused most on if we were to improve repurchases.

Can the feature set contain the parts (P1, P2, P3, P4) as well as the sum (P_main) as inputs to the model? Or does it introduce multicollinearity problems? I am removing features that have multicollinearity using the Variance Inflation Factor and then applying Lasso regression.

First, I recommend NOT binning continous variables in any way. The amount of information lost and numerical problems makes it rarely a good decision. This is especially true in your case where you will be sacking a lot of degrees of freedom in your model by binning (which are necessary if you want statistical power, i.e. rejecting the null when you should). If you want to account for non linearities, either use splines or better yet, use an algorithm that doesn't assume a linear relationship between (some function of) y and your predictors.

Second, I would be extremely cautious with doing any sort of statistical inference (determining importance/significance of predictors) with machine learning algorithms that primarily (if not fully) were created for predictive, rather than explanatory reasons. In other words, I would stick with logistic regression in this case, since most ML algorithms don't allow for much, if any, proper statistical inference.

You are right to consider multicollinearity in this case. Severe collinearity will adversely affect p values and confidence intervals, and make inference hard, if not impossible. But this is why you need to have a hypothesis first as to what could possibly influence your target, to carefully specify your model, and to check the assumptions related to your chosen model (and fix or note any violations). Simply throwing a bunch of predictors into a model won't work here since you actually care about interpreting the model and its statistical output.

To be honest, I recommend going to the cross validated SE for more information rather than here, which primarily focuses on predictions, not inference.


Deep learning basics I am looking for a paper detailing the very basics of deep learning. Ideally like the Andrew Ng course for deep learning. Do you know where I can find this ?

This link contains an amazing amount of deep learning literature.
Summarizing it here(going in the order a beginner ideally should)-
NOTE: All these resources mainly use python.

1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net.

Andrew Ng's Coursera course is pretty good too.

2) For Neural networks, nobody explains it better than Dr.Patrick Winston.
The assignments should be tried out for better understanding. They are in python.

3) For a better understanding of Neural Networks, Michael Nielsen's course should be done(as suggested by Alexey). It is pretty basic but it works.

4) For deep neural networks, and implementing them faster on GPUs, there are multiple frameworks available, such as Theano, Caffe, Pybrain, Torch,etc.
Out of these Theano provides a better low level functionality that allows its user to create custom NNs. It is a python library, so being able to use numpy,scikit-learn, matplotlib, scipy along with it is a big plus.
The deep learning tutorial written by Lisa Lab should be tried out for a better understanding of theano.

5) For Convolutional Neural Networks, follow andrej karpathy's tutorial.

6) For unsupervised learning, follow here and here.

7) For an intersection of deep learning and NLP, follow Richard Socher's class.

8) For LSTMs, read Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780 and Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012.

Here is LSTM's Theano code.


Explaining GRU equations from two sources I really confused about how GRU computation really works. I am not really good at math btw.

I read the original source of GRU equations from Cho, et al. (2014), and also a Colah's blog post on the same topic.

And somehow, GRU equations at Colah's blog post Understanding LSTM Networks are different than equations in the Cho's original paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.

These are the equations from Colah's blog:

$$z_t = \sigma(W_z \cdot [h_{t-1},x_t])$$

$$r_t = \sigma(W_r \cdot [h_{t-1},x_t])$$

$$h_t = (1-z_t) \ast h_{t-1} + z_t \ast \tilde{h}_t$$

$$\tilde h_t = tanh(W \cdot [r_t \ast h_{t-1}, x_t]_j)$$

These are the equations from the original paper:

$$z_j = \sigma([\mathbf W_z \mathbf x]_j + [\mathbf U_z \mathbf h_{(t-1)}]_j)$$

$$r_j = \sigma([\mathbf W_r \mathbf x]_j + [\mathbf U_r \mathbf h_{(t-1)}]_j)$$

$$h_j^{(t)} = z_jh_j^{(t-1)} + (1-z_j) \tilde h_j^{(t)}$$

$$\tilde h_j^{(t)} = \phi([\mathbf W \mathbf x]_j + [\mathbf U(\mathbf r \odot h_{(t-1)})]_j)$$

Can someone explain if Colah's equations are the same as the equations in the paper version?

The equations are almost the same. First of all, they are written in a slightly different form. In Cho, et al. (2014), it is written that $W_r$ and $U_r$ are weight matrices which are learned. While in Colah's blog, both matrices are referred to as $W$.

For example you could write these two equivalent forms:

$z_j = \sigma([W_z x]_j + [W_z h_{t-1}]_j)$

$z_j = \sigma(W_z[x,h_{t-1}]_j)$

Secondly, there is a small mistake in Colah's for $h_t$. Let's ignore the different notation $h_j^{(t)}$ used in the paper and use Colah's notation $h_t$ instead. Then the equation for $h$ in Cho, et al. (2014) can be written as:

$h_t = z_t \ast h_{t - 1} + (1 - z_t) \ast \tilde{h_t}$

and if we use the same order as Colah's equation:

Paper's equation: $h_t = (1 - z_t) \ast \tilde{h_t} + z_t \ast h_{t - 1}$

Colah's equation: $h_t = (1 - z_t) \ast h_{t - 1} + z_t \ast \tilde{h_t}$

So we can actually see that for Colah's equation to match the one in the paper, we have to swap $\tilde{h_t}$ and $h_{t - 1}$ in this equation.
If you check the comments next to the GRU example on Colah's blog, you can actually see that some other people found the same mistake. Well spotted, I didn't saw it at first! :)


Perceptron - Convergence proof I studied the perceptron algorithm and I'm trying to prove the convergence by myself. However, I'm wrong somewhere and I am not able to find the error.

Assumption:

We assume that there is some $\gamma > 0$ such
that $$y_{t}(\theta ^{*})^{T}x_{t} \geq \gamma $$  for all $t = 1, \ldots , n$. The additional number $\gamma > 0$ is used to ensure that each example is classified correctly with a finite margin.
We will assume that all the (training) images have bounded
Euclidean norms, i.e., $$\left \| \bar{x_{t}} \right \|\leq R$$ for all $t$ and some finite $R$

By hypothesis the learning rule is:

$$\theta ^{(k)}= \theta ^{(k-1)} + \mu y_{t}\bar{x_{t}}$$

Now, $$(\theta ^{*})^{T}\theta ^{(k)}=(\theta ^{*})^{T}\theta ^{(k-1)} + \mu y_{t}\bar{x_{t}} \geq  (\theta ^{*})^{T}\theta ^{(k-1)} + \mu \gamma $$
so , by induction
$$(\theta ^{*})^{T}\theta ^{(k)}\geq k\mu \gamma $$

At the same time,
$$\left \| \theta ^{(k)} \right \|^{2} = \left \| \theta ^{(k-1)}+\mu y_{t}\bar{x_{t}}  \right \|^{2} = \left \| \theta ^{(k-1)} \right \|^{2}+2\mu y_{t}(\theta ^{(k-1)^{^{T}}})\bar{x_{t}}+\left \| \mu \bar{x_{t}} \right \|^{2} \leq \left \| \theta ^{(k-1)} \right \|^{2}+\left \| \mu\bar{x_{t}} \right \|^{2}\leq \left \| \theta ^{(k-1)} \right \|^{2}+\mu ^{2}R^{2}$$

So, by induction

$$\left \| \theta ^{(k)} \right \|^{2} \leq k\mu ^{2}R^{2}$$

We can now combine parts 1) and 2) to bound the cosine of the angle between $\theta^∗$ and $\theta(k)$:

$$\cos(\theta ^{*},\theta ^{(k)}) =\frac{\theta ^{*}\theta ^{(k)}}{\left \| \theta ^{*} \right \|\left \|\theta ^{(k)}  \right \|} \geq \frac{k\mu \gamma }{\sqrt{k\mu ^{2}R^{2}}\left \|\theta  ^{2} \right \|}$$

Since cosine is bounded by one, we get:

$$k \leq \frac{R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$

The problem is that the correct result should be:

$$k \leq \frac{\mu ^{2}R^{2}\left \|\theta ^{*}  \right \|^{2}}{\gamma ^{2}}$$

What you presented is the  typical proof of convergence of perceptron proof indeed is independent of $\mu$. Hence the conclusion is right.

Typically $\theta^*x$ represents a hyperplane that perfectly separate the two classes.

The formula $k \le \frac{\mu^2 R^2 \|\theta^*\|^2}{\gamma^2}$ doesn't make sense as it implies that if you set $\mu$ to be small, then $k$ is arbitarily close to $0$. It is saying that with small learning rate, it converges immediately. You might want to look at the termination condition for your perceptron algorithm carefully.


Genetic neural network to satisfy variable number of inputs and outputs I have what I propose as a solution to my problem, however I haven't ever seen it mentioned in this way, so I worry that there is a valid reason not to do things this way.

I have a dataset of > 100,000 events, where each event has a winner.
I have plenty of data points, some data on the event itself, and some data on each entrant.

The number of entrants in each event is variable, and I want to build a neural network around picking a likely winner of the events.

As the number of entrants is variable, what appears to be common advice is to have enough inputs for the maximum case scenario, and 0 them out for events where there are empty slots.

This feels somewhat inelegant, and I had a slightly different idea.

I was going to have a NN where the inputs are information about the event, and information about 1 entrant. I would then have a single output (a float between 0 and 1). I would run this through, getting 1 output for each entrant in an event, then I would be left with a number of floats, equal to the number of entrants in the event. I would then select the highest value, and use the entrant that refers to as the choice for the winner.

Is there a reason I shouldn't be doing it this way? Is there a better solution I haven't yet come across?

Is there a reason I shouldn't be doing it this way?

Depends on the nature of the data. There might be an element of "Scissor/Paper/Stone" in the competition you are scoring, where different strengths and weaknesses of competitors can combine such that Player A beats Player B, Player B beats Player C, but Player C beats Player A. In that case, you cannot produce reliable ranking between players by considering each entrant separately, and a network that rates each player individually will perform less well than one that can compare players.

If players are in more of a race-to-finish or score max points separately in a competition, then separately rating each player in each competition should be more reliable. And it is definitely easier to build and train a neural network to predict that.

An alternative, if your events are more like tournaments where entrants oppose each other (even if within some larger free-for-all), is to predict relative rank between pairs of players. This may not be consistent, so you will need to use a pairwise ranking method to resolve that for the final winner. If it really is a knockout tournament, and you know how the initial draw and team combinations will work, then you could maybe make a prediction by simulating the possible games.

There is nothing preventing you from combining these approaches in some way either.

Whichever method you use, you will want to think a little about what your metric is going to be to select the best approach. If you only care about predicting the winner, then accuracy of that prediction might be enough. If you care about where the eventual winner is placed, perhaps mean reciprocal rank would be better (score 1 for correct prediction, 1/2 for predicting winner as ranked second, 1/3 if third etc).


High-dimensional data: What are useful techniques to know? Due to various curses of dimensionality, the accuracy and speed of many of the common predictive techniques degrade on high dimensional data. What are some of the most useful techniques/tricks/heuristics that help deal with high-dimensional data effectively? For example,

Do certain statistical/modeling methods perform well on high-dimensional datasets?
Can we improve the performance of our predictive models on high-dimensional data by using certain (that define alternative notions of distance) or kernels (that define alternative notions of dot product)?
What are the most useful techniques of dimensionality reduction for high-dimensional data?

This is very broad question, which I think it's impossible to cover comprehensively in a single answer. Therefore, I think that it would be more beneficial to provide some pointers to relevant answers and/or resources. This is exactly what I will do by providing the following information and thoughts of mine.

First of all, I should mention the excellent and comprehensive tutorial on dimensionality reduction by Burges (2009) from Microsoft Research. He touches on high-dimensional aspects of data frequently throughout the monograph. This work, referring to dimensionality reduction as dimension reduction, presents a theoretical introduction into the problem, suggests a taxonomy of dimensionality reduction methods, consisting of projective methods and manifold modeling methods, as well as provides an overview of multiple methods in each category.

The "projective pursuit" methods reviewed include independent component analysis (ICA), principal component analysis (PCA) and its variations, such as kernel PCA and probabilistic PCA, canonical correlation analysis (CCA) and its kernel CCA variation, linear discriminant analysis (LDA), kernel dimension reduction (KDR) and some others. The manifold methods reviewed include multidimensional scaling (MDS) and its landmark MDS variation, Isomap, Locally Linear Embedding and graphical methods, such as Laplacian eigenmaps and spectral clustering. I'm listing the most of the reviewed methods here in case, if the original publication is inaccessible for you, either online (link above), or offline (References).

There is a caveat for the term "comprehensive" that I've applied to the above-mentioned work. While it is indeed rather comprehensive, this is relative, as some of the approaches to dimensionality reduction are not discussed in the monograph, in particular, the ones, focused on unobservable (latent) variables. Some of them are mentioned, though, with references to another source - a book on dimensionality reduction.

Now, I will briefly cover several narrower aspects of the topic in question by referring to my relevant or related answers. In regard to nearest neighbors (NN)-type approaches to high-dimensional data, please see my answers here (I especially recommend to check the paper #4 in my list). One of the effects of the curse of dimensionality is that high-dimensional data is frequently sparse. Considering this fact, I believe that my relevant answers here and here on regression and PCA for sparse and high-dimensional data might be helpful.

References

Burges, C. J. C. (2010). Dimension reduction: A guided tour. Foundations and Trends® in Machine Learning, 2(4), 275-365. doi:10.1561/2200000002


What is the effect of oversampling on logistic regression? I am building a model to predict if a customer will use a coupon or not for a given campaign. I am using logistic regression for this model. I took 5 previous campaigns and generally for each campaign conversion rate is around 10%. Thus, to handle this imbalanced data set and to capture more info I took a stratified sample from this data(whole 5 campaigns) such that there are 50% of sample with negatives and 50% positives. Thus, I am oversampling my positives.
My doubt is if I use logistic regression where it estimates coefficients using maximum log likelihood. Will this oversampling will generate bias results?
Also, I think this oversampling won't create any problem with random forest?

The effect will be increasing the intercept. I don't recommend doing oversampling unless any other solution doesn't work. Besides, 10% is not such a big imbalance.
I've been in kaggle competitions with way more imbalance where no imbalance solutions were adopted, logistic regression and random forest work quite well without the need of these.

Edit
After @Ben Reininger input, here's a theoretical justification on how does the intercept change.
Also, a quick experiment showcasing that over-sampling doesn't help improve a metric like AUC, and that it indeed increases the intercept of the logistic regression model.


YOLO: What is better? multiple networks per label or multiple labels in single network? I would like to use a YOLOv3 network,load in the preexisting trained weights, then retrain the ending layer to recognize say, 20 labels.

Would adding labels to a single network in training reduce its accurary?

Would it be better to train multiple models on 5,10 and 5 labels respectively.

e.g.

Model A : recognizes 5 labels (ferrari,honda,ford,toyota,tesla)
Model B : regonzies 10 labels (store front / stopsign, speed sign, super charger station , etccc. )
Model C : recongizes 5 labels (pedestrian, dog, marathon runner, bicycler, scooter rider)

OR

Model D: Recognizes 20 labels (all labels above put together)

Would Model D be as accurate as Model A,B,C put together?

It depends specially on data size and model capacity.

Therefore, we cannot give a definite "yes" or "no" prior to any experiment. However, we can enumerate possible cases, causes, and ways to improve, regardless of any specific model or dataset.

If the number of data points per category is insufficient, which is frequently the case in reality, a single model for combined categories performs better, since the effect of data shortage will be attenuated by sharing a model among different but related tasks, i.e. shared knowledge. However, we should not forget about the capacity of model, if model capacity is not enough for the combined task (which depends on the architecture and size of the model), this sharing would backfire as the model spreads its limited capacity across multiple tasks and becomes "jack of all trades, and master of none". Therefore, this approach is favorable if model capacity permits, which is less of a concern for neural networks with flexible size constraints.
If the number of data points per category is sufficient, which is rarely the case, or model capacity is limited (for example, when model performance stops improving when trained on more than 10% of data), single model per category is better since no capacity will be wasted on different (although related) tasks. From a different point of view, although this choice might be less favorable in terms of model accuracy, but it allows parallel development on smaller tasks, i.e.  faster production, which could be a justified benefit for the cost we pay in terms of model accuracy.

Both approaches

We can also combine the two approaches. First we train a model on combined categories, second we feed a part (or parts) of the trained model (such as a layer, the final prediction, etc.) to a specific model that will be trained on one category. This is exactly what a pre-trained model does for us, it is trained on a more general task with more data (possibly even a different but related task), and then we use it for a specific task.

In your case, you can go for three levels:

The pre-trained YOLO which is already built,
Training MyYOLO on all categories (for example, by using one input from the last layer of YOLO, and one input directly from image), and
Training MyYOLO2 on specific categories (for example, by using one input from MyYOLO and one input directly from image). Noting that at each level, weights of previous model remain constant to serve a constant input.


Why there are specific number of layers with same depth in any CNN architecture? why each architecture have specific number of layers with same depth

like vgg had 2 with 64 depth and 2 with 128d and 4 layers

same with ResNet and 34 layer plain like 6 layers with depth of 64 and 8 layers with depth 128.

and why depth increasing by last depth *2 ?

is there is way or equation to decide number of layers with each depth ?

There is no equation or theory to decide the number of layers at each depth in a neural network. The most common technique is trail and error. There can be systematic search through cross validation.
Generally more layers are added until performance stops improving.


Resampling pandas Dataframe keeping other columns I'm facing a problem with a pandas dataframe. Actually my Dataframe contains 3 columns: DATE_TIME, SITE_NB, VALUE.
For some SITE_NB there are missing rows. For example:

DATE_TIME;SITE_NB; VALUE

2011-01-03 01:00; 1; 10.7

2011-01-03 04:00; 1; 3.2

2011-01-03 05:00; 1; -2.1

So here, rows for 2011-01-03 00:00, 2011-01-03 02:00 and 2011-01-03 03:00 are missing. What I want is add these rows with the same SITE_NB (=1) and with VALUE (=NaN)

I want to do the same for all different SITE_NB in my dataframe. So for each SITE_NB, add missing rows based on DATE_TIME with a frequency of 1 Hour, and putting NaN in VALUE for freshly added rows.

I tried resampling but did not get the right output...

Can somebody help me to solve this issue?

Thanks!

To add missing indices, use:

full_idx = pd.date_range(start='<start_date>', end='<end_date>', freq='H')

df.reindex(full_idx)

That will put NaN's in SITE_NB and VALUE columns.

If each dataframe has only one value of SITE_NB, you can use:

df['SITE_NB'].fillna(df['SITE_NB'].unique()[0], inplace=True)

which replaces all NaN's with the first non-null values in the column.

If you actually have one dataframe with multiple SITE_NB values, could you please show what that looks like? Are the time indices overlapping in some cases or not?

Also, this answer might work for you: https://stackoverflow.com/questions/32275540/pandas-reindex-dates-in-groupby


ML algorithms for defining NORMAL user behavior I am trying to create a model that would capture the usual behavior of a user: i.e., create a model for user profiling.

I do have 2 million rows of data indicating normal behavior of a user and want to define a model that can detect any variation from the normal behavior of the user.

I am contemplating of using one class svm / isolation forest trained on normal behavior. I would like to get an opinion whether that looks as a good direction.

Or should I venture into deep learning or unsupervised clustering based model and gather data for abnormal behavior of a user as well to possibly use supervised models.

I think it depends mostly on how difficult/expensive/time-consuming it is to gather data for abnormal behavior.It also depends on whether you'd expect this abnormal behavior to belong to one (or a fixed number of) classes or it could be anything that deviates from normal.

If you can afford to collect labelled data and expect the abnormal behavior to be limited to a fixed number of classes, then the supervised approach is very likely going to give you better results.

Instead, if any of the above hypothesis do not hold, then I think the semi-supervised approach (as you said, One Class SVM or similar techniques) is in my opinion a good way to go.


Is this a good approach to classify tickets which contains description and logs? I want to classify a dataset of support tickets which mostly contain text in the description field and sometimes server logs in a separate field.

The log field is not always there but when it's present, it's a good indicator of the target class of the ticket.

I have created a CNN based classifier which can classify the tickets based on the log field, and a SVM clf with TFIDF based features for the description field.

I am thinking of adding the output probabilities of the CNN classifier in TFIDF based  SVM classifier to combine the models as a feature column.

Is there a better way to combine these models?

Is there a better way to approach this problem, without having two separate models?

First, you should get clear about your non-functional requirements (speed, memory consumption). Make sure you have a clear idea who is going to maintain the software and what the effort will be if you make some changes. Higher accuracy might not be worth it.

The simplest way to combine n models is averaging their predictions. This technique is often used on Kaggle. See chapter 2.6 of https://arxiv.org/pdf/1707.09725

Did you try a simple MLP with tf-idf features? How well does that perform? Usually it executes way faster than SVMs, if you have many classes.


Difference between RMSProp with momentum and Adam Optimizers According to this scintillating blogpost Adam is very similar to RMSProp with momentum.

From tensorflow documentation we see that tf.train.RMSPropOptimizer has following parameters

__init__(
    learning_rate,
    decay=0.9,
    momentum=0.0,
    epsilon=1e-10,
    use_locking=False,
    centered=False,
    name='RMSProp'
)

while tf.train.AdamOptimizer:

__init__(
    learning_rate=0.001,
    beta1=0.9,
    beta2=0.999,
    epsilon=1e-08,
    use_locking=False,
    name='Adam'
)

What is the conceptual difference if we put beta1 = decay and beta2 = momentum?

(My answer is based mostly on Adam: A Method for Stochastic Optimization (the original Adam paper) and on the implementation of rmsprop with momentum in Tensorflow (which is operator() of struct ApplyRMSProp), as rmsprop is unpublished - it was described in a lecture by Geoffrey Hinton .)

Some Background

Adam and rmsprop with momentum are both methods (used by a gradient descent algorithm) to determine the step.

Let $\Delta x^{(t)}_j$ be the $j^{\text{th}}$ component of the $t^{\text{th}}$ step. Then:

In Adam: $$\Delta x_{j}^{(t)}=-\frac{\text{learning_rate}}{\sqrt{\text{BCMA}\left(g_{j}^{2}\right)}}\cdot\text{BCMA}\left(g_{j}\right)$$
while:

$\text{learning_rate}$ is a hyperparameter.
$\text{BCMA}$ is short for "bias-corrected (exponential) moving average" (I made up the acronym for brevity).

All of the moving averages I am going to talk about are exponential moving averages, so I would just refer to them as "moving averages".

$g_j$ is the $j^{\text{th}}$ component of the gradient, and so $\text{BCMA}\left(g_{j}\right)$ is a bias-corrected moving average of the $j^{\text{th}}$ components of the gradients that were calculated. Similarly, $\text{BCMA}\left(g_{j}^{2}\right)$ is a bias-corrected moving average of the squares of the $j^{\text{th}}$ components of the gradients that were calculated.
For each moving average, the decay factor (aka smoothing factor) is a hyperparameter.
Both the Adam paper and TensorFlow use the following notation:

$\beta_1$ is the decay factor for $\text{BCMA}\left(g_{j}\right)$
$\beta_2$ is the decay factor for $\text{BCMA}\left(g^2_{j}\right)$

The denominator is actually $\sqrt{\text{BCMA}\left(g_{j}^{2}\right)}+\epsilon$, while $\epsilon$ is a small hyperparameter, but I would ignore it for simplicity.

In rmsprop with momentum: $$\Delta x_{j}^{\left(t\right)}=\text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(t-1\right)}-\frac{\text{learning_rate}}{\sqrt{\text{MA}\left(g_{j}^{2}\right)}}\cdot g_{j}^{\left(t\right)}$$
while:

$\text{momentum_decay_factor}$ is a hyperparameter, and I would assume it is in $(0,1)$ (as it usually is).
In TensorFlow, this is the momentum argument of RMSPropOptimizer.
$g^{(t)}_j$ is the $j^{\text{th}}$ component of the gradient in the $t^{\text{th}}$ step.
$\text{MA}\left(g_{j}^{2}\right)$ is a moving average of the squares of the $j^{\text{th}}$ components of the gradients that were calculated.
The decay factor of this moving average is a hyperparameter, and in TensorFlow, this is the decay argument of RMSPropOptimizer.

High-Level Comparison

Now we are finally ready to talk about the differences between the two.

The denominator is quite similar (except for the bias-correction, which I explain about later).
However, the momentum-like behavior that both share (Adam due to $\text{BCMA}\left(g_{j}\right)$, and rmsprop with momentum due to explicitly taking a fraction of the previous step) is somewhat different.
E.g. this is how Sebastian Ruder describes this difference in his blog post An overview of gradient descent optimization algorithms:

  Whereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface [...]

This description is based on the paper GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, so check it out if you want to dive deeper.

Next, I would describe 2 simple scenarios to demonstrate the difference in the momentum-like behaviors of the methods.

Lastly, I would describe the difference with regard to bias-correction.

Accumulating Momentum

Consider the following scenario: The gradient was constant in every step in the recent past, and $\Delta x_{j}^{(t-1)}=0$. Also, to keep it simple, $g_{j}>0$.
I.e. we can imagine our algorithm as a stationary ball on a linear slope.
What would happen when we use each of the methods?

In Adam

The gradient was constant in the recent past, so $\text{BCMA}\left(g_{j}^{2}\right)\approx g_{j}^{2}$ and $\text{BCMA}\left(g_{j}\right)\approx g_j$.
Thus we get:
$$\begin{gathered}\\
\Delta x_{j}^{(t)}=-\frac{\text{learning_rate}}{\sqrt{g_{j}^{2}}}\cdot g_{j}=-\frac{\text{learning_rate}}{|g_{j}|}\cdot g_{j}\\
\downarrow\\
\Delta x_{j}^{\left(t\right)}=-\text{learning_rate}
\end{gathered}
$$

I.e. the "ball" immediately starts moving downhill in a constant speed.

In rmsprop with momentum

Similarly, we get:
$$\Delta x_{j}^{\left(t\right)}=\text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(t-1\right)}-\text{learning_rate}$$
This case is a little more complicated, but we can see that:
$$\begin{gathered}\\
\Delta x_{j}^{\left(t\right)}=-\text{learning_rate}\\
\Delta x_{j}^{\left(t+1\right)}=-\text{learning_rate}\cdot(1+\text{momentum_decay_factor})
\end{gathered}
$$
So the "ball" starts accelerating downhill.
Given that the gradient stays constant, you can prove that if:
$$-\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}<\Delta x_{j}^{\left(k\right)}$$
then: $$-\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}<\Delta x_{j}^{\left(k+1\right)}<\Delta x_{j}^{\left(k\right)}$$
Therefore, we conclude that the step converges, i.e. $\Delta x_{j}^{\left(k\right)}\approx \Delta x_{j}^{\left(k-1\right)}$ for some $k>t$, and then:
$$\begin{gathered}\\
\Delta x_{j}^{\left(k\right)}\approx \text{momentum_decay_factor}\cdot\Delta x_{j}^{\left(k\right)}-\text{learning_rate}\\
\downarrow\\
\Delta x_{j}^{\left(k\right)}\approx -\frac{\text{learning_rate}}{1-\text{momentum_decay_factor}}
\end{gathered}
$$
Thus, the "ball" accelerates downhill and approaches a speed $\frac{1}{1-\text{momentum_decay_factor}}$ times as large as the constant speed of Adam's "ball". (E.g. for a typical $\text{momentum_decay_factor}=0.9$, it can approach $10\times$ speed!)

Changing Direction

Now, consider a scenario following the previous one:
After going down the slope (in the previous scenario) for quite some time (i.e. enough time for rmsprop with momentum to reach a nearly constant step size), suddenly a slope with an opposite and smaller constant gradient is reached.
What would happen when we use each of the methods?

This time I would just describe the results of my simulation of the scenario (my Python code is at the end of the answer).
Note that I have chosen for Adam's $\text{BCMA}\left(g_{j}\right)$ a decay factor equal to $\text{momentum_decay_factor}$. Choosing differently would have changed the following results:

Adam is slower to change its direction, and then much slower to get back to the minimum.
However, rmsprop with momentum reaches much further before it changes direction (when both use the same $\text{learning_rate}$).
Note that this further reach is because rmsprop with momentum first reaches the opposite slope with much higher speed than Adam. If both reached the opposite slope with the same speed (which would happen if Adam's $\text{learning_rate}$ were $\frac{1}{1-\text{momentum_decay_factor}}$ times as large as that of rmsprop with momentum), then Adam would reach further before changing direction.

Bias-Correction

What do we mean by a biased/bias-corrected moving average? (Or at least, what does the Adam paper mean by that?)

Generally speaking, a moving average is a weighted average of:

The moving average of all of the previous terms
The current term

Then what is the moving average in the first step?

A natural choice for a programmer would be to initialize the "moving average of all of the previous terms" to $0$. We say that in this case the moving average is biased towards $0$.
When you only have one term, by definition the average should be equal to that term.
Thus, we say that the moving average is bias-corrected in case the moving average in the first step is the first term (and the moving average works as usual for the rest of the steps).

So here is another difference: The moving averages in Adam are bias-corrected, while the moving average in rmsprop with momentum is biased towards $0$.

For more about the bias-correction in Adam, see section 3 in the paper and also this answer.

Simulation Python Code

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

###########################################
# script parameters
def f(x):
    if x > 0:
        return x
    else:
        return -0.1 * x

def f_grad(x):
    if x > 0:
        return 1
    else:
        return -0.1

METHOD_TO_LEARNING_RATE = {
    'Adam': 0.01,
    'GD': 0.00008,
    'rmsprop_with_Nesterov_momentum': 0.008,
    'rmsprop_with_momentum': 0.001,
    'rmsprop': 0.02,
    'momentum': 0.00008,
    'Nesterov': 0.008,
    'Adadelta': None,
    }
X0 = 2
METHOD = 'rmsprop'
METHOD = 'momentum'
METHOD = 'GD'
METHOD = 'rmsprop_with_Nesterov_momentum'
METHOD = 'Nesterov'
METHOD = 'Adadelta'
METHOD = 'rmsprop_with_momentum'
METHOD = 'Adam'
LEARNING_RATE = METHOD_TO_LEARNING_RATE[METHOD]

MOMENTUM_DECAY_FACTOR = 0.9
RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR = 0.9
ADADELTA_DECAY_FACTOR = 0.9
RMSPROP_EPSILON = 1e-10
ADADELTA_EPSILON = 1e-6
ADAM_EPSILON = 1e-10
ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR = 0.999
ADAM_GRADS_AVG_DECAY_FACTOR = 0.9

INTERVAL = 9e2
INTERVAL = 1
INTERVAL = 3e2
INTERVAL = 3e1
###########################################

def plot_func(axe, f):
    xs = np.arange(-X0 * 0.5, X0 * 1.05, abs(X0) / 100)
    vf = np.vectorize(f)
    ys = vf(xs)
    return axe.plot(xs, ys, color='grey')

def next_color(color, f):
    color[1] -= 0.01
    if color[1] < 0:
        color[1] = 1
    return color[:]

def update(frame):
    global k, x, prev_step, squared_grads_decaying_avg, \
           squared_prev_steps_decaying_avg, grads_decaying_avg

    if METHOD in ('momentum', 'Nesterov', 'rmsprop_with_momentum',
                  'rmsprop_with_Nesterov_momentum'):
        step_momentum_portion = MOMENTUM_DECAY_FACTOR * prev_step
    if METHOD in ('Nesterov', 'rmsprop_with_Nesterov_momentum'):
        gradient = f_grad(x + step_momentum_portion)
    else:
        gradient = f_grad(x)

    if METHOD == 'GD':
        step = -LEARNING_RATE * gradient
    elif METHOD in ('momentum', 'Nesterov'):
        step = step_momentum_portion - LEARNING_RATE * gradient
    elif METHOD in ('rmsprop', 'rmsprop_with_momentum',
                    'rmsprop_with_Nesterov_momentum'):
        squared_grads_decaying_avg = (
            RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - RMSPROP_SQUARED_GRADS_AVG_DECAY_FACTOR) * gradient ** 2)
        grads_rms = np.sqrt(squared_grads_decaying_avg + RMSPROP_EPSILON)
        if METHOD == 'rmsprop':
            step = -LEARNING_RATE / grads_rms * gradient
        else:
            assert(METHOD in ('rmsprop_with_momentum',
                              'rmsprop_with_Nesterov_momentum'))
            print(f'LEARNING_RATE / grads_rms * gradient: {LEARNING_RATE / grads_rms * gradient}')
            step = step_momentum_portion - LEARNING_RATE / grads_rms * gradient
    elif METHOD == 'Adadelta':
        gradient = f_grad(x)
        squared_grads_decaying_avg = (
            ADADELTA_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - ADADELTA_DECAY_FACTOR) * gradient ** 2)
        grads_rms = np.sqrt(squared_grads_decaying_avg + ADADELTA_EPSILON)
        squared_prev_steps_decaying_avg = (
            ADADELTA_DECAY_FACTOR * squared_prev_steps_decaying_avg +
            (1 - ADADELTA_DECAY_FACTOR) * prev_step ** 2)
        prev_steps_rms = np.sqrt(squared_prev_steps_decaying_avg + ADADELTA_EPSILON)
        step = - prev_steps_rms / grads_rms * gradient
    elif METHOD == 'Adam':
        squared_grads_decaying_avg = (
            ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR * squared_grads_decaying_avg +
            (1 - ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR) * gradient ** 2)
        unbiased_squared_grads_decaying_avg = (
            squared_grads_decaying_avg /
            (1 - ADAM_SQUARED_GRADS_AVG_DECAY_FACTOR ** (k + 1)))
        grads_decaying_avg = (
            ADAM_GRADS_AVG_DECAY_FACTOR * grads_decaying_avg +
            (1 - ADAM_GRADS_AVG_DECAY_FACTOR) * gradient)
        unbiased_grads_decaying_avg = (
            grads_decaying_avg /
            (1 - ADAM_GRADS_AVG_DECAY_FACTOR ** (k + 1)))
        step = - (LEARNING_RATE /
                  (np.sqrt(unbiased_squared_grads_decaying_avg) + ADAM_EPSILON) *
                  unbiased_grads_decaying_avg)

    x += step
    prev_step = step
    k += 1

    color = next_color(cur_color, f)

    print(f'k: {k}\n'
          f'x: {x}\n'
          f'step: {step}\n'
          f'gradient: {gradient}\n')

    k_x_marker, = k_and_x.plot(k, x, '.', color=color)
    x_y_marker, = x_and_y.plot(x, f(x), '.', color=color)

    return k_x_marker, x_y_marker

k = 0
x = X0
cur_color = [0, 1, 1]
prev_step = 0
squared_grads_decaying_avg = 0
squared_prev_steps_decaying_avg = 0
grads_decaying_avg = 0

fig, (k_and_x, x_and_y) = plt.subplots(1, 2, figsize=(9,5))
k_and_x.set_xlabel('k')
k_and_x.set_ylabel('x', rotation=0)
x_and_y.set_xlabel('x')
x_and_y.set_ylabel('y', rotation=0)
plot_func(x_and_y, f)
x_and_y.plot(x, f(x), '.', color=cur_color[:])
k_and_x.plot(k, x, '.', color=cur_color[:])
plt.tight_layout()

ani = FuncAnimation(fig, update, blit=False, repeat=False, interval=INTERVAL)
plt.show()


Summary statistics by category using Python I have a datset with Scores and Categories and I would like to calculate the summary statistics for each of these categories. The data look something like this:

Category    Score
AAAA        1
AAAA        3
AAAA        1
BBBB        1
BBBB        100
BBBB        159
CCCC        -10
CCCC        9

What I would then like would be something like this

Category    Count   Mean   Std   Min   25%   50%   75%   Max
AAAA
AAAA
AAAA
BBBB
BBBB
BBBB
CCCC
CCCC

I have been looking at using pandas with a combination of both .groupby() and .describe() like this

df.groupby('Category')['Score'].describe()

and this almost looks like what I want but when I come to view this as a Dataset, all of the stats are in the index. I would like the data to be in the form of a table so I can output it and create a visualization off of the back of it.

Any ideas?

Thanks


How would you visualize data that comes in the millions of records? Developing a application where sometimes when I make a query I get millions or 100 of thousands of records in the results.

My problem is , when I get these huge result sets, how do I visualize (in charts such as bar, radar, line graphs, etc.) this data?

Do I just take a sample of the data and visualize that? Do I just go and plot million records? What is the best way to go about this ?

Holoviews visual library can handle very large data http://holoviews.org/ http://holoviews.org/user_guide/Large_Data.html


What tools are available for programming language parsing for ML? I want to preform a machine learning task (e.g. supervised classification, clustering) on a corpus of programming language source code (lets say Python), and I'm looking for tools for parsing and constructing structures out of a document of Python (for example) code, similar in concept to NLP tokenization and higher level language processing, adjusted for programming languages.

While I'm able to find a lot of general NLP related material, it seems to me a better starting point would be previous research specifically focusing on programming languages.

I'm looking for tools, resources, academia articles and keywords to search for, basically any help is appreciated!

NLP stands for Natural Language Processing. Programming language source code are synthetic (or unnatural) languages. Thus, NLP tools are not useful for processing programming language source code.

Understanding programming language source code is done by the compiler or interpreter. Compilers and interpreters perform many functions, including lexical analysis, parsing, and semantic analysis.

For Python language, language analysis is relatively straightforward since the main Python implementation, named CPython, compiles Python programs into intermediate bytecode, which is executed by the virtual machine. The dis module supports the analysis of CPython bytecode by disassembling it.


NameError: name 'MyStreamListener' is not defined I'd import 'tweepy' package for accessing twitter data for analysis and visualization in Python. Afterwards I did setup oauth authentication and everything else but on the next step, which is STREAMING TWEETS for me am using MyStreamListener. This causing me a Name Error--

  NameError: name 'MyStreamListener' is not defined

Here's my code-

# API Authentication

# Importing TWEEPY Library
import tweepy

# Store OAuth authentication credentials in relevant variables
access_token = "provided"
access_token_secret = "provided"
consumer_key = "provided"
consumer_secret = "provided"

# Pass OAuth details to tweepy's OAuth handler
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Streaming Tweets

# Initialize Stream listener
l = MyStreamListener()

# Create you Stream object with authentication
stream = tweepy.Stream(auth, l)

# Filter Twitter Streams to capture data by the keywords:
stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])

What should I do now? Any thoughts?
Cheers!

You need to override StreamListener() class.

# API Authentication

# Importing TWEEPY Library
import tweepy

# Store OAuth authentication credentials in relevant variables
access_token = "provided"
access_token_secret = "provided"
consumer_key = "provided"
consumer_secret = "provided"

# Pass OAuth details to tweepy's OAuth handler
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Streaming Tweets
#override tweepy.StreamListener to add logic to on_status
class MyStreamListener(tweepy.StreamListener):
    def on_status(self, status):
        print(status.text)

# Initialize Stream listener
l = MyStreamListener()

# Create you Stream object with authentication
stream = tweepy.Stream(auth, l)

# Filter Twitter Streams to capture data by the keywords:
stream.filter(track = ['clinton', 'trump', 'sanders', 'cruz'])


Reflective padding as pure keras verision I am using keras with the plaidML backend and need to implement reflective padding.
With a tensorflow backend that is simply tf.pad with mode set to REFLECT.

How can implement that functionality with K. functions or plaidml tile functions ?
Or is there an implementation somewhere i could use ?

With K. function simply slicing, inverting and concatenating the value back together would probably be possible, but all my tries in that direction where a mess and didn't really worked out.

Here a version for reflective padding as pure K function, which should (but not tested) work with every backend:

def reflection_padding(inp, paddings):
    paddings = [(x, x) if isinstance(x, int) else x for x in paddings]
    ishape = inp.shape.dims
    ndims = inp.shape.ndims
    if len(ishape) != len(paddings):
        raise ValueError("Padding dims != input dims")
    last = inp
    _all_slice = slice(None, None, None)

    def _get_slices(ndims, axis, slice_):
        ret = [_all_slice for _ in range(ndims)]
        ret[axis] = slice_
        return tuple(ret)

    for axis, pads in ((i, x) for i, x in enumerate(paddings) if x[0]+x[1] != 0):
        pad_data = []
        if pads[0]:
            pre = last[_get_slices(ndims, axis, slice(pads[0], 0, -1))]
            pad_data.append(pre)
        pad_data.append(last)
        if pads[1]:
            post = last[_get_slices(ndims, axis, slice(-2, -pads[1]-2, -1))]
            pad_data.append(post)
        last = K.concatenate(pad_data, axis)
        ishape = last.shape.dims
    return last

# USAGE: reflection_padding(image_batch, [0, [2,2], [2,2], 0])

I am going to accept my own answer. If somebody has a better answer i'll gladly switch that over to theirs


r - rpart in text mining for classification of documents I am using rpart to classify the documents whether it belongs to class a or class b. I want to prove or provide reasons  on what basis the document was classified correctly or wrongly. How can i do that?

I displayed the summary of the model. But it was not so convincing.

Please help

rpart is a decision tree model and as such is very much interpretable.
You should visualize your decision tree. See the examples here, one of them is

Then for each given document you can follow the tree path and see

what were the features on the way responsible for the classification
what leaf your document arrived at and what is the percentage of correctly classified documents at these leaf.


Principal components analysis need standardization or normalization? Principal components analysis need standardization or normalization?
After some google, I get confused. pca need the scalar be same. So which should I use.
Which technique needs to do before PCA?
Does pca need standardization?  standardized values will always be zero, and the standard deviation will always be one.
Does pca need normalization? range zero to one
or both ?

I believe Normalization refers to scaling the variable in between 0 and 1. Standardization refers to making the empirical distribution $Y\sim N(0,1)$. Principal component analysis, and similar methods such as Ridge Regression and Partial Least Squares regression, require standardization before training, i.e. $y_{i}=\frac{y_i-\mu_y}{\sigma_{y}}$, reference: Elements of Statistical Learning, Ch. 3.4


What is the relationship between the accuracy and the loss in deep learning? I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:

First Model: Accuracy: 98.1% Loss: 0.1882
Second Model: Accuracy: 98.5% Loss: 0.0997
Third Model: Accuracy: 99.1% Loss: 0.2544

My questions are:

What is the relationship between the loss and accuracy values?
Why the loss of the third model is the higher even though the accuracy is higher?

There is no relationship between these two metrics.
Loss can be seen as a distance between the true values of the problem and the values predicted by the model. Greater the loss is, more huge is the errors you made on the data.
Accuracy can be seen as the number of error you made on the data.
That means:

a low accuracy and huge loss means you made huge errors on a lot of data
a low accuracy but low loss means you made little errors on a lot of data
a great accuracy with low loss means you made low errors on a few data (best case)
your situation: a great accuracy but a huge loss, means you made huge errors on a few data.

For you case, the third model can correctly predict more examples, but on those where it was wrong, it made more errors (the distance between true value and predicted values is more huge).
NOTE:
Don't forget that low or huge loss is a subjective metric, which depends on the problem and the data. It's a distance between the true value of the prediction, and the prediction made by the model. It depends also on the loss you use.
Think:

If your data are between 0 and 1, a loss of 0.5 is huge, but if your data are between 0 and 255, an error of 0.5 is low.
Maybe think of cancer detection, and probability of detecting a cancer. Maybe an error of 0.1 is huge for this problem, whereas an error f 0.1 for image classification is fine.


How to improve a CNN without changing the architecture? I'm currently using an autoencoder CNN that's built upon the VGG-16 architecture that was designed by someone else. I want to replicate their results using their dataset first but I'm finding that:

-Validation losses diverge from training losses fairly early on (I get to around 10 epochs and it already looks like it's overfitting)
-At its best, the validation losses aren't even close to being as low as training losses
-In general, the accuracy is still worse than reported in their paper.

I'm new to machine learning and want to know if there are hyperparameters I should try to change or what I can do to maybe tinker with it without changing its architecture?

Are you in fact using the same architecture as they are? If not that could potentially be the problem.

Otherwise, are you using the same trainings protocol as they, i.e. optimizer, learning rate, learning rate schedule, batch size, preprocessing, weight initialization, number of training epochs? Depending on the size of your model and the amount of training data, 10 epochs might not be enough to judge about your models performance.

Can you link the paper?


After one hot encoding, instead of columns, my number of rows are increasing This is the code:

housing_cat1 = X_train[:,[2,7,8,10,11,13,14,15,16,21,22,23,24,25,27,28,29,30,31,32,33,35,39,40,41,42,53,55,57,58,60,63,64,65,73,74,78,79]].reshape(1,-1)

from future_encoders import OneHotEncoder

cat_encoder = OneHotEncoder(sparse=False)                  # Set to True if sparse matrix is needed
housing_cat_1_1hot = cat_encoder.fit_transform(housing_cat)
housing_cat_1_1hot

get_dummies function of pandas can also be used for this. Just create a pandas dataframe with your data and subject it to this function. There is no need to list categorical variables:

X_train_df = pandas.DataFrame(data=X_train)
X_train_df_1hot = pandas.get_dummies(X_train_df)

You can specify prefix etc also for new columns is required:

pandas.get_dummies(data, prefix=None, prefix_sep='_',
     dummy_na=False, columns=None,
     sparse=False, drop_first=False,
     dtype=None)

Data as a list of lists is available with .values attribute which can be converted to a numpy array:

housing_cat_1hot = numpy.array(X_train_df_1hot.values)


How to encode a class with 24,000 categories? I'm currently working on a logistic regression model for genomics. One of the input fields I want to include as a covariate is genes. There are around 24,000 known genes. There are many features with this level of variability in computational biology and hundreds of thousands of samples are needed.

If I LabelEncoder() those 24K genes
and then OneHotEncoder() them ...

Is 24,000 columns going to make my keras training times unreasonable for a 2.2 GHz quad-core i7 CPU?

If so, is there a different approach to encoding that I can take with this?

Should I somehow try to dedicate a layer of my model to this feature?

Does this mean I need 24K input nodes?

Yes, using one-hot encoding on 24k features requires 24k input nodes. However this should not be a problem for Keras (or any other deep learning library). Natural language processing often uses one-hot encoding on words with a vocabulary size in the same ballpark.

If you are using a "deep" model, one of your hidden layers should take care of reducing the dimensionality of your data. A separate pre-processing step is usually not needed.

The training time should not be unreasonable.


Appropriate ML algorithm for allocating students to seats I'm new to machine learning and am trying to use it to solve a specific problem related to seating students in a classroom. I want to take a list of students and allocate them each to a seat, such that a certain output value is maximised.

An example:

Students (each one containing all the data from which a compatibility score can be calculated):
A
B
C
D
E
F

Seats:
1
2
3
4
5
6
7
8

Each pair of students has a predefined 'compatibility score' and each pair of seats has a geometric distance between them. After sorting students into seats, I would calculate a ratio between those two values, the average of which would be my outcome to be optimised.

A class of 25 students and 25 seats has something like 15,000,000,000,000,000,000,000,000 possible arrangements, so a brute force method is rather unfeasible. My ideal outcome here is that machine learning can develop an optimised algorithm for sorting them.

Any ideas for what kind of ML algorithm I want?

That problem is an optimization problem, selecting the selection of the best element (with regard to some criterion) from some set of available alternatives.

It falls in the general class of computational geometry. Specifically, a bin packing problem, items of different values must be packed into a finite number of containers.